docs: make the contribution rules clearer
This commit is contained in:
+50
-10
@@ -63,6 +63,37 @@ Please follow these coding conventions as we do when writing code for Scrapling:
|
|||||||
> Please don’t put your name in the code you contribute; git provides enough metadata to identify the author of the code.
|
> Please don’t put your name in the code you contribute; git provides enough metadata to identify the author of the code.
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
|
### Getting started
|
||||||
|
|
||||||
|
1. Fork the repository and clone your fork:
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/<your-username>/Scrapling.git
|
||||||
|
cd Scrapling
|
||||||
|
git checkout dev
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Create a virtual environment and install dependencies:
|
||||||
|
```bash
|
||||||
|
python -m venv .venv
|
||||||
|
source .venv/bin/activate # On Windows: .venv\Scripts\activate
|
||||||
|
pip install -e ".[all]"
|
||||||
|
pip install -r tests/requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Install browser dependencies:
|
||||||
|
```bash
|
||||||
|
scrapling install
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Set up pre-commit hooks:
|
||||||
|
```bash
|
||||||
|
pip install pre-commit
|
||||||
|
pre-commit install
|
||||||
|
```
|
||||||
|
|
||||||
|
### Tips
|
||||||
|
|
||||||
Setting the scrapling logging level to `debug` makes it easier to know what's happening in the background.
|
Setting the scrapling logging level to `debug` makes it easier to know what's happening in the background.
|
||||||
```python
|
```python
|
||||||
import logging
|
import logging
|
||||||
@@ -73,15 +104,6 @@ Bonus: You can install the beta of the upcoming update from the dev branch as fo
|
|||||||
pip3 install git+https://github.com/D4Vinci/Scrapling.git@dev
|
pip3 install git+https://github.com/D4Vinci/Scrapling.git@dev
|
||||||
```
|
```
|
||||||
|
|
||||||
## Building Documentation
|
|
||||||
Documentation is built using [Zensical](https://zensical.org/). You can build it locally using the following commands:
|
|
||||||
```bash
|
|
||||||
pip install zensical
|
|
||||||
pip install -r docs/requirements.txt
|
|
||||||
zensical build --clean # Build the static site
|
|
||||||
zensical serve # Local preview
|
|
||||||
```
|
|
||||||
|
|
||||||
## Tests
|
## Tests
|
||||||
Scrapling includes a comprehensive test suite that can be executed with pytest. However, first, you need to install all libraries and `pytest-plugins` listed in `tests/requirements.txt`. Then, running the tests will result in an output like this:
|
Scrapling includes a comprehensive test suite that can be executed with pytest. However, first, you need to install all libraries and `pytest-plugins` listed in `tests/requirements.txt`. Then, running the tests will result in an output like this:
|
||||||
```bash
|
```bash
|
||||||
@@ -100,9 +122,27 @@ Scrapling includes a comprehensive test suite that can be executed with pytest.
|
|||||||
|
|
||||||
=============================== 271 passed in 52.68s ==============================
|
=============================== 271 passed in 52.68s ==============================
|
||||||
```
|
```
|
||||||
Hence, we used `-n auto` in the command above to run tests in threads to increase speed.
|
Here, `-n auto` runs tests in parallel across multiple processes to increase speed.
|
||||||
|
|
||||||
|
**Note:** You may need to run browser tests sequentially (`DynamicFetcher`/`StealthyFetcher`) to avoid conflicts. To run non-browser tests in parallel and browser tests separately:
|
||||||
|
```bash
|
||||||
|
# Non-browser tests (parallel)
|
||||||
|
pytest tests/ -k "not (DynamicFetcher or StealthyFetcher)" -n auto
|
||||||
|
|
||||||
|
# Browser tests (sequential)
|
||||||
|
pytest tests/ -k "DynamicFetcher or StealthyFetcher"
|
||||||
|
```
|
||||||
|
|
||||||
Bonus: You can also see the test coverage with the `pytest` plugin below
|
Bonus: You can also see the test coverage with the `pytest` plugin below
|
||||||
```bash
|
```bash
|
||||||
pytest --cov=scrapling tests/
|
pytest --cov=scrapling tests/
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Building Documentation
|
||||||
|
Documentation is built using [Zensical](https://zensical.org/). You can build it locally using the following commands:
|
||||||
|
```bash
|
||||||
|
pip install zensical
|
||||||
|
pip install -r docs/requirements.txt
|
||||||
|
zensical build --clean # Build the static site
|
||||||
|
zensical serve # Local preview
|
||||||
|
```
|
||||||
|
|||||||
Reference in New Issue
Block a user