diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e972abc..3967c1d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -63,6 +63,37 @@ Please follow these coding conventions as we do when writing code for Scrapling: > Please don’t put your name in the code you contribute; git provides enough metadata to identify the author of the code. ## Development + +### Getting started + +1. Fork the repository and clone your fork: + ```bash + git clone https://github.com//Scrapling.git + cd Scrapling + git checkout dev + ``` + +2. Create a virtual environment and install dependencies: + ```bash + python -m venv .venv + source .venv/bin/activate # On Windows: .venv\Scripts\activate + pip install -e ".[all]" + pip install -r tests/requirements.txt + ``` + +3. Install browser dependencies: + ```bash + scrapling install + ``` + +4. Set up pre-commit hooks: + ```bash + pip install pre-commit + pre-commit install + ``` + +### Tips + Setting the scrapling logging level to `debug` makes it easier to know what's happening in the background. ```python import logging @@ -73,15 +104,6 @@ Bonus: You can install the beta of the upcoming update from the dev branch as fo pip3 install git+https://github.com/D4Vinci/Scrapling.git@dev ``` -## Building Documentation -Documentation is built using [Zensical](https://zensical.org/). You can build it locally using the following commands: -```bash -pip install zensical -pip install -r docs/requirements.txt -zensical build --clean # Build the static site -zensical serve # Local preview -``` - ## Tests Scrapling includes a comprehensive test suite that can be executed with pytest. However, first, you need to install all libraries and `pytest-plugins` listed in `tests/requirements.txt`. Then, running the tests will result in an output like this: ```bash @@ -93,16 +115,34 @@ Scrapling includes a comprehensive test suite that can be executed with pytest. configfile: pytest.ini plugins: asyncio-1.2.0, anyio-4.11.0, xdist-3.8.0, httpbin-2.1.0, cov-7.0.0 asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function - 10 workers [515 items] - scheduling tests via LoadScheduling - + 10 workers [515 items] + scheduling tests via LoadScheduling + ...... - + =============================== 271 passed in 52.68s ============================== ``` -Hence, we used `-n auto` in the command above to run tests in threads to increase speed. +Here, `-n auto` runs tests in parallel across multiple processes to increase speed. + +**Note:** You may need to run browser tests sequentially (`DynamicFetcher`/`StealthyFetcher`) to avoid conflicts. To run non-browser tests in parallel and browser tests separately: +```bash +# Non-browser tests (parallel) +pytest tests/ -k "not (DynamicFetcher or StealthyFetcher)" -n auto + +# Browser tests (sequential) +pytest tests/ -k "DynamicFetcher or StealthyFetcher" +``` Bonus: You can also see the test coverage with the `pytest` plugin below ```bash pytest --cov=scrapling tests/ ``` + +## Building Documentation +Documentation is built using [Zensical](https://zensical.org/). You can build it locally using the following commands: +```bash +pip install zensical +pip install -r docs/requirements.txt +zensical build --clean # Build the static site +zensical serve # Local preview +```