freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Updating the contribution guide

Browse Source
This commit is contained in:
Karim shoair
2025-11-17 01:00:10 +02:00
parent 64aa29cd40
commit 763363457f
4 changed files with 102 additions and 139 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/PULL_REQUEST_TEMPLATE.md
+5 -7
View File
@@ -5,10 +5,8 @@
## Proposed change
<!--
Describe the big picture of your changes here to communicate to the
maintainers why we should accept this pull request. If it fixes a bug
or resolves a feature request, be sure to link to that issue in the
additional information section.
Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request.
If it fixes a bug or resolves a feature request, be sure to link to that issue in the additional information section.
-->
@@ -34,12 +32,12 @@
### Additional information
<!--
Details are important, and help maintainers processing your PR.
Details are important and help maintainers processing your PR.
Please be sure to fill out additional details, if applicable.
-->
- This PR fixes or closes issue: fixes #
- This PR is related to issue:
- This PR fixes or closes an issue: fixes #
- This PR is related to an issue: #
- Link to documentation pull request: **
### Checklist:
CONTRIBUTING.md
+96 -29
View File
@@ -1,39 +1,106 @@
# Contributing to Scrapling
Everybody is invited and welcome to contribute to Scrapling. Smaller changes have a better chance to get included in a timely manner. Adding unit tests for new features or test cases for bugs you've fixed help us to ensure that the Pull Request (PR) is fine.
There is a lot to do...
- If you are not a developer perhaps you would like to help with the [documentation](https://github.com/D4Vinci/Scrapling/tree/docs)?
- If you are a developer, most of the features I'm planning to add in the future are moved to [roadmap file](https://github.com/D4Vinci/Scrapling/blob/main/ROADMAP.md) so consider reading it.
Thank you for your interest in contributing to Scrapling!
Scrapling includes a comprehensive test suite which can be executed with pytest:
```bash
$ pytest
=============================== test session starts ===============================
platform darwin -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0
rootdir: /<some_where>/Scrapling
configfile: pytest.ini
plugins: cov-5.0.0, anyio-4.6.0
collected 16 items
Everybody is invited and welcome to contribute to Scrapling.
tests/test_parser_functions.py ................ [100%]
Minor changes have a better chance of being included promptly. Adding unit tests for new features or test cases for bugs you've fixed helps us ensure that the Pull Request (PR) is acceptable.
=============================== 16 passed in 0.22s ================================
```
Also, consider setting the scrapling logging level to `debug` so it's easier to know what's happening in the background.
There are many ways to contribute to Scrapling. Here are some of them:
- Report bugs and request features using the [GitHub issues](https://github.com/D4Vinci/Scrapling/issues). Please follow the issue template to help us resolve your issue quickly.
- Blog about Scrapling. Tell the world how you’re using Scrapling. This will help newcomers with more examples and increase the Scrapling project's visibility.
- Join the [Discord community](https://discord.gg/EMgGbDceNQ) and share your ideas on how to improve Scrapling. We’re always open to suggestions.
- If you are not a developer, perhaps you would like to help with translating the [documentation](https://github.com/D4Vinci/Scrapling/tree/docs)?
## Finding work
If you have decided to make a contribution to Scrapling, but you do not know what to contribute, here are some ways to find pending work:
- Check out the [contribution](https://github.com/D4Vinci/Scrapling/contribute) GitHub page, which lists open issues tagged as good first issue. These issues provide a good starting point.
- There are also the [help wanted](https://github.com/D4Vinci/Scrapling/issues?q=is%3Aissue%20label%3A%22help%20wanted%22%20state%3Aopen) issues, but know that some may require familiarity with the Scrapling code base first. You can also target any other issue, provided it is not tagged as `invalid`, `wontfix`, or similar tags.
- If you enjoy writing automated tests, you can work on increasing our test coverage. Currently, the test coverage is around 90–92%.
- Join the [Discord community](https://discord.gg/EMgGbDceNQ) and ask questions in the `#help` channel.
## Coding style
Please follow these coding conventions as we do when writing code for Scrapling:
- We use [pre-commit](https://pre-commit.com/) to automatically address simple code issues before every commit, so please install it and run `pre-commit install` to set it up. This will install hooks to run [ruff](https://docs.astral.sh/ruff/), [bandit](https://github.com/PyCQA/bandit), and [vermin](https://github.com/netromdk/vermin) on every commit. We are currently using a workflow to automatically run these tools on every PR, so if your code doesn't pass these checks, the PR will be rejected.
- We use type hints for better code clarity and [pyright](https://github.com/microsoft/pyright) for static type checking, which depends on the type hints, of course.
- We use the conventional commit messages format as [here](https://gist.github.com/qoomon/5dfcdf8eec66a051ecd85625518cfd13#types), so for example, we use the following prefixes for commit messages:
| Prefix | When to use it |
|-------------|--------------------------|
| `feat:` | New feature added |
| `fix:` | Bug fix |
| `docs:` | Documentation change/add |
| `test:` | Tests |
| `refactor:` | Code refactoring |
| `chore:` | Maintenance tasks |
Then include the details of the change in the body/description of the commit message.
Example:
```
feat: add `adaptive` for similar elements
- Added find_similar() method
- Implemented pattern matching
- Added tests and documentation
```
> Please don’t put your name in the code you contribute; git provides enough metadata to identify the author of the code.
## Development
Setting the scrapling logging level to `debug` makes it easier to know what's happening in the background.
```python
>>> import logging
>>> logging.getLogger("scrapling").setLevel(logging.DEBUG)
import logging
logging.getLogger("scrapling").setLevel(logging.DEBUG)
```
### The process is straight-forward.
- Read [How to get faster PR reviews](https://github.com/kubernetes/community/blob/master/contributors/guide/pull-requests.md#best-practices-for-faster-reviews) by Kubernetes (but skip step 0 and 1)
- Fork Scrapling [git repository](https://github.com/D4Vinci/Scrapling).
- Make your changes.
- Ensure tests work.
- Create a Pull Request against the [**dev**](https://github.com/D4Vinci/Scrapling/tree/dev) branch of Scrapling.
### Installing the latest changes from the dev branch
Bonus: You can install the beta of the upcoming update from the dev branch as follows
```commandline
pip3 install git+https://github.com/D4Vinci/Scrapling.git@dev
```
## Building Documentation
Documentation is built using [MkDocs](https://www.mkdocs.org/). You can build it locally using the following commands:
```bash
pip install mkdocs-material
mkdocs serve # Local preview
mkdocs build # Build the static site
```
## Tests
Scrapling includes a comprehensive test suite that can be executed with pytest. However, first, you need to install all libraries and `pytest-plugins` listed in `tests/requirements.txt`. Then, running the tests will result in an output like this:
```bash
$ pytest tests -n auto
=============================== test session starts ===============================
platform darwin -- Python 3.13.8, pytest-8.4.2, pluggy-1.6.0 -- /Users/<redacted>/.venv/bin/python3.13
cachedir: .pytest_cache
rootdir: /Users/<redacted>/scrapling
configfile: pytest.ini
plugins: asyncio-1.2.0, anyio-4.11.0, xdist-3.8.0, httpbin-2.1.0, cov-7.0.0
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
10 workers [271 items]
scheduling tests via LoadScheduling
...<shortened>...
=============================== 271 passed in 52.68s ==============================
```
Hence, we used `-n auto` in the command above to run tests in threads to increase speed.
Bonus: You can also see the test coverage with the `pytest` plugin below
```bash
pytest --cov=scrapling tests/
```
## Making a Pull Request
To ensure that your PR gets accepted, please make sure that your PR is based on the latest changes from the dev branch and that it satisfies the following requirements:
- The PR should be made against the [**dev**](https://github.com/D4Vinci/Scrapling/tree/dev) branch of Scrapling. Any PR made against the main branch will be rejected.
- The code should be passing all available tests. We are using tox with GitHub's CI to run the current tests on all supported Python versions with every commit.
- The code should be passing all code quality checks we mentioned above. We are using GitHub's CI to enforce the code style checks performed by pre-commit. If you were using the pre-commit hooks we discussed above, you should not see any issues when committing your changes.
- Make your changes, keep the code clean with an explanation of any part that might be vague, and remember to create a separate virtual environment for this project.
- If you are adding a new feature, please add tests for it.
- If you are fixing a bug, please add code with the PR that reproduces the bug.
docs/contributing.md
-102
View File
@@ -1,102 +0,0 @@
Thank you for your interest in contributing to Scrapling!
Everybody is invited and welcome to contribute to Scrapling.
Smaller changes have a better chance of getting included in a timely manner. Adding unit tests for new features or test cases for bugs you've fixed helps us to ensure that the Pull Request (PR) is acceptable.
There is a lot to do...
- If you are not a developer, you can help us improve the documentation.
- If you are a developer, most of the features I'm planning to add in the future are moved to [roadmap file](https://github.com/D4Vinci/Scrapling/blob/main/ROADMAP.md), so consider reading it.
## Running tests
Scrapling includes a comprehensive test suite that can be executed with pytest, but first, you need to install all libraries and `pytest-plugins` inside `tests/requirements.txt`. Then, running the tests will result in an output like this:
```bash
$ pytest tests
=============================== test session starts ===============================
platform darwin -- Python 3.12.8, pytest-8.3.3, pluggy-1.5.0 -- /Users/<redacted>/.venv/bin/python3.12
cachedir: .pytest_cache
rootdir: /Users/<redacted>/scrapling
configfile: pytest.ini
plugins: cov-5.0.0, asyncio-0.25.0, base-url-2.1.0, httpbin-2.1.0, playwright-0.5.2, anyio-4.6.2.post1, xdist-3.6.1, typeguard-4.3.0
asyncio: mode=Mode.AUTO, asyncio_default_fixture_loop_scope=function
collected 83 items
...<shortened>...
=============================== 83 passed in 157.52s (0:02:37) =====================
```
Hence, you can add `-n auto` to the command above to run tests in threads to increase speed.
Bonus: You can also see the test coverage with the pytest plugin below
```bash
pytest --cov=scrapling tests/
```
## Installing the latest unstable version from the dev branch
```bash
pip3 install git+https://github.com/D4Vinci/Scrapling.git@dev
```
## Development
Setting the scrapling logging level to `debug` makes it easier to know what's happening in the background.
```python
>>> import logging
>>> logging.getLogger("scrapling").setLevel(logging.DEBUG)
```
### Code Style
We use:
1. Type hints for better code clarity
2. Flake8, bandit, isort, and other hooks through `pre-commit`. <br/>Please install the hooks before committing with:
```bash
pip install pre-commit
pre-commit install
```
It will run automatically on the code you push with each commit.
3. Conventional commit messages format. We use the below format for commit messages
| Prefix | When to use it |
|-------------|--------------------------|
| `feat:` | New feature added |
| `fix:` | Bug fix |
| `docs:` | Documentation change/add |
| `test:` | Tests |
| `refactor:` | Code refactoring |
| `chore:` | Maintenance tasks |
Example:
```
feat: add `adaptive` for similar elements
- Added find_similar() method
- Implemented pattern matching
- Added tests and documentation
```
### Push changes to the library
Then, the process is straightforward.
- Read [How to get faster PR reviews](https://github.com/kubernetes/community/blob/master/contributors/guide/pull-requests.md#best-practices-for-faster-reviews) by Kubernetes (but skip step 0 and 1)
- Fork Scrapling [Git repository](https://github.com/D4Vinci/Scrapling.git).
- Make your changes, and don't forget to create a separate virtual environment for this project.
- Ensure all tests are passing.
- Create a Pull Request against the [**dev**](https://github.com/D4Vinci/Scrapling/tree/dev) branch of Scrapling.
A bonus: if you have more than one version of Python installed, you can use tox to run tests on each version with:
```bash
pip install tox
tox
```
> Note: All tests are automatically run with each push on Github on all supported Python versions using tox, so ensure all tests pass, or your PR will not be accepted.
## Building Documentation
```bash
pip install mkdocs-material
mkdocs serve # Local preview
mkdocs build # Build the static site
```
mkdocs.yml
+1 -1
View File
@@ -79,7 +79,7 @@ nav:
- Writing your retrieval system: development/adaptive_storage_system.md
- Using Scrapling's custom types: development/scrapling_custom_types.md
- Support and Advertisement: donate.md
- Contributing: contributing.md
- Contributing: 'https://github.com/D4Vinci/Scrapling/blob/main/CONTRIBUTING.md'
- Changelog: 'https://github.com/D4Vinci/Scrapling/releases'
markdown_extensions: