Abdullah a134fdb8cc feat(spiders): enable robots.txt compliance by default ...

robots_txt_obey now defaults to True. Spiders must explicitly opt out
with robots_txt_obey = False rather than opt in, making ethical
crawling the default behaviour.

File: scrapling/spiders/spider.py

2026-04-04 03:10:17 +02:00

..

core

docs: style adjustment

2026-03-30 03:53:56 +02:00

engines

fix(requests): handle missing type hint returned by curl-cffi

2026-04-03 17:26:52 +02:00

fetchers

fix: update code and docstrings to remove the old google referer logic

2026-03-09 00:25:19 +02:00

spiders

feat(spiders): enable robots.txt compliance by default

2026-04-04 03:10:17 +02:00

__init__.py

build: pump version up

2026-04-01 17:38:54 +02:00

cli.py

feat(cli): Add an option to make content safe/targets AI

2026-03-30 02:44:14 +02:00

parser.py

- Add pre-compiled XPath text selector

2026-03-07 17:11:39 -08:00

py.typed

Update and rename py.typeda to py.typed

2024-11-13 21:56:02 +13:00