Abdullah a134fdb8cc feat(spiders): enable robots.txt compliance by default ...

robots_txt_obey now defaults to True. Spiders must explicitly opt out
with robots_txt_obey = False rather than opt in, making ethical
crawling the default behaviour.

File: scrapling/spiders/spider.py

2026-04-04 03:10:17 +02:00

..

__init__.py

style: remove unwanted import from spiders main init file

2026-01-12 01:21:26 +02:00

checkpoint.py

fix(checkpoint): correct error message

2026-02-15 04:36:43 +02:00

engine.py

feat(spiders): pre-warm robots.txt cache before crawl loop starts

2026-04-04 03:00:15 +02:00

request.py

fix(spiders): corrections and removal of dead code

2026-02-15 04:25:32 +02:00

result.py

feat(spiders): integrate robots.txt compliance into the crawl engine

2026-04-03 15:08:33 +02:00

robotstxt.py

refactor(spiders): simplify robots.txt cache to domain-only key

2026-04-04 03:00:15 +02:00

scheduler.py

feat(spiders/requests): Use a reliable method for fingerprint generation

2026-01-20 20:21:42 +02:00

session.py

fix: preserve HTTP method across retries in spider session

2026-03-17 00:53:52 +09:00

spider.py

feat(spiders): enable robots.txt compliance by default

2026-04-04 03:10:17 +02:00