Abdullah 5c40c6a853 feat(spiders): integrate robots.txt compliance into the crawl engine

2026-04-03 15:08:33 +02:00

..

core

docs: style adjustment

2026-03-30 03:53:56 +02:00

engines

feat(deps): add protego for robots.txt parsing and fix pyright type error in static.py

2026-04-03 15:08:33 +02:00

fetchers

fix: update code and docstrings to remove the old google referer logic

2026-03-09 00:25:19 +02:00

spiders

feat(spiders): integrate robots.txt compliance into the crawl engine

2026-04-03 15:08:33 +02:00

__init__.py

build: pump version up

2026-04-01 17:38:54 +02:00

cli.py

feat(cli): Add an option to make content safe/targets AI

2026-03-30 02:44:14 +02:00

parser.py

- Add pre-compiled XPath text selector

2026-03-07 17:11:39 -08:00

py.typed

Update and rename py.typeda to py.typed

2024-11-13 21:56:02 +13:00