Karim shoair 556a90f645 refactor(spider): prefetch robots.txt from start_urls only ...

Stop using allowed_domains for robots.txt prefetch since bare domain strings have no scheme info.
Domains discovered mid-crawl via requests still fetch robots.txt lazily.

2026-04-05 01:43:06 +02:00

..

core

docs: style adjustment

2026-03-30 03:53:56 +02:00

engines

fix(requests): handle missing type hint returned by curl-cffi

2026-04-03 17:26:52 +02:00

fetchers

fix: update code and docstrings to remove the old google referer logic

2026-03-09 00:25:19 +02:00

spiders

refactor(spider): prefetch robots.txt from start_urls only

2026-04-05 01:43:06 +02:00

__init__.py

build: pump version up

2026-04-01 17:38:54 +02:00

cli.py

feat(cli): Add an option to make content safe/targets AI

2026-03-30 02:44:14 +02:00

parser.py

- Add pre-compiled XPath text selector

2026-03-07 17:11:39 -08:00

py.typed

Update and rename py.typeda to py.typed

2024-11-13 21:56:02 +13:00