Stop using allowed_domains for robots.txt prefetch since bare domain strings have no scheme info.
Domains discovered mid-crawl via requests still fetch robots.txt lazily.
robots_txt_obey now defaults to True. Spiders must explicitly opt out
with robots_txt_obey = False rather than opt in, making ethical
crawling the default behaviour.
File: scrapling/spiders/spider.py
Previously robots.txt was fetched lazily on the first request per
domain, causing early concurrent requests to each stall waiting for
the same network fetch. The cache is now warmed before the crawl loop
starts, making all subsequent robots.txt lookups a local read.
- RobotsTxtManager gains a prefetch(urls, sid) method that fetches all domains concurrently via a task group
- CrawlerEngine._prefetch_robots_txt() is called after on_start():
uses allowed_domains if configured, otherwise falls back to unique
domains extracted from start_urls
- Mid-crawl domain discovery (not covered by prefetch) still fetches
lazily; two concurrent callbacks on the same new domain can each
trigger a fetch — accepted tradeoff, documented in _get_domain_delay
Files: scrapling/spiders/robotstxt.py, scrapling/spiders/engine.py, tests/spiders/test_engine.py
robots.txt is a domain-level document and does not vary by session.
Keying the cache by (domain, sid) was both wasteful and incorrect —
it caused redundant fetches when the same domain was accessed by different sessions.
- Cache is now keyed by domain string only; all sessions share one entry
- Removed asyncio.Event inflight-deduplication mechanism (superseded by the prefetch approach added in the next commit)
- clear_cache() loses the `sid` parameter (breaking change); clearing a domain now evicts the single shared entry for all sessions
- Updated tests to reflect shared-cache semantics
Files: scrapling/spiders/robotstxt.py, tests/spiders/test_robotstxt.py
protego is only used by the spider framework for robots.txt compliance.
Moving it from core dependencies to the optional 'fetchers' group reduces
the dependency footprint for users who don't need the spider framework.
for pyproject.toml file
The stub shadows the real implementation, and proxy rotation always hits NotImplementedError.
Possible fix for #215
Co-Authored-By: Yuval Dinodia <102706514+yetval@users.noreply.github.com>
Karim shoair