Stop using allowed_domains for robots.txt prefetch since bare domain strings have no scheme info.
Domains discovered mid-crawl via requests still fetch robots.txt lazily.
robots_txt_obey now defaults to True. Spiders must explicitly opt out
with robots_txt_obey = False rather than opt in, making ethical
crawling the default behaviour.
File: scrapling/spiders/spider.py
Previously robots.txt was fetched lazily on the first request per
domain, causing early concurrent requests to each stall waiting for
the same network fetch. The cache is now warmed before the crawl loop
starts, making all subsequent robots.txt lookups a local read.
- RobotsTxtManager gains a prefetch(urls, sid) method that fetches all domains concurrently via a task group
- CrawlerEngine._prefetch_robots_txt() is called after on_start():
uses allowed_domains if configured, otherwise falls back to unique
domains extracted from start_urls
- Mid-crawl domain discovery (not covered by prefetch) still fetches
lazily; two concurrent callbacks on the same new domain can each
trigger a fetch — accepted tradeoff, documented in _get_domain_delay
Files: scrapling/spiders/robotstxt.py, scrapling/spiders/engine.py, tests/spiders/test_engine.py
robots.txt is a domain-level document and does not vary by session.
Keying the cache by (domain, sid) was both wasteful and incorrect —
it caused redundant fetches when the same domain was accessed by different sessions.
- Cache is now keyed by domain string only; all sessions share one entry
- Removed asyncio.Event inflight-deduplication mechanism (superseded by the prefetch approach added in the next commit)
- clear_cache() loses the `sid` parameter (breaking change); clearing a domain now evicts the single shared entry for all sessions
- Updated tests to reflect shared-cache semantics
Files: scrapling/spiders/robotstxt.py, tests/spiders/test_robotstxt.py
protego is only used by the spider framework for robots.txt compliance.
Moving it from core dependencies to the optional 'fetchers' group reduces
the dependency footprint for users who don't need the spider framework.
for pyproject.toml file
The stub shadows the real implementation, and proxy rotation always hits NotImplementedError.
Possible fix for #215
Co-Authored-By: Yuval Dinodia <102706514+yetval@users.noreply.github.com>
- `get()` now delegates to `bulk_get([url])[0]` (was a separate sync implementation)
- `fetch()` now delegates to `bulk_fetch([url])[0]` (eliminated duplicate fetcher call)
- `stealthy_fetch()` now delegates to `bulk_stealthy_fetch([url])[0]` (same)
- Replaced 6x repeated `_content_translator(Convertor._extract_content(...), page)` with a single `_translate_response()` helper
- Removed unused imports (`Fetcher`, `DynamicFetcher`, `StealthyFetcher`, `Generator`)
Shortened the code by 210 lines. Also, removed docstrings because they are not needed for CLI commands (more maintenance burden).
- `_common_http_options`: shared decorator for 10 Click options used by get/post/put/delete (was repeated 4x)
- `_common_browser_options`: shared decorator for 11 Click options used by fetch/stealthy_fetch (was repeated 2x)
- `_data_options`: shared decorator for `--data`/`--json` options used by post/put
- `__http_command()`: shared implementation body for all HTTP commands (was 4 separate `from scrapling.fetchers import Fetcher` + `__Request_and_Save` blocks)
- `__build_browser_kwargs()`: shared kwargs builder for fetch/stealthy_fetch (was duplicated)
Karim shoair