Abdullah
a2bf00c995
Merge branch 'D4Vinci:dev' into dev
2026-04-03 17:38:28 +02:00
Abdullah
132f33c846
test(spiders): add comprehensive test suite for robots.txt compliance
2026-04-03 15:08:34 +02:00
Abdullah
5c40c6a853
feat(spiders): integrate robots.txt compliance into the crawl engine
2026-04-03 15:08:33 +02:00
Abdullah
0bbe62fc7f
feat(spiders): implement RobotsTxtManager with concurrent fetch deduplication
2026-04-03 15:08:33 +02:00
Abdullah
1d15349e07
feat(deps): add protego for robots.txt parsing and fix pyright type error in static.py
2026-04-03 15:08:33 +02:00
Abdullah
7ba333c3c9
perf(session): improve page management and speed up execution
...
Close pages immediately after tasks finish to reduce resource usage and
improve overall performance. Update corresponding test files to reflect
the new page lifecycle management.
2025-09-17 18:49:45 +03:00
Abdullah