freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,323 Commits 1 Branch 0 Tags
132f33c84611c10d8c2fb2082fb63266bf1cc809
Commit Graph
5 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Abdullah 132f33c846 test(spiders): add comprehensive test suite for robots.txt compliance 2026-04-03 15:08:34 +02:00
Abdullah 5c40c6a853 feat(spiders): integrate robots.txt compliance into the crawl engine 2026-04-03 15:08:33 +02:00
Abdullah 0bbe62fc7f feat(spiders): implement RobotsTxtManager with concurrent fetch deduplication 2026-04-03 15:08:33 +02:00
Abdullah 1d15349e07 feat(deps): add protego for robots.txt parsing and fix pyright type error in static.py 2026-04-03 15:08:33 +02:00
Abdullah 7ba333c3c9 perf(session): improve page management and speed up execution 
Close pages immediately after tasks finish to reduce resource usage and
improve overall performance. Update corresponding test files to reflect
the new page lifecycle management.
 2025-09-17 18:49:45 +03:00