freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(spiders): Make Robots.txt compliance turned off by default

Browse Source 
Scrapy is turning it off by default
This commit is contained in:
Karim shoair
2026-04-05 01:55:17 +02:00
parent 556a90f645
commit ea2dd7866b
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scrapling/spiders/spider.py
+1 -1
View File
@@ -73,7 +73,7 @@ class Spider(ABC):
allowed_domains: Set[str] = set() allowed_domains: Set[str] = set()
# Robots.txt compliance # Robots.txt compliance
robots_txt_obey: bool = True robots_txt_obey: bool = False
# Concurrency settings # Concurrency settings
concurrent_requests: int = 4 concurrent_requests: int = 4