feat(spiders): enable robots.txt compliance by default

robots_txt_obey now defaults to True. Spiders must explicitly opt out
with robots_txt_obey = False rather than opt in, making ethical
crawling the default behaviour.

File: scrapling/spiders/spider.py

scrapling/spiders/spider.py

@@ -73,7 +73,7 @@ class Spider(ABC):
     allowed_domains: Set[str] = set()
 
     # Robots.txt compliance
-    robots_txt_obey: bool = False
+    robots_txt_obey: bool = True
 
     # Concurrency settings
     concurrent_requests: int = 4