docs: improving the code copy-paste experience and use less tokens for the agent skill

2026-04-22 16:52:59 +02:00
parent b626b4d585
commit 9af644ef84
17 changed files with 345 additions and 363 deletions
@@ -27,23 +27,23 @@ The following table compares them and can be quickly used for guidance.
 ## Parser configuration in all fetchers
 All fetchers share the same import method, as you will see in the upcoming pages
 ```python
->>> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
+from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
 ```
 Then you use it right away without initializing like this, and it will use the default parser settings:
 ```python
->>> page = StealthyFetcher.fetch('https://example.com') 
+page = StealthyFetcher.fetch('https://example.com') 
 ```
 If you want to configure the parser ([Selector class](parsing/main_classes.md#selector)) that will be used on the response before returning it for you, then do this first:
 ```python
->>> from scrapling.fetchers import Fetcher
->>> Fetcher.configure(adaptive=True, keep_comments=False, keep_cdata=False)  # and the rest
+from scrapling.fetchers import Fetcher
+Fetcher.configure(adaptive=True, keep_comments=False, keep_cdata=False)  # and the rest
 ```
 or
 ```python
->>> from scrapling.fetchers import Fetcher
->>> Fetcher.adaptive=True
->>> Fetcher.keep_comments=False
->>> Fetcher.keep_cdata=False  # and the rest
+from scrapling.fetchers import Fetcher
+Fetcher.adaptive=True
+Fetcher.keep_comments=False
+Fetcher.keep_cdata=False  # and the rest
 ```
 Then, continue your code as usual.

@@ -59,19 +59,19 @@ If your use case requires a different configuration for each request/fetch, you
 ## Response Object
 The `Response` object is the same as the [Selector](parsing/main_classes.md#selector) class, but it has additional details about the response, like response headers, status, cookies, etc., as shown below:
 ```python
->>> from scrapling.fetchers import Fetcher
->>> page = Fetcher.get('https://example.com')
+from scrapling.fetchers import Fetcher
+page = Fetcher.get('https://example.com')

->>> page.status          # HTTP status code
->>> page.reason          # Status message
->>> page.cookies         # Response cookies as a dictionary
->>> page.headers         # Response headers
->>> page.request_headers # Request headers
->>> page.history         # Response history of redirections, if any
->>> page.body            # Raw response body as bytes
->>> page.encoding        # Response encoding
->>> page.meta            # Response metadata dictionary (e.g., proxy used). Mainly helpful with the spiders system.
->>> page.captured_xhr    # List of captured XHR/fetch responses (when capture_xhr is enabled on a browser session)
+page.status          # HTTP status code
+page.reason          # Status message
+page.cookies         # Response cookies as a dictionary
+page.headers         # Response headers
+page.request_headers # Request headers
+page.history         # Response history of redirections, if any
+page.body            # Raw response body as bytes
+page.encoding        # Response encoding
+page.meta            # Response metadata dictionary (e.g., proxy used). Mainly helpful with the spiders system.
+page.captured_xhr    # List of captured XHR/fetch responses (when capture_xhr is enabled on a browser session)
 ```
 All fetchers return the `Response` object.

@@ -8,7 +8,7 @@ As we will explain later, to automate the page, you need some knowledge of [Play
 You have one primary way to import this Fetcher, which is the same for all fetchers.

 ```python
->>> from scrapling.fetchers import DynamicFetcher
+from scrapling.fetchers import DynamicFetcher
 ```
 Check out how to configure the parsing options [here](choosing.md#parser-configuration-in-all-fetchers)

@@ -6,7 +6,7 @@ The `Fetcher` class provides rapid and lightweight HTTP requests using the high-
 Import the Fetcher (same import pattern for all fetchers):

 ```python
->>> from scrapling.fetchers import Fetcher
+from scrapling.fetchers import Fetcher
 ```
 Check out how to configure the parsing options [here](choosing.md#parser-configuration-in-all-fetchers)

@@ -47,41 +47,41 @@ Examples are the best way to explain this:
 > Hence: `OPTIONS` and `HEAD` methods are not supported.
 #### GET
 ```python
->>> from scrapling.fetchers import Fetcher
->>> # Basic GET
->>> page = Fetcher.get('https://example.com')
->>> page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
->>> page = Fetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')
->>> # With parameters
->>> page = Fetcher.get('https://example.com/search', params={'q': 'query'})
->>>
->>> # With headers
->>> page = Fetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
->>> # Basic HTTP authentication
->>> page = Fetcher.get("https://example.com", auth=("my_user", "password123"))
->>> # Browser impersonation
->>> page = Fetcher.get('https://example.com', impersonate='chrome')
->>> # HTTP/3 support
->>> page = Fetcher.get('https://example.com', http3=True)
+from scrapling.fetchers import Fetcher
+# Basic GET
+page = Fetcher.get('https://example.com')
+page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
+page = Fetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')
+# With parameters
+page = Fetcher.get('https://example.com/search', params={'q': 'query'})
+
+# With headers
+page = Fetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
+# Basic HTTP authentication
+page = Fetcher.get("https://example.com", auth=("my_user", "password123"))
+# Browser impersonation
+page = Fetcher.get('https://example.com', impersonate='chrome')
+# HTTP/3 support
+page = Fetcher.get('https://example.com', http3=True)
 ```
 And for asynchronous requests, it's a small adjustment 
 ```python
->>> from scrapling.fetchers import AsyncFetcher
->>> # Basic GET
->>> page = await AsyncFetcher.get('https://example.com')
->>> page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
->>> page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')
->>> # With parameters
->>> page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})
->>>
->>> # With headers
->>> page = await AsyncFetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
->>> # Basic HTTP authentication
->>> page = await AsyncFetcher.get("https://example.com", auth=("my_user", "password123"))
->>> # Browser impersonation
->>> page = await AsyncFetcher.get('https://example.com', impersonate='chrome110')
->>> # HTTP/3 support
->>> page = await AsyncFetcher.get('https://example.com', http3=True)
+from scrapling.fetchers import AsyncFetcher
+# Basic GET
+page = await AsyncFetcher.get('https://example.com')
+page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
+page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030')
+# With parameters
+page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})
+
+# With headers
+page = await AsyncFetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
+# Basic HTTP authentication
+page = await AsyncFetcher.get("https://example.com", auth=("my_user", "password123"))
+# Browser impersonation
+page = await AsyncFetcher.get('https://example.com', impersonate='chrome110')
+# HTTP/3 support
+page = await AsyncFetcher.get('https://example.com', http3=True)
 ```
 The `page` object in all cases is a [Response](choosing.md#response-object) object, which is a [Selector](parsing/main_classes.md#selector), so you can use it directly
 ```python
@@ -102,62 +102,62 @@ The `page` object in all cases is a [Response](choosing.md#response-object) obje
 ```
 #### POST
 ```python
->>> from scrapling.fetchers import Fetcher
->>> # Basic POST
->>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, params={'q': 'query'})
->>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True)
->>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")
->>> # Another example of form-encoded data
->>> page = Fetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)
->>> # JSON data
->>> page = Fetcher.post('https://example.com/api', json={'key': 'value'})
+from scrapling.fetchers import Fetcher
+# Basic POST
+page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, params={'q': 'query'})
+page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True)
+page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")
+# Another example of form-encoded data
+page = Fetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)
+# JSON data
+page = Fetcher.post('https://example.com/api', json={'key': 'value'})
 ```
 And for asynchronous requests, it's a small adjustment
 ```python
->>> from scrapling.fetchers import AsyncFetcher
->>> # Basic POST
->>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})
->>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True)
->>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")
->>> # Another example of form-encoded data
->>> page = await AsyncFetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)
->>> # JSON data
->>> page = await AsyncFetcher.post('https://example.com/api', json={'key': 'value'})
+from scrapling.fetchers import AsyncFetcher
+# Basic POST
+page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})
+page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True)
+page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome")
+# Another example of form-encoded data
+page = await AsyncFetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True)
+# JSON data
+page = await AsyncFetcher.post('https://example.com/api', json={'key': 'value'})
 ```
 #### PUT
 ```python
->>> from scrapling.fetchers import Fetcher
->>> # Basic PUT
->>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'})
->>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome")
->>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')
->>> # Another example of form-encoded data
->>> page = Fetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})
+from scrapling.fetchers import Fetcher
+# Basic PUT
+page = Fetcher.put('https://example.com/update', data={'status': 'updated'})
+page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome")
+page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')
+# Another example of form-encoded data
+page = Fetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})
 ```
 And for asynchronous requests, it's a small adjustment
 ```python
->>> from scrapling.fetchers import AsyncFetcher
->>> # Basic PUT
->>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'})
->>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome")
->>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')
->>> # Another example of form-encoded data
->>> page = await AsyncFetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})
+from scrapling.fetchers import AsyncFetcher
+# Basic PUT
+page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'})
+page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome")
+page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')
+# Another example of form-encoded data
+page = await AsyncFetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']})
 ```

 #### DELETE
 ```python
->>> from scrapling.fetchers import Fetcher
->>> page = Fetcher.delete('https://example.com/resource/123')
->>> page = Fetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome")
->>> page = Fetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')
+from scrapling.fetchers import Fetcher
+page = Fetcher.delete('https://example.com/resource/123')
+page = Fetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome")
+page = Fetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')
 ```
 And for asynchronous requests, it's a small adjustment
 ```python
->>> from scrapling.fetchers import AsyncFetcher
->>> page = await AsyncFetcher.delete('https://example.com/resource/123')
->>> page = await AsyncFetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome")
->>> page = await AsyncFetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')
+from scrapling.fetchers import AsyncFetcher
+page = await AsyncFetcher.delete('https://example.com/resource/123')
+page = await AsyncFetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome")
+page = await AsyncFetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')
 ```

 ## Session Management
@@ -6,7 +6,7 @@
 You have one primary way to import this Fetcher, which is the same for all fetchers.

 ```python
->>> from scrapling.fetchers import StealthyFetcher
+from scrapling.fetchers import StealthyFetcher
 ```
 Check out how to configure the parsing options [here](choosing.md#parser-configuration-in-all-fetchers)

@@ -68,22 +68,21 @@ To extract the Questions button from the old design, a selector like `#hmenus >

 Testing the same selector in both versions:
 ```python
->> from scrapling import Fetcher
->> selector = '#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a'
->> old_url = "https://web.archive.org/web/20100102003420/http://stackoverflow.com/"
->> new_url = "https://stackoverflow.com/"
->> Fetcher.configure(adaptive = True, adaptive_domain='stackoverflow.com')
->> 
->> page = Fetcher.get(old_url, timeout=30)
->> element1 = page.css(selector, auto_save=True)[0]
->> 
->> # Same selector but used in the updated website
->> page = Fetcher.get(new_url)
->> element2 = page.css(selector, adaptive=True)[0]
->> 
->> if element1.text == element2.text:
+from scrapling import Fetcher
+selector = '#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a'
+old_url = "https://web.archive.org/web/20100102003420/http://stackoverflow.com/"
+new_url = "https://stackoverflow.com/"
+Fetcher.configure(adaptive = True, adaptive_domain='stackoverflow.com')
+
+page = Fetcher.get(old_url, timeout=30)
+element1 = page.css(selector, auto_save=True)[0]
+
+# Same selector but used in the updated website
+page = Fetcher.get(new_url)
+element2 = page.css(selector, adaptive=True)[0]
+
+if element1.text == element2.text:
 ...    print('Scrapling found the same element in the old and new designs!')
-'Scrapling found the same element in the old and new designs!'
 ```
 The `adaptive_domain` argument is used here because Scrapling sees `archive.org` and `stackoverflow.com` as two different domains and would isolate their `adaptive` data. Passing `adaptive_domain` tells Scrapling to treat them as the same website for adaptive data storage.

@@ -127,11 +126,11 @@ First, enable the `adaptive` feature by passing `adaptive=True` to the [Selector

 Examples:
 ```python
->>> from scrapling import Selector, Fetcher
->>> page = Selector(html_doc, adaptive=True)
+from scrapling import Selector, Fetcher
+page = Selector(html_doc, adaptive=True)
 # OR
->>> Fetcher.adaptive = True
->>> page = Fetcher.get('https://example.com')
+Fetcher.adaptive = True
+page = Fetcher.get('https://example.com')
 ```
 When using the [Selector](main_classes.md#selector) class, pass the URL of the website with the `url` argument so Scrapling can separate the properties saved for each element by domain.

@@ -159,11 +158,11 @@ Elements can be manually saved, retrieved, and relocated within the `adaptive` f

 Example of getting an element by text:
 ```python
->>> element = page.find_by_text('Tipping the Velvet', first_match=True)
+element = page.find_by_text('Tipping the Velvet', first_match=True)
 ```
 Save its unique properties using the `save` method. The identifier must be set manually (use a meaningful identifier):
 ```python
->>> page.save(element, 'my_special_element')
+page.save(element, 'my_special_element')
 ```
 Later, retrieve and relocate the element inside the page with `adaptive`:
 ```python
@@ -131,14 +131,14 @@ Getting the attributes of the element
 ```
 Access a specific attribute with any of the following
 ```python
->>> article.attrib['class']
->>> article.attrib.get('class')
->>> article['class']  # new in v0.3
+article.attrib['class']
+article.attrib.get('class')
+article['class']  # new in v0.3
 ```
 Check if the attributes contain a specific attribute with any of the methods below
 ```python
->>> 'class' in article.attrib
->>> 'class' in article  # new in v0.3
+'class' in article.attrib
+'class' in article  # new in v0.3
 ```
 Get the HTML content of the element
 ```python
@@ -279,13 +279,13 @@ In the [Selector](#selector) class, all methods/properties that should return a
 Starting with v0.4, all selection methods consistently return [Selector](#selector)/[Selectors](#selectors) objects, even for text nodes and attribute values. Text nodes (selected via `::text`, `/text()`, `::attr()`, `/@attr`) are wrapped in [Selector](#selector) objects. These text node selectors have `tag` set to `"#text"`, and their `text` property returns the text value. You can still access the text value directly, and all other properties return empty/default values gracefully.

 ```python
->>> page.css('a::text')              # -> Selectors (of text node Selectors)
->>> page.xpath('//a/text()')         # -> Selectors
->>> page.css('a::text').get()        # -> TextHandler (the first text value)
->>> page.css('a::text').getall()     # -> TextHandlers (all text values)
->>> page.css('a::attr(href)')        # -> Selectors
->>> page.xpath('//a/@href')          # -> Selectors
->>> page.css('.price_color')         # -> Selectors
+page.css('a::text')              # -> Selectors (of text node Selectors)
+page.xpath('//a/text()')         # -> Selectors
+page.css('a::text').get()        # -> TextHandler (the first text value)
+page.css('a::text').getall()     # -> TextHandlers (all text values)
+page.css('a::attr(href)')        # -> Selectors
+page.xpath('//a/@href')          # -> Selectors
+page.css('.price_color')         # -> Selectors
 ```

 ### Data extraction methods
@@ -346,8 +346,8 @@ It filters all elements in the current page/element in the following order:

 ### Examples
 ```python
->>> from scrapling.fetchers import Fetcher
->>> page = Fetcher.get('https://quotes.toscrape.com/')
+from scrapling.fetchers import Fetcher
+page = Fetcher.get('https://quotes.toscrape.com/')
 ```
 Find all elements with the tag name `div`.
 ```python