diff --git a/docs/ai/mcp-server.md b/docs/ai/mcp-server.md index 4387624..316f455 100644 --- a/docs/ai/mcp-server.md +++ b/docs/ai/mcp-server.md @@ -189,7 +189,7 @@ We will gradually go from simple prompts to more complex ones. We will use Claud Get all product titles from https://shop.example.com using the CSS selector '.product-title'. If the request fails, retry up to 5 times every 10 seconds. ``` - The server will extract only the elements matching your selector and return them as a structured list. Notice I told it to set the tool to try only 3 times in case the website has connection issues, but the default setting should be fine for most cases. + The server will extract only the elements matching your selector and return them as a structured list. Notice I told it to set the tool to try up to 5 times in case the website has connection issues, but the default setting should be fine for most cases. 3. **E-commerce Data Collection** diff --git a/docs/api-reference/mcp-server.md b/docs/api-reference/mcp-server.md index 55fa91e..03cb102 100644 --- a/docs/api-reference/mcp-server.md +++ b/docs/api-reference/mcp-server.md @@ -19,7 +19,7 @@ Or import the server class directly: from scrapling.core.ai import ScraplingMCPServer server = ScraplingMCPServer() -server.serve() +server.serve(http=False, host="0.0.0.0", port=8000) ``` ## Response Model diff --git a/docs/cli/extract-commands.md b/docs/cli/extract-commands.md index 8f6c19e..fa622d0 100644 --- a/docs/cli/extract-commands.md +++ b/docs/cli/extract-commands.md @@ -280,7 +280,7 @@ We will go through each command in detail below. -s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches. --wait-selector TEXT CSS selector to wait for before proceeding --locale TEXT Specify user locale. Defaults to the system default locale. - ---real-chrome/--no-real-chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False) + --real-chrome/--no-real-chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False) --proxy TEXT Proxy URL in format "http://username:password@host:port" -H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times) --help Show this message and exit. @@ -320,8 +320,7 @@ We will go through each command in detail below. --solve-cloudflare / --no-solve-cloudflare Solve Cloudflare challenges (default: False) --allow-webgl / --block-webgl Allow WebGL (default: True) --network-idle / --no-network-idle Wait for network idle (default: False) - ---real-chrome/--no-real-chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False) - --hide-canvas/--show-canvas Add noise to canvas operations (default: False) + --real-chrome/--no-real-chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False) --timeout INTEGER Timeout in milliseconds (default: 30000) --wait INTEGER Additional wait time in milliseconds after page load (default: 0) -s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches. diff --git a/docs/development/adaptive_storage_system.md b/docs/development/adaptive_storage_system.md index e086dae..788ad6e 100644 --- a/docs/development/adaptive_storage_system.md +++ b/docs/development/adaptive_storage_system.md @@ -56,7 +56,7 @@ class RedisStorage(StorageSystemMixin): orjson.dumps(element_dict) ) - def retrieve(self, identifier: str) -> dict: + def retrieve(self, identifier: str) -> dict | None: # Get data key = f"scrapling:{self._get_base_url()}:{identifier}" data = self.redis.get(key) diff --git a/docs/fetching/choosing.md b/docs/fetching/choosing.md index 5f78c73..dcba9b9 100644 --- a/docs/fetching/choosing.md +++ b/docs/fetching/choosing.md @@ -40,19 +40,18 @@ Then you use it right away without initializing like this, and it will use the d If you want to configure the parser ([Selector class](../parsing/main_classes.md#selector)) that will be used on the response before returning it for you, then do this first: ```python >>> from scrapling.fetchers import Fetcher ->>> Fetcher.configure(adaptive=True, encoding="utf-8", keep_comments=False, keep_cdata=False) # and the rest +>>> Fetcher.configure(adaptive=True, keep_comments=False, keep_cdata=False) # and the rest ``` or ```python >>> from scrapling.fetchers import Fetcher >>> Fetcher.adaptive=True ->>> Fetcher.encoding="utf-8" >>> Fetcher.keep_comments=False >>> Fetcher.keep_cdata=False # and the rest ``` Then, continue your code as usual. -The available configuration arguments are: `adaptive`, `huge_tree`, `keep_comments`, `keep_cdata`, `storage`, and `storage_args`, which are the same ones you give to the [Selector](../parsing/main_classes.md#selector) class. You can display the current configuration anytime by running `.display_config()`. +The available configuration arguments are: `adaptive`, `adaptive_domain`, `huge_tree`, `keep_comments`, `keep_cdata`, `storage`, and `storage_args`, which are the same ones you give to the [Selector](../parsing/main_classes.md#selector) class. You can display the current configuration anytime by running `.display_config()`. !!! info diff --git a/docs/fetching/stealthy.md b/docs/fetching/stealthy.md index 7cd65e8..fcbc8ea 100644 --- a/docs/fetching/stealthy.md +++ b/docs/fetching/stealthy.md @@ -181,7 +181,7 @@ def scrape_amazon_product(url): 'rating': page.css('[data-feature-name="averageCustomerReviews"] .a-popover-trigger .a-color-base::text').get(), 'reviews_count': page.css('#acrCustomerReviewText::text').re_first(r'[\d,]+'), 'features': [ - li.clean() for li in page.css('#feature-bullets li span::text') + li.get().clean() for li in page.css('#feature-bullets li span::text') ], 'availability': page.css('#availability')[0].get_all_text(strip=True), 'images': [ diff --git a/docs/parsing/adaptive.md b/docs/parsing/adaptive.md index 0fc7255..23dcaf3 100644 --- a/docs/parsing/adaptive.md +++ b/docs/parsing/adaptive.md @@ -145,7 +145,7 @@ Examples: >>> page = Selector(html_doc, adaptive=True) # OR >>> Fetcher.adaptive = True ->>> page = Fetcher.fetch('https://example.com') +>>> page = Fetcher.get('https://example.com') ``` If you are using the [Selector](main_classes.md#selector) class, you need to pass the url of the website you are using with the argument `url` so Scrapling can separate the properties saved for each element by domain. diff --git a/docs/parsing/main_classes.md b/docs/parsing/main_classes.md index 9f8ec3b..f310037 100644 --- a/docs/parsing/main_classes.md +++ b/docs/parsing/main_classes.md @@ -343,7 +343,7 @@ Apart from the standard operations on Python lists, such as iteration and slicin You can do the following: -Execute CSS and XPath selectors directly on the [Selector](#selector) instances it has, while the arguments and the return types are the same as [Selector](#selector)'s `css` and `xpath` methods. This, of course, makes chaining methods very straightforward. +Execute CSS and XPath selectors directly on the [Selector](#selector) instances it has, while the return types are the same as [Selector](#selector)'s `css` and `xpath` methods. The arguments are similar, except the `adaptive` argument is not available here. This, of course, makes chaining methods very straightforward. ```python >>> page.css('.product_pod a') [ ] ``` -Find all elements that don't have children. +Find all elements that have children. ```python >>> page.find_all(lambda element: len(element.children) > 0) [`target_sibling = element.find_previous_siblings("a")` | `target_sibling = element.siblings.filter(lambda s: s.tag == 'a')` | | Searching for an element in the next elements of an element | `target_parent = element.find_next("a")` | `target_parent = element.below_elements.search(lambda p: p.tag == 'a')` | | Searching for elements in the next elements of an element | `target_parent = element.find_all_next("a")` | `target_parent = element.below_elements.filter(lambda p: p.tag == 'a')` | -| Searching for an element in the previous elements of an element | `target_parent = element.find_previous("a")` | `target_parent = element.path.search(lambda p: p.tag == 'a')` | -| Searching for elements in the previous elements of an element | `target_parent = element.find_all_previous("a")` | `target_parent = element.path.filter(lambda p: p.tag == 'a')` | +| Searching for an element in the ancestors of an element | `target_parent = element.find_previous("a")` ¹ | `target_parent = element.path.search(lambda p: p.tag == 'a')` | +| Searching for elements in the ancestors of an element | `target_parent = element.find_all_previous("a")` ¹ | `target_parent = element.path.filter(lambda p: p.tag == 'a')` | | Get previous sibling of an element | `prev_element = element.previous_sibling` | `prev_element = element.previous` | | Navigating to children | `children = list(element.children)` | `children = element.children` | | Get all descendants of an element | `children = list(element.descendants)` | `children = element.below_elements` | | Filtering a group of elements that satisfies a condition | `group = soup.find('p', 'story').css.filter('a')` | `group = page.find_all('p', 'story').filter(lambda p: p.tag == 'a')` | +¹ **Note:** BS4's `find_previous`/`find_all_previous` searches all preceding elements in document order, while Scrapling's `path` only returns ancestors (the parent chain). These are not exact equivalents, but ancestor search covers the most common use case. + **One key point to remember**: BeautifulSoup offers features for modifying and manipulating the page after it has been parsed. Scrapling focuses more on scraping the page faster for you, and then you can do what you want with the extracted information. So, two different tools can be used in Web Scraping, but one of them specializes in Web Scraping :) ### Putting It All Together