From 1f87210b93d6ca6109c1ad85755751061d4aebef Mon Sep 17 00:00:00 2001 From: Karim shoair Date: Thu, 28 Aug 2025 04:36:49 +0300 Subject: [PATCH] docs: Add `max_pages` explanation to DynamicFetcher --- docs/fetching/dynamic.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/fetching/dynamic.md b/docs/fetching/dynamic.md index 9b38333..5d572ae 100644 --- a/docs/fetching/dynamic.md +++ b/docs/fetching/dynamic.md @@ -121,7 +121,7 @@ page = DynamicFetcher.fetch( ``` ### Browser Automation -This is where your knowledge about [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) comes into play. The function you pass here takes the page object from Playwright's API, performs the desired action, and then returns it for the current fetcher to continue working on it. +This is where your knowledge about [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) comes into play. The function you pass here takes the page object from Playwright's API, performs the desired action, and then returns it for the current fetcher to continue processing. This function is executed immediately after waiting for `network_idle` (if enabled) and before waiting for the `wait_selector` argument, allowing it to be used for various purposes, not just automation. You can alter the page as you want. @@ -260,7 +260,8 @@ async def scrape_multiple_sites(): async with AsyncDynamicSession( stealth=True, network_idle=True, - timeout=30000 + timeout=30000, + max_pages=3 ) as session: # Make async requests with shared browser configuration pages = await asyncio.gather( @@ -271,6 +272,10 @@ async def scrape_multiple_sites(): return pages ``` +You may have noticed the `max_pages` argument. This is a new argument that enables the fetcher to create a **pool of Browser tabs** that will be rotated automatically. Instead of waiting for one browser tab to become ready, it checks if the next tab in the pool is ready to be used and uses it. This allows for multiple websites to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :) + +When all tabs inside the pool are busy, the fetcher checks every subsecond if a tab becomes ready. If none become free within a 30-second interval, it raises a `TimeoutError` error. This can happen when the website you are fetching becomes unresponsive for some reason. + ### Session Benefits - **Browser reuse**: Much faster subsequent requests by reusing the same browser instance.