Commit Graph

29 Commits

Author SHA1 Message Date
Karim shoair 614d136f8c build: pump up version and deps 2026-04-15 20:44:38 +02:00
Jules Omlor 00897dad2f feat(mcp): add optional session_id parameter to open_session
Allow users to specify a custom session_id when opening a browser session,
rather than always generating a random UUID. Useful for naming sessions
for easier management across multiple tool calls.

- Add session_id: Optional[str] = None parameter
- Validate session_id doesn't already exist before starting browser
- Fall back to uuid4().hex[:12] if not provided
- Add tests for custom session_id and duplicate detection

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-04-14 01:17:52 -04:00
Karim shoair ad6fd52845 perf: force ad blocking on the MCP server and when the AI mode is activated on CLI 2026-04-12 18:11:02 +02:00
Karim shoair e7f9adb40a feat(security): default follow_redirects to "safe" for SSRF protection
curl_cffi v0.15.0 introduced CurlFollow.SAFE, which follows redirects but rejects those targeting internal/private IPs (loopback, private networks, link-local). This is now the default for all HTTP fetchers, the MCP server, and the shell curl converter.

Added FollowRedirects type alias supporting all curl_cffi redirect
modes: bool, "safe", "all", "obeycode", "firstonly".
2026-04-05 18:41:50 +02:00
Karim shoair a403156b2e docs: style adjustment 2026-03-30 03:53:56 +02:00
Karim shoair 0f6dcccf5f fix(mcp): remove unneeded code and fix type hint for mypy 2026-03-28 17:19:42 +02:00
Karim shoair c458ab65a2 feat(mcp): Add three new tools to control browser sessions
Now you can open a browser, keep using it for other requests as you want, and close it when you want.
2026-03-28 00:16:14 +02:00
Karim shoair a356dd2f2b refactor(mcp)!: Cleaning and unifying functions to async
- `get()` now delegates to `bulk_get([url])[0]` (was a separate sync implementation)
- `fetch()` now delegates to `bulk_fetch([url])[0]` (eliminated duplicate fetcher call)
- `stealthy_fetch()` now delegates to `bulk_stealthy_fetch([url])[0]` (same)
- Replaced 6x repeated `_content_translator(Convertor._extract_content(...), page)` with a single `_translate_response()` helper
- Removed unused imports (`Fetcher`, `DynamicFetcher`, `StealthyFetcher`, `Generator`)
2026-03-27 19:30:58 +02:00
Karim shoair a929de6ca4 fix: update code and docstrings to remove the old google referer logic 2026-03-09 00:25:19 +02:00
Karim shoair 921314c314 fix: complete MCP schema validation for remaining bare array types
- Fix params (Dict|List|Tuple → Dict), urls (Tuple[str,...] → List[str]),
- Renamed _NormalizeCredentials/_ContentTranslator to snake_case.
- Also, raise a ValueError on invalid credentials instead of returning None silently.
2026-02-27 00:28:44 +02:00
Robin Ede b77a1b9419 fix: make MCP get schemas validator-safe
Use JSON-object input types for cookies and basic-auth fields in get and bulk_get so strict MCP schema validators can register tools reliably.

Normalize auth dictionaries to the tuple format expected by fetchers to preserve runtime behavior.
2026-02-23 19:49:39 -06:00
Karim shoair 93bcb1681b style: docstrings corrections for accuracy 2026-02-15 03:57:14 +02:00
Karim shoair 5e3845f060 docs: shorten docstrings for some functions 2025-12-28 21:56:28 +02:00
Karim shoair 1b838eabd0 fix: update the cookies type hint for the mcp server 2025-12-28 00:08:44 +02:00
Karim shoair cfc667f7dc refactor(fetchers)!: Replace Camoufox with patchright and many optimizations
- DynamicFetcher became 20% faster
- StealthyFetcher became 99% faster
- Scrapling size decreased
- Code became ~400 lines shorter
- Most importantly, scrapling is more stable and reliable now.
- Less confusing for new users.
- More...
2025-12-26 03:21:13 +02:00
Karim shoair 3565f9d500 feat(fetcher): Make impersonate able to randomize fingerprint 2025-11-16 16:33:58 +02:00
Karim shoair debe03256b refactor: Making all the codebase acceptable by PyRight
Also fixes #97
2025-10-05 04:03:39 +03:00
Karim shoair f6c122b87f style: Removing dead code/docstrings 2025-09-29 05:09:23 +03:00
Karim shoair 8ad7cd6343 docs: Update all pages/docstrings to reflect recent changes 2025-09-29 03:57:40 +03:00
Karim shoair 74f20d2a0e refactor: better implementation for the mcp mode 2025-09-28 20:14:04 +03:00
Karim shoair 4a661b4875 feat: Make mcp able to use http transport 2025-09-28 04:54:35 +03:00
Karim shoair f848e6b7f2 fix: Fixes to handle the new extra deps group 2025-09-15 00:19:19 +03:00
Karim shoair 330d03559c style: applying the new ruff rules to all files 2025-09-13 03:22:53 +03:00
Karim shoair ae9ccaec79 style: A lot of type hints correction
Since we are using Py3.10 as minimum version now, we remove Union when possible
2025-07-30 02:46:57 +03:00
Karim shoair 18660f8132 style: type hints corrections and docstrings 2025-07-30 00:32:39 +03:00
Karim shoair ba585c0adc perf: imports optimizing 2025-07-30 00:22:06 +03:00
Karim shoair 264ae02aa7 refactor: huge change, many features/class got a better naming
- `Adaptor` became `Selector`
- `Adaptors` became `Selectors`
- `auto_match` argument/feature became `adaptive`
- `adaptor_arguments` argument became `selector_config`
- `automatch_domain` argument became `adaptive_domain`
- `additional_arguments` argument became `additional_args`
- `storage_adaptors` file became just `storage`
2025-07-29 04:20:23 +03:00
Karim shoair d1aa0be6e4 refactor(StealthyFetcher): Remove virtual mode and use persistent context
Solves #64 completely too
2025-07-27 03:41:28 +03:00
Karim shoair 172a5b4a0a feat: Add an mcp server 2025-07-26 22:57:08 +03:00