freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
540 Commits 1 Branch 0 Tags
7f11b6f59b8a3dc66952567f343a9db9e3d3fc9b
Commit Graph

286 Commits

Author SHA1 Message Date
Karim shoair 7f11b6f59b fix(storage): possible threading issue with recursion + optimizations 2025-07-28 00:16:58 +03:00
Karim shoair 3b0237e402 docs: improve All storage classes doc strings 2025-07-27 23:37:47 +03:00
Karim shoair b60fcbb884 refactor(Adaptor): Cleaner approach to find_similar method 
This code is slower than before by about 2-5μs, but it's worth it.
 2025-07-27 23:34:22 +03:00
Karim shoair ffde7e8ad9 fix(Adaptor): Add cleanup function to handle possible memory leak 2025-07-27 23:31:55 +03:00
Karim shoair 6ae1810405 docs: improve All Adaptor class doc strings 2025-07-27 22:41:37 +03:00
Karim shoair 93bd131bb6 fix(parser): Solve the ignored elements children issue while keeping speed 
solves #61
 2025-07-27 05:34:48 +03:00
Karim shoair d1aa0be6e4 refactor(StealthyFetcher): Remove virtual mode and use persistent context 
Solves #64 completely too
 2025-07-27 03:41:28 +03:00
Karim shoair 172a5b4a0a feat: Add an mcp server 2025-07-26 22:57:08 +03:00
Karim shoair 1076001e29 refactor: changes to be used by the mcp server 2025-07-14 05:38:29 +03:00
Karim shoair ecef751468 fix: small fix for fetcher 2025-07-14 02:14:54 +03:00
Karim shoair 3184dd8581 build: set minimum version to Python 3.10 2025-07-07 00:09:02 +03:00
Karim shoair 2b73fdbfd7 fix(DynamicFetcher): Remove old argument 2025-07-06 05:02:16 +03:00
Karim shoair 63a9db21a1 refactor(DynamicSession): Optimization + Removing max_pages from sync version 2025-07-06 04:59:22 +03:00
Karim shoair 6d79927233 refactor(Fetcher): Fix the issue of caching impersonation state + Less code duplication 
It now creates a session with each request, but at the same time, it's still faster than the fetcher in v0.2.99 by 20% with more features enabled.
 2025-07-06 02:21:15 +03:00
Karim shoair 18e56a8c50 refactor: Optimizing extract command and cleaning code 2025-06-29 19:40:03 +03:00
Karim shoair 29e485bdf2 refactor: Optimizations to CLI 2025-06-29 19:10:52 +03:00
Karim shoair 1c9e48b1c6 feat(extract): Adding new command to CLI options + Optimizations 
Users can now fetch websites directly without code and extract full/selected HTML content as HTML, Markdown, or extract text content.
 2025-06-29 18:56:09 +03:00
Karim shoair dbf7aa7761 feat(DynamicFetcher): Use persistent context by default for better stealth 
And might solve the sandbox issue with GitHub
 2025-06-27 03:02:46 +03:00
Karim shoair 8395db9121 fix(controllers): Fix issue with async adding stealth scripts 2025-06-27 01:39:32 +03:00
Karim shoair 37e842b796 fix(controllers): Fix issue with adding stealth scripts 2025-06-27 01:04:58 +03:00
Karim shoair 53318f3a2a fix(StealthyFetcher): Fix passed proxy type 2025-06-26 16:36:30 +03:00
Karim shoair 8d44e9c51c fix(proxy): fix proxy unpacking 2025-06-26 15:49:53 +03:00
Karim shoair 38851698ad refactor: optimize imports and docstrings correction 2025-06-25 23:15:32 +03:00
Karim shoair a41be48047 refactor: optimize imports and docstrings correction 2025-06-25 21:36:53 +03:00
Karim shoair a3a5e34408 refactor: optimize imports and docstrings correction 2025-06-25 21:36:43 +03:00
Karim shoair 724e3a648e fix(FetcherSession): Use Sentinel pattern to solve arguments precedence issue 2025-06-25 03:53:54 +03:00
Karim shoair 63ff12bcd6 docs: correcting a small mistake 2025-06-23 03:38:34 +03:00
Karim shoair 78b81c537f fix(fetchers): Fix the bug of referer and google_search argument conflict 2025-06-23 03:36:11 +03:00
Karim shoair 69b983b42f feat(fetchers): Improve StealthyFetcher + Adding StealthySession/AsyncStealthySession classes 2025-06-23 03:23:28 +03:00
Karim shoair 3754547e58 refactor: removing unused code 2025-06-23 03:20:49 +03:00
Karim shoair fc530677de refactor(DynamicFetcher): Cleaner exit for fetch 2025-06-23 02:58:01 +03:00
Karim shoair 22490db85f fix(DynamicFetcher): Set number of tabs to 1 
There is no logical use for it here; seek the session classes to use it.
 2025-06-23 02:42:43 +03:00
Karim shoair 460b4443c2 refactor(controllers): Improve type validation 2025-06-22 19:59:40 +03:00
Karim shoair 0fe04e499f refactor(validators): Optimize imports 2025-06-22 18:55:13 +03:00
Karim shoair fc42487a5f refactor(controllers): Optimize imports 2025-06-22 18:53:44 +03:00
Karim shoair 9c44ad1930 fix(AsyncFetcher): Fix stealthy_headers issue 2025-06-22 03:54:36 +03:00
Karim shoair b77c5b7abf refactor: updating top level shortcuts 2025-06-22 02:11:26 +03:00
Karim shoair 8a272cf19b feat/refactor(fetchers): Replacing PlayWrightFetcher with DynamicFetcher and adding session classes 2025-06-22 02:10:45 +03:00
Karim shoair 1e51fc9738 refactor: Removing defaults as said 2025-06-22 02:07:34 +03:00
Karim shoair 0a570a7ca7 docs: Updating doc strings 2025-06-22 01:59:38 +03:00
Karim shoair d049f63404 fix(fetcher): Fix impersonate and headers generation conflict 2025-06-20 03:29:05 +03:00
Karim shoair 9bfc900140 feat(navigation tools): Improvements 2025-06-19 04:07:45 +03:00
Karim shoair 145b710960 feat(fetchers): Adding the foundation of the new browser-based fetchers logic 2025-06-19 04:06:52 +03:00
Karim shoair 3e63fa2523 fix(shell): Hide the automatic tips and solve the namespace issue 2025-06-19 04:03:13 +03:00
Karim shoair 2bdee60004 fix(fingerprints): Fixing headers generation for requests 2025-06-15 03:37:22 +03:00
Karim shoair 1f34ac22f6 style(fetcher): Move the default UA line 2025-06-15 03:21:21 +03:00
Karim shoair 7efbcd33c3 refactor(fingerprints): Remove unused function and improve another 2025-06-15 03:16:27 +03:00
Karim shoair 6e837f6a12 refactor(StealthyFetcher): Improve OS name extraction 2025-06-14 18:57:31 +03:00
Karim shoair 3068bb1356 build: update deps 2025-06-14 18:56:31 +03:00
Karim shoair 178eed7533 feat/fix(Fetcher): Adding http3 support + Fix stealthy_headers 
And some fixes here and there to the docs
 2025-05-31 03:35:46 +03:00