freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,023 Commits 1 Branch 0 Tags
d0f52c38555cb683d192b4045b452b5ea9a36573
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair d0f52c3855 test: remove the tests for the removed html_utils module 2026-01-20 16:37:17 +02:00
Karim shoair 95955af3be build: Add w3lib to deps 2026-01-20 16:36:34 +02:00
Karim shoair 0ec78e2335 test: update tests accordingly 2026-01-19 16:40:18 +02:00
Karim shoair 6947066761 feat(spiders): Make graceful shutdown always enabled 2026-01-19 16:40:02 +02:00
Karim shoair 94bc759c6a test: updates tests to include new logic 2026-01-19 16:13:29 +02:00
Karim shoair 25850db92c fix(spiders): handle errors with manual pause during stream mode 2026-01-19 16:13:03 +02:00
Karim shoair fbc8be5bcc fix(spiders): fix a bug with session stats counter 2026-01-19 16:00:06 +02:00
Karim shoair 4ecf8a64d3 fix(spiders): Fix for when the pausing system has large queue 2026-01-19 15:04:40 +02:00
Karim shoair cf0de7ea6d Merge branch 'main' into v4 2026-01-19 14:47:59 +02:00
Karim shoair 9229cef895 docs: adding new sponsor 2026-01-19 14:12:32 +02:00
Karim shoair d247b5b691 test: add tests for the spiders system 
It's generated by Opus on Claude Code. It's very good as per my review and instructions, but I will have another look later.
 2026-01-19 03:46:00 +02:00
Karim shoair 0d003f62e3 ops: update bandit checklist 2026-01-19 03:45:39 +02:00
Karim shoair 1f0f86e3a2 feat(spiders): Add pause/resume system for crawls 2026-01-19 03:43:10 +02:00
Karim shoair acdb39988c ops: update bandit checklist 2026-01-19 03:40:57 +02:00
Karim shoair bf5aa021d4 fix(browsers): solving an issues with leaving playwright loop open when cdp connection fails 
This has been causing issues with tests for a long time, and now finally found the reason.
 2026-01-18 16:46:48 +02:00
Karim shoair 02c9dffc77 style(spiders): make parse function abstract 2026-01-18 00:01:04 +02:00
Karim shoair afe6b2e97b feat(spiders): Change items hook to make it for processing items + add a stat for this 2026-01-17 20:52:49 +02:00
Karim shoair 41c7a7e793 fix(spiders engine): Improve logging 2026-01-17 19:57:04 +02:00
Karim shoair dac854a129 fix(parser): handle responses with empty body 
It makes root's value None
 2026-01-13 02:35:26 +02:00
Karim shoair 35d570120a feat(spiders): Add option to logging format and date format 2026-01-12 17:11:52 +02:00
Karim shoair 3466bcab2c feat: add hook to run after collecting an item 2026-01-12 15:58:33 +02:00
Karim shoair e0f94cf7d0 feat(spiders): Add a stream mode to collect items in realtime 2026-01-12 15:53:29 +02:00
Karim shoair 927cfd85ff feat(spiders): The option to start sessions lazily 2026-01-12 02:09:50 +02:00
Karim shoair f2c6216968 style: remove unwanted import from spiders main init file 2026-01-12 01:21:26 +02:00
Karim shoair 72c3c2792a feat(spiders): add simple export system for the results 2026-01-12 01:21:05 +02:00
Karim shoair a17bb1d976 feat(browsers): Add option to retry tabs that gives errors 2026-01-11 23:40:26 +02:00
Karim shoair 5a11c2466b fix: make retry_blocked_request take the response as well 2026-01-11 22:54:17 +02:00
Karim shoair 736873a7c6 feat(spiders): Add logic to detect blocked requests and retry them 2026-01-11 21:56:57 +02:00
Karim shoair 774364b6d3 style(spiders): make the usage of uvloop optional 2026-01-11 19:57:03 +02:00
Karim shoair 0bfdab7158 feat: spiders use uvloop/winloop when available 2026-01-11 18:39:42 +02:00
Karim shoair d5b9ed11b2 fix(browsers): Close pages that give error 
This was causing the browser and the whole code to stand still.
 2026-01-11 18:34:13 +02:00
Karim shoair 32f4f2adf4 build: Add AnyIO to the dependencies 2026-01-11 17:01:33 +02:00
Karim shoair 059a708b6d feat(spiders system): a prototype of the new spiders system 
- A modern spider design that uses AnyIO and asyncio, yet it's very similar to Scrapy spiders API because it's the easiest design for users, and to make it easier for new users.
- Spiders can have multiple sessions per crawl, and users decide which session to use with each request.
- A scheduler system that uses heapq logic.
- The user can set the number of concurrent requests for a spider globally or per domain.
- The user can set a download delay to control the speed of the spider more.
- There's a global function that can be overridden to handle errors for all requests. (Similar to errback in scrapy).
- There's a spider argument to set the allowed domains for the spider to stay in.
- Each spider has a very detailed crawl stats that can be accessed right away from the code after the crawl finishes. Same case with scraped items.
- The whole spider as written as any other script and you just run it. No command-line arguments, and no need to run it from the terminal through the library like other known alternatives.
- Each spider has its own logger that forces sessions to use it.
- Each spider has functions to override that run before start and after close.
- There's a spider argument to set the logging level and another one to make the spider write to a log file.

- This is only the start. A lot more features are coming in the way.
 2026-01-11 16:53:18 +02:00
Karim shoair dd5c30c521 feat(response): Option to set the referer as the previous response url. 2026-01-11 03:24:30 +02:00
Karim shoair 1443a469a3 fix: handle a bug with response follow referer flow argument 2026-01-11 03:22:51 +02:00
Karim shoair aa4817f349 feat: Change logger to be flexible enough for the spiders classes 2026-01-11 03:22:20 +02:00
Karim shoair c5bc82deb8 fix(browsers): solve a bug with setting referer on the request level 2026-01-11 03:21:03 +02:00
Karim shoair 8e36962177 ops: add local Claude files to gitignore 2026-01-10 20:41:23 +02:00
Karim shoair 0634f5796c feat(spiders): Add follow function to the response 2026-01-10 20:39:43 +02:00
Karim shoair f80e8828f7 refactor: internal API changes 2026-01-08 01:53:48 +02:00
Karim shoair 3a360acced build: pump version up temporary 2026-01-08 01:37:47 +02:00
Karim shoair 08f3e6efad fix: add missing line 2026-01-08 01:01:00 +02:00
Karim shoair 6553d90e0f ops: update workflows to run for this branch 2026-01-08 00:53:29 +02:00
Karim shoair 9eb5f68fcb test: update tests accordingly 2026-01-08 00:52:35 +02:00
Karim shoair 6a73f9dcd7 refactor: internal API changes to be easily used as indicators for spiders 2026-01-08 00:52:06 +02:00
Karim shoair de79fe80bb v0.3.14 2026-01-03 22:29:20 +02:00
Karim shoair 2f87507afe test: update database test to suite windows 2026-01-03 22:02:03 +02:00
Karim shoair f99baf65f7 test: remove the check for incognito 2026-01-03 01:50:46 +02:00
Karim shoair c8a2456b81 fix(StealthyFetcher): disable incognito mode to solve #123 2026-01-03 01:50:02 +02:00
Karim shoair 30a77de2be build: pump version up and browserforge version 2026-01-03 01:48:34 +02:00
Karim shoair 6ad2ecb9f2 v0.3.13 2026-01-01 22:07:00 +02:00
Karim shoair b5fdb71b52 docs: improving the API reference section 2026-01-01 21:07:20 +02:00
Karim shoair 4947152be2 docs: change docstring style for some methods 2026-01-01 21:06:35 +02:00
Karim shoair 9742b239be docs: fix for rtd build 2026-01-01 17:42:47 +02:00
Karim shoair e275509ece docs: fix for rtd build 
test
 2026-01-01 17:05:10 +02:00
Karim shoair 185a9c594f docs: fix for rtd build 2026-01-01 17:01:24 +02:00
Karim shoair c6e4482837 docs: Improve docs website 2026-01-01 16:56:57 +02:00
Karim shoair cb5051ca56 docs: update contributing page 2026-01-01 03:18:15 +02:00
Karim shoair 4d478eb84c docs: update the donate page 2026-01-01 03:16:10 +02:00
Karim shoair 2d5e2f41c4 docs: update dev articles 2026-01-01 03:14:12 +02:00
Karim shoair 0a3bb6cfdb docs: update the migrating from BS article 2025-12-31 18:32:34 +02:00
Karim shoair 55ff20c238 docs: update replacing AI article 2025-12-31 18:28:28 +02:00
Karim shoair 7d5b907d6c docs: adding more data to the adaptive feature page 2025-12-31 16:56:57 +02:00
Karim shoair 17202d23a8 docs: Update the MCP server page 2025-12-31 16:32:21 +02:00
Karim shoair ad234165e9 docs: update the cli section 2025-12-31 03:00:26 +02:00
Karim shoair 3b9a6d021e fix: correct cli argument 2025-12-31 03:00:06 +02:00
Karim shoair d4ae6c49c0 docs: update the StealthyFetcher page 2025-12-31 00:22:33 +02:00
Karim shoair f6f0ed4b37 fix(async stealth): wait the coroutine of the bounding box 2025-12-31 00:19:53 +02:00
Karim shoair 33c5633c58 test: correcting tests with the new arguments 2025-12-30 22:06:37 +02:00
Karim shoair 0c7e26ae5c fix(fingerprints): do not create google referer for IPs 2025-12-30 21:58:09 +02:00
Karim shoair a2b5628a73 docs: update DynamicFetcher page 2025-12-29 22:02:01 +02:00
Karim shoair 67e8f4d515 docs: update DynamicFetcher page 2025-12-29 20:48:45 +02:00
Karim shoair e1cac0f69f docs: update requests page 2025-12-29 20:48:32 +02:00
Karim shoair bc420da19b docs: update deps 2025-12-29 20:39:13 +02:00
Karim shoair ee13af9516 build: update deps 
and docstrings
 2025-12-29 16:08:51 +02:00
Karim shoair 6c25dccb05 docs: updating the Fetcher Basics and the HTTP requests pages 2025-12-28 21:57:21 +02:00
Karim shoair 5e3845f060 docs: shorten docstrings for some functions 2025-12-28 21:56:28 +02:00
Karim shoair b1cd5c5a82 test: fix old assert test 2025-12-28 00:16:37 +02:00
Karim shoair 9bfe744840 fix: remove forgotten unused code 2025-12-28 00:14:47 +02:00
Karim shoair 1b838eabd0 fix: update the cookies type hint for the mcp server 2025-12-28 00:08:44 +02:00
Karim shoair a4ad6b5fce test: remove old test 2025-12-28 00:03:09 +02:00
Karim shoair 09bbd416b8 fix: remove old proxy logic 
When I was caching it with the rest of the data
 2025-12-28 00:02:50 +02:00
Karim shoair a8e646160f docs: update the parsing section 2025-12-27 23:48:31 +02:00
Karim shoair a35cc4bdd6 docs: updating overview and benchmarks pages with the new changes 2025-12-27 20:51:28 +02:00
Karim shoair fb3a1b4c17 style: removing unused code 2025-12-27 20:50:08 +02:00
Karim shoair 8ca0646172 docs: make the cli section clearer for README files of all languages 2025-12-27 16:55:27 +02:00
Karim shoair 423de087d4 docs: update README for all languages with the new updates 2025-12-27 16:40:47 +02:00
Karim shoair ab76f1bee7 docs: update the website main page 2025-12-27 16:26:30 +02:00
Karim shoair 423755fe82 fix: remove firefox from test workflow 2025-12-27 14:33:24 +02:00
Karim shoair 842718ba94 fix: update shell signatures for the autocompletion 
Added missing fields and reordered classes as well for easier comparisons
 2025-12-27 14:30:48 +02:00
Karim shoair 2af21e240d test: remove problematic sync tests 
The async version are the same tests
 2025-12-27 14:02:02 +02:00
Karim shoair 4753743f49 fix: pinning the deps version for tox 2025-12-26 14:44:26 +02:00
Karim shoair 5638755558 ops: update tests workflow accordingly 2025-12-26 03:23:28 +02:00
Karim shoair fdfd5bf96b test: update all tests accordingly 2025-12-26 03:22:56 +02:00
Karim shoair 67c8cb7de3 ops: update tox file accordingly 2025-12-26 03:22:28 +02:00
Karim shoair c5f32e495c ops: update docker file accordingly 2025-12-26 03:22:11 +02:00
Karim shoair ccf97803d5 build: update deps 2025-12-26 03:21:40 +02:00
Karim shoair cfc667f7dc refactor(fetchers)!: Replace Camoufox with patchright and many optimizations 
- DynamicFetcher became 20% faster
- StealthyFetcher became 99% faster
- Scrapling size decreased
- Code became ~400 lines shorter
- Most importantly, scrapling is more stable and reliable now.
- Less confusing for new users.
- More...
 2025-12-26 03:21:13 +02:00
Karim shoair f9f34e6ee9 fix: improve the handling of useragent in chrome fetcher 2025-12-24 23:08:30 +02:00
Karim shoair b40c16a2cd build: pump version up 2025-12-24 21:35:41 +02:00