freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,034 Commits 1 Branch 0 Tags
aa7a95fb70acea193c5987006a7c92b959208d45
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair aa7a95fb70 feat(parser)!: Make all selection return selector objects by default 
- The strings/Texthandlers are only returned by `get`/`getall`/`extract`/`extract_first`. This makes the type checking/autocompletion experience consistent.
- Removed `css_first` and `xpath_first` since it doesn't make sense to leave them now.
- Made the type hints more accurate in multiple places.
 2026-02-06 02:43:51 +02:00
Karim shoair f88502718f feat(proxy control): Force a proxy at request level at any given point 
And merge request's meta with response's meta
 2026-02-02 14:26:52 +02:00
Karim shoair 6e0277f23b test: add new tests for the proxy rotation logic 2026-02-02 00:16:40 +02:00
Karim shoair ef8c5bc7d6 feat(spiders/fetchers): Adding proxy rotation logic and change retry logic 
- User passes a single proxy to the browser session, and it will keep using the same context. Pass a proxy manager that automatically refreshes the IP with each proxy, and you get speed but sacrifice some stealth.

- User imports the proxy rotator class and passes proxies to it and a rotation strategy (Round robin by default). Then pass the instance to the session class, which will create a context and a tab for each proxy returned by the rotator. This way, you sacrifice speed a bit, but you get maximum stealth since each context is created with the proxy that will be used.

- Normal requests use what you pass without issues, of course.

- All errors are retried now, and proxies are rotated on retries automatically.
 2026-02-02 00:16:22 +02:00
Karim shoair 65f1a23358 fix(browsers): Improve stealth and speed by adjusting flags 2026-01-29 00:58:04 +02:00
Karim shoair 0c3d359b46 fix(browsers): disable autoplay for all browsers 2026-01-29 00:57:30 +02:00
Karim shoair 90c52c45c7 feat(parser): replacing tldextract with tld library 
This might break the adaptive data users have for websites BUT:
1. tld uses ~3.7x less memory during extraction operations (1.5 MB vs 5.7 MB).
2. tld uses ~56% less memory on import (5.2 MB vs 11.9 MB).
3. Zero dependencies (vs 3 for tldextract).

In return, it's 30ms slower for extracting 5000 URLs, which is negligible. Also, the type hints aren't always accurate, but it's fine; I corrected them.
 2026-01-23 00:00:48 +02:00
Karim shoair 5dbbd84b3d fix(requests): return the cached fingerprint on second call 2026-01-20 20:28:13 +02:00
Karim shoair d90189e675 test: update tests accordingly 2026-01-20 20:22:47 +02:00
Karim shoair 122923c80e feat(spiders/requests): Use a reliable method for fingerprint generation 2026-01-20 20:21:42 +02:00
Karim shoair 5aa178a76e style: update post requests type hints to be more accurate 2026-01-20 18:27:52 +02:00
Karim shoair d0f52c3855 test: remove the tests for the removed html_utils module 2026-01-20 16:37:17 +02:00
Karim shoair 95955af3be build: Add w3lib to deps 2026-01-20 16:36:34 +02:00
Karim shoair 0ec78e2335 test: update tests accordingly 2026-01-19 16:40:18 +02:00
Karim shoair 6947066761 feat(spiders): Make graceful shutdown always enabled 2026-01-19 16:40:02 +02:00
Karim shoair 94bc759c6a test: updates tests to include new logic 2026-01-19 16:13:29 +02:00
Karim shoair 25850db92c fix(spiders): handle errors with manual pause during stream mode 2026-01-19 16:13:03 +02:00
Karim shoair fbc8be5bcc fix(spiders): fix a bug with session stats counter 2026-01-19 16:00:06 +02:00
Karim shoair 4ecf8a64d3 fix(spiders): Fix for when the pausing system has large queue 2026-01-19 15:04:40 +02:00
Karim shoair cf0de7ea6d Merge branch 'main' into v4 2026-01-19 14:47:59 +02:00
Karim shoair 9229cef895 docs: adding new sponsor 2026-01-19 14:12:32 +02:00
Karim shoair d247b5b691 test: add tests for the spiders system 
It's generated by Opus on Claude Code. It's very good as per my review and instructions, but I will have another look later.
 2026-01-19 03:46:00 +02:00
Karim shoair 0d003f62e3 ops: update bandit checklist 2026-01-19 03:45:39 +02:00
Karim shoair 1f0f86e3a2 feat(spiders): Add pause/resume system for crawls 2026-01-19 03:43:10 +02:00
Karim shoair acdb39988c ops: update bandit checklist 2026-01-19 03:40:57 +02:00
Karim shoair bf5aa021d4 fix(browsers): solving an issues with leaving playwright loop open when cdp connection fails 
This has been causing issues with tests for a long time, and now finally found the reason.
 2026-01-18 16:46:48 +02:00
Karim shoair 02c9dffc77 style(spiders): make parse function abstract 2026-01-18 00:01:04 +02:00
Karim shoair afe6b2e97b feat(spiders): Change items hook to make it for processing items + add a stat for this 2026-01-17 20:52:49 +02:00
Karim shoair 41c7a7e793 fix(spiders engine): Improve logging 2026-01-17 19:57:04 +02:00
Karim shoair dac854a129 fix(parser): handle responses with empty body 
It makes root's value None
 2026-01-13 02:35:26 +02:00
Karim shoair 35d570120a feat(spiders): Add option to logging format and date format 2026-01-12 17:11:52 +02:00
Karim shoair 3466bcab2c feat: add hook to run after collecting an item 2026-01-12 15:58:33 +02:00
Karim shoair e0f94cf7d0 feat(spiders): Add a stream mode to collect items in realtime 2026-01-12 15:53:29 +02:00
Karim shoair 927cfd85ff feat(spiders): The option to start sessions lazily 2026-01-12 02:09:50 +02:00
Karim shoair f2c6216968 style: remove unwanted import from spiders main init file 2026-01-12 01:21:26 +02:00
Karim shoair 72c3c2792a feat(spiders): add simple export system for the results 2026-01-12 01:21:05 +02:00
Karim shoair a17bb1d976 feat(browsers): Add option to retry tabs that gives errors 2026-01-11 23:40:26 +02:00
Karim shoair 5a11c2466b fix: make retry_blocked_request take the response as well 2026-01-11 22:54:17 +02:00
Karim shoair 736873a7c6 feat(spiders): Add logic to detect blocked requests and retry them 2026-01-11 21:56:57 +02:00
Karim shoair 774364b6d3 style(spiders): make the usage of uvloop optional 2026-01-11 19:57:03 +02:00
Karim shoair 0bfdab7158 feat: spiders use uvloop/winloop when available 2026-01-11 18:39:42 +02:00
Karim shoair d5b9ed11b2 fix(browsers): Close pages that give error 
This was causing the browser and the whole code to stand still.
 2026-01-11 18:34:13 +02:00
Karim shoair 32f4f2adf4 build: Add AnyIO to the dependencies 2026-01-11 17:01:33 +02:00
Karim shoair 059a708b6d feat(spiders system): a prototype of the new spiders system 
- A modern spider design that uses AnyIO and asyncio, yet it's very similar to Scrapy spiders API because it's the easiest design for users, and to make it easier for new users.
- Spiders can have multiple sessions per crawl, and users decide which session to use with each request.
- A scheduler system that uses heapq logic.
- The user can set the number of concurrent requests for a spider globally or per domain.
- The user can set a download delay to control the speed of the spider more.
- There's a global function that can be overridden to handle errors for all requests. (Similar to errback in scrapy).
- There's a spider argument to set the allowed domains for the spider to stay in.
- Each spider has a very detailed crawl stats that can be accessed right away from the code after the crawl finishes. Same case with scraped items.
- The whole spider as written as any other script and you just run it. No command-line arguments, and no need to run it from the terminal through the library like other known alternatives.
- Each spider has its own logger that forces sessions to use it.
- Each spider has functions to override that run before start and after close.
- There's a spider argument to set the logging level and another one to make the spider write to a log file.

- This is only the start. A lot more features are coming in the way.
 2026-01-11 16:53:18 +02:00
Karim shoair dd5c30c521 feat(response): Option to set the referer as the previous response url. 2026-01-11 03:24:30 +02:00
Karim shoair 1443a469a3 fix: handle a bug with response follow referer flow argument 2026-01-11 03:22:51 +02:00
Karim shoair aa4817f349 feat: Change logger to be flexible enough for the spiders classes 2026-01-11 03:22:20 +02:00
Karim shoair c5bc82deb8 fix(browsers): solve a bug with setting referer on the request level 2026-01-11 03:21:03 +02:00
Karim shoair 8e36962177 ops: add local Claude files to gitignore 2026-01-10 20:41:23 +02:00
Karim shoair 0634f5796c feat(spiders): Add follow function to the response 2026-01-10 20:39:43 +02:00
Karim shoair f80e8828f7 refactor: internal API changes 2026-01-08 01:53:48 +02:00
Karim shoair 3a360acced build: pump version up temporary 2026-01-08 01:37:47 +02:00
Karim shoair 08f3e6efad fix: add missing line 2026-01-08 01:01:00 +02:00
Karim shoair 6553d90e0f ops: update workflows to run for this branch 2026-01-08 00:53:29 +02:00
Karim shoair 9eb5f68fcb test: update tests accordingly 2026-01-08 00:52:35 +02:00
Karim shoair 6a73f9dcd7 refactor: internal API changes to be easily used as indicators for spiders 2026-01-08 00:52:06 +02:00
Karim shoair de79fe80bb v0.3.14 2026-01-03 22:29:20 +02:00
Karim shoair 2f87507afe test: update database test to suite windows 2026-01-03 22:02:03 +02:00
Karim shoair f99baf65f7 test: remove the check for incognito 2026-01-03 01:50:46 +02:00
Karim shoair c8a2456b81 fix(StealthyFetcher): disable incognito mode to solve #123 2026-01-03 01:50:02 +02:00
Karim shoair 30a77de2be build: pump version up and browserforge version 2026-01-03 01:48:34 +02:00
Karim shoair 6ad2ecb9f2 v0.3.13 2026-01-01 22:07:00 +02:00
Karim shoair b5fdb71b52 docs: improving the API reference section 2026-01-01 21:07:20 +02:00
Karim shoair 4947152be2 docs: change docstring style for some methods 2026-01-01 21:06:35 +02:00
Karim shoair 9742b239be docs: fix for rtd build 2026-01-01 17:42:47 +02:00
Karim shoair e275509ece docs: fix for rtd build 
test
 2026-01-01 17:05:10 +02:00
Karim shoair 185a9c594f docs: fix for rtd build 2026-01-01 17:01:24 +02:00
Karim shoair c6e4482837 docs: Improve docs website 2026-01-01 16:56:57 +02:00
Karim shoair cb5051ca56 docs: update contributing page 2026-01-01 03:18:15 +02:00
Karim shoair 4d478eb84c docs: update the donate page 2026-01-01 03:16:10 +02:00
Karim shoair 2d5e2f41c4 docs: update dev articles 2026-01-01 03:14:12 +02:00
Karim shoair 0a3bb6cfdb docs: update the migrating from BS article 2025-12-31 18:32:34 +02:00
Karim shoair 55ff20c238 docs: update replacing AI article 2025-12-31 18:28:28 +02:00
Karim shoair 7d5b907d6c docs: adding more data to the adaptive feature page 2025-12-31 16:56:57 +02:00
Karim shoair 17202d23a8 docs: Update the MCP server page 2025-12-31 16:32:21 +02:00
Karim shoair ad234165e9 docs: update the cli section 2025-12-31 03:00:26 +02:00
Karim shoair 3b9a6d021e fix: correct cli argument 2025-12-31 03:00:06 +02:00
Karim shoair d4ae6c49c0 docs: update the StealthyFetcher page 2025-12-31 00:22:33 +02:00
Karim shoair f6f0ed4b37 fix(async stealth): wait the coroutine of the bounding box 2025-12-31 00:19:53 +02:00
Karim shoair 33c5633c58 test: correcting tests with the new arguments 2025-12-30 22:06:37 +02:00
Karim shoair 0c7e26ae5c fix(fingerprints): do not create google referer for IPs 2025-12-30 21:58:09 +02:00
Karim shoair a2b5628a73 docs: update DynamicFetcher page 2025-12-29 22:02:01 +02:00
Karim shoair 67e8f4d515 docs: update DynamicFetcher page 2025-12-29 20:48:45 +02:00
Karim shoair e1cac0f69f docs: update requests page 2025-12-29 20:48:32 +02:00
Karim shoair bc420da19b docs: update deps 2025-12-29 20:39:13 +02:00
Karim shoair ee13af9516 build: update deps 
and docstrings
 2025-12-29 16:08:51 +02:00
Karim shoair 6c25dccb05 docs: updating the Fetcher Basics and the HTTP requests pages 2025-12-28 21:57:21 +02:00
Karim shoair 5e3845f060 docs: shorten docstrings for some functions 2025-12-28 21:56:28 +02:00
Karim shoair b1cd5c5a82 test: fix old assert test 2025-12-28 00:16:37 +02:00
Karim shoair 9bfe744840 fix: remove forgotten unused code 2025-12-28 00:14:47 +02:00
Karim shoair 1b838eabd0 fix: update the cookies type hint for the mcp server 2025-12-28 00:08:44 +02:00
Karim shoair a4ad6b5fce test: remove old test 2025-12-28 00:03:09 +02:00
Karim shoair 09bbd416b8 fix: remove old proxy logic 
When I was caching it with the rest of the data
 2025-12-28 00:02:50 +02:00
Karim shoair a8e646160f docs: update the parsing section 2025-12-27 23:48:31 +02:00
Karim shoair a35cc4bdd6 docs: updating overview and benchmarks pages with the new changes 2025-12-27 20:51:28 +02:00
Karim shoair fb3a1b4c17 style: removing unused code 2025-12-27 20:50:08 +02:00
Karim shoair 8ca0646172 docs: make the cli section clearer for README files of all languages 2025-12-27 16:55:27 +02:00
Karim shoair 423de087d4 docs: update README for all languages with the new updates 2025-12-27 16:40:47 +02:00
Karim shoair ab76f1bee7 docs: update the website main page 2025-12-27 16:26:30 +02:00
Karim shoair 423755fe82 fix: remove firefox from test workflow 2025-12-27 14:33:24 +02:00