freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
334 Commits 1 Branch 0 Tags
9ea245eab4350ec3bb5f0242485eb1f1d23cbdbf
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair 9ea245eab4 doc(README): be more specific about what the regex match 2025-01-31 00:48:59 +02:00
Karim shoair 3f23f18473 style(TextHandler): fix re overload 2025-01-31 00:48:27 +02:00
Karim shoair 1962c11f64 fix(parser/find_all): Logic issues made conditions used sometimes in a (and) fashion and other times (OR) 
Also, the speed boost is 6-14% not 95% as I said first :(
 2025-01-31 00:23:54 +02:00
Karim shoair 57090f90a4 refactor(parser/find_all): about ~95% speed increase 
Boom!
 2025-01-30 23:39:46 +02:00
Karim shoair f37538320f feat(parser): adding below_elements method 2025-01-30 23:05:10 +02:00
Karim shoair ae60c2c0ad fix(playwright engine): bug in checking nstbrowser_config type 2025-01-30 20:07:19 +02:00
Karim shoair 148103f3c8 fix(translator): increase CSS to XPath convertor cache limit 2025-01-30 20:01:25 +02:00
Karim shoair c0d912d16a fix(asyncFetcher): A typo made async version of put do post request instead 
Awkward moment
 2025-01-30 19:53:13 +02:00
Karim shoair 7882cf2527 style(TextHandler): Setting overload on check_match argument for auto completion 2025-01-30 02:44:57 +02:00
Karim shoair e507778c48 feat(Parser): Set regex methods new return types as TextHandlers 2025-01-30 02:27:38 +02:00
Karim shoair 93be416572 feat(TextHandler): Now regex methods return TextHandler/TextHandlers 2025-01-30 02:15:09 +02:00
Karim shoair 4e2e7a4b22 fix(parser): More on correcting type hints and unifying return type 2025-01-30 02:03:09 +02:00
Karim shoair 7ee0f6114d fix(parser): Code restructure for speed boost & better type hints 
- Now return types are consistent across all the parser engine
- Parser got a 5-30% performance boost across different methods.
- Renamed some of the internal methods for clearer code.
- A lot better auto-completion experience after a lot of adjustments.
 2025-01-30 01:46:33 +02:00
Karim shoair 1d5fcc060d feat(TextHandler): Make slicing return TextHandlers + autocompletion fixes 
and access by index return `TextHandler`
 2025-01-29 14:22:47 +02:00
Karim shoair 70ffdd09e2 feat(TextHandler): Make split string method return Texthandlers 2025-01-29 01:22:07 +02:00
Karim shoair 5047e886a5 style(autocompletion): Improving type hints for custom types 
This will provide a better autocompletion experience inside IDEs, commit affects:
- TextHandler
- AttributesHandler
 2025-01-29 00:58:14 +02:00
Karim shoair 85f98708b2 feat(StealthyFetcher): Use memory cache by default for performance boost 
This is instead of the default disk-based cache.
 2025-01-28 23:55:24 +02:00
Karim shoair e9e606bca5 feat(StealthyFetcher): Disable Cross-Origin-Opener-Policy by default for async_fetch 
This allows elements in cross-origin iframes, such as the Turnstile checkbox, to be clicked.
.
Forgot to add it with the other commit
 2025-01-28 23:53:12 +02:00
Karim shoair 675c0110e6 feat(StealthyFetcher): Make ads enabled by default for performance boost 2025-01-28 23:51:31 +02:00
Karim shoair 52b66c98ec feat(StealthyFetcher): Disable Cross-Origin-Opener-Policy by default 
This allows elements in cross-origin iframes, such as the Turnstile checkbox, to be clicked.
 2025-01-28 23:47:47 +02:00
Karim shoair 1332be7e63 fix(autocompletion): Fix the defaults import 
It runs autocompletion on some IDEs
 2025-01-28 23:44:58 +02:00
Karim shoair c2447e069e build: Pumping version up 2025-01-28 23:41:48 +02:00
Karim shoair 99fe842bc9 build: updating dependencies 2025-01-28 23:40:56 +02:00
Karim shoair 21291958f1 docs(README file): Better more-friendly description 2025-01-28 23:10:56 +02:00
Karim shoair 198b76f5d5 Test: disable stealth parameter 
It causes issues with Github Actions
 2025-01-28 13:43:36 +02:00
Karim shoair 7776e191e8 Update test_playwright.py 2025-01-28 13:42:11 +02:00
Karim shoair 75a965e802 Update FUNDING.yml 2025-01-27 21:14:50 +02:00
Karim shoair 0e1dcd4f29 Adding Scrapeless sponsor 2025-01-06 11:59:19 +02:00
Karim shoair 32d96606ea Merge pull request #27 from D4Vinci/dev 
v0.2.92
 2024-12-26 20:03:24 +02:00
Karim shoair c5d83a11ad build: Pumping version up 2024-12-26 19:57:25 +02:00
Karim shoair ef284234e6 docs: update README file to reflect new installation instructions 2024-12-26 19:56:37 +02:00
Karim shoair 03b52aa74c feat(cli): adding terminal command scrapling install 
Instead of making users install browsers by themself
 2024-12-26 19:46:22 +02:00
Karim shoair 2006be2173 fix(StealthyFetcher): Use more dependable response data 2024-12-26 00:29:42 +02:00
Karim shoair f9b85cfbac fix(PlaywrightFetcher): Use more dependable response data 2024-12-26 00:27:13 +02:00
Karim shoair 3cca3fd56b style(parser): more accurate type hints 2024-12-21 16:17:02 +02:00
Karim shoair ee599141fe Merge pull request #26 from D4Vinci/dev 
v0.2.91
 2024-12-19 13:51:53 +02:00
Karim shoair bb398693b8 feat(Fetcher): Adding support to proxies using the SOCKS protocol 2024-12-19 13:30:43 +02:00
Karim shoair f74db648bc build: Pumping up the version to 0.2.91 2024-12-19 13:29:45 +02:00
Karim shoair 5be8c2a95e style(Fetchers): Setting correct type hint for wait_selector_state argument 2024-12-19 13:22:12 +02:00
Karim shoair 94745b1555 fix(StealthyFetcher): Wait for the real response status code 
This will return the real status response after redirection for websites that have Cloudflare instead of Playwright API's default behavior.
 2024-12-19 13:08:41 +02:00
Karim shoair 844cb0c6a2 fix(PlaywrightFetcher): Wait for the real response status code 
This will return the real status response after redirection for websites that have Cloudflare instead of Playwright's default behavior.
 2024-12-19 13:00:19 +02:00
Karim shoair 6652b8ac18 fix: solve the response log issue so now you get log for every response from Fetchers 2024-12-19 12:29:34 +02:00
Karim shoair 60df72cb23 Merge pull request #25 from D4Vinci/dev 
v0.2.9
 2024-12-16 14:42:13 +02:00
Karim shoair 838b8b1196 style: fixing a typo in logging 2024-12-16 13:39:30 +02:00
Karim shoair c282cb3cb7 fix: Stopped the log spamming that happens with multiple instances creation 2024-12-16 13:35:48 +02:00
Karim shoair ac95600650 docs: add an example that uses page.urljoin 2024-12-16 13:27:14 +02:00
Karim shoair 20ef45349a test: Rewrite automatch tests to a cleaner version and adding async test 2024-12-16 13:18:19 +02:00
Karim shoair 69e31612a5 test: Rewrite parser tests to a cleaner version and adding more tests 2024-12-16 13:17:09 +02:00
Karim shoair 3ff0d55c3e chore: Delete former-sponsor banner 2024-12-16 12:47:43 +02:00
Karim shoair 43b41b3733 docs: fix TOC header 2024-12-16 12:41:07 +02:00
Karim shoair 6cf5ce9719 docs: Adding async examples and fixing some typos 2024-12-16 12:39:44 +02:00
Karim shoair 05c6eeb387 fix: add AsyncFetcher to top-level shortcuts 2024-12-16 12:37:29 +02:00
Karim shoair d66ebc19a5 fix(Fetchers/disable_resources): Fixing the logic for intercepting resources 2024-12-16 12:35:25 +02:00
Karim shoair eceef48ef5 fix(Fetchers/page_action): Fixing logic 2024-12-16 12:34:23 +02:00
Karim shoair 3ecffcb81c style: Rewrite sync Fetchers tests to a cleaner version 2024-12-16 01:27:18 +02:00
Karim shoair 2606f7a12d test: clearer naming for async tests 2024-12-16 01:01:59 +02:00
Karim shoair 448587ff41 test: add tests for StealthyFetcher async fetch 2024-12-16 01:00:05 +02:00
Karim shoair 79a911e24e feat(StealthyFetcher): Add async fetch support 2024-12-16 00:59:19 +02:00
Karim shoair efb3270c67 style: Rewrite asyncFetcher tests to a cleaner version 2024-12-16 00:47:02 +02:00
Karim shoair 361ee440aa test: adding pytest-asyncio plugin to tests requirements file 2024-12-16 00:20:26 +02:00
Karim shoair de015f2b18 chore: Restructuring fetchers into cleaner structure 2024-12-16 00:19:48 +02:00
Karim shoair aac77e488c test: add test for PlaywrightFetcher Async support 2024-12-16 00:17:13 +02:00
Karim shoair af4f2c0f74 feat(PlaywrightFetcher): Add async support for PlaywrightFetcher 2024-12-16 00:15:47 +02:00
Karim shoair 889c111855 refactor(Playwright Engine): Separate what we can for cleaner code and the async function later 
The caching will give a slight performance increase with bulk requests
 2024-12-15 23:25:02 +02:00
Karim shoair 6c17bd886e style: using better data structures for constants 
This will cause a slight performance increase
 2024-12-15 23:19:47 +02:00
Karim shoair a90192df03 build: adding tests for AsyncFetcher class 2024-12-15 16:54:52 +02:00
Karim shoair b10cfd3643 feat: Adding AsyncFetcher class version of Fetcher 
The first step in fully supporting async
 2024-12-15 16:51:58 +02:00
Karim shoair faf728a794 style: moving repeated arguments from inside the functions to __init__ 
Might give a slight performance increase too
 2024-12-15 16:49:48 +02:00
Karim shoair 445af3c702 perf: Give repeated usage of Fetcher a slight performance increase 
By caching the `StaticEngine` class instance
 2024-12-15 16:13:08 +02:00
Karim shoair 299793af3f feat: adding the retries argument for all methods of Fetcher class 2024-12-15 15:59:15 +02:00
Karim shoair 838dd62ee4 build: pumping up camoufox 
For the issue with geoip
 2024-12-13 20:43:53 +02:00
Karim shoair f9bee4cbbb build: Pumping up camoufox version to solve browserforge issue 
Dropped browserforge from the requirements here so its version gets controlled by camoufox
 2024-12-12 14:37:31 +02:00
Karim shoair bfe906315e feat: adding geoip parameter to the StealthyFetcher 2024-12-12 12:43:49 +02:00
Karim shoair 6f87420e35 build: disable the 501 error test for playwright 2024-12-11 21:44:03 +02:00
Karim shoair e254341974 fix: forgot to stage it with last commit 2024-12-11 21:43:36 +02:00
Karim shoair 193827e27b refactor(api)!: Unifying log under 1 logger and removing debug parameter 
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
 2024-12-11 21:41:37 +02:00
Karim shoair f30eb6ab6c build: disable the 404 error test for playwright 2024-12-11 20:27:07 +02:00
Karim shoair dcf51876a5 fix: Adaptor.body returns raw HTML without processing 
If possible, otherwise returns `Adaptor.html_content`
 2024-12-11 20:18:05 +02:00
Karim shoair 9f0001af26 feat: logging for response status 2024-12-11 20:14:32 +02:00
Karim shoair e332444411 Update tests.yml 2024-12-10 23:35:21 +02:00
Karim shoair 70b54241e4 Update tox.ini 2024-12-10 23:32:54 +02:00
Karim shoair ba11c433ff Preparing to release 0.2.9 soon 2024-12-10 22:56:51 +02:00
Karim shoair e60a57c63f Python 3.8 is not supported anymore 
It's not supported in the new version of Playwright and is problematic in some situations while being slower than newer versions.
 2024-12-10 22:37:37 +02:00
Karim shoair 93ed768312 Bumping up the libraries versions for better stealth and speed! 2024-12-10 22:22:34 +02:00
Karim shoair b4f90615c3 Adding keep_cdata argument for Adaptor and Response classes 
This will force lxml to keep cdata while parsing html if you want
 2024-12-10 22:06:08 +02:00
Karim shoair 5f9c3980dc Adding urljoin method to Adaptors and Responses 2024-12-10 22:02:43 +02:00
Karim shoair d0f18951ac fix: Enable WebGL by default 2024-12-04 23:38:30 +02:00
Karim shoair 45e86f50a3 Adding empty methods (get/get_all/extract/extract_all) 
For easy copy-paste from Scrapy/parsel code when needed :)
 2024-12-04 14:28:37 +02:00
Karim shoair 31e838ce9b Make standard string methods return TextHandler again instead of str 2024-12-04 14:27:36 +02:00
Karim shoair 012820c635 Merge pull request #21 from D4Vinci/dev 
v0.2.8
 2024-11-30 18:16:06 +02:00
Karim shoair c481a1c8ad Pumping version up to 0.2.8 2024-11-30 18:13:19 +02:00
Karim shoair a3a4f1ff19 A cleanup script to use with development 2024-11-30 18:01:35 +02:00
Karim shoair f7916e504e Update tox.ini 2024-11-30 14:32:42 +02:00
Karim shoair 8321cfc180 Add vermin hook to check for minimum python version 2024-11-30 14:28:10 +02:00
Karim shoair b38c38b03b Forcing Tox to always use pytest config file 2024-11-30 14:14:59 +02:00
Karim shoair f81e3b3cc3 Turn off threaded testing for Python 3.8 on tox 
part 2 :)
 2024-11-30 14:08:18 +02:00
Karim shoair 1a17b2ce46 Sorting all library imports with isort 2024-11-30 14:02:06 +02:00
Karim shoair 26bfa88e50 Turn off threaded testing for Python 3.8 on tox 2024-11-30 14:01:30 +02:00
Karim shoair 4dddebbe24 Update .bandit.yml 2024-11-30 13:47:46 +02:00
Karim shoair 7608d86bf2 Pumping up camoufox version 2024-11-30 11:24:47 +02:00