freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,052 Commits 1 Branch 0 Tags
09cd5e618307cb50b60993fa17f7955475ddeb37
Commit Graph

1052 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair 2006be2173 fix(StealthyFetcher): Use more dependable response data 2024-12-26 00:29:42 +02:00
Karim shoair f9b85cfbac fix(PlaywrightFetcher): Use more dependable response data 2024-12-26 00:27:13 +02:00
Karim shoair 3cca3fd56b style(parser): more accurate type hints 2024-12-21 16:17:02 +02:00
Karim shoair ee599141fe Merge pull request #26 from D4Vinci/dev 
v0.2.91
 2024-12-19 13:51:53 +02:00
Karim shoair bb398693b8 feat(Fetcher): Adding support to proxies using the SOCKS protocol 2024-12-19 13:30:43 +02:00
Karim shoair f74db648bc build: Pumping up the version to 0.2.91 2024-12-19 13:29:45 +02:00
Karim shoair 5be8c2a95e style(Fetchers): Setting correct type hint for wait_selector_state argument 2024-12-19 13:22:12 +02:00
Karim shoair 94745b1555 fix(StealthyFetcher): Wait for the real response status code 
This will return the real status response after redirection for websites that have Cloudflare instead of Playwright API's default behavior.
 2024-12-19 13:08:41 +02:00
Karim shoair 844cb0c6a2 fix(PlaywrightFetcher): Wait for the real response status code 
This will return the real status response after redirection for websites that have Cloudflare instead of Playwright's default behavior.
 2024-12-19 13:00:19 +02:00
Karim shoair 6652b8ac18 fix: solve the response log issue so now you get log for every response from Fetchers 2024-12-19 12:29:34 +02:00
Karim shoair 60df72cb23 Merge pull request #25 from D4Vinci/dev 
v0.2.9
 2024-12-16 14:42:13 +02:00
Karim shoair 838b8b1196 style: fixing a typo in logging 2024-12-16 13:39:30 +02:00
Karim shoair c282cb3cb7 fix: Stopped the log spamming that happens with multiple instances creation 2024-12-16 13:35:48 +02:00
Karim shoair ac95600650 docs: add an example that uses page.urljoin 2024-12-16 13:27:14 +02:00
Karim shoair 20ef45349a test: Rewrite automatch tests to a cleaner version and adding async test 2024-12-16 13:18:19 +02:00
Karim shoair 69e31612a5 test: Rewrite parser tests to a cleaner version and adding more tests 2024-12-16 13:17:09 +02:00
Karim shoair 3ff0d55c3e chore: Delete former-sponsor banner 2024-12-16 12:47:43 +02:00
Karim shoair 43b41b3733 docs: fix TOC header 2024-12-16 12:41:07 +02:00
Karim shoair 6cf5ce9719 docs: Adding async examples and fixing some typos 2024-12-16 12:39:44 +02:00
Karim shoair 05c6eeb387 fix: add AsyncFetcher to top-level shortcuts 2024-12-16 12:37:29 +02:00
Karim shoair d66ebc19a5 fix(Fetchers/disable_resources): Fixing the logic for intercepting resources 2024-12-16 12:35:25 +02:00
Karim shoair eceef48ef5 fix(Fetchers/page_action): Fixing logic 2024-12-16 12:34:23 +02:00
Karim shoair 3ecffcb81c style: Rewrite sync Fetchers tests to a cleaner version 2024-12-16 01:27:18 +02:00
Karim shoair 2606f7a12d test: clearer naming for async tests 2024-12-16 01:01:59 +02:00
Karim shoair 448587ff41 test: add tests for StealthyFetcher async fetch 2024-12-16 01:00:05 +02:00
Karim shoair 79a911e24e feat(StealthyFetcher): Add async fetch support 2024-12-16 00:59:19 +02:00
Karim shoair efb3270c67 style: Rewrite asyncFetcher tests to a cleaner version 2024-12-16 00:47:02 +02:00
Karim shoair 361ee440aa test: adding pytest-asyncio plugin to tests requirements file 2024-12-16 00:20:26 +02:00
Karim shoair de015f2b18 chore: Restructuring fetchers into cleaner structure 2024-12-16 00:19:48 +02:00
Karim shoair aac77e488c test: add test for PlaywrightFetcher Async support 2024-12-16 00:17:13 +02:00
Karim shoair af4f2c0f74 feat(PlaywrightFetcher): Add async support for PlaywrightFetcher 2024-12-16 00:15:47 +02:00
Karim shoair 889c111855 refactor(Playwright Engine): Separate what we can for cleaner code and the async function later 
The caching will give a slight performance increase with bulk requests
 2024-12-15 23:25:02 +02:00
Karim shoair 6c17bd886e style: using better data structures for constants 
This will cause a slight performance increase
 2024-12-15 23:19:47 +02:00
Karim shoair a90192df03 build: adding tests for AsyncFetcher class 2024-12-15 16:54:52 +02:00
Karim shoair b10cfd3643 feat: Adding AsyncFetcher class version of Fetcher 
The first step in fully supporting async
 2024-12-15 16:51:58 +02:00
Karim shoair faf728a794 style: moving repeated arguments from inside the functions to __init__ 
Might give a slight performance increase too
 2024-12-15 16:49:48 +02:00
Karim shoair 445af3c702 perf: Give repeated usage of Fetcher a slight performance increase 
By caching the `StaticEngine` class instance
 2024-12-15 16:13:08 +02:00
Karim shoair 299793af3f feat: adding the retries argument for all methods of Fetcher class 2024-12-15 15:59:15 +02:00
Karim shoair 838dd62ee4 build: pumping up camoufox 
For the issue with geoip
 2024-12-13 20:43:53 +02:00
Karim shoair f9bee4cbbb build: Pumping up camoufox version to solve browserforge issue 
Dropped browserforge from the requirements here so its version gets controlled by camoufox
 2024-12-12 14:37:31 +02:00
Karim shoair bfe906315e feat: adding geoip parameter to the StealthyFetcher 2024-12-12 12:43:49 +02:00
Karim shoair 6f87420e35 build: disable the 501 error test for playwright 2024-12-11 21:44:03 +02:00
Karim shoair e254341974 fix: forgot to stage it with last commit 2024-12-11 21:43:36 +02:00
Karim shoair 193827e27b refactor(api)!: Unifying log under 1 logger and removing debug parameter 
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
 2024-12-11 21:41:37 +02:00
Karim shoair f30eb6ab6c build: disable the 404 error test for playwright 2024-12-11 20:27:07 +02:00
Karim shoair dcf51876a5 fix: Adaptor.body returns raw HTML without processing 
If possible, otherwise returns `Adaptor.html_content`
 2024-12-11 20:18:05 +02:00
Karim shoair 9f0001af26 feat: logging for response status 2024-12-11 20:14:32 +02:00
Karim shoair e332444411 Update tests.yml 2024-12-10 23:35:21 +02:00
Karim shoair 70b54241e4 Update tox.ini 2024-12-10 23:32:54 +02:00
Karim shoair ba11c433ff Preparing to release 0.2.9 soon 2024-12-10 22:56:51 +02:00