freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
317 Commits 1 Branch 0 Tags
e9e606bca51e07b7d56bea6e86d225517d8e06fa
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair e9e606bca5 feat(StealthyFetcher): Disable Cross-Origin-Opener-Policy by default for async_fetch 
This allows elements in cross-origin iframes, such as the Turnstile checkbox, to be clicked.
.
Forgot to add it with the other commit
 2025-01-28 23:53:12 +02:00
Karim shoair 675c0110e6 feat(StealthyFetcher): Make ads enabled by default for performance boost 2025-01-28 23:51:31 +02:00
Karim shoair 52b66c98ec feat(StealthyFetcher): Disable Cross-Origin-Opener-Policy by default 
This allows elements in cross-origin iframes, such as the Turnstile checkbox, to be clicked.
 2025-01-28 23:47:47 +02:00
Karim shoair 1332be7e63 fix(autocompletion): Fix the defaults import 
It runs autocompletion on some IDEs
 2025-01-28 23:44:58 +02:00
Karim shoair c2447e069e build: Pumping version up 2025-01-28 23:41:48 +02:00
Karim shoair 99fe842bc9 build: updating dependencies 2025-01-28 23:40:56 +02:00
Karim shoair 21291958f1 docs(README file): Better more-friendly description 2025-01-28 23:10:56 +02:00
Karim shoair 198b76f5d5 Test: disable stealth parameter 
It causes issues with Github Actions
 2025-01-28 13:43:36 +02:00
Karim shoair 7776e191e8 Update test_playwright.py 2025-01-28 13:42:11 +02:00
Karim shoair 75a965e802 Update FUNDING.yml 2025-01-27 21:14:50 +02:00
Karim shoair 0e1dcd4f29 Adding Scrapeless sponsor 2025-01-06 11:59:19 +02:00
Karim shoair 32d96606ea Merge pull request #27 from D4Vinci/dev 
v0.2.92
 2024-12-26 20:03:24 +02:00
Karim shoair c5d83a11ad build: Pumping version up 2024-12-26 19:57:25 +02:00
Karim shoair ef284234e6 docs: update README file to reflect new installation instructions 2024-12-26 19:56:37 +02:00
Karim shoair 03b52aa74c feat(cli): adding terminal command scrapling install 
Instead of making users install browsers by themself
 2024-12-26 19:46:22 +02:00
Karim shoair 2006be2173 fix(StealthyFetcher): Use more dependable response data 2024-12-26 00:29:42 +02:00
Karim shoair f9b85cfbac fix(PlaywrightFetcher): Use more dependable response data 2024-12-26 00:27:13 +02:00
Karim shoair 3cca3fd56b style(parser): more accurate type hints 2024-12-21 16:17:02 +02:00
Karim shoair ee599141fe Merge pull request #26 from D4Vinci/dev 
v0.2.91
 2024-12-19 13:51:53 +02:00
Karim shoair bb398693b8 feat(Fetcher): Adding support to proxies using the SOCKS protocol 2024-12-19 13:30:43 +02:00
Karim shoair f74db648bc build: Pumping up the version to 0.2.91 2024-12-19 13:29:45 +02:00
Karim shoair 5be8c2a95e style(Fetchers): Setting correct type hint for wait_selector_state argument 2024-12-19 13:22:12 +02:00
Karim shoair 94745b1555 fix(StealthyFetcher): Wait for the real response status code 
This will return the real status response after redirection for websites that have Cloudflare instead of Playwright API's default behavior.
 2024-12-19 13:08:41 +02:00
Karim shoair 844cb0c6a2 fix(PlaywrightFetcher): Wait for the real response status code 
This will return the real status response after redirection for websites that have Cloudflare instead of Playwright's default behavior.
 2024-12-19 13:00:19 +02:00
Karim shoair 6652b8ac18 fix: solve the response log issue so now you get log for every response from Fetchers 2024-12-19 12:29:34 +02:00
Karim shoair 60df72cb23 Merge pull request #25 from D4Vinci/dev 
v0.2.9
 2024-12-16 14:42:13 +02:00
Karim shoair 838b8b1196 style: fixing a typo in logging 2024-12-16 13:39:30 +02:00
Karim shoair c282cb3cb7 fix: Stopped the log spamming that happens with multiple instances creation 2024-12-16 13:35:48 +02:00
Karim shoair ac95600650 docs: add an example that uses page.urljoin 2024-12-16 13:27:14 +02:00
Karim shoair 20ef45349a test: Rewrite automatch tests to a cleaner version and adding async test 2024-12-16 13:18:19 +02:00
Karim shoair 69e31612a5 test: Rewrite parser tests to a cleaner version and adding more tests 2024-12-16 13:17:09 +02:00
Karim shoair 3ff0d55c3e chore: Delete former-sponsor banner 2024-12-16 12:47:43 +02:00
Karim shoair 43b41b3733 docs: fix TOC header 2024-12-16 12:41:07 +02:00
Karim shoair 6cf5ce9719 docs: Adding async examples and fixing some typos 2024-12-16 12:39:44 +02:00
Karim shoair 05c6eeb387 fix: add AsyncFetcher to top-level shortcuts 2024-12-16 12:37:29 +02:00
Karim shoair d66ebc19a5 fix(Fetchers/disable_resources): Fixing the logic for intercepting resources 2024-12-16 12:35:25 +02:00
Karim shoair eceef48ef5 fix(Fetchers/page_action): Fixing logic 2024-12-16 12:34:23 +02:00
Karim shoair 3ecffcb81c style: Rewrite sync Fetchers tests to a cleaner version 2024-12-16 01:27:18 +02:00
Karim shoair 2606f7a12d test: clearer naming for async tests 2024-12-16 01:01:59 +02:00
Karim shoair 448587ff41 test: add tests for StealthyFetcher async fetch 2024-12-16 01:00:05 +02:00
Karim shoair 79a911e24e feat(StealthyFetcher): Add async fetch support 2024-12-16 00:59:19 +02:00
Karim shoair efb3270c67 style: Rewrite asyncFetcher tests to a cleaner version 2024-12-16 00:47:02 +02:00
Karim shoair 361ee440aa test: adding pytest-asyncio plugin to tests requirements file 2024-12-16 00:20:26 +02:00
Karim shoair de015f2b18 chore: Restructuring fetchers into cleaner structure 2024-12-16 00:19:48 +02:00
Karim shoair aac77e488c test: add test for PlaywrightFetcher Async support 2024-12-16 00:17:13 +02:00
Karim shoair af4f2c0f74 feat(PlaywrightFetcher): Add async support for PlaywrightFetcher 2024-12-16 00:15:47 +02:00
Karim shoair 889c111855 refactor(Playwright Engine): Separate what we can for cleaner code and the async function later 
The caching will give a slight performance increase with bulk requests
 2024-12-15 23:25:02 +02:00
Karim shoair 6c17bd886e style: using better data structures for constants 
This will cause a slight performance increase
 2024-12-15 23:19:47 +02:00
Karim shoair a90192df03 build: adding tests for AsyncFetcher class 2024-12-15 16:54:52 +02:00
Karim shoair b10cfd3643 feat: Adding AsyncFetcher class version of Fetcher 
The first step in fully supporting async
 2024-12-15 16:51:58 +02:00
Karim shoair faf728a794 style: moving repeated arguments from inside the functions to __init__ 
Might give a slight performance increase too
 2024-12-15 16:49:48 +02:00
Karim shoair 445af3c702 perf: Give repeated usage of Fetcher a slight performance increase 
By caching the `StaticEngine` class instance
 2024-12-15 16:13:08 +02:00
Karim shoair 299793af3f feat: adding the retries argument for all methods of Fetcher class 2024-12-15 15:59:15 +02:00
Karim shoair 838dd62ee4 build: pumping up camoufox 
For the issue with geoip
 2024-12-13 20:43:53 +02:00
Karim shoair f9bee4cbbb build: Pumping up camoufox version to solve browserforge issue 
Dropped browserforge from the requirements here so its version gets controlled by camoufox
 2024-12-12 14:37:31 +02:00
Karim shoair bfe906315e feat: adding geoip parameter to the StealthyFetcher 2024-12-12 12:43:49 +02:00
Karim shoair 6f87420e35 build: disable the 501 error test for playwright 2024-12-11 21:44:03 +02:00
Karim shoair e254341974 fix: forgot to stage it with last commit 2024-12-11 21:43:36 +02:00
Karim shoair 193827e27b refactor(api)!: Unifying log under 1 logger and removing debug parameter 
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
 2024-12-11 21:41:37 +02:00
Karim shoair f30eb6ab6c build: disable the 404 error test for playwright 2024-12-11 20:27:07 +02:00
Karim shoair dcf51876a5 fix: Adaptor.body returns raw HTML without processing 
If possible, otherwise returns `Adaptor.html_content`
 2024-12-11 20:18:05 +02:00
Karim shoair 9f0001af26 feat: logging for response status 2024-12-11 20:14:32 +02:00
Karim shoair e332444411 Update tests.yml 2024-12-10 23:35:21 +02:00
Karim shoair 70b54241e4 Update tox.ini 2024-12-10 23:32:54 +02:00
Karim shoair ba11c433ff Preparing to release 0.2.9 soon 2024-12-10 22:56:51 +02:00
Karim shoair e60a57c63f Python 3.8 is not supported anymore 
It's not supported in the new version of Playwright and is problematic in some situations while being slower than newer versions.
 2024-12-10 22:37:37 +02:00
Karim shoair 93ed768312 Bumping up the libraries versions for better stealth and speed! 2024-12-10 22:22:34 +02:00
Karim shoair b4f90615c3 Adding keep_cdata argument for Adaptor and Response classes 
This will force lxml to keep cdata while parsing html if you want
 2024-12-10 22:06:08 +02:00
Karim shoair 5f9c3980dc Adding urljoin method to Adaptors and Responses 2024-12-10 22:02:43 +02:00
Karim shoair d0f18951ac fix: Enable WebGL by default 2024-12-04 23:38:30 +02:00
Karim shoair 45e86f50a3 Adding empty methods (get/get_all/extract/extract_all) 
For easy copy-paste from Scrapy/parsel code when needed :)
 2024-12-04 14:28:37 +02:00
Karim shoair 31e838ce9b Make standard string methods return TextHandler again instead of str 2024-12-04 14:27:36 +02:00
Karim shoair 012820c635 Merge pull request #21 from D4Vinci/dev 
v0.2.8
 2024-11-30 18:16:06 +02:00
Karim shoair c481a1c8ad Pumping version up to 0.2.8 2024-11-30 18:13:19 +02:00
Karim shoair a3a4f1ff19 A cleanup script to use with development 2024-11-30 18:01:35 +02:00
Karim shoair f7916e504e Update tox.ini 2024-11-30 14:32:42 +02:00
Karim shoair 8321cfc180 Add vermin hook to check for minimum python version 2024-11-30 14:28:10 +02:00
Karim shoair b38c38b03b Forcing Tox to always use pytest config file 2024-11-30 14:14:59 +02:00
Karim shoair f81e3b3cc3 Turn off threaded testing for Python 3.8 on tox 
part 2 :)
 2024-11-30 14:08:18 +02:00
Karim shoair 1a17b2ce46 Sorting all library imports with isort 2024-11-30 14:02:06 +02:00
Karim shoair 26bfa88e50 Turn off threaded testing for Python 3.8 on tox 2024-11-30 14:01:30 +02:00
Karim shoair 4dddebbe24 Update .bandit.yml 2024-11-30 13:47:46 +02:00
Karim shoair 7608d86bf2 Pumping up camoufox version 2024-11-30 11:24:47 +02:00
Karim shoair 82f8e4a073 Fixing import typo, thanks to #20 2024-11-30 11:23:07 +02:00
Karim shoair 1cf7a1d205 Make Github tests workflow work on main and dev branches only 
The docs branch doesn't need tests
 2024-11-29 21:19:39 +02:00
Karim shoair 97fecc9d5f Style adjustments by flake8 2024-11-29 21:14:52 +02:00
Karim shoair d6cc07deaa Add scrapless banner 2024-11-29 11:26:48 +02:00
Karim shoair f8ad18ab8a Rephrasing some parts to be clearer 2024-11-27 11:50:20 +02:00
Karim shoair 26aebbace2 Merge pull request #18 from D4Vinci/dev 
v0.2.7
 2024-11-26 23:10:49 +02:00
Karim shoair 06a47f9644 Small fix for the ResponseEncoding class 2024-11-26 22:55:22 +02:00
Karim shoair 28b783e86b Improving the stealth mode a bit for Playwright 2024-11-26 22:41:04 +02:00
Karim shoair 14ffdbd6fc Handling problematic encodings 2024-11-26 22:27:05 +02:00
Karim shoair d3fbcab3df Pumping version up to 0.2.7 2024-11-26 21:27:22 +02:00
Karim shoair ccf38606d8 Making the README clearer a bit 2024-11-26 21:25:16 +02:00
Karim shoair dbd674aa7c Adding proxy support for basic requests 2024-11-26 21:24:41 +02:00
Karim shoair 1cf7cee1a2 Fetchers - Wait for page to be stable again if the user wanted to wait for a selector 2024-11-26 20:26:54 +02:00
Karim shoair d6a59774ea StealthyFetcher - The option to disable ads with ublock origin addon 2024-11-26 20:23:02 +02:00
Karim shoair 60d551d839 PlaywrightFetcher - Add the option to set locale of the browser 2024-11-26 17:39:03 +02:00
Karim shoair 0176bc0523 PlaywrightFetcher - Make proxies available without enabling stealth 
Thanks for @AbdullahY36
 2024-11-26 16:48:35 +02:00
Karim shoair 468d9b8f63 Update README.md 2024-11-25 12:28:06 +02:00