freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
272 Commits 1 Branch 0 Tags
af4f2c0f7486329fae0563c1a10896b5dc0dadbe
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair af4f2c0f74 feat(PlaywrightFetcher): Add async support for PlaywrightFetcher 2024-12-16 00:15:47 +02:00
Karim shoair 889c111855 refactor(Playwright Engine): Separate what we can for cleaner code and the async function later 
The caching will give a slight performance increase with bulk requests
 2024-12-15 23:25:02 +02:00
Karim shoair 6c17bd886e style: using better data structures for constants 
This will cause a slight performance increase
 2024-12-15 23:19:47 +02:00
Karim shoair a90192df03 build: adding tests for AsyncFetcher class 2024-12-15 16:54:52 +02:00
Karim shoair b10cfd3643 feat: Adding AsyncFetcher class version of Fetcher 
The first step in fully supporting async
 2024-12-15 16:51:58 +02:00
Karim shoair faf728a794 style: moving repeated arguments from inside the functions to __init__ 
Might give a slight performance increase too
 2024-12-15 16:49:48 +02:00
Karim shoair 445af3c702 perf: Give repeated usage of Fetcher a slight performance increase 
By caching the `StaticEngine` class instance
 2024-12-15 16:13:08 +02:00
Karim shoair 299793af3f feat: adding the retries argument for all methods of Fetcher class 2024-12-15 15:59:15 +02:00
Karim shoair 838dd62ee4 build: pumping up camoufox 
For the issue with geoip
 2024-12-13 20:43:53 +02:00
Karim shoair f9bee4cbbb build: Pumping up camoufox version to solve browserforge issue 
Dropped browserforge from the requirements here so its version gets controlled by camoufox
 2024-12-12 14:37:31 +02:00
Karim shoair bfe906315e feat: adding geoip parameter to the StealthyFetcher 2024-12-12 12:43:49 +02:00
Karim shoair 6f87420e35 build: disable the 501 error test for playwright 2024-12-11 21:44:03 +02:00
Karim shoair e254341974 fix: forgot to stage it with last commit 2024-12-11 21:43:36 +02:00
Karim shoair 193827e27b refactor(api)!: Unifying log under 1 logger and removing debug parameter 
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
 2024-12-11 21:41:37 +02:00
Karim shoair f30eb6ab6c build: disable the 404 error test for playwright 2024-12-11 20:27:07 +02:00
Karim shoair dcf51876a5 fix: Adaptor.body returns raw HTML without processing 
If possible, otherwise returns `Adaptor.html_content`
 2024-12-11 20:18:05 +02:00
Karim shoair 9f0001af26 feat: logging for response status 2024-12-11 20:14:32 +02:00
Karim shoair e332444411 Update tests.yml 2024-12-10 23:35:21 +02:00
Karim shoair 70b54241e4 Update tox.ini 2024-12-10 23:32:54 +02:00
Karim shoair ba11c433ff Preparing to release 0.2.9 soon 2024-12-10 22:56:51 +02:00
Karim shoair e60a57c63f Python 3.8 is not supported anymore 
It's not supported in the new version of Playwright and is problematic in some situations while being slower than newer versions.
 2024-12-10 22:37:37 +02:00
Karim shoair 93ed768312 Bumping up the libraries versions for better stealth and speed! 2024-12-10 22:22:34 +02:00
Karim shoair b4f90615c3 Adding keep_cdata argument for Adaptor and Response classes 
This will force lxml to keep cdata while parsing html if you want
 2024-12-10 22:06:08 +02:00
Karim shoair 5f9c3980dc Adding urljoin method to Adaptors and Responses 2024-12-10 22:02:43 +02:00
Karim shoair d0f18951ac fix: Enable WebGL by default 2024-12-04 23:38:30 +02:00
Karim shoair 45e86f50a3 Adding empty methods (get/get_all/extract/extract_all) 
For easy copy-paste from Scrapy/parsel code when needed :)
 2024-12-04 14:28:37 +02:00
Karim shoair 31e838ce9b Make standard string methods return TextHandler again instead of str 2024-12-04 14:27:36 +02:00
Karim shoair 012820c635 Merge pull request #21 from D4Vinci/dev 
v0.2.8
 2024-11-30 18:16:06 +02:00
Karim shoair c481a1c8ad Pumping version up to 0.2.8 2024-11-30 18:13:19 +02:00
Karim shoair a3a4f1ff19 A cleanup script to use with development 2024-11-30 18:01:35 +02:00
Karim shoair f7916e504e Update tox.ini 2024-11-30 14:32:42 +02:00
Karim shoair 8321cfc180 Add vermin hook to check for minimum python version 2024-11-30 14:28:10 +02:00
Karim shoair b38c38b03b Forcing Tox to always use pytest config file 2024-11-30 14:14:59 +02:00
Karim shoair f81e3b3cc3 Turn off threaded testing for Python 3.8 on tox 
part 2 :)
 2024-11-30 14:08:18 +02:00
Karim shoair 1a17b2ce46 Sorting all library imports with isort 2024-11-30 14:02:06 +02:00
Karim shoair 26bfa88e50 Turn off threaded testing for Python 3.8 on tox 2024-11-30 14:01:30 +02:00
Karim shoair 4dddebbe24 Update .bandit.yml 2024-11-30 13:47:46 +02:00
Karim shoair 7608d86bf2 Pumping up camoufox version 2024-11-30 11:24:47 +02:00
Karim shoair 82f8e4a073 Fixing import typo, thanks to #20 2024-11-30 11:23:07 +02:00
Karim shoair 1cf7a1d205 Make Github tests workflow work on main and dev branches only 
The docs branch doesn't need tests
 2024-11-29 21:19:39 +02:00
Karim shoair 97fecc9d5f Style adjustments by flake8 2024-11-29 21:14:52 +02:00
Karim shoair d6cc07deaa Add scrapless banner 2024-11-29 11:26:48 +02:00
Karim shoair f8ad18ab8a Rephrasing some parts to be clearer 2024-11-27 11:50:20 +02:00
Karim shoair 26aebbace2 Merge pull request #18 from D4Vinci/dev 
v0.2.7
 2024-11-26 23:10:49 +02:00
Karim shoair 06a47f9644 Small fix for the ResponseEncoding class 2024-11-26 22:55:22 +02:00
Karim shoair 28b783e86b Improving the stealth mode a bit for Playwright 2024-11-26 22:41:04 +02:00
Karim shoair 14ffdbd6fc Handling problematic encodings 2024-11-26 22:27:05 +02:00
Karim shoair d3fbcab3df Pumping version up to 0.2.7 2024-11-26 21:27:22 +02:00
Karim shoair ccf38606d8 Making the README clearer a bit 2024-11-26 21:25:16 +02:00
Karim shoair dbd674aa7c Adding proxy support for basic requests 2024-11-26 21:24:41 +02:00
Karim shoair 1cf7cee1a2 Fetchers - Wait for page to be stable again if the user wanted to wait for a selector 2024-11-26 20:26:54 +02:00
Karim shoair d6a59774ea StealthyFetcher - The option to disable ads with ublock origin addon 2024-11-26 20:23:02 +02:00
Karim shoair 60d551d839 PlaywrightFetcher - Add the option to set locale of the browser 2024-11-26 17:39:03 +02:00
Karim shoair 0176bc0523 PlaywrightFetcher - Make proxies available without enabling stealth 
Thanks for @AbdullahY36
 2024-11-26 16:48:35 +02:00
Karim shoair 468d9b8f63 Update README.md 2024-11-25 12:28:06 +02:00
Karim shoair 1ab0f74cce Updating SerpApi Logo 2024-11-25 12:26:19 +02:00
Karim shoair 109933c04a Merge pull request #17 from D4Vinci/dev 
Update SerpApi.png
 2024-11-25 11:47:38 +02:00
Karim shoair b3817c5b3a Update SerpApi.png 2024-11-25 11:46:13 +02:00
Karim shoair cdc7ed8fa8 Removing CapSolver 2024-11-25 11:26:42 +02:00
Karim shoair bbbc97aef9 Merge pull request #16 from D4Vinci/dev 
v0.2.6
 2024-11-24 15:36:28 +02:00
Karim shoair 438867e0be Pumping the version up to 0.2.6 2024-11-24 15:21:58 +02:00
Karim shoair 12d638ac2e Updating the doc to reflect new changes 2024-11-24 15:20:18 +02:00
Karim shoair c1429074a7 Setting the compatible version of playwright with cdp patches 2024-11-24 15:15:58 +02:00
Karim shoair a8c576ad28 Adding the real_chrome option to the fetcher 2024-11-24 15:15:08 +02:00
Karim shoair e96e7d4440 Adding the option to use real chrome browser directly 2024-11-24 15:14:42 +02:00
Karim shoair 088117441d Making the type hints more accurate before update 2024-11-24 13:49:31 +02:00
Karim shoair 15b7e3acee Stop using generated headers by browserforge 
It turns out its headers format is pretty outdated, which made Google reCaptcha not work sometimes
 2024-11-24 13:47:21 +02:00
Karim shoair 7f330efc52 Turn-off the hide_canvas option by default 2024-11-24 13:44:16 +02:00
Karim shoair 58151c12e8 Update the minimum browser version for headers generated 2024-11-24 13:41:06 +02:00
Karim shoair e94c5034e3 Merge pull request #15 from D4Vinci/docs 
Doc adjustments to use Sphinx soon
 2024-11-23 17:52:12 +02:00
Karim shoair 1929d9bebb Fix for docs 2024-11-23 17:48:05 +02:00
Karim shoair e51556f80c Merge branch 'main' into docs 2024-11-23 16:48:22 +02:00
Karim shoair 4cea8c9926 Merge pull request #14 from D4Vinci/dev 
v0.2.5
 2024-11-23 16:39:26 +02:00
Karim shoair 0e01c1cc03 Update test_general.py 2024-11-23 16:24:25 +02:00
Karim shoair 34b645c9a5 Fix for playwright tests on GitHub actions 2024-11-23 16:19:50 +02:00
Karim shoair e7f2a83e16 Increasing the time a little bit for the performance test 
It's prolonged on GitHub Actions for obvious reasons
 2024-11-23 16:12:00 +02:00
Karim shoair 2b925b4a33 Pumping version up to 0.2.5 2024-11-23 16:06:11 +02:00
Karim shoair 454b79e00f Make tests run in parallel on GitHub actions 2024-11-23 16:04:47 +02:00
Karim shoair 723a545061 Fixing the "Unknown Status Code" test 2024-11-23 16:01:51 +02:00
Karim shoair 7017823136 Covering more parts in Playwright tests 2024-11-23 15:57:59 +02:00
Karim shoair 4d9a494e35 Covering more parts in StealthyFetcher tests 2024-11-23 15:55:50 +02:00
Karim shoair f33ff247dd Adding tests for response encoding and response status text 2024-11-23 15:55:21 +02:00
Karim shoair 073ab2c7e3 Universally handling different response encoding and charsets for all responses 
Thanks for @andyfcx
 2024-11-23 15:40:41 +02:00
Karim shoair 869412761a Fix for the wait_selector argument resolving to more than 1 element 2024-11-22 22:15:52 +02:00
Karim shoair 18eb96adb1 Merge pull request #13 from andyfcx/main 
fix: make sure to default utf-8 as encoding and strip unwanted single or double quotes
 2024-11-21 11:21:45 +02:00
Karim shoair 3f39bd52d4 Merge branch 'main' into docs 2024-11-20 14:36:30 +02:00
Karim shoair e9b0102d55 Merge pull request #11 from D4Vinci/dev 
v0.2.4
 2024-11-20 13:35:06 +02:00
Karim shoair 17d7934484 Pump version to 0.2.4 2024-11-20 13:27:39 +02:00
Karim shoair 19ad82caef Calculate status text manually if it's not returned by PlayWright API 2024-11-20 13:27:18 +02:00
Karim shoair 4c74d9bc97 Fixing a bug with reading response bytes in playwright/camoufox when network_idle is used 
PlayWright doesn't provide a way to get the response in bytes after all wait ends like `page.content()` so that's more efficient to do anyway
 2024-11-20 12:33:54 +02:00
Karim shoair 14738030e0 Merge pull request #10 from D4Vinci/dev 
v0.2.3
 2024-11-20 01:48:17 +02:00
Karim shoair 649f4cfae7 Pumping up camoufox version 2024-11-20 01:41:46 +02:00
Karim shoair 526c516693 Pump version up to 0.2.3 2024-11-20 01:28:01 +02:00
Karim shoair cb3ab4a011 Merge branch 'main' into dev 2024-11-19 20:48:05 +02:00
Karim shoair b69cc7c983 Fix for the stealth mode on playwright fetcher 
The JS files weren't being moved while installing with PIP!
 2024-11-19 14:35:24 +02:00
Karim shoair f443d774c1 Fixing the doc-string to match Sphinx 2024-11-19 14:33:54 +02:00
Karim shoair 50cd40cc42 Merge pull request #9 from D4Vinci/dev 
v0.2.2
 2024-11-16 22:04:03 +02:00
Karim shoair 0c6e7708d1 Update README.md 2024-11-16 21:55:53 +02:00
Karim shoair d8a8af8d05 Updating the README to reflect new changes 2024-11-16 21:29:05 +02:00
Karim shoair b60ac7f56a Change debug mode to be off by default 2024-11-16 21:11:58 +02:00