freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
212 Commits 1 Branch 0 Tags
438867e0be8e3aebd50cd6a9c2f8e2eea26446d0
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair 438867e0be Pumping the version up to 0.2.6 2024-11-24 15:21:58 +02:00
Karim shoair 12d638ac2e Updating the doc to reflect new changes 2024-11-24 15:20:18 +02:00
Karim shoair c1429074a7 Setting the compatible version of playwright with cdp patches 2024-11-24 15:15:58 +02:00
Karim shoair a8c576ad28 Adding the real_chrome option to the fetcher 2024-11-24 15:15:08 +02:00
Karim shoair e96e7d4440 Adding the option to use real chrome browser directly 2024-11-24 15:14:42 +02:00
Karim shoair 088117441d Making the type hints more accurate before update 2024-11-24 13:49:31 +02:00
Karim shoair 15b7e3acee Stop using generated headers by browserforge 
It turns out its headers format is pretty outdated, which made Google reCaptcha not work sometimes
 2024-11-24 13:47:21 +02:00
Karim shoair 7f330efc52 Turn-off the hide_canvas option by default 2024-11-24 13:44:16 +02:00
Karim shoair 58151c12e8 Update the minimum browser version for headers generated 2024-11-24 13:41:06 +02:00
Karim shoair e94c5034e3 Merge pull request #15 from D4Vinci/docs 
Doc adjustments to use Sphinx soon
 2024-11-23 17:52:12 +02:00
Karim shoair 1929d9bebb Fix for docs 2024-11-23 17:48:05 +02:00
Karim shoair e51556f80c Merge branch 'main' into docs 2024-11-23 16:48:22 +02:00
Karim shoair 4cea8c9926 Merge pull request #14 from D4Vinci/dev 
v0.2.5
 2024-11-23 16:39:26 +02:00
Karim shoair 0e01c1cc03 Update test_general.py 2024-11-23 16:24:25 +02:00
Karim shoair 34b645c9a5 Fix for playwright tests on GitHub actions 2024-11-23 16:19:50 +02:00
Karim shoair e7f2a83e16 Increasing the time a little bit for the performance test 
It's prolonged on GitHub Actions for obvious reasons
 2024-11-23 16:12:00 +02:00
Karim shoair 2b925b4a33 Pumping version up to 0.2.5 2024-11-23 16:06:11 +02:00
Karim shoair 454b79e00f Make tests run in parallel on GitHub actions 2024-11-23 16:04:47 +02:00
Karim shoair 723a545061 Fixing the "Unknown Status Code" test 2024-11-23 16:01:51 +02:00
Karim shoair 7017823136 Covering more parts in Playwright tests 2024-11-23 15:57:59 +02:00
Karim shoair 4d9a494e35 Covering more parts in StealthyFetcher tests 2024-11-23 15:55:50 +02:00
Karim shoair f33ff247dd Adding tests for response encoding and response status text 2024-11-23 15:55:21 +02:00
Karim shoair 073ab2c7e3 Universally handling different response encoding and charsets for all responses 
Thanks for @andyfcx
 2024-11-23 15:40:41 +02:00
Karim shoair 869412761a Fix for the wait_selector argument resolving to more than 1 element 2024-11-22 22:15:52 +02:00
Karim shoair 18eb96adb1 Merge pull request #13 from andyfcx/main 
fix: make sure to default utf-8 as encoding and strip unwanted single or double quotes
 2024-11-21 11:21:45 +02:00
Karim shoair 3f39bd52d4 Merge branch 'main' into docs 2024-11-20 14:36:30 +02:00
Karim shoair e9b0102d55 Merge pull request #11 from D4Vinci/dev 
v0.2.4
 2024-11-20 13:35:06 +02:00
Karim shoair 17d7934484 Pump version to 0.2.4 2024-11-20 13:27:39 +02:00
Karim shoair 19ad82caef Calculate status text manually if it's not returned by PlayWright API 2024-11-20 13:27:18 +02:00
Karim shoair 4c74d9bc97 Fixing a bug with reading response bytes in playwright/camoufox when network_idle is used 
PlayWright doesn't provide a way to get the response in bytes after all wait ends like `page.content()` so that's more efficient to do anyway
 2024-11-20 12:33:54 +02:00
Karim shoair 14738030e0 Merge pull request #10 from D4Vinci/dev 
v0.2.3
 2024-11-20 01:48:17 +02:00
Karim shoair 649f4cfae7 Pumping up camoufox version 2024-11-20 01:41:46 +02:00
Karim shoair 526c516693 Pump version up to 0.2.3 2024-11-20 01:28:01 +02:00
Karim shoair cb3ab4a011 Merge branch 'main' into dev 2024-11-19 20:48:05 +02:00
Karim shoair b69cc7c983 Fix for the stealth mode on playwright fetcher 
The JS files weren't being moved while installing with PIP!
 2024-11-19 14:35:24 +02:00
Karim shoair f443d774c1 Fixing the doc-string to match Sphinx 2024-11-19 14:33:54 +02:00
Karim shoair 50cd40cc42 Merge pull request #9 from D4Vinci/dev 
v0.2.2
 2024-11-16 22:04:03 +02:00
Karim shoair 0c6e7708d1 Update README.md 2024-11-16 21:55:53 +02:00
Karim shoair d8a8af8d05 Updating the README to reflect new changes 2024-11-16 21:29:05 +02:00
Karim shoair b60ac7f56a Change debug mode to be off by default 2024-11-16 21:11:58 +02:00
Karim shoair 422c135aae Adding a file import initialized fetchers for cleaner looking code 
and lazy people like me :)
 2024-11-16 21:10:26 +02:00
Karim shoair cefc95cc3d Pump version up to 0.2.2 2024-11-16 13:57:33 +02:00
Karim shoair 572df6b3b4 Fixing the way Response object handles sub items in some edge cases 
+ Do it with the least impact on performance
 2024-11-16 13:54:55 +02:00
Karim shoair 105ec5b067 Update README.md 2024-11-15 23:18:02 +02:00
Karim shoair e2a64a0707 Adding SerpApi as a sponsor 2024-11-15 23:07:59 +02:00
Karim shoair 773fcd59bf Merge pull request #8 from D4Vinci/dev 
v0.2.1
 2024-11-15 18:25:14 +02:00
Karim shoair 90e38af552 Correcting the return string for the Response object in all functions/methods 2024-11-15 16:44:47 +02:00
Karim shoair 5e8275c3eb Adding the option to randomize the OS fingerprints with the StealthyFetcher 2024-11-15 16:36:32 +02:00
Karim shoair 2abb70218f Adding the proxy support to browser-based Fetchers 2024-11-15 14:43:03 +02:00
Karim shoair b993adb4a7 Rephrasing some parts 2024-11-15 11:26:41 +02:00
Karim shoair ae7959b4a4 Update README.md 2024-11-15 02:50:24 +02:00
Karim shoair 7dd9d3090d Pumping the version up to 0.2.1 2024-11-15 02:43:29 +02:00
Karim shoair c334755377 Reflecting the new changes in the README and adding small notes 2024-11-15 02:33:47 +02:00
Karim shoair 095d50cd75 Making the Response returned from all fetchers same as Adaptor object 
but with added attributes
 2024-11-15 02:23:14 +02:00
Karim shoair b0c2b1818b Return to the default text behaviour 
The logic to remove comment tags before returning text was causing errors sometimes so from now on leave `keep_comments` set to True better as it is.
 2024-11-14 22:21:45 +02:00
Karim shoair ea81e0284d Update tests Github action 2024-11-14 16:33:28 +02:00
Karim shoair 81f4e18a86 Merge remote-tracking branch 'origin/main' into dev 2024-11-14 16:30:41 +02:00
Karim shoair 9e337251ca Update README.md 2024-11-14 16:09:05 +02:00
Karim shoair e7e9cf8b3d Adding Evomi as a sponsor 2024-11-14 16:05:37 +02:00
Karim shoair b9c64e8dbf Update and rename README.md to Adding Evomi as a sponsor 2024-11-14 16:04:48 +02:00
Karim shoair eaa7da27c6 Better logic to handle json responses 2024-11-14 00:03:26 +02:00
Karim shoair d4b896b4e9 Parser - New logic to handle JSON responses passed from Fetchers 2024-11-13 23:14:27 +02:00
Karim shoair ab7b0d45ad StaticEngine - Fix the headers bug 2024-11-13 23:13:39 +02:00
Karim shoair 4e5f7d385a Merge pull request #5 from D4Vinci/main 
Keeping `dev` up to date with `main` branch
 2024-11-13 22:51:29 +02:00
Karim shoair 2b3c9b8ff3 Merge pull request #4 from marksmayo/main 
Tiny change - py.typed had a space at the end of the file name, which was causing git commands to fail with invalid path
 2024-11-13 11:18:53 +02:00
Karim shoair 4b5f9e12a2 Merge pull request #3 from D4Vinci/dev 
Releasing version 0.2 (BIG CHANGES)
 2024-11-11 19:29:41 +02:00
Karim shoair 64c9fe905e Corrections and rephrasing 2024-11-11 16:40:54 +02:00
Karim shoair bd38fdb347 Corrections and rephrasing 2024-11-11 16:40:46 +02:00
Karim shoair b3418f7f50 Corrections and rephrasing 2024-11-11 16:40:35 +02:00
Karim shoair d6b454dc43 Corrections and rephrasing 2024-11-11 16:40:30 +02:00
Karim shoair eb100e45f7 Corrections and rephrasing 2024-11-11 16:40:21 +02:00
Karim shoair 79bf4060f4 Updating find_all examples 2024-11-11 12:51:40 +02:00
Karim shoair e8935b1af8 Update README.md 2024-11-10 23:45:48 +02:00
Karim shoair 3d4b427168 Rework Selectors generation 
- Renamed both `css_selector` and `xpath_selector` both to `generate_css_selector` and `generate_xpath_selector` for clarity and not interrupting the auto-completion while coding.
- Added `generate_full_css_selector` and `generate_full_xpath_selector` methods
 2024-11-10 18:19:48 +02:00
Karim shoair de2d65856c Adding playwright/camoufox tests for cookies and improve the timeout test 2024-11-10 16:31:28 +02:00
Karim shoair 239e2c73bf Correctly get page content after JS Execution 
- Thanks for @AbdullahY36 for the heads-up.

**Update reference** https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event
 2024-11-10 16:11:46 +02:00
Karim shoair 43045e3eb2 Get page content after JS Execution 
Thanks for @AbdullahY36
 2024-11-10 15:54:44 +02:00
Karim shoair 7d2f55bd4c First version of the 0.2 README 
I probably forgot some features and made some mistakes here and there so I will review it again later
 2024-11-10 00:09:30 +02:00
Karim shoair 4b2814404a Fixing the functions-based filtering logic 2024-11-09 22:32:05 +02:00
Karim shoair 6b62579a7e Making find_all/find methods on steroids 
Adding the ability to find elements by regex patterns and functions.
 2024-11-09 21:25:17 +02:00
Karim shoair b90c8adb02 Adding the argument automatch_domain to all fetchers 2024-11-09 15:38:27 +02:00
Karim shoair 03091beb8b Small clarification 2024-11-09 13:40:50 +02:00
Karim shoair fd7d787ab0 Add extra_headers argument to PW and isolate google referer feature from stealth mode as google_search 2024-11-09 00:54:11 +02:00
Karim shoair ee8f78feb1 Update camo.py 2024-11-09 00:48:11 +02:00
Karim shoair fd34cd9afa Adding arguments google_search and extra_headers to StealthyFetcher 2024-11-09 00:43:24 +02:00
Karim shoair 530b54d964 Clearer doc-string 2024-11-09 00:17:41 +02:00
Karim shoair 1b015bbd07 Type correction 2024-11-08 23:59:12 +02:00
Karim shoair b919536527 Updating the keywords whitelisting logic in find/find_all functions 2024-11-07 12:56:32 +02:00
Karim shoair 51ce40a561 Update ROADMAP.md 2024-11-07 00:44:23 +02:00
Karim shoair 91f6b46e6d Adding new filter and search functions to parser 2024-11-07 00:31:14 +02:00
Karim shoair 846a185b49 Updating parser tests to use find and css_first functions 2024-11-07 00:02:36 +02:00
Karim shoair b25d2ee79b Adding new find_all and find functions to parser 2024-11-07 00:01:14 +02:00
Karim shoair 6b40af426f Better logic for css_first and xpath_first 2024-11-06 22:14:33 +02:00
Karim shoair 5e848ef046 Adding new class type TextHandlers 2024-11-06 21:03:19 +02:00
Karim shoair 7a84511b7a Making Adaptor and Adaptor re/re_first arguments consistent with the TextHandler ones 2024-11-06 20:47:21 +02:00
Karim shoair 7af75f78a5 Better logic for Adaptors re_first method 2024-11-06 20:24:30 +02:00
Karim shoair a1fd5ddc7e Correcting headless argument type 2024-11-06 20:18:44 +02:00
Karim shoair a0a3e62075 Adding addons and humanize arguments to StealthyFetcher 2024-11-06 20:14:42 +02:00
Karim shoair 5ad2126030 Small speed boost for fetching by some caching 2024-11-06 19:56:39 +02:00
Karim shoair a1513ba769 Update tests.yml 2024-11-06 12:54:52 +02:00