freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
134 Commits 1 Branch 0 Tags
239e2c73bfe458f42b11f35cfb9ed9fe1d3a4479
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair 239e2c73bf Correctly get page content after JS Execution 
- Thanks for @AbdullahY36 for the heads-up.

**Update reference** https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event
 2024-11-10 16:11:46 +02:00
Karim shoair 43045e3eb2 Get page content after JS Execution 
Thanks for @AbdullahY36
 2024-11-10 15:54:44 +02:00
Karim shoair 7d2f55bd4c First version of the 0.2 README 
I probably forgot some features and made some mistakes here and there so I will review it again later
 2024-11-10 00:09:30 +02:00
Karim shoair 4b2814404a Fixing the functions-based filtering logic 2024-11-09 22:32:05 +02:00
Karim shoair 6b62579a7e Making find_all/find methods on steroids 
Adding the ability to find elements by regex patterns and functions.
 2024-11-09 21:25:17 +02:00
Karim shoair b90c8adb02 Adding the argument automatch_domain to all fetchers 2024-11-09 15:38:27 +02:00
Karim shoair 03091beb8b Small clarification 2024-11-09 13:40:50 +02:00
Karim shoair fd7d787ab0 Add extra_headers argument to PW and isolate google referer feature from stealth mode as google_search 2024-11-09 00:54:11 +02:00
Karim shoair ee8f78feb1 Update camo.py 2024-11-09 00:48:11 +02:00
Karim shoair fd34cd9afa Adding arguments google_search and extra_headers to StealthyFetcher 2024-11-09 00:43:24 +02:00
Karim shoair 530b54d964 Clearer doc-string 2024-11-09 00:17:41 +02:00
Karim shoair 1b015bbd07 Type correction 2024-11-08 23:59:12 +02:00
Karim shoair b919536527 Updating the keywords whitelisting logic in find/find_all functions 2024-11-07 12:56:32 +02:00
Karim shoair 51ce40a561 Update ROADMAP.md 2024-11-07 00:44:23 +02:00
Karim shoair 91f6b46e6d Adding new filter and search functions to parser 2024-11-07 00:31:14 +02:00
Karim shoair 846a185b49 Updating parser tests to use find and css_first functions 2024-11-07 00:02:36 +02:00
Karim shoair b25d2ee79b Adding new find_all and find functions to parser 2024-11-07 00:01:14 +02:00
Karim shoair 6b40af426f Better logic for css_first and xpath_first 2024-11-06 22:14:33 +02:00
Karim shoair 5e848ef046 Adding new class type TextHandlers 2024-11-06 21:03:19 +02:00
Karim shoair 7a84511b7a Making Adaptor and Adaptor re/re_first arguments consistent with the TextHandler ones 2024-11-06 20:47:21 +02:00
Karim shoair 7af75f78a5 Better logic for Adaptors re_first method 2024-11-06 20:24:30 +02:00
Karim shoair a1fd5ddc7e Correcting headless argument type 2024-11-06 20:18:44 +02:00
Karim shoair a0a3e62075 Adding addons and humanize arguments to StealthyFetcher 2024-11-06 20:14:42 +02:00
Karim shoair 5ad2126030 Small speed boost for fetching by some caching 2024-11-06 19:56:39 +02:00
Karim shoair a1513ba769 Update tests.yml 2024-11-06 12:54:52 +02:00
Karim shoair b542b5cc39 Recaching TOX environments 2024-11-06 12:50:14 +02:00
Karim shoair c30853ee2c Update tox.ini 2024-11-06 12:39:38 +02:00
Karim shoair fc30cfed4c Update tests.yml 2024-11-06 12:36:46 +02:00
Karim shoair cd1ff846a7 Update ROADMAP.md 2024-11-06 12:31:57 +02:00
Karim shoair d632bb14af Moving parser tests to a better structure 
and covering a bit more parts
 2024-11-06 12:30:41 +02:00
Karim shoair 5548941cf1 Correcting return type for css_first and xpath_first 2024-11-06 12:28:43 +02:00
Karim shoair e4eb442709 Adding tests for PlayWrightFetcher class 2024-11-06 12:21:55 +02:00
Karim shoair 4222626ec1 Adding tests for Fetcher class 2024-11-06 12:21:18 +02:00
Karim shoair 4400363146 Renaming fetchers tests folder for readability 2024-11-06 12:21:06 +02:00
Karim shoair 9a7f489eb7 make tests verbose for easier debugging later 2024-11-06 12:20:05 +02:00
Karim shoair 10bcdfb596 Moving bypasses folder deeper 2024-11-06 12:19:26 +02:00
Karim shoair 6cf506cb22 PlaywrightEngine - Validate CDP URLs in all cases 2024-11-06 11:24:56 +02:00
Karim shoair 6497b1b559 Type correction 2024-11-06 02:20:33 +02:00
Karim shoair 24806aecdc Update .gitignore 2024-11-05 23:41:57 +02:00
Karim shoair 3622040440 Adding css_first and xpath_first for easier usage 2024-11-05 23:41:51 +02:00
Karim shoair 009c2914aa Fixing re_first logic in Adaptors type 2024-11-05 22:59:47 +02:00
Karim shoair d50497dff3 Update .gitignore 2024-11-05 22:56:57 +02:00
Karim shoair ca4b5f205c Since tests can finally now run in Github actions without issues 
Let's miss it up again lol
 2024-11-05 13:10:51 +02:00
Karim shoair 2a275c436c Update tests.yml 2024-11-05 13:02:11 +02:00
Karim shoair 69cfbe45be Update tests.yml 2024-11-05 12:59:39 +02:00
Karim shoair 6f91d06b46 Update tests.yml 2024-11-05 12:57:57 +02:00
Karim shoair 2020214784 please work 2024-11-05 11:45:59 +02:00
Karim shoair 92d002032d Update tests.yml 2024-11-05 11:44:10 +02:00
Karim shoair 1bbb5dd75a Update tests.yml 2024-11-05 11:41:58 +02:00
Karim shoair d4358d32a8 Update tests.yml 2024-11-05 11:38:38 +02:00
Karim shoair a60662af4a Update camo.py 2024-11-05 11:38:35 +02:00
Karim shoair 8933a99a6e Update tests.yml 2024-11-05 11:29:28 +02:00
Karim shoair f200e9b290 looks like playwright need to run as normal user 2024-11-05 11:28:12 +02:00
Karim shoair 1dd04f8250 Update tests.yml 2024-11-05 02:39:15 +02:00
Karim shoair 6f7d925a2a Revert "Update tests.yml" 
This reverts commit 33f44587e6.
 2024-11-05 02:34:43 +02:00
Karim shoair 33f44587e6 Update tests.yml 2024-11-05 02:33:21 +02:00
Karim shoair 80cc0b77f3 Update tests.yml 2024-11-05 02:31:34 +02:00
Karim shoair 4121fe73c9 Update tests.yml 2024-11-05 02:27:30 +02:00
Karim shoair 72b36d6a26 Fix werkzeug issue and try caching 2024-11-05 02:22:21 +02:00
Karim shoair a5ffc0be5d Update tests.yml 2024-11-05 02:09:32 +02:00
Karim shoair 7fd50a4b94 Update tests.yml 2024-11-05 02:06:45 +02:00
Karim shoair 3b25a248fc Fixing tests action 2024-11-05 02:04:49 +02:00
Karim shoair e594bba0c4 Testing some of the new tests on Github action 2024-11-05 01:19:50 +02:00
Karim shoair bd612bb0e2 Updating the disable_resources doc-string 2024-11-04 20:16:04 +02:00
Karim shoair 544d23b696 Toolbelt fingerprint functions - Doc strings and type annotations stuff 2024-11-03 23:31:09 +02:00
Karim shoair e270b00811 Toolbelt subpackage custom functions general adjustments 2024-11-03 23:17:20 +02:00
Karim shoair 2ff0fbb0e5 Engines utils - Navigation functions 
- Adding doc strings and type annotations stuff.
- Making error messages more detailed.
- Renaming the `construct_websocket_url` function for more clarity.
 2024-11-03 22:46:40 +02:00
Karim shoair b8afb390ae Renaming fetcher file to fetchers for clarity 2024-11-03 22:43:16 +02:00
Karim shoair 282b130864 Static Engine - Doc strings and type annotations stuff 2024-11-03 22:25:34 +02:00
Karim shoair 3af174526f PlayWright Engine - Doc strings and type annotations stuff 2024-11-03 22:24:44 +02:00
Karim shoair 54cacb8a4e Camoufox Engine - Doc strings and type annotations stuff 2024-11-03 22:20:29 +02:00
Karim shoair 61fd592a6a Adding documentation to PlaywrightFetcher class 2024-11-03 21:04:11 +02:00
Karim shoair c1575b8e2e Updating doc-strings for more clarity 2024-11-03 16:49:33 +02:00
Karim shoair 1d2c68125a Update fetcher.py 2024-11-03 16:40:42 +02:00
Karim shoair 3556d11fb2 StealthyFetcher - Add disable_resources option and its explanation 2024-11-03 16:33:08 +02:00
Karim shoair 711de81106 PlaywrightFetcher - The option to drop unnecessary resources requests for speed boost 2024-11-03 16:27:55 +02:00
Karim shoair b32e1203f2 StealthyFetcher - The option to drop unnecessary resources requests for speed boost 2024-11-03 16:27:32 +02:00
Karim shoair c8c494e73f Adding a function to intercept and drop unwanted requests 2024-11-03 16:25:55 +02:00
Karim shoair 0867d9e78a Update README.md 2024-11-03 14:48:33 +02:00
Karim shoair e7e499d64b Add documentation for StealthyFetcher class + add webgl option 2024-11-03 14:45:20 +02:00
Karim shoair e7300ecba0 StealthFetcher - Add the option to allow webgl in the browser 2024-11-03 14:40:19 +02:00
Karim shoair 2528d4e023 Add documentation for the Fetcher class 2024-11-03 14:39:24 +02:00
Karim shoair be350f21dd Adding documentation to the base fetcher... 
+ Configure logger from here.
+ Enable auto_match by default.
 2024-11-03 13:45:13 +02:00
Karim shoair 55ab1d63b7 Turn auto_match off when not needed in testing 2024-11-03 13:37:27 +02:00
Karim shoair eaae5908c9 Make auto_match enabled by default 2024-11-03 13:33:24 +02:00
Karim shoair a0022f7322 Pumping up camoufox version 
Now, all progress bars are hidden
 2024-11-03 13:20:00 +02:00
Karim shoair 1ff1c93bcf Give response's body bytes priority over response's text 2024-11-03 13:04:02 +02:00
Karim shoair afe53332b8 Convert the request headers to dictionary to unify response object shape across engines 2024-11-03 13:01:48 +02:00
Karim shoair 942aa08c85 Handling JSON responses better 2024-11-03 13:00:23 +02:00
Karim shoair d845cae816 Update CONTRIBUTING.md 2024-11-03 12:42:53 +02:00
Karim shoair 263236a302 Update README.md 2024-11-03 12:42:26 +02:00
Karim shoair 26265327b5 Adding the option to update browserforge with camoufox 
Since my PRs to Camoufox got accepted 🚬
 2024-11-03 12:24:52 +02:00
Karim shoair 145c03daff Big structure changes (check commit description) 
- Moved most of the parser functions/files to the core package.
- Converted tools file to a package and made separate files for similar functions.
- Now all fetcher engines return a Response object
- Instead of selecting an engine to use and passing config to it, we have separate fetcher classes so the user can choose what to use while importing.
- I added a new custom fetcher so the user can create and use an engine.
- More...
 2024-11-03 01:04:02 +02:00
Karim shoair 7f1c06e7bf Pumping camoufox version up 2024-11-02 21:19:13 +02:00
Karim shoair 78c718c1d2 I don't know how that happened! 2024-11-02 21:17:56 +02:00
Karim shoair 088334b40b Reflecting the new changes in the ReadMe for testers 2024-10-31 02:16:09 +03:00
Karim shoair be2847ffb8 Bumping the library for version 0.2 for now and more adjustments 2024-10-31 02:14:41 +03:00
Karim shoair 3cb1696d6c Adding the main fetcher file to use for fetching pages 2024-10-31 02:13:35 +03:00
Karim shoair aadae4a25b Adding the JavaScript bypasses files for stealth mode 2024-10-31 02:12:37 +03:00
Karim shoair 8f8d08b568 Adding browsing engines 2024-10-31 02:11:33 +03:00