freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
157 Commits 1 Branch 0 Tags
b0c2b1818b9abcc31eec86c630de6be1e4e428fa
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair b0c2b1818b Return to the default text behaviour 
The logic to remove comment tags before returning text was causing errors sometimes so from now on leave `keep_comments` set to True better as it is.
 2024-11-14 22:21:45 +02:00
Karim shoair ea81e0284d Update tests Github action 2024-11-14 16:33:28 +02:00
Karim shoair 81f4e18a86 Merge remote-tracking branch 'origin/main' into dev 2024-11-14 16:30:41 +02:00
Karim shoair 9e337251ca Update README.md 2024-11-14 16:09:05 +02:00
Karim shoair e7e9cf8b3d Adding Evomi as a sponsor 2024-11-14 16:05:37 +02:00
Karim shoair b9c64e8dbf Update and rename README.md to Adding Evomi as a sponsor 2024-11-14 16:04:48 +02:00
Karim shoair eaa7da27c6 Better logic to handle json responses 2024-11-14 00:03:26 +02:00
Karim shoair d4b896b4e9 Parser - New logic to handle JSON responses passed from Fetchers 2024-11-13 23:14:27 +02:00
Karim shoair ab7b0d45ad StaticEngine - Fix the headers bug 2024-11-13 23:13:39 +02:00
Karim shoair 4e5f7d385a Merge pull request #5 from D4Vinci/main 
Keeping `dev` up to date with `main` branch
 2024-11-13 22:51:29 +02:00
Karim shoair 2b3c9b8ff3 Merge pull request #4 from marksmayo/main 
Tiny change - py.typed had a space at the end of the file name, which was causing git commands to fail with invalid path
 2024-11-13 11:18:53 +02:00
Karim shoair 4b5f9e12a2 Merge pull request #3 from D4Vinci/dev 
Releasing version 0.2 (BIG CHANGES)
 2024-11-11 19:29:41 +02:00
Karim shoair 64c9fe905e Corrections and rephrasing 2024-11-11 16:40:54 +02:00
Karim shoair bd38fdb347 Corrections and rephrasing 2024-11-11 16:40:46 +02:00
Karim shoair b3418f7f50 Corrections and rephrasing 2024-11-11 16:40:35 +02:00
Karim shoair d6b454dc43 Corrections and rephrasing 2024-11-11 16:40:30 +02:00
Karim shoair eb100e45f7 Corrections and rephrasing 2024-11-11 16:40:21 +02:00
Karim shoair 79bf4060f4 Updating find_all examples 2024-11-11 12:51:40 +02:00
Karim shoair e8935b1af8 Update README.md 2024-11-10 23:45:48 +02:00
Karim shoair 3d4b427168 Rework Selectors generation 
- Renamed both `css_selector` and `xpath_selector` both to `generate_css_selector` and `generate_xpath_selector` for clarity and not interrupting the auto-completion while coding.
- Added `generate_full_css_selector` and `generate_full_xpath_selector` methods
 2024-11-10 18:19:48 +02:00
Karim shoair de2d65856c Adding playwright/camoufox tests for cookies and improve the timeout test 2024-11-10 16:31:28 +02:00
Karim shoair 239e2c73bf Correctly get page content after JS Execution 
- Thanks for @AbdullahY36 for the heads-up.

**Update reference** https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event
 2024-11-10 16:11:46 +02:00
Karim shoair 43045e3eb2 Get page content after JS Execution 
Thanks for @AbdullahY36
 2024-11-10 15:54:44 +02:00
Karim shoair 7d2f55bd4c First version of the 0.2 README 
I probably forgot some features and made some mistakes here and there so I will review it again later
 2024-11-10 00:09:30 +02:00
Karim shoair 4b2814404a Fixing the functions-based filtering logic 2024-11-09 22:32:05 +02:00
Karim shoair 6b62579a7e Making find_all/find methods on steroids 
Adding the ability to find elements by regex patterns and functions.
 2024-11-09 21:25:17 +02:00
Karim shoair b90c8adb02 Adding the argument automatch_domain to all fetchers 2024-11-09 15:38:27 +02:00
Karim shoair 03091beb8b Small clarification 2024-11-09 13:40:50 +02:00
Karim shoair fd7d787ab0 Add extra_headers argument to PW and isolate google referer feature from stealth mode as google_search 2024-11-09 00:54:11 +02:00
Karim shoair ee8f78feb1 Update camo.py 2024-11-09 00:48:11 +02:00
Karim shoair fd34cd9afa Adding arguments google_search and extra_headers to StealthyFetcher 2024-11-09 00:43:24 +02:00
Karim shoair 530b54d964 Clearer doc-string 2024-11-09 00:17:41 +02:00
Karim shoair 1b015bbd07 Type correction 2024-11-08 23:59:12 +02:00
Karim shoair b919536527 Updating the keywords whitelisting logic in find/find_all functions 2024-11-07 12:56:32 +02:00
Karim shoair 51ce40a561 Update ROADMAP.md 2024-11-07 00:44:23 +02:00
Karim shoair 91f6b46e6d Adding new filter and search functions to parser 2024-11-07 00:31:14 +02:00
Karim shoair 846a185b49 Updating parser tests to use find and css_first functions 2024-11-07 00:02:36 +02:00
Karim shoair b25d2ee79b Adding new find_all and find functions to parser 2024-11-07 00:01:14 +02:00
Karim shoair 6b40af426f Better logic for css_first and xpath_first 2024-11-06 22:14:33 +02:00
Karim shoair 5e848ef046 Adding new class type TextHandlers 2024-11-06 21:03:19 +02:00
Karim shoair 7a84511b7a Making Adaptor and Adaptor re/re_first arguments consistent with the TextHandler ones 2024-11-06 20:47:21 +02:00
Karim shoair 7af75f78a5 Better logic for Adaptors re_first method 2024-11-06 20:24:30 +02:00
Karim shoair a1fd5ddc7e Correcting headless argument type 2024-11-06 20:18:44 +02:00
Karim shoair a0a3e62075 Adding addons and humanize arguments to StealthyFetcher 2024-11-06 20:14:42 +02:00
Karim shoair 5ad2126030 Small speed boost for fetching by some caching 2024-11-06 19:56:39 +02:00
Karim shoair a1513ba769 Update tests.yml 2024-11-06 12:54:52 +02:00
Karim shoair b542b5cc39 Recaching TOX environments 2024-11-06 12:50:14 +02:00
Karim shoair c30853ee2c Update tox.ini 2024-11-06 12:39:38 +02:00
Karim shoair fc30cfed4c Update tests.yml 2024-11-06 12:36:46 +02:00
Karim shoair cd1ff846a7 Update ROADMAP.md 2024-11-06 12:31:57 +02:00
Karim shoair d632bb14af Moving parser tests to a better structure 
and covering a bit more parts
 2024-11-06 12:30:41 +02:00
Karim shoair 5548941cf1 Correcting return type for css_first and xpath_first 2024-11-06 12:28:43 +02:00
Karim shoair e4eb442709 Adding tests for PlayWrightFetcher class 2024-11-06 12:21:55 +02:00
Karim shoair 4222626ec1 Adding tests for Fetcher class 2024-11-06 12:21:18 +02:00
Karim shoair 4400363146 Renaming fetchers tests folder for readability 2024-11-06 12:21:06 +02:00
Karim shoair 9a7f489eb7 make tests verbose for easier debugging later 2024-11-06 12:20:05 +02:00
Karim shoair 10bcdfb596 Moving bypasses folder deeper 2024-11-06 12:19:26 +02:00
Karim shoair 6cf506cb22 PlaywrightEngine - Validate CDP URLs in all cases 2024-11-06 11:24:56 +02:00
Karim shoair 6497b1b559 Type correction 2024-11-06 02:20:33 +02:00
Karim shoair 24806aecdc Update .gitignore 2024-11-05 23:41:57 +02:00
Karim shoair 3622040440 Adding css_first and xpath_first for easier usage 2024-11-05 23:41:51 +02:00
Karim shoair 009c2914aa Fixing re_first logic in Adaptors type 2024-11-05 22:59:47 +02:00
Karim shoair d50497dff3 Update .gitignore 2024-11-05 22:56:57 +02:00
Karim shoair ca4b5f205c Since tests can finally now run in Github actions without issues 
Let's miss it up again lol
 2024-11-05 13:10:51 +02:00
Karim shoair 2a275c436c Update tests.yml 2024-11-05 13:02:11 +02:00
Karim shoair 69cfbe45be Update tests.yml 2024-11-05 12:59:39 +02:00
Karim shoair 6f91d06b46 Update tests.yml 2024-11-05 12:57:57 +02:00
Karim shoair 2020214784 please work 2024-11-05 11:45:59 +02:00
Karim shoair 92d002032d Update tests.yml 2024-11-05 11:44:10 +02:00
Karim shoair 1bbb5dd75a Update tests.yml 2024-11-05 11:41:58 +02:00
Karim shoair d4358d32a8 Update tests.yml 2024-11-05 11:38:38 +02:00
Karim shoair a60662af4a Update camo.py 2024-11-05 11:38:35 +02:00
Karim shoair 8933a99a6e Update tests.yml 2024-11-05 11:29:28 +02:00
Karim shoair f200e9b290 looks like playwright need to run as normal user 2024-11-05 11:28:12 +02:00
Karim shoair 1dd04f8250 Update tests.yml 2024-11-05 02:39:15 +02:00
Karim shoair 6f7d925a2a Revert "Update tests.yml" 
This reverts commit 33f44587e6.
 2024-11-05 02:34:43 +02:00
Karim shoair 33f44587e6 Update tests.yml 2024-11-05 02:33:21 +02:00
Karim shoair 80cc0b77f3 Update tests.yml 2024-11-05 02:31:34 +02:00
Karim shoair 4121fe73c9 Update tests.yml 2024-11-05 02:27:30 +02:00
Karim shoair 72b36d6a26 Fix werkzeug issue and try caching 2024-11-05 02:22:21 +02:00
Karim shoair a5ffc0be5d Update tests.yml 2024-11-05 02:09:32 +02:00
Karim shoair 7fd50a4b94 Update tests.yml 2024-11-05 02:06:45 +02:00
Karim shoair 3b25a248fc Fixing tests action 2024-11-05 02:04:49 +02:00
Karim shoair e594bba0c4 Testing some of the new tests on Github action 2024-11-05 01:19:50 +02:00
Karim shoair bd612bb0e2 Updating the disable_resources doc-string 2024-11-04 20:16:04 +02:00
Karim shoair 544d23b696 Toolbelt fingerprint functions - Doc strings and type annotations stuff 2024-11-03 23:31:09 +02:00
Karim shoair e270b00811 Toolbelt subpackage custom functions general adjustments 2024-11-03 23:17:20 +02:00
Karim shoair 2ff0fbb0e5 Engines utils - Navigation functions 
- Adding doc strings and type annotations stuff.
- Making error messages more detailed.
- Renaming the `construct_websocket_url` function for more clarity.
 2024-11-03 22:46:40 +02:00
Karim shoair b8afb390ae Renaming fetcher file to fetchers for clarity 2024-11-03 22:43:16 +02:00
Karim shoair 282b130864 Static Engine - Doc strings and type annotations stuff 2024-11-03 22:25:34 +02:00
Karim shoair 3af174526f PlayWright Engine - Doc strings and type annotations stuff 2024-11-03 22:24:44 +02:00
Karim shoair 54cacb8a4e Camoufox Engine - Doc strings and type annotations stuff 2024-11-03 22:20:29 +02:00
Karim shoair 61fd592a6a Adding documentation to PlaywrightFetcher class 2024-11-03 21:04:11 +02:00
Karim shoair c1575b8e2e Updating doc-strings for more clarity 2024-11-03 16:49:33 +02:00
Karim shoair 1d2c68125a Update fetcher.py 2024-11-03 16:40:42 +02:00
Karim shoair 3556d11fb2 StealthyFetcher - Add disable_resources option and its explanation 2024-11-03 16:33:08 +02:00
Karim shoair 711de81106 PlaywrightFetcher - The option to drop unnecessary resources requests for speed boost 2024-11-03 16:27:55 +02:00
Karim shoair b32e1203f2 StealthyFetcher - The option to drop unnecessary resources requests for speed boost 2024-11-03 16:27:32 +02:00
Karim shoair c8c494e73f Adding a function to intercept and drop unwanted requests 2024-11-03 16:25:55 +02:00
Karim shoair 0867d9e78a Update README.md 2024-11-03 14:48:33 +02:00