Commit Graph

12 Commits

Author SHA1 Message Date
Karim shoair 239e2c73bf Correctly get page content after JS Execution
- Thanks for @AbdullahY36 for the heads-up.

**Update reference** https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event
2024-11-10 16:11:46 +02:00
Karim shoair b90c8adb02 Adding the argument automatch_domain to all fetchers 2024-11-09 15:38:27 +02:00
Karim shoair 5ad2126030 Small speed boost for fetching by some caching 2024-11-06 19:56:39 +02:00
Karim shoair 10bcdfb596 Moving bypasses folder deeper 2024-11-06 12:19:26 +02:00
Karim shoair 6cf506cb22 PlaywrightEngine - Validate CDP URLs in all cases 2024-11-06 11:24:56 +02:00
Karim shoair 544d23b696 Toolbelt fingerprint functions - Doc strings and type annotations stuff 2024-11-03 23:31:09 +02:00
Karim shoair e270b00811 Toolbelt subpackage custom functions general adjustments 2024-11-03 23:17:20 +02:00
Karim shoair 2ff0fbb0e5 Engines utils - Navigation functions
- Adding doc strings and type annotations stuff.
- Making error messages more detailed.
- Renaming the `construct_websocket_url` function for more clarity.
2024-11-03 22:46:40 +02:00
Karim shoair c8c494e73f Adding a function to intercept and drop unwanted requests 2024-11-03 16:25:55 +02:00
Karim shoair be350f21dd Adding documentation to the base fetcher...
+ Configure logger from here.
+ Enable auto_match by default.
2024-11-03 13:45:13 +02:00
Karim shoair 1ff1c93bcf Give response's body bytes priority over response's text 2024-11-03 13:04:02 +02:00
Karim shoair 145c03daff Big structure changes (check commit description)
- Moved most of the parser functions/files to the core package.
- Converted tools file to a package and made separate files for similar functions.
- Now all fetcher engines return a Response object
- Instead of selecting an engine to use and passing config to it, we have separate fetcher classes so the user can choose what to use while importing.
- I added a new custom fetcher so the user can create and use an engine.
- More...
2024-11-03 01:04:02 +02:00