7ee0f6114d
fix(parser): Code restructure for speed boost & better type hints
...
- Now return types are consistent across all the parser engine
- Parser got a 5-30% performance boost across different methods.
- Renamed some of the internal methods for clearer code.
- A lot better auto-completion experience after a lot of adjustments.
2025-01-30 01:46:33 +02:00
1d5fcc060d
feat(TextHandler): Make slicing return TextHandlers + autocompletion fixes
...
and access by index return `TextHandler`
2025-01-29 14:22:47 +02:00
70ffdd09e2
feat(TextHandler): Make split string method return Texthandlers
2025-01-29 01:22:07 +02:00
5047e886a5
style(autocompletion): Improving type hints for custom types
...
This will provide a better autocompletion experience inside IDEs, commit affects:
- TextHandler
- AttributesHandler
2025-01-29 00:58:14 +02:00
85f98708b2
feat(StealthyFetcher): Use memory cache by default for performance boost
...
This is instead of the default disk-based cache.
2025-01-28 23:55:24 +02:00
e9e606bca5
feat(StealthyFetcher): Disable Cross-Origin-Opener-Policy by default for async_fetch
...
This allows elements in cross-origin iframes, such as the Turnstile checkbox, to be clicked.
.
Forgot to add it with the other commit
2025-01-28 23:53:12 +02:00
675c0110e6
feat(StealthyFetcher): Make ads enabled by default for performance boost
2025-01-28 23:51:31 +02:00
52b66c98ec
feat(StealthyFetcher): Disable Cross-Origin-Opener-Policy by default
...
This allows elements in cross-origin iframes, such as the Turnstile checkbox, to be clicked.
2025-01-28 23:47:47 +02:00
1332be7e63
fix(autocompletion): Fix the defaults import
...
It runs autocompletion on some IDEs
2025-01-28 23:44:58 +02:00
c2447e069e
build: Pumping version up
2025-01-28 23:41:48 +02:00
c5d83a11ad
build: Pumping version up
2024-12-26 19:57:25 +02:00
03b52aa74c
feat(cli): adding terminal command scrapling install
...
Instead of making users install browsers by themself
2024-12-26 19:46:22 +02:00
2006be2173
fix(StealthyFetcher): Use more dependable response data
2024-12-26 00:29:42 +02:00
f9b85cfbac
fix(PlaywrightFetcher): Use more dependable response data
2024-12-26 00:27:13 +02:00
3cca3fd56b
style(parser): more accurate type hints
2024-12-21 16:17:02 +02:00
f74db648bc
build: Pumping up the version to 0.2.91
2024-12-19 13:29:45 +02:00
5be8c2a95e
style(Fetchers): Setting correct type hint for wait_selector_state argument
2024-12-19 13:22:12 +02:00
94745b1555
fix(StealthyFetcher): Wait for the real response status code
...
This will return the real status response after redirection for websites that have Cloudflare instead of Playwright API's default behavior.
2024-12-19 13:08:41 +02:00
844cb0c6a2
fix(PlaywrightFetcher): Wait for the real response status code
...
This will return the real status response after redirection for websites that have Cloudflare instead of Playwright's default behavior.
2024-12-19 13:00:19 +02:00
6652b8ac18
fix: solve the response log issue so now you get log for every response from Fetchers
2024-12-19 12:29:34 +02:00
838b8b1196
style: fixing a typo in logging
2024-12-16 13:39:30 +02:00
c282cb3cb7
fix: Stopped the log spamming that happens with multiple instances creation
2024-12-16 13:35:48 +02:00
05c6eeb387
fix: add AsyncFetcher to top-level shortcuts
2024-12-16 12:37:29 +02:00
d66ebc19a5
fix(Fetchers/disable_resources): Fixing the logic for intercepting resources
2024-12-16 12:35:25 +02:00
eceef48ef5
fix(Fetchers/page_action): Fixing logic
2024-12-16 12:34:23 +02:00
79a911e24e
feat(StealthyFetcher): Add async fetch support
2024-12-16 00:59:19 +02:00
af4f2c0f74
feat(PlaywrightFetcher): Add async support for PlaywrightFetcher
2024-12-16 00:15:47 +02:00
889c111855
refactor(Playwright Engine): Separate what we can for cleaner code and the async function later
...
The caching will give a slight performance increase with bulk requests
2024-12-15 23:25:02 +02:00
6c17bd886e
style: using better data structures for constants
...
This will cause a slight performance increase
2024-12-15 23:19:47 +02:00
b10cfd3643
feat: Adding AsyncFetcher class version of Fetcher
...
The first step in fully supporting async
2024-12-15 16:51:58 +02:00
faf728a794
style: moving repeated arguments from inside the functions to __init__
...
Might give a slight performance increase too
2024-12-15 16:49:48 +02:00
445af3c702
perf: Give repeated usage of Fetcher a slight performance increase
...
By caching the `StaticEngine` class instance
2024-12-15 16:13:08 +02:00
299793af3f
feat: adding the retries argument for all methods of Fetcher class
2024-12-15 15:59:15 +02:00
bfe906315e
feat: adding geoip parameter to the StealthyFetcher
2024-12-12 12:43:49 +02:00
e254341974
fix: forgot to stage it with last commit
2024-12-11 21:43:36 +02:00
193827e27b
refactor(api)!: Unifying log under 1 logger and removing debug parameter
...
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
2024-12-11 21:41:37 +02:00
dcf51876a5
fix: Adaptor.body returns raw HTML without processing
...
If possible, otherwise returns `Adaptor.html_content`
2024-12-11 20:18:05 +02:00
9f0001af26
feat: logging for response status
2024-12-11 20:14:32 +02:00
ba11c433ff
Preparing to release 0.2.9 soon
2024-12-10 22:56:51 +02:00
b4f90615c3
Adding keep_cdata argument for Adaptor and Response classes
...
This will force lxml to keep cdata while parsing html if you want
2024-12-10 22:06:08 +02:00
5f9c3980dc
Adding urljoin method to Adaptors and Responses
2024-12-10 22:02:43 +02:00
d0f18951ac
fix: Enable WebGL by default
2024-12-04 23:38:30 +02:00
45e86f50a3
Adding empty methods (get/get_all/extract/extract_all)
...
For easy copy-paste from Scrapy/parsel code when needed :)
2024-12-04 14:28:37 +02:00
31e838ce9b
Make standard string methods return TextHandler again instead of str
2024-12-04 14:27:36 +02:00
c481a1c8ad
Pumping version up to 0.2.8
2024-11-30 18:13:19 +02:00
1a17b2ce46
Sorting all library imports with isort
2024-11-30 14:02:06 +02:00
97fecc9d5f
Style adjustments by flake8
2024-11-29 21:14:52 +02:00
06a47f9644
Small fix for the ResponseEncoding class
2024-11-26 22:55:22 +02:00
28b783e86b
Improving the stealth mode a bit for Playwright
2024-11-26 22:41:04 +02:00
14ffdbd6fc
Handling problematic encodings
2024-11-26 22:27:05 +02:00
Karim shoair