 Karim shoair
|
844cb0c6a2
|
fix(PlaywrightFetcher): Wait for the real response status code
This will return the real status response after redirection for websites that have Cloudflare instead of Playwright's default behavior.
|
2024-12-19 13:00:19 +02:00 |
|
|
Karim shoair
|
6652b8ac18
|
fix: solve the response log issue so now you get log for every response from Fetchers
|
2024-12-19 12:29:34 +02:00 |
|
|
Karim shoair
|
60df72cb23
|
Merge pull request #25 from D4Vinci/dev
v0.2.9
|
2024-12-16 14:42:13 +02:00 |
|
|
Karim shoair
|
838b8b1196
|
style: fixing a typo in logging
|
2024-12-16 13:39:30 +02:00 |
|
|
Karim shoair
|
c282cb3cb7
|
fix: Stopped the log spamming that happens with multiple instances creation
|
2024-12-16 13:35:48 +02:00 |
|
|
Karim shoair
|
ac95600650
|
docs: add an example that uses page.urljoin
|
2024-12-16 13:27:14 +02:00 |
|
|
Karim shoair
|
20ef45349a
|
test: Rewrite automatch tests to a cleaner version and adding async test
|
2024-12-16 13:18:19 +02:00 |
|
|
Karim shoair
|
69e31612a5
|
test: Rewrite parser tests to a cleaner version and adding more tests
|
2024-12-16 13:17:09 +02:00 |
|
|
Karim shoair
|
3ff0d55c3e
|
chore: Delete former-sponsor banner
|
2024-12-16 12:47:43 +02:00 |
|
|
Karim shoair
|
43b41b3733
|
docs: fix TOC header
|
2024-12-16 12:41:07 +02:00 |
|
|
Karim shoair
|
6cf5ce9719
|
docs: Adding async examples and fixing some typos
|
2024-12-16 12:39:44 +02:00 |
|
|
Karim shoair
|
05c6eeb387
|
fix: add AsyncFetcher to top-level shortcuts
|
2024-12-16 12:37:29 +02:00 |
|
|
Karim shoair
|
d66ebc19a5
|
fix(Fetchers/disable_resources): Fixing the logic for intercepting resources
|
2024-12-16 12:35:25 +02:00 |
|
|
Karim shoair
|
eceef48ef5
|
fix(Fetchers/page_action): Fixing logic
|
2024-12-16 12:34:23 +02:00 |
|
|
Karim shoair
|
3ecffcb81c
|
style: Rewrite sync Fetchers tests to a cleaner version
|
2024-12-16 01:27:18 +02:00 |
|
|
Karim shoair
|
2606f7a12d
|
test: clearer naming for async tests
|
2024-12-16 01:01:59 +02:00 |
|
|
Karim shoair
|
448587ff41
|
test: add tests for StealthyFetcher async fetch
|
2024-12-16 01:00:05 +02:00 |
|
|
Karim shoair
|
79a911e24e
|
feat(StealthyFetcher): Add async fetch support
|
2024-12-16 00:59:19 +02:00 |
|
|
Karim shoair
|
efb3270c67
|
style: Rewrite asyncFetcher tests to a cleaner version
|
2024-12-16 00:47:02 +02:00 |
|
|
Karim shoair
|
361ee440aa
|
test: adding pytest-asyncio plugin to tests requirements file
|
2024-12-16 00:20:26 +02:00 |
|
|
Karim shoair
|
de015f2b18
|
chore: Restructuring fetchers into cleaner structure
|
2024-12-16 00:19:48 +02:00 |
|
|
Karim shoair
|
aac77e488c
|
test: add test for PlaywrightFetcher Async support
|
2024-12-16 00:17:13 +02:00 |
|
|
Karim shoair
|
af4f2c0f74
|
feat(PlaywrightFetcher): Add async support for PlaywrightFetcher
|
2024-12-16 00:15:47 +02:00 |
|
|
Karim shoair
|
889c111855
|
refactor(Playwright Engine): Separate what we can for cleaner code and the async function later
The caching will give a slight performance increase with bulk requests
|
2024-12-15 23:25:02 +02:00 |
|
|
Karim shoair
|
6c17bd886e
|
style: using better data structures for constants
This will cause a slight performance increase
|
2024-12-15 23:19:47 +02:00 |
|
|
Karim shoair
|
a90192df03
|
build: adding tests for AsyncFetcher class
|
2024-12-15 16:54:52 +02:00 |
|
|
Karim shoair
|
b10cfd3643
|
feat: Adding AsyncFetcher class version of Fetcher
The first step in fully supporting async
|
2024-12-15 16:51:58 +02:00 |
|
|
Karim shoair
|
faf728a794
|
style: moving repeated arguments from inside the functions to __init__
Might give a slight performance increase too
|
2024-12-15 16:49:48 +02:00 |
|
|
Karim shoair
|
445af3c702
|
perf: Give repeated usage of Fetcher a slight performance increase
By caching the `StaticEngine` class instance
|
2024-12-15 16:13:08 +02:00 |
|
|
Karim shoair
|
299793af3f
|
feat: adding the retries argument for all methods of Fetcher class
|
2024-12-15 15:59:15 +02:00 |
|
|
Karim shoair
|
838dd62ee4
|
build: pumping up camoufox
For the issue with geoip
|
2024-12-13 20:43:53 +02:00 |
|
|
Karim shoair
|
f9bee4cbbb
|
build: Pumping up camoufox version to solve browserforge issue
Dropped browserforge from the requirements here so its version gets controlled by camoufox
|
2024-12-12 14:37:31 +02:00 |
|
|
Karim shoair
|
bfe906315e
|
feat: adding geoip parameter to the StealthyFetcher
|
2024-12-12 12:43:49 +02:00 |
|
|
Karim shoair
|
6f87420e35
|
build: disable the 501 error test for playwright
|
2024-12-11 21:44:03 +02:00 |
|
|
Karim shoair
|
e254341974
|
fix: forgot to stage it with last commit
|
2024-12-11 21:43:36 +02:00 |
|
|
Karim shoair
|
193827e27b
|
refactor(api)!: Unifying log under 1 logger and removing debug parameter
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
|
2024-12-11 21:41:37 +02:00 |
|
|
Karim shoair
|
f30eb6ab6c
|
build: disable the 404 error test for playwright
|
2024-12-11 20:27:07 +02:00 |
|
|
Karim shoair
|
dcf51876a5
|
fix: Adaptor.body returns raw HTML without processing
If possible, otherwise returns `Adaptor.html_content`
|
2024-12-11 20:18:05 +02:00 |
|
|
Karim shoair
|
9f0001af26
|
feat: logging for response status
|
2024-12-11 20:14:32 +02:00 |
|
|
Karim shoair
|
e332444411
|
Update tests.yml
|
2024-12-10 23:35:21 +02:00 |
|
|
Karim shoair
|
70b54241e4
|
Update tox.ini
|
2024-12-10 23:32:54 +02:00 |
|
|
Karim shoair
|
ba11c433ff
|
Preparing to release 0.2.9 soon
|
2024-12-10 22:56:51 +02:00 |
|
|
Karim shoair
|
e60a57c63f
|
Python 3.8 is not supported anymore
It's not supported in the new version of Playwright and is problematic in some situations while being slower than newer versions.
|
2024-12-10 22:37:37 +02:00 |
|
|
Karim shoair
|
93ed768312
|
Bumping up the libraries versions for better stealth and speed!
|
2024-12-10 22:22:34 +02:00 |
|
|
Karim shoair
|
b4f90615c3
|
Adding keep_cdata argument for Adaptor and Response classes
This will force lxml to keep cdata while parsing html if you want
|
2024-12-10 22:06:08 +02:00 |
|
|
Karim shoair
|
5f9c3980dc
|
Adding urljoin method to Adaptors and Responses
|
2024-12-10 22:02:43 +02:00 |
|
|
Karim shoair
|
d0f18951ac
|
fix: Enable WebGL by default
|
2024-12-04 23:38:30 +02:00 |
|
|
Karim shoair
|
45e86f50a3
|
Adding empty methods (get/get_all/extract/extract_all)
For easy copy-paste from Scrapy/parsel code when needed :)
|
2024-12-04 14:28:37 +02:00 |
|
|
Karim shoair
|
31e838ce9b
|
Make standard string methods return TextHandler again instead of str
|
2024-12-04 14:27:36 +02:00 |
|
|
Karim shoair
|
012820c635
|
Merge pull request #21 from D4Vinci/dev
v0.2.8
|
2024-11-30 18:16:06 +02:00 |
|