 Karim shoair
|
2606f7a12d
|
test: clearer naming for async tests
|
2024-12-16 01:01:59 +02:00 |
|
|
Karim shoair
|
448587ff41
|
test: add tests for StealthyFetcher async fetch
|
2024-12-16 01:00:05 +02:00 |
|
|
Karim shoair
|
79a911e24e
|
feat(StealthyFetcher): Add async fetch support
|
2024-12-16 00:59:19 +02:00 |
|
|
Karim shoair
|
efb3270c67
|
style: Rewrite asyncFetcher tests to a cleaner version
|
2024-12-16 00:47:02 +02:00 |
|
|
Karim shoair
|
361ee440aa
|
test: adding pytest-asyncio plugin to tests requirements file
|
2024-12-16 00:20:26 +02:00 |
|
|
Karim shoair
|
de015f2b18
|
chore: Restructuring fetchers into cleaner structure
|
2024-12-16 00:19:48 +02:00 |
|
|
Karim shoair
|
aac77e488c
|
test: add test for PlaywrightFetcher Async support
|
2024-12-16 00:17:13 +02:00 |
|
|
Karim shoair
|
af4f2c0f74
|
feat(PlaywrightFetcher): Add async support for PlaywrightFetcher
|
2024-12-16 00:15:47 +02:00 |
|
|
Karim shoair
|
889c111855
|
refactor(Playwright Engine): Separate what we can for cleaner code and the async function later
The caching will give a slight performance increase with bulk requests
|
2024-12-15 23:25:02 +02:00 |
|
|
Karim shoair
|
6c17bd886e
|
style: using better data structures for constants
This will cause a slight performance increase
|
2024-12-15 23:19:47 +02:00 |
|
|
Karim shoair
|
a90192df03
|
build: adding tests for AsyncFetcher class
|
2024-12-15 16:54:52 +02:00 |
|
|
Karim shoair
|
b10cfd3643
|
feat: Adding AsyncFetcher class version of Fetcher
The first step in fully supporting async
|
2024-12-15 16:51:58 +02:00 |
|
|
Karim shoair
|
faf728a794
|
style: moving repeated arguments from inside the functions to __init__
Might give a slight performance increase too
|
2024-12-15 16:49:48 +02:00 |
|
|
Karim shoair
|
445af3c702
|
perf: Give repeated usage of Fetcher a slight performance increase
By caching the `StaticEngine` class instance
|
2024-12-15 16:13:08 +02:00 |
|
|
Karim shoair
|
299793af3f
|
feat: adding the retries argument for all methods of Fetcher class
|
2024-12-15 15:59:15 +02:00 |
|
|
Karim shoair
|
838dd62ee4
|
build: pumping up camoufox
For the issue with geoip
|
2024-12-13 20:43:53 +02:00 |
|
|
Karim shoair
|
f9bee4cbbb
|
build: Pumping up camoufox version to solve browserforge issue
Dropped browserforge from the requirements here so its version gets controlled by camoufox
|
2024-12-12 14:37:31 +02:00 |
|
|
Karim shoair
|
bfe906315e
|
feat: adding geoip parameter to the StealthyFetcher
|
2024-12-12 12:43:49 +02:00 |
|
|
Karim shoair
|
6f87420e35
|
build: disable the 501 error test for playwright
|
2024-12-11 21:44:03 +02:00 |
|
|
Karim shoair
|
e254341974
|
fix: forgot to stage it with last commit
|
2024-12-11 21:43:36 +02:00 |
|
|
Karim shoair
|
193827e27b
|
refactor(api)!: Unifying log under 1 logger and removing debug parameter
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
|
2024-12-11 21:41:37 +02:00 |
|
|
Karim shoair
|
f30eb6ab6c
|
build: disable the 404 error test for playwright
|
2024-12-11 20:27:07 +02:00 |
|
|
Karim shoair
|
dcf51876a5
|
fix: Adaptor.body returns raw HTML without processing
If possible, otherwise returns `Adaptor.html_content`
|
2024-12-11 20:18:05 +02:00 |
|
|
Karim shoair
|
9f0001af26
|
feat: logging for response status
|
2024-12-11 20:14:32 +02:00 |
|
|
Karim shoair
|
e332444411
|
Update tests.yml
|
2024-12-10 23:35:21 +02:00 |
|
|
Karim shoair
|
70b54241e4
|
Update tox.ini
|
2024-12-10 23:32:54 +02:00 |
|
|
Karim shoair
|
ba11c433ff
|
Preparing to release 0.2.9 soon
|
2024-12-10 22:56:51 +02:00 |
|
|
Karim shoair
|
e60a57c63f
|
Python 3.8 is not supported anymore
It's not supported in the new version of Playwright and is problematic in some situations while being slower than newer versions.
|
2024-12-10 22:37:37 +02:00 |
|
|
Karim shoair
|
93ed768312
|
Bumping up the libraries versions for better stealth and speed!
|
2024-12-10 22:22:34 +02:00 |
|
|
Karim shoair
|
b4f90615c3
|
Adding keep_cdata argument for Adaptor and Response classes
This will force lxml to keep cdata while parsing html if you want
|
2024-12-10 22:06:08 +02:00 |
|
|
Karim shoair
|
5f9c3980dc
|
Adding urljoin method to Adaptors and Responses
|
2024-12-10 22:02:43 +02:00 |
|
|
Karim shoair
|
d0f18951ac
|
fix: Enable WebGL by default
|
2024-12-04 23:38:30 +02:00 |
|
|
Karim shoair
|
45e86f50a3
|
Adding empty methods (get/get_all/extract/extract_all)
For easy copy-paste from Scrapy/parsel code when needed :)
|
2024-12-04 14:28:37 +02:00 |
|
|
Karim shoair
|
31e838ce9b
|
Make standard string methods return TextHandler again instead of str
|
2024-12-04 14:27:36 +02:00 |
|
|
Karim shoair
|
012820c635
|
Merge pull request #21 from D4Vinci/dev
v0.2.8
|
2024-11-30 18:16:06 +02:00 |
|
|
Karim shoair
|
c481a1c8ad
|
Pumping version up to 0.2.8
|
2024-11-30 18:13:19 +02:00 |
|
|
Karim shoair
|
a3a4f1ff19
|
A cleanup script to use with development
|
2024-11-30 18:01:35 +02:00 |
|
|
Karim shoair
|
f7916e504e
|
Update tox.ini
|
2024-11-30 14:32:42 +02:00 |
|
|
Karim shoair
|
8321cfc180
|
Add vermin hook to check for minimum python version
|
2024-11-30 14:28:10 +02:00 |
|
|
Karim shoair
|
b38c38b03b
|
Forcing Tox to always use pytest config file
|
2024-11-30 14:14:59 +02:00 |
|
|
Karim shoair
|
f81e3b3cc3
|
Turn off threaded testing for Python 3.8 on tox
part 2 :)
|
2024-11-30 14:08:18 +02:00 |
|
|
Karim shoair
|
1a17b2ce46
|
Sorting all library imports with isort
|
2024-11-30 14:02:06 +02:00 |
|
|
Karim shoair
|
26bfa88e50
|
Turn off threaded testing for Python 3.8 on tox
|
2024-11-30 14:01:30 +02:00 |
|
|
Karim shoair
|
4dddebbe24
|
Update .bandit.yml
|
2024-11-30 13:47:46 +02:00 |
|
|
Karim shoair
|
7608d86bf2
|
Pumping up camoufox version
|
2024-11-30 11:24:47 +02:00 |
|
|
Karim shoair
|
82f8e4a073
|
Fixing import typo, thanks to #20
|
2024-11-30 11:23:07 +02:00 |
|
|
Karim shoair
|
1cf7a1d205
|
Make Github tests workflow work on main and dev branches only
The docs branch doesn't need tests
|
2024-11-29 21:19:39 +02:00 |
|
|
Karim shoair
|
97fecc9d5f
|
Style adjustments by flake8
|
2024-11-29 21:14:52 +02:00 |
|
|
Karim shoair
|
d6cc07deaa
|
Add scrapless banner
|
2024-11-29 11:26:48 +02:00 |
|
|
Karim shoair
|
f8ad18ab8a
|
Rephrasing some parts to be clearer
|
2024-11-27 11:50:20 +02:00 |
|