32cb76604c
style: Adjustments to the translator
2025-07-30 03:28:16 +03:00
5bb1266fa5
style: using isinstance function as the main way for type checking
2025-07-30 03:14:23 +03:00
ae9ccaec79
style: A lot of type hints correction
...
Since we are using Py3.10 as minimum version now, we remove Union when possible
2025-07-30 02:46:57 +03:00
9415acccce
fix: shortcuts for backward compatibility
2025-07-30 01:50:48 +03:00
7300efa77e
style(parser): optimize selectors instances creation
2025-07-30 01:45:26 +03:00
e7cdd39695
style: replacing os with Pathlib and small optimizations
2025-07-30 01:15:28 +03:00
9bcb9e9d93
style: General type hints fixes and imports optimizing
2025-07-29 23:43:06 +03:00
715dfb4243
feat: add length property to Selectors to write less code
2025-07-29 06:19:45 +03:00
29b77a96c8
refactor(parser): optimize imports
2025-07-29 06:18:12 +03:00
b9c7a5af2e
refactor: replace's Selector inpt (text/body) with 1 argument called content
2025-07-29 06:08:15 +03:00
264ae02aa7
refactor: huge change, many features/class got a better naming
...
- `Adaptor` became `Selector`
- `Adaptors` became `Selectors`
- `auto_match` argument/feature became `adaptive`
- `adaptor_arguments` argument became `selector_config`
- `automatch_domain` argument became `adaptive_domain`
- `additional_arguments` argument became `additional_args`
- `storage_adaptors` file became just `storage`
2025-07-29 04:20:23 +03:00
a5e4b91653
refactor: remove clean up function for Adaptor + make adaptor attributes accessible directly
2025-07-28 14:48:22 +03:00
297e14230b
refactor(parser): Multiple optimizations and fixes
2025-07-28 03:32:22 +03:00
d220050160
refactor(parser): Make get_all_text method 40% faster
2025-07-28 03:29:13 +03:00
b60fcbb884
refactor(Adaptor): Cleaner approach to find_similar method
...
This code is slower than before by about 2-5μs, but it's worth it.
2025-07-27 23:34:22 +03:00
ffde7e8ad9
fix(Adaptor): Add cleanup function to handle possible memory leak
2025-07-27 23:31:55 +03:00
6ae1810405
docs: improve All Adaptor class doc strings
2025-07-27 22:41:37 +03:00
93bd131bb6
fix(parser): Solve the ignored elements children issue while keeping speed
...
solves #61
2025-07-27 05:34:48 +03:00
0c8dd63f87
chore: migrating to ruff and updating pre-commit hooks
2025-04-13 17:32:00 +02:00
811cd2d9b0
refactor: make auto_match disabled by default
2025-03-26 19:25:19 +02:00
68ac9d4d05
fix(parser): Lowering memory with replacing body property with html_content
2025-03-17 05:58:40 +02:00
e5a022ec6a
fix(parser): Improving CSS selecting memory usage by ~144%
...
and speed by ~2%
2025-03-16 22:33:46 +02:00
94909099a9
fix(Parser): Change default text content to empty string instead of None
...
For compatibility with previous fix/commit
2025-03-15 00:56:19 +02:00
3d2418b67a
feat(Fetcher): Add the redirections history to Fetcher
...
Feature requested in #32
2025-02-22 16:58:54 +02:00
390bd51cb1
fix(TextHandler): correcting case_sensitive argument logic
...
This is awkward, I can't believe how this slipped from me!
2025-02-22 15:20:11 +02:00
7c35341d85
refactor(parser/get_all_text): Cleaner and a bit faster implementation
2025-01-31 02:47:49 +02:00
a6fd25f64c
fix(parser): fix traverse selectors
2025-01-31 01:57:19 +02:00
a8d475afb0
feat(parser): Make all returned html as TextHandler
...
So you can do regex on raw html if wanted etc...
2025-01-31 01:38:48 +02:00
3330e9f475
refactor(parser/automatch): Cleaner and faster implementation
...
~5% speed boost
2025-01-31 01:34:02 +02:00
37a31a120e
refactor(parser/find_by_regex): Better implementation for ~60% speed boost
2025-01-31 01:19:11 +02:00
2c6142d96d
refactor(parser/find_by_text): Better implementation for ~20% speed boost
2025-01-31 01:12:00 +02:00
1962c11f64
fix(parser/find_all): Logic issues made conditions used sometimes in a (and) fashion and other times (OR)
...
Also, the speed boost is 6-14% not 95% as I said first :(
2025-01-31 00:23:54 +02:00
57090f90a4
refactor(parser/find_all): about ~95% speed increase
...
Boom!
2025-01-30 23:39:46 +02:00
f37538320f
feat(parser): adding below_elements method
2025-01-30 23:05:10 +02:00
e507778c48
feat(Parser): Set regex methods new return types as TextHandlers
2025-01-30 02:27:38 +02:00
4e2e7a4b22
fix(parser): More on correcting type hints and unifying return type
2025-01-30 02:03:09 +02:00
7ee0f6114d
fix(parser): Code restructure for speed boost & better type hints
...
- Now return types are consistent across all the parser engine
- Parser got a 5-30% performance boost across different methods.
- Renamed some of the internal methods for clearer code.
- A lot better auto-completion experience after a lot of adjustments.
2025-01-30 01:46:33 +02:00
3cca3fd56b
style(parser): more accurate type hints
2024-12-21 16:17:02 +02:00
6652b8ac18
fix: solve the response log issue so now you get log for every response from Fetchers
2024-12-19 12:29:34 +02:00
838b8b1196
style: fixing a typo in logging
2024-12-16 13:39:30 +02:00
193827e27b
refactor(api)!: Unifying log under 1 logger and removing debug parameter
...
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
2024-12-11 21:41:37 +02:00
dcf51876a5
fix: Adaptor.body returns raw HTML without processing
...
If possible, otherwise returns `Adaptor.html_content`
2024-12-11 20:18:05 +02:00
b4f90615c3
Adding keep_cdata argument for Adaptor and Response classes
...
This will force lxml to keep cdata while parsing html if you want
2024-12-10 22:06:08 +02:00
5f9c3980dc
Adding urljoin method to Adaptors and Responses
2024-12-10 22:02:43 +02:00
45e86f50a3
Adding empty methods (get/get_all/extract/extract_all)
...
For easy copy-paste from Scrapy/parsel code when needed :)
2024-12-04 14:28:37 +02:00
1a17b2ce46
Sorting all library imports with isort
2024-11-30 14:02:06 +02:00
572df6b3b4
Fixing the way Response object handles sub items in some edge cases
...
+ Do it with the least impact on performance
2024-11-16 13:54:55 +02:00
b0c2b1818b
Return to the default text behaviour
...
The logic to remove comment tags before returning text was causing errors sometimes so from now on leave `keep_comments` set to True better as it is.
2024-11-14 22:21:45 +02:00
eaa7da27c6
Better logic to handle json responses
2024-11-14 00:03:26 +02:00
d4b896b4e9
Parser - New logic to handle JSON responses passed from Fetchers
2024-11-13 23:14:27 +02:00
Karim shoair