32cb76604c
style: Adjustments to the translator
2025-07-30 03:28:16 +03:00
5bb1266fa5
style: using isinstance function as the main way for type checking
2025-07-30 03:14:23 +03:00
ae9ccaec79
style: A lot of type hints correction
...
Since we are using Py3.10 as minimum version now, we remove Union when possible
2025-07-30 02:46:57 +03:00
ca12a11b7e
style(utils): optimize imports
2025-07-30 02:08:01 +03:00
db781dcc32
style: Removing dead code
2025-07-30 02:03:39 +03:00
3c5de8e0f2
perf: Speed up clean functions
2025-07-30 02:03:27 +03:00
e7cdd39695
style: replacing os with Pathlib and small optimizations
2025-07-30 01:15:28 +03:00
18660f8132
style: type hints corrections and docstrings
2025-07-30 00:32:39 +03:00
ba585c0adc
perf: imports optimizing
2025-07-30 00:22:06 +03:00
9bcb9e9d93
style: General type hints fixes and imports optimizing
2025-07-29 23:43:06 +03:00
9ce3279488
fix(TextHandler): Increase speed of clean method 5 times
2025-07-29 22:31:03 +03:00
264ae02aa7
refactor: huge change, many features/class got a better naming
...
- `Adaptor` became `Selector`
- `Adaptors` became `Selectors`
- `auto_match` argument/feature became `adaptive`
- `adaptor_arguments` argument became `selector_config`
- `automatch_domain` argument became `adaptive_domain`
- `additional_arguments` argument became `additional_args`
- `storage_adaptors` file became just `storage`
2025-07-29 04:20:23 +03:00
7f11b6f59b
fix(storage): possible threading issue with recursion + optimizations
2025-07-28 00:16:58 +03:00
3b0237e402
docs: improve All storage classes doc strings
2025-07-27 23:37:47 +03:00
d1aa0be6e4
refactor(StealthyFetcher): Remove virtual mode and use persistent context
...
Solves #64 completely too
2025-07-27 03:41:28 +03:00
172a5b4a0a
feat: Add an mcp server
2025-07-26 22:57:08 +03:00
1076001e29
refactor: changes to be used by the mcp server
2025-07-14 05:38:29 +03:00
3184dd8581
build: set minimum version to Python 3.10
2025-07-07 00:09:02 +03:00
29e485bdf2
refactor: Optimizations to CLI
2025-06-29 19:10:52 +03:00
1c9e48b1c6
feat(extract): Adding new command to CLI options + Optimizations
...
Users can now fetch websites directly without code and extract full/selected HTML content as HTML, Markdown, or extract text content.
2025-06-29 18:56:09 +03:00
38851698ad
refactor: optimize imports and docstrings correction
2025-06-25 23:15:32 +03:00
a41be48047
refactor: optimize imports and docstrings correction
2025-06-25 21:36:53 +03:00
8a272cf19b
feat/refactor(fetchers): Replacing PlayWrightFetcher with DynamicFetcher and adding session classes
2025-06-22 02:10:45 +03:00
145b710960
feat(fetchers): Adding the foundation of the new browser-based fetchers logic
2025-06-19 04:06:52 +03:00
3e63fa2523
fix(shell): Hide the automatic tips and solve the namespace issue
2025-06-19 04:03:13 +03:00
3068bb1356
build: update deps
2025-06-14 18:56:31 +03:00
9e68e60161
feat(Fetcher): Replacing httpx + Adding FetcherSession
...
Check out the discord server for details
2025-05-25 21:10:20 +03:00
fc535fa208
perf: drop w3lib dependency
2025-05-11 20:22:15 +03:00
74fa1bfbed
feat(shell): Add support to curl -b argument
2025-04-30 04:23:05 +03:00
8f3b2092c2
fix(cli): Update page object after curl request
2025-04-29 03:14:41 +03:00
98ad50d5aa
feat(cli): Adding two new commands (uncurl/curl2fetcher)
2025-04-29 02:45:50 +03:00
49b7ae13f6
feat(cli): Adding Scrapling Shell feature
2025-04-22 05:15:57 +02:00
0c8dd63f87
chore: migrating to ruff and updating pre-commit hooks
2025-04-13 17:32:00 +02:00
ff3962b1ad
fix: Adjusting cache size in all the library
2025-03-17 06:46:41 +02:00
e5a022ec6a
fix(parser): Improving CSS selecting memory usage by ~144%
...
and speed by ~2%
2025-03-16 22:33:46 +02:00
7bef44ea4c
fix: Limit caching in all the library
...
To lower memory usage with high numbers of requests. Possibly improve case #48
2025-03-15 02:01:16 +02:00
bf6a3a9c20
fix(TextHandler): auto handling non-string type variables
...
Fix for #45
2025-03-08 01:42:58 +02:00
e9344a7384
fix(TextHandler): Fixing an issue with slicing
...
Slicing was returning `TextHandlers` instead of `TextHandler`. This solves #41 and corrects the type hint
2025-03-05 03:11:46 +02:00
390bd51cb1
fix(TextHandler): correcting case_sensitive argument logic
...
This is awkward, I can't believe how this slipped from me!
2025-02-22 15:20:11 +02:00
3f23f18473
style(TextHandler): fix re overload
2025-01-31 00:48:27 +02:00
148103f3c8
fix(translator): increase CSS to XPath convertor cache limit
2025-01-30 20:01:25 +02:00
7882cf2527
style(TextHandler): Setting overload on check_match argument for auto completion
2025-01-30 02:44:57 +02:00
93be416572
feat(TextHandler): Now regex methods return TextHandler/TextHandlers
2025-01-30 02:15:09 +02:00
1d5fcc060d
feat(TextHandler): Make slicing return TextHandlers + autocompletion fixes
...
and access by index return `TextHandler`
2025-01-29 14:22:47 +02:00
70ffdd09e2
feat(TextHandler): Make split string method return Texthandlers
2025-01-29 01:22:07 +02:00
5047e886a5
style(autocompletion): Improving type hints for custom types
...
This will provide a better autocompletion experience inside IDEs, commit affects:
- TextHandler
- AttributesHandler
2025-01-29 00:58:14 +02:00
5be8c2a95e
style(Fetchers): Setting correct type hint for wait_selector_state argument
2024-12-19 13:22:12 +02:00
193827e27b
refactor(api)!: Unifying log under 1 logger and removing debug parameter
...
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
2024-12-11 21:41:37 +02:00
9f0001af26
feat: logging for response status
2024-12-11 20:14:32 +02:00
45e86f50a3
Adding empty methods (get/get_all/extract/extract_all)
...
For easy copy-paste from Scrapy/parsel code when needed :)
2024-12-04 14:28:37 +02:00
Karim shoair