Karim shoair
67aa1a2ae7
tests: Add a test for using stealth and real_chrome arguments together
2025-09-20 00:23:55 +03:00
Karim shoair
67db865aee
feat(DynamicFetcher): Replace rebrowser with patchright
2025-09-20 00:23:29 +03:00
Karim shoair
386da62ec9
tests: fix CLI tests according to last changes
2025-09-19 23:17:54 +03:00
Karim shoair
0ee3f93227
Merge branch 'main' into dev
2025-09-19 07:01:53 +03:00
Karim shoair
165dc524f8
docs: add a new sponsor
2025-09-19 06:06:30 +03:00
Karim shoair
2d704b2a8b
fix(shell): Fixing a bug with content converting
2025-09-19 04:50:15 +03:00
Karim shoair
a9d05cc0ef
style: Removing dead code/docstrings and correcting type hints
2025-09-19 04:49:30 +03:00
Karim shoair
bf5fe6d451
fix(shell): Fix view command edge case
2025-09-19 04:21:15 +03:00
Karim shoair
504ecdf7dc
fix(shell): Fix parsing curl --data-raw
2025-09-19 04:20:16 +03:00
Karim shoair
b197f8a5f9
fix(parser): Improve selectors re function
2025-09-18 13:56:37 +03:00
Karim shoair
41c8a2dcca
Merge branch 'main' into dev
2025-09-18 13:55:48 +03:00
Karim shoair
189f323904
misc: update sponsor poster
2025-09-18 13:31:17 +03:00
Karim shoair
ddf5b52e3b
Improve page management and speed up execution ( #87 )
2025-09-18 03:34:40 +03:00
Karim shoair
6b2675c7dd
v0.3.4 ( #86 )
2025-09-16 15:50:22 +03:00
Karim shoair
1550ba64bb
docs: Adding new sponsor
2025-09-16 15:35:22 +03:00
Karim shoair
570303f938
fix(parser): Make html_content and prettify return strings not bytes (depends on the encoding)
2025-09-16 01:58:51 +03:00
Karim shoair
5683965367
fix(sessions): Fixing a bug with max pages counter
2025-09-16 01:47:31 +03:00
Karim shoair
b7327466cf
build: pump version up
2025-09-15 23:35:08 +03:00
Karim shoair
05c2d9aa00
feat(CLI): Make the session classes available in the shell
2025-09-15 23:34:36 +03:00
Karim shoair
5bcd4c2927
fix: A workaround for Playwright issue with page content retrieving on windows
2025-09-15 21:34:20 +03:00
Karim shoair
bdcf96fa9e
Merge branch 'main' into dev
2025-09-15 21:33:49 +03:00
Karim shoair
1547366c85
fix: fixing body issue with cli view command
2025-09-15 21:32:57 +03:00
Karim shoair
8a54745e47
v0.3.3 ( #84 )
2025-09-15 14:06:24 +03:00
Karim shoair
f3a1f1ecb4
Merge branch 'main' into dev
2025-09-15 13:52:40 +03:00
Karim shoair
a6aabda67c
build: Pump version up
2025-09-15 13:52:09 +03:00
Karim shoair
7a7097b775
fix: Stop removing browser default page
...
Looks like it's problematic with some systems so it doesn't worth the trouble
2025-09-15 13:43:47 +03:00
Karim shoair
fac92ef260
v0.3.2 ( #83 )
2025-09-15 04:28:55 +03:00
Karim shoair
3bf06a8521
docs: clarifying updates
2025-09-15 01:39:19 +03:00
Karim shoair
f848e6b7f2
fix: Fixes to handle the new extra deps group
2025-09-15 00:19:19 +03:00
Karim shoair
c7b9d45232
ops: Make tests workflow doesn't run when the changes isn't code related
2025-09-14 21:35:58 +03:00
Karim shoair
d259b85c5b
docs: Updating benchmarks with the latest version of all libraries
2025-09-14 21:26:28 +03:00
Karim shoair
fdac239fcc
build: Move click library to be part of the fetchers extra
2025-09-14 20:32:46 +03:00
Karim shoair
eae088a07b
tests: Multiple changes to tests
...
- Remove all tests for the old encoding logic
- Stop using the `nopecha` test page to test Cloudflare solver
- Remove useless tests like testing for infinite timeout
- Fixes to make the code compatible with new changes
2025-09-14 20:24:46 +03:00
Karim shoair
ce4fc31f2c
feat: Make .body return the passed content as it is without any processing
...
This makes it possible to download files and deal with non-HTML requests (ex: #81 )
2025-09-14 20:21:45 +03:00
Karim shoair
67ca139ff9
fix: Fixes for multiple encoding issues ( #80 & #81 )
2025-09-14 20:20:29 +03:00
Karim shoair
68615b2347
tests: remove the cloudflare test from the sync version
...
To not put pressure on the website
2025-09-14 16:31:38 +03:00
Karim shoair
1024ba6916
tests: Update test to be up-to-date with current version of the code
2025-09-14 16:30:59 +03:00
Karim shoair
0b3509a295
fix(shell): Fix a small typo
2025-09-14 05:05:55 +03:00
Karim shoair
f604485a09
docs: Update all pages related to last changes
2025-09-14 04:45:56 +03:00
Karim shoair
d6844232eb
tests: Update all tests according to new update
2025-09-13 16:07:54 +03:00
Karim shoair
13d7e70cb7
refactor: Make all fetchers as an optional dependency group
...
+ Removing some dead code
2025-09-13 16:07:48 +03:00
Karim shoair
d7e3deae2a
fix: fix invalid dep version
2025-09-13 05:10:56 +03:00
Karim shoair
521366e77e
build: bump up deps
2025-09-13 04:29:09 +03:00
Karim shoair
80077953f6
docs: update all pages related to last changes
2025-09-13 04:19:16 +03:00
Karim shoair
8c4d6158d6
feat(browser fetchers): A new option to control page JS
2025-09-13 03:59:12 +03:00
Karim shoair
aaec00563c
ops: update pre-commit hooks
2025-09-13 03:34:28 +03:00
Karim shoair
330d03559c
style: applying the new ruff rules to all files
2025-09-13 03:22:53 +03:00
Karim shoair
60be9dc816
feat: Validate all fetch-level parameters
...
So it's validated like the session-level ones. Needs improvement, but that's for later
2025-09-13 03:22:08 +03:00
Karim shoair
831eafca34
ops: Update ruff pre-commit rules
2025-09-13 03:07:15 +03:00
Karim shoair
0c7db3c9d9
fix: validation keywords
2025-09-13 02:07:04 +03:00
Karim shoair
adb82008e5
style: Using keywords for validation
...
Less lines of code and more stylish
2025-09-12 04:33:00 +03:00
Karim shoair
e4b1e00e63
build: Pump up the version
2025-09-11 04:14:32 +03:00
Karim shoair
05a714980d
docs: Update all docstring according to the new changes
2025-09-11 04:13:36 +03:00
Karim shoair
9838d741d0
refactor/feat(browser fetchers): make it possible to have a configuration per page in sessions
...
- Also, no need for the `page_action` argument function to return the page again
2025-09-11 03:48:55 +03:00
Karim shoair
931993f901
tests: changing tests accordingly
2025-09-08 16:09:24 +03:00
Karim shoair
ecaec65049
refactor/fix(browser fetchers): Solving a logic bug in page rotation
...
- Pages that get reused in rotation are possibly contaminated from previous settings used on them, and some are stubborn to remove, so the new approach replaces finished pages with new ones.
- Removed the `max_pages` from sync `StealthySession` to match `DynamicSession` (Doesn't mean anything in sync code)
2025-09-08 16:08:57 +03:00
Karim shoair
f298b76b8e
perf(StealthySession): Close persistent context default page
2025-09-07 18:48:51 +03:00
Karim shoair
710f994339
perf(DynamicSession): Close persistent context default page
2025-09-07 18:48:40 +03:00
Karim shoair
05761d4070
docs: Add YT video to the mcp page
2025-09-07 05:06:35 +03:00
Karim shoair
4341477a66
perf(StealthyFetcher): Slight optimization
2025-09-07 04:58:55 +03:00
Karim shoair
9e5ff5bb1b
chore: Add issue template for other issues
2025-09-07 04:36:00 +03:00
Karim shoair
72e26c23e8
fix(DynamicFetcher): Improve stealth mode
2025-09-07 03:13:51 +03:00
Karim shoair
31c14b3997
[ImgBot] Optimize images ( #79 )
2025-09-05 13:51:45 +03:00
Karim shoair
fa76d64d61
docs: add new sponsor to the README
2025-09-05 13:46:17 +03:00
Karim shoair
6f232b3874
misc: add new sponsor logo
2025-09-05 13:44:37 +03:00
Karim shoair
23997f2755
v0.3.1 ( #78 )
2025-09-02 17:20:53 +03:00
Karim shoair
934d8a1452
build: Pump up the version
2025-09-02 16:33:52 +03:00
Karim shoair
e06f8d36db
docs: add init_script new argument
2025-09-02 16:32:19 +03:00
Karim shoair
6ad1bab2e4
feat(DynamicFetcher): add the ability to add init_script to context ( #56 )
2025-09-02 16:25:13 +03:00
Karim shoair
9543f171aa
feat(StealthyFetcher): add the ability to add init_script to context
2025-09-02 16:19:22 +03:00
Karim shoair
d49e047736
tests: change tests accordingly
2025-09-02 15:23:30 +03:00
Karim shoair
af9a62a539
fix: fix for shell imported by mistake issue ( #76 )
2025-09-02 15:23:15 +03:00
Karim shoair
e7482a267a
docs: add interactive shell video to README.md
2025-09-01 22:11:40 +03:00
Karim shoair
9021c4f9b7
docs: Changing sponsors order
2025-09-01 16:03:47 +03:00
Karim shoair
d5aaeb9816
v0.3
2025-09-01 07:26:15 +03:00
Karim shoair
812cf2d21b
build: Pump up the version
2025-09-01 06:43:41 +03:00
Karim shoair
9e9e79d9b0
docs: fixes and adjustments
2025-09-01 06:39:17 +03:00
Karim shoair
35b88fbb62
docs: add an api reference to the mcp server
2025-09-01 01:02:08 +03:00
Karim shoair
7befcd54bc
chore: update the setup file description
2025-08-31 23:50:33 +03:00
Karim shoair
e7858d7307
docs: updating the mkdocs file accordingly
2025-08-31 23:41:53 +03:00
Karim shoair
e0c005b2cc
docs: add a page to the mcp server
2025-08-31 23:40:50 +03:00
Karim shoair
734e5f73db
docs: Changing fonts and fixing some mistakes
2025-08-31 18:52:15 +03:00
Karim shoair
878be9ccbf
tests: update tests accordingly
2025-08-31 02:57:30 +03:00
Karim shoair
967c0956a1
feat(DynamicSession): Improve stealth mode
2025-08-31 02:57:12 +03:00
Karim shoair
6eb580a6d0
feat(DynamicSession): Improve normal fetches speed
2025-08-31 02:56:34 +03:00
Karim shoair
b5be2547d7
fix(dynamicSession): issue with headless
2025-08-31 01:29:47 +03:00
Karim shoair
7153fdbcce
Merge branch 'dev' of https://github.com/D4Vinci/Scrapling into dev
2025-08-30 22:42:09 +03:00
Karim shoair
beebddd6fe
docs: add cli overview page
2025-08-30 22:42:05 +03:00
Karim shoair
c97a82f2d4
docs: add the page for the extract command
2025-08-30 22:35:44 +03:00
Karim shoair
47dd2ed28f
Merge branch 'main' into dev
2025-08-30 22:35:25 +03:00
Karim shoair
76721f906e
docs(cli): Correcting docstrings
2025-08-30 22:12:33 +03:00
Karim shoair
9dfca4c6f2
misc: Add ko-fi username to FUNDING.yml
2025-08-30 22:08:25 +03:00
Karim shoair
bf55205436
docs: Update README.md
2025-08-30 20:11:43 +03:00
Karim shoair
8edcd611d3
docs: replace the test website
2025-08-30 18:51:31 +03:00
Karim shoair
c66c48e3ae
fix(extract): Correcting docs for the css-selector option
2025-08-29 21:34:39 +03:00
Karim shoair
cd388059e8
docs: Add a page about the interactive shell
2025-08-29 20:31:00 +03:00
Karim shoair
badc3b21e1
docs: update article A Free Alternative to AI for Robust Web Scraping
2025-08-28 23:18:56 +03:00
Karim shoair
6d419e0f4d
docs: update BS article
2025-08-28 05:01:38 +03:00
Karim shoair
e14522f57a
docs: update the StealthyFetcher page
2025-08-28 04:53:22 +03:00
Karim shoair
1f87210b93
docs: Add max_pages explanation to DynamicFetcher
2025-08-28 04:36:49 +03:00
Karim shoair