freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Commit Graph

  • d4766a070b build: update the website description Karim shoair 2026-02-10 21:54:17 +02:00
  • 0489923b06 build: pump up the version status Karim shoair 2026-02-10 21:53:00 +02:00
  • d3ee9a6cbb ops: remove branch v4 from workflows Karim shoair 2026-02-10 21:51:22 +02:00
  • 2dbd402000 docs: update links to use the current branch Karim shoair 2026-02-10 21:51:02 +02:00
  • f350800fd7 docs: updating Dynamic websites with protection page and some corrections Karim shoair 2026-02-10 21:48:09 +02:00
  • dba3e9d53e docs: updating dynamic websites page and some corrections Karim shoair 2026-02-10 21:23:14 +02:00
  • 8b1704a87e docs: use better import for ProxyRotator class Karim shoair 2026-02-10 21:20:43 +02:00
  • 4e3829fced style: expose ProxyRotator class to the public API Karim shoair 2026-02-10 21:20:07 +02:00
  • 8df74dc57b docs: updating http requests page and some corrections Karim shoair 2026-02-10 19:07:09 +02:00
  • f25fe64d3c docs: updating fetcher choosing page and some corrections Karim shoair 2026-02-10 19:01:50 +02:00
  • 87b99a407b docs: updating adaptive page and some corrections Karim shoair 2026-02-10 19:00:36 +02:00
  • 11bd4e4295 docs: updating main classes page and some corrections Karim shoair 2026-02-10 18:58:58 +02:00
  • 44e2c42eaf docs: updating selection page and some corrections Karim shoair 2026-02-10 18:42:37 +02:00
  • 989fdca201 docs: updating benchmarks page Karim shoair 2026-02-10 18:28:23 +02:00
  • a5abf1f3ba docs: updating overview page and some corrections Karim shoair 2026-02-10 18:25:31 +02:00
  • 8505c4a91c docs: possible fix for RTD zensical build Karim shoair 2026-02-09 01:50:22 +02:00
  • 33eb2ff56b docs: possible fix for RTD zensical build Karim shoair 2026-02-09 01:41:50 +02:00
  • 9b46cb35ab docs: possible fix for RTD zensical build Karim shoair 2026-02-09 01:16:04 +02:00
  • dc071ee6b8 docs: possible fix for RTD zensical build Karim shoair 2026-02-09 01:10:00 +02:00
  • 8ed71e4864 V4 (#130) Karim shoair 2026-02-09 00:55:21 +02:00
  • e688093974 docs: update the translated readme files with the new content Karim shoair 2026-02-09 00:46:58 +02:00
  • 3dc4bdcb4a docs: small adjustments to the readme file Karim shoair 2026-02-09 00:36:52 +02:00
  • 2a4d008927 docs: full update for main readme Karim shoair 2026-02-09 00:28:24 +02:00
  • 1f9eb17414 ops: update benchmarks script Karim shoair 2026-02-08 23:33:30 +02:00
  • 0c86d70307 docs: update the website main page Karim shoair 2026-02-08 23:33:10 +02:00
  • ca46070a1b docs: Improve README.md description formatting Karim shoair 2026-02-08 22:43:00 +02:00
  • 8d82d3c90d docs: update readme file Karim shoair 2026-02-08 22:39:50 +02:00
  • 9a108160c2 docs: move translated README files to docs folder again Karim shoair 2026-02-08 22:18:36 +02:00
  • 12b731b14b docs: move readme files to the root directory Karim shoair 2026-02-08 21:04:15 +02:00
  • 09cd5e6183 docs(readme): update translation links Karim shoair 2026-02-08 20:54:26 +02:00
  • 2ae11d21da docs: full switch from mkdocs to zensical Karim shoair 2026-02-08 20:46:38 +02:00
  • db41a6c4b6 docs: update pages with the new media for testing zensical Karim shoair 2026-02-08 20:46:14 +02:00
  • 0e2e9fb2ed docs: moving readme files to be outside of the website build Karim shoair 2026-02-08 20:45:03 +02:00
  • 219db79195 docs: update logo and media Karim shoair 2026-02-08 20:44:04 +02:00
  • 3eaa7ffc25 ops: exclude site directory from git Karim shoair 2026-02-08 20:42:35 +02:00
  • b1ce0b414b ops: Add mypy and pyright to the code quality workflow Karim shoair 2026-02-07 17:28:44 +02:00
  • 16852cb1ac style: Use shorter and more accurate naming for constants Karim shoair 2026-02-07 16:37:47 +02:00
  • ec7c7f8153 Merge branch 'main' into v4 Karim shoair 2026-02-07 16:30:37 +02:00
  • e777342741 build: pump up version and deps Karim shoair 2026-02-07 16:30:32 +02:00
  • 5ec929435b style: Fix all mypy errors and add type hints to untyped function bodies Karim shoair 2026-02-07 16:30:00 +02:00
  • 5f557a7f9f test: add tests for the new feature Karim shoair 2026-02-07 01:28:58 +02:00
  • 66476d42be feat(browsers): Add option to block requests to specific domains Karim shoair 2026-02-07 01:26:21 +02:00
  • d00c34ee7f style(parser): Improve the type hint for find_by_text and find_by_regex Karim shoair 2026-02-06 03:19:00 +02:00
  • 05f62fba81 fix(response): Force the body of the response to always be bytes Karim shoair 2026-02-06 03:14:23 +02:00
  • 9acc8194c5 test: update the tests accordingly Karim shoair 2026-02-06 02:45:13 +02:00
  • aa7a95fb70 feat(parser)!: Make all selection return selector objects by default Karim shoair 2026-02-06 02:43:51 +02:00
  • d8fe909c8b docs: make translation urls absolute Karim shoair 2026-02-05 02:44:12 +02:00
  • 151e4e7a7c docs: adding new sponsor Karim shoair 2026-02-05 02:39:51 +02:00
  • f88502718f feat(proxy control): Force a proxy at request level at any given point Karim shoair 2026-02-02 14:26:52 +02:00
  • 6e0277f23b test: add new tests for the proxy rotation logic Karim shoair 2026-02-02 00:16:40 +02:00
  • ef8c5bc7d6 feat(spiders/fetchers): Adding proxy rotation logic and change retry logic Karim shoair 2026-02-02 00:16:22 +02:00
  • 65f1a23358 fix(browsers): Improve stealth and speed by adjusting flags Karim shoair 2026-01-29 00:58:04 +02:00
  • 0c3d359b46 fix(browsers): disable autoplay for all browsers Karim shoair 2026-01-29 00:57:30 +02:00
  • 90c52c45c7 feat(parser): replacing tldextract with tld library Karim shoair 2026-01-23 00:00:48 +02:00
  • 5dbbd84b3d fix(requests): return the cached fingerprint on second call Karim shoair 2026-01-20 20:28:13 +02:00
  • d90189e675 test: update tests accordingly Karim shoair 2026-01-20 20:22:47 +02:00
  • 122923c80e feat(spiders/requests): Use a reliable method for fingerprint generation Karim shoair 2026-01-20 20:21:42 +02:00
  • 5aa178a76e style: update post requests type hints to be more accurate Karim shoair 2026-01-20 18:27:52 +02:00
  • d0f52c3855 test: remove the tests for the removed html_utils module Karim shoair 2026-01-20 16:37:17 +02:00
  • 95955af3be build: Add w3lib to deps Karim shoair 2026-01-20 16:36:34 +02:00
  • 0ec78e2335 test: update tests accordingly Karim shoair 2026-01-19 16:40:18 +02:00
  • 6947066761 feat(spiders): Make graceful shutdown always enabled Karim shoair 2026-01-19 16:40:02 +02:00
  • 94bc759c6a test: updates tests to include new logic Karim shoair 2026-01-19 16:13:29 +02:00
  • 25850db92c fix(spiders): handle errors with manual pause during stream mode Karim shoair 2026-01-19 16:13:03 +02:00
  • fbc8be5bcc fix(spiders): fix a bug with session stats counter Karim shoair 2026-01-19 16:00:06 +02:00
  • 4ecf8a64d3 fix(spiders): Fix for when the pausing system has large queue Karim shoair 2026-01-19 15:04:40 +02:00
  • cf0de7ea6d Merge branch 'main' into v4 Karim shoair 2026-01-19 14:47:59 +02:00
  • 9229cef895 docs: adding new sponsor Karim shoair 2026-01-19 14:12:32 +02:00
  • d247b5b691 test: add tests for the spiders system Karim shoair 2026-01-19 03:46:00 +02:00
  • 0d003f62e3 ops: update bandit checklist Karim shoair 2026-01-19 03:45:39 +02:00
  • 1f0f86e3a2 feat(spiders): Add pause/resume system for crawls Karim shoair 2026-01-19 03:43:10 +02:00
  • acdb39988c ops: update bandit checklist Karim shoair 2026-01-19 03:40:57 +02:00
  • bf5aa021d4 fix(browsers): solving an issues with leaving playwright loop open when cdp connection fails Karim shoair 2026-01-18 16:46:48 +02:00
  • 02c9dffc77 style(spiders): make parse function abstract Karim shoair 2026-01-18 00:01:04 +02:00
  • afe6b2e97b feat(spiders): Change items hook to make it for processing items + add a stat for this Karim shoair 2026-01-17 20:52:49 +02:00
  • 41c7a7e793 fix(spiders engine): Improve logging Karim shoair 2026-01-17 19:57:04 +02:00
  • dac854a129 fix(parser): handle responses with empty body Karim shoair 2026-01-13 02:35:26 +02:00
  • 35d570120a feat(spiders): Add option to logging format and date format Karim shoair 2026-01-12 17:11:52 +02:00
  • 3466bcab2c feat: add hook to run after collecting an item Karim shoair 2026-01-12 15:58:33 +02:00
  • e0f94cf7d0 feat(spiders): Add a stream mode to collect items in realtime Karim shoair 2026-01-12 15:53:29 +02:00
  • 927cfd85ff feat(spiders): The option to start sessions lazily Karim shoair 2026-01-12 02:09:50 +02:00
  • f2c6216968 style: remove unwanted import from spiders main init file Karim shoair 2026-01-12 01:21:26 +02:00
  • 72c3c2792a feat(spiders): add simple export system for the results Karim shoair 2026-01-12 01:21:05 +02:00
  • a17bb1d976 feat(browsers): Add option to retry tabs that gives errors Karim shoair 2026-01-11 23:40:26 +02:00
  • 5a11c2466b fix: make retry_blocked_request take the response as well Karim shoair 2026-01-11 22:54:17 +02:00
  • 736873a7c6 feat(spiders): Add logic to detect blocked requests and retry them Karim shoair 2026-01-11 21:56:57 +02:00
  • 774364b6d3 style(spiders): make the usage of uvloop optional Karim shoair 2026-01-11 19:57:03 +02:00
  • 0bfdab7158 feat: spiders use uvloop/winloop when available Karim shoair 2026-01-11 18:39:42 +02:00
  • d5b9ed11b2 fix(browsers): Close pages that give error Karim shoair 2026-01-11 18:34:13 +02:00
  • 32f4f2adf4 build: Add AnyIO to the dependencies Karim shoair 2026-01-11 17:01:33 +02:00
  • 059a708b6d feat(spiders system): a prototype of the new spiders system Karim shoair 2026-01-11 16:53:18 +02:00
  • dd5c30c521 feat(response): Option to set the referer as the previous response url. Karim shoair 2026-01-11 03:24:30 +02:00
  • 1443a469a3 fix: handle a bug with response follow referer flow argument Karim shoair 2026-01-11 03:22:51 +02:00
  • aa4817f349 feat: Change logger to be flexible enough for the spiders classes Karim shoair 2026-01-11 03:22:20 +02:00
  • c5bc82deb8 fix(browsers): solve a bug with setting referer on the request level Karim shoair 2026-01-11 03:21:03 +02:00
  • 8e36962177 ops: add local Claude files to gitignore Karim shoair 2026-01-10 20:41:23 +02:00
  • 0634f5796c feat(spiders): Add follow function to the response Karim shoair 2026-01-10 20:39:43 +02:00
  • f80e8828f7 refactor: internal API changes Karim shoair 2026-01-08 01:53:48 +02:00
  • 3a360acced build: pump version up temporary Karim shoair 2026-01-08 01:37:47 +02:00
  • 08f3e6efad fix: add missing line Karim shoair 2026-01-08 01:01:00 +02:00