freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update the cli section

Browse Source
This commit is contained in:
Karim shoair
2025-12-31 03:00:26 +02:00
parent 3b9a6d021e
commit ad234165e9
3 changed files with 22 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cli/extract-commands.md
+17 -20
View File
@@ -2,7 +2,7 @@
**Web Scraping through the terminal without requiring any programming!**
The `scrapling extract` Command lets you download and extract content from websites directly from your terminal without writing any code. Ideal for beginners, researchers, and anyone requiring rapid web data extraction.
The `scrapling extract` command lets you download and extract content from websites directly from your terminal without writing any code. Ideal for beginners, researchers, and anyone requiring rapid web data extraction.
> 💡 **Prerequisites:**
>
@@ -30,7 +30,7 @@ The extract command is a set of simple terminal tools that:
```bash
scrapling extract get "https://example.com" page_content.txt
```
This does an HTTP GET request and saves the text content of the webpage to `page_content.txt`.
This makes an HTTP GET request and saves the webpage's text content to `page_content.txt`.
- **Save as Different Formats**
@@ -73,13 +73,13 @@ Commands:
stealthy-fetch Use StealthyFetcher to fetch content with advanced...
```
We will go through each Command in detail below.
We will go through each command in detail below.
### HTTP Requests
1. **GET Request**
The most common Command for downloading website content:
The most common command for downloading website content:
```bash
scrapling extract get [URL] [OUTPUT_FILE] [OPTIONS]
@@ -105,7 +105,7 @@ We will go through each Command in detail below.
# Add multiple headers
scrapling extract get "https://site.com" page.html -H "Accept: text/html" -H "Accept-Language: en-US"
```
Get the available options for the Command with `scrapling extract get --help` as follows:
Get the available options for the command with `scrapling extract get --help` as follows:
```bash
Usage: scrapling extract get [OPTIONS] URL OUTPUT_FILE
@@ -143,7 +143,7 @@ We will go through each Command in detail below.
# Send JSON data
scrapling extract post "https://api.site.com" response.json --json '{"username": "test", "action": "search"}'
```
Get the available options for the Command with `scrapling extract post --help` as follows:
Get the available options for the command with `scrapling extract post --help` as follows:
```bash
Usage: scrapling extract post [OPTIONS] URL OUTPUT_FILE
@@ -182,7 +182,7 @@ We will go through each Command in detail below.
# Send JSON data
scrapling extract put "https://scrapling.requestcatcher.com/put" response.json --json '{"username": "test", "action": "search"}'
```
Get the available options for the Command with `scrapling extract put --help` as follows:
Get the available options for the command with `scrapling extract put --help` as follows:
```bash
Usage: scrapling extract put [OPTIONS] URL OUTPUT_FILE
@@ -220,7 +220,7 @@ We will go through each Command in detail below.
# Send JSON data
scrapling extract delete "https://scrapling.requestcatcher.com/" response.txt --impersonate "chrome"
```
Get the available options for the Command with `scrapling extract delete --help` as follows:
Get the available options for the command with `scrapling extract delete --help` as follows:
```bash
Usage: scrapling extract delete [OPTIONS] URL OUTPUT_FILE
@@ -263,7 +263,7 @@ We will go through each Command in detail below.
# Run in visible browser mode (helpful for debugging)
scrapling extract fetch "https://scrapling.requestcatcher.com/" page.html --no-headless --disable-resources
```
Get the available options for the Command with `scrapling extract fetch --help` as follows:
Get the available options for the command with `scrapling extract fetch --help` as follows:
```bash
Usage: scrapling extract fetch [OPTIONS] URL OUTPUT_FILE
@@ -279,10 +279,8 @@ We will go through each Command in detail below.
--wait INTEGER Additional wait time in milliseconds after page load (default: 0)
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
--wait-selector TEXT CSS selector to wait for before proceeding
--locale TEXT Browser locale (default: en-US)
--stealth / --no-stealth Enable stealth mode (default: False)
--hide-canvas / --show-canvas Add noise to canvas operations (default: False)
--disable-webgl / --enable-webgl Disable WebGL support (default: False)
--locale TEXT Specify user locale. Defaults to the system default locale.
---real-chrome/--no-real-chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)
--proxy TEXT Proxy URL in format "http://username:password@host:port"
-H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times)
--help Show this message and exit.
@@ -304,10 +302,10 @@ We will go through each Command in detail below.
# Solve Cloudflare challenges
scrapling extract stealthy-fetch "https://nopecha.com/demo/cloudflare" data.txt --solve-cloudflare --css-selector "#padded_content a"
# Use proxy for anonymity
# Use a proxy for anonymity.
scrapling extract stealthy-fetch "https://site.com" content.md --proxy "http://proxy-server:8080"
```
Get the available options for the Command with `scrapling extract stealthy-fetch --help` as follows:
Get the available options for the command with `scrapling extract stealthy-fetch --help` as follows:
```bash
Usage: scrapling extract stealthy-fetch [OPTIONS] URL OUTPUT_FILE
@@ -317,25 +315,24 @@ We will go through each Command in detail below.
Options:
--headless / --no-headless Run browser in headless mode (default: True)
--block-images / --allow-images Block image loading (default: False)
--disable-resources / --enable-resources Drop unnecessary resources for speed boost (default: False)
--block-webrtc / --allow-webrtc Block WebRTC entirely (default: False)
--humanize / --no-humanize Humanize cursor movement (default: False)
--solve-cloudflare / --no-solve-cloudflare Solve Cloudflare challenges (default: False)
--allow-webgl / --block-webgl Allow WebGL (default: True)
--network-idle / --no-network-idle Wait for network idle (default: False)
--disable-ads / --allow-ads Install uBlock Origin addon (default: False)
---real-chrome/--no-real-chrom If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)
--hide-canvas/--show-canvas Add noise to canvas operations (default: False)
--timeout INTEGER Timeout in milliseconds (default: 30000)
--wait INTEGER Additional wait time in milliseconds after page load (default: 0)
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
--wait-selector TEXT CSS selector to wait for before proceeding
--geoip / --no-geoip Use IP/Proxy geolocation for timezone/locale (default: False)
--hide-canvas / --show-canvas Add noise to canvas operations (default: False)
--proxy TEXT Proxy URL in format "http://username:password@host:port"
-H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times)
--help Show this message and exit.
```
## When to use each Command
## When to use each command
If you are not a Web Scraping expert and can't decide what to choose, you can use the following formula to help you decide:
docs/cli/interactive-shell.md
+4 -2
View File
@@ -4,7 +4,7 @@
**Powerful Web Scraping REPL for Developers and Data Scientists**
The Scrapling Interactive Shell is an enhanced IPython-based environment designed specifically for Web Scraping tasks. It provides instant access to all Scrapling features, clever shortcuts, automatic page management, and advanced tools like curl command conversion.
The Scrapling Interactive Shell is an enhanced IPython-based environment designed specifically for Web Scraping tasks. It provides instant access to all Scrapling features, clever shortcuts, automatic page management, and advanced tools, such as conversion of the curl command.
> 💡 **Prerequisites:**
>
@@ -129,7 +129,9 @@ View scraped pages in your browser:
### Curl Command Integration
The shell provides a few functions to help you convert curl commands from the browser DevTools to `Fetcher` requests, which are `uncurl` and `curl2fetcher`. First, you need to copy a request as a curl command like the following:
The shell provides a few functions to help you convert curl commands from the browser DevTools to `Fetcher` requests: `uncurl` and `curl2fetcher`.
First, you need to copy a request as a curl command like the following:
<img src="../../assets/scrapling_shell_curl.png" title="Copying a request as a curl command from Chrome" alt="Copying a request as a curl command from Chrome" style="width: 70%;"/>
docs/cli/overview.md
+1 -1
View File
@@ -27,4 +27,4 @@ and the installation of the fetchers' dependencies with the following command
```bash
scrapling install
```
This downloads all browsers with their system dependencies and fingerprint manipulation dependencies.
This downloads all browsers, along with their system dependencies and fingerprint manipulation dependencies.