Big structure changes (check commit description)

- Moved most of the parser functions/files to the core package.
- Converted tools file to a package and made separate files for similar functions.
- Now all fetcher engines return a Response object
- Instead of selecting an engine to use and passing config to it, we have separate fetcher classes so the user can choose what to use while importing.
- I added a new custom fetcher so the user can create and use an engine.
- More...
This commit is contained in:
Karim shoair
2024-11-03 01:04:02 +02:00
parent 7f1c06e7bf
commit 145c03daff
21 changed files with 565 additions and 329 deletions
+6 -6
View File
@@ -1,12 +1,12 @@
import os
from difflib import SequenceMatcher
from scrapling.translator import HTMLTranslator
from scrapling.mixins import SelectorsGeneration
from scrapling.custom_types import TextHandler, AttributesHandler
from scrapling.storage_adaptors import SQLiteStorageSystem, StorageSystemMixin, _StorageTools
from scrapling.utils import setup_basic_logging, logging, clean_spaces, flatten, html_forbidden
from scrapling._types import Any, Dict, List, Tuple, Optional, Pattern, Union, Callable, Generator, SupportsIndex
from scrapling.core.translator import HTMLTranslator
from scrapling.core.mixins import SelectorsGeneration
from scrapling.core.custom_types import TextHandler, AttributesHandler
from scrapling.core.storage_adaptors import SQLiteStorageSystem, StorageSystemMixin, _StorageTools
from scrapling.core.utils import setup_basic_logging, logging, clean_spaces, flatten, html_forbidden
from scrapling.core._types import Any, Dict, List, Tuple, Optional, Pattern, Union, Callable, Generator, SupportsIndex
from lxml import etree, html
from cssselect import SelectorError, SelectorSyntaxError, parse as split_selectors