feat(parser): replacing tldextract with tld library
This might break the adaptive data users have for websites BUT: 1. tld uses ~3.7x less memory during extraction operations (1.5 MB vs 5.7 MB). 2. tld uses ~56% less memory on import (5.2 MB vs 11.9 MB). 3. Zero dependencies (vs 3 for tldextract). In return, it's 30ms slower for extracting 5000 URLs, which is negligible. Also, the type hints aren't always accurate, but it's fine; I corrected them.
This commit is contained in:
@@ -128,6 +128,9 @@ def install(force): # pragma: no cover
|
||||
],
|
||||
"Playwright dependencies",
|
||||
)
|
||||
from tld.utils import update_tld_names
|
||||
|
||||
update_tld_names(fail_silently=True)
|
||||
# if no errors raised by the above commands, then we add the below file
|
||||
__PACKAGE_DIR__.joinpath(".scrapling_dependencies_installed").touch()
|
||||
else:
|
||||
|
||||
Reference in New Issue
Block a user