feat(parser): replacing tldextract with tld library

This might break the adaptive data users have for websites BUT:
1. tld uses ~3.7x less memory during extraction operations (1.5 MB vs 5.7 MB).
2. tld uses ~56% less memory on import (5.2 MB vs 11.9 MB).
3. Zero dependencies (vs 3 for tldextract).

In return, it's 30ms slower for extracting 5000 URLs, which is negligible. Also, the type hints aren't always accurate, but it's fine; I corrected them.
This commit is contained in:
Karim shoair
2026-01-23 00:00:48 +02:00
parent 5dbbd84b3d
commit 90c52c45c7
4 changed files with 24 additions and 11 deletions
+1 -1
View File
@@ -60,7 +60,7 @@ dependencies = [
"lxml>=6.0.2",
"cssselect>=1.3.0",
"orjson>=3.11.5",
"tldextract>=5.3.1",
"tld>=0.13.1",
"w3lib>=2.3.0"
]