approach for handling Internet domain names. For switching back to Aho-Corasick it is necessary to edit ndpi-typedefs.h and uncomment the line // #define USE_LEGACY_AHO_CORASICK [1] With Aho-Corasick $ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory nDPI Memory statistics: nDPI Memory (once): 37.34 KB Flow Memory (per flow): 960 B Actual Memory: 33.09 MB Peak Memory: 33.09 MB [2] With the new algorithm $ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory nDPI Memory statistics: nDPI Memory (once): 37.31 KB Flow Memory (per flow): 960 B Actual Memory: 7.42 MB Peak Memory: 7.42 MB In essence from ~33 MB to ~7 MB This new algorithm will enable larger lists to be loaded (e.g. top 1M domans https://s3-us-west-1.amazonaws.com/umbrella-static/index.html) In ./lists there are file names that are named as <category>_<string>.list With -G ndpiReader can load all of them at startup
23 lines
940 B
Bash
Executable File
23 lines
940 B
Bash
Executable File
#!/usr/bin/env bash
|
|
|
|
set -e
|
|
|
|
cd "$(dirname "${0}")" || exit 1
|
|
. ./common.sh || exit 1
|
|
|
|
# NDPI_PROTOCOL_CATEGORY_GAMBLING = 107
|
|
LIST=../lists/107_gambling.list
|
|
|
|
printf '(1) %s\n' "Scraping Illegal Gambling Sites (Belgium)"
|
|
DOMAINS="$(curl -s 'https://www.gamingcommission.be/en/gaming-commission/illegal-games-of-chance/list-of-illegal-gambling-sites' | sed -n 's/^<td[^>]\+>\(.\+\.[a-zA-Z0-9]\+\)\(\|\/.*[^<]*\)<\/td>/\1/gp' || exit 1)"
|
|
is_str_empty "${DOMAINS}" "Please check gambling sites URL and sed REGEX."
|
|
|
|
printf '(2) %s\n' "Downloading Gambling Sites (Poland)"
|
|
DOMAINS_PL="$(curl -s https://hazard.mf.gov.pl/api/Register | xmllint --xpath "/*[local-name(.)='Rejestr']/*[local-name(.)='PozycjaRejestru']/*[local-name(.)='AdresDomeny']/text()" -)"
|
|
is_str_empty "${DOMAINS_PL}" "Please check gambling sites URL and XPath."
|
|
|
|
echo "${DOMAINS}" "${DOMAINS_PL}" | sort | uniq >${LIST}
|
|
|
|
printf '(3) %s\n' "List ${LIST} is now ready"
|
|
exit 0
|