Commit Graph
7 Commits
Author SHA1 Message Date
Toni f4d7aa45fe Improved Polish gambling sites fetch script. (#2315)
* fails quite often in the CI, so ignore potential xmllint error

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-02-10 13:33:56 +01:00
Toni 5fb631c8fe Improved belgium gambling sites regex. (#2184)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-29 16:15:23 +01:00
Luca Deri 36abf06c6f Swap from Aho-Corasick to an experimental/home-grown algorithm that uses a probabilistic
approach for handling Internet domain names.

For switching back to Aho-Corasick it is necessary to edit
ndpi-typedefs.h and uncomment the line
// #define USE_LEGACY_AHO_CORASICK

[1] With Aho-Corasick
$ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory
nDPI Memory statistics:
nDPI Memory (once):      37.34 KB
Flow Memory (per flow):  960 B
Actual Memory:           33.09 MB
Peak Memory:             33.09 MB

[2] With the new algorithm
$ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory
nDPI Memory statistics:
nDPI Memory (once):      37.31 KB
Flow Memory (per flow):  960 B
Actual Memory:           7.42 MB
Peak Memory:             7.42 MB

In essence from ~33 MB to ~7 MB

This new algorithm will enable larger lists to be loaded (e.g. top 1M domans
https://s3-us-west-1.amazonaws.com/umbrella-static/index.html)

In ./lists there are file names that are named as <category>_<string>.list
With -G ndpiReader can load all of them at startup
2023-08-29 17:34:04 +02:00
Luca Deri eeeee46b1e Changes for supporinng more efficient sub-string matching 2023-08-26 17:55:50 +02:00
snicket2100 abee1a2a6f Included Gambling website data from the Polish hazard.mf.gov.pl list (#2041)
* Refreshed the Belgium Gambling Site list data

Unfortunately some hostnames have been removed from that list,
which means they are disappearing from the `ndpi_gambling_match.c.inc`
file as well.

* build: added `libxml2-utils` (for `xmllint`)

* Included Gambling website data from the Polish `hazard.mf.gov.pl` list

The list contains over 30k gambling website hostnames as of today.
2023-07-14 09:55:46 +02:00
Toni 6da3474203 Improved helper scripts. (#1986)
* added additional (more restrictive) checks

Signed-off-by: lns <matzeton@googlemail.com>
2023-05-28 12:45:44 +02:00
Toni c97e2d67ff Added scripts to auto generate hostname/SNI *.inc files. (#1984)
* add illegal gambling sites (Belgium)

Signed-off-by: lns <matzeton@googlemail.com>
2023-05-20 15:41:15 +02:00