Commit Graph
4262 Commits
Author SHA1 Message Date
Luca Deri 3dbc4f65fc Added README file 2023-08-29 17:45:36 +02:00
Luca Deri 36abf06c6f Swap from Aho-Corasick to an experimental/home-grown algorithm that uses a probabilistic
approach for handling Internet domain names.

For switching back to Aho-Corasick it is necessary to edit
ndpi-typedefs.h and uncomment the line
// #define USE_LEGACY_AHO_CORASICK

[1] With Aho-Corasick
$ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory
nDPI Memory statistics:
nDPI Memory (once):      37.34 KB
Flow Memory (per flow):  960 B
Actual Memory:           33.09 MB
Peak Memory:             33.09 MB

[2] With the new algorithm
$ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory
nDPI Memory statistics:
nDPI Memory (once):      37.31 KB
Flow Memory (per flow):  960 B
Actual Memory:           7.42 MB
Peak Memory:             7.42 MB

In essence from ~33 MB to ~7 MB

This new algorithm will enable larger lists to be loaded (e.g. top 1M domans
https://s3-us-west-1.amazonaws.com/umbrella-static/index.html)

In ./lists there are file names that are named as <category>_<string>.list
With -G ndpiReader can load all of them at startup
2023-08-29 17:34:04 +02:00
Toni Uhlig 1f693c3f5a Added lists/gambling.list to extra dist.
* make dist in `./fuzz`: fixed inconsistent `*.dict` file pattern

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-27 15:53:31 +02:00
Toni Uhlig afd250f5d9 fix compilation and symbol check
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-27 15:32:42 +02:00
Toni Uhlig 25f3a7cd1d fix compilation
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-27 15:17:56 +02:00
Luca Deri 34986de297 Search fixes 2023-08-26 19:47:50 +02:00
Luca Deri eeeee46b1e Changes for supporinng more efficient sub-string matching 2023-08-26 17:55:50 +02:00
Luca Deri 4ca94369e1 Improved domain search tet unit 2023-08-26 10:43:19 +02:00
Luca Deri ac00542df2 LEak fix 2023-08-26 10:10:49 +02:00
Luca Deri 2c565c77c9 Added ndpi_domain_classify_XXX(0 API 2023-08-26 00:24:33 +02:00
Luca Deri 42fcb011ad Warning fix 2023-08-25 19:29:43 +02:00
Luca Deri eb9932164f added bimap and/or with allocation 2023-08-24 07:45:38 +02:00
Luca Deri ff4f291afe Minor improvements 2023-08-23 11:04:31 +02:00
Luca 95fcd08687 Added ndpi_bitmap_is_empty() and ndpi_bitmap_optimize() API calls 2023-08-23 11:03:59 +02:00
Luca 9933b5910d Boundary check 2023-08-21 23:19:57 +02:00
Luca bd25df3334 Improved Wireguard detection 2023-08-21 13:40:42 +02:00
Luca 9e4bcc7771 Removed empty line 2023-08-21 13:40:30 +02:00
Luca 6e5466f9e5 Added ndpi_bitmap_andnot API call 2023-08-21 13:26:38 +02:00
Ivan Nardi cc4461f424 fuzz: extend coverage (#2073) 2023-08-20 15:18:19 +02:00
Ivan Nardi 5867f43fae Fix compilation on Windows (#2072) 2023-08-14 22:51:09 +02:00
Luca Deri cfbb7a423e Replaces free() with ndpi_free() 2023-08-14 18:51:11 +02:00
Luca Deri 049fbf819c Reworked ndpi_filter_xxx implementation using compressed bitmaps 2023-08-14 12:50:54 +02:00
Luca Deri 3b34941be1 Typo 2023-08-12 11:39:19 +02:00
Luca Deri 9a44954365 ARM compilation fix 2023-08-12 07:39:35 +02:00
Luca Deri fe634b8d55 Fixed to address issue
Run ./utils/check_symbols.sh || { FAILED=$?; echo "::error file=${NDPI_LIB}::Unwanted libc symbols found: ${FAILED}. Please make sure to use only ndpi_malloc/ndpi_calloc/ndpi_realloc/ndpi_free wrapper instead of malloc/calloc/realloc/free."; false; }
[ndpi_filter.o]: calloc
Unwanted symbols found: 1
Please make sure to use only ndpi_malloc/ndpi_calloc/ndpi_realloc/ndpi_free wrapper instead of malloc/calloc/realloc/free
Error: Unwanted libc symbols found: 1. Please make sure to use only ndpi_malloc/ndpi_calloc/ndpi_realloc/ndpi_free wrapper instead of
2023-08-12 00:27:37 +02:00
Luca Deri 8e17f50853 Added ndpi_filter_add_multi() API call 2023-08-11 23:55:44 +02:00
Luca Deri 890dcf91d9 Minor initialization fixes 2023-08-11 23:51:16 +02:00
Luca Deri ec7adc212e Added new API calls for implementing Bloom-filter like data structures
ndpi_filter* ndpi_filter_alloc(uint32_t elements_number);
bool         ndpi_filter_add(ndpi_filter *f, uint64_t value);
bool         ndpi_filter_contains(ndpi_filter *f, uint64_t value);
void         ndpi_filter_free(ndpi_filter *f);
2023-08-11 22:49:31 +02:00
Luca 5fad72ebd9 Typo fix 2023-08-07 10:20:21 +02:00
Luca 33e9dbc24d Added ndpi_bitmap_copy() API call 2023-08-06 09:35:56 +02:00
snicket2100 1fbe8a2385 Mullvad VPN service added (based on entry node IP addresses) (#2062) 2023-08-02 19:44:16 +02:00
Luca Deri 196395398e Compilation fixes for older C compilers 2023-08-01 15:35:33 +02:00
Toni e4d3d619bc Add Service Location Protocol dissector. (#2036)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-01 08:50:46 +02:00
Toni 083275f617 Added slackb.com SNI. (#2067)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-01 08:39:10 +02:00
Ivan Nardi 5019022e13 DNS: extract geolocation information, if available (#2065)
The option NSID (RFC5001) is used by Google DNS to report the
airport code of the metro where the DNS query is handled.

This option is quite rare, but the added overhead in DNS code is pretty
much zero for "normal" DNS traffic
2023-07-31 07:44:43 +02:00
snicket2100 d477da997f Typo in a comment fixed (#2063) 2023-07-28 17:58:25 +02:00
Ivan Nardi bc91192aca ProtonVPN: split the ip list (#2060)
Use two separate lists:
* one for the ingress nodes, which triggers a ProtonVPN classification
* one for the egress nodes, which triggers the
`NDPI_ANONYMOUS_SUBSCRIBER` risk

Add a command line option (to `ndpiReader`) to easily test IP/port
matching.

Add another example of custom rule.
2023-07-27 09:05:22 +02:00
Ivan Nardi 3326fa258e Add an heuristic to detect fully encrypted flows (#2058)
A fully encrypted session is a flow where every bytes of the
payload is encrypted in an attempt to “look like nothing”.
The heuristic needs only the very first packet of the flow.
See: https://www.usenix.org/system/files/sec23fall-prepub-234-wu-mingshi.pdf

A basic, but generic, inplementation of the popcpunt alg has been added
2023-07-26 09:09:12 +02:00
Toni 2b230e28e0 Allow init of app protocols w/o any hostnames set. (#2057) 2023-07-22 14:51:33 +02:00
Toni 88a757c4cd Fix string truncation. (#2056)
ndpi_main.c: In function ‘ndpi_load_ip_category’:
ndpi_main.c:6598:3: warning: ‘strncpy’ specified bound 64 equals destination size [-Wstringop-truncation]
 6598 |   strncpy(ipbuf, ip_address_and_mask, sizeof(ipbuf));
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-07-22 12:45:45 +02:00
Luca Deri 6831187d62 Debian 12 fixes 2023-07-21 12:24:34 -04:00
Ivan Nardi 7a1509bf60 zabbix: improve detection (#2055) 2023-07-21 03:42:36 +02:00
Ivan Nardi c85f2fb0f4 TLS: add basic, basic, detection of Encrypted ClientHello (#2053) 2023-07-21 03:41:43 +02:00
Ivan Nardi fa0bd515b5 Add detection of Roblox games (#2054) 2023-07-21 03:39:40 +02:00
Ivan Nardi 3edfad01a1 fuzz: extend fuzzing coverage (#2052)
Added/merged some traces.
Improved Socks identification
2023-07-18 07:41:56 +02:00
Ivan Nardi 09548bb7cf tests: restore some old paths as symbolic links (#2050) 2023-07-16 13:47:35 +02:00
Ivan Nardi 5267858b3d HTTP: fix another memory access error (#2049)
```
=================================================================
==199079==ERROR: AddressSanitizer: negative-size-param: (size=-1)
    #0 0x559a2a6efd4f in strncpy (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader+0x94ad4f) (BuildId: 34aaabba403c6bc5482553ef355360fd2762a157)
    #1 0x559a2a9890f0 in ndpi_http_check_content /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:300:8
    #2 0x559a2a9812c0 in check_content_type_and_change_protocol /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:910:46
    #3 0x559a2a978fee in process_response /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1289:3
    #4 0x559a2a97622f in ndpi_check_http_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1382:9
    #5 0x559a2a975d95 in ndpi_search_http_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1468:3
    #6 0x559a2a864970 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5948:4
    #7 0x559a2a8660df in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6013:12
    #8 0x559a2a865d7f in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6032:12
    #9 0x559a2a876fd6 in ndpi_internal_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7038:15
    #10 0x559a2a87311f in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7205:22
    #11 0x559a2a77381e in packet_processing /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:1710:31
    #12 0x559a2a77381e in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:2427:10
[...]
```

Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=60605
2023-07-15 22:23:52 +02:00
Ivan Nardi 2bbde5bad3 Fix compilation in CI jobs (#2048) 2023-07-15 14:26:51 +02:00
Ivan Nardi 890f17788b ndpireader: fix detection of DoH traffic based on packet distributions (#2045) 2023-07-14 23:20:06 +02:00
Ivan Nardi 5811a5613b Add a configuration knob to enable/disable loading of gambling list (#2047) 2023-07-14 23:19:30 +02:00