* Label SMTP w/ STARTTLS as SMTPS *and* dissect TLS clho.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Revert "SMTP with STARTTLS is now identified as SMTPS"
This reverts commit 52d987b603.
* Revert "Compilation fix"
This reverts commit c019946f60.
* Sync unit tests.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
TargusDataspeed dissector doesn't perform any real DPI checks but it only
looks at the TCP/UDP ports.
Delete it, and use standard logic to classify these flows by port.
Protocol classification should always be set via
`ndpi_set_detected_protocol()`: this way, the values in
`flow->detected_protocol_stack[]` are always coherent.
With clang-15 (nightly build)
```
In file included from ndpi_bitmap.c:39:
./third_party/src/roaring.cc:14233:13: warning: variable 'run_count' set but not used [-Wunused-but-set-variable]
int run_count = 0;
```
* use -ltcmalloc_and_profiler and try to get rid of LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libprofiler.so
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* fixed possible memory leak caused by an invalid call to `node_proto_guess_walker()` during serialization
* execute serialization code while running regression tests
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add some statistics to ndpiReader
The purpose of this version of ndpiReader is too adding some other statistics printed by ndpiReader. In this simple version the domain names(in the flows) that are collected are:
flow-> ssh_tls.server_info
flow-> host_server_name
and are placed in a UT_hash_table, ordering them by number of occurrences.
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update example/ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
* Update ndpiReader.c
* Update ndpiReader.c
* Update ndpiReader.c
* Update ndpiReader.c
Co-authored-by: Toni <matzeton@googlemail.com>
We can access `flow->protos` union only after checking the protocol.
Checking `flow->detected_protocol.master_protocol` is redundant because
we already check it in `is_ndpi_proto`
```
AddressSanitizer:DEADLYSIGNAL
=================================================================
==29739==ERROR: AddressSanitizer: SEGV on unknown address 0x000000353820 (pc 0x7f9b64dd2717 bp 0x7fff161a52f0 sp 0x7fff161a4aa8 T0)
==29739==The signal is caused by a READ memory access.
#0 0x7f9b64dd2717 /build/glibc-SzIz7B/glibc-2.31/string/../sysdeps/x86_64/multiarch/strlen-avx2.S:96
#1 0x555c65e597d8 in __interceptor_strlen (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x6407d8) (BuildId: 11ac8ec30f1d49fb0276c9b03368e491505d2bba)
#2 0x555c65fd85fa in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:269:13
#3 0x555c65f3e8c6 in process_ndpi_collected_info /home/ivan/svnrepos/nDPI/example/reader_util.c:1188:36
#4 0x555c65f52cab in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1567:2
#5 0x555c65f4b632 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2110:10
#6 0x555c65f04d29 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:109:7
#7 0x555c65f054bb in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:181:17
#8 0x7f9b64c6e082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
#9 0x555c65e4253d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x62953d) (BuildId: 11ac8ec30f1d49fb0276c9b03368e491505d2bba)
```
Found by oss-fuzzer.
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=48020
Proper fix for the error already reported in 9040bc74
```
Uninitialized bytes in __interceptor_strlen at offset 3 inside [0x7ffc7a147390, 4)
==111876==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x55e3e4f32e5b in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:268:13
#1 0x55e3e4ef7391 in ndpi_set_risk /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:2254:12
#2 0x55e3e5022fdf in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1523:8
#3 0x55e3e503af44 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:865:5
#4 0x55e3e50397cd in ndpi_search_tls_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1024:2
#5 0x55e3e503570c in ndpi_search_tls_wrapper /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2453:5
#6 0x55e3e4f84a6a in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5150:6
#7 0x55e3e4f85778 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5198:12
#8 0x55e3e4f851e1 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5217:12
#9 0x55e3e4f96c7a in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6076:15
#10 0x55e3e4ed91ef in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:29:5
#11 0x55e3e4eda27e in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:101:17
#12 0x7f5cb3146082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
#13 0x55e3e4e5140d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0xa340d) (BuildId: 0c02c433e039970dd13a60382b94dd5a8e19f625)
```
GCC analyzer won't complain about possible use-after-free (false positive).
* tests/do.sh prints word diff's only once and not the same over and over again
* sync unit tests
Signed-off-by: lns <matzeton@googlemail.com>
```
==300852==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f108951f060 at pc 0x5641db0ee78c bp 0x7fff3b10b910 sp 0x7fff3b10b0d0
WRITE of size 116 at 0x7f108951f060 thread T0
#0 0x5641db0ee78b in __interceptor_strncpy (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x64e78b) (BuildId: 23cb34bbaf8ac11eb97563bbdc12e29ead9fb0fa)
#1 0x5641db28efad in tlsCheckUncommonALPN /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1224:7
#2 0x5641db27ec76 in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1533:6
#3 0x5641db295677 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:865:5
#4 0x5641db2935bb in ndpi_search_tls_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1024:2
#5 0x5641db28f692 in ndpi_search_tls_wrapper /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2446:5
#6 0x5641db1d87ce in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5150:6
#7 0x5641db1d91e4 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5198:12
#8 0x5641db1d8e87 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5217:12
#9 0x5641db1eb4a7 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6076:15
#10 0x5641db140b45 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:29:5
#11 0x5641db14130b in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:101:17
#12 0x7f108bcab082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
#13 0x5641db07f46d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x5df46d) (BuildId: 23cb34bbaf8ac11eb97563bbdc12e29ead9fb0fa)
Address 0x7f108951f060 is located in stack of thread T0 at offset 96 in frame
#0 0x5641db28ec4f in tlsCheckUncommonALPN /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1204
This frame has 1 object(s):
[32, 96) 'str' (line 1218) <== Memory access at offset 96 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
(longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x64e78b) (BuildId: 23cb34bbaf8ac11eb97563bbdc12e29ead9fb0fa) in __interceptor_strncpy
Shadow bytes around the buggy address:
```
Avoid zeroing the entire string.
Found by oss-fuzzer
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=47730
```
Uninitialized bytes in __interceptor_strlen at offset 0 inside [0x7ffc987443a0, 7)
==282918==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x5651277e001b in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:268:13
#1 0x5651277a45c1 in ndpi_set_risk /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:2254:12
#2 0x5651278cfaa6 in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1515:8
#3 0x5651278e7834 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:865:5
#4 0x5651278e60bd in ndpi_search_tls_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1024:2
#5 0x5651278e1ffc in ndpi_search_tls_wrapper /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2445:5
#6 0x565127831c2a in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5150:6
#7 0x565127832938 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5198:12
#8 0x5651278323a1 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5217:12
#9 0x565127843e3a in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6076:15
#10 0x56512778641f in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:29:5
#11 0x5651277874ae in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:101:17
#12 0x7f4c46794082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
#13 0x5651276fe40d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0xa340d)
```
Found by oss-fuzzer
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=47728
```
==222479==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60b000014739 at pc 0x55af06f2364f bp 0x7ffd7b6f4bf0 sp 0x7ffd7b6f4378
READ of size 12 at 0x60b000014739 thread T0
#0 0x55af06f2364e in printf_common(void*, char const*, __va_list_tag*) asan_interceptors.cpp.o
#1 0x55af06f24f70 in __interceptor_snprintf (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x613f70) (BuildId: f6545ec2bd7663bc3f16aeeb87bddc64d173a2a8)
#2 0x55af0720927f in ndpi_check_http_header /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:982:2
#3 0x55af071f4797 in ndpi_check_http_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1219:5
#4 0x55af071f05c5 in ndpi_search_http_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1402:3
#5 0x55af07080d1e in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5150:6
#6 0x55af07081734 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5198:12
#7 0x55af070813d7 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5217:12
#8 0x55af070939f7 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6076:15
#9 0x55af06fc7e1f in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1541
#10 0x55af06fc7e1f in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2110
#11 0x55af06fc2859 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:109:7
#12 0x55af06fc2feb in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:181:17
#13 0x7efe5eaac082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
#14 0x55af06f0055d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x5ef55d) (BuildId: f6545ec2bd7663bc3f16aeeb87bddc64d173a2a8)
0x60b000014739 is located 0 bytes to the right of 105-byte region [0x60b0000146d0,0x60b000014739)
allocated by thread T0 here:
#0 0x55af06f84bae in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x673bae) (BuildId: f6545ec2bd7663bc3f16aeeb87bddc64d173a2a8)
#1 0x55af06fc2673 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:103:31
#2 0x55af06fc2feb in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:181:17
#3 0x7efe5eaac082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
```
Fiund by oss-fuzzer.
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=47724
This fixes some build/test issues resulting when using tarballs.
* nDPI uses autotools (especially autoconf) in a wrong way, see #1163
Signed-off-by: lns <matzeton@googlemail.com>
* It is outdated (from 2018) and will most likely not work anymore
due to changed protocol files and outdated API usage in the example integration
* Removal is the only option besides fixing those issues and
provide a CI integration for future automatic checks
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Final RFC (https://www.rfc-editor.org/rfc/rfc9250.txt) changed the
default port to 853 (from the 784 used on the early drafts).
Since there is at least one deployed implementation still stuck at
draft-02, keep also the old value, for the time being.
* #1532 did fx TLS appdata detection only partially
* use flow->l4.tcp.tls.message.buffer_used instead of packet->payload
Signed-off-by: lns <matzeton@googlemail.com>
* Improved ASN update script
* Ran `utils/update_every_lists.sh'
* `tests/do.sh.in' prints the amount of failed pcap(s)
* `utils/asn_update.sh' prints the amount of failed download(s)
Signed-off-by: lns <matzeton@googlemail.com>
* Fixes#1528
* Serialization Interface should also fuzzed
* libjson-c may only be used in the unit test to verify the internal serialization interface
* Serialization Interface supports tlv(broken), csv and json
* Unit test does work again and requires libjson-c
Signed-off-by: lns <matzeton@googlemail.com>
```
==120637==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x606000000488 at pc 0x55a0598d97ec bp 0x7ffcfe13f2e0 sp 0x7ffcfe13f2d8
READ of size 2 at 0x606000000488 thread T0
#0 0x55a0598d97eb in ndpi_search_raknet /home/ivan/svnrepos/nDPI/src/lib/protocols/raknet.c:152:38
#1 0x55a05966c48e in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5060:6
#2 0x55a05966e428 in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5119:10
#3 0x55a05966dddc in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5152:12
#4 0x55a05967fa7a in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5971:15
#5 0x55a05962b860 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:24:3
#6 0x55a05962bd9b in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:84:17
#7 0x7f4aad7d80b2 in __libc_start_main /build/glibc-sMfBJT/glibc-2.31/csu/../csu/libc-start.c:308:16
#8 0x55a05956b46d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x5d746d) (BuildId: 9429d3d08edc3836e5536f93c07f140716d8b82e)
0x606000000488 is located 9 bytes to the right of 63-byte region [0x606000000440,0x60600000047f)
allocated by thread T0 here:
#0 0x55a0595ee80e in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x65a80e) (BuildId: 9429d3d08edc3836e5536f93c07f140716d8b82e)
#1 0x55a05962bd06 in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:70:17
#2 0x7f4aad7d80b2 in __libc_start_main /build/glibc-sMfBJT/glibc-2.31/csu/../csu/libc-start.c:308:16
```
Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=47000
* Replaced return statements in `utils/*.sh' with exit's (such scripts should never source'd)
* Ran `utils/update_every_lists.sh'
Signed-off-by: lns <matzeton@googlemail.com>
* Removed Visual Studio leftovers. Maintaining an autotools project with VS integration requires some additional overhead.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
```
==19724==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60e00000045e at pc 0x5620b8b3d3cc bp 0x7ffe0fda6b50 sp 0x7ffe0fda6310
READ of size 2 at 0x60e00000045e thread T0
#0 0x5620b8b3d3cb in __interceptor_strncpy (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x63f3cb) (BuildId: ee53ff920c8cd4c226d8520a0d4846d8864726b6)
#1 0x5620b8d9b69c in strncpy_lower /home/ivan/svnrepos/nDPI/src/lib/protocols/kerberos.c:208:4
#2 0x5620b8d995a0 in krb_parse /home/ivan/svnrepos/nDPI/src/lib/protocols/kerberos.c:316:5
#3 0x5620b8d97a90 in ndpi_search_kerberos /home/ivan/svnrepos/nDPI/src/lib/protocols/kerberos.c:687:12
#4 0x5620b8bcef35 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:4996:4
#5 0x5620b8bd1be8 in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5072:10
#6 0x5620b8bd159c in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5105:12
#7 0x5620b8be323a in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5924:15
#8 0x5620b8b8f7e0 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:24:3
#9 0x5620b8b8fd1b in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:84:17
#10 0x7f45b32b90b2 in __libc_start_main /build/glibc-sMfBJT/glibc-2.31/csu/../csu/libc-start.c:308:16
#11 0x5620b8acf47d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x5d147d) (BuildId: ee53ff920c8cd4c226d8520a0d4846d8864726b6)
0x60e00000045e is located 0 bytes to the right of 158-byte region [0x60e0000003c0,0x60e00000045e)
allocated by thread T0 here:
#0 0x5620b8b5283e in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x65483e) (BuildId: ee53ff920c8cd4c226d8520a0d4846d8864726b6)
#1 0x5620b8b8fc86 in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:70:17
#2 0x7f45b32b90b2 in __libc_start_main /build/glibc-sMfBJT/glibc-2.31/csu/../csu/libc-start.c:308:16
```
```
protocols/kerberos.c:79:52: runtime error: left shift of 255 by 24 places cannot be represented in type 'int'
```
Found by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=46670https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=46636
```
=================================================================
==19324==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60600061be96 at pc 0x55b4a4cb4460 bp 0x7ffc7b461a70 sp 0x7ffc7b461a68
READ of size 1 at 0x60600061be96 thread T0
#0 0x55b4a4cb445f in ndpi_check_tinc /home/ivan/svnrepos/nDPI/src/lib/protocols/tinc.c:105:9
#1 0x55b4a4cb1888 in ndpi_search_tinc /home/ivan/svnrepos/nDPI/src/lib/protocols/tinc.c:135:5
#2 0x55b4a4b4a6e1 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5013:6
#3 0x55b4a4b4c2d4 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5084:12
#4 0x55b4a4b4bf77 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5103:12
#5 0x55b4a4b5dcca in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5924:15
#6 0x55b4a4a87734 in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1519:31
#7 0x55b4a4a80761 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2093:10
#8 0x55b4a4a39c8d in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7
#9 0x55b4a4a3a46b in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:179:17
#10 0x7f69c63760b2 in __libc_start_main /build/glibc-sMfBJT/glibc-2.31/csu/../csu/libc-start.c:308:16
#11 0x55b4a497954d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x61654d) (BuildId: 705ebc5c412d267294a65cb01f03a1f012aeaf20)
0x60600061be96 is located 0 bytes to the right of 54-byte region [0x60600061be60,0x60600061be96)
allocated by thread T0 here:
[...]
```
Found by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=46499
* This is a quick fix, the Kerberos protocol dissector requires some refactoring effort.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
* QUIC: handle retransmissions and overlapping fragments in reassembler
* Trigger CI
* minor fix: parentheses
* Changing ndpi_malloc to ndpi_calloc
* fix memory leak
* quic_reasm_buf calloc to malloc
* change order of is_ch_complete && is_reasm_buf_complete call
* is_reasm_buf_complete: added handling for case where frame size is not multiple of 8
* add extra check
- partial revert of:
commit 51cfdfb0d8
Author: Luca Deri <deri@ntop.org>
Date: Sun Nov 17 17:51:45 2019 +0100
Removed unused JSON-C code
- Json option is changed from 'j' to 'k' as it's used in the new codebase.
- use HAVE_LIBJSON_C instead of HAVE_JSON_C.
- tabs vs spaces clean ups.
Signed-off-by: Vitaliy Ivanov <vitaliyi@interfacemasters.com>
Conflicts:
example/ndpiReader.c
The ndpi_detection_module_struct structure contains 5 arrays "struct
ndpi_call_function_struct" size 286*144=41 kB size,
which are occupied by a small number of elements.
At the moment we have callback_buffer_size = 172, tcp_with_payload=114,
tcp_no_payload=8, udp=93, other 8.
NDPI_MAX_SUPPORTED_PROTOCOLS = 285.
Size of struct ndpi_detection_module_struct is 253136 bytes.
Size of all structs ndpi_call_function_struct 5*286*144=205920 bytes.
Real use memory size for struct ndpi_call_function_struct is
(173+224)*144=57168 bytes.
Support for v2-00 has been removed (it has never been used in real
networks and it is incompatible with v2-01).
Chrome already supports v2-01 in latest versions in Chrome Beta channel.
* handling QUIC out-of-order fragments
* minor fix
* updated quic_frags_ch_out_of_order_same_packet_craziness.pcapng.out
* quic test: buf_len + last_pos
* QUIC: comment update in __reassemble function and minor change is_ch_complete function
Now there is at least one flow under `tests/pcap` for 249 protocols out
of the 284 ones supported by nDPI.
The 35 protocols without any tests are:
* P2P/sharing protocols: DIRECT_DOWNLOAD_LINK, OPENFT, FASTTRACK,
EDONKEY, SOPCAST, THUNDER, APPLEJUICE, DIRECTCONNECT, STEALTHNET
* games: CSGO, HALFLIFE2, ARMAGETRON, CROSSFIRE, DOFUS, FIESTA,
FLORENSIA, GUILDWARS, MAPLESTORY, WORLD_OF_KUNG_FU
* voip/streaming: VHUA, ICECAST, SHOUTCAST, TVUPLAYER, TRUPHONE
* other: AYIYA, SOAP, TARGUS_GETDATA, RPC, ZMQ, REDIS, VMWARE, NOE,
LOTUS_NOTES, EGP, SAP
Most of these protocols (expecially the P2P and games ones) have been
inherited by OpenDPI and have not been updated since then: even if they
are still used, the detection rules might be outdated.
However code coverage (of `lib/protocols`) only increases from 65.6% to
68.9%.
Improve Citrix, Corba, Fix, Aimini, Megaco, PPStream, SNMP and Some/IP
dissection.
Treat IPP as a HTTP sub protocol.
Fix Cassandra false positives.
Remove `NDPI_PROTOCOL_QQLIVE` and `NDPI_PROTOCOL_REMOTE_SCAN`:
these protocol ids are defined but they are never used.
Remove Collectd support: its code has never been called. If someone is
really interested in this protocol, we can re-add it later, updating the
dissector.
Add decoding of PPI (Per-Packet Information) data link type.
FTP: if the authentication fails, stop analyzing the flow
WSD: call the initialization routine; the dissector code has never been
triggered
MINING: fix dissection
* Extracting the Azure Origin url from the download link
* Response code check to address double quotes
Co-authored-by: Sharon Enoch <sharone@amzetta.com>
Fixed errors for bigendian platforms in ndpiReader.
All address and port comparisons and hash calculations are done with
endian in mind.
The get_ndpi_flow_info() function searched for an existing flow for the
forward and reverse direction of the packet.
The ndpi_workflow_node_cmp() function looked for a flow regardless of
the packet's direction. This is what led to an error in determining the
direction of transmission of the packet.
Fixed error in "synscan" test: the number of packets in the forward and
reverse direction is incorrectly defined (verified via tcpdump).
Fixed bug with icmp protocol checksum check for big endian platforms.
Right now, using external libgcrypt, nDPI is not linked to libgpg-error
because configure script never checks for it.
```
ivan@ivan-Latitude-E6540:~/svnrepos/nDPI(dev)$ CC=gcc-11 CXX=g++-11 CFLAGS="-O3 -g -Werror" ./autogen.sh --enable-debug-messages --with-pcre --with-local-libgcrypt && make -s -j
[...]
checking for numa_available in -lnuma... yes
checking for pcap_open_live in -lpcap... yes
checking for pthread_setaffinity_np in -lpthread... yes
checking for gcry_cipher_checktag in -lgcrypt... yes <------- missing check for libgpg-error
checking for pcre_compile in -lpcre... yes
checking that generated files are newer than configure... done
[...]
ivan@ivan-Latitude-E6540:~/svnrepos/nDPI(dev)$ grep HAVE_LIBGPG_ERROR src/include/ndpi_config.h
/* #undef HAVE_LIBGPG_ERROR */
```
Make both libgcrypt and libgpg-error mandatory if
`--with-local-libgcrypt` is used.
Technically speaking, libgpg-error might be optional, because it is used
only for debug messages. However having both libraries mandatory
slightly simplified the logic.
In most environments, libgpg-error is a dependency of libgcrypt anyway,
so having both libraries should be the standard case.
* Testing more code in CI environments.
* Added strict option checking for `./configure' in CI environments.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* As there is now a builtin, lightweight libgcrypt
there is no need to disable tls-clho decryption.
* It is still possible to use a host libgcrypt
with `--with-local-libgcrypt'.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Some QUIC flows are not properly decoded while using internal crypto
code: the authentication buffer is too small.
The new value (like the old one) is arbitrary.
Close#1463
```
==20492==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60300000578c at pc 0x55c47455e3ea bp 0x7ffc62ca1eb0 sp 0x7ffc62ca1ea8
READ of size 4 at 0x60300000578c thread T0
#0 0x55c47455e3e9 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:1840:16
#1 0x55c47451b9cd in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7
#2 0x55c47451c1ab in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:179:17
#3 0x7f661b50e0b2 in __libc_start_main /build/glibc-sMfBJT/glibc-2.31/csu/../csu/libc-start.c:308:16
#4 0x55c47445b54d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x61054d) (BuildId: eba4bb4cd43b7101e4f0028ec0fb79087bae0e37)
0x60300000578d is located 0 bytes to the right of 29-byte region [0x603000005770,0x60300000578d)
```
* The current behaviour ignores any user preferences
and was also incorrectly implemented, because the
flow->num_processed_pkts wraps every 65535 and nDPI
will process packets again until
NDPI_MAX_NUM_PKTS_PER_FLOW_TO_DISSECT reached.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
The main goal of a DPI engine is usually to determine "what", i.e. which
types of traffic flow on the network.
However the applications using DPI are often interested also in "who",
i.e. which "user/subscriber" generated that traffic.
The association between a flow and a subscriber is usually done via some
kind of DHCP/GTP/RADIUS/NAT mappings. In all these cases the key element
of the flow used to identify the user is the source ip address.
That usually happens for the vast majority of the traffic.
However, depending on the protocols involved and on the position on the net
where the traffic is captured, the source ip address might have been
changed/anonymized. In that case, that address is useless for any
flow-username association.
Example: iCloud Private Relay traffic captured between the exit relay and
the server.
See the picture at page 5 on:
https://www.apple.com/privacy/docs/iCloud_Private_Relay_Overview_Dec2021.PDF
This commit adds new generic flow risk `NDPI_ANONYMOUS_SUBSCRIBER` hinting
that the ip addresses shouldn't be used to identify the user associated
with the flow.
As a first example of this new feature, the entire list of the relay ip
addresses used by Private Relay is added.
A key point to note is that list is NOT used for flow classification
(unlike all the other ip lists present in nDPI) but only for setting this
new flow risk.
TODO: IPv6
```
==38674==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60400008957f at pc 0x5653fcee6434 bp 0x7ffe9b554b50 sp 0x7ffe9b554b48
READ of size 1 at 0x60400008957f thread T0
#0 0x5653fcee6433 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2050:18
#1 0x5653fce9fc0d in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7
#2 0x5653fcea03eb in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:179:17
#3 0x7fe71dc3d0b2 in __libc_start_main /build/glibc-YbNSs7/glibc-2.31/csu/../csu/libc-start.c:308:16
#4 0x5653fcddf67d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x58a67d) (BuildId: 525418a27e8c37d6c492cc3220e0e97809c40f98)
0x60400008957f is located 0 bytes to the right of 47-byte region [0x604000089550,0x60400008957f)
allocated by thread T0 here:
```
Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=45036
While the lists in a6ff0dd0 and 2f5f445f are somehow provided by the
companies themselves (or by some interested parties), these new lists
are directly extracted from BGP information, via AS prefixes.
*Usually*, these new lists are far more stable than the previous ones.
TODO:
* add some other ASNs (see `src/lib/ndpi_content_match.c.inc`)
* IPv6, as usual :-(
At every fuzz iteration (i.e for every trace file):
* keep the same ndpi context (`ndpi_init_detection_module` is very
slow);
* reset the flow table, otherwise it grows indefinitely.
This change should fix the "out-of-memory" errors reported by oss-fuzz.
```
protocols/tls.c:650:54: runtime error: member access within null pointer of type 'const struct ndpi_tcphdr'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior protocols/tls.c:650:54 in
protocols/tls.c:650:54: runtime error: load of null pointer of type 'const u_int16_t' (aka 'const unsigned short')
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior protocols/tls.c:650:54 in
AddressSanitizer:DEADLYSIGNAL
=================================================================
==47401==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x55f7a61b661c bp 0x7f38190f91b0 sp 0x7f38190f70e0 T1)
==47401==The signal is caused by a READ memory access.
==47401==Hint: address points to the zero page.
#0 0x55f7a61b661c in processCertificateElements /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:650:41
#1 0x55f7a61ac3cc in processCertificate /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:792:7
#2 0x55f7a61d34e1 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:846:13
```
Implementation borrowed from the
https://github.com/ARMmbed/mbedtls.git project (v3.1.0)
Speed testing (Xeon(R) CPU E3-1230 V2 @ 3.30GHz):
gcrypt-gnu Test md 2897 ms enc 2777 ms dec 942 ms
gcrypt-int Test md 3668 ms enc 1312 ms dec 2836 ms
gcrypt-int-noaesni Test md 3652 ms enc 1916 ms dec 4458 ms
gcrypt-gnu-nonopt Test md 3763 ms enc 4978 ms dec 3999 ms
gcrypt-gnu-nonopt - libgcrypt compiled without hardware acceleration
--disable-padlock-support --disable-aesni-support \
--disable-shaext-support --disable-pclmul-support \
--disable-sse41-support --disable-drng-support \
--disable-avx-support --disable-avx2-support \
--disable-neon-support --disable-arm-crypto-support \
--disable-ppc-crypto-support
--disable-amd64-as-feature-detection
* Add few scripts to easily update some IPs lists
Some IPs lists should be updated frequently: try to easy the process.
The basic idea is taken from d59fefd0 and a8fe74e5 (for Azure
addresses): one specific .c.inc file and one script for each protocol.
Add the possibility to don't load a specific list.
Rename the old NDPI_PROTOCOL_HOTMAIL id to NDPI_PROTOCOL_MS_OUTLOOK,
to identify Hotmail/Outlook/Exchange flows.
TODO: ipv6
Remove the 9 addresses associated to BitTorrent: they have been added in
e2f21116 but it is not clear why all the traffic to/from these ips
should be classified as BitTorrent.
* Added quotes
* Added quotes
Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
Memory allocation or ndpi_tsearch might fail, so the two values should be
incremented only when insertion actually happened.
Co-authored-by: Andrey Izrailev <Andrey.Izrailev@oktetlabs.ru>
* Checking for port 5353/5355 is not enough.
* Added additional multicast address and header checks.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Found by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=44280
```
==263603==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x592478 in ndpi_is_printable_string ndpi/src/lib/ndpi_utils.c:2200:9
#1 0x5b047c in processCertificateElements ndpi/src/lib/protocols/tls.c:400:7
#2 0x5ac880 in processCertificate ndpi/src/lib/protocols/tls.c:790:7
#3 0x5c3a32 in processTLSBlock ndpi/src/lib/protocols/tls.c:844:13
#4 0x5c2c61 in ndpi_search_tls_tcp ndpi/src/lib/protocols/tls.c:973:2
#5 0x5c117d in ndpi_search_tls_wrapper ndpi/src/lib/protocols/tls.c:2367:5
#6 0x552a50 in check_ndpi_detection_func ndpi/src/lib/ndpi_main.c:4792:6
```
Remove the last uses of `struct ndpi_id_struct`.
That code is not really used and it has not been updated for a very long
time: see #1279 for details.
Correlation among flows is achieved via LRU caches.
This change allows to further reduce memory consumption (see also
91bb77a8).
At nDPI 4.0 (more precisly, at a6b10cf, because memory stats
were wrong until that commit):
```
nDPI Memory statistics:
nDPI Memory (once): 221.15 KB
Flow Memory (per flow): 2.94 KB
```
Now:
```
nDPI Memory statistics:
nDPI Memory (once): 235.27 KB
Flow Memory (per flow): 688 B <--------
```
i.e. memory usage per flow has been reduced by 77%.
Close#1279
We should have two protocols in classification results only when the
"master" protocol allows some sub-protocols.
Classifications like `AmazonAWS`, `TLS/AmazonAWS`, `DNS/AmazonAWS` are
fine. However classifications like `NTP/Apple`, `BitTorrent/Azure`,
`DNScrypt.AmazonAWS` or `NestLogSink.Google` are misleading.
For example, `ndpiReader`shows `BitTorrent/Azure` flows under `Azure`
statistics; that seems to be wrong or, at least, very misleading.
This is quite important since we have lots of addresses from CDN
operators.
The only drawback of this solution is that right now ICMP traffic is
classified simply as `ICMP`; if we are really interested in ICMP stuff
we can restore the old behaviour later.
Add detection of AccuWeather site/app and Google Classroom.
Improve detection of Azure, Zattoo, Whatsapp, MQTT and LDAP.
Fix some RX false positives.
Fix some "Uncommon TLS ALPN"-risk false positives.
Fix "confidence" value for some Zoom/Torrent classifications.
Minor fix in Lua script for Wireshark extcap.
Update .gitignore file.
Let GitHub correctly detect the language type of *.inc files.
Zattoo example has been provided by @subhajit-cdot in #1148.
* Added u32 pads to `union ip_tuple` so btree search should now work as expected.
The bug caused new flow's when the remote answers, resulting in two Flows per direction. Fail.
* Fixed a race condition during shutdown phase.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
The check for grease was too broad and filtered some valid values.
In particular, the value 257 was skipped because it matched the previous check.
This has been discovered while parsing tests/pcap/443-firefox.pcap
expected ja3:
771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,0-23-65281-10-11-35-16-5-51-43-13-45-28-21,29-23-24-25-256-257,0
previously generated ja3:
771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,0-23-65281-10-11-35-16-5-51-43-13-45-28-21,29-23-24-25-256,0
Signed-off-by: Patrick Havelange <patrick.havelange_ext@softathome.com>
* build: update m4/ax_pthread.m4 from serial 23 -> serial 31
Update ax_pthread.m4 to the latest version from the autoconf-archive
project.
Signed-off-by: Sam James <sam@gentoo.org>
* build: properly detect AR, CC, RANLIB
It's necessary to be able to override choice of AR/CC/RANLIB and other toolchain
variables/tools for cross-compilation, testing with other toolchains, and
to ensure the compiler chosen by the user is actually used for the build.
Previously, GNU_PREFIX was kind-of used for this but this isn't a standard
variable (at all) and it wasn't applied consistently anyway.
We now use the standard autoconf mechanisms for finding these tools.
(RANLIB is already covered by LT_INIT.)
Signed-off-by: Sam James <sam@gentoo.org>
* build: use $(MAKE)
This ensures that parallel make works correctly, as otherwise, a fresh
make job will be started without the jobserver fd, and hence
not know about its parent, forcing -j1.
* build: respect CPPFLAGS, LDFLAGS
- CPPFLAGS is for the C preprocessor (usually for setting defines)
- LDFLAGS should be placed before objects for certain flags to work
(e.g. -Wl,--as-needed)
Signed-off-by: Sam James <sam@gentoo.org>
Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
These dissectors have *never* been triggered because their registration
functions use the wrong parameter/bitmask.
Diameter code is buggy since the origianl commit (1d108234), while
XBox code since 5266c726.
Fix some false positives in Xbox code.
This allows to avoid triggering alerts for trusted albeit private certificate issuers.
Extended the example/protos.txt with the new syntax for specifying trusted issueDN.
Example:
trusted_issuer_dn:"CN=813845657003339838, O=Code42, OU=TEST, ST=MN, C=US"
Reported by oss-fuzz:
```
==685288==ERROR: AddressSanitizer: SEGV on unknown address 0x61a100000687 (pc 0x0000005aba64 bp 0x7ffe3f29f510 sp 0x7ffe3f29f400 T0)
==685288==The signal is caused by a READ memory access.
SCARINESS: 20 (wild-addr-read)
#0 0x5aba64 in quic_len ndpi/src/lib/protocols/quic.c:203:12
#1 0x5aba64 in decrypt_initial_packet ndpi/src/lib/protocols/quic.c:993:16
#2 0x5aba64 in get_clear_payload ndpi/src/lib/protocols/quic.c:1302:21
#3 0x5aba64 in ndpi_search_quic ndpi/src/lib/protocols/quic.c:1658:19
#4 0x579f00 in check_ndpi_detection_func ndpi/src/lib/ndpi_main.c:4683:6
#5 0x57abe6 in ndpi_check_flow_func ndpi/src/lib/ndpi_main.c:0
#6 0x583b2c in ndpi_detection_process_packet ndpi/src/lib/ndpi_main.c:5545:15
#7 0x55e75e in LLVMFuzzerTestOneInput ndpi/fuzz/fuzz_process_packet.c:30:3
[...]
```
As a general rule, the higher the confidence value, the higher the
"reliability/precision" of the classification.
In other words, this new field provides an hint about "how" the flow
classification has been obtained.
For example, the application may want to ignore classification "by-port"
(they are not real DPI classifications, after all) or give a second
glance at flows classified via LRU caches (because of false positives).
Setting only one value for the confidence field is a bit tricky: more
work is probably needed in the next future to tweak/fix/improve the logic.
Remove some unused fields and re-organize other ones.
In particular:
* Update the parameters of `ndpi_ssl_version2str()` function
* Zattoo, Thunder: these timestamps aren't really used.
* Ftp/mail: these protocols are dissected only over TCP.
* Attention must be paid to TLS.Bittorrent flows to avoid invalid
read/write to `flow->protos.bittorrent.hash` field.
This is the last(?) commit of a long series (see 22241a1d, 227e586e,
730c2360, a8ffcd8b) aiming to reduce library memory consumption.
Before, at nDPI 4.0 (more precisly, at a6b10cf7, because memory stats
were wrong until that commit):
```
nDPI Memory statistics:
nDPI Memory (once): 221.15 KB
Flow Memory (per flow): 2.94 KB
```
Now:
```
nDPI Memory statistics:
nDPI Memory (once): 231.71 KB
Flow Memory (per flow): 1008 B <---------
```
i.e. memory usage per flow has been reduced by 66%, dropping below the
psychological threshold of 1 KB.
To further reduce this value, we probably need to look into #1279:
let's fight this battle another day.
Improve Microsoft, GMail, Likee, Whatsapp, DisneyPlus and Tiktok
detection.
Add Vimeo, Fuze, Alibaba and Firebase Crashlytics detection.
Try to differentiate between Messenger/Signal standard flows (i.e chat)
and their VOIP (video)calls (like we already do for Whatsapp and
Snapchat).
Add a partial list of some ADS/Tracking stuff.
Fix Cassandra, Radius and GTP false positives.
Fix DNS, Syslog and SIP false negatives.
Improve GTP (sub)classification: differentiate among GTP-U, GTP_C and
GTP_PRIME.
Fix 3 LGTM warnings.
There are no valid reasons for a (generic) protocol to ignore IPv6
traffic.
Note that:
* I have not found the specifications of "CheckPoint High Availability
Protocol", so I don't know how/if it supports IPv6
* all LRU caches are still IPv4 only
Even if src_id/dst_id stuff is probably useless (see #1279), the right
way to update the protocol classification is via `ndpi_set_detected_protocol()`
Looking at `struct ndpi_flow_struct` the two bigger fields are
`host_server_name[240]` (mainly for HTTP hostnames and DNS domains) and
`protos.tls_quic.client_requested_server_name[256]`
(for TLS/QUIC SNIs).
This commit aims to reduce `struct ndpi_flow_struct` size, according to
two simple observations:
1) maximum one of these two fields is used for each flow. So it seems safe
to merge them;
2) even if hostnames/SNIs might be very long, in practice they are rarely
longer than a fews tens of bytes. So, using a (single) large buffer is a
waste of memory for all kinds of flows. If we need to truncate the name,
we keep the *last* characters, easing domain matching.
Analyzing some real traffic, it seems safe to assume that the vast
majority of hostnames/SNIs is shorter than 80 bytes.
Hostnames/SNIs are always converted to lowercase.
Attention was given so as to be sure that unit-tests outputs are not
affected by this change.
Because of a bug, TLS/QUIC SNI were always truncated to 64 bytes (the
*first* 64 ones): as a consequence, there were some "Suspicious DGA
domain name" and "TLS Certificate Mismatch" false positives.
We can write to `flow->protos` only after a proper classification.
This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP,
SMTP, IMAP and POP code.
There are two kinds of fixes:
* write to `flow->protos` only if a final protocol has been detected
* move protocol state out of `flow->protos`
The hard part is to find, for each protocol, the right tradeoff between
memory usage and code complexity.
Handle Kerberos like DNS: if we find a request, we set the protocol
and an extra callback to further parsing the reply.
For all the other protocols, move the state out of `flow->protos`. This
is an issue only for the FTP/MAIL stuff.
Add DHCP Class Identification value to the output of ndpiReader and to
the Jason serialization.
Extend code coverage of fuzz tests.
Close#1343Close#1342
`ndpiReader` is only an example, aiming to show nDPI capabilities
and integration, without any claim about performances.
Nonetheless its memory usage per flow is *huge*, limiting the kinds
of traces that we can test on a "normal" hardware (example: scan
attacks).
The key reason of that behaviour is that we preallocate all the memory
needed for *all* the available features.
Try to reduce memory usage simply allocating some structures only
when they are really needed. Most significant example: JOY algorithms.
This way we should use a lot less memory in the two most common
user-cases:
* `ndpiReader` invoked without any particular flag (i.e `ndpiReader -i
$FILENAME_OR_IFACE`)
* internal unit tests
Before (on x86_64):
```
struct ndpi_flow_info {
[...]
/* size: 7320, cachelines: 115, members: 72 */
```
After:
```
struct ndpi_flow_info {
[...]
/* size: 2128, cachelines: 34, members: 75 */
```
This protocol is detected via HTTP Content-Type header.
Until 89d548f9, nDPI had a dedicated automa (`content_automa`) to
classify a HTTP flow according to this header. Since then, this automa has
been useless because it is always empty.
Re-enable it to match only a string seems overkilling.
Remove all `content_automa` leftovers.
Disable unit tests on CI for big-endian target. We know we have multiple
issues on big-endian architectures (see #1312) and so the unit tests
always fail there. Ignore this error for the time being and let the CI
pass if we don't have other issues.
Remove an unused automa definition
We should avoid updating any valid protocol in `ndpi_detection_giveup`; we
should try to find a proper classification only if the flow is still
completely unclassified.
For example in the attached pcap there is a valid TLS session, recognized
as such by TLS dissector. However, the `ndpi_detection_giveup`function
updates it to "HTTP/TLS" (!?) simply because the server port is 80.
Note that the real issue is not the wrong classification, but the
wrong access to `flow->protos` union. If we already set some fields of
`flow->protos` and we change the protocol in `ndpi_detection_giveup`, we
might end up freeing some invalid pointers in `ndpi_free_flow_data`
(no wonder this issue has been found while fuzzing #1354)
Fix GIT and TLS dissectors (issues found by CI fuzzer)
`ndpi_detection_giveup()` (and any functions called by it) can't access
`ndpi_detection_module_struct->packet` anymore since 730c236.
Sync unit tests results
Close#1348
* fixed several memory errors (heap-overflow, unitialized memory, etc)
* ability to build fuzz_process_packet with a main()
allowing to replay crash data generated with fuzz_process_packet
by LLVMs libfuzzer
* temporarily disable fuzzing if `tests/do.sh`
executed with env FUZZY_TESTING_ENABLED=1
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Tor via TLS should be detectable via DGA as a risk
* protocol limitations should be part of the official documentation in `doc/`
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
There are no real reasons to embed `struct ndpi_packet_struct` (i.e. "packet")
in `struct ndpi_flow_struct` (i.e. "flow"). In other words, we can avoid
saving dissection information of "current packet" into the "flow" state,
i.e. in the flow management table.
The nDPI detection module processes only one packet at the time, so it is
safe to save packet dissection information in `struct ndpi_detection_module_struct`,
reusing always the same "packet" instance and saving a huge amount of memory.
Bottom line: we need only one copy of "packet" (for detection module),
not one for each "flow".
It is not clear how/why "packet" ended up in "flow" in the first place.
It has been there since the beginning of the GIT history, but in the original
OpenDPI code `struct ipoque_packet_struct` was embedded in
`struct ipoque_detection_module_struct`, i.e. there was the same exact
situation this commit wants to achieve.
Most of the changes in this PR are some boilerplate to update something
like "flow->packet" into something like "module->packet" throughout the code.
Some attention has been paid to update `ndpi_init_packet()` since we need
to reset some "packet" fields before starting to process another packet.
There has been one important change, though, in ndpi_detection_giveup().
Nothing changed for the applications/users, but this function can't access
"packet" anymore.
The reason is that this function can be called "asynchronously" with respect
to the data processing, i.e in context where there is no valid notion of
"current packet"; for example ndpiReader calls it after having processed all
the traffic, iterating the entire session table.
Mining LRU stuff seems a bit odd (even before this patch): probably we need
to rethink it, as a follow-up.
Most (all?) protocols don't care about (tcp) retransmissions.
If a protocol registers itself with a
NDPI_SELECTION_BITMASK_PROTOCOL_*_WITHOUT_RETRANSMISSION value, its
callback is never triggered with a retransmitted packet.
This field is an exact copy of `ndpi_flow_struct->detected_protocol_stack[2]`:
* at the very beginning of packet dissection, the value saved in
`flow->detected_protocol_stack` is copied in `packet->detected_protocol_stack`
(via `ndpi_detection_process_packet()` -> `ndpi_init_packet_header()`)
* every time we update `flow->detected_protocol_stack` we update
`packet->detected_protocol_stack` too (via `ndpi_int_change_protocol()`
-> `ndpi_int_change_packet_protocol()`)
These two fields are always in sync: keeping the same value in two
different places is useless.
When TLS-over-FTP is used, the credentials are encrypted. So we must not
wait for the username and the password commands, otherwise we elaborate a
lot of packets for nothing.
Decoding of ipv6 traffic with extension headers was completely broken,
since the beginning of the L4 header was always set to a wrong value.
Handle the ipv6 fragments in the same way as the ipv4 ones: keep the first
one and drop the others.
Follow-up of 22241a1d
Only trivial changes:
* remove completely unused fields
* remove fields only written (but never read)
* CSGO protocol only handles UDP traffic
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
* Add files via upload
Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
Zeroing large structures (i.e. size > KB) is quite costly (from a CPU point
of view): we can safely avoid doing that for a couple of big structures.
Standard and Valgrind tests have been diverging quite a lot: it is time
to re-sync them. Use the same script and enable Valgrind via an
enviroment variable:
NDPI_TESTS_VALGRIND=1 ./tests/do.sh
Not sure if this is the right fix (from a logical point-of-view): this
code hasn't changed since OpenDPI era (!) and I have't found a trace
triggering this code path.
Anyway, the use-of-uninitialized-value error itself should be fixed.
Fix all the warnings.
Getting rid of "-Wno-unused-parameter" is quite complex because some
parameters usage depends on compilation variable (i.e.
`--enable-debug-messages`).
The "-Werror" flag has been added only in Travis builds to avoid
breaking the builds to users using uncommon/untested
OS/compiler/enviroment.
Tested on:
* x86_64; Ubuntu 20.04; gcc 7,8,9,10,11; clang 7,8,9,10,11,12
* x86_64; CentOS 7.7; gcc 4.8.5 (with "--disable-gcrypt" flag)
* Raspberry 4; Debian 10.10; gcc 8.3.0
* Added ARM build and unit test run for SonarCloud-CI.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fixed Mingw64 build.
* adapted to SonarCloud-CI workflow
* removed broken and incomplete Windows example (tested on VS2017/VS2019)
* removed unnecessary include (e.g. pthread.h for the library which does not make use of it)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Added new API calls
- u_int8_t ndpi_is_valid_protoId(u_int16_t protoId);
Cleaned up code and organized in a function.
- u_int8_t ndpi_is_encrypted_proto(struct ndpi_detection_module_struct *ndpi_str, ndpi_protocol proto);
Check whether a protocol is encrypted or not based on master and app protocols
* Added flow risk: TLS certificate too long
* Added flow risk: TLS certificate too long
* Date for TLS limit added
* TLS certificate check fixed
Co-authored-by: pacant <a.pace97@outlook.com>
The goal is to have a (roughly) idea about how many packets nDPI needs
to properly classify a flow.
Log this information (and guessed flows number too) during unit tests,
to keep track of improvements/regressions across commits.
Simplified the process of adding lines to AC_AUTOMATA_t.
Use the ndpi_string_to_automa() function to add patterns with domain names.
For other cases can use ndpi_add_string_value_to_automa().
ac_automata_feature(ac_automa, AC_FEATURE_LC) allows adding
and compare data in a case insensitive manner. For mandatory pattern comparison
from the end of the line, the "ac_pattern.rep.at_end=1" flag is used.
This eliminated unnecessary conversions to lowercase and adding "$" for
end-of-line matching in domain name patterns.
ac_match_handler() has been renamed ac_domain_match_handler() and has been greatly simplified.
ac_domain_match_handler() looks for the template with the highest domain level.
For special cases it is possible to manually specify the domain level.
Added test for checking ambiguous domain names like:
- short.weixin.qq.com is QQ, not Wechat
- instagram.faae1-1.fna.fbcdn.net is Instagram, not Facebook
If you specify a NULL handler when creating the AC_AUTOMATA_t structure,
then a pattern with the maximum length that satisfies the search conditions will be found
(exact match, from the beginning of the string, from the end of the string, or a substring).
Added debugging for ac_automata_search.
To do this, you need to enable debugging globally using ac_automata_enable_debug(1) and
enable debugging in the AC_AUTOMATA_t structure using ac_automata_name("name", AC_FEATURE_DEBUG).
The search will display "name" and a list of matching patterns.
Running "AHO_DEBUG=1 ndpiReader ..." will show the lines that were searched for templates
and which templates were found.
The ac_automata_dump() prototype has been changed. Now it outputs data to a file.
If it is specified as NULL, then the output will be directed to stdout.
If you need to get data as a string, then use open_memstream().
Added the ability to run individual tests via the do.sh script
* Revert "Fix return value of ndpi_match_string_subprotocol() (#1230)"
This reverts commit 58665e93a9.
* Checking the return code after calling ndpi_match_string_subprotocol()
ndpi_api.h: Description of the returned error codes for
the ndpi_match_string_subprotocol() function.
If the ndpi_match_string_subprotocol() function returned an error,
then return NDPI_PROTOCOL_UNKNOWN.
http: The "Content-type" header is only checked if it is not empty.
* RTSP is no subprotocol of HTTP (most of the time)
* detection patterns should stay in rtsp.c
* set detected HTTP protocol only if at least a valid HTTP method detected
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
protocols/tls.c:1856:5: runtime error: index 256 out of bounds for type 'char [256]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior protocols/tls.c:1856:5
Add support for Likee app (https://likee.video/) and GitLab
Improve detection of Google domains ("gtv1.com" added in 6dd42d19f was
likely a typo)
Improve Snapchat and Tiktok detection
Non-critical bugs.
If a file list is used, then all files except the last are not closed.
Opening the next file loses the memory allocated via pcap_open_offline() for the previous file.
If a bpf filter is used, then no memory is freed after pcap_compile.
Removed bigram_automata, impossible_bigram_automata, trigram_automata.
The ahocorasick structure is replaced with a bitmap.
The bitmap size for ndpi_en_bigram is 176 bytes.
The bitmap size for ndpi_en_trigram is 2201 bytes.
On the test machine, the test execution time was reduced from 27.3 seconds to 24.7 (9%).
The common actions required to call the ac_automata_search() function
have been moved to the ndpi_match_string_common function. This made it
possible to simplify the ndpi_match_string, ndpi_match_string_protocol_id,
ndpi_match_string_value, ndpi_match_custom_category, ndpi_match_string_subprotocol,
ndpi_match_bigram, ndpi_match_trigram functions.
Using u_int16_t type for protocol identifiers when working with the
ahocorasick library (changes src/include/ndpi_api.h.in and src/include/ndpi_typedefs.h).
Reworked "finalization" of all AC_AUTOMATA_t structures.
Changing the order of fields in the ndpi_call_function_struct structure
reduces the size of the ndpi_detection_module_struct structure by 10 kB (for x86_64).
The new version is about 25% faster with -O2 and 45% faster with -O3.
No recursion is used (smaller stack size required).
Uses less memory (by valgrind info)
bigram:
- original 1796 allocs, 247864 bytes allocated
- new 1232 allocs, 158880 bytes allocated
host_match:
- original 18038 allocs, 3004576 bytes allocated
- new 6861 allocs, 396624 bytes allocated
The function ac_automata_search() is thread safe.
Optional case-insensitive comparison.
Matching at the beginning and at the end of the string is supported.
One code file and one header file.
Create a separated proto field entry for each possible flow risk.
This way, filtering will be more natural: you can use something like
"ndpi.flow_risk.desktop_file_sharing_session"
```
In file included from protocols/fasttrack.c:29:
../include/ndpi_api.h:1504:3: warning: type qualifiers ignored on function return type [-Wignored-qualifiers]
1504 | const ndpi_risk_severity ndpi_risk2severity(ndpi_risk_enum risk);
| ^~~~~
In file included from protocols/amazon_video.c:28:
../include/ndpi_api.h:1504:3: warning: type qualifiers ignored on function return type [-Wignored-qualifiers]
1504 | const ndpi_risk_severity ndpi_risk2severity(ndpi_risk_enum risk);
| ^~~~~
...
ndpi_utils.c: In function ‘ndpi_risk2severity’:
ndpi_utils.c:1834:1: warning: control reaches end of non-void function [-Wreturn-type]
1834 | }
| ^
```
```
==69562==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6100009000fb at pc 0x7f41882003a7 bp 0x7f4183cfbfc0 sp 0x7f4183cfb768
READ of size 32 at 0x6100009000fb thread T1
#0 0x7f41882003a6 in __interceptor_memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:827
#1 0x560b2d7462a1 in processClientServerHello protocols/tls.c:1647
#2 0x560b2d73be6a in processTLSBlock protocols/tls.c:712
#3 0x560b2d73e61f in ndpi_search_tls_udp protocols/tls.c:968
```
Error detected with valgrind.
==13127== Conditional jump or move depends on uninitialised value(s)
==13127== at 0x483EF58: strlen (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==13127== by 0x1A93B6: ndpi_strdup (ndpi_main.c:159)
==13127== by 0x1C07CC: processClientServerHello (tls.c:1678)
==13127== by 0x1C0C4C: processTLSBlock (tls.c:712)
==13127== by 0x1C0C4C: ndpi_search_tls_tcp.part.0 (tls.c:849)
See also 8c3674e9
Error detected with valgrind.
==125883== Conditional jump or move depends on uninitialised value(s)
==125883== at 0x438F57: processClientServerHello (tls.c:1421)
==125883== by 0x43B35A: processTLSBlock (tls.c:712)
==125883== by 0x43B1C4: ndpi_search_tls_tcp (tls.c:849)
==125883== by 0x42C60B: check_ndpi_detection_func (ndpi_main.c:4426)
==125883== by 0x42E920: ndpi_detection_process_packet (ndpi_main.c:5301)
==125916== Conditional jump or move depends on uninitialised value(s)
==125916== at 0x438D7D: processClientServerHello (tls.c:1379)
==125916== by 0x43B35A: processTLSBlock (tls.c:712)
==125916== by 0x43B1C4: ndpi_search_tls_tcp (tls.c:849)
==125916== by 0x42C60B: check_ndpi_detection_func (ndpi_main.c:4426)
==125932== Conditional jump or move depends on uninitialised value(s)
==125932== at 0x438C1D: processClientServerHello (tls.c:1298)
==125932== by 0x43B35A: processTLSBlock (tls.c:712)
==125932== by 0x43B1C4: ndpi_search_tls_tcp (tls.c:849)
==125932== by 0x42C60B: check_ndpi_detection_func (ndpi_main.c:4426)
==125950== Conditional jump or move depends on uninitialised value(s)
==125950== at 0x438D4F: processClientServerHello (tls.c:1371)
==125950== by 0x43B35A: processTLSBlock (tls.c:712)
==125950== by 0x43B1C4: ndpi_search_tls_tcp (tls.c:849)
==125950== by 0x42C079: check_ndpi_detection_func (ndpi_main.c:4443)
Message length checks and basic headers are not uniform across GTP-U,
GTP-C and GTP-PRIME.
Note that, even if the length checks were wrong, the GTP sessions were almost
always correctly classified because of the "guessing" algorithm.
This patch has been tested with GTP-U, GTP-C-V1, GTP-C-V2 and GPT-PRIME-V2
traffic using ndpiReader with "-d" flag (to avoid "guessing" algorithm) and
without "-t" flag (to avoid GTP-U de-tunneling).
See #1148
* Refactored and merged callback buffer routines for non-udp-tcp / udp / tcp / tcp-wo-payload.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Try to detect one subprotocol if a detected protocol can have one.
* This adds a performance overhead due to much more protocol detection routine calls.
See #1148 for more information.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Refactor subprotocol handling (1/2).
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Refactor subprotocol handling (2/2).
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Prevent some code duplication by using macros for ndpi_int_one_line_struct string comparision.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Refactored aimini HTTP detection parts (somehow related to #1148).
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Added aimini client/server test pcap.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Removed master protocol as it was only used for STUN and via also removed API function ndpi_get_protocol_id_master_proto
* Adjusted Python code to conform to the changes made during the refactoring process.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Add support for Snapchat voip calls
Snapchat multiplexes some of its audio/video real time traffic with QUIC
sessions. The peculiarity of these sessions is that they are Q046 and
don't have any SNI.
* Fix tests with libgcrypt disabled
* An external file which contains known malicious SSL certificate SHA-1 hashes
can be loaded via ndpi_load_malicious_sha1_file(...)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Fixed memory leaks caused by conditional free'ing for some TLS connections.
* Members of tls_quic struct should also free'd if the detected master protocol is IMAPS / POPS / SMTPS / etc.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Prevent reader_util.c from exit()'ing if maximum flow count reached.
This confuses the fuzzer.
* Improved fuzz/Makefile.am to use LDADD for ../example/libndpiReader.a instead of LDFLAGS.
That way, fuzz_ndpi_reader re-links to ../example/libndpiReader.a if something changed there.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Introduced in 5f7b9d802
reader_util.c: In function ‘process_ndpi_collected_info’:
reader_util.c:1148:60: warning: ‘%s’ directive output may be truncated writing up to 255 bytes into a region of size 64 [-Wformat-truncation=]
1148 | sizeof(flow->ssh_tls.client_requested_server_name), "%s",
| ^~
reader_util.c:1147:5: note: ‘snprintf’ output between 1 and 256 bytes into a destination of size 64
1147 | snprintf(flow->ssh_tls.client_requested_server_name,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1148 | sizeof(flow->ssh_tls.client_requested_server_name), "%s",
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1149 | flow->ndpi_flow->protos.tls_quic_stun.tls_quic.client_requested_server_name);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
User-agent information is used to try to detect the user OS; since the
UA is extracted for QUIC traffic too, the "detected_os" field must be
generic and not associated to HTTP flows only.
Otherwise, you might overwrite some "tls_quic_stun" fields (SNI...) with
random data.
Strangely enough, the "detected_os" field is never used: it is never
logged, or printed, or exported...
According to RFC 4366, SNI host names can be up to 255 bytes.
Previous size of 64 resulted in failed application matches due to truncation.
For example:
0976e041e65b1aece3e720df36ac6bd7.safeframe.googlesyndication.co|m
Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
FB_ZERO was an experimental protocol run by Facebook.
They switched to QUIC/TLS1.3 more than 2 years ago; no one ever used it but
them so it is definitely dead.
See: https://engineering.fb.com/2018/08/06/security/fizz/
* Fixed stack overflow caused by missing buffer space for the trailing \0 added by sprintf()
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Remove the autoconf cache value from the previous and failed check before checking again.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* QUIC: fix return value on error path on quic_cipher_init()
* QUIC: allow dissection of sessions forcing version negotiation
Enhance heuristic to avoid false positives.
* Add AmazonAlexa protocol.
* Add AmazonAlexa test file and result.
* Include pcapng as file format.
* Rename Category to VirtualAssistant.
* Add AppleSiri virtual assistant.
* Fix pcapng test files format support.
Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
* Implement dga evaluation helper.
* Add test set for DGA classification.
* Add DGA classification performances tracking as part of Travis.
* Add DGA evaluation doc.
* Fix CI on OSX.
* Add missing backquote.
* Syntax error caused buffer pointer to equal 0x1
Possible copy-paste from lines 141-142?
* Another comma operator
* whitespace matching
* another comma operator
* another comma operator
* another comma operator
* Check for non-zero payload
* Switched to PKG_CHECK_MODULES to keep pkg-config checks more portable.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Improved GCrypt autoconf check to detect a possible gpg-error inter-library dependency.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
The "offset" field is a variable-length integer.
This bug hasn't any practical effects right now, since we are ignoring any
packet with "offset" != 0 (and the value 0 is always encoded in only one byte).
But extracting a correct "offset" is important if we are ever going to handle
fragmented Client Hello messages.
LGTM found a real issue on a boundary check
Fix unit tests: a pcap ha been uploaded twice (with different names)
Fix compilation when using DPDK (see #990)
Add QUIC payload and header decryption: most of the crypto code has been
"copied-and-incolled" from Wireshark. That code has been clearly marked
as such. All credits for that code should go to the original authors.
I tried to keep the Wireshark code as similar as possible to the original,
comments included, to ease future backporting of fixes.
Inevitably, glibc data types and data structures, tvbuff abstraction and
allocation functions have been converted.
Latest QUIC versions use TLS for the encryption layer: reuse existing code
to allow Client Hello parsing and sub-classification based on SNI value.
Side effect: we might have J3AC, TLS negotiated version, SNI value and
supported cipher list for QUIC, too.
To support QUIC payload and header decryption, it is necessary to choose an
external crypto library to handle the low-level crypto stuff. Since we will
use some Wireshark code, it is quite natural to choose the same library used
by Wireshark itself: libgcrypt.
More precisely, we will use libgcrypt and libgpg-error.
Both libraries have LGPL license, so there should be no issue from this point
of view.
These libraries are not required to build nDPI, and their usage is optional:
nDPI will keep working (and compiling) even if they are not available.
However, without them, QUIC sub-classification is next to impossible.
The configure flag "--disable-gcrypt" forces the build system to ignore these
libraries.
libgpg-error is only used for debug to have meaningful error messages and its
usage is trivial.
The same cannot be said for libgcrypt because its initialization is a significant
issue.
The rest of this commit message try explaining how libgcrypt is
initialized.
According to the documentation
https://gnupg.org/documentation/manuals/gcrypt/Initializing-the-library.html
https://gnupg.org/documentation/manuals/gcrypt/Multi_002dThreading.html#Multi_002dThreading
libgcrypt must be initialized before using it, but such initialization should
be performed by the actual application and not by any library.
Forcing the users to proper initialize libgcrypt in their own code seems
unreasonable: most people using nDPI might be complete unaware of any crypto
stuff and update each and every one application linking to nDPI with specific
libgcrypt code should be out of question, anyway.
Fortunately, it seems a workaround exists to initialize libgcrypt in a library
https://lists.gnupg.org/pipermail/gcrypt-devel/2003-August/000458.html
Therefore, we could provide a wrapper to this initialization stuff in a nDPI
function. Unfortunately nDPI API lacks a global init function that must be
called only once, before any other functions. We could add it, but that would
be a major API break.
AFAIK, ndpi_init_detection_module() might be called multiple times, for example
to create multiple independent dpi engines in the same program.
The proposed solution is to (optionally) initialize libgcrypt in
ndpi_init_detection_module() anyway:
* if the actual application doesn't directly use libgcrypt and only calls
ndpi_init_detection_module() once, everything is formally correct and it
should work out of the box [by far the most common user case];
* if the actual application already uses libgcrypt directly, it already
performs the required initialization. In this case the ndpi_prefs.ndpi_dont_init_libgcrypt
flag should be passed to ndpi_init_detection_module() to avoid further
initializations.
The only scenario not supported by this solution is when the application is
unaware of libgcrypt and calls ndpi_init_detection_module() multiple times
concurrently. But this scenario should be uncommon.
A completely different option should be to switch to another crypto library,
with a huge impact on the QUIC dissector code.
Bottom line: crypto is hard, using libgcrypt is complex and the proposed
initialization, even if not perfect, should cover the most frequent user
cases and should work, for the time being.
If anyone has some suggestions...
Some BSD APIs called in example/ return `struct bpf_timeval`, where nDPI
APIs expect `struct timeval`. These two structs, besides having
a different name, share the exact same set of fields.
In a Client Hello, the presence of both SNI and ESNI may obfuscate the real
domain of an HTTPS connection, fooling DPI engines and firewalls, similarly
to Domain Fronting.
Such technique is reported in a presentation at DEF CON 28:
"Domain Fronting is Dead, Long Live Domain Fronting: Using TLS 1.3 to evade
censors, bypass network defenses, and blend in with the noise"
Full credit for the idea must go the original author
At the moment, the only way to get the pdf presention and related video is via
https://forum.defcon.org/node/234492
Hopefully a direct link (and an example pcap) will be available soon
This leak is clearly happening because of missing/disabled code in "bins"
handling.
This is a trivial workaround useful until such code is completed.
In this way tavis-ci will stop complaining
* Added compiler search list for AC_PROG_CC, AM_PROG_CC_C_O, AC_PROG_CXX, AC_PROG_CC_STDC
for Mac OS X only
The list rendered the AX_CHECK_COMPILE_FLAG([-fsanitize=fuzzer]) useless as it did use
clang for AX_CHECK_COMPILE_FLAG but gcc during the compile process. Seems broken somehow.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
* Changing the OSS-Fuzz script from our side is much more easier then
opening a PR to google/oss-fuzz every time we have to change a single line.
* https://github.com/google/oss-fuzz/pull/4041 will be updated once this PR is merged
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
./tests/do.sh can supports SMP/MT via environment variables.
Removed -fno-sanitize=shift as well, was fixed by 317d3ffd.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Error messages:
ndpiReader.c:3211:2: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
ndpiReader.c:3207:5: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
The errors started popping up since 3d9285f1
NDPI_LOG* macros dereference ndpi_detection_module_struct object which is
private to ndpi library (via NDPI_LIB_COMPILATION define). So we can't use
them outside the library itself, i.e. in ndpiReader code
Therefore, in files in example/, convert all (rare) uses of NDPI_LOG* macros
to a new very simple macro, private to ndpiReader program. If necessary,
such macro may be improved.
According to a comment in ndpi_define.h, each dissector must define its own
NDPI_CURRENT_PROTO macro before including ndpi_api.h file
* renamed --with-llvm-sanitizer to --with-sanitizer
* disable all optimisations (-O0) if --with-sanitizer set,
no functions/paramaters/variables will be optimised, improves debugging
* enable undefined behaviour sanitizer (ubsan)
* enable leak sanitizer (lsan)
* check if -fsanitize=fuzzer is available and --enable-fuzztargets set,
fail if not (only supported by clang)
* include level 3 debugging information (-g3), improves macro debugging
* disabled ubsan misaligned pointer access and lshift overflow
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Example
- Single IP address
ip:213.75.170.11@CustomProtocol
- IP address with CIDR
ip:213.75.170.11/32@CustomProtocol
- IP address with CIDR and port
ip:213.75.170.11/32:443@CustomProtocol
Please note that there are some restrictions on the port
usage. They have been listed in example/protos.txt
Add support for 4in4, 6in6 and 4in6 encapsulations
Add support for ipv6 traffic in gtp tunnels, too
To allow gtp unit test, gtp detunneling flag has been globally enabled
in the test suite
Created function correct_csv_data_field to pevent duplicated code. Additionally used for _flow->ndpi_flow->protos.stun_ssl.ssl.alpn_ and _flow->ndpi_flow->protos.stun_ssl.ssl.tls_supported_versions_ to guarantee a valid csv output (commas replaced by semicolon) .
If "-v" is used as an argument, the line breaks are missing in the csv file ("-C"), because the argument "-J" is assumed. Moving the condition for the CSV file handler out of the scope of the "enable_joy_stats" condition removes this dependency and the CSV file is formatted correctly.
To enable the leaks detector, set the ASAN_OPTIONS=detect_leaks=1
environment variable, e.g:
ASAN_OPTIONS=detect_leaks=1 example/ndpiReader -i test.pcap
ndpiReader.c:24:0: warning: "_GNU_SOURCE" redefined
#define _GNU_SOURCE
<command-line>:0:0: note: this is the location of the previous definition
Signed-off-by: Vitaliy Ivanov <vitaliyi@interfacemasters.com>
reader_util.c:1708:43: error: 'ETHER_MAX_LEN' undeclared here (not in a function)
.rxmode = { .max_rx_pkt_len = ETHER_MAX_LEN }
^~~~~~~~~~~~~
DPDK before 19.08 had a macro ETHER_MAX_LEN, in later versions it was
changed to RTE_ETHER_MAX_LEN.
Signed-off-by: Vitaliy Ivanov <vitaliyi@interfacemasters.com>
In this corner case, the "prefix" variable was not initialized leading to a failed assertion and crash:
ndpi_patricia_search_best2: Assertion `prefix->bitlen <= patricia->maxbits' failed.
Removed ndpi_pref_http_dont_dissect_response and ndpi_pref_dns_dont_dissect_response as the ndpi_extra_dissection_possible() call will now handle everything
ndpi fails to cross build from source, because configure.seed hard codes
the build architecture pkg-config. It is best to use PKG_CHECK_MODULES
instead. Please consider applying the attached patch.
See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=939286
SSL dissector now reports the protocol version
Fixed bug in ndpiReader that ivalidates in some cases the protocl dissection and used to slow down the dissection
relative link path can preserve link relation after packaging and
doplying to new machine
ex:
$ make DESTDIR=/tmp/ndpi install
$ tar czf ndpi.tar.gz /tmp/ndpi
$ tar xf ndpi.tar.gz -C /lib
Orignal install script will caused the /lib/libndpi.so.2 links to
/tmp/ndpi/libndpi.so.2.9.0 which is not intended
Signed-off-by: chiehminw <chiehminw@synology.com>
commit 6ac83ce7eb use CXX to link ndpiReader however CXX is not
properly set in configure step
Fixes: 6ac83ce7eb ("Fixes broken hyperscan compilation")
Signed-off-by: chiehminw <chiehminw@synology.com>
in order to make builds reproducible.
See https://reproducible-builds.org/ for why this is good.
This date call works with GNU date and BSD date.
Also use UTC/gmtime to be independent of timezone.
2019-01-24 14:27:10 +01:00
1105 changed files with 247708 additions and 34824 deletions
* nDPI now reports the protocol ASAP even when specific fields have not yet been dissected because such packets have not yet been observed. This is important for inline applications that can immediately act on traffic. Applications that need full dissection need to call the new API function ndpi_extra_dissection_possible() to check if metadata dissection has been completely performed or if there is more to read before declaring it completed.
* TLS (formerly identified as SSL in nDPI v2.x) is now dissected more deeply, certificate validity is extracted as well certificate SHA-1.
* nDPIreader can now export data in CSV format with option `-C`
* Implemented Sequence of Packet Length and Time (SPLT) and Byte Distribution (BD) as specified by Cisco Joy (https://github.com/cisco/joy). This allows malware activities on encrypted TLS streams. Read more at https://blogs.cisco.com/security/detecting-encrypted-malware-traffic-without-decryption
* Available as library and in `ndpiReader` with option `-J`
* Promoted usage of protocol categories rather than protocol identifiers in order to classify protocols. This allows application protocols to be clustered in families and thus better managed by users/developers rather than using hundred of protocols unknown to most of the people.
* Added Inter-Arrival Time (IAT) calculation used to detect protocol misbehaviour (e.g. slow-DoS detection)
* Added data analysis features for computign metrics such as entropy, average, stddev, variance on a single and consistent place that will prevent when possible. This should ease traffic analysis on monitoring/security applications. New API calls have been implemented such as ndpi_data_XXX() to handle these calculations.
* Initial release of Python bindings available under nDPI/python.
* Implemented search of human readable strings for promoting data exfiltration detection
* Available as library and in `ndpiReader` with option `-e`
* Fingerprints
* JA3 (https://github.com/salesforce/ja3)
* HASSH (https://github.com/salesforce/hassh)
* DHCP
* Implemented a library to serialize/deserialize data in both Type-Length-Value (TLV) and JSON format
* Used by nProbe/ntopng to exchange data via ZMQ
## New Supported Protocols and Services
* DTLS (i.e. TLS over UDP)
* Hulu
* TikTok/Musical.ly
* WhatsApp Video
* DNSoverHTTPS
* Datasaver
* Line protocol
* Google Duo and Hangout merged
* WireGuard VPN
* IMO
* Zoom.us
## Improvements
* TLS
* Organizations
* Ciphers
* Certificate analysis
* Added PUBLISH/SUBSCRIBE methods to SIP
* Implemented STUN cache to enhance matching of STUN-based protocols
* Dissection improvements
* Viber
* WhatsApp
* AmazonVideo
* SnapChat
* FTP
* QUIC
* OpenVPN support for UDP-based VPNs
* Facebook Messenger mobile
* Various improvements for STUN, Hangout and Duo
* Added new categories: CUSTOM_CATEGORY_ANTIMALWARE, NDPI_PROTOCOL_CATEGORY_MUSIC, NDPI_PROTOCOL_CATEGORY_VIDEO, NDPI_PROTOCOL_CATEGORY_SHOPPING, NDPI_PROTOCOL_CATEGORY_PRODUCTIVITY and NDPI_PROTOCOL_CATEGORY_FILE_SHARING
* Added NDPI_PROTOCOL_DANGEROUS classification
## Fixes
* Fixed the dissection of certain invalid DNS responses
* Fixed Spotify dissection
* Fixed false positives with FTP and FTP_DATA
* Fix to discard STUN over TCP flows
* Fixed MySQL dissector
* Fix category detection due to missing initialization
nDPI is an open source LGPLv3 library for deep-packet inspection. Based on OpenDPI it includes ntop extensions. We have tried to push them into the OpenDPI source tree but nobody answered emails so we have decided to create our own source tree
nDPI® is an open source LGPLv3 library for deep-packet inspection. Based on OpenDPI it includes ntop extensions. We have tried to push them into the OpenDPI source tree but nobody answered emails so we have decided to create our own source tree
### How To Compile nDPI
In order to compile this library do
In order to compile this project do
- ./autogen.sh
-./configure
-make
To compile the library w/o any tools or tests:
- ./autogen.sh --with-only-libndpi
- make
To run tests do additionally:
-cd tests; ./do.sh
-./tests/do.sh # Generate and check for diff's in PCAP files
- ./tests/do-unit.sh # Run unit tests
- ./tests/do-dga.sh # Run DGA detection test
Please note that the pre-requisites for compilation include:
- GNU tools (autogen, automake, autoconf, libtool)
- GNU C compiler (gcc)
or run all with: `make check`
Please note that the (minimal) pre-requisites for compilation include:
- GNU tools (autoconf automake libtool pkg-config gettext flex bison)
Use the builtin python3 webserver to view documentation:
- make doc-view
### How To Add A New Protocol Dissector
The entire procedure of adding new protocols in detail:
1. Add new protocol together with its unique ID to: src/include/ndpi_protocol_ids.h
2. Create a new protocol in: src/lib/protocols/
3. Variables to be kept for the duration of the entire flow (as state variables) need to be placed in: src/include/ndpi_typedefs.h in ndpi_flow_tcp_struct (for TCP only), ndpi_flow_udp_struct (for UDP only), or ndpi_flow_struct (for both).
4. Add a new entry for the search function for the new protocol in: src/include/ndpi_protocols.h
5. Choose (do not change anything) a selection bitmask from: src/include/ndpi_define.h
6.Add a new entry in ndpi_set_protocol_detection_bitmask2 in: src/lib/ndpi_main.c
7.Set protocol default ports in ndpi_init_protocol_defaults in: src/lib/ndpi_main.c
8. ./autogen.sh
9. make
10. make check
1. Add new protocol together with its unique ID to: `src/include/ndpi_protocol_ids.h`
2. Create a new protocol in: `src/lib/protocols/`
3. Variables to be kept for the duration of the entire flow (as state variables) need to be placed in: `src/include/ndpi_typedefs.h` in `ndpi_flow_tcp_struct` (for TCP only), `ndpi_flow_udp_struct` (for UDP only), or `ndpi_flow_struct` (for both).
4. Add a new entry for the search function for the new protocol in: `src/include/ndpi_protocols.h`
5. Choose (do not change anything) a selection bitmask from: `src/include/ndpi_define.h`
6.Set protocol default ports in `ndpi_init_protocol_defaults` in: `src/lib/ndpi_main.c`
7.`./autogen.sh`
8.`make`
9.`make check`
### How to use nDPI to Block Selected Traffic
You can use nDPI to selectively block selected Internet traffic by embedding it onto an application (remember that nDPI is just a library). Both [ntopng](https://github.com/ntop/ntopng) and [nProbe cento](http://www.ntop.org/products/netflow/nprobe-cento/) can do this.
### nDPI Paper Citation
- Deri, Luca, et al. [nDPI: Open-source high-speed deep packet inspection](http://luca.ntop.org/nDPI.pdf) 2014 International Wireless Communications and Mobile Computing Conference (IWCMC). IEEE, 2014.
While we do our best to detect network protocols, we cannot guarantee that our software is error free and 100% accurate in protocol detection. Please make sure that you respect the privacy of users and you have proper authorization to listen, capture and inspect network traffic.
### Creating A Source File Tar Ball
If you want to distribute a source tar file of nDPI do:
- make dist
To ensure that a tar file includes all necessary files and to run tests on distribution do:
AC_ARG_WITH(hyperscan, [ --with-hyperscan Enable nDPI build with Intel Hyperscan])
if test "${with_hyperscan+set}" = set; then
BKP=$LIBS
LIBS="$LIBS -lstdc++ -lm"
AC_CHECK_LIB([hs], [hs_compile_multi], AC_DEFINE_UNQUOTED(HAVE_HYPERSCAN, 1, [Intel Hyperscan is present]))
LIBS=$BKP
if test "x$ac_cv_lib_hs_hs_compile_multi" = xyes; then :
AC_CHECK_LIB([m], [pow])
AC_CHECK_LIB([stdc++], [main])
HS_INC=`pkg-config --cflags libhs`
HS_LIB=`pkg-config --libs libhs`
LDFLAGS="$LDFLAGS $HS_LIB"
AC_MSG_RESULT([compiling with Intel Hyperscan])
else
AC_MSG_RESULT([Intel Hyperscan not found, exiting. See https://github.com/intel/hyperscan/blob/master/doc/dev-reference/getting_started.rst for install/build instructions])
$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
nDPI is designed not just to detect application protocols in traffic flows but also to evaluate potential security risks associated with the traffic. In nDPI parlance this is called a "flow risk". Flows can have multiple risks detected hence nDPI reports them with a bitmap. Each risk detected corresponds to a bit in the flow risk bitmap. You can read more about `ndpi_risk_enum <https://github.com/ntop/nDPI/blob/dev/src/include/ndpi_typedefs.h>`_ for the list of all numeric risks currently supported.
Below you can find a description of each flow risk so that you can easily understand when a risk is triggered and its meaning. The flow risks are listed in numerical order as they are defined in ndpi_risk_enum.
.._Risk 001:
NDPI_URL_POSSIBLE_XSS
=====================
HTTP only: this risk indicates a possible `XSS (Cross Side Scripting) <https://en.wikipedia.org/wiki/Cross-site_scripting>`_ attack.
.._Risk 002:
NDPI_URL_POSSIBLE_SQL_INJECTION
===============================
HTTP only: this risk indicates a possible `SQL Injection attack <https://en.wikipedia.org/wiki/SQL_injection>`_.
.._Risk 003:
NDPI_URL_POSSIBLE_RCE_INJECTION
===============================
HTTP only: this risk indicates a possible `RCE (Remote Code Execution) attack <https://en.wikipedia.org/wiki/Arbitrary_code_execution>`_.
.._Risk 004:
NDPI_BINARY_APPLICATION_TRANSFER
================================
HTTP only: this risk indicates that a binary application is downloaded/uploaded. Detected applications include Windows binaries, Linux executables, Unix scripts and Android apps.
.._Risk 005:
NDPI_KNOWN_PROTOCOL_ON_NON_STANDARD_PORT
========================================
This risk indicates a known protocol used on a non standard port. Example HTTP is supposed to use TCP/80, and in case it is detected on TCP/1234 this risk is detected.
.._Risk 006:
NDPI_TLS_SELFSIGNED_CERTIFICATE
===============================
TLS/QUIC only: this risk is triggered when a `self-signed certificate <https://en.wikipedia.org/wiki/Self-signed_certificate>`_ is used.
.._Risk 007:
NDPI_TLS_OBSOLETE_VERSION
=========================
Risk triggered when TLS version is older than 1.1.
.._Risk 008:
NDPI_TLS_WEAK_CIPHER
====================
Risk triggered when an unsafe TLS cipher is used. See `this page <https://community.qualys.com/thread/18212-how-does-qualys-determine-the-server-cipher-suites>`_ for a list of insecure ciphers.
.._Risk 009:
NDPI_TLS_CERTIFICATE_EXPIRED
============================
Risk triggered when a TLS certificate is expired, i.e. the current date falls outside of the certificate validity dates.
.._Risk 010:
NDPI_TLS_CERTIFICATE_MISMATCH
=============================
Risk triggered when a TLS certificate does not match the hostname we're accessing. Example you do http://www.aaa.com and the TLS certificate returned is for www.bbb.com.
.._Risk 011:
NDPI_HTTP_SUSPICIOUS_USER_AGENT
===============================
HTTP only: this risk is triggered whenever the user agent contains suspicious characters or its format is suspicious. Example: <?php something ?> is a typical suspicious user agent.
.._Risk 012:
NDPI_HTTP_NUMERIC_IP_HOST
=========================
HTTP only: this risk is triggered whenever we're accessing a host using its IP rather than its symbolic name. Example http://1.2.3.4.
.._Risk 013:
NDPI_HTTP_SUSPICIOUS_URL
========================
HTTP only: this risk is triggered whenever the accessed URL is suspicious. Example: http://127.0.0.1/msadc/..%255c../..%255c../..%255c../winnt/system32/cmd.exe.
.._Risk 014:
NDPI_HTTP_SUSPICIOUS_HEADER
===========================
HTTP only: this risk is triggered whenever the HTTP peader contains suspicious entries such as Uuid, TLS_version, Osname that are unexpected on the HTTP header.
.._Risk 015:
NDPI_TLS_NOT_CARRYING_HTTPS
===========================
TLS only: this risk indicates that this TLS flow will not be used to transport HTTP content. Example VPNs use TLS to encrypt data rather to carry HTTP. This is useful to spot this type of cases.
.._Risk 016:
NDPI_SUSPICIOUS_DGA_DOMAIN
==========================
A `DGA <https://en.wikipedia.org/wiki/Domain_generation_algorithm>`_ is used to generate domain names often used by malwares. This risk indicates that this domain name can (but it's not 100% sure) a DGA as its name is suspicious.
.._Risk 017:
NDPI_MALFORMED_PACKET
=====================
This risk is generated when a packet (e.g. a DNS packet) has an unexpected formt. This can indicate a protocol error or more often an attempt to jeopardize a valid protocol to carry other type of data.
.._Risk 018:
NDPI_SSH_OBSOLETE_CLIENT_VERSION_OR_CIPHER
==========================================
This risk is generated whenever a SSH client uses an obsolete SSH protocol version or insecure ciphers.
.._Risk 019:
NDPI_SSH_OBSOLETE_SERVER_VERSION_OR_CIPHER
==========================================
This risk is generated whenever a SSH server uses an obsolete SSH protocol version or insecure ciphers.
.._Risk 020:
NDPI_SMB_INSECURE_VERSION
=========================
This risk indicates that the `SMB <https://en.wikipedia.org/wiki/Server_Message_Block>`_ version used is insecure (i.e. v1).
.._Risk 021:
NDPI_TLS_SUSPICIOUS_ESNI_USAGE
==============================
`SNI <https://en.wikipedia.org/wiki/Server_Name_Indication>`_ is a way to carry in TLS the host/domain name we're accessing. ESNI means encrypted SNI and it is a way to mask SNI (carried in clear text in the TLS header) with encryption. While this practice is legal, it could be used for hiding data or for attacks such as a suspicious `domain fronting <https://github.com/SixGenInc/Noctilucent/blob/master/docs/>`_.
.._Risk 022:
NDPI_UNSAFE_PROTOCOL
====================
This risk indicates that the protocol used is insecure and that a secure protocol should be used (e.g. Telnet vs SSH).
.._Risk 023:
NDPI_DNS_SUSPICIOUS_TRAFFIC
===========================
This risk is returned when DNS traffic returns an unexpected/obsolete `record type <https://en.wikipedia.org/wiki/List_of_DNS_record_types>`_.
.._Risk 024:
NDPI_TLS_MISSING_SNI
====================
TLS needs to carry the the `SNI <https://en.wikipedia.org/wiki/Server_Name_Indication>`_ of the remote server we're accessing. Unfortunately SNI is optional in TLS so it can be omitted. In this case this risk is triggered as this is a non-standard situation that indicates a potential security problem or a protocol using TLS for other purposes (or a protocol bug).
.._Risk 025:
NDPI_HTTP_SUSPICIOUS_CONTENT
============================
HTTP only: risk reported when HTTP carries content in expected format. Example the HTTP header indicates that the context is text/html but the real content is not readeable (i.e. it can transport binary data). In general this is an attempt to use a valid MIME type to carry data that does not match the type.
.._Risk 026:
NDPI_RISKY_ASN
==============
This is a placeholder for traffic exchanged with `ASN <https://en.wikipedia.org/wiki/Autonomous_system_(Internet)>`_ that are considered risky. nDPI does not fill this risk that instead should be filled by aplications sitting on top of nDPI (e.g. ntopng).
.._Risk 027:
NDPI_RISKY_DOMAIN
=================
This is a placeholder for traffic exchanged with domain names that are considered risky. nDPI does not fill this risk that instead should be filled by aplications sitting on top of nDPI (e.g. ntopng).
.._Risk 028:
NDPI_MALICIOUS_JA3
==================
`JA3 <https://engineering.salesforce.com/tls-fingerprinting-with-ja3-and-ja3s-247362855967>`_ is a method to fingerprint TLS traffic. This risk indicates that the JA3 of the TLS connection is considered suspicious (i.e. it has been found in known malware JA3 blacklists). nDPI does not fill this risk that instead should be filled by aplications sitting on top of nDPI (e.g. ntopng).
.._Risk 029:
NDPI_MALICIOUS_SHA1_CERTIFICATE
===============================
TLS certificates are uniquely identified with a `SHA1 <https://en.wikipedia.org/wiki/SHA-1>`_ hash value. If such hash is found on a blacklist, this risk can be used. As for other risks, this is a placeholder as nDPI does not fill this risk that instead should be filled by aplications sitting on top of nDPI (e.g. ntopng).
.._Risk 030:
NDPI_DESKTOP_OR_FILE_SHARING_SESSION
====================================
This risk is set when the flow carries desktop or file sharing sessions (e.g. TeamViewer or AnyDesk just to mention two).
.._Risk 031:
NDPI_TLS_UNCOMMON_ALPN
======================
This risk is set when the `ALPN <https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation>`_ (it indicates the protocol carried into this TLS flow, for instance HTTP/1.1) is uncommon with respect to the list of expected values.
.._Risk 032:
NDPI_TLS_CERT_VALIDITY_TOO_LONG
===============================
From 01/09/2020 TLS certificates lifespan is limited to `13 months <https://www.appviewx.com/blogs/tls-certificate-lifespans-now-capped-at-13-months/>`_. This risk is triggered for certificates not respecting this directive.
.._Risk 033:
NDPI_TLS_SUSPICIOUS_EXTENSION
=============================
This risk is triggered when the domain name (SNI extension) is not printable and thus it is a problem. In TLS extensions can be dynamically specified by the client in the hello packet.
.._Risk 034:
NDPI_TLS_FATAL_ALERT
====================
This risk is triggered when a TLS fatal alert is detected in the TLS flow. See `this page <https://techcommunity.microsoft.com/t5/iis-support-blog/ssl-tls-alert-protocol-and-the-alert-codes/ba-p/377132>`_ for details.
.._Risk 035:
NDPI_SUSPICIOUS_ENTROPY
=======================
This risk is used to detect suspicious data carried in ICMP packets whose entropy (used to measure how data is distributed, hence to indirectly guess the type of data carried on) is suspicious and thus that it can indicate a data leak.
.._Risk 036:
NDPI_CLEAR_TEXT_CREDENTIALS
===========================
Clear text protocols are not bad per-se, but they should be avoided when they carry credentials as they can be intercepted by malicious users. This risk is triggered whenever clear text protocols (e.g. FTP, HTTP, IMAP...) contain credentials in clear text (read it as nDPI does not trigger this risk for HTTP connections that do not carry credentials).
.._Risk 037:
NDPI_DNS_LARGE_PACKET
=====================
`DNS <https://en.wikipedia.org/wiki/Domain_Name_System>`_ packets over UDP should be limited to 512 bytes. DNS packets over this threshold indicate a potential security risk (e.g. use DNS to carry data) or a misconfiguration.
.._Risk 038:
NDPI_DNS_FRAGMENTED
===================
UDP `DNS <https://en.wikipedia.org/wiki/Domain_Name_System>`_ packets cannot be fragmented. If so, this indicates a potential security risk (e.g. use DNS to carry data) or a misconfiguration.
.._Risk 039:
NDPI_INVALID_CHARACTERS
=======================
The risk is set whenever a dissected protocol contains characters not allowed in that protocol field.
For example a DNS hostname must only contain a subset of all printable characters or else this risk is set.
Additionally, some TLS protocol fields are checked for printable characters as well.
.._Risk 040:
NDPI_POSSIBLE_EXPLOIT
=====================
The risk is set whenever a possible exploit (e.g. `Log4J/Log4Shell <https://en.wikipedia.org/wiki/Log4Shell>`_) is detected.
.._Risk 041:
NDPI_TLS_CERTIFICATE_ABOUT_TO_EXPIRE
===================================
The risk is set whenever a TLS certificate is close to the expiration date.
.._Risk 042:
NDPI_PUNYCODE_IDN
===================================
The risk is set whenever a domain name is specified in IDN format as they are sometimes used in `IDN homograph attacks <https://en.wikipedia.org/wiki/IDN_homograph_attack>`_.
.._Risk 043:
NDPI_ERROR_CODE_DETECTED
===================================
The risk is set whenever an error code is detected in the underlying protocol (e.g. HTTP and DNS).
.._Risk 044:
NDPI_HTTP_CRAWLER_BOT
===================================
The risk is set whenever a crawler/bot/robot has been detected
.._Risk 045:
NDPI_ANONYMOUS_SUBSCRIBER
===================================
The risk is set whenever the (source) ip address has been anonymized and it can't be used to identify the subscriber.
Example: the flow is generated by an iCloud-private-relay exit node.
.._Risk 046:
NDPI_UNIDIRECTIONAL_TRAFFIC_SUBSCRIBER
===================================
The risk is set whenever the flow has unidirectional traffic (typically no traffic on the server to client direction). THis
risk is not triggered for multicast/broadcast destinations.
nDPI is a DPI (Deep Packet Inspection) toolkit able to:
- Detect application protocol on traffic flows
- Analize encrypted traffic flows via ET (Encrypted Traffic Analysis)
- Extract selected protocol metadata from traffic
- Implement APIs for analysing traffic
Releases and Features
---------------------
nDPI development lifecycle is typically 6 to 9 months. The history of changes and features implemented by every release, is available on its `Changelog <https://github.com/ntop/nDPI/blob/dev/CHANGELOG.md>`_.
Installation
============
nDPI is open source and available on `GitHub
<https://github.com/ntop/nDPI>`_. In addition, pre-compiled, binary
nDPI packages are available both for Linux and other platforms. Installation
instructions for binary packages are available below.
Installing on Linux
-------------------
Installation instructions can be found at
http://packages.ntop.org/. Development and stable builds are
available. Stable builds are intended for production environments whereas
development builds are intended for testing or early feature access.
Software Updates
================
General instructions for updating the software can be found at
http://packages.ntop.org/ together with the installation instructions.
Depending on the Operating System, nDPI supports also automatic updates
through the GUI as described in the below sections.
Updating the Software on Linux
------------------------------
Instructions for updating the software via command line can be found
at http://packages.ntop.org/. For example on Ubuntu/Debian systems the
below commands will update the repository, check for updates and install
You need to first compile the nDPI library as usual:
- ./autogen.sh
- ./configure
- make
Then open the Xcode project and you are ready to go. The default behavior is to analyze an embeded pcap file `capture.pcap`. You can change the behavior by changing command line input in `ViewController.m` file.
# What does the XCode project do?
It's a dummy Mac App project with a **Run** button. It doesn't modify any nDPI code except that it renamed the `main` function to `orginal_main` in `ndpiReader.c` (because the Mac App has it's own main function) and call the `orginal_main` with synthetic command line input from `ViewController.m` file when the **Run** button is clicked.
It also fixes some problems when compiling with Xcode. Some are listed below:
- Add missed `NDPI_LOG_DEBUG2` macro definition implementation (defined as `NDPI_LOG_DEBUG2_XCODE_PROJ` in `ViewController.m`)
- Add an empty ndpi_utils.h file to make `protocols/attic/ftp.c` and `protocols/attic/secondlife.c` can compile
- Specially treat `ndpi_patricia.c` by not adding it into compilation source, since it's directly included in `ndpi_main.c`
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.