Commit Graph
2217 Commits
Author SHA1 Message Date
Toni Uhlig c3df3a12aa Fixed msys2 build warnings and re-activated CI Mingw64 build.
* Removed Visual Studio leftovers. Maintaining an autotools project with VS integration requires some additional overhead.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-14 19:17:48 +02:00
Ivan Nardi 4775be3d85 Kerberos: fix some memory access errors (#1514)
```
==19724==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60e00000045e at pc 0x5620b8b3d3cc bp 0x7ffe0fda6b50 sp 0x7ffe0fda6310
READ of size 2 at 0x60e00000045e thread T0
    #0 0x5620b8b3d3cb in __interceptor_strncpy (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x63f3cb) (BuildId: ee53ff920c8cd4c226d8520a0d4846d8864726b6)
    #1 0x5620b8d9b69c in strncpy_lower /home/ivan/svnrepos/nDPI/src/lib/protocols/kerberos.c:208:4
    #2 0x5620b8d995a0 in krb_parse /home/ivan/svnrepos/nDPI/src/lib/protocols/kerberos.c:316:5
    #3 0x5620b8d97a90 in ndpi_search_kerberos /home/ivan/svnrepos/nDPI/src/lib/protocols/kerberos.c:687:12
    #4 0x5620b8bcef35 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:4996:4
    #5 0x5620b8bd1be8 in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5072:10
    #6 0x5620b8bd159c in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5105:12
    #7 0x5620b8be323a in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5924:15
    #8 0x5620b8b8f7e0 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:24:3
    #9 0x5620b8b8fd1b in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:84:17
    #10 0x7f45b32b90b2 in __libc_start_main /build/glibc-sMfBJT/glibc-2.31/csu/../csu/libc-start.c:308:16
    #11 0x5620b8acf47d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x5d147d) (BuildId: ee53ff920c8cd4c226d8520a0d4846d8864726b6)

0x60e00000045e is located 0 bytes to the right of 158-byte region [0x60e0000003c0,0x60e00000045e)
allocated by thread T0 here:
    #0 0x5620b8b5283e in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x65483e) (BuildId: ee53ff920c8cd4c226d8520a0d4846d8864726b6)
    #1 0x5620b8b8fc86 in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:70:17
    #2 0x7f45b32b90b2 in __libc_start_main /build/glibc-sMfBJT/glibc-2.31/csu/../csu/libc-start.c:308:16
```

```
protocols/kerberos.c:79:52: runtime error: left shift of 255 by 24 places cannot be represented in type 'int'
```

Found by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=46670
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=46636
2022-04-13 18:10:56 +02:00
Luca Deri e00997ea7c Extended list of cybersecurity domains 2022-04-13 18:05:29 +02:00
Wayneand林文烽 939572c61e fix(ndpi_main):Fix memory leak about ndpi_str; (#1513)
Co-authored-by: 林文烽 <wenfeng.lin@baishan.com>
2022-04-12 15:34:37 +02:00
Ivan Nardi 06a0abb6fd TINC: fix invalid memory read (#1512)
```
=================================================================
==19324==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60600061be96 at pc 0x55b4a4cb4460 bp 0x7ffc7b461a70 sp 0x7ffc7b461a68
READ of size 1 at 0x60600061be96 thread T0
    #0 0x55b4a4cb445f in ndpi_check_tinc /home/ivan/svnrepos/nDPI/src/lib/protocols/tinc.c:105:9
    #1 0x55b4a4cb1888 in ndpi_search_tinc /home/ivan/svnrepos/nDPI/src/lib/protocols/tinc.c:135:5
    #2 0x55b4a4b4a6e1 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5013:6
    #3 0x55b4a4b4c2d4 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5084:12
    #4 0x55b4a4b4bf77 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5103:12
    #5 0x55b4a4b5dcca in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5924:15
    #6 0x55b4a4a87734 in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1519:31
    #7 0x55b4a4a80761 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2093:10
    #8 0x55b4a4a39c8d in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7
    #9 0x55b4a4a3a46b in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:179:17
    #10 0x7f69c63760b2 in __libc_start_main /build/glibc-sMfBJT/glibc-2.31/csu/../csu/libc-start.c:308:16
    #11 0x55b4a497954d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x61654d) (BuildId: 705ebc5c412d267294a65cb01f03a1f012aeaf20)

0x60600061be96 is located 0 bytes to the right of 54-byte region [0x60600061be60,0x60600061be96)
allocated by thread T0 here:
[...]
```
Found by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=46499
2022-04-10 16:59:34 +02:00
Toni e3faecf9de Improved ASN.1 parsing for Keberos. Fixes #1492. (#1497)
* This is a quick fix, the Kerberos protocol dissector requires some refactoring effort.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-10 10:36:10 +02:00
Vinicius Silva Nogueira 713cdd0b13 QUIC: handle retransmissions and overlapping fragments in reassembler (#1195) (#1498)
* QUIC: handle retransmissions and overlapping fragments in reassembler

* Trigger CI

* minor fix: parentheses

* Changing ndpi_malloc to ndpi_calloc

* fix memory leak

* quic_reasm_buf calloc to malloc

* change order of is_ch_complete && is_reasm_buf_complete call

* is_reasm_buf_complete: added handling for case where frame size is not multiple of 8

* add extra check
2022-04-07 21:18:58 +02:00
Luca Deri a6e2f4a15a Added ndpi_find_outliers() API call using Z-Score 2022-04-04 10:02:45 +02:00
Luca Deri 8080b3d14f Removed un-necessary guess in mining 2022-04-01 19:23:13 +02:00
Luca Deri 6eb5bd88a7 DGA improvements 2022-04-01 17:39:29 +02:00
Luca Deri 8238e68275 Improvements for CUSTOM_NDPI_PROTOCOLS 2022-04-01 09:02:05 +02:00
Luca Deri 1e8cdd893c Moved geneated file to a separate folder 2022-04-01 08:58:53 +02:00
Luca Deri 0ff3e77c42 Improved twitter detection 2022-04-01 08:50:56 +02:00
Luca Deri 6677d692c4 Removed SRV record from suspicious DNS traffic 2022-03-31 11:32:30 +02:00
Luca Deri 98a548c642 Improved DGA detection 2022-03-31 11:29:30 +02:00
Ivan Nardi 6acf7a6abe Add support for Pluralsight site (#1503) 2022-03-27 15:13:12 +02:00
Vitaly Lavrov 4896dabb6b Reducing the size of the ndpi_detection_module_struct structure. (#1490)
The ndpi_detection_module_struct structure contains 5 arrays "struct
ndpi_call_function_struct" size 286*144=41 kB size,
which are occupied by a small number of elements.

At the moment we have callback_buffer_size = 172, tcp_with_payload=114,
tcp_no_payload=8, udp=93, other 8.
NDPI_MAX_SUPPORTED_PROTOCOLS = 285.

Size of struct ndpi_detection_module_struct is 253136 bytes.
Size of all structs ndpi_call_function_struct 5*286*144=205920 bytes.
Real use memory size for struct ndpi_call_function_struct is
(173+224)*144=57168 bytes.
2022-03-27 12:00:21 +02:00
Darryl Sokoloski d668ab4bec [SSDP] Extract HTTP user-agent when available. (#1500)
[SSDP] Added capture file with UA header.
[SSDP] Added pcap test output log file.

Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
2022-03-27 11:10:44 +02:00
Luca Deri 856d7d2916 Improved DGA detection skipping names containign at least 3 consecutive digits in the first word 2022-03-26 09:59:55 +01:00
Ivan Nardi aeb5f1f947 QUIC: add support for version 2 draft 01 (#1493)
Support for v2-00 has been removed (it has never been used in real
networks and it is incompatible with v2-01).

Chrome already supports v2-01 in latest versions in Chrome Beta channel.
2022-03-25 10:16:30 +01:00
Ivan Nardi af1d20bca1 Mining: cleanup registration (#1496)
Use the same pattern of all the other dissectors: one registration and
one callback.
Spotted by @dsokoloski
2022-03-25 10:07:47 +01:00
Vinicius Silva Nogueira 26df1403e6 Trying to improve QUIC reassembler (#1195) (#1489)
* handling QUIC out-of-order fragments

* minor fix

* updated quic_frags_ch_out_of_order_same_packet_craziness.pcapng.out

* quic test: buf_len + last_pos

* QUIC: comment update in __reassemble function and minor change is_ch_complete function
2022-03-24 10:51:31 +01:00
Luca Deri 93f723d50f Extended the list of cybersecurity protocol 2022-03-21 15:12:50 +01:00
Vitaly Lavrov 8dd70b7023 Bug fixing. (#1487)
Using the protocol_id instead of its index.
2022-03-15 19:40:18 +01:00
Ivan Nardi 9e22c88e0a QUIC: convert logs to standard mechanism (#1485) 2022-03-15 16:55:47 +01:00
Ivan Nardi b83e9f20e7 QUIC: fix dissection of draft-34 (#1484)
QUIC-34 is probably not used in production, but fixing it is trivial and
it doesn't add any noise to the already complex QUIC code.
2022-03-09 22:37:56 +01:00
Ivan Nardi 7aee856aa0 Extend tests coverage (#1476)
Now there is at least one flow under `tests/pcap` for 249 protocols out
of the 284 ones supported by nDPI.

The 35 protocols without any tests are:

* P2P/sharing protocols: DIRECT_DOWNLOAD_LINK, OPENFT, FASTTRACK,
EDONKEY, SOPCAST, THUNDER, APPLEJUICE, DIRECTCONNECT, STEALTHNET

* games: CSGO, HALFLIFE2, ARMAGETRON, CROSSFIRE, DOFUS, FIESTA,
FLORENSIA, GUILDWARS, MAPLESTORY, WORLD_OF_KUNG_FU

* voip/streaming: VHUA, ICECAST, SHOUTCAST, TVUPLAYER, TRUPHONE

* other: AYIYA, SOAP, TARGUS_GETDATA, RPC, ZMQ, REDIS, VMWARE, NOE,
LOTUS_NOTES, EGP, SAP

Most of these protocols (expecially the P2P and games ones) have been
inherited by OpenDPI and have not been updated since then: even if they
are still used, the detection rules might be outdated.

However code coverage (of `lib/protocols`) only increases from 65.6% to
68.9%.

Improve Citrix, Corba, Fix, Aimini, Megaco, PPStream, SNMP and Some/IP
dissection.
Treat IPP as a HTTP sub protocol.
Fix Cassandra false positives.

Remove `NDPI_PROTOCOL_QQLIVE` and `NDPI_PROTOCOL_REMOTE_SCAN`:
these protocol ids are defined but they are never used.

Remove Collectd support: its code has never been called. If someone is
really interested in this protocol, we can re-add it later, updating the
dissector.

Add decoding of PPI (Per-Packet Information) data link type.
2022-03-09 22:37:35 +01:00
Toni f646a4bce0 Improved ASN/IP update scripts and CI integration. (#1474)
* CI will print a warning if ASN/IP addresses changed.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-09 13:53:04 +01:00
Ivan Nardi 269be6c7ef Some small fixes (#1481)
FTP: if the authentication fails, stop analyzing the flow
WSD: call the initialization routine; the dissector code has never been
triggered
MINING: fix dissection
2022-03-08 00:21:51 +01:00
Vitaly Lavrov a1451935b8 Errors fixed (#1482)
Fixed errors for bigendian platforms in ndpiReader.
All address and port comparisons and hash calculations are done with
endian in mind.
The get_ndpi_flow_info() function searched for an existing flow for the
forward and reverse direction of the packet.
The ndpi_workflow_node_cmp() function looked for a flow regardless of
the packet's direction. This is what led to an error in determining the
direction of transmission of the packet.

Fixed error in "synscan" test: the number of packets in the forward and
reverse direction is incorrectly defined (verified via tcpdump).

Fixed bug with icmp protocol checksum check for big endian platforms.
2022-03-08 00:20:56 +01:00
Ivan Nardi c345b3c7af EthernetIP: fix integer conversion on big-endian archs (#1477) 2022-03-05 15:05:05 +01:00
Vitaly Lavrov 438f04066b Fixed a bug for BE architectures (#1478)
Fixed a bug in the internal implementation of libgcrypt for bigendian architectures
2022-03-05 11:44:21 +01:00
Ivan Nardi a173d3e98f configure: fix usage of libgpg-error with --with-local-libgcrypt (#1472)
Right now, using external libgcrypt, nDPI is not linked to libgpg-error
because configure script never checks for it.

```
ivan@ivan-Latitude-E6540:~/svnrepos/nDPI(dev)$ CC=gcc-11 CXX=g++-11 CFLAGS="-O3 -g -Werror" ./autogen.sh --enable-debug-messages --with-pcre  --with-local-libgcrypt && make -s -j
[...]
checking for numa_available in -lnuma... yes
checking for pcap_open_live in -lpcap... yes
checking for pthread_setaffinity_np in -lpthread... yes
checking for gcry_cipher_checktag in -lgcrypt... yes             <------- missing check for libgpg-error
checking for pcre_compile in -lpcre... yes
checking that generated files are newer than configure... done
[...]
ivan@ivan-Latitude-E6540:~/svnrepos/nDPI(dev)$ grep HAVE_LIBGPG_ERROR src/include/ndpi_config.h
/* #undef HAVE_LIBGPG_ERROR */

```

Make both libgcrypt and libgpg-error mandatory if
`--with-local-libgcrypt` is used.

Technically speaking, libgpg-error might be optional, because it is used
only for debug messages. However having both libraries mandatory
slightly simplified the logic.
In most environments, libgpg-error is a dependency of libgcrypt anyway,
so having both libraries should be the standard case.
2022-03-05 11:41:37 +01:00
Toni cb62dfd249 Drop support for non-gcrypt builds. (#1469)
* As there is now a builtin, lightweight libgcrypt
   there is no need to disable tls-clho decryption.
 * It is still possible to use a host libgcrypt
   with `--with-local-libgcrypt'.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-02 19:48:46 +01:00
Ivan Nardi e23507d3e9 Internal crypto: increase size of authentication buffer (#1468)
Some QUIC flows are not properly decoded while using internal crypto
code: the authentication buffer is too small.
The new value (like the old one) is arbitrary.

Close #1463
2022-03-02 15:14:57 +01:00
Toni e8559a4127 Add ICMP checksum check and set risk if mismatch detected. (#1464)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-02 13:12:01 +01:00
Toni 4da1453d90 Added configureable ndpi packet processing limit. (#1466)
* The current behaviour ignores any user preferences
   and was also incorrectly implemented, because the
   flow->num_processed_pkts wraps every 65535 and nDPI
   will process packets again until
   NDPI_MAX_NUM_PKTS_PER_FLOW_TO_DISSECT reached.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-02 12:02:09 +01:00
Toni 736a80f9ce Fix libgcrypt(-light/-internal) compile error. (#1465)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-02 11:03:47 +01:00
Ivan Nardi 7a7e4ee69f Add a new flow risk NDPI_ANONYMOUS_SUBSCRIBER (#1462)
The main goal of a DPI engine is usually to determine "what", i.e. which
types of traffic flow on the network.
However the applications using DPI are often interested also in "who",
i.e. which "user/subscriber" generated that traffic.

The association between a flow and a subscriber is usually done via some
kind of DHCP/GTP/RADIUS/NAT mappings. In all these cases the key element
of the flow used to identify the user is the source ip address.

That usually happens for the vast majority of the traffic.

However, depending on the protocols involved and on the position on the net
where the traffic is captured, the source ip address might have been
changed/anonymized. In that case, that address is useless for any
flow-username association.

Example: iCloud Private Relay traffic captured between the exit relay and
the server.
See the picture at page 5 on:
https://www.apple.com/privacy/docs/iCloud_Private_Relay_Overview_Dec2021.PDF

This commit adds new generic flow risk `NDPI_ANONYMOUS_SUBSCRIBER` hinting
that the ip addresses shouldn't be used to identify the user associated
with the flow.
As a first example of this new feature, the entire list of the relay ip
addresses used by Private Relay is added.

A key point to note is that list is NOT used for flow classification
(unlike all the other ip lists present in nDPI) but only for setting this
new flow risk.

TODO: IPv6
2022-02-28 15:25:58 +01:00
Ivan Nardi e193f3713d Removed some unused fields (#1461) 2022-02-28 15:03:49 +01:00
Vitaly Lavrov c390085f91 Bug fixing. (#1459)
The '--enable-debug-messages' option works again.
Fixed warning in ahocorasick.c
Fixed integer overflow in ndpiReader.c for 32bit systems.
2022-02-28 15:01:00 +01:00
Ivan Nardi 8b062295cc Add some scripts to easily update some IPs lists (#1449)
While the lists in a6ff0dd0 and 2f5f445f are somehow provided by the
companies themselves (or by some interested parties), these new lists
are directly extracted from BGP information, via AS prefixes.

*Usually*, these new lists are far more stable than the previous ones.

TODO:
* add some other ASNs (see `src/lib/ndpi_content_match.c.inc`)
* IPv6, as usual :-(
2022-02-25 14:26:26 +01:00
Toni f14dcb8344 Provide some API functions for convenience. (#1456)
* Extended JSON serializsation: risk, risk score, confidence

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-25 14:26:04 +01:00
Luca Deri 34e020ac35 Win fixes 2022-02-23 21:00:53 +01:00
Luca Deri 16ed0fd3f8 Replaced strdup with ndpi_strup 2022-02-23 21:37:52 +01:00
Ivan Nardi d506a0e12f Directly drop malformed packets (#1455)
TCP/UDP/ICMP/ICMPv6 packets with invalid L4 header length should be
ignored.
2022-02-22 22:15:46 +01:00
Ivan Nardi 6c1accd2bd DTLS: fix access to certificate cache (#1450)
```
protocols/tls.c:650:54: runtime error: member access within null pointer of type 'const struct ndpi_tcphdr'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior protocols/tls.c:650:54 in
protocols/tls.c:650:54: runtime error: load of null pointer of type 'const u_int16_t' (aka 'const unsigned short')
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior protocols/tls.c:650:54 in
AddressSanitizer:DEADLYSIGNAL
=================================================================
==47401==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x55f7a61b661c bp 0x7f38190f91b0 sp 0x7f38190f70e0 T1)
==47401==The signal is caused by a READ memory access.
==47401==Hint: address points to the zero page.
    #0 0x55f7a61b661c in processCertificateElements /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:650:41
    #1 0x55f7a61ac3cc in processCertificate /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:792:7
    #2 0x55f7a61d34e1 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:846:13
```
2022-02-21 20:30:22 +01:00
Ivan Nardi 59993e294e EthernetIP: add missing initialization (#1448)
Fix:1e1cfb89
2022-02-20 10:20:46 +01:00
Ivan Nardi 2f5f445f72 Add support for Google Cloud (#1447)
Differentiate between Google its own apps/services and Google Cloud.
We already do something similar for Amazon vs AWS and Microsoft vs Azure.
2022-02-20 10:19:52 +01:00
Vitaly Lavrov f2411958d7 Added lightweight implementation of libgcrypt. (#1444)
Implementation borrowed from the
https://github.com/ARMmbed/mbedtls.git project (v3.1.0)

Speed testing (Xeon(R) CPU E3-1230 V2 @ 3.30GHz):

gcrypt-gnu         Test md   2897 ms enc   2777 ms dec    942 ms
gcrypt-int         Test md   3668 ms enc   1312 ms dec   2836 ms
gcrypt-int-noaesni Test md   3652 ms enc   1916 ms dec   4458 ms

gcrypt-gnu-nonopt  Test md   3763 ms enc   4978 ms dec   3999 ms

gcrypt-gnu-nonopt - libgcrypt compiled without hardware acceleration
  --disable-padlock-support --disable-aesni-support \
  --disable-shaext-support --disable-pclmul-support \
  --disable-sse41-support --disable-drng-support \
  --disable-avx-support --disable-avx2-support \
  --disable-neon-support --disable-arm-crypto-support \
  --disable-ppc-crypto-support
  --disable-amd64-as-feature-detection
2022-02-20 10:16:46 +01:00