Commit Graph
293 Commits
Author SHA1 Message Date
Ivan Nardi 231748bb0e LRU cache: move to 64 bits long keys (#2346)
Tradeoff between key comparison efficiency (i.e. no `memcmp`) and key
length.
At least in the ipv4 cases, we have no more different entries with the
same key.
2024-03-15 10:12:51 +01:00
Ivan Nardi c09b4dacdf Fix memory leak (#2340)
```
Direct leak of 12 byte(s) in 1 object(s) allocated from:
    #0 0x55779e1a46ff in malloc (/home/ivan/svnrepos/nDPI/example/ndpiReader+0x8706ff) (BuildId: 14c2fc626744710d49d652ea1c5bbb24a8cbab4f)
    #1 0x55779e2120c7 in ndpi_malloc_wrapper /home/ivan/svnrepos/nDPI/example/ndpiReader.c:298:10
    #2 0x55779e5fa215 in ndpi_malloc /home/ivan/svnrepos/nDPI/src/lib/ndpi_memory.c:60:25
    #3 0x55779e5fa500 in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_memory.c:113:13
    #4 0x55779e42153c in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2554:46
    #5 0x55779e4359a1 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:908:5
    #6 0x55779e432de7 in ndpi_search_tls_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1097:2
    #7 0x55779e4133f9 in ndpi_search_tls_wrapper /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2913:5
```

Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=67250
2024-03-07 13:11:13 +01:00
Nardi Ivan b287d6ec85 TLS: avoid setting some flow risks for webrtc traffic
Is quite rare to have a SNI or an ALPN on Client Hello of STUN/DTLS/SRTP
traffic
2024-02-26 10:20:05 +01:00
Ivan Nardi ffd6c3edee TLS: fix disabling of JA3C fingerprint (#2319) 2024-02-19 11:11:57 +01:00
Ivan Nardi 31f0608b9a TLS: add configuration of JA* fingerprints (#2313) 2024-02-10 09:16:13 +01:00
Ivan Nardi f352e4f77b Improve normalization of flow->host_server_name (#2310)
Follow-up of 4543385d10

Remove trailing spaces for any HTTP header (we already remove leading
spaces)

We want:
* a "normalized" string in `flow->host_server_name`, but
* to parse the original string for flow risk checking

`ndpi_hostname_sni_set()` is a private function, so there is no need to
export its flags.
2024-02-09 09:20:02 +01:00
Vitaly Lavrov 4543385d10 Normalization of host_server_name (#2299)
* Normalization of host_server_name

The ndpi_hostname_sni_set() function replaces all non-printable
characters with the "?" character and removing whitespace characters
at the end of the line.

* Added conditional hostname normalization.
2024-02-05 19:44:45 +01:00
Toni f8aa642c3c Simplify and fix JA4 string computation. (#2298)
* additional JA4 string buffer is not needed and may cause a string truncation warning

```
protocols/tls.c: In function ‘ndpi_compute_ja4’:
protocols/tls.c:1738:3: warning: ‘strncpy’ output may be truncated copying 36 bytes from a string of length 1023 [-Wstringop-truncation]
 1738 |   strncpy(flow->protos.tls_quic.ja4_client, ja_str, 36);
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-02-02 14:25:43 +01:00
Nardi Ivan c669044a44 config: configure TLS certificate expiration with the new API 2024-01-18 10:21:24 +01:00
Nardi Ivan 311d8b6dae config: move cfg of aggressiviness and opportunistic TLS to the new API 2024-01-18 10:21:24 +01:00
Nardi Ivan d72a760ac3 New API for library configuration
This is the first step into providing (more) configuration options in nDPI.

The idea is to have a simple way to configure (most of) nDPI: only one
function (`ndpi_set_config()`) to set any configuration parameters
(in the present or on in the future) and we try to keep this function
prototype as agnostic as possible.

You can configure the library:
* via API, using `ndpi_set_config()`
* via a configuration file, in a text format

This way, anytime we need to add a new configuration parameter:
* we don't need to add two public functions (a getter and a setter)
* we don't break API/ABI compatibility of the library; even changing
the parameter type (from integer to a list of integer, for example)
doesn't break the compatibility.

The complete list of configuration options is provided in
`doc/configuration_parameters.md`.

As a first example, two configuration knobs are provided:
* the ability to enable/disable the extraction of the sha1 fingerprint of
the TLS certificates.
* the upper limit on the number of packets per flow that will be subject
to inspection
2024-01-18 10:21:24 +01:00
Ivan Nardi 308f71a6e8 TLS: fix heap-buffer-overflow (#2235)
```
==53992==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x511000007e04 at pc 0x555da2165fd0 bp 0x7ffddf7e3990 sp 0x7ffddf7e3988
READ of size 2 at 0x511000007e04 thread T0
    #0 0x555da2165fcf in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2384:50
    #1 0x555da217c31f in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:908:5
    #2 0x555da2176720 in ndpi_search_tls_udp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1273:11
    #3 0x555da215a628 in ndpi_search_tls_wrapper /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2883:5
    #4 0x555da1e95c30 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6720:6
    #5 0x555da1e969f3 in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6756:10
    #6 0x555da1e96394 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6789:12
    #7 0x555da1ea7991 in ndpi_internal_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7929:15
    #8 0x555da1ea547f in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:8104:22
    #9 0x555da1de137f in packet_processing /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:1721:31
    #10 0x555da1de137f in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:2438:1
```

Found by oss-fuzzer
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=65362
2024-01-02 11:37:40 +01:00
Luca Deri 99d4838328 New ndpi_sha256() nDPI API call (#2230)
* Added ndpi_sha256.c to the Windows project

* Added ndpi_sha256() nDPI API call
2023-12-23 10:47:28 +01:00
Luca Deri caa223d2ba Changes to avoid type redefinition on windows 2023-12-23 09:15:42 +01:00
Luca Deri 5c7200f2bb Buffer overflow fix 2023-12-23 08:07:28 +01:00
Luca Deri 53b9ae23e8 Warning fix 2023-12-22 21:10:03 +01:00
Luca Deri 8285fffdae Implements JA4 Support (#2191) 2023-12-22 20:40:42 +01:00
Luca Deri b90c18e906 Changes to make room for JA4 2023-12-22 20:40:42 +01:00
Ivan Nardi 8ea16ffd77 TLS: be sure to always set a category value (#2206)
We probably need to check everywhere we set the category in the
"extra dissection" data path: a flow with a proper classification can't
have NDPI_PROTOCOL_CATEGORY_UNSPECIFIED as category.
2023-12-14 08:07:04 +01:00
Ivan Nardi adf8982d8e fuzz: extend fuzzing coverage (#2205) 2023-12-11 12:48:50 +01:00
Ivan Nardi 7b0c16a70d TLS: remove JA3+ fingerprints. (#2192)
See: #2191
2023-12-05 08:05:44 +01:00
Ivan Nardi 7ff22a7e3c STUN: improve demultiplexing of DTLS packets (#2153)
Keep demultiplexing STUN/RTP/RTCP packets after DTLS ones.

We might end up processing the session a little longer, because we will
process the STUN/RTP/RTCP packets after the DTLS handshake.
2023-11-27 11:10:38 +01:00
Ivan Nardi b08c787fe2 Have a clear distinction between public and private/internal API (#2137)
1) Public API/headers in `src/include/` [as it has always been]
2) Private API/headers in `src/lib/`

Try to keep the "ndpi_" prefix only for the public functions
2023-11-09 10:50:59 +01:00
Toni a98d7ff433 Added HAProxy protocol. (#2088)
* fixed tests/do.sh.in failure print

Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-02 18:10:47 +02:00
Luca Deri 770e460ba9 Added NDPI_TLS_ALPN_SNI_MISMATCH flow risk 2023-09-07 23:42:42 +02:00
Ivan Nardi c85f2fb0f4 TLS: add basic, basic, detection of Encrypted ClientHello (#2053) 2023-07-21 03:41:43 +02:00
Ivan Nardi 950f5cc4e3 fuzz: extend fuzzing coverage (#2040)
Some notes:
* libinjection: according to https://github.com/libinjection/libinjection/issues/44,
it seems NULL characters are valid in the input string;
* RTP: `rtp_get_stream_type()` is called only for RTP packets; if you
want to tell RTP from RTCP you should use `is_rtp_or_rtcp()`;
* TLS: unnecessary check; we already make the same check just above, at
the beginning of the `while` loop
2023-07-11 10:12:08 +02:00
Toni 859d9ea3c3 Fixed heap-overflow if compiled with --enable-tls-sigs. (#2038)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-07-07 18:52:06 +02:00
Ivan Nardi 31a9da238c STUN: add dissection of DTLS handshake (#2018) 2023-06-26 12:07:26 +02:00
Luca Deri cdc0bb8c61 Compilation fix 2023-06-22 16:04:14 +02:00
Luca Deri 8dbd591053 Waring fix 2023-06-21 21:26:29 +02:00
headshog 7dcceb4daf Numeric truncation at tls.c:1010 (#2005)
* fixed numtrunc error in protocols/tls.c

* fixed build error for tls.c
2023-06-08 07:52:18 +02:00
Ivan Nardi 4d11941d32 Ookla: rework detection (#1922)
The logic of the LRU cache has been changed: once we know an ip has
connected to an Ookla server, all the following (unknown) flows (for
a short time interval) from the same ip to the port 8080 are treated
as Ookla ones.

Most of the changes in this commit are about introducing the concept of
"aggressive detection". In some cases, to properly detect a
protocol we might use some statistical/behavior logic that, from one
side, let us to identify the protocol more often but, from the other
side, might lead to some false positives.
To allow the user/application to easily detect when such logic has been
triggered, the new confidence value `NDPI_CONFIDENCE_DPI_AGGRESSIVE` has been
added.
It is always possible to disable/configure this kind of logic via the
API.

Detection of Ookla flows using plain TLS over port 8080 is the first
example of aggressive detection in nDPI.

Tested with:
* Android 9.0 with app 4.8.3
* Ubuntu 20.04 with Firefox 110
* Win 10 with app 1.15 and 1.16
* Win 10 with Chrome 108, Edge 108 and Firefox 106
2023-03-30 17:13:51 +02:00
Ivan Nardi 6e0cd16225 TLS: fix another interger overflow in certificate processing (#1915) 2023-03-30 05:12:40 +02:00
Ivan Nardi ddb9aa0f95 TLS: fix parsing of certificate elements (#1910)
```
==1228==ERROR: AddressSanitizer: SEGV on unknown address 0x6040000bed05 (pc 0x00000056e148 bp 0x7ffcca534320 sp 0x7ffcca5330c0 T0)
==1228==The signal is caused by a WRITE memory access.
    #0 0x56e148 in processCertificateElements ndpi/src/lib/protocols/tls.c:682:79
    #1 0x56c60f in LLVMFuzzerTestOneInput ndpi/fuzz/fuzz_tls_certificate.c:43:3
    #2 0x43de63 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:611:15
    #3 0x4295c2 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:324:6
    #4 0x42ee6c in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:860:9
    #5 0x4583a2 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #6 0x7f8c021c9082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16
    #7 0x41f78d in _start
```

Found by oss-fuzz.
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=57317
2023-03-22 18:18:36 +01:00
Ivan Nardi 0e80828e14 fuzz: add a new fuzzer to test TLS certificates (#1901) 2023-03-20 17:00:46 +01:00
Ivan Nardi 89cae9ddf2 Add a new flow risk about literal IP addresses used as SNI (#1892)
RFC 6066 3: "Literal IPv4 and IPv6 addresses are not permitted in
"HostName"."

Don't set this risk if we have a valid sub-classification (example:
via certificate)

Since a similar risk already exists for HTTP hostnames, reuse it, with a
more generic name.
2023-03-02 15:27:30 +01:00
Ivan Nardi 1b98bec0ab LRU caches: add a generic (optional and configurable) expiration logic (#1855)
Two caches already implemented a similar mechanism: make it generic.
2023-01-18 18:18:36 +01:00
Ivan Nardi ad6bfbad4d Add protocol disabling feature (#1808)
The application may enable only some protocols.
Disabling a protocol means:
*) don't register/use the protocol dissector code (if any)
*) disable classification by-port for such a protocol
*) disable string matchings for domains/certificates involving this protocol
*) disable subprotocol registration (if any)

This feature can be tested with `ndpiReader -B list_of_protocols_to_disable`.

Custom protocols are always enabled.

Technically speaking, this commit doesn't introduce any API/ABI
incompatibility. However, calling `ndpi_set_protocol_detection_bitmask2()`
is now mandatory, just after having called `ndpi_init_detection_module()`.

Most of the diffs (and all the diffs in `/src/lib/protocols/`) are due to
the removing of some function parameters.

Fix the low level macro `NDPI_LOG`. This issue hasn't been detected
sooner simply because almost all the code uses only the helpers `NDPI_LOG_*`
2022-12-18 08:10:57 +00:00
Ivan Nardi 8c7071e040 DTLS: handle (certificate) fragments (#1811)
Keep using the existing function to handle reassembling buffer: rename
it from `ndpi_search_tls_tcp_memory` to
`ndpi_search_tls_memory` and make it "transport" agnostic
2022-12-10 18:32:25 +01:00
Ivan Nardi ada4fe4aa8 fuzz: add a new fuzzer testing memory allocation failures (#1818)
Try to fuzz error paths triggered by allocation errors.
Fix some errors already found by this new fuzzer.
Basic idea taken from: https://github.com/harfbuzz/harfbuzz/pull/2566/files

`FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is a standard define used to
(not)compile specific code in fuzzing builds.
See: https://llvm.org/docs/LibFuzzer.html
2022-12-06 17:41:58 +01:00
Ivan Nardi 81e1ea545c Make LRU caches ipv6 aware (#1810)
Simplest solution, keeping the existing cache data structure

TLS certificate cache is used for DTLS traffic, too.

Note that Ookla cache already works with ipv6 flows.

TODO:
 * make the key/hashing more robust (extending the key size?)
 * update bittorrent cache too. That task is quite difficult because
   ntopng uses a public function (`ndpi_guess_undetected_protocol()`)
   intrinsically ipv4 only...
2022-12-03 12:07:32 +01:00
Ivan Nardi 2b65a4e05e TLS: be sure to always set ssl_version field (#1806)
Useful with asymmetric traffic with (D)TLS <= 1.2
2022-11-22 16:13:28 +01:00
Ivan Nardi ca5ffc4988 TLS: improve handling of ALPN(s) (#1784)
Tell "Advertised" ALPN list from "Negotiated" ALPN; the former is
extracted from the CH, the latter from the SH.

Add some entries to the known ALPN list.

Fix printing of "TLS Supported Versions" field.
2022-10-25 17:06:29 +02:00
Nardi Ivan 9c0caa5362 TLS: allow sub-classification via ALPN
In some rare cases, it is possible to sub-classify the flow via ALPN
matching. This is particularly usefull for asymmetric traffic where the
Client Hello doens't have the SNI.

For the time being there is only one rule, about ANYDESK.
2022-10-20 22:00:12 +02:00
Ivan Nardi 35fdbc8148 TLS: explicit ignore client certificate (#1776)
TLS classification usually stops after processing *server* certificates
(if any). That means, that *client* certificate, if present, is usually
ignored.
However in some corner cases (i.e. unidirectional traffic) we might end
up processing client certificate and exposing its metadata: the issue is
that the application will think that this metadata are about the server
and not about the client.
So, for the time being, always ignore client certificate processing.

As a future work, we might find an efficient way to process and export both
certificates.
2022-10-18 16:40:15 +02:00
Nardi Ivan f3a74d97d8 TLS/DTLS: we process certificate for UDP flows, too
Note that current code access `certificate_processed` state even before
setting the protocol classification, so this piece of information can't
be saved in `flow->protos` union.
2022-09-30 06:53:29 +02:00
Nardi Ivan 1f345b311f Sizes of LRU caches are now configurable
0 as size value disable the cache.

The diffs in unit tests are due to the fact that some lookups are
performed before the first insert: before this change these lookups
weren't counted because the cache was not yet initialized, now they are.
2022-09-23 18:33:48 +02:00
Ivan Nardi a7c2734b38 Remove classification "by-ip" from protocol stack (#1743)
Basically:
* "classification by-ip" (i.e. `flow->guessed_protocol_id_by_ip` is
NEVER returned in the protocol stack (i.e.
`flow->detected_protocol_stack[]`);
* if the application is interested into such information, it can access
`ndpi_protocol->protocol_by_ip` itself.

There are mainly 4 points in the code that set the "classification
by-ip" in the protocol stack:  the generic `ndpi_set_detected_protocol()`/
`ndpi_detection_giveup()` functions and the HTTP/STUN  dissectors.

In the unit tests output, a print about `ndpi_protocol->protocol_by_ip`
has been added for each flow: the huge diff of this commit is mainly due
to that.

Strictly speaking, this change is NOT an API/ABI breakage, but there are
important differences in the classification results. For examples:
* TLS flows without the initial handshake (or without a matching
SNI/certificate) are simply classified as `TLS`;
* similar for HTTP or QUIC flows;
* DNS flows without a matching request domain are simply classified as
`DNS`; we don't have `DNS/Google` anymore just because the server is
8.8.8.8 (that was an outrageous behaviour...);
* flows previusoly classified only "by-ip" are now classified as
`NDPI_PROTOCOL_UNKNOWN`.

See #1425 for other examples of why adding the "classification by-ip" in
the protocol stack is a bad idea.

Please, note that IPV6 is not supported :(  (long standing issue in nDPI) i.e.
`ndpi_protocol->protocol_by_ip` wil be always `NDPI_PROTOCOL_UNKNOWN` for
IPv6 flows.

Define `NDPI_CONFIDENCE_MATCH_BY_IP` has been removed.

Close #1687
2022-09-20 22:24:47 +02:00
Nardi Ivan 7571f48392 Dissectors shouldn't update flow->guessed_host_protocol_id
The field `flow->guessed_host_protocol_id` is set at the beginning of
the flow analysis and it represents the "classification by ip" of the flow
itself.

This field should never be changed. Dissectors which want to provide an
"hint" about the classification, should update `flow->guessed_protocol_id`
instead. Such "hint" is useless if the dissector set the "extra-dissection"
data-path.

Rename such field to `guessed_protocol_id_by_ip` to better describe its
role.

Preliminary work necessary for #1687
2022-09-14 17:51:39 +02:00