100 Commits
Author SHA1 Message Date
Nardi Ivan 556f892a56 wireshark: lua: export some metadata
Export some metadata (for the moment, SNI and TLS fingerprints) to
Wireshark/tshark via extcap.
Note that:
* metadata are exported only once per flow
* metadata are exported (all together) when nDPI stopped processing
the flow

Still room for a lot of improvements!
In particular:
* we need to add some boundary checks (if we are going to export other
attributes)
* we should try to have a variable length trailer
2024-06-25 16:39:45 +02:00
Nardi Ivan f44832cc51 wireshark: lua: filtering and trailer dissection work with tshark, too
```
ivan@ivan-Latitude-E6540:~/$ tshark -C "nDPI extcap" -i ndpi -o extcap.ndpi.i:/home/ivan/svnrepos/nDPI/tests/pcap/anydesk.pcapng -Y "ndpi.protocol.name contains DNS"
Capturing on 'nDPI interface: ndpi'
   62 22635386.425683 192.168.1.187 DNS.AnyDesk 192.168.1.1  128   Standard query 0xec22 A relay-3185a847.net.anydesk.com
   63 22635386.439540  192.168.1.1 DNS.AnyDesk 192.168.1.187 144   Standard query response 0xec22 A relay-3185a847.net.anydesk.com A 37.61.223.15
   64 22635386.721277 192.168.1.187 DNS.AnyDesk 192.168.1.1  128   Standard query 0xea89 A relay-9b6827f2.net.anydesk.com
   65 22635386.732444  192.168.1.1 DNS.AnyDesk 192.168.1.187 144   Standard query response 0xea89 A relay-9b6827f2.net.anydesk.com A 138.199.36.115
4 packets captured

```
2024-06-25 16:39:45 +02:00
Nardi Ivan 3f0ea18866 wireshark: lua: fix DNS dissection
Not sure when we (or Wireshark, or Lua...) broke it, but we can't call
tonumber() on Bool variables.
2024-06-25 16:39:45 +02:00
Nardi Ivan b5afa165f0 wireshark: extcap: restore filtering mechanism 2024-06-25 16:39:45 +02:00
Nardi Ivan 2daab3f248 wireshark: lua: latest Wireshark versions correctly handle 64 bit mask 2024-06-25 16:39:45 +02:00
Nardi Ivan 46e450456b wireshark: lua: minor improvements
* Use a proper TVB to parse the nDPI trailer
* Fix some flow risks definitions
2024-06-25 16:39:45 +02:00
Nardi Ivan a35fae6b75 Sync unit tests results 2024-06-17 11:23:39 +02:00
Nardi Ivan bbe52da5cf Zoom: harden RTP/RTCP detection 2024-06-17 10:19:55 +02:00
Nardi Ivan 526cf6f291 Zoom: remove "stun_zoom" LRU cache
Since 070a0908b we are able to detect P2P calls directly from the packet
content, without any correlation among flows
2024-06-17 10:19:55 +02:00
Nardi Ivan 521ce609c8 Sync unit tests results 2024-05-22 19:00:33 +02:00
Nardi Ivan f494bdd653 Fix parameters check
```
==17==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x000000546050 bp 0x7fff113c82a0 sp 0x7fff113c7a58 T0)
==17==The signal is caused by a READ memory access.
==17==Hint: address points to the zero page.
SCARINESS: 10 (null-deref)
    #0 0x546050 in __sanitizer::internal_strlen(char const*) /src/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_libc.cpp:167:10
    #1 0x4c6ba5 in __interceptor_strrchr /src/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:740:5
    #2 0x5fb9b9 in ndpi_get_host_domain_suffix /src/ndpi/src/lib/ndpi_domains.c:105:20
    #3 0x578058 in LLVMFuzzerTestOneInput /src/ndpi/fuzz/fuzz_config.cpp:503:3
```

Found while fuzzing
2024-04-21 12:29:09 +02:00
Nardi Ivan 95f4d0b852 Fix a warning
```
nDPI/PcDebug64/src/include/ndpi_api.h:1970:3: error: function declaration isn’t a prototype [-Werror=strict-prototypes]
```
2024-04-20 19:30:42 +02:00
Nardi Ivan 17965c8852 Update unit tests results 2024-04-11 18:36:43 +02:00
Nardi Ivan aacbc9e918 Update unit tests results 2024-04-03 16:18:24 +02:00
Nardi Ivan 15a80527c6 STUN: remove workaround to identify RTP traffic
We are able to demultiplex RTP packets in STUN flows since 3608ab01b, at
least; no need to explicity call the RTP dissector
2024-03-20 09:39:15 +01:00
Nardi Ivan 5ccf41739b STUN: fix fallback when DTL dissection fails 2024-03-20 09:39:15 +01:00
Nardi Ivan f758f3605a STUN: dissection of TURN Channel Data 2024-03-20 09:39:15 +01:00
Nardi Ivan ee87c4bc66 STUN: add dissection of XOR-PEER-ADDRESS with ipv6 address 2024-03-20 09:39:15 +01:00
Nardi Ivan 58894dd788 STUN: improve heurstic to detect old classic-stun 2024-03-20 09:39:15 +01:00
Nardi Ivan b287d6ec85 TLS: avoid setting some flow risks for webrtc traffic
Is quite rare to have a SNI or an ALPN on Client Hello of STUN/DTLS/SRTP
traffic
2024-02-26 10:20:05 +01:00
Nardi Ivan 478c31b6c9 utils: update script to download Cloudflare ips 2024-02-26 09:26:21 +01:00
Nardi Ivan ed5ba179f6 Telegram: improve identification
Follow up of 31c706c3db and
75485e177c.

Allow fast classification by ip, but give time to other dissectors to
kick in (for example, the TLS code for the Telegram Web flows).

Even if we don't classify it anymore at the very first packet (i.e. SYN)
we fully classify Telegram traffic at the first packet with payload, as
*any* other protocol.
This way, we always have the proper category, the proper confidence
for the UDP flows and we don't overwrite previous classifications (TLS
or ICMP)

Remove old and stale identification logic for TCP flows
2024-02-26 09:26:21 +01:00
Nardi Ivan 0d36d9c8c2 Remove spurious call to exit() 2024-02-12 17:10:45 +01:00
Nardi Ivan c78ed93ec3 Sync unit tests results 2024-02-01 15:45:36 +01:00
Nardi Ivan 0712d496fe config: allow configuration of guessing algorithms 2024-01-18 10:21:24 +01:00
Nardi Ivan 6c85f10cd5 config: move debug/log configuration to the new API 2024-01-18 10:21:24 +01:00
Nardi Ivan c704be1a20 config: DNS: add two configuration options
* Enable/disable sub-classification of DNS flows
* Enable/disable processing of DNS responses
2024-01-18 10:21:24 +01:00
Nardi Ivan 950f209a17 config: HTTP: enable/disable processing of HTTP responses 2024-01-18 10:21:24 +01:00
Nardi Ivan c669044a44 config: configure TLS certificate expiration with the new API 2024-01-18 10:21:24 +01:00
Nardi Ivan 88720331ae config: remove enum ndpi_prefs 2024-01-18 10:21:24 +01:00
Nardi Ivan 1289951b32 config: remove ndpi_set_detection_preferences() 2024-01-18 10:21:24 +01:00
Nardi Ivan 311d8b6dae config: move cfg of aggressiviness and opportunistic TLS to the new API 2024-01-18 10:21:24 +01:00
Nardi Ivan 4cbe2674ab config: move IP lists configurations to the new API 2024-01-18 10:21:24 +01:00
Nardi Ivan f55358973f config: move LRU cache configurations to the new API 2024-01-18 10:21:24 +01:00
Nardi Ivan 3107a95881 Make ndpi_finalize_initialization() returns an error code
We should check if the initialization was fine or not
2024-01-18 10:21:24 +01:00
Nardi Ivan d72a760ac3 New API for library configuration
This is the first step into providing (more) configuration options in nDPI.

The idea is to have a simple way to configure (most of) nDPI: only one
function (`ndpi_set_config()`) to set any configuration parameters
(in the present or on in the future) and we try to keep this function
prototype as agnostic as possible.

You can configure the library:
* via API, using `ndpi_set_config()`
* via a configuration file, in a text format

This way, anytime we need to add a new configuration parameter:
* we don't need to add two public functions (a getter and a setter)
* we don't break API/ABI compatibility of the library; even changing
the parameter type (from integer to a list of integer, for example)
doesn't break the compatibility.

The complete list of configuration options is provided in
`doc/configuration_parameters.md`.

As a first example, two configuration knobs are provided:
* the ability to enable/disable the extraction of the sha1 fingerprint of
the TLS certificates.
* the upper limit on the number of packets per flow that will be subject
to inspection
2024-01-18 10:21:24 +01:00
Nardi Ivan 0891ad0e77 Added missing lists file to the distribution tarball 2024-01-15 20:00:36 +01:00
Nardi Ivan b22fa558ff ndpiReader: fix memory leak
Change the working directory of `ndpiReader` in the Github Actions so
that it can load the domain suffix list during `domainsUnitTest()`
2024-01-15 19:49:27 +01:00
Nardi Ivan 16b4913be6 fuzz: extend fuzzing coverage 2023-10-15 12:00:26 +02:00
Nardi Ivan 4a0eda69ad QUIC: export QUIC version as metadata 2023-10-11 15:15:20 +02:00
Nardi Ivan 1366d94156 fuzzing: extend fuzzing coverage
Try fuzzing some functions which write to file/file descriptor; to avoid
slowing the fuzzer, close its stdout
2023-10-09 15:41:46 +02:00
Nardi Ivan 86115a8a65 fuzz: extend fuzzing coverage 2023-10-07 13:34:37 +02:00
Nardi Ivan 70814002a9 fuzz: extend fuzzing coverage 2023-09-16 11:26:11 +02:00
Nardi Ivan 570c75d601 STUN: fix detection over TCP
TCP framing is optional
2023-06-21 10:38:44 +02:00
Nardi Ivan 2ac240ce6a STUN: improve WhatsappCall detection 2023-06-21 10:38:44 +02:00
Nardi Ivan 96340ccea2 oss-fuzz: sync build script with upstream
File copied from https://github.com/google/oss-fuzz/blob/master/projects/ndpi/build.sh

The general idea is to keep the build script in our repository and use
it from oss-fuzz builder: updating it from our side is easier and faster
then passing via an oss-fuzz PR.

The original idea is from @utoni in 3068306b60.

Once this change has been merged, we can update the code in oss-fuzz.
2023-06-12 15:17:56 +02:00
Nardi Ivan 7150b40c63 QUIC: fix dissection of packets forcing VN 2023-06-08 17:07:25 +02:00
Nardi Ivan 3e640a13a4 QUIC: fix a memory access error
Found while fuzzing
2023-06-08 17:07:25 +02:00
Nardi Ivan fd4cb10190 QUIC: add support for QUIC version 2
See: https://www.rfc-editor.org/rfc/rfc9369.txt

Old v2-01 version has been removed, since it has never been really used.
2023-06-08 17:07:25 +02:00
Nardi Ivan 1ab5318f9c Sync unit tests results 2023-05-20 15:09:51 +02:00
Nardi Ivan e0ad5b49c0 Sync unit tests results 2023-03-01 20:34:56 +01:00
Nardi Ivan d65f8cf15f Keep master protocol in ndpi_reconcile_protocols
As a side effect of this change, the following memory leak has been
fixed.
```
==55523==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 2 byte(s) in 1 object(s) allocated from:
    #0 0x55657d01b7be in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_alloc_fail+0x5b27be) (BuildId: 9dfa7bd9081421417b8b1c2e8b44633a9bfcfe52)
    #1 0x55657d169182 in ndpi_malloc /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:243:25
    #2 0x55657d1694c8 in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:310:13
    #3 0x55657d25ddc7 in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2308:46
    #4 0x55657d2701b9 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:895:5
    #5 0x55657d26a408 in ndpi_search_tls_udp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1223:11
```

Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=56466
2023-03-01 20:34:56 +01:00
Nardi Ivan 9eb9664516 SoftEther: fix invalid memory access
We can't write `flow->protos` union until we are really sure about protocol
classification

```
==28334==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x558db5554512 bp 0x000000000000 sp 0x7ffcb22c2880 T0)
==28334==The signal is caused by a READ memory access.
==28334==Hint: this fault was caused by a dereference of a high value address (see register values below).  Disassemble the provided pc to learn which register was used.
    #0 0x558db5554512 in __asan::Allocator::Deallocate(void*, unsigned long, unsigned long, __sanitizer::BufferedStackTrace*, __asan::AllocType) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet+0x48e512) (BuildId: 2f71e395637a7b748f36d5a04c7281f18b1128d7)
    #1 0x558db55ea54b in __interceptor_free (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet+0x52454b) (BuildId: 2f71e395637a7b748f36d5a04c7281f18b1128d7)
    #2 0x558db56977ca in ndpi_free /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:274:7
    #3 0x558db56c20e3 in ndpi_free_flow_data /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5175:2
    #4 0x558db569783f in ndpi_free_flow /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:8394:5
    #5 0x558db5627936 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:38:3
```

Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=56272
2023-03-01 20:34:56 +01:00
Nardi Ivan 52b562c328 Fix json export of ipv6 addresses
The "string" buffer was to short; better start using `INET6_ADDRSTRLEN`
as reported in the man page of `inet_ntop`.

Close: #1794
2022-11-07 20:36:55 +01:00
Nardi Ivan aa5dd25392 RTP: remove a superfluous check
Static analyzer complains about dereferencing `packet->udp` before
checking.
Since this function is called only with UDP flows, remove the check.

Close: #1792
2022-11-07 18:47:19 +01:00
Nardi Ivan 9c0caa5362 TLS: allow sub-classification via ALPN
In some rare cases, it is possible to sub-classify the flow via ALPN
matching. This is particularly usefull for asymmetric traffic where the
Client Hello doens't have the SNI.

For the time being there is only one rule, about ANYDESK.
2022-10-20 22:00:12 +02:00
Nardi Ivan c1b6aa9483 Sync unit tests results 2022-10-19 13:01:34 +02:00
Nardi Ivan 223a6fb9f7 Fix a use-of-uninitialized-value error on PCRE code
This is likely a false positive, triggered by the fact that libpcre is
usually compiled without MASAN support.
It it was a real error, ASAN would complain loudly with a invalid-free
error at the end of the same function.

```
==83793==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x562296111174 in ndpi_compile_rce_regex /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1631:3
    #1 0x5622960e3e4a in ndpi_is_rce_injection /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1636:5
    #2 0x5622960de7cd in ndpi_validate_url /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1741:12
    #3 0x5622960dae45 in ndpi_dpi2json /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1362:29
    #4 0x5622960e2751 in ndpi_flow2json /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1512:10
    #5 0x562296033b0f in process_ndpi_collected_info /home/ivan/svnrepos/nDPI/example/reader_util.c:1310:9
    #6 0x5622960501f9 in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1659:2
    #7 0x562296045aef in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2202:10
    #8 0x562295e85374 in ndpi_process_packet /home/ivan/svnrepos/nDPI/example/ndpiReader.c:3937:7
    #9 0x7f1235053466  (/lib/x86_64-linux-gnu/libpcap.so.0.8+0x23466) (BuildId: b84c893ea2516d6fb2c1c6726b1fe93b3be78f61)
    #10 0x7f1235041f67 in pcap_loop (/lib/x86_64-linux-gnu/libpcap.so.0.8+0x11f67) (BuildId: b84c893ea2516d6fb2c1c6726b1fe93b3be78f61)
    #11 0x562295e53139 in runPcapLoop /home/ivan/svnrepos/nDPI/example/ndpiReader.c:4060:15
    #12 0x562295e51e7f in processing_thread /home/ivan/svnrepos/nDPI/example/ndpiReader.c:4130:3
    #13 0x7f1234e53608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
    #14 0x7f1234d2f132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

  Uninitialized value was created by an allocation of 'pcreErrorStr' in the stack frame
    #0 0x5622961108a6 in ndpi_compile_rce_regex /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1603:3
```
2022-10-17 23:15:07 +02:00
Nardi Ivan 3d0c36cf22 Add some popular streaming applications
A simply back-porting from https://github.com/vel21ripn/nDPI/pull/142;
full credits to @ChrisNelson-CyberReef.

Close #1716
2022-10-17 23:08:25 +02:00
Nardi Ivan cca585053e Fix compilation and sync utests results 2022-10-04 22:17:05 +02:00
Nardi Ivan f3a74d97d8 TLS/DTLS: we process certificate for UDP flows, too
Note that current code access `certificate_processed` state even before
setting the protocol classification, so this piece of information can't
be saved in `flow->protos` union.
2022-09-30 06:53:29 +02:00
Nardi Ivan a27538b764 Remove unused code
LRU callbacks have been added in 460ff3c7a, but they have never been
used and they have never been extended to the other LRU caches.

`ndpi_search_tcp_or_udp()` basically returns the classification by port/ip
of the flow; calling it from the dissector is useless.
The same for TOR detection: ips are checked in the generic code
2022-09-28 21:09:27 +02:00
Nardi Ivan ac7a3669da Fix value of ndpi_protocol->protocol_by_ip
Fix: a7c2734b
2022-09-25 13:31:57 +02:00
Nardi Ivan 39f9b8cadf DNS: change category of DNS flows
DNS flows should have `NDPI_PROTOCOL_CATEGORY_NETWORK` as category,
regardless of the subprotocol (if any).
Follow-up of 83de3e47
2022-09-25 13:31:04 +02:00
Nardi Ivan 1f345b311f Sizes of LRU caches are now configurable
0 as size value disable the cache.

The diffs in unit tests are due to the fact that some lookups are
performed before the first insert: before this change these lookups
weren't counted because the cache was not yet initialized, now they are.
2022-09-23 18:33:48 +02:00
Nardi Ivan 83de3e4716 DNS: change category of DNS flows
DNS flows should have `NDPI_PROTOCOL_CATEGORY_NETWORK` as category,
regardless of the subprotocol (if any).
2022-09-22 13:23:55 +02:00
Nardi Ivan b250f9d86c Extend content match lists 2022-09-22 09:30:09 +02:00
Nardi Ivan c6201284d7 NAT-PMP: fix metadata extraction 2022-09-21 20:02:23 +02:00
Nardi Ivan 9ce4d40d14 Remove a case of guessed sub-classification
This code is triggered only for "unknown" flows with a valid
sni/hostname.
Why in that case the guessed classification should be
something like `DNS/Subprotocol_depending_on_hostname`? Why DNS as
master and not HTTP or TLS or QUIC?

Furthermore, I have not been able to trigger a positive match from that
lookup. I strongly think that if we had a valid subprotocol, we would
have a valid master in the first place.
In doubt, remove it completely.

As a follow up, we should investigate why some dissectors (the HTTP one,
at least) set the sni/hostname field without setting a valid protocol,
in the first place.
This behaviour seems quite suspicious, if not plainly buggy.
2022-09-14 17:52:01 +02:00
Nardi Ivan 7571f48392 Dissectors shouldn't update flow->guessed_host_protocol_id
The field `flow->guessed_host_protocol_id` is set at the beginning of
the flow analysis and it represents the "classification by ip" of the flow
itself.

This field should never be changed. Dissectors which want to provide an
"hint" about the classification, should update `flow->guessed_protocol_id`
instead. Such "hint" is useless if the dissector set the "extra-dissection"
data-path.

Rename such field to `guessed_protocol_id_by_ip` to better describe its
role.

Preliminary work necessary for #1687
2022-09-14 17:51:39 +02:00
Nardi Ivan eea3c448de HTTP: remove some wrong code
This code seems wrong or in the wrong place, at least:
* "classification by port" and "classification by ip" protocols (i.e
"guessed" protocols) should be used to set the protocol stack only after
trying all the dissectors, and only by the generic code
* there are no reason (for a dissector) to update the "guessed"
information using the protocol stack values: it is usually the other way
around (see previous point)
2022-09-13 20:48:20 +02:00
Nardi Ivan 678dd61866 STUN: several improvements
Add detection over TCP and fix detection over IPv6.
Rename some variables since Stun dissector is no more "udp-centric".
Stun dissector should always classified the flow as `STUN` or
`STUN/Something`.
Don't touch `flow->guessed_host_protocol_id` field, which should be
always be related to "ip-classification" only.
2022-09-11 13:33:32 +02:00
Nardi Ivan f44413e039 Update Changelog file (version 4.4)
Notes shameless taken from https://github.com/ntop/nDPI/releases
2022-09-10 22:51:40 +02:00
Nardi Ivan 1b3de449a1 HTTP: correctly set the classification protocols
Classification should always be set via `ndpi_set_detected_protocol()`
to be sure to set a correct `confidence` value, too.
Having a "known" protocol stack with `NDPI_CONFIDENCE_UNKNOWN` as
confidence, is not valid.

This code in HTTP dissector likely needs some more thoughts (the
classification itself of the attached example doesn't make a lot of
sense), but the goal of this commit is only to always have a valid
`confidence` value.
2022-09-06 14:58:31 +02:00
Nardi Ivan e5063f9883 Fix some warnings
Ubuntu-20.04, clang-16 (nightly build)

```
Making all in src/lib
protocols/smpp.c:70:17: warning: variable 'pdu_c' set but not used [-Wunused-but-set-variable]
      u_int16_t pdu_c = 1;
                ^
1 warning generated.
third_party/src/ahocorasick.c:173:20: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion]
  thiz->root->root = 1;
                   ^ ~
third_party/src/ahocorasick.c:336:15: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion]
        n->ff = 1;
              ^ ~
third_party/src/ahocorasick.c:716:21: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion]
        node->final = 1;
[...]
```
2022-09-05 23:44:51 +02:00
Nardi Ivan 95c1daa851 Fix endianess in IP-Port lookup
Port parameter in `ndpi_network_port_ptree_match()` must be in network
order.
2022-09-05 10:17:58 +02:00
Nardi Ivan b9cb391756 Add support to opportunistic TLS
A lot of protocols provide the feature to upgrade their plain text
connections to an encrypted one, via some kind of "STARTTLS" command.

Add generic code to support this extension, and allow dissection of the
entire TLS handshake.

As examples, SMTP, POP, IMAP and FTP dissectors have been updated.

Since this feature requires to process more packets per flow, add the
possibility to disable it.

Fix some log messages.

Slight improvement on TCP sequence number tracking.

As a side effect, this commit fix also a memory leak found by
oss-fuzzer
```
==108966==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 22 byte(s) in 1 object(s) allocated from:
    #0 0x55f8b367a0be in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x5480be) (BuildId: 94debacb4a6784c30420ab748c8bf3cc59621063)
    #1 0x55f8b36e1345 in ndpi_malloc_wrapper /home/ivan/svnrepos/nDPI/example/reader_util.c:321:10
    #2 0x55f8b379c7d2 in ndpi_malloc /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:212:25
    #3 0x55f8b379cb18 in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:279:13
    #4 0x55f8b386ce46 in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2153:34
    #5 0x55f8b385ebf7 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:867:5
    #6 0x55f8b39e708c in ndpi_extra_search_mail_smtp_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/mail_smtp.c:422:9
    #7 0x55f8b37e636c in ndpi_process_extra_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5884:9
    #8 0x55f8b37edc05 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6276:5
    #9 0x55f8b3701ffc in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1619:31
    #10 0x55f8b36faf14 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2189:10
    #11 0x55f8b36b6a50 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7

```
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=50765
2022-09-04 17:22:19 +02:00
Nardi Ivan 79653f99e0 Internal gcrypt: fix memory leak
Fix: ef99eb67
2022-08-30 11:12:00 +02:00
Nardi Ivan a329730d24 QUIC: fix heap-buffer-overflow
```
==12318==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x604000000032 at pc 0x55a59ec97959 bp 0x7fffee67fdd0 sp 0x7fffee67fdc8
READ of size 1 at 0x604000000032 thread T0
    #0 0x55a59ec97958 in may_be_0rtt /home/ivan/svnrepos/nDPI/src/lib/protocols/quic.c:1483:24
    #1 0x55a59ec9515f in ndpi_search_quic /home/ivan/svnrepos/nDPI/src/lib/protocols/quic.c:1708:13
    #2 0x55a59ec32e95 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5428:6
    #3 0x55a59ec33c5b in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5464:10
    #4 0x55a59ec335fc in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5497:12
    #5 0x55a59ec44615 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6322:15
    #6 0x55a59eb8884e in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:29:5
    #7 0x55a59eb889c7 in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:101:17
    #8 0x7fb5b3ba2082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
    #9 0x55a59eac742d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x47d42d) (BuildId: 712c87b21cf5c05f64174745909c693d3ba0b62e)

0x604000000032 is located 0 bytes to the right of 34-byte region [0x604000000010,0x604000000032)
allocated by thread T0 here:
    #0 0x55a59eb4bfee in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x501fee) (BuildId: 712c87b21cf5c05f64174745909c693d3ba0b62e)
    #1 0x55a59eb8899c in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:87:17

```

Found by CI tests.
See: https://github.com/ntop/nDPI/runs/7996151458?check_suite_focus=true
2022-08-25 06:44:38 +02:00
Nardi Ivan 8bfb1712d8 QUIC: add support for 0-RTT packets received before the Initial
RFC9001 4.6.1: "A client that wishes to send 0-RTT packets uses the
early_data extension in the ClientHello message of a subsequent handshake;
see Section 4.2.10 of [TLS13]. It then sends application data in 0-RTT
packets."
That means the client sends before the CH (in the Initial) and then the
0-RTT (in the same UDP datagram or not)".

However, because of packet loss or out-of-order delivery, it might
happens that a 0-RTT packet is received before the Initial (the original
one or a retransmission).
For example, Google and Facebook servers save 0-RTT packets for a small
amount of time in hopes of receiving the corresponding Initial.

Update the QUIC dissector to detect 0-RTT packets and keep looking for
the Initial.

Issue found by @utoni in #1706; the trace example has been taken from that
PR.
2022-08-24 15:38:30 +02:00
Nardi Ivan ac0d7ccb7e HTTP, SoftEther, Florensia: fix some memory corruptions
```
==12308==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60600068f4b4 at pc 0x5603d2b56098 bp 0x7ffc967bd3b0 sp 0x7ffc967bd3a8
READ of size 2 at 0x60600068f4b4 thread T0
    #0 0x5603d2b56097 in ndpi_search_florensia /home/ivan/svnrepos/nDPI/src/lib/protocols/florensia.c:93:40
    #1 0x5603d2a2e835 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5414:6
    #2 0x5603d2a2f247 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5462:12
    #3 0x5603d2a2eee7 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5481:12
    #4 0x5603d2a3ffb5 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6308:15
    #5 0x5603d2955f4c in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1612:31
    #6 0x5603d294ee64 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2182:10
    #7 0x5603d290aa50 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7
    #8 0x5603d290ab8b in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:179:17
    #9 0x7f972cdc9082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
    #10 0x5603d28494fd in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader_with_main+0x4be4fd) (BuildId: e81373a75c70e248ba9ae985fcb4134e7113121a)
```
```
==14164==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60d0000000c1 at pc 0x561608bada2c bp 0x7ffc40d38ed0 sp 0x7ffc40d38690
READ of size 10 at 0x60d0000000c1 thread T0
    #0 0x561608bada2b in __interceptor_strncpy (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x4e9a2b) (BuildId: 2e4fd67f1aeb93c491a04ae424331669c993ee9b)
    #1 0x561608e66f2a in dissect_softether_host_fqdn /home/ivan/svnrepos/nDPI/src/lib/protocols/softether.c:226:2
    #2 0x561608e64e88 in ndpi_search_softether /home/ivan/svnrepos/nDPI/src/lib/protocols/softether.c:316:8
    #3 0x561608ca7475 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5414:6
    #4 0x561608ca823b in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5450:10
    #5 0x561608ca7bdc in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5483:12
    #6 0x561608cb8bf5 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6308:15
    #7 0x561608bff84e in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:29:5
    #8 0x561608bff9c7 in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:101:17
    #9 0x7fc50662d082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
    #10 0x561608b3e42d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x47a42d) (BuildId: 2e4fd67f1aeb93c491a04ae424331669c993ee9b)
```

Found by oss-fuzzer.
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=50105
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=50110
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=50268
2022-08-22 17:53:52 +02:00
Nardi Ivan 00eb2393f4 STUN: improve detection of Hangout/Duo and FB voip calls 2022-07-20 20:36:31 +02:00
Nardi Ivan d66aa49787 DTLS: fix exclusion of DTLS protocol
Add an helper to exclude a generic protocol
2022-07-20 19:16:18 +02:00
Nardi Ivan e1edb08f06 SKYPE: fix detection over UDP
Commit ba6a48c9 is completely bogus: we can't set extra dissection
without having set a proper classification.

The idea behind that commit seems to be that we need to look for 2
(consecutives?) packets with the same crc/pattern: try to implement this
logic in a saner way.
2022-07-20 16:13:55 +02:00
Nardi Ivan 5702c6fb08 SKYPE: remove detection over TCP
Skype detection over TCP has been completely disable since 659f75138 (3
years ago!).
Since that logic was too weak anyway, remove it.
2022-07-20 16:13:55 +02:00
Nardi Ivan 2636c07571 MONGODB: avoid false positives 2022-07-07 15:36:05 +02:00
Nardi Ivan a31e79fc3c TLS: ignore invalid Content Type values 2022-07-07 15:36:05 +02:00
Nardi Ivan 017c1a4239 FTP: fix support for START-TLS sessions
When TLS-over-FTP is used, the credentials are encrypted. So we must not
wait for the username and the password commands, otherwise we elaborate a
lot of packets for nothing.
2021-09-21 20:53:04 +02:00
Nardi Ivan 95ac8fd41a STUN: fix extraction of Realm attribute
While at it, improve detection of Facebook Messenger
2021-09-20 11:04:54 +02:00
Nardi Ivan 03d3e1bafc Fix parsing of ipv6 packets with extension headers
Decoding of ipv6 traffic with extension headers was completely broken,
since the beginning of the L4 header was always set to a wrong value.

Handle the ipv6 fragments in the same way as the ipv4 ones: keep the first
one and drop the others.
2021-09-19 17:29:22 +02:00
Nardi Ivan dcac633878 QUIC: add support for MVFST EXPERIMENTAL version 2020-09-20 16:38:28 +02:00
Nardi Ivan 8db084ab06 TLS: fix memory accesses in QUIC transport parameters extension 2020-09-10 18:19:18 +02:00
Nardi Ivan 7d5a0e1f04 QUIC: extract User Agent information 2020-09-08 11:03:22 +02:00
Nardi Ivan a1014e8895 http: create a common function to parse User Agent field
Prepare the code to handle UA information from flows other than HTTP
2020-09-08 10:34:05 +02:00
Nardi Ivan 7da4abe6ad QUIC: add support for GQUIC T050 and T051
QUIC versioning wasn't complex enough without T05X family...
These versions are very similar to Q050, but use TLS as their handshake
protocol.
2020-08-30 20:51:33 +02:00
Nardi Ivan 97b80a8838 QUIC: minor fixes
LGTM found a real issue on a boundary check
Fix unit tests: a pcap ha been uploaded twice (with different names)
Fix compilation when using DPDK (see #990)
2020-08-24 13:53:36 +02:00
Nardi Ivan b23cfd6b84 Add sub-classification for GQUIC >= Q050 and (IETF-)QUIC
Add QUIC payload and header decryption: most of the crypto code has been
"copied-and-incolled" from Wireshark. That code has been clearly marked
as such. All credits for that code should go to the original authors.

I tried to keep the Wireshark code as similar as possible to the original,
comments included, to ease future backporting of fixes.
Inevitably, glibc data types and data structures, tvbuff abstraction and
allocation functions have been converted.
2020-08-21 22:04:55 +02:00
Nardi Ivan 7a1147d733 Update TLS dissector to handle QUIC flows
Latest QUIC versions use TLS for the encryption layer: reuse existing code
to allow Client Hello parsing and sub-classification based on SNI value.
Side effect: we might have J3AC, TLS negotiated version, SNI value and
supported cipher list for QUIC, too.
2020-08-21 22:04:55 +02:00
Nardi Ivan d62ae567d1 Add (optional) dependency on external libraries: libgcrypt and libgpg-error
To support QUIC payload and header decryption, it is necessary to choose an
external crypto library to handle the low-level crypto stuff. Since we will
use some Wireshark code, it is quite natural to choose the same library used
by Wireshark itself: libgcrypt.

More precisely, we will use libgcrypt and libgpg-error.
Both libraries have LGPL license, so there should be no issue from this point
of view.

These libraries are not required to build nDPI, and their usage is optional:
nDPI will keep working (and compiling) even if they are not available.
However, without them, QUIC sub-classification is next to impossible.
The configure flag "--disable-gcrypt" forces the build system to ignore these
libraries.

libgpg-error is only used for debug to have meaningful error messages and its
usage is trivial.
The same cannot be said for libgcrypt because its initialization is a significant
issue.

The rest of this commit message try explaining how libgcrypt is
initialized.

According to the documentation
   https://gnupg.org/documentation/manuals/gcrypt/Initializing-the-library.html
   https://gnupg.org/documentation/manuals/gcrypt/Multi_002dThreading.html#Multi_002dThreading
libgcrypt must be initialized before using it, but such initialization should
be performed by the actual application and not by any library.

Forcing the users to proper initialize libgcrypt in their own code seems
unreasonable: most people using nDPI might be complete unaware of any crypto
stuff and update each and every one application linking to nDPI with specific
libgcrypt code should be out of question, anyway.

Fortunately, it seems a workaround exists to initialize libgcrypt in a library
    https://lists.gnupg.org/pipermail/gcrypt-devel/2003-August/000458.html

Therefore, we could provide a wrapper to this initialization stuff in a nDPI
function. Unfortunately nDPI API lacks a global init function that must be
called only once, before any other functions. We could add it, but that would
be a major API break.
AFAIK, ndpi_init_detection_module() might be called multiple times, for example
to create multiple independent dpi engines in the same program.

The proposed solution is to (optionally) initialize libgcrypt in
ndpi_init_detection_module() anyway:
* if the actual application doesn't directly use libgcrypt and only calls
  ndpi_init_detection_module() once, everything is formally correct and it
  should work out of the box [by far the most common user case];
* if the actual application already uses libgcrypt directly, it already
  performs the required initialization. In this case the ndpi_prefs.ndpi_dont_init_libgcrypt
  flag should be passed to ndpi_init_detection_module() to avoid further
  initializations.

The only scenario not supported by this solution is when the application is
unaware of libgcrypt and calls ndpi_init_detection_module() multiple times
concurrently. But this scenario should be uncommon.

A completely different option should be to switch to another crypto library,
with a huge impact on the QUIC dissector code.

Bottom line: crypto is hard, using libgcrypt is complex and the proposed
initialization, even if not perfect, should cover the most frequent user
cases and should work, for the time being.
If anyone has some suggestions...
2020-08-21 22:04:55 +02:00
Nardi Ivan 23ec82b59d Major rework of QUIC dissector
Improve support for GQUIC (up to Q046) and add support for Q050 and (IETF-)QUIC
Still no sub-classification for Q050 and QUIC
2020-08-21 22:04:55 +02:00