Commit Graph
2716 Commits
Author SHA1 Message Date
Ivan Nardi f95bdaf625 Bittorrent: fix heap-buffer-overflow (#1863)
```
==258287==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60600068ff9d at pc 0x5653a6e35def bp 0x7ffeef5aa620 sp 0x7ffeef5a9dc8
READ of size 22 at 0x60600068ff9d thread T0
    #0 0x5653a6e35dee in strncmp (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader+0x4d2dee) (BuildId: 133b8c3c8ff99408109fcb9be2538bb8341f07f7)
    #1 0x5653a70d6624 in ndpi_search_bittorrent /home/ivan/svnrepos/nDPI/src/lib/protocols/bittorrent.c:500:71
    #2 0x5653a6ff255a in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5686:6
    #3 0x5653a6ff331b in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5722:10
    #4 0x5653a6ff2cbc in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5755:12
    #5 0x5653a70016bf in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6578:15
    #6 0x5653a6f1836d in packet_processing /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:1678:31
    #7 0x5653a6f140a1 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:2256:10
```
Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=55218
Fix: 470eaa6f
2023-01-19 14:12:51 +01:00
Luca Deri bc44334599 Shrinked symbolic flow risks labels 2023-01-19 10:14:32 +01:00
Ivan Nardi 1b98bec0ab LRU caches: add a generic (optional and configurable) expiration logic (#1855)
Two caches already implemented a similar mechanism: make it generic.
2023-01-18 18:18:36 +01:00
Ivan Nardi de24206adc POP3: improve detection (#1856) 2023-01-18 07:19:44 +01:00
Ivan Nardi 97014c53f3 Improve support for Snapchat voip calls (#1858)
Latest Snapchat versions use QUICv1 for their audio/video real time
sessions. See c50a8d480
2023-01-17 22:02:23 +01:00
Ivan Nardi ccc5a20710 Add detection of Tailscale (#1857)
Extend the example of wireguard traffic
2023-01-17 21:55:38 +01:00
Ivan Nardi 470eaa6ff3 Add Meraki Cloud protocol and improve other protocols (#1854)
Avoid some LineCall and Jabber false positives.
Detect Discord mid flows.
Fix Bittorrent detection.
2023-01-17 11:38:39 +01:00
Ivan Nardi 29be01ef3a Add some fuzzers to test algorithms and data structures (#1852)
Fix some issues found with these new fuzzers
2023-01-17 08:31:59 +01:00
Ivan Nardi ebb9ebd2a0 Fix classification "by-port" (#1851)
Classification "by-port" should be the last possible effort, *after*
having test all the LRU caches.

Remove some dead code from `ndpi_detection_giveup()`:
`flow->guessed_protocol_id` is never set to any od those voip protocols
and at that point in this function we never have both a master *and* a
application protocols.
Coverage reports (both from unit tests and from fuzzing) confirms that
was dead code.
2023-01-17 08:26:42 +01:00
Luca Deri 1f7c57deff Improved DGA detection 2023-01-12 10:06:31 +01:00
Ivan Nardi 5a5e01a9e2 PPSTREAM: fix some heap-buffer overflows (#1846)
Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=54802
2023-01-05 16:41:53 +01:00
Ege ÇetinandNardi Ivan 663df7e475 Fix missing bracket at ppstream (#1843)
* add missing bracket

* Sync unit test results

Co-authored-by: Nardi Ivan <nardi.ivan@gmail.com>
2023-01-03 20:11:24 +01:00
Ivan Nardi 9c83f0f3c5 Fix some warnings and two errors found while fuzzing (#1844)
Fix CI
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=54614
2023-01-03 13:21:54 +01:00
Luca Deri cb15d2b328 Added missing prototype 2023-01-03 12:58:52 +01:00
Luca Deri 2c551afbd7 Added NDPI_MINOR_ISSUES risk used for storing generic/relevant information about issues found on traffic. 2022-12-31 16:55:55 +01:00
Luca Deri 1735931f67 Added NDPI_PERIODIC_FLOW flow risk to be used by apps based on nDPI 2022-12-30 19:20:07 +01:00
Luca Deri 8f91b8ba72 Implemented EDNS(0) support in DNS dissector
Improved DNS dissection
2022-12-29 19:38:25 +01:00
Ivan Nardi 560280e6f0 fuzz: add fuzzer testing nDPI (initial) configurations (#1830)
The goal of this fuzzer is to test init and deinit of the library, with
different configurations. In details:
* random memory allocation failures, even during init phase
* random `ndpi_init_prefs` parameter of `ndpi_init_detection_module()`
* random LRU caches sizes
* random bitmask of enabled protocols
* random parameters of `ndpi_set_detection_preferences()`
* random initialization of opportunistic TLS
* random load/don't load of configuration files

This new fuzzer is a C++ file, because it uses `FuzzedDataProvider`
class (see
https://github.com/google/fuzzing/blob/master/docs/split-inputs.md).
Note that the (existing) fuzzers need to be linked with C++ compiler
anyway, so this new fuzzer doesn't add any new requirements.
2022-12-23 19:07:13 +01:00
Ivan Nardi 3de76812d9 Remove some old protocols (#1832)
These protocols:
* have been addeded in the OpenDPI era
* have never been updated since then
* we don't have any pcap examples [*]

If (and it is a big if...) some of these protocols are still somehow
used and if someone is still interested in them, we can probably
re-add them starting from scratch (because the current detection
rules are probably outdated)

Protocols removed: DIRECT_DOWNLOAD_LINK, APPLEJUICE, DIRECTCONNECT,
OPENFT, FASTTRACK, SHOUTCAST, THUNDER, AYIYA, STEALTHNET, FIESTA,
FLORENSIA, AIMINI, SOPCAST

PPSTREAM dissector works (...) only on UDP.

[*]: with do have an AIMINI test pcap but it was some trivial http
traffic detected only by hostname matching, on domains no more
available...
2022-12-23 10:04:08 +01:00
Ivan Nardi 5fafe8374a postgres: improve detection (#1831)
Remove some dead code (found via coverage report)
2022-12-22 21:41:32 +01:00
Ivan Nardi e9d5e72fb5 Fix infinite loop when a custom rule has port 65535 (#1833)
Close #1829
2022-12-21 10:55:14 +01:00
Ivan Nardi 22821241e2 Fix some errors found by oss-fuzz (#1834) 2022-12-20 22:50:18 +01:00
Darryl Sokoloski b8972d1b38 Added TP-LINK Smart Home Protocol dissector. (#1841)
Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>

Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
2022-12-20 20:46:06 +01:00
Darryl Sokoloski 9854aed408 Added TUYA LAN Protocol dissector. (#1838)
Tuya IoTOS Embedded Wi-Fi and BLE SDK for bk7231n.  Used by many "smart"
devices such as LED light strips, bulbs, etc.

Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>

Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
2022-12-19 17:02:19 +01:00
Ivan Nardi ad6bfbad4d Add protocol disabling feature (#1808)
The application may enable only some protocols.
Disabling a protocol means:
*) don't register/use the protocol dissector code (if any)
*) disable classification by-port for such a protocol
*) disable string matchings for domains/certificates involving this protocol
*) disable subprotocol registration (if any)

This feature can be tested with `ndpiReader -B list_of_protocols_to_disable`.

Custom protocols are always enabled.

Technically speaking, this commit doesn't introduce any API/ABI
incompatibility. However, calling `ndpi_set_protocol_detection_bitmask2()`
is now mandatory, just after having called `ndpi_init_detection_module()`.

Most of the diffs (and all the diffs in `/src/lib/protocols/`) are due to
the removing of some function parameters.

Fix the low level macro `NDPI_LOG`. This issue hasn't been detected
sooner simply because almost all the code uses only the helpers `NDPI_LOG_*`
2022-12-18 08:10:57 +00:00
Luca Deri de289f1305 Added ability to decode DNS PTR records used for reverse address resolution 2022-12-15 21:22:53 +01:00
Luca Deri dbff2fbadd Minor RTP fix 2022-12-15 19:17:05 +01:00
Ivan Nardi 5704e4c142 STUN: add detection of ZOOM peer-to-peer flows (#1825)
See: "Enabling Passive Measurement of Zoom Performance in Production Networks"
https://dl.acm.org/doi/pdf/10.1145/3517745.3561414
2022-12-11 23:07:35 +01:00
Ivan Nardi 48a7f6d487 fuzz: some enhancements (#1827)
Load some custom configuration (like in the unit tests) and factorize some
(fuzzing) common code.

There is no way to pass file paths to the fuzzers as parameters. The safe
solution seems to be to load them from the process working dir. Anyway,
missing file is not a blocking error.

Remove some dead code (found looking at the coverage report)
2022-12-10 19:49:11 +01:00
Ivan Nardi 8c7071e040 DTLS: handle (certificate) fragments (#1811)
Keep using the existing function to handle reassembling buffer: rename
it from `ndpi_search_tls_tcp_memory` to
`ndpi_search_tls_memory` and make it "transport" agnostic
2022-12-10 18:32:25 +01:00
Ivan Nardi 79931b2ec7 Fix undefined-behavior when setting empty user-agent (#1821)
```
ndpi_main.c:9111:35: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:44:28: note: nonnull attribute specified here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ndpi_main.c:9111:35 in
```
2022-12-10 18:31:51 +01:00
Ivan Nardi 6004e55d7b Build: fix make install when --prefix= is used in configure script (#1824)
Use the same logic already used in `example/Makefile.in`

Close #1823
2022-12-10 09:13:20 +01:00
Luca Deri eacc2b8e32 Added Zoom screen share detection 2022-12-09 21:32:45 +01:00
Luca Deri fc7b070030 Added RTP stream type in flow metadata 2022-12-09 14:26:53 +01:00
Luca Deri 63f349319f Improved Zoom protocol detection 2022-12-08 22:00:24 +01:00
Ivan Nardi ada4fe4aa8 fuzz: add a new fuzzer testing memory allocation failures (#1818)
Try to fuzz error paths triggered by allocation errors.
Fix some errors already found by this new fuzzer.
Basic idea taken from: https://github.com/harfbuzz/harfbuzz/pull/2566/files

`FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is a standard define used to
(not)compile specific code in fuzzing builds.
See: https://llvm.org/docs/LibFuzzer.html
2022-12-06 17:41:58 +01:00
Ivan Nardi 946c3dba0f Fix compilation (#1819) 2022-12-05 22:33:29 +01:00
Luca Deri e0afc16aa2 Exported HTTP server in metadata 2022-12-05 21:27:30 +01:00
Ivan Nardi 3e4ab39b52 Add support for LTO and Gold linker (#1812)
This commit add (optional) support for Link-Time-Optimization and Gold
linker.
This is the first, mandatory step needed to make nDPI compliant with
"introspector" sanitizer requirements in OSS-Fuzz: see
https://github.com/google/oss-fuzz/issues/8939

Gold linker is not supported by Windows and by macOS, so this feature is
disabled by default. It has been enable in CI in two linux targets
("latest" gcc and clang).

Fix some warnings triggered by LTO.

The changes in `src/lib/ndpi_serializer.c` seams reasonable.
However, the change in `tests/unit/unit.c` is due to the following
warning, which seems to be a false positive.

```
unit.c: In function ‘serializerUnitTest’:
ndpi_serializer.c:2258:13: error: ‘MEM[(struct ndpi_private_serializer *)&deserializer].buffer.size’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
unit.c:67:31: note: ‘MEM[(struct ndpi_private_serializer *)&deserializer].buffer.size’ was declared here
   67 |   ndpi_serializer serializer, deserializer;
      |                               ^
ndpi_serializer.c:2605:10: error: ‘MEM[(struct ndpi_private_serializer *)&deserializer].status.buffer.size_used’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
unit.c:67:31: note: ‘MEM[(struct ndpi_private_serializer *)&deserializer].status.buffer.size_used’ was declared here
   67 |   ndpi_serializer serializer, deserializer;
```
Since this warning is triggered only with an old version of gcc and
`tests/unit/unit.c` is used only during the tests, the easiest fix has
been applied.

Some (unknown to me) combinations of OS and compiler trigger the
following warnings at linker time (with sanitizer and gold linker)
```
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load1_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load2_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load4_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load8_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load16_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store1_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store2_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store4_asm'
[..]
```
I have not found any references to this kind of message, with the only
exception of https://sourceware.org/bugzilla/show_bug.cgi?id=25975
which seems to suggest that these messages can be safely ignored.
In any case, the compilation results are sound.

Fix `clean` target in the Makefile in the `example` directory.

In OSS-Fuzz enviroments, `fuzz_ndpi_reader` reports a strange link error
(as always, when the gold linker is involved...).
It's come out that the culprit was the `tempnam` function: the code has
been changed to use `tmpfile` instead. No sure why... :(

Fuzzing target `fuzz_ndpi_reader.c` doesn't use `libndpiReader.a`
anymore: this way we can use `--with-only-libndpi` flag on Oss-Fuzz builds
as workaround for the "missing dependencies errors" described in
https://github.com/google/oss-fuzz/issues/8939
2022-12-05 10:21:42 +01:00
Ivan Nardi b9f63458e6 CI: GitHub is updating Ubuntu runners (#1817)
GitHub is moving `ubuntu-latest` to `ubuntu-22.04`: update our
dependencies.
See: https://github.blog/changelog/2022-11-09-github-actions-ubuntu-latest-workflows-will-use-ubuntu-22-04/

This is the reason of the recent random failures in CI.

Update "newest" tested gcc to gcc-12.

Fix a memory error introduced in 557bbcfc5a
2022-12-03 18:29:44 +01:00
Ivan Nardi 81e1ea545c Make LRU caches ipv6 aware (#1810)
Simplest solution, keeping the existing cache data structure

TLS certificate cache is used for DTLS traffic, too.

Note that Ookla cache already works with ipv6 flows.

TODO:
 * make the key/hashing more robust (extending the key size?)
 * update bittorrent cache too. That task is quite difficult because
   ntopng uses a public function (`ndpi_guess_undetected_protocol()`)
   intrinsically ipv4 only...
2022-12-03 12:07:32 +01:00
Luca Deri 2a8d0d1fe4 Added check for empty/missing user-agent in HTTP 2022-12-03 11:32:39 +01:00
Luca Deri 557bbcfc5a Improved HTTP header parsing skipping lines that won't match 2022-12-03 11:32:29 +01:00
Ivan Nardi 197a9a6cfc Extend CodeQL configuration and fix two warnings (#1816)
```
Call to `memset` may be deleted
Comparison of narrow type with wide type in loop condition
```

See https://www.usenix.org/system/files/conference/usenixsecurity17/sec17-yang.pdf
for the new portable function `secure_memzero`.
2022-12-02 23:26:22 +01:00
Ivan Nardi 02e7e3c23b Add a new fuzzer for QUIC (#1800)
QUIC packets are encrypted/obfuscated; that means that we need to
decrypt them before parsing the real (TLS) message.
Fuzzing is not effective here, since a random buffer is hardly a valid
encrypted QUIC packet.

Add a new fuzzer, testing *decrypted* QUIC packets.

Add a basic corpus.

Fix a few bugs already found by this fuzzer.
2022-12-01 12:21:04 +01:00
Ivan Nardi afb5c3e6ba Fix classification "by-port" of QUIC flows (#1785)
nDPI is able to properly classify QUIC flows only if it elaborates the
very first packets of the flow.

The protocol list in `is_udp_guessable_protocol()` is basically a list
of protocols which can be detected from *any* packets in the flow.
Rename such function to `is_udp_not_guessable_protocol()`: the name is
still quite cryptic, but at least not plainly wrong
2022-11-22 18:00:02 +01:00
Ivan Nardi 2b65a4e05e TLS: be sure to always set ssl_version field (#1806)
Useful with asymmetric traffic with (D)TLS <= 1.2
2022-11-22 16:13:28 +01:00
Ivan Nardi 388d01a345 DNS: fix an infinite loop, due to an integer overflow (#1804)
Found by sydr-fuzz
Close #1803
2022-11-15 16:29:10 +01:00
Toni b5e8d86def Improved Teamspeak3 License/Weblist detection. (#1802)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-11-15 08:59:45 +01:00
Ivan Nardi cd41ab7c8f Improve export/print of L4 protocol information (#1799)
Close #1797
2022-11-13 22:35:46 +01:00