Commit Graph
4007 Commits
Author SHA1 Message Date
Ivan Nardi 9fc724de5a Add some fuzzers to test other data structures. (#1870)
Start using a dictionary for fuzzing (see:
https://llvm.org/docs/LibFuzzer.html#dictionaries).
Remove some dead code.
Fuzzing with debug enabled is not usually a great idea (from performance
POV). Keep the code since it might be useful while debugging.
2023-01-25 11:44:59 +01:00
Ivan Nardi 29c5cc39fb Some small changes (#1869)
All dissector callbacks should not be exported by the library; make static
some other local functions.
The callback logic in `ndpiReader` has never been used.
With internal libgcrypt, `gcry_control()` should always return no
errors.
We can check `categories` length at compilation time.
2023-01-25 11:44:09 +01:00
Luca Deri 5849863ef9 Added new risk NDPI_TCP_ISSUES 2023-01-24 22:58:17 +01:00
Ivan Nardi 5e8c1ebbb7 fuzz: fix memory allocation failure logic (#1867)
We *do* want to have some allocation errors.
Fix some related bugs
Fix: 29be01ef
2023-01-20 14:27:33 +01:00
Ivan Nardi 496b284c98 Bittorrent: fix detection over TCP (#1868)
Close #1866
2023-01-19 22:02:23 +01:00
Ivan Nardi c4274a4aec Minor changes in ndpi_detection_giveup() (#1861)
Check the mining cache at the end of the function, like all the others
LRU caches.
Rewrite the STUN checks: same identical semantic but in a cleaner way,
hopefully
2023-01-19 21:45:35 +01:00
Ivan Nardi f5c2f9280c Fix undefined-behaviour in ahocorasick callback (#1864)
```
ndpi_main.c:2119:28: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
```
Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=55226
2023-01-19 14:13:17 +01:00
Ivan Nardi f95bdaf625 Bittorrent: fix heap-buffer-overflow (#1863)
```
==258287==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60600068ff9d at pc 0x5653a6e35def bp 0x7ffeef5aa620 sp 0x7ffeef5a9dc8
READ of size 22 at 0x60600068ff9d thread T0
    #0 0x5653a6e35dee in strncmp (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader+0x4d2dee) (BuildId: 133b8c3c8ff99408109fcb9be2538bb8341f07f7)
    #1 0x5653a70d6624 in ndpi_search_bittorrent /home/ivan/svnrepos/nDPI/src/lib/protocols/bittorrent.c:500:71
    #2 0x5653a6ff255a in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5686:6
    #3 0x5653a6ff331b in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5722:10
    #4 0x5653a6ff2cbc in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5755:12
    #5 0x5653a70016bf in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6578:15
    #6 0x5653a6f1836d in packet_processing /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:1678:31
    #7 0x5653a6f140a1 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:2256:10
```
Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=55218
Fix: 470eaa6f
2023-01-19 14:12:51 +01:00
Ivan Nardi edb8165ab9 Sync unit tests results (#1865) 2023-01-19 11:03:45 +01:00
Luca Deri bc44334599 Shrinked symbolic flow risks labels 2023-01-19 10:14:32 +01:00
Ivan Nardi 02f52207be Fuzz: fix compilation with nBPF (#1860)
Close #1859
2023-01-18 18:26:54 +01:00
Ivan Nardi 1b98bec0ab LRU caches: add a generic (optional and configurable) expiration logic (#1855)
Two caches already implemented a similar mechanism: make it generic.
2023-01-18 18:18:36 +01:00
Ivan Nardi de24206adc POP3: improve detection (#1856) 2023-01-18 07:19:44 +01:00
Ivan Nardi 97014c53f3 Improve support for Snapchat voip calls (#1858)
Latest Snapchat versions use QUICv1 for their audio/video real time
sessions. See c50a8d480
2023-01-17 22:02:23 +01:00
Ivan Nardi ccc5a20710 Add detection of Tailscale (#1857)
Extend the example of wireguard traffic
2023-01-17 21:55:38 +01:00
Ivan Nardi 470eaa6ff3 Add Meraki Cloud protocol and improve other protocols (#1854)
Avoid some LineCall and Jabber false positives.
Detect Discord mid flows.
Fix Bittorrent detection.
2023-01-17 11:38:39 +01:00
Ivan Nardi 29be01ef3a Add some fuzzers to test algorithms and data structures (#1852)
Fix some issues found with these new fuzzers
2023-01-17 08:31:59 +01:00
Ivan Nardi ebb9ebd2a0 Fix classification "by-port" (#1851)
Classification "by-port" should be the last possible effort, *after*
having test all the LRU caches.

Remove some dead code from `ndpi_detection_giveup()`:
`flow->guessed_protocol_id` is never set to any od those voip protocols
and at that point in this function we never have both a master *and* a
application protocols.
Coverage reports (both from unit tests and from fuzzing) confirms that
was dead code.
2023-01-17 08:26:42 +01:00
Luca Deri 1f7c57deff Improved DGA detection 2023-01-12 10:06:31 +01:00
Ivan Nardi a944514dde Reduce the size of some traces used as unit test (#1845)
No traces and no flows has been removed; only long sessions
has been reduced, keeping only their first packets.

This is quite important in fuzzing systems, since these pcaps are used
as initial seed. There is no value in fuzzing long sessions, because
only the very first packets are really used/processed by nDPI.

Before:
```
du -h tests/pcap/
200M	tests/pcap/
```
After:
```
du -h tests/pcap/
98M	tests/pcap/
```
2023-01-05 16:43:35 +01:00
Ivan Nardi 5a5e01a9e2 PPSTREAM: fix some heap-buffer overflows (#1846)
Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=54802
2023-01-05 16:41:53 +01:00
Ege ÇetinandNardi Ivan 663df7e475 Fix missing bracket at ppstream (#1843)
* add missing bracket

* Sync unit test results

Co-authored-by: Nardi Ivan <nardi.ivan@gmail.com>
2023-01-03 20:11:24 +01:00
Ivan Nardi 9c83f0f3c5 Fix some warnings and two errors found while fuzzing (#1844)
Fix CI
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=54614
2023-01-03 13:21:54 +01:00
Luca Deri cb15d2b328 Added missing prototype 2023-01-03 12:58:52 +01:00
Luca Deri 2c551afbd7 Added NDPI_MINOR_ISSUES risk used for storing generic/relevant information about issues found on traffic. 2022-12-31 16:55:55 +01:00
Luca Deri 1735931f67 Added NDPI_PERIODIC_FLOW flow risk to be used by apps based on nDPI 2022-12-30 19:20:07 +01:00
Luca Deri 8f91b8ba72 Implemented EDNS(0) support in DNS dissector
Improved DNS dissection
2022-12-29 19:38:25 +01:00
Ivan Nardi 560280e6f0 fuzz: add fuzzer testing nDPI (initial) configurations (#1830)
The goal of this fuzzer is to test init and deinit of the library, with
different configurations. In details:
* random memory allocation failures, even during init phase
* random `ndpi_init_prefs` parameter of `ndpi_init_detection_module()`
* random LRU caches sizes
* random bitmask of enabled protocols
* random parameters of `ndpi_set_detection_preferences()`
* random initialization of opportunistic TLS
* random load/don't load of configuration files

This new fuzzer is a C++ file, because it uses `FuzzedDataProvider`
class (see
https://github.com/google/fuzzing/blob/master/docs/split-inputs.md).
Note that the (existing) fuzzers need to be linked with C++ compiler
anyway, so this new fuzzer doesn't add any new requirements.
2022-12-23 19:07:13 +01:00
Ivan Nardi 3de76812d9 Remove some old protocols (#1832)
These protocols:
* have been addeded in the OpenDPI era
* have never been updated since then
* we don't have any pcap examples [*]

If (and it is a big if...) some of these protocols are still somehow
used and if someone is still interested in them, we can probably
re-add them starting from scratch (because the current detection
rules are probably outdated)

Protocols removed: DIRECT_DOWNLOAD_LINK, APPLEJUICE, DIRECTCONNECT,
OPENFT, FASTTRACK, SHOUTCAST, THUNDER, AYIYA, STEALTHNET, FIESTA,
FLORENSIA, AIMINI, SOPCAST

PPSTREAM dissector works (...) only on UDP.

[*]: with do have an AIMINI test pcap but it was some trivial http
traffic detected only by hostname matching, on domains no more
available...
2022-12-23 10:04:08 +01:00
Ivan Nardi 5fafe8374a postgres: improve detection (#1831)
Remove some dead code (found via coverage report)
2022-12-22 21:41:32 +01:00
Ivan Nardi e9d5e72fb5 Fix infinite loop when a custom rule has port 65535 (#1833)
Close #1829
2022-12-21 10:55:14 +01:00
Ivan Nardi 22821241e2 Fix some errors found by oss-fuzz (#1834) 2022-12-20 22:50:18 +01:00
Ivan Nardi fa67166fcc Remove any references to LGTM (#1837) 2022-12-20 22:49:34 +01:00
Darryl Sokoloski b8972d1b38 Added TP-LINK Smart Home Protocol dissector. (#1841)
Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>

Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
2022-12-20 20:46:06 +01:00
Ivan Nardi d8de8eba36 Sync unit tests results (#1840)
Update the documentation, hinting how to test/update *all* the unit
tests.
2022-12-19 17:57:44 +01:00
Darryl Sokoloski 9854aed408 Added TUYA LAN Protocol dissector. (#1838)
Tuya IoTOS Embedded Wi-Fi and BLE SDK for bk7231n.  Used by many "smart"
devices such as LED light strips, bulbs, etc.

Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>

Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
2022-12-19 17:02:19 +01:00
Ivan Nardi ad6bfbad4d Add protocol disabling feature (#1808)
The application may enable only some protocols.
Disabling a protocol means:
*) don't register/use the protocol dissector code (if any)
*) disable classification by-port for such a protocol
*) disable string matchings for domains/certificates involving this protocol
*) disable subprotocol registration (if any)

This feature can be tested with `ndpiReader -B list_of_protocols_to_disable`.

Custom protocols are always enabled.

Technically speaking, this commit doesn't introduce any API/ABI
incompatibility. However, calling `ndpi_set_protocol_detection_bitmask2()`
is now mandatory, just after having called `ndpi_init_detection_module()`.

Most of the diffs (and all the diffs in `/src/lib/protocols/`) are due to
the removing of some function parameters.

Fix the low level macro `NDPI_LOG`. This issue hasn't been detected
sooner simply because almost all the code uses only the helpers `NDPI_LOG_*`
2022-12-18 08:10:57 +00:00
Luca Deri fb0a73c0c7 Updated decription 2022-12-17 19:36:00 +01:00
Ivan Nardi 3c4b12e061 README: another try to fix build badge (#1836) 2022-12-16 12:22:07 +01:00
Ivan Nardi 390dc1d155 README: fix build badge link (#1835)
See: https://github.com/badges/shields/issues/8671
2022-12-16 12:04:16 +01:00
Luca Deri de289f1305 Added ability to decode DNS PTR records used for reverse address resolution 2022-12-15 21:22:53 +01:00
Luca Deri dbff2fbadd Minor RTP fix 2022-12-15 19:17:05 +01:00
Ivan Nardi 5704e4c142 STUN: add detection of ZOOM peer-to-peer flows (#1825)
See: "Enabling Passive Measurement of Zoom Performance in Production Networks"
https://dl.acm.org/doi/pdf/10.1145/3517745.3561414
2022-12-11 23:07:35 +01:00
Luca Deri 2edbe3c11f Centos7 fix 2022-12-11 10:24:42 +01:00
Luca Deri 88d4a25cb3 Rocky9 fix 2022-12-11 10:21:44 +01:00
Ivan Nardi 48a7f6d487 fuzz: some enhancements (#1827)
Load some custom configuration (like in the unit tests) and factorize some
(fuzzing) common code.

There is no way to pass file paths to the fuzzers as parameters. The safe
solution seems to be to load them from the process working dir. Anyway,
missing file is not a blocking error.

Remove some dead code (found looking at the coverage report)
2022-12-10 19:49:11 +01:00
Ivan Nardi 8c7071e040 DTLS: handle (certificate) fragments (#1811)
Keep using the existing function to handle reassembling buffer: rename
it from `ndpi_search_tls_tcp_memory` to
`ndpi_search_tls_memory` and make it "transport" agnostic
2022-12-10 18:32:25 +01:00
Ivan Nardi 79931b2ec7 Fix undefined-behavior when setting empty user-agent (#1821)
```
ndpi_main.c:9111:35: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:44:28: note: nonnull attribute specified here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ndpi_main.c:9111:35 in
```
2022-12-10 18:31:51 +01:00
Luca Deri 9dc433f709 Packaing changes 2022-12-10 13:14:33 +01:00
Ivan Nardi 9f7ef72332 fuzz: fix signed-integer-overflow (#1822)
```
fuzz_ndpi_reader.c:33:29: runtime error: signed integer overflow: 214013 * 24360337 cannot be represented in type 'int'
    #0 0x4c1cf7 in fastrand ndpi/fuzz/fuzz_ndpi_reader.c:33:29
    #1 0x4c1cf7 in malloc_wrapper ndpi/fuzz/fuzz_ndpi_reader.c:38:11
    #2 0x523057 in ndpi_malloc ndpi/src/lib/ndpi_main.c:220:25
```
Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=54112
2022-12-10 09:18:36 +01:00