4747 Commits
Author SHA1 Message Date
Ivan Nardi 33d3d254ac CI: add support for ubuntu-24.04 runners (#2443) 2024-05-16 17:05:13 +02:00
Vladimir Gavrilov 3d1da00d8d Add Call of Duty Mobile support (#2438) 2024-05-15 12:46:02 +02:00
Ivan Nardi d6b1c24079 Parallel execution of unit tests (#2435)
Running unit tests is quite a bottleneck while developing or while
waiting for GitHub CI results...
Try to run the tests in parallel, using the `parallel` tool.

By default, tests still run one after the other, as usual; to enable
parallel execution you need `NDPI_FORCE_PARALLEL_UTESTS=1 ./tests/do.sh`

Please note that the output is quite different in parallel mode!

A big part of the script has been rewritten to avoid code dupication
between "serial" and "parallel" path

On my notebook:
```
ivan@ivan-Latitude-E6540:~/svnrepos/nDPI(parallel)$ time ./tests/do.sh
[...]
real	3m12,684s
[...]
ivan@ivan-Latitude-E6540:~/svnrepos/nDPI(parallel)$ time NDPI_FORCE_PARALLEL_UTESTS=1 ./tests/do.sh
[...]
real	0m58,463s
```
2024-05-15 08:17:12 +02:00
Ivan Nardi 0110623b4e H323: improve detection and avoid false positives (#2432) 2024-05-11 23:39:54 +02:00
Ivan Nardi a064261e85 Revert ndpi_strnstr() optimization introduced in a813121e0 (#2439)
New implementation fails tests 11b, 12 and 13.
Revert to the original (BSD) implementation (with also some basic
parameters check)
2024-05-11 23:37:31 +02:00
Vladimir Gavrilov 2d33431948 Add Ethernet Global Data support (#2437) 2024-05-11 16:49:16 +02:00
Ivan Nardi 8c5e6e9761 IRC: simplify detection (#2423)
IRC has its best times well behind, but there are still some servers
using it.

We should try to simplify the detection logic, still based on OpenDPI
logic.

Let's start with some easy changes:
* try to detect TLS connection via standard hostname/SNI matching,
removing an old heuristic (we have never had any trace matching it);
* add some basic server names;
* once we detect that the flow is IRC, we don't have to perform
anything else;
* remove HTTP stuff; real HTTP flows never trigger that data path
* use `ndpi_memmem()` when possible
2024-05-11 14:12:13 +02:00
Vladimir GavrilovandIvan Nardi 70c4920184 Remove Vevo support (#2436)
Co-authored-by: Ivan Nardi <12729895+IvanNardi@users.noreply.github.com>
2024-05-11 11:57:53 +02:00
Ivan Nardi b116456fc5 Viber: add detection of voip calls and avoid false positives (#2434) 2024-05-11 09:21:13 +02:00
Vladimir Gavrilov a813121e0a ndpi_strnstr() optimization (#2433) 2024-05-10 22:43:59 +02:00
Ivan Nardi 4b4b358562 Line: use common helper to detect RTP/RTCP packets (#2429)
Add an explicit upper limit on the number of packets processed before
giving up.
2024-05-10 19:23:50 +02:00
Luca Deri 6bbae9a70a Update README.md 2024-05-10 08:53:19 +02:00
Ivan Nardi d4650f0f81 Raknet/RTP: avoid Raknet false positives and harden RTP heuristic (#2427)
There is some overlap between RTP and Raknet detection: give precedence
to RTP logic.
Consequences:
* Raknet might require a little bit more packets for some flows (not a
big issue)
* some very small (1-2 pkts) Raknet flows are not classified (not sure
what do do about that..)
2024-05-09 21:18:18 +02:00
Luca Deri e4fd6e47a3 Added #ifdef for Windows builds (#2431) 2024-05-09 17:27:25 +02:00
Ivan Nardi 2e7fc82e55 Protobuf: fix false positives (#2428) 2024-05-09 15:38:06 +02:00
Toni 18e03a26ca Add extra entropy checks and more precise(?) analysis. (#2383)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-05-09 15:24:11 +02:00
Ivan Nardi b65a755e85 Fix a memory access error and some leaks (#2425)
```
SCARINESS: 12 (1-byte-read-heap-buffer-overflow)
    #0 0x557f3a5b5100 in ndpi_get_host_domain /src/ndpi/src/lib/ndpi_domains.c:158:8
    #1 0x557f3a59b561 in ndpi_check_dga_name /src/ndpi/src/lib/ndpi_main.c:10412:17
    #2 0x557f3a51163a in process_chlo /src/ndpi/src/lib/protocols/quic.c:1467:7
    #3 0x557f3a469f4b in LLVMFuzzerTestOneInput /src/ndpi/fuzz/fuzz_quic_get_crypto_data.c:44:7
    #4 0x557f3a46abc8 in NaloFuzzerTestOneInput (/out/fuzz_quic_get_crypto_data+0x4cfbc8)
```

Some notes about the leak: if the insertion into the uthash fails (because of an
allocation failure), we need to free the just allocated entry. But the only
way to check if the `HASH_ADD_*` failed, is to perform a new lookup: a bit
costly, but we don't use that code in the fast-path.
See also efb261a95c

Credits for finding the issues to Philippe Antoine (@catenacyber) and his
`nallocfuzz` fuzzing engine
See: https://github.com/catenacyber/nallocfuzz
See: https://github.com/google/oss-fuzz/pull/9902
2024-05-08 11:46:46 +02:00
Ivan Nardi 7c6910d9e5 Fix/improve fuzzing (#2426) 2024-05-08 11:46:02 +02:00
Toni e9dc035c5c Added optimized memmem/strlcpy version (#2424)
* credits goes to Vladimir Gavrilov

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-05-08 11:38:53 +02:00
Luca Deri 2b4e2f9c9a Removed trace 2024-05-07 23:42:32 +02:00
Luca Deri 5f8ad0527d Reworked code and added two new API calls
- int ndpi_load_ipv4_ptree_file(ndpi_ptree_t *tree, const char *path, u_int16_t protocol_id);
- int ndpi_load_ipv6_ptree_file(ndpi_ptree_t *tree, const char *path, u_int16_t protocol_id);
2024-05-07 23:05:31 +02:00
Luca Deri 3d2b611ea6 ndpi_mod is now optional (albeit better to spcify it) in ndpi_domain_classify_xxx calls 2024-05-07 13:09:00 +02:00
Luca Deri 0d495294bf Win warning fixes 2024-05-07 10:04:11 +02:00
Luca Deri 6195dbad76 Renamed radius source file to avoid name clashes on Windows 2024-05-07 09:05:15 +02:00
Ivan Nardi 1773d7ff2d Add support for Mastodon, Bluesky and (FB-)Threads (#2418) 2024-05-06 13:37:18 +02:00
Ivan Nardi 95fe21015d Remove "zoom" cache (#2420)
This cache was added in b6b4967aa, when there was no real Zoom support.
With 63f349319, a proper identification of multimedia stream has been
added, making this cache quite useless: any improvements on Zoom
classification should be properly done in Zoom dissector.

Tested for some months with a few 10Gbits links of residential traffic: the
cache pretty much never returned a valid hit.
2024-05-06 12:51:45 +02:00
Ivan Nardi b7d77972bc Remove workaround for TCP flows with multiple SYNs (#2421)
Deciding when a session starts and ends is responsability of the
applicationi (via its flow manager)i, not of the library.
BTW, the removed code is incomplete at beast
2024-05-06 10:20:40 +02:00
Ivan Nardi e31ef00715 TLS: avoid setting NDPI_TLS_SELFSIGNED_CERTIFICATE for webrtc traffic (#2417)
See RFC8122: it is quite likely that STUN/DTLS/SRTP flows use
self-signed certificates

Follow-up of b287d6ec8
2024-05-06 10:20:07 +02:00
Ivan Nardi 266af02752 Merge RTP and RTCP logic (#2416)
Avoid code duplication between these two protocols.

We remove support for RTCP over TCP; it is quite rare to find this kind
of traffic and, more important, we have never had support for RTP
over TCP: we should try to add both detecion as follow-up.

Fix a message log in the LINE code
2024-05-06 10:19:46 +02:00
Ivan Nardi a1845da359 TLS: fix Ja4 fingerprint computation (#2419)
The new values has been checked against the ones reported by Wireshark.

Found while fixing a Use-of-uninitialized-value error reported by
oss-fuzz

```
==7582==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x5a6549abc368 in ndpi_compute_ja4 ndpi/src/lib/protocols/tls.c:1762:10
    #1 0x5a6549ab88a0 in processClientServerHello ndpi/src/lib/protocols/tls.c:2863:10
    #2 0x5a6549ac1452 in processTLSBlock ndpi/src/lib/protocols/tls.c:909:5
    #3 0x5a6549abf588 in ndpi_search_tls_tcp ndpi/src/lib/protocols/tls.c:1098:2
    #4 0x5a65499c53ec in check_ndpi_detection_func ndpi/src/lib/ndpi_main.c:7215:6
```

See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=68449&q=ndpi&can=1&sort=-id
2024-05-05 17:27:29 +02:00
Ivan Nardi a6fd981fcf eDonkey: improve/update classification (#2410)
eDonkey is definitely not as used as >10 years ago, but it seems it is
still active.

While having a basic TCP support seems easy, identification over UDP doesn't
work and it is hard to do it rightly (packets might be only 2 bytes long):
remove it.

Credits to V.G <v.gavrilov@securitycode.ru>
2024-05-04 19:11:31 +02:00
Luca Deri 57ecbf38c0 Updated JA4 test results 2024-05-02 17:40:24 +02:00
Luca Deri 66133d1501 Fixes JA4 computation adding a better GREASE detect funzion 2024-05-02 16:58:44 +02:00
Luca Deri af955169e1 typo 2024-05-02 16:03:30 +02:00
Ivan Nardi 9e549ba781 Workaround for fixing GitHub runners on macOS (#2411)
GitHub switched "macos-latest" from "macos-12" to "macos-14", which is only
on ARM64!
https://github.com/actions/runner/issues/3256
https://github.blog/changelog/2024-01-30-github-actions-macos-14-sonoma-is-now-available/

However we are having some issues build nDPI on macos-14 with external
libraries:

```
configure: error: libgpg-error required (because of --with-local-libgcrypt) but not found or too old.
```
See: https://github.com/ntop/nDPI/actions/runs/8869020568/job/24350356867

```
ndpi_utils.c:69:10: fatal error: 'pcre2.h' file not found
         ^~~~~~~~~
1 error generated.
```
See: https://github.com/ntop/nDPI/actions/runs/8869020568/job/24349242251

Everything is still fine with macos-14 and no external dependencies

As workaround, test only macos-12 and macos-13 in our main matrix.
2024-04-28 22:17:39 +02:00
Ivan Nardi d5bda47efe DTLS: add support for Alert message type (similar to TLS) (#2406) 2024-04-25 11:23:05 +02:00
0x41CEA55 7040847eed Add Adobe Connect support (#2407) 2024-04-24 12:41:25 +02:00
0x41CEA55 66036a14f4 Remove PPStream protocol and add iQIYI (#2403)
P2P video player PPStream was discontinued shortly after the purchase of PPS.tv by Baidu (iQIYI) on 2013 (see https://www.techinasia.com/report-baidu-acquires-video-rival-pps)
So we remove the old `NDPI_PROTOCOL_PPSTREAM` logic and add `NDPI_PROTOCOL_IQIYI` id to handle all the iQIYI traffic, which is basically video streaming traffic.

A video hosting service, called PPS.tv, is still offered by the same company: for the time being we classified both services with the same protocol id.
2024-04-23 18:01:36 +02:00
0x41CEA55 fd388845d5 Add BFCP protocol support (#2401) 2024-04-23 15:35:19 +02:00
Ivan Nardi abc7e430e2 STUN: slightly faster sub-classification with DTLS (#2404) 2024-04-23 09:31:48 +02:00
Ivan Nardi a62679952c fuzz: fix shoco fuzzer (#2405)
```
==22779==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f0900701020 at pc 0x555bcd2a6f02 bp 0x7ffe3ba5e790 sp 0x7ffe3ba5e788
READ of size 1 at 0x7f0900701020 thread T0
    #0 0x555bcd2a6f01 in shoco_decompress /home/ivan/svnrepos/nDPI/src/lib/third_party/src/shoco.c:184:26
    #1 0x555bcd2a4018 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco.cpp:18:3
    #2 0x555bcd1aa816 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco+0x4f816) (BuildId: c54d1c32163c9937e06f62127348ae6bd26d9309)
    #3 0x555bcd193be8 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco+0x38be8) (BuildId: c54d1c32163c9937e06f62127348ae6bd26d9309)
    #4 0x555bcd1996fa in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco+0x3e6fa) (BuildId: c54d1c32163c9937e06f62127348ae6bd26d9309)
    #5 0x555bcd1c3c92 in main (/home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco+0x68c92) (BuildId: c54d1c32163c9937e06f62127348ae6bd26d9309)
    #6 0x7f090257a082 in __libc_start_main /build/glibc-e2p3jK/glibc-2.31/csu/../csu/libc-start.c:308:16
    #7 0x555bcd18e96d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco+0x3396d) (BuildId: c54d1c32163c9937e06f62127348ae6bd26d9309)

Address 0x7f0900701020 is located in stack of thread T0 at offset 4128 in frame
    #0 0x555bcd2a3d97 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco.cpp:5

  This frame has 4 object(s):
    [32, 4128) 'out' (line 9) <== Memory access at offset 4128 overflows this variable
    [4256, 8352) 'orig' (line 9)
```

Found by oss-fuzzer.
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=68211
2024-04-22 18:39:23 +02:00
Nardi Ivan f494bdd653 Fix parameters check
```
==17==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x000000546050 bp 0x7fff113c82a0 sp 0x7fff113c7a58 T0)
==17==The signal is caused by a READ memory access.
==17==Hint: address points to the zero page.
SCARINESS: 10 (null-deref)
    #0 0x546050 in __sanitizer::internal_strlen(char const*) /src/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_libc.cpp:167:10
    #1 0x4c6ba5 in __interceptor_strrchr /src/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:740:5
    #2 0x5fb9b9 in ndpi_get_host_domain_suffix /src/ndpi/src/lib/ndpi_domains.c:105:20
    #3 0x578058 in LLVMFuzzerTestOneInput /src/ndpi/fuzz/fuzz_config.cpp:503:3
```

Found while fuzzing
2024-04-21 12:29:09 +02:00
0x41CEA55 246e972f9c Replace my personal email with my corporate one in all my contributions (#2399) 2024-04-20 21:37:01 +02:00
Nardi Ivan 95f4d0b852 Fix a warning
```
nDPI/PcDebug64/src/include/ndpi_api.h:1970:3: error: function declaration isn’t a prototype [-Werror=strict-prototypes]
```
2024-04-20 19:30:42 +02:00
Ivan Nardi ef89183469 fuzz: improvements (#2400)
Create the zip file with all the traces only once.

Add a new fuzzer to test "shoco" compression algorithm
2024-04-20 18:15:23 +02:00
0x41CEA55 905120588b Remove obsolete protocols: tuenty, tvuplayer and kontiki (#2398) 2024-04-19 21:35:32 +02:00
0x41CEA55 1b2e2cd968 Add strlcpy implementation (#2395) 2024-04-19 17:16:40 +02:00
0x41CEA55 e75d7a620e Add KNXnet/IP protocol support (#2397)
* Add KNXnet/IP protocol support

* Improve KNXnet/IP over TCP detection
2024-04-19 12:54:00 +02:00
Luca Deri ad117bfaab Domain Classification Improvements (#2396)
* Added
size_t ndpi_compress_str(const char * in, size_t len, char * out, size_t bufsize);
size_t ndpi_decompress_str(const char * in, size_t len, char * out, size_t bufsize);

used to compress short strings such as domain names. This code is based on
https://github.com/Ed-von-Schleck/shoco

* Major code rewrite for ndpi_hash and ndpi_domain_classify

* Improvements to make sure custom categories are loaded and enabled

* Fixed string encoding

* Extended SalesForce/Cloudflare domains list
2024-04-18 23:21:40 +02:00
Luca Deri 108b8331d5 Invalid initialization 2024-04-15 18:13:17 +02:00