Compare commits

...
419 Commits
Author SHA1 Message Date
Ivan Nardi ebcea42e2b fuzz: improve fuzzers using pl7m (#2486)
For some unclear reasons, fuzzers using pl7m create huge corpus,
triggering OOM in oss-fuzz runs (where the memory RSS limit is set to
2560Mb). Example:

```
==25340== ERROR: libFuzzer: out-of-memory (used: 2564Mb; limit: 2560Mb)
To change the out-of-memory limit use -rss_limit_mb=<N>

Live Heap Allocations: 2364004039 bytes in 133791 chunks; quarantined: 60662293 bytes in 3664 chunks; 176432 other chunks; total chunks: 313887; showing top 95% (at most 8 unique contexts)
1285841683 byte(s) (54%) in 2956 allocation(s)
    #0 0x56f814ef4bde in __interceptor_malloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3
    #1 0x56f814e04416 in operator new(unsigned long) cxa_noexception.cpp:0
    #2 0x56f814de6b2d in assign<unsigned char *, 0> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1443:3
    #3 0x56f814de6b2d in operator= /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1412:9
    #4 0x56f814de6b2d in fuzzer::InputCorpus::AddToCorpus(std::__Fuzzer::vector<unsigned char, std::__Fuzzer::allocator<unsigned char>> const&, unsigned long, bool, bool, bool, std::__Fuzzer::chrono::duration<long long, std::__Fuzzer::ratio<1l, 1000000l>>, std::__Fuzzer::vector<unsigned int, std::__Fuzzer::allocator<unsigned int>> const&, fuzzer::DataFlowTrace const&, fuzzer::InputInfo const*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerCorpus.h:221:10
    #5 0x56f814de60e5 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:539:16
    #6 0x56f814de7df2 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:829:7
    #7 0x56f814de8127 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:867:3
    #8 0x56f814dd6736 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:914:6
    #9 0x56f814e02c62 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #10 0x7fa11e2c3082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16

1031350683 byte(s) (43%) in 2468 allocation(s)
    #0 0x56f814ef4bde in __interceptor_malloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3
    #1 0x56f814e04416 in operator new(unsigned long) cxa_noexception.cpp:0
    #2 0x56f814de6b2d in assign<unsigned char *, 0> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1443:3
    #3 0x56f814de6b2d in operator= /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1412:9
    #4 0x56f814de6b2d in fuzzer::InputCorpus::AddToCorpus(std::__Fuzzer::vector<unsigned char, std::__Fuzzer::allocator<unsigned char>> const&, unsigned long, bool, bool, bool, std::__Fuzzer::chrono::duration<long long, std::__Fuzzer::ratio<1l, 1000000l>>, std::__Fuzzer::vector<unsigned int, std::__Fuzzer::allocator<unsigned int>> const&, fuzzer::DataFlowTrace const&, fuzzer::InputInfo const*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerCorpus.h:221:10
    #5 0x56f814de60e5 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:539:16
    #6 0x56f814de7635 in fuzzer::Fuzzer::MutateAndTestOne() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:760:19
    #7 0x56f814de8425 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:905:5
    #8 0x56f814dd6736 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:914:6
    #9 0x56f814e02c62 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #10 0x7fa11e2c3082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16
```

See: https://oss-fuzz.com/testcase-detail/4717811415449600
See: https://oss-fuzz.com/testcase-detail/6164130982068224

Let's *try* the following workaround: set the parameter `-max-len` to
512K, to force the engine to not genereate inputs (i.e. pcap files...)
larger than 512K. Right now the value used is 1MB, i.e the default,
because we have file larger than 1MB in the initial seeds (i.e.
`/tests/pcaps/*`).

Let's hope than having smaller files lead to smaller corpus...

Update pl7m code (fix a Use-of-uninitialized-value error)
2024-06-29 18:01:05 +02:00
Ivan Nardi 83e6e753af fuzz: pl7m: add a custom mutator for better fuzzing of pcap files (#2483)
Pl7m is a custom mutator (used for structure aware fuzzing) for network
traffic packet captures (i.e. pcap files).

The output of the mutator is always a valid pcap file, containing the
same flows/sessions of the input file. That's it: the mutator only
changes the packet payload after the TCP/UDP header, keeping all the
original L2/L3 information (IP addresses and L4 ports).

See: https://github.com/IvanNardi/pl7m
2024-06-27 18:07:43 +02:00
Anshul ThakurandAnshul Thakur 8f6f73505d Remove duplicate API for ndpi_serialize_uint32_double in ndpi_api.h (#2480)
Co-authored-by: Anshul Thakur <athakur@cdot.in>
2024-06-26 14:10:57 +02:00
Nardi Ivan 556f892a56 wireshark: lua: export some metadata
Export some metadata (for the moment, SNI and TLS fingerprints) to
Wireshark/tshark via extcap.
Note that:
* metadata are exported only once per flow
* metadata are exported (all together) when nDPI stopped processing
the flow

Still room for a lot of improvements!
In particular:
* we need to add some boundary checks (if we are going to export other
attributes)
* we should try to have a variable length trailer
2024-06-25 16:39:45 +02:00
Nardi Ivan f44832cc51 wireshark: lua: filtering and trailer dissection work with tshark, too
```
ivan@ivan-Latitude-E6540:~/$ tshark -C "nDPI extcap" -i ndpi -o extcap.ndpi.i:/home/ivan/svnrepos/nDPI/tests/pcap/anydesk.pcapng -Y "ndpi.protocol.name contains DNS"
Capturing on 'nDPI interface: ndpi'
   62 22635386.425683 192.168.1.187 DNS.AnyDesk 192.168.1.1  128   Standard query 0xec22 A relay-3185a847.net.anydesk.com
   63 22635386.439540  192.168.1.1 DNS.AnyDesk 192.168.1.187 144   Standard query response 0xec22 A relay-3185a847.net.anydesk.com A 37.61.223.15
   64 22635386.721277 192.168.1.187 DNS.AnyDesk 192.168.1.1  128   Standard query 0xea89 A relay-9b6827f2.net.anydesk.com
   65 22635386.732444  192.168.1.1 DNS.AnyDesk 192.168.1.187 144   Standard query response 0xea89 A relay-9b6827f2.net.anydesk.com A 138.199.36.115
4 packets captured

```
2024-06-25 16:39:45 +02:00
Nardi Ivan 3f0ea18866 wireshark: lua: fix DNS dissection
Not sure when we (or Wireshark, or Lua...) broke it, but we can't call
tonumber() on Bool variables.
2024-06-25 16:39:45 +02:00
Nardi Ivan b5afa165f0 wireshark: extcap: restore filtering mechanism 2024-06-25 16:39:45 +02:00
Nardi Ivan 2daab3f248 wireshark: lua: latest Wireshark versions correctly handle 64 bit mask 2024-06-25 16:39:45 +02:00
Nardi Ivan 46e450456b wireshark: lua: minor improvements
* Use a proper TVB to parse the nDPI trailer
* Fix some flow risks definitions
2024-06-25 16:39:45 +02:00
Luca Deri e92568e4bb Improved logic for checking invalid DNS queries 2024-06-23 22:27:19 +02:00
Mark Jeffery aa1d7247d1 Added default port mappings to ndpiReader help -H (#2477)
Close #2125
2024-06-19 13:47:18 +02:00
Ivan Nardi 9feede9a14 Zoom: fix detection of screen sharing (#2476) 2024-06-17 17:30:24 +02:00
Ivan Nardi 26cc1f131f fuzz: improve fuzzing coverage (#2474)
Remove some code never triggered

AFP: the removed check is included in the following one
MQTT: fix flags extraction
2024-06-17 13:45:47 +02:00
Nardi Ivan a35fae6b75 Sync unit tests results 2024-06-17 11:23:39 +02:00
Toni 8fd649ab1e Add Ripe Atlas probe protocol. (#2473)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-06-17 11:00:38 +02:00
Nardi Ivan bbe52da5cf Zoom: harden RTP/RTCP detection 2024-06-17 10:19:55 +02:00
Nardi Ivan 526cf6f291 Zoom: remove "stun_zoom" LRU cache
Since 070a0908b we are able to detect P2P calls directly from the packet
content, without any correlation among flows
2024-06-17 10:19:55 +02:00
Ivan Nardi 2bedd14aae Update main Readme and remove commented out code (#2471)
Avoid that useless code is copy-and-pasted into new dissector, as in
https://github.com/ntop/nDPI/pull/2470#discussion_r1639384434
2024-06-17 10:11:17 +02:00
Mark Jeffery f796c94375 Added protocol - JRMI - Java Remote Method Invocation (#2470) 2024-06-15 10:52:28 +02:00
Luca 7c7c375b45 Improved detection of Android connectiity checks 2024-06-12 10:55:07 +02:00
Mark Jeffery 312dc424bd Added NDPI_PROTOCOL_NTOP assert and removed percentage comparison (#2460)
Close #2413
2024-06-10 19:45:19 +02:00
Ivan Nardi aee2c81f76 Zoom: fix integer overflow (#2469)
```
AddressSanitizer:DEADLYSIGNAL
=================================================================
==29508==ERROR: AddressSanitizer: SEGV on unknown address 0x50710145d51d (pc 0x55cb788f25fe bp 0x7ffcfefa15f0 sp 0x7ffcfefa1240 T0)
==29508==The signal is caused by a READ memory access.
    #0 0x55cb788f25fe in ndpi_search_zoom /home/ivan/svnrepos/nDPI/src/lib/protocols/zoom.c:210:24
    #1 0x55cb787e9418 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7174:6
    #2 0x55cb7883f753 in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7209:10
    #3 0x55cb7883bc9d in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7240:12
```
Found by oss-fuzzer
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=69520
2024-06-10 11:06:17 +02:00
Ivan Nardi b90d39c4ac RTP/STUN: look for STUN packets after RTP/RTCP classification (#2465)
After a flow has been classified as RTP or RTCP, nDPI might analyse more
packets to look for STUN/DTLS packets, i.e. to try to tell if this flow
is a "pure" RTP/RTCP flow or if the RTP/RTCP packets are multiplexed with
STUN/DTLS.
Useful for proper (sub)classification when the beginning of the flows
are not captured or if there are lost packets in the the captured traffic.

Disabled by default
2024-06-07 13:12:04 +02:00
Ivan Nardi 070a0908b3 Zoom: faster detection of P2P flows (#2467) 2024-06-07 09:50:41 +02:00
Ivan Nardi 619005c5b2 STUN: add support for Microsoft Multiplexed TURN channels (#2464) 2024-06-05 16:55:58 +02:00
Ivan Nardi b6beeef623 TLS: add support for DTLS (over STUN) over TCP (#2463)
TODO: TCP reassembler on top of UDP reassembler

See: #2414
2024-06-05 13:49:34 +02:00
Ivan Nardi 6a9d4b1ab6 CI: more parallel work (#2459) 2024-06-05 11:56:03 +02:00
Ivan Nardi 7be482f5b1 Update unit tests results (#2466) 2024-06-05 11:55:30 +02:00
Luca Deri 85587a8f98 Fixes accounting of number of processed packets in ndpi_internal_detection_process_packet() 2024-06-04 17:00:45 +02:00
Ivan Nardi 81e42b748e RTP: fix detection over TCP (#2462)
RFC4571 is not the only way to wrap RTP messages in TCP streams.
For example, when RTP is encapsulated over TURN flows (i.e. via DATA
attribute) there is no additional framing.
See also 6127e0490
2024-05-29 18:31:10 +02:00
Maatuq 6127e04900 support rtp/rtcp over tcp (#2422) (#2457)
Support rtp/rtcp over tcp as per rfc4571.

Signed-off-by: mmaatuq <mahmoudmatook.mm@gmail.com>
2024-05-28 22:01:08 +02:00
Toni 80171dbcf3 Add ZUG consensus protocol dissector. (#2458)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-05-28 20:29:48 +02:00
Ivan Nardi 25f8964a23 CiscoVPN: we detect it only over UDP (#2454)
The original code handled also TCP/TLS, but it was removed in 6fc29b3ae
2024-05-28 14:07:48 +02:00
ToniandNardi Ivan abce6d4023 Improved Kafka dissector. (#2456)
* detect more Kafka request packet's
 * requires less flow memory
 * same detection behavior as before e.g. no asym detection implemented
   (can be done by dissecting responses, requires more effort)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Co-authored-by: Nardi Ivan <nardi.ivan@gmail.com>
2024-05-27 09:46:32 +02:00
Vladimir Gavrilov d1a59c0308 Rename Messenger to FacebookMessenger (#2453) 2024-05-23 09:23:31 +02:00
Ivan Nardi 41050af5b5 fuzz: fix build of oss-introspector (#2452) 2024-05-22 19:41:59 +02:00
Nardi Ivan 521ce609c8 Sync unit tests results 2024-05-22 19:00:33 +02:00
Luca 44a290286b More NDPI_PROBING_ATTEMPT changes 2024-05-22 18:04:33 +02:00
Luca 74d3843ebe Added NDPI_PROBING_ATTEMPT risk 2024-05-22 17:55:16 +02:00
Vladimir Gavrilov 15643547fe Replace ndpi_strnstr() implementation with an optimal one (#2447) 2024-05-22 12:47:27 +02:00
Luca Deri 5a25f89ab3 Enlarged buffer 2024-05-22 11:00:01 +02:00
Ivan Nardi fd02baa13a DTLS: fix JA4 fingerprint (#2446) 2024-05-21 18:13:25 +02:00
Toni 3639d2045b Remove unused code. (#2450)
* some `#ifdef`ed code dates back to 2019, 2020 and 2021
 * some function signatures were still present in `ndpi_main.h`
   which may cause linker errors for libnDPI dependee's
 * return an error while trying to serialize a double instead
   of `fprintf(stderr, ...)`

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-05-21 13:08:07 +02:00
Ivan Nardi 83840f1bb9 DTLS: add support for DTLS 1.3 (#2445) 2024-05-21 12:49:27 +02:00
Ivan Nardi 0109014f2c Follow-up of 2093ac5bf (#2451) 2024-05-21 12:47:25 +02:00
Ivan Nardi bc8bc69beb CI: enable parallel tests (for x86_64, at least) (#2444)
TODO: enable parallel tests when using docker with no-x86_64 archs.

When I tried the obviuos solutions:
```
NDPI_FORCE_PARALLEL_UTESTS=1 NDPI_SKIP_PARALLEL_BAR=1 make check VERBOSE=1
```
I got:
```
Run configuration "caches_cfg" [--cfg=lru.ookla.size,0 --cfg=lru.msteams.ttl,1]
ookla.pcap	/bin/sh: 1: run_single_pcap: not found
teams.pcap	/bin/sh: 1: run_single_pcap: not found
Run configuration "caches_global" [--cfg=lru.ookla.scope,1 --cfg=lru.bittorrent.scope,1 --cfg=lru.stun.scope,1 --cfg=lru.tls_cert.scope,1 --cfg=lru.mining.scope,1 --cfg=lru.msteams.scope,1 --cfg=lru.stun_zoom.scope,1]
bittorrent.pcap	/bin/sh: 1: run_single_pcap: not found
lru_ipv6_caches.pcapng	/bin/sh: 1: run_single_pcap: not found
mining.pcapng	/bin/sh: 1: run_single_pcap: not found
...
```
2024-05-20 19:07:16 +02:00
Ivan Nardi 399be12585 Small fixes after API cleanup done in c63446e59 (#2449) 2024-05-20 19:06:24 +02:00
Luca Deri 2093ac5bf6 Minor dissector optimizations 2024-05-20 12:17:04 +02:00
Luca Deri 42dba2e4af Added dpi.compute_entropy configuration parameter 2024-05-18 09:46:15 +02:00
Luca Deri c63446e592 Cleaned up API
Removed
-  int ndpi_load_ipv4_ptree_file(ndpi_ptree_t *tree, const char *path, u_int16_t protocol_id);
-  int ndpi_load_ipv6_ptree_file(ndpi_ptree_t *tree, const char *path, u_int16_t protocol_id);

Added (it supports both IPv4 and v6)
+  int ndpi_load_ptree_file(ndpi_ptree_t *tree, const char *path, u_int16_t protocol_id);
2024-05-17 16:38:02 +02:00
Ivan Nardi 33d3d254ac CI: add support for ubuntu-24.04 runners (#2443) 2024-05-16 17:05:13 +02:00
Vladimir Gavrilov 3d1da00d8d Add Call of Duty Mobile support (#2438) 2024-05-15 12:46:02 +02:00
Ivan Nardi d6b1c24079 Parallel execution of unit tests (#2435)
Running unit tests is quite a bottleneck while developing or while
waiting for GitHub CI results...
Try to run the tests in parallel, using the `parallel` tool.

By default, tests still run one after the other, as usual; to enable
parallel execution you need `NDPI_FORCE_PARALLEL_UTESTS=1 ./tests/do.sh`

Please note that the output is quite different in parallel mode!

A big part of the script has been rewritten to avoid code dupication
between "serial" and "parallel" path

On my notebook:
```
ivan@ivan-Latitude-E6540:~/svnrepos/nDPI(parallel)$ time ./tests/do.sh
[...]
real	3m12,684s
[...]
ivan@ivan-Latitude-E6540:~/svnrepos/nDPI(parallel)$ time NDPI_FORCE_PARALLEL_UTESTS=1 ./tests/do.sh
[...]
real	0m58,463s
```
2024-05-15 08:17:12 +02:00
Ivan Nardi 0110623b4e H323: improve detection and avoid false positives (#2432) 2024-05-11 23:39:54 +02:00
Ivan Nardi a064261e85 Revert ndpi_strnstr() optimization introduced in a813121e0 (#2439)
New implementation fails tests 11b, 12 and 13.
Revert to the original (BSD) implementation (with also some basic
parameters check)
2024-05-11 23:37:31 +02:00
Vladimir Gavrilov 2d33431948 Add Ethernet Global Data support (#2437) 2024-05-11 16:49:16 +02:00
Ivan Nardi 8c5e6e9761 IRC: simplify detection (#2423)
IRC has its best times well behind, but there are still some servers
using it.

We should try to simplify the detection logic, still based on OpenDPI
logic.

Let's start with some easy changes:
* try to detect TLS connection via standard hostname/SNI matching,
removing an old heuristic (we have never had any trace matching it);
* add some basic server names;
* once we detect that the flow is IRC, we don't have to perform
anything else;
* remove HTTP stuff; real HTTP flows never trigger that data path
* use `ndpi_memmem()` when possible
2024-05-11 14:12:13 +02:00
Vladimir GavrilovandIvan Nardi 70c4920184 Remove Vevo support (#2436)
Co-authored-by: Ivan Nardi <12729895+IvanNardi@users.noreply.github.com>
2024-05-11 11:57:53 +02:00
Ivan Nardi b116456fc5 Viber: add detection of voip calls and avoid false positives (#2434) 2024-05-11 09:21:13 +02:00
Vladimir Gavrilov a813121e0a ndpi_strnstr() optimization (#2433) 2024-05-10 22:43:59 +02:00
Ivan Nardi 4b4b358562 Line: use common helper to detect RTP/RTCP packets (#2429)
Add an explicit upper limit on the number of packets processed before
giving up.
2024-05-10 19:23:50 +02:00
Luca Deri 6bbae9a70a Update README.md 2024-05-10 08:53:19 +02:00
Ivan Nardi d4650f0f81 Raknet/RTP: avoid Raknet false positives and harden RTP heuristic (#2427)
There is some overlap between RTP and Raknet detection: give precedence
to RTP logic.
Consequences:
* Raknet might require a little bit more packets for some flows (not a
big issue)
* some very small (1-2 pkts) Raknet flows are not classified (not sure
what do do about that..)
2024-05-09 21:18:18 +02:00
Luca Deri e4fd6e47a3 Added #ifdef for Windows builds (#2431) 2024-05-09 17:27:25 +02:00
Ivan Nardi 2e7fc82e55 Protobuf: fix false positives (#2428) 2024-05-09 15:38:06 +02:00
Toni 18e03a26ca Add extra entropy checks and more precise(?) analysis. (#2383)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-05-09 15:24:11 +02:00
Ivan Nardi b65a755e85 Fix a memory access error and some leaks (#2425)
```
SCARINESS: 12 (1-byte-read-heap-buffer-overflow)
    #0 0x557f3a5b5100 in ndpi_get_host_domain /src/ndpi/src/lib/ndpi_domains.c:158:8
    #1 0x557f3a59b561 in ndpi_check_dga_name /src/ndpi/src/lib/ndpi_main.c:10412:17
    #2 0x557f3a51163a in process_chlo /src/ndpi/src/lib/protocols/quic.c:1467:7
    #3 0x557f3a469f4b in LLVMFuzzerTestOneInput /src/ndpi/fuzz/fuzz_quic_get_crypto_data.c:44:7
    #4 0x557f3a46abc8 in NaloFuzzerTestOneInput (/out/fuzz_quic_get_crypto_data+0x4cfbc8)
```

Some notes about the leak: if the insertion into the uthash fails (because of an
allocation failure), we need to free the just allocated entry. But the only
way to check if the `HASH_ADD_*` failed, is to perform a new lookup: a bit
costly, but we don't use that code in the fast-path.
See also efb261a95c

Credits for finding the issues to Philippe Antoine (@catenacyber) and his
`nallocfuzz` fuzzing engine
See: https://github.com/catenacyber/nallocfuzz
See: https://github.com/google/oss-fuzz/pull/9902
2024-05-08 11:46:46 +02:00
Ivan Nardi 7c6910d9e5 Fix/improve fuzzing (#2426) 2024-05-08 11:46:02 +02:00
Toni e9dc035c5c Added optimized memmem/strlcpy version (#2424)
* credits goes to Vladimir Gavrilov

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-05-08 11:38:53 +02:00
Luca Deri 2b4e2f9c9a Removed trace 2024-05-07 23:42:32 +02:00
Luca Deri 5f8ad0527d Reworked code and added two new API calls
- int ndpi_load_ipv4_ptree_file(ndpi_ptree_t *tree, const char *path, u_int16_t protocol_id);
- int ndpi_load_ipv6_ptree_file(ndpi_ptree_t *tree, const char *path, u_int16_t protocol_id);
2024-05-07 23:05:31 +02:00
Luca Deri 3d2b611ea6 ndpi_mod is now optional (albeit better to spcify it) in ndpi_domain_classify_xxx calls 2024-05-07 13:09:00 +02:00
Luca Deri 0d495294bf Win warning fixes 2024-05-07 10:04:11 +02:00
Luca Deri 6195dbad76 Renamed radius source file to avoid name clashes on Windows 2024-05-07 09:05:15 +02:00
Ivan Nardi 1773d7ff2d Add support for Mastodon, Bluesky and (FB-)Threads (#2418) 2024-05-06 13:37:18 +02:00
Ivan Nardi 95fe21015d Remove "zoom" cache (#2420)
This cache was added in b6b4967aa, when there was no real Zoom support.
With 63f349319, a proper identification of multimedia stream has been
added, making this cache quite useless: any improvements on Zoom
classification should be properly done in Zoom dissector.

Tested for some months with a few 10Gbits links of residential traffic: the
cache pretty much never returned a valid hit.
2024-05-06 12:51:45 +02:00
Ivan Nardi b7d77972bc Remove workaround for TCP flows with multiple SYNs (#2421)
Deciding when a session starts and ends is responsability of the
applicationi (via its flow manager)i, not of the library.
BTW, the removed code is incomplete at beast
2024-05-06 10:20:40 +02:00
Ivan Nardi e31ef00715 TLS: avoid setting NDPI_TLS_SELFSIGNED_CERTIFICATE for webrtc traffic (#2417)
See RFC8122: it is quite likely that STUN/DTLS/SRTP flows use
self-signed certificates

Follow-up of b287d6ec8
2024-05-06 10:20:07 +02:00
Ivan Nardi 266af02752 Merge RTP and RTCP logic (#2416)
Avoid code duplication between these two protocols.

We remove support for RTCP over TCP; it is quite rare to find this kind
of traffic and, more important, we have never had support for RTP
over TCP: we should try to add both detecion as follow-up.

Fix a message log in the LINE code
2024-05-06 10:19:46 +02:00
Ivan Nardi a1845da359 TLS: fix Ja4 fingerprint computation (#2419)
The new values has been checked against the ones reported by Wireshark.

Found while fixing a Use-of-uninitialized-value error reported by
oss-fuzz

```
==7582==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x5a6549abc368 in ndpi_compute_ja4 ndpi/src/lib/protocols/tls.c:1762:10
    #1 0x5a6549ab88a0 in processClientServerHello ndpi/src/lib/protocols/tls.c:2863:10
    #2 0x5a6549ac1452 in processTLSBlock ndpi/src/lib/protocols/tls.c:909:5
    #3 0x5a6549abf588 in ndpi_search_tls_tcp ndpi/src/lib/protocols/tls.c:1098:2
    #4 0x5a65499c53ec in check_ndpi_detection_func ndpi/src/lib/ndpi_main.c:7215:6
```

See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=68449&q=ndpi&can=1&sort=-id
2024-05-05 17:27:29 +02:00
Ivan Nardi a6fd981fcf eDonkey: improve/update classification (#2410)
eDonkey is definitely not as used as >10 years ago, but it seems it is
still active.

While having a basic TCP support seems easy, identification over UDP doesn't
work and it is hard to do it rightly (packets might be only 2 bytes long):
remove it.

Credits to V.G <v.gavrilov@securitycode.ru>
2024-05-04 19:11:31 +02:00
Luca Deri 57ecbf38c0 Updated JA4 test results 2024-05-02 17:40:24 +02:00
Luca Deri 66133d1501 Fixes JA4 computation adding a better GREASE detect funzion 2024-05-02 16:58:44 +02:00
Luca Deri af955169e1 typo 2024-05-02 16:03:30 +02:00
Ivan Nardi 9e549ba781 Workaround for fixing GitHub runners on macOS (#2411)
GitHub switched "macos-latest" from "macos-12" to "macos-14", which is only
on ARM64!
https://github.com/actions/runner/issues/3256
https://github.blog/changelog/2024-01-30-github-actions-macos-14-sonoma-is-now-available/

However we are having some issues build nDPI on macos-14 with external
libraries:

```
configure: error: libgpg-error required (because of --with-local-libgcrypt) but not found or too old.
```
See: https://github.com/ntop/nDPI/actions/runs/8869020568/job/24350356867

```
ndpi_utils.c:69:10: fatal error: 'pcre2.h' file not found
         ^~~~~~~~~
1 error generated.
```
See: https://github.com/ntop/nDPI/actions/runs/8869020568/job/24349242251

Everything is still fine with macos-14 and no external dependencies

As workaround, test only macos-12 and macos-13 in our main matrix.
2024-04-28 22:17:39 +02:00
Ivan Nardi d5bda47efe DTLS: add support for Alert message type (similar to TLS) (#2406) 2024-04-25 11:23:05 +02:00
0x41CEA55 7040847eed Add Adobe Connect support (#2407) 2024-04-24 12:41:25 +02:00
0x41CEA55 66036a14f4 Remove PPStream protocol and add iQIYI (#2403)
P2P video player PPStream was discontinued shortly after the purchase of PPS.tv by Baidu (iQIYI) on 2013 (see https://www.techinasia.com/report-baidu-acquires-video-rival-pps)
So we remove the old `NDPI_PROTOCOL_PPSTREAM` logic and add `NDPI_PROTOCOL_IQIYI` id to handle all the iQIYI traffic, which is basically video streaming traffic.

A video hosting service, called PPS.tv, is still offered by the same company: for the time being we classified both services with the same protocol id.
2024-04-23 18:01:36 +02:00
0x41CEA55 fd388845d5 Add BFCP protocol support (#2401) 2024-04-23 15:35:19 +02:00
Ivan Nardi abc7e430e2 STUN: slightly faster sub-classification with DTLS (#2404) 2024-04-23 09:31:48 +02:00
Ivan Nardi a62679952c fuzz: fix shoco fuzzer (#2405)
```
==22779==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f0900701020 at pc 0x555bcd2a6f02 bp 0x7ffe3ba5e790 sp 0x7ffe3ba5e788
READ of size 1 at 0x7f0900701020 thread T0
    #0 0x555bcd2a6f01 in shoco_decompress /home/ivan/svnrepos/nDPI/src/lib/third_party/src/shoco.c:184:26
    #1 0x555bcd2a4018 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco.cpp:18:3
    #2 0x555bcd1aa816 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco+0x4f816) (BuildId: c54d1c32163c9937e06f62127348ae6bd26d9309)
    #3 0x555bcd193be8 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco+0x38be8) (BuildId: c54d1c32163c9937e06f62127348ae6bd26d9309)
    #4 0x555bcd1996fa in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco+0x3e6fa) (BuildId: c54d1c32163c9937e06f62127348ae6bd26d9309)
    #5 0x555bcd1c3c92 in main (/home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco+0x68c92) (BuildId: c54d1c32163c9937e06f62127348ae6bd26d9309)
    #6 0x7f090257a082 in __libc_start_main /build/glibc-e2p3jK/glibc-2.31/csu/../csu/libc-start.c:308:16
    #7 0x555bcd18e96d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco+0x3396d) (BuildId: c54d1c32163c9937e06f62127348ae6bd26d9309)

Address 0x7f0900701020 is located in stack of thread T0 at offset 4128 in frame
    #0 0x555bcd2a3d97 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_alg_shoco.cpp:5

  This frame has 4 object(s):
    [32, 4128) 'out' (line 9) <== Memory access at offset 4128 overflows this variable
    [4256, 8352) 'orig' (line 9)
```

Found by oss-fuzzer.
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=68211
2024-04-22 18:39:23 +02:00
Nardi Ivan f494bdd653 Fix parameters check
```
==17==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x000000546050 bp 0x7fff113c82a0 sp 0x7fff113c7a58 T0)
==17==The signal is caused by a READ memory access.
==17==Hint: address points to the zero page.
SCARINESS: 10 (null-deref)
    #0 0x546050 in __sanitizer::internal_strlen(char const*) /src/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_libc.cpp:167:10
    #1 0x4c6ba5 in __interceptor_strrchr /src/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:740:5
    #2 0x5fb9b9 in ndpi_get_host_domain_suffix /src/ndpi/src/lib/ndpi_domains.c:105:20
    #3 0x578058 in LLVMFuzzerTestOneInput /src/ndpi/fuzz/fuzz_config.cpp:503:3
```

Found while fuzzing
2024-04-21 12:29:09 +02:00
0x41CEA55 246e972f9c Replace my personal email with my corporate one in all my contributions (#2399) 2024-04-20 21:37:01 +02:00
Nardi Ivan 95f4d0b852 Fix a warning
```
nDPI/PcDebug64/src/include/ndpi_api.h:1970:3: error: function declaration isn’t a prototype [-Werror=strict-prototypes]
```
2024-04-20 19:30:42 +02:00
Ivan Nardi ef89183469 fuzz: improvements (#2400)
Create the zip file with all the traces only once.

Add a new fuzzer to test "shoco" compression algorithm
2024-04-20 18:15:23 +02:00
0x41CEA55 905120588b Remove obsolete protocols: tuenty, tvuplayer and kontiki (#2398) 2024-04-19 21:35:32 +02:00
0x41CEA55 1b2e2cd968 Add strlcpy implementation (#2395) 2024-04-19 17:16:40 +02:00
0x41CEA55 e75d7a620e Add KNXnet/IP protocol support (#2397)
* Add KNXnet/IP protocol support

* Improve KNXnet/IP over TCP detection
2024-04-19 12:54:00 +02:00
Luca Deri ad117bfaab Domain Classification Improvements (#2396)
* Added
size_t ndpi_compress_str(const char * in, size_t len, char * out, size_t bufsize);
size_t ndpi_decompress_str(const char * in, size_t len, char * out, size_t bufsize);

used to compress short strings such as domain names. This code is based on
https://github.com/Ed-von-Schleck/shoco

* Major code rewrite for ndpi_hash and ndpi_domain_classify

* Improvements to make sure custom categories are loaded and enabled

* Fixed string encoding

* Extended SalesForce/Cloudflare domains list
2024-04-18 23:21:40 +02:00
Luca Deri 108b8331d5 Invalid initialization 2024-04-15 18:13:17 +02:00
Luca Deri 95897c6436 Fixed minor glitches 2024-04-15 14:25:26 +02:00
Ivan Nardi d97afd6efd STUN: fix attributes list iteration (#2391)
We need to check all the attributes, to look for any possible metadata
2024-04-13 18:43:41 +02:00
Ivan Nardi 06f4814085 STUN: try to stop extra dissection earlier, if possible (#2390) 2024-04-13 14:45:17 +02:00
Ivan Nardi 8edb2f133c STUN: add support for ipv6 in some metadata (#2389) 2024-04-13 14:12:20 +02:00
Ivan Nardi f51dfc6973 STUN: simplify ip/port parsing (#2388)
Add other 2 configuration options
2024-04-13 12:00:04 +02:00
Ivan Nardi 0535e54484 STUN: fix boundary checks on attribute list parsing (#2387)
Restore all unit tests.
Add some configuration knobs.
Fix the endianess.
2024-04-12 22:55:51 +02:00
Luca Deri bb4fccd2cf Added comment 2024-04-12 20:29:14 +02:00
Luca Deri b83eb7c7a2 Implemented STUN peer_address, relayed_address, response_origin, other_address parsing
Added code to ignore invalid STUN realm
Extended JSON output with STUN information
2024-04-12 19:50:04 +02:00
Toni Uhlig 31de94ba12 fix invalid read
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-04-12 17:55:48 +02:00
Vladimir Gavrilov c63697205b Add Label Distribution Protocol support (#2385)
* Add Label Distribution Protocol support

* Fix typo

* Update unit test results
2024-04-12 17:44:36 +02:00
Ivan Nardi 93f02ee507 Fix ndpi_reconcile_msteams_udp (#2377)
Microsoft UDP traffic over port ~3478 is voip traffic, using some kind
of proprietary STUN-like protocol: so use the most specific protocol id.

More important, we definitely want `Stun/Skype_TeamsCall` and not
`Stun/Skype_Teams`
2024-04-12 17:05:28 +02:00
Toni Uhlig 65fef09ff2 Updated unit test results
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-04-12 16:52:00 +02:00
Toni Uhlig 082b00ede7 Updated unit test results
* fixed invalid read

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-04-12 16:38:44 +02:00
Luca Deri a8507c3308 STUN:
- Fixed issue with XOR-MAPPED-ADDRESS decoding
- Implemented MAPPED_ADDRESS (IPv4 only)
2024-04-12 15:49:42 +02:00
Nardi Ivan 17965c8852 Update unit tests results 2024-04-11 18:36:43 +02:00
Luca Deri fd2838e9a1 Minor code cleanup 2024-04-11 17:44:44 +02:00
Toni 39a5d234fd Serialize amount of found TLS blocks and signature algos. (#2384)
* partially fixes #2282

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-04-11 16:21:41 +02:00
Luca Deri 8b4fdbc0fb Renamed ndpi_fill_ip6_protocol_category in ndpi_fill_ipv6_protocol_category 2024-04-11 12:37:39 +02:00
Vladimir Gavrilov 9ff4bece33 Add The Elder Scrolls Online support (#2376)
* Add The Elder Scrolls Online support

* Use ndpi_memmem instead of memmem from libc

* Add protocol description

* Change selection bitmask to V4_V6

* Update protocols.rst
2024-04-10 18:04:02 +02:00
Vladimir Gavrilov 142c8f5afb Add memmem() implementation (#2378)
* Add memmem() implementation

* Fix build

* Add fix to avoid too many memcmp calls
2024-04-10 15:39:08 +02:00
Toni 1d0be6c4f4 Add Shellscript risk detection. (#2375)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-04-10 15:36:54 +02:00
Ivan Nardi 54517d8e04 Fix a warning and restore a unit test result (#2379) 2024-04-10 09:56:06 +02:00
Luca Deri bd2968dcd3 Tuned DNS risk values
Modified NDPI_BINARY_TRANSFER_ATTEMPT in NDPI_BINARY_DATA_TRANSFER
2024-04-09 10:22:01 +02:00
Luca Deri 00a360f30a Disabled "known proto on non standard port" for FTP_DATA 2024-04-09 00:01:47 +02:00
Luca Deri 98bf0e243e Fixed false positives on binary application transfer risk 2024-04-08 23:52:27 +02:00
Luca Deri 48b0f8e2c1 Typo 2024-04-08 18:45:32 +02:00
Ivan Nardi 1b3ef7d7b2 STUN: improve extraction of Mapped-Address metadata (#2370)
Enable parsing of Mapped-Address attribute for all STUN flows: that
means that STUN classification might require more packets.

Add a configuration knob to enable/disable this feature.

Note that we can have (any) STUN metadata also for flows *not*
classified as STUN (because of DTLS).

Add support for ipv6.

Restore the correct extra dissection logic for Telegram flows.
2024-04-08 10:24:51 +02:00
Ivan Nardi f5905a62c7 Fix invalid memory access (#2374)
The bug is triggered when `pe_offset == (u_int32_t)-1`

```
AddressSanitizer:DEADLYSIGNAL
=================================================================
==23719==ERROR: AddressSanitizer: SEGV on unknown address 0x5081000002b3 (pc 0x55c69274ac72 bp 0x7ffffffc8e70 sp 0x7ffffffc8cc0 T0)
==23719==The signal is caused by a READ memory access.
    #0 0x55c69274ac72 in ndpi_search_portable_executable /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:8191:7
    #1 0x55c69271606b in ndpi_internal_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:8596:5
    #2 0x55c69270f58f in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:8629:22
    #3 0x55c6926a07e7 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:24:5
    #4 0x55c6925a79b6 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet+0x64e9b6) (BuildId: ec46c60ec7e03ebfb3d825bd6308d0a8d6e9803b)
    #5 0x55c692590d48 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet+0x637d48) (BuildId: ec46c60ec7e03ebfb3d825bd6308d0a8d6e9803b)
    #6 0x55c69259685a in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet+0x63d85a) (BuildId: ec46c60ec7e03ebfb3d825bd6308d0a8d6e9803b)
    #7 0x55c6925c0e02 in main (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet+0x667e02) (BuildId: ec46c60ec7e03ebfb3d825bd6308d0a8d6e9803b)
    #8 0x7f8e99793082 in __libc_start_main /build/glibc-wuryBv/glibc-2.31/csu/../csu/libc-start.c:308:16
    #9 0x55c69258baed in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet+0x632aed) (BuildId: ec46c60ec7e03ebfb3d825bd6308d0a8d6e9803b)
```

Found by oss-fuzzer
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=67881
2024-04-06 20:23:06 +02:00
Toni a5d45253c4 Add ELF risk detection (detect transmitted linux executables). (#2373)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-04-06 19:32:51 +02:00
Toni 727e72d1f1 Calculate packet entropy for unknown protocols. (#2369)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-04-06 17:01:19 +02:00
Vladimir Gavrilov b535033674 Get rid of Apache Kafka false positives (#2372) 2024-04-06 16:59:38 +02:00
Ivan Nardi 700637a162 fuzzing: extend fuzzing coverage (#2371) 2024-04-05 21:02:54 +02:00
Toni 0f77f49b77 Add PE32/PE32+ risk detection (detect transmitted windows executables). (#2312)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-04-05 14:43:28 +02:00
Toni 99e521eaf8 Limit git commit hash to 7 characters. (#2368)
* ensure consistency across all platforms

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-04-05 11:04:22 +02:00
Toni 09bb383437 Fixed yet another another "unused function" warning. (#2367)
* added `-Wextra` to the CI

```
In file included from ndpi_bitmap64_fuse.c:31:
./third_party/include/binaryfusefilter.h:31:24: error: unused function 'binary_fuse_rotl64' [-Werror,-Wunused-function]
static inline uint64_t binary_fuse_rotl64(uint64_t n, unsigned int c) {

..snip..
```

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-04-04 20:41:45 +02:00
RoboSchmied ead9aa72da Fix: 3 typos (#2366)
Signed-off-by: RoboSchmied <github@roboschmie.de>
2024-04-04 20:06:47 +02:00
Toni df29e12f5e Fixed yet another "unused function" warning. (#2365)
* seems like clang on `ubuntu-latest` warns about unused static inlined functions

```
In file included from ndpi_bitmap.c:41:
./third_party/include/roaring.h:422:19: error: unused function 'roaring_leading_zeroes' [-Werror,-Wunused-function]
static inline int roaring_leading_zeroes(unsigned long long input_num) {

..snip..
```

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-04-04 18:18:44 +02:00
Ivan Nardi c0d3f8a92e STUN: rework sub-classification (#2361)
The main goal is to have the "real" application (if any; i.e.
Signal/Whatsapp/Telegram/...) always as "application" protocol and not
as "master" one
2024-04-04 18:16:40 +02:00
Toni 6e61368cd6 Add -Wno-unused-function as CFLAG for third_party/src/gcrypt_light.c. (#2364)
* fixes failing nDPI build from an external project with clang and `-Wextra`

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-04-04 13:51:27 +02:00
Toni Uhlig dface63157 Fixed flow struct static assertion size check (this time with struct padding in mind).
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-04-04 12:40:09 +02:00
Toni Uhlig 6917acffda Fixed flow struct static assertion size check.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-04-04 12:36:42 +02:00
Luca Deri 9185c2ccc4 Added support for STUN Mapped IP address 2024-04-03 23:03:46 +02:00
Nardi Ivan aacbc9e918 Update unit tests results 2024-04-03 16:18:24 +02:00
Luca 225ff7f8a4 Added binary data transfer risk alert 2024-04-03 15:41:26 +02:00
Ivan Nardi 1acc9ee396 Patrica tree: fix handling of mac addresses (#2362)
Found looking at the fuzzer statistics...
2024-04-03 14:10:37 +02:00
Toni 41eef9246c Disable -Wno-unused-parameter -Wno-unused-function. (#2358)
* unused parameters and functions pollute the code and decrease readability

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-04-03 14:10:21 +02:00
Vitaly Lavrov b5e8bc2214 Fixed gcc version checking. (#2360)
Using the "test" utility instead of "[".
2024-04-02 10:58:55 +02:00
Vladimir Gavrilov 599cc0f4b8 Improve LoL: Wild Rift detection (#2359) 2024-03-28 16:01:01 +01:00
Vladimir Gavrilov 5b32c98a21 Add LoL: Wild Rift detection (#2356) 2024-03-26 08:11:14 +01:00
Luca Deri 51f5fc7140 Added support for roaring bitmap v3 (#2355)
* Integrated RoaringBitmap v3

* Renamed ndpi_bitmap64 ro ndpi_bitmap64_fuse

* Fixes to ndpi_bitmap for new roaring library

* Fixes for bitmap serialization

* Fixed format

* Warning fix

* Conversion fix

* Warning fix

* Added check for roaring v3 support

* Updated file name

* Updated path

* Uses clang-9 (instead of clang-7) for builds

* Fixed fuzz_ds_bitmap64_fuse

* Fixes nDPI printf handling

* Disabled printf

* Yet another printf fix

* Cleaup

* Fx for compiling on older platforms

* Fixes for old compilers

* Initialization changes

* Added compiler check

* Fixes for old compilers

* Inline function is not static inline

* Added missing include
2024-03-25 08:15:19 +01:00
Vladimir Gavrilov be0fd7444f Add more TencentGames signatures (#2354) 2024-03-21 14:38:47 +01:00
Vladimir Gavrilov 27f9ca9d04 Improve TencentGames detection (#2353)
* Improve TencentGames detection

* Add more signatures
2024-03-20 12:13:32 +01:00
Nardi Ivan 15a80527c6 STUN: remove workaround to identify RTP traffic
We are able to demultiplex RTP packets in STUN flows since 3608ab01b, at
least; no need to explicity call the RTP dissector
2024-03-20 09:39:15 +01:00
Nardi Ivan 5ccf41739b STUN: fix fallback when DTL dissection fails 2024-03-20 09:39:15 +01:00
Nardi Ivan f758f3605a STUN: dissection of TURN Channel Data 2024-03-20 09:39:15 +01:00
Nardi Ivan ee87c4bc66 STUN: add dissection of XOR-PEER-ADDRESS with ipv6 address 2024-03-20 09:39:15 +01:00
Nardi Ivan 58894dd788 STUN: improve heurstic to detect old classic-stun 2024-03-20 09:39:15 +01:00
Ivan Nardi dabf29ea61 STUN: update demultiplexing logic according to RFC9443 (#2349) 2024-03-19 09:11:51 +01:00
Vladimir Gavrilov e6474d835f Add FLUTE protocol dissector (#2351)
* Add FLUTE protocol dissector

* Add flute.c to MSVC project
2024-03-19 09:11:04 +01:00
Toni 15f61e7abe Enable USE_GLOBAL_CONTEXT via CFLAGS passed to the CC. (#2348)
* `ndpi_typedefs.h`: requires to include `ndpi_config.h` for the `HAVE_STRUCT_TIMESPEC` check
   That will never happen, because `USE_GLOBAL_CONTEXT` is defined inside `ndpi_config.h`.
   It's better to use `CFLAGS` to achieve the same.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-03-15 12:47:13 +01:00
Ivan Nardi 231748bb0e LRU cache: move to 64 bits long keys (#2346)
Tradeoff between key comparison efficiency (i.e. no `memcmp`) and key
length.
At least in the ipv4 cases, we have no more different entries with the
same key.
2024-03-15 10:12:51 +01:00
Ivan Nardi 97fae6e00a STUN: fix parsing of DATA attribute (#2345) 2024-03-14 14:20:51 +01:00
Ivan Nardi 37ddf02e63 bitmap64: fix memory access error (#2344)
```
AddressSanitizer:DEADLYSIGNAL
=================================================================
==29723==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x562910b70ddb bp 0x7ffcb22c5b70 sp 0x7ffcb22c5a80 T0)
==29723==The signal is caused by a READ memory access.
==29723==Hint: address points to the zero page.
    #0 0x562910b70ddb in binary_fuse16_contain /home/ivan/svnrepos/nDPI/src/lib/./third_party/include/binaryfusefilter.h:492:8
    #1 0x562910b70bbe in ndpi_bitmap64_isset /home/ivan/svnrepos/nDPI/src/lib/ndpi_bitmap64.c:178:10
    #2 0x562910788fd3 in ndpi_domain_classify_longest_prefix /home/ivan/svnrepos/nDPI/src/lib/ndpi_domain_classify.c:261:5
    #3 0x56291078940e in ndpi_domain_classify_contains /home/ivan/svnrepos/nDPI/src/lib/ndpi_domain_classify.c:291:9
    #4 0x56291069a392 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ds_domain_classify.cpp:52:5
```

Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=67369
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=67372
2024-03-14 14:20:35 +01:00
Ivan Nardi 25c145f26d CI actions: fix Ubuntu jobs with sanitizers (#2347)
See: https://github.com/actions/runner-images/issues/9491
2024-03-14 14:20:21 +01:00
Vladimir Gavrilov 8fad77991d Add PFCP protocol dissector (#2342) 2024-03-13 20:18:43 +01:00
Ivan Nardi 4102c8df72 CI: fix build on MacOS-13 runners (#2343)
Workaroud for Homebrew's python link error
See: https://github.com/Homebrew/homebrew-core/issues/165793#issuecomment-1991817938
2024-03-13 19:22:13 +01:00
Luca Deri 1aedbefffa Fixes bitmap memory calculation 2024-03-11 17:59:24 +01:00
Ivan Nardi 6152d595e8 STUN: add a parameter to configure how long the extra dissection lasts (#2336)
Tradeoff: performance (i.e. number of packets) vs sub-classification
2024-03-07 14:39:32 +01:00
Ivan Nardi 56ce228a8b Add a specific protocol id for audio/video calls made using Google apps (#2341)
Same logic already used for Signal/Whatsapp/Line/Facebook/...
2024-03-07 13:48:19 +01:00
Ivan Nardi c09b4dacdf Fix memory leak (#2340)
```
Direct leak of 12 byte(s) in 1 object(s) allocated from:
    #0 0x55779e1a46ff in malloc (/home/ivan/svnrepos/nDPI/example/ndpiReader+0x8706ff) (BuildId: 14c2fc626744710d49d652ea1c5bbb24a8cbab4f)
    #1 0x55779e2120c7 in ndpi_malloc_wrapper /home/ivan/svnrepos/nDPI/example/ndpiReader.c:298:10
    #2 0x55779e5fa215 in ndpi_malloc /home/ivan/svnrepos/nDPI/src/lib/ndpi_memory.c:60:25
    #3 0x55779e5fa500 in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_memory.c:113:13
    #4 0x55779e42153c in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2554:46
    #5 0x55779e4359a1 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:908:5
    #6 0x55779e432de7 in ndpi_search_tls_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1097:2
    #7 0x55779e4133f9 in ndpi_search_tls_wrapper /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2913:5
```

Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=67250
2024-03-07 13:11:13 +01:00
Toni 1fb4719e59 Disable AX_PTHREAD for MingW/MSYS builds. (#2338)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-03-07 13:09:01 +01:00
Vladimir Gavrilov e2949048e0 Add Path of Exile protocol dissector (#2337)
* Add Path of Exile protocol dissector

* Update protocols.rst
2024-03-06 19:59:09 +01:00
Ivan Nardi 21da53d3a0 ahocorasick: improve matching with subdomains (#2331)
The basic idea is to have the following logic:
* pattern "DOMAIN" matches the domain itself (i.e exact match) *and* any
subdomains (i.e. "ANYTHING.DOMAIN")
* pattern "DOMAIN." matches *also* any strings for which is a prefix
[please, note that this kind of match is handy but it is quite
dangerous...]
* pattern "-DOMAIN" matches *also* any strings for which is a postfix

Examples:
* pattern "wikipedia.it":
  * "wikipiedia.it" -> OK
  * "foo.wikipedia.it -> OK
  * "foowikipedia.it -> NO MATCH
  * "wikipedia.it.com -> NO MATCH
* pattern "wikipedia.":
  * "wikipedia.it" -> OK
  * "foo.wikipedia.it -> OK
  * "foowikipedia.it -> NO MATCH
  * "wikipedia.it.com -> OK
* pattern "-wikipedia.it":
  * "wikipedia.it" -> NO MATCH
  * "foo.wikipedia.it -> NO MATCH
  * "0001-wikipedia.it -> OK
  * "foo.0001-wikipedia.it -> OK

Bottom line:
* exact match
* prefix with "." (always, implicit)
* prefix with "-" (only if esplicitly set)
* postfix with "." (only if esplicitly set)

That means that the patterns cannot start with '.' anymore.

Close #2330
2024-03-06 19:25:59 +01:00
Luca Deri 8f63a11735 Extended connectivity checks 2024-03-05 17:45:46 +01:00
Luca Deri 65b9ad88a9 Improved alert on suspicious DNS traffic 2024-03-05 11:14:34 +01:00
Vladimir Gavrilov e7f1946ae7 Add NetEase Games detection support (#2335) 2024-03-05 11:00:52 +01:00
Vladimir Gavrilov 58fdc9fafb Add Naraka Bladepoint detection support (#2334) 2024-03-04 08:30:54 +01:00
Luca Deri 70aae6e998 Added DGA exception for Dropbox 2024-03-03 22:14:34 +01:00
Luca Deri e0587fe75e indent 2024-03-03 22:12:13 +01:00
Vladimir Gavrilov f2e3c7fb90 Add BFD protocol dissector (#2332) 2024-02-29 08:19:00 +01:00
Ivan Nardi c2b5b48fc8 ndpiReader: restore ndpiReader -x $DOMAIN_NAME functionality (#2329) 2024-02-26 20:50:07 +01:00
Nardi Ivan b287d6ec85 TLS: avoid setting some flow risks for webrtc traffic
Is quite rare to have a SNI or an ALPN on Client Hello of STUN/DTLS/SRTP
traffic
2024-02-26 10:20:05 +01:00
Nardi Ivan 478c31b6c9 utils: update script to download Cloudflare ips 2024-02-26 09:26:21 +01:00
Nardi Ivan ed5ba179f6 Telegram: improve identification
Follow up of 31c706c3db and
75485e177c.

Allow fast classification by ip, but give time to other dissectors to
kick in (for example, the TLS code for the Telegram Web flows).

Even if we don't classify it anymore at the very first packet (i.e. SYN)
we fully classify Telegram traffic at the first packet with payload, as
*any* other protocol.
This way, we always have the proper category, the proper confidence
for the UDP flows and we don't overwrite previous classifications (TLS
or ICMP)

Remove old and stale identification logic for TCP flows
2024-02-26 09:26:21 +01:00
Ivan Nardi c95e8c184e STUN: fix category when sub-classification is set in "extra-dissection" data path (#2320) 2024-02-24 09:44:24 +01:00
Luca Deri 71b7afccf5 Updated telegam out 2024-02-23 16:22:13 +01:00
Luca Deri 75485e177c Improved Telegram detection 2024-02-23 16:15:29 +01:00
Luca Deri bfb21dc09b Fixes exception handling glitch 2024-02-22 23:19:01 +01:00
Luca Deri 31c706c3db Improved telegram detection 2024-02-22 22:23:23 +01:00
Luca Deri 2399abd272 Added missing telegram networks 2024-02-22 22:04:54 +01:00
Vladimir Gavrilov 66b6e2b3f2 Add DLEP protocol dissector (#2326) 2024-02-20 16:05:41 +01:00
Ivan Nardi 4f2e642fbe make install: avoid copying private header (#2323) 2024-02-20 15:39:25 +01:00
Ivan Nardi a1280a2fd1 Move some defines (expecially log related) to the private header (#2324) 2024-02-20 15:35:38 +01:00
Ivan Nardi 12e142565e Add a script to download/update the domain suffix list (#2321) 2024-02-20 11:51:58 +01:00
Ivan Nardi 2f814c526b Add identification of Huawei generic and cloud traffic (#2325) 2024-02-20 11:51:40 +01:00
Ivan Nardi ffd6c3edee TLS: fix disabling of JA3C fingerprint (#2319) 2024-02-19 11:11:57 +01:00
Luca Deri 23536ff77a Improved modbus dissection to discard false positives 2024-02-16 15:48:34 +01:00
Luca Deri d1aed7b391 Indentation 2024-02-16 15:31:01 +01:00
Vladimir Gavrilov e93bcfd619 Add ANSI C12.22 protocol dissector (#2317)
* Add ANSI C12.22 protocol dissector

* Add UDP sample
2024-02-15 09:36:06 +01:00
Ivan Nardi ae36648c6c Skype: remove old detection logic (#1954)
Skype has been using standard protocols (STUN/ICE or TLS) for a long,
long time, now. Long gone are the days of Skype as a distribuited
protocol.

See: #2166
2024-02-12 18:39:24 +01:00
Nardi Ivan 0d36d9c8c2 Remove spurious call to exit() 2024-02-12 17:10:45 +01:00
Luca Deri 4072cb8862 Added stress test 2024-02-11 22:35:33 +01:00
Toni f4d7aa45fe Improved Polish gambling sites fetch script. (#2315)
* fails quite often in the CI, so ignore potential xmllint error

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-02-10 13:33:56 +01:00
Ivan Nardi ad25affcb7 reader_util: fix GRE detunneling (#2314) 2024-02-10 09:16:27 +01:00
Ivan Nardi 31f0608b9a TLS: add configuration of JA* fingerprints (#2313) 2024-02-10 09:16:13 +01:00
Ivan Nardi 03ecb026ff fuzz: improve fuzzing coverage (#2309) 2024-02-09 19:19:03 +01:00
Vladimir Gavrilov f5cec001f3 Add detection of Gaijin Entertainment games (#2311)
* Add detection of Gaijin Entertainment games

* Short NDPI_PROTOCOL_GAIJINENTERTAINMENT to NDPI_PROTOCOL_GAIJIN

* Add default UDP port for Gaijin Entertainment games

* Remove NDPI_PROTOCOL_CROSSOUT protocol id
2024-02-09 11:07:48 +01:00
Ivan Nardi f352e4f77b Improve normalization of flow->host_server_name (#2310)
Follow-up of 4543385d10

Remove trailing spaces for any HTTP header (we already remove leading
spaces)

We want:
* a "normalized" string in `flow->host_server_name`, but
* to parse the original string for flow risk checking

`ndpi_hostname_sni_set()` is a private function, so there is no need to
export its flags.
2024-02-09 09:20:02 +01:00
Vladimir Gavrilov 2abb68c8ca Add new AppsFlyer domain (#2307) 2024-02-08 08:10:55 +01:00
Vladimir Gavrilov 6207be43fa Add TencentGames protocol dissector (#2306) 2024-02-08 08:10:35 +01:00
Vitaly Lavrov 4543385d10 Normalization of host_server_name (#2299)
* Normalization of host_server_name

The ndpi_hostname_sni_set() function replaces all non-printable
characters with the "?" character and removing whitespace characters
at the end of the line.

* Added conditional hostname normalization.
2024-02-05 19:44:45 +01:00
Luca Deri a4da4467ac Fixed incompatibity with RH7 introduced by https://github.com/ntop/nDPI/commit/02030ac16e5016b00e0da6ff7512d97751bf87d2 2024-02-05 18:12:42 +01:00
Luca Deri b86452f2b1 Removed un-necessary include that caused configure-based applications sitting on top of nDPI to faile at compilation 2024-02-05 17:01:21 +01:00
Ivan Nardi 66371c202f Fix ndpi_get_lru_cache_stats() (#2303)
Found while fuzzing
2024-02-05 12:38:33 +01:00
Toni f2b09014f9 Serialize JA4 to JSON. (#2305)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-02-05 12:35:38 +01:00
Toni 02030ac16e Build RPM package in the CI. (#2304)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-02-05 12:13:09 +01:00
Luca Deri b15b877c51 Minor hash improvement 2024-02-04 20:23:49 +01:00
Toni b59994fa20 Completly disable all pthread related code in the library if USE_GLOBAL_CONTEXT macro is not defined. (#2302)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-02-03 11:53:49 +01:00
Ivan Nardi 9af2b9d815 collectd: fix extraction of hostname (#2301) 2024-02-03 04:11:58 +01:00
Toni ede25cc2b3 Improve ndpi_set_config error printing. (#2300)
* exit `ndpiReader` if a invalid configuration setting detected

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-02-02 15:15:30 +01:00
Toni f8aa642c3c Simplify and fix JA4 string computation. (#2298)
* additional JA4 string buffer is not needed and may cause a string truncation warning

```
protocols/tls.c: In function ‘ndpi_compute_ja4’:
protocols/tls.c:1738:3: warning: ‘strncpy’ output may be truncated copying 36 bytes from a string of length 1023 [-Wstringop-truncation]
 1738 |   strncpy(flow->protos.tls_quic.ja4_client, ja_str, 36);
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-02-02 14:25:43 +01:00
Ivan Nardi 13212711ab CIP: fix infinite-loop (#2295)
Found by oss-fuzzer
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=66342
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=66320
2024-02-01 20:30:43 +01:00
Ivan Nardi 8fbef7fb5a GitHub Actions: update to latest macOS runners (#2293)
Add a simple job with macos-14 on M1.
https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/)
There are some issues with external dependencies (they are installed but
autoconf script doens;t find them) so keep it simple.

On macos-13 it seems that:
* there is no `realpath` program (even if coreutils has been
installed...)
* most of the filesystem is read only (we can't write on /usr/lib).
So I change
```
make install DESTDIR=$(realpath _install)
ls -alhHR _install
```
to
```
DESTDIR=/tmp/ndpi make install
ls -alhHR /tmp/ndpi
```
for all the jobs

Fix a warning on GitHub logs:
```
Node.js 16 actions are deprecated. Please update the following actions
to use Node.js 20: actions/checkout@v3. For more information see:
https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.
```
2024-02-01 20:30:27 +01:00
Vladimir Gavrilov 4c9446379f Add Gearman protocol dissector (#2297) 2024-02-01 19:46:57 +01:00
Ivan Nardi 6c5ecae607 Fix function parameters check on domain code (#2296)
Found while fuzzing fuzz_config
2024-02-01 17:06:16 +01:00
Nardi Ivan c78ed93ec3 Sync unit tests results 2024-02-01 15:45:36 +01:00
Ivan Nardi 400cd516b5 Allow multiple struct ndpi_detection_module_struct to share some state (#2271)
Add the concept of "global context".

Right now every instance of `struct ndpi_detection_module_struct` (we
will call it "local context" in this description) is completely
independent from each other. This provide optimal performances in
multithreaded environment, where we pin each local context to a thread,
and each thread to a specific CPU core: we don't have any data shared
across the cores.

Each local context has, internally, also some information correlating
**different** flows; something like:
```
if flow1 (PeerA <-> Peer B) is PROTOCOL_X; then
  flow2 (PeerC <-> PeerD) will be PROTOCOL_Y
```
To get optimal classification results, both flow1 and flow2 must be
processed by the same local context. This is not an issue at all in the far
most common scenario where there is only one local context, but it might
be impractical in some more complex scenarios.

Create the concept of "global context": multiple local contexts can use
the same global context and share some data (structures) using it.
This way the data correlating multiple flows can be read/write from
different local contexts.
This is an optional feature, disabled by default.

Obviously data structures shared in a global context must be thread safe.
This PR updates the code of the LRU implementation to be, optionally,
thread safe.

Right now, only the LRU caches can be shared; the other main structures
(trees and automas) are basically read-only: there is little sense in
sharing them. Furthermore, these structures don't have any information
correlating multiple flows.

Every LRU cache can be shared, independently from the others, via
`ndpi_set_config(ndpi_struct, NULL, "lru.$CACHE_NAME.scope", "1")`.

It's up to the user to find the right trade-off between performances
(i.e. without shared data) and classification results (i.e. with some
shared data among the local contexts), depending on the specific traffic
patterns and on the algorithms used to balance the flows across the
threads/cores/local contexts.

Add some basic examples of library initialization in
`doc/library_initialization.md`.

This code needs libpthread as external dependency. It shouldn't be a big
issue; however a configure flag has been added to disable global context
support. A new CI job has been added to test it.

TODO: we should need to find a proper way to add some tests on
multithreaded enviroment... not an easy task...

*** API changes ***

If you are not interested in this feature, simply add a NULL parameter to
any `ndpi_init_detection_module()` calls.
2024-02-01 15:33:11 +01:00
Toni 44c2e59661 Provide a u64 wrapper for ndpi_set_config() (#2292)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-01-30 17:13:35 +01:00
Toni Uhlig d2f22d1308 Updated results (missing cip_io.pcap.out)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-01-29 20:49:47 +01:00
Luca Deri 47f72443fa Implemented CIP I/O (UDP version of the CIP protocol), Common Industrial protocol 2024-01-29 13:28:41 +01:00
Luca Deri 6c08601c3d Updated result 2024-01-29 12:27:52 +01:00
Luca Deri 426494b4ba Fixes invalid skype detection on UDP 2024-01-29 12:27:52 +01:00
Luca Deri 14cfa174be Added public_suffix_list.dat in RPM package 2024-01-29 11:48:39 +01:00
Luca Deri fb58f838e5 Added public_suffix_list.dat in packages 2024-01-29 11:43:23 +01:00
Luca Deri ae501e2503 Enhanced exception lists for binary application transfer 2024-01-29 11:37:04 +01:00
Ivan Nardi 92c2ac5a0f fuzz: fuzz_config: try restoring good coverage (#2291)
Last changes reduce fuzzing coverage of this fuzzer :(
2024-01-29 10:53:28 +01:00
Vladimir Gavrilov fb095a339d Add missing NDPI_LOG_DBG in some dissectors (#2290)
* Add missing NDPI_LOG_DBG

* Add missing NDPI_LOG_DBG in yojimbo.c
2024-01-29 09:25:29 +01:00
Vladimir Gavrilov c807d84054 Fix RESP detection (#2289)
* Rename redis_net.c to resp.c

* Fix RESP detection
2024-01-27 21:19:34 +01:00
Luca Deri 8e562c4ecd Removed check fir skipping .arpa domains 2024-01-27 20:54:46 +01:00
Luca Deri ec83a592b8 Check DGA on domain name (as it should be) rather than on the host name 2024-01-27 20:46:00 +01:00
Luca Deri 7a7df2b7d7 Fixed warning 2024-01-27 20:45:52 +01:00
Luca Deri 65b9c68d7d Fixed loading of non-ICANN domains that caused false positives with ndpi_load_domain_suffixes
Minor hash optimization
2024-01-27 20:40:27 +01:00
Toni bcca89b78c Add Raft protocol dissector. (#2286)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-01-25 20:26:18 +01:00
Vladimir Gavrilov 9a0affc6aa Improve MySQL detection (#2279)
* Improve MySQL detection

* Update copyright
2024-01-25 11:12:08 +01:00
Ivan Nardi b3efa7d3fe fuzz: fuzz_config: we need bigegr inputs (#2285) 2024-01-25 09:58:20 +01:00
Vladimir Gavrilov 4e712e3ab5 Add Radmin protocol dissector (#2283)
* Add Radmin protocol dissector

* Update test results
2024-01-25 08:10:29 +01:00
Ivan Nardi d577508727 fuzz: extend fuzzing coverage (#2281) 2024-01-24 21:16:58 +01:00
Ivan Nardi 7a83a8dc91 QUIC: fix decryption with CH fragments with different Destination CID (#2278)
QUIC decryption fails when the Client Hello is split into multiple UDP
packets and these packets have different Destination Connection IDs
(because the server told the client to switch to a different CID; see
RFC 9000 7.2)

```
The Destination Connection ID field from the first Initial packet sent by
a client is used to determine packet protection keys for Initial packets.
[..]
Upon first receiving an Initial or Retry packet from the server, the
client uses the Source Connection ID supplied by the server as the
Destination Connection ID for subsequent packets

```

From a logical point of view, the ciphers used for decryption should be
initialized only once, with the first Initial pkt sent by the client and
kept for later usage with the following packets (if any).

However it seems that we can safely initialize them at each packet, if
we keep using the DCID of the **first** packet sent by the client.
Keep initializing the ciphers at each packet greatly simplifie this patch.

This issue has been undetected for so long because:
* in the vast majority of the cases we only decrypt one packet per flow;
* the available traces with the Client Hello split into multiple packets
(i.e. cases where we need to decrypt at least two packets per flow) were
created in a simple test environment to simulate Post-Quantum handshake,
and in that scenario the client sent all the packets (with the same
DCID) before any reply from the server.

However, in the last months all major browsers started supporting PQ
key, so it is now common to have split CH in real traffic.

Please note that in the attached example, the CH is split into 2
(in-order) fragments (in different UDP packets) and the second one in
turn is divided into 9 (out-of-order) CRYPTO frames; the reassembler
code works out-of-the-box even in this (new) scenario.
2024-01-24 09:57:28 +01:00
Vladimir Gavrilov f04b4450a1 Add STOMP protocol dissector (#2280) 2024-01-23 21:08:13 +01:00
Ivan Nardi 9b26e74bb7 example: rework code between ndpiReader.c and reader_util.c (#2273) 2024-01-22 18:12:06 +01:00
Ivan Nardi 82e8bf91dd Improve handling of custom rules (#2276)
Avoid collisions between user-ids and internal-ids protocols in the
`example/protos.txt` file.
Add a new value for the classification confidence:
`NDPI_CONFIDENCE_CUSTOM_RULE`

With `./example/ndpiReader -p example/protos.txt -H` we now see also the
custom protocols and their internal/external ids:

```
nDPI supported protocols:
 Id Userd-id Protocol               Layer_4    Nw_Proto Breed        Category
  0        0 Unknown                TCP        X        Unrated      Unspecified

...

387      387 Mumble                 UDP        X        Fun          VoIP
388      388 iSCSI                  TCP                 Acceptable   Unspecified
389      389 Kibana                 TCP                 Acceptable   Unspecified
390      390 TestProto              TCP                 Acceptable   Unspecified
391      391 HomeRouter             TCP                 Acceptable   Unspecified
392      392 CustomProtocol         TCP                 Acceptable   Unspecified
393      393 AmazonPrime            TCP                 Acceptable   Unspecified
394      394 CustomProtocolA        TCP                 Acceptable   Unspecified
395      395 CustomProtocolB        TCP                 Acceptable   Unspecified
396      800 CustomProtocolC        TCP                 Acceptable   Unspecified
397     1024 CustomProtocolD        TCP                 Acceptable   Unspecified
398     2048 CustomProtocolE        TCP                 Acceptable   Unspecified
399     2049 CustomProtocolF        TCP                 Acceptable   Unspecified
400     2050 CustomProtocolG        TCP                 Acceptable   Unspecified
401    65535 CustomProtocolH        TCP                 Acceptable   Unspecified
```

We likely need to take a better look in general at the iteration between
internal and external protocols ids...

This PR fixes the issue observed in
https://github.com/ntop/nDPI/pull/2274#discussion_r1460674874 and in
https://github.com/ntop/nDPI/pull/2275.
2024-01-21 19:53:32 +01:00
Vladimir Gavrilov 5620e10742 Add ElectronicArts detection support (#2274)
* Add ElectronicArts detection support

* Merge electronicarts.pcapng into sites.pcapng
2024-01-21 18:58:12 +01:00
Toni 7d24e1258d Add Yojimbo (netcode) protocol dissector (#2277)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-01-21 17:47:42 +01:00
Ivan Nardi 49591dda9b Zoom: improve detection (#2270) 2024-01-21 12:17:34 +01:00
Ivan Nardi 42d23cff6a config: follow-up (#2268)
Some changes in the parameters names.
Add a fuzzer to fuzz the configuration file format.
Add the infrastructure to configuratin callbacks.
Add an helper to map LRU cache indexes to names.
2024-01-20 16:14:41 +01:00
Vladimir Gavrilov 8651ce9811 Fix detection of new Cassandra versions (#2272)
* Fix detection of new Cassandra versions

* Add Cassandra Internode Communication protocol support

* Add default port for Cassandra Internode Communication protocol
2024-01-20 15:58:14 +01:00
Ivan Nardi eb129297e9 Add a dedicated dissector for Zoom (#2265)
Move it from the RTP code and extend it
2024-01-19 10:01:38 +01:00
Ivan Nardi c83698c957 STUN: fix flow risks when DTLS packets are found (#2266)
When switching to (D)TLS dissector from the STUN one, we need to clear
any flow risks set from the latter (because we don't have anymore
`NDPI_PROTOCOL_STUN` in the classification results)
2024-01-19 08:57:39 +01:00
Vladimir Gavrilov 6ac2ce84f8 Add Mumble detection support (#2269) 2024-01-19 07:46:51 +01:00
Vladimir Gavrilov 248f3d5588 Rework Steam detection (part 1) (#2264)
* Clean up Steam dissector

* Add Steam Datagram Relay dissector

* Update docs

* Update test results

* Remove csgo.c from MSVC project

* Small fixes

* Add Steam TLS pcap sample

* Merge Steam pcap samples into single one

* Fix typo

* Update test results
2024-01-18 19:23:44 +01:00
Nardi Ivan 0712d496fe config: allow configuration of guessing algorithms 2024-01-18 10:21:24 +01:00
Nardi Ivan 6c85f10cd5 config: move debug/log configuration to the new API 2024-01-18 10:21:24 +01:00
Nardi Ivan c704be1a20 config: DNS: add two configuration options
* Enable/disable sub-classification of DNS flows
* Enable/disable processing of DNS responses
2024-01-18 10:21:24 +01:00
Nardi Ivan 950f209a17 config: HTTP: enable/disable processing of HTTP responses 2024-01-18 10:21:24 +01:00
Nardi Ivan c669044a44 config: configure TLS certificate expiration with the new API 2024-01-18 10:21:24 +01:00
Nardi Ivan 88720331ae config: remove enum ndpi_prefs 2024-01-18 10:21:24 +01:00
Nardi Ivan 1289951b32 config: remove ndpi_set_detection_preferences() 2024-01-18 10:21:24 +01:00
Nardi Ivan 311d8b6dae config: move cfg of aggressiviness and opportunistic TLS to the new API 2024-01-18 10:21:24 +01:00
Nardi Ivan 4cbe2674ab config: move IP lists configurations to the new API 2024-01-18 10:21:24 +01:00
Nardi Ivan f55358973f config: move LRU cache configurations to the new API 2024-01-18 10:21:24 +01:00
Nardi Ivan 3107a95881 Make ndpi_finalize_initialization() returns an error code
We should check if the initialization was fine or not
2024-01-18 10:21:24 +01:00
Nardi Ivan d72a760ac3 New API for library configuration
This is the first step into providing (more) configuration options in nDPI.

The idea is to have a simple way to configure (most of) nDPI: only one
function (`ndpi_set_config()`) to set any configuration parameters
(in the present or on in the future) and we try to keep this function
prototype as agnostic as possible.

You can configure the library:
* via API, using `ndpi_set_config()`
* via a configuration file, in a text format

This way, anytime we need to add a new configuration parameter:
* we don't need to add two public functions (a getter and a setter)
* we don't break API/ABI compatibility of the library; even changing
the parameter type (from integer to a list of integer, for example)
doesn't break the compatibility.

The complete list of configuration options is provided in
`doc/configuration_parameters.md`.

As a first example, two configuration knobs are provided:
* the ability to enable/disable the extraction of the sha1 fingerprint of
the TLS certificates.
* the upper limit on the number of packets per flow that will be subject
to inspection
2024-01-18 10:21:24 +01:00
Luca ca7df1db82 Improved ndpi_get_host_domain 2024-01-16 07:25:03 +01:00
Luca 1637a991a4 Added ndpi_get_host_domain() for returning the host domain
vs ndpi_get_host_domain_prefix() that instead returnd the host TLD
2024-01-16 06:56:51 +01:00
Ivan Nardi 111015b872 ndpiReader: improve the check on max number of pkts processed per flow (#2261)
Allow to disable this check.

I don't know how much sense these limits have in the application
(especially with those default values...) since we have always had a
hard limit on the library itself (`max_packets_to_process` set to 32).
The only value might be that they provide different limits for TCP and
UDP traffic.

Keep them for the time being...
2024-01-15 20:12:57 +01:00
Nardi Ivan 0891ad0e77 Added missing lists file to the distribution tarball 2024-01-15 20:00:36 +01:00
Nardi Ivan b22fa558ff ndpiReader: fix memory leak
Change the working directory of `ndpiReader` in the Github Actions so
that it can load the domain suffix list during `domainsUnitTest()`
2024-01-15 19:49:27 +01:00
Luca d29c2f3ad1 Added ndpi tracing 2024-01-15 19:26:28 +01:00
Luca a1634604e1 Removes extraneous parentheses that caused macOS to complain 2024-01-15 19:12:31 +01:00
Luca 162c38f18f Added new API calls
- ndpi_load_domain_suffixes()
- ndpi_get_host_domain_suffix()

whose goal is to find the domain name of a hostname. Example:

www.bbc.co.uk   -> co.uk
mail.apple.com  -> com
2024-01-15 19:03:46 +01:00
Ivan Nardi 61a18623e7 Bittorrent: improve detection of UTPv1 (#2259)
* fix header length check: we must take into account optional extension
header
* it seems the check on UTPv1 header is weak only in the DATA case: wait
for multiple packets only in that case
2024-01-12 18:53:24 +01:00
Ivan Nardi dd8be1fcb1 Fix some warnings reported by CODESonar (#2227)
Remove some unreached/duplicated code.

Add error checking for `atoi()` calls.

About `isdigit()` and similar functions. The warning reported is:
```
Negative Character Value help
isdigit() is invoked here with an argument of signed type char, but only
has defined behavior for int arguments that are either representable
as unsigned char or equal to the value of macro EOF(-1).
Casting the argument to unsigned char will avoid the undefined behavior.
In a number of libc implementations, isdigit() is implemented using lookup
tables (arrays): passing in a negative value can result in a read underrun.
```
Switching to our macros fix that.
Add a check to `check_symbols.sh` to avoid using the original functions
from libc.
2024-01-12 13:30:43 +01:00
Toni 0aea509e23 Add KCP protocol dissector. (#2257)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-01-12 12:14:59 +01:00
Vladimir Gavrilov 75d3f78f7d Add more tracker/ad hosts (#2256) 2024-01-12 10:44:31 +01:00
Toni 7b2bbb2309 Improved uTorrent via utp (TCP-like streams over UDP). (#2255)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-01-11 16:49:25 +01:00
Luca Deri 30c2386ee2 Added missing (C) 2024-01-09 22:27:41 +01:00
Luca Deri 4f9a08cf05 Moved ndpi_private.h to include directory 2024-01-09 22:00:40 +01:00
Vladimir Gavrilov 3d57dec6b4 Add PIA (Private Internet Access) support (#2250)
* Fix typo in protocols.rst

* Add PIA (Private Internet Access) support

* Update ndpi_main.c
2024-01-09 18:28:57 +01:00
Vladimir Gavrilov ead5bf2adb Add more adult content hostnames (#2247) 2024-01-09 07:57:01 +01:00
Toni c5b0b05b80 Add Roughtime protocol dissector. (#2248)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-01-09 07:56:27 +01:00
Toni 6c3d162cd6 Add realtime protocol output to ndpiReader. (#2197)
* support for using a new flow callback invoked before the flow memory is free'd
 * minor fixes

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-01-09 00:39:59 +01:00
Ivan Nardi 90b999e3f1 Fix default port conflicts between AJP and CiscoVPN (#2245)
We shouldn't have the same default port associated with multiple
protocols. The key reason is that we use only one of them (for
classification by-port, for example) and which one we really use
depends on the internal order of the protocolsi's initialization (which
is unknown to the user).

We have 3 port collisions:
* 8009: AJP and CiscoVPN
* 445: SMBv1 and SMBv23
* 8080: HTTP_Connect and HTTP_Proxy
I think that the two last cases don't really matter...

About the first one: it seems that CiscoVPN is retired from 2011/2014
(see
https://www.cisco.com/c/en/us/obsolete/security/cisco-vpn-client.html)
and superseded by AnyConnect, which uses different ports (see
https://community.cisco.com/t5/network-security/what-protocols-are-used-on-anyconnect-mobility-client/td-p/4713525).
Therefore it should be safe to remove 8009 from the CiscoVPN default
ports.
2024-01-08 17:04:46 +01:00
Ivan Nardi 8321b79539 Make some test traces smaller (#2243)
Having smaller traces help fuzzing: we want the fuzzers to mutate
"interesting" packets analyzed by nDPI, i.e. the first packets of each
flows.
Try hard to keep the same classification and extraction capabilities
2024-01-08 17:04:12 +01:00
Vladimir Gavrilov ce08291ccd Add Google Chat support (#2244) 2024-01-07 18:04:59 +01:00
Ivan Nardi 40797521af ndpiReader: add breed stats on output used for CI (#2236) 2024-01-05 13:02:39 +01:00
Vladimir Gavrilov 3d09b25653 Add Ceph protocol dissector (#2242)
* Add Ceph protocol dissector

* Update protocols.rst
2024-01-04 13:22:23 +01:00
Ivan Nardi f23e9dc7bb Add an implementation of the BSD function strtonum (#2238)
The main difference with the original function is that we allow to
specify the base.
Credit for the original idea and the first implementation to @0xA50C1A1
2024-01-04 13:16:39 +01:00
Vladimir Gavrilov 7f9973bd0c Add HL7 protocol dissector (#2240)
* Add HL7 protocol dissector

* Small fixes

* Small fixes
2024-01-02 20:57:05 +01:00
Ivan Nardi 88d1416b70 STUN: fix detection of Google Meet over IPv6 (#2241) 2024-01-02 19:30:59 +01:00
Vladimir GavrilovandNardi Ivan 0180c1f04a Add IEC62056 (DLMS/COSEM) protocol dissector (#2229)
* Add IEC62056 (DLMS/COSEM) protocol dissector

* Fix detection on big endian architectures

* Update protocols.rst

* Add ndpi_crc16_x25 to fuzz/fuzz_alg_crc32_md5.c

* Update pcap sample

* Remove empty .out file

* iec62056: add some documentation

---------

Co-authored-by: Nardi Ivan <nardi.ivan@gmail.com>
2024-01-02 16:45:54 +01:00
Ivan Nardi 3c7ed34ce9 fuzz: improve fuzzing coverage (#2239) 2024-01-02 15:22:44 +01:00
Vladimir Gavrilov 0f4d9f5054 Remove Google Hangouts/Duo stuff (#2233)
* Remove Google Hangouts/Duo support

* Update protocols.rst
2024-01-02 14:01:33 +01:00
Ivan Nardi 3dbd89eee8 sha256: fix undefined-shift error (#2237)
```
third_party/src/ndpi_sha256.c:51:21: runtime error: left shift of 128 by 24 places cannot be represented in type 'int'
```

Found by oss-fuzzer
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=65305
2024-01-02 12:29:20 +01:00
Ivan Nardi 308f71a6e8 TLS: fix heap-buffer-overflow (#2235)
```
==53992==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x511000007e04 at pc 0x555da2165fd0 bp 0x7ffddf7e3990 sp 0x7ffddf7e3988
READ of size 2 at 0x511000007e04 thread T0
    #0 0x555da2165fcf in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2384:50
    #1 0x555da217c31f in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:908:5
    #2 0x555da2176720 in ndpi_search_tls_udp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1273:11
    #3 0x555da215a628 in ndpi_search_tls_wrapper /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2883:5
    #4 0x555da1e95c30 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6720:6
    #5 0x555da1e969f3 in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6756:10
    #6 0x555da1e96394 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6789:12
    #7 0x555da1ea7991 in ndpi_internal_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7929:15
    #8 0x555da1ea547f in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:8104:22
    #9 0x555da1de137f in packet_processing /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:1721:31
    #10 0x555da1de137f in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:2438:1
```

Found by oss-fuzzer
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=65362
2024-01-02 11:37:40 +01:00
Ivan Nardi d886a6107f Teamviewer: varius fixes (#2228)
We already have a generic (and up to date) logic to handle ip addresses:
remove that stale list.

Teamviewer uses TCP and UDP, both; we can't access `flow->l4.udp`.

According to a comment, we set the flow risk
`NDPI_DESKTOP_OR_FILE_SHARING_SESSION` only for the UDP flows.
2024-01-02 11:22:43 +01:00
Vladimir Gavrilov 2796bc9b47 Add NoMachine NX protocol dissector (#2234)
* Add NoMachine protocol dissector

* Fix detection on big endian architectures

* Make NoMachine over UDP check more strict

* Small fixes
2024-01-02 10:23:42 +01:00
Luca 2f657cb8f9 Implemented ndpi_is_outlier() for detecting outliers using z-score 2023-12-28 19:59:54 +01:00
Luca Deri 1366518bff Implements ndpi_pearson_correlation for measuring how correlated are two series 2023-12-27 22:42:37 +01:00
Luca Deri 99d4838328 New ndpi_sha256() nDPI API call (#2230)
* Added ndpi_sha256.c to the Windows project

* Added ndpi_sha256() nDPI API call
2023-12-23 10:47:28 +01:00
Luca Deri f2be972224 Removed type redefinition 2023-12-23 09:16:35 +01:00
Luca Deri caa223d2ba Changes to avoid type redefinition on windows 2023-12-23 09:15:42 +01:00
Luca Deri 5c7200f2bb Buffer overflow fix 2023-12-23 08:07:28 +01:00
Luca Deri 7c4be6d077 Updated result 2023-12-22 21:29:45 +01:00
Luca Deri 53b9ae23e8 Warning fix 2023-12-22 21:10:03 +01:00
Luca Deri cbea1b2203 Fixed check (when using ./autogen.sh --enable-option-checking=fatal --enable-debug-messages --with-thread-sanitizer --with-pcre2 --with-maxminddb --enable-tls-sigs) 2023-12-22 21:01:14 +01:00
Luca Deri a95d4744b6 Fixes nDPI datastructs size check 2023-12-22 20:52:56 +01:00
Luca Deri 8285fffdae Implements JA4 Support (#2191) 2023-12-22 20:40:42 +01:00
Luca Deri b90c18e906 Changes to make room for JA4 2023-12-22 20:40:42 +01:00
Vladimir Gavrilov 5eb468d07b Add Apache Kafka protocol dissector (#2226) 2023-12-22 14:42:47 +01:00
Vladimir Gavrilov 6fc8aa4e61 Add WebDAV detection support (#2224)
* Add WebDAV detection support

* Add pcap example

* Update test results

* Remove redundant checks

* Add WebDAV related HTTP methods to fuzz/dictionary.dict

* Add note about WebDAV
2023-12-22 13:23:37 +01:00
Luca 41aa2f96b6 Various MDNS flow risks fixes 2023-12-21 12:24:30 +01:00
Luca 28b1ce41cb MDNS fix 2023-12-21 11:11:28 +01:00
Ivan Nardi a5595d16c0 CI: update list of compilers (#2223)
Try using latest gcc and clang versions.
We still care about RHEL7: since handling a RHEL7 runner on GitHub is
quite complex, let try to use a similar version of gcc, at least
2023-12-20 19:22:22 +01:00
Vladimir Gavrilov 149067b3fc Add JSON-RPC protocol dissector (#2217)
* Add JSON-RPC protocol dissector

* Small fixes

* Improve detection
2023-12-20 12:42:25 +01:00
Vladimir Gavrilov 33f11cb10f Add OpenFlow protocol dissector (#2222) 2023-12-20 10:48:45 +01:00
Ivan Nardi 8aa09f9c99 mining: a better identification logic (#2221)
It is quite simple (and not so efficient) but it should fix all the
false positives reported in #2216. Add support for Ethereum mining.

Merge all the mining traces.

Remove duplicated function.

Close #2216
2023-12-20 10:46:57 +01:00
Ivan Nardi 308b266333 fuzz: improve fuzzing coverage (#2220) 2023-12-19 20:33:08 +01:00
Vladimir Gavrilov 6af93c09c8 Rename NDPI_PROTOCOL_RPC to NDPI_PROTOCOL_MS_RPCH (#2218)
* Rename NDPI_PROTOCOL_RPC to NDPI_PROTOCOL_MS_RPCH

* Add protocol description

* Improve MS-RPCH detection
2023-12-19 11:56:26 +01:00
Vladimir Gavrilov 59c8eabc0e Add UFTP protocol dissector (#2215)
* Add UFTP protocol dissector

* Update docs

* Merge pcap files
2023-12-18 11:21:07 +01:00
Vladimir Gavrilov d8c7a76611 Add HiSLIP protocol dissector (#2214)
* Add HiSLIP protocol dissector

* Fix error
2023-12-17 11:52:55 +01:00
Vladimir Gavrilov 0f3e6d832b Add PROFINET/IO protocol dissector (#2213)
* Add PROFINET/IO protocol dissector

* Add LE (Little Endian) to the file name

* Rework dissector

* Remove redundant check
2023-12-16 13:30:21 +01:00
Ivan Nardi 8ea16ffd77 TLS: be sure to always set a category value (#2206)
We probably need to check everywhere we set the category in the
"extra dissection" data path: a flow with a proper classification can't
have NDPI_PROTOCOL_CATEGORY_UNSPECIFIED as category.
2023-12-14 08:07:04 +01:00
Toni ef62391dba Add Monero protocol classification. (#2196)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-12-13 19:55:18 +01:00
Ivan Nardi bda037c57f Sync unit tests results after latest merge (#2211) 2023-12-13 17:47:20 +01:00
Ivan Nardi 193f28582b QUIC: add heuristic to detect unidirectional *G*QUIC flows (#2207)
Fix extraction of `flow->protos.tls_quic.quic_version` metadata.
2023-12-13 17:14:04 +01:00
Ivan Nardi 8e14aac5e0 ndpiReader: avoid creating two detection modules when processing traffic/traces (#2209) 2023-12-12 19:44:29 +01:00
Ivan Nardi 241c42ad7e ndpiReader: fix guessed_flow_protocols statistic (#2203)
Increment the counter only if the flow has been guessed
2023-12-12 19:44:03 +01:00
Ivan Nardi 7b5354588b fuzz: extend fuzzing coverage (#2208) 2023-12-11 19:24:17 +01:00
Ivan Nardi b3f2b1bb7f STUN: rework extra dissection (#2202)
Keep looking for RTP packets but remove the monitoring concept.
We will re-introduce a more general concept of "flow in monitoring
state" later.
The function was disabled by default.
Some configuration knobs will be provided when/if #2190 is merged.
2023-12-11 14:53:12 +01:00
Ivan Nardi 673b6e7345 HTTP: faster processing of asymmetric flows (#2198) 2023-12-11 14:52:31 +01:00
Ivan Nardi adf8982d8e fuzz: extend fuzzing coverage (#2205) 2023-12-11 12:48:50 +01:00
rl1987 59d476195c Fix typos (#2204)
* Fix typo in ndpiSimpleIntegration.c

* Fix misspelling in a comment
2023-12-10 19:58:22 +01:00
Vladimir Gavrilov c809e7c069 Replace complicated TPKT header validation with an helper function (#2201)
* Replace complicated TPKT header validation with an helper function

* Move tpkt_verify_hdr function definition to ndpi_utils.c
2023-12-10 12:10:50 +01:00
Ivan Nardi f74cf16c36 OpenVPN: rework detection (#2199)
Close #1873
2023-12-06 10:24:26 +01:00
Vladimir Gavrilov ad20846fad Add Ether-S-Bus protocol dissector (#2200) 2023-12-05 17:20:38 +01:00
Vladimir Gavrilov 72601a69ff Add some fast CRC16 algorithms implementation (#2195)
* Add some fast CRC16 algorithms implementation

* Update ndpi_crc.c

* Move crc16 stuff to ndpi_analyze.c

* IEEE C37.118: use new fast CRC-16/CCITT-FALSE implementation
2023-12-05 15:58:11 +01:00
Vladimir Gavrilov be50493f44 Add IEEE C37.118 protocol dissector (#2193) 2023-12-05 08:06:15 +01:00
Ivan Nardi 7b0c16a70d TLS: remove JA3+ fingerprints. (#2192)
See: #2191
2023-12-05 08:05:44 +01:00
Vladimir Gavrilov c34bded4ef Add ISO 9506-1 MMS protocol dissector (#2189)
* Add ISO 9506-1 MMS protocol dissector
* Fix detection on big-endian architectures
2023-12-01 09:03:07 +01:00
Ivan Nardi 983b8e8eee STUN: parsing of DATA attribute (#2179) 2023-12-01 07:01:49 +01:00
Vladimir Gavrilov 24df1913ac Add Beckhoff ADS protocol dissector (#2181)
* Add Beckhoff ADS protocol dissector

* Remove redundant le32toh

* Fix detection on big-endian architectures
2023-11-30 09:13:45 +01:00
Ivan Nardi 6f046df0dc STUN: fix detection of DTLS (#2187)
Fix a memory leak
```
==97697==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 16 byte(s) in 1 object(s) allocated from:
    #0 0x55a6967cfa7e in malloc (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader+0x701a7e) (BuildId: c7124999fa1ccc54346fa7bd536d8eab88c3ea01)
    #1 0x55a696972ab5 in ndpi_malloc /home/ivan/svnrepos/nDPI/src/lib/ndpi_memory.c:60:25
    #2 0x55a696972da0 in ndpi_strdup /home/ivan/svnrepos/nDPI/src/lib/ndpi_memory.c:113:13
    #3 0x55a696b7658d in processClientServerHello /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2394:46
    #4 0x55a696b86e81 in processTLSBlock /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:897:5
    #5 0x55a696b80649 in ndpi_search_tls_udp /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1262:11
    #6 0x55a696b67a57 in ndpi_search_tls_wrapper /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:2751:5
    #7 0x55a696b67758 in switch_to_tls /home/ivan/svnrepos/nDPI/src/lib/protocols/tls.c:1408:3
    #8 0x55a696c47810 in stun_search_again /home/ivan/svnrepos/nDPI/src/lib/protocols/stun.c:422:4
    #9 0x55a6968a22af in ndpi_process_extra_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7247:9
    #10 0x55a6968acd6f in ndpi_internal_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7746:5
    #11 0x55a6968aba3f in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:8013:22
    #12 0x55a69683d30e in packet_processing /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:1723:31
    #13 0x55a69683d30e in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:2440:10
    #14 0x55a69680f08f in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:135:7
[...]
SUMMARY: AddressSanitizer: 16 byte(s) leaked in 1 allocation(s).
```
Found by oss-fuzzer
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=64564
2023-11-30 09:09:40 +01:00
Ivan Nardi d85f8592b8 Rework extraction of HTTP headers (#2183)
Extract only the headers that we really need/use.
Avoid too many `strcmp`; the algorithm might be a little bit
overwhelming right now but it might be useful if we have further headers
in the future.
2023-11-29 19:44:25 +01:00
Ivan Nardi 1100326edb Fix FAQ (#2186) 2023-11-29 18:56:21 +01:00
Ivan Nardi d3d9770b96 Update FAQ.md 2023-11-29 18:52:35 +01:00
Ivan Nardi cc721fd3ce Update FAQ.md 2023-11-29 18:28:26 +01:00
Ivan Nardi 0114992f5f Add a FAQ for the project (#2185) 2023-11-29 18:25:36 +01:00
Ivan Nardi 3b35cb37d9 Keep separating public and private API (#2157)
See: b08c787fe
2023-11-29 17:13:00 +01:00
Ivan Nardi ac90b1f009 Fix detection of NDPI_TCP_ISSUES flow risk (#2177)
We need to take into account retransmissions: they increase
`flow->all_packets_counter` counter but not `flows->packet_counter`
one.
Therefore, the right way to check for 3WH + RST pattern involves checking
for `flows->packet_counter == 0`
2023-11-29 16:55:39 +01:00
Toni 5fb631c8fe Improved belgium gambling sites regex. (#2184)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-29 16:15:23 +01:00
Vladimir Gavrilov c60c03766c Add Schneider Electric’s UMAS detection support (#2180)
* Add Schneider Electric’s UMAS detection support

* Swap proto IDs in ndpi_set_detected_protocol

* Update unit test result
2023-11-28 18:03:00 +01:00
Vladimir Gavrilov 688a197b85 Fix FINS false positives (#2176)
* Fix FINS false positives

* Add rsv (reserved) field check
2023-11-27 21:46:14 +01:00
Vladimir Gavrilov ebb1bc2f34 Add Ether-S-I/O protocol dissector (#2174) 2023-11-27 19:04:05 +01:00
Vladimir Gavrilov 84427b0754 Add Omron FINS protocol dissector (#2172)
* Add Omron FINS protocol dissector

* Add a kludge to avoid invalid FINS over UDP detection as SkypeTeams and RTP

* Update unit test results

* Update protocols.rst

* Remove dummy flows from fins.pcap
2023-11-27 17:09:53 +01:00
3763c702f0 Rework S7Comm dissector; add S7Comm Plus support (#2165)
* Rework S7Comm dissector; add S7Comm Plus support

* Cleanup s7comm.c

* Improve S7Comm Plus detection

* s7comm/s7commplus: faster detection

---------

Co-authored-by: 0xA50C1A1 <mage.wizard88@gmail.com>
Co-authored-by: Nardi Ivan <nardi.ivan@gmail.com>
2023-11-27 14:37:48 +01:00
Vladimir Gavrilov 0b6e261523 Improve CORBA detection (#2167)
* Improve CORBA detection

* Remove dummy flow from ziop.pcap

* Merge ziop.pcap and miop.pcap into corba.pcap
2023-11-27 13:10:50 +01:00
Vladimir Gavrilov da629709f3 Add OPC UA protocol dissector (#2169) 2023-11-27 12:13:23 +01:00
Ivan Nardi 7ff22a7e3c STUN: improve demultiplexing of DTLS packets (#2153)
Keep demultiplexing STUN/RTP/RTCP packets after DTLS ones.

We might end up processing the session a little longer, because we will
process the STUN/RTP/RTCP packets after the DTLS handshake.
2023-11-27 11:10:38 +01:00
Vladimir Gavrilov 87399b3544 Add RTPS protocol dissector (#2168) 2023-11-27 07:17:39 +01:00
Vladimir Gavrilovand0xA50C1A1 27802b0134 Reduce false positives for H.323 over TCP (#2164)
Co-authored-by: 0xA50C1A1 <mage.wizard88@gmail.com>
2023-11-23 17:29:00 +01:00
fbae51ae9d Get rid of RDP false positives (#2161)
* Get rid of false positives in the RDP protocol dissector

* Remove kludge for RDP

* RDP: improve detection

---------

Co-authored-by: 0xA50C1A1 <mage.wizard88@gmail.com>
Co-authored-by: Nardi Ivan <nardi.ivan@gmail.com>
2023-11-23 09:35:43 +01:00
Vladimir Gavrilovand0xA50C1A1 5c8c5c90c2 Add HART-IP protocol dissector (#2163)
* Add HART-IP protocol dissector

* Update docs

* Update protocols.rst

* Reuse free proto id and re-run tests

* docs: move HART-IP to top of list

---------

Co-authored-by: 0xA50C1A1 <mage.wizard88@gmail.com>
2023-11-22 22:04:22 +01:00
Toni 21f2574033 Improved TFTP. Fixes #2075. (#2149)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-21 16:56:46 +01:00
Vladimir Gavrilovand0xA50C1A1 35abafec4f Get rid of Apache Cassandra false positives (#2159)
* Rewrite Apache Cassandra dissector

* Replace memcmp with strncmp

* Add payload length check

* Update Cassandra dissector

* Update test results

---------

Co-authored-by: 0xA50C1A1 <mage.wizard88@gmail.com>
2023-11-21 16:56:01 +01:00
Vladimir Gavrilovand0xA50C1A1 ae6e6d61f0 Add IEEE 1588-2008 (PTPv2) dissector (#2156)
* Add IEEE 1588-2008 (PTPv2) dissector

PTPv2 is a time synchronization protocol in computer networks, similar to NTP.

* Add default protocol ports

* Update default test result for PTPv2

* Update copyright

---------

Co-authored-by: 0xA50C1A1 <mage.wizard88@gmail.com>
2023-11-21 13:39:54 +01:00
Ivan Nardi 55664392a9 Ip address list: aggregate Mullvad and Tor lists too (#2154)
Missing from bdb73db1a
See #2150
2023-11-21 08:28:32 +01:00
Vladimir Gavrilovand0xA50C1A1 8d71998670 Remove Google+ support (#2155)
* Remove Google+ support

Google+ was discontiued in 2019, so I think that its protocol id can be freed for reuse.

* Fix typo

* Update tests

---------

Co-authored-by: 0xA50C1A1 <mage.wizard88@gmail.com>
2023-11-21 08:14:20 +01:00
Ivan Nardi bdb73db1a4 IP lists: aggregate addresses wherever possible (#2152)
See #2150
2023-11-17 12:26:23 +01:00
Ivan Nardi 6c9571d9a9 Remove duplicate addreess list (#2151)
We are loading the same AS list as GOTO
See #2150
2023-11-16 20:08:06 +01:00
Maatuq 344ac69468 protocols/ethereum: enhance node discovery protocol detection. (#2145)
*  packet-type == 0x0 shouldn't be allowed.
* allow more packet-type e.g. ENRRequest, ENRResponse.

Signed-off-by: Mahmoud Maatuq <mahmoudmatook.mm@gmail.com>
2023-11-15 19:45:48 +01:00
Toni 38f9a74713 Added TeslaServices and improved TikTok host names. Fixes #2140. (#2144)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-10 11:25:10 +01:00
Toni 0cb6f4cb75 Fixed hash buffer size in ndpiSimpleIntegration. (#2143)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-10 10:23:37 +01:00
Ivan Nardi b08c787fe2 Have a clear distinction between public and private/internal API (#2137)
1) Public API/headers in `src/include/` [as it has always been]
2) Private API/headers in `src/lib/`

Try to keep the "ndpi_" prefix only for the public functions
2023-11-09 10:50:59 +01:00
Toni 0673da54b5 Fixed implicit u32 cast in ndpi_data_min() / ndpi_data_max(). (#2139)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-09 10:16:57 +01:00
Ivan Nardi b539b0d090 fuzz: improve coverage and remove dead code (#2135)
We are not able to remove custom rules: remove the empty stubs (which
originate from the original OpenDPI code).

`ndpi_guess_protocol_id()` is only called on the first packet of the
flow, so the bitmask `flow->excluded_protocol_bitmask` is always empty,
since we didn't call any dissectors yet.

Move another hash function to the dedicated source file.
2023-11-07 17:46:29 +01:00
Ivan Nardi 58a9e2d967 Remove struct ndpi_packet_struct from the public API (#2129)
Right now, the only instance of `struct ndpi_packet_struct` is embedded
into `struct ndpi_detection_module_struct`. Since the latter is a
private structure (because of `NDPI_LIB_COMPILATION` ) there is no way for
the application to get a pointer to `ndpi_struct->packet`.

Bottom line: the application can't use any API functions having `struct
ndpi_packet_struct *` as parameter. Remove them all (since they are
completly unused and unusable).

There are no public helper functions to initialize/populate/deinit a
`struct ndpi_packet_struct` object, so the application can't neither
create its own instance of this object.

Protect `struct ndpi_packet_struct` via the same define
`NDPI_LIB_COMPILATION`.
2023-11-07 15:30:56 +01:00
Luca Deri 0db12b1390 In case a pattern is redefined (for instance with protos.txt), the custom definition overwrites the previous one (so far it was ignored) 2023-11-02 20:46:14 +01:00
Toni 6dcecd73d3 Added malicious sites from the polish cert. (#2121)
* added handling of parsing errors

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-02 09:04:04 +01:00
Christian Marangi d5c9a16a48 Move from PCRE to PCRE2 (#2134)
Move from PCRE to PCRE2. PCRE is EOL and won't receive any security
updates anymore. Convert to PCRE2 by converting any function PCRE2 new
API.

Also update every entry in github workflows and README to point to the
new configure flag. (--with-pcre2)

Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
2023-11-01 14:16:49 +01:00
Ivan KapranovandIvan Kapranov 5a2666ce9e Fix proto_name and proto_id missmatch for Sina and SinaWeibo (#2131)
* minor fixes

fixed 'handle leak' in ndpi_load_malicious_sha1_file and removed the redundant comparison ndpi_search_eaq

* fix Stack overflow caused by invalid write in ndpi_automa_match_string_subprotocol

* fix compile errors

* fix

* Fix name missmatch for Sina and Sina Weibo

* fix

* add Sina Weibo to doc

* fix

* add Sina Weibo to doc

---------

Co-authored-by: Ivan Kapranov <i.kapranov@securitycode.ru>
2023-11-01 14:12:49 +01:00
Ivan Nardi cd59485f7e Custom rules: add missing check (#2132)
Fix an use-of-uninitialized-value error

```
==28646==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55a9b7a3dea2 in ndpi_patricia_lookup /home/ivan/svnrepos/nDPI/src/lib/third_party/src/ndpi_patricia.c:739:8
    #1 0x55a9b7442cac in add_to_ptree /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:2588:10
    #2 0x55a9b7469290 in ndpi_add_ip_risk_mask /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:4161:12
    #3 0x55a9b746cf14 in ndpi_handle_rule /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:4297:11
    #4 0x55a9b74842ae in ndpi_load_protocols_file_fd /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:4944:8
    #5 0x55a9b7424706 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_filecfg_protocols.c:38:3
[...]
```

Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=63754
2023-10-31 20:02:37 +01:00
Ivan Nardi 50006516b3 Custom rules: fixes for Windows (#2130)
Even after adding `WSAStartup()` call, the behavior with IPv6 addresses
was different on Windows; it is the same issue reported on:
https://stackoverflow.com/questions/66755681/getaddrinfo-c-on-windows-not-handling-ipv6-correctly-returning-error-code-1

Fix bitmasks.
2023-10-31 16:04:30 +01:00
Luca Deri 773e9b8170 Added pointer check 2023-10-31 15:48:06 +01:00
Luca Deri f49cedc1e8 Added missing freeaddrinfo() call 2023-10-30 22:58:17 +01:00
Luca Deri 5e9f10f820 Replaced gethostbyname2 with getaddrinfo (portable code) 2023-10-30 22:55:50 +01:00
Luca Deri c6af36990e Warning fixes 2023-10-30 13:10:28 +01:00
Ivan Nardi 42d24f8799 STUN: major code rework (#2116)
Try to have a faster classification, on first packet; use standard extra
dissection data path for sub-classification, metadata extraction and
monitoring.

STUN caches:
* use the proper confidence value
* lookup into the caches only once per flow, after having found a proper
STUN classification

Add identification of Telegram VoIP calls.
2023-10-30 10:28:19 +01:00
Ivan Nardi e399bd7e3d Custom rules: fix a stack overflow (#2128)
```
==19255==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f515bb3bf80 at pc 0x55796e01394a bp 0x7fff4fb5c050 sp 0x7fff4fb5b7e0
WRITE of size 58 at 0x7f515bb3bf80 thread T0
    #0 0x55796e013949 in scanf_common(void*, int, bool, char const*, __va_list_tag*) asan_interceptors.cpp.o
    #1 0x55796e0147df in __isoc99_sscanf (/home/ivan/svnrepos/nDPI/fuzz/fuzz_filecfg_protocols+0x77f7df) (BuildId: a88601afb2c538ead3968648f39b9aa4da53427c)
    #2 0x55796e0fc74a in ndpi_add_host_ip_subprotocol /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:2771:13
    #3 0x55796e0fb029 in ndpi_handle_rule /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:4411:16
    #4 0x55796e103738 in ndpi_load_protocols_file_fd /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:4901:8
    #5 0x55796e0ca96d in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_filecfg_protocols.c:38:3
    #6 0x55796dfd78e0 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_filecfg_protocols+0x7428e0) (BuildId: a88601afb2c538ead3968648f39b9aa4da53427c)
    #7 0x55796dfc0e93 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_filecfg_protocols+0x72be93) (BuildId: a88601afb2c538ead3968648f39b9aa4da53427c)
    #8 0x55796dfc6d96 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_filecfg_protocols+0x731d96) (BuildId: a88601afb2c538ead3968648f39b9aa4da53427c)
    #9 0x55796dff1672 in main (/home/ivan/svnrepos/nDPI/fuzz/fuzz_filecfg_protocols+0x75c672) (BuildId: a88601afb2c538ead3968648f39b9aa4da53427c)
    #10 0x7f515df19082 in __libc_start_main /build/glibc-BHL3KM/glibc-2.31/csu/../csu/libc-start.c:308:16
    #11 0x55796dfbbb0d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_filecfg_protocols+0x726b0d) (BuildId: a88601afb2c538ead3968648f39b9aa4da53427c)

Address 0x7f515bb3bf80 is located in stack of thread T0 at offset 128 in frame
    #0 0x55796e0fb977 in ndpi_add_host_ip_subprotocol /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:2703

  This frame has 4 object(s):
    [32, 36) 'pin' (line 2705)
    [48, 64) 'pin6' (line 2706)
    [80, 96) 'd' (line 2769)
    [112, 128) 'tail' (line 2770) <== Memory access at offset 128 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow asan_interceptors.cpp.o in scanf_common(void*, int, bool, char const*, __va_list_tag*)
Shadow bytes around the buggy address:

```
2023-10-30 09:21:57 +01:00
Luca Deri 8769a2282b Disbling AES-NI on FreeBSD as the feature checke does not work well
and it enables it by mistake on IntelB. Coreb" i3-3240 Processor
2023-10-30 00:05:25 +01:00
Luca Deri 9fb7a63576 FreeBSD compilation fix (missingcvg #include <netdb.h>) 2023-10-29 23:18:17 +01:00
Luca Deri 76829b413f Implements support for symbolic host names (#2123) 2023-10-29 22:54:45 +01:00
Luca Deri b8e0602bfd Added check 2023-10-29 22:54:45 +01:00
Ivan Nardi 2c2eaf8e79 Rename some functions with more useful/clear names (#2127) 2023-10-29 20:21:24 +01:00
Ivan Nardi 03fd155ae3 IPv6: add support for custom categories (#2126) 2023-10-29 12:56:44 +01:00
Ivan Nardi 32b50f5aa4 IPv6: add support for IPv6 risk exceptions (#2122) 2023-10-29 12:14:20 +01:00
Ivan Nardi c711251578 IPv6: add support for custom rules (#2120) 2023-10-29 11:26:35 +01:00
Ivan Nardi e8e4b9e8ff IPv6: add support for IPv6 risk tree (#2118)
Fix the script to download crawler addressess
2023-10-27 13:58:15 +02:00
Toni ed17f4d658 Improved Protobuf dissector. (#2119)
* tag extraction/validation was done wrong

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-27 10:19:47 +02:00
Ivan Nardi 8b07be4b9f Jabber: remove support for UDP (#2115)
Jabber/XMPP is only over TCP (even the name `ndpi_search_jabber_tcp`
suggests that...).

Bug introduced in 5266c726f
2023-10-26 20:16:27 +02:00
Ivan Nardi 611c3b66f0 ipv6: add support for ipv6 addresses lists (#2113) 2023-10-26 20:15:44 +02:00
Luca Deri 1832d247b3 Tool for creating bitcoing IP files 2023-10-26 00:01:44 +02:00
Toni 9509cd507a Fixed endian issue while DEBUG_PROTOBUF is enabled. (#2112)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-25 13:11:17 +02:00
Maatuq 4a8e7105b2 add ethereum protocol dissector. (#2111)
as explained here for bitcoin https://www.ntop.org/guides/nDPI/protocols.html#ndpi-protocol-bitcoin
the same is applicable for ethereum.
ethereum detection was removed from mining protocol and is now handled separately.

Signed-off-by: Mahmoud Maatuq <mahmoudmatook.mm@gmail.com>
2023-10-25 12:44:33 +02:00
Toni e70333de87 Added generic Google Protobuf dissector. (#2109)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-24 12:18:31 +02:00
Luca Deri 71d768d036 Improved detection as non DGA for hostnames belnging to a CDN (#2068) 2023-10-23 17:49:50 +02:00
Toni Uhlig a443bba0dd Add CAN over Ethernet dissector.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-23 13:45:56 +02:00
Toni Uhlig 25c54dd6d7 Improved CryNetwork protocol dissector.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-23 13:44:34 +02:00
Luca Deri 7122599287 Added CLA section 2023-10-23 12:05:56 +02:00
Luca Deri 23759c6d2d Moved to 4.9 2023-10-23 11:46:04 +02:00
1738 changed files with 253462 additions and 86570 deletions
+2
View File
@@ -1,3 +1,5 @@
mpile
# Codige Code Review ignore
# Prevent Codiga complaining about Sphinx config.py
+1 -1
View File
@@ -21,7 +21,7 @@ A clear and concise description of what happening.
* OS version: [e.g. 18.04]
* Architecture: [e.g. arm64]
* nDPI version or commit hash: [e.g. 4.0-stable or 937357e4bc55610f116f66d15a8e0fc1e260c02c].
* nDPI compilation flags used: if you are building from source [e.g. --with-pcre --with-local-libgcrypt].
* nDPI compilation flags used: if you are building from source [e.g. --with-pcre2 --with-local-libgcrypt].
* Attach the `config.log` file generated after `./configure` ran (if you are building from source).
## How to reproduce the reported bug
+3 -3
View File
@@ -15,10 +15,10 @@ jobs:
runs-on: windows-latest
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Add msbuild to PATH
uses: microsoft/setup-msbuild@v1.1
uses: microsoft/setup-msbuild@v2
- name: Download NuGet packages
run: msbuild windows/nDPI.sln -t:restore -p:RestorePackagesConfig=true
@@ -36,7 +36,7 @@ jobs:
run: msbuild windows/nDPI.sln -t:rebuild -property:Configuration=Debug-ndpiReader
- name: Upload artifacts
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: build-artifacts
path: windows/bin
+31
View File
@@ -0,0 +1,31 @@
name: RPM Build
on:
workflow_dispatch:
schedule:
# At the end of every day
- cron: '0 0 * * *'
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Install Ubuntu Prerequisites
run: |
sudo apt-get update
sudo apt-get install autoconf automake libtool pkg-config gettext libjson-c-dev flex bison libpcap-dev
- name: Configure nDPI
run: |
# fake CentOS 7 env
echo 'CentOS Linux release 7.9.2009 (Core)' | sudo tee /etc/centos-release
# symlink source directory for ndpi.spec
ln -sr . $HOME/nDPI
./autogen.sh
cd packages/rpm
./configure
- name: Build RPM package
run: |
make -C packages/rpm package
+105 -79
View File
@@ -14,13 +14,13 @@ jobs:
name: Python Bindings (ubuntu-latest)
runs-on: ubuntu-latest
env:
CFLAGS: -Werror
CFLAGS: -Wextra -Werror
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Python 3.9
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "3.9"
- name: Install Ubuntu Prerequisites
@@ -30,7 +30,7 @@ jobs:
- name: Build nDPI library
run: |
./autogen.sh
make
make -j
sudo make install
- name: Generate Python bindings
run: |
@@ -47,7 +47,7 @@ jobs:
name: Test Utils (ubuntu-latest)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install Ubuntu Prerequisites
@@ -67,17 +67,17 @@ jobs:
git diff-index --quiet HEAD -- || true
test:
name: ${{ matrix.os }} ${{ matrix.arch }} ${{ matrix.gcrypt }} ${{ matrix.compiler }} ${{ matrix.pcre }} ${{ matrix.maxminddb }} ${{ matrix.msan }} ${{ matrix.nBPF }} ${{matrix.lto_gold_linker}}
name: ${{ matrix.os }} ${{ matrix.arch }} ${{ matrix.gcrypt }} ${{ matrix.compiler }} ${{ matrix.pcre }} ${{ matrix.maxminddb }} ${{ matrix.msan }} ${{ matrix.nBPF }} ${{matrix.lto_gold_linker}} ${{matrix.global_context}}
runs-on: ${{ matrix.os }}
env:
CC: ${{ matrix.compiler }}
CFLAGS: -Werror
CFLAGS: -Wextra -Werror -DNDPI_EXTENDED_SANITY_CHECKS
strategy:
fail-fast: true
matrix:
# ubuntu-latest == ubuntu-22.04
# macOS-latest == macos-12
os: ["ubuntu-latest", "ubuntu-20.04", "macOS-latest", "macos-11", "windows-latest"]
# macOS-latest == macos-14 on **ARM64**
# There are some issues with external dependencies on macOS-latest. Disable it for the time being
os: ["ubuntu-20.04", "ubuntu-22.04", "ubuntu-24.04", "macOS-12", "macOS-13", "windows-latest"]
arch: ["x86_64"]
gcrypt: ["--with-local-libgcrypt", ""]
compiler: ["cc"]
@@ -88,48 +88,64 @@ jobs:
msan: [""]
nBPF: [""]
lto_gold_linker: [""]
global_context: [""] #Enable by default
include:
- compiler: "gcc-7" # "Oldest" gcc easily available
- compiler: "gcc-4.9" # "Oldest" gcc easily available. To simulate RHEL7
os: ubuntu-20.04
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre"
pcre: "--with-pcre2"
maxminddb: "--with-maxminddb"
msan: "--with-sanitizer"
nBPF: ""
- compiler: "gcc-12" # "Newest" gcc easily available
os: ubuntu-22.04
- compiler: "gcc-14" # "Newest" gcc easily available
os: ubuntu-24.04
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre"
pcre: "--with-pcre2"
maxminddb: "--with-maxminddb"
msan: "--with-sanitizer"
nBPF: ""
lto_gold_linker: "--with-lto-and-gold-linker"
- compiler: "clang-7" # "Oldest" clang easily available
- compiler: "clang-9" # "Oldest" clang easily available
os: ubuntu-20.04
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre"
pcre: "--with-pcre2"
maxminddb: "--with-maxminddb"
msan: "--with-sanitizer"
nBPF: ""
- compiler: "clang-14" # "Newest" clang easily available
ar: "llvm-ar-14"
ranlib: "llvm-ranlib-14"
os: ubuntu-22.04
- compiler: "clang-17" # "Newest" clang easily available. See also below...
ar: "llvm-ar-17"
ranlib: "llvm-ranlib-17"
os: ubuntu-24.04
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre"
pcre: "--with-pcre2"
maxminddb: "--with-maxminddb"
msan: "--with-sanitizer"
nBPF: ""
lto_gold_linker: "--with-lto-and-gold-linker"
- compiler: "clang-18" # "The latest clang version easily available should be 18,
# but it is buggy on ubuntu-24.04 with LTO and Gold linker:
# https://github.com/llvm/llvm-project/issues/87553
# https://bugs.launchpad.net/ubuntu/+source/llvm-toolchain-18/+bug/2064187
# Waiting for a fix...
ar: "llvm-ar-18"
ranlib: "llvm-ranlib-18"
os: ubuntu-24.04
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre2"
maxminddb: "--with-maxminddb"
msan: "--with-sanitizer"
nBPF: ""
lto_gold_linker: ""
- compiler: "cc"
os: ubuntu-latest
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre"
pcre: "--with-pcre2"
maxminddb: "--with-maxminddb"
msan: "--with-thread-sanitizer"
nBPF: ""
@@ -137,39 +153,48 @@ jobs:
os: ubuntu-latest
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre"
pcre: "--with-pcre2"
maxminddb: "--with-maxminddb"
msan: "--with-sanitizer"
nBPF: "nBPF"
- compiler: "clang-14"
- compiler: "cc"
os: ubuntu-latest
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre2"
maxminddb: "--with-maxminddb"
msan: "--with-sanitizer"
nBPF: ""
global_context: "--disable-global-context-support"
- compiler: "clang" #TODO: some issues with masan/clang/ubuntu-24.04
os: ubuntu-22.04
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre"
pcre: "--with-pcre2"
maxminddb: "--with-maxminddb"
msan: "--with-memory-sanitizer"
nBPF: ""
- compiler: "cc"
os: macOS-latest
os: macOS-13
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre"
pcre: "--with-pcre2"
maxminddb: "--with-maxminddb"
msan: "" # Disable sanitizer on macos
nBPF: ""
- compiler: "cc"
os: macos-12
os: macos-14
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre"
maxminddb: "--with-maxminddb"
pcre: ""
maxminddb: ""
msan: "" # Disable sanitizer on macos
nBPF: ""
- compiler: "cc"
os: ubuntu-latest
arch: "arm64"
gcrypt: ""
pcre: "--with-pcre"
pcre: "--with-pcre2"
maxminddb: "--with-maxminddb"
msan: "" # Disable sanitizer on arm64
nBPF: ""
@@ -177,7 +202,7 @@ jobs:
os: ubuntu-latest
arch: "armhf"
gcrypt: ""
pcre: "--with-pcre"
pcre: "--with-pcre2"
maxminddb: "--with-maxminddb"
msan: "--with-sanitizer"
nBPF: ""
@@ -185,7 +210,7 @@ jobs:
os: ubuntu-latest
arch: "s390x"
gcrypt: ""
pcre: "--with-pcre"
pcre: "--with-pcre2"
maxminddb: "--with-maxminddb"
msan: ""
nBPF: ""
@@ -194,9 +219,15 @@ jobs:
if: startsWith(matrix.os, 'ubuntu') && !startsWith(matrix.arch, 'x86_64')
run: |
docker run --rm --privileged multiarch/qemu-user-static:register --reset
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Shallow clones should be disabled for a better relevancy of analysis
- name: Fix kernel mmap rnd bits on Ubuntu
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64')
run: |
# Workaround for compatinility between latest kernel and sanitizer
# See https://github.com/actions/runner-images/issues/9491
sudo sysctl vm.mmap_rnd_bits=28
- name: Install Ubuntu Prerequisites
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64')
run: |
@@ -204,13 +235,13 @@ jobs:
sudo apt-get install autoconf automake libtool pkg-config gettext libjson-c-dev flex bison libpcap-dev
sudo apt-get install gcc-mingw-w64 libc6-dev
sudo apt-get install doxygen python3-sphinx python3-sphinx-rtd-theme python3-breathe python3-pip
sudo apt-get install rrdtool librrd-dev
sudo apt-get install rrdtool librrd-dev parallel
- name: Install Ubuntu Prerequisites (libgcrypt)
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.gcrypt, '--with-local-libgcrypt')
run: |
sudo apt-get install libgcrypt20-dev
- name: Install Ubuntu Prerequisites (libpcre)
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.pcre, '--with-pcre')
- name: Install Ubuntu Prerequisites (libpcre2)
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.pcre, '--with-pcre2')
run: |
sudo apt-get install libpcre3-dev
- name: Install Ubuntu Prerequisites (maxminddb)
@@ -226,8 +257,13 @@ jobs:
make
cd -
- name: Setup Ubuntu specified compiler
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64') && ! startsWith(matrix.compiler, 'cc')
if: startsWith(matrix.os, 'ubuntu-20.04') && startsWith(matrix.arch, 'x86_64') && ! startsWith(matrix.compiler, 'cc')
run: |
#For gcc-4.9 (on ubuntu-20.04)
echo "deb http://dk.archive.ubuntu.com/ubuntu/ xenial main" | sudo tee -a /etc/apt/sources.list
echo "deb http://dk.archive.ubuntu.com/ubuntu/ xenial universe" | sudo tee -a /etc/apt/sources.list
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 40976EAF437D05B5
sudo apt-get update
sudo apt-get install ${{ matrix.compiler }}
- name: Install Windows msys2 prerequisites
if: startsWith(matrix.os, 'windows')
@@ -235,39 +271,22 @@ jobs:
with:
msystem: MINGW64
update: true
install: git mingw-w64-x86_64-toolchain automake1.16 automake-wrapper autoconf libtool make mingw-w64-x86_64-json-c mingw-w64-x86_64-crt-git mingw-w64-x86_64-pcre mingw-w64-x86_64-libpcap
- name: Setup Windows msys2 prerequisites (libgcrypt)
if: startsWith(matrix.os, 'windows') && startsWith(matrix.gcrypt, '--with-local-libgcrypt')
run: |
msys2 -c 'git clone --branch libgpg-error-1.42 https://github.com/gpg/libgpg-error'
cd libgpg-error
msys2 -c './autogen.sh'
msys2 -c './configure -enable-maintainer-mode --enable-static --enable-shared --with-pic --disable-doc --disable-nls'
msys2 -c 'make'
msys2 -c 'make install'
msys2 -c 'cd ..'
msys2 -c 'rm -rf libgpg-error/'
msys2 -c 'git clone --branch libgcrypt-1.8.8 https://github.com/gpg/libgcrypt'
cd libgcrypt
msys2 -c './autogen.sh'
msys2 -c './configure -enable-maintainer-mode --enable-static --enable-shared --with-pic --disable-doc'
msys2 -c 'make'
msys2 -c 'make install'
msys2 -c 'cd ..'
msys2 -c 'rm -rf libgcrypt/'
env:
CC:
CFLAGS:
install: git mingw-w64-x86_64-toolchain automake1.16 automake-wrapper autoconf libtool make mingw-w64-x86_64-json-c mingw-w64-x86_64-crt-git mingw-w64-x86_64-pcre mingw-w64-x86_64-libpcap mingw-w64-x86_64-libgcrypt parallel
- name: Installing MacOS prerequisites
if: startsWith(matrix.os, 'macOS') && startsWith(matrix.arch, 'x86_64')
run: |
brew install coreutils wdiff colordiff autoconf automake libtool pkg-config gettext json-c rrdtool
# A workaround for "The `brew link` step did not complete successfully" error.
# See https://github.com/Homebrew/homebrew-core/issues/165793#issuecomment-1991817938
find /usr/local/bin -lname '*/Library/Frameworks/Python.framework/*' -delete
sudo rm -rf /Library/Frameworks/Python.framework/
brew install --force python3 && brew unlink python3 && brew unlink python3 && brew link --overwrite python3
brew install coreutils wdiff colordiff autoconf automake libtool pkg-config gettext json-c rrdtool parallel
- name: Install MacOS Prerequisites (libgcrypt)
if: startsWith(matrix.os, 'macOS') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.gcrypt, '--with-local-libgcrypt')
run: |
brew install libgcrypt
- name: Install MacOS Prerequisites (libpcre)
if: startsWith(matrix.os, 'macOS') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.pcre, '--with-pcre')
- name: Install MacOS Prerequisites (libpcre2)
if: startsWith(matrix.os, 'macOS') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.pcre, '--with-pcre2')
run: |
brew install pcre
- name: Install MacOS Prerequisites (maxminddb)
@@ -277,7 +296,7 @@ jobs:
- name: Configure nDPI on Ubuntu
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64')
run: |
AR=${{ matrix.ar }} RANLIB=${{ matrix.ranlib }} ./autogen.sh --enable-option-checking=fatal --enable-debug-messages ${{ matrix.gcrypt }} ${{ matrix.msan }} ${{ matrix.pcre }} ${{ matrix.maxminddb }} --enable-tls-sigs ${{matrix.lto_gold_linker}}
AR=${{ matrix.ar }} RANLIB=${{ matrix.ranlib }} ./autogen.sh --enable-option-checking=fatal --enable-debug-messages ${{ matrix.gcrypt }} ${{ matrix.msan }} ${{ matrix.pcre }} ${{ matrix.maxminddb }} --enable-tls-sigs ${{matrix.lto_gold_linker}} ${{matrix.global_context}}
- name: Configure nDPI on MacOS
if: startsWith(matrix.os, 'macOS') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.compiler, 'cc')
run: |
@@ -289,23 +308,30 @@ jobs:
- name: Build nDPI on Windows msys2
if: startsWith(matrix.os, 'windows') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.compiler, 'cc')
run: |
msys2 -c 'make all'
msys2 -c 'make -j all'
msys2 -c 'ldd ./example/ndpiReader.exe'
- name: Build nDPI
if: startsWith(matrix.arch, 'x86_64') && !startsWith(matrix.os, 'windows')
if: startsWith(matrix.arch, 'x86_64') && !startsWith(matrix.os, 'windows') && !startsWith(matrix.os, 'macos-14')
run: |
make all
make -j all
make -C example ndpiSimpleIntegration
make -C rrdtool
- name: Build nDPI (MacOS M1)
if: startsWith(matrix.os, 'macos-14')
run: |
make -j all
make -C example ndpiSimpleIntegration
#There are somes issues with librrd
#make -C rrdtool
- name: Print nDPI long help
if: startsWith(matrix.arch, 'x86_64') && !startsWith(matrix.os, 'windows')
run: |
./example/ndpiReader -H
cd ./example && ./ndpiReader -H
- name: Install nDPI
if: startsWith(matrix.arch, 'x86_64') && !startsWith(matrix.os, 'windows')
run: |
make install DESTDIR=$(realpath _install)
ls -alhHR _install
DESTDIR=/tmp/ndpi make install
ls -alhHR /tmp/ndpi
- name: Test nDPI [SYMBOLS]
if: (startsWith(matrix.os, 'ubuntu') || startsWith(matrix.os, 'mac')) && startsWith(matrix.arch, 'x86_64')
run: |
@@ -315,10 +341,10 @@ jobs:
- name: Test nDPI [DIFF]
if: startsWith(matrix.arch, 'x86_64') && !startsWith(matrix.os, 'windows')
run: |
./tests/do.sh
NDPI_FORCE_PARALLEL_UTESTS=1 NDPI_SKIP_PARALLEL_BAR=1 ./tests/do.sh
- name: Test nDPI [UNIT]
#Some issues with masan + json-c. Disable the test as workaround
if: startsWith(matrix.arch, 'x86_64') && !startsWith(matrix.os, 'windows') && !startsWith(matrix.msan, '--with-memory-sanitizer')
if: startsWith(matrix.arch, 'x86_64') && !startsWith(matrix.os, 'windows') && !startsWith(matrix.msan, '--with-memory-sanitizer') && !startsWith(matrix.os, 'macos-14')
run: |
./tests/do-unit.sh
- name: Test nDPI [DGA]
@@ -328,7 +354,7 @@ jobs:
- name: Test nDPI [DIFF] (runs only on windows jobs)
if: startsWith(matrix.arch, 'x86_64') && startsWith(matrix.os, 'windows')
run: |
msys2 -c './tests/do.sh'
msys2 -c 'NDPI_FORCE_PARALLEL_UTESTS=1 NDPI_SKIP_PARALLEL_BAR=1 ./tests/do.sh'
- name: Test nDPI [UNIT] (runs only on windows jobs)
if: startsWith(matrix.arch, 'x86_64') && startsWith(matrix.os, 'windows')
run: |
@@ -347,7 +373,7 @@ jobs:
run: |
make distclean
./autogen.sh --enable-option-checking=fatal --enable-debug-messages --enable-tls-sigs --host=x86_64-w64-mingw32
make all
make -j $(nproc) all
env:
CC:
- name: Display qemu specified architecture (arm64 - little endian)
@@ -369,7 +395,7 @@ jobs:
apt-get -y install git wdiff colordiff autoconf automake libtool pkg-config gettext libjson-c-dev flex bison libpcap-dev libgcrypt20-dev libpcre3-dev libmaxminddb-dev rrdtool librrd-dev &&
git config --global --add safe.directory $(realpath .) &&
env CC=gcc CFLAGS='-Werror' ./autogen.sh --enable-option-checking=fatal --enable-debug-messages ${{ matrix.gcrypt }} ${{ matrix.msan }} ${{ matrix.pcre }} ${{ matrix.maxminddb }} --enable-tls-sigs &&
make all &&
make -j all &&
make -C example ndpiSimpleIntegration &&
make -C rrdtool &&
make check VERBOSE=1
@@ -393,7 +419,7 @@ jobs:
apt-get -y install git wdiff colordiff autoconf automake libtool pkg-config gettext libjson-c-dev flex bison libpcap-dev libgcrypt20-dev libpcre3-dev libmaxminddb-dev rrdtool librrd-dev &&
git config --global --add safe.directory $(realpath .) &&
env CC=gcc CFLAGS='-Werror' ./autogen.sh --enable-option-checking=fatal --enable-debug-messages ${{ matrix.gcrypt }} ${{ matrix.msan }} ${{ matrix.pcre }} ${{ matrix.maxminddb }} --enable-tls-sigs &&
make all &&
make -j all &&
make -C example ndpiSimpleIntegration &&
make -C rrdtool &&
make check VERBOSE=1
@@ -417,7 +443,7 @@ jobs:
apt-get -y install git wdiff colordiff autoconf automake libtool pkg-config gettext libjson-c-dev flex bison libpcap-dev libgcrypt20-dev libpcre3-dev libmaxminddb-dev rrdtool librrd-dev &&
git config --global --add safe.directory $(realpath .) &&
env CC=gcc CFLAGS='-Werror' ./autogen.sh --enable-option-checking=fatal --enable-debug-messages ${{ matrix.gcrypt }} ${{ matrix.msan }} ${{ matrix.pcre }} ${{ matrix.maxminddb }} --enable-tls-sigs &&
make all &&
make -j all &&
make -C example ndpiSimpleIntegration &&
make -C rrdtool &&
make check VERBOSE=1
+11 -11
View File
@@ -12,7 +12,7 @@ jobs:
env:
CFLAGS: -Werror
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install Ubuntu Prerequisites
@@ -23,7 +23,7 @@ jobs:
sudo apt-get install libpcre3-dev libmaxminddb-dev lcov
sudo apt-get install wdiff colordiff
- name: Configure
run: ./autogen.sh --enable-option-checking=fatal --enable-debug-messages --enable-code-coverage --with-pcre --with-maxminddb --enable-tls-sigs
run: ./autogen.sh --enable-option-checking=fatal --enable-debug-messages --enable-code-coverage --with-pcre2 --with-maxminddb --enable-tls-sigs
- name: Build
run: make all
- name: Test
@@ -32,7 +32,7 @@ jobs:
- name: Generate Coverage Report
run: |
make coverage
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4
with:
name: ndpi-coverage-report
path: coverage_report
@@ -42,7 +42,7 @@ jobs:
name: Documentation (ubuntu-latest)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Python 3.9
@@ -63,7 +63,7 @@ jobs:
make doc
mkdir -vp doc/_build/ndpi-documentation-upload/ndpi-documentation
mv -v doc/_build/html doc/_build/ndpi-documentation-upload/ndpi-documentation/html
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4
with:
name: ndpi-documentation
path: doc/_build/ndpi-documentation-upload
@@ -76,7 +76,7 @@ jobs:
GO111MODULE: on
CFLAGS: -Werror
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install Ubuntu Prerequisites
@@ -84,27 +84,27 @@ jobs:
sudo add-apt-repository ppa:longsleep/golang-backports
sudo apt-get update
sudo apt-get install libunwind-dev
sudo apt-get install autoconf automake libtool pkg-config gettext flex bison libpcap-dev
sudo apt-get install autoconf automake libtool pkg-config gettext flex bison libpcap-dev parallel
sudo apt-get install golang-1.16-go libgoogle-perftools-dev graphviz
go install github.com/google/pprof@latest
sudo ln -s ${HOME}/go/bin/pprof /usr/bin/pprof
pprof -h
- name: Configure nDPI library
run: |
./autogen.sh --enable-gprof --enable-option-checking=fatal --with-pcre --with-maxminddb --enable-tls-sigs
./autogen.sh --enable-gprof --enable-option-checking=fatal --with-pcre2 --with-maxminddb --enable-tls-sigs
- name: Build nDPI library
run: |
make
make -j
- name: Performance Profiling
run: |
./tests/do.sh
NDPI_FORCE_PARALLEL_UTESTS=1 NDPI_SKIP_PARALLEL_BAR=1 ./tests/do.sh
mkdir ndpi-performance-upload
for d in $(find ./tests/cfgs/* -type d -maxdepth 0 2>/dev/null) ; do
PROFILE="$(basename $d)"
mv -v tests/cfgs/${PROFILE}/result/cpu_profile.png ndpi-performance-upload/${PROFILE}_cpu_profile.png
mv -v tests/cfgs/${PROFILE}/result/heap_profile.png ndpi-performance-upload/${PROFILE}_heap_profile.png
done
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4
with:
name: ndpi-performance
path: ndpi-performance-upload
+2 -2
View File
@@ -18,7 +18,7 @@ jobs:
uses: google/oss-fuzz/infra/cifuzz/actions/run_fuzzers@master
with:
oss-fuzz-project-name: 'ndpi'
fuzz-seconds: 1200
fuzz-seconds: 1320
dry-run: false
sanitizer: ${{ matrix.sanitizer }}
- name: Check Crash (fails when a crash is detected)
@@ -27,7 +27,7 @@ jobs:
run: |
exit $(ls out/artifacts |wc -l)
- name: Upload Crash (upload detected crash as artifacts)
uses: actions/upload-artifact@v1
uses: actions/upload-artifact@v4
if: failure()
with:
name: ${{ matrix.sanitizer }}-artifacts
+6 -6
View File
@@ -24,10 +24,10 @@ jobs:
strategy:
fail-fast: false
matrix:
language: [ 'cpp', 'python', 'ruby' ]
language: [ 'cpp', 'python' ]
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install Ubuntu Prerequisites
@@ -42,18 +42,18 @@ jobs:
pprof -h
- name: Configure nDPI library
run: |
./autogen.sh --enable-gprof --enable-option-checking=fatal --with-pcre --with-maxminddb --enable-tls-sigs --enable-debug-messages
./autogen.sh --enable-gprof --enable-option-checking=fatal --with-pcre2 --with-maxminddb --enable-tls-sigs --enable-debug-messages
- name: Initialize CodeQL
uses: github/codeql-action/init@v2
uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
- name: Build nDPI library
run: |
make
make -j
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v2
uses: github/codeql-action/analyze@v3
with:
category: "/language:${{matrix.language}}"
+31 -7
View File
@@ -61,6 +61,7 @@
/fuzz/fuzz_alg_crc32_md5
/fuzz/fuzz_alg_ses_des
/fuzz/fuzz_alg_bytestream
/fuzz/fuzz_alg_shoco
/fuzz/fuzz_config
/fuzz/fuzz_community_id
/fuzz/fuzz_serialization
@@ -70,49 +71,71 @@
/fuzz/fuzz_ds_ptree
/fuzz/fuzz_ds_hash
/fuzz/fuzz_ds_ahocorasick
/fuzz/fuzz_ds_bitmap64
/fuzz/fuzz_ds_bitmap64_fuse
/fuzz/fuzz_ds_domain_classify
/fuzz/fuzz_libinjection
/fuzz/fuzz_binaryfusefilter
/fuzz/fuzz_tls_certificate
/fuzz/fuzz_dga
/fuzz/fuzz_is_stun_udp
/fuzz/fuzz_is_stun_tcp
/fuzz/fuzz_ds_cmsketch
/fuzz/fuzz_gcrypt_light
/fuzz/fuzz_gcrypt_aes
/fuzz/fuzz_gcrypt_gcm
/fuzz/fuzz_gcrypt_cipher
/fuzz/fuzz_ndpi_reader_payload_analyzer
/fuzz/fuzz_ndpi_reader_pl7m_simplest
/fuzz/fuzz_ndpi_reader_pl7m_simplest_internal
/fuzz/fuzz_ndpi_reader_pl7m
/fuzz/fuzz_ndpi_reader_pl7m_64k
/fuzz/fuzz_ndpi_reader_pl7m_internal
/fuzz/fuzz_filecfg_protocols
/fuzz/fuzz_filecfg_categories
/fuzz/fuzz_filecfg_category
/fuzz/fuzz_filecfg_malicious_sha1
/fuzz/fuzz_filecfg_malicious_ja3
/fuzz/fuzz_filecfg_risk_domains
/fuzz/fuzz_filecfg_config
/fuzz/fuzz_readerutils_workflow
/fuzz/fuzz_readerutils_parseprotolist
/fuzz/fuzz_ndpi_reader_alloc_fail_seed_corpus.zip
/fuzz/fuzz_ndpi_reader_seed_corpus.zip
/fuzz/fuzz_ndpi_reader_pl7m_seed_corpus.zip
/fuzz/fuzz_ndpi_reader_pl7m_64k_seed_corpus.zip
/fuzz/fuzz_ndpi_reader_pl7m_internal_seed_corpus.zip
/fuzz/fuzz_ndpi_reader_pl7m_simplest_seed_corpus.zip
/fuzz/fuzz_ndpi_reader_pl7m_simplest_internal_seed_corpus.zip
/fuzz/fuzz_quic_get_crypto_data_seed_corpus.zip
/fuzz/fuzz_config_seed_corpus.zip
/fuzz/fuzz_community_id_seed_corpus.zip
/fuzz/fuzz_is_stun_udp_seed_corpus.zip
/fuzz/fuzz_is_stun_tcp_seed_corpus.zip
/fuzz/fuzz_serialization_seed_corpus.zip
/fuzz/fuzz_alg_ses_des_seed_corpus.zip
/fuzz/fuzz_alg_bins_seed_corpus.zip
/fuzz/fuzz_alg_hll_seed_corpus.zip
/fuzz/fuzz_alg_jitter_seed_corpus.zip
/fuzz/fuzz_alg_crc32_md5_seed_corpus.zip
/fuzz/fuzz_alg_hw_rsi_outliers_da_seed_corpus.zip
/fuzz/fuzz_alg_bytestream_seed_corpus.zip
/fuzz/fuzz_ds_patricia_seed_corpus.zip
/fuzz/fuzz_ds_libcache_seed_corpus.zip
/fuzz/fuzz_ds_tree_seed_corpus.zip
/fuzz/fuzz_ds_ptree_seed_corpus.zip
/fuzz/fuzz_ds_ahocorasick_seed_corpus.zip
/fuzz/fuzz_ds_bitmap64_seed_corpus.zip
/fuzz/fuzz_ds_bitmap64_fuse_seed_corpus.zip
/fuzz/fuzz_ds_domain_classify_seed_corpus.zip
/fuzz/fuzz_libinjection_seed_corpus.zip
/fuzz/fuzz_tls_certificate_seed_corpus.zip
/fuzz/fuzz_filecfg_protocols_seed_corpus.zip
/fuzz/fuzz_filecfg_categories_seed_corpus.zip
/fuzz/fuzz_filecfg_category_seed_corpus.zip
/fuzz/fuzz_filecfg_malicious_sha1_seed_corpus.zip
/fuzz/fuzz_filecfg_malicious_ja3_seed_corpus.zip
/fuzz/fuzz_filecfg_risk_domains_seed_corpus.zip
/fuzz/fuzz_filecfg_config_seed_corpus.zip
/fuzz/fuzz_dga_seed_corpus.zip
/fuzz/fuzz_ndpi_reader_payload_analyzer_seed_corpus.zip
/fuzz/fuzz_readerutils_workflow_seed_corpus.zip
/fuzz/fuzz_readerutils_parseprotolist_seed_corpus.zip
/fuzz/fuzz_*.dict
/fuzz/testpcaps.zip
/influxdb/Makefile
/install-sh
/libndpi.pc
@@ -139,6 +162,7 @@
/src/lib/libndpi.so*
/src/lib/protocols/.deps/
/src/lib/third_party/src/.deps/
/src/lib/third_party/src/fuzz/.deps/
/tags
/test-driver
/tests/Makefile
+1 -1
View File
@@ -384,7 +384,7 @@
- Fix stop of extra dissection in HTTP, Bittorrent, Kerberos
- Fix signed integer overflow in ASN1/BER dissector
- Fix char/uchar bug in ahocorasick
- Fix endianess in IP-Port lookup
- Fix endianness in IP-Port lookup
- Fix FastCGI memory allocation issue
- Fix metadata extraction in NAT-PMP
- Fix invalid unidirectional traffic alert for unidirectional protocols (e.g. sFlow)
+6
View File
@@ -24,4 +24,10 @@ Please keep all code, comments, issues and pull requests in English, otherwise t
- If there is an open issue for the bug, ensure the description of the pull request contains `Fixes #issue_number`.
- Ensure the pull request passes CI testing. Any pull request which fails CI will be **ignored**.
## Before you submit your code
As we want to make sure that we can incorporate your changes in nDPI, we ask all contributors to read and sign the
[Contributor License Agreement](https://github.com/ntop/legal/blob/main/individual-contributor-licence-agreement.md)
that guarantees that your code is original and royalty-free to keep nDPI free and available for everybody,
Finally, thank you for your contributions.
+4 -1
View File
@@ -14,12 +14,15 @@ EXTRA_DIST = README.md README.fuzzer.md CHANGELOG.md CONTRIBUTING.md \
rrdtool/Makefile.in rrdtool/README.txt rrdtool/rrd_anomaly.c rrdtool/rrd_similarity.c \
doc/requirements.txt doc/conf.py doc/flow_risks.rst doc/protocols.rst doc/guide/nDPI_QuickStartGuide.pages \
doc/guide/nDPI_QuickStartGuide.pdf doc/img/logo.png doc/index.rst \
doc/Makefile doc/what_is_ndpi.rst \
doc/Makefile doc/what_is_ndpi.rst doc/FAQ.md doc/configuration_parameters.md \
doc/library_initialization.md \
python/DEV_GUIDE.md python/dev_requirements.txt python/ndpi_example.py python/ndpi/__init__.py \
python/ndpi/ndpi_build.py python/ndpi/ndpi.py python/README.md \
python/requirements.txt python/setup.py python/tests.py \
lists/100_malware.list \
lists/107_gambling.list \
lists/107_gambling_custom.list \
lists/public_suffix_list.dat \
lists/README.md \
sonar-project.properties .github .ci-ignore
+6 -4
View File
@@ -1,4 +1,3 @@
![ntop][ntopng_logo] ![ntop][ntop_logo]
# nDPI
[![Build Status](https://img.shields.io/github/actions/workflow/status/ntop/nDPI/build.yml?branch=dev&logo=github)](https://github.com/ntop/nDPI/actions?query=workflow%3ABuild)
@@ -8,6 +7,8 @@
nDPI® is an open source LGPLv3 library for deep-packet inspection. Based on OpenDPI it includes ntop extensions. We have tried to push them into the OpenDPI source tree but nobody answered emails so we have decided to create our own source tree
A generic FAQ about nDPI® is available [here](https://github.com/ntop/nDPI/blob/dev/doc/FAQ.md)
### How To Compile nDPI
In order to compile this project do
@@ -51,7 +52,7 @@ On Windows:
There are three supported ways to build nDPI:
1. MSYS2 (assuming [MSYS2](https://www.msys2.org/) already installed):
- msys2 -c "pacman --noconfirm -S --needed --overwrite '\*' git mingw-w64-x86\_64-toolchain automake1.16 automake-wrapper autoconf libtool make mingw-w64-x86\_64-json-c mingw-w64-x86\_64-crt-git mingw-w64-x86\_64-pcre mingw-w64-x86\_64-libpcap"
- msys2 -c "pacman --noconfirm -S --needed --overwrite '\*' git mingw-w64-x86\_64-toolchain automake1.16 automake-wrapper autoconf libtool make mingw-w64-x86\_64-json-c mingw-w64-x86\_64-crt-git mingw-w64-x86\_64-pcre2 mingw-w64-x86\_64-libpcap"
2. Mingw-w64
@@ -78,11 +79,12 @@ The entire procedure of adding new protocols in detail:
4. Add a new entry for the search function for the new protocol in: `src/include/ndpi_protocols.h`
5. Choose (do not change anything) a selection bitmask from: `src/include/ndpi_define.h`
6. Set protocol default ports in `ndpi_init_protocol_defaults` in: `src/lib/ndpi_main.c`
7. Be sure to have nBPF support, cloning `PF_RING` in the same directory where you cloned `nDPI`: `git clone https://github.com/ntop/PF_RING/ && cd PF_RING/userland/nbpf && ./configure && make`
8. From the `nDPI` root directory, `./autogen.sh --with-pcre` (nBPF and PCRE are usually optional, but they are needed to run/update *all* the unit tests)
7. Be sure to have nBPF support, cloning `PF_RING` in the same directory where you cloned `nDPI`: `git clone https://github.com/ntop/PF_RING/ && cd PF_RING/userland/nbpf && ./configure && make`. You can ignore the `/bin/sh: 1: ../lib/pfring_config: not found` error
8. From the `nDPI` root directory, `./autogen.sh --with-pcre2` (nBPF and PCRE2 are usually optional, but they are needed to run/update *all* the unit tests)
9. `make`
10. `make check`
11. Update the documentation, adding this new protocol to `doc/protocols.rst`
12. Update the Windows Visual Studio configuration, adding the new c file in `windows/nDPI.vcxproj`
### How to use nDPI to Block Selected Traffic
+66 -23
View File
@@ -1,4 +1,4 @@
AC_INIT([libndpi],[4.7.0])
AC_INIT([libndpi],[4.9.0])
AC_CONFIG_AUX_DIR([.])
AC_CONFIG_MACRO_DIR([m4])
@@ -29,6 +29,7 @@ AC_ARG_ENABLE(npcap, AS_HELP_STRING([--disable-npcap], [msys2 only: Disable link
AC_ARG_WITH(nbpf-path, AS_HELP_STRING([--with-nbpf-path], [nBPF library custom path; default: ${PWD}/../PF_RING/userland/nbpf]),[NBPF_HOME=$withval],[NBPF_HOME=${PWD}/../PF_RING/userland/nbpf])
AC_ARG_WITH(lto-and-gold-linker, AS_HELP_STRING([--with-lto-and-gold-linker], [Build with LTO and Gold linker]))
AC_ARG_ENABLE(debug-build, AS_HELP_STRING([--enable-debug-build], [Enable debug build (`-g` flag)]),[enable_debugbuild=$enableval],[enable_debugbuild=no])
AC_ARG_ENABLE(global-context-support, AS_HELP_STRING([--disable-global-context-support], [Disable support for global context. No external dependency on libpthread]))
NDPI_CFLAGS="${NDPI_CFLAGS} -D_DEFAULT_SOURCE=1 -D_GNU_SOURCE=1"
@@ -64,8 +65,14 @@ AS_IF([test "${with_sanitizer+set}" = set -o "${with_thread_sanitizer+set}" = se
])
AS_IF([test "${with_sanitizer+set}" = set],[
NDPI_CFLAGS="${NDPI_CFLAGS} -fsanitize=address -fsanitize=undefined -fno-sanitize=alignment -fsanitize=leak -fno-omit-frame-pointer"
NDPI_LDFLAGS="${NDPI_LDFLAGS} -fsanitize=address -fsanitize=undefined -fno-sanitize=alignment -fsanitize=leak"
NDPI_CFLAGS="${NDPI_CFLAGS} -fsanitize=address -fsanitize=undefined -fsanitize=leak -fno-omit-frame-pointer"
NDPI_LDFLAGS="${NDPI_LDFLAGS} -fsanitize=address -fsanitize=undefined -fsanitize=leak"
#Sanitizers should work on any compilers that we support (or that we test on CI, at least)
#Exception: "-fsanitize=alignment" is not supported in gcc 4.9
AX_CHECK_COMPILE_FLAG([-fno-sanitize=alignment], [
NDPI_CFLAGS="${NDPI_CFLAGS} -fno-sanitize=alignment"
NDPI_LDFLAGS="${NDPI_LDFLAGS} -fno-sanitize=alignment"
])
])
AS_IF([test "${with_thread_sanitizer+set}" = set],[
@@ -113,6 +120,29 @@ AC_CHECK_TOOL(AR, ar, [false])
AC_LANG_WERROR
AC_MSG_CHECKING([whether roaring v3 works])
AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
#if __has_include (<stdio.h>)
#endif
]])],
[
AC_MSG_RESULT([yes])
],
[
AC_MSG_RESULT([no])
AC_DEFINE_UNQUOTED(USE_ROARING_V2, "1", [Use CRoaring 2.1.x])
])
GCC_VERSION=`gcc --version | cut -d ' ' -f 3 | head -1|cut -d '.' -f 1`
if test "${GCC_VERSION}" == "version" ; then
GCC_VERSION=`gcc --version | cut -d ' ' -f 4 | head -1|cut -d '.' -f 1`
fi
if test "${GCC_VERSION}" -lt "7" ; then
AC_DEFINE_UNQUOTED(USE_ROARING_V2, "1", [Use CRoaring 2.1.x])
fi
NDPI_MAJOR=`echo "${PACKAGE_VERSION}" | cut -d . -f 1`
NDPI_MINOR=`echo "${PACKAGE_VERSION}" | cut -d . -f 2`
NDPI_PATCH=`echo "${PACKAGE_VERSION}" | cut -d . -f 3`
@@ -125,8 +155,8 @@ AC_DEFINE_UNQUOTED(NDPI_PATCH_LEVEL, "${NDPI_PATCH}", [nDPI patch level])
# .git as directory in a cloned repo
# .git as file in submodule based integration
if test -d ".git" || test -r ".git"; then
GIT_TAG=`git log -1 --format=%h`
GIT_DATE=`git log -1 --format=%cd`
GIT_TAG=`git log -1 --abbrev=7 --format=%h`
GIT_DATE=`git log -1 --abbrev=7 --format=%cd`
#
# On CentOS 6 `git rev-list HEAD --count` does not work
#
@@ -171,12 +201,19 @@ if ! test "${with_only_libndpi+set}" = set; then :
JSONC_LIBS=""
JSONC_CFLAGS=""
])
AX_PTHREAD([],[
AC_MSG_WARN([POSIX Threads not available. Building library only.])
JSONC_LIBS=""
JSONC_CFLAGS=""
EXTRA_TARGETS=""
])
dnl> AX_PTHREAD test does not work along with MingW/MSYS anymore
case "$host" in
*-*-mingw32*|*-*-msys)
;;
*)
AX_PTHREAD([],[
AC_MSG_WARN([POSIX Threads not available. Building library only.])
JSONC_LIBS=""
JSONC_CFLAGS=""
EXTRA_TARGETS=""
])
;;
esac
fi
AM_CONDITIONAL([BUILD_UNITTESTS], [test "x$build_unittests" = "xyes"])
@@ -185,9 +222,7 @@ AS_IF([test "x${enable_tls_sigs}" = "xyes"],[
HANDLE_TLS_SIGS=""
])
NDPI_CFLAGS="-W -Wall -Wno-unused-parameter -Wno-unused-function -Wno-address-of-packed-member ${NDPI_CFLAGS} -Wno-attributes"
#Workaround for RoaringBitmap with gcc-7
NDPI_CFLAGS="-Wno-attributes ${NDPI_CFLAGS}"
NDPI_CFLAGS="-W -Wall -Wno-address-of-packed-member ${NDPI_CFLAGS}"
AS_IF([test "${with_lto_and_gold_linker+set}" = set], [
NDPI_CFLAGS="${NDPI_CFLAGS} -flto -fuse-ld=gold -Wno-unused-command-line-argument"
@@ -203,6 +238,13 @@ if test "x$ac_cv_lib_rrd_rrd_fetch_r" = xyes; then :
EXTRA_TARGETS="$EXTRA_TARGETS rrdtool"
fi
GLOBAL_CONTEXT_ENABLED=0
AS_IF([test "x$enable_global_context_support" != "xno"], [
AC_CHECK_LIB([pthread], [pthread_mutex_init])
NDPI_CFLAGS="${NDPI_CFLAGS} -DUSE_GLOBAL_CONTEXT"
GLOBAL_CONTEXT_ENABLED=1
])
ADDITIONAL_INCS=
ADDITIONAL_LIBS="$LIBM"
PCAP_HOME=$HOME/PF_RING/userland
@@ -359,14 +401,14 @@ AS_IF([test "${with_local_libgcrypt+set}" = set],[
AC_DEFINE_UNQUOTED(USE_HOST_LIBGCRYPT, 1, [Use locally installed libgcrypt instead of builtin gcrypt-light])
])
dnl> PCRE
PCRE_ENABLED=0
AC_ARG_WITH(pcre, AS_HELP_STRING([--with-pcre], [Enable nDPI build with libpcre]))
if test "${with_pcre+set}" = set; then :
AC_CHECK_LIB(pcre, pcre_compile, AC_DEFINE_UNQUOTED(HAVE_PCRE, 1, [libpcre(-dev) is present]))
if test "x$ac_cv_lib_pcre_pcre_compile" = xyes; then :
ADDITIONAL_LIBS="${ADDITIONAL_LIBS} -lpcre"
PCRE_ENABLED=1
dnl> PCRE2
PCRE2_ENABLED=0
AC_ARG_WITH(pcre2, AS_HELP_STRING([--with-pcre2], [Enable nDPI build with libpcre2]))
if test "${with_pcre2+set}" = set; then :
AC_CHECK_LIB(pcre2-8, pcre2_compile_8, AC_DEFINE_UNQUOTED(HAVE_PCRE2, 1, [libpcre2(-dev) is present]))
if test "x$ac_cv_lib_pcre2_8_pcre2_compile_8" = xyes; then :
ADDITIONAL_LIBS="${ADDITIONAL_LIBS} -lpcre2-8"
PCRE2_ENABLED=1
fi
fi
@@ -420,8 +462,9 @@ AC_SUBST(GPROF_CFLAGS)
AC_SUBST(GPROF_LIBS)
AC_SUBST(GPROF_ENABLED)
AC_SUBST(USE_HOST_LIBGCRYPT)
AC_SUBST(PCRE_ENABLED)
AC_SUBST(PCRE2_ENABLED)
AC_SUBST(NBPF_ENABLED)
AC_SUBST(GLOBAL_CONTEXT_ENABLED)
AC_SUBST(HANDLE_TLS_SIGS)
AC_SUBST(DISABLE_NPCAP)
AC_SUBST(EXE_SUFFIX)
+36
View File
@@ -0,0 +1,36 @@
# FAQ
From https://www.ntop.org/ndpi/ndpi-internals-and-frequent-questions/
**Q**: How nDPI implements protocol detection?\
**A**: nDPI includes a list of protocol dissectors (364 as of today) that are able to dissect protocols such as WhatsApp or TLS. As soon as a new flow is submitted to nDPI, the library applies in sequence dissectors that can potentially match the protocols (i.e. telnet is a TCP-based protocol and it will not be considered for UDP flows). We start from the dissector that can most probably match using the port number. This means for traffic on TCP/22 nDPI will start with the SSH dissectors and if not matching continue with the others. Dissection completes as soon as a protocol matches or when none of them matched and in this case the flow will be labelled as Unknown.
**Q**: What is the nDPI release cycle?\
**A**: We cut release a approximately every 6-8 months, fixes and improvements are on a daily basis (check the nDPI code on GitHub).
**Q**: Is nDPI running on all popular platforms?\
**A**: Yes it runs on Linux, macOS, Windows… and also on not-so-popular ones such as IBM mainframes. We support ARM, Intel, RISC… architectures.
**Q**: How many packets does nDPI need in order to implement detection?\
**A**: It depends on the protocol. For UDP-based protocols such as DNS one packet is enough, for more complex protocols such as TLS about 10 packets. For sure if after 15-20 packets nDPI has not detected the application protocol, then the protocol is labelled as Unknown.
**Q**: Is nDPI detection only based on protocol dissectors?\
**A**: No, payload inspection is the main technique, but nDPI can also use IP address, ports, TLS certificates etc as signatures for protocols. In this case, after detection is complete, nDPI will report if the match was performed on payload inspection or other means (e.g. IP address).
**Q**: Does nDPI contain list of known IP addresses?\
**A**: Yes it includes lists of well known IPs such as those provided by Microsoft of Meta for identifying known service.
**Q**: Can I extend nDPI by defining new protocols with a configuration file?\
**A**: Yes you can. See this [file](https://github.com/ntop/nDPI/blob/dev/example/protos.txt) as an example for defining new protocols..
**Q**: Is nDPI able to detect VPNs?\
**A**: Yes it can detect VPNS such as Tailscale, WireGuard, OpenVPN, FortiClient.. and also in-app VPNs such as UltraSurf or OperaVPN.
**Q**: Is nDPI able to detect malware and viruses?\
**A**: It can detect anomalous behaviour that can be caused by a malware, but nDPI is not a signature-based tool so it does not include signatures for malware A or B. This is because [signature-based tools](https://en.wikipedia.org/wiki/Intrusion_detection_system) have various limitations and resource intensive, whereas nDPI has been designed to be used also in high-speed (100 Gbit+) networks.
**Q**: Is nDPI able to detect security issues?\
**A**: Yes it can by means of a technique called [flow risk](https://github.com/ntop/nDPI/blob/dev/doc/flow_risks.rst). It can identify 50+ threats (e.g. a host that is talking with a malware host).
**Q**: Is nDPI able to block traffic?\
**A**: No, nDPI is a passive traffic analysis library that does not manipulate packets. You can create applications on top of it for policing (i.e. blocking or shaping) traffic. Examples of such applications are ntopng Edge, nProbe IPS and nProbe Cento.
+49
View File
@@ -0,0 +1,49 @@
# Configuration knobs
TODO
| Protocol | Parameter | Default value | Min value | Max value | Description | Notes |
| ------ | ------ | ------ | ------ | ------ | ------ | ------ |
| NULL | "packets_limit_per_flow" | 32 | 0 | 255 | The upper limit on the number of packets per flow that will be subject to DPI, after which classification will be considered complete (0 = no limit) |
| NULL | "flow.direction_detection" | enable | NULL | NULL | Enable/disable internal detection of packet direction (client to server or server to client) |
| NULL | "flow.track_payload" | disable | NULL | NULL | Enable/disable tracking/export of flow payload (i.e. L5/7 data): if enabled, the library exports the first 1024 bytes of payload for each flow |
| NULL | "tcp_ack_payload_heuristic" | disable | NULL | NULL | In some networks, there are some anomalous TCP flows where the smallest ACK packets have some kind of zero padding. It looks like the IP and TCP headers in those frames wrongly consider the 0x00 Ethernet padding bytes as part of the TCP payload. While this kind of packets is perfectly valid per-se, in some conditions they might be treated by the TCP reassembler logic as (partial) overlaps, deceiving the classification engine. This parameter enable/disable an heuristic to detect these packets and to ignore them, allowing correct detection/classification. See #1946 for other details |
| NULL | "fully_encrypted_heuristic" | enable | NULL | NULL | Enable/disable an heuristic to detect fully encrypted sessions, i.e. flows where every bytes of the payload is encrypted in an attempt to “look like nothing”. This heuristic only analyzes the first packet of the flow. See: https://www.usenix.org/system/files/sec23fall-prepub-234-wu-mingshi.pdf |
| NULL | "libgcrypt.init" | 1 | NULL | NULL | Enable/disable initialization of libgcrypt. When using the external libgcrypt (instead of the internal crypto code) the libgcrypt runtime must be initialized. If, for whatever reasons, the application alread does it, nDPI must be told to skip it. Note that, by default, nDPI uses the crypto code and not libgcrypt: in that case this parameter is ignored |
| NULL | "dpi.compute_entropy" | 1 | NULL | NULL | Enable/disable computation of flow entropy |
| NULL | "dpi.guess_on_giveup" | 0x03 | 0x00 | 0x03 | Tell the library to guess flow classification, if any DPI algorithms/logics fail. The value is a bitmask. Values: 0x0 = disabled; 0x01 = enable guessing by port; 0x02 = enable guessing by ip |
| NULL | "flow_risk_lists.load" | 1 | NULL | NULL | Enable/disable loading of every IP addresses lists used to check any flow risks |
| NULL | "flow_risk.anonymous_subscriber.list.icloudprivaterelay.load" | 1 | NULL | NULL | Enable/disable loading of internal iCouldPrivateRealy IP address list used to check `NDPI_ANONYMOUS_SUBSCRIBER` flow risk |
| NULL | "flow_risk.anonymous_subscriber.list.protonvpn.load" | 1 | NULL | NULL | Enable/disable loading of internal IP address list of ProtonVPN exit nodes used to check `NDPI_ANONYMOUS_SUBSCRIBER` flow risk |
| NULL | "flow_risk.crawler_bot.list.load" | 1 | NULL | NULL | Enable/disable loading of internal IP address list used to check `NDPI_HTTP_CRAWLER_BOT` flow risk |
| NULL | "filename.config" | NULL | NULL | NULL | Name of the file containing a list of configuration knobs itself (one per line)!. Useful to configure nDPI via text file instead of via API |
| NULL | "log.level" | 0 | 0 | 3 | Configure the log/debug level. Possible values: 0 = error, 1 = trace, 2 = debug, 3 = extra debug |
| NULL | "lru.$CACHE_NAME.size" | See description | 0 | 16777215 | Set the size (in number of elements) of the specified LRU cache (0 = the cache is disabled). The keyword "$CACHE_NAME" is a placeholder for the cache name and the possible values are: ookla, bittorrent, stun, tls_cert, mining, msteams. The default value is "32768" for the bittorrent cache and "1024" for all the other caches |
| NULL | "lru.$CACHE_NAME.ttl" | See description | 0 | 16777215 | Set the TTL (in seconds) for the elements of the specified LRU cache (0 = the elements never explicitly expire). The keyword "$CACHE_NAME" is a placeholder for the cache name and the possible values are: ookla, bittorrent, stun, tls_cert, mining, msteams. The default value is "120" for the ookla cache, "60" for the msteams cache and "0" for all the other caches |
| NULL | "lru.$CACHE_NAME.scope" | 0 | 0 | 1 | Set the scope of the specified LRU cache (0 = the cache is local, 1 = the cache is global). The keyword "$CACHE_NAME" is a placeholder for the cache name and the possible values are: ookla, bittorrent, stun, tls_cert, mining, msteams. The global scope con be set only if a global context has been initialized |
| "tls" | "certificate_expiration_threshold" | 30 | 0 | 365 | The threshold (in days) used to trigger the `NDPI_TLS_CERTIFICATE_ABOUT_TO_EXPIRE` flow risk |
| "tls" | "application_blocks_tracking" | disable | NULL | NULL | Enable/disable processing of TLS Application Blocks (post handshake) to extract statistical information about the flow |
| "tls" | "metadata.sha1_fingerprint" | enable | NULL | NULL | Enable/disable computation and export of SHA1 fingerprint for TLS flows. Note that if it is disable, the flow risk `NDPI_MALICIOUS_SHA1_CERTIFICATE` is not checked |
| "tls" | "metadata.ja3c_fingerprint" | enable | NULL | NULL | Enable/disable computation and export of JA3C fingerprint for TLS flows. Note that if it is disable, the flow risk `NDPI_MALICIOUS_JA3` is not checked |
| "tls" | "metadata.ja3s_fingerprint" | enable | NULL | NULL | Enable/disable computation and export of JA3S fingerprint for TLS flows |
| "tls" | "metadata.ja4c_fingerprint" | enable | NULL | NULL | Enable/disable computation and export of JA4C fingerprint for TLS flows |
| "smtp" | "tls_dissection" | enable | NULL | NULL | Enable/disable dissection of TLS packets in cleartext SMTP flows (because of opportunistic TLS, via STARTTLS msg) |
| "imap" | "tls_dissection" | enable | NULL | NULL | Enable/disable dissection of TLS packets in cleartext IMAP flows (because of opportunistic TLS, via STARTTLS msg) |
| "pop" | "tls_dissection" | enable | NULL | NULL | Enable/disable dissection of TLS packets in cleartext POP flows (because of opportunistic TLS, via STARTTLS msg) |
| "ftp" | "tls_dissection" | enable | NULL | NULL | Enable/disable dissection of TLS packets in cleartext FTP flows (because of opportunistic TLS, via AUTH TLS msg) |
| "stun" | "max_packets_extra_dissection" | 4 | 0 | 255 | After a flow has been classified has STUN, nDPI might analyse more packets to look for a sub-classification or for metadata. This parameter set the upper limit on the number of these packets |
| "stun" | "tls_dissection" | enable | NULL | NULL | Enable/disable dissection of TLS packets multiplexed into STUN flows |
| "stun" | "metadata.attribute.mapped_address" | enable | NULL | NULL | Enable/disable extraction of (xor)-mapped-address attribute for STUN flows. If it is disabled, STUN classification might be significant faster |
| "stun" | "metadata.attribute.response_origin" | enable | NULL | NULL | Enable/disable extraction of response-origin attribute for STUN flows. If it is disabled, STUN classification might be significant faster |
| "stun" | "metadata.attribute.other_address" | enable | NULL | NULL | Enable/disable extraction of other-address attribute for STUN flows. If it is disabled, STUN classification might be significant faster |
| "stun" | "metadata.attribute.relayed_address" | enable | NULL | NULL | Enable/disable extraction of (xor-)relayed-address attribute for STUN flows. If it is disabled, STUN classification might be significant faster |
| "stun" | "metadata.attribute.peer_address" | enable | NULL | NULL | Enable/disable extraction of (xor-)peer-address attribute for STUN flows. If it is disabled, STUN classification might be significant faster; however sub-classification capability might be negatively impacted |
| "dns" | "subclassification" | enable | NULL | NULL | Enable/disable sub-classification of DNS flows (via query/response domain name). If disabled, some flow risks are not checked |
| "dns" | "process_response" | enable | NULL | NULL | Enable/disable processing of DNS responses. By default, DNS flows are fully classified after the first request/response pair (or after the first response, if the request is missing). If this parameter is disabled, the flows are fully classified after the first packet, i.e. usually after the first request; in that case, some flow risks are not checked and some metadata are not exported |
| "http" | "process_response" | enable | NULL | NULL | Enable/disable processing of HTTP responses. By default, HTTP flows are usually fully classified after the first request/response pair. If this parameter is disabled, the flows are fully classified after the first request (or after the first response, if the request is missing); in that case, some flow risks are not checked and some metadata are not exported |
| "ookla" | "dpi.aggressiveness", | 0x01 | 0x00 | 0x01 | Detection aggressiveness for Ookla. The value is a bitmask. Values: 0x0 = disabled; 0x01 = enable heuristic for detection over TLS (via Ookla LRU cache) |
| "zoom" | "max_packets_extra_dissection" | 4 | 0 | 255 | After a flow has been classified has Zoom, nDPI might analyse more packets to look for a sub-classification or for metadata. This parameter set the upper limit on the number of these packets |
| "rtp" | "search_for_stun" | disable | NULL | NULL | After a flow has been classified as RTP or RTCP, nDPI might analyse more packets to look for STUN/DTLS packets, i.e. to try to tell if this flow is a "pure" RTP/RTCP flow or if the RTP/RTCP packets are multiplexed with STUN/DTLS. Useful for proper (sub)classification when the beginning of the flows are not captured or if there are lost packets in the the captured traffic. If enabled, nDPI requires more packets to process for each RTP/RTCP flow. |
| $PROTO_NAME | "log" | disable | NULL | NULL | Enable/disable logging/debug for specific protocol. Use "any" as protocol name if you want to easily enable/disable logging/debug for all protocols |
| $PROTO_NAME | "ip_list.load" | 1 | NULL | NULL | Enable/disable loading of internal list of IP addresses (used for (sub)classification) specific to that protocol. Use "any" as protocol name if you want to easily enable/disable all lists. This knob is valid only for the following protocols: Alibaba, Amazon AWS, Apple, Avast, Bloomberg, Cachefly, Cloudflare, Discord, Disney+, Dropbox, Edgecast, EpicGames, Ethereum, Facebook, Github, Google, Google Cloud, GoTo, Hotspot Shield, Hulu, Line, Microsoft 365, Microsoft Azure, Microsoft One Drive, Microsoft Outlook, Mullvad, Netflix, Nvidia, OpenDNS, ProtonVPN, RiotGames, Roblox, Skype/Teams, Starcraft, Steam, Teamviewer, Telegram, Tencent, Threema, TOR, Twitch, Twitter, UbuntuONE, VK, Yandex, Yandex Cloud, Webex, Whatsapp, Zoom |
+15 -3
View File
@@ -105,7 +105,7 @@ A `DGA <https://en.wikipedia.org/wiki/Domain_generation_algorithm>`_ is used to
NDPI_MALFORMED_PACKET
=====================
This risk is generated when a packet (e.g. a DNS packet) has an unexpected formt. This can indicate a protocol error or more often an attempt to jeopardize a valid protocol to carry other type of data.
This risk is generated when a packet (e.g. a DNS packet) has an unexpected format. This can indicate a protocol error or more often an attempt to jeopardize a valid protocol to carry other type of data.
.. _Risk 018:
@@ -247,7 +247,7 @@ Additionally, some TLS protocol fields are checked for printable characters as w
NDPI_POSSIBLE_EXPLOIT
=====================
The risk is set whenever a possible exploit (e.g. `Log4J/Log4Shell <https://en.wikipedia.org/wiki/Log4Shell>`_) is detected.
The risk is set whenever a possible exploit attempt (e.g. `Log4J/Log4Shell <https://en.wikipedia.org/wiki/Log4Shell>`_) is detected.
.. _Risk 041:
@@ -277,7 +277,7 @@ The risk is set whenever a crawler/bot/robot has been detected
NDPI_ANONYMOUS_SUBSCRIBER
===================================
The risk is set whenever the (source) ip address has been anonymized and it can't be used to identify the subscriber.
The risk is set whenever the (source) IP address has been anonymized and it can't be used to identify the subscriber.
Example: the flow is generated by an iCloud-private-relay exit node.
.. _Risk 046:
@@ -328,3 +328,15 @@ Invalid TLS ALPN/SNI mismatch. For instance ALPN advertises the flow as h2 (HTTP
NDPI_MALWARE_CONTACTED
======================
Client contacted a server host labelled as malware.
.. _Risk 054:
NDPI_BINARY_DATA_TRANSFER
============================
HTTP only: this risk indicates that a binary file/data application transfer (attempt).
.. _Risk 055:
NDPI_PROBING_ATTEMPT
====================
Connection with no data exchagef that looks like a probing attempt
+84
View File
@@ -0,0 +1,84 @@
A simple, common example
```
struct ndpi_detection_module_struct *ndpi_struct;
ndpi_cfg_error rc;
int ret;
ndpi_struct = ndpi_init_detection_module(NULL);
if(!ndpi_struct) {
ERROR;
}
/* Configuration */
rc = ndpi_set_config(ndpi_struct, "tls", "certificate_expiration_threshold", "10");
if(rc != NDPI_CFG_OK) {
ERROR;
}
/* Finalization */
ret = ndpi_finalize_initialization(ndpi_struct);
if(ret != 0) {
ERROR;
}
/* Initialization done, now you can feed packets to the library */
/* Cleanup */
ndpi_exit_detection_module(ndpi_struct);
```
A more complex example, with global context and a shared Oookla LRU cache (all the others caches are local)
```
struct ndpi_global_context *g_ctx;
struct ndpi_detection_module_struct *ndpi_structs[num_local_contexts];
ndpi_cfg_error rc;
int i, ret;
g_ctx = ndpi_global_init();
if(!g_ctx) {
ERROR;
}
for(i = 0; i < num_local_contexts; i++) {
ndpi_structs[i] = ndpi_init_detection_module(g_ctx);
if(!ndpi_struct[i]) {
ERROR;
}
rc = ndpi_set_config(ndpi_structs[i], NULL, "lru.ookla.scope", "1");
if(rc != NDPI_CFG_OK) {
ERROR;
}
ret = ndpi_finalize_initialization(ndpi_structs[i]);
if(ret != 0) {
ERROR;
}
}
/* Initialization done */
/* Cleanup */
for(i = 0; i < num_local_contexts; i++) {
ndpi_exit_detection_module(ndpi_structs[i]);
}
ndpi_global_deinit(g_ctx);
```
+722 -14
View File
@@ -5,13 +5,126 @@ This page provides the list of the protocols/applications supported by nDPI. For
Work in progress!
.. _Proto 32:
`NDPI_PROTOCOL_BFCP`
=======================
BFCP (Binary Floor Control Protocol) is used for controlling and coordinating real-time data sharing and collaboration during video conferencing sessions.
References: `RFC: <https://datatracker.ietf.org/doc/html/rfc8855>`_
.. _Proto 54:
`NDPI_PROTOCOL_IQIYI`
===========================
iQIYI is a Chinese online video platform that offers a wide range of original and licensed content including movies, dramas, variety shows, and anime.
References: `Main site: <https://www.iqiyi.com/>`_
.. _Proto 59:
`NDPI_PROTOCOL_ADOBE_CONNECT`
===========================
Adobe Connect is a web conferencing platform that allows users to conduct online meetings, webinars, and virtual classrooms.
References: `Main site: <https://www.adobe.com/products/adobeconnect.html>`_
.. _Proto 65:
`NDPI_PROTOCOL_IRC`
==================
IRC (Internet Relay Chat) is a text-based chat system for instant messaging.
References: `Wikipiedia: <https://en.wikipedia.org/wiki/IRC>`_ and `Some statistics: <https://netsplit.de/networks/top10.php>`
.. _Proto 72:
`NDPI_PROTOCOL_HART_IP`
=======================
Highway Addressable Remote Transducer over IP
References: `Protocol Specs: <https://library.fieldcommgroup.org/20085/TS20085>`_
.. _Proto 96:
`NDPI_PROTOCOL_TFTP`
====================
TFTP is a very simple protocol used to transfer files. It is from this that its name comes, Trivial File Transfer Protocol or TFTP.
References: `RFC1350 <https://datatracker.ietf.org/doc/html/rfc1350>`_ and `RFC2347 <https://datatracker.ietf.org/doc/html/rfc2347>`_ and `RFC2349 <https://datatracker.ietf.org/doc/html/rfc2349>`_
.. _Proto 127:
`NDPI_PROTOCOL_MS_RPCH`
=======================
A Remote Procedure Call protocol over HTTP from Microsoft.
References: `Protocol Specs: <https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-rpch/c0f4c9c5-1a61-4d10-b8e2-005378d1d212>`_
.. _Proto 149:
`NDPI_PROTOCOL_EGD`
=========================
Ethernet Global Data (EGD) is a communication protocol developed by GE Fanuc Automation for real-time data exchange between automation devices and control systems using standard Ethernet technology. It is widely adopted in industrial environments for its simplicity and reliability in data communication.
References: `Wikipedia <https://en.wikipedia.org/wiki/Ethernet_Global_Data_Protocol>`_
Notes:
- This dissector only works for data packets, not configuration commands.
- IPv6 is not supported
.. _Proto 182:
`NDPI_PROTOCOL_RESP`
=======================
Redis Serialization Protocol
References: `Protocol Specs: <https://redis.io/docs/reference/protocol-spec/>`_
.. _Proto 186:
`NDPI_PROTOCOL_COD_MOBILE`
=======================
Call of Duty: Mobile is a free-to-play shooter game for iOS and Android devices. It has amassed over 650 million downloads worldwide, making it one of the most popular mobile games.
References: `Main site: <https://www.callofduty.com/mobile/>`_
.. _Proto 201:
`NDPI_PROTOCOL_GOOGLE_MEET`
===========================
Google Meet is a video conferencing service from Google.
References: `Main site: <https://meet.google.com/>`_
.. _Proto 235:
`NDPI_PROTOCOL_VALVE_SDR`
===========================
Steam Datagram Relay (SDR) is Valve's virtual private gaming network. Used in all modern games from Valve, but can also be used by developers via the Steamworks SDK for their games.
References: `Main site: <https://partner.steamgames.com/doc/features/multiplayer/steamdatagramrelay>`_
.. _Proto 338:
`NDPI_PROTOCOL_SRTP`
====================
The Secure Real-time Transport Protocol (SRTP) is a profile for Real-time Transport Protocol (RTP) intended to provide encryption, message authentication, integrity, and replay attack protection to the RTP data.
References: `RFC3711 <https://datatracker.ietf.org/doc/html/rfc3711>`_.
References: `RFC3711 <https://datatracker.ietf.org/doc/html/rfc3711>`_
Notes:
@@ -25,7 +138,7 @@ Notes:
=========================
Opera VPN is a free VPN included with Opera Browser.
References: `Main site <https://www.opera.com/it/features/free-vpn>`_.
References: `Main site <https://www.opera.com/it/features/free-vpn>`_
.. _Proto 340:
@@ -34,7 +147,7 @@ References: `Main site <https://www.opera.com/it/features/free-vpn>`_.
=========================
Epic Games is a video game company developing the Unreal Engine and some successful games as Fortnite and Gears of War.
References: `Main site <https://store.epicgames.com/en-US/>`_ and `Fortnite <https://www.fortnite.com/>`_.
References: `Main site <https://store.epicgames.com/en-US/>`_ and `Fortnite <https://www.fortnite.com/>`_
.. _Proto 341:
@@ -43,7 +156,7 @@ References: `Main site <https://store.epicgames.com/en-US/>`_ and `Fortnite <htt
==========================
GeForce Now is the brand used by Nvidia for its cloud gaming service.
References: `Main site <https://www.nvidia.com/en-us/geforce-now/>`_.
References: `Main site <https://www.nvidia.com/en-us/geforce-now/>`_
.. _Proto 342:
@@ -52,7 +165,7 @@ References: `Main site <https://www.nvidia.com/en-us/geforce-now/>`_.
======================
Generic web traffic from Nvidia sites.
References: `Main site <https://www.nvidia.com>`_.
References: `Main site <https://www.nvidia.com>`_
.. _Proto 343:
@@ -61,7 +174,7 @@ References: `Main site <https://www.nvidia.com>`_.
=======================
Bitcoin is one of the most common crypto currencies.
References: `Main site <https://en.bitcoin.it/wiki/Protocol_documentation>`_.
References: `Main site <https://en.bitcoin.it/wiki/Protocol_documentation>`_
Notes:
@@ -76,7 +189,7 @@ Notes:
=========================
Proton VPN is a VPN service operated by the Swiss company Proton AG, the company behind the email service Proton Mail
References: `Main site <https://protonvpn.com/>`_.
References: `Main site <https://protonvpn.com/>`_
.. _Proto 345:
@@ -85,7 +198,7 @@ References: `Main site <https://protonvpn.com/>`_.
======================
Apache Thrift is a generic data interchange framework that supports a bunch of different languages and platforms.
References: `Official site <https://thrift.apache.org>`_ and `Github <https://github.com/apache/thrift>`_.
References: `Official site <https://thrift.apache.org>`_ and `Github <https://github.com/apache/thrift>`_
.. _Proto 346:
@@ -94,7 +207,7 @@ References: `Official site <https://thrift.apache.org>`_ and `Github <https://gi
======================
Roblox is an online game platform and game creation system.
References: `Main site <https://www.roblox.com/>`_.
References: `Main site <https://www.roblox.com/>`_
Notes:
@@ -107,7 +220,7 @@ Notes:
================================
The Service Location Protocol is a service discovery protocol that allows computers and other devices to find services in a local area network without prior configuration.
References: `SLPv1 <https://datatracker.ietf.org/doc/html/rfc2165>`_ and `SLPv2 <https://datatracker.ietf.org/doc/html/rfc2608>`_.
References: `SLPv1 <https://datatracker.ietf.org/doc/html/rfc2165>`_ and `SLPv2 <https://datatracker.ietf.org/doc/html/rfc2608>`_
.. _Proto 348:
@@ -116,7 +229,7 @@ References: `SLPv1 <https://datatracker.ietf.org/doc/html/rfc2165>`_ and `SLPv2
=======================
Mullvad is a VPN service operated by Mullvad VPN AB, based in Sweden
References: `Main site <https://mullvad.net/>`_.
References: `Main site <https://mullvad.net/>`_
.. _Proto 349:
@@ -125,7 +238,7 @@ References: `Main site <https://mullvad.net/>`_.
=====================
HTTP/2 (originally named HTTP/2.0) is a major revision of the HTTP network protocol used by the World Wide Web.
References: `RFC <https://datatracker.ietf.org/doc/html/rfc9113>`_.
References: `RFC <https://datatracker.ietf.org/doc/html/rfc9113>`_
Notes:
@@ -138,7 +251,7 @@ Notes:
=======================
HAProxy is a free and open source software that provides a high availability load balancer and reverse proxy for TCP and HTTP-based applications that spreads requests across multiple servers.
References: `Main site: <https://www.haproxy.org>`_.
References: `Main site: <https://www.haproxy.org>`_
.. _Proto 351:
@@ -147,4 +260,599 @@ References: `Main site: <https://www.haproxy.org>`_.
====================
The Intelligent Platform Management Interface (IPMI) is a set of computer interface specifications for an autonomous computer subsystem that provides management and monitoring capabilities independently of the host system's CPU, firmware (BIOS or UEFI) and operating system.
References: `Protocol Specs: <https://www.dmtf.org/sites/default/files/standards/documents/DSP0114.pdf>`_.
References: `Protocol Specs: <https://www.dmtf.org/sites/default/files/standards/documents/DSP0114.pdf>`_
.. _Proto 352:
`NDPI_PROTOCOL_CAN`
===================
Controller Area Network (CAN) is used extensively in automotive applications, with in excess of 400 million CAN enabled microcontrollers manufactured each year.
References: `Protocol Specs: <https://www.iso.org/standard/63648.html>`_
.. _Proto 353:
`NDPI_PROTOCOL_PROTOBUF`
========================
Protocol Buffers (Protobuf) is a free and open-source cross-platform data format used to serialize structured data.
References: `Encoding: <https://protobuf.dev/programming-guides/encoding>`_
.. _Proto 354:
`NDPI_PROTOCOL_ETHEREUM`
=======================
Ethereum is a decentralized, open-source blockchain with smart contract functionality.
References: `Main site <https://ethereum.org/en/developers/docs/intro-to-ethereum/>`_
Notes:
- same as Bitcoin, not each crypto exchange is a mining, it could be a normal transaction, sending or receving or even blockchain exploration.
.. _Proto 355:
`NDPI_PROTOCOL_TELEGRAM_VOIP`
============================
Audio/video calls made using the telegram app.
References: `Wikipedia <https://en.wikipedia.org/wiki/telegram_(software)/>`_
.. _Proto 356:
`NDPI_PROTOCOL_SINA_WEIBO`
============================
Chinese microblogging (weibo) website.
References: `Wikipedia <https://en.wikipedia.org/wiki/Sina_Weibo>`_
.. _Proto 358:
`NDPI_PROTOCOL_PTPV2`
============================
IEEE 1588-2008 Precision Time Protocol (PTP) Version 2.
References: `Protocol Specs: <https://standards.ieee.org/ieee/1588/4355/>`_
.. _Proto 359:
`NDPI_PROTOCOL_RTPS`
============================
Real-Time Publish Subscribe Protocol
References: `Protocol Specs: <https://www.omg.org/spec/DDSI-RTPS/>`_
.. _Proto 360:
`NDPI_PROTOCOL_OPC_UA`
============================
IEC62541 OPC Unified Architecture
References: `Protocol Specs: <https://reference.opcfoundation.org/>`_
.. _Proto 361:
`NDPI_PROTOCOL_S7COMM_PLUS`
============================
A proprietary protocol from Siemens used for data exchange between PLCs and access PLC data via SCADA systems.
Completely different from classic S7Comm, but also uses TPKT/COTP as a transport.
References: `Unofficial description: <https://plc4x.apache.org/protocols/s7/s7comm-plus.html>`_
.. _Proto 362:
`NDPI_PROTOCOL_FINS`
============================
Factory Interface Network Service (FINS) is a network protocol used by Omron PLCs.
References: `Protocol Specs: <https://assets.omron.eu/downloads/manual/en/v4/w421_cj1w-etn21_cs1w-etn21_ethernet_units_-_construction_of_applications_operation_manual_en.pdf>`_
.. _Proto 363:
`NDPI_PROTOCOL_ETHERSIO`
============================
Ether-S-I/O is a proprietary protocol used by Saia-Burgess's PLCs.
References: `Wireshark wiki: <https://wiki.wireshark.org/EtherSIO.md>`_
.. _Proto 364:
`NDPI_PROTOCOL_UMAS`
============================
UMAS is a proprietary Schneider Electric protocol based on Modbus. It's used in Modicon M580 and Modicon M340 CPU-based PLCs.
References: `Unofficial article: <https://ics-cert.kaspersky.com/publications/reports/2022/09/29/the-secrets-of-schneider-electrics-umas-protocol/>`_
.. _Proto 365:
`NDPI_PROTOCOL_BECKHOFF_ADS`
============================
Automation Device Specification is the protocol used for interfacing with Beckhoff PLCs via TwinCAT.
References: `Protocol Specs: <https://infosys.beckhoff.com/english.php?content=../content/1033/tc3_ads_intro/115847307.html>`_
.. _Proto 366:
`NDPI_PROTOCOL_ISO9506_1_MMS`
============================
The international standard MMS (Manufacturing Message Specification) is an OSI application layer messaging protocol origionally designed for the remote control and monitoring of devices such as Remote Terminal Units (RTU), Programmable Logic Controllers (PLC), Numerical Controllers (NC), or Robot Controllers (RC).
References: `Paid Specs: <https://www.iso.org/ru/standard/37079.html>`_
.. _Proto 367:
`NDPI_PROTOCOL_IEEE_C37118`
============================
IEEE Standard for Synchrophasor Data Transfer for Power Systems
References: `Paid Specs: <https://standards.ieee.org/ieee/C37.118.1/4902/>`_
.. _Proto 368:
`NDPI_PROTOCOL_ETHERSBUS`
============================
Ether-S-Bus is a proprietary protocol used for the communication with and between PLCs manufactured by Saia-Burgess Controls Ltd.
References: `Wireshark wiki: <https://wiki.wireshark.org/EtherSBus>`_
.. _Proto 369:
`NDPI_PROTOCOL_MONERO`
======================
Monero is a private and decentralized cryptocurrency with focus on confidentiality and security.
.. _Proto 370:
`NDPI_PROTOCOL_DCERPC`
======================
DCE/RPC is a specification for a remote procedure call mechanism that defines both APIs and an over-the-network protocol.
References: `Wireshark wiki: <https://wiki.wireshark.org/DCE/RPC>`_
.. _Proto 371:
`NDPI_PROTOCOL_PROFINET_IO`
===========================
PROFINET/IO is a field bus protocol based on connectionless DCE/RPC.
References: `Protocol Specs: <https://www.profibus.com/download/profinet-specification>`_
.. _Proto 372:
`NDPI_PROTOCOL_HISLIP`
======================
High-Speed LAN Instrument Protocol (HiSLIP) is a protocol for remote instrument control of LAN-based test and measurement instruments.
References: `Protocol Specs: <https://www.ivifoundation.org/downloads/Protocol%20Specifications/IVI-6.1_HiSLIP-2.0-2020-04-23.pdf>`_
.. _Proto 373:
`NDPI_PROTOCOL_UFTP`
====================
Encrypted UDP based FTP with multicast.
References: `Protocol Specs: <https://uftp-multicast.sourceforge.net/protocol.txt>`_.
.. _Proto 374:
`NDPI_PROTOCOL_OPENFLOW`
========================
OpenFlow protocol is a network protocol closely associated with Software-Defined Networking (SDN).
References: `Protocol Specs: <https://opennetworking.org/wp-content/uploads/2014/10/openflow-switch-v1.5.1.pdf>`_
.. _Proto 375:
`NDPI_PROTOCOL_JSON_RPC`
========================
JSON-RPC is a remote procedure call protocol encoded in JSON.
References: `Protocol Specs: <https://www.jsonrpc.org/specification>`_
.. _Proto 376:
`NDPI_PROTOCOL_WEBDAV`
======================
WebDAV is a set of extensions to the HTTP protocol that allows WebDAV clients to collaboratively edit and manage files on remote Web servers.
References: `RFC4918: <https://datatracker.ietf.org/doc/html/rfc4918>`_
Notes:
- WebDAV is almost always encrypted, i.e. transported over TLS.
.. _Proto 377:
`NDPI_PROTOCOL_APACHE_KAFKA`
============================
Apache Kafka is a distributed event store and stream-processing platform.
References: `Official site <https://kafka.apache.org>`_ and `Github <https://github.com/apache/kafka>`_
.. _Proto 378:
`NDPI_PROTOCOL_NOMACHINE`
=========================
NoMachine is a popular proprietary remote desktop software.
References: `Main site <https://www.nomachine.com/>`_
.. _Proto 379:
`NDPI_PROTOCOL_IEC62056`
============================
IEC 62056-4-7 DLMS/COSEM is a transport layer for IP networks.
References: `Paid Specs: <https://webstore.iec.ch/publication/22487>`_
Notes:
- Wireshark is not able to recognize this protocol. Some old plugins/code (with some documentation) are available `here <https://github.com/bearxiong99/wireshark-dlms>` and `here <https://github.com/matousp/dlms-analysis/tree/master>`.
.. _Proto 380:
`NDPI_PROTOCOL_HL7`
=========================
HL7 is a range of global standards for the transfer of clinical and administrative health data between applications.
References: `Main site <https://www.hl7.org/>`_
.. _Proto 381:
`NDPI_PROTOCOL_CEPH`
=========================
Ceph is a scalable distributed storage system.
References: `Main site <https://ceph.io/en/>`_
.. _Proto 382:
`NDPI_PROTOCOL_GOOGLE_CHAT`
=========================
Google Chat is an instant messaging service from Google, which replaced Hangouts.
References: `Main site <https://chat.google.com/>`_
.. _Proto 383:
`NDPI_PROTOCOL_ROUGHTIME`
=========================
A protocol that aims to achieve rough time synchronization while detecting servers that provide inaccurate time and providing cryptographic proof of their malfeasance.
References: `IETF Draft <https://www.ietf.org/archive/id/draft-ietf-ntp-roughtime-08.html>`_
.. _Proto 384:
`NDPI_PROTOCOL_PIA`
=========================
Private Internet Access (PIA) is a popular VPN service from Kape Technologies.
References: `Main site <https://www.privateinternetaccess.com/>`_
.. _Proto 385:
`NDPI_PROTOCOL_KCP`
===================
KCP - A Fast and Reliable ARQ Protocol. It provides TCP-like stream support with low latency at the cost of bandwidth usage - used by lot's of Open Source / Third Party applications.
References: `Protocol Specs: <https://github.com/skywind3000/kcp/blob/master/protocol.txt>`_
.. _Proto 386:
`NDPI_PROTOCOL_DOTA2`
=========================
Dota 2 is an extremely popular multiplayer MOBA game from Valve.
References: `Main site <https://www.dota2.com/>`_
.. _Proto 387:
`NDPI_PROTOCOL_MUMBLE`
=========================
Mumble is a free, open source, low latency, high quality voice chat application.
References: `Main site <https://www.mumble.info/>`_
.. _Proto 388:
`NDPI_PROTOCOL_YOJIMBO`
=======================
Yojimbo (netcode) is a secure connection-based client/server protocol built on top of UDP.
References: `Protocol Specs: <https://github.com/mas-bandwidth/netcode/blob/main/STANDARD.md>`_
.. _Proto 389:
`NDPI_PROTOCOL_ELECTRONICARTS`
=========================
Electronic Arts is a leading publisher of games on Console, PC and Mobile.
References: `Main site <https://www.ea.com/>`_
Notes:
- Almost all of that traffic is related to their EA Origin game store.
.. _Proto 390:
`NDPI_PROTOCOL_STOMP`
========================
STOMP is a simple interoperable protocol designed for asynchronous message passing between clients via mediating servers. Supported in ActiveMQ and RabbitMQ.
References: `Protocol Specs: <https://stomp.github.io/stomp-specification-1.2.html>`_
.. _Proto 391:
`NDPI_PROTOCOL_RADMIN`
=========================
Radmin is remote access software for the Microsoft Windows platform.
References: `Main site <https://www.radmin.com/>`_
.. _Proto 392:
`NDPI_PROTOCOL_RAFT`
====================
Raft is a consensus algorithm and protocol for managing a replicated log.
References: `C implementation <https://github.com/canonical/raft>`_ and `Paper <https://raft.github.io/raft.pdf>`_
.. _Proto 394:
`NDPI_PROTOCOL_GEARMAN`
====================
Gearman is a network-based job-queuing system that was initially developed by Danga Interactive in order to process large volumes of jobs.
References: `Main site <http://gearman.org/>`_
.. _Proto 395:
`NDPI_PROTOCOL_TENCENTGAMES`
====================
A protocol used by various games from Tencent (mostly mobile games).
References: `Main site <https://www.tencentgames.com/>`_
.. _Proto 396:
`NDPI_PROTOCOL_GAIJIN`
====================
Protocols used in various games from Gaijin Entertainment.
References: `Main site <https://gaijin.net/>`_
.. _Proto 397:
`NDPI_PROTOCOL_C1222`
====================
ANSI C12.22 (IEEE Std 1703) describe a protocol for transporting ANSI C12.19 table data over networks. It's mostly used to communicate with electric meters.
References: `Paid specs <https://www.nema.org/Standards/view/American-National-Standard-for-Protocol-Specification-for-Interfacing-to-Data-Communication-Networks/>`_
.. _Proto 398:
`NDPI_PROTOCOL_HUAWEI`
======================
Generic Huawei traffic.
References: `Main site <https://www.huawei.com/>`_
.. _Proto 399:
`NDPI_PROTOCOL_HUAWEI_CLOUD`
============================
Huawei Mobile Cloud.
References: `Main site <https://cloud.huawei.com/>`_
.. _Proto 400:
`NDPI_PROTOCOL_DLEP`
=====================
The Dynamic Link ExchangeProtocol (DLEP) is a radio aware routing (RAR) protocol.
References: `RFC <https://datatracker.ietf.org/doc/html/rfc8175>`_
.. _Proto 401:
`NDPI_PROTOCOL_BFD`
=====================
Bidirectional Forwarding Detection is a network protocol that is used to detect faults between two routers or switches.
References: `RFC <https://datatracker.ietf.org/doc/html/rfc5880>`_
.. _Proto 402:
`NDPI_PROTOCOL_NETEASE_GAMES`
============================
Traffic of various NetEase games.
References: `Main site <https://www.neteasegames.com/>`_
.. _Proto 403:
`NDPI_PROTOCOL_PATHOFEXILE`
============================
Path of Exile is a free-to-play online Action RPG.
References: `Main site <https://pathofexile.com/>`_
.. _Proto 404:
`NDPI_PROTOCOL_GOOGLE_CALL`
===========================
Audio/video calls made by (any) Google applications (i.e Google Meet).
References: `Main site: <https://meet.google.com/>`_
Notes:
- nDPI usually uses different protocol ids for the generic application traffic and for its "realtime" traffic (examples: NDPI_PROTOCOL_MEET/NDPI_PROTOCOL_GOOGLE_CALL, NDPI_PROTOCOL_WHATSAPP/NDPI_PROTOCOL_WHATSAPP_CALL, ...)
.. _Proto 405:
`NDPI_PROTOCOL_PFCP`
=====================
PFCP is a protocol used for communicating between control plane (CP) and user plane (UP) functions in 4G and 5G networks.
References: `Protocol Specs: <https://www.etsi.org/deliver/etsi_ts/129200_129299/129244/16.05.00_60/ts_129244v160500p.pdf>`_
.. _Proto 406:
`NDPI_PROTOCOL_FLUTE`
=====================
File Delivery over Unidirectional Transport.
References: `RFC <https://datatracker.ietf.org/doc/html/rfc6726>`_
.. _Proto 407:
`NDPI_PROTOCOL_LOLWILDRIFT`
============================
League of Legends: Wild Rift is a mobile MOBA game.
References: `Main site <https://wildrift.leagueoflegends.com/>`_
.. _Proto 408:
`NDPI_PROTOCOL_TESO`
============================
The Elder Scrolls Online is a MMORPG set in the fantasy world of Tamriel.
References: `Main site <https://www.elderscrollsonline.com/>`_
.. _Proto 409:
`NDPI_PROTOCOL_LDP`
=====================
The Label Distribution Protocol (LDP) is a routing protocol used to establish and maintain label-switched paths in a Multiprotocol Label Switching (MPLS) network.
References: `RFC <https://datatracker.ietf.org/doc/html/rfc5036>`_
.. _Proto 410:
`NDPI_PROTOCOL_KNXNET_IP`
=====================
KNXnet/IP is a building automation protocol that enables the exchange of data and control information over IP networks, extending the KNX standard for home and building automation.
References: `Paid Specs: <https://webstore.ansi.org/standards/ds/dsiso225102019>`_
.. _Proto 411:
`NDPI_PROTOCOL_BLUESKY`
======================
Bluesky, also known as Bluesky Social, is a decentralized microblogging social platform.
References: `Main site: <https://bsky.app/>`_
.. _Proto 412:
`NDPI_PROTOCOL_MASTODON`
=======================
Mastodon is free and open-source software for running self-hosted social networking services. It has microblogging features similar to Twitter.
References: `Main site: <https://joinmastodon.org/>`_
.. _Proto 413:
`NDPI_PROTOCOL_THREADS`
======================
Threads is an online social media and social networking service operated by Meta Platforms.
References: `Main site: <https://www.threads.net>`_
.. _Proto 414:
`NDPI_PROTOCOL_VIBER_VOIP`
=========================
Audio/video calls made using the Viber app.
References: `Wikipedia <https://en.wikipedia.org/wiki/Viber>`_
.. _Proto 415:
`NDPI_PROTOCOL_ZUG`
=========================
The ZUG protocol is part of the Casper 2.0 consensus model.
References: `Main Site <https://casperlabs.io>`_ and `Blog Post <https://casperlabs.io/blog/beyond-eth-30-theres-casper-20>`_
.. _Proto 416:
`NDPI_PROTOCOL_JRMI`
=========================
The JRMI protocol is the Java Remote Method Invocation protocol.
References: `Oracle site <https://docs.oracle.com/en/java/javase/21/docs/specs/rmi/protocol.html>`_
.. _Proto 417:
`NDPI_PROTOCOL_RIPE_ATLAS`
==========================
The RIPE Atlas probe protocol is used for the world's largest active Internet measurement network.
References: `Main Site <https://atlas.ripe.net/>`_ and `Documentation <https://ripe-atlas-tools.readthedocs.io/en/latest/index.html>`_
+3
View File
@@ -1,6 +1,9 @@
AR=@AR@
CC=@CC@
CXX=@CXX@
ifneq ($(OS),Windows_NT)
OS := $(shell uname)
endif
ifeq ($(OS),Darwin)
CC=clang -fno-color-diagnostics
endif
+3
View File
@@ -1,3 +1,6 @@
# Format: name\tcategory_id
internetbadguys.com 100
144.139.247.220 100
172.30.69.0/24 100
[2001:1670:8:40a6:a08e:332b:aa69:18dc] 100
[2001:db8:1::1]/127 100
+13
View File
@@ -0,0 +1,13 @@
#Format: <proto, param, value>
#proto might be empty
#Basic example of how to set nDPI configuration parameters via file
#We set only some default values
#See doc/configuration_parameters.md for a complete list and description of all the accepted knobs
packets_limit_per_flow,32
ookla,dpi.aggressiveness,0x1
tls,metadata.sha1_fingerprint,1
lru.bittorrent.ttl,0
+965 -426
View File
File diff suppressed because it is too large Load Diff
+46 -28
View File
@@ -188,8 +188,7 @@ static struct nDPI_workflow * init_workflow(char const * const file_or_device)
printf("pcap_setfilter error: '%s'\n", pcap_geterr(workflow->pcap_handle));
}
ndpi_init_prefs init_prefs = ndpi_no_prefs;
workflow->ndpi_struct = ndpi_init_detection_module(init_prefs);
workflow->ndpi_struct = ndpi_init_detection_module(NULL);
if (workflow->ndpi_struct == NULL) {
free_workflow(&workflow);
return NULL;
@@ -230,6 +229,7 @@ static void ndpi_flow_info_freer(void * const node)
static void free_workflow(struct nDPI_workflow ** const workflow)
{
struct nDPI_workflow * const w = *workflow;
size_t i;
if (w == NULL) {
return;
@@ -243,7 +243,7 @@ static void free_workflow(struct nDPI_workflow ** const workflow)
if (w->ndpi_struct != NULL) {
ndpi_exit_detection_module(w->ndpi_struct);
}
for(size_t i = 0; i < w->max_active_flows; i++) {
for(i = 0; i < w->max_active_flows; i++) {
ndpi_tdestroy(w->ndpi_flows_active[i], ndpi_flow_info_freer);
}
ndpi_free(w->ndpi_flows_active);
@@ -272,6 +272,7 @@ static int setup_reader_threads(char const * const file_or_device)
{
char * file_or_default_device;
char pcap_error_buffer[PCAP_ERRBUF_SIZE];
int i;
if (reader_thread_count > MAX_READER_THREADS) {
return 1;
@@ -290,7 +291,7 @@ static int setup_reader_threads(char const * const file_or_device)
}
}
for (int i = 0; i < reader_thread_count; ++i) {
for (i = 0; i < reader_thread_count; ++i) {
reader_threads[i].workflow = init_workflow(file_or_default_device);
if (reader_threads[i].workflow == NULL)
{
@@ -500,8 +501,10 @@ static int ndpi_workflow_node_cmp(void const * const A, void const * const B) {
static void check_for_idle_flows(struct nDPI_workflow * const workflow)
{
size_t idle_scan_index;
if (workflow->last_idle_scan_time + IDLE_SCAN_PERIOD < workflow->last_time) {
for (size_t idle_scan_index = 0; idle_scan_index < workflow->max_active_flows; ++idle_scan_index) {
for (idle_scan_index = 0; idle_scan_index < workflow->max_active_flows; ++idle_scan_index) {
ndpi_twalk(workflow->ndpi_flows_active[idle_scan_index], ndpi_idle_scan_walker, workflow);
while (workflow->cur_idle_flows > 0) {
@@ -530,7 +533,7 @@ static void ndpi_process_packet(uint8_t * const args,
struct nDPI_reader_thread * const reader_thread =
(struct nDPI_reader_thread *)args;
struct nDPI_workflow * workflow;
struct nDPI_flow_info flow = {};
struct nDPI_flow_info flow;
size_t hashed_index;
void * tree_result;
@@ -551,6 +554,8 @@ static void ndpi_process_packet(uint8_t * const args,
uint16_t type;
uint32_t thread_index = INITIAL_THREAD_HASH; // generated with `dd if=/dev/random bs=1024 count=1 |& hd'
memset(&flow, '\0', sizeof(flow));
if (reader_thread == NULL) {
return;
}
@@ -736,24 +741,33 @@ static void ndpi_process_packet(uint8_t * const args,
print_packet_info(reader_thread, header, l4_len, &flow);
#endif
/* calculate flow hash for btree find, search(insert) */
if (flow.l3_type == L3_IP) {
if (ndpi_flowv4_flow_hash(flow.l4_protocol, flow.ip_tuple.v4.src, flow.ip_tuple.v4.dst,
flow.src_port, flow.dst_port, 0, 0,
(uint8_t *)&flow.hashval, sizeof(flow.hashval)) != 0)
{
uint64_t tmp[4] = {};
/* calculate flow hash for btree find, search(insert) */
if (flow.l3_type == L3_IP) {
if (ndpi_flowv4_flow_hash(flow.l4_protocol, flow.ip_tuple.v4.src, flow.ip_tuple.v4.dst,
flow.src_port, flow.dst_port, 0, 0,
(uint8_t *)&tmp[0], sizeof(tmp)) != 0)
{
flow.hashval = flow.ip_tuple.v4.src + flow.ip_tuple.v4.dst; // fallback
flow.hashval = flow.ip_tuple.v4.src + flow.ip_tuple.v4.dst; // fallback
} else {
flow.hashval = tmp[0] + tmp[1] + tmp[2] + tmp[3];
}
} else if (flow.l3_type == L3_IP6) {
if (ndpi_flowv6_flow_hash(flow.l4_protocol, &ip6->ip6_src, &ip6->ip6_dst,
flow.src_port, flow.dst_port, 0, 0,
(uint8_t *)&flow.hashval, sizeof(flow.hashval)) != 0)
} else if (flow.l3_type == L3_IP6) {
if (ndpi_flowv6_flow_hash(flow.l4_protocol, &ip6->ip6_src, &ip6->ip6_dst,
flow.src_port, flow.dst_port, 0, 0,
(uint8_t *)&tmp[0], sizeof(tmp)) != 0)
{
flow.hashval = flow.ip_tuple.v6.src[0] + flow.ip_tuple.v6.src[1];
flow.hashval += flow.ip_tuple.v6.dst[0] + flow.ip_tuple.v6.dst[1];
flow.hashval = flow.ip_tuple.v6.src[0] + flow.ip_tuple.v6.src[1];
flow.hashval += flow.ip_tuple.v6.dst[0] + flow.ip_tuple.v6.dst[1];
} else {
flow.hashval = tmp[0] + tmp[1] + tmp[2] + tmp[3];
}
}
flow.hashval += flow.l4_protocol + flow.src_port + flow.dst_port;
}
flow.hashval += flow.l4_protocol + flow.src_port + flow.dst_port;
hashed_index = flow.hashval % workflow->max_active_flows;
tree_result = ndpi_tfind(&flow, &workflow->ndpi_flows_active[hashed_index], ndpi_workflow_node_cmp);
@@ -866,7 +880,7 @@ static void ndpi_process_packet(uint8_t * const args,
flow_to_process->guessed_protocol =
ndpi_detection_giveup(workflow->ndpi_struct,
flow_to_process->ndpi_flow,
1, &protocol_was_guessed);
&protocol_was_guessed);
if (protocol_was_guessed != 0) {
printf("[%8llu, %d, %4d][GUESSED] protocol: %s | app protocol: %s | category: %s\n",
workflow->packets_captured,
@@ -886,8 +900,7 @@ static void ndpi_process_packet(uint8_t * const args,
ip != NULL ? (uint8_t *)ip : (uint8_t *)ip6,
ip_size, time_ms, NULL);
if (ndpi_is_protocol_detected(workflow->ndpi_struct,
flow_to_process->detected_l7_protocol) != 0 &&
if (ndpi_is_protocol_detected(flow_to_process->detected_l7_protocol) != 0 &&
flow_to_process->detection_completed == 0)
{
if (flow_to_process->detected_l7_protocol.master_protocol != NDPI_PROTOCOL_UNKNOWN ||
@@ -1019,7 +1032,9 @@ static void * processing_thread(void * const ndpi_thread_arg)
static int processing_threads_error_or_eof(void)
{
for (int i = 0; i < reader_thread_count; ++i) {
int i;
for (i = 0; i < reader_thread_count; ++i) {
if (__sync_fetch_and_add(&reader_threads[i].workflow->error_or_eof, 0) == 0) {
return 0;
}
@@ -1029,6 +1044,8 @@ static int processing_threads_error_or_eof(void)
static int start_reader_threads(void)
{
int i;
#ifndef WIN32
sigset_t thread_signal_set, old_signal_set;
@@ -1041,7 +1058,7 @@ static int start_reader_threads(void)
}
#endif
for (int i = 0; i < reader_thread_count; ++i) {
for (i = 0; i < reader_thread_count; ++i) {
reader_threads[i].array_index = i;
if (reader_threads[i].workflow == NULL) {
@@ -1067,6 +1084,7 @@ static int start_reader_threads(void)
static int stop_reader_threads(void)
{
int i;
unsigned long long int total_packets_captured = 0;
unsigned long long int total_packets_processed = 0;
unsigned long long int total_l4_data_len = 0;
@@ -1074,13 +1092,13 @@ static int stop_reader_threads(void)
unsigned long long int total_flows_idle = 0;
unsigned long long int total_flows_detected = 0;
for (int i = 0; i < reader_thread_count; ++i) {
for (i = 0; i < reader_thread_count; ++i) {
break_pcap_loop(&reader_threads[i]);
}
printf("------------------------------------ Stopping reader threads\n");
for (int i = 0; i < reader_thread_count; ++i) {
for (i = 0; i < reader_thread_count; ++i) {
if (reader_threads[i].workflow == NULL) {
continue;
}
@@ -1105,7 +1123,7 @@ static int stop_reader_threads(void)
/* total packets captured: same value for all threads as packet2thread distribution happens later */
total_packets_captured = reader_threads[0].workflow->packets_captured;
for (int i = 0; i < reader_thread_count; ++i) {
for (i = 0; i < reader_thread_count; ++i) {
if (reader_threads[i].workflow == NULL) {
continue;
}
@@ -1137,7 +1155,7 @@ static void sighandler(int signum)
int main(int argc, char ** argv)
{
if (argc == 0) {
printf("usage: ndpiSimpleIntegration Mdevice name>\n");
printf("usage: ndpiSimpleIntegration <device name>\n");
return 1;
}
+20 -2
View File
@@ -27,6 +27,7 @@ host:"api-global.netflix.com"@Netflix
# IP based Subprotocols
# Format:
# ip:<value>,ip:<value>,.....@<subproto>
# ipv6:[<value>],ipv6:[<value>],.....@<subproto>
#
# NOTES
@@ -45,14 +46,29 @@ host:"api-global.netflix.com"@Netflix
# detected as <L7 proto>.Google but only
# as <L7 proto>
#
ip:213.75.170.11/32:443@CustomProtocol
ip:8.248.73.247:443@AmazonPrime
ip:54.80.47.130@AmazonPrime
#You can specify a protocol Id. In that case you probably want to avoid conflict with internal ids.
#You can use any number up to 65535
ip:3.3.3.3:443@CustomProtocolA
ip:3.3.3.3:444@CustomProtocolB
ip:3.3.3.3:446@CustomProtocolC=400
ip:3.3.3.3:446@CustomProtocolC=800
ipv6:[3ffe:507:0:1:200:86ff:fe05:80da]@CustomProtocolD=1024
ipv6:[247f:855b:5e16:3caf::]/64:100@CustomProtocolE=2048
ipv6:[247f:855b:5e16:3caf::]/64@CustomProtocolF=2049
ipv6:[fe80::76ac:b9ff:fe6c:c124]:12717@CustomProtocolG=2050
ipv6:[fe80::76ac:b9ff:fe6c:c124]:12718@CustomProtocolH=65535
ipv6:[fe80::76ac:b9ff:fe6c:c124]:12719@CustomProtocolI=65534
#
# You can use symbolic IP addreses if you want
#
ip:www.ntop.org@ntop
ipv6:www.ntop.org@ntop
#
# Risk Exceptions
@@ -67,6 +83,8 @@ ip:3.3.3.3:446@CustomProtocolC=400
#ip_risk_mask:192.168.1.0/24=0
ip_risk_mask:10.10.120.0/24=0
ip_risk_mask:10.196.157.228=0
ipv6_risk_mask:[fe80::356b:e047:3695:0]/112=0
ipv6_risk_mask:[fe80::7c0:e74e:87c3:5d93]=0
host_risk_mask:".home"=0
+48 -108
View File
@@ -1,7 +1,7 @@
/*
* reader_util.c
*
* Copyright (C) 2011-23 - ntop.org
* Copyright (C) 2011-24 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
@@ -42,6 +42,7 @@
#include <netinet/in.h>
#include <netinet/ip.h>
#endif
#include <assert.h>
#include "reader_util.h"
@@ -73,18 +74,14 @@
#include "reader_util.h"
#include "ndpi_classify.h"
extern u_int8_t enable_protocol_guess, enable_flow_stats, enable_payload_analyzer;
extern u_int8_t enable_flow_stats, enable_payload_analyzer;
extern u_int8_t verbose, human_readeable_string_len;
extern u_int8_t max_num_udp_dissected_pkts /* 24 */, max_num_tcp_dissected_pkts /* 80 */;
static u_int32_t flow_id = 0;
u_int8_t enable_doh_dot_detection = 0;
extern ndpi_init_prefs init_prefs;
extern int malloc_size_stats;
extern struct ndpi_bin malloc_bins;
extern int max_malloc_bins;
extern int enable_malloc_bins;
/* ****************************************************** */
@@ -118,7 +115,6 @@ u_int16_t max_pattern_len = 8;
/* *********************************************************** */
int ndpi_analyze_payload(struct ndpi_flow_info *flow,
u_int8_t src_to_dst_direction,
u_int8_t *payload,
u_int16_t payload_len,
u_int32_t packet_id) {
@@ -129,7 +125,7 @@ int ndpi_analyze_payload(struct ndpi_flow_info *flow,
#ifdef DEBUG_PAYLOAD
u_int16_t i;
for(i=0; i<payload_len; i++)
printf("%c", isprint(payload[i]) ? payload[i] : '.');
printf("%c", ndpi_isprint(payload[i]) ? payload[i] : '.');
printf("\n");
#endif
@@ -198,7 +194,6 @@ int ndpi_analyze_payload(struct ndpi_flow_info *flow,
/* *********************************************************** */
void ndpi_payload_analyzer(struct ndpi_flow_info *flow,
u_int8_t src_to_dst_direction,
u_int8_t *payload, u_int16_t payload_len,
u_int32_t packet_id) {
u_int16_t i, j;
@@ -219,7 +214,7 @@ void ndpi_payload_analyzer(struct ndpi_flow_info *flow,
for(i=0; i<scan_len; i++) {
for(j=min_pattern_len; j <= max_pattern_len; j++) {
if((i+j) < payload_len) {
if(ndpi_analyze_payload(flow, src_to_dst_direction, &payload[i], j, packet_id) == -1) {
if(ndpi_analyze_payload(flow, &payload[i], j, packet_id) == -1) {
LOG(NDPI_LOG_ERROR, "Error ndpi_analyze_payload (allocation failure)\n");
}
}
@@ -247,7 +242,7 @@ static void print_payload_stat(struct payload_stats *p, FILE *out) {
fprintf(out, "\t[");
for(i=0; i<p->pattern_len; i++) {
fprintf(out, "%c", isprint(p->pattern[i]) ? p->pattern[i] : '.');
fprintf(out, "%c", ndpi_isprint(p->pattern[i]) ? p->pattern[i] : '.');
}
fprintf(out, "]");
@@ -255,7 +250,7 @@ static void print_payload_stat(struct payload_stats *p, FILE *out) {
fprintf(out, "[");
for(i=0; i<p->pattern_len; i++) {
fprintf(out, "%s%02X", (i > 0) ? " " : "", isprint(p->pattern[i]) ? p->pattern[i] : '.');
fprintf(out, "%s%02X", (i > 0) ? " " : "", ndpi_isprint(p->pattern[i]) ? p->pattern[i] : '.');
}
fprintf(out, "]");
@@ -336,45 +331,6 @@ void ndpi_free_flow_info_half(struct ndpi_flow_info *flow) {
/* ***************************************************** */
extern u_int32_t current_ndpi_memory, max_ndpi_memory;
static u_int32_t __slot_malloc_bins(u_int64_t v)
{
int i;
/* 0-2,3-4,5-8,9-16,17-32,33-64,65-128,129-256,257-512,513-1024,1025-2048,2049-4096,4097-8192,8193- */
for(i=0; i < max_malloc_bins - 1; i++)
if((1ULL << (i + 1)) >= v)
return i;
return i;
}
/**
* @brief ndpi_malloc wrapper function
*/
static void *ndpi_malloc_wrapper(size_t size) {
current_ndpi_memory += size;
if(current_ndpi_memory > max_ndpi_memory)
max_ndpi_memory = current_ndpi_memory;
if(enable_malloc_bins && malloc_size_stats)
ndpi_inc_bin(&malloc_bins, __slot_malloc_bins(size), 1);
return(malloc(size)); /* Don't change to ndpi_malloc !!!!! */
}
/* ***************************************************** */
/**
* @brief free wrapper function
*/
static void free_wrapper(void *freeable) {
free(freeable); /* Don't change to ndpi_free !!!!! */
}
/* ***************************************************** */
static uint16_t ndpi_get_proto_id(struct ndpi_detection_module_struct *ndpi_mod, const char *name) {
uint16_t proto_id;
char *e;
@@ -410,7 +366,7 @@ int parse_proto_name_list(char *str, NDPI_PROTOCOL_BITMASK *bitmask, int inverte
else
op = 0; /* Default action: remove from the bitmask */
/* Use a temporary module with all protocols enabled */
module = ndpi_init_detection_module(0);
module = ndpi_init_detection_module(NULL);
if(!module)
return 1;
NDPI_BITMASK_SET_ALL(all);
@@ -449,24 +405,14 @@ int parse_proto_name_list(char *str, NDPI_PROTOCOL_BITMASK *bitmask, int inverte
/* ***************************************************** */
extern char *_debug_protocols;
struct ndpi_workflow* ndpi_workflow_init(const struct ndpi_workflow_prefs * prefs,
pcap_t * pcap_handle, int do_init_flows_root,
ndpi_serialization_format serialization_format) {
ndpi_serialization_format serialization_format,
struct ndpi_global_context *g_ctx) {
struct ndpi_detection_module_struct * module;
struct ndpi_workflow * workflow;
static NDPI_PROTOCOL_BITMASK debug_bitmask;
static int _debug_protocols_ok = 0;
/* On some fuzzers we don't want to use these memory allocators, but some custom ones */
#ifndef DISABLE_CUSTOM_ALLOCATOR_ON_READERUTILS
set_ndpi_malloc(ndpi_malloc_wrapper), set_ndpi_free(free_wrapper);
set_ndpi_flow_malloc(NULL), set_ndpi_flow_free(NULL);
#endif
/* TODO: just needed here to init ndpi ndpi_malloc wrapper */
module = ndpi_init_detection_module(init_prefs);
module = ndpi_init_detection_module(g_ctx);
if(module == NULL) {
LOG(NDPI_LOG_ERROR, "global structure initialization failed\n");
@@ -486,20 +432,6 @@ struct ndpi_workflow* ndpi_workflow_init(const struct ndpi_workflow_prefs * pref
ndpi_set_user_data(module, workflow);
ndpi_set_log_level(module, nDPI_LogLevel);
if(_debug_protocols != NULL && ! _debug_protocols_ok) {
NDPI_BITMASK_RESET(debug_bitmask);
if(parse_proto_name_list(_debug_protocols, &debug_bitmask, 0)) {
ndpi_exit_detection_module(module);
ndpi_free(workflow);
return NULL;
}
_debug_protocols_ok = 1;
}
if(_debug_protocols_ok)
ndpi_set_debug_bitmask(module, debug_bitmask);
if(do_init_flows_root) {
workflow->ndpi_flows_root = ndpi_calloc(workflow->prefs.num_roots, sizeof(void *));
if(!workflow->ndpi_flows_root) {
@@ -526,11 +458,11 @@ void ndpi_flow_info_freer(void *node) {
/* ***************************************************** */
static void ndpi_free_flow_tls_data(struct ndpi_flow_info *flow) {
if(flow->dhcp_fingerprint) {
ndpi_free(flow->dhcp_fingerprint);
flow->dhcp_fingerprint = NULL;
}
if(flow->dhcp_class_ident) {
ndpi_free(flow->dhcp_class_ident);
flow->dhcp_class_ident = NULL;
@@ -545,6 +477,7 @@ static void ndpi_free_flow_tls_data(struct ndpi_flow_info *flow) {
ndpi_free(flow->telnet.username);
flow->telnet.username = NULL;
}
if(flow->telnet.password) {
ndpi_free(flow->telnet.password);
flow->telnet.password = NULL;
@@ -754,7 +687,6 @@ static struct ndpi_flow_info *get_ndpi_flow_info(struct ndpi_workflow * workflow
ndpi_packet_tunnel tunnel_type,
const struct ndpi_iphdr *iph,
const struct ndpi_ipv6hdr *iph6,
u_int16_t ip_offset,
u_int16_t ipsize,
u_int16_t l4_packet_len,
u_int16_t l4_offset,
@@ -1040,7 +972,6 @@ static struct ndpi_flow_info *get_ndpi_flow_info6(struct ndpi_workflow * workflo
u_int16_t vlan_id,
ndpi_packet_tunnel tunnel_type,
const struct ndpi_ipv6hdr *iph6,
u_int16_t ip_offset,
u_int16_t ipsize,
struct ndpi_tcphdr **tcph,
struct ndpi_udphdr **udph,
@@ -1069,7 +1000,7 @@ static struct ndpi_flow_info *get_ndpi_flow_info6(struct ndpi_workflow * workflo
iph.protocol = l4proto;
return(get_ndpi_flow_info(workflow, 6, vlan_id, tunnel_type,
&iph, iph6, ip_offset, ipsize,
&iph, iph6, ipsize,
ip_len, l4ptr - (const u_int8_t *)iph6,
tcph, udph, sport, dport,
proto, payload,
@@ -1148,11 +1079,13 @@ void process_ndpi_collected_info(struct ndpi_workflow * workflow, struct ndpi_fl
u_int j;
if(flow->ndpi_flow->protos.bittorrent.hash[0] != '\0') {
flow->bittorent_hash = ndpi_malloc(sizeof(flow->ndpi_flow->protos.bittorrent.hash) * 2 + 1);
u_int avail = sizeof(flow->ndpi_flow->protos.bittorrent.hash) * 2 + 1;
flow->bittorent_hash = ndpi_malloc(avail);
if(flow->bittorent_hash) {
for(i=0, j = 0; i < sizeof(flow->ndpi_flow->protos.bittorrent.hash); i++) {
sprintf(&flow->bittorent_hash[j], "%02x",
snprintf(&flow->bittorent_hash[j], avail-j, "%02x",
flow->ndpi_flow->protos.bittorrent.hash[i]);
j += 2;
@@ -1326,6 +1259,8 @@ void process_ndpi_collected_info(struct ndpi_workflow * workflow, struct ndpi_fl
flow->ssh_tls.notAfter = flow->ndpi_flow->protos.tls_quic.notAfter;
ndpi_snprintf(flow->ssh_tls.ja3_client, sizeof(flow->ssh_tls.ja3_client), "%s",
flow->ndpi_flow->protos.tls_quic.ja3_client);
ndpi_snprintf(flow->ssh_tls.ja4_client, sizeof(flow->ssh_tls.ja4_client), "%s",
flow->ndpi_flow->protos.tls_quic.ja4_client);
ndpi_snprintf(flow->ssh_tls.ja3_server, sizeof(flow->ssh_tls.ja3_server), "%s",
flow->ndpi_flow->protos.tls_quic.ja3_server);
flow->ssh_tls.server_unsafe_cipher = flow->ndpi_flow->protos.tls_quic.server_unsafe_cipher;
@@ -1380,6 +1315,12 @@ void process_ndpi_collected_info(struct ndpi_workflow * workflow, struct ndpi_fl
}
}
memcpy(&flow->stun.mapped_address, &flow->ndpi_flow->stun.mapped_address, sizeof(ndpi_address_port));
memcpy(&flow->stun.peer_address, &flow->ndpi_flow->stun.peer_address, sizeof(ndpi_address_port));
memcpy(&flow->stun.relayed_address, &flow->ndpi_flow->stun.relayed_address, sizeof(ndpi_address_port));
memcpy(&flow->stun.response_origin, &flow->ndpi_flow->stun.response_origin, sizeof(ndpi_address_port));
memcpy(&flow->stun.other_address, &flow->ndpi_flow->stun.other_address, sizeof(ndpi_address_port));
flow->multimedia_flow_type = flow->ndpi_flow->flow_multimedia_type;
/* HTTP metadata are "global" not in `flow->ndpi_flow->protos` union; for example, we can have
@@ -1412,7 +1353,7 @@ void process_ndpi_collected_info(struct ndpi_workflow * workflow, struct ndpi_fl
flow->detected_protocol,
&flow->ndpi_flow_serializer) != 0) {
LOG(NDPI_LOG_ERROR, "flow2json failed\n");
exit(-1);
return;
}
ndpi_serialize_string_uint32(&flow->ndpi_flow_serializer, "detection_completed", flow->detection_completed);
@@ -1423,6 +1364,9 @@ void process_ndpi_collected_info(struct ndpi_workflow * workflow, struct ndpi_fl
flow->flow_payload = flow->ndpi_flow->flow_payload, flow->flow_payload_len = flow->ndpi_flow->flow_payload_len;
flow->ndpi_flow->flow_payload = NULL; /* We'll free the memory */
if(workflow->flow_callback != NULL)
workflow->flow_callback(workflow, flow, workflow->flow_callback_userdata);
ndpi_free_flow_info_half(flow);
}
}
@@ -1492,12 +1436,12 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
ndpi_packet_tunnel tunnel_type,
const struct ndpi_iphdr *iph,
struct ndpi_ipv6hdr *iph6,
u_int16_t ip_offset,
u_int16_t ipsize, u_int16_t rawsize,
const struct pcap_pkthdr *header,
const u_char *packet,
pkt_timeval when,
ndpi_risk *flow_risk) {
ndpi_risk *flow_risk,
struct ndpi_flow_info **flow_ext) {
struct ndpi_flow_info *flow = NULL;
struct ndpi_flow_struct *ndpi_flow = NULL;
u_int8_t proto;
@@ -1515,7 +1459,7 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
if(iph)
flow = get_ndpi_flow_info(workflow, IPVERSION, vlan_id,
tunnel_type, iph, NULL,
ip_offset, ipsize,
ipsize,
ntohs(iph->tot_len) ? (ntohs(iph->tot_len) - (iph->ihl * 4)) : ipsize - (iph->ihl * 4) /* TSO */,
iph->ihl * 4,
&tcph, &udph, &sport, &dport,
@@ -1523,7 +1467,7 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
&payload, &payload_len, &src_to_dst_direction, when);
else
flow = get_ndpi_flow_info6(workflow, vlan_id,
tunnel_type, iph6, ip_offset, ipsize,
tunnel_type, iph6, ipsize,
&tcph, &udph, &sport, &dport,
&proto,
&payload, &payload_len, &src_to_dst_direction, when);
@@ -1616,7 +1560,7 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
#endif
if(enable_payload_analyzer && (payload_len > 0))
ndpi_payload_analyzer(flow, src_to_dst_direction,
ndpi_payload_analyzer(flow,
payload, payload_len,
workflow->stats.ip_packet_count);
@@ -1675,7 +1619,7 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
}
if((!skip) && ((flow->src2dst_packets+flow->dst2src_packets) < 100)) {
if(ndpi_has_human_readeable_string(workflow->ndpi_struct, (char*)packet, header->caplen,
if(ndpi_has_human_readeable_string((char*)packet, header->caplen,
human_readeable_string_len,
flow->human_readeable_string_buffer,
sizeof(flow->human_readeable_string_buffer)) == 1)
@@ -1700,8 +1644,8 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
struct ndpi_flow_input_info input_info;
u_int enough_packets =
(((proto == IPPROTO_UDP) && ((flow->src2dst_packets + flow->dst2src_packets) > max_num_udp_dissected_pkts))
|| ((proto == IPPROTO_TCP) && ((flow->src2dst_packets + flow->dst2src_packets) > max_num_tcp_dissected_pkts))) ? 1 : 0;
((proto == IPPROTO_UDP && (max_num_udp_dissected_pkts > 0 && flow->src2dst_packets + flow->dst2src_packets > max_num_udp_dissected_pkts)) ||
(proto == IPPROTO_TCP && (max_num_tcp_dissected_pkts > 0 && flow->src2dst_packets + flow->dst2src_packets > max_num_tcp_dissected_pkts))) ? 1 : 0;
#if 0
printf("%s()\n", __FUNCTION__);
@@ -1728,23 +1672,17 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
if(enough_packets || (flow->detected_protocol.app_protocol != NDPI_PROTOCOL_UNKNOWN)) {
if((!enough_packets)
&& ndpi_extra_dissection_possible(workflow->ndpi_struct, ndpi_flow))
; /* Wait for certificate fingerprint */
; /* Wait for further metadata */
else {
/* New protocol detected or give up */
flow->detection_completed = 1;
#if 0
/* Check if we should keep checking extra packets */
if(ndpi_flow && ndpi_flow->check_extra_packets)
flow->check_extra_packets = 1;
#endif
if(flow->detected_protocol.app_protocol == NDPI_PROTOCOL_UNKNOWN) {
u_int8_t proto_guessed;
flow->detected_protocol = ndpi_detection_giveup(workflow->ndpi_struct, flow->ndpi_flow,
enable_protocol_guess, &proto_guessed);
if(enable_protocol_guess) workflow->stats.guessed_flow_protocols++;
&proto_guessed);
if(proto_guessed) workflow->stats.guessed_flow_protocols++;
}
process_ndpi_collected_info(workflow, flow);
@@ -1765,6 +1703,7 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
#endif
*flow_risk = flow->risk;
*flow_ext = flow;
return(flow->detected_protocol);
}
@@ -1851,10 +1790,9 @@ static uint32_t ndpi_is_valid_gre_tunnel(const struct pcap_pkthdr *header,
struct ndpi_gre_basehdr *grehdr = (struct ndpi_gre_basehdr*)&packet[offset];
offset += sizeof(struct ndpi_gre_basehdr);
/*
rfc-1701
The GRE flags are encoded in the first two octets. Bit 0 is the
most significant bit, bit 15 is the least significant bit. Bits
13 through 15 are reserved for the Version field. Bits 5 through
13 through 15 are reserved for the Version field. Bits 9 through
12 are reserved for future use and MUST be transmitted as zero.
*/
if(NDPI_GRE_IS_FLAGS(grehdr->flags))
@@ -1915,7 +1853,8 @@ static uint32_t ndpi_is_valid_gre_tunnel(const struct pcap_pkthdr *header,
struct ndpi_proto ndpi_workflow_process_packet(struct ndpi_workflow * workflow,
const struct pcap_pkthdr *header,
const u_char *packet,
ndpi_risk *flow_risk) {
ndpi_risk *flow_risk,
struct ndpi_flow_info **flow) {
/*
* Declare pointers to packet headers
*/
@@ -1964,6 +1903,7 @@ struct ndpi_proto ndpi_workflow_process_packet(struct ndpi_workflow * workflow,
u_int8_t vlan_packet = 0;
*flow_risk = 0 /* NDPI_NO_RISK */;
*flow = NULL;
/* Increment raw packet counter */
workflow->stats.raw_packet_count++;
@@ -2438,9 +2378,9 @@ struct ndpi_proto ndpi_workflow_process_packet(struct ndpi_workflow * workflow,
/* process the packet */
return(packet_processing(workflow, time_ms, vlan_id, tunnel_type, iph, iph6,
ip_offset, header->caplen - ip_offset,
header->caplen - ip_offset,
header->caplen, header, packet, header->ts,
flow_risk));
flow_risk, flow));
}
/* *********************************************** */
+24 -6
View File
@@ -273,7 +273,7 @@ typedef struct ndpi_flow_info {
client_hassh[33], server_hassh[33], *server_names,
*advertised_alpns, *negotiated_alpn, *tls_supported_versions,
*tls_issuerDN, *tls_subjectDN,
ja3_client[33], ja3_server[33],
ja3_client[33], ja3_server[33], ja4_client[37],
sha1_cert_fingerprint[20];
u_int8_t sha1_cert_fingerprint_set;
struct tls_heuristics browser_heuristics;
@@ -295,10 +295,16 @@ typedef struct ndpi_flow_info {
} ssh_tls;
struct {
char url[256], request_content_type[64], content_type[64], user_agent[256], server[128], nat_ip[32], filename[256];
char url[256], request_content_type[64], content_type[64],
user_agent[256], server[128], nat_ip[32], filename[256];
u_int response_status_code;
} http;
struct {
ndpi_address_port mapped_address, peer_address,
relayed_address, response_origin, other_address;
} stun;
struct {
char *username, *password;
} telnet;
@@ -374,12 +380,16 @@ typedef struct ndpi_workflow {
struct ndpi_workflow_prefs prefs;
struct ndpi_stats stats;
ndpi_workflow_callback_ptr flow_callback;
void * flow_callback_userdata;
/* outside referencies */
pcap_t *pcap_handle;
/* allocated by prefs */
void **ndpi_flows_root;
struct ndpi_detection_module_struct *ndpi_struct;
struct ndpi_global_context *g_ctx;
u_int32_t num_allocated_flows;
/* CSV,TLV,JSON serialization interface */
@@ -388,7 +398,7 @@ typedef struct ndpi_workflow {
/* TODO: remove wrappers parameters and use ndpi global, when their initialization will be fixed... */
struct ndpi_workflow * ndpi_workflow_init(const struct ndpi_workflow_prefs * prefs, pcap_t * pcap_handle, int do_init_flows_root, ndpi_serialization_format serialization_format);
struct ndpi_workflow * ndpi_workflow_init(const struct ndpi_workflow_prefs * prefs, pcap_t * pcap_handle, int do_init_flows_root, ndpi_serialization_format serialization_format, struct ndpi_global_context *g_ctx);
/* workflow main free function */
@@ -406,7 +416,15 @@ void ndpi_free_flow_info_half(struct ndpi_flow_info *flow);
struct ndpi_proto ndpi_workflow_process_packet(struct ndpi_workflow * workflow,
const struct pcap_pkthdr *header,
const u_char *packet,
ndpi_risk *flow_risk);
ndpi_risk *flow_risk,
struct ndpi_flow_info **flow);
/* Flow callback for completed flows, before the flow memory will be freed. */
static inline void ndpi_workflow_set_flow_callback(struct ndpi_workflow * workflow, ndpi_workflow_callback_ptr callback, void * userdata) {
workflow->flow_callback = callback;
workflow->flow_callback_userdata = userdata;
}
int ndpi_is_datalink_supported(int datalink_type);
@@ -418,12 +436,12 @@ void ndpi_flow_info_freer(void *node);
const char* print_cipher_id(u_int32_t cipher);
int parse_proto_name_list(char *str, NDPI_PROTOCOL_BITMASK *bitmask, int inverted_logic);
extern int nDPI_LogLevel;
extern int reader_log_level;
#if defined(NDPI_ENABLE_DEBUG_MESSAGES) && !defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION)
#define LOG(log_level, args...) \
{ \
if(log_level <= nDPI_LogLevel) \
if(log_level <= reader_log_level) \
printf(args); \
}
#else
+3 -1
View File
@@ -7,6 +7,8 @@
################################################################
#
# Listingdate,SHA1,Listingreason
#The first entry (and only the first one) is fake. It is used to trigger the risk in the tests
1970-01-01 00:00:00,0DDB34F875632C7E1EC09D75827F82D2336DFEB6, Fake
2022-07-02 07:02:11,3784422d67a6ae0c82d60eeaf3850960a9896cb2,AsyncRAT C&C
2022-07-01 18:28:10,a0d78b3ddb27cd4f110a0292cab800beadaf9bd2,AsyncRAT C&C
2022-07-01 09:44:14,4c319ec8b5a66a256a0137d70cd0f9a76bb23fd7,AsyncRAT C&C
@@ -4878,4 +4880,4 @@
2014-05-04 08:10:31,86d6aade4ba1414a91b1e7fb3cdd7d503692f410,Shylock C&C
2014-05-04 08:10:26,5afc236d1dd00c9c45457b75226b501b815a59c7,Shylock C&C
2014-05-04 08:09:56,b08a4939fb88f375a2757eaddc47b1fb8b554439,Shylock C&C
# END (4871) entries
# END (4871) entries
1 ################################################################
7 ################################################################
8 #
9 # Listingdate,SHA1,Listingreason
10 #The first entry (and only the first one) is fake. It is used to trigger the risk in the tests
11 1970-01-01 00:00:00,0DDB34F875632C7E1EC09D75827F82D2336DFEB6, Fake
12 2022-07-02 07:02:11,3784422d67a6ae0c82d60eeaf3850960a9896cb2,AsyncRAT C&C
13 2022-07-01 18:28:10,a0d78b3ddb27cd4f110a0292cab800beadaf9bd2,AsyncRAT C&C
14 2022-07-01 09:44:14,4c319ec8b5a66a256a0137d70cd0f9a76bb23fd7,AsyncRAT C&C
4880 2014-05-04 08:10:31,86d6aade4ba1414a91b1e7fb3cdd7d503692f410,Shylock C&C
4881 2014-05-04 08:10:26,5afc236d1dd00c9c45457b75226b501b815a59c7,Shylock C&C
4882 2014-05-04 08:09:56,b08a4939fb88f375a2757eaddc47b1fb8b554439,Shylock C&C
4883 # END (4871) entries
+275 -58
View File
@@ -1,16 +1,18 @@
bin_PROGRAMS = fuzz_process_packet fuzz_ndpi_reader fuzz_ndpi_reader_alloc_fail fuzz_ndpi_reader_payload_analyzer fuzz_quic_get_crypto_data fuzz_config fuzz_community_id fuzz_serialization fuzz_tls_certificate fuzz_dga
bin_PROGRAMS = fuzz_process_packet fuzz_ndpi_reader fuzz_ndpi_reader_alloc_fail fuzz_ndpi_reader_payload_analyzer fuzz_quic_get_crypto_data fuzz_config fuzz_community_id fuzz_serialization fuzz_tls_certificate fuzz_dga fuzz_is_stun_udp fuzz_is_stun_tcp
#Alghoritms
bin_PROGRAMS += fuzz_alg_bins fuzz_alg_hll fuzz_alg_hw_rsi_outliers_da fuzz_alg_jitter fuzz_alg_ses_des fuzz_alg_crc32_md5 fuzz_alg_bytestream
bin_PROGRAMS += fuzz_alg_bins fuzz_alg_hll fuzz_alg_hw_rsi_outliers_da fuzz_alg_jitter fuzz_alg_ses_des fuzz_alg_crc32_md5 fuzz_alg_bytestream fuzz_alg_shoco
#Data structures
bin_PROGRAMS += fuzz_ds_patricia fuzz_ds_ahocorasick fuzz_ds_libcache fuzz_ds_tree fuzz_ds_ptree fuzz_ds_hash fuzz_ds_cmsketch fuzz_ds_bitmap64 fuzz_ds_domain_classify
bin_PROGRAMS += fuzz_ds_patricia fuzz_ds_ahocorasick fuzz_ds_libcache fuzz_ds_tree fuzz_ds_ptree fuzz_ds_hash fuzz_ds_cmsketch fuzz_ds_bitmap64_fuse fuzz_ds_domain_classify
#Third party
bin_PROGRAMS += fuzz_libinjection fuzz_binaryfusefilter
#Internal crypto
bin_PROGRAMS += fuzz_gcrypt_light fuzz_gcrypt_aes fuzz_gcrypt_gcm fuzz_gcrypt_cipher
#Configuration files
bin_PROGRAMS += fuzz_filecfg_protocols
bin_PROGRAMS += fuzz_filecfg_protocols fuzz_filecfg_categories fuzz_filecfg_malicious_sha1 fuzz_filecfg_malicious_ja3 fuzz_filecfg_risk_domains fuzz_filecfg_config fuzz_filecfg_category
#Reader utils
bin_PROGRAMS += fuzz_readerutils_workflow fuzz_readerutils_parseprotolist
#Mutators
bin_PROGRAMS += fuzz_ndpi_reader_pl7m fuzz_ndpi_reader_pl7m_64k fuzz_ndpi_reader_pl7m_simplest fuzz_ndpi_reader_pl7m_internal fuzz_ndpi_reader_pl7m_simplest_internal
fuzz_process_packet_SOURCES = fuzz_process_packet.c fuzz_common_code.c
fuzz_process_packet_CFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS)
@@ -65,7 +67,7 @@ fuzz_ndpi_reader_payload_analyzer_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTO
$(fuzz_ndpi_reader_payload_analyzer_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_quic_get_crypto_data_SOURCES = fuzz_quic_get_crypto_data.c fuzz_common_code.c
fuzz_quic_get_crypto_data_CFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS)
fuzz_quic_get_crypto_data_CFLAGS = -I../src/lib/ @NDPI_CFLAGS@ $(CXXFLAGS) -DNDPI_LIB_COMPILATION
fuzz_quic_get_crypto_data_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_quic_get_crypto_data_LDFLAGS = $(LIBS)
if HAS_FUZZLDFLAGS
@@ -78,8 +80,8 @@ fuzz_quic_get_crypto_data_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS)
$(fuzz_quic_get_crypto_data_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_config_SOURCES = fuzz_config.cpp fuzz_common_code.c
fuzz_config_CXXFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS)
fuzz_config_CFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS)
fuzz_config_CXXFLAGS = -I../src/lib/ @NDPI_CFLAGS@ $(CXXFLAGS) -DNDPI_LIB_COMPILATION
fuzz_config_CFLAGS = -I../src/lib/ @NDPI_CFLAGS@ $(CXXFLAGS) -DNDPI_LIB_COMPILATION
fuzz_config_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_config_LDFLAGS = $(LIBS)
if HAS_FUZZLDFLAGS
@@ -183,7 +185,7 @@ fuzz_alg_jitter_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(fuzz_alg_jitter_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_alg_crc32_md5_SOURCES = fuzz_alg_crc32_md5.c
fuzz_alg_crc32_md5_CFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS)
fuzz_alg_crc32_md5_CFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS) -I ../src/lib/third_party/include/
fuzz_alg_crc32_md5_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_alg_crc32_md5_LDFLAGS = $(LIBS)
if HAS_FUZZLDFLAGS
@@ -208,6 +210,19 @@ fuzz_alg_bytestream_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_alg_bytestream_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_alg_shoco_SOURCES = fuzz_alg_shoco.cpp
fuzz_alg_shoco_CXXFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS) -I ../src/lib/third_party/include/
fuzz_alg_shoco_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_alg_shoco_LDFLAGS = $(LIBS)
if HAS_FUZZLDFLAGS
fuzz_alg_shoco_CXXFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_alg_shoco_LDFLAGS += $(LIB_FUZZING_ENGINE)
endif
# force usage of CXX for linker
fuzz_alg_shoco_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_alg_shoco_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_alg_ses_des_SOURCES = fuzz_alg_ses_des.cpp fuzz_common_code.c
fuzz_alg_ses_des_CXXFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS)
fuzz_alg_ses_des_CFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS)
@@ -328,20 +343,20 @@ fuzz_ds_cmsketch_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_ds_cmsketch_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_ds_bitmap64_SOURCES = fuzz_ds_bitmap64.cpp fuzz_common_code.c
fuzz_ds_bitmap64_CXXFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS)
fuzz_ds_bitmap64_CFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS)
fuzz_ds_bitmap64_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_ds_bitmap64_LDFLAGS = $(LIBS)
fuzz_ds_bitmap64_fuse_SOURCES = fuzz_ds_bitmap64_fuse.cpp fuzz_common_code.c
fuzz_ds_bitmap64_fuse_CXXFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS)
fuzz_ds_bitmap64_fuse_CFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS)
fuzz_ds_bitmap64_fuse_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_ds_bitmap64_fuse_LDFLAGS = $(LIBS)
if HAS_FUZZLDFLAGS
fuzz_ds_bitmap64_CXXFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_ds_bitmap64_CFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_ds_bitmap64_LDFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_ds_bitmap64_fuse_CXXFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_ds_bitmap64_fuse_CFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_ds_bitmap64_fuse_LDFLAGS += $(LIB_FUZZING_ENGINE)
endif
# force usage of CXX for linker
fuzz_ds_bitmap64_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
fuzz_ds_bitmap64_fuse_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_ds_bitmap64_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
$(fuzz_ds_bitmap64_fuse_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_ds_domain_classify_SOURCES = fuzz_ds_domain_classify.cpp fuzz_common_code.c
fuzz_ds_domain_classify_CXXFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS)
@@ -387,7 +402,7 @@ fuzz_binaryfusefilter_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(fuzz_binaryfusefilter_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_tls_certificate_SOURCES = fuzz_tls_certificate.c fuzz_common_code.c
fuzz_tls_certificate_CFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS)
fuzz_tls_certificate_CFLAGS = -I../src/lib/ @NDPI_CFLAGS@ $(CXXFLAGS) -DNDPI_LIB_COMPILATION
fuzz_tls_certificate_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_tls_certificate_LDFLAGS = $(LIBS)
if HAS_FUZZLDFLAGS
@@ -412,6 +427,32 @@ fuzz_dga_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_dga_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_is_stun_udp_SOURCES = fuzz_is_stun.c fuzz_common_code.c
fuzz_is_stun_udp_CFLAGS = -I../src/lib/ @NDPI_CFLAGS@ $(CXXFLAGS) -DNDPI_LIB_COMPILATION
fuzz_is_stun_udp_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_is_stun_udp_LDFLAGS = $(LIBS)
if HAS_FUZZLDFLAGS
fuzz_is_stun_udp_CFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_is_stun_udp_LDFLAGS += $(LIB_FUZZING_ENGINE)
endif
# force usage of CXX for linker
fuzz_is_stun_udp_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_is_stun_udp_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_is_stun_tcp_SOURCES = fuzz_is_stun.c fuzz_common_code.c
fuzz_is_stun_tcp_CFLAGS = -I../src/lib/ @NDPI_CFLAGS@ $(CXXFLAGS) -DNDPI_LIB_COMPILATION -DSTUN_TCP
fuzz_is_stun_tcp_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_is_stun_tcp_LDFLAGS = $(LIBS)
if HAS_FUZZLDFLAGS
fuzz_is_stun_tcp_CFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_is_stun_tcp_LDFLAGS += $(LIB_FUZZING_ENGINE)
endif
# force usage of CXX for linker
fuzz_is_stun_tcp_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_is_stun_tcp_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_gcrypt_light_SOURCES = fuzz_gcrypt_light.cpp fuzz_common_code.c
fuzz_gcrypt_light_CXXFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS) -I ../src/lib/third_party/include/
fuzz_gcrypt_light_CFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS) -I ../src/lib/third_party/include/
@@ -473,7 +514,7 @@ fuzz_gcrypt_cipher_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(fuzz_gcrypt_cipher_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_filecfg_protocols_SOURCES = fuzz_filecfg_protocols.c fuzz_common_code.c
fuzz_filecfg_protocols_CFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS)
fuzz_filecfg_protocols_CFLAGS = -I../src/lib/ @NDPI_CFLAGS@ $(CXXFLAGS) -DNDPI_LIB_COMPILATION
fuzz_filecfg_protocols_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_filecfg_protocols_LDFLAGS = $(LIBS)
if HAS_FUZZLDFLAGS
@@ -485,9 +526,87 @@ fuzz_filecfg_protocols_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_filecfg_protocols_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_filecfg_categories_SOURCES = fuzz_filecfg_categories.c fuzz_common_code.c
fuzz_filecfg_categories_CFLAGS = -I../src/lib/ @NDPI_CFLAGS@ $(CXXFLAGS) -DNDPI_LIB_COMPILATION
fuzz_filecfg_categories_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_filecfg_categories_LDFLAGS = $(LIBS)
if HAS_FUZZLDFLAGS
fuzz_filecfg_categories_CFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_filecfg_categories_LDFLAGS += $(LIB_FUZZING_ENGINE)
endif
# force usage of CXX for linker
fuzz_filecfg_categories_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_filecfg_categories_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_filecfg_malicious_sha1_SOURCES = fuzz_filecfg_malicious_sha1.c fuzz_common_code.c
fuzz_filecfg_malicious_sha1_CFLAGS = -I../src/lib/ @NDPI_CFLAGS@ $(CXXFLAGS) -DNDPI_LIB_COMPILATION
fuzz_filecfg_malicious_sha1_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_filecfg_malicious_sha1_LDFLAGS = $(LIBS)
if HAS_FUZZLDFLAGS
fuzz_filecfg_malicious_sha1_CFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_filecfg_malicious_sha1_LDFLAGS += $(LIB_FUZZING_ENGINE)
endif
# force usage of CXX for linker
fuzz_filecfg_malicious_sha1_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_filecfg_malicious_sha1_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_filecfg_malicious_ja3_SOURCES = fuzz_filecfg_malicious_ja3.c fuzz_common_code.c
fuzz_filecfg_malicious_ja3_CFLAGS = -I../src/lib/ @NDPI_CFLAGS@ $(CXXFLAGS) -DNDPI_LIB_COMPILATION
fuzz_filecfg_malicious_ja3_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_filecfg_malicious_ja3_LDFLAGS = $(LIBS)
if HAS_FUZZLDFLAGS
fuzz_filecfg_malicious_ja3_CFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_filecfg_malicious_ja3_LDFLAGS += $(LIB_FUZZING_ENGINE)
endif
# force usage of CXX for linker
fuzz_filecfg_malicious_ja3_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_filecfg_malicious_ja3_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_filecfg_risk_domains_SOURCES = fuzz_filecfg_risk_domains.c fuzz_common_code.c
fuzz_filecfg_risk_domains_CFLAGS = -I../src/lib/ @NDPI_CFLAGS@ $(CXXFLAGS) -DNDPI_LIB_COMPILATION
fuzz_filecfg_risk_domains_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_filecfg_risk_domains_LDFLAGS = $(LIBS)
if HAS_FUZZLDFLAGS
fuzz_filecfg_risk_domains_CFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_filecfg_risk_domains_LDFLAGS += $(LIB_FUZZING_ENGINE)
endif
# force usage of CXX for linker
fuzz_filecfg_risk_domains_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_filecfg_risk_domains_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_filecfg_config_SOURCES = fuzz_filecfg_config.c fuzz_common_code.c
fuzz_filecfg_config_CFLAGS = -I../src/lib/ @NDPI_CFLAGS@ $(CXXFLAGS) -DNDPI_LIB_COMPILATION
fuzz_filecfg_config_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_filecfg_config_LDFLAGS = $(LIBS)
if HAS_FUZZLDFLAGS
fuzz_filecfg_config_CFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_filecfg_config_LDFLAGS += $(LIB_FUZZING_ENGINE)
endif
# force usage of CXX for linker
fuzz_filecfg_config_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_filecfg_config_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_filecfg_category_SOURCES = fuzz_filecfg_category.c fuzz_common_code.c
fuzz_filecfg_category_CFLAGS = -I../src/lib/ @NDPI_CFLAGS@ $(CXXFLAGS) -DNDPI_LIB_COMPILATION
fuzz_filecfg_category_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_filecfg_category_LDFLAGS = $(LIBS)
if HAS_FUZZLDFLAGS
fuzz_filecfg_category_CFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_filecfg_category_LDFLAGS += $(LIB_FUZZING_ENGINE)
endif
# force usage of CXX for linker
fuzz_filecfg_category_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_filecfg_category_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_readerutils_workflow_SOURCES = fuzz_readerutils_workflow.cpp fuzz_common_code.c ../example/reader_util.c
fuzz_readerutils_workflow_CXXFLAGS = -I../example/ @NDPI_CFLAGS@ $(CXXFLAGS) -DDISABLE_CUSTOM_ALLOCATOR_ON_READERUTILS
fuzz_readerutils_workflow_CFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS) -DDISABLE_CUSTOM_ALLOCATOR_ON_READERUTILS
fuzz_readerutils_workflow_CXXFLAGS = -I../example/ @NDPI_CFLAGS@ $(CXXFLAGS)
fuzz_readerutils_workflow_CFLAGS = @NDPI_CFLAGS@ $(CXXFLAGS)
fuzz_readerutils_workflow_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_readerutils_workflow_LDFLAGS = $(PCAP_LIB) $(LIBS)
if HAS_FUZZLDFLAGS
@@ -515,53 +634,127 @@ fuzz_readerutils_parseprotolist_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOL
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_readerutils_parseprotolist_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_ndpi_reader_pl7m_simplest_SOURCES = fuzz_ndpi_reader.c fuzz_common_code.c ../example/reader_util.c ../src/lib/third_party/src/fuzz/pl7m.c
fuzz_ndpi_reader_pl7m_simplest_CFLAGS = -I../example/ @NDPI_CFLAGS@ $(CXXFLAGS) -I../src/lib/third_party/include/ -DENABLE_PCAP_L7_MUTATOR -DPL7M_USE_SIMPLEST_MUTATOR
fuzz_ndpi_reader_pl7m_simplest_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_ndpi_reader_pl7m_simplest_LDFLAGS = $(PCAP_LIB) $(LIBS)
if HAS_FUZZLDFLAGS
fuzz_ndpi_reader_pl7m_simplest_CFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_ndpi_reader_pl7m_simplest_LDFLAGS += $(LIB_FUZZING_ENGINE)
endif
# force usage of CXX for linker
fuzz_ndpi_reader_pl7m_simplest_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_ndpi_reader_pl7m_simplest_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_ndpi_reader_pl7m_simplest_internal_SOURCES = fuzz_ndpi_reader.c fuzz_common_code.c ../example/reader_util.c ../src/lib/third_party/src/fuzz/pl7m.c
fuzz_ndpi_reader_pl7m_simplest_internal_CFLAGS = -I../example/ @NDPI_CFLAGS@ $(CXXFLAGS) -I../src/lib/third_party/include/ -DENABLE_PCAP_L7_MUTATOR -DPL7M_USE_SIMPLEST_MUTATOR -DPL7M_USE_INTERNAL_FUZZER_MUTATE
fuzz_ndpi_reader_pl7m_simplest_internal_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_ndpi_reader_pl7m_simplest_internal_LDFLAGS = $(PCAP_LIB) $(LIBS)
if HAS_FUZZLDFLAGS
fuzz_ndpi_reader_pl7m_simplest_internal_CFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_ndpi_reader_pl7m_simplest_internal_LDFLAGS += $(LIB_FUZZING_ENGINE)
endif
# force usage of CXX for linker
fuzz_ndpi_reader_pl7m_simplest_internal_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_ndpi_reader_pl7m_simplest_internal_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_ndpi_reader_pl7m_SOURCES = fuzz_ndpi_reader.c fuzz_common_code.c ../example/reader_util.c ../src/lib/third_party/src/fuzz/pl7m.c
fuzz_ndpi_reader_pl7m_CFLAGS = -I../example/ @NDPI_CFLAGS@ $(CXXFLAGS) -I../src/lib/third_party/include/ -DENABLE_PCAP_L7_MUTATOR
fuzz_ndpi_reader_pl7m_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_ndpi_reader_pl7m_LDFLAGS = $(PCAP_LIB) $(LIBS)
if HAS_FUZZLDFLAGS
fuzz_ndpi_reader_pl7m_CFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_ndpi_reader_pl7m_LDFLAGS += $(LIB_FUZZING_ENGINE)
endif
# force usage of CXX for linker
fuzz_ndpi_reader_pl7m_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_ndpi_reader_pl7m_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_ndpi_reader_pl7m_64k_SOURCES = fuzz_ndpi_reader.c fuzz_common_code.c ../example/reader_util.c ../src/lib/third_party/src/fuzz/pl7m.c
fuzz_ndpi_reader_pl7m_64k_CFLAGS = -I../example/ @NDPI_CFLAGS@ $(CXXFLAGS) -I../src/lib/third_party/include/ -DENABLE_PCAP_L7_MUTATOR -DPL7M_USE_64K_PACKETS
fuzz_ndpi_reader_pl7m_64k_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_ndpi_reader_pl7m_64k_LDFLAGS = $(PCAP_LIB) $(LIBS)
if HAS_FUZZLDFLAGS
fuzz_ndpi_reader_pl7m_64k_CFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_ndpi_reader_pl7m_64k_LDFLAGS += $(LIB_FUZZING_ENGINE)
endif
# force usage of CXX for linker
fuzz_ndpi_reader_pl7m_64k_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_ndpi_reader_pl7m_64k_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
fuzz_ndpi_reader_pl7m_internal_SOURCES = fuzz_ndpi_reader.c fuzz_common_code.c ../example/reader_util.c ../src/lib/third_party/src/fuzz/pl7m.c
fuzz_ndpi_reader_pl7m_internal_CFLAGS = -I../example/ @NDPI_CFLAGS@ $(CXXFLAGS) -I../src/lib/third_party/include/ -DENABLE_PCAP_L7_MUTATOR -DPL7M_USE_INTERNAL_FUZZER_MUTATE
fuzz_ndpi_reader_pl7m_internal_LDADD = ../src/lib/libndpi.a $(ADDITIONAL_LIBS)
fuzz_ndpi_reader_pl7m_internal_LDFLAGS = $(PCAP_LIB) $(LIBS)
if HAS_FUZZLDFLAGS
fuzz_ndpi_reader_pl7m_internal_CFLAGS += $(LIB_FUZZING_ENGINE)
fuzz_ndpi_reader_pl7m_internal_LDFLAGS += $(LIB_FUZZING_ENGINE)
endif
# force usage of CXX for linker
fuzz_ndpi_reader_pl7m_internal_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) @NDPI_CFLAGS@ $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_ndpi_reader_pl7m_internal_LDFLAGS) @NDPI_LDFLAGS@ $(LDFLAGS) -o $@
# required for Google oss-fuzz
# see https://github.com/google/oss-fuzz/tree/master/projects/ndpi
testpcaps := $(wildcard ../tests/cfgs/default/pcap/*.pcap*)
fuzz_ndpi_reader_seed_corpus.zip: $(testpcaps)
zip -j fuzz_ndpi_reader_seed_corpus.zip $(testpcaps)
testpcaps.zip: $(testpcaps)
zip -j testpcaps.zip $(testpcaps)
fuzz_ndpi_reader_alloc_fail_seed_corpus.zip: $(testpcaps)
zip -j fuzz_ndpi_reader_alloc_fail_seed_corpus.zip $(testpcaps)
fuzz_ndpi_reader_seed_corpus.zip: testpcaps.zip
cp testpcaps.zip fuzz_ndpi_reader_seed_corpus.zip
fuzz_ndpi_reader_payload_analyzer_seed_corpus.zip: $(testpcaps)
zip -j fuzz_ndpi_reader_payload_analyzer_seed_corpus.zip $(testpcaps)
fuzz_ndpi_reader_alloc_fail_seed_corpus.zip: testpcaps.zip
cp testpcaps.zip fuzz_ndpi_reader_alloc_fail_seed_corpus.zip
fuzz_ndpi_reader_payload_analyzer_seed_corpus.zip: testpcaps.zip
cp testpcaps.zip fuzz_ndpi_reader_payload_analyzer_seed_corpus.zip
fuzz_ndpi_reader_pl7m_simplest_seed_corpus.zip: testpcaps.zip
cp testpcaps.zip fuzz_ndpi_reader_pl7m_simplest_seed_corpus.zip
fuzz_ndpi_reader_pl7m_simplest_internal_seed_corpus.zip: testpcaps.zip
cp testpcaps.zip fuzz_ndpi_reader_pl7m_simplest_internal_seed_corpus.zip
fuzz_ndpi_reader_pl7m_seed_corpus.zip: testpcaps.zip
cp testpcaps.zip fuzz_ndpi_reader_pl7m_seed_corpus.zip
fuzz_ndpi_reader_pl7m_64k_seed_corpus.zip: testpcaps.zip
cp testpcaps.zip fuzz_ndpi_reader_pl7m_64k_seed_corpus.zip
fuzz_ndpi_reader_pl7m_internal_seed_corpus.zip: testpcaps.zip
cp testpcaps.zip fuzz_ndpi_reader_pl7m_internal_seed_corpus.zip
files_corpus_fuzz_quic_get_crypto_data := $(wildcard corpus/fuzz_quic_get_crypto_data/*)
fuzz_quic_get_crypto_data_seed_corpus.zip: $(files_corpus_fuzz_quic_get_crypto_data)
zip -j fuzz_quic_get_crypto_data_seed_corpus.zip $(files_corpus_fuzz_quic_get_crypto_data)
files_corpus_fuzz_config := $(wildcard corpus/fuzz_config/*)
fuzz_config_seed_corpus.zip: $(files_corpus_fuzz_config)
zip -j fuzz_config_seed_corpus.zip $(files_corpus_fuzz_config)
files_corpus_fuzz_serialization := $(wildcard corpus/fuzz_serialization/*)
fuzz_serialization_seed_corpus.zip: $(files_corpus_fuzz_serialization)
zip -j fuzz_serialization_seed_corpus.zip $(files_corpus_fuzz_serialization)
files_corpus_fuzz_community_id := $(wildcard corpus/fuzz_community_id/*)
fuzz_community_id_seed_corpus.zip: $(files_corpus_fuzz_community_id)
zip -j fuzz_community_id_seed_corpus.zip $(files_corpus_fuzz_community_id)
files_corpus_fuzz_ds_patricia := $(wildcard corpus/fuzz_ds_patricia/*)
fuzz_ds_patricia_seed_corpus.zip: $(files_corpus_fuzz_ds_patricia)
zip -j fuzz_ds_patricia_seed_corpus.zip $(files_corpus_fuzz_ds_patricia)
files_corpus_fuzz_is_stun_udp := $(wildcard corpus/fuzz_is_stun_udp/*)
fuzz_is_stun_udp_seed_corpus.zip: $(files_corpus_fuzz_is_stun_udp)
zip -j fuzz_is_stun_udp_seed_corpus.zip $(files_corpus_fuzz_is_stun_udp)
files_corpus_fuzz_ds_ahocorasick := $(wildcard corpus/fuzz_ds_ahocorasick/*)
fuzz_ds_ahocorasick_seed_corpus.zip: $(files_corpus_fuzz_ds_ahocorasick)
zip -j fuzz_ds_ahocorasick_seed_corpus.zip $(files_corpus_fuzz_ds_ahocorasick)
files_corpus_fuzz_is_stun_tcp := $(wildcard corpus/fuzz_is_stun_tcp/*)
fuzz_is_stun_tcp_seed_corpus.zip: $(files_corpus_fuzz_is_stun_tcp)
zip -j fuzz_is_stun_tcp_seed_corpus.zip $(files_corpus_fuzz_is_stun_tcp)
files_corpus_fuzz_alg_ses_des := $(wildcard corpus/fuzz_alg_ses_des/*)
fuzz_alg_ses_des_seed_corpus.zip: $(files_corpus_fuzz_alg_ses_des)
zip -j fuzz_alg_ses_des_seed_corpus.zip $(files_corpus_fuzz_alg_ses_des)
files_corpus_fuzz_alg_hw_rsi_outliers_da := $(wildcard corpus/fuzz_alg_hw_rsi_outliers_da/*)
fuzz_alg_hw_rsi_outliers_da_seed_corpus.zip: $(files_corpus_fuzz_alg_hw_rsi_outliers_da)
zip -j fuzz_alg_hw_rsi_outliers_da_seed_corpus.zip $(files_corpus_fuzz_alg_hw_rsi_outliers_da)
files_corpus_fuzz_alg_bins := $(wildcard corpus/fuzz_alg_bins/*)
fuzz_alg_bins_seed_corpus.zip: $(files_corpus_fuzz_alg_bins)
zip -j fuzz_alg_bins_seed_corpus.zip $(files_corpus_fuzz_alg_bins)
@@ -582,10 +775,6 @@ files_corpus_fuzz_ds_libcache := $(wildcard corpus/fuzz_ds_libcache/*)
fuzz_ds_libcache_seed_corpus.zip: $(files_corpus_fuzz_ds_libcache)
zip -j fuzz_ds_libcache_seed_corpus.zip $(files_corpus_fuzz_ds_libcache)
files_corpus_fuzz_ds_tree := $(wildcard corpus/fuzz_ds_tree/*)
fuzz_ds_tree_seed_corpus.zip: $(files_corpus_fuzz_ds_tree)
zip -j fuzz_ds_tree_seed_corpus.zip $(files_corpus_fuzz_ds_tree)
files_corpus_fuzz_ds_ptree := $(wildcard corpus/fuzz_ds_ptree/*)
fuzz_ds_ptree_seed_corpus.zip: $(files_corpus_fuzz_ds_ptree)
zip -j fuzz_ds_ptree_seed_corpus.zip $(files_corpus_fuzz_ds_ptree)
@@ -606,6 +795,30 @@ files_corpus_fuzz_filecfg_protocols := $(wildcard corpus/fuzz_filecfg_protocols
fuzz_filecfg_protocols_seed_corpus.zip: $(files_corpus_fuzz_filecfg_protocols)
zip -j fuzz_filecfg_protocols_seed_corpus.zip $(files_corpus_fuzz_filecfg_protocols)
files_corpus_fuzz_filecfg_categories := $(wildcard corpus/fuzz_filecfg_categories/*)
fuzz_filecfg_categories_seed_corpus.zip: $(files_corpus_fuzz_filecfg_categories)
zip -j fuzz_filecfg_categories_seed_corpus.zip $(files_corpus_fuzz_filecfg_categories)
files_corpus_fuzz_filecfg_malicious_sha1 := $(wildcard corpus/fuzz_filecfg_malicious_sha1/*)
fuzz_filecfg_malicious_sha1_seed_corpus.zip: $(files_corpus_fuzz_filecfg_malicious_sha1)
zip -j fuzz_filecfg_malicious_sha1_seed_corpus.zip $(files_corpus_fuzz_filecfg_malicious_sha1)
files_corpus_fuzz_filecfg_malicious_ja3 := $(wildcard corpus/fuzz_filecfg_malicious_ja3/*)
fuzz_filecfg_malicious_ja3_seed_corpus.zip: $(files_corpus_fuzz_filecfg_malicious_ja3)
zip -j fuzz_filecfg_malicious_ja3_seed_corpus.zip $(files_corpus_fuzz_filecfg_malicious_ja3)
files_corpus_fuzz_filecfg_risk_domains := $(wildcard corpus/fuzz_filecfg_risk_domains/*)
fuzz_filecfg_risk_domains_seed_corpus.zip: $(files_corpus_fuzz_filecfg_risk_domains)
zip -j fuzz_filecfg_risk_domains_seed_corpus.zip $(files_corpus_fuzz_filecfg_risk_domains)
files_corpus_fuzz_filecfg_config := $(wildcard corpus/fuzz_filecfg_config/*)
fuzz_filecfg_config_seed_corpus.zip: $(files_corpus_fuzz_filecfg_config)
zip -j fuzz_filecfg_config_seed_corpus.zip $(files_corpus_fuzz_filecfg_config)
files_corpus_fuzz_filecfg_category := $(wildcard corpus/fuzz_filecfg_category/*)
fuzz_filecfg_category_seed_corpus.zip: $(files_corpus_fuzz_filecfg_category)
zip -j fuzz_filecfg_category_seed_corpus.zip $(files_corpus_fuzz_filecfg_category)
files_corpus_fuzz_readerutils_workflow := $(wildcard corpus/fuzz_readerutils_workflow/*)
fuzz_readerutils_workflow_seed_corpus.zip: $(files_corpus_fuzz_readerutils_workflow)
zip -j fuzz_readerutils_workflow_seed_corpus.zip $(files_corpus_fuzz_readerutils_workflow)
@@ -614,15 +827,15 @@ files_corpus_fuzz_readerutils_parseprotolist := $(wildcard corpus/fuzz_readerut
fuzz_readerutils_parseprotolist_seed_corpus.zip: $(files_corpus_fuzz_readerutils_parseprotolist)
zip -j fuzz_readerutils_parseprotolist_seed_corpus.zip $(files_corpus_fuzz_readerutils_parseprotolist)
files_corpus_fuzz_ds_bitmap64 := $(wildcard corpus/fuzz_ds_bitmap64/*)
fuzz_ds_bitmap64_seed_corpus.zip: $(files_corpus_fuzz_ds_bitmap64)
zip -j fuzz_ds_bitmap64_seed_corpus.zip $(files_corpus_fuzz_ds_bitmap64)
files_corpus_fuzz_ds_bitmap64_fuse := $(wildcard corpus/fuzz_ds_bitmap64_fuse/*)
fuzz_ds_bitmap64_fuse_seed_corpus.zip: $(files_corpus_fuzz_ds_bitmap64_fuse)
zip -j fuzz_ds_bitmap64_fuse_seed_corpus.zip $(files_corpus_fuzz_ds_bitmap64_fuse)
files_corpus_fuzz_ds_domain_classify := $(wildcard corpus/fuzz_ds_domain_classify/*)
fuzz_ds_domain_classify_seed_corpus.zip: $(files_corpus_fuzz_ds_domain_classify)
zip -j fuzz_ds_domain_classify_seed_corpus.zip $(files_corpus_fuzz_ds_domain_classify)
corpus: fuzz_ndpi_reader_seed_corpus.zip fuzz_ndpi_reader_alloc_fail_seed_corpus.zip fuzz_ndpi_reader_payload_analyzer_seed_corpus.zip fuzz_quic_get_crypto_data_seed_corpus.zip fuzz_config_seed_corpus.zip fuzz_ds_patricia_seed_corpus.zip fuzz_ds_ahocorasick_seed_corpus.zip fuzz_alg_ses_des_seed_corpus.zip fuzz_alg_hw_rsi_outliers_da_seed_corpus.zip fuzz_alg_bins_seed_corpus.zip fuzz_alg_hll_seed_corpus.zip fuzz_alg_jitter_seed_corpus.zip fuzz_ds_libcache_seed_corpus.zip fuzz_community_id_seed_corpus.zip fuzz_ds_tree_seed_corpus.zip fuzz_serialization_seed_corpus.zip fuzz_ds_ptree_seed_corpus.zip fuzz_alg_crc32_md5_seed_corpus.zip fuzz_alg_bytestream_seed_corpus.zip fuzz_libinjection_seed_corpus.zip fuzz_tls_certificate_seed_corpus.zip fuzz_filecfg_protocols_seed_corpus.zip fuzz_readerutils_workflow_seed_corpus.zip fuzz_readerutils_parseprotolist_seed_corpus.zip fuzz_ds_bitmap64_seed_corpus.zip fuzz_ds_domain_classify_seed_corpus.zip
corpus: fuzz_ndpi_reader_seed_corpus.zip fuzz_ndpi_reader_alloc_fail_seed_corpus.zip fuzz_ndpi_reader_payload_analyzer_seed_corpus.zip fuzz_quic_get_crypto_data_seed_corpus.zip fuzz_alg_ses_des_seed_corpus.zip fuzz_alg_bins_seed_corpus.zip fuzz_alg_hll_seed_corpus.zip fuzz_alg_jitter_seed_corpus.zip fuzz_ds_libcache_seed_corpus.zip fuzz_community_id_seed_corpus.zip fuzz_serialization_seed_corpus.zip fuzz_ds_ptree_seed_corpus.zip fuzz_alg_crc32_md5_seed_corpus.zip fuzz_alg_bytestream_seed_corpus.zip fuzz_libinjection_seed_corpus.zip fuzz_tls_certificate_seed_corpus.zip fuzz_filecfg_protocols_seed_corpus.zip fuzz_readerutils_workflow_seed_corpus.zip fuzz_readerutils_parseprotolist_seed_corpus.zip fuzz_ds_bitmap64_fuse_seed_corpus.zip fuzz_ds_domain_classify_seed_corpus.zip fuzz_filecfg_protocols_seed_corpus.zip fuzz_is_stun_udp_seed_corpus.zip fuzz_is_stun_tcp_seed_corpus.zip fuzz_ndpi_reader_pl7m_simplest_seed_corpus.zip fuzz_ndpi_reader_pl7m_seed_corpus.zip fuzz_ndpi_reader_pl7m_64k_seed_corpus.zip fuzz_ndpi_reader_pl7m_simplest_internal_seed_corpus.zip fuzz_ndpi_reader_pl7m_internal_seed_corpus.zip
cp corpus/fuzz_*seed_corpus.zip .
#Create dictionaries exactly as expected by oss-fuzz.
@@ -643,6 +856,7 @@ distdir:
-o -name '*.cpp' \
-o -name '*.options' \
-o -name 'ipv4_addresses.txt' \
-o -name 'ipv6_addresses.txt' \
-o -name 'bd_param.txt' \
-o -name 'splt_param.txt' \
-o -name 'random_list.list' \
@@ -651,9 +865,16 @@ distdir:
-o -path './corpus/fuzz_*.zip' \
-o -path './corpus/fuzz_quic_get_crypto_data/*' \
-o -path './corpus/fuzz_filecfg_protocols/*' \
-o -path './corpus/fuzz_filecfg_categories/*' \
-o -path './corpus/fuzz_filecfg_malicious_sha1/*' \
-o -path './corpus/fuzz_filecfg_malicious_ja3/*' \
-o -path './corpus/fuzz_filecfg_risk_domains/*' \
-o -path './corpus/fuzz_filecfg_config/*' \
-o -path './corpus/fuzz_filecfg_category/*' \
-o -path './corpus/fuzz_readerutils_workflow/*' \
-o -path './corpus/fuzz_readerutils_parseprotolist/*' \
-o -path './corpus/fuzz_config/*' \
-o -path './corpus/fuzz_is_stun_udp/*' \
-o -path './corpus/fuzz_is_stun_tcp/*' \
-o -path './corpus/fuzz_serialization/*' \
-o -path './corpus/fuzz_community_id/*' \
-o -path './corpus/fuzz_libinjection/*' \
@@ -663,14 +884,10 @@ distdir:
-o -path './corpus/fuzz_alg_hll/*' \
-o -path './corpus/fuzz_alg_jitter/*' \
-o -path './corpus/fuzz_alg_crc32_md5/*' \
-o -path './corpus/fuzz_alg_hw_rsi_outliers_da/*' \
-o -path './corpus/fuzz_alg_bytestream/*' \
-o -path './corpus/fuzz_ds_ahocorasick/*' \
-o -path './corpus/fuzz_ds_libcache/*' \
-o -path './corpus/fuzz_ds_tree/*' \
-o -path './corpus/fuzz_ds_bitmap64/*' \
-o -path './corpus/fuzz_ds_bitmap64_fuse/*' \
-o -path './corpus/fuzz_ds_domain_classify/*' \
-o -path './corpus/fuzz_ds_ptree/*' \
-o -path './corpus/fuzz_ds_patricia/*' | xargs -I'{}' cp -r '{}' '$(distdir)/{}'
-o -path './corpus/fuzz_ds_ptree/*' | xargs -I'{}' cp -r '{}' '$(distdir)/{}'
all: corpus dictionaries

Some files were not shown because too many files have changed in this diff Show More