Commit Graph
266 Commits
Author SHA1 Message Date
Luca Deri 63a2b359b9 Added vlan_id in ndpi_flow2json() prototype 2023-06-09 19:50:47 +02:00
Ivan Nardi efb261a95c Fix some memory errors triggered by allocation failures (#1995)
Some low hanging fruits found using nallocfuzz.
See: https://github.com/catenacyber/nallocfuzz
See: https://github.com/google/oss-fuzz/pull/9902

Most of these errors are quite trivial to fix; the only exception is the
stuff in the uthash.
If the insertion fails (because of an allocation failure), we need to
avoid some memory leaks. But the only way to check if the `HASH_ADD_*`
failed, is to perform a new lookup: a bit costly, but we don't use that
code in any critical data-path.
2023-05-29 19:24:00 +02:00
Ivan Nardi 82fa3a0986 Fix an heap-buffer-overflow (#1994)
```
==32439==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x604000000080 at pc 0x564a9a1639ac bp 0x7ffc77d42e20 sp 0x7ffc77d42e18
READ of size 1 at 0x604000000080 thread T0
    #0 0x564a9a1639ab in ndpi_check_punycode_string /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:98:12
    #1 0x564a9a26025a in ndpi_match_host_subprotocol /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:8652:6
    #2 0x564a9a33e45c in ndpi_http_parse_subprotocol /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:446:9
    #3 0x564a9a3354b8 in check_content_type_and_change_protocol /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:921:3
    #4 0x564a9a32dc8b in process_request /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1237:3
    #5 0x564a9a329a9c in ndpi_check_http_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1351:5
    #6 0x564a9a3293ed in ndpi_search_http_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1425:3
    #7 0x564a9a21c64a in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5906:6
    #8 0x564a9a21d057 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5954:12
```

Found by oss-fuzzer.
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=59393
2023-05-29 16:53:39 +02:00
Toni 5e8f93c2d1 Improved missing usage of nDPIs malloc wrapper. Fixes #1978. (#1979)
* added CI check

Signed-off-by: lns <matzeton@googlemail.com>
2023-05-20 16:18:52 +02:00
Ivan Nardi 6b94c9675a Improve detection of crawler/bot traffic (#1956) 2023-05-04 11:27:34 +02:00
Luca Deri b39ece4428 Updated (C) 2023-04-28 23:00:33 +02:00
Ivan Nardi 40b6d5a2e1 fuzz: extend fuzzers coverage (#1952) 2023-04-25 16:37:28 +02:00
Maatuq f1193d5e6f add support for gre decapsulation (#1442) (#1921) 2023-04-04 14:20:11 +02:00
Luca Deri d398f232be Added fix for better handling exceptions rollback in case of later match 2023-03-06 17:33:08 +01:00
Ivan Nardi 89cae9ddf2 Add a new flow risk about literal IP addresses used as SNI (#1892)
RFC 6066 3: "Literal IPv4 and IPv6 addresses are not permitted in
"HostName"."

Don't set this risk if we have a valid sub-classification (example:
via certificate)

Since a similar risk already exists for HTTP hostnames, reuse it, with a
more generic name.
2023-03-02 15:27:30 +01:00
Ivan Nardi 4075324e2b fuzz: extend fuzz coverage (#1888) 2023-02-16 18:04:34 +01:00
Ivan Nardi f2f8f9d0ee Fix export/serialization of flow->risk (#1885)
Any risks bigger than 31 was never exported/serialized; found looking
a the fuzz coverage report.
2023-02-14 21:30:36 +01:00
Luca Deri 5849863ef9 Added new risk NDPI_TCP_ISSUES 2023-01-24 22:58:17 +01:00
Luca Deri bc44334599 Shrinked symbolic flow risks labels 2023-01-19 10:14:32 +01:00
Luca Deri 2c551afbd7 Added NDPI_MINOR_ISSUES risk used for storing generic/relevant information about issues found on traffic. 2022-12-31 16:55:55 +01:00
Luca Deri 1735931f67 Added NDPI_PERIODIC_FLOW flow risk to be used by apps based on nDPI 2022-12-30 19:20:07 +01:00
Luca Deri 8f91b8ba72 Implemented EDNS(0) support in DNS dissector
Improved DNS dissection
2022-12-29 19:38:25 +01:00
Ivan Nardi 560280e6f0 fuzz: add fuzzer testing nDPI (initial) configurations (#1830)
The goal of this fuzzer is to test init and deinit of the library, with
different configurations. In details:
* random memory allocation failures, even during init phase
* random `ndpi_init_prefs` parameter of `ndpi_init_detection_module()`
* random LRU caches sizes
* random bitmask of enabled protocols
* random parameters of `ndpi_set_detection_preferences()`
* random initialization of opportunistic TLS
* random load/don't load of configuration files

This new fuzzer is a C++ file, because it uses `FuzzedDataProvider`
class (see
https://github.com/google/fuzzing/blob/master/docs/split-inputs.md).
Note that the (existing) fuzzers need to be linked with C++ compiler
anyway, so this new fuzzer doesn't add any new requirements.
2022-12-23 19:07:13 +01:00
Ivan Nardi 22821241e2 Fix some errors found by oss-fuzz (#1834) 2022-12-20 22:50:18 +01:00
Ivan Nardi ada4fe4aa8 fuzz: add a new fuzzer testing memory allocation failures (#1818)
Try to fuzz error paths triggered by allocation errors.
Fix some errors already found by this new fuzzer.
Basic idea taken from: https://github.com/harfbuzz/harfbuzz/pull/2566/files

`FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is a standard define used to
(not)compile specific code in fuzzing builds.
See: https://llvm.org/docs/LibFuzzer.html
2022-12-06 17:41:58 +01:00
Ivan Nardi cd41ab7c8f Improve export/print of L4 protocol information (#1799)
Close #1797
2022-11-13 22:35:46 +01:00
Nardi Ivan 52b562c328 Fix json export of ipv6 addresses
The "string" buffer was to short; better start using `INET6_ADDRSTRLEN`
as reported in the man page of `inet_ntop`.

Close: #1794
2022-11-07 20:36:55 +01:00
Ivan Nardi ca5ffc4988 TLS: improve handling of ALPN(s) (#1784)
Tell "Advertised" ALPN list from "Negotiated" ALPN; the former is
extracted from the CH, the latter from the SH.

Add some entries to the known ALPN list.

Fix printing of "TLS Supported Versions" field.
2022-10-25 17:06:29 +02:00
Luca Deri 24cc949f14 Enhanced HTTP numeric IP check 2022-10-22 10:06:09 +02:00
Luca Deri df0dd76ed2 Reworked severities adding critical and emergency, thus shifting values down 2022-10-19 10:53:11 +02:00
Nardi Ivan 223a6fb9f7 Fix a use-of-uninitialized-value error on PCRE code
This is likely a false positive, triggered by the fact that libpcre is
usually compiled without MASAN support.
It it was a real error, ASAN would complain loudly with a invalid-free
error at the end of the same function.

```
==83793==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x562296111174 in ndpi_compile_rce_regex /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1631:3
    #1 0x5622960e3e4a in ndpi_is_rce_injection /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1636:5
    #2 0x5622960de7cd in ndpi_validate_url /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1741:12
    #3 0x5622960dae45 in ndpi_dpi2json /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1362:29
    #4 0x5622960e2751 in ndpi_flow2json /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1512:10
    #5 0x562296033b0f in process_ndpi_collected_info /home/ivan/svnrepos/nDPI/example/reader_util.c:1310:9
    #6 0x5622960501f9 in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1659:2
    #7 0x562296045aef in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2202:10
    #8 0x562295e85374 in ndpi_process_packet /home/ivan/svnrepos/nDPI/example/ndpiReader.c:3937:7
    #9 0x7f1235053466  (/lib/x86_64-linux-gnu/libpcap.so.0.8+0x23466) (BuildId: b84c893ea2516d6fb2c1c6726b1fe93b3be78f61)
    #10 0x7f1235041f67 in pcap_loop (/lib/x86_64-linux-gnu/libpcap.so.0.8+0x11f67) (BuildId: b84c893ea2516d6fb2c1c6726b1fe93b3be78f61)
    #11 0x562295e53139 in runPcapLoop /home/ivan/svnrepos/nDPI/example/ndpiReader.c:4060:15
    #12 0x562295e51e7f in processing_thread /home/ivan/svnrepos/nDPI/example/ndpiReader.c:4130:3
    #13 0x7f1234e53608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
    #14 0x7f1234d2f132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

  Uninitialized value was created by an allocation of 'pcreErrorStr' in the stack frame
    #0 0x5622961108a6 in ndpi_compile_rce_regex /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1603:3
```
2022-10-17 23:15:07 +02:00
Toni 01622c4c63 Updated protocol dissector serialisation code. (#1767)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-10-15 12:58:44 +02:00
Luca 37c88b129f Added new flow risk NDPI_HTTP_OBSOLETE_SERVER. Currently Apache and nginx are supported 2022-10-04 22:32:45 +02:00
Toni 644ad34962 Improved NATPMP dissection. (#1745)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-21 18:24:04 +02:00
Toni Uhlig d6701e8979 Build ndpiReader and run regression tests.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-21 18:03:22 +02:00
mark andrews 5d5b46e514 Add proj and sln for compile of dynamic x64 lib under Visual Studio 2019.
* add CI support via MSBuild

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-21 18:03:22 +02:00
Nardi Ivan 678dd61866 STUN: several improvements
Add detection over TCP and fix detection over IPv6.
Rename some variables since Stun dissector is no more "udp-centric".
Stun dissector should always classified the flow as `STUN` or
`STUN/Something`.
Don't touch `flow->guessed_host_protocol_id` field, which should be
always be related to "ip-classification" only.
2022-09-11 13:33:32 +02:00
Toni 37f918322c Serialize nDPI detection/dissection data as object. (#1725)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-06 11:38:31 +02:00
Toni ac24b35b1f Add Discord dissector. (#1694)
* fixed RiotGames false positive

Signed-off-by: lns <matzeton@googlemail.com>
2022-08-03 12:03:36 +02:00
Luca Deri 63fe4b4306 Compilation fix (not all compilers like restrict) (part 2) 2022-08-03 11:12:15 +02:00
Toni b3e722e5a8 Improved nDPI JSON serialization. (#1689)
* fixed autoconf CFLAGS/LDFLAGS MSAN issue which could lead to build errors
 * introduced portable version of gmtime_r aka ndpi_gmtime_r
 * do as most as possible of the serialization work in ndpi_utils.c
 * use flow2json in ndpiReader

Signed-off-by: lns <matzeton@googlemail.com>
2022-08-02 17:54:44 +02:00
Ivan Nardi b190dab6bc Improve handling of HTTP-Proxy and HTTP-Connect (#1673)
Treat HTTP-Proxy and HTTP-Connect flows like the HTTP ones:
print/serialize all the attributes and allow parsing of replies.

The line about "1kxun" has been removed to avoid regressions in 1KXUN
classification in `tests/pcap/1kxun.pcap`. I haven't fully understod
what was happening but the comment at the beginning of `static
ndpi_category_match category_match[]` says that we can't have overlaps
between `host_match` and `category_match` lists and that is no longer true
since 938e89ca.
Bottom line: removing this line seems the right thing to do, anyway.
2022-07-25 12:57:33 +02:00
Ivan Nardi e6b332aa4a Add support for flow client/server information (#1671)
In a lot of places in ndPI we use *packet* source/dest info
(address/port/direction) when we are interested in *flow* client/server
info, instead.

Add basic logic to autodetect this kind of information.

nDPI doesn't perform any "flow management" itself but this task is
delegated to the external application. It is then likely that the
application might provide more reliable hints about flow
client/server direction and about the TCP handshake presence: in that case,
these information might be (optionally) passed to the library, disabling
the internal "autodetect" logic.

These new fields have been used in some LRU caches and in the "guessing"
algorithm.
It is quite likely that some other code needs to be updated.
2022-07-24 17:46:24 +02:00
Zied Aouini 40d7354dd7 Windows error C2466 fix (array size init with 0). 2022-07-22 21:58:25 +02:00
Zied Aouini 9d040f6317 Windows compilation fix. 2022-07-22 20:14:30 +02:00
Luca b80218a0e4 Added null pointer checks 2022-07-14 18:04:51 +02:00
Ivan Nardi 407155755d ASN1/BER: fix signed integer overflow (#1660)
```
protocols/snmp_proto.c:77:23: runtime error: signed integer overflow: 6 + 2147483647 cannot be represented in type 'int'
 #0 0x52f69e in ndpi_search_snmp ndpi/src/lib/protocols/snmp_proto.c:77:23
 #1 0x4c5347 in check_ndpi_detection_func ndpi/src/lib/ndpi_main.c:5211:4
 #2 0x4c5591 in ndpi_check_flow_func ndpi/src/lib/ndpi_main.c:0
 #3 0x4c8903 in ndpi_detection_process_packet ndpi/src/lib/ndpi_main.c:6145:15
 #4 0x4b3712 in LLVMFuzzerTestOneInput ndpi/fuzz/fuzz_process_packet.c:29:5
[...]
```
Found by oss-fuzzer.
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=49057
2022-07-12 18:39:05 +02:00
Ivan Nardi 510517126a LDAP: rewrite dissection (#1649) 2022-07-08 12:50:46 +02:00
Ivan Nardi 7645909460 Fix handling of NDPI_UNIDIRECTIONAL_TRAFFIC risk (#1636) 2022-07-05 17:01:00 +02:00
Luca Deri 461589517e Updated ndpi_check_flow_risk_exceptions() signature 2022-07-04 21:38:38 +02:00
Luca Deri e7a5eaecde Cleaned-up issuer DN check code adding
u_int8_t ndpi_check_issuerdn_risk_exception(struct ndpi_detection_module_struct *ndpi_str, char *issuerDN);

Added new API function for checking nDPI-configured exceptions
u_int8_t ndpi_check_flow_risk_exception(struct ndpi_detection_module_struct *ndpi_str,
                                        u_int num_params,
                                        ndpi_risk_params **params);
2022-07-04 18:41:01 +02:00
Toni Uhlig b3d3e3b210 Replaced malicious JA3-md5/SSL-cert-sha1 ac automata with hashmaps.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-04 16:05:22 +02:00
Luca Deri ab09b8ce2e Added unidirectional traffic flow risk 2022-06-20 00:22:13 +02:00
Toni 9c8b2d63da Replaced nDPI's internal hashmap with uthash. (#1602)
Signed-off-by: lns <matzeton@googlemail.com>
2022-06-17 19:50:31 +02:00
Toni 09fbe0a64a Fixed syslog false positives. (#1577)
* syslog: removed unnecessary/unreliable printable string check
 * added `ndpi_isalnum()`
 * splitted `ndpi_is_printable_string()` into `ndpi_is_printable_buffer()` and `ndpi_normalize_printable_string()`

Signed-off-by: lns <matzeton@googlemail.com>
2022-06-03 18:21:29 +02:00