Commit Graph
2404 Commits
Author SHA1 Message Date
Ivan Nardi c0732eda45 HTTP: improve sub-classification (#1696)
Content-matched sub-protocols (DASH, IPP, MPEGDASH...) shouldn't
ovewrite the previous master protocol (if any; usually HTTP).

Furthermore. the HTTP dissector shouldn't update the classification
(in the extra-dissection code path) if a content-matched sub-protocols
has already been found.

This commit should address the first part of the changes described in #1687.
2022-08-05 14:14:38 +02:00
Ivan Nardi 346e274a1b SoftEther: fix two heap-buffer-overflows (#1695)
The first change is a proper (hopefully) fix for the bug reported in
8b6a00f8.
The second one is related to:
```
==15096==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60f0000d7a00 at pc 0x55a2c593bd0b bp 0x7ffc92021cd0 sp 0x7ffc92021478
READ of size 3 at 0x60f0000d7a00 thread T0
    #0 0x55a2c593bd0a in strncmp (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader+0x56fd0a) (BuildId: ee8631c0950a8cded5ba60c17f09709bbebbe5d8)
    #1 0x55a2c5d1d9f9 in dissect_softether_host_fqdn /home/ivan/svnrepos/nDPI/src/lib/protocols/softether.c:249:9
    #2 0x55a2c5d1b55b in ndpi_search_softether /home/ivan/svnrepos/nDPI/src/lib/protocols/softether.c:348:9
    #3 0x55a2c5b0e9c5 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5407:6
    #4 0x55a2c5b0f78b in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5443:10
    #5 0x55a2c5b0f12c in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5476:12
    #6 0x55a2c5b20f39 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6303:15
    #7 0x55a2c5a3014c in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1600:31
    #8 0x55a2c5a29062 in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2170:10
    #9 0x55a2c59e51a2 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader.c:107:7
    #10 0x55a2c590acb2 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader+0x53ecb2) (BuildId: ee8631c0950a8cded5ba60c17f09709bbebbe5d8)
    #11 0x55a2c590a3c5 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader+0x53e3c5) (BuildId: ee8631c0950a8cded5ba60c17f09709bbebbe5d8)
    #12 0x55a2c590c0f6 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile>>&) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader+0x5400f6) (BuildId: ee8631c0950a8cded5ba60c17f09709bbebbe5d8)
    #13 0x55a2c590c663 in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile>>&) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader+0x540663) (BuildId: ee8631c0950a8cded5ba60c17f09709bbebbe5d8)
    #14 0x55a2c58faff2 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader+0x52eff2) (BuildId: ee8631c0950a8cded5ba60c17f09709bbebbe5d8)
    #15 0x55a2c5923c82 in main (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader+0x557c82) (BuildId: ee8631c0950a8cded5ba60c17f09709bbebbe5d8)
    #16 0x7f504ab98082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
    #17 0x55a2c58efb1d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader+0x523b1d) (BuildId: ee8631c0950a8cded5ba60c17f09709bbebbe5d8)

0x60f0000d7a00 is located 0 bytes to the right of 176-byte region [0x60f0000d7950,0x60f0000d7a00)
```
Found by oss-fuzzer
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=49736
2022-08-05 13:43:09 +02:00
Luca Deri bcf7f1713c Win and indentation fixes 2022-08-05 11:17:09 +02:00
Toni ac24b35b1f Add Discord dissector. (#1694)
* fixed RiotGames false positive

Signed-off-by: lns <matzeton@googlemail.com>
2022-08-03 12:03:36 +02:00
Luca Deri c3c73e7bff Cosmetic changes 2022-08-03 11:31:32 +02:00
Luca Deri 63fe4b4306 Compilation fix (not all compilers like restrict) (part 2) 2022-08-03 11:12:15 +02:00
Toni b3e722e5a8 Improved nDPI JSON serialization. (#1689)
* fixed autoconf CFLAGS/LDFLAGS MSAN issue which could lead to build errors
 * introduced portable version of gmtime_r aka ndpi_gmtime_r
 * do as most as possible of the serialization work in ndpi_utils.c
 * use flow2json in ndpiReader

Signed-off-by: lns <matzeton@googlemail.com>
2022-08-02 17:54:44 +02:00
Toni 26aafd931c Add Activision dissector. (#1693)
Signed-off-by: lns <matzeton@googlemail.com>
2022-08-02 16:15:07 +02:00
Ivan Nardi 013212c67c TLS: avoid useless extra dissection (#1690)
If we have seen Application Data blocks in both directions, it means we
are after the handshake. At that point, extra dissection is useless.

Useful feature with TLS mid-sessions, i.e. sessions without initial
packets.
2022-07-31 16:57:17 +02:00
Ivan Nardi 2a596c79e6 HTTP: fix classification (#1692)
If we have a valid HTTP sessions, we should ignore
`flow->guessed_protocol_id` field (i.e. classification "by-port")
altogheter.

The attached trace was classified as "SIP/HTTP" only because the *client*
port was 5060...
As a general rule, having a classification such as "XXXX/HTTP" is
*extremely* suspicious.
2022-07-30 22:57:20 +02:00
Ivan Nardi 8b6a00f84b SoftEther: fix heap-buffer-overflow (#1691)
```
==160665==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x604000000038 at pc 0x55f7250d9a5c bp 0x7fff02c82b90 sp 0x7fff02c82350
READ of size 4 at 0x604000000038 thread T0
    #0 0x55f7250d9a5b in __interceptor_strncpy (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x548a5b) (BuildId: 12fd06e7a171f035d3a25f016184ac357088379c)
    #1 0x55f7253e6495 in dissect_softether_ip_port /home/ivan/svnrepos/nDPI/src/lib/protocols/softether.c:303:3
    #2 0x55f7253e5703 in ndpi_search_softether /home/ivan/svnrepos/nDPI/src/lib/protocols/softether.c:330:9
    #3 0x55f7251d87c5 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5397:6
    #4 0x55f7251d958b in check_ndpi_udp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5433:10
    #5 0x55f7251d8f2c in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5466:12
    #6 0x55f7251ead39 in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6293:15
    #7 0x55f72512b87e in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:29:5
    #8 0x55f72512b9f7 in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:101:17
    #9 0x7fdef837b082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
    #10 0x55f72506a45d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x4d945d) (BuildId: 12fd06e7a171f035d3a25f016184ac357088379c)
```

Found by oss-fuzz.
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=49638
2022-07-30 22:55:46 +02:00
Ivan Nardi d54d5083b3 SMTPS, POPS, IMAPS: fix classification and extra dissection (#1685)
The big change in TLS code is to allow "master" protocols other than
TLS/DTLS, like SMTPS, POPS and IMAPS.
This change will allow, in a future, a proper and complete TLS dissection
for all these protocols with "STARTTLS"-like messages.
2022-07-30 12:05:43 +02:00
Toni 8402bd68ad Tiny Softether dissector improvement. (#1688)
Signed-off-by: lns <matzeton@googlemail.com>
2022-07-30 10:03:42 +02:00
Toni ed4f106a0d Add Softether dissector. (#1679)
Signed-off-by: lns <matzeton@googlemail.com>
2022-07-29 19:29:54 +02:00
Ivan Nardi 405a52ed65 Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1683)
Add (basic) internal stats to the main data structures used by the
library; they might be usefull to check how effective these structures
are.

Add an option to `ndpiReader` to dump them; enabled by default in the
unit tests.
This new option enables/disables dumping of "num dissectors calls"
values, too (see b4cb14ec).
2022-07-29 15:25:00 +02:00
Ivan Nardi df2e11ef51 Revert "Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1677)" (#1682)
This reverts commit bb83899985.
2022-07-29 12:08:40 +02:00
Ivan Nardi bb83899985 Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1677)
Add (basic) internal stats to the main data structures used by the
library; they might be usefull to check how effective these structures
are.

Add an option to `ndpiReader` to dump them; disabled by default to avoid
too much fuss with the unit tests.
2022-07-29 12:07:41 +02:00
Ivan Nardi 95e16872fa First step in simplify ndpi_process_extra_packet() (#1680)
Move the prottocol specific logic into the proper dissector code, where
it belongs.

Next step: remove that list of protocols. Long goal: remove this
function altogether...
2022-07-29 12:07:14 +02:00
Ivan Nardi 172e698bb8 TINC: avoid processing SYN packets (#1676)
Since e6b332aa, we have proper support for detecting client/server
direction. So Tinc dissector is now able to properly initialize the
cache entry only when needed and not anymore at the SYN time; initializing
that entry for **every** SYN packets was a complete waste of resources.

Since 4896dabb, the various `struct ndpi_call_function_struct`
structures are not more separate objects and therefore comparing them
using only their pointers is bogus: this bug was triggered by this
change because `ndpi_str->callback_buffer_size_tcp_no_payload` is now 0.
2022-07-28 12:39:18 +02:00
Ivan Nardi d8d525fff2 Update the protocol bitmask for some protocols (#1675)
Tcp retransmissions should be ignored.

Remove some unused protocol bitmasks.

Update script to download Whatsapp IP list.
2022-07-27 11:46:45 +02:00
Toni ab3a678ad4 Add AVAST dissector. (#1674)
Signed-off-by: lns <matzeton@googlemail.com>
2022-07-25 18:07:44 +02:00
Ivan Nardi b190dab6bc Improve handling of HTTP-Proxy and HTTP-Connect (#1673)
Treat HTTP-Proxy and HTTP-Connect flows like the HTTP ones:
print/serialize all the attributes and allow parsing of replies.

The line about "1kxun" has been removed to avoid regressions in 1KXUN
classification in `tests/pcap/1kxun.pcap`. I haven't fully understod
what was happening but the comment at the beginning of `static
ndpi_category_match category_match[]` says that we can't have overlaps
between `host_match` and `category_match` lists and that is no longer true
since 938e89ca.
Bottom line: removing this line seems the right thing to do, anyway.
2022-07-25 12:57:33 +02:00
Ivan Nardi 86a3e4c8c3 Add support for Memory sanitizer (#1652) 2022-07-24 18:12:25 +02:00
Ivan Nardi e6b332aa4a Add support for flow client/server information (#1671)
In a lot of places in ndPI we use *packet* source/dest info
(address/port/direction) when we are interested in *flow* client/server
info, instead.

Add basic logic to autodetect this kind of information.

nDPI doesn't perform any "flow management" itself but this task is
delegated to the external application. It is then likely that the
application might provide more reliable hints about flow
client/server direction and about the TCP handshake presence: in that case,
these information might be (optionally) passed to the library, disabling
the internal "autodetect" logic.

These new fields have been used in some LRU caches and in the "guessing"
algorithm.
It is quite likely that some other code needs to be updated.
2022-07-24 17:46:24 +02:00
Toni a25b2a7e37 Added AliCloud server access dissector. (#1672)
Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-23 11:21:49 +02:00
Zied Aouini 40d7354dd7 Windows error C2466 fix (array size init with 0). 2022-07-22 21:58:25 +02:00
Zied Aouini 9d040f6317 Windows compilation fix. 2022-07-22 20:14:30 +02:00
Ivan Nardi 52005e88ed TLS: improve reassembler (#1669)
* TLS: cosmetic changes

* TLS: improve reassembler

We might need to contemporary re-order messages from both directions:
use one buffer per direction.
2022-07-22 12:19:21 +02:00
Ivan Nardi ce6f11840d Bittorrent: fix keys in cache code (#1670) 2022-07-22 12:10:43 +02:00
Nardi Ivan 00eb2393f4 STUN: improve detection of Hangout/Duo and FB voip calls 2022-07-20 20:36:31 +02:00
Nardi Ivan d66aa49787 DTLS: fix exclusion of DTLS protocol
Add an helper to exclude a generic protocol
2022-07-20 19:16:18 +02:00
Nardi Ivan e1edb08f06 SKYPE: fix detection over UDP
Commit ba6a48c9 is completely bogus: we can't set extra dissection
without having set a proper classification.

The idea behind that commit seems to be that we need to look for 2
(consecutives?) packets with the same crc/pattern: try to implement this
logic in a saner way.
2022-07-20 16:13:55 +02:00
Nardi Ivan 5702c6fb08 SKYPE: remove detection over TCP
Skype detection over TCP has been completely disable since 659f75138 (3
years ago!).
Since that logic was too weak anyway, remove it.
2022-07-20 16:13:55 +02:00
Ivan Nardi c72660d7d3 reader_util: stop processing a flow (#1666)
We should stop processing a flow if all protocols have been excluded or
if we have already processed too many packets.
2022-07-20 14:48:09 +02:00
Ivan Nardi b472a49271 BITTORRENT: fix confidence (#1664)
Remove two unused parameters.
2022-07-20 13:59:51 +02:00
Ivan Nardi ac1113b9ac ahocorasick: fix char/uchar bug (#1597)
Porting of https://github.com/vel21ripn/nDPI/commit/06e2967d0c26ab214683a2f5565f4012fd523537
Close #1506
2022-07-20 10:59:17 +02:00
Toni be1ac1b430 Fix unitialized use of W[16]. (#1662)
Signed-off-by: lns <matzeton@googlemail.com>
2022-07-15 14:15:37 +02:00
Luca b80218a0e4 Added null pointer checks 2022-07-14 18:04:51 +02:00
Toni 7c19de4904 Do not interfere with CFLAGS/LDFLAGS env anymore. (#1659)
* CI fixes
 * some build systems do not like that (e.g. OpenWrt)
 * fixed some rrdtool related build warnings/errors

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-13 19:44:18 +02:00
Toni ae2bedce3a Improved Jabber/XMPP detection. (#1661)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-13 17:55:33 +02:00
Ivan Nardi 407155755d ASN1/BER: fix signed integer overflow (#1660)
```
protocols/snmp_proto.c:77:23: runtime error: signed integer overflow: 6 + 2147483647 cannot be represented in type 'int'
 #0 0x52f69e in ndpi_search_snmp ndpi/src/lib/protocols/snmp_proto.c:77:23
 #1 0x4c5347 in check_ndpi_detection_func ndpi/src/lib/ndpi_main.c:5211:4
 #2 0x4c5591 in ndpi_check_flow_func ndpi/src/lib/ndpi_main.c:0
 #3 0x4c8903 in ndpi_detection_process_packet ndpi/src/lib/ndpi_main.c:6145:15
 #4 0x4b3712 in LLVMFuzzerTestOneInput ndpi/fuzz/fuzz_process_packet.c:29:5
[...]
```
Found by oss-fuzzer.
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=49057
2022-07-12 18:39:05 +02:00
Luca 9c235796af Introduced risk accountability 2022-07-12 10:36:40 +02:00
Luca cba2d12051 Cosmetic change 2022-07-12 10:36:32 +02:00
Ivan Nardi b4cb14ec19 Keep track of how many dissectors calls we made for each flow (#1657) 2022-07-11 09:47:47 +02:00
Ivan Nardi df599e5eff HTTP: improve detection of WindowsUpdate (#1658)
WindowsUpdate is also transported over HTTP, using a numeric IP as
hostname (some kinds of CDN?)
2022-07-10 17:08:37 +02:00
Ivan Nardi 1fcd03a6b6 Remove unsafe access to flow->protos union (#1656)
We can access `flow->protos` only if we already have set a valid
classification.

It is quite likely that this code is never trigger, anyway.
2022-07-10 17:08:22 +02:00
Ivan Nardi 2edfaeba4a SNMP: fix detection (#1655)
We can write to `flow->protos` only when we are sure about SNMP
classification.

Use the generic wrapper to decode ASN1 BER integer
2022-07-10 17:07:52 +02:00
Ivan Nardi 997dce0f04 SIP: improve detection (#1654) 2022-07-09 05:45:42 +02:00
Ivan Nardi dfe6557e18 TFTP: fix memory access (#1653) 2022-07-08 18:39:05 +02:00
Ivan Nardi 510517126a LDAP: rewrite dissection (#1649) 2022-07-08 12:50:46 +02:00