Commit Graph
118 Commits
Author SHA1 Message Date
Ivan Nardi fe043d3af5 Further reduction of the size of some traces used as unit test (#1879)
See a944514d. No flow/classification/metadata have been removed.
2023-01-30 11:10:54 +01:00
Luca Deri eea9956430 Improved connection refused detection 2023-01-25 22:18:50 +01:00
Ivan Nardi ccc5a20710 Add detection of Tailscale (#1857)
Extend the example of wireguard traffic
2023-01-17 21:55:38 +01:00
Ivan Nardi 470eaa6ff3 Add Meraki Cloud protocol and improve other protocols (#1854)
Avoid some LineCall and Jabber false positives.
Detect Discord mid flows.
Fix Bittorrent detection.
2023-01-17 11:38:39 +01:00
Ivan Nardi ebb9ebd2a0 Fix classification "by-port" (#1851)
Classification "by-port" should be the last possible effort, *after*
having test all the LRU caches.

Remove some dead code from `ndpi_detection_giveup()`:
`flow->guessed_protocol_id` is never set to any od those voip protocols
and at that point in this function we never have both a master *and* a
application protocols.
Coverage reports (both from unit tests and from fuzzing) confirms that
was dead code.
2023-01-17 08:26:42 +01:00
Ivan Nardi 3de76812d9 Remove some old protocols (#1832)
These protocols:
* have been addeded in the OpenDPI era
* have never been updated since then
* we don't have any pcap examples [*]

If (and it is a big if...) some of these protocols are still somehow
used and if someone is still interested in them, we can probably
re-add them starting from scratch (because the current detection
rules are probably outdated)

Protocols removed: DIRECT_DOWNLOAD_LINK, APPLEJUICE, DIRECTCONNECT,
OPENFT, FASTTRACK, SHOUTCAST, THUNDER, AYIYA, STEALTHNET, FIESTA,
FLORENSIA, AIMINI, SOPCAST

PPSTREAM dissector works (...) only on UDP.

[*]: with do have an AIMINI test pcap but it was some trivial http
traffic detected only by hostname matching, on domains no more
available...
2022-12-23 10:04:08 +01:00
Darryl Sokoloski b8972d1b38 Added TP-LINK Smart Home Protocol dissector. (#1841)
Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>

Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
2022-12-20 20:46:06 +01:00
Darryl Sokoloski 9854aed408 Added TUYA LAN Protocol dissector. (#1838)
Tuya IoTOS Embedded Wi-Fi and BLE SDK for bk7231n.  Used by many "smart"
devices such as LED light strips, bulbs, etc.

Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>

Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
2022-12-19 17:02:19 +01:00
Ivan Nardi 5704e4c142 STUN: add detection of ZOOM peer-to-peer flows (#1825)
See: "Enabling Passive Measurement of Zoom Performance in Production Networks"
https://dl.acm.org/doi/pdf/10.1145/3517745.3561414
2022-12-11 23:07:35 +01:00
Luca Deri 57562b3dfe Updated test results 2022-12-05 21:32:45 +01:00
Ivan Nardi 1796a1d814 LINE_CALL: add detection of LINE voip calls (#1761)
These flows are classifed as `LINE_CALL`; another option was
`RTP/LINE_CALL`. No sure about the best solution...

Extend LINE domains list.

Remove RTP dead code.
2022-10-06 17:09:26 +02:00
Luca 37c88b129f Added new flow risk NDPI_HTTP_OBSOLETE_SERVER. Currently Apache and nginx are supported 2022-10-04 22:32:45 +02:00
Nardi Ivan 1f345b311f Sizes of LRU caches are now configurable
0 as size value disable the cache.

The diffs in unit tests are due to the fact that some lookups are
performed before the first insert: before this change these lookups
weren't counted because the cache was not yet initialized, now they are.
2022-09-23 18:33:48 +02:00
Ivan Nardi a7c2734b38 Remove classification "by-ip" from protocol stack (#1743)
Basically:
* "classification by-ip" (i.e. `flow->guessed_protocol_id_by_ip` is
NEVER returned in the protocol stack (i.e.
`flow->detected_protocol_stack[]`);
* if the application is interested into such information, it can access
`ndpi_protocol->protocol_by_ip` itself.

There are mainly 4 points in the code that set the "classification
by-ip" in the protocol stack:  the generic `ndpi_set_detected_protocol()`/
`ndpi_detection_giveup()` functions and the HTTP/STUN  dissectors.

In the unit tests output, a print about `ndpi_protocol->protocol_by_ip`
has been added for each flow: the huge diff of this commit is mainly due
to that.

Strictly speaking, this change is NOT an API/ABI breakage, but there are
important differences in the classification results. For examples:
* TLS flows without the initial handshake (or without a matching
SNI/certificate) are simply classified as `TLS`;
* similar for HTTP or QUIC flows;
* DNS flows without a matching request domain are simply classified as
`DNS`; we don't have `DNS/Google` anymore just because the server is
8.8.8.8 (that was an outrageous behaviour...);
* flows previusoly classified only "by-ip" are now classified as
`NDPI_PROTOCOL_UNKNOWN`.

See #1425 for other examples of why adding the "classification by-ip" in
the protocol stack is a bad idea.

Please, note that IPV6 is not supported :(  (long standing issue in nDPI) i.e.
`ndpi_protocol->protocol_by_ip` wil be always `NDPI_PROTOCOL_UNKNOWN` for
IPv6 flows.

Define `NDPI_CONFIDENCE_MATCH_BY_IP` has been removed.

Close #1687
2022-09-20 22:24:47 +02:00
Toni Uhlig a966d37a21 Add CryNetwork dissector.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-16 16:48:43 +02:00
Toni Uhlig ec1981c20c Add Syncthing dissector.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-16 15:21:49 +02:00
Toni de020b1747 Add NATPMP dissector. (#1738)
Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-16 13:32:30 +02:00
Ivan Nardi 4f584f78a0 Fix ndpi_do_guess() (#1731)
Avoid a double call of `ndpi_guess_host_protocol_id()`.
Some code paths work for ipv4/6 both
Remove some never used code.
2022-09-12 19:28:41 +02:00
Ivan Nardi 0a47f745cc Avoid useless host automa lookup (#1724)
The host automa is used for two tasks:
* protocol sub-classification (obviously);
* DGA evaluation: the idea is that if a domain is present in this
automa, it can't be a DGA, regardless of its format/name.

In most dissectors both checks are executed, i.e. the code is something
like:

```
ndpi_match_host_subprotocol(..., flow->host_server_name, ...);
ndpi_check_dga_name(..., flow->host_server_name,...);

```

In that common case, we can perform only one automa lookup: if we check the
sub-classification before the DGA, we can avoid the second lookup in
the DGA function itself.
2022-09-05 13:59:51 +02:00
Toni 2e25c36396 Add TiVoConnect dissector. Fixes #1697. (#1699)
* added static assert if supported, to complain if the flow struct changes

Signed-off-by: lns <matzeton@googlemail.com>
2022-08-08 19:04:20 +02:00
Toni ac24b35b1f Add Discord dissector. (#1694)
* fixed RiotGames false positive

Signed-off-by: lns <matzeton@googlemail.com>
2022-08-03 12:03:36 +02:00
Toni 26aafd931c Add Activision dissector. (#1693)
Signed-off-by: lns <matzeton@googlemail.com>
2022-08-02 16:15:07 +02:00
Toni ed4f106a0d Add Softether dissector. (#1679)
Signed-off-by: lns <matzeton@googlemail.com>
2022-07-29 19:29:54 +02:00
Ivan Nardi 405a52ed65 Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1683)
Add (basic) internal stats to the main data structures used by the
library; they might be usefull to check how effective these structures
are.

Add an option to `ndpiReader` to dump them; enabled by default in the
unit tests.
This new option enables/disables dumping of "num dissectors calls"
values, too (see b4cb14ec).
2022-07-29 15:25:00 +02:00
Ivan Nardi 172e698bb8 TINC: avoid processing SYN packets (#1676)
Since e6b332aa, we have proper support for detecting client/server
direction. So Tinc dissector is now able to properly initialize the
cache entry only when needed and not anymore at the SYN time; initializing
that entry for **every** SYN packets was a complete waste of resources.

Since 4896dabb, the various `struct ndpi_call_function_struct`
structures are not more separate objects and therefore comparing them
using only their pointers is bogus: this bug was triggered by this
change because `ndpi_str->callback_buffer_size_tcp_no_payload` is now 0.
2022-07-28 12:39:18 +02:00
Ivan Nardi d8d525fff2 Update the protocol bitmask for some protocols (#1675)
Tcp retransmissions should be ignored.

Remove some unused protocol bitmasks.

Update script to download Whatsapp IP list.
2022-07-27 11:46:45 +02:00
Nardi Ivan d66aa49787 DTLS: fix exclusion of DTLS protocol
Add an helper to exclude a generic protocol
2022-07-20 19:16:18 +02:00
Toni ae2bedce3a Improved Jabber/XMPP detection. (#1661)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-13 17:55:33 +02:00
Ivan Nardi b4cb14ec19 Keep track of how many dissectors calls we made for each flow (#1657) 2022-07-11 09:47:47 +02:00
Ivan Nardi ff4e010501 Avoid spurious calls to extra dissection (#1648)
If the extra callabck is not set, calling the extra dissection is only a
waste of resources...
2022-07-07 17:49:35 +02:00
Ivan Nardi 7645909460 Fix handling of NDPI_UNIDIRECTIONAL_TRAFFIC risk (#1636) 2022-07-05 17:01:00 +02:00
Luca Deri 227ab5c105 Enhanced TLS risk info reported to users 2022-06-28 00:01:00 +02:00
Luca Deri ab09b8ce2e Added unidirectional traffic flow risk 2022-06-20 00:22:13 +02:00
Luca Deri 1da9f1a36f Updated tests results
Code cleanup
2022-05-30 00:54:17 +02:00
Toni 4319d760e0 Improved TLS alert detection. (#1542)
Signed-off-by: lns <matzeton@googlemail.com>
2022-05-08 20:33:59 +02:00
Ivan Nardi 71636dcafd Sync unit tests results (#1533) 2022-04-27 18:22:11 +02:00
Ivan Nardi 86b97ffb73 Sync unit tests results (#1423)
Fix: 7a3aa41a
2022-01-28 17:17:39 +01:00
Ivan Nardi 3a087e951d Add a "confidence" field about the reliability of the classification. (#1395)
As a general rule, the higher the confidence value, the higher the
"reliability/precision" of the classification.

In other words, this new field provides an hint about "how" the flow
classification has been obtained.
For example, the application may want to ignore classification "by-port"
(they are not real DPI classifications, after all) or give a second
glance at flows classified via LRU caches (because of false positives).

Setting only one value for the confidence field is a bit tricky: more
work is probably needed in the next future to tweak/fix/improve the logic.
2022-01-11 15:23:39 +01:00
Ivan Nardi 7153b8933c Improve/add several protocols (#1383)
Improve Microsoft, GMail, Likee, Whatsapp, DisneyPlus and Tiktok
detection.
Add Vimeo, Fuze, Alibaba and Firebase Crashlytics detection.

Try to differentiate between Messenger/Signal standard flows (i.e chat)
and their VOIP (video)calls (like we already do for Whatsapp and
Snapchat).

Add a partial list of some ADS/Tracking stuff.

Fix Cassandra, Radius and GTP false positives.
Fix DNS, Syslog and SIP false negatives.

Improve GTP (sub)classification: differentiate among GTP-U, GTP_C and
GTP_PRIME.

Fix 3 LGTM warnings.
2021-12-18 13:24:51 +01:00
Ivan Nardi b1e9245d94 ndpiReader: slight simplificaton of the output (#1378) 2021-11-27 17:32:23 +01:00
Ivan Nardi 5464bad6db Differentiate between standard Amazon stuff (i.e market) and AWS (#1369) 2021-11-04 00:20:45 +01:00
Ivan Nardi b6d9536533 Fixed cleartext protocol assignment (#1357) 2021-10-25 15:04:04 +02:00
Luca Deri 9e97d20c25 Refreshed results list 2021-10-16 12:03:16 +02:00
Luca Deri c2e7fbce94 Updated test results after latest commit 2021-10-16 11:57:32 +02:00
Luca Deri 408d78e628 Improved DGA detection for skipping potential DGAs of known/popular domain names 2021-10-05 16:51:24 +02:00
Ivan Nardi f3fcf1e7c0 Update unit tests results after da8eed5a (#1323) 2021-10-05 15:06:34 +02:00
Luca Deri e8455236bd Updated output 2021-08-07 17:38:33 +02:00
Ivan Nardi cccf794265 ndpiReader: add statistics about nDPI performance (#1240)
The goal is to have a (roughly) idea about how many packets nDPI needs
to properly classify a flow.

Log this information (and guessed flows number too) during unit tests,
to keep track of improvements/regressions across commits.
2021-07-13 12:28:39 +02:00
Luca Deri 732bcecd17 Added flow risk score 2021-05-18 21:05:47 +02:00
Luca Deri 43a8576efb Reworked human readeable string search in flows
Removed fragment manager code
2021-05-17 20:55:06 +02:00