Commit Graph
86 Commits
Author SHA1 Message Date
Ivan Nardi 7153b8933c Improve/add several protocols (#1383)
Improve Microsoft, GMail, Likee, Whatsapp, DisneyPlus and Tiktok
detection.
Add Vimeo, Fuze, Alibaba and Firebase Crashlytics detection.

Try to differentiate between Messenger/Signal standard flows (i.e chat)
and their VOIP (video)calls (like we already do for Whatsapp and
Snapchat).

Add a partial list of some ADS/Tracking stuff.

Fix Cassandra, Radius and GTP false positives.
Fix DNS, Syslog and SIP false negatives.

Improve GTP (sub)classification: differentiate among GTP-U, GTP_C and
GTP_PRIME.

Fix 3 LGTM warnings.
2021-12-18 13:24:51 +01:00
Ivan Nardi 7cba34a1e3 Improve IPv6 support, enabling IPv6 traffic on (almost) all dissectors. (#1380)
There are no valid reasons for a (generic) protocol to ignore IPv6
traffic.
Note that:
* I have not found the specifications of "CheckPoint High Availability
Protocol", so I don't know how/if it supports IPv6
* all LRU caches are still IPv4 only

Even if src_id/dst_id stuff is probably useless (see #1279), the right
way to update the protocol classification is via `ndpi_set_detected_protocol()`
2021-12-04 13:30:10 +01:00
Ivan Nardi a8ffcd8bb0 Rework how hostname/SNI info is saved (#1330)
Looking at `struct ndpi_flow_struct` the two bigger fields are
`host_server_name[240]` (mainly for HTTP hostnames and DNS domains) and
`protos.tls_quic.client_requested_server_name[256]`
(for TLS/QUIC SNIs).

This commit aims to reduce `struct ndpi_flow_struct` size, according to
two simple observations:
 1) maximum one of these two fields is used for each flow. So it seems safe
to merge them;
 2) even if hostnames/SNIs might be very long, in practice they are rarely
longer than a fews tens of bytes. So, using a (single) large buffer is a
waste of memory for all kinds of flows. If we need to truncate the name,
we keep the *last* characters, easing domain matching.

Analyzing some real traffic, it seems safe to assume that the vast
majority of hostnames/SNIs is shorter than 80 bytes.

Hostnames/SNIs are always converted to lowercase.

Attention was given so as to be sure that unit-tests outputs are not
affected by this change.

Because of a bug, TLS/QUIC SNI were always truncated to 64 bytes (the
*first* 64 ones): as a consequence, there were some "Suspicious DGA
domain name" and "TLS Certificate Mismatch" false positives.
2021-11-24 10:46:48 +01:00
Ivan Nardi afc2b641eb Fix writes to flow->protos union fields (#1354)
We can write to `flow->protos` only after a proper classification.

This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP,
SMTP, IMAP and POP code.
There are two kinds of fixes:
 * write to `flow->protos` only if a final protocol has been detected
 * move protocol state out of `flow->protos`
The hard part is to find, for each protocol, the right tradeoff between
memory usage and code complexity.

Handle Kerberos like DNS: if we find a request, we set the protocol
and an extra callback to further parsing the reply.

For all the other protocols, move the state out of `flow->protos`. This
is an issue only for the FTP/MAIL stuff.

Add DHCP Class Identification value to the output of ndpiReader and to
the Jason serialization.

Extend code coverage of fuzz tests.

Close #1343
Close #1342
2021-11-15 16:20:57 +01:00
Luca Deri f41d8ac106 Improved STUN and RTP detection 2021-10-27 00:06:05 +02:00
Ivan Nardi 2dfc478ad4 Fix compilation with clang-13 or if some debug macros are enabled (#1326) 2021-10-06 18:32:15 +02:00
Ivan Nardi 730c2360bd Remove struct ndpi_packet_struct from struct ndpi_flow_struct (#1319)
There are no real reasons to embed `struct ndpi_packet_struct` (i.e. "packet")
in `struct ndpi_flow_struct` (i.e. "flow"). In other words, we can avoid
saving dissection information of "current packet" into the "flow" state,
i.e. in the flow management table.

The nDPI detection module processes only one packet at the time, so it is
safe to save packet dissection information in `struct ndpi_detection_module_struct`,
reusing always the same "packet" instance and saving a huge amount of memory.
Bottom line: we need only one copy of "packet" (for detection module),
not one for each "flow".

It is not clear how/why "packet" ended up in "flow" in the first place.
It has been there since the beginning of the GIT history, but in the original
OpenDPI code `struct ipoque_packet_struct` was embedded in
`struct ipoque_detection_module_struct`, i.e. there was the same exact
situation this commit wants to achieve.

Most of the changes in this PR are some boilerplate to update something
like "flow->packet" into something like "module->packet" throughout the code.
Some attention has been paid to update `ndpi_init_packet()` since we need
to reset some "packet" fields before starting to process another packet.

There has been one important change, though, in ndpi_detection_giveup().
Nothing changed for the applications/users, but this function can't access
"packet" anymore.
The reason is that this function can be called "asynchronously" with respect
to the data processing, i.e in context where there is no valid notion of
"current packet"; for example ndpiReader calls it after having processed all
the traffic, iterating the entire session table.

Mining LRU stuff seems a bit odd (even before this patch): probably we need
to rethink it, as a follow-up.
2021-10-05 15:49:52 +02:00
Nardi Ivan 95ac8fd41a STUN: fix extraction of Realm attribute
While at it, improve detection of Facebook Messenger
2021-09-20 11:04:54 +02:00
Luca Deri a31bd5ac3c Cleaned up tls/quic datatypes 2021-01-21 19:17:33 +01:00
Luca 0809956e5f Rewored UPnP protocol that in essence was WSD hence it has been renamed
Cleaned up TLS code for DTLS detection by defining a new DTLS protocol
2021-01-20 09:53:30 +01:00
Luca Deri e9f43516ec Improves STUN dissection removing an invalid termination condition that prevented Skype calls to be properly identified 2021-01-13 16:46:51 +01:00
Luca Deri 68b6ac7da8 (C) Update 2021-01-07 11:13:36 +01:00
Ivan Nardi ec2735694b STUN: avoid false positives (#1110)
STUN traffic doesn't use multicast addresses
2021-01-07 10:58:22 +01:00
Luca Deri 044ed14b4f Various optimizations to reduce not-necessary calls
Optimized various UDP dissectors
Removed dead protocols such as pando and pplive
2020-09-24 23:26:03 +02:00
Luca Deri 362baa6321 Minor change for alignment issue 2020-09-21 21:15:52 +02:00
Luca Deri 460ff3c7ab Added (optional) notifier for LRU add 2020-08-31 08:47:16 +02:00
Luca Deri 07d9e4f9ba Fixed valse positive whatsapp detection
Cleaned Microsoft IP addresses list
2020-05-20 23:28:21 +02:00
Luca Deri 29dd45838d Updated (C) 2020-01-05 18:24:58 +01:00
Philippe Antoine 843204299c Fix read buffer overflow in stun 2019-12-18 13:48:33 +01:00
Luca Deri 6d7f3976f3 Code cleanup 2019-12-09 18:02:48 +01:00
Luca Deri 64804a2a24 Fixed some false positivies with skype and stun-based protocols 2019-10-27 12:23:57 +01:00
Luca Deri b603cb4530 nDPI TLS improvements using the server certificate 2019-10-26 10:54:32 +02:00
Luca Deri 194cdcf581 Fixed false positive with STUN detection 2019-09-26 22:18:56 +02:00
Luca Deri 0e9918464b Added Zoom protocol support removing invalid STUN/Skype detections 2019-09-26 21:52:42 +02:00
marco-testa bd83847027 Changed the packets handling with STUN msg_type > 0x000C and other fixes.
Eliminated double call to the ndpi_int_stun_add_connection function.
2019-09-24 09:43:47 +02:00
Luca Deri ab9246f51f Adedd DTLS check in STUN
Uodated (C)
2019-09-21 09:16:56 +02:00
Luca bdcc6dc173 STUN protocol dissector code cleanup 2019-09-20 18:47:39 +02:00
Luca f9716c5139 Unified WhatsApp Video and Audio under WhatsAppCall 2019-09-20 17:37:41 +02:00
Luca f678486393 Improved STUN-based protocol heuristic both in terms of accuracy and packets necessary for the detection 2019-09-20 07:29:02 +02:00
Luca Deri cf0ca7db51 Improved STUN caching 2019-09-18 23:31:25 +02:00
Luca Deri ac1f7d71dd New instagram testing set 2019-09-18 12:23:55 +02:00
Luca Deri 64ed0ac6b3 Various STUN improvements 2019-09-17 00:11:17 +02:00
Luca Deri 6966e0d19b Added STUN check to avoid false positives
Added fingerprint comments in SSH/TLS
Added netflow test pcap
2019-09-11 17:13:49 +02:00
Luca a0a8149833 Skype STUN enhancements 2019-09-06 10:41:53 +02:00
Luca 241cfb319b Enhanced Signal detection 2019-09-05 10:45:08 +02:00
Luca 9b80c6314d Enhanced STUN cache 2019-09-05 10:23:07 +02:00
Luca fdeecf5d41 Implemented STUN cache to enhance matching of STUN-based protocols 2019-08-12 14:16:28 +02:00
Luca 51ac583d83 Improved google duo detection 2019-08-12 11:59:57 +02:00
Luca c6e832be18 Various TLS/STUN improvememnts 2019-08-08 18:09:12 +02:00
Luca cce8a6026f Reworked SSL/TLS field naming 2019-08-08 15:20:05 +02:00
Luca 419160f351 Implemented DTLS support
Renamed ssl to tls
2019-08-08 12:45:27 +02:00
Luca Deri 5c38e7b126 Better messenger traffic guess 2019-07-25 17:23:32 +02:00
Luca Deri b715467da6 Added -e option to ndpiReader for searchign human readeable strings lenght
Default human readeable strings lenght is not 5 chars (used to be 8)
2019-07-24 00:25:00 +02:00
Luca Deri 63173e7360 Updated results with new dissection 2019-07-24 00:13:07 +02:00
Luca Deri 974abaa275 Merged Google Hangout and Duo as they are pretty similar from the network standpoint and from the features they implement 2019-07-22 23:24:40 +02:00
Luca Deri ccea4669de Improved WhatsApp detection 2019-07-22 23:06:25 +02:00
Luca Deri 72b05fed75 STUN, Hangout, Duo dissection improvements 2019-07-21 08:17:28 +02:00
Luca Deri 6bce0c4a9c Added support for Google Duo 2019-07-19 00:48:07 +02:00
Luca Deri 46499a53aa Improved Facebook messnger mobile detection 2019-07-18 15:13:48 +02:00
Luca Deri 5a018e0bc0 Added Line protocol dissection
Add fix for discarding STUN over TCP flows
2019-07-15 00:19:45 +02:00