Commit Graph
2244 Commits
Author SHA1 Message Date
Luca Deri 708d4ea33a Improved user agent analysis 2022-01-09 18:47:47 +01:00
Ivan Nardi bf69321a29 GTP: fix some false positives (#1394) 2022-01-08 20:40:24 +01:00
Ivan Nardi 4d27001d1d Remove some unused fields (#1393) 2022-01-08 20:40:10 +01:00
Luca Deri eac3ae9ded Improved bittorrent detection 2022-01-06 19:50:31 +01:00
Luca Deri f24f68f59d Invalid check 2022-01-06 07:15:45 +01:00
Alfredo Cardigliano 23a4761276 Update copyright 2022-01-03 11:00:45 +01:00
Luca Deri d72a40a0fc Improved BitTorrent guess 2021-12-26 19:41:35 +01:00
Luca Deri c4ac53a03f Added support for Log4J/Log4Shell detection in nDPI via a new flow risk named NDPI_POSSIBLE_EXPLOIT 2021-12-23 21:30:16 +01:00
Ivan Nardi fdb6481cd6 Add support for ICloud Private Relay (#1390)
See:
https://www.apple.com/privacy/docs/iCloud_Private_Relay_Overview_Dec2021.PDF

TODO: an up-to-date list of egress IP ranges is publicly available. Can
we use it somehow?
2021-12-22 19:55:57 +01:00
Ivan Nardi 91bb77a880 A final(?) effort to reduce memory usage per flow (#1389)
Remove some unused fields and re-organize other ones.
In particular:
* Update the parameters of `ndpi_ssl_version2str()` function
* Zattoo, Thunder: these timestamps aren't really used.
* Ftp/mail: these protocols are dissected only over TCP.
* Attention must be paid to TLS.Bittorrent flows to avoid invalid
read/write to `flow->protos.bittorrent.hash` field.

This is the last(?) commit of a long series (see 22241a1d, 227e586e,
730c2360, a8ffcd8b) aiming to reduce library memory consumption.

Before, at nDPI 4.0 (more precisly, at a6b10cf7, because memory stats
were wrong until that commit):
```
nDPI Memory statistics:
	nDPI Memory (once):      221.15 KB
	Flow Memory (per flow):  2.94 KB
```
Now:
```
nDPI Memory statistics:
	nDPI Memory (once):      231.71 KB
	Flow Memory (per flow):  1008 B       <---------
```
i.e. memory usage per flow has been reduced by 66%, dropping below the
psychological threshold of 1 KB.

To further reduce this value, we probably need to look into #1279:
let's fight this battle another day.
2021-12-22 19:54:06 +01:00
Luca Deri 4c02c81bf4 Improved bittorrent heuristic 2021-12-21 23:04:27 +01:00
Ivan Nardi a16b4aa564 Fix a global-buffer-overflow when loading Azure list (#1388) 2021-12-19 22:29:25 +01:00
Luca Deri 7aad97fffb Added Microsoft Azure support 2021-12-19 18:59:45 +01:00
Ivan Nardi 7153b8933c Improve/add several protocols (#1383)
Improve Microsoft, GMail, Likee, Whatsapp, DisneyPlus and Tiktok
detection.
Add Vimeo, Fuze, Alibaba and Firebase Crashlytics detection.

Try to differentiate between Messenger/Signal standard flows (i.e chat)
and their VOIP (video)calls (like we already do for Whatsapp and
Snapchat).

Add a partial list of some ADS/Tracking stuff.

Fix Cassandra, Radius and GTP false positives.
Fix DNS, Syslog and SIP false negatives.

Improve GTP (sub)classification: differentiate among GTP-U, GTP_C and
GTP_PRIME.

Fix 3 LGTM warnings.
2021-12-18 13:24:51 +01:00
Ivan Nardi 58b33dcb2d TLS: add support for IPV6 in Subject Alt Names field (#1385) 2021-12-18 12:03:46 +01:00
Ivan Nardi ec6eea0c23 TLS: fix usage of certificate cache (#1384) 2021-12-18 12:03:32 +01:00
Luca Deri 74519f6e10 Improved bittorrent detection 2021-12-17 21:17:18 +01:00
Ivan Nardi f11b215cfb QUIC: fix old versions of GQUIC on big-endian machines (#1387)
Credits to @viniciussn (see #1312)
2021-12-17 12:10:34 +01:00
Luca Deri 8e512ff503 Improved BitTorrent classification 2021-12-07 21:49:00 +01:00
Luca Deri 519d70fa04 Optimized ndpi_serialize_string_int32 2021-12-07 15:54:23 +01:00
Luca Deri 85b396ca43 Fixed issue that prevented alt certificate names to be fully detected when ipAddress and rfc822Name were specified in certificates 2021-12-07 14:45:42 +01:00
Ivan Nardi 947896ad7d Fix configure script (after fb85dac9) (#1381)
Fix/disable some LGTM warnings
2021-12-04 20:22:05 +01:00
Ivan Nardi 7cba34a1e3 Improve IPv6 support, enabling IPv6 traffic on (almost) all dissectors. (#1380)
There are no valid reasons for a (generic) protocol to ignore IPv6
traffic.
Note that:
* I have not found the specifications of "CheckPoint High Availability
Protocol", so I don't know how/if it supports IPv6
* all LRU caches are still IPv4 only

Even if src_id/dst_id stuff is probably useless (see #1279), the right
way to update the protocol classification is via `ndpi_set_detected_protocol()`
2021-12-04 13:30:10 +01:00
Ivan Nardi 6e86e6d924 QUIC: add support for QUICv2 (draft 00) (#1379)
It is already time to start looking at the new QUIC version.
See: https://datatracker.ietf.org/doc/html/draft-ietf-quic-v2-00
2021-12-04 13:29:30 +01:00
Toni 4d60c70c8b Make serialize risk and proto not dependant on any flow. (#1377)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-12-04 13:28:55 +01:00
Luca Deri fe2822c6a8 Added example for finding similarities in RRDs using nDPI statistical APIs 2021-12-04 10:09:01 +01:00
Luca Deri 1b077ba65f Demo tool for finding anomalies in RRD files 2021-12-03 17:36:49 +01:00
Alfredo Cardigliano 3dd9b5176e Remove printf when not required 2021-12-01 10:24:12 +01:00
Alfredo Cardigliano 1d3428d538 Improve JSON serialization 2021-12-01 10:04:17 +01:00
Luca Deri 33557c5fbc Improved CSV serialization 2021-11-30 22:15:26 +01:00
Luca Deri 5e0befd56c Fixed seriealizer unit 2021-11-30 17:24:42 +01:00
Luca Deri 4f81cdc103 Removed spaces from some protocol names as they create problems with some applications 2021-11-30 10:12:24 +01:00
Luca Deri 320bcfafa2 Added ndpi_serializer_skip_header() serialization API 2021-11-26 22:18:59 +01:00
Luca Deri a7b5e09195 Added Salesforce detection 2021-11-26 19:07:45 +01:00
Luca Deri 9d59b8b1a9 HTTP proxy improvement 2021-11-25 23:17:27 +01:00
Luca Deri ea435c46f5 Reworked HTTP protocol dissection including HTTP proxy and HTTP connect 2021-11-25 22:53:46 +01:00
Luca Deri 772957349b Aded default port to HTTP_CONNECT 2021-11-25 21:46:00 +01:00
Luca Deri 40ff2c54c1 Added googletagmanager.com to the host_match list 2021-11-25 21:45:38 +01:00
Ivan Nardi a8ffcd8bb0 Rework how hostname/SNI info is saved (#1330)
Looking at `struct ndpi_flow_struct` the two bigger fields are
`host_server_name[240]` (mainly for HTTP hostnames and DNS domains) and
`protos.tls_quic.client_requested_server_name[256]`
(for TLS/QUIC SNIs).

This commit aims to reduce `struct ndpi_flow_struct` size, according to
two simple observations:
 1) maximum one of these two fields is used for each flow. So it seems safe
to merge them;
 2) even if hostnames/SNIs might be very long, in practice they are rarely
longer than a fews tens of bytes. So, using a (single) large buffer is a
waste of memory for all kinds of flows. If we need to truncate the name,
we keep the *last* characters, easing domain matching.

Analyzing some real traffic, it seems safe to assume that the vast
majority of hostnames/SNIs is shorter than 80 bytes.

Hostnames/SNIs are always converted to lowercase.

Attention was given so as to be sure that unit-tests outputs are not
affected by this change.

Because of a bug, TLS/QUIC SNI were always truncated to 64 bytes (the
*first* 64 ones): as a consequence, there were some "Suspicious DGA
domain name" and "TLS Certificate Mismatch" false positives.
2021-11-24 10:46:48 +01:00
Ivan Nardi fd02e1b304 QUIC: fix extra dissection (#1376)
When we have fully reassembled the Client Hello, we need to stop extra
dissection.
2021-11-23 10:57:13 +01:00
Luca Deri 4c39ed293e BitTorrent dissector code cleanup and merge of https://github.com/ntop/nDPI/pull/1374 2021-11-17 23:18:22 +01:00
Luca Deri 2e2d4c911a BitTorrent detection improvements 2021-11-17 22:41:35 +01:00
Luca Deri 2a6e5c6341 Improved BT hashing to minimize false positives 2021-11-17 10:47:21 +01:00
Luca Deri a6545ea095 bittorrent old code cleanup. Enlarged BT cache 2021-11-16 10:05:18 +01:00
Luca Deri f2a279e441 Improved BitTorrent detection 2021-11-15 17:25:13 +01:00
Ivan Nardi afc2b641eb Fix writes to flow->protos union fields (#1354)
We can write to `flow->protos` only after a proper classification.

This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP,
SMTP, IMAP and POP code.
There are two kinds of fixes:
 * write to `flow->protos` only if a final protocol has been detected
 * move protocol state out of `flow->protos`
The hard part is to find, for each protocol, the right tradeoff between
memory usage and code complexity.

Handle Kerberos like DNS: if we find a request, we set the protocol
and an extra callback to further parsing the reply.

For all the other protocols, move the state out of `flow->protos`. This
is an issue only for the FTP/MAIL stuff.

Add DHCP Class Identification value to the output of ndpiReader and to
the Jason serialization.

Extend code coverage of fuzz tests.

Close #1343
Close #1342
2021-11-15 16:20:57 +01:00
ragostino da47357762 Update ndpi_main.c for HTTP line parser optimization and cleaning (#1372)
* Update ndpi_main.c

HTTP line parser: optimization and cleaning

* Update ndpi_main.c
2021-11-11 12:38:31 +01:00
Ivan Nardi 3e5491fa10 Add detection of OCSP (#1370)
This protocol is detected via HTTP Content-Type header.

Until 89d548f9, nDPI had a dedicated automa (`content_automa`) to
classify a HTTP flow according to this header. Since then, this automa has
been useless because it is always empty.
Re-enable it to match only a string seems overkilling.

Remove all `content_automa` leftovers.
2021-11-11 12:36:55 +01:00
Ivan Nardi 0f168d9150 IMAP, POP3, SMTP: improve dissection (#1368)
Avoid NATS false positives
2021-11-11 11:55:56 +01:00
Luca Deri 9abf528c8e BitTorrent cache debug 2021-11-08 12:42:15 +01:00