Commit Graph
59 Commits
Author SHA1 Message Date
Ivan NardiandLuca Deri a6ff0dd0e3 Add few scripts to easily update some IPs lists (#1436)
* Add few scripts to easily update some IPs lists

Some IPs lists should be updated frequently: try to easy the process.
The basic idea is taken from d59fefd0 and a8fe74e5 (for Azure
addresses): one specific .c.inc file and one script for each protocol.

Add the possibility to don't load a specific list.

Rename the old NDPI_PROTOCOL_HOTMAIL id to NDPI_PROTOCOL_MS_OUTLOOK,
to identify Hotmail/Outlook/Exchange flows.

TODO: ipv6

Remove the 9 addresses associated to BitTorrent: they have been added in
e2f21116 but it is not clear why all the traffic to/from these ips
should be classified as BitTorrent.

* Added quotes

* Added quotes

Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2022-02-09 11:45:48 +01:00
Ivan Nardi 3a087e951d Add a "confidence" field about the reliability of the classification. (#1395)
As a general rule, the higher the confidence value, the higher the
"reliability/precision" of the classification.

In other words, this new field provides an hint about "how" the flow
classification has been obtained.
For example, the application may want to ignore classification "by-port"
(they are not real DPI classifications, after all) or give a second
glance at flows classified via LRU caches (because of false positives).

Setting only one value for the confidence field is a bit tricky: more
work is probably needed in the next future to tweak/fix/improve the logic.
2022-01-11 15:23:39 +01:00
Ivan Nardi 7153b8933c Improve/add several protocols (#1383)
Improve Microsoft, GMail, Likee, Whatsapp, DisneyPlus and Tiktok
detection.
Add Vimeo, Fuze, Alibaba and Firebase Crashlytics detection.

Try to differentiate between Messenger/Signal standard flows (i.e chat)
and their VOIP (video)calls (like we already do for Whatsapp and
Snapchat).

Add a partial list of some ADS/Tracking stuff.

Fix Cassandra, Radius and GTP false positives.
Fix DNS, Syslog and SIP false negatives.

Improve GTP (sub)classification: differentiate among GTP-U, GTP_C and
GTP_PRIME.

Fix 3 LGTM warnings.
2021-12-18 13:24:51 +01:00
Ivan Nardi b1e9245d94 ndpiReader: slight simplificaton of the output (#1378) 2021-11-27 17:32:23 +01:00
Ivan Nardi 5464bad6db Differentiate between standard Amazon stuff (i.e market) and AWS (#1369) 2021-11-04 00:20:45 +01:00
Ivan Nardi b6d9536533 Fixed cleartext protocol assignment (#1357) 2021-10-25 15:04:04 +02:00
Luca Deri 9e97d20c25 Refreshed results list 2021-10-16 12:03:16 +02:00
Luca Deri e8455236bd Updated output 2021-08-07 17:38:33 +02:00
Ivan Nardi cccf794265 ndpiReader: add statistics about nDPI performance (#1240)
The goal is to have a (roughly) idea about how many packets nDPI needs
to properly classify a flow.

Log this information (and guessed flows number too) during unit tests,
to keep track of improvements/regressions across commits.
2021-07-13 12:28:39 +02:00
Luca Deri 43a8576efb Reworked human readeable string search in flows
Removed fragment manager code
2021-05-17 20:55:06 +02:00
Luca Deri ac1eaca8a6 Added browser TLS heuristic 2021-05-13 20:00:27 +02:00
Luca Deri fba61adf5e Improved DNS dissector 2021-02-26 00:42:15 +01:00
Luca Deri be58d78417 (Temporarely) Disabled fragment manager (#1129) 2021-02-05 23:03:08 +01:00
Luca Deri 155b1aaa68 Updated results due to the new fragment manager handler 2021-02-03 11:52:46 +01:00
Luca Deri 044ed14b4f Various optimizations to reduce not-necessary calls
Optimized various UDP dissectors
Removed dead protocols such as pando and pplive
2020-09-24 23:26:03 +02:00
Luca Deri d81bc1add6 Reworked MDNS dissector that is not based on the DNS dissector 2020-09-17 23:24:02 +02:00
Nardi Ivan 23ec82b59d Major rework of QUIC dissector
Improve support for GQUIC (up to Q046) and add support for Q050 and (IETF-)QUIC
Still no sub-classification for Q050 and QUIC
2020-08-21 22:04:55 +02:00
Luca Deri 2ae4c6675d Fixed partial TLS dissection 2020-07-30 18:30:07 +02:00
Luca Deri e71df49b3e Changed due to bin size extension 2020-07-30 00:06:46 +02:00
Luca Deri 12abcd516b Updated test results due to bin changes 2020-07-09 17:28:02 +02:00
Luca Deri d9af1562f0 Fixes #906
Packet bins are not printed wehn empty
2020-06-22 14:30:26 +02:00
Luca Deri 1a62f4c799 Added ndpi_bin_XXX API
Added packet lenght distribution bins
2020-06-22 01:02:54 +02:00
Luca Deri 4a09b4efa0 Added TLS issuerDN and subjectDN 2020-05-07 18:44:51 +02:00
Luca Deri 748df7972a Results update 2020-04-17 10:36:27 +02:00
emanuele-f fd94270507 Remove decimals in test results for IAT, packet lengths and goodput ratio 2020-02-14 11:42:20 +01:00
Luca Deri 5cad39f0e8 Added export of TLS supported version in TLS header 2020-02-07 23:26:03 +01:00
Luca Deri 3be263aafc Added TLS ALPN support 2020-02-07 21:54:04 +01:00
Luca Deri 0703ab5ac5 Improved DNS response decoding
The first decoded address is now reported by ndpiReader
2020-02-04 22:16:54 +01:00
Luca daae1cc9b1 Reworked TLS dissection 2020-01-01 12:59:19 +01:00
Luca Deri e98b994a39 Updated results 2019-11-21 13:35:04 +01:00
Luca Deri 0974075fa0 Major cleanup
Removed ndpi_pref_http_dont_dissect_response and ndpi_pref_dns_dont_dissect_response as the ndpi_extra_dissection_possible() call will now handle everything
2019-10-24 19:48:55 +02:00
Luca Deri 4fd7e5734a Manual merge of pull #769 2019-10-02 23:01:29 +02:00
Luca Deri f2a5bbef17 Reworked categories handling
Removed GenericProtocol and replaced with categories
Removed ndpi_pref_enable_category_substring_match option: substring matching is now default
2019-09-29 21:46:41 +02:00
Luca Deri c839dcb74c Improved category handlign in subprotocols
Further DNS dissection fixes
Fixed WeChat invalid category
2019-09-27 17:34:22 +02:00
Luca 0ed679e795 Improves IAT calculation 2019-09-24 16:37:42 +02:00
Luca Deri 11e9fd8cef As TLS certificate fingerprint is computed, TLS without certificate protocol has been removed
Various improvemenets in detection quality
2019-09-15 12:18:24 +02:00
Luca Deri 0f9107bef3 Fixed timezone conversion 2019-09-10 22:15:23 +02:00
Luca Deri 068611e8ba Changes in validity format 2019-09-10 21:44:42 +02:00
Luca Deri 65bdd0de9a Uddated results 2019-09-10 00:43:03 +02:00
Luca Deri 086c511a11 Parsed TLS certificate validity 2019-09-10 00:34:48 +02:00
Luca 886d575157 Added -C to generate CSV analysis files
Improved IAT and byte distribution
2019-09-03 18:38:54 +02:00
Luca b1270fc7bb Uodated results 2019-08-29 15:23:01 +02:00
Luca e4e40e3c70 Added entropy, average, stddev, variance, bytes ratio calculation 2019-08-28 14:02:39 +02:00
Luca c6e832be18 Various TLS/STUN improvememnts 2019-08-08 18:09:12 +02:00
Luca 419160f351 Implemented DTLS support
Renamed ssl to tls
2019-08-08 12:45:27 +02:00
Luca Deri 3c4a959c1e Updated results when adding client hello 2019-08-07 20:06:31 +02:00
Luca Deri b715467da6 Added -e option to ndpiReader for searchign human readeable strings lenght
Default human readeable strings lenght is not 5 chars (used to be 8)
2019-07-24 00:25:00 +02:00
Luca Deri 63173e7360 Updated results with new dissection 2019-07-24 00:13:07 +02:00
Luca Deri b8867642fc Refresh after data leak detection 2019-07-18 11:49:53 +02:00
Luca Deri 5c758d4564 Reworked output
-v 1 now prints only unknown flows
-v 2 now prints all flows
2019-07-15 14:45:25 +02:00