Commit Graph
172 Commits
Author SHA1 Message Date
Luca Deri 00857abf2c Added new risk for clear text credentials 2021-09-10 22:00:04 +02:00
Ivan Nardi 9e0f0ce3df Fix access to some TLS fields in flow structure (#1277)
Fields 'tls.hello_processed` and `tls.subprotocol_detected` are used by
QUIC (i.e UDP...), too.
2021-08-20 18:11:37 +02:00
Ivan Nardi 8fdffbf3a1 Compile everything with "-W -Wall -Wno-unused-parameter" flags (#1276)
Fix all the warnings.

Getting rid of "-Wno-unused-parameter" is quite complex because some
parameters usage depends on compilation variable (i.e.
`--enable-debug-messages`).

The "-Werror" flag has been added only in Travis builds to avoid
breaking the builds to users using uncommon/untested
OS/compiler/enviroment.

Tested on:
* x86_64; Ubuntu 20.04; gcc 7,8,9,10,11; clang 7,8,9,10,11,12
* x86_64; CentOS 7.7; gcc 4.8.5 (with "--disable-gcrypt" flag)
* Raspberry 4; Debian 10.10; gcc 8.3.0
2021-08-20 18:11:13 +02:00
Luca Deri 0df8aa19ec Clode cleanup (after last merge) 2021-08-08 17:10:08 +02:00
ToniandLuca Deri 259b60f574 Added entropy calculation to check for suspicious (encrypted) payload. (#1270)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2021-08-08 17:03:23 +02:00
Luca Deri 4183718952 Added TLS fatal alert flow risk 2021-08-07 19:40:44 +02:00
Luca Deri 46ebd7128f Added ability to report whether a protocol is encrypted
Added new API calls
- u_int8_t ndpi_is_valid_protoId(u_int16_t protoId);
Cleaned up code and organized in a function.

- u_int8_t ndpi_is_encrypted_proto(struct ndpi_detection_module_struct *ndpi_str, ndpi_protocol proto);
Check whether a protocol is encrypted or not based on master and app protocols
2021-08-07 17:10:18 +02:00
Luca Deri 4cafa7cb1e Improved risk detection mask algorithm 2021-07-26 09:38:53 +02:00
Luca Deri 51e4888442 Implemented ALPN automa for checking uncommon ALPNs 2021-07-24 17:50:32 +02:00
Luca Deri 526568fcd5 Risk check improvement 2021-07-24 16:18:18 +02:00
Luca Deri 61fc5be202 Reworked flow risk implementation 2021-07-23 17:27:15 +02:00
Toni 6ad0d6666c Implemented function to retrieve flow information. #1253 (#1254)
* fixed [h]euristic typo

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-07-23 10:37:20 +02:00
Luca Deri b01b60a2b5 Implementation of flow risk eception (work in progress) 2021-07-22 01:35:57 +02:00
Toni 32275543c4 Added risk: TLS_EXTENSION_SUSPICIOUS (#1252)
* validates client/server hello TLS extensions
 * inspects content for some extensions

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-07-19 16:23:24 +02:00
Vitaly Lavrov daf1fbfc7b Code cleanup. (#1246)
ndpi_utils.c: use ndpi_malloc,ndpi_calloc,ndpi_free
genshin_impact.c, git.c, hpvirtgrp.c, http.c, z3950.c: removed "#include stdlib.h"
2021-07-16 13:51:54 +02:00
pacantandpacant 19a29e1e22 TLS Risks - Certificate Validity Too Long (#1239)
* Added flow risk: TLS certificate too long

* Added flow risk: TLS certificate too long

* Date for TLS limit added

* TLS certificate check fixed

Co-authored-by: pacant <a.pace97@outlook.com>
2021-07-14 11:13:22 +02:00
Vitaly Lavrov c418b7110b ahoсorasick. Code review. Part 2. (#1236)
Simplified the process of adding lines to AC_AUTOMATA_t.
Use the ndpi_string_to_automa() function to add patterns with domain names.
For other cases can use ndpi_add_string_value_to_automa().

ac_automata_feature(ac_automa, AC_FEATURE_LC) allows adding
and compare data in a case insensitive manner. For mandatory pattern comparison
from the end of the line, the "ac_pattern.rep.at_end=1" flag is used.
This eliminated unnecessary conversions to lowercase and adding "$" for
end-of-line matching in domain name patterns.

ac_match_handler() has been renamed ac_domain_match_handler() and has been greatly simplified.
ac_domain_match_handler() looks for the template with the highest domain level.
For special cases it is possible to manually specify the domain level.
Added test for checking ambiguous domain names like:
 - short.weixin.qq.com is QQ, not Wechat
 - instagram.faae1-1.fna.fbcdn.net is Instagram, not Facebook

If you specify a NULL handler when creating the AC_AUTOMATA_t structure,
then a pattern with the maximum length that satisfies the search conditions will be found
(exact match, from the beginning of the string, from the end of the string, or a substring).

Added debugging for ac_automata_search.
To do this, you need to enable debugging globally using ac_automata_enable_debug(1) and
enable debugging in the AC_AUTOMATA_t structure using ac_automata_name("name", AC_FEATURE_DEBUG).
The search will display "name" and a list of matching patterns.
Running "AHO_DEBUG=1 ndpiReader ..." will show the lines that were searched for templates
and which templates were found.

The ac_automata_dump() prototype has been changed. Now it outputs data to a file.
If it is specified as NULL, then the output will be directed to stdout.
If you need to get data as a string, then use open_memstream().

Added the ability to run individual tests via the do.sh script
2021-07-12 17:39:43 +02:00
Toni 8222ac5bdc Removed ht_hash as it is not used anymore. (#1220)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-29 15:32:48 +02:00
Vitaly Lavrov 35fc6a6de5 Speed and memory size optimisation (#1214)
Removed bigram_automata, impossible_bigram_automata, trigram_automata.
The ahocorasick structure is replaced with a bitmap.
The bitmap size for ndpi_en_bigram is 176 bytes.
The bitmap size for ndpi_en_trigram is 2201 bytes.
On the test machine, the test execution time was reduced from 27.3 seconds to 24.7 (9%).
2021-06-23 12:03:52 +02:00
Luca Deri 03ae090d39 Warnign fix 2021-06-03 09:05:53 +02:00
Luca c620858671 Reworked ndpi flow risk score adding client and server score 2021-06-01 09:17:26 +02:00
Alfredo Cardigliano c4084ca3c7 Fix warnings 2021-05-25 15:17:48 +02:00
Luca Deri d485c1cecb Improved TLS browser detection heuristics 2021-05-19 21:28:33 +02:00
Luca Deri ca15e3295e Added risk/score dump (ndpiReader -h)
Added ndpi_dump_risks_score() API score
2021-05-18 19:34:17 +02:00
Luca Deri 0b3d627185 Updated API for ndpi_risk2score()
Added ndpi_get_upper_proto() API call
2021-05-17 22:39:07 +02:00
Luca Deri 43a8576efb Reworked human readeable string search in flows
Removed fragment manager code
2021-05-17 20:55:06 +02:00
Luca Deri a62be9b8ec Implemented heuristic to detect Safari and Firefox TLS browsing 2021-05-13 12:37:07 +02:00
Luca Deri 4297a65ce8 Implemented flow score in Wireshark integration 2021-05-10 22:43:05 +02:00
Ivan Nardi 1deae1132b Fix some warnings (#1181)
```
In file included from protocols/fasttrack.c:29:
../include/ndpi_api.h:1504:3: warning: type qualifiers ignored on function return type [-Wignored-qualifiers]
 1504 |   const ndpi_risk_severity ndpi_risk2severity(ndpi_risk_enum risk);
      |   ^~~~~
In file included from protocols/amazon_video.c:28:
../include/ndpi_api.h:1504:3: warning: type qualifiers ignored on function return type [-Wignored-qualifiers]
 1504 |   const ndpi_risk_severity ndpi_risk2severity(ndpi_risk_enum risk);
      |   ^~~~~

...

ndpi_utils.c: In function ‘ndpi_risk2severity’:
ndpi_utils.c:1834:1: warning: control reaches end of non-void function [-Wreturn-type]
 1834 | }
      | ^
```
2021-05-09 15:11:05 +02:00
Luca ce54a6ab57 Added ndpi_risk2severity() API call 2021-05-02 19:35:01 +02:00
Toni da3e6bd61b Check for common ALPNs and set a flow risk if not known. (#1175)
* Increased risk bitmask to 64bit (instead of 32bit).
 * Removed annoying "Unknown datalink" error message for fuzzers.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-27 07:22:04 +02:00
Alfredo Cardigliano d95cd9912c Add vxlan enum to ndpi_packet_tunnel 2021-04-21 12:43:12 +02:00
Luca Deri bf318e0b86 Added NDPI_DESKTOP_OR_FILE_SHARING_SESSION flow risk 2021-04-11 14:42:27 +02:00
Luca Deri 56bfb439f8 Improved DGA detection with trigrams. Disadvantage: slower startup time
Reworked Tor dissector embedded in TLS (fixes #1141)
Removed false positive on HTTP User-Agent
2021-03-03 00:41:07 +01:00
Ivan Nardi 4c00ff89df DTLS: improve support (#1146)
* DTLS: add some pcap tests

* DTLS: fix parsing of Client/Server Helllo message

* DTLS: add parsing of server certificates
2021-03-02 21:15:40 +01:00
Toni 16890a6632 Added NDPI_MALICIOUS_SHA1 flow risk. (#1142)
* An external file which contains known malicious SSL certificate SHA-1 hashes
   can be loaded via ndpi_load_malicious_sha1_file(...)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-26 17:00:05 +01:00
Toni 70c35addcb Added protocol breed to JSON serializer. (#1137)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-25 08:46:38 +01:00
Luca Deri 4bd175b07e Modified JA3 fingerprint message 2021-02-24 10:42:26 +01:00
Luca Deri f1b22b199f Added NDPI_MALICIOUS_JA3 flow risk
Added ndpi_load_malicious_ja3_file() API call
2021-02-22 23:19:23 +01:00
Luca Deri 5ad056e99b Removed unused NDPI_RISKY_COUNTRY 2021-02-21 21:45:46 +01:00
Luca Deri fb4cdecb68 Fixes due to the fragment mananegr code 2021-02-18 13:31:02 +01:00
Luca Deri 4609e2084b Added new risks (future use)
-  NDPI_RISKY_ASN
-  NDPI_RISKY_DOMAIN
-  NDPI_RISKY_COUNTRY
2021-02-16 18:23:19 +01:00
Luca Deri ea0309d21b Removed now obsolete NDPI_DETECTION_SUPPORT_IPV6: code is more readeable now 2021-02-10 19:09:11 +01:00
Luca Deri 45e9c3c438 Partial fix for #1129 2021-02-05 22:22:33 +01:00
Alfredo Cardigliano 95a6a2072b Code cleanup and safety checks in the fragment manager (#1129) 2021-02-05 17:16:11 +01:00
Luca Deri 6e87daab69 Cosmetic fixes 2021-02-03 11:52:32 +01:00
b70ad0e2f1 fragments management added (#1122)
Management of tcp segments managements.

Co-authored-by: ragostino <ragostino73@gmail.com>
Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2021-02-03 10:28:51 +01:00
Luca Deri a31bd5ac3c Cleaned up tls/quic datatypes 2021-01-21 19:17:33 +01:00
Luca Deri 15295ef4c5 Reworked TLS fingerprint calcolation
Modified TLS memory free
2021-01-21 19:06:05 +01:00
Luca Deri 3e5e9569ff Added simple hash implementation to the nDPI API 2021-01-20 21:30:19 +01:00