Commit Graph
712 Commits
Author SHA1 Message Date
Luca Deri bf318e0b86 Added NDPI_DESKTOP_OR_FILE_SHARING_SESSION flow risk 2021-04-11 14:42:27 +02:00
Luca Deri 28879c570e Reworked ndpi patricia includes to avoid compilation issues on some platforms 2021-03-31 19:43:55 +02:00
Luca Deri 48726301e1 Fixed mispelled word 2021-03-31 12:52:54 +02:00
Luca Deri c1d6e3f145 Improved mining detection support 2021-03-30 17:50:19 +02:00
Toni b040407683 Refactored nDPI subprotocol handling and aimini protocol detection. (#1156)
* Refactored and merged callback buffer routines for non-udp-tcp / udp / tcp / tcp-wo-payload.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Try to detect one subprotocol if a detected protocol can have one.

 * This adds a performance overhead due to much more protocol detection routine calls.
   See #1148 for more information.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Refactor subprotocol handling (1/2).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Refactor subprotocol handling (2/2).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Prevent some code duplication by using macros for ndpi_int_one_line_struct string comparision.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Refactored aimini HTTP detection parts (somehow related to #1148).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Added aimini client/server test pcap.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Removed master protocol as it was only used for STUN and via also removed API function ndpi_get_protocol_id_master_proto

 * Adjusted Python code to conform to the changes made during the refactoring process.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-03-23 11:46:12 +01:00
Luca Deri 9419015711 Implemented square erro rollup to avoid overflow 2021-03-14 11:01:51 +01:00
Luca 192fad4402 Added double exponential smoothing implementation 2021-03-11 09:39:52 +01:00
Luca Deri 6833ee2bbe Added single exponential smoothing API
int ndpi_ses_init(struct ndpi_ses_struct *ses, double alpha, float significance);
int ndpi_ses_add_value(struct ndpi_ses_struct *ses, const u_int32_t _value, double *forecast, double *confidence_band);
2021-03-11 00:04:33 +01:00
Luca Deri 5c6f070097 Fixed nDPI prefs definition 2021-03-09 23:50:31 +01:00
Luca Deri f6ad16d8f8 Added experiemntal JA3+ implementation that can be used with -z i ndpiReader 2021-03-09 23:38:29 +01:00
Luca Deri 3032864ec9 Added Ookla detection over IPv6 2021-03-09 12:55:14 +01:00
Luca Deri bb6423a79f Added the ability to define a custom DGA detection function by overwriting
the value of the function pointer ndpi_dga_function curently set to NULL
(that means the nDPI internal DGA function will be used)
2021-03-08 22:57:30 +01:00
Ivan Nardi c50a8d4808 Add support for Snapchat voip calls (#1147)
* Add support for Snapchat voip calls

Snapchat multiplexes some of its audio/video real time traffic with QUIC
sessions. The peculiarity of these sessions is that they are Q046 and
don't have any SNI.

* Fix tests with libgcrypt disabled
2021-03-06 05:48:36 +01:00
Alfredo Cardigliano 73e4348570 Add ndpi_serialize_binary_boolean for consistency. Fix comments. 2021-03-04 10:31:03 +01:00
Luca Deri 56bfb439f8 Improved DGA detection with trigrams. Disadvantage: slower startup time
Reworked Tor dissector embedded in TLS (fixes #1141)
Removed false positive on HTTP User-Agent
2021-03-03 00:41:07 +01:00
Toni 16890a6632 Added NDPI_MALICIOUS_SHA1 flow risk. (#1142)
* An external file which contains known malicious SSL certificate SHA-1 hashes
   can be loaded via ndpi_load_malicious_sha1_file(...)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-02-26 17:00:05 +01:00
Luca Deri 545f270132 Windows code cleanup 2021-02-24 11:04:04 +01:00
Alfredo Cardigliano 566d5db402 Add more utility functions to work with patricia trees 2021-02-23 11:53:15 +01:00
Alfredo Cardigliano 888ddfcd17 Update ndpi_patricia_walk_inorder API 2021-02-23 10:23:52 +01:00
Alfredo Cardigliano f8e83f7e35 Add support for MAC to Patricia tree. Expose full API to applications. Add utility functions. 2021-02-23 10:01:56 +01:00
Luca Deri f1b22b199f Added NDPI_MALICIOUS_JA3 flow risk
Added ndpi_load_malicious_ja3_file() API call
2021-02-22 23:19:23 +01:00
Luca Deri fc3db8f169 Implemented TLS Certificate Sibject matching
Improved AnyDesk detection
2021-02-22 22:37:33 +01:00
Luca Deri 5ad056e99b Removed unused NDPI_RISKY_COUNTRY 2021-02-21 21:45:46 +01:00
Luca Deri fc16c9368e Added risky domain flow-risk support 2021-02-21 21:45:46 +01:00
Luca Deri 92e8d95f38 Added ndpi_get_geoip() APi call 2021-02-18 23:15:27 +01:00
Luca Deri bce54079d8 Initial geoip support 2021-02-18 22:35:25 +01:00
Luca Deri fb4cdecb68 Fixes due to the fragment mananegr code 2021-02-18 13:31:02 +01:00
Luca Deri 4609e2084b Added new risks (future use)
-  NDPI_RISKY_ASN
-  NDPI_RISKY_DOMAIN
-  NDPI_RISKY_COUNTRY
2021-02-16 18:23:19 +01:00
Luca Deri ea0309d21b Removed now obsolete NDPI_DETECTION_SUPPORT_IPV6: code is more readeable now 2021-02-10 19:09:11 +01:00
Luca Deri e2f6569adb Fixed CPHA missing protocol initialization
Improved IEC104 and IRC detection
2021-02-10 15:22:20 +01:00
Luca Deri 1331e0aec9 Extended the API to calculate jitter
- ndpi_jitter_init()
- ndpi_jitter_free()
- ndpi_jitter_add_value()
2021-02-09 15:56:03 +01:00
Luca Deri 732579b72b Added timeseries forecasting support implementing Holt-Winters with confidence interval
New API calls added
- ndpi_hw_init()
- ndpi_hw_add_value()
- ndpi_hw_free()
2021-02-08 19:10:25 +01:00
Luca Deri 45e9c3c438 Partial fix for #1129 2021-02-05 22:22:33 +01:00
Luca Deri 8dd7716ae5 Implemented more efficient and memory savvy RSI 2021-02-05 12:38:41 +01:00
Luca Deri 60b58dbd67 RSI enhancements 2021-02-05 10:59:09 +01:00
Luca Deri 1eedf734be Implemented API for computing RSI (Relative Strenght Index)
void  ndpi_init_rsi(struct ndpi_rsi_struct *s, u_int16_t num_learning_values);
void  ndpi_free_rsi(struct ndpi_rsi_struct *s);
float ndpi_rsi_add_value(struct ndpi_rsi_struct *s, const u_int32_t value);
2021-02-04 23:52:33 +01:00
Ivan Nardi 8c0ea694f8 HTTP: fix user-agent parsing (#1124)
User-agent information is used to try to detect the user OS; since the
UA is extracted for QUIC traffic too, the "detected_os" field must be
generic and not associated to HTTP flows only.

Otherwise, you might overwrite some "tls_quic_stun" fields (SNI...) with
random data.

Strangely enough, the "detected_os" field is never used: it is never
logged, or printed, or exported...
2021-02-03 11:54:11 +01:00
b70ad0e2f1 fragments management added (#1122)
Management of tcp segments managements.

Co-authored-by: ragostino <ragostino73@gmail.com>
Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2021-02-03 10:28:51 +01:00
Luca Deri a31bd5ac3c Cleaned up tls/quic datatypes 2021-01-21 19:17:33 +01:00
Luca Deri 15295ef4c5 Reworked TLS fingerprint calcolation
Modified TLS memory free
2021-01-21 19:06:05 +01:00
Luca Deri 3e5e9569ff Added simple hash implementation to the nDPI API 2021-01-20 21:30:19 +01:00
Luca 0809956e5f Rewored UPnP protocol that in essence was WSD hence it has been renamed
Cleaned up TLS code for DTLS detection by defining a new DTLS protocol
2021-01-20 09:53:30 +01:00
Luca Deri 68b6ac7da8 (C) Update 2021-01-07 11:13:36 +01:00
Darryl Sokoloski 5f7b9d8024 Increase SNI hostname buffer length to 256. (#1111)
According to RFC 4366, SNI host names can be up to 255 bytes.
Previous size of 64 resulted in failed application matches due to truncation.

For example:
 0976e041e65b1aece3e720df36ac6bd7.safeframe.googlesyndication.co|m

Signed-off-by: Darryl Sokoloski <darryl@sokoloski.ca>
2021-01-07 10:58:48 +01:00
Ivan Nardi 2080cc7365 QUIC: add suppport for DNS-over-QUIC (#1107)
Even if it is only an early internet draft, DoQ has already (at least)
one deployed implementation.
See: https://www.zdnet.com/article/ad-blocker-adguard-deploys-worlds-first-dns-over-quic-resolver/
Draft: https://tools.ietf.org/html/draft-huitema-dprive-dnsoquic-00

In the future, if this protocol will be really used, it might be worth to
rename NDPI_PROTOCOL_DOH_DOT in NDPI_PROTOCOL_DOH_DOT_DOQ
2021-01-07 10:56:39 +01:00
Luca Deri eb37f8f1fb Split HTTP request from response Content-Type. Request Content-Type should be present with POSTs and not with other methods such as GET 2021-01-06 18:28:24 +01:00
Luca Deri cd21f0d316 Added check for invalid HTTP content 2021-01-06 12:25:32 +01:00
Ivan Nardi 23b84cd3ee Remove FB_ZERO protocol (#1102)
FB_ZERO was an experimental protocol run by Facebook.
They switched to QUIC/TLS1.3 more than 2 years ago; no one ever used it but
them so it is definitely dead.
See: https://engineering.fb.com/2018/08/06/security/fizz/
2021-01-04 15:49:19 +01:00
Toni 53ee6db795 Added a new API function `ndpi_free_flow_data' which free's all members of ndpi_flow_struct but not the struct itself. (#1101)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-01-04 12:47:05 +01:00
Luca Deri 05d76525b0 Added HTTP suspicious content securirty risk (useful for tracking trickbot) 2021-01-02 21:11:42 +01:00