Commit Graph
69 Commits
Author SHA1 Message Date
headshog a8d2eeddd2 Numeric truncation at ndpi_analyze.c at lines 101, 104, 107, 110 (#1999)
* fixed numeric truncation error in ndpi_analyze.c

* fixed numeric truncation error in ndpi_analyze.c x2

* fixed numeric truncation error in ndpi_analyze.c x3

* fixed numeric truncation error in ndpi_analyze.c and printf format

* fixed tests
2023-05-30 12:27:47 +02:00
headshog dd7198fcaf fixed numeric truncation error 2023-05-20 15:09:51 +02:00
Luca Deri 5ca6f0ac62 Implemented ndpi_predict_linear() for predicting a timeseries value overtime 2023-05-19 11:46:03 +02:00
Luca Deri 791d3e6ad3 Added missing check 2023-04-28 23:00:33 +02:00
Ivan Nardi 40b6d5a2e1 fuzz: extend fuzzers coverage (#1952) 2023-04-25 16:37:28 +02:00
Luca Deri c47e9d201d Implemented ndpi_XXX_reset() API calls whre XXX is ses, des, hw 2023-04-08 09:52:14 +02:00
Ivan Nardi b51a2ac72a fuzz: some improvements and add two new fuzzers (#1881)
Remove `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` define from
`fuzz/Makefile.am`; it is already included by the main configure script
(when fuzzing).

Add a knob to force disabling of AESNI optimizations: this way we can
fuzz also no-aesni crypto code.

Move CRC32 algorithm into the library.

Add some fake traces to extend fuzzing coverage. Note that these traces
are hand-made (via scapy/curl) and must not be used as "proof" that the
dissectors are really able to identify this kind of traffic.

Some small updates to some dissectors:

CSGO: remove a wrong rule (never triggered, BTW). Any UDP packet starting
with "VS01" will be classified as STEAM (see steam.c around line 111).
Googling it, it seems right so.

XBOX: XBOX only analyses UDP flows while HTTP only TCP ones; therefore
that condition is false.

RTP, STUN: removed useless "break"s

Zattoo: `flow->zattoo_stage` is never set to any values greater or equal
to 5, so these checks are never true.

PPStream: `flow->l4.udp.ppstream_stage` is never read. Delete it.

TeamSpeak: we check for `flow->packet_counter == 3` just above, so the
following check `flow->packet_counter >= 3` is always false.
2023-02-09 20:02:12 +01:00
Ivan Nardi 9fc724de5a Add some fuzzers to test other data structures. (#1870)
Start using a dictionary for fuzzing (see:
https://llvm.org/docs/LibFuzzer.html#dictionaries).
Remove some dead code.
Fuzzing with debug enabled is not usually a great idea (from performance
POV). Keep the code since it might be useful while debugging.
2023-01-25 11:44:59 +01:00
Ivan Nardi 5e8c1ebbb7 fuzz: fix memory allocation failure logic (#1867)
We *do* want to have some allocation errors.
Fix some related bugs
Fix: 29be01ef
2023-01-20 14:27:33 +01:00
Ivan Nardi 29be01ef3a Add some fuzzers to test algorithms and data structures (#1852)
Fix some issues found with these new fuzzers
2023-01-17 08:31:59 +01:00
Ivan Nardi ada4fe4aa8 fuzz: add a new fuzzer testing memory allocation failures (#1818)
Try to fuzz error paths triggered by allocation errors.
Fix some errors already found by this new fuzzer.
Basic idea taken from: https://github.com/harfbuzz/harfbuzz/pull/2566/files

`FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is a standard define used to
(not)compile specific code in fuzzing builds.
See: https://llvm.org/docs/LibFuzzer.html
2022-12-06 17:41:58 +01:00
Luca Deri 7578d02de9 Fixes on hw calculation 2022-09-01 22:06:57 +02:00
Luca Deri a53f476585 Modified definition of hll_add 2022-08-17 16:25:10 +02:00
Ivan Nardi b5fb2066cb bins: add support for 64bit bins (#1626) 2022-07-03 19:25:15 +02:00
Toni a83ca555fb Removed superfluous ifdef'd includes. (#1519)
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-19 17:51:48 +02:00
Toni Uhlig c3df3a12aa Fixed msys2 build warnings and re-activated CI Mingw64 build.
* Removed Visual Studio leftovers. Maintaining an autotools project with VS integration requires some additional overhead.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-14 19:17:48 +02:00
Luca Deri a6e2f4a15a Added ndpi_find_outliers() API call using Z-Score 2022-04-04 10:02:45 +02:00
Alfredo Cardigliano 23a4761276 Update copyright 2022-01-03 11:00:45 +01:00
Luca Deri fe2822c6a8 Added example for finding similarities in RRDs using nDPI statistical APIs 2021-12-04 10:09:01 +01:00
Luca Deri 1b077ba65f Demo tool for finding anomalies in RRD files 2021-12-03 17:36:49 +01:00
Luca Deri 937357e4bc Implemented ndpi_ses_fitting() and ndpi_des_fitting()
for comuting the best alpha/beta values for exponential smoothing
2021-10-12 13:08:58 +02:00
Luca Deri d13bc0e01a Fixed DES initialisation 2021-10-12 12:38:40 +02:00
Ivan Nardi 3e06e6f5b8 Fix some ndpi_malloc/ndpi_free pairs (#1298) 2021-09-13 18:25:22 +02:00
Alfredo Cardigliano 8e7b063804 Fix possible segfault/leak 2021-07-05 15:09:48 +02:00
Luca Deri 613e21002a Upgraded exponential smoothing to 64 bit values 2021-06-18 22:56:56 +02:00
Toni b040407683 Refactored nDPI subprotocol handling and aimini protocol detection. (#1156)
* Refactored and merged callback buffer routines for non-udp-tcp / udp / tcp / tcp-wo-payload.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Try to detect one subprotocol if a detected protocol can have one.

 * This adds a performance overhead due to much more protocol detection routine calls.
   See #1148 for more information.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Refactor subprotocol handling (1/2).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Refactor subprotocol handling (2/2).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Prevent some code duplication by using macros for ndpi_int_one_line_struct string comparision.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Refactored aimini HTTP detection parts (somehow related to #1148).

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Added aimini client/server test pcap.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Removed master protocol as it was only used for STUN and via also removed API function ndpi_get_protocol_id_master_proto

 * Adjusted Python code to conform to the changes made during the refactoring process.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-03-23 11:46:12 +01:00
Luca Deri 9419015711 Implemented square erro rollup to avoid overflow 2021-03-14 11:01:51 +01:00
Luca 192fad4402 Added double exponential smoothing implementation 2021-03-11 09:39:52 +01:00
Luca Deri 6833ee2bbe Added single exponential smoothing API
int ndpi_ses_init(struct ndpi_ses_struct *ses, double alpha, float significance);
int ndpi_ses_add_value(struct ndpi_ses_struct *ses, const u_int32_t _value, double *forecast, double *confidence_band);
2021-03-11 00:04:33 +01:00
Luca Deri 2d558afb54 Added HW checks 2021-03-01 19:59:14 +01:00
Luca Deri 2724a14bf7 Added further HW checks 2021-03-01 19:34:20 +01:00
Luca Deri 4bff595733 Holt-Winters calculation improvement 2021-02-27 11:32:51 +01:00
Luca Deri 09559611cc Fixes #1136 2021-02-19 15:15:02 +01:00
Ivan Nardi 421609475e Fix small memory leak (#1133)
Now function definition matches the prototype in ndpi_api.h.in
2021-02-10 15:24:34 +01:00
Luca Deri 1331e0aec9 Extended the API to calculate jitter
- ndpi_jitter_init()
- ndpi_jitter_free()
- ndpi_jitter_add_value()
2021-02-09 15:56:03 +01:00
Luca Deri b960809d3e Minor code improvements 2021-02-09 12:55:11 +01:00
Luca Deri 732579b72b Added timeseries forecasting support implementing Holt-Winters with confidence interval
New API calls added
- ndpi_hw_init()
- ndpi_hw_add_value()
- ndpi_hw_free()
2021-02-08 19:10:25 +01:00
Luca Deri 8dd7716ae5 Implemented more efficient and memory savvy RSI 2021-02-05 12:38:41 +01:00
Luca Deri 60b58dbd67 RSI enhancements 2021-02-05 10:59:09 +01:00
Luca Deri 1eedf734be Implemented API for computing RSI (Relative Strenght Index)
void  ndpi_init_rsi(struct ndpi_rsi_struct *s, u_int16_t num_learning_values);
void  ndpi_free_rsi(struct ndpi_rsi_struct *s);
float ndpi_rsi_add_value(struct ndpi_rsi_struct *s, const u_int32_t value);
2021-02-04 23:52:33 +01:00
Luca Deri 23a15bae5f Fixes #1029 2020-11-27 18:51:56 +01:00
Luca Deri 6c119c0b06 Fixes #1032 2020-11-27 18:48:25 +01:00
Luca Deri 948a906037 Added -D flag for detecting DoH in the wild
Removed heuristic from CiscoVPN as it leads to false positives
2020-10-26 21:40:59 +01:00
Luca Deri f597086386 Stddev calculation changes 2020-08-30 12:48:32 +02:00
Luca Deri dd75060932 Fixed false positive in suspicous user agent
Optimized stddev calculation
2020-08-30 12:25:15 +02:00
Luca Deri 21e2e57614 Fixed bin similarity 2020-07-24 01:30:58 +02:00
Luca Deri 439558f6a3 Improved bin clustering 2020-07-22 23:56:50 +02:00
Luca Deri 64fb40d21e Added ndpi_reset_data_analysis() API call 2020-07-17 13:16:01 +02:00
Luca Deri acd05b171e Added ndpi_data_last() API call 2020-07-16 11:22:06 +02:00
Luca Deri 46e97d5da3 Added ndpi_data_window_variance() and ndpi_data_window_stddev() API calls 2020-07-15 18:13:06 +02:00