Vladimir Gavrilov and Nardi Ivan
0180c1f04a
Add IEC62056 (DLMS/COSEM) protocol dissector ( #2229 )
...
* Add IEC62056 (DLMS/COSEM) protocol dissector
* Fix detection on big endian architectures
* Update protocols.rst
* Add ndpi_crc16_x25 to fuzz/fuzz_alg_crc32_md5.c
* Update pcap sample
* Remove empty .out file
* iec62056: add some documentation
---------
Co-authored-by: Nardi Ivan <nardi.ivan@gmail.com >
2024-01-02 16:45:54 +01:00
Ivan Nardi
3c7ed34ce9
fuzz: improve fuzzing coverage ( #2239 )
2024-01-02 15:22:44 +01:00
Luca
2f657cb8f9
Implemented ndpi_is_outlier() for detecting outliers using z-score
2023-12-28 19:59:54 +01:00
Luca Deri
1366518bff
Implements ndpi_pearson_correlation for measuring how correlated are two series
2023-12-27 22:42:37 +01:00
Luca Deri
99d4838328
New ndpi_sha256() nDPI API call ( #2230 )
...
* Added ndpi_sha256.c to the Windows project
* Added ndpi_sha256() nDPI API call
2023-12-23 10:47:28 +01:00
Ivan Nardi
7b5354588b
fuzz: extend fuzzing coverage ( #2208 )
2023-12-11 19:24:17 +01:00
Ivan Nardi
b3f2b1bb7f
STUN: rework extra dissection ( #2202 )
...
Keep looking for RTP packets but remove the monitoring concept.
We will re-introduce a more general concept of "flow in monitoring
state" later.
The function was disabled by default.
Some configuration knobs will be provided when/if #2190 is merged.
2023-12-11 14:53:12 +01:00
Vladimir Gavrilov
72601a69ff
Add some fast CRC16 algorithms implementation ( #2195 )
...
* Add some fast CRC16 algorithms implementation
* Update ndpi_crc.c
* Move crc16 stuff to ndpi_analyze.c
* IEEE C37.118: use new fast CRC-16/CCITT-FALSE implementation
2023-12-05 15:58:11 +01:00
Ivan Nardi
3b35cb37d9
Keep separating public and private API ( #2157 )
...
See: b08c787fe
2023-11-29 17:13:00 +01:00
Toni
0673da54b5
Fixed implicit u32 cast in ndpi_data_min() / ndpi_data_max(). ( #2139 )
...
Signed-off-by: Toni Uhlig <matzeton@googlemail.com >
2023-11-09 10:16:57 +01:00
Ivan Nardi
2c2eaf8e79
Rename some functions with more useful/clear names ( #2127 )
2023-10-29 20:21:24 +01:00
Ivan Nardi
03fd155ae3
IPv6: add support for custom categories ( #2126 )
2023-10-29 12:56:44 +01:00
Ivan Nardi
611c3b66f0
ipv6: add support for ipv6 addresses lists ( #2113 )
2023-10-26 20:15:44 +02:00
Luca Deri
b6179467eb
Added NDPI_MALWARE_HOST_CONTACTED flow risk
2023-10-13 00:01:55 +02:00
Nardi Ivan
4a0eda69ad
QUIC: export QUIC version as metadata
2023-10-11 15:15:20 +02:00
Toni
ef3adb9830
Added printf/fprintf replacement for some internal modules. ( #1974 )
...
* logging is instead redirected to `ndpi_debug_printf`
Signed-off-by: lns <matzeton@googlemail.com >
Signed-off-by: Toni Uhlig <matzeton@googlemail.com >
2023-09-26 23:10:57 +02:00
Ivan Nardi
d4e0b79dd6
Fix some prototypes ( #2085 )
...
```
error: function declaration isn’t a prototype [-Werror=strict-prototypes]
```
2023-09-18 14:06:47 +02:00
Ivan Nardi
c5cd4df056
Add ndpi_domain_classify_finalize() function ( #2084 )
...
The "domain classify" data structure is immutable, since it uses
"bitmap64".
Allow to finalize it before starting to process packets (i.e. before
calling `ndpi_domain_classify_contains()`) to avoid, in the data-path,
all the memory allocations due to compression.
Calling `ndpi_domain_classify_finalize()` is optional.
2023-09-12 19:47:14 +02:00
Ivan Nardi
ef6085370f
fuzz: add fuzzers to test bitmap64 and domain_classify data structures ( #2082 )
2023-09-10 18:44:50 +02:00
Luca
47fb5e9f3d
Cleanup
2023-09-07 11:12:37 +02:00
Luca Deri
978df906b3
Improved classification further reducing memory used
2023-09-05 17:03:20 +02:00
Luca Deri
c5ebd84fd8
Added ndpi_bitmap64 support
2023-09-05 00:21:16 +02:00
Luca Deri
4f2ce2d43b
Added ndpi_murmur_hash to the nDPI API
2023-09-04 12:53:42 +02:00
Luca Deri
1d480c18e3
Reworked domain classification based on binary filters
2023-09-02 19:16:40 +02:00
Luca Deri
74fdac7d77
Code cleanup
2023-09-01 08:50:55 +02:00
Luca Deri
133f45f3f3
Added ndpi_binary_bitmap datastruture
...
It is similar to ndpi_filter but based on binary search and with
the ability to store a category per value (as ndpi_domain_classify)
2023-08-31 23:51:50 +02:00
Luca Deri
16b0ce3710
Code cleanup
2023-08-31 09:14:17 +02:00
Luca Deri
f824939662
Added comment
2023-08-31 09:00:36 +02:00
Luca Deri
36abf06c6f
Swap from Aho-Corasick to an experimental/home-grown algorithm that uses a probabilistic
...
approach for handling Internet domain names.
For switching back to Aho-Corasick it is necessary to edit
ndpi-typedefs.h and uncomment the line
// #define USE_LEGACY_AHO_CORASICK
[1] With Aho-Corasick
$ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory
nDPI Memory statistics:
nDPI Memory (once): 37.34 KB
Flow Memory (per flow): 960 B
Actual Memory: 33.09 MB
Peak Memory: 33.09 MB
[2] With the new algorithm
$ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory
nDPI Memory statistics:
nDPI Memory (once): 37.31 KB
Flow Memory (per flow): 960 B
Actual Memory: 7.42 MB
Peak Memory: 7.42 MB
In essence from ~33 MB to ~7 MB
This new algorithm will enable larger lists to be loaded (e.g. top 1M domans
https://s3-us-west-1.amazonaws.com/umbrella-static/index.html )
In ./lists there are file names that are named as <category>_<string>.list
With -G ndpiReader can load all of them at startup
2023-08-29 17:34:04 +02:00
Luca Deri
2c565c77c9
Added ndpi_domain_classify_XXX(0 API
2023-08-26 00:24:33 +02:00
Luca Deri
eb9932164f
added bimap and/or with allocation
2023-08-24 07:45:38 +02:00
Luca Deri
ff4f291afe
Minor improvements
2023-08-23 11:04:31 +02:00
Luca
95fcd08687
Added ndpi_bitmap_is_empty() and ndpi_bitmap_optimize() API calls
2023-08-23 11:03:59 +02:00
Luca
6e5466f9e5
Added ndpi_bitmap_andnot API call
2023-08-21 13:26:38 +02:00
Ivan Nardi
cc4461f424
fuzz: extend coverage ( #2073 )
2023-08-20 15:18:19 +02:00
Luca Deri
049fbf819c
Reworked ndpi_filter_xxx implementation using compressed bitmaps
2023-08-14 12:50:54 +02:00
Luca Deri
8e17f50853
Added ndpi_filter_add_multi() API call
2023-08-11 23:55:44 +02:00
Luca Deri
ec7adc212e
Added new API calls for implementing Bloom-filter like data structures
...
ndpi_filter* ndpi_filter_alloc(uint32_t elements_number);
bool ndpi_filter_add(ndpi_filter *f, uint64_t value);
bool ndpi_filter_contains(ndpi_filter *f, uint64_t value);
void ndpi_filter_free(ndpi_filter *f);
2023-08-11 22:49:31 +02:00
Luca
33e9dbc24d
Added ndpi_bitmap_copy() API call
2023-08-06 09:35:56 +02:00
Ivan Nardi
3326fa258e
Add an heuristic to detect fully encrypted flows ( #2058 )
...
A fully encrypted session is a flow where every bytes of the
payload is encrypted in an attempt to “look like nothing”.
The heuristic needs only the very first packet of the flow.
See: https://www.usenix.org/system/files/sec23fall-prepub-234-wu-mingshi.pdf
A basic, but generic, inplementation of the popcpunt alg has been added
2023-07-26 09:09:12 +02:00
Toni
2b230e28e0
Allow init of app protocols w/o any hostnames set. ( #2057 )
2023-07-22 14:51:33 +02:00
Luca Deri
fea09e825b
Fixes risk mask exception handling while improving the overall performance
2023-07-14 19:52:34 +02:00
Luca Deri
1f55dc511f
Implemented Count-Min Sketch [count how many times a value has been observed]
...
- ndpi_cm_sketch_init()
- ndpi_cm_sketch_add()
- ndpi_cm_sketch_count()
- ndpi_cm_sketch_destroy()
2023-07-13 21:54:51 +02:00
Ivan Nardi
3608ab01b6
STUN: keep monitoring/processing STUN flows ( #2012 )
...
Look for RTP packets in the STUN sessions.
TODO: tell RTP from RTCP
2023-06-21 09:16:20 +02:00
Luca Deri
63a2b359b9
Added vlan_id in ndpi_flow2json() prototype
2023-06-09 19:50:47 +02:00
headshog
a8d2eeddd2
Numeric truncation at ndpi_analyze.c at lines 101, 104, 107, 110 ( #1999 )
...
* fixed numeric truncation error in ndpi_analyze.c
* fixed numeric truncation error in ndpi_analyze.c x2
* fixed numeric truncation error in ndpi_analyze.c x3
* fixed numeric truncation error in ndpi_analyze.c and printf format
* fixed tests
2023-05-30 12:27:47 +02:00
Toni
04f5c5196e
Changed logging callback function sig. ( #2000 )
...
* make user data available for any build config
Signed-off-by: lns <matzeton@googlemail.com >
2023-05-30 08:27:37 +02:00
headshog
dd7198fcaf
fixed numeric truncation error
2023-05-20 15:09:51 +02:00
Luca Deri
5ca6f0ac62
Implemented ndpi_predict_linear() for predicting a timeseries value overtime
2023-05-19 11:46:03 +02:00
Yuriy Rymarchuk
a8b3baf2b8
Add support for roaring_bitmap_xor_inplace ( #1983 )
2023-05-18 13:28:32 +02:00