Commit Graph
80 Commits
Author SHA1 Message Date
Luca Deri c5ebd84fd8 Added ndpi_bitmap64 support 2023-09-05 00:21:16 +02:00
Ivan Nardi 5867f43fae Fix compilation on Windows (#2072) 2023-08-14 22:51:09 +02:00
Luca Deri 049fbf819c Reworked ndpi_filter_xxx implementation using compressed bitmaps 2023-08-14 12:50:54 +02:00
Luca Deri 3b34941be1 Typo 2023-08-12 11:39:19 +02:00
Luca Deri 9a44954365 ARM compilation fix 2023-08-12 07:39:35 +02:00
Luca Deri 890dcf91d9 Minor initialization fixes 2023-08-11 23:51:16 +02:00
Luca Deri ec7adc212e Added new API calls for implementing Bloom-filter like data structures
ndpi_filter* ndpi_filter_alloc(uint32_t elements_number);
bool         ndpi_filter_add(ndpi_filter *f, uint64_t value);
bool         ndpi_filter_contains(ndpi_filter *f, uint64_t value);
void         ndpi_filter_free(ndpi_filter *f);
2023-08-11 22:49:31 +02:00
Ivan Nardi efb261a95c Fix some memory errors triggered by allocation failures (#1995)
Some low hanging fruits found using nallocfuzz.
See: https://github.com/catenacyber/nallocfuzz
See: https://github.com/google/oss-fuzz/pull/9902

Most of these errors are quite trivial to fix; the only exception is the
stuff in the uthash.
If the insertion fails (because of an allocation failure), we need to
avoid some memory leaks. But the only way to check if the `HASH_ADD_*`
failed, is to perform a new lookup: a bit costly, but we don't use that
code in any critical data-path.
2023-05-29 19:24:00 +02:00
Toni b6629ba2be Improved debug output. (#1951)
* try to get rid of some `printf(..)`s as they do not belong to a shared library
 * replaced all `exit(..)`s with `abort()`s to indicate an abnormal process termination

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-04-21 12:40:26 +02:00
Ivan Nardi cc5aec5f0a fuzz: add fuzzer to test internal gcrypt code (#1920) 2023-04-04 14:19:41 +02:00
Ivan Nardi d7662379b4 Update libinjection code (#1918)
Update libinjection code to the current master https://github.com/libinjection/libinjection/commit/7e4b74e824dc3f0623ba4894df2a0d817351ec39

The goal is to finally fix #1820
See: https://github.com/libinjection/libinjection/issues/33

Update the corpus of the libinjection fuzzers

Close #1820
2023-04-04 14:18:57 +02:00
Luca Deri bf413afba1 Update roaring bitmap code 2023-02-12 12:50:24 +01:00
Ivan Nardi 9fc724de5a Add some fuzzers to test other data structures. (#1870)
Start using a dictionary for fuzzing (see:
https://llvm.org/docs/LibFuzzer.html#dictionaries).
Remove some dead code.
Fuzzing with debug enabled is not usually a great idea (from performance
POV). Keep the code since it might be useful while debugging.
2023-01-25 11:44:59 +01:00
Ivan Nardi 29be01ef3a Add some fuzzers to test algorithms and data structures (#1852)
Fix some issues found with these new fuzzers
2023-01-17 08:31:59 +01:00
Ivan Nardi 197a9a6cfc Extend CodeQL configuration and fix two warnings (#1816)
```
Call to `memset` may be deleted
Comparison of narrow type with wide type in loop condition
```

See https://www.usenix.org/system/files/conference/usenixsecurity17/sec17-yang.pdf
for the new portable function `secure_memzero`.
2022-12-02 23:26:22 +01:00
Nardi Ivan e5063f9883 Fix some warnings
Ubuntu-20.04, clang-16 (nightly build)

```
Making all in src/lib
protocols/smpp.c:70:17: warning: variable 'pdu_c' set but not used [-Wunused-but-set-variable]
      u_int16_t pdu_c = 1;
                ^
1 warning generated.
third_party/src/ahocorasick.c:173:20: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion]
  thiz->root->root = 1;
                   ^ ~
third_party/src/ahocorasick.c:336:15: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion]
        n->ff = 1;
              ^ ~
third_party/src/ahocorasick.c:716:21: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion]
        node->final = 1;
[...]
```
2022-09-05 23:44:51 +02:00
Luca Deri a53f476585 Modified definition of hll_add 2022-08-17 16:25:10 +02:00
Ivan Nardi 405a52ed65 Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1683)
Add (basic) internal stats to the main data structures used by the
library; they might be usefull to check how effective these structures
are.

Add an option to `ndpiReader` to dump them; enabled by default in the
unit tests.
This new option enables/disables dumping of "num dissectors calls"
values, too (see b4cb14ec).
2022-07-29 15:25:00 +02:00
Ivan Nardi df2e11ef51 Revert "Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1677)" (#1682)
This reverts commit bb83899985.
2022-07-29 12:08:40 +02:00
Ivan Nardi bb83899985 Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1677)
Add (basic) internal stats to the main data structures used by the
library; they might be usefull to check how effective these structures
are.

Add an option to `ndpiReader` to dump them; disabled by default to avoid
too much fuss with the unit tests.
2022-07-29 12:07:41 +02:00
Ivan Nardi ac1113b9ac ahocorasick: fix char/uchar bug (#1597)
Porting of https://github.com/vel21ripn/nDPI/commit/06e2967d0c26ab214683a2f5565f4012fd523537
Close #1506
2022-07-20 10:59:17 +02:00
Toni 9c8b2d63da Replaced nDPI's internal hashmap with uthash. (#1602)
Signed-off-by: lns <matzeton@googlemail.com>
2022-06-17 19:50:31 +02:00
Toni Uhlig c3df3a12aa Fixed msys2 build warnings and re-activated CI Mingw64 build.
* Removed Visual Studio leftovers. Maintaining an autotools project with VS integration requires some additional overhead.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-14 19:17:48 +02:00
Vitaly Lavrov 438f04066b Fixed a bug for BE architectures (#1478)
Fixed a bug in the internal implementation of libgcrypt for bigendian architectures
2022-03-05 11:44:21 +01:00
Toni cb62dfd249 Drop support for non-gcrypt builds. (#1469)
* As there is now a builtin, lightweight libgcrypt
   there is no need to disable tls-clho decryption.
 * It is still possible to use a host libgcrypt
   with `--with-local-libgcrypt'.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-02 19:48:46 +01:00
Ivan Nardi e23507d3e9 Internal crypto: increase size of authentication buffer (#1468)
Some QUIC flows are not properly decoded while using internal crypto
code: the authentication buffer is too small.
The new value (like the old one) is arbitrary.

Close #1463
2022-03-02 15:14:57 +01:00
Vitaly Lavrov f2411958d7 Added lightweight implementation of libgcrypt. (#1444)
Implementation borrowed from the
https://github.com/ARMmbed/mbedtls.git project (v3.1.0)

Speed testing (Xeon(R) CPU E3-1230 V2 @ 3.30GHz):

gcrypt-gnu         Test md   2897 ms enc   2777 ms dec    942 ms
gcrypt-int         Test md   3668 ms enc   1312 ms dec   2836 ms
gcrypt-int-noaesni Test md   3652 ms enc   1916 ms dec   4458 ms

gcrypt-gnu-nonopt  Test md   3763 ms enc   4978 ms dec   3999 ms

gcrypt-gnu-nonopt - libgcrypt compiled without hardware acceleration
  --disable-padlock-support --disable-aesni-support \
  --disable-shaext-support --disable-pclmul-support \
  --disable-sse41-support --disable-drng-support \
  --disable-avx-support --disable-avx2-support \
  --disable-neon-support --disable-arm-crypto-support \
  --disable-ppc-crypto-support
  --disable-amd64-as-feature-detection
2022-02-20 10:16:46 +01:00
Luca Deri 519c69c5cc Reworked bitmap code 2021-09-27 10:29:03 +02:00
Luca Deri 1efabef4cf Added API for handling compressed bitmaps
ndpi_bitmap* ndpi_bitmap_alloc();
void ndpi_bitmap_free(ndpi_bitmap* b);
u_int64_t ndpi_bitmap_cardinality(ndpi_bitmap* b);
void ndpi_bitmap_set(ndpi_bitmap* b, u_int32_t value);
void ndpi_bitmap_unset(ndpi_bitmap* b, u_int32_t value);
bool ndpi_bitmap_isset(ndpi_bitmap* b, u_int32_t value);
void ndpi_bitmap_clear(ndpi_bitmap* b);
size_t ndpi_bitmap_serialize(ndpi_bitmap* b, char **buf);
ndpi_bitmap* ndpi_bitmap_deserialize(char *buf);

based on https://github.com/RoaringBitmap/CRoaring
2021-09-26 22:55:15 +02:00
Luca Deri dc841574af Win fix 2021-08-03 09:44:50 +02:00
Luca Deri b01b60a2b5 Implementation of flow risk eception (work in progress) 2021-07-22 01:35:57 +02:00
Vitaly Lavrov c418b7110b ahoсorasick. Code review. Part 2. (#1236)
Simplified the process of adding lines to AC_AUTOMATA_t.
Use the ndpi_string_to_automa() function to add patterns with domain names.
For other cases can use ndpi_add_string_value_to_automa().

ac_automata_feature(ac_automa, AC_FEATURE_LC) allows adding
and compare data in a case insensitive manner. For mandatory pattern comparison
from the end of the line, the "ac_pattern.rep.at_end=1" flag is used.
This eliminated unnecessary conversions to lowercase and adding "$" for
end-of-line matching in domain name patterns.

ac_match_handler() has been renamed ac_domain_match_handler() and has been greatly simplified.
ac_domain_match_handler() looks for the template with the highest domain level.
For special cases it is possible to manually specify the domain level.
Added test for checking ambiguous domain names like:
 - short.weixin.qq.com is QQ, not Wechat
 - instagram.faae1-1.fna.fbcdn.net is Instagram, not Facebook

If you specify a NULL handler when creating the AC_AUTOMATA_t structure,
then a pattern with the maximum length that satisfies the search conditions will be found
(exact match, from the beginning of the string, from the end of the string, or a substring).

Added debugging for ac_automata_search.
To do this, you need to enable debugging globally using ac_automata_enable_debug(1) and
enable debugging in the AC_AUTOMATA_t structure using ac_automata_name("name", AC_FEATURE_DEBUG).
The search will display "name" and a list of matching patterns.
Running "AHO_DEBUG=1 ndpiReader ..." will show the lines that were searched for templates
and which templates were found.

The ac_automata_dump() prototype has been changed. Now it outputs data to a file.
If it is specified as NULL, then the output will be directed to stdout.
If you need to get data as a string, then use open_memstream().

Added the ability to run individual tests via the do.sh script
2021-07-12 17:39:43 +02:00
Toni 8222ac5bdc Removed ht_hash as it is not used anymore. (#1220)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-06-29 15:32:48 +02:00
Luca Deri 177f9456bd Win changes 2021-06-08 18:15:46 +02:00
Vitaly Lavrov 65678dbeea New version of the ahocorasick library (#1200)
The new version is about 25% faster with -O2 and 45% faster with -O3.
No recursion is used (smaller stack size required).
Uses less memory (by valgrind info)
 bigram:
  - original 1796 allocs,   247864 bytes allocated
  - new      1232 allocs,   158880 bytes allocated
 host_match:
  - original 18038 allocs, 3004576 bytes allocated
  - new       6861 allocs,  396624 bytes allocated

The function ac_automata_search() is thread safe.
Optional case-insensitive comparison.
Matching at the beginning and at the end of the string is supported.
One code file and one header file.
2021-06-07 14:19:40 +02:00
Luca Deri 28879c570e Reworked ndpi patricia includes to avoid compilation issues on some platforms 2021-03-31 19:43:55 +02:00
Alfredo Cardigliano f8e83f7e35 Add support for MAC to Patricia tree. Expose full API to applications. Add utility functions. 2021-02-23 10:01:56 +01:00
Luca Deri d964c3e081 Code cleanup: third party uthash is at the right place 2021-01-20 19:11:36 +01:00
Luca Deri dc401f8a74 Split ptree user data in 32 and 64 bit entries 2020-12-30 12:12:33 +01:00
Luca Deri 8de62c6d34 Values stored in patricia tree are now 32 bit (they used to be 16 bit) long 2020-06-26 17:49:45 +02:00
Luca Deri 60aaa80570 Added HyperLogLog cardinality estimator API calls
/* Memory lifecycle */
int ndpi_hll_init(struct ndpi_hll *hll, u_int8_t bits);
void ndpi_hll_destroy(struct ndpi_hll *hll);

/* Add values */
void ndpi_hll_add(struct ndpi_hll *hll, const char *data, size_t data_len);
void ndpi_hll_add_number(struct ndpi_hll *hll, u_int32_t value) ;

/* Get cardinality estimation */
double ndpi_hll_count(struct ndpi_hll *hll);
2020-06-10 23:43:35 +02:00
Luca Deri 64fab4a478 Win fixes 2020-06-08 23:08:12 +02:00
Luca Deri 2ccd2c204b API cleanup for indetifying explicitly in automa's what we're searching (protocol or category)
Removed hyperscan support that is apperently unused
2020-05-06 23:19:59 +02:00
Luca Deri 263547e77d Updated automa API to use 32 bit values splits from protocol/categpry 2020-05-06 21:57:32 +02:00
Luca Deri 84f66b4d6b Introduced custom protocols with IP and (optional) port support
Example

- Single IP address
  ip:213.75.170.11@CustomProtocol

- IP address with CIDR
  ip:213.75.170.11/32@CustomProtocol

- IP address with CIDR and port
  ip:213.75.170.11/32:443@CustomProtocol

Please note that there are some restrictions on the port
usage. They have been listed in example/protos.txt
2020-05-06 12:51:44 +02:00
Luca Deri 7855e0318d Various fixes to patricia tree handling 2020-05-06 11:13:57 +02:00
Luca Deri 48282369e2 False positive fixes 2020-05-06 01:34:55 +02:00
Luca Deri 427002d14f Reworked protocol handling chnging it is u_int16_t 2020-05-06 00:31:40 +02:00
MrTiz9 3422574c17 FIXED - nDPI now detect RCE injections via PCRE instead Intel Hyperscan 2020-02-01 17:18:35 +01:00
MrTiz9 5c8c2d843a nDPI now detect RCE injections via PCRE instead Intel Hyperscan - BUGGY, DOES NOT COMPILE 2020-01-30 15:08:26 +01:00