Commit Graph
146 Commits
Author SHA1 Message Date
Luca Deri 049fbf819c Reworked ndpi_filter_xxx implementation using compressed bitmaps 2023-08-14 12:50:54 +02:00
Luca Deri 3b34941be1 Typo 2023-08-12 11:39:19 +02:00
Luca Deri 9a44954365 ARM compilation fix 2023-08-12 07:39:35 +02:00
Luca Deri 890dcf91d9 Minor initialization fixes 2023-08-11 23:51:16 +02:00
Luca Deri ec7adc212e Added new API calls for implementing Bloom-filter like data structures
ndpi_filter* ndpi_filter_alloc(uint32_t elements_number);
bool         ndpi_filter_add(ndpi_filter *f, uint64_t value);
bool         ndpi_filter_contains(ndpi_filter *f, uint64_t value);
void         ndpi_filter_free(ndpi_filter *f);
2023-08-11 22:49:31 +02:00
Ivan Nardi 950f5cc4e3 fuzz: extend fuzzing coverage (#2040)
Some notes:
* libinjection: according to https://github.com/libinjection/libinjection/issues/44,
it seems NULL characters are valid in the input string;
* RTP: `rtp_get_stream_type()` is called only for RTP packets; if you
want to tell RTP from RTCP you should use `is_rtp_or_rtcp()`;
* TLS: unnecessary check; we already make the same check just above, at
the beginning of the `while` loop
2023-07-11 10:12:08 +02:00
Toni 4e284b5e40 Set _DEFAULT_SOURCE and _GNU_SOURCE globally. (#2010)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-06-12 19:53:57 +02:00
Ivan Nardi efb261a95c Fix some memory errors triggered by allocation failures (#1995)
Some low hanging fruits found using nallocfuzz.
See: https://github.com/catenacyber/nallocfuzz
See: https://github.com/google/oss-fuzz/pull/9902

Most of these errors are quite trivial to fix; the only exception is the
stuff in the uthash.
If the insertion fails (because of an allocation failure), we need to
avoid some memory leaks. But the only way to check if the `HASH_ADD_*`
failed, is to perform a new lookup: a bit costly, but we don't use that
code in any critical data-path.
2023-05-29 19:24:00 +02:00
Toni 334b43579e Fixed invalid use of ndpi_free(). Sorry, my fault. (#1988)
* Fixed invalid use of ndpi_free(). Sorry, my fault.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Fine tuned symbol check script.

 * added check for expected syms in modules

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

---------

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-05-24 13:19:06 +02:00
Toni 5e8f93c2d1 Improved missing usage of nDPIs malloc wrapper. Fixes #1978. (#1979)
* added CI check

Signed-off-by: lns <matzeton@googlemail.com>
2023-05-20 16:18:52 +02:00
Toni b6629ba2be Improved debug output. (#1951)
* try to get rid of some `printf(..)`s as they do not belong to a shared library
 * replaced all `exit(..)`s with `abort()`s to indicate an abnormal process termination

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-04-21 12:40:26 +02:00
Ivan Nardi cc5aec5f0a fuzz: add fuzzer to test internal gcrypt code (#1920) 2023-04-04 14:19:41 +02:00
Ivan Nardi d7662379b4 Update libinjection code (#1918)
Update libinjection code to the current master https://github.com/libinjection/libinjection/commit/7e4b74e824dc3f0623ba4894df2a0d817351ec39

The goal is to finally fix #1820
See: https://github.com/libinjection/libinjection/issues/33

Update the corpus of the libinjection fuzzers

Close #1820
2023-04-04 14:18:57 +02:00
Luca Deri 85bab9e660 Centos7 fixes 2023-02-13 01:02:27 +01:00
Luca Deri bf413afba1 Update roaring bitmap code 2023-02-12 12:50:24 +01:00
Ivan Nardi b51a2ac72a fuzz: some improvements and add two new fuzzers (#1881)
Remove `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` define from
`fuzz/Makefile.am`; it is already included by the main configure script
(when fuzzing).

Add a knob to force disabling of AESNI optimizations: this way we can
fuzz also no-aesni crypto code.

Move CRC32 algorithm into the library.

Add some fake traces to extend fuzzing coverage. Note that these traces
are hand-made (via scapy/curl) and must not be used as "proof" that the
dissectors are really able to identify this kind of traffic.

Some small updates to some dissectors:

CSGO: remove a wrong rule (never triggered, BTW). Any UDP packet starting
with "VS01" will be classified as STEAM (see steam.c around line 111).
Googling it, it seems right so.

XBOX: XBOX only analyses UDP flows while HTTP only TCP ones; therefore
that condition is false.

RTP, STUN: removed useless "break"s

Zattoo: `flow->zattoo_stage` is never set to any values greater or equal
to 5, so these checks are never true.

PPStream: `flow->l4.udp.ppstream_stage` is never read. Delete it.

TeamSpeak: we check for `flow->packet_counter == 3` just above, so the
following check `flow->packet_counter >= 3` is always false.
2023-02-09 20:02:12 +01:00
Ivan Nardi 9fc724de5a Add some fuzzers to test other data structures. (#1870)
Start using a dictionary for fuzzing (see:
https://llvm.org/docs/LibFuzzer.html#dictionaries).
Remove some dead code.
Fuzzing with debug enabled is not usually a great idea (from performance
POV). Keep the code since it might be useful while debugging.
2023-01-25 11:44:59 +01:00
Ivan Nardi 29c5cc39fb Some small changes (#1869)
All dissector callbacks should not be exported by the library; make static
some other local functions.
The callback logic in `ndpiReader` has never been used.
With internal libgcrypt, `gcry_control()` should always return no
errors.
We can check `categories` length at compilation time.
2023-01-25 11:44:09 +01:00
Ivan Nardi 5e8c1ebbb7 fuzz: fix memory allocation failure logic (#1867)
We *do* want to have some allocation errors.
Fix some related bugs
Fix: 29be01ef
2023-01-20 14:27:33 +01:00
Ivan Nardi 29be01ef3a Add some fuzzers to test algorithms and data structures (#1852)
Fix some issues found with these new fuzzers
2023-01-17 08:31:59 +01:00
Ivan Nardi 9c83f0f3c5 Fix some warnings and two errors found while fuzzing (#1844)
Fix CI
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=54614
2023-01-03 13:21:54 +01:00
Ivan Nardi 560280e6f0 fuzz: add fuzzer testing nDPI (initial) configurations (#1830)
The goal of this fuzzer is to test init and deinit of the library, with
different configurations. In details:
* random memory allocation failures, even during init phase
* random `ndpi_init_prefs` parameter of `ndpi_init_detection_module()`
* random LRU caches sizes
* random bitmask of enabled protocols
* random parameters of `ndpi_set_detection_preferences()`
* random initialization of opportunistic TLS
* random load/don't load of configuration files

This new fuzzer is a C++ file, because it uses `FuzzedDataProvider`
class (see
https://github.com/google/fuzzing/blob/master/docs/split-inputs.md).
Note that the (existing) fuzzers need to be linked with C++ compiler
anyway, so this new fuzzer doesn't add any new requirements.
2022-12-23 19:07:13 +01:00
Ivan Nardi ada4fe4aa8 fuzz: add a new fuzzer testing memory allocation failures (#1818)
Try to fuzz error paths triggered by allocation errors.
Fix some errors already found by this new fuzzer.
Basic idea taken from: https://github.com/harfbuzz/harfbuzz/pull/2566/files

`FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is a standard define used to
(not)compile specific code in fuzzing builds.
See: https://llvm.org/docs/LibFuzzer.html
2022-12-06 17:41:58 +01:00
Ivan Nardi 197a9a6cfc Extend CodeQL configuration and fix two warnings (#1816)
```
Call to `memset` may be deleted
Comparison of narrow type with wide type in loop condition
```

See https://www.usenix.org/system/files/conference/usenixsecurity17/sec17-yang.pdf
for the new portable function `secure_memzero`.
2022-12-02 23:26:22 +01:00
Luca Deri 175981a422 Removed unused variable 2022-10-26 16:14:32 +02:00
Alfredo Cardigliano f30479b4a4 Fix typo 2022-10-26 16:05:32 +02:00
Luca Deri a16cd9634f Cleanup 2022-10-26 13:59:07 +02:00
Luca Deri 15d58db2f9 Typo 2022-10-26 13:56:24 +02:00
Luca Deri 55f885f3ed Improved AESNI check 2022-10-26 13:51:00 +02:00
Luca Deri e4dcec560e Windows changes 2022-10-06 11:58:16 +02:00
mark andrews 5d5b46e514 Add proj and sln for compile of dynamic x64 lib under Visual Studio 2019.
* add CI support via MSBuild

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-21 18:03:22 +02:00
Nardi Ivan e5063f9883 Fix some warnings
Ubuntu-20.04, clang-16 (nightly build)

```
Making all in src/lib
protocols/smpp.c:70:17: warning: variable 'pdu_c' set but not used [-Wunused-but-set-variable]
      u_int16_t pdu_c = 1;
                ^
1 warning generated.
third_party/src/ahocorasick.c:173:20: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion]
  thiz->root->root = 1;
                   ^ ~
third_party/src/ahocorasick.c:336:15: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion]
        n->ff = 1;
              ^ ~
third_party/src/ahocorasick.c:716:21: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion]
        node->final = 1;
[...]
```
2022-09-05 23:44:51 +02:00
Alfredo Cardigliano 981a72cd80 Improve AES-NI check 2022-08-30 16:37:30 +02:00
Nardi Ivan 79653f99e0 Internal gcrypt: fix memory leak
Fix: ef99eb67
2022-08-30 11:12:00 +02:00
Luca Deri ef99eb674e Improved AES-NI presence check on Linux 2022-08-29 17:42:28 +02:00
Luca Deri a53f476585 Modified definition of hll_add 2022-08-17 16:25:10 +02:00
Ivan Nardi 405a52ed65 Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1683)
Add (basic) internal stats to the main data structures used by the
library; they might be usefull to check how effective these structures
are.

Add an option to `ndpiReader` to dump them; enabled by default in the
unit tests.
This new option enables/disables dumping of "num dissectors calls"
values, too (see b4cb14ec).
2022-07-29 15:25:00 +02:00
Ivan Nardi df2e11ef51 Revert "Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1677)" (#1682)
This reverts commit bb83899985.
2022-07-29 12:08:40 +02:00
Ivan Nardi bb83899985 Patricia tree, Ahocarasick automa, LRU cache: add statistics (#1677)
Add (basic) internal stats to the main data structures used by the
library; they might be usefull to check how effective these structures
are.

Add an option to `ndpiReader` to dump them; disabled by default to avoid
too much fuss with the unit tests.
2022-07-29 12:07:41 +02:00
Ivan Nardi 86a3e4c8c3 Add support for Memory sanitizer (#1652) 2022-07-24 18:12:25 +02:00
Ivan Nardi ac1113b9ac ahocorasick: fix char/uchar bug (#1597)
Porting of https://github.com/vel21ripn/nDPI/commit/06e2967d0c26ab214683a2f5565f4012fd523537
Close #1506
2022-07-20 10:59:17 +02:00
Toni be1ac1b430 Fix unitialized use of W[16]. (#1662)
Signed-off-by: lns <matzeton@googlemail.com>
2022-07-15 14:15:37 +02:00
Ivan Nardi 5f6fa6d164 Fix a compilation warning (#1620)
With clang-15 (nightly build)
```
In file included from ndpi_bitmap.c:39:
./third_party/src/roaring.cc:14233:13: warning: variable 'run_count' set but not used [-Wunused-but-set-variable]
        int run_count = 0;
```
2022-07-03 17:43:38 +02:00
Toni 9c8b2d63da Replaced nDPI's internal hashmap with uthash. (#1602)
Signed-off-by: lns <matzeton@googlemail.com>
2022-06-17 19:50:31 +02:00
Toni d8d806c79f Force roaring bitmap to use ndpi memory wrappers. (#1569)
GCC analyzer won't complain about possible use-after-free (false positive).

 * tests/do.sh prints word diff's only once and not the same over and over again
 * sync unit tests

Signed-off-by: lns <matzeton@googlemail.com>
2022-05-31 12:06:01 +02:00
Luca Deri 77dc884a4c Compilation fixes for old gcc compiler 2022-05-28 16:06:34 +02:00
Toni 02d0b5fe13 Improved AES-NI check. (#1536)
* A library should not open a subshell

Signed-off-by: lns <matzeton@googlemail.com>
2022-04-30 20:46:20 +02:00
Luca Deri e4318ffc2d Improved AES-NI check on Linux to avoid crashes on CPUs that do
not support it (e.g. Intel Celeron N2930)
2022-04-29 14:50:21 +02:00
Toni Uhlig c3df3a12aa Fixed msys2 build warnings and re-activated CI Mingw64 build.
* Removed Visual Studio leftovers. Maintaining an autotools project with VS integration requires some additional overhead.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-14 19:17:48 +02:00
Vitaly Lavrov 438f04066b Fixed a bug for BE architectures (#1478)
Fixed a bug in the internal implementation of libgcrypt for bigendian architectures
2022-03-05 11:44:21 +01:00