48 Commits
Author SHA1 Message Date
Luca Deri b83eb7c7a2 Implemented STUN peer_address, relayed_address, response_origin, other_address parsing
Added code to ignore invalid STUN realm
Extended JSON output with STUN information
2024-04-12 19:50:04 +02:00
Luca Deri 1d480c18e3 Reworked domain classification based on binary filters 2023-09-02 19:16:40 +02:00
Ivan Nardi 3e4ab39b52 Add support for LTO and Gold linker (#1812)
This commit add (optional) support for Link-Time-Optimization and Gold
linker.
This is the first, mandatory step needed to make nDPI compliant with
"introspector" sanitizer requirements in OSS-Fuzz: see
https://github.com/google/oss-fuzz/issues/8939

Gold linker is not supported by Windows and by macOS, so this feature is
disabled by default. It has been enable in CI in two linux targets
("latest" gcc and clang).

Fix some warnings triggered by LTO.

The changes in `src/lib/ndpi_serializer.c` seams reasonable.
However, the change in `tests/unit/unit.c` is due to the following
warning, which seems to be a false positive.

```
unit.c: In function ‘serializerUnitTest’:
ndpi_serializer.c:2258:13: error: ‘MEM[(struct ndpi_private_serializer *)&deserializer].buffer.size’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
unit.c:67:31: note: ‘MEM[(struct ndpi_private_serializer *)&deserializer].buffer.size’ was declared here
   67 |   ndpi_serializer serializer, deserializer;
      |                               ^
ndpi_serializer.c:2605:10: error: ‘MEM[(struct ndpi_private_serializer *)&deserializer].status.buffer.size_used’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
unit.c:67:31: note: ‘MEM[(struct ndpi_private_serializer *)&deserializer].status.buffer.size_used’ was declared here
   67 |   ndpi_serializer serializer, deserializer;
```
Since this warning is triggered only with an old version of gcc and
`tests/unit/unit.c` is used only during the tests, the easiest fix has
been applied.

Some (unknown to me) combinations of OS and compiler trigger the
following warnings at linker time (with sanitizer and gold linker)
```
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load1_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load2_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load4_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load8_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load16_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store1_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store2_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store4_asm'
[..]
```
I have not found any references to this kind of message, with the only
exception of https://sourceware.org/bugzilla/show_bug.cgi?id=25975
which seems to suggest that these messages can be safely ignored.
In any case, the compilation results are sound.

Fix `clean` target in the Makefile in the `example` directory.

In OSS-Fuzz enviroments, `fuzz_ndpi_reader` reports a strange link error
(as always, when the gold linker is involved...).
It's come out that the culprit was the `tempnam` function: the code has
been changed to use `tmpfile` instead. No sure why... :(

Fuzzing target `fuzz_ndpi_reader.c` doesn't use `libndpiReader.a`
anymore: this way we can use `--with-only-libndpi` flag on Oss-Fuzz builds
as workaround for the "missing dependencies errors" described in
https://github.com/google/oss-fuzz/issues/8939
2022-12-05 10:21:42 +01:00
Toni 7c19de4904 Do not interfere with CFLAGS/LDFLAGS env anymore. (#1659)
* CI fixes
 * some build systems do not like that (e.g. OpenWrt)
 * fixed some rrdtool related build warnings/errors

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-13 19:44:18 +02:00
Toni df0ff9bcbd Added gprof CPU/HEAP profiling support. (#1592)
* Some small auto{conf,make} improvements

Signed-off-by: lns <matzeton@googlemail.com>
2022-06-12 21:00:41 +02:00
Toni 341f58fd80 Removed Makefile references to legacy code. (#1589)
- feature removed with f5545a80f9

Signed-off-by: lns <matzeton@googlemail.com>
2022-06-08 13:37:11 +02:00
Toni 00ee364a7e Updated README.md (#1562)
* make check great again (not so much)
 * make doc/doc-view
 * CI updates

Signed-off-by: lns <matzeton@googlemail.com>
2022-05-30 19:48:05 +02:00
Toni 78dfe959cb Added script to compare and verify the output of `make dist'. (#1551)
This fixes some build/test issues resulting when using tarballs.

 * nDPI uses autotools (especially autoconf) in a wrong way, see #1163

Signed-off-by: lns <matzeton@googlemail.com>
2022-05-26 15:07:22 +02:00
Toni Uhlig c3df3a12aa Fixed msys2 build warnings and re-activated CI Mingw64 build.
* Removed Visual Studio leftovers. Maintaining an autotools project with VS integration requires some additional overhead.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-14 19:17:48 +02:00
Sam JamesandLuca Deri c2b7d77784 build: respect environment options more (#1392)
* build: update m4/ax_pthread.m4 from serial 23 -> serial 31

Update ax_pthread.m4 to the latest version from the autoconf-archive
project.

Signed-off-by: Sam James <sam@gentoo.org>

* build: properly detect AR, CC, RANLIB

It's necessary to be able to override choice of AR/CC/RANLIB and other toolchain
variables/tools for cross-compilation, testing with other toolchains, and
to ensure the compiler chosen by the user is actually used for the build.

Previously, GNU_PREFIX was kind-of used for this but this isn't a standard
variable (at all) and it wasn't applied consistently anyway.

We now use the standard autoconf mechanisms for finding these tools.

(RANLIB is already covered by LT_INIT.)

Signed-off-by: Sam James <sam@gentoo.org>

* build: use $(MAKE)

This ensures that parallel make works correctly, as otherwise, a fresh
make job will be started without the jobserver fd, and hence
not know about its parent, forcing -j1.

* build: respect CPPFLAGS, LDFLAGS

- CPPFLAGS is for the C preprocessor (usually for setting defines)
- LDFLAGS should be placed before objects for certain flags to work
  (e.g. -Wl,--as-needed)

Signed-off-by: Sam James <sam@gentoo.org>

Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2022-01-18 14:30:14 +01:00
Toni 8d0c7b1fae Fixed Mingw64 build, SonerCloud-CI and more. (#1273)
* Added ARM build and unit test run for SonarCloud-CI.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Fixed Mingw64 build.

 * adapted to SonarCloud-CI workflow
 * removed broken and incomplete Windows example (tested on VS2017/VS2019)
 * removed unnecessary include (e.g. pthread.h for the library which does not make use of it)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-08-18 11:34:16 +02:00
Toni 13c5d6801e Improved libm check via Autoconf. (#1263)
* libm will now be part of @ADDITIONAL_LIBS@ (if required)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-07-28 22:24:03 +02:00
Toni 8c28613eb2 Check datalink during fuzzing to prevent console / logfile spam. See #1175 for more information. (#1177)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-05-09 15:09:43 +02:00
Toni da3e6bd61b Check for common ALPNs and set a flow risk if not known. (#1175)
* Increased risk bitmask to 64bit (instead of 32bit).
 * Removed annoying "Unknown datalink" error message for fuzzers.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-04-27 07:22:04 +02:00
Toni c2d8955c18 Fixed missing PCAP include directories in Makefiles. (#1034)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-10-19 11:25:22 +02:00
Toni 656323c334 Added missing files to `make dist' target which are not required to build nDPI but still somehow essential. (#1024)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-29 17:58:33 +02:00
Luca Deri ea9177cc93 Merge pull request #1018 from lnslbrty/fix/make-dist
Fixed broken `make dist' and added CI check.
2020-09-20 18:33:57 +02:00
Toni Uhlig fdef6f3545 Fixed broken `make dist' and added CI check.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-19 02:23:53 +02:00
Toni Uhlig bcce122665 Fixed mingw build w/o examples/tests/fuzzer.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-18 19:46:07 +02:00
Nardi Ivan d62ae567d1 Add (optional) dependency on external libraries: libgcrypt and libgpg-error
To support QUIC payload and header decryption, it is necessary to choose an
external crypto library to handle the low-level crypto stuff. Since we will
use some Wireshark code, it is quite natural to choose the same library used
by Wireshark itself: libgcrypt.

More precisely, we will use libgcrypt and libgpg-error.
Both libraries have LGPL license, so there should be no issue from this point
of view.

These libraries are not required to build nDPI, and their usage is optional:
nDPI will keep working (and compiling) even if they are not available.
However, without them, QUIC sub-classification is next to impossible.
The configure flag "--disable-gcrypt" forces the build system to ignore these
libraries.

libgpg-error is only used for debug to have meaningful error messages and its
usage is trivial.
The same cannot be said for libgcrypt because its initialization is a significant
issue.

The rest of this commit message try explaining how libgcrypt is
initialized.

According to the documentation
   https://gnupg.org/documentation/manuals/gcrypt/Initializing-the-library.html
   https://gnupg.org/documentation/manuals/gcrypt/Multi_002dThreading.html#Multi_002dThreading
libgcrypt must be initialized before using it, but such initialization should
be performed by the actual application and not by any library.

Forcing the users to proper initialize libgcrypt in their own code seems
unreasonable: most people using nDPI might be complete unaware of any crypto
stuff and update each and every one application linking to nDPI with specific
libgcrypt code should be out of question, anyway.

Fortunately, it seems a workaround exists to initialize libgcrypt in a library
    https://lists.gnupg.org/pipermail/gcrypt-devel/2003-August/000458.html

Therefore, we could provide a wrapper to this initialization stuff in a nDPI
function. Unfortunately nDPI API lacks a global init function that must be
called only once, before any other functions. We could add it, but that would
be a major API break.
AFAIK, ndpi_init_detection_module() might be called multiple times, for example
to create multiple independent dpi engines in the same program.

The proposed solution is to (optionally) initialize libgcrypt in
ndpi_init_detection_module() anyway:
* if the actual application doesn't directly use libgcrypt and only calls
  ndpi_init_detection_module() once, everything is formally correct and it
  should work out of the box [by far the most common user case];
* if the actual application already uses libgcrypt directly, it already
  performs the required initialization. In this case the ndpi_prefs.ndpi_dont_init_libgcrypt
  flag should be passed to ndpi_init_detection_module() to avoid further
  initializations.

The only scenario not supported by this solution is when the application is
unaware of libgcrypt and calls ndpi_init_detection_module() multiple times
concurrently. But this scenario should be uncommon.

A completely different option should be to switch to another crypto library,
with a huge impact on the QUIC dissector code.

Bottom line: crypto is hard, using libgcrypt is complex and the proposed
initialization, even if not perfect, should cover the most frequent user
cases and should work, for the time being.
If anyone has some suggestions...
2020-08-21 22:04:55 +02:00
Toni Uhlig f75984633b ndpiSimpleIntegration should not be included in a static library
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-07-08 23:47:51 +02:00
Toni Uhlig 17c26911fb ndpiSimpleIntegration: added another integration example
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-06-24 22:03:18 +02:00
Toni Uhlig 8d8e61b256 Added missing ndpiReader dependency for the install target
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-04-22 21:51:57 +02:00
emanuele-f cb34c6b233 Add fpic in the reader 2020-02-14 12:06:55 +01:00
Luca Deri c68ea3067b Added libpcre lib if present 2020-01-30 22:24:24 +01:00
Philippe Antoine fe312ffbe1 Compile libndpireader 2020-01-14 08:20:15 +01:00
Luca Deri f389969c0b Use share instead of sbin for sample files
Fixes #826
2020-01-05 18:50:56 +01:00
Luca Deri d1fb41a161 Minor cleanup 2020-01-05 18:42:36 +01:00
Luca Deri 23b0b8625d Minor changes to the IDS score calculation 2019-12-09 00:11:32 +01:00
Luca Deri 51cfdfb0d8 Removed unused JSON-C code 2019-11-17 17:51:45 +01:00
emanuele-f 2c4495e483 Add missing lib 2019-08-29 17:55:51 +02:00
Ravi Kerur 1ec160d4c9 nDPI ovs changes.
Signed-off-by: Ravi Kerur <ravi.kerur@viasat.com>
2019-07-23 09:56:24 -07:00
Luca Deri b50635cb89 Renamed ndpi_util.c -> reader_util.c
Added crash fix with IPv6
2019-07-18 11:38:16 +02:00
chiehminw 518dbbc96b Respect CXX variable during configure
commit 6ac83ce7eb use CXX to link ndpiReader however CXX is not
properly set in configure step

Fixes: 6ac83ce7eb ("Fixes broken hyperscan compilation")
Signed-off-by: chiehminw <chiehminw@synology.com>
2019-03-06 15:27:50 +08:00
Pramode 81649d9979 Propagate prefix settings to Makefiles
A "make install" was failing because the --prefix flag
setting was not being propagated to the Makefiles.
2019-01-20 19:09:07 +05:30
Luca Deri f3b0878a37 Warnign fixes
Updated test results
2019-01-17 00:39:45 +01:00
Vitaliy Ivanov fd963538f3 build: cleanup of dpdk examples.
- clean install dependencies on example binary
- add more rules to clean command

Signed-off-by: Vitaliy Ivanov <vitaliyi@interfacemasters.com>
2018-11-29 13:40:37 +02:00
Vitaliy Ivanov a3abbc3ebb build: do not fail if example app is not present.
Signed-off-by: Vitaliy Ivanov <vitaliyi@interfacemasters.com>
2018-11-28 19:23:27 +02:00
Vitaliy Ivanov 8df6b5c1bf build: install example binaries to dst dir.
make install DESTDIR=/build/nDPI/

Signed-off-by: Vitaliy Ivanov <vitaliyi@interfacemasters.com>
2018-11-28 17:40:01 +02:00
Vitaliy Ivanov ea7e7c753b build: correct example dpdk linking with pcap.
- take libpcap from configure information.
- clean up distclean.

Signed-off-by: Vitaliy Ivanov <vitaliyi@interfacemasters.com>
2018-11-28 16:40:34 +02:00
Alfredo Cardigliano 21504c8492 rpm spec file 2018-11-19 19:01:53 +01:00
Alfredo Cardigliano 1537669178 rpm package 2018-11-19 18:02:06 +01:00
Luca Deri 71b2c19cf2 Added DPDK support to ndpiReader 2018-11-10 16:10:22 +01:00
Guido Falsi b7b12d5b4e Make the examples Makefile respect DESTDIR and PREFIX env vars. 2018-10-12 17:22:13 +02:00
root 1d56c9aa1b fix remake 2018-10-02 23:54:01 +08:00
Simone Mainardi 6ac83ce7eb Fixes broken hyperscan compilation 2018-09-11 10:43:04 +02:00
Luca ed5b04791e Added missing -lpthread 2018-08-30 11:26:49 +02:00
Luca 33d6f2d52f Added missing file 2018-08-30 11:17:19 +02:00