Commit Graph
44 Commits
Author SHA1 Message Date
snicket2100 abee1a2a6f Included Gambling website data from the Polish hazard.mf.gov.pl list (#2041)
* Refreshed the Belgium Gambling Site list data

Unfortunately some hostnames have been removed from that list,
which means they are disappearing from the `ndpi_gambling_match.c.inc`
file as well.

* build: added `libxml2-utils` (for `xmllint`)

* Included Gambling website data from the Polish `hazard.mf.gov.pl` list

The list contains over 30k gambling website hostnames as of today.
2023-07-14 09:55:46 +02:00
Toni 5e8f93c2d1 Improved missing usage of nDPIs malloc wrapper. Fixes #1978. (#1979)
* added CI check

Signed-off-by: lns <matzeton@googlemail.com>
2023-05-20 16:18:52 +02:00
Ivan Nardi 684e041998 Improve detection of crawlers/bots (#1968)
Add support for Facebook crawler
2023-05-09 16:42:29 +02:00
Ivan Nardi 04e017a854 Rework CI jobs to try reducing CI duration (#1903)
CI duration is quite long: the longest jobs is the "Performance" one.
Try to reduce the overall duration: that job (and some others) will not
be triggered for each PR/commit anymore, but asynchronously, once a day
(this scheduling seems right since the frequency of the PR/commits in
the project).
It should be possibly to trigger them manually, via GUI, anyway.

Remove two identical jobs; we already tests ASAN with 4 different
compilers.

After 9eff0754 it is safe to reduce fuzzing time.

Bottom line: try to have as upper-time of CI tests the duration of the
fuzzing jobs
2023-03-22 18:17:25 +01:00
Ivan Nardi 06d4f8c7e5 Update GitHub runners versions (#1889)
See: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners
2023-02-14 21:30:54 +01:00
Ivan Nardi 3e4ab39b52 Add support for LTO and Gold linker (#1812)
This commit add (optional) support for Link-Time-Optimization and Gold
linker.
This is the first, mandatory step needed to make nDPI compliant with
"introspector" sanitizer requirements in OSS-Fuzz: see
https://github.com/google/oss-fuzz/issues/8939

Gold linker is not supported by Windows and by macOS, so this feature is
disabled by default. It has been enable in CI in two linux targets
("latest" gcc and clang).

Fix some warnings triggered by LTO.

The changes in `src/lib/ndpi_serializer.c` seams reasonable.
However, the change in `tests/unit/unit.c` is due to the following
warning, which seems to be a false positive.

```
unit.c: In function ‘serializerUnitTest’:
ndpi_serializer.c:2258:13: error: ‘MEM[(struct ndpi_private_serializer *)&deserializer].buffer.size’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
unit.c:67:31: note: ‘MEM[(struct ndpi_private_serializer *)&deserializer].buffer.size’ was declared here
   67 |   ndpi_serializer serializer, deserializer;
      |                               ^
ndpi_serializer.c:2605:10: error: ‘MEM[(struct ndpi_private_serializer *)&deserializer].status.buffer.size_used’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
unit.c:67:31: note: ‘MEM[(struct ndpi_private_serializer *)&deserializer].status.buffer.size_used’ was declared here
   67 |   ndpi_serializer serializer, deserializer;
```
Since this warning is triggered only with an old version of gcc and
`tests/unit/unit.c` is used only during the tests, the easiest fix has
been applied.

Some (unknown to me) combinations of OS and compiler trigger the
following warnings at linker time (with sanitizer and gold linker)
```
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load1_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load2_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load4_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load8_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_load16_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store1_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store2_asm'
/usr/bin/ld.gold: warning: Cannot export local symbol '__asan_report_store4_asm'
[..]
```
I have not found any references to this kind of message, with the only
exception of https://sourceware.org/bugzilla/show_bug.cgi?id=25975
which seems to suggest that these messages can be safely ignored.
In any case, the compilation results are sound.

Fix `clean` target in the Makefile in the `example` directory.

In OSS-Fuzz enviroments, `fuzz_ndpi_reader` reports a strange link error
(as always, when the gold linker is involved...).
It's come out that the culprit was the `tempnam` function: the code has
been changed to use `tmpfile` instead. No sure why... :(

Fuzzing target `fuzz_ndpi_reader.c` doesn't use `libndpiReader.a`
anymore: this way we can use `--with-only-libndpi` flag on Oss-Fuzz builds
as workaround for the "missing dependencies errors" described in
https://github.com/google/oss-fuzz/issues/8939
2022-12-05 10:21:42 +01:00
Ivan Nardi b9f63458e6 CI: GitHub is updating Ubuntu runners (#1817)
GitHub is moving `ubuntu-latest` to `ubuntu-22.04`: update our
dependencies.
See: https://github.blog/changelog/2022-11-09-github-actions-ubuntu-latest-workflows-will-use-ubuntu-22-04/

This is the reason of the recent random failures in CI.

Update "newest" tested gcc to gcc-12.

Fix a memory error introduced in 557bbcfc5a
2022-12-03 18:29:44 +01:00
Ivan Nardi 6e6f1e4489 GitHub Actions: fix a warning (#1807)
```
The `set-output` command is deprecated and will be disabled soon.
Please upgrade to using Environment Files. For more information see:
https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
```
2022-11-22 16:44:08 +01:00
Ivan Nardi 07872b3527 GitHub Actions: update to Node.js 16 actions (#1801)
Fix warnings on recent CI results; example:
https://github.com/ntop/nDPI/actions/runs/3455588082

See: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/
2022-11-13 19:42:13 +01:00
mark andrews 5d5b46e514 Add proj and sln for compile of dynamic x64 lib under Visual Studio 2019.
* add CI support via MSBuild

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-21 18:03:22 +02:00
Ivan Nardi 03d217eae6 Fix CI after nBPF integration (#1746)
Add one CI job testing nBPF
2022-09-21 16:49:51 +02:00
Ivan Nardi 4bb81f7609 Update Github CI OS and compilers (#1733)
ubuntu-18.04 is deprecated (ubuntu-latest points to 20.04).
macos-latest points to macos-11, so it makes sense to test macos-12,
too.

About the compilers, the general idea it to test the oldest and the
newest versions easily available: switch to gcc-11 and clang-14.

See: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners
2022-09-12 19:11:43 +02:00
Toni 2e25c36396 Add TiVoConnect dissector. Fixes #1697. (#1699)
* added static assert if supported, to complain if the flow struct changes

Signed-off-by: lns <matzeton@googlemail.com>
2022-08-08 19:04:20 +02:00
Ivan Nardi 86a3e4c8c3 Add support for Memory sanitizer (#1652) 2022-07-24 18:12:25 +02:00
Toni 7c19de4904 Do not interfere with CFLAGS/LDFLAGS env anymore. (#1659)
* CI fixes
 * some build systems do not like that (e.g. OpenWrt)
 * fixed some rrdtool related build warnings/errors

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-13 19:44:18 +02:00
Toni Uhlig 69ccb39741 Generate profiling results as PNG.
* use -ltcmalloc_and_profiler and try to get rid of LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libprofiler.so

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-03 17:38:43 +02:00
lns f2d1edbedf gprof test/CI integration
Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-03 17:38:43 +02:00
Toni 5ca82ad84b Added TSAN support. (#1613)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-03 14:35:21 +02:00
Toni d8d806c79f Force roaring bitmap to use ndpi memory wrappers. (#1569)
GCC analyzer won't complain about possible use-after-free (false positive).

 * tests/do.sh prints word diff's only once and not the same over and over again
 * sync unit tests

Signed-off-by: lns <matzeton@googlemail.com>
2022-05-31 12:06:01 +02:00
Toni 00ee364a7e Updated README.md (#1562)
* make check great again (not so much)
 * make doc/doc-view
 * CI updates

Signed-off-by: lns <matzeton@googlemail.com>
2022-05-30 19:48:05 +02:00
Toni 48065d80e9 Support word diff for tests/do.sh for better readability. (#1565)
* Sync unit tests

Signed-off-by: lns <matzeton@googlemail.com>
2022-05-30 17:33:03 +02:00
Toni 6b7b23b01d Use Doxygen to generate the API documentation. (#1558)
* Integrated Doxygen documentation into Sphinx

Signed-off-by: lns <matzeton@googlemail.com>
2022-05-29 13:44:52 +02:00
Toni 78dfe959cb Added script to compare and verify the output of `make dist'. (#1551)
This fixes some build/test issues resulting when using tarballs.

 * nDPI uses autotools (especially autoconf) in a wrong way, see #1163

Signed-off-by: lns <matzeton@googlemail.com>
2022-05-26 15:07:22 +02:00
Toni 7010d17ccc Fixed windows-latest build error. (#1552)
* The warning itself looks like a bug

Signed-off-by: lns <matzeton@googlemail.com>
2022-05-26 10:19:57 +02:00
Toni Uhlig c3df3a12aa Fixed msys2 build warnings and re-activated CI Mingw64 build.
* Removed Visual Studio leftovers. Maintaining an autotools project with VS integration requires some additional overhead.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-14 19:17:48 +02:00
aouinizied 1207ea118a Add HOWTO Python. 2022-03-22 14:56:44 +01:00
aouinizied c47d710d8e Fix python bindings CI. 2022-03-22 13:40:31 +01:00
aouinizied beef4f997b Complete rework of nDPI Python bindings (cffi API, automatic generation, packaging and CI integration) 2022-03-22 13:19:27 +01:00
Toni f646a4bce0 Improved ASN/IP update scripts and CI integration. (#1474)
* CI will print a warning if ASN/IP addresses changed.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-09 13:53:04 +01:00
Zied Aouini 74ae315e36 Implement CI on Windows. (#1483)
* Switch fail fast to True.
* Windows CI.
2022-03-09 10:57:21 +01:00
Vitaly Lavrov a1451935b8 Errors fixed (#1482)
Fixed errors for bigendian platforms in ndpiReader.
All address and port comparisons and hash calculations are done with
endian in mind.
The get_ndpi_flow_info() function searched for an existing flow for the
forward and reverse direction of the packet.
The ndpi_workflow_node_cmp() function looked for a flow regardless of
the packet's direction. This is what led to an error in determining the
direction of transmission of the packet.

Fixed error in "synscan" test: the number of packets in the forward and
reverse direction is incorrectly defined (verified via tcpdump).

Fixed bug with icmp protocol checksum check for big endian platforms.
2022-03-08 00:20:56 +01:00
Toni 95a3d4fffe Added autoconf option `--enable-tls-sigs'. (#1471)
* Testing more code in CI environments.
 * Added strict option checking for `./configure' in CI environments.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-03 10:16:08 +01:00
Toni cb62dfd249 Drop support for non-gcrypt builds. (#1469)
* As there is now a builtin, lightweight libgcrypt
   there is no need to disable tls-clho decryption.
 * It is still possible to use a host libgcrypt
   with `--with-local-libgcrypt'.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-02 19:48:46 +01:00
Toni 00d9a16b6a Added `--enable-code-coverage' build using lcov for coverage generation. (#1430)
* CI job generates a coverage report

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-28 14:58:45 +01:00
Ivan Nardi 7f69de0b51 Restore a unit test result (#1403)
Deleted, probably by mistake, in 406ac7e8

Fix Makefile and add compilation of `rrdtool` in CI tests
2022-01-13 23:14:28 +01:00
Ivan Nardi 55880e4ae4 TLS: fix two warnings (#1365)
Disable unit tests on CI for big-endian target. We know we have multiple
issues on big-endian architectures (see #1312) and so the unit tests
always fail there. Ignore this error for the time being and let the CI
pass if we don't have other issues.

Remove an unused automa definition
2021-11-02 22:08:15 +01:00
Toni 549134ef10 Fix make dist and add it to the CI along with make -C doc text. Fixes #1324 (#1327)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-10 13:15:38 +02:00
Toni cf931fda6b Get rid of configure.seed as it comes with some disadvantages. (#1328)
* using Autotools best-practices to achieve (hopefully) the same result

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-10 13:13:52 +02:00
Zied Aouini c40cf4e0ec Add arm64, armhf and s390x as part of CI. (#1314)
* Add arm64, armhf and s390x as part on CI.

* Minor fix.

* Fix code inspector complaints.
2021-09-23 14:41:56 +02:00
Zied Aouini 0994771974 Improve CI (#1303)
* Improve CI pipeline

* Fix branch name.

* Fix branch name.

* Fix libgcrypt configuration.

* Update build.yml

* Move to Github Actions instead of Travis CI.

* Fix mingw on ubuntu bionic.

* Reactivate cross compile on Ubuntu Bionic.

* Switch to single line steps.

* Add several compilers versions

* Minor fix.

* Fix build all and delete cxx

* Fix RCE detection.

* Fix PCRE configuration.

* Add condition on PCRE test pcap.

* Update WebattackRCE.pcap.out

* Add missing SUBST.

* Delete WebattackRCE.pcap.out

* Update WebAttackRCE result.

* Fix typo.

* Extend jobs with pcre+msan+maxminddb.

* Fix code inpector warnings.

* Delete .appveyor.yml
2021-09-22 12:10:52 +02:00
Ivan Nardi 951bcb0c39 Cassandra: fix compilation when "--enable-debug-messages" option is used (#1294)
Let's try adding a dedicated compilation in GitHub Actions to easily
detect this kind of errors in the future
2021-09-11 11:10:46 +02:00
Toni 8d0c7b1fae Fixed Mingw64 build, SonerCloud-CI and more. (#1273)
* Added ARM build and unit test run for SonarCloud-CI.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Fixed Mingw64 build.

 * adapted to SonarCloud-CI workflow
 * removed broken and incomplete Windows example (tested on VS2017/VS2019)
 * removed unnecessary include (e.g. pthread.h for the library which does not make use of it)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-08-18 11:34:16 +02:00
Toni 8e996f7f7c Fixed broken SonarCloud-CI. (#1272)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-08-08 16:59:36 +02:00
Luca Deri a5ecdf9df8 Changes for sonarcloud integration 2021-05-11 21:36:25 +02:00