Commit Graph
3315 Commits
Author SHA1 Message Date
Luca Deri b97dc6baa4 Removed outdated (and broken) soulseek dissector 2021-10-15 17:27:27 +02:00
Luca Deri e696f31487 Updated test results 2021-10-14 22:18:34 +02:00
Luca Deri b1a43cbcff Added missing call to ndpi_reconcile_protocols() with protocol guess 2021-10-14 22:08:20 +02:00
Luca Deri 6e649e0cb6 Fixed issue on DGA numeric IP detection 2021-10-13 19:21:27 +02:00
Luca Deri 09668496c3 Indemt 2021-10-13 19:18:05 +02:00
Zied Aouini a2e9730b23 Add issue templates. (#1340)
* Add issue template.
* Add config log instruction.
2021-10-13 18:39:10 +02:00
Luca Deri 937357e4bc Implemented ndpi_ses_fitting() and ndpi_des_fitting()
for comuting the best alpha/beta values for exponential smoothing
2021-10-12 13:08:58 +02:00
Alfredo Cardigliano d947437233 Return 0 on ndpi_init_serializer_ll success for consistency 2021-10-12 10:47:44 +00:00
Luca Deri d13bc0e01a Fixed DES initialisation 2021-10-12 12:38:40 +02:00
Ivan Nardi 550e6fe6fc QUIC: fix an integer overflow (#1337)
Long standing bug: credits to @lnslbrty for digging into it and to
@aouinizied for the CI improvements
2021-10-11 23:08:10 +02:00
Toni a8f938f7ac Removed README.protocols because: (#1333)
* Tor via TLS should be detectable via DGA as a risk
 * protocol limitations should be part of the official documentation in `doc/`

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-11 23:07:36 +02:00
Luca Deri 960e8ab22a Win fix 2021-10-11 20:59:28 +02:00
Zied Aouini ca827ae548 Fix FuzzCI. (#1338) 2021-10-11 20:17:12 +02:00
Zied Aouini 7e0724d576 Enable sanitizers on CIFuzz. (#1336) 2021-10-11 19:01:50 +02:00
Luca Deri 246d115d7b Cleaned up code moving specific includes in files their are using it. Thi prevents ndpi_config.h to be included everywhere in apps using nDPI that might leade to #define redefinitions after the latest changes 2021-10-11 11:45:45 +02:00
Toni 60eca79fdf Additional fix related to cf931fda6b. (#1332)
* configure.seed references removed

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-10 14:14:02 +02:00
Toni 549134ef10 Fix make dist and add it to the CI along with make -C doc text. Fixes #1324 (#1327)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-10 13:15:38 +02:00
Toni cf931fda6b Get rid of configure.seed as it comes with some disadvantages. (#1328)
* using Autotools best-practices to achieve (hopefully) the same result

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-10 13:13:52 +02:00
Toni 6ff67b5043 Add more WindowsUpdate URLs as requested in #698. (#1329)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-08 18:29:49 +02:00
Luca Deri 3ffb65a81c Added extra check 2021-10-07 08:21:40 +02:00
Ivan Nardi 2dfc478ad4 Fix compilation with clang-13 or if some debug macros are enabled (#1326) 2021-10-06 18:32:15 +02:00
Luca Deri 944e8ed105 Added check 2021-10-06 18:28:09 +02:00
Luca Deri 2b4821a97b Compilation fix for old Linux distributions 2021-10-06 09:28:04 +02:00
Luca Deri d9c72999ee Test update 2021-10-06 00:06:34 +02:00
Luca Deri 181a03c5ad Compilation fix 2021-10-05 17:17:12 +02:00
Luca Deri 8957c52a48 Numeric IPs are not considered for DGA checks 2021-10-05 17:14:31 +02:00
Luca Deri 408d78e628 Improved DGA detection for skipping potential DGAs of known/popular domain names 2021-10-05 16:51:24 +02:00
Luca Deri 10576eab0a TLS obsolete protocol is set when TLS < 1.2 (used to be 1.1) 2021-10-05 16:44:56 +02:00
Ivan Nardi 730c2360bd Remove struct ndpi_packet_struct from struct ndpi_flow_struct (#1319)
There are no real reasons to embed `struct ndpi_packet_struct` (i.e. "packet")
in `struct ndpi_flow_struct` (i.e. "flow"). In other words, we can avoid
saving dissection information of "current packet" into the "flow" state,
i.e. in the flow management table.

The nDPI detection module processes only one packet at the time, so it is
safe to save packet dissection information in `struct ndpi_detection_module_struct`,
reusing always the same "packet" instance and saving a huge amount of memory.
Bottom line: we need only one copy of "packet" (for detection module),
not one for each "flow".

It is not clear how/why "packet" ended up in "flow" in the first place.
It has been there since the beginning of the GIT history, but in the original
OpenDPI code `struct ipoque_packet_struct` was embedded in
`struct ipoque_detection_module_struct`, i.e. there was the same exact
situation this commit wants to achieve.

Most of the changes in this PR are some boilerplate to update something
like "flow->packet" into something like "module->packet" throughout the code.
Some attention has been paid to update `ndpi_init_packet()` since we need
to reset some "packet" fields before starting to process another packet.

There has been one important change, though, in ndpi_detection_giveup().
Nothing changed for the applications/users, but this function can't access
"packet" anymore.
The reason is that this function can be called "asynchronously" with respect
to the data processing, i.e in context where there is no valid notion of
"current packet"; for example ndpiReader calls it after having processed all
the traffic, iterating the entire session table.

Mining LRU stuff seems a bit odd (even before this patch): probably we need
to rethink it, as a follow-up.
2021-10-05 15:49:52 +02:00
Ivan Nardi f3fcf1e7c0 Update unit tests results after da8eed5a (#1323) 2021-10-05 15:06:34 +02:00
Luca Deri da8eed5a32 Updated description 2021-10-05 11:08:16 +02:00
Ivan Nardi c1e794366f WHOIS: enhance detection, avoiding false positives (#1320)
We are interested only in the domain name required, not in the long reply.
2021-10-05 09:35:04 +02:00
Luca Deri bb7aff6526 Added -a <num> to ndpiReader for generating OPNsense configuration
See https://github.com/ntop/opnsense
2021-10-04 22:34:49 +02:00
Luca Deri fd0e65cb57 Removed trace 2021-10-03 23:41:31 +02:00
Ivan Nardi bc5be1e76c Fix how some protocols handle tcp retransmissions (#1321)
Most (all?) protocols don't care about (tcp) retransmissions.

If a protocol registers itself with a
NDPI_SELECTION_BITMASK_PROTOCOL_*_WITHOUT_RETRANSMISSION value, its
callback is never triggered with a retransmitted packet.
2021-10-03 21:38:49 +02:00
Toni e84da386c7 Added nDPI pkg-config file to Debian / Ubuntu ndpi-dev packaging. (#1318)
* Added nDPI pkg-config file to Debian / Ubuntu ndpi-dev packaging.

 * fixed missing gcrypt library dependency in libndpi.pc

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-03 21:29:56 +02:00
Simone Mainardi 06f763e6c3 Adds sections labels with risk id to the docs 2021-10-01 15:02:14 +02:00
Luca Deri a9d4f5d8d0 Warning fix 2021-10-01 11:36:35 +02:00
Luca Deri 8e6de00ce3 Initial attempt to write nDPI documentation. Starting with flow risks. Please contribute 2021-10-01 11:32:27 +02:00
Ivan Nardi 5cf3fef8f7 Remove detected_protocol_stack field from ndpi_packet_struct (#1317)
This field is an exact copy of `ndpi_flow_struct->detected_protocol_stack[2]`:
* at the very beginning of packet dissection, the value saved in
`flow->detected_protocol_stack` is copied in `packet->detected_protocol_stack`
(via `ndpi_detection_process_packet()` -> `ndpi_init_packet_header()`)

* every time we update `flow->detected_protocol_stack` we update
`packet->detected_protocol_stack` too (via `ndpi_int_change_protocol()`
-> `ndpi_int_change_packet_protocol()`)

These two fields are always in sync: keeping the same value in two
different places is useless.
2021-09-29 13:11:32 +02:00
Alfredo Cardigliano 721031210b Fix warning 2021-09-28 09:44:31 +02:00
Luca Deri af35d6b963 Added unit test for bitmap iteration 2021-09-27 12:47:05 +02:00
Luca Deri b9ccff0761 Compilation fixed on CentOS 7
Bitmap APi changes
2021-09-27 12:41:40 +02:00
Luca Deri 241ee32cbc Bitmap code 2021-09-27 12:06:46 +02:00
Luca Deri 519c69c5cc Reworked bitmap code 2021-09-27 10:29:03 +02:00
Luca Deri 72df138a7d Warnign fix 2021-09-27 09:08:04 +02:00
eac7024489 Fix armhf (#1315)
* Fix unaligned memory accesses with get_u_int64_t at armhf

see: https://bugs.debian.org/993627

* Use get_u_int64_t to avoid unaligned memory access at armhf

see: https://bugs.debian.org/993627

* Update src/include/ndpi_define.h.in

Drop const type from get_u_int64_t, from lnslbrty

Co-authored-by: Bernhard Übelacker <bernhardu@mailbox.org>
Co-authored-by: Toni <matzeton@googlemail.com>
2021-09-26 23:53:00 +02:00
Luca Deri 5ed1930a93 Added include for defining bools (not present on all platforms) 2021-09-26 23:00:13 +02:00
Luca Deri 1efabef4cf Added API for handling compressed bitmaps
ndpi_bitmap* ndpi_bitmap_alloc();
void ndpi_bitmap_free(ndpi_bitmap* b);
u_int64_t ndpi_bitmap_cardinality(ndpi_bitmap* b);
void ndpi_bitmap_set(ndpi_bitmap* b, u_int32_t value);
void ndpi_bitmap_unset(ndpi_bitmap* b, u_int32_t value);
bool ndpi_bitmap_isset(ndpi_bitmap* b, u_int32_t value);
void ndpi_bitmap_clear(ndpi_bitmap* b);
size_t ndpi_bitmap_serialize(ndpi_bitmap* b, char **buf);
ndpi_bitmap* ndpi_bitmap_deserialize(char *buf);

based on https://github.com/RoaringBitmap/CRoaring
2021-09-26 22:55:15 +02:00
Zied Aouini c40cf4e0ec Add arm64, armhf and s390x as part of CI. (#1314)
* Add arm64, armhf and s390x as part on CI.

* Minor fix.

* Fix code inspector complaints.
2021-09-23 14:41:56 +02:00