Commit Graph
2174 Commits
Author SHA1 Message Date
Luca Deri 06f13fd2de Added clertext alert with telnet 2021-10-19 08:44:29 +02:00
Toni ed51987e3a Fix broken fuzz_process_packet fuzzer by adding a call to ndpi_finalize_initialization(). (#1334)
* fixed several memory errors (heap-overflow, unitialized memory, etc)
 * ability to build fuzz_process_packet with a main()
   allowing to replay crash data generated with fuzz_process_packet
   by LLVMs libfuzzer
 * temporarily disable fuzzing if `tests/do.sh`
   executed with env FUZZY_TESTING_ENABLED=1

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-18 23:16:32 +02:00
Luca Deri 7d3c3b23f8 Implemented RDP over UDP dissection 2021-10-18 22:12:28 +02:00
Luca Deri b08b6a6a28 Fixed cleartext protocol assignment 2021-10-16 12:02:04 +02:00
Luca Deri 1b3c41a752 Reworked flow risks asignment
Added esceptions for windows update and binary application transfer risk
2021-10-16 11:55:08 +02:00
Luca Deri b97dc6baa4 Removed outdated (and broken) soulseek dissector 2021-10-15 17:27:27 +02:00
Luca Deri b1a43cbcff Added missing call to ndpi_reconcile_protocols() with protocol guess 2021-10-14 22:08:20 +02:00
Luca Deri 6e649e0cb6 Fixed issue on DGA numeric IP detection 2021-10-13 19:21:27 +02:00
Luca Deri 09668496c3 Indemt 2021-10-13 19:18:05 +02:00
Luca Deri 937357e4bc Implemented ndpi_ses_fitting() and ndpi_des_fitting()
for comuting the best alpha/beta values for exponential smoothing
2021-10-12 13:08:58 +02:00
Alfredo Cardigliano d947437233 Return 0 on ndpi_init_serializer_ll success for consistency 2021-10-12 10:47:44 +00:00
Luca Deri d13bc0e01a Fixed DES initialisation 2021-10-12 12:38:40 +02:00
Ivan Nardi 550e6fe6fc QUIC: fix an integer overflow (#1337)
Long standing bug: credits to @lnslbrty for digging into it and to
@aouinizied for the CI improvements
2021-10-11 23:08:10 +02:00
Luca Deri 960e8ab22a Win fix 2021-10-11 20:59:28 +02:00
Luca Deri 246d115d7b Cleaned up code moving specific includes in files their are using it. Thi prevents ndpi_config.h to be included everywhere in apps using nDPI that might leade to #define redefinitions after the latest changes 2021-10-11 11:45:45 +02:00
Toni 6ff67b5043 Add more WindowsUpdate URLs as requested in #698. (#1329)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-08 18:29:49 +02:00
Luca Deri 3ffb65a81c Added extra check 2021-10-07 08:21:40 +02:00
Ivan Nardi 2dfc478ad4 Fix compilation with clang-13 or if some debug macros are enabled (#1326) 2021-10-06 18:32:15 +02:00
Luca Deri 944e8ed105 Added check 2021-10-06 18:28:09 +02:00
Luca Deri 2b4821a97b Compilation fix for old Linux distributions 2021-10-06 09:28:04 +02:00
Luca Deri 181a03c5ad Compilation fix 2021-10-05 17:17:12 +02:00
Luca Deri 8957c52a48 Numeric IPs are not considered for DGA checks 2021-10-05 17:14:31 +02:00
Luca Deri 408d78e628 Improved DGA detection for skipping potential DGAs of known/popular domain names 2021-10-05 16:51:24 +02:00
Luca Deri 10576eab0a TLS obsolete protocol is set when TLS < 1.2 (used to be 1.1) 2021-10-05 16:44:56 +02:00
Ivan Nardi 730c2360bd Remove struct ndpi_packet_struct from struct ndpi_flow_struct (#1319)
There are no real reasons to embed `struct ndpi_packet_struct` (i.e. "packet")
in `struct ndpi_flow_struct` (i.e. "flow"). In other words, we can avoid
saving dissection information of "current packet" into the "flow" state,
i.e. in the flow management table.

The nDPI detection module processes only one packet at the time, so it is
safe to save packet dissection information in `struct ndpi_detection_module_struct`,
reusing always the same "packet" instance and saving a huge amount of memory.
Bottom line: we need only one copy of "packet" (for detection module),
not one for each "flow".

It is not clear how/why "packet" ended up in "flow" in the first place.
It has been there since the beginning of the GIT history, but in the original
OpenDPI code `struct ipoque_packet_struct` was embedded in
`struct ipoque_detection_module_struct`, i.e. there was the same exact
situation this commit wants to achieve.

Most of the changes in this PR are some boilerplate to update something
like "flow->packet" into something like "module->packet" throughout the code.
Some attention has been paid to update `ndpi_init_packet()` since we need
to reset some "packet" fields before starting to process another packet.

There has been one important change, though, in ndpi_detection_giveup().
Nothing changed for the applications/users, but this function can't access
"packet" anymore.
The reason is that this function can be called "asynchronously" with respect
to the data processing, i.e in context where there is no valid notion of
"current packet"; for example ndpiReader calls it after having processed all
the traffic, iterating the entire session table.

Mining LRU stuff seems a bit odd (even before this patch): probably we need
to rethink it, as a follow-up.
2021-10-05 15:49:52 +02:00
Luca Deri da8eed5a32 Updated description 2021-10-05 11:08:16 +02:00
Ivan Nardi c1e794366f WHOIS: enhance detection, avoiding false positives (#1320)
We are interested only in the domain name required, not in the long reply.
2021-10-05 09:35:04 +02:00
Luca Deri bb7aff6526 Added -a <num> to ndpiReader for generating OPNsense configuration
See https://github.com/ntop/opnsense
2021-10-04 22:34:49 +02:00
Ivan Nardi bc5be1e76c Fix how some protocols handle tcp retransmissions (#1321)
Most (all?) protocols don't care about (tcp) retransmissions.

If a protocol registers itself with a
NDPI_SELECTION_BITMASK_PROTOCOL_*_WITHOUT_RETRANSMISSION value, its
callback is never triggered with a retransmitted packet.
2021-10-03 21:38:49 +02:00
Simone Mainardi 06f763e6c3 Adds sections labels with risk id to the docs 2021-10-01 15:02:14 +02:00
Ivan Nardi 5cf3fef8f7 Remove detected_protocol_stack field from ndpi_packet_struct (#1317)
This field is an exact copy of `ndpi_flow_struct->detected_protocol_stack[2]`:
* at the very beginning of packet dissection, the value saved in
`flow->detected_protocol_stack` is copied in `packet->detected_protocol_stack`
(via `ndpi_detection_process_packet()` -> `ndpi_init_packet_header()`)

* every time we update `flow->detected_protocol_stack` we update
`packet->detected_protocol_stack` too (via `ndpi_int_change_protocol()`
-> `ndpi_int_change_packet_protocol()`)

These two fields are always in sync: keeping the same value in two
different places is useless.
2021-09-29 13:11:32 +02:00
Luca Deri b9ccff0761 Compilation fixed on CentOS 7
Bitmap APi changes
2021-09-27 12:41:40 +02:00
Luca Deri 241ee32cbc Bitmap code 2021-09-27 12:06:46 +02:00
Luca Deri 519c69c5cc Reworked bitmap code 2021-09-27 10:29:03 +02:00
eac7024489 Fix armhf (#1315)
* Fix unaligned memory accesses with get_u_int64_t at armhf

see: https://bugs.debian.org/993627

* Use get_u_int64_t to avoid unaligned memory access at armhf

see: https://bugs.debian.org/993627

* Update src/include/ndpi_define.h.in

Drop const type from get_u_int64_t, from lnslbrty

Co-authored-by: Bernhard Übelacker <bernhardu@mailbox.org>
Co-authored-by: Toni <matzeton@googlemail.com>
2021-09-26 23:53:00 +02:00
Luca Deri 5ed1930a93 Added include for defining bools (not present on all platforms) 2021-09-26 23:00:13 +02:00
Luca Deri 1efabef4cf Added API for handling compressed bitmaps
ndpi_bitmap* ndpi_bitmap_alloc();
void ndpi_bitmap_free(ndpi_bitmap* b);
u_int64_t ndpi_bitmap_cardinality(ndpi_bitmap* b);
void ndpi_bitmap_set(ndpi_bitmap* b, u_int32_t value);
void ndpi_bitmap_unset(ndpi_bitmap* b, u_int32_t value);
bool ndpi_bitmap_isset(ndpi_bitmap* b, u_int32_t value);
void ndpi_bitmap_clear(ndpi_bitmap* b);
size_t ndpi_bitmap_serialize(ndpi_bitmap* b, char **buf);
ndpi_bitmap* ndpi_bitmap_deserialize(char *buf);

based on https://github.com/RoaringBitmap/CRoaring
2021-09-26 22:55:15 +02:00
Ivan Nardi 7eb73bc92a QUIC: fix old GQUIC versions on big-endian machines (#1313) 2021-09-23 13:24:49 +02:00
Zied Aouini 0994771974 Improve CI (#1303)
* Improve CI pipeline

* Fix branch name.

* Fix branch name.

* Fix libgcrypt configuration.

* Update build.yml

* Move to Github Actions instead of Travis CI.

* Fix mingw on ubuntu bionic.

* Reactivate cross compile on Ubuntu Bionic.

* Switch to single line steps.

* Add several compilers versions

* Minor fix.

* Fix build all and delete cxx

* Fix RCE detection.

* Fix PCRE configuration.

* Add condition on PCRE test pcap.

* Update WebattackRCE.pcap.out

* Add missing SUBST.

* Delete WebattackRCE.pcap.out

* Update WebAttackRCE result.

* Fix typo.

* Extend jobs with pcre+msan+maxminddb.

* Fix code inpector warnings.

* Delete .appveyor.yml
2021-09-22 12:10:52 +02:00
Nardi Ivan 017c1a4239 FTP: fix support for START-TLS sessions
When TLS-over-FTP is used, the credentials are encrypted. So we must not
wait for the username and the password commands, otherwise we elaborate a
lot of packets for nothing.
2021-09-21 20:53:04 +02:00
Toni bb52f1362c Merge pull request #1309 from IvanNardi/ipv6-headers
Fix parsing of ipv6 packets with extension headers
2021-09-20 12:54:01 +02:00
Nardi Ivan 95ac8fd41a STUN: fix extraction of Realm attribute
While at it, improve detection of Facebook Messenger
2021-09-20 11:04:54 +02:00
Nardi Ivan 03d3e1bafc Fix parsing of ipv6 packets with extension headers
Decoding of ipv6 traffic with extension headers was completely broken,
since the beginning of the L4 header was always set to a wrong value.

Handle the ipv6 fragments in the same way as the ipv4 ones: keep the first
one and drop the others.
2021-09-19 17:29:22 +02:00
Ivan Nardi 994bd0696b Small fixes after latest commits (#1308) 2021-09-18 09:31:00 +02:00
Ivan Nardi 227e586e63 A (tiny) effort to reduce memory usage per flow (#1304)
Follow-up of 22241a1d
Only trivial changes:
* remove completely unused fields
* remove fields only written (but never read)
* CSGO protocol only handles UDP traffic
2021-09-18 09:29:45 +02:00
Luca Deri 3303dac7cf Improved fragmented DNS detection 2021-09-17 15:17:39 +02:00
Luca Deri 119a14c6e3 DNS dissection fixes 2021-09-17 12:20:19 +02:00
deboracerretiniandLuca Deri 8f113c1d37 Progetto esame Gestione di Reti - Debora Cerretini (#1290)
* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2021-09-17 11:01:50 +02:00
Ivan Nardi 6325aebda6 TLS: avoid zeroing large structures (#1300)
Zeroing large structures (i.e. size > KB) is quite costly (from a CPU point
of view): we can safely avoid doing that for a couple of big structures.

Standard and Valgrind tests have been diverging quite a lot: it is time
to re-sync them. Use the same script and enable Valgrind via an
enviroment variable:

  NDPI_TESTS_VALGRIND=1 ./tests/do.sh
2021-09-16 14:34:59 +02:00
Simone Mainardi 978c9cfda3 Adds instructions to update flow risks in ntopng 2021-09-15 17:49:47 +02:00