Commit Graph
92 Commits
Author SHA1 Message Date
Ivan Nardi 730c2360bd Remove struct ndpi_packet_struct from struct ndpi_flow_struct (#1319)
There are no real reasons to embed `struct ndpi_packet_struct` (i.e. "packet")
in `struct ndpi_flow_struct` (i.e. "flow"). In other words, we can avoid
saving dissection information of "current packet" into the "flow" state,
i.e. in the flow management table.

The nDPI detection module processes only one packet at the time, so it is
safe to save packet dissection information in `struct ndpi_detection_module_struct`,
reusing always the same "packet" instance and saving a huge amount of memory.
Bottom line: we need only one copy of "packet" (for detection module),
not one for each "flow".

It is not clear how/why "packet" ended up in "flow" in the first place.
It has been there since the beginning of the GIT history, but in the original
OpenDPI code `struct ipoque_packet_struct` was embedded in
`struct ipoque_detection_module_struct`, i.e. there was the same exact
situation this commit wants to achieve.

Most of the changes in this PR are some boilerplate to update something
like "flow->packet" into something like "module->packet" throughout the code.
Some attention has been paid to update `ndpi_init_packet()` since we need
to reset some "packet" fields before starting to process another packet.

There has been one important change, though, in ndpi_detection_giveup().
Nothing changed for the applications/users, but this function can't access
"packet" anymore.
The reason is that this function can be called "asynchronously" with respect
to the data processing, i.e in context where there is no valid notion of
"current packet"; for example ndpiReader calls it after having processed all
the traffic, iterating the entire session table.

Mining LRU stuff seems a bit odd (even before this patch): probably we need
to rethink it, as a follow-up.
2021-10-05 15:49:52 +02:00
Luca Deri bb7aff6526 Added -a <num> to ndpiReader for generating OPNsense configuration
See https://github.com/ntop/opnsense
2021-10-04 22:34:49 +02:00
eac7024489 Fix armhf (#1315)
* Fix unaligned memory accesses with get_u_int64_t at armhf

see: https://bugs.debian.org/993627

* Use get_u_int64_t to avoid unaligned memory access at armhf

see: https://bugs.debian.org/993627

* Update src/include/ndpi_define.h.in

Drop const type from get_u_int64_t, from lnslbrty

Co-authored-by: Bernhard Übelacker <bernhardu@mailbox.org>
Co-authored-by: Toni <matzeton@googlemail.com>
2021-09-26 23:53:00 +02:00
Ivan Nardi 7eb73bc92a QUIC: fix old GQUIC versions on big-endian machines (#1313) 2021-09-23 13:24:49 +02:00
Ivan Nardi 994bd0696b Small fixes after latest commits (#1308) 2021-09-18 09:31:00 +02:00
Luca Deri 90c6fd99a7 ARM fixes 2021-08-25 09:46:29 +02:00
Ivan Nardi 9e0f0ce3df Fix access to some TLS fields in flow structure (#1277)
Fields 'tls.hello_processed` and `tls.subprotocol_detected` are used by
QUIC (i.e UDP...), too.
2021-08-20 18:11:37 +02:00
Ivan Nardi 8fdffbf3a1 Compile everything with "-W -Wall -Wno-unused-parameter" flags (#1276)
Fix all the warnings.

Getting rid of "-Wno-unused-parameter" is quite complex because some
parameters usage depends on compilation variable (i.e.
`--enable-debug-messages`).

The "-Werror" flag has been added only in Travis builds to avoid
breaking the builds to users using uncommon/untested
OS/compiler/enviroment.

Tested on:
* x86_64; Ubuntu 20.04; gcc 7,8,9,10,11; clang 7,8,9,10,11,12
* x86_64; CentOS 7.7; gcc 4.8.5 (with "--disable-gcrypt" flag)
* Raspberry 4; Debian 10.10; gcc 8.3.0
2021-08-20 18:11:13 +02:00
Luca Deri 61fc5be202 Reworked flow risk implementation 2021-07-23 17:27:15 +02:00
Ivan Nardi f8fe3ee520 QUIC: add basic support for fragmented Client Hello (#1216)
Only in-order and non overlapping fragments are handled
See #1195
2021-06-24 18:30:34 +02:00
Ivan Nardi c50a8d4808 Add support for Snapchat voip calls (#1147)
* Add support for Snapchat voip calls

Snapchat multiplexes some of its audio/video real time traffic with QUIC
sessions. The peculiarity of these sessions is that they are Q046 and
don't have any SNI.

* Fix tests with libgcrypt disabled
2021-03-06 05:48:36 +01:00
Luca Deri 56bfb439f8 Improved DGA detection with trigrams. Disadvantage: slower startup time
Reworked Tor dissector embedded in TLS (fixes #1141)
Removed false positive on HTTP User-Agent
2021-03-03 00:41:07 +01:00
Luca Deri a31bd5ac3c Cleaned up tls/quic datatypes 2021-01-21 19:17:33 +01:00
Luca Deri 68b6ac7da8 (C) Update 2021-01-07 11:13:36 +01:00
Luca Deri 367184628e Warning fix 2021-01-07 11:07:17 +01:00
Ivan Nardi 2080cc7365 QUIC: add suppport for DNS-over-QUIC (#1107)
Even if it is only an early internet draft, DoQ has already (at least)
one deployed implementation.
See: https://www.zdnet.com/article/ad-blocker-adguard-deploys-worlds-first-dns-over-quic-resolver/
Draft: https://tools.ietf.org/html/draft-huitema-dprive-dnsoquic-00

In the future, if this protocol will be really used, it might be worth to
rename NDPI_PROTOCOL_DOH_DOT in NDPI_PROTOCOL_DOH_DOT_DOQ
2021-01-07 10:56:39 +01:00
Ivan Nardi 00dabce65e Quic fixes (#1106)
* QUIC: fix heap-buffer-overflow

* TLS: fix parsing of QUIC Transport Parameters
2021-01-07 10:55:55 +01:00
Ivan Nardi b8a5358e80 QUIC: improve handling of SNI (#1105)
* QUIC: SNI should be always saved in flow->protos.stun_ssl.ssl.client_requested_server_name

Close #1077

* QUIC: fix matching of custom categories

* QUIC: add NDPI_TLS_MISSING_SNI support for older GQUIC versions

* QUIC: fix serialization

* QUIC: add DGA check for older GQUIC versions
2021-01-07 10:55:23 +01:00
Ivan Nardi 1b524f5538 QUIC: update to draft-33 (#1104)
QUIC (final!?) constants for v1 are defined in draft-33
2021-01-04 15:50:14 +01:00
Ivan Nardi 860ef58ace QUIC: sync with Wireshark latest changes (#1074)
Most of the QUIC crypto code has been "copied-and-pasted" from Wireshark;
try to stay in sync with the original sources to ease backporting of fixes.

Only cosmetic changes and code refactoring; no behaviour changes or bugfixes.
See:
https://gitlab.com/wireshark/wireshark/-/commit/5e45f770fd79ca979c41ed397fee72d2e8fb5f1e
https://gitlab.com/wireshark/wireshark/-/commit/5798b91c1526747bf688b6746b33562c1b24a9e0
2020-12-08 15:47:58 +01:00
Alfredo Cardigliano 7437ecaab0 Warning fix 2020-11-23 10:19:35 +01:00
Ivan Nardi 53a5c354d8 Quic fixes (#1067)
* QUIC: fix return value on error path on quic_cipher_init()

* QUIC: allow dissection of sessions forcing version negotiation

Enhance heuristic to avoid false positives.
2020-11-22 11:04:10 +01:00
Ivan Nardi a9547da138 QUIC: fix dissection of Initial packets coalesced with 0-RTT one (#1044)
* QUIC: fix dissection of Initial packets coalesced with 0-RTT one

* QUIC: fix a memory leak
2020-11-03 11:35:52 +01:00
Ivan Nardi 2c1b7cf1bd QUIC: fix dissection of "offset" field (#1025)
The "offset" field is a variable-length integer.
This bug hasn't any practical effects right now, since we are ignoring any
packet with "offset" != 0 (and the value 0 is always encoded in only one byte).
But extracting a correct "offset" is important if we are ever going to handle
fragmented Client Hello messages.
2020-09-29 17:59:03 +02:00
Luca Deri 5f99433ee7 Minor UA handling improvement to avoid heap-overflow 2020-09-22 08:56:35 +02:00
Nardi Ivan dcac633878 QUIC: add support for MVFST EXPERIMENTAL version 2020-09-20 16:38:28 +02:00
Luca Deri 753b5dde16 Merge pull request #1012 from IvanNardi/ua
QUIC: extract User Agent information
2020-09-17 21:32:25 +02:00
Luca Deri 9189aee83b Disabled QUIC tracing that pollutes the output 2020-09-17 14:37:56 +02:00
Nardi Ivan 8db084ab06 TLS: fix memory accesses in QUIC transport parameters extension 2020-09-10 18:19:18 +02:00
Nardi Ivan 7d5a0e1f04 QUIC: extract User Agent information 2020-09-08 11:03:22 +02:00
Nardi Ivan 7da4abe6ad QUIC: add support for GQUIC T050 and T051
QUIC versioning wasn't complex enough without T05X family...
These versions are very similar to Q050, but use TLS as their handshake
protocol.
2020-08-30 20:51:33 +02:00
Toni Uhlig e998fc28d1 Fixed use-of-uninitialized-value in QUIC clho decryption probably caused by a BUG in libgcrypt (not verified).
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-08-27 16:30:05 +02:00
Nardi Ivan 97b80a8838 QUIC: minor fixes
LGTM found a real issue on a boundary check
Fix unit tests: a pcap ha been uploaded twice (with different names)
Fix compilation when using DPDK (see #990)
2020-08-24 13:53:36 +02:00
Luca Deri 93003d0523 Warning fix 2020-08-22 16:41:42 +02:00
Nardi Ivan b23cfd6b84 Add sub-classification for GQUIC >= Q050 and (IETF-)QUIC
Add QUIC payload and header decryption: most of the crypto code has been
"copied-and-incolled" from Wireshark. That code has been clearly marked
as such. All credits for that code should go to the original authors.

I tried to keep the Wireshark code as similar as possible to the original,
comments included, to ease future backporting of fixes.
Inevitably, glibc data types and data structures, tvbuff abstraction and
allocation functions have been converted.
2020-08-21 22:04:55 +02:00
Nardi Ivan 23ec82b59d Major rework of QUIC dissector
Improve support for GQUIC (up to Q046) and add support for Q050 and (IETF-)QUIC
Still no sub-classification for Q050 and QUIC
2020-08-21 22:04:55 +02:00
Luca Deri ea10b8e757 Added memory checks 2020-08-02 13:00:31 +02:00
Luca Deri 69f140878c Boundary check on QUIC 2020-07-27 07:40:50 +02:00
Philippe Antoine 369dc65c1a Fix integer overflow in quic 2020-04-15 15:45:32 +02:00
Nardi Ivan 4f399356a6 quic: fix heap-overflow error 2020-03-27 11:09:53 +01:00
Luca Deri 4227be7989 Update for QUIC 046 2020-03-17 16:48:34 +01:00
Luca Deri ef16591f3f Merge branch 'dev' of https://github.com/ntop/nDPI into dev 2020-01-05 18:25:44 +01:00
Luca Deri 29dd45838d Updated (C) 2020-01-05 18:24:58 +01:00
Luca 257ec7cc5f Removed disable_metadata_export preference that is no longer useful
since ndpi_process_extra_packet() can drive limited or full metadata export
2019-12-29 08:07:35 +01:00
Luca Deri ab9246f51f Adedd DTLS check in STUN
Uodated (C)
2019-09-21 09:16:56 +02:00
Luca Deri ff1234cce3 Changes in QUIC dissector to avoid miing QUIC with STUN 2019-08-06 23:13:47 +02:00
Luca Deri 57ef7c1276 Added QUIC 046 support 2019-07-18 15:00:24 +02:00
Luca Deri 3e2e616507 Dissection improvents
Fixed QUIC dissection
Disabled debug message
2019-07-11 22:35:50 +02:00
Luca Deri e9e647f043 ntohl Portability fixes 2019-06-29 17:34:08 +02:00
Guido Falsi b1282da542 The le32toh() function used in some places on BSD OSes require sys/endian.h to be included. 2019-03-12 18:12:18 +01:00