Commit Graph
4126 Commits
Author SHA1 Message Date
Ivan Nardi ace32c9dfe Add support for SRTP (#1977)
The goal is to have Zoom flows classified as "Encrypted" and not as
"Cleartext".

Start documenting the list of protocols supported by nDPI;
format, verbosity and content are still a work-in-progress.
2023-05-16 16:02:14 +02:00
Ivan Nardi 4e186f6bfb HTTP: rework state machine (#1966)
The goal if to correlate the right request-response pair, exporting
metadata from only one transaction (for example, the right url & return
state pair)

As a nice side effect, the code should be much cleaner, but that is a
matter of taste.

Two differences respect to the previous code:
* as it happens in the CI, if in the flow there are only one response
(before) and one request (after), only the metadata of the response are
saved/exported
* for performance reasons, we don't call `ndpi_parse_packet_line_info()`
anymore for ALL packets triggering the HTTP dissector, but only for the
packets that we already know belong to an HTTP flow. This is the reason
for the changes in RTSP/SOAP/... code
2023-05-16 16:01:55 +02:00
Ivan Nardi 8c224b464f Fix MS Teams detection with heuristic (#1972)
When we reconcile a TLS session to Teams, we need to keep TLs as master.
This way:
* we keep exporting all the TLS metadata
* we avoid some memory leaks (of these metadata themeselves)
2023-05-10 21:00:27 +02:00
Luca Deri 596a5fd5dc Updated results 2023-05-10 17:11:14 +02:00
Luca 64853dff02 Improved MS Teams detection with heuristic 2023-05-10 16:59:28 +02:00
Luca 98f029a091 Enhanced MS teams STUN/Azure detection 2023-05-10 16:28:34 +02:00
Luca 7d5e4d0f7e MS Teams enhancement 2023-05-10 15:41:26 +02:00
Luca 8de7c49298 MS Teams improvements 2023-05-10 15:16:39 +02:00
Ivan Nardi 31d23aee56 All protocols should be excluded sooner or later (#1969)
For a lot of protocols, reduce the number of packets after which the
protocols dissector gives up.
The values are quite arbitary, tring to not impact on classification
2023-05-10 12:50:24 +02:00
Ivan Nardi 99d7066ea0 Remove special handling of some TCP flows without SYN (#1965)
This piece of code has multiple problems:
* nDPI is able to detect some TCP protocols even with mid-flows (i.e.
without the initial packets of the session); TLS is the most
significative example
* since e6b332aa4a it is perfectly valid
to not pass the TCP Handshake packets to nDPI
* in any case, we shouldn't call `ndpi_detection_giveup()`. That
function is usually called by the application and we end up calling it
twice in some cases.

The simple solution is to completely remove that code: process these
kinds of flows like everyone else.

Note that the application can always avoid to pass to nDPI any TCP flows
without the initial handshake; the flow managemnt is always up to the
application.

Looking at the CI results, some rare flows are now processed significantly
longer. As a follow-up we could look into that.
2023-05-09 19:36:02 +02:00
Ivan Nardi 684e041998 Improve detection of crawlers/bots (#1968)
Add support for Facebook crawler
2023-05-09 16:42:29 +02:00
Ivan Nardi 79c1dbe0b2 Extend content-match list (#1967) 2023-05-09 16:41:12 +02:00
Luca Deri 2f9059dc54 Disabled query string validation in MDNS in order to avoid zapping chars that in DNS (instead) are not permitted 2023-05-09 15:04:02 +02:00
Luca Deri bfe79243bc Refreshed lists 2023-05-08 16:51:38 +02:00
Clément BRUGUERA 998bedbbb7 add 2 ns from fdn.fr to DoH section (#1964) 2023-05-05 18:08:13 +02:00
Ivan Nardi 0223d3c4f5 HTTP: improve extraction of metadata and of flow risks (#1959) 2023-05-05 13:35:20 +02:00
Ivan Nardi 02a2c80453 ndpiReader: fix print of flow payload (#1960) 2023-05-04 11:52:48 +02:00
Ivan Nardi a11ddd2dc4 HTTP: remove useless code about XBOX (#1958)
Old nDPI versions were able to detect XBOX flows over HTTP via
user-agent matching. This feature has been removed from a long time
(89d548f9d, at very least)
2023-05-04 11:28:03 +02:00
Ivan Nardi 6b94c9675a Improve detection of crawler/bot traffic (#1956) 2023-05-04 11:27:34 +02:00
Ivan Nardi 9a5d60bff6 Sync unit tests results (#1962) 2023-05-04 09:47:52 +02:00
Luca Deri 3681de4795 Updated pcap detection results after Facebook Reel/Stories support 2023-05-03 22:52:58 +02:00
Luca Deri de13c6c0bd Added detection of Facebook Reels and Stories
https://www.switcherstudio.com/blog/the-ultimate-guide-to-reels-and-stories#:~:text=Audience%3A%20Reels%20are%20served%20to,things%20like%20photos%20and%20boomerangs.
2023-05-03 22:50:02 +02:00
Ivan Nardi bb370f5ef0 Make Bittorrent LRU cache IPv6 aware. (#1909)
It was the only remaining LRU cache without IPv6 support.
See 81e1ea545c
2023-04-29 11:09:23 +02:00
Luca Deri 6b803f48dc Added missing check 2023-04-28 23:08:54 +02:00
Luca Deri 6e9dce357b Added ndpi_guess_undetected_protocol_v4() to restore logic that
was removed in https://github.com/ntop/nDPI/commit/032e778a6dc1ee2173c4cdb2c671180c5b5df245
2023-04-28 23:00:33 +02:00
Luca Deri f7138f07a6 Missing module termination 2023-04-28 23:00:33 +02:00
Luca Deri b39ece4428 Updated (C) 2023-04-28 23:00:33 +02:00
Luca Deri 791d3e6ad3 Added missing check 2023-04-28 23:00:33 +02:00
Ivan Nardi 8934f7b45f Add an heuristic to detect/ignore some anomalous TCP ACK packets (#1948)
In some networks, there are some anomalous TCP flows where the smallest
ACK packets have some kind of zero padding.
It looks like the IP and TCP headers in those frames wrongly consider the
0x00 Ethernet padding bytes as part of the TCP payload.
While this kind of packets is perfectly valid per-se, in some conditions
they might be treated by the TCP reassembler logic as (partial) overlaps,
deceiving the classification engine.
Add an heuristic to detect these packets and to ignore them, allowing
correct detection/classification.

This heuristic is configurable. Default value:
* in the library, it is disabled
* in `ndpiReader` and in the fuzzers, it is enabled (to ease testing)

Credit to @vel21ripn for the initial patch.

Close #1946
2023-04-25 19:25:07 +02:00
Ivan Nardi 40b6d5a2e1 fuzz: extend fuzzers coverage (#1952) 2023-04-25 16:37:28 +02:00
Ivan Nardi de693cbbc9 RTP: remove dead-code (#1953)
The checks `isValidMSRTPType(..) == 1` is a subset of
`is_valid_rtp_payload_type()` so this if-branch is never reached.

More importantly, the article describing how to detect Microsoft Lync and
Skype for Business is from 2014. These payload types are static or they
are in the dynamic range: in both cases, these values might be used (and
they are used indeed) pretty much by every application.
Bottom line: we can't use PT alone to identify a specific protocol.

Keep the list, since it is used to tell audio streams from video ones.
2023-04-25 14:16:27 +02:00
Luca Deri c4ba981370 Compilation fix on RH7 2023-04-24 08:49:11 +02:00
nikitamishagin d3e67faf20 Add "Heroes of the Storm" video game signature detection. (#1949) 2023-04-22 18:28:43 +02:00
Toni b6629ba2be Improved debug output. (#1951)
* try to get rid of some `printf(..)`s as they do not belong to a shared library
 * replaced all `exit(..)`s with `abort()`s to indicate an abnormal process termination

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-04-21 12:40:26 +02:00
Toni 85751ba9d7 Added OICQ dissector. (#1950)
Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-04-21 11:30:10 +02:00
Ivan Nardi ba99330031 ndpiReader: fix flow stats (#1943) 2023-04-14 11:17:37 +02:00
Ivan Nardi 032e778a6d Simplify ndpi_internal_guess_undetected_protocol() (#1941)
`ndpi_guess_undetected_protocol()/ndpi_internal_guess_undetected_protocol()`
is a strange function:
* it is exported by the library and it is actively used by `ntopng`
* it is intrinsecally ipv4-only
* it returns basically something like "classification_by_ip"/"classification_by_port"
(these information have already been calculated in `ndpi_do_guess()`...)
* it access the bittorrent LRU caches (similarly to
`ndpi_detection_giveup()` but without all the other caches...)

So:
* make the interface IPv4/6 agnostic
* use the classifications already available

This work will allow to make the Bittorrent caches IPV6-aware (see
81e1ea5).

Handle Dropbox classification "by-port" in the "standard" way.
2023-04-12 14:39:10 +02:00
Toni 6660feed35 Added BACnet dissector. (#1940)
Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-04-11 20:26:27 +02:00
lns 77ea80862f Improved debug logging.
Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-04-11 15:39:09 +02:00
lns 6da958655c Improved invalid logging via printf().
Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-04-11 15:39:09 +02:00
Toni bebcce5281 Added Source Engine dissector. (#1937)
Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-04-11 14:31:57 +02:00
Luca Deri 9ba16a4837 Test files for riit games 2023-04-11 10:14:18 +02:00
Jacopo a469009842 RiotGames: add detection of flows (#1935)
* RiotGames: add detection of flows

* remove of akamai domains
2023-04-11 10:04:09 +02:00
Ivan Nardi c7ef3608d9 CI: fix Performance job (#1936)
Fix: 7714507f
2023-04-09 10:16:28 +02:00
Luca Deri c47e9d201d Implemented ndpi_XXX_reset() API calls whre XXX is ses, des, hw 2023-04-08 09:52:14 +02:00
Ivan Nardi 7714507f81 Test multiple ndpiReader configurations (#1931)
Extend internal unit tests to handle multiple configurations.
As some examples, add tests about:
* disabling some protocols
* disabling Ookla aggressiveness

Every configurations data is stored in a dedicated directory under
`tests\cfgs`
2023-04-06 11:30:36 +02:00
Ivan Nardi 7a627296f0 Fix LRU/Patricia/Automa stats in ndpiReader with multiple threads (#1934) 2023-04-06 09:36:11 +02:00
Ivan Nardi b639744c3a Extend content list of Microsoft protocols (#1930) 2023-04-06 09:35:41 +02:00
Ivan Nardi 22cf8b5d8f configure: add an option to enable debug build, i.e -g (#1929)
Add this new flag to one CI job, to test it

Close #1925
2023-04-06 09:35:27 +02:00
Ivan Nardi 06a63dcbaa MySql: improve detection (#1928)
Add support for flows with "caching_sha2_password" authentication plugin.

See #1924
2023-04-06 09:34:39 +02:00