Commit Graph
2155 Commits
Author SHA1 Message Date
Ivan Nardi f229068fd3 Fix memory access in ndpi_strnstr() (#1438)
Found by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=44482

It should be the same error reported (and only partially fixed) in
79968f32
2022-02-09 11:46:39 +01:00
Ivan NardiandLuca Deri a6ff0dd0e3 Add few scripts to easily update some IPs lists (#1436)
* Add few scripts to easily update some IPs lists

Some IPs lists should be updated frequently: try to easy the process.
The basic idea is taken from d59fefd0 and a8fe74e5 (for Azure
addresses): one specific .c.inc file and one script for each protocol.

Add the possibility to don't load a specific list.

Rename the old NDPI_PROTOCOL_HOTMAIL id to NDPI_PROTOCOL_MS_OUTLOOK,
to identify Hotmail/Outlook/Exchange flows.

TODO: ipv6

Remove the 9 addresses associated to BitTorrent: they have been added in
e2f21116 but it is not clear why all the traffic to/from these ips
should be classified as BitTorrent.

* Added quotes

* Added quotes

Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2022-02-09 11:45:48 +01:00
Luca Deri 9838ceb7cd Added ndpi_serialize_string_string_len() APi call
Fixed CSV string serialization
2022-02-08 22:17:26 +01:00
Luca Deri be5a29ba9d Added HSRP protocol detection
Removed attic directory now obsolete
2022-02-08 18:04:57 +01:00
Luca Deri 922a7be897 Added check to ignore multicast packets marking the as Skype 2022-02-08 17:23:25 +01:00
Toni d59f0327a5 Improved MDNS/LLMNR detection. (#1437)
* Checking for port 5353/5355 is not enough.
 * Added additional multicast address and header checks.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-02-07 18:05:23 +01:00
Ivan Nardi ff38d4b980 TLS: fix parsing of certificate elements (#1435)
Found by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=44280
```
==263603==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x592478 in ndpi_is_printable_string ndpi/src/lib/ndpi_utils.c:2200:9
    #1 0x5b047c in processCertificateElements ndpi/src/lib/protocols/tls.c:400:7
    #2 0x5ac880 in processCertificate ndpi/src/lib/protocols/tls.c:790:7
    #3 0x5c3a32 in processTLSBlock ndpi/src/lib/protocols/tls.c:844:13
    #4 0x5c2c61 in ndpi_search_tls_tcp ndpi/src/lib/protocols/tls.c:973:2
    #5 0x5c117d in ndpi_search_tls_wrapper ndpi/src/lib/protocols/tls.c:2367:5
    #6 0x552a50 in check_ndpi_detection_func ndpi/src/lib/ndpi_main.c:4792:6
```
2022-02-07 18:05:12 +01:00
Ivan Nardi 79968f3245 Sync utests (#1433)
* Sync utest results

* Fix read-heap-buffer-overflow error reported by CI

See: https://github.com/ntop/nDPI/runs/5055876515?check_suite_focus=true
2022-02-04 15:57:17 +01:00
Luca Deri 7aef27f85e Added NDPI_ERROR_CODE_DETECTED risk 2022-02-03 13:20:54 +01:00
Luca Deri 0dd5ac89f8 Renamed DCERPC to more generic RPC protocol so we can use also for other types of RPCs (not limited to DCE)
Extended HTTP plugin to support RPC
Improved HTTP crear text detection to limit it to Basic and Digest
2022-02-03 13:06:27 +01:00
Luca 37ff626e78 Added new IDN/Punycode risk for spotting internationalized domain names 2022-02-03 09:17:54 +01:00
Ivan Nardi 5bb5bec477 Remove struct ndpi_id_struct (#1427)
Remove the last uses of `struct ndpi_id_struct`.
That code is not really used and it has not been updated for a very long
time: see #1279 for details.

Correlation among flows is achieved via LRU caches.

This change allows to further reduce memory consumption (see also
91bb77a8).

At nDPI 4.0 (more precisly, at a6b10cf, because memory stats
were wrong until that commit):
```
nDPI Memory statistics:
	nDPI Memory (once):      221.15 KB
	Flow Memory (per flow):  2.94 KB
```
Now:
```
nDPI Memory statistics:
	nDPI Memory (once):      235.27 KB
	Flow Memory (per flow):  688 B        <--------
```
i.e. memory usage per flow has been reduced by 77%.

Close #1279
2022-01-30 19:18:12 +01:00
Ivan Nardi 1fdcddb33d Remove Playstation VUE protocol (#1426)
PS VUE service has been discontinued on January 30, 2020
https://en.wikipedia.org/wiki/PlayStation_Vue
2022-01-30 19:17:53 +01:00
Luca Deri 5f745ee425 Commented old code (see https://github.com/ntop/nDPI/pull/1425) 2022-01-30 19:17:23 +01:00
Ivan Nardi f455421870 Improve protocol stacks (#1425)
We should have two protocols in classification results only when the
"master" protocol allows some sub-protocols.

Classifications like `AmazonAWS`, `TLS/AmazonAWS`, `DNS/AmazonAWS` are
fine. However classifications like `NTP/Apple`, `BitTorrent/Azure`,
`DNScrypt.AmazonAWS` or `NestLogSink.Google` are misleading.
For example, `ndpiReader`shows  `BitTorrent/Azure` flows under `Azure`
statistics; that seems to be wrong or, at least, very misleading.

This is quite important since we have lots of addresses from CDN
operators.

The only drawback of this solution is that right now ICMP traffic is
classified simply as `ICMP`; if we are really interested in ICMP stuff
we can restore the old behaviour later.
2022-01-30 19:15:42 +01:00
Ivan Nardi 513e386959 Extend protocols support (#1422)
Add detection of AccuWeather site/app and Google Classroom.
Improve detection of Azure, Zattoo, Whatsapp, MQTT and LDAP.

Fix some RX false positives.

Fix some "Uncommon TLS ALPN"-risk false positives.

Fix "confidence" value for some Zoom/Torrent classifications.

Minor fix in Lua script for Wireshark extcap.

Update .gitignore file.

Let GitHub correctly detect the language type of *.inc files.

Zattoo example has been provided by @subhajit-cdot in #1148.
2022-01-29 09:19:26 +01:00
Toni 9b8679a320 Fix some race conditions by using atomic operations. (#1420)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-29 09:18:51 +01:00
Ivan Nardi 0c70411b1b Make some protocols more "big-endian" friendly (#1402)
See #1312
2022-01-29 09:18:32 +01:00
Luca 7a3aa41afc Updated alert description case 2022-01-28 15:02:42 +01:00
Toni c53c82d482 Fixed heap overflow in nDPI realloc wrapper if new size < old size. (#1421)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-27 15:29:16 +01:00
Ivan Nardi 9dcc4a2334 Kerberos, TLS, example: fix some memory errors (#1419)
Detected by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43823
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43921
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43925
2022-01-27 10:59:35 +01:00
Luca Deri 6d185c052e Added support for the .goog Google TLD 2022-01-26 15:22:33 +01:00
Toni 2cd0479204 Serializing empty `risk blocks' pollutes the resulting string. (#1417)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-26 13:36:45 +01:00
Luca Deri 58a9aff17c Added NDPI_TLS_CERTIFICATE_ABOUT_TO_EXPIRE flow risk
Added ndpi_set_tls_cert_expire_days() API call to modify the number of days for triggering the above alert that by default is set to 30 days
2022-01-26 09:23:23 +01:00
Luca Deri 14e8808f03 Removed pandora.tv from pandora protocol as they are different services that shouldd not be mixed 2022-01-25 19:11:16 +01:00
Luca Deri bbb3b8349f Improved pandora TV detection 2022-01-24 18:28:43 +01:00
Luca Deri b6b4967aa6 Improved Zoom protocol detection 2022-01-23 20:59:36 +01:00
Alfredo Cardigliano 20b9f01f12 Fix ndpi_serialize_string_int64 2022-01-21 17:35:38 +01:00
havsah e16b6a18be Fix Grease values parsing (#1416)
The check for grease was too broad and filtered some valid values.
In particular, the value 257 was skipped because it matched the previous check.
This has been discovered while parsing tests/pcap/443-firefox.pcap
expected ja3:
771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,0-23-65281-10-11-35-16-5-51-43-13-45-28-21,29-23-24-25-256-257,0
previously generated ja3:
771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,0-23-65281-10-11-35-16-5-51-43-13-45-28-21,29-23-24-25-256,0

Signed-off-by: Patrick Havelange <patrick.havelange_ext@softathome.com>
2022-01-21 12:40:35 +01:00
Luca Deri 6f336f8307 Added JA3 in risj exceptions 2022-01-20 22:22:56 +01:00
Luca Deri d91477b18d Fixed certificate mismatch check 2022-01-19 23:10:10 +01:00
Ivan Nardi 9f050fa0a6 TLS, H323, examples: fix some memory errors (#1414)
Detected by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=26880
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=26906
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43782
https://oss-fuzz.com/testcase-detail/6334089358082048
2022-01-18 21:52:37 +01:00
Ivan Nardi bd036f96f9 Netbios, CSGO: fix two memory errors (#1413)
Detected by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43754
https://oss-fuzz.com/testcase-detail/5329842395021312
2022-01-18 19:11:01 +01:00
Sam JamesandLuca Deri c2b7d77784 build: respect environment options more (#1392)
* build: update m4/ax_pthread.m4 from serial 23 -> serial 31

Update ax_pthread.m4 to the latest version from the autoconf-archive
project.

Signed-off-by: Sam James <sam@gentoo.org>

* build: properly detect AR, CC, RANLIB

It's necessary to be able to override choice of AR/CC/RANLIB and other toolchain
variables/tools for cross-compilation, testing with other toolchains, and
to ensure the compiler chosen by the user is actually used for the build.

Previously, GNU_PREFIX was kind-of used for this but this isn't a standard
variable (at all) and it wasn't applied consistently anyway.

We now use the standard autoconf mechanisms for finding these tools.

(RANLIB is already covered by LT_INIT.)

Signed-off-by: Sam James <sam@gentoo.org>

* build: use $(MAKE)

This ensures that parallel make works correctly, as otherwise, a fresh
make job will be started without the jobserver fd, and hence
not know about its parent, forcing -j1.

* build: respect CPPFLAGS, LDFLAGS

- CPPFLAGS is for the C preprocessor (usually for setting defines)
- LDFLAGS should be placed before objects for certain flags to work
  (e.g. -Wl,--as-needed)

Signed-off-by: Sam James <sam@gentoo.org>

Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2022-01-18 14:30:14 +01:00
Ivan Nardi e4a7990a4e H323: fix a use-after-poison error (#1412)
Detected by oss-fuzz
See: https://oss-fuzz.com/testcase-detail/6730505580576768

Fix a function prototype
Update a unit test results
2022-01-17 19:47:27 +01:00
Luca Deri d19bd96555 Improved Badoo detection (missing mobile app domain) 2022-01-17 18:54:27 +01:00
Luca Deri 6877e46273 Added Badoo detection 2022-01-17 15:13:14 +01:00
Luca 6679453d86 Adds some risk exceptions for popular services and domain names
via a new (internal) function named ndpi_add_domain_risk_exceptions()
2022-01-17 09:00:25 +01:00
Ivan Nardi 1ff308af09 TLS: fix a use-of-uninitialized-value error (#1411)
Detected by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43705
2022-01-16 20:31:48 +01:00
Ivan Nardi 05734f3d31 Zattoo: fix Null-dereference READ with ipv6 traffic (#1410)
Fix: 20b5f6d7
Detected by oss-fux:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43700
2022-01-16 19:28:39 +01:00
Ivan Nardi 611c753da5 XBox, Diameter: fix dissectors initialization (#1405)
These dissectors have *never* been triggered because their registration
functions use the wrong parameter/bitmask.
Diameter code is buggy since the origianl commit (1d108234), while
XBox code since 5266c726.

Fix some false positives in Xbox code.
2022-01-16 16:19:00 +01:00
Luca Deri f3af39ee42 Added performance tests tools 2022-01-16 12:47:56 +01:00
Ivan Nardi e46b508a36 Kerberos: fix use-of-uninitialized-value error (#1409)
Detected by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43677
2022-01-15 20:53:36 +01:00
Ivan Nardi eb5d7b07af TLS: fix heap-buffer-overflow error (#1408)
Detected by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43664
2022-01-15 18:47:59 +01:00
Ivan Nardi 63c44a24b6 STUN: fix "confidence" value for some classifications (#1407) 2022-01-15 17:35:06 +01:00
Ivan Nardi 20b5f6d7cc Improve IPv6 support, enabling IPv6 traffic on (almost) all dissectors. (#1406)
Follow-up of 7cba34a1
2022-01-15 17:34:56 +01:00
Luca Deri 406ac7e8c8 Added the ability to specify trusted issueDN often used in companies to self-signed certificates
This allows to avoid triggering alerts for trusted albeit private certificate issuers.

Extended the example/protos.txt with the new syntax for specifying trusted issueDN.
Example:

trusted_issuer_dn:"CN=813845657003339838, O=Code42, OU=TEST, ST=MN, C=US"
2022-01-13 19:06:21 +01:00
Vinicius Silva Nogueira dc60cd09c4 fix ahocorasick on big-endian machines (#1401) 2022-01-13 17:35:06 +01:00
Luca Deri d59fefd06e Improved MicrosoftAzure detection 2022-01-12 21:49:31 +01:00
Luca Deri 1e1cfb89d2 Added EthernetIP dissector 2022-01-12 21:48:39 +01:00