Compare commits

..
258 Commits
Author SHA1 Message Date
Luca 8b5c6af71b Updated release to 4.2 2022-02-01 09:14:05 +01:00
Alfredo Cardigliano 03cb8efad6 Update changelog 2022-01-31 18:38:02 +01:00
Ivan Nardi 5bb5bec477 Remove struct ndpi_id_struct (#1427)
Remove the last uses of `struct ndpi_id_struct`.
That code is not really used and it has not been updated for a very long
time: see #1279 for details.

Correlation among flows is achieved via LRU caches.

This change allows to further reduce memory consumption (see also
91bb77a8).

At nDPI 4.0 (more precisly, at a6b10cf, because memory stats
were wrong until that commit):
```
nDPI Memory statistics:
	nDPI Memory (once):      221.15 KB
	Flow Memory (per flow):  2.94 KB
```
Now:
```
nDPI Memory statistics:
	nDPI Memory (once):      235.27 KB
	Flow Memory (per flow):  688 B        <--------
```
i.e. memory usage per flow has been reduced by 77%.

Close #1279
2022-01-30 19:18:12 +01:00
Ivan Nardi 1fdcddb33d Remove Playstation VUE protocol (#1426)
PS VUE service has been discontinued on January 30, 2020
https://en.wikipedia.org/wiki/PlayStation_Vue
2022-01-30 19:17:53 +01:00
Luca Deri 5f745ee425 Commented old code (see https://github.com/ntop/nDPI/pull/1425) 2022-01-30 19:17:23 +01:00
Ivan Nardi f455421870 Improve protocol stacks (#1425)
We should have two protocols in classification results only when the
"master" protocol allows some sub-protocols.

Classifications like `AmazonAWS`, `TLS/AmazonAWS`, `DNS/AmazonAWS` are
fine. However classifications like `NTP/Apple`, `BitTorrent/Azure`,
`DNScrypt.AmazonAWS` or `NestLogSink.Google` are misleading.
For example, `ndpiReader`shows  `BitTorrent/Azure` flows under `Azure`
statistics; that seems to be wrong or, at least, very misleading.

This is quite important since we have lots of addresses from CDN
operators.

The only drawback of this solution is that right now ICMP traffic is
classified simply as `ICMP`; if we are really interested in ICMP stuff
we can restore the old behaviour later.
2022-01-30 19:15:42 +01:00
Ivan Nardi 513e386959 Extend protocols support (#1422)
Add detection of AccuWeather site/app and Google Classroom.
Improve detection of Azure, Zattoo, Whatsapp, MQTT and LDAP.

Fix some RX false positives.

Fix some "Uncommon TLS ALPN"-risk false positives.

Fix "confidence" value for some Zoom/Torrent classifications.

Minor fix in Lua script for Wireshark extcap.

Update .gitignore file.

Let GitHub correctly detect the language type of *.inc files.

Zattoo example has been provided by @subhajit-cdot in #1148.
2022-01-29 09:19:26 +01:00
Toni 9b8679a320 Fix some race conditions by using atomic operations. (#1420)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-29 09:18:51 +01:00
Ivan Nardi 0c70411b1b Make some protocols more "big-endian" friendly (#1402)
See #1312
2022-01-29 09:18:32 +01:00
Ivan Nardi 86b97ffb73 Sync unit tests results (#1423)
Fix: 7a3aa41a
2022-01-28 17:17:39 +01:00
Luca 7a3aa41afc Updated alert description case 2022-01-28 15:02:42 +01:00
Toni c53c82d482 Fixed heap overflow in nDPI realloc wrapper if new size < old size. (#1421)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-27 15:29:16 +01:00
Ivan Nardi 9dcc4a2334 Kerberos, TLS, example: fix some memory errors (#1419)
Detected by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43823
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43921
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43925
2022-01-27 10:59:35 +01:00
Toni 011ee3ecbd Fixed wrong ip tuple comparison. #1386 (#1418)
* Added u32 pads to `union ip_tuple` so btree search should now work as expected.
   The bug caused new flow's when the remote answers, resulting in two Flows per direction. Fail.
 * Fixed a race condition during shutdown phase.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-26 15:47:38 +01:00
Luca Deri 6d185c052e Added support for the .goog Google TLD 2022-01-26 15:22:33 +01:00
Toni 2cd0479204 Serializing empty `risk blocks' pollutes the resulting string. (#1417)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-01-26 13:36:45 +01:00
Luca Deri 81b33f0d06 Updated test results after the risk NDPI_TLS_CERTIFICATE_ABOUT_TO_EXPIRE has been added 2022-01-26 09:46:46 +01:00
Luca Deri 58a9aff17c Added NDPI_TLS_CERTIFICATE_ABOUT_TO_EXPIRE flow risk
Added ndpi_set_tls_cert_expire_days() API call to modify the number of days for triggering the above alert that by default is set to 30 days
2022-01-26 09:23:23 +01:00
Luca Deri 14e8808f03 Removed pandora.tv from pandora protocol as they are different services that shouldd not be mixed 2022-01-25 19:11:16 +01:00
Luca Deri bbb3b8349f Improved pandora TV detection 2022-01-24 18:28:43 +01:00
Luca Deri b6b4967aa6 Improved Zoom protocol detection 2022-01-23 20:59:36 +01:00
Luca Deri a8fe74e502 Tool for generating automatically the Azure IP list 2022-01-23 18:56:51 +01:00
Alfredo Cardigliano 20b9f01f12 Fix ndpi_serialize_string_int64 2022-01-21 17:35:38 +01:00
Alfredo Cardigliano a0bb0ab0ca Add unit test for ndpi_serialize_string_int64 2022-01-21 13:05:58 +01:00
havsah e16b6a18be Fix Grease values parsing (#1416)
The check for grease was too broad and filtered some valid values.
In particular, the value 257 was skipped because it matched the previous check.
This has been discovered while parsing tests/pcap/443-firefox.pcap
expected ja3:
771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,0-23-65281-10-11-35-16-5-51-43-13-45-28-21,29-23-24-25-256-257,0
previously generated ja3:
771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,0-23-65281-10-11-35-16-5-51-43-13-45-28-21,29-23-24-25-256,0

Signed-off-by: Patrick Havelange <patrick.havelange_ext@softathome.com>
2022-01-21 12:40:35 +01:00
Luca Deri 6f336f8307 Added JA3 in risj exceptions 2022-01-20 22:22:56 +01:00
Luca Deri d91477b18d Fixed certificate mismatch check 2022-01-19 23:10:10 +01:00
Luca Deri 771b6606c8 Fixed an issue on CentOS 7 introduced by
https://github.com/ntop/nDPI/commit/c2b7d77784beeae5f86f19e33ba3da5ddce55898
2022-01-19 10:19:58 +01:00
Luca Deri 9eee257be0 Updated confidence type 2022-01-18 22:28:58 +01:00
Ivan Nardi 9f050fa0a6 TLS, H323, examples: fix some memory errors (#1414)
Detected by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=26880
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=26906
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43782
https://oss-fuzz.com/testcase-detail/6334089358082048
2022-01-18 21:52:37 +01:00
Ivan Nardi bd036f96f9 Netbios, CSGO: fix two memory errors (#1413)
Detected by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43754
https://oss-fuzz.com/testcase-detail/5329842395021312
2022-01-18 19:11:01 +01:00
Sam JamesandLuca Deri c2b7d77784 build: respect environment options more (#1392)
* build: update m4/ax_pthread.m4 from serial 23 -> serial 31

Update ax_pthread.m4 to the latest version from the autoconf-archive
project.

Signed-off-by: Sam James <sam@gentoo.org>

* build: properly detect AR, CC, RANLIB

It's necessary to be able to override choice of AR/CC/RANLIB and other toolchain
variables/tools for cross-compilation, testing with other toolchains, and
to ensure the compiler chosen by the user is actually used for the build.

Previously, GNU_PREFIX was kind-of used for this but this isn't a standard
variable (at all) and it wasn't applied consistently anyway.

We now use the standard autoconf mechanisms for finding these tools.

(RANLIB is already covered by LT_INIT.)

Signed-off-by: Sam James <sam@gentoo.org>

* build: use $(MAKE)

This ensures that parallel make works correctly, as otherwise, a fresh
make job will be started without the jobserver fd, and hence
not know about its parent, forcing -j1.

* build: respect CPPFLAGS, LDFLAGS

- CPPFLAGS is for the C preprocessor (usually for setting defines)
- LDFLAGS should be placed before objects for certain flags to work
  (e.g. -Wl,--as-needed)

Signed-off-by: Sam James <sam@gentoo.org>

Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2022-01-18 14:30:14 +01:00
Ivan Nardi e4a7990a4e H323: fix a use-after-poison error (#1412)
Detected by oss-fuzz
See: https://oss-fuzz.com/testcase-detail/6730505580576768

Fix a function prototype
Update a unit test results
2022-01-17 19:47:27 +01:00
Luca Deri d19bd96555 Improved Badoo detection (missing mobile app domain) 2022-01-17 18:54:27 +01:00
Luca Deri 6877e46273 Added Badoo detection 2022-01-17 15:13:14 +01:00
Luca 6679453d86 Adds some risk exceptions for popular services and domain names
via a new (internal) function named ndpi_add_domain_risk_exceptions()
2022-01-17 09:00:25 +01:00
Ivan Nardi 1ff308af09 TLS: fix a use-of-uninitialized-value error (#1411)
Detected by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43705
2022-01-16 20:31:48 +01:00
Ivan Nardi 05734f3d31 Zattoo: fix Null-dereference READ with ipv6 traffic (#1410)
Fix: 20b5f6d7
Detected by oss-fux:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43700
2022-01-16 19:28:39 +01:00
Ivan Nardi 611c753da5 XBox, Diameter: fix dissectors initialization (#1405)
These dissectors have *never* been triggered because their registration
functions use the wrong parameter/bitmask.
Diameter code is buggy since the origianl commit (1d108234), while
XBox code since 5266c726.

Fix some false positives in Xbox code.
2022-01-16 16:19:00 +01:00
Luca Deri f3af39ee42 Added performance tests tools 2022-01-16 12:47:56 +01:00
Luca Deri 42d74171b2 Minor cosmetic changes 2022-01-16 12:47:56 +01:00
Luca Deri dd325248f1 Reduced Patricia tree bucket memory footprint 2022-01-16 12:47:56 +01:00
Ivan Nardi e46b508a36 Kerberos: fix use-of-uninitialized-value error (#1409)
Detected by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43677
2022-01-15 20:53:36 +01:00
Ivan Nardi eb5d7b07af TLS: fix heap-buffer-overflow error (#1408)
Detected by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=43664
2022-01-15 18:47:59 +01:00
Ivan Nardi 63c44a24b6 STUN: fix "confidence" value for some classifications (#1407) 2022-01-15 17:35:06 +01:00
Ivan Nardi 20b5f6d7cc Improve IPv6 support, enabling IPv6 traffic on (almost) all dissectors. (#1406)
Follow-up of 7cba34a1
2022-01-15 17:34:56 +01:00
Ivan Nardi 7f69de0b51 Restore a unit test result (#1403)
Deleted, probably by mistake, in 406ac7e8

Fix Makefile and add compilation of `rrdtool` in CI tests
2022-01-13 23:14:28 +01:00
Luca Deri 533b3ab0e3 FreeBSD fixes 2022-01-13 21:15:21 +01:00
Luca Deri 406ac7e8c8 Added the ability to specify trusted issueDN often used in companies to self-signed certificates
This allows to avoid triggering alerts for trusted albeit private certificate issuers.

Extended the example/protos.txt with the new syntax for specifying trusted issueDN.
Example:

trusted_issuer_dn:"CN=813845657003339838, O=Code42, OU=TEST, ST=MN, C=US"
2022-01-13 19:06:21 +01:00
Vinicius Silva Nogueira dc60cd09c4 fix ahocorasick on big-endian machines (#1401) 2022-01-13 17:35:06 +01:00
Luca Deri d59fefd06e Improved MicrosoftAzure detection 2022-01-12 21:49:31 +01:00
Luca Deri 1e1cfb89d2 Added EthernetIP dissector 2022-01-12 21:48:39 +01:00
Ivan Nardi d4da3b6516 Run fuzzing tests also on push event (like the CI tests) (#1400) 2022-01-12 21:04:01 +01:00
Ivan Nardi f30fbc3212 Fix DGA false positive (#1399)
Close #1397
2022-01-12 20:51:29 +01:00
Ivan Nardi b080a1c136 Fix two use-of-uninitialized-value errors (#1398)
Found by oss-fuzz:
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=40269
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41432

Fix fuzz compilation (follow-up of f5545a80)
2022-01-12 20:24:57 +01:00
Luca Deri 552d199d2e Removed outdated comment 2022-01-11 21:46:30 +01:00
Luca Deri f5545a80f9 Removed legacy code 2022-01-11 21:45:27 +01:00
Ivan Nardi 06f3b07909 QUIC: fix an integer overflow (#1396)
Reported by oss-fuzz:
```
==685288==ERROR: AddressSanitizer: SEGV on unknown address 0x61a100000687 (pc 0x0000005aba64 bp 0x7ffe3f29f510 sp 0x7ffe3f29f400 T0)
==685288==The signal is caused by a READ memory access.
SCARINESS: 20 (wild-addr-read)
 #0 0x5aba64 in quic_len ndpi/src/lib/protocols/quic.c:203:12
 #1 0x5aba64 in decrypt_initial_packet ndpi/src/lib/protocols/quic.c:993:16
 #2 0x5aba64 in get_clear_payload ndpi/src/lib/protocols/quic.c:1302:21
 #3 0x5aba64 in ndpi_search_quic ndpi/src/lib/protocols/quic.c:1658:19
 #4 0x579f00 in check_ndpi_detection_func ndpi/src/lib/ndpi_main.c:4683:6
 #5 0x57abe6 in ndpi_check_flow_func ndpi/src/lib/ndpi_main.c:0
 #6 0x583b2c in ndpi_detection_process_packet ndpi/src/lib/ndpi_main.c:5545:15
 #7 0x55e75e in LLVMFuzzerTestOneInput ndpi/fuzz/fuzz_process_packet.c:30:3
[...]
```
2022-01-11 17:57:34 +01:00
Ivan Nardi 3a087e951d Add a "confidence" field about the reliability of the classification. (#1395)
As a general rule, the higher the confidence value, the higher the
"reliability/precision" of the classification.

In other words, this new field provides an hint about "how" the flow
classification has been obtained.
For example, the application may want to ignore classification "by-port"
(they are not real DPI classifications, after all) or give a second
glance at flows classified via LRU caches (because of false positives).

Setting only one value for the confidence field is a bit tricky: more
work is probably needed in the next future to tweak/fix/improve the logic.
2022-01-11 15:23:39 +01:00
Luca Deri a2916d2e4c Readme for using fuzzer with nDPI 2022-01-11 15:12:45 +01:00
Luca Deri c02574807e Updated result 2022-01-09 18:52:30 +01:00
Luca Deri 708d4ea33a Improved user agent analysis 2022-01-09 18:47:47 +01:00
Ivan Nardi bf69321a29 GTP: fix some false positives (#1394) 2022-01-08 20:40:24 +01:00
Ivan Nardi 4d27001d1d Remove some unused fields (#1393) 2022-01-08 20:40:10 +01:00
Luca Deri eac3ae9ded Improved bittorrent detection 2022-01-06 19:50:31 +01:00
Luca Deri f24f68f59d Invalid check 2022-01-06 07:15:45 +01:00
Luca Deri 8f805a8163 Added missing dependency 2022-01-05 11:11:58 +01:00
Alfredo Cardigliano 948c0367f9 Update issue tempalte 2022-01-03 17:26:23 +01:00
Alfredo Cardigliano 23a4761276 Update copyright 2022-01-03 11:00:45 +01:00
Luca Deri d72a40a0fc Improved BitTorrent guess 2021-12-26 19:41:35 +01:00
Luca Deri c4ac53a03f Added support for Log4J/Log4Shell detection in nDPI via a new flow risk named NDPI_POSSIBLE_EXPLOIT 2021-12-23 21:30:16 +01:00
Ivan Nardi fdb6481cd6 Add support for ICloud Private Relay (#1390)
See:
https://www.apple.com/privacy/docs/iCloud_Private_Relay_Overview_Dec2021.PDF

TODO: an up-to-date list of egress IP ranges is publicly available. Can
we use it somehow?
2021-12-22 19:55:57 +01:00
Ivan Nardi 91bb77a880 A final(?) effort to reduce memory usage per flow (#1389)
Remove some unused fields and re-organize other ones.
In particular:
* Update the parameters of `ndpi_ssl_version2str()` function
* Zattoo, Thunder: these timestamps aren't really used.
* Ftp/mail: these protocols are dissected only over TCP.
* Attention must be paid to TLS.Bittorrent flows to avoid invalid
read/write to `flow->protos.bittorrent.hash` field.

This is the last(?) commit of a long series (see 22241a1d, 227e586e,
730c2360, a8ffcd8b) aiming to reduce library memory consumption.

Before, at nDPI 4.0 (more precisly, at a6b10cf7, because memory stats
were wrong until that commit):
```
nDPI Memory statistics:
	nDPI Memory (once):      221.15 KB
	Flow Memory (per flow):  2.94 KB
```
Now:
```
nDPI Memory statistics:
	nDPI Memory (once):      231.71 KB
	Flow Memory (per flow):  1008 B       <---------
```
i.e. memory usage per flow has been reduced by 66%, dropping below the
psychological threshold of 1 KB.

To further reduce this value, we probably need to look into #1279:
let's fight this battle another day.
2021-12-22 19:54:06 +01:00
Alfredo Cardigliano 02da143e45 Add links to other user guides 2021-12-22 10:23:14 +01:00
Luca Deri 4c02c81bf4 Improved bittorrent heuristic 2021-12-21 23:04:27 +01:00
Ivan Nardi a16b4aa564 Fix a global-buffer-overflow when loading Azure list (#1388) 2021-12-19 22:29:25 +01:00
Luca Deri 7aad97fffb Added Microsoft Azure support 2021-12-19 18:59:45 +01:00
Ivan Nardi 7153b8933c Improve/add several protocols (#1383)
Improve Microsoft, GMail, Likee, Whatsapp, DisneyPlus and Tiktok
detection.
Add Vimeo, Fuze, Alibaba and Firebase Crashlytics detection.

Try to differentiate between Messenger/Signal standard flows (i.e chat)
and their VOIP (video)calls (like we already do for Whatsapp and
Snapchat).

Add a partial list of some ADS/Tracking stuff.

Fix Cassandra, Radius and GTP false positives.
Fix DNS, Syslog and SIP false negatives.

Improve GTP (sub)classification: differentiate among GTP-U, GTP_C and
GTP_PRIME.

Fix 3 LGTM warnings.
2021-12-18 13:24:51 +01:00
Ivan Nardi 58b33dcb2d TLS: add support for IPV6 in Subject Alt Names field (#1385) 2021-12-18 12:03:46 +01:00
Ivan Nardi ec6eea0c23 TLS: fix usage of certificate cache (#1384) 2021-12-18 12:03:32 +01:00
Luca Deri 74519f6e10 Improved bittorrent detection 2021-12-17 21:17:18 +01:00
Ivan Nardi f11b215cfb QUIC: fix old versions of GQUIC on big-endian machines (#1387)
Credits to @viniciussn (see #1312)
2021-12-17 12:10:34 +01:00
Luca Deri 8e512ff503 Improved BitTorrent classification 2021-12-07 21:49:00 +01:00
Luca Deri 519d70fa04 Optimized ndpi_serialize_string_int32 2021-12-07 15:54:23 +01:00
Luca Deri 85b396ca43 Fixed issue that prevented alt certificate names to be fully detected when ipAddress and rfc822Name were specified in certificates 2021-12-07 14:45:42 +01:00
Luca Deri cefcc25b13 Added missing install target 2021-12-07 09:00:58 +01:00
Ivan Nardi 947896ad7d Fix configure script (after fb85dac9) (#1381)
Fix/disable some LGTM warnings
2021-12-04 20:22:05 +01:00
Ivan Nardi 7cba34a1e3 Improve IPv6 support, enabling IPv6 traffic on (almost) all dissectors. (#1380)
There are no valid reasons for a (generic) protocol to ignore IPv6
traffic.
Note that:
* I have not found the specifications of "CheckPoint High Availability
Protocol", so I don't know how/if it supports IPv6
* all LRU caches are still IPv4 only

Even if src_id/dst_id stuff is probably useless (see #1279), the right
way to update the protocol classification is via `ndpi_set_detected_protocol()`
2021-12-04 13:30:10 +01:00
Ivan Nardi 6e86e6d924 QUIC: add support for QUICv2 (draft 00) (#1379)
It is already time to start looking at the new QUIC version.
See: https://datatracker.ietf.org/doc/html/draft-ietf-quic-v2-00
2021-12-04 13:29:30 +01:00
Toni 4d60c70c8b Make serialize risk and proto not dependant on any flow. (#1377)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-12-04 13:28:55 +01:00
Luca Deri fb85dac905 Configure improvements
Fixed invalid anomaly detection in rrd_anomaly
2021-12-04 13:21:57 +01:00
Luca Deri fe2822c6a8 Added example for finding similarities in RRDs using nDPI statistical APIs 2021-12-04 10:09:01 +01:00
Luca Deri 6ab1367846 Added quick mode (-q) 2021-12-03 21:18:03 +01:00
Luca Deri 8c4ae7527b Updated rrdtool makefile 2021-12-03 21:10:43 +01:00
Luca Deri 1b077ba65f Demo tool for finding anomalies in RRD files 2021-12-03 17:36:49 +01:00
Alfredo Cardigliano 3dd9b5176e Remove printf when not required 2021-12-01 10:24:12 +01:00
Alfredo Cardigliano 1d3428d538 Improve JSON serialization 2021-12-01 10:04:17 +01:00
Luca Deri 33557c5fbc Improved CSV serialization 2021-11-30 22:15:26 +01:00
Luca Deri 5e0befd56c Fixed seriealizer unit 2021-11-30 17:24:42 +01:00
Luca Deri 4f81cdc103 Removed spaces from some protocol names as they create problems with some applications 2021-11-30 10:12:24 +01:00
Ivan Nardi b1e9245d94 ndpiReader: slight simplificaton of the output (#1378) 2021-11-27 17:32:23 +01:00
Luca Deri 320bcfafa2 Added ndpi_serializer_skip_header() serialization API 2021-11-26 22:18:59 +01:00
Luca Deri a7b5e09195 Added Salesforce detection 2021-11-26 19:07:45 +01:00
Luca Deri 9d59b8b1a9 HTTP proxy improvement 2021-11-25 23:17:27 +01:00
Luca Deri ea435c46f5 Reworked HTTP protocol dissection including HTTP proxy and HTTP connect 2021-11-25 22:53:46 +01:00
Luca Deri 772957349b Aded default port to HTTP_CONNECT 2021-11-25 21:46:00 +01:00
Luca Deri 40ff2c54c1 Added googletagmanager.com to the host_match list 2021-11-25 21:45:38 +01:00
Ivan Nardi a8ffcd8bb0 Rework how hostname/SNI info is saved (#1330)
Looking at `struct ndpi_flow_struct` the two bigger fields are
`host_server_name[240]` (mainly for HTTP hostnames and DNS domains) and
`protos.tls_quic.client_requested_server_name[256]`
(for TLS/QUIC SNIs).

This commit aims to reduce `struct ndpi_flow_struct` size, according to
two simple observations:
 1) maximum one of these two fields is used for each flow. So it seems safe
to merge them;
 2) even if hostnames/SNIs might be very long, in practice they are rarely
longer than a fews tens of bytes. So, using a (single) large buffer is a
waste of memory for all kinds of flows. If we need to truncate the name,
we keep the *last* characters, easing domain matching.

Analyzing some real traffic, it seems safe to assume that the vast
majority of hostnames/SNIs is shorter than 80 bytes.

Hostnames/SNIs are always converted to lowercase.

Attention was given so as to be sure that unit-tests outputs are not
affected by this change.

Because of a bug, TLS/QUIC SNI were always truncated to 64 bytes (the
*first* 64 ones): as a consequence, there were some "Suspicious DGA
domain name" and "TLS Certificate Mismatch" false positives.
2021-11-24 10:46:48 +01:00
Ivan Nardi fd02e1b304 QUIC: fix extra dissection (#1376)
When we have fully reassembled the Client Hello, we need to stop extra
dissection.
2021-11-23 10:57:13 +01:00
Luca Deri 4c39ed293e BitTorrent dissector code cleanup and merge of https://github.com/ntop/nDPI/pull/1374 2021-11-17 23:18:22 +01:00
Luca Deri 2e2d4c911a BitTorrent detection improvements 2021-11-17 22:41:35 +01:00
Luca Deri 2a6e5c6341 Improved BT hashing to minimize false positives 2021-11-17 10:47:21 +01:00
Luca Deri a6545ea095 bittorrent old code cleanup. Enlarged BT cache 2021-11-16 10:05:18 +01:00
Luca Deri f2a279e441 Improved BitTorrent detection 2021-11-15 17:25:13 +01:00
Ivan Nardi afc2b641eb Fix writes to flow->protos union fields (#1354)
We can write to `flow->protos` only after a proper classification.

This issue has been found in Kerberos, DHCP, HTTP, STUN, IMO, FTP,
SMTP, IMAP and POP code.
There are two kinds of fixes:
 * write to `flow->protos` only if a final protocol has been detected
 * move protocol state out of `flow->protos`
The hard part is to find, for each protocol, the right tradeoff between
memory usage and code complexity.

Handle Kerberos like DNS: if we find a request, we set the protocol
and an extra callback to further parsing the reply.

For all the other protocols, move the state out of `flow->protos`. This
is an issue only for the FTP/MAIL stuff.

Add DHCP Class Identification value to the output of ndpiReader and to
the Jason serialization.

Extend code coverage of fuzz tests.

Close #1343
Close #1342
2021-11-15 16:20:57 +01:00
ragostino da47357762 Update ndpi_main.c for HTTP line parser optimization and cleaning (#1372)
* Update ndpi_main.c

HTTP line parser: optimization and cleaning

* Update ndpi_main.c
2021-11-11 12:38:31 +01:00
Ivan Nardi acb1de69aa Reduce memory used by ndpiReader (#1371)
`ndpiReader` is only an example, aiming to show nDPI capabilities
and integration, without any claim about performances.

Nonetheless its memory usage per flow is *huge*, limiting the kinds
of traces that we can test on a "normal" hardware (example: scan
attacks).

The key reason of that behaviour is that we preallocate all the memory
needed for *all* the available features.

Try to reduce memory usage simply allocating some structures only
when they are really needed. Most significant example: JOY algorithms.

This way we should use a lot less memory in the two most common
user-cases:
 * `ndpiReader` invoked without any particular flag (i.e `ndpiReader -i
$FILENAME_OR_IFACE`)
 * internal unit tests

Before (on x86_64):
```
struct ndpi_flow_info {
[...]
	/* size: 7320, cachelines: 115, members: 72 */
```
After:
```
struct ndpi_flow_info {
[...]
	/* size: 2128, cachelines: 34, members: 75 */
```
2021-11-11 12:37:25 +01:00
Ivan Nardi 3e5491fa10 Add detection of OCSP (#1370)
This protocol is detected via HTTP Content-Type header.

Until 89d548f9, nDPI had a dedicated automa (`content_automa`) to
classify a HTTP flow according to this header. Since then, this automa has
been useless because it is always empty.
Re-enable it to match only a string seems overkilling.

Remove all `content_automa` leftovers.
2021-11-11 12:36:55 +01:00
Ivan Nardi 0f168d9150 IMAP, POP3, SMTP: improve dissection (#1368)
Avoid NATS false positives
2021-11-11 11:55:56 +01:00
Luca Deri 9abf528c8e BitTorrent cache debug 2021-11-08 12:42:15 +01:00
Ivan Nardi 5464bad6db Differentiate between standard Amazon stuff (i.e market) and AWS (#1369) 2021-11-04 00:20:45 +01:00
Luca Deri 4173775b60 Improved BitTorrent detection 2021-11-04 00:20:26 +01:00
Ivan Nardi a8a3c6f7c2 Fix use-of-uninitialized-value (#1366)
Close #1346
2021-11-03 15:40:07 +01:00
Ivan Nardi 55880e4ae4 TLS: fix two warnings (#1365)
Disable unit tests on CI for big-endian target. We know we have multiple
issues on big-endian architectures (see #1312) and so the unit tests
always fail there. Ignore this error for the time being and let the CI
pass if we don't have other issues.

Remove an unused automa definition
2021-11-02 22:08:15 +01:00
Luca Deri 1f6c94e663 Updated test results 2021-11-02 12:08:20 +01:00
Luca Deri b11065b2e3 Fixed certificate mismatch for TLS flows with no client hello observed 2021-11-02 12:06:18 +01:00
Luca Deri 9f8c1e8630 BitTorrent detection improvements 2021-10-31 18:45:13 +01:00
Luca Deri a9aa46287d Little change in TLS debug traces 2021-10-28 10:16:33 +02:00
Luca Deri a25d9d871e Updated results 2021-10-27 23:04:29 +02:00
Luca Deri 5283755afa Fixed TLS certificate dissection of subjectAltName 2021-10-27 22:51:59 +02:00
Luca Deri aaa474a37d Minor code cleanup 2021-10-27 09:32:11 +02:00
Ivan Nardi 6edb7bedd7 Avoid overwriting valid protocol in ndpi_detection_giveup (#1360)
We should avoid updating any valid protocol in `ndpi_detection_giveup`; we
should try to find a proper classification only if the flow is still
completely unclassified.

For example in the attached pcap there is a valid TLS session, recognized
as such by TLS dissector. However, the `ndpi_detection_giveup`function
updates it to "HTTP/TLS" (!?) simply because the server port is 80.

Note that the real issue is not the wrong classification, but the
wrong access to `flow->protos` union. If we already set some fields of
`flow->protos` and we change the protocol in `ndpi_detection_giveup`, we
might end up freeing some invalid pointers in `ndpi_free_flow_data`
(no wonder this issue has been found while fuzzing #1354)

Fix GIT and TLS dissectors (issues found by CI fuzzer)
2021-10-27 09:23:07 +02:00
Toni c4eebaaf5e Serialize additional information stored in the flow struct. (#1362)
* Changed function signature of ndpi_flow2json (removed unused vlan_id; API break)
 * Serialize NTP information.
 * Improved QUIC serialization.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-27 09:22:19 +02:00
Luca Deri f41d8ac106 Improved STUN and RTP detection 2021-10-27 00:06:05 +02:00
Toni 41765efcf8 Detect invalid characters in text and set a risk. Fixes #1347. (#1363)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-26 21:34:01 +02:00
Luca Deri 5ccc61d1cb Fixed Git protocol dissection (#1355) 2021-10-25 18:52:27 +02:00
Ivan Nardi b6d9536533 Fixed cleartext protocol assignment (#1357) 2021-10-25 15:04:04 +02:00
Ivan Nardi 333a6d60e8 TLS: fix a heap-buffer-overflow (#1356)
Revert of c3d1c697
Error reproducible with the attached pcap and valgrind
2021-10-22 14:57:49 +02:00
Luca Deri c3d1c69722 Fixed heap-buffer-overflow in TLS dissector 2021-10-20 22:07:47 +02:00
Ivan Nardi 8db8c8ae34 Fix QUIC log and remove SoulSeek leftovers after b97dc6ba (#1351)
Update .gitignore file
Fix a function prototype
Close #1349
2021-10-19 21:42:55 +02:00
Ivan Nardi fea3991ad1 Fix some invalid memory reads (#1350)
`ndpi_detection_giveup()` (and any functions called by it) can't access
`ndpi_detection_module_struct->packet` anymore since 730c236.

Sync unit tests results

Close #1348
2021-10-19 12:07:47 +02:00
Luca Deri 06f13fd2de Added clertext alert with telnet 2021-10-19 08:44:29 +02:00
Toni ed51987e3a Fix broken fuzz_process_packet fuzzer by adding a call to ndpi_finalize_initialization(). (#1334)
* fixed several memory errors (heap-overflow, unitialized memory, etc)
 * ability to build fuzz_process_packet with a main()
   allowing to replay crash data generated with fuzz_process_packet
   by LLVMs libfuzzer
 * temporarily disable fuzzing if `tests/do.sh`
   executed with env FUZZY_TESTING_ENABLED=1

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-18 23:16:32 +02:00
Luca Deri 7d3c3b23f8 Implemented RDP over UDP dissection 2021-10-18 22:12:28 +02:00
Luca Deri 9e97d20c25 Refreshed results list 2021-10-16 12:03:16 +02:00
Luca Deri b08b6a6a28 Fixed cleartext protocol assignment 2021-10-16 12:02:04 +02:00
Luca Deri c2e7fbce94 Updated test results after latest commit 2021-10-16 11:57:32 +02:00
Luca Deri 1b3c41a752 Reworked flow risks asignment
Added esceptions for windows update and binary application transfer risk
2021-10-16 11:55:08 +02:00
Luca Deri b97dc6baa4 Removed outdated (and broken) soulseek dissector 2021-10-15 17:27:27 +02:00
Luca Deri e696f31487 Updated test results 2021-10-14 22:18:34 +02:00
Luca Deri b1a43cbcff Added missing call to ndpi_reconcile_protocols() with protocol guess 2021-10-14 22:08:20 +02:00
Luca Deri 6e649e0cb6 Fixed issue on DGA numeric IP detection 2021-10-13 19:21:27 +02:00
Luca Deri 09668496c3 Indemt 2021-10-13 19:18:05 +02:00
Zied Aouini a2e9730b23 Add issue templates. (#1340)
* Add issue template.
* Add config log instruction.
2021-10-13 18:39:10 +02:00
Luca Deri 937357e4bc Implemented ndpi_ses_fitting() and ndpi_des_fitting()
for comuting the best alpha/beta values for exponential smoothing
2021-10-12 13:08:58 +02:00
Alfredo Cardigliano d947437233 Return 0 on ndpi_init_serializer_ll success for consistency 2021-10-12 10:47:44 +00:00
Luca Deri d13bc0e01a Fixed DES initialisation 2021-10-12 12:38:40 +02:00
Ivan Nardi 550e6fe6fc QUIC: fix an integer overflow (#1337)
Long standing bug: credits to @lnslbrty for digging into it and to
@aouinizied for the CI improvements
2021-10-11 23:08:10 +02:00
Toni a8f938f7ac Removed README.protocols because: (#1333)
* Tor via TLS should be detectable via DGA as a risk
 * protocol limitations should be part of the official documentation in `doc/`

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-11 23:07:36 +02:00
Luca Deri 960e8ab22a Win fix 2021-10-11 20:59:28 +02:00
Zied Aouini ca827ae548 Fix FuzzCI. (#1338) 2021-10-11 20:17:12 +02:00
Zied Aouini 7e0724d576 Enable sanitizers on CIFuzz. (#1336) 2021-10-11 19:01:50 +02:00
Luca Deri 246d115d7b Cleaned up code moving specific includes in files their are using it. Thi prevents ndpi_config.h to be included everywhere in apps using nDPI that might leade to #define redefinitions after the latest changes 2021-10-11 11:45:45 +02:00
Toni 60eca79fdf Additional fix related to cf931fda6b. (#1332)
* configure.seed references removed

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-10 14:14:02 +02:00
Toni 549134ef10 Fix make dist and add it to the CI along with make -C doc text. Fixes #1324 (#1327)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-10 13:15:38 +02:00
Toni cf931fda6b Get rid of configure.seed as it comes with some disadvantages. (#1328)
* using Autotools best-practices to achieve (hopefully) the same result

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-10 13:13:52 +02:00
Toni 6ff67b5043 Add more WindowsUpdate URLs as requested in #698. (#1329)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-08 18:29:49 +02:00
Luca Deri 3ffb65a81c Added extra check 2021-10-07 08:21:40 +02:00
Ivan Nardi 2dfc478ad4 Fix compilation with clang-13 or if some debug macros are enabled (#1326) 2021-10-06 18:32:15 +02:00
Luca Deri 944e8ed105 Added check 2021-10-06 18:28:09 +02:00
Luca Deri 2b4821a97b Compilation fix for old Linux distributions 2021-10-06 09:28:04 +02:00
Luca Deri d9c72999ee Test update 2021-10-06 00:06:34 +02:00
Luca Deri 181a03c5ad Compilation fix 2021-10-05 17:17:12 +02:00
Luca Deri 8957c52a48 Numeric IPs are not considered for DGA checks 2021-10-05 17:14:31 +02:00
Luca Deri 408d78e628 Improved DGA detection for skipping potential DGAs of known/popular domain names 2021-10-05 16:51:24 +02:00
Luca Deri 10576eab0a TLS obsolete protocol is set when TLS < 1.2 (used to be 1.1) 2021-10-05 16:44:56 +02:00
Ivan Nardi 730c2360bd Remove struct ndpi_packet_struct from struct ndpi_flow_struct (#1319)
There are no real reasons to embed `struct ndpi_packet_struct` (i.e. "packet")
in `struct ndpi_flow_struct` (i.e. "flow"). In other words, we can avoid
saving dissection information of "current packet" into the "flow" state,
i.e. in the flow management table.

The nDPI detection module processes only one packet at the time, so it is
safe to save packet dissection information in `struct ndpi_detection_module_struct`,
reusing always the same "packet" instance and saving a huge amount of memory.
Bottom line: we need only one copy of "packet" (for detection module),
not one for each "flow".

It is not clear how/why "packet" ended up in "flow" in the first place.
It has been there since the beginning of the GIT history, but in the original
OpenDPI code `struct ipoque_packet_struct` was embedded in
`struct ipoque_detection_module_struct`, i.e. there was the same exact
situation this commit wants to achieve.

Most of the changes in this PR are some boilerplate to update something
like "flow->packet" into something like "module->packet" throughout the code.
Some attention has been paid to update `ndpi_init_packet()` since we need
to reset some "packet" fields before starting to process another packet.

There has been one important change, though, in ndpi_detection_giveup().
Nothing changed for the applications/users, but this function can't access
"packet" anymore.
The reason is that this function can be called "asynchronously" with respect
to the data processing, i.e in context where there is no valid notion of
"current packet"; for example ndpiReader calls it after having processed all
the traffic, iterating the entire session table.

Mining LRU stuff seems a bit odd (even before this patch): probably we need
to rethink it, as a follow-up.
2021-10-05 15:49:52 +02:00
Ivan Nardi f3fcf1e7c0 Update unit tests results after da8eed5a (#1323) 2021-10-05 15:06:34 +02:00
Luca Deri da8eed5a32 Updated description 2021-10-05 11:08:16 +02:00
Ivan Nardi c1e794366f WHOIS: enhance detection, avoiding false positives (#1320)
We are interested only in the domain name required, not in the long reply.
2021-10-05 09:35:04 +02:00
Luca Deri bb7aff6526 Added -a <num> to ndpiReader for generating OPNsense configuration
See https://github.com/ntop/opnsense
2021-10-04 22:34:49 +02:00
Luca Deri fd0e65cb57 Removed trace 2021-10-03 23:41:31 +02:00
Ivan Nardi bc5be1e76c Fix how some protocols handle tcp retransmissions (#1321)
Most (all?) protocols don't care about (tcp) retransmissions.

If a protocol registers itself with a
NDPI_SELECTION_BITMASK_PROTOCOL_*_WITHOUT_RETRANSMISSION value, its
callback is never triggered with a retransmitted packet.
2021-10-03 21:38:49 +02:00
Toni e84da386c7 Added nDPI pkg-config file to Debian / Ubuntu ndpi-dev packaging. (#1318)
* Added nDPI pkg-config file to Debian / Ubuntu ndpi-dev packaging.

 * fixed missing gcrypt library dependency in libndpi.pc

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-10-03 21:29:56 +02:00
Simone Mainardi 06f763e6c3 Adds sections labels with risk id to the docs 2021-10-01 15:02:14 +02:00
Luca Deri a9d4f5d8d0 Warning fix 2021-10-01 11:36:35 +02:00
Luca Deri 8e6de00ce3 Initial attempt to write nDPI documentation. Starting with flow risks. Please contribute 2021-10-01 11:32:27 +02:00
Ivan Nardi 5cf3fef8f7 Remove detected_protocol_stack field from ndpi_packet_struct (#1317)
This field is an exact copy of `ndpi_flow_struct->detected_protocol_stack[2]`:
* at the very beginning of packet dissection, the value saved in
`flow->detected_protocol_stack` is copied in `packet->detected_protocol_stack`
(via `ndpi_detection_process_packet()` -> `ndpi_init_packet_header()`)

* every time we update `flow->detected_protocol_stack` we update
`packet->detected_protocol_stack` too (via `ndpi_int_change_protocol()`
-> `ndpi_int_change_packet_protocol()`)

These two fields are always in sync: keeping the same value in two
different places is useless.
2021-09-29 13:11:32 +02:00
Alfredo Cardigliano 721031210b Fix warning 2021-09-28 09:44:31 +02:00
Luca Deri af35d6b963 Added unit test for bitmap iteration 2021-09-27 12:47:05 +02:00
Luca Deri b9ccff0761 Compilation fixed on CentOS 7
Bitmap APi changes
2021-09-27 12:41:40 +02:00
Luca Deri 241ee32cbc Bitmap code 2021-09-27 12:06:46 +02:00
Luca Deri 519c69c5cc Reworked bitmap code 2021-09-27 10:29:03 +02:00
Luca Deri 72df138a7d Warnign fix 2021-09-27 09:08:04 +02:00
eac7024489 Fix armhf (#1315)
* Fix unaligned memory accesses with get_u_int64_t at armhf

see: https://bugs.debian.org/993627

* Use get_u_int64_t to avoid unaligned memory access at armhf

see: https://bugs.debian.org/993627

* Update src/include/ndpi_define.h.in

Drop const type from get_u_int64_t, from lnslbrty

Co-authored-by: Bernhard Übelacker <bernhardu@mailbox.org>
Co-authored-by: Toni <matzeton@googlemail.com>
2021-09-26 23:53:00 +02:00
Luca Deri 5ed1930a93 Added include for defining bools (not present on all platforms) 2021-09-26 23:00:13 +02:00
Luca Deri 1efabef4cf Added API for handling compressed bitmaps
ndpi_bitmap* ndpi_bitmap_alloc();
void ndpi_bitmap_free(ndpi_bitmap* b);
u_int64_t ndpi_bitmap_cardinality(ndpi_bitmap* b);
void ndpi_bitmap_set(ndpi_bitmap* b, u_int32_t value);
void ndpi_bitmap_unset(ndpi_bitmap* b, u_int32_t value);
bool ndpi_bitmap_isset(ndpi_bitmap* b, u_int32_t value);
void ndpi_bitmap_clear(ndpi_bitmap* b);
size_t ndpi_bitmap_serialize(ndpi_bitmap* b, char **buf);
ndpi_bitmap* ndpi_bitmap_deserialize(char *buf);

based on https://github.com/RoaringBitmap/CRoaring
2021-09-26 22:55:15 +02:00
Zied Aouini c40cf4e0ec Add arm64, armhf and s390x as part of CI. (#1314)
* Add arm64, armhf and s390x as part on CI.

* Minor fix.

* Fix code inspector complaints.
2021-09-23 14:41:56 +02:00
Ivan Nardi 7eb73bc92a QUIC: fix old GQUIC versions on big-endian machines (#1313) 2021-09-23 13:24:49 +02:00
Zied Aouini 0994771974 Improve CI (#1303)
* Improve CI pipeline

* Fix branch name.

* Fix branch name.

* Fix libgcrypt configuration.

* Update build.yml

* Move to Github Actions instead of Travis CI.

* Fix mingw on ubuntu bionic.

* Reactivate cross compile on Ubuntu Bionic.

* Switch to single line steps.

* Add several compilers versions

* Minor fix.

* Fix build all and delete cxx

* Fix RCE detection.

* Fix PCRE configuration.

* Add condition on PCRE test pcap.

* Update WebattackRCE.pcap.out

* Add missing SUBST.

* Delete WebattackRCE.pcap.out

* Update WebAttackRCE result.

* Fix typo.

* Extend jobs with pcre+msan+maxminddb.

* Fix code inpector warnings.

* Delete .appveyor.yml
2021-09-22 12:10:52 +02:00
Toni 9717afb2da Merge pull request #1311 from IvanNardi/ftp-start-tls
FTP: fix support for START-TLS sessions
2021-09-22 12:08:04 +02:00
Nardi Ivan 017c1a4239 FTP: fix support for START-TLS sessions
When TLS-over-FTP is used, the credentials are encrypted. So we must not
wait for the username and the password commands, otherwise we elaborate a
lot of packets for nothing.
2021-09-21 20:53:04 +02:00
Toni bb52f1362c Merge pull request #1309 from IvanNardi/ipv6-headers
Fix parsing of ipv6 packets with extension headers
2021-09-20 12:54:01 +02:00
Toni d74c2739d5 Merge pull request #1310 from IvanNardi/stun-realm
STUN: fix extraction of Realm attribute
2021-09-20 11:53:40 +02:00
Nardi Ivan 95ac8fd41a STUN: fix extraction of Realm attribute
While at it, improve detection of Facebook Messenger
2021-09-20 11:04:54 +02:00
Nardi Ivan 03d3e1bafc Fix parsing of ipv6 packets with extension headers
Decoding of ipv6 traffic with extension headers was completely broken,
since the beginning of the L4 header was always set to a wrong value.

Handle the ipv6 fragments in the same way as the ipv4 ones: keep the first
one and drop the others.
2021-09-19 17:29:22 +02:00
Ivan Nardi 994bd0696b Small fixes after latest commits (#1308) 2021-09-18 09:31:00 +02:00
Ivan Nardi 227e586e63 A (tiny) effort to reduce memory usage per flow (#1304)
Follow-up of 22241a1d
Only trivial changes:
* remove completely unused fields
* remove fields only written (but never read)
* CSGO protocol only handles UDP traffic
2021-09-18 09:29:45 +02:00
Luca Deri 4539e96ce2 Added DNS fragmented test pcap 2021-09-17 15:56:18 +02:00
Luca Deri 3303dac7cf Improved fragmented DNS detection 2021-09-17 15:17:39 +02:00
Luca Deri 119a14c6e3 DNS dissection fixes 2021-09-17 12:20:19 +02:00
deboracerretiniandLuca Deri 8f113c1d37 Progetto esame Gestione di Reti - Debora Cerretini (#1290)
* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

* Add files via upload

Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2021-09-17 11:01:50 +02:00
Ivan Nardi 6325aebda6 TLS: avoid zeroing large structures (#1300)
Zeroing large structures (i.e. size > KB) is quite costly (from a CPU point
of view): we can safely avoid doing that for a couple of big structures.

Standard and Valgrind tests have been diverging quite a lot: it is time
to re-sync them. Use the same script and enable Valgrind via an
enviroment variable:

  NDPI_TESTS_VALGRIND=1 ./tests/do.sh
2021-09-16 14:34:59 +02:00
Simone Mainardi 978c9cfda3 Adds instructions to update flow risks in ntopng 2021-09-15 17:49:47 +02:00
Ivan Nardi 0eb7a0388c Hangout: fix usage of STUN function (#1299) 2021-09-13 18:25:43 +02:00
Ivan Nardi 3e06e6f5b8 Fix some ndpi_malloc/ndpi_free pairs (#1298) 2021-09-13 18:25:22 +02:00
Ivan Nardi 31404f4865 Update the list of default ports for QUIC protocol (#1297)
There are no reasons to register UDP/80 as a default port for QUIC
2021-09-13 18:25:04 +02:00
Ivan Nardi 17774bf09c Bittorrent: fix a potential use-of-uninitialized-value error (#1296)
Not sure if this is the right fix (from a logical point-of-view): this
code hasn't changed since OpenDPI era (!) and I have't found a trace
triggering this code path.
Anyway, the use-of-uninitialized-value error itself should be fixed.
2021-09-13 18:24:42 +02:00
Vitaly Lavrov 22241a1d36 Removing unused fields from the id_struct structure. (#1293)
Reduced by 40 bytes.
2021-09-13 18:19:09 +02:00
Ivan Nardi 951bcb0c39 Cassandra: fix compilation when "--enable-debug-messages" option is used (#1294)
Let's try adding a dedicated compilation in GitHub Actions to easily
detect this kind of errors in the future
2021-09-11 11:10:46 +02:00
Ivan Nardi 083e464929 Fix unit test results after 00857abf (#1295) 2021-09-11 11:09:55 +02:00
Luca Deri 00857abf2c Added new risk for clear text credentials 2021-09-10 22:00:04 +02:00
lucasbaileandLucas Santos 1fadf4754a Add Cassandra protocol dissector (#1285)
Co-authored-by: Lucas Santos <lucas.santos@zerum.com>
2021-09-09 22:47:58 +02:00
Luca Deri 36f08b8d2e Win fix 2021-09-07 22:37:22 +02:00
Alfredo Cardigliano 57fce4e350 Fix unused var 2021-09-03 17:46:04 +02:00
Alfredo Cardigliano 89af9b7e3c Print API version 2021-09-01 09:48:37 +02:00
Luca Deri a6b10cf73f Fixed memory stats 2021-08-26 12:12:22 +02:00
Luca Deri 90c6fd99a7 ARM fixes 2021-08-25 09:46:29 +02:00
Luca Deri 88040f0449 Compilation fix 2021-08-20 22:47:46 +02:00
Ivan Nardi 9e0f0ce3df Fix access to some TLS fields in flow structure (#1277)
Fields 'tls.hello_processed` and `tls.subprotocol_detected` are used by
QUIC (i.e UDP...), too.
2021-08-20 18:11:37 +02:00
Ivan Nardi 8fdffbf3a1 Compile everything with "-W -Wall -Wno-unused-parameter" flags (#1276)
Fix all the warnings.

Getting rid of "-Wno-unused-parameter" is quite complex because some
parameters usage depends on compilation variable (i.e.
`--enable-debug-messages`).

The "-Werror" flag has been added only in Travis builds to avoid
breaking the builds to users using uncommon/untested
OS/compiler/enviroment.

Tested on:
* x86_64; Ubuntu 20.04; gcc 7,8,9,10,11; clang 7,8,9,10,11,12
* x86_64; CentOS 7.7; gcc 4.8.5 (with "--disable-gcrypt" flag)
* Raspberry 4; Debian 10.10; gcc 8.3.0
2021-08-20 18:11:13 +02:00
Ivan Nardi 55eec29c08 wireshark/lua: restore full flow risks dissection (#1275)
It was partially disabled in 3eba8cc5.
Wireshark and Wireshark-Lua bindings don't handle 64 bit integer very well.
(see https://www.wireshark.org/docs/wsdg_html_chunked/lua_module_Int64.html).

As workaround, only for visualization purpose, split the (64 bit) risk mask
into two 32 bit integer values.
2021-08-18 11:34:46 +02:00
Toni 8d0c7b1fae Fixed Mingw64 build, SonerCloud-CI and more. (#1273)
* Added ARM build and unit test run for SonarCloud-CI.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Fixed Mingw64 build.

 * adapted to SonarCloud-CI workflow
 * removed broken and incomplete Windows example (tested on VS2017/VS2019)
 * removed unnecessary include (e.g. pthread.h for the library which does not make use of it)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-08-18 11:34:16 +02:00
Luca Deri 677b513f70 Fixes a crash on ARM (Raspberry Pi 4 Model B Rev 1.1) 2021-08-18 11:12:06 +02:00
Luca Deri 821f4c9249 Fixed some invalid TLS guesses 2021-08-17 19:04:07 +02:00
Luca Deri a2ea67152c Added check to avoid clases on similar protocols (FTP adn SMTP) on setting hostname 2021-08-11 12:11:58 +02:00
Luca Deri 5c33fbf19b Added extraction of hostname in SMTP
Fixed mail incalid subprotocol calculation
2021-08-11 11:52:24 +02:00
Alfredo Cardigliano 846b546dbc Fix typo 2021-08-10 11:43:32 +02:00
Luca Deri c1acb6d801 Typo 2021-08-09 18:36:36 +02:00
Luca Deri 0fbe895103 Addef ETH_ARP definition 2021-08-09 18:33:33 +02:00
Luca Deri 05492ec8e8 Defined more standard ethernet protocol types 2021-08-09 17:01:10 +02:00
Luca Deri 58ca7b5a7f Added protocol defines 2021-08-09 16:45:03 +02:00
Luca Deri 0df8aa19ec Clode cleanup (after last merge) 2021-08-08 17:10:08 +02:00
ToniandLuca Deri 259b60f574 Added entropy calculation to check for suspicious (encrypted) payload. (#1270)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2021-08-08 17:03:23 +02:00
Toni 4e856a41d6 Skip whitespaces between HTTP method and URL. (#1271)
* be less case-restrictive, RFC2616 wants it that way

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-08-08 17:00:10 +02:00
Toni 8e996f7f7c Fixed broken SonarCloud-CI. (#1272)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-08-08 16:59:36 +02:00
Luca Deri 3706267d68 Updated test output 2021-08-07 19:43:55 +02:00
Luca Deri 0522e562e8 Added testing pcap for TLS fatal alert 2021-08-07 19:42:53 +02:00
Luca Deri 4183718952 Added TLS fatal alert flow risk 2021-08-07 19:40:44 +02:00
Luca Deri e8455236bd Updated output 2021-08-07 17:38:33 +02:00
Luca Deri a13f1fe52f Report whether a protocol is encrypted 2021-08-07 17:35:34 +02:00
Luca Deri 46ebd7128f Added ability to report whether a protocol is encrypted
Added new API calls
- u_int8_t ndpi_is_valid_protoId(u_int16_t protoId);
Cleaned up code and organized in a function.

- u_int8_t ndpi_is_encrypted_proto(struct ndpi_detection_module_struct *ndpi_str, ndpi_protocol proto);
Check whether a protocol is encrypted or not based on master and app protocols
2021-08-07 17:10:18 +02:00
Luca Deri dc841574af Win fix 2021-08-03 09:44:50 +02:00
Luca Deri 6b7e5fa8d2 Added legacy whatsapp rules used to match old pcaps that need to be removed in the future 2021-07-31 23:49:48 +02:00
Toni ad57af9f79 Improved RTSP detection and fixed HTTP false-positive. Fixes #1229. (#1266)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-07-31 23:31:49 +02:00
Fábio Depin ce597b4806 Updated WhatsAPP and Instagram addresses (#1264)
* Updated WhatsAPP addresses

* Added Instagram addresses
2021-07-31 23:30:34 +02:00
Toni 13c5d6801e Improved libm check via Autoconf. (#1263)
* libm will now be part of @ADDITIONAL_LIBS@ (if required)

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-07-28 22:24:03 +02:00
Luca Deri cb26a8c82d Moved to 4.1.0 2021-07-26 09:40:50 +02:00
568 changed files with 80294 additions and 15575 deletions
-17
View File
@@ -1,17 +0,0 @@
mversion: 3.1.{build}
branches:
only:
- dev
environment:
matrix:
- COMPILER: mingw
MINGW_DIR: c:\msys64\mingw32
before_build:
- set Path=%MINGW_DIR%\bin;%Path%
build_script:
- set Path=%MINGW_DIR%\bin;c:\msys64\usr\bin;
- bash autogen.sh
- cd src/lib
- make
+3
View File
@@ -1,3 +1,6 @@
*.sh -crlf
*.ac -crlf
*.am -crlf
#Content files are C file
*.inc linguist-language=C
+50
View File
@@ -0,0 +1,50 @@
---
name: "🐞 Report a Bug"
about: Create a report to help us improve nDPI
title: ''
labels: 'bug'
assignees: ''
---
## Describe the bug
A clear and concise description of what the bug is.
### Expected behavior
A clear and concise description of what you expected to happen.
### Obtained behavior
A clear and concise description of what happening.
## nDPI Environment (please complete the following information):
* OS name: [e.g. Ubuntu].
* OS version: [e.g. 18.04]
* Architecture: [e.g. arm64]
* nDPI version or commit hash: [e.g. 4.0-stable or 937357e4bc55610f116f66d15a8e0fc1e260c02c].
* nDPI compilation flags used: if you are building from source [e.g. --with-pcre --disable-gcrypt].
* Attach the `config.log` file generated after `./configure` ran (if you are building from source).
## How to reproduce the reported bug
### Reproducible using ndpiReader?
- [x] The reported bug is reproducible using ndpiReader.
- [x] The reported bug is not reproducible using ndpiReader.
### If applicable, the used ndpiReader options:
``` shell
ndpiReader -q -t -i wlo1 -T 20 -U 20
```
### If your bug is reproducible using a pcap, please attach a pcap file (or a valid link to download it)
Example: test.pcap
### Steps to reproduce the behavior:
1. Run '...'
2. Set '....'
3. Do '....'
4. See error
## Additional context
Add any other context about the problem here.
@@ -0,0 +1,20 @@
---
name: "💡 Suggest an Improvement or Feature"
about: Suggest an idea or a feature for nDPI
title: ''
labels: 'enhancement'
assignees: ''
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I always need to [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
@@ -0,0 +1,10 @@
---
name: General question
about: Ask a question about nDPI
title: ''
labels: 'question'
assignees: ''
---
Type your question in a clear and concise way to help the community answering it.
+278 -9
View File
@@ -2,19 +2,288 @@ name: Build
on:
push:
branches:
- master
- dev
pull_request:
branches:
- dev
types: [opened, synchronize, reopened]
release:
types: [created]
jobs:
sonarcloud:
name: SonarCloud
runs-on: ubuntu-latest
test:
name: ${{ matrix.os }} ${{ matrix.arch }} ${{ matrix.gcrypt }} ${{ matrix.compiler }} ${{ matrix.pcre }} ${{ matrix.maxminddb }} ${{ matrix.msan }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false # Debugging purposes, switch to true when all issues on big endian is solved
matrix:
os: ["ubuntu-latest", "ubuntu-18.04", "macOS-latest", "macos-11"]
arch: ["x86_64"]
gcrypt: ["--disable-gcrypt", ""]
compiler: ["default-cc"]
pcre: [""]
maxminddb: [""]
msan: [""]
include:
- compiler: "gcc-7"
os: ubuntu-latest
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre"
maxminddb: "--with-maxminddb"
msan: "--with-sanitizer"
- compiler: "gcc-10"
os: ubuntu-latest
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre"
maxminddb: "--with-maxminddb"
msan: "--with-sanitizer"
- compiler: "clang-7"
os: ubuntu-latest
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre"
maxminddb: "--with-maxminddb"
msan: "--with-sanitizer"
- compiler: "clang-12"
os: ubuntu-latest
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre"
maxminddb: "--with-maxminddb"
msan: "--with-sanitizer"
- compiler: "default-cc"
os: ubuntu-latest
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre"
maxminddb: "--with-maxminddb"
msan: "--with-sanitizer"
- compiler: "default-cc"
os: ubuntu-18.04
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre"
maxminddb: "--with-maxminddb"
msan: "--with-sanitizer"
- compiler: "default-cc"
os: macOS-latest
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre"
maxminddb: "--with-maxminddb"
msan: "" # Disable sanitizer on macos
- compiler: "default-cc"
os: macos-11
arch: "x86_64"
gcrypt: ""
pcre: "--with-pcre"
maxminddb: "--with-maxminddb"
msan: "" # Disable sanitizer on macos
- compiler: "default-cc"
os: ubuntu-latest
arch: "arm64"
gcrypt: ""
pcre: "--with-pcre"
maxminddb: "--with-maxminddb"
msan: "" # Disable sanitizer on arm64
- compiler: "default-cc"
os: ubuntu-latest
arch: "armhf"
gcrypt: ""
pcre: "--with-pcre"
maxminddb: "--with-maxminddb"
msan: "--with-sanitizer"
- compiler: "default-cc"
os: ubuntu-latest
arch: "s390x"
gcrypt: ""
pcre: "--with-pcre"
maxminddb: "--with-maxminddb"
msan: ""
steps:
- name: Setup multiarch/qemu-user-static
if: startsWith(matrix.os, 'ubuntu') && !startsWith(matrix.arch, 'x86_64')
run: |
docker run --rm --privileged multiarch/qemu-user-static:register --reset
- uses: actions/checkout@v2
with:
fetch-depth: 0 # Shallow clones should be disabled for a better relevancy of analysis
- name: SonarCloud Scan
uses: SonarSource/sonarcloud-github-action@master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # Needed to get PR information, if any
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
- name: Install Ubuntu Prerequisites
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64')
run: |
sudo apt-get update
sudo apt-get install autoconf automake libtool pkg-config gettext libjson-c-dev flex bison libpcap-dev
sudo apt-get install gcc-mingw-w64 libc6-dev
sudo apt-get install doxygen python3-sphinx python3-sphinx-rtd-theme python3-breathe python3-pip
sudo apt-get install rrdtool librrd-dev
- name: Install Ubuntu Prerequisites (libgcrypt)
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64') && !startsWith(matrix.gcrypt, '--disable-gcrypt')
run: |
sudo apt-get install libgcrypt20-dev
- name: Install Ubuntu Prerequisites (libpcre)
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.pcre, '--with-pcre')
run: |
sudo apt-get install libpcre3-dev
- name: Install Ubuntu Prerequisites (maxminddb)
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.maxminddb, '--with-maxminddb')
run: |
sudo apt-get install libmaxminddb-dev
- name: Setup Ubuntu specified compiler (gcc)
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.compiler, 'gcc')
run: |
sudo apt-get install ${{ matrix.compiler }}
- name: Setup Ubuntu specified compiler (clang)
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.compiler, 'clang')
run: |
sudo apt-get install ${{ matrix.compiler }}
- name: Installing MacOS prerequisites
if: startsWith(matrix.os, 'macOS') && startsWith(matrix.arch, 'x86_64')
run: |
brew install autoconf automake libtool pkg-config gettext json-c
brew install coreutils
brew install rrdtool
- name: Install MacOS Prerequisites (libgcrypt)
if: startsWith(matrix.os, 'macOS') && startsWith(matrix.arch, 'x86_64') && !startsWith(matrix.gcrypt, '--disable-gcrypt')
run: |
brew install libgcrypt
- name: Install MacOS Prerequisites (libpcre)
if: startsWith(matrix.os, 'macOS') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.pcre, '--with-pcre')
run: |
brew install pcre
- name: Install MacOS Prerequisites (maxminddb)
if: startsWith(matrix.os, 'macOS') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.maxminddb, '--with-maxminddb')
run: |
brew install libmaxminddb
- name: Configure nDPI on Ubuntu
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.compiler, 'default-cc')
run: |
env CC=gcc CFLAGS='-Werror' ./autogen.sh --enable-debug-messages ${{ matrix.gcrypt }} ${{ matrix.msan }} ${{ matrix.pcre }} ${{ matrix.maxminddb }}
- name: Configure nDPI on MacOS
if: startsWith(matrix.os, 'macOS') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.compiler, 'default-cc')
run: |
env CC=clang CFLAGS='-Werror' ./autogen.sh --enable-debug-messages ${{ matrix.gcrypt }} ${{ matrix.msan }} ${{ matrix.pcre }} ${{ matrix.maxminddb }}
- name: Configure nDPI with specified GCC version on Ubuntu
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.compiler, 'gcc')
run: |
env CC=${{ matrix.compiler }} CFLAGS='-Werror' ./autogen.sh --enable-debug-messages ${{ matrix.gcrypt }} ${{ matrix.msan }} ${{ matrix.pcre }} ${{ matrix.maxminddb }}
- name: Configure nDPI with specified CLANG on Ubuntu
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64') && startsWith(matrix.compiler, 'clang')
run: |
env CC=${{ matrix.compiler }} CFLAGS='-Werror' ./autogen.sh --enable-debug-messages ${{ matrix.gcrypt }} ${{ matrix.msan }} ${{ matrix.pcre }} ${{ matrix.maxminddb }}
- name: Build nDPI
if: startsWith(matrix.arch, 'x86_64')
run: |
make all
make -C example ndpiSimpleIntegration
make -C rrdtool
make -C python
- name: Install nDPI
if: startsWith(matrix.arch, 'x86_64')
run: |
make install DESTDIR=$(realpath _install)
ls -alhHR _install
- name: Test nDPI [DIFF]
if: startsWith(matrix.arch, 'x86_64')
run: |
./tests/do.sh
- name: Test nDPI [UNIT]
if: startsWith(matrix.arch, 'x86_64')
run: |
./tests/do-unit.sh
- name: Test nDPI [DGA]
if: startsWith(matrix.arch, 'x86_64')
run: |
./tests/do-dga.sh
- name: Generate tarball
if: startsWith(matrix.os, 'ubuntu-latest') && startsWith(matrix.arch, 'x86_64')
run: |
make dist
- name: Generate documentation
if: startsWith(matrix.os, 'ubuntu-latest') && startsWith(matrix.arch, 'x86_64')
run: |
pip3 install --user sphinxcontrib.swaggerdoc
make -C doc text
make -C doc html
- name: Configure nDPI [Mingw-w64] (runs only on ubuntu jobs)
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64')
run: |
make distclean && ./autogen.sh --host=x86_64-w64-mingw32 ${{ matrix.gcrypt }} ${{ matrix.pcre }} ${{ matrix.maxminddb }}
- name: Build nDPI [Mingw-w64] (runs only on ubuntu jobs)
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'x86_64')
run: |
make all
- name: Display qemu specified architecture (arm64 - little endian)
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'arm64')
uses: docker://multiarch/ubuntu-core:arm64-bionic
with:
args: >
bash -c
"uname -a &&
lscpu | grep Endian
"
- name: Configure, compile and test using qemu for the specified architecture (arm64 - little endian)
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'arm64')
uses: docker://multiarch/ubuntu-core:arm64-bionic
with:
args: >
bash -c
"apt-get -y update &&
apt-get -y install git autoconf automake libtool pkg-config gettext libjson-c-dev flex bison libpcap-dev libgcrypt20-dev libpcre3-dev libmaxminddb-dev rrdtool librrd-dev &&
env CC=gcc CFLAGS='-Werror' ./autogen.sh --enable-debug-messages ${{ matrix.gcrypt }} ${{ matrix.msan }} ${{ matrix.pcre }} ${{ matrix.maxminddb }} &&
make all &&
make -C example ndpiSimpleIntegration &&
make -C rrdtool &&
make -C python &&
./tests/do.sh &&
./tests/do-unit.sh
"
- name: Display qemu specified architecture (armhf - little endian)
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'armhf')
uses: docker://multiarch/ubuntu-core:armhf-bionic
with:
args: >
bash -c
"uname -a &&
lscpu | grep Endian
"
- name: Configure, compile and test using qemu for the specified architecture (armhf - little endian)
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 'armhf')
uses: docker://multiarch/ubuntu-core:armhf-bionic
with:
args: >
bash -c
"apt-get -y update &&
apt-get -y install git autoconf automake libtool pkg-config gettext libjson-c-dev flex bison libpcap-dev libgcrypt20-dev libpcre3-dev libmaxminddb-dev rrdtool librrd-dev &&
env CC=gcc CFLAGS='-Werror' ./autogen.sh --enable-debug-messages ${{ matrix.gcrypt }} ${{ matrix.msan }} ${{ matrix.pcre }} ${{ matrix.maxminddb }} &&
make all &&
make -C example ndpiSimpleIntegration &&
make -C rrdtool &&
make -C python &&
./tests/do.sh &&
./tests/do-unit.sh
"
- name: Display qemu specified architecture (s390x - big endian)
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 's390x')
uses: docker://multiarch/ubuntu-core:s390x-bionic
with:
args: >
bash -c
"uname -a &&
lscpu | grep Endian
"
- name: Configure and compile (no tests) using qemu for the specified architecture (s390x - big endian)
if: startsWith(matrix.os, 'ubuntu') && startsWith(matrix.arch, 's390x')
uses: docker://multiarch/ubuntu-core:s390x-bionic
with: #./tests/do.sh disabled because we know we have some problems with big-endian machines
args: >
bash -c
"apt-get -y update &&
apt-get -y install git autoconf automake libtool pkg-config gettext libjson-c-dev flex bison libpcap-dev libgcrypt20-dev libpcre3-dev libmaxminddb-dev rrdtool librrd-dev &&
env CC=gcc CFLAGS='-Werror' ./autogen.sh --enable-debug-messages ${{ matrix.gcrypt }} ${{ matrix.msan }} ${{ matrix.pcre }} ${{ matrix.maxminddb }} &&
make all &&
make -C example ndpiSimpleIntegration &&
make -C rrdtool &&
make -C python &&
./tests/do-unit.sh
"
+32 -21
View File
@@ -1,23 +1,34 @@
name: CIFuzz
on: [pull_request]
on: [push, pull_request]
jobs:
Fuzzing:
runs-on: ubuntu-latest
steps:
- name: Build Fuzzers
uses: google/oss-fuzz/infra/cifuzz/actions/build_fuzzers@master
with:
oss-fuzz-project-name: 'ndpi'
dry-run: false
- name: Run Fuzzers
uses: google/oss-fuzz/infra/cifuzz/actions/run_fuzzers@master
with:
oss-fuzz-project-name: 'ndpi'
fuzz-seconds: 600
dry-run: false
- name: Upload Crash
uses: actions/upload-artifact@v1
if: failure()
with:
name: artifacts
path: ./out/artifacts
Fuzzing:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
sanitizer: [address, undefined, memory]
steps:
- name: Build Fuzzers - ${{ matrix.sanitizer }}
uses: google/oss-fuzz/infra/cifuzz/actions/build_fuzzers@master
with:
oss-fuzz-project-name: 'ndpi'
dry-run: false
sanitizer: ${{ matrix.sanitizer }}
- name: Run Fuzzers - ${{ matrix.sanitizer }}
uses: google/oss-fuzz/infra/cifuzz/actions/run_fuzzers@master
with:
oss-fuzz-project-name: 'ndpi'
fuzz-seconds: 600
dry-run: false
sanitizer: ${{ matrix.sanitizer }}
- name: Check Crash (fails when a crash is detected)
# Run Fuzzers return success even when setting dry-run to false.
# A temporal workaround is to trigger failure manually if we fing crash files.
run: |
exit $(ls out/artifacts |wc -l)
- name: Upload Crash (upload detected crash as artifacts)
uses: actions/upload-artifact@v1
if: failure()
with:
name: ${{ matrix.sanitizer }}-artifacts
path: ./out/artifacts
+26 -1
View File
@@ -11,7 +11,6 @@ compile
/config.status
/config.sub
/configure
/configure.ac
/configure.tmp
.cproject
/depcomp
@@ -21,6 +20,7 @@ compile
/example/ndpiReader
/example/Makefile.dpdk
/example/build
/example/ndpiSimpleIntegration
*.in
/install-sh
*.ko
@@ -45,6 +45,7 @@ compile
.settings
/src/include/Makefile
/src/include/ndpi_config.h
/src/include/ndpi_api.h
/src/include/ndpi_define.h
/src/lib/.deps/
/src/lib/Makefile
@@ -54,3 +55,27 @@ stamp-h1
*.swp
/test-driver
/tests/Makefile
/fuzz/.deps/
/fuzz/Makefile
/fuzz/fuzz_ndpi_reader
/fuzz/fuzz_ndpi_reader_with_main
/fuzz/fuzz_process_packet
/fuzz/fuzz_process_packet_with_main
/python/Makefile
/tests/dga/Makefile
/tests/do.sh
/tests/unit/Makefile
/example/libndpiReader.a
/example/ndpiReader.exe
/src/lib/libndpi.a
/src/lib/libndpi.so*
/src/lib/libndpi
/src/lib/libndpi-4.1.0.dll
/src/lib/libndpi.4
/tests/dga/dga_evaluate
/tests/unit/unit
/tags
/rrdtool/Makefile
/rrdtool/rrd_anomaly
/rrdtool/rrd_similarity
/tests/performance/Makefile
-238
View File
@@ -1,238 +0,0 @@
language: c
matrix:
include:
- os: osx
compiler: clang
- os: linux
env: BUILD_DPKG_PACKAGE=y
compiler: gcc
dist: bionic
addons:
apt:
packages:
- fakeroot
- debhelper
- dpkg-dev
- libpcap-dev
- libgcrypt20-dev
- libjson-c-dev
- autogen
- os: linux
env: DISABLE_GCRYPT=y
compiler: gcc
dist: bionic
addons:
apt:
packages:
- fakeroot
- debhelper
- dpkg-dev
- libpcap-dev
- libgcrypt20-dev
- libjson-c-dev
- autogen
- os: linux
compiler: clang
dist: bionic
addons:
apt:
packages:
- libpcap-dev
- libgcrypt20-dev
- libjson-c-dev
- autogen
- os: linux
compiler: gcc
dist: bionic
arch: arm64
addons:
apt:
packages:
- libpcap-dev
- libgcrypt20-dev
- libjson-c-dev
- autogen
- os: linux
compiler: clang
dist: bionic
arch: arm64
addons:
apt:
packages:
- libpcap-dev
- libgcrypt20-dev
- libjson-c-dev
- autogen
# Targets below have been disabled as we have no way
# to debug on tsuch platforms at the moment
# - os: linux
# compiler: gcc
# dist: bionic
# arch: s390x
# addons:
# apt:
# packages:
# - libpcap-dev
# - libjson-c-dev
# - autogen
#
# - os: linux
# compiler: clang
# dist: bionic
# arch: s390x
# addons:
# apt:
# packages:
# - libpcap-dev
# - libjson-c-dev
# - autogen
- os: linux
compiler: gcc-8
addons:
apt:
sources:
- ubuntu-toolchain-r-test
packages:
- g++-8
- libpcap-dev
- libgcrypt20-dev
- libjson-c-dev
- autogen
env:
- MATRIX_EVAL="CC=gcc-8"
- os: linux
compiler: gcc-9
addons:
apt:
sources:
- ubuntu-toolchain-r-test
packages:
- g++-9
- libpcap-dev
- libgcrypt20-dev
- libjson-c-dev
- autogen
env:
- MATRIX_EVAL="CC=gcc-9"
- os: linux
compiler: clang-8
addons:
apt:
sources:
- ubuntu-toolchain-r-test
- llvm-toolchain-trusty-8
packages:
- clang-8
- libpcap-dev
- libgcrypt20-dev
- libjson-c-dev
- autogen
env:
- MATRIX_EVAL="CC=clang-8"
- os: linux
compiler: clang-7
addons:
apt:
sources:
- ubuntu-toolchain-r-test
- llvm-toolchain-trusty-7
packages:
- clang-7
- libpcap-dev
- libgcrypt20-dev
- libjson-c-dev
- autogen
env:
- MATRIX_EVAL="CC=clang-7"
- name: fuzza
env: CXXFLAGS="-g3 -O0 -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize=fuzzer-no-link" CFLAGS="-g3 -O0 -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize=fuzzer-no-link" LDFLAGS="-g3 -O0 -fsanitize=address" QA_FUZZ=asan CC=clang-7 && CXX=clang++-7 ASAN_SYMBOLIZER_PATH=/usr/local/clang-7.0.0/bin/llvm-symbolizer
os: linux
compiler: clang-7
addons:
apt:
sources:
- ubuntu-toolchain-r-test
- llvm-toolchain-trusty-7
packages:
- clang-7
- libpcap-dev
- libgcrypt20-dev
- libjson-c-dev
- autogen
- name: fuzzm
env: CXXFLAGS="-g3 -O0 -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=memory -fsanitize=fuzzer-no-link" CFLAGS="-g3 -O0 -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=memory -fsanitize=fuzzer-no-link" LDFLAGS="-g3 -O0 -fsanitize=memory" QA_FUZZ=msan CC=clang && CXX=clang++
os: linux
compiler: clang
dist: bionic
addons:
apt:
packages:
- libpcap-dev
- libgcrypt20-dev
- libjson-c-dev
- autogen
- name: fuzzu
env: CXXFLAGS="-g3 -Og -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=undefined -fsanitize=fuzzer-no-link -fno-sanitize=alignment" CFLAGS="-g3 -Og -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=undefined -fno-sanitize-recover=undefined,integer -fsanitize=fuzzer-no-link -fno-sanitize=alignment" LDFLAGS="-g3 -Og -fsanitize=undefined -fno-sanitize=alignment" QA_FUZZ=ubsan CC=clang-7 && CXX=clang++-7
os: linux
compiler: clang-7
addons:
apt:
sources:
- ubuntu-toolchain-r-test
- llvm-toolchain-trusty-7
packages:
- clang-7
- libpcap-dev
- libgcrypt20-dev
- libjson-c-dev
- autogen
before_install:
- eval "${MATRIX_EVAL}"
before_script:
- ./autogen.sh
# - lcov --directory . --zerocounters
script:
- if [ -n "$QA_FUZZ" ]; then ./configure --enable-fuzztargets ; else
./configure ; fi
- if [ -n "$BUILD_DPKG_PACKAGE" ]; then cd ./packages/ubuntu ;
./configure && dpkg-buildpackage -b -us -uc ; cd ../.. ; fi
- if [ -n "$DISABLE_GCRYPT" ]; then ./configure --disable-gcrypt ; fi
- make
- make -C example ndpiSimpleIntegration
- make dist
#after_script:
- cd tests
- ./do.sh
- ./do-unit.sh
- ./do-dga.sh
#after_success:
#- cd ${TRAVIS_BUILD_DIR}
#- lcov --directory . --capture --output-file coverage.info # capture coverage info
#- lcov --remove coverage.info 'tests/*' 'packages/*' 'm4/*' 'doc/*' --output-file coverage.info
#- lcov --list coverage.info # debug before upload
#- coveralls-lcov --repo-token ${COVERALLS_TOKEN} coverage.info # uploads to coveralls
notifications:
email:
recipients:
- packager@ntop.org
on_success: never
on_failure: always
+96
View File
@@ -1,5 +1,101 @@
# CHANGELOG
#### nDPI 4.2 (Feb 2022)
## New Features
- Add a "confidence" field indicating the reliability of the classification
- Add risk exceptions for services and domain names via ndpi_add_domain_risk_exceptions()
- Add ability to report whether a protocol is encrypted
## New Supported Protocols and Services
- Add protocol detection for:
- Badoo
- Cassandra
- EthernetIP
## Improvements
- Reduce memory footprint
- Improve protocol detection for:
- BitTorrent
- ICloud Private Relay
- IMAP, POP3, SMTP
- Log4J/Log4Shell
- Microsoft Azure
- Pandora TV
- RTP
- RTSP
- Salesforce
- STUN
- Whatsapp
- QUICv2
- Zoom
- Add flow risk:
- NDPI_CLEAR_TEXT_CREDENTIALS
- NDPI_POSSIBLE_EXPLOIT (Log4J)
- NDPI_TLS_FATAL_ALERT
- NDPI_TLS_CERTIFICATE_ABOUT_TO_EXPIRE
- Update WhatsAPP and Instagram addresses
- Update the list of default ports for QUIC
- Update WindowsUpdate URLs
- Add support for the .goog Google TLD
- Add googletagmanager.com
- Add bitmaps and API for handling compressed bitmaps
- Add JA3 in risk exceptions
- Add entropy calculation to check for suspicious (encrypted) payload
- Add extraction of hostname in SMTP
- Add RDP over UDP dissection
- Add support for TLS over IPV6 in Subject Alt Names field
- Improve JSON and CSV serialization
- Improve IPv6 support for almost all dissectors
- Improve CI and unit tests, add arm64, armhf and s390x as part of CI
- Improve WHOIS detection, reduce false positives
- Improve DGA detection for skipping potential DGAs of known/popular domain names
- Improve user agent analysis
- Reworked HTTP protocol dissection including HTTP proxy and HTTP connect
## Changes
- TLS obsolete protocol is set when TLS < 1.2 (used to be 1.1)
- Numeric IPs are not considered for DGA checks
- Differentiate between standard Amazon stuff (i.e market) and AWS
- Remove Playstation VUE protocol
- Remove pandora.tv from Pandora protocol
- Remove outdated SoulSeek dissector
## Fixes
- Fix race conditions
- Fix dissectors to be big-endian friendly
- Fix heap overflow in realloc wrapper
- Fix errors in Kerberos, TLS, H323, Netbios, CSGO, Bittorrent
- Fix wrong tuple comparison
- Fix ndpi_serialize_string_int64
- Fix Grease values parsing
- Fix certificate mismatch check
- Fix null-dereference read for Zattoo with IPv6
- Fix dissectors initialization for XBox, Diameter
- Fix confidence for STUN classifications
- Fix FreeBSD support
- Fix old GQUIC versions on big-endian machines
- Fix aho-corasick on big-endian machines
- Fix DGA false positive
- Fix integer overflow for QUIC
- Fix HTTP false positives
- Fix SonarCloud-CI support
- Fix clashes setting the hostname on similar protocols (FTP, SMTP)
- Fix some invalid TLS guesses
- Fix crash on ARM (Raspberry)
- Fix DNS (including fragmented DNS) dissection
- Fix parsing of IPv6 packets with extension headers
- Fix extraction of Realm attribute in STUN
- Fix support for START-TLS sessions in FTP
- Fix TCP retransmissions for multiple dissectors
- Fix DES initialisation
- Fix Git protocol dissection
- Fix certificate mismatch for TLS flows with no client hello observed
- Fix old versions of GQUIC on big-endian machines
## Misc
- Add tool for generating automatically the Azure IP list
#### nDPI 4.0 (July 2021)
## New Features
+1 -1
View File
@@ -8,7 +8,7 @@ endif
pkgconfigdir = $(libdir)/pkgconfig
pkgconfig_DATA = libndpi.pc
EXTRA_DIST = README.md CHANGELOG.md CONTRIBUTING.md README.protocols autogen.sh configure.seed wireshark python windows utils packages doc/nDPI_QuickStartGuide.pages doc/nDPI_QuickStartGuide.pdf example/MacOS example/Win32
EXTRA_DIST = README.md CHANGELOG.md CONTRIBUTING.md autogen.sh wireshark python windows utils packages example/MacOS doc/api.rst doc/conf.py doc/flow_risks.rst doc/guide/nDPI_QuickStartGuide.pages doc/guide/nDPI_QuickStartGuide.pdf doc/img/logo.png doc/index.rst doc/Makefile doc/what_is_ndpi.rst
changelog:
git log --since={`curl -s https://github.com/ntop/ndpi/releases | grep datetime | head -n1 | egrep -o "[0-9]+\-[0-9]+\-[0-9]+"`} --name-only --pretty=format:" - %s" | grep "^ " > CHANGELOG.latest
+41
View File
@@ -0,0 +1,41 @@
## Compiling nDPI with Fuzzer
- Install the latest clang (sudo apt-get install clang-10)
- Export environamental variables to prefer clang over g++ (of installed)
- export CC=/usr/bin/clang-10
- export CPP=/usr/bin/clang-cpp-10
- export CXX=/usr/bin/clang++-10
- export LD=/usr/bin/ld.lld-10
- Run autogen.sh
- ./autogen.sh --with-sanitizer --enable-fuzztargets
## Testing nDPI with ClusterFuzz Artifacts
- Download the artifact (example clusterfuzz-testcase-fuzz_process_packet-4992218834796544)
- Run nDPI against the artifact
- Example: ./fuzz/fuzz_process_packet_with_main clusterfuzz-testcase-fuzz_process_packet-4992218834796544
The output is the error report
```
./fuzz/fuzz_process_packet_with_main /tmp/clusterfuzz-testcase-fuzz_process_packet-4992218834796544
AddressSanitizer:DEADLYSIGNAL
=================================================================
==11590==ERROR: AddressSanitizer: SEGV on unknown address 0x61a100000087 (pc 0x00000056e6a4 bp 0x7ffd624fa170 sp 0x7ffd624fa090 T0)
==11590==The signal is caused by a READ memory access.
#0 0x56e6a4 in quic_len /home/deri/nDPI/src/lib/protocols/quic.c:203:12
#1 0x575d6b in decrypt_initial_packet /home/deri/nDPI/src/lib/protocols/quic.c:993:16
#2 0x571776 in get_clear_payload /home/deri/nDPI/src/lib/protocols/quic.c:1302:21
#3 0x56f149 in ndpi_search_quic /home/deri/nDPI/src/lib/protocols/quic.c:1658:19
#4 0x503935 in check_ndpi_detection_func /home/deri/nDPI/src/lib/ndpi_main.c:4683:6
#5 0x5056fb in check_ndpi_udp_flow_func /home/deri/nDPI/src/lib/ndpi_main.c:4742:10
#6 0x505152 in ndpi_check_flow_func /home/deri/nDPI/src/lib/ndpi_main.c:4775:12
#7 0x5174cf in ndpi_detection_process_packet /home/deri/nDPI/src/lib/ndpi_main.c:5545:15
#8 0x4c709b in LLVMFuzzerTestOneInput /home/deri/nDPI/fuzz/fuzz_process_packet.c:30:3
#9 0x4c7640 in main /home/deri/nDPI/fuzz/fuzz_process_packet.c:90:17
#10 0x7f888e5dabf6 in __libc_start_main /build/glibc-S9d2JN/glibc-2.27/csu/../csu/libc-start.c:310
#11 0x41c399 in _start (/home/deri/nDPI/fuzz/fuzz_process_packet_with_main+0x41c399)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/deri/nDPI/src/lib/protocols/quic.c:203:12 in quic_len
==11590==ABORTING
```
+1 -1
View File
@@ -1,7 +1,7 @@
![ntop][ntopng_logo] ![ntop][ntop_logo]
# nDPI
[![Build Status](https://travis-ci.org/ntop/nDPI.png?branch=dev)](https://travis-ci.org/ntop/nDPI)
[![Build Status](https://img.shields.io/github/workflow/status/ntop/nDPI/Build/dev?logo=github)](https://github.com/ntop/nDPI/actions?query=workflow%3ABuild)
[![Code Quality: Cpp](https://img.shields.io/lgtm/grade/cpp/g/ntop/nDPI.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/ntop/nDPI/context:cpp)
[![Total Alerts](https://img.shields.io/lgtm/alerts/g/ntop/nDPI.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/ntop/nDPI/alerts)
[![Fuzzing Status](https://oss-fuzz-build-logs.storage.googleapis.com/badges/ndpi.svg)](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:ndpi)
-18
View File
@@ -1,18 +0,0 @@
Tor
---
Tor protocol can use SSL to hide itself. These are examples:
TCP 37.128.208.46:9001 <-> 172.16.253.130:2078 [VLAN: 0][proto: 91/SSL][132 pkts/93834 bytes][SSL client: www.jwrpsthzrih.com]
TCP 172.16.253.130:2021 <-> 75.147.140.249:443 [VLAN: 0][proto: 91/SSL][28 pkts/8053 bytes][SSL client: www.5akw23dx.com]
TCP 172.16.253.130:2077 <-> 77.247.181.163:443 [VLAN: 0][proto: 91/SSL][136 pkts/94329 bytes][SSL client: www.fk4pprq42hsvl2wey.com]
It can be detected by analyzing the SSL client certificate and checking the name that does not match to a real host in
addition of being a bit weird. As doing DNS resolution is not a task for nDPI we let applications do and then recognize
SSL-tunnelled connections.
See http://www.netresec.com/?page=Blog&month=2013-04&post=Detecting-TOR-Communication-in-Network-Traffic
For this reason nDPI uses a heuristic, non-DNS based, approach to detect tor communications. If possible, apps
should validate the certificate using the DNS. This is not something nDPI can afford to do for performance
reasons
-21
View File
@@ -1,10 +1,5 @@
#!/bin/sh
NDPI_MAJOR="4"
NDPI_MINOR="0"
NDPI_PATCH="0"
NDPI_VERSION_SHORT="$NDPI_MAJOR.$NDPI_MINOR.$NDPI_PATCH"
rm -f configure config.h config.h.in
AUTOCONF=$(command -v autoconf)
@@ -13,16 +8,10 @@ LIBTOOL=$(command -v libtool)
LIBTOOLIZE=$(command -v libtoolize)
AUTORECONF=$(command -v autoreconf)
PKG_CONFIG=$(command -v pkg-config)
FUZZY=
if test -z $AUTOCONF; then
echo "autoconf is missing: please install it and try again"
exit
else
V=`autoconf --version | head -1 | cut -d ' ' -f 4`
if [ "$V" = '2.63' ]; then
FUZZY="dnl> "
fi
fi
if test -z $AUTOMAKE; then
@@ -45,17 +34,7 @@ if test -z $PKG_CONFIG; then
exit
fi
cat configure.seed | sed \
-e "s/@NDPI_MAJOR@/$NDPI_MAJOR/g" \
-e "s/@NDPI_MINOR@/$NDPI_MINOR/g" \
-e "s/@NDPI_PATCH@/$NDPI_PATCH/g" \
-e "s/@NDPI_VERSION_SHORT@/$NDPI_VERSION_SHORT/g" \
-e "s/@FUZZY@/$FUZZY/g" \
> configure.ac
autoreconf -ivf
cat configure | sed "s/#define PACKAGE/#define NDPI_PACKAGE/g" | sed "s/#define VERSION/#define NDPI_VERSION/g" > configure.tmp
cat configure.tmp > configure
echo "./configure $@"
chmod +x configure
+115 -70
View File
@@ -1,4 +1,4 @@
AC_INIT([libndpi], [@NDPI_VERSION_SHORT@])
AC_INIT([libndpi], [4.2.0])
AC_CONFIG_AUX_DIR([.])
AC_CONFIG_MACRO_DIR([m4])
@@ -13,25 +13,19 @@ AS_IF([test "${with_only_libndpi+set}" = set],[
EXTRA_TARGETS=""
])
GNU_PREFIX=""
AC_ARG_WITH(mipsel, AS_HELP_STRING([--with-mipsel], [Build only libndpi (no examples, tests etc) for mipsel]))
AS_IF([test "${with_mipsel+set}" = set],[
EXTRA_TARGETS=""
GNU_PREFIX="mipsel-linux-gnu-"
])
AC_ARG_WITH(sanitizer, AS_HELP_STRING([--with-sanitizer], [Build with support for address, undefined and leak sanitizer]))
AC_ARG_ENABLE(fuzztargets, AS_HELP_STRING([--enable-fuzztargets], [Enable fuzz targets]),[enable_fuzztargets=$enableval],[enable_fuzztargets=no])
AS_IF([test "x$enable_fuzztargets" = "xyes"], [BUILD_FUZZTARGETS=1], [BUILD_FUZZTARGETS=0])
AM_CONDITIONAL([BUILD_FUZZTARGETS], [test "x$enable_fuzztargets" = "xyes"])
AS_IF([test "${with_sanitizer+set}" = set],[
CFLAGS="${CFLAGS} -g3 -O0 -Wno-unused-value -fsanitize=address -fsanitize=undefined -fno-sanitize=alignment -fsanitize=leak -fno-omit-frame-pointer"
CFLAGS="${CFLAGS} -g3 -O0 -fsanitize=address -fsanitize=undefined -fno-sanitize=alignment -fsanitize=leak -fno-omit-frame-pointer"
LDFLAGS="${LDFLAGS} -fsanitize=address -fsanitize=undefined -fno-sanitize=alignment -fsanitize=leak"
])
LT_INIT
LT_LIB_M
PKG_PROG_PKG_CONFIG
SYSTEM=`uname -s`
if test $SYSTEM = "Darwin"; then
@@ -40,18 +34,20 @@ dnl> AC_PROG_CC(clang gcc)
AC_PROG_CXX(clang++ g++)
AC_PROG_CC_STDC(clang gcc)
else
dnl> AC_PROG_CC
AM_PROG_CC_C_O
AC_PROG_CC_C_O
AC_PROG_CXX
AC_PROG_CC_STDC
fi
AC_LANG_WERROR
AX_PTHREAD
NDPI_VERSION_SHORT="@NDPI_VERSION_SHORT@"
NDPI_MAJOR="@NDPI_MAJOR@"
NDPI_MINOR="@NDPI_MINOR@"
NDPI_PATCH="@NDPI_PATCH@"
dnl> Can't iuse AM_PROG_AR because not all of our Makefiles are automake (yet?)
AC_CHECK_TOOL(AR, ar, [false])
AC_LANG_WERROR
NDPI_MAJOR=`echo "${PACKAGE_VERSION}" | cut -d . -f 1`
NDPI_MINOR=`echo "${PACKAGE_VERSION}" | cut -d . -f 2`
NDPI_PATCH=`echo "${PACKAGE_VERSION}" | cut -d . -f 3`
NDPI_VERSION_SHORT="$NDPI_MAJOR.$NDPI_MINOR.$NDPI_PATCH"
AC_DEFINE_UNQUOTED(NDPI_MAJOR_RELEASE, "${NDPI_MAJOR}", [nDPI major release])
AC_DEFINE_UNQUOTED(NDPI_MINOR_RELEASE, "${NDPI_MINOR}", [nDPI minor release])
@@ -84,20 +80,41 @@ fi
NDPI_API_VERSION=`echo $NDPI_API_VERSION | sed 's/^0*//'`
echo "Setting API version to $NDPI_API_VERSION"
AC_DEFINE_UNQUOTED(NDPI_GIT_RELEASE, "${GIT_RELEASE}", [GIT Release])
AC_DEFINE_UNQUOTED(NDPI_GIT_DATE, "${GIT_DATE}", [Last GIT change])
if ! test "${with_mipsel+set}" = set && ! test "${with_only_libndpi+set}" = set; then :
dnl> used by json-c for unit tests
PKG_CHECK_MODULES([JSONC], [json-c], [JSONC_LIBS=`pkg-config --libs json-c` JSONC_CFLAGS=`pkg-config --cflags json-c`])
if ! test "${with_only_libndpi+set}" = set; then :
dnl> used by json-c for unit tests
PKG_CHECK_MODULES([JSONC], [json-c], [JSONC_LIBS="${pkg_cv_JSONC_LIBS}" JSONC_CFLAGS="${pkg_cv_JSONC_CFLAGS}"])
AC_CHECK_LIB([json-c], [json_object_put], [], [
AC_MSG_WARN([JSON-C not available. Disabled unit test.])
JSONC_LIBS=""
JSONC_CFLAGS=""
EXTRA_TARGETS="example tests tests/dga"
])
AX_PTHREAD([],[
AC_MSG_WARN([POSIX Threads not available. Building library only.])
JSONC_LIBS=""
JSONC_CFLAGS=""
EXTRA_TARGETS=""
])
fi
LIBS="$LIBS $JSONC_LIBS"
CFLAGS="$CFLAGS $JSONC_CFLAGS"
CFLAGS="-W -Wall -Wno-unused-parameter -Wno-unused-function $CFLAGS $JSONC_CFLAGS"
AC_CHECK_HEADERS([netinet/in.h stdint.h stdlib.h string.h unistd.h json.h])
AC_CHECK_HEADERS([netinet/in.h stdint.h stdlib.h string.h unistd.h math.h float.h])
AC_CHECK_LIB([m], [sqrt], [], [LIBM="-lm"])
AC_CHECK_LIB([rrd], [rrd_fetch_r], [LIBRRD=-lrrd])
ADDITIONAL_LIBS=
if test "x$ac_cv_lib_rrd_rrd_fetch_r" = xyes; then :
AC_DEFINE_UNQUOTED(HAVE_RRDTOOL, "1", [rrdtool is present])
EXTRA_TARGETS="$EXTRA_TARGETS rrdtool"
fi
ADDITIONAL_LIBS="$LIBRRD $LIBM"
PCAP_HOME=$HOME/PF_RING/userland
DPDK_TARGET=
@@ -134,28 +151,44 @@ fi
case "$host" in
*-*-mingw32*|*-*-msys)
CFLAGS="${CFLAGS} -DOS_WIN32"
LDFLAGS="${LDFLAGS} -lws2_32 -lucrtbase"
PCAP_INC=""
PCAP_LIB=""
case "$host" in
x86_64-*)
BUILD_MINGW_X64=1
;;
i686-*)
;;
*)
AC_MSG_ERROR([Unsupported MingW cross compiler.])
;;
esac
CFLAGS="${CFLAGS}"
LIBS="${LIBS} -lws2_32"
BUILD_MINGW=1
;;
*)
if test -f $PCAP_HOME/libpcap/libpcap.a; then :
echo "Using libpcap from $PCAP_HOME"
PCAP_INC="-I $PCAP_HOME/libpcap"
PCAP_LIB="$PCAP_HOME/libpcap/libpcap.a $PCAP_HOME/lib/libpfring.a $LIBNUMA `$PCAP_HOME/lib/pfring_config --libs`"
AC_CHECK_LIB([rt], [clock_gettime], [PCAP_LIB="$PCAP_LIB -lrt"])
PFRING_LIB=
if test -f $PCAP_HOME/lib/libpfring.a; then :
PFRING_LIB=$PCAP_HOME/lib/libpfring.a
fi
PCAP_LIB="$PCAP_HOME/libpcap/libpcap.a $PFRING_LIB $LIBNUMA `$PCAP_HOME/lib/pfring_config --libs`"
AC_CHECK_LIB([rt], [clock_gettime], [PCAP_LIB="$PCAP_LIB -lrt"])
AC_CHECK_LIB([nl], [nl_handle_alloc], [PCAP_LIB="$PCAP_LIB -lnl"])
# The dlopen() function is in libdl on GLIBC-based systems
# and in the C library for *BSD systems
AC_CHECK_LIB([dl], [dlopen, dlsym], [DL_LIB="-ldl"],[AC_CHECK_LIB([c], [dlopen, dlsym], [DL_LIB="-lc"],[AC_MSG_ERROR([unable to find the dlopen(), dlsym() functions]) ]) ])
else
AC_CHECK_LIB([dl], [dlopen], [DL_LIB="-ldl"],[AC_CHECK_LIB([c],
[dlopen], [DL_LIB="-lc"],
[AC_MSG_ERROR([unable to find the dlopen(), dlsym() functions]) ]) ])
elif test "${with_only_libndpi+set}" != set; then :
AC_CHECK_LIB([pcap], [pcap_open_live], [PCAP_LIB="-lpcap"])
if test $ac_cv_lib_pcap_pcap_open_live = "no"; then :
echo ""
echo "ERROR: Missing libpcap(-dev) library required to compile the example application"
echo "ERROR: Please install it and try again"
exit
fi
if test $ac_cv_lib_pcap_pcap_open_live = "no"; then :
AC_MSG_ERROR([Missing libpcap(-dev) library required to compile the example application.])
fi
fi
;;
esac
@@ -164,35 +197,35 @@ AC_ARG_ENABLE([debug-messages],
AS_HELP_STRING([--enable-debug-messages], [Define NDPI_ENABLE_DEBUG_MESSAGES=1]), [
AC_DEFINE(NDPI_ENABLE_DEBUG_MESSAGES, 1, [Enable ndpi_debug_messages]) ])
@FUZZY@ AS_IF([test "x$enable_fuzztargets" = "xyes"], [
@FUZZY@ AC_PROG_CXX
@FUZZY@ AC_LANG_PUSH(C++)
@FUZZY@ tmp_saved_flags=$[]_AC_LANG_PREFIX[]FLAGS
@FUZZY@ AX_CHECK_COMPILE_FLAG([-fsanitize=fuzzer],,
@FUZZY@ [AC_MSG_ERROR([--enable-fuzztargets requires -fsanitize=fuzzer which is only supported by LLVM])],
@FUZZY@ [-Werror])
@FUZZY@ AS_IF([test "x$LIB_FUZZING_ENGINE" = "x"], [
@FUZZY@ LIB_FUZZING_ENGINE=-fsanitize=fuzzer
@FUZZY@ AC_SUBST(LIB_FUZZING_ENGINE)
@FUZZY@ ])
@FUZZY@ _AC_LANG_PREFIX[]FLAGS="$[]_AC_LANG_PREFIX[]FLAGS $LIB_FUZZING_ENGINE"
@FUZZY@ AC_MSG_CHECKING([whether $CXX accepts $LIB_FUZZING_ENGINE])
@FUZZY@ AC_LINK_IFELSE([AC_LANG_SOURCE([[
@FUZZY@ #include <sys/types.h>
@FUZZY@ extern "C" int LLVMFuzzerTestOneInput(const unsigned char *Data, size_t Size);
@FUZZY@ extern "C" int LLVMFuzzerTestOneInput(const unsigned char *Data, size_t Size) {
@FUZZY@ (void)Data;
@FUZZY@ (void)Size;
@FUZZY@ return 0;
@FUZZY@ }
@FUZZY@ ]])],
@FUZZY@ [ AC_MSG_RESULT(yes)
@FUZZY@ has_sanitizefuzzer=yes],
@FUZZY@ [ AC_MSG_RESULT(no) ]
@FUZZY@ )
@FUZZY@ _AC_LANG_PREFIX[]FLAGS=$tmp_saved_flags
@FUZZY@ AC_LANG_POP()
@FUZZY@ ])
AS_IF([test "x$enable_fuzztargets" = "xyes"], [
AC_PROG_CXX
AC_LANG_PUSH(C++)
tmp_saved_flags=$[]_AC_LANG_PREFIX[]FLAGS
AX_CHECK_COMPILE_FLAG([-fsanitize=fuzzer],,
[AC_MSG_ERROR([--enable-fuzztargets requires -fsanitize=fuzzer which is only supported by LLVM])],
[-Werror])
AS_IF([test "x$LIB_FUZZING_ENGINE" = "x"], [
LIB_FUZZING_ENGINE=-fsanitize=fuzzer
AC_SUBST(LIB_FUZZING_ENGINE)
])
_AC_LANG_PREFIX[]FLAGS="$[]_AC_LANG_PREFIX[]FLAGS $LIB_FUZZING_ENGINE"
AC_MSG_CHECKING([whether $CXX accepts $LIB_FUZZING_ENGINE])
AC_LINK_IFELSE([AC_LANG_SOURCE([[
#include <sys/types.h>
extern "C" int LLVMFuzzerTestOneInput(const unsigned char *Data, size_t Size);
extern "C" int LLVMFuzzerTestOneInput(const unsigned char *Data, size_t Size) {
(void)Data;
(void)Size;
return 0;
}
]])],
[ AC_MSG_RESULT(yes)
has_sanitizefuzzer=yes],
[ AC_MSG_RESULT(no) ]
)
_AC_LANG_PREFIX[]FLAGS=$tmp_saved_flags
AC_LANG_POP()
])
AM_CONDITIONAL([HAS_FUZZLDFLAGS], [test "x$has_sanitizefuzzer" = "xyes"])
AC_CHECK_LIB(pthread, pthread_setaffinity_np, AC_DEFINE_UNQUOTED(HAVE_PTHREAD_SETAFFINITY_NP, 1, [libc has pthread_setaffinity_np]))
@@ -203,7 +236,7 @@ AC_ARG_ENABLE([gcrypt],
[AS_HELP_STRING([--disable-gcrypt], [Avoid compiling with libgcrypt/libgpg-error, even if they are present. QUIC sub-classification may be missing])],
[GCRYPT_ENABLED=0],
[AC_CHECK_LIB(gcrypt, gcry_cipher_checktag)])
AS_IF([test "x$enable_gcrypt" = "xyes"], [
AS_IF([test ${GCRYPT_ENABLED} -eq 1], [
if test "x$ac_cv_lib_gcrypt_gcry_cipher_checktag" = xyes; then :
ADDITIONAL_LIBS="${ADDITIONAL_LIBS} -lgcrypt"
else
@@ -212,16 +245,20 @@ AS_IF([test "x$enable_gcrypt" = "xyes"], [
AC_CHECK_LIB(gcrypt, gcry_cipher_checktag)
if test "x$ac_cv_lib_gcrypt_gcry_cipher_checktag" = xyes -a "x$ac_cv_lib_gpg_error_gpg_strerror_r" = xyes; then :
ADDITIONAL_LIBS="${ADDITIONAL_LIBS} -lgcrypt -lgpg-error"
else
GCRYPT_ENABLED=0
fi
fi
])
dnl> PCRE
PCRE_ENABLED=0
AC_ARG_WITH(pcre, [ --with-pcre Enable nDPI build with libpcre])
if test "${with_pcre+set}" = set; then :
AC_CHECK_LIB(pcre, pcre_compile, AC_DEFINE_UNQUOTED(HAVE_PCRE, 1, [libpcre(-dev) is present]))
if test "x$ac_cv_lib_pcre_pcre_compile" = xyes; then :
ADDITIONAL_LIBS="${ADDITIONAL_LIBS} -lpcre"
PCRE_ENABLED=1
fi
fi
@@ -236,9 +273,17 @@ if test "${with_maxminddb+set}" = set; then :
fi
fi
AC_CONFIG_FILES([Makefile example/Makefile example/Makefile.dpdk tests/Makefile tests/unit/Makefile tests/dga/Makefile libndpi.pc src/include/ndpi_define.h src/lib/Makefile python/Makefile fuzz/Makefile src/include/ndpi_api.h])
dnl> Curl
dnl> AC_CHECK_LIB([curl], [curl_easy_perform])
dnl> if test ${ac_cv_lib_curl_curl_easy_perform} = "no"; then
dnl> echo "Please install libcurl4-openssl-dev (http://curl.haxx.se/) in order to enable curl support"
dnl> else
dnl> ADDITIONAL_LIBS="${ADDITIONAL_LIBS} -lcurl"
dnl> AC_DEFINE_UNQUOTED(HAVE_CURL, 1, [curl is present])
dnl> fi
AC_CONFIG_FILES([Makefile example/Makefile example/Makefile.dpdk tests/Makefile tests/unit/Makefile tests/performance/Makefile tests/dga/Makefile rrdtool/Makefile libndpi.pc src/include/ndpi_define.h src/lib/Makefile python/Makefile fuzz/Makefile src/include/ndpi_api.h])
AC_CONFIG_FILES([tests/do.sh], [chmod +x tests/do.sh])
AC_CONFIG_FILES([tests/do_valgrind.sh], [chmod +x tests/do_valgrind.sh])
AC_CONFIG_HEADERS(src/include/ndpi_config.h)
AC_SUBST(GIT_RELEASE)
AC_SUBST(NDPI_MAJOR)
@@ -255,10 +300,10 @@ AC_SUBST(CUSTOM_NDPI)
AC_SUBST(NDPI_API_VERSION)
AC_SUBST(EXTRA_TARGETS)
AC_SUBST(BUILD_MINGW)
AC_SUBST(BUILD_MINGW_X64)
AC_SUBST(BUILD_FUZZTARGETS)
AC_SUBST(JSONC_CFLAGS)
AC_SUBST(JSONC_LIBS)
AC_SUBST(GCRYPT_ENABLED)
AC_SUBST(GNU_PREFIX)
AC_SUBST(FRAG_MAN_ENABLED)
AC_SUBST(PCRE_ENABLED)
AC_OUTPUT
+231
View File
@@ -0,0 +1,231 @@
# Makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
PAPER =
BUILDDIR = _build
# User-friendly check for sphinx-build
ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
endif
# Internal variables.
PAPEROPT_a4 = -D latex_paper_size=a4
PAPEROPT_letter = -D latex_paper_size=letter
ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
# the i18n builder cannot share the environment and doctrees with the others
I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
.PHONY: help
help:
@echo "Please use \`make <target>' where <target> is one of"
@echo " release to make standalone HTML files for ntop documentation release"
@echo " html to make standalone HTML files"
@echo " dirhtml to make HTML files named index.html in directories"
@echo " singlehtml to make a single large HTML file"
@echo " pickle to make pickle files"
@echo " json to make JSON files"
@echo " htmlhelp to make HTML files and a HTML help project"
@echo " qthelp to make HTML files and a qthelp project"
@echo " applehelp to make an Apple Help Book"
@echo " devhelp to make HTML files and a Devhelp project"
@echo " epub to make an epub"
@echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
@echo " latexpdf to make LaTeX files and run them through pdflatex"
@echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
@echo " text to make text files"
@echo " man to make manual pages"
@echo " texinfo to make Texinfo files"
@echo " info to make Texinfo files and run them through makeinfo"
@echo " gettext to make PO message catalogs"
@echo " changes to make an overview of all changed/added/deprecated items"
@echo " xml to make Docutils-native XML files"
@echo " pseudoxml to make pseudoxml-XML files for display purposes"
@echo " linkcheck to check all external links for integrity"
@echo " doctest to run all doctests embedded in the documentation (if enabled)"
@echo " coverage to run coverage check of the documentation (if enabled)"
.PHONY: clean
clean:
rm -rf $(BUILDDIR)/*
cd doxygen; make clean
.PHONY: html
html:
cd doxygen; make
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
.PHONY: release
release:
make clean
@{ \
git_root=$$(git rev-parse --show-toplevel);\
version=$$(grep -Po '^NTOPNG_SHORT_VERSION="\K.+(?=")' $${git_root}/configure.ac);\
echo Release version $${version};\
sed -i "s/version = u'.*'/version = u'$${version}'/" conf.py;\
sed -i "s/release = u'.*'/release = u'$${version}'/" conf.py;\
}
make html
.PHONY: dirhtml
dirhtml:
$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
.PHONY: singlehtml
singlehtml:
$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
@echo
@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
.PHONY: pickle
pickle:
$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
@echo
@echo "Build finished; now you can process the pickle files."
.PHONY: json
json:
$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
@echo
@echo "Build finished; now you can process the JSON files."
.PHONY: htmlhelp
htmlhelp:
$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
@echo
@echo "Build finished; now you can run HTML Help Workshop with the" \
".hhp project file in $(BUILDDIR)/htmlhelp."
.PHONY: qthelp
qthelp:
$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
@echo
@echo "Build finished; now you can run "qcollectiongenerator" with the" \
".qhcp project file in $(BUILDDIR)/qthelp, like this:"
@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/ntop.qhcp"
@echo "To view the help file:"
@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/ntop.qhc"
.PHONY: applehelp
applehelp:
$(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp
@echo
@echo "Build finished. The help book is in $(BUILDDIR)/applehelp."
@echo "N.B. You won't be able to view it unless you put it in" \
"~/Library/Documentation/Help or install it in your application" \
"bundle."
.PHONY: devhelp
devhelp:
$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
@echo
@echo "Build finished."
@echo "To view the help file:"
@echo "# mkdir -p $$HOME/.local/share/devhelp/ntop"
@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/ntop"
@echo "# devhelp"
.PHONY: epub
epub:
$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
@echo
@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
.PHONY: latex
latex:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo
@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
@echo "Run \`make' in that directory to run these through (pdf)latex" \
"(use \`make latexpdf' here to do that automatically)."
.PHONY: latexpdf
latexpdf:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through pdflatex..."
$(MAKE) -C $(BUILDDIR)/latex all-pdf
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
.PHONY: latexpdfja
latexpdfja:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through platex and dvipdfmx..."
$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
.PHONY: text
text:
$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
@echo
@echo "Build finished. The text files are in $(BUILDDIR)/text."
.PHONY: man
man:
$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
@echo
@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
.PHONY: texinfo
texinfo:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo
@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
@echo "Run \`make' in that directory to run these through makeinfo" \
"(use \`make info' here to do that automatically)."
.PHONY: info
info:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo "Running Texinfo files through makeinfo..."
make -C $(BUILDDIR)/texinfo info
@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
.PHONY: gettext
gettext:
$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
@echo
@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
.PHONY: changes
changes:
$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
@echo
@echo "The overview file is in $(BUILDDIR)/changes."
.PHONY: linkcheck
linkcheck:
$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
@echo
@echo "Link check complete; look for any errors in the above output " \
"or in $(BUILDDIR)/linkcheck/output.txt."
.PHONY: doctest
doctest:
$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
@echo "Testing of doctests in the sources finished, look at the " \
"results in $(BUILDDIR)/doctest/output.txt."
.PHONY: coverage
coverage:
$(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage
@echo "Testing of coverage in the sources finished, look at the " \
"results in $(BUILDDIR)/coverage/python.txt."
.PHONY: xml
xml:
$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
@echo
@echo "Build finished. The XML files are in $(BUILDDIR)/xml."
.PHONY: pseudoxml
pseudoxml:
$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
@echo
@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."
+4
View File
@@ -0,0 +1,4 @@
nDPI Programming API
####################
Placeholder
+291
View File
@@ -0,0 +1,291 @@
# -*- coding: utf-8 -*-
import sys
import os
import sphinx_rtd_theme
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#sys.path.insert(0, os.path.abspath('.'))
sys.path.append( "breathe/" )
# -- General configuration ------------------------------------------------
# If documentation needs a minimal Sphinx version, state it here.
#needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.intersphinx',
'breathe'
]
# Workaround for platforms where swaggerdoc is not available
if not os.environ.get("PYTHON_SKIP_SWAGGERDOC"):
extensions.append('sphinxcontrib.swaggerdoc')
breathe_projects = { "apidoc" : "doxygen/xml/" }
breathe_default_project = "apidoc"
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
source_suffix = ['.rst', '.md']
#source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'nDPI'
copyright = u'2021, ntop and contributors'
author = u'ntop.org'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = u'4.1'
# The full version, including alpha/beta/rc tags.
release = u'4.1'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['_build', 'README.md', 'example.rst']
# The reST default role (used for this markup: `text`) to use for all
# documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
#modindex_common_prefix = []
# If true, keep warnings as "system message" paragraphs in the built documents.
#keep_warnings = False
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#html_theme = 'alabaster'
html_theme = "sphinx_rtd_theme"
#html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
#html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
html_logo = "img/logo.png"
# The name of an image file (relative to this directory) to use as a favicon of
# the docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
# directly to the root of the documentation.
#html_extra_path = []
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
#html_domain_indices = True
# If false, no index is generated.
#html_use_index = True
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None
# Language to be used for generating the HTML full-text search index.
# Sphinx supports the following languages:
# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'
# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr'
#html_search_language = 'en'
# A dictionary with options for the search language support, empty by default.
# Now only 'ja' uses this config value
#html_search_options = {'type': 'default'}
# The name of a javascript file (relative to the configuration directory) that
# implements a search results scorer. If empty, the default will be used.
#html_search_scorer = 'scorer.js'
# Output file base name for HTML help builder.
htmlhelp_basename = 'ntopdoc'
# -- Options for LaTeX output ---------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#'preamble': '',
# Latex figure (float) alignment
#'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'nDPI.tex', u'nDPI Documentation',
u'ntop', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False
# If true, show page references after internal links.
#latex_show_pagerefs = False
# If true, show URL addresses after external links.
#latex_show_urls = False
# Documents to append as an appendix to all manuals.
#latex_appendices = []
# If false, no module index is generated.
#latex_domain_indices = True
# -- Options for manual page output ---------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'nDPI', u'nDPI Documentation',
[author], 1)
]
# If true, show URL addresses after external links.
#man_show_urls = False
# -- Options for Texinfo output -------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'nDPI', u'nDPI Documentation',
author, 'ntop and contributors', 'Deep Packet Inspection toolkit',
'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
#texinfo_appendices = []
# If false, no module index is generated.
#texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
#texinfo_show_urls = 'footnote'
# If true, do not generate a @detailmenu in the "Top" node's menu.
#texinfo_no_detailmenu = False
intersphinx_mapping = {'https://docs.python.org/': None}
def setup(app):
app.add_stylesheet('css/ntop.css')
+255
View File
@@ -0,0 +1,255 @@
nDPI Flow Risks
###############
nDPI is designed not just to detect application protocols in traffic flows but also to evaluate potential security risks associated with the traffic. In nDPI parlance this is called a "flow risk". Flows can have multiple risks detected hence nDPI reports them with a bitmap. Each risk detected corresponds to a bit in the flow risk bitmap. You can read more about `ndpi_risk_enum <https://github.com/ntop/nDPI/blob/dev/src/include/ndpi_typedefs.h>`_ for the list of all numeric risks currently supported.
Below you can find a description of each flow risk so that you can easily understand when a risk is triggered and its meaning. The flow risks are listed in numerical order as they are defined in ndpi_risk_enum.
.. _Risk 001:
NDPI_URL_POSSIBLE_XSS
=====================
HTTP only: this risk indicates a possible `XSS (Cross Side Scripting) <https://en.wikipedia.org/wiki/Cross-site_scripting>`_ attack.
.. _Risk 002:
NDPI_URL_POSSIBLE_SQL_INJECTION
===============================
HTTP only: this risk indicates a possible `SQL Injection attack <https://en.wikipedia.org/wiki/SQL_injection>`_.
.. _Risk 003:
NDPI_URL_POSSIBLE_RCE_INJECTION
===============================
HTTP only: this risk indicates a possible `RCE (Remote Code Execution) attack <https://en.wikipedia.org/wiki/Arbitrary_code_execution>`_.
.. _Risk 004:
NDPI_BINARY_APPLICATION_TRANSFER
================================
HTTP only: this risk indicates that a binary application is downloaded/uploaded. Detected applications include Windows binaries, Linux executables, Unix scripts and Android apps.
.. _Risk 005:
NDPI_KNOWN_PROTOCOL_ON_NON_STANDARD_PORT
========================================
This risk indicates a known protocol used on a non standard port. Example HTTP is supposed to use TCP/80, and in case it is detected on TCP/1234 this risk is detected.
.. _Risk 006:
NDPI_TLS_SELFSIGNED_CERTIFICATE
===============================
TLS/QUIC only: this risk is triggered when a `self-signed certificate <https://en.wikipedia.org/wiki/Self-signed_certificate>`_ is used.
.. _Risk 007:
NDPI_TLS_OBSOLETE_VERSION
=========================
Risk 0triggered when TLS version is older than 1.1.
.. _Risk 008:
NDPI_TLS_WEAK_CIPHER
====================
Risk 0triggered when an unsafe TLS cipher is used. See `this page <https://community.qualys.com/thread/18212-how-does-qualys-determine-the-server-cipher-suites>`_ for a list of insecure ciphers.
.. _Risk 009:
NDPI_TLS_CERTIFICATE_EXPIRED
============================
Risk 0triggered when a TLS certificate is expired, i.e. the current date falls outside of the certificate validity dates.
.. _Risk 010:
NDPI_TLS_CERTIFICATE_MISMATCH
=============================
Risk 0triggered when a TLS certificate does not match the hostname we're accessing. Example you do http://www.aaa.com and the TLS certificate returned is for www.bbb.com.
.. _Risk 011:
NDPI_HTTP_SUSPICIOUS_USER_AGENT
===============================
HTTP only: this risk is triggered whenever the user agent contains suspicious characters or its format is suspicious. Example: <?php something ?> is a typical suspicious user agent.
.. _Risk 012:
NDPI_HTTP_NUMERIC_IP_HOST
=========================
HTTP only: this risk is triggered whenever we're accessing a host using its IP rather than its symbolic name. Example http://1.2.3.4.
.. _Risk 013:
NDPI_HTTP_SUSPICIOUS_URL
========================
HTTP only: this risk is triggered whenever the accessed URL is suspicious. Example: http://127.0.0.1/msadc/..%255c../..%255c../..%255c../winnt/system32/cmd.exe.
.. _Risk 014:
NDPI_HTTP_SUSPICIOUS_HEADER
===========================
HTTP only: this risk is triggered whenever the HTTP peader contains suspicious entries such as Uuid, TLS_version, Osname that are unexpected on the HTTP header.
.. _Risk 015:
NDPI_TLS_NOT_CARRYING_HTTPS
===========================
TLS only: this risk indicates that this TLS flow will not be used to transport HTTP content. Example VPNs use TLS to encrypt data rather to carry HTTP. This is useful to spot this type of cases.
.. _Risk 016:
NDPI_SUSPICIOUS_DGA_DOMAIN
==========================
A `DGA <https://en.wikipedia.org/wiki/Domain_generation_algorithm>`_ is used to generate domain names often used by malwares. This risk indicates that this domain name can (but it's not 100% sure) a DGA as its name is suspicious.
.. _Risk 017:
NDPI_MALFORMED_PACKET
=====================
This risk is generated when a packet (e.g. a DNS packet) has an unexpected formt. This can indicate a protocol error or more often an attempt to jeopardize a valid protocol to carry other type of data.
.. _Risk 018:
NDPI_SSH_OBSOLETE_CLIENT_VERSION_OR_CIPHER
==========================================
This risk is generated whenever a SSH client uses an obsolete SSH protocol version or insecure ciphers.
.. _Risk 019:
NDPI_SSH_OBSOLETE_SERVER_VERSION_OR_CIPHER
==========================================
This risk is generated whenever a SSH server uses an obsolete SSH protocol version or insecure ciphers.
.. _Risk 020:
NDPI_SMB_INSECURE_VERSION
=========================
This risk indicates that the `SMB <https://en.wikipedia.org/wiki/Server_Message_Block>`_ version used is insecure (i.e. v1).
.. _Risk 021:
NDPI_TLS_SUSPICIOUS_ESNI_USAGE
==============================
`SNI <https://en.wikipedia.org/wiki/Server_Name_Indication>`_ is a way to carry in TLS the host/domain name we're accessing. ESNI means encrypted SNI and it is a way to mask SNI (carried in clear text in the TLS header) with encryption. While this practice is legal, it could be used for hiding data or for attacks such as a suspicious `domain fronting <https://github.com/SixGenInc/Noctilucent/blob/master/docs/>`_.
.. _Risk 022:
NDPI_UNSAFE_PROTOCOL
====================
This risk indicates that the protocol used is insecure and that a secure protocol should be used (e.g. Telnet vs SSH).
.. _Risk 023:
NDPI_DNS_SUSPICIOUS_TRAFFIC
===========================
This risk is returned when DNS traffic returns an unexpected/obsolete `record type <https://en.wikipedia.org/wiki/List_of_DNS_record_types>`_.
.. _Risk 024:
NDPI_TLS_MISSING_SNI
====================
TLS needs to carry the the `SNI <https://en.wikipedia.org/wiki/Server_Name_Indication>`_ of the remote server we're accessing. Unfortunately SNI is optional in TLS so it can be omitted. In this case this risk is triggered as this is a non-standard situation that indicates a potential security problem or a protocol using TLS for other purposes (or a protocol bug).
.. _Risk 025:
NDPI_HTTP_SUSPICIOUS_CONTENT
============================
HTTP only: risk reported when HTTP carries content in expected format. Example the HTTP header indicates that the context is text/html but the real content is not readeable (i.e. it can transport binary data). In general this is an attempt to use a valid MIME type to carry data that does not match the type.
.. _Risk 026:
NDPI_RISKY_ASN
==============
This is a placeholder for traffic exchanged with `ASN <https://en.wikipedia.org/wiki/Autonomous_system_(Internet)>`_ that are considered risky. nDPI does not fill this risk that instead should be filled by aplications sitting on top of nDPI (e.g. ntopng).
.. _Risk 027:
NDPI_RISKY_DOMAIN
=================
This is a placeholder for traffic exchanged with domain names that are considered risky. nDPI does not fill this risk that instead should be filled by aplications sitting on top of nDPI (e.g. ntopng).
.. _Risk 028:
NDPI_MALICIOUS_JA3
==================
`JA3 <https://engineering.salesforce.com/tls-fingerprinting-with-ja3-and-ja3s-247362855967>`_ is a method to fingerprint TLS traffic. This risk indicates that the JA3 of the TLS connection is considered suspicious (i.e. it has been found in known malware JA3 blacklists). nDPI does not fill this risk that instead should be filled by aplications sitting on top of nDPI (e.g. ntopng).
.. _Risk 029:
NDPI_MALICIOUS_SHA1_CERTIFICATE
===============================
TLS certificates are uniquely identified with a `SHA1 <https://en.wikipedia.org/wiki/SHA-1>`_ hash value. If such hash is found on a blacklist, this risk can be used. As for other risks, this is a placeholder as nDPI does not fill this risk that instead should be filled by aplications sitting on top of nDPI (e.g. ntopng).
.. _Risk 030:
NDPI_DESKTOP_OR_FILE_SHARING_SESSION
====================================
This risk is set when the flow carries desktop or file sharing sessions (e.g. TeamViewer or AnyDesk just to mention two).
.. _Risk 031:
NDPI_TLS_UNCOMMON_ALPN
======================
This risk is set when the `ALPN <https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation>`_ (it indicates the protocol carried into this TLS flow, for instance HTTP/1.1) is uncommon with respect to the list of expected values.
.. _Risk 032:
NDPI_TLS_CERT_VALIDITY_TOO_LONG
===============================
From 01/09/2020 TLS certificates lifespan is limited to 13 months. This risk is triggered for certificates not respecting this directive.
.. _Risk 033:
NDPI_TLS_SUSPICIOUS_EXTENSION
=============================
This risk is triggered when the domain name (SNI extension) is not printable and thus it is a problem.
.. _Risk 034:
NDPI_TLS_FATAL_ALERT
====================
This risk is triggered when a TLS fatal alert is detected in the TLS flow. See `this page <https://techcommunity.microsoft.com/t5/iis-support-blog/ssl-tls-alert-protocol-and-the-alert-codes/ba-p/377132>`_ for details.
.. _Risk 035:
NDPI_SUSPICIOUS_ENTROPY
=======================
This risk is used to detect suspicious data carried in ICMP packets whose entropy (used to measure how data is distributed, hence to indirectly guess the type of data carried on) is suspicious and thus that it can indicate a data leak.
.. _Risk 036:
NDPI_CLEAR_TEXT_CREDENTIALS
===========================
Clear text protocols are not bad per-se, but they should be avoided when they carry credentials as they can be intercepted by malicious users. This risk is triggered whenever clear text protocols (e.g. FTP, HTTP, IMAP...) contain credentials in clear text (read it as nDPI does not trigger this risk for HTTP connections that do not carry credentials).
.. _Risk 037:
NDPI_DNS_LARGE_PACKET
=====================
`DNS <https://en.wikipedia.org/wiki/Domain_Name_System>`_ packets over UDP should be limited to 512 bytes. DNS packets over this threshold indicate a potential security risk (e.g. use DNS to carry data) or a misconfiguration.
.. _Risk 038:
NDPI_DNS_FRAGMENTED
===================
UDP `DNS <https://en.wikipedia.org/wiki/Domain_Name_System>`_ packets cannot be fragmented. If so, this indicates a potential security risk (e.g. use DNS to carry data) or a misconfiguration.
.. _Risk 039:
NDPI_INVALID_CHARACTERS
=======================
The risk is set whenever a dissected protocol contains characters not allowed in that protocol field.
For example a DNS hostname must only contain a subset of all printable characters or else this risk is set.
Additionally, some TLS protocol fields are checked for printable characters as well.
.. _Risk 040:
NDPI_POSSIBLE_EXPLOIT
=====================
The risk is set whenever a possible exploit (e.g. Log4J/Log4Shell) is detected.
.. _Risk 041:
NDPI_TLS_CERTIFICATE_ABOUT_TO_EXPIRE
===================================
The risk is set whenever a TLS certificate is close to the expiration date.
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 47 KiB

+37
View File
@@ -0,0 +1,37 @@
nDPI Documentation
====================
nDPI is an open source DPI (Deep Packet Inspection) toolkit for traffic analysis.
.. toctree::
:maxdepth: 2
:caption: User's Guide
what_is_ndpi
.. toctree::
:maxdepth: 2
:caption: Developer's Guide
api
flow_risks
.. toctree::
:caption: Other Products
ntopng <https://www.ntop.org/guides/ntopng/>
nProbe <https://www.ntop.org/guides/nprobe/>
nProbe Cento <https://www.ntop.org/guides/cento/>
n2disk <https://www.ntop.org/guides/n2disk/>
PF_RING <https://www.ntop.org/guides/pf_ring/>
nEdge <https://www.ntop.org/guides/nedge/>
nScrub <https://www.ntop.org/guides/nscrub/>
nBox <https://www.ntop.org/guides/nbox/>
.. Indices and tables
.. ==================
..
.. * :ref:`genindex`
.. * :ref:`modindex`
.. * :ref:`search`
+56
View File
@@ -0,0 +1,56 @@
What is nDPI
##############
nDPI is a DPI (Deep Packet Inspection) toolkit able to:
- Detect application protocol on traffic flows
- Analize encrypted traffic flows via ET (Encrypted Traffic Analysis)
- Extract selected protocol metadata from traffic
- Implement APIs for analysing traffic
Releases and Features
---------------------
nDPI development lifecycle is typically 6 to 9 months. The history of changes and features implemented by every release, is available on its `Changelog <https://github.com/ntop/nDPI/blob/dev/CHANGELOG.md>`_.
Installation
============
nDPI is open source and available on `GitHub
<https://github.com/ntop/nDPI>`_. In addition, pre-compiled, binary
nDPI packages are available both for Linux and other platforms. Installation
instructions for binary packages are available below.
Installing on Linux
-------------------
Installation instructions can be found at
http://packages.ntop.org/. Development and stable builds are
available. Stable builds are intended for production environments whereas
development builds are intended for testing or early feature access.
Software Updates
================
General instructions for updating the software can be found at
http://packages.ntop.org/ together with the installation instructions.
Depending on the Operating System, nDPI supports also automatic updates
through the GUI as described in the below sections.
Updating the Software on Linux
------------------------------
Instructions for updating the software via command line can be found
at http://packages.ntop.org/. For example on Ubuntu/Debian systems the
below commands will update the repository, check for updates and install
the latest software update if any:
.. code:: bash
apt-get update
apt-get upgrade
.. _AvailableVersions:
+1 -1
View File
@@ -21,7 +21,7 @@ SRCS-y := reader_util.c intrusion_detection.c ndpiReader.c
CFLAGS += -g
CFLAGS += -Wno-strict-prototypes -Wno-missing-prototypes -Wno-missing-declarations -Wno-unused-parameter -I $(PWD)/../src/include @CFLAGS@ -DUSE_DPDK
LDLIBS = $(LIBNDPI) @PCAP_LIB@ @LIBS@ -lpthread -lm @LDFLAGS@
LDLIBS = $(LIBNDPI) @PCAP_LIB@ @LIBS@ @ADDITIONAL_LIBS@ -lpthread @LDFLAGS@
include $(RTE_SDK)/mk/rte.extapp.mk
+29 -16
View File
@@ -1,55 +1,70 @@
AR=@AR@
CC=@CC@
CXX=@CXX@
BUILD_MINGW=@BUILD_MINGW@
BUILD_MINGW_X64=@BUILD_MINGW_X64@
SRCHOME=../src
CFLAGS=-g -fPIC -DPIC -I$(SRCHOME)/include @PCAP_INC@ @CFLAGS@
LDFLAGS=@LDFLAGS@
LIBNDPI=$(SRCHOME)/lib/libndpi.a
LDFLAGS=$(LIBNDPI) @PCAP_LIB@ @LIBS@ @ADDITIONAL_LIBS@ -lpthread -lm @LDFLAGS@
LIBS=$(LIBNDPI) @PCAP_LIB@ @LIBS@ @ADDITIONAL_LIBS@
HEADERS=intrusion_detection.h reader_util.h $(SRCHOME)/include/ndpi_api.h \
$(SRCHOME)/include/ndpi_typedefs.h $(SRCHOME)/include/ndpi_protocol_ids.h
OBJS=ndpiReader.o reader_util.o intrusion_detection.o
PREFIX?=@prefix@
ifneq ($(BUILD_MINGW),)
all:
@echo 'Examples disabled due to mingw build.'
SUFFIX:=.exe
CFLAGS+=-I@srcdir@/../windows/WpdPack/Include -I@srcdir@/../windows/WpdPack/Include/pcap
LIBS+=-Wl,-Bstatic -lpthread -Wl,-Bdynamic
ifneq ($(BUILD_MINGW_X64),)
LIBS+=@srcdir@/../windows/WpdPack/Lib/x64/wpcap.lib
else
LIBS+=@srcdir@/../windows/WpdPack/Lib/wpcap.lib
endif
else
all: ndpiReader @DPDK_TARGET@
LIBS+=-lpthread
endif
all: ndpiReader$(SUFFIX) @DPDK_TARGET@
EXECUTABLE_SOURCES := ndpiReader.c ndpiSimpleIntegration.c
COMMON_SOURCES := $(filter-out $(EXECUTABLE_SOURCES),$(wildcard *.c ))
libndpiReader.a: $(COMMON_SOURCES:%.c=%.o) $(LIBNDPI)
ar rsv libndpiReader.a $(COMMON_SOURCES:%.c=%.o)
$(AR) rsv libndpiReader.a $(COMMON_SOURCES:%.c=%.o)
ndpiReader: libndpiReader.a $(LIBNDPI) ndpiReader.o
$(CC) $(CFLAGS) ndpiReader.o libndpiReader.a -o $@ $(LDFLAGS)
ndpiReader$(SUFFIX): libndpiReader.a $(LIBNDPI) ndpiReader.o
$(CC) $(CFLAGS) $(LDFLAGS) ndpiReader.o libndpiReader.a $(LIBS) -o $@
ndpiSimpleIntegration: ndpiSimpleIntegration.o
$(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
ndpiSimpleIntegration$(SUFFIX): ndpiSimpleIntegration.o
$(CC) $(CFLAGS) $(LDFLAGS) $< $(LIBS) -o $@
%.o: %.c $(HEADERS) Makefile
$(CC) $(CFLAGS) -c $< -o $@
$(CC) $(CFLAGS) $(CPPFLAGS) -c $< -o $@
install: ndpiReader
install: ndpiReader$(SUFFIX)
mkdir -p $(DESTDIR)$(PREFIX)/bin/
mkdir -p $(DESTDIR)$(PREFIX)/share/ndpi
cp ndpiReader $(DESTDIR)$(PREFIX)/bin/
cp ndpiReader$(SUFFIX) $(DESTDIR)$(PREFIX)/bin/
cp protos.txt $(DESTDIR)$(PREFIX)/share/ndpi/ndpiProtos.txt
cp mining_hosts.txt $(DESTDIR)$(PREFIX)/share/ndpi/ndpiCustomCategory.txt
[ -f ndpiSimpleIntegration$(SUFFIX) ] && cp ndpiSimpleIntegration$(SUFFIX) $(DESTDIR)$(PREFIX)/bin/ndpiSimpleIntegration$(SUFFIX) || true
[ -f build/app/ndpiReader.dpdk ] && cp build/app/ndpiReader.dpdk $(DESTDIR)$(PREFIX)/bin/ || true
[ -f ndpiReader.dpdk ] && cp ndpiReader.dpdk $(DESTDIR)$(PREFIX)/bin/ || true
dpdk:
make -f Makefile.dpdk
$(MAKE) -f Makefile.dpdk
check:
cppcheck --template='{file}:{line}:{severity}:{message}' --quiet --enable=all --force -I$(SRCHOME)/include *.c
clean:
/bin/rm -f *.o ndpiReader ndpiReader.dpdk
/bin/rm -f *.o ndpiReader ndpiSimpleIntegration ndpiReader$(SUFFIX) ndpiSimpleIntegration$(SUFFIX) ndpiReader.dpdk
/bin/rm -f .*.dpdk.cmd .*.o.cmd *.dpdk.map .*.o.d
/bin/rm -f _install _postbuild _postinstall _preinstall
/bin/rm -rf build
@@ -61,5 +76,3 @@ distdir:
distclean: clean
/bin/rm -f Makefile.dpdk
/bin/rm -f Makefile
endif
-20
View File
@@ -1,20 +0,0 @@
Microsoft Visual Studio Solution File, Format Version 11.00
# Visual C++ Express 2010
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "pcapExample", "pcapExample\pcapExample.vcxproj", "{F6A2C0AE-2110-438A-87E4-7C1CFCE064C6}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Win32 = Debug|Win32
Release|Win32 = Release|Win32
EndGlobalSection
GlobalSection(ProjectConfigurationPlatforms) = postSolution
{F6A2C0AE-2110-438A-87E4-7C1CFCE064C6}.Debug|Win32.ActiveCfg = Debug|Win32
{F6A2C0AE-2110-438A-87E4-7C1CFCE064C6}.Debug|Win32.Build.0 = Debug|Win32
{F6A2C0AE-2110-438A-87E4-7C1CFCE064C6}.Release|Win32.ActiveCfg = Release|Win32
{F6A2C0AE-2110-438A-87E4-7C1CFCE064C6}.Release|Win32.Build.0 = Release|Win32
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
EndGlobalSection
EndGlobal
Binary file not shown.
-11
View File
@@ -1,11 +0,0 @@
// pcapExample.cpp : Defines the entry point for the console application.
//
#include "stdafx.h"
int _tmain(int argc, _TCHAR* argv[])
{
return 0;
}
@@ -1,231 +0,0 @@
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup Label="ProjectConfigurations">
<ProjectConfiguration Include="Debug|Win32">
<Configuration>Debug</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|Win32">
<Configuration>Release</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
</ItemGroup>
<PropertyGroup Label="Globals">
<ProjectGuid>{F6A2C0AE-2110-438A-87E4-7C1CFCE064C6}</ProjectGuid>
<Keyword>Win32Proj</Keyword>
<RootNamespace>pcapExample</RootNamespace>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" Label="Configuration">
<ConfigurationType>Application</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
<ImportGroup Label="ExtensionSettings">
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<PropertyGroup Label="UserMacros" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<LinkIncremental>true</LinkIncremental>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<LinkIncremental>false</LinkIncremental>
</PropertyGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<ClCompile>
<PrecompiledHeader>NotUsing</PrecompiledHeader>
<WarningLevel>Level3</WarningLevel>
<Optimization>Disabled</Optimization>
<PreprocessorDefinitions>WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<AdditionalIncludeDirectories>C:\ntop\nprobe\nprobe-win-32;C:\ntop\winpcap\Include;C:\ntop\dependencies\nDPI\src\lib\third_party\include;C:\ntop\dependencies\nDPI\src\include;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<AdditionalDependencies>Wsock32.lib;Ws2_32.lib;C:\ntop\winpcap\Lib\Packet.lib;C:\ntop\winpcap\Lib\wpcap.lib;%(AdditionalDependencies)</AdditionalDependencies>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<ClCompile>
<WarningLevel>Level3</WarningLevel>
<PrecompiledHeader>Use</PrecompiledHeader>
<Optimization>MaxSpeed</Optimization>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<PreprocessorDefinitions>WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
</Link>
</ItemDefinitionGroup>
<ItemGroup>
<ClCompile Include="..\..\..\..\..\nprobe\nprobe-win-32\getopt.c" />
<ClCompile Include="..\..\..\..\..\nprobe\nprobe-win-32\getopt1.c" />
<ClCompile Include="..\..\..\src\lib\ndpi_main.c" />
<ClCompile Include="..\..\..\src\lib\protocols\afp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\aimini.c" />
<ClCompile Include="..\..\..\src\lib\protocols\applejuice.c" />
<ClCompile Include="..\..\..\src\lib\protocols\armagetron.c" />
<ClCompile Include="..\..\..\src\lib\protocols\battlefield.c" />
<ClCompile Include="..\..\..\src\lib\protocols\bgp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\bittorrent.c" />
<ClCompile Include="..\..\..\src\lib\protocols\ciscovpn.c" />
<ClCompile Include="..\..\..\src\lib\protocols\citrix.c" />
<ClCompile Include="..\..\..\src\lib\protocols\corba.c" />
<ClCompile Include="..\..\..\src\lib\protocols\crossfire.c" />
<ClCompile Include="..\..\..\src\lib\protocols\dcerpc.c" />
<ClCompile Include="..\..\..\src\lib\protocols\dhcp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\dhcpv6.c" />
<ClCompile Include="..\..\..\src\lib\protocols\directconnect.c" />
<ClCompile Include="..\..\..\src\lib\protocols\directdownloadlink.c" />
<ClCompile Include="..\..\..\src\lib\protocols\dns.c" />
<ClCompile Include="..\..\..\src\lib\protocols\dofus.c" />
<ClCompile Include="..\..\..\src\lib\protocols\dropbox.c" />
<ClCompile Include="..\..\..\src\lib\protocols\edonkey.c" />
<ClCompile Include="..\..\..\src\lib\protocols\fasttrack.c" />
<ClCompile Include="..\..\..\src\lib\protocols\feidian.c" />
<ClCompile Include="..\..\..\src\lib\protocols\fiesta.c" />
<ClCompile Include="..\..\..\src\lib\protocols\filetopia.c" />
<ClCompile Include="..\..\..\src\lib\protocols\flash.c" />
<ClCompile Include="..\..\..\src\lib\protocols\florensia.c" />
<ClCompile Include="..\..\..\src\lib\protocols\ftp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\gadu_gadu.c" />
<ClCompile Include="..\..\..\src\lib\protocols\gnutella.c" />
<ClCompile Include="..\..\..\src\lib\protocols\gtp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\guildwars.c" />
<ClCompile Include="..\..\..\src\lib\protocols\h323.c" />
<ClCompile Include="..\..\..\src\lib\protocols\halflife2_and_mods.c" />
<ClCompile Include="..\..\..\src\lib\protocols\http.c" />
<ClCompile Include="..\..\..\src\lib\protocols\http_activesync.c" />
<ClCompile Include="..\..\..\src\lib\protocols\i23v5.c" />
<ClCompile Include="..\..\..\src\lib\protocols\iax.c" />
<ClCompile Include="..\..\..\src\lib\protocols\icecast.c" />
<ClCompile Include="..\..\..\src\lib\protocols\imesh.c" />
<ClCompile Include="..\..\..\src\lib\protocols\ipp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\irc.c" />
<ClCompile Include="..\..\..\src\lib\protocols\jabber.c" />
<ClCompile Include="..\..\..\src\lib\protocols\kerberos.c" />
<ClCompile Include="..\..\..\src\lib\protocols\kontiki.c" />
<ClCompile Include="..\..\..\src\lib\protocols\ldap.c" />
<ClCompile Include="..\..\..\src\lib\protocols\lotus_notes.c" />
<ClCompile Include="..\..\..\src\lib\protocols\mail_imap.c" />
<ClCompile Include="..\..\..\src\lib\protocols\mail_pop.c" />
<ClCompile Include="..\..\..\src\lib\protocols\mail_smtp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\maplestory.c" />
<ClCompile Include="..\..\..\src\lib\protocols\mdns.c" />
<ClCompile Include="..\..\..\src\lib\protocols\meebo.c" />
<ClCompile Include="..\..\..\src\lib\protocols\mgcp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\mms.c" />
<ClCompile Include="..\..\..\src\lib\protocols\msn.c" />
<ClCompile Include="..\..\..\src\lib\protocols\mssql.c" />
<ClCompile Include="..\..\..\src\lib\protocols\mysql.c" />
<ClCompile Include="..\..\..\src\lib\protocols\netbios.c" />
<ClCompile Include="..\..\..\src\lib\protocols\netflow.c" />
<ClCompile Include="..\..\..\src\lib\protocols\nfs.c" />
<ClCompile Include="..\..\..\src\lib\protocols\noe.c" />
<ClCompile Include="..\..\..\src\lib\protocols\non_tcp_udp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\ntp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\openft.c" />
<ClCompile Include="..\..\..\src\lib\protocols\openvpn.c" />
<ClCompile Include="..\..\..\src\lib\protocols\oracle.c" />
<ClCompile Include="..\..\..\src\lib\protocols\oscar.c" />
<ClCompile Include="..\..\..\src\lib\protocols\pando.c" />
<ClCompile Include="..\..\..\src\lib\protocols\pcanywhere.c" />
<ClCompile Include="..\..\..\src\lib\protocols\popo.c" />
<ClCompile Include="..\..\..\src\lib\protocols\postgres.c" />
<ClCompile Include="..\..\..\src\lib\protocols\pplive.c" />
<ClCompile Include="..\..\..\src\lib\protocols\ppstream.c" />
<ClCompile Include="..\..\..\src\lib\protocols\pptp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\qq.c" />
<ClCompile Include="..\..\..\src\lib\protocols\quake.c" />
<ClCompile Include="..\..\..\src\lib\protocols\quic.c" />
<ClCompile Include="..\..\..\src\lib\protocols\radius.c" />
<ClCompile Include="..\..\..\src\lib\protocols\rdp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\rsync.c" />
<ClCompile Include="..\..\..\src\lib\protocols\rtcp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\rtp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\rtsp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\sflow.c" />
<ClCompile Include="..\..\..\src\lib\protocols\shoutcast.c" />
<ClCompile Include="..\..\..\src\lib\protocols\sip.c" />
<ClCompile Include="..\..\..\src\lib\protocols\skinny.c" />
<ClCompile Include="..\..\..\src\lib\protocols\skype.c" />
<ClCompile Include="..\..\..\src\lib\protocols\smb.c" />
<ClCompile Include="..\..\..\src\lib\protocols\snmp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\socrates.c" />
<ClCompile Include="..\..\..\src\lib\protocols\sopcast.c" />
<ClCompile Include="..\..\..\src\lib\protocols\soulseek.c" />
<ClCompile Include="..\..\..\src\lib\protocols\spotify.c" />
<ClCompile Include="..\..\..\src\lib\protocols\ssdp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\ssh.c" />
<ClCompile Include="..\..\..\src\lib\protocols\ssl.c" />
<ClCompile Include="..\..\..\src\lib\protocols\stealthnet.c" />
<ClCompile Include="..\..\..\src\lib\protocols\steam.c" />
<ClCompile Include="..\..\..\src\lib\protocols\stun.c" />
<ClCompile Include="..\..\..\src\lib\protocols\syslog.c" />
<ClCompile Include="..\..\..\src\lib\protocols\tcp_udp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\tds.c" />
<ClCompile Include="..\..\..\src\lib\protocols\teamspeak.c" />
<ClCompile Include="..\..\..\src\lib\protocols\teamviewer.c" />
<ClCompile Include="..\..\..\src\lib\protocols\telnet.c" />
<ClCompile Include="..\..\..\src\lib\protocols\tftp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\thunder.c" />
<ClCompile Include="..\..\..\src\lib\protocols\tor.c" />
<ClCompile Include="..\..\..\src\lib\protocols\tvants.c" />
<ClCompile Include="..\..\..\src\lib\protocols\tvuplayer.c" />
<ClCompile Include="..\..\..\src\lib\protocols\usenet.c" />
<ClCompile Include="..\..\..\src\lib\protocols\veohtv.c" />
<ClCompile Include="..\..\..\src\lib\protocols\viber.c" />
<ClCompile Include="..\..\..\src\lib\protocols\vmware.c" />
<ClCompile Include="..\..\..\src\lib\protocols\vnc.c" />
<ClCompile Include="..\..\..\src\lib\protocols\warcraft3.c" />
<ClCompile Include="..\..\..\src\lib\protocols\winmx.c" />
<ClCompile Include="..\..\..\src\lib\protocols\world_of_kung_fu.c" />
<ClCompile Include="..\..\..\src\lib\protocols\world_of_warcraft.c" />
<ClCompile Include="..\..\..\src\lib\protocols\xbox.c" />
<ClCompile Include="..\..\..\src\lib\protocols\xdmcp.c" />
<ClCompile Include="..\..\..\src\lib\protocols\yahoo.c" />
<ClCompile Include="..\..\..\src\lib\protocols\zattoo.c" />
<ClCompile Include="..\..\..\src\lib\third_party\src\ahocorasick.c" />
<ClCompile Include="..\..\..\src\lib\third_party\src\node.c" />
<ClCompile Include="..\..\..\src\lib\third_party\src\sort.c" />
<ClCompile Include="..\..\pcapReader.c" />
</ItemGroup>
<ItemGroup>
<ClInclude Include="..\..\..\..\..\nprobe\nprobe-win-32\getopt.h" />
<ClInclude Include="..\..\..\src\include\linux_compat.h" />
<ClInclude Include="..\..\..\src\include\ndpi_api.h" />
<ClInclude Include="..\..\..\src\include\ndpi_debug_functions.h" />
<ClInclude Include="..\..\..\src\include\ndpi_define.h" />
<ClInclude Include="..\..\..\src\include\ndpi_macros.h" />
<ClInclude Include="..\..\..\src\include\ndpi_main.h" />
<ClInclude Include="..\..\..\src\include\ndpi_protocols.h" />
<ClInclude Include="..\..\..\src\include\ndpi_protocols_osdpi.h" />
<ClInclude Include="..\..\..\src\include\ndpi_protocol_history.h" />
<ClInclude Include="..\..\..\src\include\ndpi_public_functions.h" />
<ClInclude Include="..\..\..\src\include\ndpi_structs.h" />
<ClInclude Include="..\..\..\src\include\ndpi_utils.h" />
<ClInclude Include="..\..\..\src\lib\third_party\include\actypes.h" />
<ClInclude Include="..\..\..\src\lib\third_party\include\ahocorasick.h" />
<ClInclude Include="..\..\..\src\lib\third_party\include\node.h" />
<ClInclude Include="..\..\..\src\lib\third_party\include\sort.h" />
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets">
</ImportGroup>
</Project>
@@ -1,465 +0,0 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<Filter Include="Source Files">
<UniqueIdentifier>{4FC737F1-C7A5-4376-A066-2A32D752A2FF}</UniqueIdentifier>
<Extensions>cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx</Extensions>
</Filter>
<Filter Include="Header Files">
<UniqueIdentifier>{93995380-89BD-4b04-88EB-625FBE52EBFB}</UniqueIdentifier>
<Extensions>h;hpp;hxx;hm;inl;inc;xsd</Extensions>
</Filter>
<Filter Include="Resource Files">
<UniqueIdentifier>{67DA6AB6-F800-4c08-8B7A-83BB121AAD01}</UniqueIdentifier>
<Extensions>rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav;mfcribbon-ms</Extensions>
</Filter>
</ItemGroup>
<ItemGroup>
<ClCompile Include="..\..\pcapReader.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\ndpi_main.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\zattoo.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\afp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\aimini.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\applejuice.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\armagetron.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\battlefield.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\bgp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\bittorrent.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\ciscovpn.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\citrix.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\corba.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\crossfire.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\dcerpc.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\dhcp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\dhcpv6.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\directconnect.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\directdownloadlink.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\dns.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\dofus.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\dropbox.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\edonkey.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\fasttrack.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\feidian.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\fiesta.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\filetopia.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\flash.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\florensia.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\ftp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\gadu_gadu.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\gnutella.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\gtp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\guildwars.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\h323.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\halflife2_and_mods.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\http.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\http_activesync.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\i23v5.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\iax.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\icecast.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\imesh.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\ipp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\irc.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\jabber.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\kerberos.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\kontiki.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\ldap.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\lotus_notes.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\mail_imap.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\mail_pop.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\mail_smtp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\maplestory.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\mdns.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\meebo.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\mgcp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\mms.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\msn.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\mssql.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\mysql.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\netbios.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\netflow.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\nfs.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\noe.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\non_tcp_udp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\ntp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\openft.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\openvpn.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\oracle.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\oscar.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\pando.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\pcanywhere.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\popo.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\postgres.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\pplive.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\ppstream.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\pptp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\qq.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\quake.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\quic.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\radius.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\rdp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\rsync.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\rtcp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\rtp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\rtsp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\sflow.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\shoutcast.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\sip.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\skinny.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\skype.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\smb.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\snmp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\socrates.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\sopcast.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\soulseek.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\spotify.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\ssdp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\ssh.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\ssl.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\stealthnet.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\steam.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\stun.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\syslog.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\tcp_udp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\tds.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\teamspeak.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\teamviewer.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\telnet.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\tftp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\thunder.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\tor.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\tvants.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\tvuplayer.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\usenet.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\veohtv.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\viber.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\vmware.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\vnc.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\warcraft3.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\winmx.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\world_of_kung_fu.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\world_of_warcraft.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\xbox.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\xdmcp.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\protocols\yahoo.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\third_party\src\ahocorasick.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\third_party\src\node.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\src\lib\third_party\src\sort.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\..\..\nprobe\nprobe-win-32\getopt.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\..\..\..\..\nprobe\nprobe-win-32\getopt1.c">
<Filter>Source Files</Filter>
</ClCompile>
</ItemGroup>
<ItemGroup>
<ClInclude Include="..\..\..\src\lib\third_party\include\actypes.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="..\..\..\src\lib\third_party\include\ahocorasick.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="..\..\..\src\lib\third_party\include\node.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="..\..\..\src\lib\third_party\include\sort.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="..\..\..\src\include\ndpi_api.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="..\..\..\src\include\ndpi_debug_functions.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="..\..\..\src\include\ndpi_define.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="..\..\..\src\include\ndpi_macros.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="..\..\..\src\include\ndpi_main.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="..\..\..\src\include\ndpi_protocol_history.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="..\..\..\src\include\ndpi_protocols.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="..\..\..\src\include\ndpi_protocols_osdpi.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="..\..\..\src\include\ndpi_public_functions.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="..\..\..\src\include\ndpi_structs.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="..\..\..\src\include\ndpi_utils.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="..\..\..\src\include\linux_compat.h">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="..\..\..\..\..\nprobe\nprobe-win-32\getopt.h">
<Filter>Header Files</Filter>
</ClInclude>
</ItemGroup>
</Project>
@@ -1,3 +0,0 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
</Project>
-462
View File
@@ -1,462 +0,0 @@
/*
* intrusion_detection.c
*
* Copyright (C) 2011-21 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
*
* nDPI is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* nDPI is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public License
* along with nDPI. If not, see <http://www.gnu.org/licenses/>.
*
*/
#include "intrusion_detection.h"
double normalize(ndpi_norm_value* tresholds){
if(tresholds->upper_bound != tresholds->lower_bound){
tresholds->norm_value = (tresholds->value - tresholds->lower_bound) / (tresholds->upper_bound - tresholds->lower_bound);
}else{
if(tresholds->value > tresholds->upper_bound){
tresholds->norm_value = 1 + (tresholds->value - tresholds->lower_bound) / tresholds->upper_bound;
}else{
tresholds->norm_value = 1 - (tresholds->value - tresholds->lower_bound) / tresholds->upper_bound;
}
}
if(tresholds->norm_value >= 0){
return tresholds->norm_value * tresholds->weight;
}
else{
return (1 - tresholds->norm_value) * tresholds->weight;
}
}
double get_flow_score(ndpi_norm_value* scores, int n_metrics){
double flow_score = 0;
for(int i=0; i<n_metrics; i++){
flow_score += normalize(&scores[i]);
}
return flow_score;
}
/* ********************************** */
double Ddos_score(struct ndpi_flow_info* flow){
int n_metrics = 6;
ndpi_norm_value* scores = malloc(n_metrics * sizeof(ndpi_norm_value));
/* pktlen_c_to_s_avg */
int i = 0;
scores[i].lower_bound = 70.0;
scores[i].upper_bound = 263.4799999999999;
scores[i].weight = 0.21257330032661592;
scores[i].value = ndpi_data_average(flow->pktlen_c_to_s);
/* pktlen_s_to_c_max */
i++;
scores[i].lower_bound = 90.0;
scores[i].upper_bound = 2974.0;
scores[i].weight = 0.21073785073559176;
scores[i].value = ndpi_data_max(flow->pktlen_s_to_c);
/* pktlen_s_to_c_avg */
i++;
scores[i].lower_bound = 72.7;
scores[i].upper_bound = 1130.4199999999996;
scores[i].weight = 0.21257330032661592;
scores[i].value = ndpi_data_average(flow->pktlen_s_to_c);
/* pktlen_s_to_c_stddev */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 906.0;
scores[i].weight = 0.20990954527912953;
scores[i].value = ndpi_data_stddev(flow->pktlen_s_to_c);
/* fin */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 2.0;
scores[i].weight = 0.07710300166602348;
scores[i].value = flow->fin_count;
/* s_to_c_fin */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 2.0;
scores[i].weight = 0.07710300166602348;
scores[i].value = flow->dst2src_fin_count;
// sum = 1.0
double flow_score = get_flow_score(scores, n_metrics);
free(scores);
return flow_score;
}
double Dos_goldeneye_score(struct ndpi_flow_info* flow){
int n_metrics = 6;
ndpi_norm_value* scores = malloc(n_metrics * sizeof(ndpi_norm_value));
/* pktlen_s_to_c_max */
int i = 0;
scores[i].lower_bound = 74.0;
scores[i].upper_bound = 3292.6699999999764;
scores[i].weight = 0.3123007140611667;
scores[i].value = ndpi_data_max(flow->pktlen_s_to_c);
/* pktlen_s_to_c_avg */
i++;
scores[i].lower_bound = 68.7;
scores[i].upper_bound = 1354.0569999999987;
scores[i].weight = 0.23802038891633356;
scores[i].value = ndpi_data_average(flow->pktlen_s_to_c);
/* pktlen_s_to_c_stddev */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 959.4469999999993;
scores[i].weight = 0.3111779763775991;
scores[i].value = ndpi_data_stddev(flow->pktlen_s_to_c);
/* syn */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 2.0;
scores[i].weight = 0.0464364305923564;
scores[i].value = flow->syn_count;
/* c_to_s_syn */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 1.0;
scores[i].weight = 0.04562805946018772;
scores[i].value = flow->src2dst_syn_count;
/* s_to_c_syn */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 2.0;
scores[i].weight = 0.0464364305923564;
scores[i].value = flow->dst2src_syn_count;
// sum = 0.9999999999999998
double flow_score = get_flow_score(scores, n_metrics);
free(scores);
return flow_score;
}
double Dos_hulk_score(struct ndpi_flow_info* flow){
double f = (double)flow->first_seen_ms/1000.0, l = (double)flow->last_seen_ms/1000.0;
int n_metrics = 6;
ndpi_norm_value* scores = malloc(n_metrics * sizeof(ndpi_norm_value));
/* duration */
int i = 0;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 539.40668006422;
scores[i].weight = 0.16666666666666666;
scores[i].value = (l - f);
/* src2dst_packets */
i++;
scores[i].lower_bound = 2.0;
scores[i].upper_bound = 41.0;
scores[i].weight = 0.16666666666666666;
scores[i].value = flow->src2dst_packets;
/* dst2src_packets */
i++;
scores[i].lower_bound = 2.0;
scores[i].upper_bound = 45.0;
scores[i].weight = 0.16666666666666666;
scores[i].value = flow->dst2src_packets;
/* src2dst_bytes */
i++;
scores[i].lower_bound = 146.0;
scores[i].upper_bound = 6306.300000000001;
scores[i].weight = 0.16666666666666666;
scores[i].value = flow->src2dst_bytes;
/* ack */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 82.0;
scores[i].weight = 0.16666666666666666;
scores[i].value = flow->ack_count;
/* syn */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 2.0;
scores[i].weight = 0.16666666666666666;
scores[i].value = flow->syn_count;
// sum = 0.9999999999999999
double flow_score = get_flow_score(scores, n_metrics);
free(scores);
return flow_score;
}
double Dos_slow_score(struct ndpi_flow_info* flow){
int n_metrics = 6;
ndpi_norm_value* scores = malloc(n_metrics * sizeof(ndpi_norm_value));
/* pktlen_s_to_c_max */
int i = 0;
scores[i].lower_bound = 90.0;
scores[i].upper_bound = 3135.0;
scores[i].weight = 0.1760747755022144;
scores[i].value = ndpi_data_max(flow->pktlen_s_to_c);
/* pktlen_s_to_c_avg */
i++;
scores[i].lower_bound = 80.37100000000001;
scores[i].upper_bound = 1292.5900000000008;
scores[i].weight = 0.17600137023171597;
scores[i].value = ndpi_data_average(flow->pktlen_s_to_c);
/* dst2src_bytes */
i++;
scores[i].lower_bound = 262.0;
scores[i].upper_bound = 53227.80000000002;
scores[i].weight = 0.16919914849886225;
scores[i].value = flow->dst2src_bytes;
/* syn */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 2.0;
scores[i].weight = 0.168000195747388;
scores[i].value = flow->syn_count;
/* c_to_s_syn */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 1.0;
scores[i].weight = 0.14272431427243143;
scores[i].value = flow->src2dst_syn_count;
/* s_to_c_syn */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 2.0;
scores[i].weight = 0.168000195747388;
scores[i].value = flow->dst2src_syn_count;
// sum = 1.0
double flow_score = get_flow_score(scores, n_metrics);
free(scores);
return flow_score;
}
double Ftp_patator_score(struct ndpi_flow_info* flow){
int n_metrics = 6;
ndpi_norm_value* scores = malloc(n_metrics * sizeof(ndpi_norm_value));
/* iat_flow_min */
int i = 0;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 24.0;
scores[i].weight = 0.002732919254658385;
scores[i].value = ndpi_data_min(flow->iat_flow);
/* pktlen_s_to_c_max */
i++;
scores[i].lower_bound = 90.0;
scores[i].upper_bound = 3393.0;
scores[i].weight = 0.007453416149068323;
scores[i].value = ndpi_data_max(flow->pktlen_s_to_c);
/* pktlen_s_to_c_avg */
i++;
scores[i].lower_bound = 81.3;
scores[i].upper_bound = 1315.021;
scores[i].weight = 0.9833540372670807;
scores[i].value = ndpi_data_average(flow->pktlen_s_to_c);
/* dst2src_bytes */
i++;
scores[i].lower_bound = 256.0;
scores[i].upper_bound = 56434.0;
scores[i].weight = 0.0034782608695652175;
scores[i].value = flow->dst2src_bytes;
/* fin */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 2.0;
scores[i].weight = 0.0014906832298136647;
scores[i].value = flow->fin_count;
/* rst */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 2.0;
scores[i].weight = 0.0014906832298136647;
scores[i].value = flow->rst_count;
// sum = 1.0
double flow_score = get_flow_score(scores, n_metrics);
free(scores);
return flow_score;
}
double Hearthbleed_score(struct ndpi_flow_info* flow){
double f = (double)flow->first_seen_ms/1000.0, l = (double)flow->last_seen_ms/1000.0;
int n_metrics = 6;
ndpi_norm_value* scores = malloc(n_metrics * sizeof(ndpi_norm_value));
/* iat_flow_max */
int i = 0;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 595213.3999999999;
scores[i].weight = 0.16666666666666666;
scores[i].value = ndpi_data_max(flow->iat_flow);
/* iat_flow_stddev */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 245377.74799999973;
scores[i].weight = 0.16666666666666666;
scores[i].value = ndpi_data_stddev(flow->iat_flow);
/* pktlen_s_to_c_max */
i++;
scores[i].lower_bound = 74.0;
scores[i].upper_bound = 3380.0;
scores[i].weight = 0.16666666666666666;
scores[i].value = ndpi_data_max(flow->pktlen_s_to_c);
/* pktlen_s_to_c_avg */
i++;
scores[i].lower_bound = 70.0;
scores[i].upper_bound = 1344.6399999999996;
scores[i].weight = 0.16666666666666666;
scores[i].value = ndpi_data_average(flow->pktlen_s_to_c);
/* pktlen_s_to_c_stddev */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 944.6399999999996;
scores[i].weight = 0.16666666666666666;
scores[i].value = ndpi_data_stddev(flow->pktlen_s_to_c);
/* duration */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 711.6677598000391;
scores[i].weight = 0.16666666666666666;
scores[i].value = (l - f);
// sum = 0.9999999999999999
double flow_score = get_flow_score(scores, n_metrics);
free(scores);
return flow_score;
}
double Infiltration_score(struct ndpi_flow_info* flow){
int n_metrics = 6;
ndpi_norm_value* scores = malloc(n_metrics * sizeof(ndpi_norm_value));
/* pktlen_c_to_s_max */
int i = 0;
scores[i].lower_bound = 72.0;
scores[i].upper_bound = 1840.739999999998;
scores[i].weight = 0.11937557392102846;
scores[i].value = ndpi_data_max(flow->pktlen_c_to_s);
/* pktlen_c_to_s_avg */
i++;
scores[i].lower_bound = 70.0;
scores[i].upper_bound = 296.56599999999816;
scores[i].weight = 0.12526782981328435;
scores[i].value = ndpi_data_average(flow->pktlen_c_to_s);
/* pktlen_s_to_c_max */
i++;
scores[i].lower_bound = 90.0;
scores[i].upper_bound = 3496.1399999999776;
scores[i].weight = 0.13927150290786652;
scores[i].value = ndpi_data_max(flow->pktlen_s_to_c);
/* pktlen_s_to_c_avg */
i++;
scores[i].lower_bound = 72.6;
scores[i].upper_bound = 1367.7959999999991;
scores[i].weight = 0.12182430364248545;
scores[i].value = ndpi_data_average(flow->pktlen_s_to_c);
/* src2dst_bytes */
i++;
scores[i].lower_bound = 144.0;
scores[i].upper_bound = 7847.69999999999;
scores[i].weight = 0.12059993878175697;
scores[i].value = flow->src2dst_bytes;
/* dst2src_bytes */
i++;
scores[i].lower_bound = 236.0;
scores[i].upper_bound = 74486.7799999998;
scores[i].weight = 0.3736608509335782;
scores[i].value = flow->dst2src_bytes;
// sum = 1.0
double flow_score = get_flow_score(scores, n_metrics);
free(scores);
return flow_score;
}
double Ssh_patator_score(struct ndpi_flow_info* flow){
int n_metrics = 6;
ndpi_norm_value* scores = malloc(n_metrics * sizeof(ndpi_norm_value));
/* fin */
int i = 0;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 2.0;
scores[i].weight = 0.0033738191632928477;
scores[i].value = flow->fin_count;
/* psh */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 30.0;
scores[i].weight = 0.33076923076923076;
scores[i].value = flow->psh_count;
/* c_to_s_syn */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 1.0;
scores[i].weight = 0.0004048582995951417;
scores[i].value = flow->src2dst_syn_count;
/* c_to_s_psh */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 12.0;
scores[i].weight = 0.33130904183535764;
scores[i].value = flow->src2dst_psh_count;
/* s_to_c_fin */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 2.0;
scores[i].weight = 0.0033738191632928477;
scores[i].value = flow->dst2src_fin_count;
/* s_to_c_psh */
i++;
scores[i].lower_bound = 0.0;
scores[i].upper_bound = 30.0;
scores[i].weight = 0.33076923076923076;
scores[i].value = flow->dst2src_psh_count;
// sum = 1.0
double flow_score = get_flow_score(scores, n_metrics);
free(scores);
return flow_score;
}
-69
View File
@@ -1,69 +0,0 @@
/*
* intrusion_detection.h
*
* Copyright (C) 2011-21 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
*
* nDPI is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* nDPI is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public License
* along with nDPI. If not, see <http://www.gnu.org/licenses/>.
*
*/
#ifndef _INTRUSION_DETECTION_H_
#define _INTRUSION_DETECTION_H_
/*
Code to detect attacks reported in
https://www.unb.ca/cic/datasets/ids-2017.html
https://www.unb.ca/cic/datasets/ids-2018.html
*/
#include <stdio.h>
#include <stdlib.h>
#include "reader_util.h"
#include "ndpi_api.h"
typedef struct norm_values{
double upper_bound;
double lower_bound;
double weight;
double value;
double norm_value;
}ndpi_norm_value;
double normalize(ndpi_norm_value* tresholds);
double get_flow_score(ndpi_norm_value* scores, int n_metrics);
/* ********************************** */
double Ddos_score(struct ndpi_flow_info* flow);
double Dos_goldeneye_score(struct ndpi_flow_info* flow);
double Dos_hulk_score(struct ndpi_flow_info* flow);
double Dos_slow_score(struct ndpi_flow_info* flow);
double Ftp_patator_score(struct ndpi_flow_info* flow);
double Hearthbleed_score(struct ndpi_flow_info* flow);
double Infiltration_score(struct ndpi_flow_info* flow);
double Ssh_patator_score(struct ndpi_flow_info* flow);
#endif /* _INTRUSION_DETECTION_H_ */
+217 -170
View File
@@ -1,7 +1,7 @@
/*
* ndpiReader.c
*
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* nDPI is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published by
@@ -37,6 +37,8 @@
#else
#include <unistd.h>
#include <netinet/in.h>
#include <sys/socket.h>
#include <sys/mman.h>
#endif
#include <string.h>
#include <stdarg.h>
@@ -44,7 +46,6 @@
#include <pcap.h>
#include <signal.h>
#include <pthread.h>
#include <sys/socket.h>
#include <assert.h>
#include <math.h>
#include "ndpi_api.h"
@@ -52,11 +53,9 @@
#include "../src/lib/third_party/include/ahocorasick.h"
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <libgen.h>
#include "reader_util.h"
#include "intrusion_detection.h"
#define ntohl64(x) ( ( (uint64_t)(ntohl( (uint32_t)((x << 32) >> 32) )) << 32) | ntohl( ((uint32_t)(x >> 32)) ) )
#define htonl64(x) ntohl64(x)
@@ -66,7 +65,9 @@
/** Client parameters **/
static char *_pcap_file[MAX_NUM_READER_THREADS]; /**< Ingress pcap file/interfaces */
#ifndef USE_DPDK
static FILE *playlist_fp[MAX_NUM_READER_THREADS] = { NULL }; /**< Ingress playlist */
#endif
static FILE *results_file = NULL;
static char *results_path = NULL;
static char * bpfFilter = NULL; /**< bpf filter */
@@ -82,7 +83,7 @@ static char* domain_to_check = NULL;
static u_int8_t ignore_vlanid = 0;
/** User preferences **/
u_int8_t enable_protocol_guess = 1, enable_payload_analyzer = 0, num_bin_clusters = 0, extcap_exit = 0;
u_int8_t verbose = 0, enable_joy_stats = 0;
u_int8_t verbose = 0, enable_flow_stats = 0;
int nDPI_LogLevel = 0;
char *_debug_protocols = NULL;
u_int8_t human_readeable_string_len = 5;
@@ -99,7 +100,10 @@ static struct timeval startup_time, begin, end;
static int core_affinity[MAX_NUM_READER_THREADS];
#endif
static struct timeval pcap_start = { 0, 0}, pcap_end = { 0, 0 };
static struct bpf_program bpf_code,*bpf_cfilter = NULL;
#ifndef USE_DPDK
static struct bpf_program bpf_code;
#endif
static struct bpf_program *bpf_cfilter = NULL;
/** Detection parameters **/
static time_t capture_for = 0;
static time_t capture_until = 0;
@@ -261,7 +265,7 @@ u_int check_bin_doh_similarity(struct ndpi_bin *bin, float *similarity) {
float lowest_similarity = 9999999999.0f;
for(i=0; i<NUM_DOH_BINS; i++) {
*similarity = ndpi_bin_similarity(&doh_ndpi_bins[i], bin, 0);
*similarity = ndpi_bin_similarity(&doh_ndpi_bins[i], bin, 0, 0);
if(*similarity <= doh_max_distance)
return(1);
@@ -280,7 +284,6 @@ void ndpiCheckHostStringMatch(char *testChar) {
ndpi_protocol_match_result match = { NDPI_PROTOCOL_UNKNOWN,
NDPI_PROTOCOL_CATEGORY_UNSPECIFIED, NDPI_PROTOCOL_UNRATED };
int testRes;
u_int8_t is_host_match = 1;
char appBufStr[64];
ndpi_protocol detected_protocol;
struct ndpi_detection_module_struct *ndpi_str;
@@ -297,8 +300,7 @@ void ndpiCheckHostStringMatch(char *testChar) {
// ac_automata_display( module->host_automa.ac_automa, 'n');
testRes = ndpi_match_string_subprotocol(ndpi_str,
testChar, strlen(testChar), &match,
is_host_match);
testChar, strlen(testChar), &match);
if(testRes) {
memset( &detected_protocol, 0, sizeof(ndpi_protocol) );
@@ -329,29 +331,6 @@ void ndpiCheckHostStringMatch(char *testChar) {
*/
static void setupDetection(u_int16_t thread_id, pcap_t * pcap_handle);
#if 0
static void reduceBDbits(uint32_t *bd, unsigned int len) {
int mask = 0;
int shift = 0;
unsigned int i = 0;
for(i = 0; i < len; i++)
mask = mask | bd[i];
mask = mask >> 8;
for(i = 0; i < 24 && mask; i++) {
mask = mask >> 1;
if (mask == 0) {
shift = i+1;
break;
}
}
for(i = 0; i < len; i++)
bd[i] = bd[i] >> shift;
}
#endif
/**
* @brief Get flow byte distribution mean and variance
*/
@@ -362,66 +341,56 @@ flowGetBDMeanandVariance(struct ndpi_flow_info* flow) {
uint32_t tmp[256], i;
unsigned int num_bytes;
double mean = 0.0, variance = 0.0;
struct ndpi_entropy last_entropy = flow->last_entropy;
struct ndpi_entropy *last_entropy = flow->last_entropy;
fflush(out);
if(!last_entropy)
return;
/*
* Sum up the byte_count array for outbound and inbound flows,
* if this flow is bidirectional
*/
if (!flow->bidirectional) {
array = last_entropy.src2dst_byte_count;
num_bytes = last_entropy.src2dst_l4_bytes;
array = last_entropy->src2dst_byte_count;
num_bytes = last_entropy->src2dst_l4_bytes;
for (i=0; i<256; i++) {
tmp[i] = last_entropy.src2dst_byte_count[i];
tmp[i] = last_entropy->src2dst_byte_count[i];
}
if (last_entropy.src2dst_num_bytes != 0) {
mean = last_entropy.src2dst_bd_mean;
variance = last_entropy.src2dst_bd_variance/(last_entropy.src2dst_num_bytes - 1);
if (last_entropy->src2dst_num_bytes != 0) {
mean = last_entropy->src2dst_bd_mean;
variance = last_entropy->src2dst_bd_variance/(last_entropy->src2dst_num_bytes - 1);
variance = sqrt(variance);
if (last_entropy.src2dst_num_bytes == 1) {
if (last_entropy->src2dst_num_bytes == 1) {
variance = 0.0;
}
}
} else {
for (i=0; i<256; i++) {
tmp[i] = last_entropy.src2dst_byte_count[i] + last_entropy.dst2src_byte_count[i];
tmp[i] = last_entropy->src2dst_byte_count[i] + last_entropy->dst2src_byte_count[i];
}
array = tmp;
num_bytes = last_entropy.src2dst_l4_bytes + last_entropy.dst2src_l4_bytes;
num_bytes = last_entropy->src2dst_l4_bytes + last_entropy->dst2src_l4_bytes;
if (last_entropy.src2dst_num_bytes + last_entropy.dst2src_num_bytes != 0) {
mean = ((double)last_entropy.src2dst_num_bytes)/((double)(last_entropy.src2dst_num_bytes+last_entropy.dst2src_num_bytes))*last_entropy.src2dst_bd_mean +
((double)last_entropy.dst2src_num_bytes)/((double)(last_entropy.dst2src_num_bytes+last_entropy.src2dst_num_bytes))*last_entropy.dst2src_bd_mean;
if (last_entropy->src2dst_num_bytes + last_entropy->dst2src_num_bytes != 0) {
mean = ((double)last_entropy->src2dst_num_bytes)/((double)(last_entropy->src2dst_num_bytes+last_entropy->dst2src_num_bytes))*last_entropy->src2dst_bd_mean +
((double)last_entropy->dst2src_num_bytes)/((double)(last_entropy->dst2src_num_bytes+last_entropy->src2dst_num_bytes))*last_entropy->dst2src_bd_mean;
variance = ((double)last_entropy.src2dst_num_bytes)/((double)(last_entropy.src2dst_num_bytes+last_entropy.dst2src_num_bytes))*last_entropy.src2dst_bd_variance +
((double)last_entropy.dst2src_num_bytes)/((double)(last_entropy.dst2src_num_bytes+last_entropy.src2dst_num_bytes))*last_entropy.dst2src_bd_variance;
variance = ((double)last_entropy->src2dst_num_bytes)/((double)(last_entropy->src2dst_num_bytes+last_entropy->dst2src_num_bytes))*last_entropy->src2dst_bd_variance +
((double)last_entropy->dst2src_num_bytes)/((double)(last_entropy->dst2src_num_bytes+last_entropy->src2dst_num_bytes))*last_entropy->dst2src_bd_variance;
variance = variance/((double)(last_entropy.src2dst_num_bytes + last_entropy.dst2src_num_bytes - 1));
variance = variance/((double)(last_entropy->src2dst_num_bytes + last_entropy->dst2src_num_bytes - 1));
variance = sqrt(variance);
if (last_entropy.src2dst_num_bytes + last_entropy.dst2src_num_bytes == 1) {
if (last_entropy->src2dst_num_bytes + last_entropy->dst2src_num_bytes == 1) {
variance = 0.0;
}
}
}
if(enable_joy_stats) {
#if 0
if(verbose > 1) {
reduceBDbits(tmp, 256);
array = tmp;
fprintf(out, " [byte_dist: ");
for(i = 0; i < 255; i++)
fprintf(out, "%u,", (unsigned char)array[i]);
fprintf(out, "%u]", (unsigned char)array[i]);
}
#endif
if(enable_flow_stats) {
/* Output the mean */
if(num_bytes != 0) {
double entropy = ndpi_flow_get_byte_count_entropy(array, num_bytes);
@@ -454,7 +423,8 @@ static void help(u_int long_help) {
"[-f <filter>][-s <duration>][-m <duration>][-b <num bin clusters>]\n"
" [-p <protos>][-l <loops> [-q][-d][-J][-h][-D][-e <len>][-t][-v <level>]\n"
" [-n <threads>][-w <file>][-c <file>][-C <file>][-j <file>][-x <file>]\n"
" [-r <file>][-j <file>][-S <file>][-T <num>][-U <num>] [-x <domain>][-z]\n\n"
" [-r <file>][-j <file>][-S <file>][-T <num>][-U <num>] [-x <domain>][-z]\n"
" [-a <mode>]\n\n"
"Usage:\n"
" -i <file.pcap|device> | Specify a pcap file/playlist to read packets from or a\n"
" | device for live capture (comma-separated list)\n"
@@ -469,11 +439,14 @@ static void help(u_int long_help) {
#ifdef linux
" -g <id:id...> | Thread affinity mask (one core id per thread)\n"
#endif
" -a <mode> | Generates option values for GUIs\n"
" | 0 - List known protocols\n"
" | 1 - List known categories\n"
" | 2 - List known risks\n"
" -d | Disable protocol guess and use only DPI\n"
" -e <len> | Min human readeable string match len. Default %u\n"
" -q | Quiet mode\n"
" -J | Display flow SPLT (sequence of packet length and time)\n"
" | and BD (byte distribution). See https://github.com/cisco/joy\n"
" -F | Enable flow stats\n"
" -t | Dissect GTP/TZSP tunnels\n"
" -P <a>:<b>:<c>:<d>:<e> | Enable payload analysis:\n"
" | <a> = min pattern len to search\n"
@@ -564,6 +537,7 @@ static struct option longopts[] = {
{ "csv-dump", required_argument, NULL, 'C'},
{ "interface", required_argument, NULL, 'i'},
{ "filter", required_argument, NULL, 'f'},
{ "flow-stats", required_argument, NULL, 'F'},
{ "cpu-bind", required_argument, NULL, 'g'},
{ "loops", required_argument, NULL, 'l'},
{ "num-threads", required_argument, NULL, 'n'},
@@ -578,7 +552,6 @@ static struct option longopts[] = {
{ "ndpi-log-level", required_argument, NULL, 'V'},
{ "dbg-proto", required_argument, NULL, 'u'},
{ "help", no_argument, NULL, 'h'},
{ "joy", required_argument, NULL, 'J'},
{ "payload-analysis", required_argument, NULL, 'P'},
{ "result-path", required_argument, NULL, 'w'},
{ "quiet", no_argument, NULL, 'q'},
@@ -731,8 +704,7 @@ void extcap_capture() {
void printCSVHeader() {
if(!csv_fp) return;
fprintf(csv_fp, "#flow_id,protocol,first_seen,last_seen,duration,src_ip,src_port,dst_ip,dst_port,ndpi_proto_num,ndpi_proto,server_name,");
fprintf(csv_fp, "benign_score,dos_slow_score,dos_goldeneye_score,dos_hulk_score,ddos_score,hearthbleed_score,ftp_patator_score,ssh_patator_score,infiltration_score,");
fprintf(csv_fp, "#flow_id,protocol,first_seen,last_seen,duration,src_ip,src_port,dst_ip,dst_port,ndpi_proto_num,ndpi_proto,server_name_sni,");
fprintf(csv_fp, "c_to_s_pkts,c_to_s_bytes,c_to_s_goodput_bytes,s_to_c_pkts,s_to_c_bytes,s_to_c_goodput_bytes,");
fprintf(csv_fp, "data_ratio,str_data_ratio,c_to_s_goodput_ratio,s_to_c_goodput_ratio,");
@@ -756,7 +728,7 @@ void printCSVHeader() {
fprintf(csv_fp, "c_to_s_init_win,s_to_c_init_win,");
/* Flow info */
fprintf(csv_fp, "client_info,server_info,");
fprintf(csv_fp, "server_info,");
fprintf(csv_fp, "tls_version,ja3c,tls_client_unsafe,");
fprintf(csv_fp, "ja3s,tls_server_unsafe,");
fprintf(csv_fp, "tls_alpn,tls_supported_versions,");
@@ -765,8 +737,7 @@ void printCSVHeader() {
#endif
fprintf(csv_fp, "ssh_client_hassh,ssh_server_hassh,flow_info,plen_bins");
/* Joy */
if(enable_joy_stats) {
if(enable_flow_stats) {
fprintf(csv_fp, ",byte_dist_mean,byte_dist_std,entropy,total_entropy");
}
@@ -779,12 +750,16 @@ void printCSVHeader() {
* @brief Option parser
*/
static void parseOptions(int argc, char **argv) {
int option_idx = 0, do_capture = 0;
char *__pcap_file = NULL, *bind_mask = NULL;
int thread_id, opt;
int option_idx = 0;
int opt;
#ifndef USE_DPDK
char *__pcap_file = NULL;
int thread_id, do_capture = 0;
#ifdef linux
char *bind_mask = NULL;
u_int num_cores = sysconf(_SC_NPROCESSORS_ONLN);
#endif
#endif
#ifdef USE_DPDK
{
@@ -797,13 +772,17 @@ static void parseOptions(int argc, char **argv) {
}
#endif
while((opt = getopt_long(argc, argv, "b:e:c:C:dDf:g:i:Ij:S:hp:pP:l:r:s:tu:v:V:n:Jrp:x:w:zq0123:456:7:89:m:T:U:",
while((opt = getopt_long(argc, argv, "a:b:e:c:C:dDf:g:i:Ij:S:hp:pP:l:r:s:tu:v:V:n:Jrp:x:w:zq0123:456:7:89:m:T:U:",
longopts, &option_idx)) != EOF) {
#ifdef DEBUG_TRACE
if(trace) fprintf(trace, " #### Handling option -%c [%s] #### \n", opt, optarg ? optarg : "");
#endif
switch (opt) {
case 'a':
ndpi_generate_options(atoi(optarg));
break;
case 'b':
if((num_bin_clusters = atoi(optarg)) > 32)
num_bin_clusters = 32;
@@ -847,9 +826,13 @@ static void parseOptions(int argc, char **argv) {
bpfFilter = optarg;
break;
#ifndef USE_DPDK
#ifdef linux
case 'g':
bind_mask = optarg;
break;
#endif
#endif
case 'l':
num_loops = atoi(optarg);
@@ -895,21 +878,21 @@ static void parseOptions(int argc, char **argv) {
if(nDPI_LogLevel > NDPI_LOG_DEBUG_EXTRA) {
nDPI_LogLevel = NDPI_LOG_DEBUG_EXTRA;
ndpi_free(_debug_protocols);
_debug_protocols = strdup("all");
_debug_protocols = ndpi_strdup("all");
}
break;
case 'u':
ndpi_free(_debug_protocols);
_debug_protocols = strdup(optarg);
_debug_protocols = ndpi_strdup(optarg);
break;
case 'h':
help(1);
break;
case 'J':
enable_joy_stats = 1;
case 'F':
enable_flow_stats = 1;
break;
case 'P':
@@ -940,7 +923,7 @@ static void parseOptions(int argc, char **argv) {
break;
case 'w':
results_path = strdup(optarg);
results_path = ndpi_strdup(optarg);
if((results_file = fopen(results_path, "w")) == NULL) {
printf("Unable to write in file %s: quitting\n", results_path);
return;
@@ -969,12 +952,14 @@ static void parseOptions(int argc, char **argv) {
extcap_config();
break;
#ifndef USE_DPDK
case '5':
do_capture = 1;
break;
#endif
case '7':
extcap_capture_fifo = strdup(optarg);
extcap_capture_fifo = ndpi_strdup(optarg);
break;
case '9':
@@ -1041,12 +1026,14 @@ static void parseOptions(int argc, char **argv) {
}
#ifdef linux
#ifndef USE_DPDK
for(thread_id = 0; thread_id < num_threads; thread_id++)
core_affinity[thread_id] = -1;
if(num_cores > 1 && bind_mask != NULL) {
char *core_id = strtok(bind_mask, ":");
thread_id = 0;
while(core_id != NULL && thread_id < num_threads) {
core_affinity[thread_id++] = atoi(core_id) % num_cores;
core_id = strtok(NULL, ":");
@@ -1054,6 +1041,7 @@ static void parseOptions(int argc, char **argv) {
}
#endif
#endif
#endif
}
/* ********************************** */
@@ -1160,7 +1148,7 @@ static char* is_unsafe_cipher(ndpi_cipher_weakness c) {
/* ********************************** */
void print_bin(FILE *fout, const char *label, struct ndpi_bin *b) {
u_int8_t i;
u_int16_t i;
const char *sep = label ? "," : ";";
ndpi_normalize_bin(b);
@@ -1193,42 +1181,13 @@ static void printFlow(u_int32_t id, struct ndpi_flow_info *flow, u_int16_t threa
FILE *out = results_file ? results_file : stdout;
u_int8_t known_tls;
char buf[32], buf1[64];
char buf_ver[16];
u_int i;
double dos_ge_score;
double dos_slow_score;
double dos_hulk_score;
double ddos_score;
double hearthbleed_score;
double ftp_patator_score;
double ssh_patator_score;
double inf_score;
if(csv_fp != NULL) {
float data_ratio = ndpi_data_ratio(flow->src2dst_bytes, flow->dst2src_bytes);
double f = (double)flow->first_seen_ms, l = (double)flow->last_seen_ms;
/* PLEASE KEEP IN SYNC WITH printCSVHeader() */
dos_ge_score = Dos_goldeneye_score(flow);
dos_slow_score = Dos_slow_score(flow);
dos_hulk_score = Dos_hulk_score(flow);
ddos_score = Ddos_score(flow);
hearthbleed_score = Hearthbleed_score(flow);
ftp_patator_score = Ftp_patator_score(flow);
ssh_patator_score = Ssh_patator_score(flow);
inf_score = Infiltration_score(flow);
double benign_score = dos_ge_score < 1 && dos_slow_score < 1 && \
dos_hulk_score < 1 && ddos_score < 1 && hearthbleed_score < 1 && \
ftp_patator_score < 1 && ssh_patator_score < 1 && inf_score < 1 ? 1.1 : 0;
fprintf(csv_fp, "%u,%u,%.3f,%.3f,%.3f,%s,%u,%s,%u,",
flow->flow_id,
flow->protocol,
@@ -1247,11 +1206,6 @@ static void printFlow(u_int32_t id, struct ndpi_flow_info *flow, u_int16_t threa
flow->detected_protocol, buf, sizeof(buf)),
flow->host_server_name);
fprintf(csv_fp, "%.4lf,%.4lf,%.4lf,%.4lf,%.4lf,%.4lf,%.4lf,%.4lf,%.4lf,", \
benign_score, dos_slow_score, dos_ge_score, dos_hulk_score, \
ddos_score, hearthbleed_score, ftp_patator_score, \
ssh_patator_score, inf_score);
fprintf(csv_fp, "%u,%llu,%llu,", flow->src2dst_packets,
(long long unsigned int) flow->src2dst_bytes, (long long unsigned int) flow->src2dst_goodput_bytes);
fprintf(csv_fp, "%u,%llu,%llu,", flow->dst2src_packets,
@@ -1283,12 +1237,11 @@ static void printFlow(u_int32_t id, struct ndpi_flow_info *flow, u_int16_t threa
/* TCP window */
fprintf(csv_fp, "%u,%u,", flow->c_to_s_init_win, flow->s_to_c_init_win);
fprintf(csv_fp, "%s,%s,",
(flow->ssh_tls.client_requested_server_name[0] != '\0') ? flow->ssh_tls.client_requested_server_name : "",
fprintf(csv_fp, "%s,",
(flow->ssh_tls.server_info[0] != '\0') ? flow->ssh_tls.server_info : "");
fprintf(csv_fp, "%s,%s,%s,%s,%s,",
(flow->ssh_tls.ssl_version != 0) ? ndpi_ssl_version2str(flow->ndpi_flow, flow->ssh_tls.ssl_version, &known_tls) : "0",
(flow->ssh_tls.ssl_version != 0) ? ndpi_ssl_version2str(buf_ver, sizeof(buf_ver), flow->ssh_tls.ssl_version, &known_tls) : "0",
(flow->ssh_tls.ja3_client[0] != '\0') ? flow->ssh_tls.ja3_client : "",
(flow->ssh_tls.ja3_client[0] != '\0') ? is_unsafe_cipher(flow->ssh_tls.client_unsafe_cipher) : "0",
(flow->ssh_tls.ja3_server[0] != '\0') ? flow->ssh_tls.ja3_server : "",
@@ -1319,7 +1272,7 @@ static void printFlow(u_int32_t id, struct ndpi_flow_info *flow, u_int16_t threa
}
if((verbose != 1) && (verbose != 2)) {
if(csv_fp && enable_joy_stats) {
if(csv_fp && enable_flow_stats) {
flowGetBDMeanandVariance(flow);
}
@@ -1349,11 +1302,11 @@ static void printFlow(u_int32_t id, struct ndpi_flow_info *flow, u_int16_t threa
if(enable_payload_analyzer) fprintf(out, "[flowId: %u]", flow->flow_id);
}
if(enable_joy_stats) {
if(enable_flow_stats) {
/* Print entropy values for monitored flows. */
flowGetBDMeanandVariance(flow);
fflush(out);
fprintf(out, "[score: %.4f]", flow->entropy.score);
fprintf(out, "[score: %.4f]", flow->entropy->score);
}
if(csv_fp) fprintf(csv_fp, "\n");
@@ -1368,6 +1321,11 @@ static void printFlow(u_int32_t id, struct ndpi_flow_info *flow, u_int16_t threa
ndpi_protocol2name(ndpi_thread_info[thread_id].workflow->ndpi_struct,
flow->detected_protocol, buf1, sizeof(buf1)));
fprintf(out, "[%s]",
ndpi_is_encrypted_proto(ndpi_thread_info[thread_id].workflow->ndpi_struct, flow->detected_protocol) ? "Encrypted" : "ClearText");
fprintf(out, "[Confidence: %s]", ndpi_confidence_get_name(flow->confidence));
if(flow->detected_protocol.category != 0)
fprintf(out, "[cat: %s/%u]",
ndpi_category_get_name(ndpi_thread_info[thread_id].workflow->ndpi_struct,
@@ -1388,9 +1346,10 @@ static void printFlow(u_int32_t id, struct ndpi_flow_info *flow, u_int16_t threa
else
fprintf(out, "[< 1 sec]");
if(flow->telnet.username[0] != '\0') fprintf(out, "[Username: %s]", flow->telnet.username);
if(flow->telnet.password[0] != '\0') fprintf(out, "[Password: %s]", flow->telnet.password);
if(flow->host_server_name[0] != '\0') fprintf(out, "[Host: %s]", flow->host_server_name);
if(flow->telnet.username) fprintf(out, "[Username: %s]", flow->telnet.username);
if(flow->telnet.password) fprintf(out, "[Password: %s]", flow->telnet.password);
if(flow->host_server_name[0] != '\0') fprintf(out, "[Hostname/SNI: %s]", flow->host_server_name);
if(flow->info[0] != '\0') fprintf(out, "[%s]", flow->info);
if(flow->flow_extra_info[0] != '\0') fprintf(out, "[%s]", flow->flow_extra_info);
@@ -1449,9 +1408,9 @@ static void printFlow(u_int32_t id, struct ndpi_flow_info *flow, u_int16_t threa
fprintf(out, "[Risk Score: %u]", ndpi_risk2score(flow->risk, &cli_score, &srv_score));
}
if(flow->ssh_tls.ssl_version != 0) fprintf(out, "[%s]", ndpi_ssl_version2str(flow->ndpi_flow, flow->ssh_tls.ssl_version, &known_tls));
if(flow->ssh_tls.client_requested_server_name[0] != '\0') fprintf(out, "[Client: %s]", flow->ssh_tls.client_requested_server_name);
if(flow->ssh_tls.ssl_version != 0) fprintf(out, "[%s]", ndpi_ssl_version2str(buf_ver, sizeof(buf_ver), flow->ssh_tls.ssl_version, &known_tls));
if(flow->ssh_tls.client_hassh[0] != '\0') fprintf(out, "[HASSH-C: %s]", flow->ssh_tls.client_hassh);
if(flow->ssh_tls.ja3_client[0] != '\0') fprintf(out, "[JA3C: %s%s]", flow->ssh_tls.ja3_client,
@@ -1489,7 +1448,7 @@ static void printFlow(u_int32_t id, struct ndpi_flow_info *flow, u_int16_t threa
if(flow->ssh_tls.browser_heuristics.is_firefox_tls) fprintf(out, "[Firefox]");
if(flow->ssh_tls.browser_heuristics.is_chrome_tls) fprintf(out, "[Chrome]");
#endif
if(flow->ssh_tls.notBefore && flow->ssh_tls.notAfter) {
char notBefore[32], notAfter[32];
struct tm a, b;
@@ -1504,10 +1463,12 @@ static void printFlow(u_int32_t id, struct ndpi_flow_info *flow, u_int16_t threa
if(flow->ssh_tls.server_cipher != '\0') fprintf(out, "[Cipher: %s]",
ndpi_cipher2str(flow->ssh_tls.server_cipher));
if(flow->bittorent_hash[0] != '\0') fprintf(out, "[BT Hash: %s]",
if(flow->bittorent_hash) fprintf(out, "[BT Hash: %s]",
flow->bittorent_hash);
if(flow->dhcp_fingerprint[0] != '\0') fprintf(out, "[DHCP Fingerprint: %s]",
if(flow->dhcp_fingerprint) fprintf(out, "[DHCP Fingerprint: %s]",
flow->dhcp_fingerprint);
if(flow->dhcp_class_ident) fprintf(out, "[DHCP Class Ident: %s]",
flow->dhcp_class_ident);
if(flow->has_human_readeable_strings) fprintf(out, "[PLAIN TEXT (%s)]",
flow->human_readeable_string_buffer);
@@ -1590,6 +1551,7 @@ static void node_proto_guess_walker(const void *node, ndpi_VISIT which, int dept
ndpi_thread_info[thread_id].workflow->stats.protocol_counter[proto] += flow->src2dst_packets + flow->dst2src_packets;
ndpi_thread_info[thread_id].workflow->stats.protocol_counter_bytes[proto] += flow->src2dst_bytes + flow->dst2src_bytes;
ndpi_thread_info[thread_id].workflow->stats.protocol_flows[proto]++;
ndpi_thread_info[thread_id].workflow->stats.flow_confidence[flow->confidence]++;
}
}
@@ -1665,7 +1627,8 @@ int updateIpTree(u_int32_t key, u_int8_t version,
q->addr = key;
q->version = version;
strncpy(q->proto, proto, sizeof(q->proto));
strncpy(q->proto, proto, sizeof(q->proto) - 1);
q->proto[sizeof(q->proto) - 1] = '\0';
q->count = UPDATED_TREE;
q->left = q->right = (addr_node *)0;
@@ -1700,7 +1663,8 @@ void updateTopIpAddress(u_int32_t addr, u_int8_t version, const char *proto,
pair.addr = addr;
pair.version = version;
pair.count = count;
strncpy(pair.proto, proto, sizeof(pair.proto));
strncpy(pair.proto, proto, sizeof(pair.proto) - 1);
pair.proto[sizeof(pair.proto) - 1] = '\0';
for(i=0; i<size; i++) {
/* if the same ip with a bigger
@@ -1758,7 +1722,8 @@ static void updatePortStats(struct port_stats **stats, u_int32_t port,
s->addr_tree->addr = addr;
s->addr_tree->version = version;
strncpy(s->addr_tree->proto, proto, sizeof(s->addr_tree->proto));
strncpy(s->addr_tree->proto, proto, sizeof(s->addr_tree->proto) - 1);
s->addr_tree->proto[sizeof(s->addr_tree->proto) - 1] = '\0';
s->addr_tree->count = 1;
s->addr_tree->left = NULL;
s->addr_tree->right = NULL;
@@ -1968,12 +1933,14 @@ static void port_stats_walker(const void *node, ndpi_VISIT which, int depth, voi
sport = ntohs(flow->src_port), dport = ntohs(flow->dst_port);
/* get app level protocol */
if(flow->detected_protocol.master_protocol)
if(flow->detected_protocol.master_protocol) {
ndpi_protocol2name(ndpi_thread_info[thread_id].workflow->ndpi_struct,
flow->detected_protocol, proto, sizeof(proto));
else
} else {
strncpy(proto, ndpi_get_proto_name(ndpi_thread_info[thread_id].workflow->ndpi_struct,
flow->detected_protocol.app_protocol),sizeof(proto));
flow->detected_protocol.app_protocol),sizeof(proto) - 1);
proto[sizeof(proto) - 1] = '\0';
}
if(((r = strcmp(ipProto2Name(flow->protocol), "TCP")) == 0)
&& (flow->src2dst_packets == 1) && (flow->dst2src_packets == 0)) {
@@ -2109,6 +2076,8 @@ static void setupDetection(u_int16_t thread_id, pcap_t * pcap_handle) {
sizeof(ndpi_thread_info[thread_id].workflow->stats.protocol_counter_bytes));
memset(ndpi_thread_info[thread_id].workflow->stats.protocol_flows, 0,
sizeof(ndpi_thread_info[thread_id].workflow->stats.protocol_flows));
memset(ndpi_thread_info[thread_id].workflow->stats.flow_confidence, 0,
sizeof(ndpi_thread_info[thread_id].workflow->stats.flow_confidence));
if(_protoFilePath != NULL)
ndpi_load_protocols_file(ndpi_thread_info[thread_id].workflow->ndpi_struct, _protoFilePath);
@@ -2351,7 +2320,7 @@ static void printFlowsStats() {
ndpi_host_ja3_fingerprints *ja3ByHostsHashT = NULL; // outer hash table
ndpi_ja3_fingerprints_host *hostByJA3C_ht = NULL; // for client
ndpi_ja3_fingerprints_host *hostByJA3S_ht = NULL; // for server
int i;
unsigned int i;
ndpi_host_ja3_fingerprints *ja3ByHost_element = NULL;
ndpi_ja3_info *info_of_element = NULL;
ndpi_host_ja3_fingerprints *tmp = NULL;
@@ -2386,7 +2355,7 @@ static void printFlowsStats() {
newHost->host_server_info_hasht = NULL;
newHost->ip_string = all_flows[i].flow->src_name;
newHost->ip = all_flows[i].flow->src_ip;
newHost->dns_name = all_flows[i].flow->ssh_tls.client_requested_server_name;
newHost->dns_name = all_flows[i].flow->host_server_name;
ndpi_ja3_info *newJA3 = ndpi_malloc(sizeof(ndpi_ja3_info));
newJA3->ja3 = all_flows[i].flow->ssh_tls.ja3_client;
@@ -2419,7 +2388,7 @@ static void printFlowsStats() {
newHost->ip = all_flows[i].flow->src_ip;
newHost->ip_string = all_flows[i].flow->src_name;
newHost->dns_name = all_flows[i].flow->ssh_tls.client_requested_server_name;;
newHost->dns_name = all_flows[i].flow->host_server_name;
ndpi_ja3_fingerprints_host *newElement = ndpi_malloc(sizeof(ndpi_ja3_fingerprints_host));
newElement->ja3 = all_flows[i].flow->ssh_tls.ja3_client;
@@ -2436,7 +2405,7 @@ static void printFlowsStats() {
ndpi_ip_dns *newInnerElement = ndpi_malloc(sizeof(ndpi_ip_dns));
newInnerElement->ip = all_flows[i].flow->src_ip;
newInnerElement->ip_string = all_flows[i].flow->src_name;
newInnerElement->dns_name = all_flows[i].flow->ssh_tls.client_requested_server_name;
newInnerElement->dns_name = all_flows[i].flow->host_server_name;
HASH_ADD_INT(hostByJA3Found->ipToDNS_ht, ip, newInnerElement);
}
}
@@ -2789,10 +2758,10 @@ static void printFlowsStats() {
print_bin(out, NULL, &bins[i]);
printf("][similarity: %f]",
(similarity = ndpi_bin_similarity(&centroids[j], &bins[i], 0)));
(similarity = ndpi_bin_similarity(&centroids[j], &bins[i], 0, 0)));
if(all_flows[i].flow->ssh_tls.client_requested_server_name[0] != '\0')
fprintf(out, "[%s]", all_flows[i].flow->ssh_tls.client_requested_server_name);
if(all_flows[i].flow->host_server_name[0] != '\0')
fprintf(out, "[%s]", all_flows[i].flow->host_server_name);
if(enable_doh_dot_detection) {
if(((all_flows[i].flow->detected_protocol.master_protocol == NDPI_PROTOCOL_TLS)
@@ -2857,7 +2826,7 @@ static void printFlowsStats() {
for(i=0; i<num_flows; i++)
printFlow(i+1, all_flows[i].flow, all_flows[i].thread_id);
} else if(csv_fp != NULL) {
int i;
unsigned int i;
num_flows = 0;
for(thread_id = 0; thread_id < num_threads; thread_id++) {
@@ -2880,7 +2849,6 @@ static void printFlowsStats() {
*/
static void printResults(u_int64_t processing_time_usec, u_int64_t setup_time_usec) {
u_int32_t i;
u_int64_t total_flow_bytes = 0;
u_int32_t avg_pkt_size = 0;
int thread_id;
char buf[32];
@@ -2932,6 +2900,9 @@ static void printResults(u_int64_t processing_time_usec, u_int64_t setup_time_us
cumulative_stats.dpi_packet_count[0] += ndpi_thread_info[thread_id].workflow->stats.dpi_packet_count[0];
cumulative_stats.dpi_packet_count[1] += ndpi_thread_info[thread_id].workflow->stats.dpi_packet_count[1];
cumulative_stats.dpi_packet_count[2] += ndpi_thread_info[thread_id].workflow->stats.dpi_packet_count[2];
for(i = 0; i < sizeof(cumulative_stats.flow_confidence)/sizeof(cumulative_stats.flow_confidence[0]); i++)
cumulative_stats.flow_confidence[i] += ndpi_thread_info[thread_id].workflow->stats.flow_confidence[i];
}
if(cumulative_stats.total_wire_bytes == 0)
@@ -2940,7 +2911,7 @@ static void printResults(u_int64_t processing_time_usec, u_int64_t setup_time_us
if(!quiet_mode) {
printf("\nnDPI Memory statistics:\n");
printf("\tnDPI Memory (once): %-13s\n", formatBytes(ndpi_get_ndpi_detection_module_size(), buf, sizeof(buf)));
printf("\tFlow Memory (per flow): %-13s\n", formatBytes(sizeof(struct ndpi_flow_struct), buf, sizeof(buf)));
printf("\tFlow Memory (per flow): %-13s\n", formatBytes( ndpi_detection_get_sizeof_ndpi_flow_struct(), buf, sizeof(buf)));
printf("\tActual Memory: %-13s\n", formatBytes(current_ndpi_memory, buf, sizeof(buf)));
printf("\tPeak Memory: %-13s\n", formatBytes(max_ndpi_memory, buf, sizeof(buf)));
printf("\tSetup Time: %lu msec\n", (unsigned long)(setup_time_usec/1000));
@@ -2993,9 +2964,23 @@ static void printResults(u_int64_t processing_time_usec, u_int64_t setup_time_us
t = 0;
b = 0;
}
strftime(when, sizeof(when), "%d/%b/%Y %H:%M:%S", localtime_r(&pcap_start.tv_sec, &result));
#ifdef WIN32
/* localtime() on Windows is thread-safe */
struct tm * tm_ptr = localtime(&pcap_start.tv_sec);
result = *tm_ptr;
#else
localtime_r(&pcap_start.tv_sec, &result);
#endif
strftime(when, sizeof(when), "%d/%b/%Y %H:%M:%S", &result);
printf("\tAnalysis begin: %s\n", when);
strftime(when, sizeof(when), "%d/%b/%Y %H:%M:%S", localtime_r(&pcap_end.tv_sec, &result));
#ifdef WIN32
/* localtime() on Windows is thread-safe */
tm_ptr = localtime(&pcap_end.tv_sec);
result = *tm_ptr;
#else
localtime_r(&pcap_end.tv_sec, &result);
#endif
strftime(when, sizeof(when), "%d/%b/%Y %H:%M:%S", &result);
printf("\tAnalysis end: %s\n", when);
printf("\tTraffic throughput: %s pps / %s/sec\n", formatPackets(t, buf), formatTraffic(b, 1, buf1));
printf("\tTraffic duration: %.3f sec\n", traffic_duration/1000000);
@@ -3016,6 +3001,12 @@ static void printResults(u_int64_t processing_time_usec, u_int64_t setup_time_us
printf("\tDPI Packets (other): %-13llu (%.2f pkts/flow)\n",
(long long unsigned int)cumulative_stats.dpi_packet_count[2],
cumulative_stats.dpi_packet_count[2] / (float)cumulative_stats.flow_count[2]);
for(i = 0; i < sizeof(cumulative_stats.flow_confidence)/sizeof(cumulative_stats.flow_confidence[0]); i++) {
if(cumulative_stats.flow_confidence[i] != 0)
printf("\tConfidence: %-10s %-13llu (flows)\n", ndpi_confidence_get_name(i),
(long long unsigned int)cumulative_stats.flow_confidence[i]);
}
}
if(results_file) {
@@ -3034,6 +3025,14 @@ static void printResults(u_int64_t processing_time_usec, u_int64_t setup_time_us
fprintf(results_file, "DPI Packets (other):\t%llu\t(%.2f pkts/flow)\n",
(long long unsigned int)cumulative_stats.dpi_packet_count[2],
cumulative_stats.dpi_packet_count[2] / (float)cumulative_stats.flow_count[2]);
for(i = 0; i < sizeof(cumulative_stats.flow_confidence)/sizeof(cumulative_stats.flow_confidence[0]); i++) {
if(cumulative_stats.flow_confidence[i] != 0)
fprintf(results_file, "Confidence %-17s: %llu (flows)\n",
ndpi_confidence_get_name(i),
(long long unsigned int)cumulative_stats.flow_confidence[i]);
}
fprintf(results_file, "\n");
}
@@ -3059,8 +3058,6 @@ static void printResults(u_int64_t processing_time_usec, u_int64_t setup_time_us
(long long unsigned int)cumulative_stats.protocol_counter_bytes[i],
cumulative_stats.protocol_flows[i]);
}
total_flow_bytes += cumulative_stats.protocol_counter_bytes[i];
}
}
@@ -3076,8 +3073,6 @@ static void printResults(u_int64_t processing_time_usec, u_int64_t setup_time_us
}
}
// printf("\n\nTotal Flow Traffic: %llu (diff: %llu)\n", total_flow_bytes, cumulative_stats.total_ip_bytes-total_flow_bytes);
printRiskStats();
printFlowsStats();
@@ -3148,6 +3143,8 @@ void sigproc(int sig) {
}
#ifndef USE_DPDK
/**
* @brief Get the next pcap file from a passed playlist
*/
@@ -3171,7 +3168,6 @@ next_line:
}
}
/**
* @brief Configure the pcap handle
*/
@@ -3194,14 +3190,17 @@ static void configurePcapHandle(pcap_t * pcap_handle) {
}
}
#endif
/**
* @brief Open a pcap file or a specified device - Always returns a valid pcap_t
*/
static pcap_t * openPcapFileOrDevice(u_int16_t thread_id, const u_char * pcap_file) {
#ifndef USE_DPDK
u_int snaplen = 1536;
int promisc = 1;
char pcap_error_buffer[PCAP_ERRBUF_SIZE];
#endif
pcap_t * pcap_handle = NULL;
/* trying to open a live interface */
@@ -3336,7 +3335,7 @@ static void ndpi_process_packet(u_char *args,
u_int32_t *crc, delta = sizeof(struct ndpi_packet_trailer) + 4 /* ethernet trailer */;
struct ndpi_packet_trailer *trailer;
u_int16_t cli_score, srv_score;
memcpy(&h, header, sizeof(h));
if(h.caplen > (sizeof(extcap_buf)-sizeof(struct ndpi_packet_trailer) - 4)) {
@@ -3369,8 +3368,8 @@ static void ndpi_process_packet(u_char *args,
printf("INTERNAL ERROR: ingress packet was modified by nDPI: this should not happen [thread_id=%u, packetId=%lu, caplen=%u]\n",
thread_id, (unsigned long)ndpi_thread_info[thread_id].workflow->stats.raw_packet_count, header->caplen);
if((pcap_end.tv_sec-pcap_start.tv_sec) > pcap_analysis_duration) {
int i;
if((u_int32_t)(pcap_end.tv_sec-pcap_start.tv_sec) > pcap_analysis_duration) {
unsigned int i;
u_int64_t processing_time_usec, setup_time_usec;
gettimeofday(&end, NULL);
@@ -3403,6 +3402,7 @@ static void ndpi_process_packet(u_char *args,
}
}
#ifndef USE_DPDK
/**
* @brief Call pcap_loop() to process packets from a live capture or savefile
*/
@@ -3418,13 +3418,16 @@ static void runPcapLoop(u_int16_t thread_id) {
printf("Error while reading pcap file: '%s'\n", pcap_geterr(ndpi_thread_info[thread_id].workflow->pcap_handle));
}
}
#endif
/**
* @brief Process a running thread
*/
void * processing_thread(void *_thread_id) {
long thread_id = (long) _thread_id;
#ifndef USE_DPDK
char pcap_error_buffer[PCAP_ERRBUF_SIZE];
#endif
#if defined(linux) && defined(HAVE_PTHREAD_SETAFFINITY_NP)
if(core_affinity[thread_id] >= 0) {
@@ -3644,8 +3647,6 @@ static void dgaUnitTest() {
};
const char *non_dga[] = {
"dns.msftncsi.com",
"www.confindustriabrescia.it",
"mz.gov.pl",
"zoomam104zc.zoom.us",
"5CI_DOMBIN",
@@ -3694,12 +3695,12 @@ static void dgaUnitTest() {
if(debug) printf("Checking non DGA %s\n", non_dga[i]);
assert(ndpi_check_dga_name(ndpi_str, NULL, (char*)non_dga[i], 1) == 0);
}
for(i=0; dga[i] != NULL; i++) {
if(debug) printf("Checking DGA %s\n", non_dga[i]);
assert(ndpi_check_dga_name(ndpi_str, NULL, (char*)dga[i], 1) == 1);
}
ndpi_exit_detection_module(ndpi_str);
}
@@ -3761,8 +3762,8 @@ void automataUnitTest() {
void *automa = ndpi_init_automa();
assert(automa);
assert(ndpi_add_string_to_automa(automa, strdup("hello")) == 0);
assert(ndpi_add_string_to_automa(automa, strdup("world")) == 0);
assert(ndpi_add_string_to_automa(automa, ndpi_strdup("hello")) == 0);
assert(ndpi_add_string_to_automa(automa, ndpi_strdup("world")) == 0);
ndpi_finalize_automa(automa);
assert(ndpi_match_string(automa, "This is the wonderful world of nDPI") == 1);
ndpi_free_automa(automa);
@@ -4141,6 +4142,8 @@ void sesUnitTest() {
}
if(fd) fclose(fd);
ndpi_ses_fitting(v, num, &alpha); /* Compute the best alpha */
}
/* *********************************************** */
@@ -4200,7 +4203,7 @@ void desUnitTest() {
if(trace) {
printf("%2u)\t%12.3f\t%.3f\t%12.3f\t%12.3f\t %s [%.3f]\n", i, v[i], prediction, lower, upper,
((rc == 0) || ((v[i] >= lower) && (v[i] <= upper))) ? "OK" : "ANOMALY",
(rc == 0) ? "LEARNING" : (((v[i] >= lower) && (v[i] <= upper)) ? "OK" : "ANOMALY"),
confidence_band);
if(fd)
@@ -4211,6 +4214,8 @@ void desUnitTest() {
}
if(fd) fclose(fd);
ndpi_des_fitting(v, num, &alpha, &beta); /* Compute the best alpha/beta */
}
/* *********************************************** */
@@ -4315,6 +4320,46 @@ void jitterUnitTest() {
/* *********************************************** */
void compressedBitmapUnitTest() {
ndpi_bitmap *b = ndpi_bitmap_alloc(), *b1;
u_int i, trace = 0;
size_t ser;
char *buf;
ndpi_bitmap_iterator *it;
u_int32_t value;
for(i=0; i<1000; i++) {
u_int32_t v = rand();
if(trace) printf("%u ", v);
ndpi_bitmap_set(b, v);
assert(ndpi_bitmap_isset(b, v));
}
if(trace) printf("\n");
ser = ndpi_bitmap_serialize(b, &buf);
assert(ser > 0);
if(trace) printf("len: %u\n", (unsigned int)ser);
b1 = ndpi_bitmap_deserialize(buf);
assert(b1);
assert((it = ndpi_bitmap_iterator_alloc(b)));
while(ndpi_bitmap_iterator_next(it, &value)) {
if(trace) printf("%u ", value);
}
if(trace) printf("\n");
ndpi_bitmap_iterator_free(it);
ndpi_free(buf);
ndpi_bitmap_free(b);
ndpi_bitmap_free(b1);
}
/* *********************************************** */
/**
@brief MAIN FUNCTION
**/
@@ -4375,9 +4420,10 @@ int original_main(int argc, char **argv) {
ndpi_self_check_host_match();
analysisUnitTest();
rulesUnitTest();
compressedBitmapUnitTest();
#endif
}
gettimeofday(&startup_time, NULL);
memset(ndpi_thread_info, 0, sizeof(ndpi_thread_info));
@@ -4421,7 +4467,7 @@ int original_main(int argc, char **argv) {
if(ndpi_info_mod) ndpi_exit_detection_module(ndpi_info_mod);
if(csv_fp) fclose(csv_fp);
ndpi_free(_debug_protocols);
#ifdef DEBUG_TRACE
if(trace) fclose(trace);
#endif
@@ -4439,11 +4485,12 @@ int original_main(int argc, char **argv) {
/**
@brief Timezone
**/
#ifndef __GNUC__
struct timezone {
int tz_minuteswest; /* minutes W of Greenwich */
int tz_dsttime; /* type of dst correction */
};
#endif
/**
@brief Set time
+145 -145
View File
@@ -1,6 +1,10 @@
#ifndef WIN32
#include <arpa/inet.h>
#include <errno.h>
#include <netinet/in.h>
#else
#include <windows.h>
#endif
#include <errno.h>
#include <ndpi_api.h>
#include <ndpi_main.h>
#include <ndpi_typedefs.h>
@@ -12,6 +16,7 @@
#include <string.h>
#include <unistd.h>
//#define VERBOSE 1
#define MAX_FLOW_ROOTS_PER_THREAD 2048
#define MAX_IDLE_FLOWS_PER_THREAD 64
#define TICK_RESOLUTION 1000
@@ -47,12 +52,19 @@ struct nDPI_flow_info {
union {
struct {
uint32_t src;
uint32_t pad_00[3];
uint32_t dst;
uint32_t pad_01[3];
} v4;
struct {
uint64_t src[2];
uint64_t dst[2];
} v6;
struct {
uint32_t src[4];
uint32_t dst[4];
} u32;
} ip_tuple;
unsigned long long int total_l4_data_len;
@@ -73,16 +85,12 @@ struct nDPI_flow_info {
struct ndpi_proto guessed_protocol;
struct ndpi_flow_struct * ndpi_flow;
struct ndpi_id_struct * ndpi_src;
struct ndpi_id_struct * ndpi_dst;
};
struct nDPI_workflow {
pcap_t * pcap_handle;
uint8_t error_or_eof:1;
uint8_t reserved_00:7;
uint8_t reserved_01[3];
uint8_t error_or_eof;
unsigned long long int packets_captured;
unsigned long long int packets_processed;
@@ -130,12 +138,16 @@ static struct nDPI_workflow * init_workflow(char const * const file_or_device)
if (access(file_or_device, R_OK) != 0 && errno == ENOENT) {
workflow->pcap_handle = pcap_open_live(file_or_device, /* 1536 */ 65535, 1, 250, pcap_error_buffer);
} else {
#ifdef WIN32
workflow->pcap_handle = pcap_open_offline(file_or_device, pcap_error_buffer);
#else
workflow->pcap_handle = pcap_open_offline_with_tstamp_precision(file_or_device, PCAP_TSTAMP_PRECISION_MICRO,
pcap_error_buffer);
#endif
}
if (workflow->pcap_handle == NULL) {
fprintf(stderr, "pcap_open_live / pcap_open_offline_with_tstamp_precision: %.*s\n",
fprintf(stderr, "pcap_open_live / pcap_open_offline: %.*s\n",
(int) PCAP_ERRBUF_SIZE, pcap_error_buffer);
free_workflow(&workflow);
return NULL;
@@ -176,8 +188,6 @@ static void ndpi_flow_info_freer(void * const node)
{
struct nDPI_flow_info * const flow = (struct nDPI_flow_info *)node;
ndpi_free(flow->ndpi_dst);
ndpi_free(flow->ndpi_src);
ndpi_flow_free(flow->ndpi_flow);
ndpi_free(flow);
}
@@ -336,60 +346,72 @@ static void print_packet_info(struct nDPI_reader_thread const * const reader_thr
}
#endif
static int ip_tuples_equal(struct nDPI_flow_info const * const A,
struct nDPI_flow_info const * const B)
static int ip_tuples_compare(struct nDPI_flow_info const * const A, struct nDPI_flow_info const * const B)
{
if (A->l3_type == L3_IP && B->l3_type == L3_IP6) {
return A->ip_tuple.v4.src == B->ip_tuple.v4.src &&
A->ip_tuple.v4.dst == B->ip_tuple.v4.dst;
} else if (A->l3_type == L3_IP6 && B->l3_type == L3_IP6) {
return A->ip_tuple.v6.src[0] == B->ip_tuple.v6.src[0] &&
A->ip_tuple.v6.src[1] == B->ip_tuple.v6.src[1] &&
A->ip_tuple.v6.dst[0] == B->ip_tuple.v6.dst[0] &&
A->ip_tuple.v6.dst[1] == B->ip_tuple.v6.dst[1];
// generate a warning if the enum changes
switch (A->l3_type)
{
case L3_IP:
case L3_IP6:
break;
}
return 0;
}
static int ip_tuples_compare(struct nDPI_flow_info const * const A,
struct nDPI_flow_info const * const B)
{
if (A->l3_type == L3_IP && B->l3_type == L3_IP6) {
if (A->ip_tuple.v4.src < B->ip_tuple.v4.src ||
A->ip_tuple.v4.dst < B->ip_tuple.v4.dst)
{
return -1;
}
if (A->ip_tuple.v4.src > B->ip_tuple.v4.src ||
A->ip_tuple.v4.dst > B->ip_tuple.v4.dst)
{
return 1;
}
} else if (A->l3_type == L3_IP6 && B->l3_type == L3_IP6) {
if ((A->ip_tuple.v6.src[0] < B->ip_tuple.v6.src[0] &&
A->ip_tuple.v6.src[1] < B->ip_tuple.v6.src[1]) ||
(A->ip_tuple.v6.dst[0] < B->ip_tuple.v6.dst[0] &&
A->ip_tuple.v6.dst[1] < B->ip_tuple.v6.dst[1]))
{
return -1;
}
if ((A->ip_tuple.v6.src[0] > B->ip_tuple.v6.src[0] &&
A->ip_tuple.v6.src[1] > B->ip_tuple.v6.src[1]) ||
(A->ip_tuple.v6.dst[0] > B->ip_tuple.v6.dst[0] &&
A->ip_tuple.v6.dst[1] > B->ip_tuple.v6.dst[1]))
{
return 1;
}
}
if (A->src_port < B->src_port ||
A->dst_port < B->dst_port)
if (A->l3_type == L3_IP && B->l3_type == L3_IP)
{
if (A->ip_tuple.v4.src < B->ip_tuple.v4.src)
{
return -1;
} else if (A->src_port > B->src_port ||
A->dst_port > B->dst_port)
}
if (A->ip_tuple.v4.src > B->ip_tuple.v4.src)
{
return 1;
}
if (A->ip_tuple.v4.dst < B->ip_tuple.v4.dst)
{
return -1;
}
if (A->ip_tuple.v4.dst > B->ip_tuple.v4.dst)
{
return 1;
}
}
else if (A->l3_type == L3_IP6 && B->l3_type == L3_IP6)
{
if (A->ip_tuple.v6.src[0] < B->ip_tuple.v6.src[0] && A->ip_tuple.v6.src[1] < B->ip_tuple.v6.src[1])
{
return -1;
}
if (A->ip_tuple.v6.src[0] > B->ip_tuple.v6.src[0] && A->ip_tuple.v6.src[1] > B->ip_tuple.v6.src[1])
{
return 1;
}
if (A->ip_tuple.v6.dst[0] < B->ip_tuple.v6.dst[0] && A->ip_tuple.v6.dst[1] < B->ip_tuple.v6.dst[1])
{
return -1;
}
if (A->ip_tuple.v6.dst[0] > B->ip_tuple.v6.dst[0] && A->ip_tuple.v6.dst[1] > B->ip_tuple.v6.dst[1])
{
return 1;
}
}
if (A->src_port < B->src_port)
{
return -1;
}
if (A->src_port > B->src_port)
{
return 1;
}
if (A->dst_port < B->dst_port)
{
return -1;
}
if (A->dst_port > B->dst_port)
{
return 1;
}
return 0;
}
@@ -438,13 +460,6 @@ static int ndpi_workflow_node_cmp(void const * const A, void const * const B) {
return(1);
}
if (ip_tuples_equal(flow_info_a, flow_info_b) != 0 &&
flow_info_a->src_port == flow_info_b->src_port &&
flow_info_a->dst_port == flow_info_b->dst_port)
{
return(0);
}
return ip_tuples_compare(flow_info_a, flow_info_b);
}
@@ -486,10 +501,6 @@ static void ndpi_process_packet(uint8_t * const args,
void * tree_result;
struct nDPI_flow_info * flow_to_process;
int direction_changed = 0;
struct ndpi_id_struct * ndpi_src;
struct ndpi_id_struct * ndpi_dst;
const struct ndpi_ethhdr * ethernet;
const struct ndpi_iphdr * ip;
struct ndpi_ipv6hdr * ip6;
@@ -687,7 +698,7 @@ static void ndpi_process_packet(uint8_t * const args,
workflow->total_l4_data_len += l4_len;
#ifdef VERBOSE
print_packet_info(reader_thread, header, l4_data_len, &flow);
print_packet_info(reader_thread, header, l4_len, &flow);
#endif
/* calculate flow hash for btree find, search(insert) */
@@ -713,27 +724,38 @@ static void ndpi_process_packet(uint8_t * const args,
tree_result = ndpi_tfind(&flow, &workflow->ndpi_flows_active[hashed_index], ndpi_workflow_node_cmp);
if (tree_result == NULL) {
/* flow not found in btree: switch src <-> dst and try to find it again */
uint64_t orig_src_ip[2] = { flow.ip_tuple.v6.src[0], flow.ip_tuple.v6.src[1] };
uint64_t orig_dst_ip[2] = { flow.ip_tuple.v6.dst[0], flow.ip_tuple.v6.dst[1] };
uint32_t orig_src_ip[4] = { flow.ip_tuple.u32.src[0], flow.ip_tuple.u32.src[1],
flow.ip_tuple.u32.src[2], flow.ip_tuple.u32.src[3] };
uint32_t orig_dst_ip[4] = { flow.ip_tuple.u32.dst[0], flow.ip_tuple.u32.dst[1],
flow.ip_tuple.u32.dst[2], flow.ip_tuple.u32.dst[3] };
uint16_t orig_src_port = flow.src_port;
uint16_t orig_dst_port = flow.dst_port;
flow.ip_tuple.v6.src[0] = orig_dst_ip[0];
flow.ip_tuple.v6.src[1] = orig_dst_ip[1];
flow.ip_tuple.v6.dst[0] = orig_src_ip[0];
flow.ip_tuple.v6.dst[1] = orig_src_ip[1];
flow.ip_tuple.u32.src[0] = orig_dst_ip[0];
flow.ip_tuple.u32.src[1] = orig_dst_ip[1];
flow.ip_tuple.u32.src[2] = orig_dst_ip[2];
flow.ip_tuple.u32.src[3] = orig_dst_ip[3];
flow.ip_tuple.u32.dst[0] = orig_src_ip[0];
flow.ip_tuple.u32.dst[1] = orig_src_ip[1];
flow.ip_tuple.u32.dst[2] = orig_src_ip[2];
flow.ip_tuple.u32.dst[3] = orig_src_ip[3];
flow.src_port = orig_dst_port;
flow.dst_port = orig_src_port;
tree_result = ndpi_tfind(&flow, &workflow->ndpi_flows_active[hashed_index], ndpi_workflow_node_cmp);
if (tree_result != NULL) {
direction_changed = 1;
}
flow.ip_tuple.v6.src[0] = orig_src_ip[0];
flow.ip_tuple.v6.src[1] = orig_src_ip[1];
flow.ip_tuple.v6.dst[0] = orig_dst_ip[0];
flow.ip_tuple.v6.dst[1] = orig_dst_ip[1];
flow.ip_tuple.u32.src[0] = orig_src_ip[0];
flow.ip_tuple.u32.src[1] = orig_src_ip[1];
flow.ip_tuple.u32.src[2] = orig_src_ip[2];
flow.ip_tuple.u32.src[3] = orig_src_ip[3];
flow.ip_tuple.u32.dst[0] = orig_dst_ip[0];
flow.ip_tuple.u32.dst[1] = orig_dst_ip[1];
flow.ip_tuple.u32.dst[2] = orig_dst_ip[2];
flow.ip_tuple.u32.dst[3] = orig_dst_ip[3];
flow.src_port = orig_src_port;
flow.dst_port = orig_dst_port;
}
@@ -757,7 +779,7 @@ static void ndpi_process_packet(uint8_t * const args,
workflow->cur_active_flows++;
workflow->total_active_flows++;
memcpy(flow_to_process, &flow, sizeof(*flow_to_process));
flow_to_process->flow_id = flow_id++;
flow_to_process->flow_id = __sync_fetch_and_add(&flow_id, 1);
flow_to_process->ndpi_flow = (struct ndpi_flow_struct *)ndpi_flow_malloc(SIZEOF_FLOW_STRUCT);
if (flow_to_process->ndpi_flow == NULL) {
@@ -767,20 +789,6 @@ static void ndpi_process_packet(uint8_t * const args,
}
memset(flow_to_process->ndpi_flow, 0, SIZEOF_FLOW_STRUCT);
flow_to_process->ndpi_src = (struct ndpi_id_struct *)ndpi_calloc(1, SIZEOF_ID_STRUCT);
if (flow_to_process->ndpi_src == NULL) {
fprintf(stderr, "[%8llu, %d, %4u] Not enough memory for src id struct\n",
workflow->packets_captured, reader_thread->array_index, flow_to_process->flow_id);
return;
}
flow_to_process->ndpi_dst = (struct ndpi_id_struct *)ndpi_calloc(1, SIZEOF_ID_STRUCT);
if (flow_to_process->ndpi_dst == NULL) {
fprintf(stderr, "[%8llu, %d, %4u] Not enough memory for dst id struct\n",
workflow->packets_captured, reader_thread->array_index, flow_to_process->flow_id);
return;
}
printf("[%8llu, %d, %4u] new %sflow\n", workflow->packets_captured, thread_index,
flow_to_process->flow_id,
(flow_to_process->is_midstream_flow != 0 ? "midstream-" : ""));
@@ -788,19 +796,8 @@ static void ndpi_process_packet(uint8_t * const args,
/* Possible Leak, but should not happen as we'd abort earlier. */
return;
}
ndpi_src = flow_to_process->ndpi_src;
ndpi_dst = flow_to_process->ndpi_dst;
} else {
flow_to_process = *(struct nDPI_flow_info **)tree_result;
if (direction_changed != 0) {
ndpi_src = flow_to_process->ndpi_dst;
ndpi_dst = flow_to_process->ndpi_src;
} else {
ndpi_src = flow_to_process->ndpi_src;
ndpi_dst = flow_to_process->ndpi_dst;
}
}
flow_to_process->packets_processed++;
@@ -851,7 +848,7 @@ static void ndpi_process_packet(uint8_t * const args,
flow_to_process->detected_l7_protocol =
ndpi_detection_process_packet(workflow->ndpi_struct, flow_to_process->ndpi_flow,
ip != NULL ? (uint8_t *)ip : (uint8_t *)ip6,
ip_size, time_ms, ndpi_src, ndpi_dst);
ip_size, time_ms);
if (ndpi_is_protocol_detected(workflow->ndpi_struct,
flow_to_process->detected_l7_protocol) != 0 &&
@@ -906,41 +903,43 @@ static void ndpi_process_packet(uint8_t * const args,
flow_to_process->detected_l7_protocol.app_protocol == NDPI_PROTOCOL_TLS)
{
if (flow_to_process->tls_client_hello_seen == 0 &&
flow_to_process->ndpi_flow->l4.tcp.tls.hello_processed != 0)
flow_to_process->ndpi_flow->protos.tls_quic.hello_processed != 0)
{
uint8_t unknown_tls_version = 0;
char buf_ver[16];
printf("[%8llu, %d, %4d][TLS-CLIENT-HELLO] version: %s | sni: %s | alpn: %s\n",
workflow->packets_captured,
reader_thread->array_index,
flow_to_process->flow_id,
ndpi_ssl_version2str(flow_to_process->ndpi_flow,
flow_to_process->ndpi_flow->protos.tls_quic_stun.tls_quic.ssl_version,
ndpi_ssl_version2str(buf_ver, sizeof(buf_ver),
flow_to_process->ndpi_flow->protos.tls_quic.ssl_version,
&unknown_tls_version),
flow_to_process->ndpi_flow->protos.tls_quic_stun.tls_quic.client_requested_server_name,
(flow_to_process->ndpi_flow->protos.tls_quic_stun.tls_quic.alpn != NULL ?
flow_to_process->ndpi_flow->protos.tls_quic_stun.tls_quic.alpn : "-"));
flow_to_process->ndpi_flow->host_server_name,
(flow_to_process->ndpi_flow->protos.tls_quic.alpn != NULL ?
flow_to_process->ndpi_flow->protos.tls_quic.alpn : "-"));
flow_to_process->tls_client_hello_seen = 1;
}
if (flow_to_process->tls_server_hello_seen == 0 &&
flow_to_process->ndpi_flow->l4.tcp.tls.certificate_processed != 0)
{
uint8_t unknown_tls_version = 0;
char buf_ver[16];
printf("[%8llu, %d, %4d][TLS-SERVER-HELLO] version: %s | common-name(s): %.*s | "
"issuer: %s | subject: %s\n",
workflow->packets_captured,
reader_thread->array_index,
flow_to_process->flow_id,
ndpi_ssl_version2str(flow_to_process->ndpi_flow,
flow_to_process->ndpi_flow->protos.tls_quic_stun.tls_quic.ssl_version,
ndpi_ssl_version2str(buf_ver, sizeof(buf_ver),
flow_to_process->ndpi_flow->protos.tls_quic.ssl_version,
&unknown_tls_version),
(flow_to_process->ndpi_flow->protos.tls_quic_stun.tls_quic.server_names_len == 0 ?
1 : flow_to_process->ndpi_flow->protos.tls_quic_stun.tls_quic.server_names_len),
(flow_to_process->ndpi_flow->protos.tls_quic_stun.tls_quic.server_names == NULL ?
"-" : flow_to_process->ndpi_flow->protos.tls_quic_stun.tls_quic.server_names),
(flow_to_process->ndpi_flow->protos.tls_quic_stun.tls_quic.issuerDN != NULL ?
flow_to_process->ndpi_flow->protos.tls_quic_stun.tls_quic.issuerDN : "-"),
(flow_to_process->ndpi_flow->protos.tls_quic_stun.tls_quic.subjectDN != NULL ?
flow_to_process->ndpi_flow->protos.tls_quic_stun.tls_quic.subjectDN : "-"));
(flow_to_process->ndpi_flow->protos.tls_quic.server_names_len == 0 ?
1 : flow_to_process->ndpi_flow->protos.tls_quic.server_names_len),
(flow_to_process->ndpi_flow->protos.tls_quic.server_names == NULL ?
"-" : flow_to_process->ndpi_flow->protos.tls_quic.server_names),
(flow_to_process->ndpi_flow->protos.tls_quic.issuerDN != NULL ?
flow_to_process->ndpi_flow->protos.tls_quic.issuerDN : "-"),
(flow_to_process->ndpi_flow->protos.tls_quic.subjectDN != NULL ?
flow_to_process->ndpi_flow->protos.tls_quic.subjectDN : "-"));
flow_to_process->tls_server_hello_seen = 1;
}
}
@@ -957,7 +956,7 @@ static void run_pcap_loop(struct nDPI_reader_thread const * const reader_thread)
fprintf(stderr, "Error while reading pcap file: '%s'\n",
pcap_geterr(reader_thread->workflow->pcap_handle));
reader_thread->workflow->error_or_eof = 1;
__sync_fetch_and_add(&reader_thread->workflow->error_or_eof, 1);
}
}
}
@@ -978,14 +977,14 @@ static void * processing_thread(void * const ndpi_thread_arg)
printf("Starting Thread %d\n", reader_thread->array_index);
run_pcap_loop(reader_thread);
reader_thread->workflow->error_or_eof = 1;
__sync_fetch_and_add(&reader_thread->workflow->error_or_eof, 1);
return NULL;
}
static int processing_threads_error_or_eof(void)
{
for (int i = 0; i < reader_thread_count; ++i) {
if (reader_threads[i].workflow->error_or_eof == 0) {
if (__sync_fetch_and_add(&reader_threads[i].workflow->error_or_eof, 0) == 0) {
return 0;
}
}
@@ -994,6 +993,7 @@ static int processing_threads_error_or_eof(void)
static int start_reader_threads(void)
{
#ifndef WIN32
sigset_t thread_signal_set, old_signal_set;
sigfillset(&thread_signal_set);
@@ -1003,6 +1003,7 @@ static int start_reader_threads(void)
fprintf(stderr, "pthread_sigmask: %s\n", strerror(errno));
return 1;
}
#endif
for (int i = 0; i < reader_thread_count; ++i) {
reader_threads[i].array_index = i;
@@ -1030,6 +1031,7 @@ static int start_reader_threads(void)
static int stop_reader_threads(void)
{
unsigned long long int total_packets_captured = 0;
unsigned long long int total_packets_processed = 0;
unsigned long long int total_l4_data_len = 0;
unsigned long long int total_flows_captured = 0;
@@ -1047,6 +1049,10 @@ static int stop_reader_threads(void)
continue;
}
if (pthread_join(reader_threads[i].thread_id, NULL) != 0) {
fprintf(stderr, "pthread_join: %s\n", strerror(errno));
}
total_packets_processed += reader_threads[i].workflow->packets_processed;
total_l4_data_len += reader_threads[i].workflow->total_l4_data_len;
total_flows_captured += reader_threads[i].workflow->total_active_flows;
@@ -1059,27 +1065,25 @@ static int stop_reader_threads(void)
reader_threads[i].workflow->total_l4_data_len, reader_threads[i].workflow->total_active_flows,
reader_threads[i].workflow->total_idle_flows, reader_threads[i].workflow->detected_flow_protocols);
}
/* total packets captured: same value for all threads as packet2thread distribution happens later */
printf("Total packets captured.: %llu\n",
reader_threads[0].workflow->packets_captured);
printf("Total packets processed: %llu\n", total_packets_processed);
printf("Total layer4 data size.: %llu\n", total_l4_data_len);
printf("Total flows captured...: %llu\n", total_flows_captured);
printf("Total flows timed out..: %llu\n", total_flows_idle);
printf("Total flows detected...: %llu\n", total_flows_detected);
total_packets_captured = reader_threads[0].workflow->packets_captured;
for (int i = 0; i < reader_thread_count; ++i) {
if (reader_threads[i].workflow == NULL) {
continue;
}
if (pthread_join(reader_threads[i].thread_id, NULL) != 0) {
fprintf(stderr, "pthread_join: %s\n", strerror(errno));
}
free_workflow(&reader_threads[i].workflow);
}
printf("Total packets captured.: %llu\n", total_packets_captured);
printf("Total packets processed: %llu\n", total_packets_processed);
printf("Total layer4 data size.: %llu\n", total_l4_data_len);
printf("Total flows captured...: %llu\n", total_flows_captured);
printf("Total flows timed out..: %llu\n", total_flows_idle);
printf("Total flows detected...: %llu\n", total_flows_detected);
return 0;
}
@@ -1087,12 +1091,8 @@ static void sighandler(int signum)
{
fprintf(stderr, "Received SIGNAL %d\n", signum);
if (main_thread_shutdown == 0) {
main_thread_shutdown = 1;
if (stop_reader_threads() != 0) {
fprintf(stderr, "Failed to stop reader threads!\n");
exit(EXIT_FAILURE);
}
if (__sync_fetch_and_add(&main_thread_shutdown, 0) == 0) {
__sync_fetch_and_add(&main_thread_shutdown, 1);
} else {
fprintf(stderr, "Reader threads are already shutting down, please be patient.\n");
}
@@ -1126,11 +1126,11 @@ int main(int argc, char ** argv)
signal(SIGINT, sighandler);
signal(SIGTERM, sighandler);
while (main_thread_shutdown == 0 && processing_threads_error_or_eof() == 0) {
while (__sync_fetch_and_add(&main_thread_shutdown, 0) == 0 && processing_threads_error_or_eof() == 0) {
sleep(1);
}
if (main_thread_shutdown == 0 && stop_reader_threads() != 0) {
if (stop_reader_threads() != 0) {
fprintf(stderr, "%s: stop_reader_threads\n", argv[0]);
return 1;
}
+4 -2
View File
@@ -51,6 +51,8 @@ ip:54.80.47.130@AmazonPrime
# For Flows with a hostname (e.g. TLS) the risk is also put in AND with the host_risk_mask
#ip_risk_mask:192.168.1.0/24=0
#ip_risk_mask:10.196.157.228=0
host_risk_mask:".local"=0
host_risk_mask:".msftconnecttest.com"=0
host_risk_mask:".home"=0
# Custom certification autorities we trust
trusted_issuer_dn:"CN=813845657003339838, O=Code42, OU=TEST, ST=MN, C=US"
+255 -227
View File
@@ -1,7 +1,7 @@
/*
* reader_util.c
*
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
@@ -26,36 +26,22 @@
#endif
#include <stdlib.h>
#include <math.h>
#include <float.h>
#ifdef WIN32
#include <winsock2.h> /* winsock.h is included automatically */
#include <process.h>
#include <io.h>
#include <ip6_misc.h>
#else
#include <unistd.h>
#include <netinet/in.h>
#include <math.h>
#include <float.h>
#endif
#include "reader_util.h"
#ifndef ETH_P_IP
#define ETH_P_IP 0x0800 /* IPv4 */
#endif
#ifndef ETH_P_IPv6
#define ETH_P_IPV6 0x86dd /* IPv6 */
#endif
#define SLARP 0x8035 /* Cisco Slarp */
#define CISCO_D_PROTO 0x2000 /* Cisco Discovery Protocol */
#define VLAN 0x8100
#define MPLS_UNI 0x8847
#define MPLS_MULTI 0x8848
#define PPPoE 0x8864
#define SNAP 0xaa
#define SNAP 0XAA
#define BSTP 0x42 /* Bridge Spanning Tree Protocol */
/* Keep last 32 packets */
@@ -83,7 +69,7 @@
#include "reader_util.h"
#include "ndpi_classify.h"
extern u_int8_t enable_protocol_guess, enable_joy_stats, enable_payload_analyzer;
extern u_int8_t enable_protocol_guess, enable_flow_stats, enable_payload_analyzer;
extern u_int8_t verbose, human_readeable_string_len;
extern u_int8_t max_num_udp_dissected_pkts /* 24 */, max_num_tcp_dissected_pkts /* 80 */;
static u_int32_t flow_id = 0;
@@ -314,8 +300,6 @@ void ndpi_report_payload_stats() {
void ndpi_free_flow_info_half(struct ndpi_flow_info *flow) {
if(flow->ndpi_flow) { ndpi_flow_free(flow->ndpi_flow); flow->ndpi_flow = NULL; }
if(flow->src_id) { ndpi_free(flow->src_id); flow->src_id = NULL; }
if(flow->dst_id) { ndpi_free(flow->dst_id); flow->dst_id = NULL; }
}
/* ***************************************************** */
@@ -461,6 +445,29 @@ void ndpi_flow_info_freer(void *node) {
static void ndpi_free_flow_tls_data(struct ndpi_flow_info *flow) {
if(flow->dhcp_fingerprint) {
ndpi_free(flow->dhcp_fingerprint);
flow->dhcp_fingerprint = NULL;
}
if(flow->dhcp_class_ident) {
ndpi_free(flow->dhcp_class_ident);
flow->dhcp_class_ident = NULL;
}
if(flow->bittorent_hash) {
ndpi_free(flow->bittorent_hash);
flow->bittorent_hash = NULL;
}
if(flow->telnet.username) {
ndpi_free(flow->telnet.username);
flow->telnet.username = NULL;
}
if(flow->telnet.password) {
ndpi_free(flow->telnet.password);
flow->telnet.password = NULL;
}
if(flow->ssh_tls.server_names) {
ndpi_free(flow->ssh_tls.server_names);
flow->ssh_tls.server_names = NULL;
@@ -502,6 +509,9 @@ static void ndpi_free_flow_data_analysis(struct ndpi_flow_info *flow) {
if(flow->pktlen_s_to_c) ndpi_free_data_analysis(flow->pktlen_s_to_c, 1);
if(flow->iat_flow) ndpi_free_data_analysis(flow->iat_flow, 1);
if(flow->entropy) ndpi_free(flow->entropy);
if(flow->last_entropy) ndpi_free(flow->last_entropy);
}
/* ***************************************************** */
@@ -590,14 +600,14 @@ ndpi_flow_update_byte_count(struct ndpi_flow_info *flow, const void *x,
* the 4000th octet has been seen for a flow.
*/
if((flow->entropy.src2dst_pkt_count+flow->entropy.dst2src_pkt_count) <= max_num_packets_per_flow) {
if((flow->entropy->src2dst_pkt_count+flow->entropy->dst2src_pkt_count) <= max_num_packets_per_flow) {
/* octet count was already incremented before processing this payload */
u_int32_t current_count;
if(src_to_dst_direction) {
current_count = flow->entropy.src2dst_l4_bytes - len;
current_count = flow->entropy->src2dst_l4_bytes - len;
} else {
current_count = flow->entropy.dst2src_l4_bytes - len;
current_count = flow->entropy->dst2src_l4_bytes - len;
}
if(current_count < ETTA_MIN_OCTETS) {
@@ -606,9 +616,9 @@ ndpi_flow_update_byte_count(struct ndpi_flow_info *flow, const void *x,
for(i=0; i<len; i++) {
if(src_to_dst_direction) {
flow->entropy.src2dst_byte_count[data[i]]++;
flow->entropy->src2dst_byte_count[data[i]]++;
} else {
flow->entropy.dst2src_byte_count[data[i]]++;
flow->entropy->dst2src_byte_count[data[i]]++;
}
current_count++;
if(current_count >= ETTA_MIN_OCTETS) {
@@ -633,22 +643,22 @@ ndpi_flow_update_byte_dist_mean_var(ndpi_flow_info_t *flow, const void *x,
unsigned int len, u_int8_t src_to_dst_direction) {
const unsigned char *data = x;
if((flow->entropy.src2dst_pkt_count+flow->entropy.dst2src_pkt_count) <= max_num_packets_per_flow) {
if((flow->entropy->src2dst_pkt_count+flow->entropy->dst2src_pkt_count) <= max_num_packets_per_flow) {
unsigned int i;
for(i=0; i<len; i++) {
double delta;
if(src_to_dst_direction) {
flow->entropy.src2dst_num_bytes += 1;
delta = ((double)data[i] - flow->entropy.src2dst_bd_mean);
flow->entropy.src2dst_bd_mean += delta/((double)flow->entropy.src2dst_num_bytes);
flow->entropy.src2dst_bd_variance += delta*((double)data[i] - flow->entropy.src2dst_bd_mean);
flow->entropy->src2dst_num_bytes += 1;
delta = ((double)data[i] - flow->entropy->src2dst_bd_mean);
flow->entropy->src2dst_bd_mean += delta/((double)flow->entropy->src2dst_num_bytes);
flow->entropy->src2dst_bd_variance += delta*((double)data[i] - flow->entropy->src2dst_bd_mean);
} else {
flow->entropy.dst2src_num_bytes += 1;
delta = ((double)data[i] - flow->entropy.dst2src_bd_mean);
flow->entropy.dst2src_bd_mean += delta/((double)flow->entropy.dst2src_num_bytes);
flow->entropy.dst2src_bd_variance += delta*((double)data[i] - flow->entropy.dst2src_bd_mean);
flow->entropy->dst2src_num_bytes += 1;
delta = ((double)data[i] - flow->entropy->dst2src_bd_mean);
flow->entropy->dst2src_bd_mean += delta/((double)flow->entropy->dst2src_num_bytes);
flow->entropy->dst2src_bd_variance += delta*((double)data[i] - flow->entropy->dst2src_bd_mean);
}
}
}
@@ -683,17 +693,16 @@ static struct ndpi_flow_info *get_ndpi_flow_info(struct ndpi_workflow * workflow
u_int16_t ip_offset,
u_int16_t ipsize,
u_int16_t l4_packet_len,
u_int16_t l4_offset,
struct ndpi_tcphdr **tcph,
struct ndpi_udphdr **udph,
u_int16_t *sport, u_int16_t *dport,
struct ndpi_id_struct **src,
struct ndpi_id_struct **dst,
u_int8_t *proto,
u_int8_t **payload,
u_int16_t *payload_len,
u_int8_t *src_to_dst_direction,
pkt_timeval when) {
u_int32_t idx, l4_offset, hashval;
u_int32_t idx, hashval;
struct ndpi_flow_info flow;
void *ret;
const u_int8_t *l3, *l4;
@@ -711,11 +720,9 @@ static struct ndpi_flow_info *get_ndpi_flow_info(struct ndpi_workflow * workflow
/* || (iph->frag_off & htons(0x1FFF)) != 0 */)
return NULL;
l4_offset = iph->ihl * 4;
l3 = (const u_int8_t*)iph;
} else {
l4_offset = sizeof(struct ndpi_ipv6hdr);
if(sizeof(struct ndpi_ipv6hdr) > ipsize)
if(l4_offset > ipsize)
return NULL;
l3 = (const u_int8_t*)iph6;
@@ -844,7 +851,7 @@ static struct ndpi_flow_info *get_ndpi_flow_info(struct ndpi_workflow * workflow
#else
ndpi_init_bin(&newflow->payload_len_bin, ndpi_bin_family8, PLEN_NUM_BINS);
#endif
if(version == IPVERSION) {
inet_ntop(AF_INET, &newflow->src_ip, newflow->src_name, sizeof(newflow->src_name));
inet_ntop(AF_INET, &newflow->dst_ip, newflow->dst_name, sizeof(newflow->dst_name));
@@ -867,30 +874,6 @@ static struct ndpi_flow_info *get_ndpi_flow_info(struct ndpi_workflow * workflow
} else
memset(newflow->ndpi_flow, 0, SIZEOF_FLOW_STRUCT);
if((newflow->src_id = ndpi_malloc(SIZEOF_ID_STRUCT)) == NULL) {
LOG(NDPI_LOG_ERROR, "[NDPI] %s(3): not enough memory\n", __FUNCTION__);
#ifdef DIRECTION_BINS
ndpi_free_bin(&newflow->payload_len_bin_src2dst), ndpi_free_bin(&newflow->payload_len_bin_dst2src);
#else
ndpi_free_bin(&newflow->payload_len_bin);
#endif
ndpi_free(newflow);
return(NULL);
} else
memset(newflow->src_id, 0, SIZEOF_ID_STRUCT);
if((newflow->dst_id = ndpi_malloc(SIZEOF_ID_STRUCT)) == NULL) {
LOG(NDPI_LOG_ERROR, "[NDPI] %s(4): not enough memory\n", __FUNCTION__);
#ifdef DIRECTION_BINS
ndpi_free_bin(&newflow->payload_len_bin_src2dst), ndpi_free_bin(&newflow->payload_len_bin_dst2src);
#else
ndpi_free_bin(&newflow->payload_len_bin);
#endif
ndpi_free(newflow);
return(NULL);
} else
memset(newflow->dst_id, 0, SIZEOF_ID_STRUCT);
ndpi_tsearch(newflow, &workflow->ndpi_flows_root[idx], ndpi_workflow_node_cmp); /* Add */
workflow->stats.ndpi_flow_count++;
if(*proto == IPPROTO_TCP)
@@ -900,17 +883,20 @@ static struct ndpi_flow_info *get_ndpi_flow_info(struct ndpi_workflow * workflow
else
workflow->stats.flow_count[2]++;
*src = newflow->src_id, *dst = newflow->dst_id;
newflow->entropy.src2dst_pkt_len[newflow->entropy.src2dst_pkt_count] = l4_data_len;
newflow->entropy.src2dst_pkt_time[newflow->entropy.src2dst_pkt_count] = when;
if(newflow->entropy.src2dst_pkt_count == 0) {
newflow->entropy.src2dst_start = when;
}
newflow->entropy.src2dst_pkt_count++;
// Non zero app data.
if(l4_data_len != 0XFEEDFACE && l4_data_len != 0) {
newflow->entropy.src2dst_opackets++;
newflow->entropy.src2dst_l4_bytes += l4_data_len;
if(enable_flow_stats) {
newflow->entropy = ndpi_calloc(1, sizeof(struct ndpi_entropy));
newflow->last_entropy = ndpi_calloc(1, sizeof(struct ndpi_entropy));
newflow->entropy->src2dst_pkt_len[newflow->entropy->src2dst_pkt_count] = l4_data_len;
newflow->entropy->src2dst_pkt_time[newflow->entropy->src2dst_pkt_count] = when;
if(newflow->entropy->src2dst_pkt_count == 0) {
newflow->entropy->src2dst_start = when;
}
newflow->entropy->src2dst_pkt_count++;
// Non zero app data.
if(l4_data_len != 0XFEEDFACE && l4_data_len != 0) {
newflow->entropy->src2dst_opackets++;
newflow->entropy->src2dst_l4_bytes += l4_data_len;
}
}
return newflow;
}
@@ -923,9 +909,9 @@ static struct ndpi_flow_info *get_ndpi_flow_info(struct ndpi_workflow * workflow
&& rflow->src_port == htons(*sport)
&& rflow->dst_port == htons(*dport)
)
*src = rflow->dst_id, *dst = rflow->src_id, *src_to_dst_direction = 0, rflow->bidirectional = 1;
*src_to_dst_direction = 0, rflow->bidirectional = 1;
else
*src = rflow->src_id, *dst = rflow->dst_id, *src_to_dst_direction = 1;
*src_to_dst_direction = 1;
}
else {
if(rflow->src_ip == iph->saddr
@@ -933,34 +919,36 @@ static struct ndpi_flow_info *get_ndpi_flow_info(struct ndpi_workflow * workflow
&& rflow->src_port == htons(*sport)
&& rflow->dst_port == htons(*dport)
)
*src = rflow->src_id, *dst = rflow->dst_id, *src_to_dst_direction = 1;
*src_to_dst_direction = 1;
else
*src = rflow->dst_id, *dst = rflow->src_id, *src_to_dst_direction = 0, rflow->bidirectional = 1;
*src_to_dst_direction = 0, rflow->bidirectional = 1;
}
if(src_to_dst_direction) {
if(rflow->entropy.src2dst_pkt_count < max_num_packets_per_flow) {
rflow->entropy.src2dst_pkt_len[rflow->entropy.src2dst_pkt_count] = l4_data_len;
rflow->entropy.src2dst_pkt_time[rflow->entropy.src2dst_pkt_count] = when;
rflow->entropy.src2dst_l4_bytes += l4_data_len;
rflow->entropy.src2dst_pkt_count++;
}
// Non zero app data.
if(l4_data_len != 0XFEEDFACE && l4_data_len != 0) {
rflow->entropy.src2dst_opackets++;
}
} else {
if(rflow->entropy.dst2src_pkt_count < max_num_packets_per_flow) {
rflow->entropy.dst2src_pkt_len[rflow->entropy.dst2src_pkt_count] = l4_data_len;
rflow->entropy.dst2src_pkt_time[rflow->entropy.dst2src_pkt_count] = when;
if(rflow->entropy.dst2src_pkt_count == 0) {
rflow->entropy.dst2src_start = when;
if(enable_flow_stats) {
if(src_to_dst_direction) {
if(rflow->entropy->src2dst_pkt_count < max_num_packets_per_flow) {
rflow->entropy->src2dst_pkt_len[rflow->entropy->src2dst_pkt_count] = l4_data_len;
rflow->entropy->src2dst_pkt_time[rflow->entropy->src2dst_pkt_count] = when;
rflow->entropy->src2dst_l4_bytes += l4_data_len;
rflow->entropy->src2dst_pkt_count++;
}
// Non zero app data.
if(l4_data_len != 0XFEEDFACE && l4_data_len != 0) {
rflow->entropy->src2dst_opackets++;
}
} else {
if(rflow->entropy->dst2src_pkt_count < max_num_packets_per_flow) {
rflow->entropy->dst2src_pkt_len[rflow->entropy->dst2src_pkt_count] = l4_data_len;
rflow->entropy->dst2src_pkt_time[rflow->entropy->dst2src_pkt_count] = when;
if(rflow->entropy->dst2src_pkt_count == 0) {
rflow->entropy->dst2src_start = when;
}
rflow->entropy->dst2src_l4_bytes += l4_data_len;
rflow->entropy->dst2src_pkt_count++;
}
// Non zero app data.
if(l4_data_len != 0XFEEDFACE && l4_data_len != 0) {
rflow->entropy->dst2src_opackets++;
}
rflow->entropy.dst2src_l4_bytes += l4_data_len;
rflow->entropy.dst2src_pkt_count++;
}
// Non zero app data.
if(l4_data_len != 0XFEEDFACE && l4_data_len != 0) {
rflow->entropy.dst2src_opackets++;
}
}
@@ -979,8 +967,6 @@ static struct ndpi_flow_info *get_ndpi_flow_info6(struct ndpi_workflow * workflo
struct ndpi_tcphdr **tcph,
struct ndpi_udphdr **udph,
u_int16_t *sport, u_int16_t *dport,
struct ndpi_id_struct **src,
struct ndpi_id_struct **dst,
u_int8_t *proto,
u_int8_t **payload,
u_int16_t *payload_len,
@@ -988,6 +974,8 @@ static struct ndpi_flow_info *get_ndpi_flow_info6(struct ndpi_workflow * workflo
pkt_timeval when) {
struct ndpi_iphdr iph;
if(ipsize < 40)
return(NULL);
memset(&iph, 0, sizeof(iph));
iph.version = IPVERSION;
iph.saddr = iph6->ip6_src.u6_addr.u6_addr32[2] + iph6->ip6_src.u6_addr.u6_addr32[3];
@@ -995,16 +983,16 @@ static struct ndpi_flow_info *get_ndpi_flow_info6(struct ndpi_workflow * workflo
u_int8_t l4proto = iph6->ip6_hdr.ip6_un1_nxt;
u_int16_t ip_len = ntohs(iph6->ip6_hdr.ip6_un1_plen);
const u_int8_t *l4ptr = (((const u_int8_t *) iph6) + sizeof(struct ndpi_ipv6hdr));
if(ndpi_handle_ipv6_extension_headers(NULL, &l4ptr, &ip_len, &l4proto) != 0) {
if(ndpi_handle_ipv6_extension_headers(ipsize - sizeof(struct ndpi_ipv6hdr), &l4ptr, &ip_len, &l4proto) != 0) {
return(NULL);
}
iph.protocol = l4proto;
return(get_ndpi_flow_info(workflow, 6, vlan_id, tunnel_type,
&iph, iph6, ip_offset, ipsize,
ntohs(iph6->ip6_hdr.ip6_un1_plen),
ip_len, l4ptr - (const u_int8_t *)iph6,
tcph, udph, sport, dport,
src, dst, proto, payload,
proto, payload,
payload_len, src_to_dst_direction, when));
}
@@ -1022,15 +1010,16 @@ static u_int8_t is_ndpi_proto(struct ndpi_flow_info *flow, u_int16_t id) {
void correct_csv_data_field(char* data) {
/* Replace , with ; to avoid issues with CSVs */
for(u_int i=0; data[i] != '\0'; i++) if(data[i] == ',') data[i] = ';';
u_int i;
for(i=0; data[i] != '\0'; i++) if(data[i] == ',') data[i] = ';';
}
/* ****************************************************** */
u_int8_t plen2slot(u_int16_t plen) {
/*
u_int8_t plen2slot(u_int16_t plen) {
/*
Slots [32 bytes lenght]
0..31, 32..63 ...
0..31, 32..63 ...
*/
if(plen > PLEN_MAX)
@@ -1046,6 +1035,8 @@ void process_ndpi_collected_info(struct ndpi_workflow * workflow, struct ndpi_fl
if(!flow->ndpi_flow) return;
flow->confidence = flow->ndpi_flow->confidence;
snprintf(flow->host_server_name, sizeof(flow->host_server_name), "%s",
flow->ndpi_flow->host_server_name);
@@ -1055,18 +1046,24 @@ void process_ndpi_collected_info(struct ndpi_workflow * workflow, struct ndpi_fl
flow->risk = flow->ndpi_flow->risk;
if(is_ndpi_proto(flow, NDPI_PROTOCOL_DHCP)) {
snprintf(flow->dhcp_fingerprint, sizeof(flow->dhcp_fingerprint), "%s", flow->ndpi_flow->protos.dhcp.fingerprint);
} else if(is_ndpi_proto(flow, NDPI_PROTOCOL_BITTORRENT)) {
u_int j, n = 0;
if(flow->ndpi_flow->protos.dhcp.fingerprint[0] != '\0')
flow->dhcp_fingerprint = ndpi_strdup(flow->ndpi_flow->protos.dhcp.fingerprint);
if(flow->ndpi_flow->protos.dhcp.class_ident[0] != '\0')
flow->dhcp_class_ident = ndpi_strdup(flow->ndpi_flow->protos.dhcp.class_ident);
} else if(is_ndpi_proto(flow, NDPI_PROTOCOL_BITTORRENT) &&
!is_ndpi_proto(flow, NDPI_PROTOCOL_TLS)) {
u_int j;
for(i=0, j = 0; j < sizeof(flow->bittorent_hash)-1; i++) {
sprintf(&flow->bittorent_hash[j], "%02x",
flow->ndpi_flow->protos.bittorrent.hash[i]);
if(flow->ndpi_flow->protos.bittorrent.hash[0] != '\0') {
flow->bittorent_hash = ndpi_malloc(sizeof(flow->ndpi_flow->protos.bittorrent.hash) * 2 + 1);
for(i=0, j = 0; i < sizeof(flow->ndpi_flow->protos.bittorrent.hash); i++) {
sprintf(&flow->bittorent_hash[j], "%02x",
flow->ndpi_flow->protos.bittorrent.hash[i]);
j += 2, n += flow->ndpi_flow->protos.bittorrent.hash[i];
j += 2;
}
flow->bittorent_hash[j] = '\0';
}
if(n == 0) flow->bittorent_hash[0] = '\0';
}
/* DNS */
else if(is_ndpi_proto(flow, NDPI_PROTOCOL_DNS)) {
@@ -1081,8 +1078,7 @@ void process_ndpi_collected_info(struct ndpi_workflow * workflow, struct ndpi_fl
}
/* MDNS */
else if(is_ndpi_proto(flow, NDPI_PROTOCOL_MDNS)) {
char *name = (char*)flow->ndpi_flow->host_server_name; /* Trick to avoid warning(s) */
snprintf(flow->info, sizeof(flow->info), "%s", name);
snprintf(flow->info, sizeof(flow->info), "%s", flow->ndpi_flow->host_server_name);
}
/* UBNTAC2 */
else if(is_ndpi_proto(flow, NDPI_PROTOCOL_UBNTAC2)) {
@@ -1093,11 +1089,11 @@ void process_ndpi_collected_info(struct ndpi_workflow * workflow, struct ndpi_fl
|| /* IMAP */ is_ndpi_proto(flow, NDPI_PROTOCOL_MAIL_IMAP)
|| /* POP */ is_ndpi_proto(flow, NDPI_PROTOCOL_MAIL_POP)
|| /* SMTP */ is_ndpi_proto(flow, NDPI_PROTOCOL_MAIL_SMTP)) {
if(flow->ndpi_flow->protos.ftp_imap_pop_smtp.username[0] != '\0')
if(flow->ndpi_flow->l4.tcp.ftp_imap_pop_smtp.username[0] != '\0')
snprintf(flow->info, sizeof(flow->info), "User: %s][Pwd: %s%s",
flow->ndpi_flow->protos.ftp_imap_pop_smtp.username,
flow->ndpi_flow->protos.ftp_imap_pop_smtp.password,
flow->ndpi_flow->protos.ftp_imap_pop_smtp.auth_failed ? "][Auth Failed" : "");
flow->ndpi_flow->l4.tcp.ftp_imap_pop_smtp.username,
flow->ndpi_flow->l4.tcp.ftp_imap_pop_smtp.password,
flow->ndpi_flow->l4.tcp.ftp_imap_pop_smtp.auth_failed ? "][Auth Failed" : "");
}
/* KERBEROS */
else if(is_ndpi_proto(flow, NDPI_PROTOCOL_KERBEROS)) {
@@ -1132,11 +1128,13 @@ void process_ndpi_collected_info(struct ndpi_workflow * workflow, struct ndpi_fl
snprintf(flow->http.user_agent, sizeof(flow->http.user_agent), "%s", flow->ndpi_flow->http.user_agent ? flow->ndpi_flow->http.user_agent : "");
}
} else if(is_ndpi_proto(flow, NDPI_PROTOCOL_TELNET)) {
snprintf(flow->telnet.username, sizeof(flow->telnet.username), "%s", flow->ndpi_flow->protos.telnet.username);
snprintf(flow->telnet.password, sizeof(flow->telnet.password), "%s", flow->ndpi_flow->protos.telnet.password);
if(flow->ndpi_flow->protos.telnet.username[0] != '\0')
flow->telnet.username = ndpi_strdup(flow->ndpi_flow->protos.telnet.username);
if(flow->ndpi_flow->protos.telnet.password[0] != '\0')
flow->telnet.password = ndpi_strdup(flow->ndpi_flow->protos.telnet.password);
} else if(is_ndpi_proto(flow, NDPI_PROTOCOL_SSH)) {
snprintf(flow->ssh_tls.client_requested_server_name,
sizeof(flow->ssh_tls.client_requested_server_name), "%s",
snprintf(flow->host_server_name,
sizeof(flow->host_server_name), "%s",
flow->ndpi_flow->protos.ssh.client_signature);
snprintf(flow->ssh_tls.server_info, sizeof(flow->ssh_tls.server_info), "%s",
flow->ndpi_flow->protos.ssh.server_signature);
@@ -1149,86 +1147,83 @@ void process_ndpi_collected_info(struct ndpi_workflow * workflow, struct ndpi_fl
else if((is_ndpi_proto(flow, NDPI_PROTOCOL_TLS))
|| ((is_quic = is_ndpi_proto(flow, NDPI_PROTOCOL_QUIC)))
|| (flow->detected_protocol.master_protocol == NDPI_PROTOCOL_TLS)
|| (flow->ndpi_flow->protos.tls_quic_stun.tls_quic.ja3_client[0] != '\0')
|| (flow->ndpi_flow->protos.tls_quic.ja3_client[0] != '\0')
) {
flow->ssh_tls.ssl_version = flow->ndpi_flow->protos.tls_quic_stun.tls_quic.ssl_version;
snprintf(flow->ssh_tls.client_requested_server_name,
sizeof(flow->ssh_tls.client_requested_server_name), "%s",
flow->ndpi_flow->protos.tls_quic_stun.tls_quic.client_requested_server_name);
flow->ssh_tls.ssl_version = flow->ndpi_flow->protos.tls_quic.ssl_version;
snprintf(flow->http.user_agent, sizeof(flow->http.user_agent), "%s", flow->ndpi_flow->http.user_agent ? flow->ndpi_flow->http.user_agent : "");
if(flow->ndpi_flow->protos.tls_quic_stun.tls_quic.server_names_len > 0 && flow->ndpi_flow->protos.tls_quic_stun.tls_quic.server_names)
flow->ssh_tls.server_names = ndpi_strdup(flow->ndpi_flow->protos.tls_quic_stun.tls_quic.server_names);
if(flow->ndpi_flow->protos.tls_quic.server_names_len > 0 && flow->ndpi_flow->protos.tls_quic.server_names)
flow->ssh_tls.server_names = ndpi_strdup(flow->ndpi_flow->protos.tls_quic.server_names);
flow->ssh_tls.notBefore = flow->ndpi_flow->protos.tls_quic_stun.tls_quic.notBefore;
flow->ssh_tls.notAfter = flow->ndpi_flow->protos.tls_quic_stun.tls_quic.notAfter;
flow->ssh_tls.notBefore = flow->ndpi_flow->protos.tls_quic.notBefore;
flow->ssh_tls.notAfter = flow->ndpi_flow->protos.tls_quic.notAfter;
snprintf(flow->ssh_tls.ja3_client, sizeof(flow->ssh_tls.ja3_client), "%s",
flow->ndpi_flow->protos.tls_quic_stun.tls_quic.ja3_client);
flow->ndpi_flow->protos.tls_quic.ja3_client);
snprintf(flow->ssh_tls.ja3_server, sizeof(flow->ssh_tls.ja3_server), "%s",
flow->ndpi_flow->protos.tls_quic_stun.tls_quic.ja3_server);
flow->ssh_tls.server_unsafe_cipher = flow->ndpi_flow->protos.tls_quic_stun.tls_quic.server_unsafe_cipher;
flow->ssh_tls.server_cipher = flow->ndpi_flow->protos.tls_quic_stun.tls_quic.server_cipher;
flow->ndpi_flow->protos.tls_quic.ja3_server);
flow->ssh_tls.server_unsafe_cipher = flow->ndpi_flow->protos.tls_quic.server_unsafe_cipher;
flow->ssh_tls.server_cipher = flow->ndpi_flow->protos.tls_quic.server_cipher;
if(flow->ndpi_flow->l4.tcp.tls.fingerprint_set) {
memcpy(flow->ssh_tls.sha1_cert_fingerprint,
flow->ndpi_flow->protos.tls_quic_stun.tls_quic.sha1_certificate_fingerprint, 20);
flow->ndpi_flow->protos.tls_quic.sha1_certificate_fingerprint, 20);
flow->ssh_tls.sha1_cert_fingerprint_set = 1;
}
flow->ssh_tls.browser_heuristics = flow->ndpi_flow->protos.tls_quic_stun.tls_quic.browser_heuristics;
if(flow->ndpi_flow->protos.tls_quic_stun.tls_quic.alpn) {
if((flow->ssh_tls.tls_alpn = ndpi_strdup(flow->ndpi_flow->protos.tls_quic_stun.tls_quic.alpn)) != NULL)
flow->ssh_tls.browser_heuristics = flow->ndpi_flow->protos.tls_quic.browser_heuristics;
if(flow->ndpi_flow->protos.tls_quic.alpn) {
if((flow->ssh_tls.tls_alpn = ndpi_strdup(flow->ndpi_flow->protos.tls_quic.alpn)) != NULL)
correct_csv_data_field(flow->ssh_tls.tls_alpn);
}
if(flow->ndpi_flow->protos.tls_quic_stun.tls_quic.issuerDN)
flow->ssh_tls.tls_issuerDN = strdup(flow->ndpi_flow->protos.tls_quic_stun.tls_quic.issuerDN);
if(flow->ndpi_flow->protos.tls_quic_stun.tls_quic.subjectDN)
flow->ssh_tls.tls_subjectDN = strdup(flow->ndpi_flow->protos.tls_quic_stun.tls_quic.subjectDN);
if(flow->ndpi_flow->protos.tls_quic.issuerDN)
flow->ssh_tls.tls_issuerDN = strdup(flow->ndpi_flow->protos.tls_quic.issuerDN);
if(flow->ndpi_flow->protos.tls_quic_stun.tls_quic.encrypted_sni.esni) {
flow->ssh_tls.encrypted_sni.esni = strdup(flow->ndpi_flow->protos.tls_quic_stun.tls_quic.encrypted_sni.esni);
flow->ssh_tls.encrypted_sni.cipher_suite = flow->ndpi_flow->protos.tls_quic_stun.tls_quic.encrypted_sni.cipher_suite;
if(flow->ndpi_flow->protos.tls_quic.subjectDN)
flow->ssh_tls.tls_subjectDN = strdup(flow->ndpi_flow->protos.tls_quic.subjectDN);
if(flow->ndpi_flow->protos.tls_quic.encrypted_sni.esni) {
flow->ssh_tls.encrypted_sni.esni = strdup(flow->ndpi_flow->protos.tls_quic.encrypted_sni.esni);
flow->ssh_tls.encrypted_sni.cipher_suite = flow->ndpi_flow->protos.tls_quic.encrypted_sni.cipher_suite;
}
if(flow->ssh_tls.tls_supported_versions) {
if((flow->ssh_tls.tls_supported_versions = ndpi_strdup(flow->ndpi_flow->protos.tls_quic_stun.tls_quic.tls_supported_versions)) != NULL)
if((flow->ssh_tls.tls_supported_versions = ndpi_strdup(flow->ndpi_flow->protos.tls_quic.tls_supported_versions)) != NULL)
correct_csv_data_field(flow->ssh_tls.tls_supported_versions);
}
if(flow->ndpi_flow->protos.tls_quic_stun.tls_quic.alpn
&& flow->ndpi_flow->protos.tls_quic_stun.tls_quic.tls_supported_versions) {
correct_csv_data_field(flow->ndpi_flow->protos.tls_quic_stun.tls_quic.alpn);
correct_csv_data_field(flow->ndpi_flow->protos.tls_quic_stun.tls_quic.tls_supported_versions);
if(flow->ndpi_flow->protos.tls_quic.alpn
&& flow->ndpi_flow->protos.tls_quic.tls_supported_versions) {
correct_csv_data_field(flow->ndpi_flow->protos.tls_quic.alpn);
correct_csv_data_field(flow->ndpi_flow->protos.tls_quic.tls_supported_versions);
if(csv_fp)
snprintf(flow->info, sizeof(flow->info), "%s",
flow->ndpi_flow->protos.tls_quic_stun.tls_quic.alpn);
flow->ndpi_flow->protos.tls_quic.alpn);
else
snprintf(flow->info, sizeof(flow->info), "ALPN: %s][TLS Supported Versions: %s",
flow->ndpi_flow->protos.tls_quic_stun.tls_quic.alpn,
flow->ndpi_flow->protos.tls_quic_stun.tls_quic.tls_supported_versions);
} else if(flow->ndpi_flow->protos.tls_quic_stun.tls_quic.alpn) {
correct_csv_data_field(flow->ndpi_flow->protos.tls_quic_stun.tls_quic.alpn);
flow->ndpi_flow->protos.tls_quic.alpn,
flow->ndpi_flow->protos.tls_quic.tls_supported_versions);
} else if(flow->ndpi_flow->protos.tls_quic.alpn) {
correct_csv_data_field(flow->ndpi_flow->protos.tls_quic.alpn);
if(csv_fp)
snprintf(flow->info, sizeof(flow->info), "%s,",
flow->ndpi_flow->protos.tls_quic_stun.tls_quic.alpn);
flow->ndpi_flow->protos.tls_quic.alpn);
else
snprintf(flow->info, sizeof(flow->info), "ALPN: %s",
flow->ndpi_flow->protos.tls_quic_stun.tls_quic.alpn);
flow->ndpi_flow->protos.tls_quic.alpn);
}
if(enable_doh_dot_detection) {
/* For TLS we use TLS block lenght instead of payload lenght */
ndpi_reset_bin(&flow->payload_len_bin);
for(i=0; i<flow->ndpi_flow->l4.tcp.tls.num_tls_blocks; i++) {
u_int16_t len = abs(flow->ndpi_flow->l4.tcp.tls.tls_application_blocks_len[i]);
/* printf("[TLS_LEN] %u\n", len); */
ndpi_inc_bin(&flow->payload_len_bin, plen2slot(len), 1);
}
@@ -1255,9 +1250,11 @@ void process_ndpi_collected_info(struct ndpi_workflow * workflow, struct ndpi_fl
*/
static void
ndpi_clear_entropy_stats(struct ndpi_flow_info *flow) {
if(flow->entropy.src2dst_pkt_count + flow->entropy.dst2src_pkt_count == max_num_packets_per_flow) {
memcpy(&flow->last_entropy, &flow->entropy, sizeof(struct ndpi_entropy));
memset(&flow->entropy, 0x00, sizeof(struct ndpi_entropy));
if(enable_flow_stats) {
if(flow->entropy->src2dst_pkt_count + flow->entropy->dst2src_pkt_count == max_num_packets_per_flow) {
memcpy(flow->last_entropy, flow->entropy, sizeof(struct ndpi_entropy));
memset(flow->entropy, 0x00, sizeof(struct ndpi_entropy));
}
}
}
@@ -1318,7 +1315,6 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
pkt_timeval when,
ndpi_risk *flow_risk,
FILE * csv_fp) {
struct ndpi_id_struct *src, *dst;
struct ndpi_flow_info *flow = NULL;
struct ndpi_flow_struct *ndpi_flow = NULL;
u_int8_t proto;
@@ -1331,21 +1327,22 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
struct ndpi_proto nproto = NDPI_PROTOCOL_NULL;
if(workflow->prefs.ignore_vlanid)
vlan_id = 0;
vlan_id = 0;
if(iph)
flow = get_ndpi_flow_info(workflow, IPVERSION, vlan_id,
tunnel_type, iph, NULL,
ip_offset, ipsize,
ntohs(iph->tot_len) - (iph->ihl * 4),
iph->ihl * 4,
&tcph, &udph, &sport, &dport,
&src, &dst, &proto,
&proto,
&payload, &payload_len, &src_to_dst_direction, when);
else
flow = get_ndpi_flow_info6(workflow, vlan_id,
tunnel_type, iph6, ip_offset, ipsize,
&tcph, &udph, &sport, &dport,
&src, &dst, &proto,
&proto,
&payload, &payload_len, &src_to_dst_direction, when);
if(flow != NULL) {
@@ -1368,8 +1365,8 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
if((tcph != NULL) && (tcph->fin || tcph->rst || tcph->syn))
begin_or_end_tcp = 1;
if(flow->entropy.flow_last_pkt_time.tv_sec) {
ndpi_timer_sub(&when, &flow->entropy.flow_last_pkt_time, &tdiff);
if(flow->flow_last_pkt_time.tv_sec) {
ndpi_timer_sub(&when, &flow->flow_last_pkt_time, &tdiff);
if(flow->iat_flow
&& (tdiff.tv_sec >= 0) /* Discard backward time */
@@ -1381,11 +1378,11 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
}
}
memcpy(&flow->entropy.flow_last_pkt_time, &when, sizeof(when));
if(src_to_dst_direction) {
if(flow->entropy.src2dst_last_pkt_time.tv_sec) {
ndpi_timer_sub(&when, &flow->entropy.src2dst_last_pkt_time, &tdiff);
memcpy(&flow->flow_last_pkt_time, &when, sizeof(when));
if(src_to_dst_direction) {
if(flow->src2dst_last_pkt_time.tv_sec) {
ndpi_timer_sub(&when, &flow->src2dst_last_pkt_time, &tdiff);
if(flow->iat_c_to_s
&& (tdiff.tv_sec >= 0) /* Discard backward time */
@@ -1398,15 +1395,15 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
ndpi_data_add_value(flow->pktlen_c_to_s, rawsize);
flow->src2dst_packets++, flow->src2dst_bytes += rawsize, flow->src2dst_goodput_bytes += payload_len;
memcpy(&flow->entropy.src2dst_last_pkt_time, &when, sizeof(when));
memcpy(&flow->src2dst_last_pkt_time, &when, sizeof(when));
#ifdef DIRECTION_BINS
if(payload_len && (flow->src2dst_packets < MAX_NUM_BIN_PKTS))
ndpi_inc_bin(&flow->payload_len_bin_src2dst, plen2slot(payload_len));
#endif
} else {
if(flow->entropy.dst2src_last_pkt_time.tv_sec && (!begin_or_end_tcp)) {
ndpi_timer_sub(&when, &flow->entropy.dst2src_last_pkt_time, &tdiff);
} else {
if(flow->dst2src_last_pkt_time.tv_sec && (!begin_or_end_tcp)) {
ndpi_timer_sub(&when, &flow->dst2src_last_pkt_time, &tdiff);
if(flow->iat_s_to_c) {
u_int32_t ms = ndpi_timeval_to_milliseconds(tdiff);
@@ -1416,7 +1413,7 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
}
ndpi_data_add_value(flow->pktlen_s_to_c, rawsize);
flow->dst2src_packets++, flow->dst2src_bytes += rawsize, flow->dst2src_goodput_bytes += payload_len;
memcpy(&flow->entropy.dst2src_last_pkt_time, &when, sizeof(when));
memcpy(&flow->dst2src_last_pkt_time, &when, sizeof(when));
#ifdef DIRECTION_BINS
if(payload_len && (flow->dst2src_packets < MAX_NUM_BIN_PKTS))
@@ -1439,29 +1436,29 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
payload, payload_len,
workflow->stats.ip_packet_count);
if(enable_joy_stats) {
if(enable_flow_stats) {
/* Update BD, distribution and mean. */
ndpi_flow_update_byte_count(flow, payload, payload_len, src_to_dst_direction);
ndpi_flow_update_byte_dist_mean_var(flow, payload, payload_len, src_to_dst_direction);
/* Update SPLT scores for first 32 packets. */
if((flow->entropy.src2dst_pkt_count+flow->entropy.dst2src_pkt_count) <= max_num_packets_per_flow) {
if((flow->entropy->src2dst_pkt_count+flow->entropy->dst2src_pkt_count) <= max_num_packets_per_flow) {
if(flow->bidirectional)
flow->entropy.score = ndpi_classify(flow->entropy.src2dst_pkt_len, flow->entropy.src2dst_pkt_time,
flow->entropy.dst2src_pkt_len, flow->entropy.dst2src_pkt_time,
flow->entropy.src2dst_start, flow->entropy.dst2src_start,
flow->entropy->score = ndpi_classify(flow->entropy->src2dst_pkt_len, flow->entropy->src2dst_pkt_time,
flow->entropy->dst2src_pkt_len, flow->entropy->dst2src_pkt_time,
flow->entropy->src2dst_start, flow->entropy->dst2src_start,
max_num_packets_per_flow, flow->src_port, flow->dst_port,
flow->src2dst_packets, flow->dst2src_packets,
flow->entropy.src2dst_opackets, flow->entropy.dst2src_opackets,
flow->entropy.src2dst_l4_bytes, flow->entropy.dst2src_l4_bytes, 1,
flow->entropy.src2dst_byte_count, flow->entropy.dst2src_byte_count);
flow->entropy->src2dst_opackets, flow->entropy->dst2src_opackets,
flow->entropy->src2dst_l4_bytes, flow->entropy->dst2src_l4_bytes, 1,
flow->entropy->src2dst_byte_count, flow->entropy->dst2src_byte_count);
else
flow->entropy.score = ndpi_classify(flow->entropy.src2dst_pkt_len, flow->entropy.src2dst_pkt_time,
NULL, NULL, flow->entropy.src2dst_start, flow->entropy.src2dst_start,
flow->entropy->score = ndpi_classify(flow->entropy->src2dst_pkt_len, flow->entropy->src2dst_pkt_time,
NULL, NULL, flow->entropy->src2dst_start, flow->entropy->src2dst_start,
max_num_packets_per_flow, flow->src_port, flow->dst_port,
flow->src2dst_packets, 0,
flow->entropy.src2dst_opackets, 0,
flow->entropy.src2dst_l4_bytes, 0, 1,
flow->entropy.src2dst_byte_count, NULL);
flow->entropy->src2dst_opackets, 0,
flow->entropy->src2dst_l4_bytes, 0, 1,
flow->entropy->src2dst_byte_count, NULL);
}
}
@@ -1472,7 +1469,13 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
/* Copy packets entropy if num packets count == 10 */
ndpi_clear_entropy_stats(flow);
/* Reset IAT reeference times (see https://github.com/ntop/nDPI/pull/1316) */
if(((flow->src2dst_packets + flow->dst2src_packets) % max_num_packets_per_flow) == 0) {
memset(&flow->src2dst_last_pkt_time, '\0', sizeof(flow->src2dst_last_pkt_time));
memset(&flow->dst2src_last_pkt_time, '\0', sizeof(flow->dst2src_last_pkt_time));
memset(&flow->flow_last_pkt_time, '\0', sizeof(flow->flow_last_pkt_time));
}
if((human_readeable_string_len != 0) && (!flow->has_human_readeable_strings)) {
u_int8_t skip = 0;
@@ -1527,8 +1530,8 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
flow->detected_protocol = ndpi_detection_process_packet(workflow->ndpi_struct, ndpi_flow,
iph ? (uint8_t *)iph : (uint8_t *)iph6,
ipsize, time_ms, src, dst);
ipsize, time_ms);
if(enough_packets || (flow->detected_protocol.app_protocol != NDPI_PROTOCOL_UNKNOWN)) {
if((!enough_packets)
&& ndpi_extra_dissection_possible(workflow->ndpi_struct, ndpi_flow))
@@ -1566,9 +1569,9 @@ static struct ndpi_proto packet_processing(struct ndpi_workflow * workflow,
}
}
#endif
*flow_risk = flow->risk;
return(flow->detected_protocol);
}
@@ -1581,8 +1584,12 @@ int ndpi_is_datalink_supported(int datalink_type) {
case DLT_PPP_SERIAL:
case DLT_C_HDLC:
case DLT_PPP:
#ifdef DLT_IPV4
case DLT_IPV4:
#endif
#ifdef DLT_IPV6
case DLT_IPV6:
#endif
case DLT_EN10MB:
case DLT_LINUX_SLL:
case DLT_IEEE802_11_RADIO:
@@ -1631,7 +1638,7 @@ struct ndpi_proto ndpi_workflow_process_packet(struct ndpi_workflow * workflow,
ndpi_packet_tunnel tunnel_type = ndpi_no_tunnel;
/* lengths and offsets */
u_int16_t eth_offset = 0;
u_int32_t eth_offset = 0;
u_int16_t radio_len;
u_int16_t fc;
u_int16_t type = 0;
@@ -1648,7 +1655,7 @@ struct ndpi_proto ndpi_workflow_process_packet(struct ndpi_workflow * workflow,
u_int8_t vlan_packet = 0;
*flow_risk = 0 /* NDPI_NO_RISK */;
/* Increment raw packet counter */
workflow->stats.raw_packet_count++;
@@ -1691,7 +1698,7 @@ struct ndpi_proto ndpi_workflow_process_packet(struct ndpi_workflow * workflow,
/* Cisco PPP in HDLC-like framing - 50 */
case DLT_PPP_SERIAL:
chdlc = (struct ndpi_chdlc *) &packet[eth_offset];
ip_offset = sizeof(struct ndpi_chdlc); /* CHDLC_OFF = 4 */
ip_offset = eth_offset + sizeof(struct ndpi_chdlc); /* CHDLC_OFF = 4 */
type = ntohs(chdlc->proto_code);
break;
@@ -1700,23 +1707,27 @@ struct ndpi_proto ndpi_workflow_process_packet(struct ndpi_workflow * workflow,
case DLT_PPP:
if(packet[0] == 0x0f || packet[0] == 0x8f) {
chdlc = (struct ndpi_chdlc *) &packet[eth_offset];
ip_offset = sizeof(struct ndpi_chdlc); /* CHDLC_OFF = 4 */
ip_offset = eth_offset + sizeof(struct ndpi_chdlc); /* CHDLC_OFF = 4 */
type = ntohs(chdlc->proto_code);
} else {
ip_offset = 2;
ip_offset = eth_offset + 2;
type = ntohs(*((u_int16_t*)&packet[eth_offset]));
}
break;
#ifdef DLT_IPV4
case DLT_IPV4:
type = ETH_P_IP;
ip_offset = 0;
break;
#endif
#ifdef DLT_IPV6
case DLT_IPV6:
type = ETH_P_IPV6;
ip_offset = 0;
break;
#endif
/* IEEE 802.3 Ethernet - 1 */
case DLT_EN10MB:
@@ -1804,14 +1815,16 @@ struct ndpi_proto ndpi_workflow_process_packet(struct ndpi_workflow * workflow,
/* check ether type */
switch(type) {
case VLAN:
case ETH_P_VLAN:
if(ip_offset+4 >= (int)header->caplen)
return(nproto);
vlan_id = ((packet[ip_offset] << 8) + packet[ip_offset+1]) & 0xFFF;
type = (packet[ip_offset+2] << 8) + packet[ip_offset+3];
ip_offset += 4;
vlan_packet = 1;
// double tagging for 802.1Q
while((type == 0x8100) && (((bpf_u_int32)ip_offset) < header->caplen)) {
while((type == 0x8100) && (((bpf_u_int32)ip_offset+4) < header->caplen)) {
vlan_id = ((packet[ip_offset] << 8) + packet[ip_offset+1]) & 0xFFF;
type = (packet[ip_offset+2] << 8) + packet[ip_offset+3];
ip_offset += 4;
@@ -1819,8 +1832,8 @@ struct ndpi_proto ndpi_workflow_process_packet(struct ndpi_workflow * workflow,
recheck_type = 1;
break;
case MPLS_UNI:
case MPLS_MULTI:
case ETH_P_MPLS_UNI:
case ETH_P_MPLS_MULTI:
mpls.u32 = *((uint32_t *) &packet[ip_offset]);
mpls.u32 = ntohl(mpls.u32);
workflow->stats.mpls_count++;
@@ -1834,7 +1847,7 @@ struct ndpi_proto ndpi_workflow_process_packet(struct ndpi_workflow * workflow,
recheck_type = 1;
break;
case PPPoE:
case ETH_P_PPPoE:
workflow->stats.pppoe_count++;
type = ETH_P_IP;
ip_offset += 8;
@@ -1878,7 +1891,11 @@ struct ndpi_proto ndpi_workflow_process_packet(struct ndpi_workflow * workflow,
ip_len = ((u_int16_t)iph->ihl * 4);
iph6 = NULL;
if(iph->protocol == IPPROTO_IPV6 || iph->protocol == IPPROTO_IPIP) {
if(iph->protocol == IPPROTO_IPV6
#ifdef IPPROTO_IPIP
|| iph->protocol == IPPROTO_IPIP
#endif
) {
ip_offset += ip_len;
if(ip_len > 0)
goto iph_check;
@@ -1900,7 +1917,7 @@ struct ndpi_proto ndpi_workflow_process_packet(struct ndpi_workflow * workflow,
} else if(iph->version == 6) {
if(header->caplen < ip_offset + sizeof(struct ndpi_ipv6hdr))
return(nproto); /* Too short for IPv6 header*/
iph6 = (struct ndpi_ipv6hdr *)&packet[ip_offset];
proto = iph6->ip6_hdr.ip6_un1_nxt;
ip_len = ntohs(iph6->ip6_hdr.ip6_un1_plen);
@@ -1909,12 +1926,17 @@ struct ndpi_proto ndpi_workflow_process_packet(struct ndpi_workflow * workflow,
return(nproto); /* Too short for IPv6 payload*/
const u_int8_t *l4ptr = (((const u_int8_t *) iph6) + sizeof(struct ndpi_ipv6hdr));
u_int16_t ipsize = header->caplen - ip_offset;
if(ndpi_handle_ipv6_extension_headers(NULL, &l4ptr, &ip_len, &proto) != 0) {
if(ndpi_handle_ipv6_extension_headers(ipsize - sizeof(struct ndpi_ipv6hdr), &l4ptr, &ip_len, &proto) != 0) {
return(nproto);
}
if(proto == IPPROTO_IPV6 || proto == IPPROTO_IPIP) {
if(proto == IPPROTO_IPV6
#ifdef IPPROTO_IPIP
|| proto == IPPROTO_IPIP
#endif
) {
if(l4ptr > packet) { /* Better safe than sorry */
ip_offset = (l4ptr - packet);
goto iph_check;
@@ -1932,7 +1954,7 @@ struct ndpi_proto ndpi_workflow_process_packet(struct ndpi_workflow * workflow,
"\n\nWARNING: only IPv4/IPv6 packets are supported in this demo (nDPI supports both IPv4 and IPv6), all other packets will be discarded\n\n");
ipv4_warning_used = 1;
}
workflow->stats.total_discarded_bytes += header->len;
return(nproto);
}
@@ -1982,9 +2004,14 @@ struct ndpi_proto ndpi_workflow_process_packet(struct ndpi_workflow * workflow,
if(iph->version == 6) {
iph6 = (struct ndpi_ipv6hdr *)&packet[ip_offset];
iph = NULL;
if(header->caplen < ip_offset + sizeof(struct ndpi_ipv6hdr))
return(nproto);
} else if(iph->version != IPVERSION) {
// printf("WARNING: not good (packet_id=%u)!\n", (unsigned int)workflow->stats.raw_packet_count);
goto v4_warning;
} else {
if(header->caplen < ip_offset + sizeof(struct ndpi_iphdr))
return(nproto);
}
}
}
@@ -2170,6 +2197,7 @@ int dpdk_port_init(int port, struct rte_mempool *mbuf_pool) {
int dpdk_port_deinit(int port) {
rte_eth_dev_stop(port);
rte_eth_dev_close(port);
return 0;
}
#endif
+17 -14
View File
@@ -1,7 +1,7 @@
/*
* ndpi_util.h
*
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* nDPI is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published by
@@ -134,7 +134,6 @@ struct flow_metrics {
struct ndpi_entropy {
// Entropy fields
pkt_timeval src2dst_last_pkt_time, dst2src_last_pkt_time, flow_last_pkt_time;
u_int16_t src2dst_pkt_len[MAX_NUM_PKTS]; /*!< array of packet appdata lengths */
pkt_timeval src2dst_pkt_time[MAX_NUM_PKTS]; /*!< array of arrival times */
u_int16_t dst2src_pkt_len[MAX_NUM_PKTS]; /*!< array of packet appdata lengths */
@@ -162,10 +161,10 @@ struct ndpi_entropy {
typedef struct ndpi_flow_info {
u_int32_t flow_id;
u_int32_t hashval;
u_int32_t src_ip;
u_int32_t dst_ip;
u_int16_t src_port;
u_int16_t dst_port;
u_int32_t src_ip; /* network order */
u_int32_t dst_ip; /* network order */
u_int16_t src_port; /* network order */
u_int16_t dst_port; /* network order */
u_int8_t detection_completed, protocol, bidirectional, check_extra_packets;
u_int16_t vlan_id;
ndpi_packet_tunnel tunnel_type;
@@ -190,21 +189,24 @@ typedef struct ndpi_flow_info {
// result only, not used for flow identification
ndpi_protocol detected_protocol;
ndpi_confidence_t confidence;
// Flow data analysis
pkt_timeval src2dst_last_pkt_time, dst2src_last_pkt_time, flow_last_pkt_time;
struct ndpi_analyze_struct *iat_c_to_s, *iat_s_to_c, *iat_flow,
*pktlen_c_to_s, *pktlen_s_to_c;
char info[255];
char flow_extra_info[16];
char host_server_name[240];
char bittorent_hash[41];
char dhcp_fingerprint[48];
char host_server_name[80]; /* Hostname/SNI */
char *bittorent_hash;
char *dhcp_fingerprint;
char *dhcp_class_ident;
ndpi_risk risk;
struct {
u_int16_t ssl_version;
char client_requested_server_name[256], server_info[64],
char server_info[64],
client_hassh[33], server_hassh[33], *server_names,
*tls_alpn, *tls_supported_versions,
*tls_issuerDN, *tls_subjectDN,
@@ -224,18 +226,18 @@ typedef struct ndpi_flow_info {
} ssh_tls;
struct {
char url[256], request_content_type[64], content_type[64], user_agent[128];
char url[256], request_content_type[64], content_type[64], user_agent[256];
u_int response_status_code;
} http;
struct {
char username[32], password[32];
char *username, *password;
} telnet;
void *src_id, *dst_id;
struct ndpi_entropy entropy;
struct ndpi_entropy last_entropy;
struct ndpi_entropy *entropy;
struct ndpi_entropy *last_entropy;
/* Payload lenght bins */
#ifdef DIRECTION_BINS
@@ -262,6 +264,7 @@ typedef struct ndpi_stats {
u_int64_t packet_len[6];
u_int16_t max_packet_len;
u_int64_t dpi_packet_count[3];
u_int64_t flow_confidence[NDPI_CONFIDENCE_MAX];
} ndpi_stats_t;
+11 -2
View File
@@ -1,4 +1,4 @@
bin_PROGRAMS = fuzz_process_packet fuzz_ndpi_reader fuzz_ndpi_reader_with_main
bin_PROGRAMS = fuzz_process_packet fuzz_process_packet_with_main fuzz_ndpi_reader fuzz_ndpi_reader_with_main
fuzz_process_packet_SOURCES = fuzz_process_packet.c
fuzz_process_packet_CFLAGS =
@@ -13,6 +13,15 @@ fuzz_process_packet_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_process_packet_LDFLAGS) $(LDFLAGS) -o $@
fuzz_process_packet_with_main_SOURCES = fuzz_process_packet.c
fuzz_process_packet_with_main_CFLAGS = -DBUILD_MAIN
fuzz_process_packet_with_main_LDADD = ../src/lib/libndpi.a
fuzz_process_packet_with_main_LDFLAGS = $(ADDITIONAL_LIBS) $(LIBS)
# force usage of CXX for linker
fuzz_process_packet_with_main_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \
$(LIBTOOLFLAGS) --mode=link $(CXX) $(AM_CXXFLAGS) $(CXXFLAGS) \
$(fuzz_process_packet_with_main_LDFLAGS) $(LDFLAGS) -o $@
fuzz_ndpi_reader_SOURCES = fuzz_ndpi_reader.c
fuzz_ndpi_reader_CFLAGS = -I../example/
fuzz_ndpi_reader_LDADD = ../example/libndpiReader.a ../src/lib/libndpi.a
@@ -37,7 +46,7 @@ fuzz_ndpi_reader_with_main_LINK=$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS
# required for Google oss-fuzz
# see https://github.com/google/oss-fuzz/tree/master/projects/ndpi
testpcaps := $(wildcard ../tests/pcap/*.pcap)
testpcaps := $(wildcard ../tests/pcap/*.pcap*)
fuzz_ndpi_reader_seed_corpus.zip: $(testpcaps)
zip -r fuzz_ndpi_reader_seed_corpus.zip $(testpcaps)
+2 -2
View File
@@ -13,7 +13,7 @@ int nDPI_LogLevel = 0;
char *_debug_protocols = NULL;
u_int32_t current_ndpi_memory = 0, max_ndpi_memory = 0;
u_int8_t enable_protocol_guess = 1, enable_payload_analyzer = 0;
u_int8_t enable_joy_stats = 0;
u_int8_t enable_flow_stats = 0;
u_int8_t human_readeable_string_len = 5;
u_int8_t max_num_udp_dissected_pkts = 16 /* 8 is enough for most protocols, Signal requires more */, max_num_tcp_dissected_pkts = 80 /* due to telnet */;
@@ -159,7 +159,7 @@ int main(int argc, char ** argv)
return 1;
}
if (fread(pcap_buffer, sizeof(*pcap_buffer), pcap_file_size, pcap_file) != pcap_file_size) {
if (fread(pcap_buffer, sizeof(*pcap_buffer), pcap_file_size, pcap_file) != (size_t)pcap_file_size) {
perror("fread failed");
fclose(pcap_file);
free(pcap_buffer);
+71 -9
View File
@@ -4,25 +4,87 @@
#include <stdio.h>
struct ndpi_detection_module_struct *ndpi_info_mod = NULL;
struct ndpi_id_struct *src;
struct ndpi_id_struct *dst;
int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
uint8_t protocol_was_guessed;
if (ndpi_info_mod == NULL) {
ndpi_info_mod = ndpi_init_detection_module(ndpi_no_prefs);
NDPI_PROTOCOL_BITMASK all;
ndpi_info_mod = ndpi_init_detection_module(ndpi_enable_ja3_plus);
NDPI_PROTOCOL_BITMASK all, debug_bitmask;
NDPI_BITMASK_SET_ALL(all);
NDPI_BITMASK_SET_ALL(debug_bitmask);
ndpi_set_protocol_detection_bitmask2(ndpi_info_mod, &all);
src = ndpi_malloc(SIZEOF_ID_STRUCT);
dst = ndpi_malloc(SIZEOF_ID_STRUCT);
ndpi_set_log_level(ndpi_info_mod, 4);
ndpi_set_debug_bitmask(ndpi_info_mod, debug_bitmask);
ndpi_finalize_initialization(ndpi_info_mod);
}
struct ndpi_flow_struct *ndpi_flow = ndpi_flow_malloc(SIZEOF_FLOW_STRUCT);
memset(ndpi_flow, 0, SIZEOF_FLOW_STRUCT);
memset(src, 0, SIZEOF_ID_STRUCT);
memset(dst, 0, SIZEOF_ID_STRUCT);
ndpi_detection_process_packet(ndpi_info_mod, ndpi_flow, Data, Size, 0, src, dst);
ndpi_detection_process_packet(ndpi_info_mod, ndpi_flow, Data, Size, 0);
ndpi_detection_giveup(ndpi_info_mod, ndpi_flow, 1, &protocol_was_guessed);
ndpi_free_flow(ndpi_flow);
return 0;
}
#ifdef BUILD_MAIN
int main(int argc, char ** argv)
{
FILE * data_file;
long data_file_size;
uint8_t * data_buffer;
int test_retval;
if (argc != 2) {
fprintf(stderr, "usage: %s: [data-file]\n",
(argc > 0 ? argv[0] : "fuzz_process_packet_with_main"));
return 1;
}
data_file = fopen(argv[1], "r");
if (data_file == NULL) {
perror("fopen failed");
return 1;
}
if (fseek(data_file, 0, SEEK_END) != 0) {
perror("fseek(SEEK_END) failed");
fclose(data_file);
return 1;
}
data_file_size = ftell(data_file);
if (data_file_size < 0) {
perror("ftell failed");
fclose(data_file);
return 1;
}
if (fseek(data_file, 0, SEEK_SET) != 0) {
perror("fseek(0, SEEK_SET) failed");
fclose(data_file);
return 1;
}
data_buffer = malloc(data_file_size);
if (data_buffer == NULL) {
perror("malloc failed");
fclose(data_file);
return 1;
}
if (fread(data_buffer, sizeof(*data_buffer), data_file_size, data_file) != (size_t)data_file_size) {
perror("fread failed");
fclose(data_file);
free(data_buffer);
return 1;
}
test_retval = LLVMFuzzerTestOneInput(data_buffer, data_file_size);
fclose(data_file);
free(data_buffer);
return test_retval;
}
#endif
+2
View File
@@ -0,0 +1,2 @@
queries:
- exclude: cpp/cleartext-storage-file
+153 -115
View File
@@ -1,5 +1,5 @@
# ===========================================================================
# http://www.gnu.org/software/autoconf-archive/ax_pthread.html
# https://www.gnu.org/software/autoconf-archive/ax_pthread.html
# ===========================================================================
#
# SYNOPSIS
@@ -14,20 +14,24 @@
# flags that are needed. (The user can also force certain compiler
# flags/libs to be tested by setting these environment variables.)
#
# Also sets PTHREAD_CC to any special C compiler that is needed for
# multi-threaded programs (defaults to the value of CC otherwise). (This
# is necessary on AIX to use the special cc_r compiler alias.)
# Also sets PTHREAD_CC and PTHREAD_CXX to any special C compiler that is
# needed for multi-threaded programs (defaults to the value of CC
# respectively CXX otherwise). (This is necessary on e.g. AIX to use the
# special cc_r/CC_r compiler alias.)
#
# NOTE: You are assumed to not only compile your program with these flags,
# but also to link with them as well. For example, you might link with
# $PTHREAD_CC $CFLAGS $PTHREAD_CFLAGS $LDFLAGS ... $PTHREAD_LIBS $LIBS
# $PTHREAD_CXX $CXXFLAGS $PTHREAD_CFLAGS $LDFLAGS ... $PTHREAD_LIBS $LIBS
#
# If you are only building threaded programs, you may wish to use these
# variables in your default LIBS, CFLAGS, and CC:
#
# LIBS="$PTHREAD_LIBS $LIBS"
# CFLAGS="$CFLAGS $PTHREAD_CFLAGS"
# CXXFLAGS="$CXXFLAGS $PTHREAD_CFLAGS"
# CC="$PTHREAD_CC"
# CXX="$PTHREAD_CXX"
#
# In addition, if the PTHREAD_CREATE_JOINABLE thread-attribute constant
# has a nonstandard name, this macro defines PTHREAD_CREATE_JOINABLE to
@@ -55,6 +59,7 @@
#
# Copyright (c) 2008 Steven G. Johnson <stevenj@alum.mit.edu>
# Copyright (c) 2011 Daniel Richard G. <skunk@iSKUNK.ORG>
# Copyright (c) 2019 Marc Stevens <marc.stevens@cwi.nl>
#
# This program is free software: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
@@ -67,7 +72,7 @@
# Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program. If not, see <http://www.gnu.org/licenses/>.
# with this program. If not, see <https://www.gnu.org/licenses/>.
#
# As a special exception, the respective Autoconf Macro's copyright owner
# gives unlimited permission to copy, distribute and modify the configure
@@ -82,7 +87,7 @@
# modified version of the Autoconf Macro, you may extend this special
# exception to the GPL to apply to your modified version as well.
#serial 23
#serial 31
AU_ALIAS([ACX_PTHREAD], [AX_PTHREAD])
AC_DEFUN([AX_PTHREAD], [
@@ -104,6 +109,7 @@ if test "x$PTHREAD_CFLAGS$PTHREAD_LIBS" != "x"; then
ax_pthread_save_CFLAGS="$CFLAGS"
ax_pthread_save_LIBS="$LIBS"
AS_IF([test "x$PTHREAD_CC" != "x"], [CC="$PTHREAD_CC"])
AS_IF([test "x$PTHREAD_CXX" != "x"], [CXX="$PTHREAD_CXX"])
CFLAGS="$CFLAGS $PTHREAD_CFLAGS"
LIBS="$PTHREAD_LIBS $LIBS"
AC_MSG_CHECKING([for pthread_join using $CC $PTHREAD_CFLAGS $PTHREAD_LIBS])
@@ -123,10 +129,12 @@ fi
# (e.g. DEC) have both -lpthread and -lpthreads, where one of the
# libraries is broken (non-POSIX).
# Create a list of thread flags to try. Items starting with a "-" are
# C compiler flags, and other items are library names, except for "none"
# which indicates that we try without any flags at all, and "pthread-config"
# which is a program returning the flags for the Pth emulation library.
# Create a list of thread flags to try. Items with a "," contain both
# C compiler flags (before ",") and linker flags (after ","). Other items
# starting with a "-" are C compiler flags, and remaining items are
# library names, except for "none" which indicates that we try without
# any flags at all, and "pthread-config" which is a program returning
# the flags for the Pth emulation library.
ax_pthread_flags="pthreads none -Kthread -pthread -pthreads -mthreads pthread --thread-safe -mt pthread-config"
@@ -194,14 +202,47 @@ case $host_os in
# that too in a future libc.) So we'll check first for the
# standard Solaris way of linking pthreads (-mt -lpthread).
ax_pthread_flags="-mt,pthread pthread $ax_pthread_flags"
ax_pthread_flags="-mt,-lpthread pthread $ax_pthread_flags"
;;
esac
# Are we compiling with Clang?
AC_CACHE_CHECK([whether $CC is Clang],
[ax_cv_PTHREAD_CLANG],
[ax_cv_PTHREAD_CLANG=no
# Note that Autoconf sets GCC=yes for Clang as well as GCC
if test "x$GCC" = "xyes"; then
AC_EGREP_CPP([AX_PTHREAD_CC_IS_CLANG],
[/* Note: Clang 2.7 lacks __clang_[a-z]+__ */
# if defined(__clang__) && defined(__llvm__)
AX_PTHREAD_CC_IS_CLANG
# endif
],
[ax_cv_PTHREAD_CLANG=yes])
fi
])
ax_pthread_clang="$ax_cv_PTHREAD_CLANG"
# GCC generally uses -pthread, or -pthreads on some platforms (e.g. SPARC)
# Note that for GCC and Clang -pthread generally implies -lpthread,
# except when -nostdlib is passed.
# This is problematic using libtool to build C++ shared libraries with pthread:
# [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=25460
# [2] https://bugzilla.redhat.com/show_bug.cgi?id=661333
# [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=468555
# To solve this, first try -pthread together with -lpthread for GCC
AS_IF([test "x$GCC" = "xyes"],
[ax_pthread_flags="-pthread -pthreads $ax_pthread_flags"])
[ax_pthread_flags="-pthread,-lpthread -pthread -pthreads $ax_pthread_flags"])
# Clang takes -pthread (never supported any other flag), but we'll try with -lpthread first
AS_IF([test "x$ax_pthread_clang" = "xyes"],
[ax_pthread_flags="-pthread,-lpthread -pthread"])
# The presence of a feature test macro requesting re-entrant function
# definitions is, on some systems, a strong hint that pthreads support is
@@ -224,101 +265,6 @@ AS_IF([test "x$ax_pthread_check_macro" = "x--"],
[ax_pthread_check_cond=0],
[ax_pthread_check_cond="!defined($ax_pthread_check_macro)"])
# Are we compiling with Clang?
AC_CACHE_CHECK([whether $CC is Clang],
[ax_cv_PTHREAD_CLANG],
[ax_cv_PTHREAD_CLANG=no
# Note that Autoconf sets GCC=yes for Clang as well as GCC
if test "x$GCC" = "xyes"; then
AC_EGREP_CPP([AX_PTHREAD_CC_IS_CLANG],
[/* Note: Clang 2.7 lacks __clang_[a-z]+__ */
# if defined(__clang__) && defined(__llvm__)
AX_PTHREAD_CC_IS_CLANG
# endif
],
[ax_cv_PTHREAD_CLANG=yes])
fi
])
ax_pthread_clang="$ax_cv_PTHREAD_CLANG"
ax_pthread_clang_warning=no
# Clang needs special handling, because older versions handle the -pthread
# option in a rather... idiosyncratic way
if test "x$ax_pthread_clang" = "xyes"; then
# Clang takes -pthread; it has never supported any other flag
# (Note 1: This will need to be revisited if a system that Clang
# supports has POSIX threads in a separate library. This tends not
# to be the way of modern systems, but it's conceivable.)
# (Note 2: On some systems, notably Darwin, -pthread is not needed
# to get POSIX threads support; the API is always present and
# active. We could reasonably leave PTHREAD_CFLAGS empty. But
# -pthread does define _REENTRANT, and while the Darwin headers
# ignore this macro, third-party headers might not.)
PTHREAD_CFLAGS="-pthread"
PTHREAD_LIBS=
ax_pthread_ok=yes
# However, older versions of Clang make a point of warning the user
# that, in an invocation where only linking and no compilation is
# taking place, the -pthread option has no effect ("argument unused
# during compilation"). They expect -pthread to be passed in only
# when source code is being compiled.
#
# Problem is, this is at odds with the way Automake and most other
# C build frameworks function, which is that the same flags used in
# compilation (CFLAGS) are also used in linking. Many systems
# supported by AX_PTHREAD require exactly this for POSIX threads
# support, and in fact it is often not straightforward to specify a
# flag that is used only in the compilation phase and not in
# linking. Such a scenario is extremely rare in practice.
#
# Even though use of the -pthread flag in linking would only print
# a warning, this can be a nuisance for well-run software projects
# that build with -Werror. So if the active version of Clang has
# this misfeature, we search for an option to squash it.
AC_CACHE_CHECK([whether Clang needs flag to prevent "argument unused" warning when linking with -pthread],
[ax_cv_PTHREAD_CLANG_NO_WARN_FLAG],
[ax_cv_PTHREAD_CLANG_NO_WARN_FLAG=unknown
# Create an alternate version of $ac_link that compiles and
# links in two steps (.c -> .o, .o -> exe) instead of one
# (.c -> exe), because the warning occurs only in the second
# step
ax_pthread_save_ac_link="$ac_link"
ax_pthread_sed='s/conftest\.\$ac_ext/conftest.$ac_objext/g'
ax_pthread_link_step=`$as_echo "$ac_link" | sed "$ax_pthread_sed"`
ax_pthread_2step_ac_link="($ac_compile) && (echo ==== >&5) && ($ax_pthread_link_step)"
ax_pthread_save_CFLAGS="$CFLAGS"
for ax_pthread_try in '' -Qunused-arguments -Wno-unused-command-line-argument unknown; do
AS_IF([test "x$ax_pthread_try" = "xunknown"], [break])
CFLAGS="-Werror -Wunknown-warning-option $ax_pthread_try -pthread $ax_pthread_save_CFLAGS"
ac_link="$ax_pthread_save_ac_link"
AC_LINK_IFELSE([AC_LANG_SOURCE([[int main(void){return 0;}]])],
[ac_link="$ax_pthread_2step_ac_link"
AC_LINK_IFELSE([AC_LANG_SOURCE([[int main(void){return 0;}]])],
[break])
])
done
ac_link="$ax_pthread_save_ac_link"
CFLAGS="$ax_pthread_save_CFLAGS"
AS_IF([test "x$ax_pthread_try" = "x"], [ax_pthread_try=no])
ax_cv_PTHREAD_CLANG_NO_WARN_FLAG="$ax_pthread_try"
])
case "$ax_cv_PTHREAD_CLANG_NO_WARN_FLAG" in
no | unknown) ;;
*) PTHREAD_CFLAGS="$ax_cv_PTHREAD_CLANG_NO_WARN_FLAG $PTHREAD_CFLAGS" ;;
esac
fi # $ax_pthread_clang = yes
if test "x$ax_pthread_ok" = "xno"; then
for ax_pthread_try_flag in $ax_pthread_flags; do
@@ -328,10 +274,10 @@ for ax_pthread_try_flag in $ax_pthread_flags; do
AC_MSG_CHECKING([whether pthreads work without any flags])
;;
-mt,pthread)
AC_MSG_CHECKING([whether pthreads work with -mt -lpthread])
PTHREAD_CFLAGS="-mt"
PTHREAD_LIBS="-lpthread"
*,*)
PTHREAD_CFLAGS=`echo $ax_pthread_try_flag | sed "s/^\(.*\),\(.*\)$/\1/"`
PTHREAD_LIBS=`echo $ax_pthread_try_flag | sed "s/^\(.*\),\(.*\)$/\2/"`
AC_MSG_CHECKING([whether pthreads work with "$PTHREAD_CFLAGS" and "$PTHREAD_LIBS"])
;;
-*)
@@ -371,7 +317,13 @@ for ax_pthread_try_flag in $ax_pthread_flags; do
# if $ax_pthread_check_cond
# error "$ax_pthread_check_macro must be defined"
# endif
static void routine(void *a) { a = 0; }
static void *some_global = NULL;
static void routine(void *a)
{
/* To avoid any unused-parameter or
unused-but-set-parameter warning. */
some_global = a;
}
static void *start_routine(void *a) { return a; }],
[pthread_t th; pthread_attr_t attr;
pthread_create(&th, 0, start_routine, 0);
@@ -393,6 +345,80 @@ for ax_pthread_try_flag in $ax_pthread_flags; do
done
fi
# Clang needs special handling, because older versions handle the -pthread
# option in a rather... idiosyncratic way
if test "x$ax_pthread_clang" = "xyes"; then
# Clang takes -pthread; it has never supported any other flag
# (Note 1: This will need to be revisited if a system that Clang
# supports has POSIX threads in a separate library. This tends not
# to be the way of modern systems, but it's conceivable.)
# (Note 2: On some systems, notably Darwin, -pthread is not needed
# to get POSIX threads support; the API is always present and
# active. We could reasonably leave PTHREAD_CFLAGS empty. But
# -pthread does define _REENTRANT, and while the Darwin headers
# ignore this macro, third-party headers might not.)
# However, older versions of Clang make a point of warning the user
# that, in an invocation where only linking and no compilation is
# taking place, the -pthread option has no effect ("argument unused
# during compilation"). They expect -pthread to be passed in only
# when source code is being compiled.
#
# Problem is, this is at odds with the way Automake and most other
# C build frameworks function, which is that the same flags used in
# compilation (CFLAGS) are also used in linking. Many systems
# supported by AX_PTHREAD require exactly this for POSIX threads
# support, and in fact it is often not straightforward to specify a
# flag that is used only in the compilation phase and not in
# linking. Such a scenario is extremely rare in practice.
#
# Even though use of the -pthread flag in linking would only print
# a warning, this can be a nuisance for well-run software projects
# that build with -Werror. So if the active version of Clang has
# this misfeature, we search for an option to squash it.
AC_CACHE_CHECK([whether Clang needs flag to prevent "argument unused" warning when linking with -pthread],
[ax_cv_PTHREAD_CLANG_NO_WARN_FLAG],
[ax_cv_PTHREAD_CLANG_NO_WARN_FLAG=unknown
# Create an alternate version of $ac_link that compiles and
# links in two steps (.c -> .o, .o -> exe) instead of one
# (.c -> exe), because the warning occurs only in the second
# step
ax_pthread_save_ac_link="$ac_link"
ax_pthread_sed='s/conftest\.\$ac_ext/conftest.$ac_objext/g'
ax_pthread_link_step=`AS_ECHO(["$ac_link"]) | sed "$ax_pthread_sed"`
ax_pthread_2step_ac_link="($ac_compile) && (echo ==== >&5) && ($ax_pthread_link_step)"
ax_pthread_save_CFLAGS="$CFLAGS"
for ax_pthread_try in '' -Qunused-arguments -Wno-unused-command-line-argument unknown; do
AS_IF([test "x$ax_pthread_try" = "xunknown"], [break])
CFLAGS="-Werror -Wunknown-warning-option $ax_pthread_try -pthread $ax_pthread_save_CFLAGS"
ac_link="$ax_pthread_save_ac_link"
AC_LINK_IFELSE([AC_LANG_SOURCE([[int main(void){return 0;}]])],
[ac_link="$ax_pthread_2step_ac_link"
AC_LINK_IFELSE([AC_LANG_SOURCE([[int main(void){return 0;}]])],
[break])
])
done
ac_link="$ax_pthread_save_ac_link"
CFLAGS="$ax_pthread_save_CFLAGS"
AS_IF([test "x$ax_pthread_try" = "x"], [ax_pthread_try=no])
ax_cv_PTHREAD_CLANG_NO_WARN_FLAG="$ax_pthread_try"
])
case "$ax_cv_PTHREAD_CLANG_NO_WARN_FLAG" in
no | unknown) ;;
*) PTHREAD_CFLAGS="$ax_cv_PTHREAD_CLANG_NO_WARN_FLAG $PTHREAD_CFLAGS" ;;
esac
fi # $ax_pthread_clang = yes
# Various other checks:
if test "x$ax_pthread_ok" = "xyes"; then
ax_pthread_save_CFLAGS="$CFLAGS"
@@ -438,7 +464,8 @@ if test "x$ax_pthread_ok" = "xyes"; then
AC_CACHE_CHECK([for PTHREAD_PRIO_INHERIT],
[ax_cv_PTHREAD_PRIO_INHERIT],
[AC_LINK_IFELSE([AC_LANG_PROGRAM([[#include <pthread.h>]],
[[int i = PTHREAD_PRIO_INHERIT;]])],
[[int i = PTHREAD_PRIO_INHERIT;
return i;]])],
[ax_cv_PTHREAD_PRIO_INHERIT=yes],
[ax_cv_PTHREAD_PRIO_INHERIT=no])
])
@@ -460,18 +487,28 @@ if test "x$ax_pthread_ok" = "xyes"; then
[#handle absolute path differently from PATH based program lookup
AS_CASE(["x$CC"],
[x/*],
[AS_IF([AS_EXECUTABLE_P([${CC}_r])],[PTHREAD_CC="${CC}_r"])],
[AC_CHECK_PROGS([PTHREAD_CC],[${CC}_r],[$CC])])])
[
AS_IF([AS_EXECUTABLE_P([${CC}_r])],[PTHREAD_CC="${CC}_r"])
AS_IF([test "x${CXX}" != "x"], [AS_IF([AS_EXECUTABLE_P([${CXX}_r])],[PTHREAD_CXX="${CXX}_r"])])
],
[
AC_CHECK_PROGS([PTHREAD_CC],[${CC}_r],[$CC])
AS_IF([test "x${CXX}" != "x"], [AC_CHECK_PROGS([PTHREAD_CXX],[${CXX}_r],[$CXX])])
]
)
])
;;
esac
fi
fi
test -n "$PTHREAD_CC" || PTHREAD_CC="$CC"
test -n "$PTHREAD_CXX" || PTHREAD_CXX="$CXX"
AC_SUBST([PTHREAD_LIBS])
AC_SUBST([PTHREAD_CFLAGS])
AC_SUBST([PTHREAD_CC])
AC_SUBST([PTHREAD_CXX])
# Finally, execute ACTION-IF-FOUND/ACTION-IF-NOT-FOUND:
if test "x$ax_pthread_ok" = "xyes"; then
@@ -483,3 +520,4 @@ else
fi
AC_LANG_POP
])dnl AX_PTHREAD
+1 -8
View File
@@ -8,7 +8,7 @@
include $(TOPDIR)/rules.mk
PKG_NAME:=libndpi
PKG_VERSION:=3.4
PKG_VERSION:=4.0
PKG_RELEASE:=1
PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.gz
@@ -58,13 +58,6 @@ endef
define Build/Prepare
$(PKG_UNPACK)
$(Build/Patch)
mv $(PKG_BUILD_DIR)/configure.seed $(PKG_BUILD_DIR)/configure.ac
$(SED) "s/@NDPI_MAJOR@/3/g" \
-e "s/@NDPI_MINOR@/4/g" \
-e "s/@NDPI_PATCH@/0/g" \
-e "s/@NDPI_VERSION_SHORT@/3.4.0/g" \
-e "s/@FUZZY@/dnl> /g" \
$(PKG_BUILD_DIR)/configure.ac
endef
define Build/InstallDev
-7
View File
@@ -58,13 +58,6 @@ define Build/Prepare
cp -r $(SOURCE_DIR)/* $(PKG_BUILD_DIR)
$(PKG_UNPACK)
$(Build/Patch)
mv $(PKG_BUILD_DIR)/configure.seed $(PKG_BUILD_DIR)/configure.ac
$(SED) "s/@NDPI_MAJOR@/3/g" \
-e "s/@NDPI_MINOR@/4/g" \
-e "s/@NDPI_PATCH@/0/g" \
-e "s/@NDPI_VERSION_SHORT@/3.4.0/g" \
-e "s/@FUZZY@/dnl> /g" \
$(PKG_BUILD_DIR)/configure.ac
endef
define Build/InstallDev
+6 -5
View File
@@ -5,14 +5,15 @@ ndpi:
\rm -rf ./debian/ndpi-tmp ./debian/ndpi-dev-tmp ./debian/ndpi ./debian/ndpi-dev
mkdir -p ./debian/ndpi-tmp ./debian/ndpi-dev-tmp
mkdir -p ./debian/ndpi-tmp/usr/lib ./debian/ndpi-tmp/usr/bin
mkdir -p ./debian/ndpi-dev-tmp/usr/lib ./debian/ndpi-dev-tmp/usr/include/ndpi
cd ${NDPI_HOME}; ./autogen.sh; ./configure; make
mkdir -p ./debian/ndpi-dev-tmp/usr/lib ./debian/ndpi-dev-tmp/usr/include/ndpi ./debian/ndpi-dev-tmp/usr/lib/pkgconfig
cd ${NDPI_HOME}; ./autogen.sh; ./configure; $(MAKE)
cp $(NDPI_HOME)/src/lib/libndpi.so.@NDPI_VERS@ ./debian/ndpi-tmp/usr/lib/
cd ./debian/ndpi-tmp/usr/lib/; ln -s libndpi.so.@NDPI_VERS@ libndpi.so; cd -
cd ./debian/ndpi-tmp/usr/lib/; ln -s libndpi.so.@NDPI_VERS@ libndpi.so.@MAJOR_RELEASE@; cd -
cp $(NDPI_HOME)/src/lib/libndpi.a ./debian/ndpi-dev-tmp/usr/lib/
cp $(NDPI_HOME)/example/ndpiReader ./debian/ndpi-tmp/usr/bin/
cp $(NDPI_HOME)/src/include/*.h ./debian/ndpi-dev-tmp/usr/include/ndpi/
cp $(NDPI_HOME)/libndpi.pc ./debian/ndpi-dev-tmp/usr/lib/pkgconfig/
-rm -fr ./debian/ndpi-dev-tmp/usr/include/ndpi/ndpi_win32.h*
@echo
@find ./debian/ndpi-tmp -name "*~" -exec /bin/rm {} ';'
@@ -32,11 +33,11 @@ ndpi:
@echo "-------------------------------"
distclean:
-cd ${NDPI_HOME} && make distclean
-cd ${NDPI_HOME} && $(MAKE) distclean
install:
cd ${NDPI_HOME}; make install DESTDIR="${NDPI_BUILD}/debian/tmp" AM_UPDATE_INFO_DIR=no
cd ${NDPI_HOME}; $(MAKE) install DESTDIR="${NDPI_BUILD}/debian/tmp" AM_UPDATE_INFO_DIR=no
clean:
-cd ${NDPI_HOME} && make clean
-cd ${NDPI_HOME} && $(MAKE) clean
-rm -rf *~ *deb ./usr ./debian/ndpi ./debian/ndpi-dev ./debian/ndpi ./debian/ndpi-dev
+3 -2
View File
@@ -2,7 +2,8 @@ CC=@CC@
CFLAGS=-I. -I../src/include -I./src/lib/third_party/include @CFLAGS@ @CUSTOM_NDPI@ -shared
#LIBNDPI=../src/lib/libndpi.so.@NDPI_VERSION_SHORT@
LIBNDPI=../src/lib/libndpi.a
LDFLAGS=$(CFILE) $(LIBNDPI) -lpcap @LIBS@ @ADDITIONAL_LIBS@
LDFLAGS=@LDFLAGS@
LIBS=$(CFILE) $(LIBNDPI) -lpcap @LIBS@ @ADDITIONAL_LIBS@
SHARE = -soname,ndpi_wrap
SO=ndpi_wrap.so
OBJS = ndpi_wrap.o
@@ -19,7 +20,7 @@ endif
all: $(SO)
$(SO): $(CFILE) $(LIBNDPI) Makefile
$(CC) $(CFLAGS) -Wl,$(SHARE) -o $@ $(PIC) $(LDFLAGS)
$(CC) $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $(PIC) -Wl,$(SHARE) -o $@ $(LIBS)
# ln -s $(LIBNDPI) .
clean:
+58 -160
View File
@@ -322,6 +322,17 @@ typedef enum {
NDPI_MALICIOUS_JA3,
NDPI_MALICIOUS_SHA1_CERTIFICATE,
NDPI_DESKTOP_OR_FILE_SHARING_SESSION,
NDPI_TLS_UNCOMMON_ALPN,
NDPI_TLS_CERT_VALIDITY_TOO_LONG,
NDPI_TLS_SUSPICIOUS_EXTENSION,
NDPI_TLS_FATAL_ALERT,
NDPI_SUSPICIOUS_ENTROPY,
NDPI_CLEAR_TEXT_CREDENTIALS,
NDPI_DNS_LARGE_PACKET,
NDPI_DNS_FRAGMENTED,
NDPI_INVALID_CHARACTERS,
NDPI_POSSIBLE_EXPLOIT,
NDPI_TLS_CERTIFICATE_ABOUT_TO_EXPIRE,
/* Leave this as last member */
NDPI_MAX_RISK
@@ -433,11 +444,6 @@ struct ndpi_id_struct {
to compare this, use:
**/
NDPI_PROTOCOL_BITMASK detected_protocol_bitmask;
/* NDPI_PROTOCOL_RTSP */
ndpi_ip_addr_t rtsp_ip_address;
/* NDPI_PROTOCOL_YAHOO */
uint32_t yahoo_video_lan_timer;
/* NDPI_PROTOCOL_IRC_MAXPORT % 2 must be 0 */
/* NDPI_PROTOCOL_IRC */
@@ -449,65 +455,31 @@ struct ndpi_id_struct {
/* NDPI_PROTOCOL_GNUTELLA */
uint32_t gnutella_ts;
/* NDPI_PROTOCOL_THUNDER */
uint32_t thunder_ts;
/* NDPI_PROTOCOL_RTSP */
uint32_t rtsp_timer;
/* NDPI_PROTOCOL_ZATTOO */
uint32_t zattoo_ts;
/* NDPI_PROTOCOL_UNENCRYPTED_JABBER */
uint32_t jabber_stun_or_ft_ts;
/* NDPI_PROTOCOL_DIRECTCONNECT */
uint32_t directconnect_last_safe_access_time;
/* NDPI_PROTOCOL_SOULSEEK */
uint32_t soulseek_last_safe_access_time;
/* NDPI_PROTOCOL_DIRECTCONNECT */
uint16_t detected_directconnect_port;
uint16_t detected_directconnect_udp_port;
uint16_t detected_directconnect_ssl_port;
/* NDPI_PROTOCOL_BITTORRENT */
#define NDPI_BT_PORTS 8
uint16_t bt_port_t[NDPI_BT_PORTS];
uint16_t bt_port_u[NDPI_BT_PORTS];
/* NDPI_PROTOCOL_UNENCRYPTED_JABBER */
#define JABBER_MAX_STUN_PORTS 6
uint16_t jabber_voice_stun_port[JABBER_MAX_STUN_PORTS];
uint16_t jabber_file_transfer_port[2];
/* NDPI_PROTOCOL_GNUTELLA */
uint16_t detected_gnutella_port;
/* NDPI_PROTOCOL_GNUTELLA */
uint16_t detected_gnutella_udp_port1;
uint16_t detected_gnutella_udp_port2;
/* NDPI_PROTOCOL_SOULSEEK */
uint16_t soulseek_listen_port;
/* NDPI_PROTOCOL_IRC */
uint8_t irc_number_of_port;
/* NDPI_PROTOCOL_UNENCRYPTED_JABBER */
uint8_t jabber_voice_stun_used_ports;
/* NDPI_PROTOCOL_SIP */
/* NDPI_PROTOCOL_YAHOO */
uint32_t yahoo_video_lan_dir:1;
/* NDPI_PROTOCOL_YAHOO */
uint32_t yahoo_conf_logged_in:1;
uint32_t yahoo_voice_conf_logged_in:1;
/* NDPI_PROTOCOL_RTSP */
uint32_t rtsp_ts_set:1;
};
struct ndpi_flow_tcp_struct {
@@ -517,18 +489,11 @@ struct ndpi_flow_tcp_struct {
/* NDPI_PROTOCOL_MAIL_POP */
uint16_t pop_command_bitmask;
/* NDPI_PROTOCOL_QQ */
uint16_t qq_nxt_len;
/* NDPI_PROTOCOL_WHATSAPP */
uint8_t wa_matched_so_far;
/* NDPI_PROTOCOL_TDS */
uint8_t tds_login_version;
/* NDPI_PROTOCOL_IRC */
uint8_t irc_stage;
uint8_t irc_port;
/* NDPI_PROTOCOL_H323 */
uint8_t h323_valid_packets;
@@ -542,38 +507,16 @@ struct ndpi_flow_tcp_struct {
uint32_t irc_direction:2;
uint32_t irc_0x1000_full:1;
/* NDPI_PROTOCOL_SOULSEEK */
uint32_t soulseek_stage:2;
/* NDPI_PROTOCOL_TDS */
uint32_t tds_stage:3;
/* NDPI_PROTOCOL_USENET */
uint32_t usenet_stage:2;
/* NDPI_PROTOCOL_IMESH */
uint32_t imesh_stage:4;
/* NDPI_PROTOCOL_HTTP */
uint32_t http_setup_dir:2;
uint32_t http_stage:2;
uint32_t http_empty_line_seen:1;
uint32_t http_wait_for_retransmission:1;
/* NDPI_PROTOCOL_GNUTELLA */
uint32_t gnutella_stage:2; // 0 - 2
/* NDPI_CONTENT_MMS */
uint32_t mms_stage:2;
/* NDPI_PROTOCOL_YAHOO */
uint32_t yahoo_sip_comm:1;
uint32_t yahoo_http_proxy_stage:2;
/* NDPI_PROTOCOL_MSN */
uint32_t msn_stage:3;
uint32_t msn_ssl_ft:2;
/* NDPI_PROTOCOL_SSH */
uint32_t ssh_stage:3;
@@ -589,10 +532,8 @@ struct ndpi_flow_tcp_struct {
unsigned buffer_len, buffer_used;
} message;
void* srv_cert_fingerprint_ctx; /* SHA-1 */
/* NDPI_PROTOCOL_TLS */
uint8_t hello_processed:1, certificate_processed:1, subprotocol_detected:1, fingerprint_set:1, _pad:4;
uint8_t certificate_processed:1, fingerprint_set:1, _pad:6;
uint8_t sha1_certificate_fingerprint[20], num_tls_blocks;
int16_t tls_application_blocks_len[NDPI_MAX_NUM_TLS_APPL_BLOCKS];
} tls;
@@ -600,8 +541,7 @@ struct ndpi_flow_tcp_struct {
/* NDPI_PROTOCOL_POSTGRES */
uint32_t postgres_stage:3;
/* NDPI_PROTOCOL_DIRECT_DOWNLOAD_LINK */
uint32_t ddlink_server_direction:1;
/* Part of the TCP header. */
uint32_t seen_syn:1;
uint32_t seen_syn_ack:1;
uint32_t seen_ack:1;
@@ -618,9 +558,6 @@ struct ndpi_flow_tcp_struct {
/* NDPI_PROTOCOL_WORLDOFWARCRAFT */
uint32_t wow_stage:2;
/* NDPI_PROTOCOL_HTTP_APPLICATION_VEOHTV */
uint32_t veoh_tv_stage:2;
/* NDPI_PROTOCOL_SHOUTCAST */
uint32_t shoutcast_stage:2;
@@ -660,12 +597,6 @@ struct ndpi_flow_tcp_struct {
};
struct ndpi_flow_udp_struct {
/* NDPI_PROTOCOL_SNMP */
uint32_t snmp_msg_id;
/* NDPI_PROTOCOL_SNMP */
uint32_t snmp_stage:2;
/* NDPI_PROTOCOL_PPSTREAM */
uint32_t ppstream_stage:3; // 0 - 7
@@ -681,9 +612,6 @@ struct ndpi_flow_udp_struct {
/* NDPI_PROTOCOL_XBOX */
uint32_t xbox_stage:1;
/* NDPI_PROTOCOL_WINDOWS_UPDATE */
uint32_t wsus_stage:1;
/* NDPI_PROTOCOL_SKYPE */
uint8_t skype_packet_id;
@@ -704,6 +632,21 @@ struct ndpi_flow_udp_struct {
/* NDPI_PROTOCOL_WIREGUARD */
uint8_t wireguard_stage;
uint32_t wireguard_peer_index[2];
/* NDPI_PROTOCOL_QUIC */
u_int8_t *quic_reasm_buf;
u_int32_t quic_reasm_buf_len;
/* NDPI_PROTOCOL_CSGO */
uint8_t csgo_strid[18],csgo_state,csgo_s2;
uint32_t csgo_id2;
/* NDPI_PROTOCOL_RDP */
u_int8_t rdp_to_srv[3], rdp_from_srv[3], rdp_to_srv_pkts, rdp_from_srv_pkts;
/* NDPI_PROTOCOL_IMO */
uint8_t imo_last_one_byte_pkt, imo_last_byte;
};
struct ndpi_int_one_line_struct {
@@ -722,7 +665,6 @@ struct ndpi_packet_struct {
uint64_t current_time_ms;
uint16_t detected_protocol_stack[2];
uint16_t protocol_stack_info;
struct ndpi_int_one_line_struct line[64];
/* HTTP headers */
@@ -746,19 +688,13 @@ struct ndpi_packet_struct {
uint8_t http_num_headers; /* number of found (valid) header lines in HTTP request or response */
uint16_t l3_packet_len;
uint16_t l4_packet_len;
uint16_t payload_packet_len;
uint16_t actual_payload_len;
uint16_t num_retried_bytes;
uint16_t parsed_lines;
uint16_t parsed_unix_lines;
uint16_t empty_line_position;
uint8_t tcp_retransmission;
uint8_t l4_protocol;
uint8_t tls_certificate_detected:4, tls_certificate_num_checks:4;
uint8_t packet_lines_parsed_complete:1,
packet_direction:1, empty_line_position_set:1, pad:5;
packet_direction:1, empty_line_position_set:1, http_check_content:1, pad:4;
};
struct ndpi_detection_module_struct;
@@ -952,8 +888,6 @@ struct ndpi_detection_module_struct {
/* HTTP/DNS/HTTPS host matching */
ndpi_automa host_automa, /* Used for DNS/HTTPS */
content_automa, /* Used for HTTP subprotocol_detection */
subprotocol_automa, /* Used for HTTP subprotocol_detection */
bigrams_automa, impossible_bigrams_automa; /* TOR */
/* IMPORTANT: please update ndpi_finalize_initalization() whenever you add a new automa */
@@ -970,18 +904,7 @@ struct ndpi_detection_module_struct {
uint32_t irc_timeout;
/* gnutella parameters */
uint32_t gnutella_timeout;
/* thunder parameters */
uint32_t thunder_timeout;
/* SoulSeek parameters */
uint32_t soulseek_connection_ip_tick_timeout;
/* rtsp parameters */
uint32_t rtsp_connection_timeout;
/* rstp */
uint32_t orb_rstp_ts_timeout;
/* yahoo */
uint8_t yahoo_detect_http_connections;
uint32_t yahoo_lan_video_timeout;
uint32_t zattoo_connection_timeout;
uint32_t jabber_stun_timeout;
uint32_t jabber_file_transfer_timeout;
uint8_t ip_version_limit;
@@ -1008,6 +931,9 @@ struct ndpi_detection_module_struct {
uint8_t direction_detect_disable:1, /* disable internal detection of packet direction */
_pad:7;
/* Current packet */
struct ndpi_packet_struct packet;
};
#define NDPI_CIPHER_SAFE 0
@@ -1022,7 +948,6 @@ typedef enum {
struct ndpi_flow_struct {
uint16_t detected_protocol_stack[2];
uint16_t protocol_stack_info;
/* init parameter, internal used to set up timestamp,... */
uint16_t guessed_protocol_id, guessed_host_protocol_id, guessed_category, guessed_header_category;
uint8_t l4_proto, protocol_id_already_guessed:1, host_already_guessed:1,
@@ -1049,13 +974,8 @@ struct ndpi_flow_struct {
/* Place textual flow info here */
char flow_extra_info[16];
/*
Pointer to src or dst that identifies the
server of this connection
*/
struct ndpi_id_struct *server_id;
/* HTTP host or DNS query */
uint8_t host_server_name[240];
char host_server_name[80];
uint8_t initial_binary_bytes[8], initial_binary_bytes_len;
uint8_t risk_checked;
ndpi_risk risk; /* Issues found with this flow [bitmask of ndpi_risk] */
@@ -1069,11 +989,11 @@ struct ndpi_flow_struct {
*/
struct {
ndpi_http_method method;
char *url, *content_type, *user_agent;
uint8_t num_request_headers, num_response_headers;
uint8_t request_version; /* 0=1.0 and 1=1.1. Create an enum for this? */
uint16_t response_status_code; /* 200, 404, etc. */
uint8_t detected_os[32]; /* Via HTTP/QUIC User-Agent */
char *url, *content_type, *user_agent;
char *detected_os; /* Via HTTP/QUIC User-Agent */
char *nat_ip; /* Via HTTP X-Forwarded-For */
} http;
@@ -1086,6 +1006,18 @@ struct ndpi_flow_struct {
char *pktbuf;
uint16_t pktbuf_maxlen, pktbuf_currlen;
} kerberos_buf;
struct {
u_int8_t num_udp_pkts, num_binding_requests;
u_int16_t num_processed_pkts;
} stun;
/* TODO: something clever to save memory */
struct {
uint8_t auth_found:1, auth_failed:1, auth_tls:1, auth_done:1, _pad:4;
char username[32], password[16];
} ftp_imap_pop_smtp;
union {
/* the only fields useful for nDPI and ntopng */
struct {
@@ -1104,37 +1036,25 @@ struct ndpi_flow_struct {
} kerberos;
struct {
struct {
char ssl_version_str[12];
uint16_t ssl_version, server_names_len;
char client_requested_server_name[64], *server_names,
*alpn, *tls_supported_versions, *issuerDN, *subjectDN;
char *server_names, *alpn, *tls_supported_versions, *issuerDN, *subjectDN;
uint32_t notBefore, notAfter;
char ja3_client[33], ja3_server[33];
uint16_t server_cipher;
u_int8_t sha1_certificate_fingerprint[20];
u_int8_t hello_processed:1, subprotocol_detected:1, _pad:6;
struct {
uint16_t cipher_suite;
char *esni;
} encrypted_sni;
ndpi_cipher_weakness server_unsafe_cipher;
} ssl;
struct {
uint8_t num_udp_pkts, num_processed_pkts, num_binding_requests;
} stun;
/* We can have STUN over SSL/TLS thus they need to live together */
} stun_ssl;
uint16_t ssl_version, server_names_len;
} tls_quic;
struct {
char client_signature[48], server_signature[48];
char hassh_client[33], hassh_server[33];
} ssh;
struct {
uint8_t last_one_byte_pkt, last_byte;
} imo;
struct {
uint8_t username_detected:1, username_found:1,
password_detected:1, password_found:1,
@@ -1151,16 +1071,6 @@ struct ndpi_flow_struct {
char version[32];
} ubntac2;
struct {
/* Via HTTP X-Forwarded-For */
uint8_t nat_ip[24];
} http;
struct {
uint8_t auth_found:1, auth_failed:1, _pad:5;
char username[16], password[16];
} ftp_imap_pop_smtp;
struct {
/* Bittorrent hash */
uint8_t hash[20];
@@ -1191,24 +1101,15 @@ struct ndpi_flow_struct {
/* NDPI_PROTOCOL_DIRECTCONNECT */
uint8_t directconnect_stage:2; // 0 - 1
/* NDPI_PROTOCOL_YAHOO */
uint8_t sip_yahoo_voice:1;
/* NDPI_PROTOCOL_HTTP */
uint8_t http_detected:1;
/* NDPI_PROTOCOL_RTSP */
uint8_t rtsprdt_stage:2, rtsp_control_flow:1;
/* NDPI_PROTOCOL_YAHOO */
uint8_t yahoo_detection_finished:2;
uint8_t rtsprdt_stage:2;
/* NDPI_PROTOCOL_ZATTOO */
uint8_t zattoo_stage:3;
/* NDPI_PROTOCOL_QQ */
uint8_t qq_stage:3;
/* NDPI_PROTOCOL_THUNDER */
uint8_t thunder_stage:2; // 0 - 3
@@ -1243,16 +1144,13 @@ struct ndpi_flow_struct {
uint8_t ovpn_session_id[8];
uint8_t ovpn_counter;
/* Flow key used to search a match into the mining cache */
u_int32_t key_mining_cache;
/* NDPI_PROTOCOL_TINC */
uint8_t tinc_state;
struct tinc_cache_entry tinc_cache_entry;
/* NDPI_PROTOCOL_CSGO */
uint8_t csgo_strid[18],csgo_state,csgo_s2;
uint32_t csgo_id2;
/* internal structures to save functions calls */
struct ndpi_packet_struct packet;
struct ndpi_flow_struct *flow;
struct ndpi_id_struct *src;
struct ndpi_id_struct *dst;
};
+36 -105
View File
@@ -251,22 +251,12 @@ NDPIDetectionModuleStruct._fields_ = [
("ndpi_num_supported_protocols", c_uint),
("ndpi_num_custom_protocols", c_uint),
("host_automa", NDPIAutoma),
("content_automa", NDPIAutoma),
("subprotocol_automa", NDPIAutoma),
("bigrams_automa", NDPIAutoma),
("impossible_bigrams_automa", NDPIAutoma),
("custom_categories", CustomCategories),
("protocols_ptree", c_void_p),
("irc_timeout", c_uint32),
("gnutella_timeout", c_uint32),
("battlefield_timeout", c_uint32),
("thunder_timeout", c_uint32),
("soulseek_connection_ip_tick_timeout", c_uint32),
("rtsp_connection_timeout", c_uint32),
("tvants_connection_timeout", c_uint32),
("orb_rstp_ts_timeout", c_uint32),
("yahoo_detect_http_connections", c_uint8),
("yahoo_lan_video_timeout", c_uint32),
("zattoo_connection_timeout", c_uint32),
("jabber_stun_timeout", c_uint32),
("jabber_file_transfer_timeout", c_uint32),
@@ -279,11 +269,9 @@ NDPIDetectionModuleStruct._fields_ = [
("tinc_cache", POINTER(Cache)),
("proto_defaults", NDPIProtoDefaultsT * (ndpi.ndpi_wrap_ndpi_max_supported_protocols() +
ndpi.ndpi_wrap_ndpi_max_num_custom_protocols())),
("http_dont_dissect_response", c_uint8, 1),
("dns_dont_dissect_response", c_uint8, 1),
("direction_detect_disable", c_uint8, 1),
("disable_metadata_export", c_uint8, 1),
("hyperscan", c_void_p)
('_pad', c_uint8, 7),
('packet', NDPIPacketStruct),
]
@@ -312,38 +300,24 @@ class NDPIIpAddrT(Union):
class NDPIIdStruct(Structure):
_fields_ = [
('detected_protocol_bitmask', NDPIProtocolBitMask),
('rtsp_ip_address', NDPIIpAddrT),
('yahoo_video_lan_timer', c_uint32),
('irc_port', c_uint16 * 8),
('last_time_port_used', c_uint32 * 8),
('irc_ts', c_uint32),
('gnutella_ts', c_uint32),
('battlefield_ts', c_uint32),
('thunder_ts', c_uint32),
('rtsp_timer', c_uint32),
('oscar_last_safe_access_time', c_uint32),
('zattoo_ts', c_uint32),
('jabber_stun_or_ft_ts', c_uint32),
('directconnect_last_safe_access_time', c_uint32),
('soulseek_last_safe_access_time', c_uint32),
('detected_directconnect_port', c_uint16),
('detected_directconnect_udp_port', c_uint16),
('detected_directconnect_ssl_port', c_uint16),
('bt_port_t', c_uint16 * 8),
('bt_port_u', c_uint16 * 8),
('jabber_voice_stun_port', c_uint16 * 6),
('jabber_file_transfer_port', c_uint16 * 2),
('detected_gnutella_port', c_uint16),
('detected_gnutella_udp_port1', c_uint16),
('detected_gnutella_udp_port2', c_uint16),
('soulseek_listen_port', c_uint16),
('irc_number_of_port', c_uint8),
('oscar_ssl_session_id', c_uint8 * 33),
('jabber_voice_stun_used_ports', c_uint8),
('yahoo_video_lan_dir', c_uint32, 1),
('yahoo_conf_logged_in', c_uint32, 1),
('yahoo_voice_conf_logged_in', c_uint32, 1),
('rtsp_ts_set', c_uint32, 1),
]
@@ -352,35 +326,21 @@ class NDPIFlowTcpStruct(Structure):
_fields_ = [
('smtp_command_bitmask', c_uint16),
('pop_command_bitmask', c_uint16),
('qq_nxt_len', c_uint16),
('wa_matched_so_far', c_uint8),
('tds_login_version', c_uint8),
('irc_stage', c_uint8),
('irc_port', c_uint8),
('h323_valid_packets', c_uint8),
('gnutella_msg_id', c_uint8 * 3),
('irc_3a_counter', c_uint32, 3),
('irc_stage2', c_uint32, 5),
('irc_direction', c_uint32, 2),
('irc_0x1000_full', c_uint32, 1),
('soulseek_stage', c_uint32, 2),
('tds_stage', c_uint32, 3),
('usenet_stage', c_uint32, 2),
('imesh_stage', c_uint32, 4),
('http_setup_dir', c_uint32, 2),
('http_stage', c_uint32, 2),
('http_empty_line_seen', c_uint32, 1),
('http_wait_for_retransmission', c_uint32, 1),
('gnutella_stage', c_uint32, 2),
('mms_stage', c_uint32, 2),
('yahoo_sip_comm', c_uint32, 1),
('yahoo_http_proxy_stage', c_uint32, 2),
('msn_stage', c_uint32, 3),
('msn_ssl_ft', c_uint32, 2),
('ssh_stage', c_uint32, 3),
('vnc_stage', c_uint32, 2),
('telnet_stage', c_uint32, 2),
('tls_srv_cert_fingerprint_ctx', c_void_p),
('tls_seen_client_cert', c_uint8, 1),
('tls_seen_server_cert', c_uint8, 1),
('tls_seen_certificate', c_uint8, 1),
@@ -391,7 +351,6 @@ class NDPIFlowTcpStruct(Structure):
('tls_fingerprint_len', c_int16),
('tls_sha1_certificate_fingerprint', c_uint8 * 20),
('postgres_stage', c_uint32, 3),
('ddlink_server_direction', c_uint32, 1),
('seen_syn', c_uint32, 1),
('seen_syn_ack', c_uint32, 1),
('seen_ack', c_uint32, 1),
@@ -399,7 +358,6 @@ class NDPIFlowTcpStruct(Structure):
('dofus_stage', c_uint32, 1),
('fiesta_stage', c_uint32, 2),
('wow_stage', c_uint32, 2),
('veoh_tv_stage', c_uint32, 2),
('shoutcast_stage', c_uint32, 2),
('rtp_special_packets_seen', c_uint32, 1),
('mail_pop_stage', c_uint32, 2),
@@ -420,16 +378,11 @@ class NDPIFlowTcpStruct(Structure):
class NDPIFlowUdpStruct(Structure):
_pack_ = 1
_fields_ = [
('battlefield_msg_id', c_uint32),
('snmp_msg_id', c_uint32),
('battlefield_stage', c_uint32, 3),
('snmp_stage', c_uint32, 2),
('ppstream_stage', c_uint32, 3),
('halflife2_stage', c_uint32, 2),
('tftp_stage', c_uint32, 2),
('aimini_stage', c_uint32, 5),
('xbox_stage', c_uint32, 1),
('wsus_stage', c_uint32, 1),
('skype_packet_id', c_uint8),
('teamviewer_stage', c_uint8),
('eaq_pkt_id', c_uint8),
@@ -439,6 +392,18 @@ class NDPIFlowUdpStruct(Structure):
('memcached_matches', c_uint8),
('wireguard_stage', c_uint8),
('wireguard_peer_index', c_uint32 * 2),
('quic_reasm_buf', POINTER(c_uint8)),
('quic_reasm_buf_len', c_uint32),
('csgo_strid', c_uint8 * 18),
('csgo_state', c_uint8),
('csgo_s2', c_uint8),
('csgo_id2', c_uint32),
('rdp_to_srv', c_uint8 * 3),
('rdp_from_srv', c_uint8 * 3),
('rdp_to_srv_pkts,', c_uint8),
('rdp_from_srv_pkts', c_uint8),
('imo_last_one_byte_pkt,', c_uint8),
('imo_last_byte', c_uint8),
]
@@ -449,13 +414,13 @@ class L4(Union):
class Http(Structure):
_fields_ = [
("method", c_int),
("url", c_char_p),
("content_type", c_char_p),
("num_request_headers", c_uint8),
("num_response_headers", c_uint8),
("request_version", c_uint8),
("response_status_code", c_uint16),
("detected_os", c_char * 32),
("url", c_char_p),
("content_type", c_char_p),
("user_agent", c_char_p),
("detected_os", c_char_p),
("nat_ip", c_char_p),
]
@@ -482,7 +447,7 @@ class Kerberos(Structure):
("realm", c_char * 24)]
class Ssl(Structure):
class QuicSsl(Structure):
_fields_ = [
("ssl_version", c_uint16),
("client_certificate", c_char * 64),
@@ -505,10 +470,6 @@ class Stun(Structure):
]
class StunSsl(Structure):
_fields_ = [("ssl", Ssl), ("stun", Stun)]
class Ssh(Structure):
_fields_ = [
("client_signature", c_char * 48),
@@ -518,13 +479,6 @@ class Ssh(Structure):
]
class Imo(Structure):
_fields_ = [
("last_one_byte_pkt", c_uint8),
("last_byte", c_uint8)
]
class Mdns(Structure):
_fields_ = [("answer", c_char * 96)]
@@ -533,12 +487,17 @@ class Ubntac2(Structure):
_fields_ = [("version", c_char * 32)]
class Http2(Structure):
class FtpImapPopSmtp(Structure):
_fields_ = [
("nat_ip", c_char * 24)
("auth_found", c_uint8, 1),
("auth_failed", c_uint8, 1),
("auth_tls", c_uint8, 1),
("auth_done", c_uint8, 1),
("_pad", c_uint8, 4),
("username", c_char * 32),
("password", c_char * 16)
]
class Bittorrent(Structure):
_fields_ = [("hash", c_char * 20)]
@@ -554,12 +513,10 @@ class Protos(Union):
_fields_ = [
("dns", Dns),
("kerberos", Kerberos),
("stun_ssl", StunSsl),
("quic_ssl", QuicSsl),
("ssh", Ssh),
("imo", Imo),
("mdns", Mdns),
("ubntac2", Ubntac2),
("http", Http2),
("bittorrent", Bittorrent),
("dhcp", Dhcp)
]
@@ -649,13 +606,6 @@ class NDPIUdpHdr(Structure):
]
class NDPIPacketStructStack(Structure):
_pack_ = 1
_fields_ = [
('protocol_stack_info', c_uint16)
]
class NDPIPacketStruct(Structure):
_fields_ = [
('iph', POINTER(NDPIIphdr)),
@@ -664,10 +614,8 @@ class NDPIPacketStruct(Structure):
('udp', POINTER(NDPIUdpHdr)),
('generic_l4_ptr', POINTER(c_uint8)),
('payload', POINTER(c_uint8)),
('tick_timestamp', c_uint32),
('tick_timestamp_l', c_uint64),
('current_timestamp_ms', c_uint64),
('detected_protocol_stack', c_uint16 * ndpi.ndpi_wrap_ndpi_procol_size()),
('ndpi_packet_stack', NDPIPacketStructStack),
('line', NDPIIntOneLineStruct * 64),
('host_line', NDPIIntOneLineStruct),
('forwarded_line', NDPIIntOneLineStruct),
@@ -687,21 +635,15 @@ class NDPIPacketStruct(Structure):
('http_response', NDPIIntOneLineStruct),
('http_num_headers', c_uint8),
('l3_packet_len', c_uint16),
('l4_packet_len', c_uint16),
('payload_packet_len', c_uint16),
('actual_payload_len', c_uint16),
('num_retried_bytes', c_uint16),
('parsed_lines', c_uint16),
('parsed_unix_lines', c_uint16),
('empty_line_position', c_uint16),
('tcp_retransmission', c_uint8),
('l4_protocol', c_uint8),
('ssl_certificate_detected', c_uint8, 4),
('ssl_certificate_num_checks', c_uint8, 4),
('packet_lines_parsed_complete', c_uint8, 1),
('packet_direction', c_uint8, 1),
('empty_line_position_set', c_uint8, 1),
('pad', c_uint8, 5),
('http_check_content', c_uint8, 1),
('pad', c_uint8, 4),
]
@@ -709,7 +651,6 @@ class NDPIFlowStructStack(Structure):
_pack_ = 1
_fields_ = [
("detected_protocol_stack", c_uint16 * ndpi.ndpi_wrap_ndpi_procol_size()),
("protocol_stack_info", c_uint16)
]
@@ -732,9 +673,10 @@ NDPIFlowStruct._fields_ = [
("num_processed_pkts", c_uint8),
("extra_packets_func", CFUNCTYPE(c_int, POINTER(NDPIDetectionModuleStruct), POINTER(NDPIFlowStruct))),
("l4", L4),
("server_id", POINTER(NDPIIdStruct)),
("host_server_name", c_ubyte * 256),
("host_server_name", c_char * 80),
("http", Http),
("stun", Stun),
("ftp_imap_pop_smtp", FtpImapPopSmtp),
("protos", Protos),
("excluded_protocol_bitmask", NDPIProtocolBitMask),
("category", c_int),
@@ -745,15 +687,11 @@ NDPIFlowStruct._fields_ = [
('byte_counter', c_uint16 * 2),
('bittorrent_stage', c_uint8),
('directconnect_stage', c_uint8, 2),
('sip_yahoo_voice', c_uint8, 1),
('http_detected', c_uint8, 1),
('http_upper_protocol', c_uint16),
('http_lower_protocol', c_uint16),
('rtsprdt_stage', c_uint8, 2),
('rtsp_control_flow', c_uint8, 1),
('yahoo_detection_finished', c_uint8, 2),
('zattoo_stage', c_uint8, 3),
('qq_stage', c_uint8, 3),
('thunder_stage', c_uint8, 2),
('oscar_ssl_voice_stage', c_uint8, 3),
('oscar_video_voice', c_uint8, 1),
@@ -773,16 +711,9 @@ NDPIFlowStruct._fields_ = [
('starcraft_udp_stage', c_uint8, 3),
('ovpn_session_id', c_uint8 * 8),
('ovpn_counter', c_uint8),
('key_mining_cache', c_uint32),
('tinc_state', c_uint8),
('TincCacheEntry', TincCacheEntry),
('csgo_strid', c_uint8 * 18),
('csgo_state', c_uint8),
('csgo_s2', c_uint8),
('csgo_id2', c_uint32),
('kxun_counter', c_uint16),
('iqiyi_counter', c_uint16),
('packet', NDPIPacketStruct),
('flow', POINTER(NDPIFlowStruct)),
('src', POINTER(NDPIIdStruct)),
('dst', POINTER(NDPIIdStruct))
]
+26
View File
@@ -0,0 +1,26 @@
CC=@CC@
INC=-I ../src/include -I/usr/local/include
LIBDPI=../src/lib/libndpi.a
LDFLAGS=@LDFLAGS@
LIB=$(LIBDPI) -lrrd -lm @LIBS@ @ADDITIONAL_LIBS@
TOOLS=rrd_anomaly rrd_similarity
all: $(TOOLS)
rrd_anomaly: rrd_anomaly.c Makefile $(LIBDPI)
$(CC) $(CPPFLAGS) -g $(INC) $(LDFLAGS) rrd_anomaly.c -o rrd_anomaly $(LIB)
rrd_similarity: rrd_similarity.c Makefile $(LIBDPI)
$(CC) $(CPPFLAGS) -g $(INC) $(LDFLAGS) rrd_similarity.c -o rrd_similarity $(LIB)
clean:
/bin/rm -f *.o $(TOOLS) *~
distclean: clean
/bin/rm -f Makefile
distdir:
install:
@echo -n ""
+10
View File
@@ -0,0 +1,10 @@
This directory contains a tool that allows to identify anomalies and similarities in RRD files
Prerequisite
- rrdtool (https://oss.oetiker.ch/rrdtool/)
If you have Ubuntu-based systems you need to do
- sudo apt-get install rrdtool librrd-dev
+184
View File
@@ -0,0 +1,184 @@
/*
* rrd_anomaly.c
*
* Copyright (C) 2011-22 - ntop.org
*
* nDPI is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* nDPI is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public License
* along with nDPI. If not, see <http://www.gnu.org/licenses/>.
*
*/
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <getopt.h>
#include <rrd.h>
#include "ndpi_api.h"
#define DEFAULT_ALPHA 0.5
#define DEFAULT_RO 0.05
#define DEFAULT_START "now-1d"
#define DEFAULT_END "now"
/* *************************************************** */
static void help() {
printf("Usage: rrd_anomaly [-v][-a <alpha>][-e <end>][-q][-s <start>] -f <filename>\n"
"-a | Set alpha. Valid range >0 .. <1. Default %.2f\n"
"-e <end> | RRD end time. Default %s\n"
"-q | Quick output (only anomalies are reported)\n"
"-s <start> | RRD start time. Default %s\n"
"-f <rrd path> | Path of the RRD filename to analyze\n"
"-v | Verbose\n"
,
DEFAULT_ALPHA, DEFAULT_END, DEFAULT_START);
printf("\n\nExample: rrd_anomaly -q -f hum.rrd\n");
exit(0);
}
/* *************************************************** */
int main(int argc, char *argv[]) {
rrd_time_value_t start_tv, end_tv;
unsigned long step = 0, ds_cnt = 0;
rrd_value_t *data, *p;
char **names, *filename = NULL, *start_s, *end_s, *cf;
u_int i, j, t, first = 1, quick_mode = 0, verbose = 0;
time_t start, end;
struct ndpi_ses_struct ses;
float alpha, ro;
char c;
/* Defaults */
alpha = DEFAULT_ALPHA;
start_s = DEFAULT_START;
end_s = DEFAULT_END;
cf = "AVERAGE";
ro = DEFAULT_RO;
while((c = getopt(argc, argv, "s:e:a:qf:r:v")) != '?') {
if(c == -1) break;
switch(c) {
case 's':
start_s = optarg;
break;
case 'e':
end_s = optarg;
break;
case 'q':
quick_mode = 1;
break;
case 'a':
{
float f = atof(optarg);
if((f > 0) && (f < 1))
alpha = f;
else
printf("Discarding -a: valid range is >0 .. <1\n");
}
break;
case 'f':
filename = optarg;
break;
case 'r':
ro = atof(optarg);
if((ro <= 0) || (ro >= 1))
ro = DEFAULT_RO;
break;
case 'v':
verbose = 1;
break;
default:
help();
break;
}
}
if(filename == NULL)
help();
ndpi_ses_init(&ses, alpha, ro);
if((rrd_parsetime(start_s, &start_tv) != NULL)) {
printf("Unable to parse start time %s\n", start_s);
return(-1);
}
if((rrd_parsetime(end_s, &end_tv) != NULL)) {
printf("Unable to parse end time %s\n", end_s);
return(-1);
}
rrd_proc_start_end(&start_tv, &end_tv, &start, &end);
if(rrd_fetch_r(filename, cf, &start, &end, &step, &ds_cnt, &names, &data) != 0) {
printf("Unable to extract data from rrd %s\n", filename);
return(-2);
}
p = data;
for(t=start+1, i=0; t<end; t+=step, i++) {
j = 0; /* Consider only the first DS */
/* for(j=0; j<ds_cnt; j++) */ {
rrd_value_t value = *p++;
if(!isnan(value)) {
double prediction, confidence_band;
double lower, upper;
char buf[32];
int rc;
u_int is_anomaly;
value *= 100; /* trick to avoid dealing with floats */
rc = ndpi_ses_add_value(&ses, value, &prediction, &confidence_band);
lower = prediction - confidence_band, upper = prediction + confidence_band;
is_anomaly = ((rc == 0) || (confidence_band == 0) || ((value >= lower) && (value <= upper))) ? 0 : 1;
if(verbose || is_anomaly) {
if(quick_mode) {
printf("%u\n", t);
} else {
const time_t _t = t;
struct tm *t_info = localtime((const time_t*)&_t);
strftime(buf, sizeof(buf), "%d/%b/%Y %H:%M:%S", t_info);
if(first) {
first = 0;
printf("%s %s\t%s %s %s\t %s [%s]\n",
"When", "Value", "Prediction", "Lower", "Upper", "Out", "Band");
}
printf("%s %12.3f\t%.3f\t%12.3f\t%12.3f\t %s [%.3f]\n",
buf, value/100., prediction/100., lower/100., upper/100., is_anomaly? "ANOMALY" : "OK",
confidence_band/100.);
}
}
}
}
}
rrd_freemem(data);
return(0);
}
+319
View File
@@ -0,0 +1,319 @@
/*
* rrd_similarity.c
*
* Copyright (C) 2011-22 - ntop.org
*
* nDPI is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* nDPI is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public License
* along with nDPI. If not, see <http://www.gnu.org/licenses/>.
*
*/
#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <getopt.h>
#include <dirent.h>
#include <sys/stat.h>
#include "rrd.h"
#include "ndpi_api.h"
#define DEFAULT_ALPHA 0.5
#define DEFAULT_START "now-1d"
#define DEFAULT_END "now"
#define MAX_NUM_RRDS 8192
#ifndef PATH_MAX
#define PATH_MAX 4096
#endif
typedef struct {
char *path;
float average, stddev;
struct ndpi_bin b;
} rrd_file_stats;
u_int verbose = 0, similarity_threshold = 100, skip_zero = 0;
/* *************************************************** */
static void help() {
printf("Usage: rrd_similarity [-v][-a <alpha>][-e <end>][-q][-s <start>]\n"
" -f <filename> -d <basedir> [-t <threshold>]\n"
"-a | Set alpha. Valid range >0 .. <1. Default %.2f\n"
"-e <end> | RRD end time. Default %s\n"
"-q | Quick output (only anomalies are reported)\n"
"-s <start> | RRD start time. Default %s\n"
"-d <basedir> | Base directory where RRD filename is searched\n"
"-f <rrd path> | Path of the RRD filename to analyze\n"
"-t <threshold> | Similarity threshold. Default %u (0 == alike)\n"
"-v | Verbose\n"
"-z | Skip zero RRDs during comparison\n"
,
DEFAULT_ALPHA, DEFAULT_END, DEFAULT_START, similarity_threshold);
printf("\n\nExample: rrd_similarity -q -f bytes.rrd -d /var/lib/ntopng/-1/snmpstats\n");
printf("\n\nGoal: find similar RRDs\n");
exit(0);
}
/* *************************************************** */
void analyze_rrd(rrd_file_stats *rrd, time_t start, time_t end) {
unsigned long step = 0, ds_cnt = 0;
rrd_value_t *data, *p;
char **names;
u_int t, i, j, num_points;
struct ndpi_analyze_struct *s;
if(rrd_fetch_r(rrd->path, "AVERAGE", &start, &end, &step, &ds_cnt, &names, &data) != 0) {
printf("Unable to extract data from rrd %s\n", rrd->path);
return;
}
p = data;
num_points = (end-start)/step;
if((s = ndpi_alloc_data_analysis(num_points)) == NULL)
return;
ndpi_init_bin(&rrd->b, ndpi_bin_family32, num_points);
/* Step 1 - Compute average and stddev */
for(t=start+1, i=0; t<end; t+=step, i++) {
double value = (double)*p++;
if(isnan(value)) value = 0;
ndpi_data_add_value(s, value);
ndpi_set_bin(&rrd->b, i, value);
}
rrd->average = ndpi_data_average(s);
rrd->stddev = ndpi_data_stddev(s);
/* Step 2 - Bin analysis */
ndpi_free_data_analysis(s, 1);
rrd_freemem(data);
}
/* *************************************************** */
int circles_touch(int x1, int r1, int x2, int r2) {
int radius_sum = r1+r2;
int x_diff = abs(x1 - x2);
return((radius_sum < x_diff) ? 0 : 1);
}
/* *************************************************** */
void find_rrd_similarities(rrd_file_stats *rrd, u_int num_rrds) {
u_int i, j, num_similar_rrds = 0, num_potentially_zero_equal = 0;
for(i=0; i<num_rrds; i++) {
for(j=i+1; j<num_rrds; j++) {
/*
Average is the circle center, and stddev is the radius
if circles touch each other then there is a chance that
the two rrds are similar
*/
if((rrd[i].average == 0) && (rrd[i].average == rrd[j].average)) {
if(!skip_zero)
printf("%s [%.1f/%.1f] - %s [%.1f/%.1f] are alike\n",
rrd[i].path, rrd[i].average, rrd[i].stddev,
rrd[j].path, rrd[j].average, rrd[j].stddev);
num_potentially_zero_equal++;
} else if(circles_touch(rrd[i].average, rrd[i].stddev, rrd[j].average, rrd[j].stddev)
) {
float similarity = ndpi_bin_similarity(&rrd[i].b, &rrd[j].b, 0, similarity_threshold);
if((similarity >= 0) && (similarity < similarity_threshold)) {
if(verbose)
printf("%s [%.1f/%.1f] - %s [%.1f/%.1f] are %s [%.1f]\n",
rrd[i].path, rrd[i].average, rrd[i].stddev,
rrd[j].path, rrd[j].average, rrd[j].stddev,
(similarity == 0) ? "alike" : "similar",
similarity
);
num_similar_rrds++;
}
}
}
}
printf("Found %u (%.3f %%) similar RRDs / %u zero alike RRDs [num_rrds: %u]\n",
num_similar_rrds,
(num_similar_rrds*100.)/(float)(num_rrds*num_rrds),
num_potentially_zero_equal,
num_rrds);
}
/* *************************************************** */
void find_rrds(char *basedir, char *filename, rrd_file_stats *rrds, u_int *num_rrds) {
struct dirent **namelist;
int n = scandir(basedir, &namelist, 0, NULL);
if(n < 0)
return; /* End of the tree */
while(n--) {
if(namelist[n]->d_name[0] != '.') {
char path[PATH_MAX];
struct stat s;
snprintf(path, sizeof(path), "%s/%s", basedir, namelist[n]->d_name);
if(stat(path, &s) == 0) {
if(S_ISDIR(s.st_mode))
find_rrds(path, filename, rrds, num_rrds);
else if(strcmp(namelist[n]->d_name, filename) == 0) {
if(*num_rrds < MAX_NUM_RRDS) {
rrds[*num_rrds].path = strdup(path);
if(rrds[*num_rrds].path != NULL)
(*num_rrds)++;
}
}
}
}
free(namelist[n]);
}
free(namelist);
}
/* *************************************************** */
int main(int argc, char *argv[]) {
rrd_time_value_t start_tv, end_tv;
char *filename = NULL, *start_s, *end_s, *dirname = NULL, *basedir = NULL;
u_int first = 1, quick_mode = 0;
float alpha;
char c;
time_t start, end;
u_int num_rrds = 0, i;
rrd_file_stats *rrds;
/* Defaults */
alpha = DEFAULT_ALPHA;
start_s = DEFAULT_START;
end_s = DEFAULT_END;
while((c = getopt(argc, argv, "d:s:e:a:qf:t:vz")) != '?') {
if(c == -1) break;
switch(c) {
case 's':
start_s = optarg;
break;
case 'd':
basedir = optarg;
break;
case 'e':
end_s = optarg;
break;
case 'q':
quick_mode = 1;
break;
case 'v':
verbose = 1;
break;
case 'a':
{
float f = atof(optarg);
if((f > 0) && (f < 1))
alpha = f;
else
printf("Discarding -a: valid range is >0 .. <1\n");
}
break;
case 'f':
filename = optarg;
break;
case 't':
similarity_threshold = atoi(optarg);
break;
case 'z':
skip_zero = 1;
break;
default:
help();
break;
}
}
if((filename == NULL) || (basedir == NULL))
help();
if((rrd_parsetime(start_s, &start_tv) != NULL)) {
printf("Unable to parse start time %s\n", start_s);
return(-1);
}
if((rrd_parsetime(end_s, &end_tv) != NULL)) {
printf("Unable to parse end time %s\n", end_s);
return(-1);
}
rrd_proc_start_end(&start_tv, &end_tv, &start, &end);
if((rrds = ndpi_calloc(sizeof(rrd_file_stats), MAX_NUM_RRDS)) == NULL) {
printf("Not enough memory !\n");
return(-1);
}
/* Find all rrd's */
find_rrds(basedir, filename, rrds, &num_rrds);
/* Read RRD's data */
for(i=0; i<num_rrds; i++)
analyze_rrd(&rrds[i], start, end);
find_rrd_similarities(rrds, num_rrds);
#if 0
if(verbose) {
for(i=0; i<num_rrds; i++)
printf("%s\t%.1f\t%.1f\n", rrds[i].path, rrds[i].average, rrds[i].stddev);
}
#endif
for(i=0; i<num_rrds; i++) {
ndpi_free_bin(&rrds[i].b);
free(rrds[i].path);
}
ndpi_free(rrds);
return(0);
}
+88 -55
View File
@@ -36,7 +36,6 @@ extern "C" {
*/
#define NDPI_API_VERSION @NDPI_API_VERSION@
#define SIZEOF_ID_STRUCT ( sizeof(struct ndpi_id_struct) )
#define SIZEOF_FLOW_STRUCT ( sizeof(struct ndpi_flow_struct) )
#define NDPI_DETECTION_ONLY_IPV4 ( 1 << 0 )
@@ -75,15 +74,6 @@ extern "C" {
u_int32_t ndpi_detection_get_sizeof_ndpi_flow_struct(void);
/**
* Get the size of the id struct
*
* @return the size of the id struct
*
*/
u_int32_t ndpi_detection_get_sizeof_ndpi_id_struct(void);
/**
* Get the size of the flow tcp struct
*
@@ -118,7 +108,8 @@ extern "C" {
void ndpi_free(void *ptr);
void * ndpi_flow_malloc(size_t size);
void ndpi_flow_free(void *ptr);
u_int32_t ndpi_get_tot_allocated_memory(void);
/**
* Search the first occurrence of substring -find- in -s-
* The search is limited to the first -slen- characters of the string
@@ -302,8 +293,6 @@ extern "C" {
* @par packet = unsigned char pointer to the Layer 3 (IP header)
* @par packetlen = the length of the packet
* @par packet_time_ms = the current timestamp for the packet (expressed in msec)
* @par src = pointer to the source subscriber state machine
* @par dst = pointer to the destination subscriber state machine
* @return void
*
*/
@@ -311,9 +300,7 @@ extern "C" {
struct ndpi_flow_struct *flow,
const unsigned char *packet,
const unsigned short packetlen,
const u_int64_t packet_time_ms,
struct ndpi_id_struct *src,
struct ndpi_id_struct *dst);
const u_int64_t packet_time_ms);
/**
* Processes one packet and returns the ID of the detected protocol.
@@ -324,8 +311,6 @@ extern "C" {
* @par packet = unsigned char pointer to the Layer 3 (IP header)
* @par packetlen = the length of the packet
* @par packet_time_ms = the current timestamp for the packet (expressed in msec)
* @par src = pointer to the source subscriber state machine
* @par dst = pointer to the destination subscriber state machine
* @return the detected ID of the protocol
*
*/
@@ -333,9 +318,7 @@ extern "C" {
struct ndpi_flow_struct *flow,
const unsigned char *packet,
const unsigned short packetlen,
const u_int64_t packet_time_ms,
struct ndpi_id_struct *src,
struct ndpi_id_struct *dst);
const u_int64_t packet_time_ms);
/**
* Get the main protocol of the passed flows for the detected module
*
@@ -424,7 +407,6 @@ extern "C" {
* @par string_to_match = the string to match
* @par string_to_match_len = the length of the string
* @par ret_match = completed returned match information
* @par is_host_match = value of the second field of struct ndpi_automa
* @return the ID of the matched subprotocol;
* -1 if automa is not finalized;
* -2 if automa==NULL or string_to_match==NULL or empty string_to_match
@@ -433,8 +415,7 @@ extern "C" {
int ndpi_match_string_subprotocol(struct ndpi_detection_module_struct *ndpi_struct,
char *string_to_match,
u_int string_to_match_len,
ndpi_protocol_match_result *ret_match,
u_int8_t is_host_match);
ndpi_protocol_match_result *ret_match);
/**
* Check if the host passed match with a protocol
*
@@ -464,24 +445,6 @@ extern "C" {
void ndpi_check_subprotocol_risk(struct ndpi_detection_module_struct *ndpi_str,
struct ndpi_flow_struct *flow, u_int16_t subprotocol_id);
/**
* Check if the string content passed match with a protocol
*
* @par ndpi_struct = the detection module
* @par flow = the flow where match the host
* @par string_to_match = the string to match
* @par string_to_match_len = the length of the string
* @par ret_match = completed returned match information
* @par master_protocol_id = value of the ID associated to the master protocol detected
* @return the ID of the matched subprotocol
*
*/
u_int16_t ndpi_match_content_subprotocol(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow,
char *string_to_match,
u_int string_to_match_len,
ndpi_protocol_match_result *ret_match,
u_int16_t master_protocol_id);
/**
* Exclude protocol from search
*
@@ -601,6 +564,15 @@ extern "C" {
const char* ndpi_category_get_name(struct ndpi_detection_module_struct *ndpi_mod,
ndpi_protocol_category_t category);
/**
* Get classification confidence as string
*
* @par confidence = the confidence value
* @return the string name of the confidence result
*
*/
const char* ndpi_confidence_get_name(ndpi_confidence_t confidence);
/**
* Set protocol category string
*
@@ -682,12 +654,19 @@ extern "C" {
*/
void ndpi_dump_protocols(struct ndpi_detection_module_struct *mod);
/**
* Generate Options list used in OPNsense firewall plugin
*
* @par opt = The Option list to generate
*/
void ndpi_generate_options(u_int opt);
/**
* Write the list of the scores and their associated risks
*
* @par ndpi_mod = the detection module
*/
void ndpi_dump_risks_score();
void ndpi_dump_risks_score(void);
/**
* Read a file and load the protocols
@@ -729,6 +708,15 @@ extern "C" {
*/
int ndpi_add_host_risk_mask(struct ndpi_detection_module_struct *ndpi_mod, char *host, ndpi_risk mask);
/**
* Add a trusted certificate issuer DN
*
* @par ndpi_mod = the detection module
* @par dn = the issuer DN as it appears in the certificate (example "CN=813845657003339838, O=Code42, OU=TEST, ST=MN, C=US")
* @return 0 if the rule is loaded correctly; < 0 in case an error is detected
*/
int ndpi_add_trusted_issuer_dn(struct ndpi_detection_module_struct *ndpi_mod, char *dn);
/**
* Read a file and load the categories
*
@@ -958,6 +946,18 @@ extern "C" {
ndpi_protocol_category_t *category,
ndpi_protocol_breed_t *breed);
/**
* Specifies the threshold used to trigger the NDPI_TLS_CERTIFICATE_ABOUT_TO_EXPIRE
* flow risk that by default is set to 30 days
*
* @par ndpi_struct = the struct created for the protocol detection
* @par days = the number of days threshold for emitting the alert
*
*/
void ndpi_set_tls_cert_expire_days(struct ndpi_detection_module_struct *ndpi_str,
u_int8_t days);
/* Utility functions to set ndpi malloc/free/print wrappers */
void set_ndpi_malloc(void* (*__ndpi_malloc)(size_t size));
void set_ndpi_free(void (*__ndpi_free)(void *ptr));
@@ -988,7 +988,7 @@ extern "C" {
/* Return a flow info string (summarized). Does only work for DNS/HTTP/TLS/QUIC. */
const char* ndpi_get_flow_info(struct ndpi_flow_struct const * const flow,
ndpi_protocol const * const l7_protocol);
char* ndpi_ssl_version2str(struct ndpi_flow_struct *flow,
char* ndpi_ssl_version2str(char *buf, int buf_len,
u_int16_t version, u_int8_t *unknown_tls_version);
int ndpi_netbios_name_interpret(u_char *in, u_int in_len, u_char *out, u_int out_len);
void ndpi_patchIPv6Address(char *str);
@@ -1007,7 +1007,7 @@ extern "C" {
int ndpi_flow2json(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow,
u_int8_t ip_version,
u_int8_t l4_protocol, u_int16_t vlan_id,
u_int8_t l4_protocol,
u_int32_t src_v4, u_int32_t dst_v4,
struct ndpi_in6_addr *src_v6, struct ndpi_in6_addr *dst_v6,
u_int16_t src_port, u_int16_t dst_port,
@@ -1054,7 +1054,9 @@ extern "C" {
/* General purpose utilities */
u_int64_t ndpi_htonll(u_int64_t v);
u_int64_t ndpi_ntohll(u_int64_t v);
u_int8_t ndpi_is_valid_protoId(u_int16_t protoId);
u_int8_t ndpi_is_encrypted_proto(struct ndpi_detection_module_struct *ndpi_str, ndpi_protocol proto);
/* DGA */
int ndpi_check_dga_name(struct ndpi_detection_module_struct *ndpi_str,
struct ndpi_flow_struct *flow,
@@ -1091,6 +1093,12 @@ extern "C" {
*/
void ndpi_reset_serializer(ndpi_serializer *serializer);
/**
* Hint to not create the header (used to avoid creaign the header when not used)
* @param serializer The serializer handle
*/
void ndpi_serializer_skip_header(ndpi_serializer *serializer);
/**
* Serialize a 32-bit unsigned int key and a 32-bit unsigned int value
* @param serializer The serializer handle
@@ -1519,12 +1527,14 @@ extern "C" {
int ndpi_ses_init(struct ndpi_ses_struct *ses, double alpha, float significance);
int ndpi_ses_add_value(struct ndpi_ses_struct *ses, const u_int64_t _value, double *forecast, double *confidence_band);
void ndpi_ses_fitting(double *values, u_int32_t num_values, float *ret_alpha);
/* ******************************* */
int ndpi_des_init(struct ndpi_des_struct *des, double alpha, double beta, float significance);
int ndpi_des_add_value(struct ndpi_des_struct *des, const u_int64_t _value, double *forecast, double *confidence_band);
void ndpi_des_fitting(double *values, u_int32_t num_values, float *ret_alpha, float *ret_beta);
/* ******************************* */
int ndpi_jitter_init(struct ndpi_jitter_struct *hw, u_int16_t num_periods);
@@ -1541,8 +1551,10 @@ extern "C" {
u_int8_t ndpi_is_protocol_detected(struct ndpi_detection_module_struct *ndpi_str,
ndpi_protocol proto);
void ndpi_serialize_risk(ndpi_serializer *serializer, struct ndpi_flow_struct *flow);
void ndpi_serialize_risk(ndpi_serializer *serializer, ndpi_risk_enum risk);
void ndpi_serialize_proto(struct ndpi_detection_module_struct *ndpi_struct,
ndpi_serializer *serializer, ndpi_risk_enum risk,
ndpi_protocol l7_protocol);
const char* ndpi_risk2str(ndpi_risk_enum risk);
const char* ndpi_severity2str(ndpi_risk_severity s);
ndpi_risk_info* ndpi_risk2severity(ndpi_risk_enum risk);
@@ -1567,16 +1579,17 @@ extern "C" {
/* ******************************* */
int ndpi_init_bin(struct ndpi_bin *b, enum ndpi_bin_family f, u_int8_t num_bins);
int ndpi_init_bin(struct ndpi_bin *b, enum ndpi_bin_family f, u_int16_t num_bins);
void ndpi_free_bin(struct ndpi_bin *b);
struct ndpi_bin* ndpi_clone_bin(struct ndpi_bin *b);
void ndpi_inc_bin(struct ndpi_bin *b, u_int8_t slot_id, u_int32_t val);
void ndpi_set_bin(struct ndpi_bin *b, u_int8_t slot_id, u_int32_t value);
u_int32_t ndpi_get_bin_value(struct ndpi_bin *b, u_int8_t slot_id);
void ndpi_inc_bin(struct ndpi_bin *b, u_int16_t slot_id, u_int32_t val);
void ndpi_set_bin(struct ndpi_bin *b, u_int16_t slot_id, u_int32_t value);
u_int32_t ndpi_get_bin_value(struct ndpi_bin *b, u_int16_t slot_id);
void ndpi_reset_bin(struct ndpi_bin *b);
void ndpi_normalize_bin(struct ndpi_bin *b);
char* ndpi_print_bin(struct ndpi_bin *b, u_int8_t normalize_first, char *out_buf, u_int out_buf_len);
float ndpi_bin_similarity(struct ndpi_bin *b1, struct ndpi_bin *b2, u_int8_t normalize_first);
float ndpi_bin_similarity(struct ndpi_bin *b1, struct ndpi_bin *b2,
u_int8_t normalize_first, float similarity_max_threshold);
int ndpi_cluster_bins(struct ndpi_bin *bins, u_int16_t num_bins,
u_int8_t num_clusters, u_int16_t *cluster_ids,
struct ndpi_bin *centroids);
@@ -1608,6 +1621,26 @@ extern "C" {
char* ndpi_get_flow_name(struct ndpi_flow_struct *flow);
/* ******************************* */
ndpi_bitmap* ndpi_bitmap_alloc(void);
void ndpi_bitmap_free(ndpi_bitmap* b);
u_int64_t ndpi_bitmap_cardinality(ndpi_bitmap* b);
void ndpi_bitmap_set(ndpi_bitmap* b, u_int32_t value);
void ndpi_bitmap_unset(ndpi_bitmap* b, u_int32_t value);
bool ndpi_bitmap_isset(ndpi_bitmap* b, u_int32_t value);
void ndpi_bitmap_clear(ndpi_bitmap* b);
size_t ndpi_bitmap_serialize(ndpi_bitmap* b, char **buf);
ndpi_bitmap* ndpi_bitmap_deserialize(char *buf);
void ndpi_bitmap_and(ndpi_bitmap* a, ndpi_bitmap* b_and);
void ndpi_bitmap_or(ndpi_bitmap* a, ndpi_bitmap* b_or);
ndpi_bitmap_iterator* ndpi_bitmap_iterator_alloc(ndpi_bitmap* b);
void ndpi_bitmap_iterator_free(ndpi_bitmap* b);
bool ndpi_bitmap_iterator_next(ndpi_bitmap_iterator* i, uint32_t *value);
/* ******************************* */
#ifdef __cplusplus
}
+93 -11
View File
@@ -27,8 +27,8 @@
gcc -E -dM - < /dev/null |grep ENDIAN
*/
#if defined(__FreeBSD__) || defined(__NetBSD__) || defined(__OpenBSD__)
#include <machine/endian.h>
#if defined(__FreeBSD__) || defined(__NetBSD__)
#include <sys/endian.h>
#endif
#ifdef __OpenBSD__
@@ -156,6 +156,7 @@
/* misc definitions */
#define NDPI_DEFAULT_MAX_TCP_RETRANSMISSION_WINDOW_SIZE 0x10000
#define NDPI_MAX_NUM_PKTS_PER_FLOW_TO_DISSECT 32
/* TODO: rebuild all memory areas to have a more aligned memory block here */
@@ -171,20 +172,11 @@
#define NDPI_GNUTELLA_CONNECTION_TIMEOUT 60
#define NDPI_BATTLEFIELD_CONNECTION_TIMEOUT 60
#define NDPI_THUNDER_CONNECTION_TIMEOUT 30
#define NDPI_RTSP_CONNECTION_TIMEOUT 5
#define NDPI_TVANTS_CONNECTION_TIMEOUT 5
#define NDPI_YAHOO_DETECT_HTTP_CONNECTIONS 1
#define NDPI_YAHOO_LAN_VIDEO_TIMEOUT 30
#define NDPI_ZATTOO_CONNECTION_TIMEOUT 120
#define NDPI_ZATTOO_FLASH_TIMEOUT 5
#define NDPI_JABBER_STUN_TIMEOUT 30
#define NDPI_JABBER_FT_TIMEOUT 5
#define NDPI_SOULSEEK_CONNECTION_IP_TICK_TIMEOUT 600
#ifndef _NDPI_CONFIG_H_
#include "ndpi_config.h" /* To have access to NDPI_ENABLE_DEBUG_MESSAGES */
#define _NDPI_CONFIG_H_
#endif
#ifdef NDPI_ENABLE_DEBUG_MESSAGES
#define NDPI_LOG(proto, m, log_level, args...) \
@@ -321,7 +313,16 @@
#define get_u_int8_t(X,O) (*(u_int8_t *)((&(((u_int8_t *)X)[O]))))
#define get_u_int16_t(X,O) (*(u_int16_t *)((&(((u_int8_t *)X)[O]))))
#define get_u_int32_t(X,O) (*(u_int32_t *)((&(((u_int8_t *)X)[O]))))
#if defined(__arm__)
static inline u_int64_t get_u_int64_t(const u_int8_t* X, int O)
{
u_int64_t tmp;
memcpy(&tmp, X + O, sizeof(tmp));
return tmp;
}
#else
#define get_u_int64_t(X,O) (*(u_int64_t *)((&(((u_int8_t *)X)[O]))))
#endif // __arm__
/* new definitions to get little endian from network bytes */
#define get_ul8(X,O) get_u_int8_t(X,O)
@@ -345,6 +346,19 @@
#define snprintf _snprintf
#endif
#if defined(WIN32)
#undef strtok_r
#define strtok_r strtok_s
#if BYTE_ORDER == LITTLE_ENDIAN
#define le16toh(x) (x)
#define le32toh(x) (x)
#else
#error "byte order not supported"
#endif
#endif /* WIN32 */
#define NDPI_MAX_DNS_REQUESTS 16
#define NDPI_MIN_NUM_STUN_DETECTION 8
@@ -382,4 +396,72 @@
#endif /* __APPLE__ */
#if defined(__MINGW32__)
#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
#define htobe16(x) htons(x)
#define htole16(x) (x)
#define be16toh(x) ntohs(x)
#define le16toh(x) (x)
#define htobe32(x) htonl(x)
#define htole32(x) (x)
#define be32toh(x) ntohl(x)
#define le32toh(x) (x)
#define htobe64(x) htonll(x)
#define htole64(x) (x)
#define be64toh(x) ntohll(x)
#define le64toh(x) (x)
#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
#define htobe16(x) (x)
#define htole16(x) __builtin_bswap16(x)
#define be16toh(x) (x)
#define le16toh(x) __builtin_bswap16(x)
#define htobe32(x) (x)
#define htole32(x) __builtin_bswap32(x)
#define be32toh(x) (x)
#define le32toh(x) __builtin_bswap32(x)
#define htobe64(x) (x)
#define htole64(x) __builtin_bswap64(x)
#define be64toh(x) (x)
#define le64toh(x) __builtin_bswap64(x)
#else
#error Unexpected __BYTE_ORDER__
#endif /* __BYTE_ORDER__ */
#endif /* __MINGW32__ */
#ifndef ETH_ARP
#define ETH_ARP 0x0806
#endif
#ifndef ETH_P_IP
#define ETH_P_IP 0x0800 /* IPv4 */
#endif
#ifndef ETH_P_IPV6
#define ETH_P_IPV6 0x86dd /* IPv6 */
#endif
#ifndef ETH_P_VLAN
#define ETH_P_VLAN 0x8100
#endif
#ifndef ETH_P_MPLS_UNI
#define ETH_P_MPLS_UNI 0x8847
#endif
#ifndef ETH_P_MPLS_MULTI
#define ETH_P_MPLS_MULTI 0x8848
#endif
#ifndef ETH_P_PPPoE
#define ETH_P_PPPoE 0x8864
#endif
#endif /* __NDPI_DEFINE_INCLUDE_FILE__ */
+1 -1
View File
@@ -1,7 +1,7 @@
/*
* ndpi_encryption.h
*
* Copyright (C) 2016-21 - ntop.org
* Copyright (C) 2016-22 - ntop.org
*
* nDPI is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published by
+1 -5
View File
@@ -32,13 +32,13 @@
#include <ctype.h>
#include <time.h>
#include <limits.h>
#include <stdbool.h>
#ifdef WIN32
#include "ndpi_win32.h"
#else
#include <sys/types.h>
#include <sys/param.h>
#include <pthread.h>
#include <arpa/inet.h>
#include <sys/time.h>
#include <sys/socket.h>
@@ -57,10 +57,6 @@
#if defined __NetBSD__ || defined __OpenBSD__
#include <netinet/in_systm.h>
#if defined __OpenBSD__
#include <pthread.h>
#endif
#endif
#endif
#endif
+13 -14
View File
@@ -1,7 +1,7 @@
/*
* ndpi_main.h
*
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
@@ -77,7 +77,8 @@ extern "C" {
void ndpi_set_detected_protocol(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow,
u_int16_t upper_detected_protocol,
u_int16_t lower_detected_protocol);
u_int16_t lower_detected_protocol,
ndpi_confidence_t confidence);
extern void ndpi_parse_packet_line_info(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow);
@@ -87,14 +88,6 @@ extern "C" {
extern u_int16_t ndpi_check_for_email_address(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow, u_int16_t counter);
extern void ndpi_int_change_packet_protocol(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow,
u_int16_t upper_detected_protocol,
u_int16_t lower_detected_protocol);
extern void ndpi_int_change_protocol(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow,
u_int16_t upper_detected_protocol,
u_int16_t lower_detected_protocol);
extern void ndpi_int_change_category(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow,
ndpi_protocol_category_t protocol_category);
@@ -103,12 +96,12 @@ extern "C" {
int protoId, ...);
extern void ndpi_set_proto_defaults(struct ndpi_detection_module_struct *ndpi_mod,
u_int8_t is_cleartext,
ndpi_protocol_breed_t protoBreed, u_int16_t protoId, char *protoName,
ndpi_protocol_category_t protoCategory,
ndpi_port_range *tcpDefPorts,
ndpi_port_range *udpDefPorts);
extern void ndpi_int_reset_packet_protocol(struct ndpi_packet_struct *packet);
extern void ndpi_int_reset_protocol(struct ndpi_flow_struct *flow);
extern int ndpi_packet_src_ip_eql(const struct ndpi_packet_struct *packet, const ndpi_ip_addr_t * ip);
@@ -148,16 +141,22 @@ extern "C" {
#define ndpi_match_strprefix(payload, payload_len, str) \
ndpi_match_prefix((payload), (payload_len), (str), (sizeof(str)-1))
int ndpi_handle_ipv6_extension_headers(struct ndpi_detection_module_struct *ndpi_str,
int ndpi_handle_ipv6_extension_headers(u_int16_t l3len,
const u_int8_t ** l4ptr, u_int16_t * l4len,
u_int8_t * nxt_hdr);
void ndpi_set_risk(struct ndpi_detection_module_struct *ndpi_str,
struct ndpi_flow_struct *flow, ndpi_risk_enum r);
int ndpi_is_printable_string(char const * const str, size_t len);
float ndpi_calculate_entropy(u_int8_t const * const buf, size_t len);
int ndpi_isset_risk(struct ndpi_detection_module_struct *ndpi_str,
struct ndpi_flow_struct *flow, ndpi_risk_enum r);
int ndpi_is_printable_string(char * const str, size_t len);
#define NDPI_ENTROPY_ENCRYPTED_OR_RANDOM(entropy) (entropy > 7.0f)
float ndpi_entropy(u_int8_t const * const buf, size_t len);
void load_common_alpns(struct ndpi_detection_module_struct *ndpi_str);
u_int8_t is_a_common_alpn(struct ndpi_detection_module_struct *ndpi_str,
const char *alpn_to_check, u_int alpn_to_check_len);
char *ndpi_hostname_sni_set(struct ndpi_flow_struct *flow, const u_int8_t *value, size_t value_len);
#ifdef __cplusplus
}
#endif
+4 -4
View File
@@ -1,7 +1,7 @@
/*
* ndpi_patricia_typedef.h
*
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
@@ -65,9 +65,7 @@
#define _NDPI_PATRICIA_TYPEDEF_H_
/* pointer to usr data (ex. route flap info) */
union ndpi_patricia_node_value_t {
void *user_data;
union ndpi_patricia_node_value_t {
/* User-defined values */
union {
struct {
@@ -75,6 +73,8 @@ union ndpi_patricia_node_value_t {
} uv32;
u_int64_t uv64;
void *user_data;
} u;
};
+21 -3
View File
@@ -1,7 +1,7 @@
/*
* ndpi_protocol_ids.h
*
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
@@ -91,8 +91,8 @@ typedef enum {
NDPI_PROTOCOL_MONGODB = 60, /* Leonn Paiva <leonn.paiva@gmail.com> */
NDPI_PROTOCOL_QQLIVE = 61,
NDPI_PROTOCOL_THUNDER = 62,
NDPI_PROTOCOL_SOULSEEK = 63,
NDPI_PROTOCOL_PS_VUE = 64,
NDPI_PROTOCOL_OCSP = 63,
NDPI_PROTOCOL_FREE_64 = 64, /* FREE */
NDPI_PROTOCOL_IRC = 65,
NDPI_PROTOCOL_AYIYA = 66,
NDPI_PROTOCOL_JABBER = 67,
@@ -292,6 +292,24 @@ typedef enum {
NDPI_PROTOCOL_LIKEE = 261,
NDPI_PROTOCOL_GITLAB = 262,
NDPI_PROTOCOL_AVAST_SECUREDNS = 263, /* Toni Uhlig <matzeton@googlemail.com> */
NDPI_PROTOCOL_CASSANDRA = 264, /* Lucas Santos <lfneiva.santos@gmail.com> */
NDPI_PROTOCOL_AMAZON_AWS = 265,
NDPI_PROTOCOL_SALESFORCE = 266,
NDPI_PROTOCOL_VIMEO = 267,
NDPI_PROTOCOL_FACEBOOK_VOIP = 268,
NDPI_PROTOCOL_SIGNAL_VOIP = 269,
NDPI_PROTOCOL_FUZE = 270,
NDPI_PROTOCOL_GTP_U = 271,
NDPI_PROTOCOL_GTP_C = 272,
NDPI_PROTOCOL_GTP_PRIME = 273,
NDPI_PROTOCOL_ALIBABA = 274,
NDPI_PROTOCOL_CRASHLYSTICS = 275,
NDPI_PROTOCOL_MICROSOFT_AZURE = 276,
NDPI_PROTOCOL_ICLOUD_PRIVATE_RELAY = 277,
NDPI_PROTOCOL_ETHERNET_IP = 278,
NDPI_PROTOCOL_BADOO = 279,
NDPI_PROTOCOL_ACCUWEATHER = 280,
NDPI_PROTOCOL_GOOGLE_CLASSROOM = 281,
#ifdef CUSTOM_NDPI_PROTOCOLS
#include "../../../nDPI-custom/custom_ndpi_protocol_ids.h"
+7 -2
View File
@@ -1,7 +1,7 @@
/*
* ndpi_protocols.h
*
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
@@ -60,6 +60,8 @@ void init_ayiya_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_in
void init_amqp_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask);
void init_bgp_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask);
void init_bittorrent_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask);
int ndpi_search_into_bittorrent_cache(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow,
u_int32_t saddr, u_int16_t sport, u_int32_t daddr, u_int16_t dport);
void init_lisp_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask);
void init_teredo_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask);
void init_ciscovpn_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask);
@@ -142,7 +144,6 @@ void init_smb_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int3
void init_snmp_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask);
void init_socrates_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask);
void init_sopcast_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask);
void init_soulseek_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask);
void init_socks_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask);
void init_spotify_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask);
void init_ssh_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask);
@@ -216,5 +217,9 @@ void init_hpvirtgrp_dissector(struct ndpi_detection_module_struct *ndpi_struct,
void init_genshin_impact_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask);
void init_z3950_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask);
void init_avast_securedns_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask);
void init_cassandra_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask);
/* ndpi_main.c */
extern u_int32_t ndpi_ip_port_hash_funct(u_int32_t ip, u_int16_t port);
#endif /* __NDPI_PROTOCOLS_H__ */
+153 -295
View File
@@ -1,7 +1,7 @@
/*
* ndpi_typedefs.h
*
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
@@ -31,10 +31,6 @@
/* Used by both nDPI core and patricia code under third-party */
#include "ndpi_patricia_typedefs.h"
#ifdef HAVE_MAXMINDDB
#include <maxminddb.h>
#endif
/* NDPI_LOG_LEVEL */
typedef enum {
NDPI_LOG_ERROR,
@@ -64,9 +60,18 @@ typedef enum {
When the typedef below is modified don't forget to update
- nDPI/wireshark/ndpi.lua
- ndpi_risk2str (in ndpi_utils.c)
- https://github.com/ntop/ntopng/blob/dev/scripts/lua/modules/flow_risk_utils.lua
- doc/flow_risks.rst
- ndpi_risk_enum (in python/ndpi.py)
- ndpi_known_risks (ndpi_utils.c)
- ndpi_known_risks (ndpi_main.c)
To make sure the risk is also seen by ntopng:
1. Add a new flow alert key to the enum FlowAlertTypeEnum in include/ntop_typedefs.h
2. Add the very same flow alert key to the table flow_alert_keys in scripts/lua/modules/alert_keys/flow_alert_keys.lua
3. Add the risk to the array risk_enum_to_alert_type in src/FlowRiskAlerts.cpp
- To initialize .alert_type use the flow alert key added in 1. and an AlertCategory
- To initialize .alert_lua_name use a unique string
Example: https://github.com/ntop/ntopng/commit/aecc1e3e6505a0522439dbb2b295a3703d3d0f9a
*/
typedef enum {
NDPI_NO_RISK = 0,
@@ -102,7 +107,15 @@ typedef enum {
NDPI_DESKTOP_OR_FILE_SHARING_SESSION, /* 30 */
NDPI_TLS_UNCOMMON_ALPN,
NDPI_TLS_CERT_VALIDITY_TOO_LONG,
NDPI_TLS_EXTENSION_SUSPICIOUS,
NDPI_TLS_SUSPICIOUS_EXTENSION,
NDPI_TLS_FATAL_ALERT,
NDPI_SUSPICIOUS_ENTROPY,
NDPI_CLEAR_TEXT_CREDENTIALS,
NDPI_DNS_LARGE_PACKET,
NDPI_DNS_FRAGMENTED,
NDPI_INVALID_CHARACTERS,
NDPI_POSSIBLE_EXPLOIT, /* Log4J and other exploits */
NDPI_TLS_CERTIFICATE_ABOUT_TO_EXPIRE,
/* Leave this as last member */
NDPI_MAX_RISK /* must be <= 63 due to (**) */
@@ -138,7 +151,6 @@ typedef struct {
risk_percentage default_client_risk_pctg; /* 0-100 */
} ndpi_risk_info;
/* NDPI_VISIT */
typedef enum {
ndpi_preorder,
@@ -478,50 +490,10 @@ struct ndpi_vxlanhdr {
typedef struct message {
u_int8_t *buffer;
u_int buffer_len, buffer_used, max_expected;
u_int buffer_len, buffer_used;
u_int32_t next_seq[2]; /* Directions */
} message_t;
/* NDPI_PROTOCOL_BITTORRENT */
typedef struct spinlock {
volatile int val;
} spinlock_t;
typedef struct atomic {
volatile int counter;
} atomic_t;
struct hash_ip4p_node {
struct hash_ip4p_node *next,*prev;
time_t lchg;
u_int16_t port,count:12,flag:4;
u_int32_t ip;
// + 12 bytes for ipv6
};
struct hash_ip4p {
struct hash_ip4p_node *top;
spinlock_t lock;
size_t len;
};
struct hash_ip4p_table {
size_t size;
int ipv6;
spinlock_t lock;
atomic_t count;
struct hash_ip4p tbl;
};
struct bt_announce { // 192 bytes
u_int32_t hash[5];
u_int32_t ip[4];
u_int32_t time;
u_int16_t port;
u_int8_t name_len,
name[192 - 4*10 - 2 - 1]; // 149 bytes
};
/* NDPI_PROTOCOL_TINC */
#define TINC_CACHE_MAX_SIZE 10
@@ -558,112 +530,31 @@ struct ndpi_lru_cache {
struct ndpi_lru_cache_entry *entries;
};
struct ndpi_id_struct {
/**
detected_protocol_bitmask:
access this bitmask to find out whether an id has used skype or not
if a flag is set here, it will not be reset
to compare this, use:
**/
NDPI_PROTOCOL_BITMASK detected_protocol_bitmask;
/* NDPI_PROTOCOL_RTSP */
ndpi_ip_addr_t rtsp_ip_address;
/* NDPI_PROTOCOL_YAHOO */
u_int32_t yahoo_video_lan_timer;
/* NDPI_PROTOCOL_IRC_MAXPORT % 2 must be 0 */
/* NDPI_PROTOCOL_IRC */
#define NDPI_PROTOCOL_IRC_MAXPORT 8
u_int16_t irc_port[NDPI_PROTOCOL_IRC_MAXPORT];
u_int32_t last_time_port_used[NDPI_PROTOCOL_IRC_MAXPORT];
u_int32_t irc_ts;
/* NDPI_PROTOCOL_GNUTELLA */
u_int32_t gnutella_ts;
/* NDPI_PROTOCOL_THUNDER */
u_int32_t thunder_ts;
/* NDPI_PROTOCOL_RTSP */
u_int32_t rtsp_timer;
/* NDPI_PROTOCOL_ZATTOO */
u_int32_t zattoo_ts;
/* NDPI_PROTOCOL_JABBER */
u_int32_t jabber_stun_or_ft_ts;
/* NDPI_PROTOCOL_DIRECTCONNECT */
u_int32_t directconnect_last_safe_access_time;
/* NDPI_PROTOCOL_SOULSEEK */
u_int32_t soulseek_last_safe_access_time;
/* NDPI_PROTOCOL_DIRECTCONNECT */
u_int16_t detected_directconnect_port;
u_int16_t detected_directconnect_udp_port;
u_int16_t detected_directconnect_ssl_port;
/* NDPI_PROTOCOL_BITTORRENT */
#define NDPI_BT_PORTS 8
u_int16_t bt_port_t[NDPI_BT_PORTS];
u_int16_t bt_port_u[NDPI_BT_PORTS];
/* NDPI_PROTOCOL_JABBER */
#define JABBER_MAX_STUN_PORTS 6
u_int16_t jabber_voice_stun_port[JABBER_MAX_STUN_PORTS];
u_int16_t jabber_file_transfer_port[2];
/* NDPI_PROTOCOL_GNUTELLA */
u_int16_t detected_gnutella_port;
/* NDPI_PROTOCOL_GNUTELLA */
u_int16_t detected_gnutella_udp_port1;
u_int16_t detected_gnutella_udp_port2;
/* NDPI_PROTOCOL_SOULSEEK */
u_int16_t soulseek_listen_port;
/* NDPI_PROTOCOL_IRC */
u_int8_t irc_number_of_port;
/* NDPI_PROTOCOL_JABBER */
u_int8_t jabber_voice_stun_used_ports;
/* NDPI_PROTOCOL_SIP */
/* NDPI_PROTOCOL_YAHOO */
u_int32_t yahoo_video_lan_dir:1;
/* NDPI_PROTOCOL_YAHOO */
u_int32_t yahoo_conf_logged_in:1;
u_int32_t yahoo_voice_conf_logged_in:1;
/* NDPI_PROTOCOL_RTSP */
u_int32_t rtsp_ts_set:1;
};
/* ************************************************** */
struct ndpi_flow_tcp_struct {
/* NDPI_PROTOCOL_MAIL_SMTP */
/* NDPI_PROTOCOL_MAIL_POP */
/* NDPI_PROTOCOL_MAIL_IMAP */
/* NDPI_PROTOCOL_MAIL_FTP */
/* TODO: something clever to save memory */
struct {
u_int8_t auth_found:1, auth_failed:1, auth_tls:1, auth_done:1, _pad:4;
char username[32], password[16];
} ftp_imap_pop_smtp;
/* NDPI_PROTOCOL_MAIL_SMTP */
u_int16_t smtp_command_bitmask;
/* NDPI_PROTOCOL_MAIL_POP */
u_int16_t pop_command_bitmask;
/* NDPI_PROTOCOL_QQ */
u_int16_t qq_nxt_len;
/* NDPI_PROTOCOL_WHATSAPP */
u_int8_t wa_matched_so_far;
/* NDPI_PROTOCOL_TDS */
u_int8_t tds_login_version;
/* NDPI_PROTOCOL_IRC */
u_int8_t irc_stage;
u_int8_t irc_port;
/* NDPI_PROTOCOL_H323 */
u_int8_t h323_valid_packets;
@@ -677,38 +568,16 @@ struct ndpi_flow_tcp_struct {
u_int32_t irc_direction:2;
u_int32_t irc_0x1000_full:1;
/* NDPI_PROTOCOL_SOULSEEK */
u_int32_t soulseek_stage:2;
/* NDPI_PROTOCOL_TDS */
u_int32_t tds_stage:3;
/* NDPI_PROTOCOL_USENET */
u_int32_t usenet_stage:2;
/* NDPI_PROTOCOL_IMESH */
u_int32_t imesh_stage:4;
/* NDPI_PROTOCOL_HTTP */
u_int32_t http_setup_dir:2;
u_int32_t http_stage:2;
u_int32_t http_empty_line_seen:1;
u_int32_t http_wait_for_retransmission:1;
/* NDPI_PROTOCOL_GNUTELLA */
u_int32_t gnutella_stage:2; // 0 - 2
/* NDPI_CONTENT_MMS */
u_int32_t mms_stage:2;
/* NDPI_PROTOCOL_YAHOO */
u_int32_t yahoo_sip_comm:1;
u_int32_t yahoo_http_proxy_stage:2;
/* NDPI_PROTOCOL_MSN */
u_int32_t msn_stage:3;
u_int32_t msn_ssl_ft:2;
/* NDPI_PROTOCOL_SSH */
u_int32_t ssh_stage:3;
@@ -721,11 +590,8 @@ struct ndpi_flow_tcp_struct {
struct {
message_t message;
void* srv_cert_fingerprint_ctx; /* SHA-1 */
/* NDPI_PROTOCOL_TLS */
u_int8_t hello_processed:1, certificate_processed:1, subprotocol_detected:1,
fingerprint_set:1, _pad:4;
u_int8_t certificate_processed:1, fingerprint_set:1, _pad:6;
u_int8_t num_tls_blocks;
int16_t tls_application_blocks_len[NDPI_MAX_NUM_TLS_APPL_BLOCKS]; /* + = src->dst, - = dst->src */
} tls;
@@ -733,8 +599,7 @@ struct ndpi_flow_tcp_struct {
/* NDPI_PROTOCOL_POSTGRES */
u_int32_t postgres_stage:3;
/* NDPI_PROTOCOL_DIRECT_DOWNLOAD_LINK */
u_int32_t ddlink_server_direction:1;
/* Part of the TCP header. */
u_int32_t seen_syn:1;
u_int32_t seen_syn_ack:1;
u_int32_t seen_ack:1;
@@ -751,9 +616,6 @@ struct ndpi_flow_tcp_struct {
/* NDPI_PROTOCOL_WORLDOFWARCRAFT */
u_int32_t wow_stage:2;
/* NDPI_PROTOCOL_HTTP_APPLICATION_VEOHTV */
u_int32_t veoh_tv_stage:2;
/* NDPI_PROTOCOL_SHOUTCAST */
u_int32_t shoutcast_stage:2;
@@ -798,12 +660,6 @@ struct ndpi_flow_tcp_struct {
/* ************************************************** */
struct ndpi_flow_udp_struct {
/* NDPI_PROTOCOL_SNMP */
u_int32_t snmp_msg_id;
/* NDPI_PROTOCOL_SNMP */
u_int32_t snmp_stage:2;
/* NDPI_PROTOCOL_PPSTREAM */
u_int32_t ppstream_stage:3; // 0 - 7
@@ -819,9 +675,6 @@ struct ndpi_flow_udp_struct {
/* NDPI_PROTOCOL_XBOX */
u_int32_t xbox_stage:1;
/* NDPI_PROTOCOL_WINDOWS_UPDATE */
u_int32_t wsus_stage:1;
/* NDPI_PROTOCOL_SKYPE */
u_int8_t skype_packet_id;
u_int8_t skype_crc[4];
@@ -847,6 +700,16 @@ struct ndpi_flow_udp_struct {
/* NDPI_PROTOCOL_QUIC */
u_int8_t *quic_reasm_buf;
u_int32_t quic_reasm_buf_len;
/* NDPI_PROTOCOL_CSGO */
u_int8_t csgo_strid[18],csgo_state,csgo_s2;
u_int32_t csgo_id2;
/* NDPI_PROTOCOL_RDP */
u_int8_t rdp_to_srv[3], rdp_from_srv[3], rdp_to_srv_pkts, rdp_from_srv_pkts;
/* NDPI_PROTOCOL_IMO */
u_int8_t imo_last_one_byte_pkt, imo_last_byte;
};
/* ************************************************** */
@@ -884,9 +747,6 @@ struct ndpi_packet_struct {
u_int64_t current_time_ms;
u_int16_t detected_protocol_stack[NDPI_PROTOCOL_SIZE];
u_int16_t protocol_stack_info;
struct ndpi_int_one_line_struct line[NDPI_MAX_PARSE_LINES_PER_PACKET];
/* HTTP headers */
struct ndpi_int_one_line_struct host_line;
@@ -895,6 +755,7 @@ struct ndpi_packet_struct {
struct ndpi_int_one_line_struct content_line;
struct ndpi_int_one_line_struct content_disposition_line;
struct ndpi_int_one_line_struct accept_line;
struct ndpi_int_one_line_struct authorization_line;
struct ndpi_int_one_line_struct user_agent_line;
struct ndpi_int_one_line_struct http_url_name;
struct ndpi_int_one_line_struct http_encoding;
@@ -910,17 +771,11 @@ struct ndpi_packet_struct {
u_int8_t http_num_headers; /* number of found (valid) header lines in HTTP request or response */
u_int16_t l3_packet_len;
u_int16_t l4_packet_len;
u_int16_t payload_packet_len;
u_int16_t actual_payload_len;
u_int16_t num_retried_bytes;
u_int16_t parsed_lines;
u_int16_t parsed_unix_lines;
u_int16_t empty_line_position;
u_int8_t tcp_retransmission;
u_int8_t l4_protocol;
u_int8_t tls_certificate_detected:4, tls_certificate_num_checks:4;
u_int8_t packet_lines_parsed_complete:1,
packet_direction:1, empty_line_position_set:1, http_check_content:1, pad:4;
};
@@ -945,6 +800,27 @@ typedef struct {
u_int16_t port_low, port_high;
} ndpi_port_range;
typedef enum {
NDPI_CONFIDENCE_UNKNOWN = 0, /* Unknown classification */
NDPI_CONFIDENCE_MATCH_BY_PORT, /* Classification obtained looking only at the L4 ports */
NDPI_CONFIDENCE_MATCH_BY_IP, /* Classification obtained looking only at the L3 addresses */
NDPI_CONFIDENCE_DPI_CACHE, /* Classification results based on same LRU cache (i.e. correlation among sessions) */
NDPI_CONFIDENCE_DPI, /* Deep packet inspection */
/*
IMPORTANT
Please keep in sync with
ndpi_confidence_get_name()
in ndpi_main.c
*/
/* Last one */
NDPI_CONFIDENCE_MAX,
} ndpi_confidence_t;
typedef enum {
NDPI_PROTOCOL_SAFE = 0, /* Surely doesn't provide risks for the network. (e.g., a news site) */
NDPI_PROTOCOL_ACCEPTABLE, /* Probably doesn't provide risks, but could be malicious (e.g., Dropbox) */
@@ -1046,7 +922,8 @@ typedef enum {
typedef struct ndpi_proto_defaults {
char *protoName;
ndpi_protocol_category_t protoCategory;
u_int16_t * subprotocols;
u_int8_t isClearTextProto;
u_int16_t *subprotocols;
u_int32_t subprotocol_count;
u_int16_t protoId, protoIdx;
u_int16_t tcp_default_ports[MAX_DEFAULT_PORTS], udp_default_ports[MAX_DEFAULT_PORTS];
@@ -1087,20 +964,16 @@ typedef struct ndpi_proto {
#define _NDPI_CONFIG_H_
#endif
#ifdef HAVE_PCRE
#include <pcre.h>
struct pcre_struct {
pcre *compiled;
pcre_extra *optimized;
};
#endif
typedef enum {
ndpi_stun_cache,
ndpi_hangout_cache
} ndpi_lru_cache_type;
typedef struct ndpi_list_struct {
char *value;
struct ndpi_list_struct *next;
} ndpi_list;
struct ndpi_detection_module_struct {
NDPI_PROTOCOL_BITMASK detection_bitmask;
NDPI_PROTOCOL_BITMASK generic_http_packet_bitmask;
@@ -1109,7 +982,8 @@ struct ndpi_detection_module_struct {
u_int32_t ticks_per_second;
u_int16_t num_tls_blocks_to_follow;
u_int8_t skip_tls_blocks_until_change_cipher:1, enable_ja3_plus:1, _notused:6;
u_int8_t tls_certificate_expire_in_x_days;
#ifdef NDPI_ENABLE_DEBUG_MESSAGES
void *user_data;
#endif
@@ -1157,13 +1031,13 @@ struct ndpi_detection_module_struct {
int ac_automa_finalized;
/* HTTP/DNS/HTTPS/QUIC host matching */
ndpi_automa host_automa, /* Used for DNS/HTTPS */
content_automa, /* Used for HTTP subprotocol_detection */
subprotocol_automa, /* Used for HTTP subprotocol_detection */
risky_domain_automa, tls_cert_subject_automa,
malicious_ja3_automa, malicious_sha1_automa,
host_risk_mask_automa, common_alpns_automa;
/* IMPORTANT: please update ndpi_finalize_initialization() whenever you add a new automa */
ndpi_list *trusted_issuer_dn;
void *ip_risk_mask_ptree;
struct {
@@ -1179,27 +1053,10 @@ struct ndpi_detection_module_struct {
u_int32_t irc_timeout;
/* gnutella parameters */
u_int32_t gnutella_timeout;
/* thunder parameters */
u_int32_t thunder_timeout;
/* SoulSeek parameters */
u_int32_t soulseek_connection_ip_tick_timeout;
/* rtsp parameters */
u_int32_t rtsp_connection_timeout;
/* rstp */
u_int32_t orb_rstp_ts_timeout;
/* yahoo */
u_int8_t yahoo_detect_http_connections;
u_int32_t yahoo_lan_video_timeout;
u_int32_t zattoo_connection_timeout;
u_int32_t jabber_stun_timeout;
u_int32_t jabber_file_transfer_timeout;
u_int8_t ip_version_limit;
/* NDPI_PROTOCOL_BITTORRENT */
struct hash_ip4p_table *bt_ht, *bt6_ht;
/* BT_ANNOUNCE */
struct bt_announce *bt_ann;
int bt_ann_len;
/* NDPI_PROTOCOL_OOKLA */
struct ndpi_lru_cache *ookla_cache;
@@ -1207,6 +1064,12 @@ struct ndpi_detection_module_struct {
/* NDPI_PROTOCOL_TINC */
struct cache *tinc_cache;
/* NDPI_PROTOCOL_BITTORRENT */
struct ndpi_lru_cache *bittorrent_cache;
/* NDPI_PROTOCOL_ZOOM */
struct ndpi_lru_cache *zoom_cache;
/* NDPI_PROTOCOL_STUN and subprotocols */
struct ndpi_lru_cache *stun_cache;
@@ -1229,10 +1092,12 @@ struct ndpi_detection_module_struct {
#include "../../../nDPI-custom/custom_ndpi_typedefs.h"
#endif
#ifdef HAVE_MAXMINDDB
MMDB_s mmdb_city, mmdb_as;
/* GeoIP */
void *mmdb_city, *mmdb_as;
u_int8_t mmdb_city_loaded, mmdb_as_loaded;
#endif
/* Current packet */
struct ndpi_packet_struct packet;
};
#endif /* NDPI_LIB_COMPILATION */
@@ -1260,12 +1125,12 @@ struct tls_heuristics {
*/
struct ndpi_flow_struct {
u_int16_t detected_protocol_stack[NDPI_PROTOCOL_SIZE];
u_int16_t protocol_stack_info;
/* init parameter, internal used to set up timestamp,... */
u_int16_t guessed_protocol_id, guessed_host_protocol_id, guessed_category, guessed_header_category;
u_int8_t l4_proto, protocol_id_already_guessed:1, host_already_guessed:1, fail_with_unknown:1,
init_finished:1, setup_packet_direction:1, packet_direction:1, check_extra_packets:1;
init_finished:1, setup_packet_direction:1, packet_direction:1, check_extra_packets:1, is_ipv6:1;
ndpi_confidence_t confidence; /* ndpi_confidence_t */
/*
if ndpi_struct->direction_detect_disable == 1
@@ -1273,6 +1138,15 @@ struct ndpi_flow_struct {
*/
u_int32_t next_tcp_seq_nr[2];
/* Flow addresses (used mainly for LRU lookups in ndpi_detection_giveup())
and ports. All in *network* byte order
TODO
- IPv6. Note that LRU is ipv4 only, for the time being
*/
u_int32_t saddr, daddr;
u_int16_t sport, dport;
// -----------------------------------------
u_int8_t max_extra_packets_to_check;
@@ -1281,6 +1155,8 @@ struct ndpi_flow_struct {
int (*extra_packets_func) (struct ndpi_detection_module_struct *, struct ndpi_flow_struct *flow);
u_int64_t last_packet_time_ms;
/*
the tcp / udp / other l4 value union
used to reduce the number of bytes for tcp or udp protocol states
@@ -1290,16 +1166,20 @@ struct ndpi_flow_struct {
struct ndpi_flow_udp_struct udp;
} l4;
/* Some protocols calculate the entropy. */
float entropy;
/* Place textual flow info here */
char flow_extra_info[16];
/*
Pointer to src or dst that identifies the
server of this connection
*/
struct ndpi_id_struct *server_id;
/* HTTP host or DNS query */
u_char host_server_name[240];
/* General purpose field used to save mainly hostname/SNI information.
* In details it used for: DNS and NETBIOS name, HTTP and DHCP hostname,
* WHOIS request, TLS/QUIC server name and STUN realm.
*
* Please, think *very* hard before increasing its size!
*/
char host_server_name[80];
u_int8_t initial_binary_bytes[8], initial_binary_bytes_len;
u_int8_t risk_checked:1, ip_risk_mask_evaluated:1, host_risk_mask_evaluated:1, _notused:5;
ndpi_risk risk_mask; /* Stores the flow risk mask for flow peers */
@@ -1314,11 +1194,11 @@ struct ndpi_flow_struct {
*/
struct {
ndpi_http_method method;
char *url, *content_type /* response */, *request_content_type /* e.g. for POST */, *user_agent;
u_int8_t num_request_headers, num_response_headers;
u_int8_t request_version; /* 0=1.0 and 1=1.1. Create an enum for this? */
u_int16_t response_status_code; /* 200, 404, etc. */
u_char detected_os[32]; /* Via HTTP/QUIC User-Agent */
char *url, *content_type /* response */, *request_content_type /* e.g. for POST */, *user_agent;
char *detected_os; /* Via HTTP/QUIC User-Agent */
char *nat_ip; /* Via HTTP X-Forwarded-For */
} http;
/*
@@ -1331,6 +1211,11 @@ struct ndpi_flow_struct {
u_int16_t pktbuf_maxlen, pktbuf_currlen;
} kerberos_buf;
struct {
u_int8_t num_udp_pkts, num_binding_requests;
u_int16_t num_processed_pkts;
} stun;
union {
/* the only fields useful for nDPI and ntopng */
struct {
@@ -1349,48 +1234,35 @@ struct ndpi_flow_struct {
} kerberos;
struct {
struct {
char ssl_version_str[12];
u_int16_t ssl_version, server_names_len;
char client_requested_server_name[256], /* SNI hostname length: RFC 4366 */
*server_names, *alpn, *tls_supported_versions, *issuerDN, *subjectDN;
u_int32_t notBefore, notAfter;
char ja3_client[33], ja3_server[33];
u_int16_t server_cipher;
u_int8_t sha1_certificate_fingerprint[20];
char *server_names, *alpn, *tls_supported_versions, *issuerDN, *subjectDN;
u_int32_t notBefore, notAfter;
char ja3_client[33], ja3_server[33];
u_int16_t server_cipher;
u_int8_t sha1_certificate_fingerprint[20];
u_int8_t hello_processed:1, subprotocol_detected:1, _pad:6;
#ifdef TLS_HANDLE_SIGNATURE_ALGORITMS
/* Under #ifdef to save memory for those who do not need them */
u_int8_t num_tls_signature_algorithms;
u_int16_t client_signature_algorithms[MAX_NUM_TLS_SIGNATURE_ALGORITHMS];
/* Under #ifdef to save memory for those who do not need them */
u_int8_t num_tls_signature_algorithms;
u_int16_t client_signature_algorithms[MAX_NUM_TLS_SIGNATURE_ALGORITHMS];
#endif
struct tls_heuristics browser_heuristics;
struct tls_heuristics browser_heuristics;
struct {
u_int16_t cipher_suite;
char *esni;
} encrypted_sni;
ndpi_cipher_weakness server_unsafe_cipher;
} tls_quic;
u_int16_t ssl_version, server_names_len;
struct {
u_int8_t num_udp_pkts, num_binding_requests;
u_int16_t num_processed_pkts;
} stun;
/* We can have STUN over SSL/TLS thus they need to live together */
} tls_quic_stun;
u_int16_t cipher_suite;
char *esni;
} encrypted_sni;
ndpi_cipher_weakness server_unsafe_cipher;
} tls_quic;
struct {
char client_signature[48], server_signature[48];
char hassh_client[33], hassh_server[33];
} ssh;
struct {
u_int8_t last_one_byte_pkt, last_byte;
} imo;
struct {
u_int8_t username_detected:1, username_found:1,
password_detected:1, password_found:1,
@@ -1403,16 +1275,9 @@ struct ndpi_flow_struct {
char version[32];
} ubntac2;
struct {
/* Via HTTP X-Forwarded-For */
u_char nat_ip[24];
} http;
struct {
u_int8_t auth_found:1, auth_failed:1, _pad:5;
char username[16], password[16];
} ftp_imap_pop_smtp;
/* In TLS.Bittorent flows there is no hash.
Nonetheless, we must pay attention to NOT write to /read from this field
with these flows */
struct {
/* Bittorrent hash */
u_char hash[20];
@@ -1439,6 +1304,7 @@ struct ndpi_flow_struct {
u_int16_t byte_counter[2];
/* NDPI_PROTOCOL_BITTORRENT */
u_int8_t bittorrent_stage; // can be 0 - 255
u_int8_t bt_check_performed : 1;
/* NDPI_PROTOCOL_DIRECTCONNECT */
u_int8_t directconnect_stage:2; // 0 - 1
@@ -1447,17 +1313,11 @@ struct ndpi_flow_struct {
u_int8_t http_detected:1;
/* NDPI_PROTOCOL_RTSP */
u_int8_t rtsprdt_stage:2, rtsp_control_flow:1;
/* NDPI_PROTOCOL_YAHOO */
u_int8_t yahoo_detection_finished:2;
u_int8_t rtsprdt_stage:2;
/* NDPI_PROTOCOL_ZATTOO */
u_int8_t zattoo_stage:3;
/* NDPI_PROTOCOL_QQ */
u_int8_t qq_stage:3;
/* NDPI_PROTOCOL_THUNDER */
u_int8_t thunder_stage:2; // 0 - 3
@@ -1492,24 +1352,16 @@ struct ndpi_flow_struct {
/* NDPI_PROTOCOL_TINC */
u_int8_t tinc_state;
struct tinc_cache_entry tinc_cache_entry;
/* NDPI_PROTOCOL_CSGO */
u_int8_t csgo_strid[18],csgo_state,csgo_s2;
u_int32_t csgo_id2;
/* internal structures to save functions calls */
struct ndpi_packet_struct packet;
struct ndpi_flow_struct *flow;
struct ndpi_id_struct *src;
struct ndpi_id_struct *dst;
};
#define NDPI_PROTOCOL_DEFAULT_LEVEL 0
typedef struct {
char *string_to_match, *proto_name;
u_int16_t protocol_id;
ndpi_protocol_category_t protocol_category;
ndpi_protocol_breed_t protocol_breed;
int level; /* 0 by default */
int level; /* NDPI_PROTOCOL_DEFAULT_LEVEL (0) by default */
} ndpi_protocol_match;
typedef struct {
@@ -1525,7 +1377,7 @@ typedef struct {
typedef struct {
u_int32_t network;
u_int8_t cidr;
u_int8_t value;
u_int16_t value;
} ndpi_network;
typedef u_int32_t ndpi_init_prefs;
@@ -1687,7 +1539,8 @@ enum ndpi_bin_family {
};
struct ndpi_bin {
u_int8_t num_bins, is_empty;
u_int8_t is_empty;
u_int16_t num_bins;
enum ndpi_bin_family family;
union {
@@ -1772,4 +1625,9 @@ typedef int (*ndpi_custom_dga_predict_fctn)(const char* domain, int domain_lengt
/* **************************************** */
typedef void ndpi_bitmap;
typedef void ndpi_bitmap_iterator;
/* **************************************** */
#endif /* __NDPI_TYPEDEFS_H__ */
-3
View File
@@ -28,9 +28,6 @@
#include <netinet/in.h>
#if defined(__NetBSD__) || defined(__OpenBSD__)
#include <netinet/in_systm.h>
#if defined(__OpenBSD__)
#include <pthread.h>
#endif
#endif
#endif
+2
View File
@@ -16,6 +16,8 @@ extern void printRawData(const uint8_t *ptr, size_t len);
//extern uint8_t add_segment_to_buffer( struct ndpi_flow_struct *flow, struct ndpi_tcphdr const * tcph, uint32_t waited);
//extern uint8_t check_for_sequence( struct ndpi_flow_struct *flow, struct ndpi_tcphdr const * tcph);
extern u_int8_t ndpi_ends_with(char *str, char *ends);
/* **************************************** */
/* Can't call libc functions from kernel space, define some stub instead */
+2 -17
View File
@@ -76,25 +76,10 @@ typedef uint u_int32_t;
typedef uint u_int;
typedef unsigned __int64 u_int64_t;
#define pthread_t HANDLE
#define pthread_mutex_t HANDLE
#define pthread_rwlock_t pthread_mutex_t
#define pthread_rwlock_init pthread_mutex_init
#define pthread_rwlock_wrlock pthread_mutex_lock
#define pthread_rwlock_rdlock pthread_mutex_lock
#define pthread_rwlock_unlock pthread_mutex_unlock
#define pthread_rwlock_destroy pthread_mutex_destroy
#define gmtime_r(a, b) memcpy(b, gmtime(a), sizeof(struct tm))
#define gmtime_r(a, b) memcpy(b, gmtime(a), sizeof(struct tm))
#define in_addr_t unsigned long
extern unsigned long waitForNextEvent(unsigned long ulDelay /* ms */);
#define sleep(a /* sec */) waitForNextEvent(1000*a /* ms */)
#define strtok_r strtok_s
#define timegm _mkgmtime
extern struct tm* localtime_r(const time_t* timep, struct tm* r);
#define sleep(a /* sec */) Sleep(1000*a /* ms */)
#endif /* __NDPI_WIN32_H__ */
+9 -8
View File
@@ -5,17 +5,18 @@
# cd src/lib
# make Makefile
#
AR = @AR@
CC = @CC@
RANLIB = @RANLIB@
#
# Installation directories
#
prefix = @prefix@
libdir = ${prefix}/lib
includedir = ${prefix}/include/ndpi
CC = @GNU_PREFIX@@CC@
CFLAGS += -fPIC -DPIC -I../include -Ithird_party/include -DNDPI_LIB_COMPILATION -Wall @CFLAGS@ @CUSTOM_NDPI@
LDFLAGS = @LDFLAGS@ @ADDITIONAL_LIBS@ @LIBS@ -lm
RANLIB = @GNU_PREFIX@ranlib
CFLAGS += -fPIC -DPIC -I../include -Ithird_party/include -DNDPI_LIB_COMPILATION @CFLAGS@ @CUSTOM_NDPI@
LDFLAGS = @LDFLAGS@
LIBS = @ADDITIONAL_LIBS@ @LIBS@
OBJECTS = $(patsubst protocols/%.c, protocols/%.o, $(wildcard protocols/*.c)) $(patsubst third_party/src/%.c, third_party/src/%.o, $(wildcard third_party/src/*.c)) $(patsubst ./%.c, ./%.o, $(wildcard ./*.c))
HEADERS = $(wildcard ../include/*.h)
@@ -47,16 +48,16 @@ all: $(NDPI_LIBS)
ndpi_main.c: ndpi_content_match.c.inc
$(NDPI_LIB_STATIC): $(OBJECTS)
@GNU_PREFIX@ar rc $@ $(OBJECTS)
$(AR) rc $@ $(OBJECTS)
$(RANLIB) $@
$(NDPI_LIB_SHARED): $(OBJECTS)
$(CC) -shared -fPIC $(SONAME_FLAG) -o $@ $(OBJECTS) $(LDFLAGS)
$(CC) -shared -fPIC $(SONAME_FLAG) -o $@ $(LDFLAGS) $(OBJECTS) $(LIBS)
ln -fs $(NDPI_LIB_SHARED) $(NDPI_LIB_SHARED_BASE)
ln -fs $(NDPI_LIB_SHARED) $(NDPI_LIB_SHARED_BASE).$(NDPI_VERSION_MAJOR)
%.o: %.c $(HEADERS) Makefile
$(CC) $(CFLAGS) -c $< -o $@
$(CC) $(CPPFLAGS) $(CFLAGS) -c $< -o $@
clean:
/bin/rm -f $(NDPI_LIB_STATIC) $(OBJECTS) *.o *.so *.lo libndpi.so*
+143 -23
View File
@@ -1,7 +1,7 @@
/*
* ndpi_analyze.c
*
* Copyright (C) 2019-21 - ntop.org
* Copyright (C) 2019-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library.
@@ -299,7 +299,7 @@ double ndpi_hll_count(struct ndpi_hll *hll) {
/* ********************************************************************************* */
/* ********************************************************************************* */
int ndpi_init_bin(struct ndpi_bin *b, enum ndpi_bin_family f, u_int8_t num_bins) {
int ndpi_init_bin(struct ndpi_bin *b, enum ndpi_bin_family f, u_int16_t num_bins) {
b->num_bins = num_bins, b->family = f, b->is_empty = 1;
switch(f) {
@@ -327,13 +327,13 @@ int ndpi_init_bin(struct ndpi_bin *b, enum ndpi_bin_family f, u_int8_t num_bins)
void ndpi_free_bin(struct ndpi_bin *b) {
switch(b->family) {
case ndpi_bin_family8:
free(b->u.bins8);
ndpi_free(b->u.bins8);
break;
case ndpi_bin_family16:
free(b->u.bins16);
ndpi_free(b->u.bins16);
break;
case ndpi_bin_family32:
free(b->u.bins32);
ndpi_free(b->u.bins32);
break;
}
}
@@ -350,7 +350,7 @@ struct ndpi_bin* ndpi_clone_bin(struct ndpi_bin *b) {
switch(out->family) {
case ndpi_bin_family8:
if((out->u.bins8 = (u_int8_t*)ndpi_calloc(out->num_bins, sizeof(u_int8_t))) == NULL) {
free(out);
ndpi_free(out);
return(NULL);
} else
memcpy(out->u.bins8, b->u.bins8, out->num_bins*sizeof(u_int8_t));
@@ -358,7 +358,7 @@ struct ndpi_bin* ndpi_clone_bin(struct ndpi_bin *b) {
case ndpi_bin_family16:
if((out->u.bins16 = (u_int16_t*)ndpi_calloc(out->num_bins, sizeof(u_int16_t))) == NULL) {
free(out);
ndpi_free(out);
return(NULL);
} else
memcpy(out->u.bins16, b->u.bins16, out->num_bins*sizeof(u_int16_t));
@@ -366,7 +366,7 @@ struct ndpi_bin* ndpi_clone_bin(struct ndpi_bin *b) {
case ndpi_bin_family32:
if((out->u.bins32 = (u_int32_t*)ndpi_calloc(out->num_bins, sizeof(u_int32_t))) == NULL) {
free(out);
ndpi_free(out);
return(NULL);
} else
memcpy(out->u.bins32, b->u.bins32, out->num_bins*sizeof(u_int32_t));
@@ -378,7 +378,7 @@ struct ndpi_bin* ndpi_clone_bin(struct ndpi_bin *b) {
/* ********************************************************************************* */
void ndpi_set_bin(struct ndpi_bin *b, u_int8_t slot_id, u_int32_t val) {
void ndpi_set_bin(struct ndpi_bin *b, u_int16_t slot_id, u_int32_t val) {
if(slot_id >= b->num_bins) slot_id = 0;
switch(b->family) {
@@ -396,7 +396,7 @@ void ndpi_set_bin(struct ndpi_bin *b, u_int8_t slot_id, u_int32_t val) {
/* ********************************************************************************* */
void ndpi_inc_bin(struct ndpi_bin *b, u_int8_t slot_id, u_int32_t val) {
void ndpi_inc_bin(struct ndpi_bin *b, u_int16_t slot_id, u_int32_t val) {
b->is_empty = 0;
if(slot_id >= b->num_bins) slot_id = 0;
@@ -416,7 +416,7 @@ void ndpi_inc_bin(struct ndpi_bin *b, u_int8_t slot_id, u_int32_t val) {
/* ********************************************************************************* */
u_int32_t ndpi_get_bin_value(struct ndpi_bin *b, u_int8_t slot_id) {
u_int32_t ndpi_get_bin_value(struct ndpi_bin *b, u_int16_t slot_id) {
if(slot_id >= b->num_bins) slot_id = 0;
switch(b->family) {
@@ -457,7 +457,7 @@ void ndpi_reset_bin(struct ndpi_bin *b) {
Each bin slot is transformed in a % with respect to the value total
*/
void ndpi_normalize_bin(struct ndpi_bin *b) {
u_int8_t i;
u_int16_t i;
u_int32_t tot = 0;
if(b->is_empty) return;
@@ -495,7 +495,7 @@ void ndpi_normalize_bin(struct ndpi_bin *b) {
/* ********************************************************************************* */
char* ndpi_print_bin(struct ndpi_bin *b, u_int8_t normalize_first, char *out_buf, u_int out_buf_len) {
u_int8_t i;
u_int16_t i;
u_int len = 0;
if(!out_buf) return(out_buf); else out_buf[0] = '\0';
@@ -555,10 +555,14 @@ char* ndpi_print_bin(struct ndpi_bin *b, u_int8_t normalize_first, char *out_buf
0 = alike
...
the higher the more different
*/
float ndpi_bin_similarity(struct ndpi_bin *b1, struct ndpi_bin *b2, u_int8_t normalize_first) {
u_int8_t i;
if similarity_max_threshold != 0, we assume that bins arent similar
*/
float ndpi_bin_similarity(struct ndpi_bin *b1, struct ndpi_bin *b2,
u_int8_t normalize_first, float similarity_max_threshold) {
u_int16_t i;
float threshold = similarity_max_threshold*similarity_max_threshold;
if(
// (b1->family != b2->family) ||
(b1->num_bins != b2->num_bins))
@@ -594,7 +598,10 @@ float ndpi_bin_similarity(struct ndpi_bin *b1, struct ndpi_bin *b2, u_int8_t nor
if(a != b) sum += pow(diff, 2);
// printf("[a: %u][b: %u][sum: %u]\n", a, b, sum);
if(threshold && (sum > threshold))
return(-2); /* Sorry they are not similar */
// printf("%u/%u) [a: %u][b: %u][sum: %u]\n", i, b1->num_bins, a, b, sum);
}
/* The lower the more similar */
@@ -720,7 +727,7 @@ int ndpi_cluster_bins(struct ndpi_bin *bins, u_int16_t num_bins,
if(centroids[j].is_empty) continue;
similarity = ndpi_bin_similarity(&bins[i], &centroids[j], 0);
similarity = ndpi_bin_similarity(&bins[i], &centroids[j], 0, 0);
if(j == cluster_ids[i])
current_similarity = similarity;
@@ -840,8 +847,8 @@ int ndpi_alloc_rsi(struct ndpi_rsi_struct *s, u_int16_t num_learning_values) {
s->last_value = 0;
return(0);
} else {
if(s->gains) free(s->gains);
if(s->losses) free(s->losses);
if(s->gains) ndpi_free(s->gains);
if(s->losses) ndpi_free(s->losses);
return(-1);
}
}
@@ -987,7 +994,7 @@ int ndpi_hw_init(struct ndpi_hw_struct *hw,
return(-1);
if((hw->s = (double*)ndpi_calloc(hw->params.num_season_periods, sizeof(double))) == NULL) {
free(hw->y);
ndpi_free(hw->y);
return(-1);
}
@@ -1226,12 +1233,67 @@ int ndpi_ses_add_value(struct ndpi_ses_struct *ses, const u_int64_t _value, doub
#ifdef SES_DEBUG
printf("[num_values: %u][[error: %.3f][forecast: %.3f][sqe: %.3f][sq: %.3f][confidence_band: %.3f]\n",
ses->num_values, error, *forecast, ses->sum_square_error, sq, *confidence_band);
ses->num_values, error, *forecast, ses->sum_square_error, sq_error, *confidence_band);
#endif
return(rc);
}
/* *********************************************************** */
/*
Computes the best alpha value using the specified values used for training
*/
void ndpi_ses_fitting(double *values, u_int32_t num_values, float *ret_alpha) {
u_int i;
float alpha, best_alpha;
double sse, lowest_sse;
int trace = 0;
lowest_sse = 0, best_alpha = 0;
for(alpha=0.1; alpha<0.99; alpha += 0.05) {
struct ndpi_ses_struct ses;
ndpi_ses_init(&ses, alpha, 0.05);
if(trace)
printf("\nDouble Exponential Smoothing [alpha: %.2f]\n", alpha);
sse = 0;
for(i=0; i<num_values; i++) {
double prediction, confidence_band;
double diff;
if(ndpi_ses_add_value(&ses, values[i], &prediction, &confidence_band) != 0) {
diff = fabs(prediction-values[i]);
if(trace)
printf("%2u)\t%12.3f\t%.3f\t%.3f\n", i, values[i], prediction, diff);
sse += diff*diff;
}
}
if(lowest_sse == 0)
lowest_sse = sse, best_alpha = alpha; /* first run */
else {
if(sse <= lowest_sse)
lowest_sse = sse, best_alpha = alpha;
}
if(trace)
printf("[alpha: %.2f] - SSE: %.2f [BEST: alpha: %.2f/SSE: %.2f]\n", alpha, sse,
best_alpha, lowest_sse);
} /* for (alpha) */
if(trace)
printf("BEST [alpha: %.2f][SSE: %.2f]\n", best_alpha, lowest_sse);
*ret_alpha = best_alpha;
}
/* *********************************************************** */
/* *********************************************************** */
@@ -1243,7 +1305,8 @@ int ndpi_des_init(struct ndpi_des_struct *des, double alpha, double beta, float
memset(des, 0, sizeof(struct ndpi_des_struct));
des->params.alpha = alpha;
des->params.beta = beta;
if((significance < 0) || (significance > 1)) significance = 0.05;
des->params.ro = ndpi_normal_cdf_inverse(1 - (significance / 2.));
@@ -1305,3 +1368,60 @@ int ndpi_des_add_value(struct ndpi_des_struct *des, const u_int64_t _value, doub
return(rc);
}
/* *********************************************************** */
/*
Computes the best alpha and beta values using the specified values used for training
*/
void ndpi_des_fitting(double *values, u_int32_t num_values, float *ret_alpha, float *ret_beta) {
u_int i;
float alpha, best_alpha, best_beta, beta = 0;
double sse, lowest_sse;
int trace = 0;
lowest_sse = 0, best_alpha = 0, best_beta = 0;
for(beta=0.1; beta<0.99; beta += 0.05) {
for(alpha=0.1; alpha<0.99; alpha += 0.05) {
struct ndpi_des_struct des;
ndpi_des_init(&des, alpha, beta, 0.05);
if(trace)
printf("\nDouble Exponential Smoothing [alpha: %.2f][beta: %.2f]\n", alpha, beta);
sse = 0;
for(i=0; i<num_values; i++) {
double prediction, confidence_band;
double diff;
if(ndpi_des_add_value(&des, values[i], &prediction, &confidence_band) != 0) {
diff = fabs(prediction-values[i]);
if(trace)
printf("%2u)\t%12.3f\t%.3f\t%.3f\n", i, values[i], prediction, diff);
sse += diff*diff;
}
}
if(lowest_sse == 0)
lowest_sse = sse, best_alpha = alpha, best_beta = beta; /* first run */
else {
if(sse <= lowest_sse)
lowest_sse = sse, best_alpha = alpha, best_beta = beta;
}
if(trace)
printf("[alpha: %.2f][beta: %.2f] - SSE: %.2f [BEST: alpha: %.2f/beta: %.2f/SSE: %.2f]\n", alpha, beta, sse,
best_alpha, best_beta, lowest_sse);
} /* for (alpha) */
} /* for (beta) */
if(trace)
printf("BEST [alpha: %.2f][beta: %.2f][SSE: %.2f]\n", best_alpha, best_beta, lowest_sse);
*ret_alpha = best_alpha, *ret_beta = best_beta;
}
File diff suppressed because it is too large Load Diff
+139
View File
@@ -0,0 +1,139 @@
/*
* ndpi_utils.c
*
* Copyright (C) 2011-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
*
* nDPI is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* nDPI is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public License
* along with nDPI. If not, see <http://www.gnu.org/licenses/>.
*
*/
#include <stdlib.h>
#include <errno.h>
#include <math.h>
#include <sys/types.h>
#define NDPI_CURRENT_PROTO NDPI_PROTOCOL_UNKNOWN
#include "ndpi_config.h"
#include "ndpi_api.h"
#include "ndpi_includes.h"
#include "ndpi_encryption.h"
#include "third_party/include/roaring.h"
#include "third_party/src/roaring.cc"
/* ******************************************* */
ndpi_bitmap* ndpi_bitmap_alloc() {
return((ndpi_bitmap*)roaring_bitmap_create());
}
/* ******************************************* */
void ndpi_bitmap_free(ndpi_bitmap* b) {
roaring_bitmap_free((const roaring_bitmap_t *)b);
}
/* ******************************************* */
u_int64_t ndpi_bitmap_cardinality(ndpi_bitmap* b) {
return(roaring_bitmap_get_cardinality((const roaring_bitmap_t *)b));
}
/* ******************************************* */
void ndpi_bitmap_set(ndpi_bitmap* b, u_int32_t value) {
roaring_bitmap_add((roaring_bitmap_t *)b, value);
}
/* ******************************************* */
void ndpi_bitmap_unset(ndpi_bitmap* b, u_int32_t value) {
roaring_bitmap_remove((roaring_bitmap_t *)b, value);
}
/* ******************************************* */
bool ndpi_bitmap_isset(ndpi_bitmap* b, u_int32_t value) {
return(roaring_bitmap_contains((const roaring_bitmap_t *)b, value));
}
/* ******************************************* */
void ndpi_bitmap_clear(ndpi_bitmap* b) {
roaring_bitmap_clear((roaring_bitmap_t *)b);
}
/* ******************************************* */
size_t ndpi_bitmap_serialize(ndpi_bitmap* b, char **buf) {
const roaring_bitmap_t *r = (const roaring_bitmap_t *)b;
size_t s = roaring_bitmap_size_in_bytes(r);
*buf = (char*)ndpi_malloc(s);
if((*buf) == NULL) return(0);
return(roaring_bitmap_serialize(r, *buf));
}
/* ******************************************* */
ndpi_bitmap* ndpi_bitmap_deserialize(char *buf) {
return((ndpi_bitmap*)roaring_bitmap_deserialize(buf));
}
/* ******************************************* */
/* b = b & b_and */
void ndpi_bitmap_and(ndpi_bitmap* a, ndpi_bitmap* b_and) {
roaring_bitmap_and_inplace((ndpi_bitmap*)a, (ndpi_bitmap*)b_and);
}
/* ******************************************* */
/* b = b | b_or */
void ndpi_bitmap_or(ndpi_bitmap* a, ndpi_bitmap* b_or) {
roaring_bitmap_or_inplace((ndpi_bitmap*)a, (ndpi_bitmap*)b_or);
}
/* ******************************************* */
ndpi_bitmap_iterator* ndpi_bitmap_iterator_alloc(ndpi_bitmap* b) {
return(roaring_create_iterator((ndpi_bitmap*)b));
}
/* ******************************************* */
void ndpi_bitmap_iterator_free(ndpi_bitmap* b) {
return(roaring_free_uint32_iterator((ndpi_bitmap*)b));
}
/* ******************************************* */
/* Return the next value in the bitmap iterator
true is returned when a value is present, false when we reached the end
*/
bool ndpi_bitmap_iterator_next(ndpi_bitmap_iterator* i, uint32_t *value) {
uint32_t num = roaring_read_uint32_iterator((roaring_uint32_iterator_t*)i, value, 1);
return((num == 1) ? true /* found */ : false /* not found */);
}
+6
View File
@@ -688,7 +688,13 @@ ndpi_log_timestamp(char *log_ts, uint32_t log_ts_len)
gettimeofday(&tv, NULL);
nowtime = tv.tv_sec;
#ifdef WIN32
/* localtime() on Windows is thread-safe */
struct tm * nowtm_r_ptr = localtime(&nowtime);
nowtm_r = *nowtm_r_ptr;
#else
localtime_r(&nowtime, &nowtm_r);
#endif
strftime(tmbuf, NDPI_TIMESTAMP_LEN, "%H:%M:%S", &nowtm_r);
snprintf(log_ts, log_ts_len, "%s.%06ld", tmbuf, (long)tv.tv_usec);
}
+3 -3
View File
@@ -1,7 +1,7 @@
/*
* ndpi_community_id.c
*
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
@@ -185,11 +185,11 @@ void ndpi_string_sha1_hash(const uint8_t *message, size_t len, u_char *hash /* 2
/* **************************************************** */
/*
https://github.com/corelight/community-id-spec/blob/bda913f617389df07cdaa23606e11bbd318e265c/community-id.py#L285
https://github.com/corelight/community-id-spec/blob/bda913f617389df07cdaa23606e11bbd318e265c/community-id.py#L285
*/
static int ndpi_community_id_finalize_and_compute_hash(u_int8_t *comm_buf, u_int16_t off, u_int8_t l4_proto,
u_int16_t src_port, u_int16_t dst_port,
char *hash_buf, u_int8_t hash_buf_len) {
char *hash_buf, size_t hash_buf_len) {
u_int8_t pad = 0;
uint32_t hash[5];
char *community_id;
File diff suppressed because it is too large Load Diff
+19 -10
View File
@@ -21,10 +21,6 @@
*
*/
#ifdef HAVE_CONFIG_H
#include "ndpi_config.h"
#endif
#include <stdlib.h>
#include <errno.h>
#include <sys/types.h>
@@ -33,6 +29,10 @@
#include "ndpi_api.h"
#include "ndpi_config.h"
#ifdef HAVE_MAXMINDDB
#include <maxminddb.h>
#endif
/* ********************************************************************************* */
int ndpi_load_geoip(struct ndpi_detection_module_struct *ndpi_str,
@@ -40,13 +40,19 @@ int ndpi_load_geoip(struct ndpi_detection_module_struct *ndpi_str,
#ifdef HAVE_MAXMINDDB
int status;
ndpi_str->mmdb_city = (void*)ndpi_malloc(sizeof(MMDB_s));
ndpi_str->mmdb_as = (void*)ndpi_malloc(sizeof(MMDB_s));
if((ndpi_str->mmdb_city == NULL) || (ndpi_str->mmdb_as == NULL))
return(-1);
/* Open the MMDB files */
if((status = MMDB_open(ip_city_data, MMDB_MODE_MMAP, &ndpi_str->mmdb_city)) != MMDB_SUCCESS)
if((status = MMDB_open(ip_city_data, MMDB_MODE_MMAP, (MMDB_s*)ndpi_str->mmdb_city)) != MMDB_SUCCESS)
return(-1);
else
ndpi_str->mmdb_city_loaded = 1;
if((status = MMDB_open(ip_as_data, MMDB_MODE_MMAP, &ndpi_str->mmdb_as)) != MMDB_SUCCESS)
if((status = MMDB_open(ip_as_data, MMDB_MODE_MMAP, (MMDB_s*)ndpi_str->mmdb_as)) != MMDB_SUCCESS)
return(-2);
else
ndpi_str->mmdb_as_loaded = 1;
@@ -61,8 +67,11 @@ int ndpi_load_geoip(struct ndpi_detection_module_struct *ndpi_str,
void ndpi_free_geoip(struct ndpi_detection_module_struct *ndpi_str) {
#ifdef HAVE_MAXMINDDB
if(ndpi_str->mmdb_city_loaded) MMDB_close(&ndpi_str->mmdb_city);
if(ndpi_str->mmdb_as_loaded) MMDB_close(&ndpi_str->mmdb_as);
if(ndpi_str->mmdb_city_loaded) MMDB_close((MMDB_s*)ndpi_str->mmdb_city);
if(ndpi_str->mmdb_as_loaded) MMDB_close((MMDB_s*)ndpi_str->mmdb_as);
ndpi_free(ndpi_str->mmdb_city);
ndpi_free(ndpi_str->mmdb_as);
#endif
}
@@ -75,7 +84,7 @@ int ndpi_get_geoip_asn(struct ndpi_detection_module_struct *ndpi_str, char *ip,
MMDB_entry_data_s entry_data;
if(ndpi_str->mmdb_as_loaded) {
result = MMDB_lookup_string(&ndpi_str->mmdb_as, ip, &gai_error, &mmdb_error);
result = MMDB_lookup_string((MMDB_s*)ndpi_str->mmdb_as, ip, &gai_error, &mmdb_error);
if((gai_error != 0)
|| (mmdb_error != MMDB_SUCCESS)
@@ -112,7 +121,7 @@ int ndpi_get_geoip_country_continent(struct ndpi_detection_module_struct *ndpi_s
if(ndpi_str->mmdb_city_loaded) {
int status;
result = MMDB_lookup_string(&ndpi_str->mmdb_city, ip, &gai_error, &mmdb_error);
result = MMDB_lookup_string((MMDB_s*)ndpi_str->mmdb_city, ip, &gai_error, &mmdb_error);
if((gai_error != 0)
|| (mmdb_error != MMDB_SUCCESS)
+1265 -953
View File
File diff suppressed because it is too large Load Diff
+227 -111
View File
@@ -1,7 +1,7 @@
/*
* ndpi_serializer.c
*
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* nDPI is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published by
@@ -47,14 +47,17 @@
#define NDPI_U64_FORMAT "%llu"
#endif
#define OPTIMIZE_NUMERIC_KEYS /* Convert numeric string keys into int keys */
#define OPTIMIZE_CSV_SERIALIZATION /* Optimize serialization speed in case of CSV */
/* ********************************** */
u_int64_t ndpi_htonll(u_int64_t v) {
union { u_int32_t lv[2]; u_int64_t llv; } u;
u.lv[0] = htonl(v >> 32);
u.lv[1] = htonl(v & 0xFFFFFFFFULL);
return(u.llv);
}
@@ -62,22 +65,24 @@ u_int64_t ndpi_htonll(u_int64_t v) {
u_int64_t ndpi_ntohll(u_int64_t v) {
union { u_int32_t lv[2]; u_int64_t llv; } u;
u.llv = v;
return((u_int64_t)ntohl(u.lv[0]) << 32) | (u_int64_t)ntohl(u.lv[1]);
}
/* ********************************** */
#ifdef OPTIMIZE_NUMERIC_KEYS
static int ndpi_is_number(const char *str, u_int32_t str_len) {
int i;
unsigned int i;
for(i = 0; i < str_len; i++)
if(!isdigit(str[i])) return(0);
return(1);
}
#endif
/* ********************************** */
@@ -159,7 +164,7 @@ static inline int ndpi_snappend(char *buf, size_t size, const char *fmt, ...) {
else if (wlen >= size)
wlen = size-1;
return wlen;
return(wlen);
}
#endif
@@ -188,12 +193,21 @@ void ndpi_reset_serializer(ndpi_serializer *_serializer) {
/* ********************************** */
void ndpi_serializer_skip_header(ndpi_serializer *_serializer) {
ndpi_private_serializer *serializer = (ndpi_private_serializer*)_serializer;
serializer->status.flags |= NDPI_SERIALIZER_STATUS_HDR_DONE;
}
/* ********************************** */
static int ndpi_init_serializer_buffer(ndpi_private_serializer_buffer *buffer, u_int32_t buffer_size) {
buffer->initial_size = buffer->size = buffer_size;
buffer->data = (u_int8_t *) calloc(buffer->size, sizeof(u_int8_t));
if(buffer->data == NULL)
return -1;
return 0;
return(-1);
return(0);
}
/* ********************************** */
@@ -209,7 +223,7 @@ int ndpi_init_serializer_ll(ndpi_serializer *_serializer,
if (ndpi_init_serializer_buffer(&serializer->buffer, buffer_size) != 0)
return(-1);
if(serializer->fmt == ndpi_serialization_format_json) {
/* nothing to do */
@@ -227,7 +241,7 @@ int ndpi_init_serializer_ll(ndpi_serializer *_serializer,
ndpi_reset_serializer(_serializer);
return(1);
return(0);
}
/* ********************************** */
@@ -272,12 +286,12 @@ static inline int ndpi_serializer_check_header_room(ndpi_private_serializer *ser
u_int32_t buff_diff = serializer->header.size - serializer->status.header.size_used;
if (buff_diff < needed)
if (ndpi_extend_serializer_buffer(&serializer->header, needed - buff_diff) < 0)
return -1;
if (ndpi_extend_serializer_buffer(&serializer->header, needed - buff_diff) < 0)
return(-1);
buff_diff = serializer->header.size - serializer->status.header.size_used;
return buff_diff;
return(buff_diff);
}
/* ********************************** */
@@ -286,17 +300,17 @@ static inline int ndpi_serializer_header_uint32(ndpi_private_serializer *seriali
int room;
if (serializer->status.flags & NDPI_SERIALIZER_STATUS_HDR_DONE)
return 0;
return(0);
room = ndpi_serializer_check_header_room(serializer, 12);
if (room < 0)
return -1;
return(-1);
serializer->status.header.size_used += snprintf((char *) &serializer->header.data[serializer->status.header.size_used],
serializer->status.header.size_used += snprintf((char *) &serializer->header.data[serializer->status.header.size_used],
room, "%s%u", (serializer->status.header.size_used > 0) ? serializer->csv_separator : "", key);
return 0;
return(0);
}
/* ********************************** */
@@ -305,12 +319,12 @@ static inline int ndpi_serializer_header_string(ndpi_private_serializer *seriali
int room;
if (serializer->status.flags & NDPI_SERIALIZER_STATUS_HDR_DONE)
return 0;
return(0);
room = ndpi_serializer_check_header_room(serializer, klen + 4);
if (room < 0)
return -1;
return(-1);
if (serializer->status.header.size_used > 0) {
int slen = strlen(serializer->csv_separator);
@@ -325,7 +339,7 @@ static inline int ndpi_serializer_header_string(ndpi_private_serializer *seriali
serializer->header.data[serializer->status.header.size_used] = '\0';
return 0;
return(0);
}
/* ********************************** */
@@ -385,7 +399,7 @@ char* ndpi_serializer_get_header(ndpi_serializer *_serializer, u_int32_t *buffer
if(buf == NULL) {
*buffer_len = 0;
return "";
return("");
}
/* NULL terminate the buffer */
@@ -401,7 +415,8 @@ char* ndpi_serializer_get_header(ndpi_serializer *_serializer, u_int32_t *buffer
ndpi_serialization_format ndpi_serializer_get_format(ndpi_serializer *_serializer) {
ndpi_private_serializer *serializer = (ndpi_private_serializer*)_serializer;
return serializer->fmt;
return(serializer->fmt);
}
/* ********************************** */
@@ -579,7 +594,7 @@ int ndpi_serialize_raw_record(ndpi_serializer *_serializer,
u_int32_t buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
u_int16_t needed = record_len;
u_int8_t add_comma = 0;
if(serializer->fmt == ndpi_serialization_format_json) {
needed += 1;
@@ -588,21 +603,21 @@ int ndpi_serialize_raw_record(ndpi_serializer *_serializer,
else
needed += 2, add_comma = 1;
}
if(buff_diff < needed) {
if(ndpi_extend_serializer_buffer(&serializer->buffer, needed - buff_diff) < 0)
return(-1);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
}
if(serializer->fmt == ndpi_serialization_format_json) {
if(serializer->fmt == ndpi_serialization_format_json) {
if(add_comma)
serializer->buffer.data[serializer->status.buffer.size_used-1] = ',';
else
serializer->status.buffer.size_used--;
}
memcpy(&serializer->buffer.data[serializer->status.buffer.size_used], record, record_len);
serializer->status.buffer.size_used += record_len;
@@ -610,9 +625,9 @@ int ndpi_serialize_raw_record(ndpi_serializer *_serializer,
serializer->buffer.data[serializer->status.buffer.size_used] = ']';
if(add_comma) serializer->status.buffer.size_used++;
}
ndpi_serialize_end_of_record(_serializer);
return(0);
}
@@ -653,7 +668,7 @@ int ndpi_serialize_end_of_record(ndpi_serializer *_serializer) {
}
serializer->status.flags &= ~NDPI_SERIALIZER_STATUS_NOT_EMPTY;
return(0);
}
@@ -665,10 +680,10 @@ static inline void ndpi_serialize_csv_pre(ndpi_private_serializer *serializer) {
} else if (serializer->status.buffer.size_used == 0) {
/* nothing to do */
} else {
u_int32_t buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used], buff_diff, "%s",
serializer->csv_separator);
if(serializer->buffer.size > serializer->status.buffer.size_used)
serializer->buffer.data[serializer->status.buffer.size_used] = serializer->csv_separator[0];
serializer->status.buffer.size_used++;
}
}
@@ -692,7 +707,7 @@ static inline void ndpi_serialize_json_pre(ndpi_serializer *_serializer) {
serializer->status.buffer.size_used--; /* Remove ']' */
if(serializer->status.flags & NDPI_SERIALIZER_STATUS_SOL)
serializer->status.flags &= ~NDPI_SERIALIZER_STATUS_SOL;
else
else
serializer->buffer.data[serializer->status.buffer.size_used++] = ',';
} else {
if(serializer->status.flags & NDPI_SERIALIZER_STATUS_SOB)
@@ -763,13 +778,13 @@ int ndpi_serialize_uint32_uint32(ndpi_serializer *_serializer,
if (!(serializer->status.flags & NDPI_SERIALIZER_STATUS_LIST)) {
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used],
&serializer->buffer.data[serializer->status.buffer.size_used],
buff_diff, "\"%u\":", key);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
}
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used],
&serializer->buffer.data[serializer->status.buffer.size_used],
buff_diff, "%u", value);
ndpi_serialize_json_post(_serializer);
@@ -830,13 +845,13 @@ int ndpi_serialize_uint32_uint64(ndpi_serializer *_serializer,
ndpi_serialize_json_pre(_serializer);
if (!(serializer->status.flags & NDPI_SERIALIZER_STATUS_LIST)) {
serializer->status.buffer.size_used += snprintf((char *)
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used], buff_diff,
"\"%u\":", key);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
}
serializer->status.buffer.size_used += snprintf((char *)
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used], buff_diff,
NDPI_U64_FORMAT, (unsigned long long)value);
@@ -892,16 +907,16 @@ int ndpi_serialize_uint32_int32(ndpi_serializer *_serializer,
if(serializer->fmt == ndpi_serialization_format_json) {
ndpi_serialize_json_pre(_serializer);
if (!(serializer->status.flags & NDPI_SERIALIZER_STATUS_LIST)) {
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used],
&serializer->buffer.data[serializer->status.buffer.size_used],
buff_diff, "\"%u\":", key);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
}
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used],
&serializer->buffer.data[serializer->status.buffer.size_used],
buff_diff, "%d", value);
ndpi_serialize_json_post(_serializer);
@@ -909,7 +924,7 @@ int ndpi_serialize_uint32_int32(ndpi_serializer *_serializer,
if (ndpi_serializer_header_uint32(serializer, key) < 0) return(-1);
ndpi_serialize_csv_pre(serializer);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
serializer->status.buffer.size_used += snprintf((char *)
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used], buff_diff,
"%d", value);
} else {
@@ -962,14 +977,14 @@ int ndpi_serialize_uint32_int64(ndpi_serializer *_serializer,
ndpi_serialize_json_pre(_serializer);
if (!(serializer->status.flags & NDPI_SERIALIZER_STATUS_LIST)) {
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used],
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used],
buff_diff, "\"%u\":", key);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
}
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used],
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used],
buff_diff, NDPI_I64_FORMAT, (long long int)value);
ndpi_serialize_json_post(_serializer);
@@ -1084,7 +1099,7 @@ static int ndpi_serialize_uint32_binary(ndpi_serializer *_serializer,
ndpi_serialize_json_pre(_serializer);
if (!(serializer->status.flags & NDPI_SERIALIZER_STATUS_LIST)) {
serializer->status.buffer.size_used += snprintf((char *) &serializer->buffer.data[serializer->status.buffer.size_used],
serializer->status.buffer.size_used += snprintf((char *) &serializer->buffer.data[serializer->status.buffer.size_used],
buff_diff, "\"%u\":", key);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
}
@@ -1096,9 +1111,9 @@ static int ndpi_serialize_uint32_binary(ndpi_serializer *_serializer,
if (ndpi_serializer_header_uint32(serializer, key) < 0) return(-1);
ndpi_serialize_csv_pre(serializer);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used], buff_diff,
"%s", value);
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used], buff_diff,
"%s", value);
} else {
ndpi_serialization_type kt;
u_int8_t type = 0;
@@ -1135,7 +1150,7 @@ int ndpi_serialize_uint32_boolean(ndpi_serializer *_serializer,
if(serializer->fmt != ndpi_serialization_format_json &&
serializer->fmt != ndpi_serialization_format_csv)
return -1;
return(-1);
if(buff_diff < needed) {
if(ndpi_extend_serializer_buffer(&serializer->buffer, needed - buff_diff) < 0)
@@ -1147,14 +1162,14 @@ int ndpi_serialize_uint32_boolean(ndpi_serializer *_serializer,
ndpi_serialize_json_pre(_serializer);
if (!(serializer->status.flags & NDPI_SERIALIZER_STATUS_LIST)) {
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used],
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used],
buff_diff, "\"%u\":", key);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
}
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used],
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used],
buff_diff, "%s", value ? "true" : "false");
ndpi_serialize_json_post(_serializer);
@@ -1180,8 +1195,10 @@ int ndpi_serialize_binary_int32(ndpi_serializer *_serializer,
u_int32_t buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
u_int32_t needed;
#ifdef OPTIMIZE_NUMERIC_KEYS
if(ndpi_is_number(key, klen))
return(ndpi_serialize_uint32_int32(_serializer, atoi(key), value));
#endif
needed =
sizeof(u_int8_t) /* type */ +
@@ -1204,13 +1221,12 @@ int ndpi_serialize_binary_int32(ndpi_serializer *_serializer,
if (!(serializer->status.flags & NDPI_SERIALIZER_STATUS_LIST)) {
serializer->status.buffer.size_used += ndpi_json_string_escape(key, klen,
(char *) &serializer->buffer.data[serializer->status.buffer.size_used], buff_diff);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used], buff_diff, ":");
serializer->buffer.data[serializer->status.buffer.size_used] = ':';
serializer->status.buffer.size_used++;
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
}
serializer->status.buffer.size_used += snprintf((char *)
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used], buff_diff, "%d", value);
ndpi_serialize_json_post(_serializer);
@@ -1238,7 +1254,7 @@ int ndpi_serialize_binary_int32(ndpi_serializer *_serializer,
}
serializer->status.flags |= NDPI_SERIALIZER_STATUS_NOT_EMPTY;
return(0);
}
@@ -1246,6 +1262,38 @@ int ndpi_serialize_binary_int32(ndpi_serializer *_serializer,
int ndpi_serialize_string_int32(ndpi_serializer *_serializer,
const char *key, int32_t value) {
#ifdef OPTIMIZE_CSV_SERIALIZATION
ndpi_private_serializer *serializer = (ndpi_private_serializer*)_serializer;
if(serializer->fmt == ndpi_serialization_format_csv) {
/* Key is ignored */
u_int32_t buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
u_int16_t needed = 11 /* 10 (billion) + CVS separator */;
int rc;
if(buff_diff < needed) {
if(ndpi_extend_serializer_buffer(&serializer->buffer, needed - buff_diff) < 0)
return(-1);
else
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
}
if(!(serializer->status.flags & NDPI_SERIALIZER_STATUS_HDR_DONE)) {
if(ndpi_serializer_header_string(serializer, key, strlen(key)) < 0)
return(-1);
}
ndpi_serialize_csv_pre(serializer);
needed--;
rc = snprintf((char*)&serializer->buffer.data[serializer->status.buffer.size_used],
needed, "%u", value);
if(rc > 0)
serializer->status.buffer.size_used += rc;
return(0);
} else
#endif
return(ndpi_serialize_binary_int32(_serializer, key, strlen(key), value));
}
@@ -1258,8 +1306,10 @@ int ndpi_serialize_binary_int64(ndpi_serializer *_serializer,
u_int32_t buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
u_int32_t needed;
#ifdef OPTIMIZE_NUMERIC_KEYS
if(ndpi_is_number(key, klen))
return(ndpi_serialize_uint32_int64(_serializer, atoi(key), value));
#endif
needed =
sizeof(u_int8_t) /* type */ +
@@ -1283,13 +1333,12 @@ int ndpi_serialize_binary_int64(ndpi_serializer *_serializer,
serializer->status.buffer.size_used += ndpi_json_string_escape(key, klen,
(char *) &serializer->buffer.data[serializer->status.buffer.size_used], buff_diff);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used], buff_diff,
":");
serializer->buffer.data[serializer->status.buffer.size_used] = ':';
serializer->status.buffer.size_used++;
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
}
serializer->status.buffer.size_used += snprintf((char *)
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used], buff_diff,
NDPI_I64_FORMAT, (long long int)value);
@@ -1306,7 +1355,7 @@ int ndpi_serialize_binary_int64(ndpi_serializer *_serializer,
} else {
serializer->buffer.data[serializer->status.buffer.size_used++] = (ndpi_serialization_string << 4) | ndpi_serialization_int64;
ndpi_serialize_single_string(serializer, key, klen);
ndpi_serialize_single_uint32(serializer, value);
ndpi_serialize_single_uint64(serializer, value);
}
}
@@ -1329,17 +1378,48 @@ int ndpi_serialize_binary_uint32(ndpi_serializer *_serializer,
u_int32_t buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
u_int32_t needed;
#ifdef OPTIMIZE_CSV_SERIALIZATION
if(serializer->fmt == ndpi_serialization_format_csv) {
/* Key is ignored */
u_int32_t buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
u_int16_t needed;
char _value[16];
snprintf(_value, sizeof(_value), "%u", value);
needed = strlen(_value) + 1 /* CVS separator */;
if(buff_diff < needed) {
if(ndpi_extend_serializer_buffer(&serializer->buffer, needed - buff_diff) < 0)
return(-1);
else
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
}
if(!(serializer->status.flags & NDPI_SERIALIZER_STATUS_HDR_DONE)) {
if(ndpi_serializer_header_string(serializer, key, klen) < 0)
return(-1);
}
ndpi_serialize_csv_pre(serializer);
needed--;
memcpy(&serializer->buffer.data[serializer->status.buffer.size_used], _value, needed);
serializer->status.buffer.size_used += needed;
return(0);
}
#endif
#ifdef OPTIMIZE_NUMERIC_KEYS
if(ndpi_is_number(key, klen))
return(ndpi_serialize_uint32_uint32(_serializer, atoi(key), value));
#endif
needed =
sizeof(u_int8_t) /* type */ +
sizeof(u_int16_t) /* key len */ +
klen /* key */ +
sizeof(u_int32_t);
if(serializer->fmt == ndpi_serialization_format_json)
needed += 16 + klen;
sizeof(u_int32_t) + /* value (as int) */
20 /* value (as string, in case of CSV) */ +
16 /* extra overhead for JSON */;
if(buff_diff < needed) {
if(ndpi_extend_serializer_buffer(&serializer->buffer, needed - buff_diff) < 0)
@@ -1354,16 +1434,14 @@ int ndpi_serialize_binary_uint32(ndpi_serializer *_serializer,
serializer->status.buffer.size_used += ndpi_json_string_escape(key, klen,
(char *) &serializer->buffer.data[serializer->status.buffer.size_used], buff_diff);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used],
buff_diff, ":");
serializer->buffer.data[serializer->status.buffer.size_used] = ':';
serializer->status.buffer.size_used++;
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
}
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used],
buff_diff, "%u", value);
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used],
buff_diff, "%u", value);
ndpi_serialize_json_post(_serializer);
} else if(serializer->fmt == ndpi_serialization_format_csv) {
@@ -1371,8 +1449,8 @@ int ndpi_serialize_binary_uint32(ndpi_serializer *_serializer,
ndpi_serialize_csv_pre(serializer);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used], buff_diff,
"%u", value);
&serializer->buffer.data[serializer->status.buffer.size_used], buff_diff,
"%u", value);
} else {
if(value <= 0xff) {
serializer->buffer.data[serializer->status.buffer.size_used++] = (ndpi_serialization_string << 4) | ndpi_serialization_uint8;
@@ -1431,8 +1509,10 @@ int ndpi_serialize_binary_uint64(ndpi_serializer *_serializer,
u_int32_t buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
u_int32_t needed;
#ifdef OPTIMIZE_NUMERIC_KEYS
if(ndpi_is_number(key, klen))
return(ndpi_serialize_uint32_uint64(_serializer, atoi(key), value));
#endif
needed =
sizeof(u_int8_t) /* type */ +
@@ -1456,13 +1536,12 @@ int ndpi_serialize_binary_uint64(ndpi_serializer *_serializer,
serializer->status.buffer.size_used += ndpi_json_string_escape(key, klen,
(char *) &serializer->buffer.data[serializer->status.buffer.size_used], buff_diff);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used], buff_diff,
":");
serializer->buffer.data[serializer->status.buffer.size_used] = ':';
serializer->status.buffer.size_used++;
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
}
serializer->status.buffer.size_used += snprintf((char *)
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used], buff_diff,
NDPI_U64_FORMAT, (unsigned long long)value);
@@ -1506,8 +1585,10 @@ int ndpi_serialize_binary_float(ndpi_serializer *_serializer,
u_int32_t buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
u_int32_t needed;
#ifdef OPTIMIZE_NUMERIC_KEYS
if(ndpi_is_number(key, klen))
return(ndpi_serialize_uint32_float(_serializer, atoi(key), value, format));
#endif
needed =
sizeof(u_int8_t) /* type */ +
@@ -1530,9 +1611,9 @@ int ndpi_serialize_binary_float(ndpi_serializer *_serializer,
if (!(serializer->status.flags & NDPI_SERIALIZER_STATUS_LIST)) {
serializer->status.buffer.size_used += ndpi_json_string_escape(key, klen,
(char *) &serializer->buffer.data[serializer->status.buffer.size_used], buff_diff);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
serializer->buffer.data[serializer->status.buffer.size_used] = ':';
serializer->status.buffer.size_used++;
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
}
serializer->status.buffer.size_used += snprintf((char *) &serializer->buffer.data[serializer->status.buffer.size_used], buff_diff, format, value);
@@ -1598,26 +1679,31 @@ static int ndpi_serialize_binary_raw(ndpi_serializer *_serializer,
if (!(serializer->status.flags & NDPI_SERIALIZER_STATUS_LIST)) {
serializer->status.buffer.size_used += ndpi_json_string_escape(key, klen,
(char *) &serializer->buffer.data[serializer->status.buffer.size_used], buff_diff);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
serializer->status.buffer.size_used += snprintf((char *) &serializer->buffer.data[serializer->status.buffer.size_used], buff_diff, ":");
serializer->buffer.data[serializer->status.buffer.size_used] = ':';
serializer->status.buffer.size_used++;
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
}
if (escape)
serializer->status.buffer.size_used += ndpi_json_string_escape(value, vlen,
(char *) &serializer->buffer.data[serializer->status.buffer.size_used], buff_diff);
else
serializer->status.buffer.size_used += snprintf((char *) &serializer->buffer.data[serializer->status.buffer.size_used], buff_diff,
value, vlen);
else {
//serializer->status.buffer.size_used += snprintf((char *) &serializer->buffer.data[serializer->status.buffer.size_used], buff_diff, value, vlen);
memcpy(&serializer->buffer.data[serializer->status.buffer.size_used], value, vlen);
serializer->status.buffer.size_used += vlen;
}
ndpi_serialize_json_post(_serializer);
} else if(serializer->fmt == ndpi_serialization_format_csv) {
if (ndpi_serializer_header_string(serializer, key, klen) < 0) return(-1);
ndpi_serialize_csv_pre(serializer);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used], buff_diff,
"%s", value);
//serializer->status.buffer.size_used += snprintf((char *)
// &serializer->buffer.data[serializer->status.buffer.size_used], buff_diff, "%s", value);
memcpy(&serializer->buffer.data[serializer->status.buffer.size_used], value, vlen);
serializer->status.buffer.size_used += vlen;
} else {
serializer->buffer.data[serializer->status.buffer.size_used++] = (ndpi_serialization_string << 4) | ndpi_serialization_string;
@@ -1631,7 +1717,7 @@ static int ndpi_serialize_binary_raw(ndpi_serializer *_serializer,
/* ********************************** */
/* Key is a <string, len> pair, value is a <string, len> pair */
/* Key is a <string, len> pair, value is a <string, len> pair */
int ndpi_serialize_binary_binary(ndpi_serializer *_serializer,
const char *key,
u_int16_t klen,
@@ -1639,10 +1725,12 @@ int ndpi_serialize_binary_binary(ndpi_serializer *_serializer,
u_int16_t vlen) {
const char *value = _value ? _value : "";
#ifdef OPTIMIZE_NUMERIC_KEYS
if(ndpi_is_number(key, klen))
return(ndpi_serialize_uint32_binary(_serializer, atoi(key), value, vlen));
#endif
return ndpi_serialize_binary_raw(_serializer, key, klen, value, vlen, 1 /* escape */);
return(ndpi_serialize_binary_raw(_serializer, key, klen, value, vlen, 1 /* escape */));
}
/* ********************************** */
@@ -1660,6 +1748,33 @@ int ndpi_serialize_string_binary(ndpi_serializer *_serializer,
int ndpi_serialize_string_string(ndpi_serializer *_serializer,
const char *key, const char *_value) {
const char *value = _value ? _value : "";
ndpi_private_serializer *serializer = (ndpi_private_serializer*)_serializer;
#ifdef OPTIMIZE_CSV_SERIALIZATION
if(serializer->fmt == ndpi_serialization_format_csv) {
/* Key is ignored */
u_int32_t buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
u_int16_t needed = strlen(_value) + 1 /* CVS separator */;
if(buff_diff < needed) {
if(ndpi_extend_serializer_buffer(&serializer->buffer, needed - buff_diff) < 0)
return(-1);
else
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
}
if(!(serializer->status.flags & NDPI_SERIALIZER_STATUS_HDR_DONE)) {
if(ndpi_serializer_header_string(serializer, key, strlen(key)) < 0)
return(-1);
}
ndpi_serialize_csv_pre(serializer);
needed--;
memcpy(&serializer->buffer.data[serializer->status.buffer.size_used], _value, needed);
serializer->status.buffer.size_used += needed;
return(0);
} else
#endif
return(ndpi_serialize_binary_binary(_serializer, key, strlen(key), value, strlen(value)));
}
@@ -1681,10 +1796,12 @@ int ndpi_serialize_binary_boolean(ndpi_serializer *_serializer,
if(serializer->fmt != ndpi_serialization_format_json &&
serializer->fmt != ndpi_serialization_format_csv)
return -1;
return(-1);
#ifdef OPTIMIZE_NUMERIC_KEYS
if(ndpi_is_number(key, klen))
return(ndpi_serialize_uint32_boolean(_serializer, atoi(key), value));
#endif
needed = klen + 16;
@@ -1700,9 +1817,8 @@ int ndpi_serialize_binary_boolean(ndpi_serializer *_serializer,
if (!(serializer->status.flags & NDPI_SERIALIZER_STATUS_LIST)) {
serializer->status.buffer.size_used += ndpi_json_string_escape(key, klen,
(char *) &serializer->buffer.data[serializer->status.buffer.size_used], buff_diff);
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
serializer->status.buffer.size_used += snprintf((char *)
&serializer->buffer.data[serializer->status.buffer.size_used], buff_diff, ":");
serializer->buffer.data[serializer->status.buffer.size_used] = ':';
serializer->status.buffer.size_used++;
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
}
@@ -1779,7 +1895,7 @@ int ndpi_serialize_start_of_list(ndpi_serializer *_serializer,
const char *key = _key ? _key : "";
u_int16_t klen = strlen(key);
return ndpi_serialize_start_of_list_binary(_serializer, key, klen);
return(ndpi_serialize_start_of_list_binary(_serializer, key, klen));
}
/* ********************************** */
@@ -1813,7 +1929,7 @@ int ndpi_serialize_start_of_block_binary(ndpi_serializer *_serializer,
u_int32_t buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
u_int32_t needed;
if(serializer->fmt != ndpi_serialization_format_json &&
if(serializer->fmt != ndpi_serialization_format_json &&
serializer->fmt != ndpi_serialization_format_tlv)
return(-1);
@@ -1833,7 +1949,7 @@ int ndpi_serialize_start_of_block_binary(ndpi_serializer *_serializer,
serializer->status.buffer.size_used += snprintf((char *) &serializer->buffer.data[serializer->status.buffer.size_used], buff_diff, ": {");
buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
ndpi_serialize_json_post(_serializer);
serializer->status.flags |= NDPI_SERIALIZER_STATUS_SOB;
} else /* ndpi_serialization_format_tlv */ {
serializer->buffer.data[serializer->status.buffer.size_used++] = ndpi_serialization_start_of_block;
@@ -1851,7 +1967,7 @@ int ndpi_serialize_start_of_block(ndpi_serializer *_serializer,
const char *key = _key ? _key : "";
u_int16_t klen = strlen(key);
return ndpi_serialize_start_of_block_binary(_serializer, key, klen);
return(ndpi_serialize_start_of_block_binary(_serializer, key, klen));
}
/* ********************************** */
@@ -1862,7 +1978,7 @@ int ndpi_serialize_end_of_block(ndpi_serializer *_serializer) {
u_int32_t buff_diff = serializer->buffer.size - serializer->status.buffer.size_used;
u_int32_t needed;
if(serializer->fmt != ndpi_serialization_format_json &&
if(serializer->fmt != ndpi_serialization_format_json &&
serializer->fmt != ndpi_serialization_format_tlv)
return(-1);
@@ -1899,8 +2015,8 @@ void ndpi_serializer_create_snapshot(ndpi_serializer *_serializer) {
(serializer->status.flags & NDPI_SERIALIZER_STATUS_ARRAY) ? " ARRAY" : "",
(serializer->status.flags & NDPI_SERIALIZER_STATUS_EOR) ? " EOR" : ""
);
#endif
#endif
memcpy(&serializer->snapshot, &serializer->status, sizeof(ndpi_private_serializer_status));
serializer->has_snapshot = 1;
}
@@ -1909,7 +2025,7 @@ void ndpi_serializer_create_snapshot(ndpi_serializer *_serializer) {
void ndpi_serializer_rollback_snapshot(ndpi_serializer *_serializer) {
ndpi_private_serializer *serializer = (ndpi_private_serializer*)_serializer;
if(serializer->has_snapshot) {
memcpy(&serializer->status, &serializer->snapshot, sizeof(ndpi_private_serializer_status));
serializer->has_snapshot = 0;
@@ -1921,7 +2037,7 @@ void ndpi_serializer_rollback_snapshot(ndpi_serializer *_serializer) {
serializer->buffer.data[0] = ' ';
serializer->buffer.data[serializer->status.buffer.size_used-1] = '}';
}
}
}
}
}
+243 -134
View File
@@ -1,7 +1,7 @@
/*
* ndpi_utils.c
*
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
@@ -59,6 +59,15 @@
// #define DEBUG_REASSEMBLY
#ifdef HAVE_PCRE
#include <pcre.h>
struct pcre_struct {
pcre *compiled;
pcre_extra *optimized;
};
#endif
/* ****************************************** */
/* implementation of the punycode check function */
@@ -853,12 +862,12 @@ static const char* ndpi_get_flow_info_by_proto_id(struct ndpi_flow_struct const
{
case NDPI_PROTOCOL_DNS:
case NDPI_PROTOCOL_HTTP:
return (char const *)flow->host_server_name;
return flow->host_server_name;
case NDPI_PROTOCOL_QUIC:
case NDPI_PROTOCOL_TLS:
if (flow->l4.tcp.tls.hello_processed != 0)
if (flow->protos.tls_quic.hello_processed != 0)
{
return flow->protos.tls_quic_stun.tls_quic.client_requested_server_name;
return flow->host_server_name;
}
break;
}
@@ -881,21 +890,24 @@ const char* ndpi_get_flow_info(struct ndpi_flow_struct const * const flow,
/* ********************************** */
char* ndpi_ssl_version2str(struct ndpi_flow_struct *flow,
char* ndpi_ssl_version2str(char *buf, int buf_len,
u_int16_t version, u_int8_t *unknown_tls_version) {
if(unknown_tls_version)
*unknown_tls_version = 0;
if(buf == NULL || buf_len <= 1)
return NULL;
switch(version) {
case 0x0300: return("SSLv3");
case 0x0301: return("TLSv1");
case 0x0302: return("TLSv1.1");
case 0x0303: return("TLSv1.2");
case 0x0304: return("TLSv1.3");
case 0XFB1A: return("TLSv1.3 (Fizz)"); /* https://engineering.fb.com/security/fizz/ */
case 0XFEFF: return("DTLSv1.0");
case 0XFEFD: return("DTLSv1.2");
case 0x0300: strncpy(buf, "SSLv3", buf_len); buf[buf_len - 1] = '\0'; return buf;
case 0x0301: strncpy(buf, "TLSv1", buf_len); buf[buf_len - 1] = '\0'; return buf;
case 0x0302: strncpy(buf, "TLSv1.1", buf_len); buf[buf_len - 1] = '\0'; return buf;
case 0x0303: strncpy(buf, "TLSv1.2", buf_len); buf[buf_len - 1] = '\0'; return buf;
case 0x0304: strncpy(buf, "TLSv1.3", buf_len); buf[buf_len - 1] = '\0'; return buf;
case 0XFB1A: strncpy(buf, "TLSv1.3 (Fizz)", buf_len); buf[buf_len - 1] = '\0'; return buf; /* https://engineering.fb.com/security/fizz/ */
case 0XFEFF: strncpy(buf, "DTLSv1.0", buf_len); buf[buf_len - 1] = '\0'; return buf;
case 0XFEFD: strncpy(buf, "DTLSv1.2", buf_len); buf[buf_len - 1] = '\0'; return buf;
case 0x0A0A:
case 0x1A1A:
case 0x2A2A:
@@ -911,22 +923,21 @@ char* ndpi_ssl_version2str(struct ndpi_flow_struct *flow,
case 0xCACA:
case 0xDADA:
case 0xEAEA:
case 0xFAFA: return("GREASE");
case 0xFAFA: strncpy(buf, "GREASE", buf_len); buf[buf_len - 1] = '\0'; return buf;
}
if((version >= 0x7f00) && (version <= 0x7fff))
return("TLSv1.3 (draft)");
if((version >= 0x7f00) && (version <= 0x7fff)) {
strncpy(buf, "TLSv1.3 (draft)", buf_len);
buf[buf_len - 1] = '\0';
return buf;
}
if(unknown_tls_version)
*unknown_tls_version = 1;
if(flow != NULL) {
snprintf(flow->protos.tls_quic_stun.tls_quic.ssl_version_str,
sizeof(flow->protos.tls_quic_stun.tls_quic.ssl_version_str), "TLS (%04X)", version);
snprintf(buf, buf_len, "TLS (%04X)", version);
return(flow->protos.tls_quic_stun.tls_quic.ssl_version_str);
} else
return("");
return buf;
}
/* ***************************************************** */
@@ -1048,7 +1059,7 @@ u_char* ndpi_base64_decode(const u_char *src, size_t len, size_t *out_len) {
char* ndpi_base64_encode(unsigned char const* bytes_to_encode, size_t in_len) {
size_t len = 0, ret_size;
char *ret;
int i = 0;
int j, i = 0;
unsigned char char_array_3[3];
unsigned char char_array_4[4];
@@ -1072,7 +1083,7 @@ char* ndpi_base64_encode(unsigned char const* bytes_to_encode, size_t in_len) {
}
if(i) {
for(int j = i; j < 3; j++)
for(j = i; j < 3; j++)
char_array_3[j] = '\0';
char_array_4[0] = (char_array_3[0] & 0xfc) >> 2;
@@ -1080,7 +1091,7 @@ char* ndpi_base64_encode(unsigned char const* bytes_to_encode, size_t in_len) {
char_array_4[2] = ((char_array_3[1] & 0x0f) << 2) + ((char_array_3[2] & 0xc0) >> 6);
char_array_4[3] = char_array_3[2] & 0x3f;
for(int j = 0; (j < i + 1); j++)
for(j = 0; (j < i + 1); j++)
ret[len++] = base64_table[char_array_4[j]];
while((i++ < 3))
@@ -1095,24 +1106,46 @@ char* ndpi_base64_encode(unsigned char const* bytes_to_encode, size_t in_len) {
/* ********************************** */
void ndpi_serialize_risk(ndpi_serializer *serializer,
struct ndpi_flow_struct *flow) {
if(flow->risk != 0) {
u_int32_t i;
ndpi_risk_enum risk)
{
u_int32_t i;
ndpi_serialize_start_of_block(serializer, "flow_risk");
for(i = 0; i < NDPI_MAX_RISK; i++) {
ndpi_risk_enum r = (ndpi_risk_enum)i;
if(NDPI_ISSET_BIT(flow->risk, r))
ndpi_serialize_uint32_string(serializer, i, ndpi_risk2str(r));
}
ndpi_serialize_end_of_block(serializer);
if (risk == NDPI_NO_RISK) {
return;
}
ndpi_serialize_start_of_block(serializer, "flow_risk");
for(i = 0; i < NDPI_MAX_RISK; i++) {
ndpi_risk_enum r = (ndpi_risk_enum)i;
if(NDPI_ISSET_BIT(risk, r))
ndpi_serialize_uint32_string(serializer, i, ndpi_risk2str(r));
}
ndpi_serialize_end_of_block(serializer);
}
/* ********************************** */
void ndpi_serialize_proto(struct ndpi_detection_module_struct *ndpi_struct,
ndpi_serializer *serializer,
ndpi_risk_enum risk,
ndpi_protocol l7_protocol)
{
char buf[64];
ndpi_serialize_start_of_block(serializer, "ndpi");
ndpi_serialize_risk(serializer, risk);
ndpi_serialize_string_string(serializer, "proto", ndpi_protocol2name(ndpi_struct, l7_protocol, buf, sizeof(buf)));
ndpi_protocol_breed_t breed =
ndpi_get_proto_breed(ndpi_struct,
(l7_protocol.app_protocol != NDPI_PROTOCOL_UNKNOWN ? l7_protocol.app_protocol : l7_protocol.master_protocol));
ndpi_serialize_string_string(serializer, "breed", ndpi_get_proto_breed_name(ndpi_struct, breed));
if(l7_protocol.category != NDPI_PROTOCOL_CATEGORY_UNSPECIFIED)
ndpi_serialize_string_string(serializer, "category", ndpi_category_get_name(ndpi_struct, l7_protocol.category));
ndpi_serialize_end_of_block(serializer);
}
/* ********************************** */
/* ********************************** */
/* NOTE: serializer must have been already initialized */
@@ -1124,21 +1157,20 @@ int ndpi_dpi2json(struct ndpi_detection_module_struct *ndpi_struct,
if(flow == NULL) return(-1);
ndpi_serialize_start_of_block(serializer, "ndpi");
ndpi_serialize_risk(serializer, flow);
ndpi_serialize_string_string(serializer, "proto", ndpi_protocol2name(ndpi_struct, l7_protocol, buf, sizeof(buf)));
ndpi_protocol_breed_t breed =
ndpi_get_proto_breed(ndpi_struct,
(l7_protocol.app_protocol != NDPI_PROTOCOL_UNKNOWN ? l7_protocol.app_protocol : l7_protocol.master_protocol));
ndpi_serialize_string_string(serializer, "breed", ndpi_get_proto_breed_name(ndpi_struct, breed));
if(l7_protocol.category != NDPI_PROTOCOL_CATEGORY_UNSPECIFIED)
ndpi_serialize_string_string(serializer, "category", ndpi_category_get_name(ndpi_struct, l7_protocol.category));
ndpi_serialize_end_of_block(serializer);
ndpi_serialize_proto(ndpi_struct, serializer, flow->risk, l7_protocol);
switch(l7_protocol.master_protocol ? l7_protocol.master_protocol : l7_protocol.app_protocol) {
case NDPI_PROTOCOL_IP_ICMP:
if (flow->entropy > 0.0f) {
ndpi_serialize_string_float(serializer, "entropy", flow->entropy, "%.6f");
}
break;
case NDPI_PROTOCOL_DHCP:
ndpi_serialize_start_of_block(serializer, "dhcp");
ndpi_serialize_string_string(serializer, "hostname", flow->host_server_name);
ndpi_serialize_string_string(serializer, "fingerprint", flow->protos.dhcp.fingerprint);
ndpi_serialize_string_string(serializer, "class_ident", flow->protos.dhcp.class_ident);
ndpi_serialize_end_of_block(serializer);
break;
@@ -1165,7 +1197,7 @@ int ndpi_dpi2json(struct ndpi_detection_module_struct *ndpi_struct,
case NDPI_PROTOCOL_DNS:
ndpi_serialize_start_of_block(serializer, "dns");
if(flow->host_server_name[0] != '\0')
ndpi_serialize_string_string(serializer, "query", (const char*)flow->host_server_name);
ndpi_serialize_string_string(serializer, "query", flow->host_server_name);
ndpi_serialize_string_uint32(serializer, "num_queries", flow->protos.dns.num_queries);
ndpi_serialize_string_uint32(serializer, "num_answers", flow->protos.dns.num_answers);
ndpi_serialize_string_uint32(serializer, "reply_code", flow->protos.dns.reply_code);
@@ -1177,9 +1209,16 @@ int ndpi_dpi2json(struct ndpi_detection_module_struct *ndpi_struct,
ndpi_serialize_end_of_block(serializer);
break;
case NDPI_PROTOCOL_NTP:
ndpi_serialize_start_of_block(serializer, "ntp");
ndpi_serialize_string_uint32(serializer, "request_code", flow->protos.ntp.request_code);
ndpi_serialize_string_uint32(serializer, "version", flow->protos.ntp.request_code);
ndpi_serialize_end_of_block(serializer);
break;
case NDPI_PROTOCOL_MDNS:
ndpi_serialize_start_of_block(serializer, "mdns");
ndpi_serialize_string_string(serializer, "answer", (const char*)flow->host_server_name);
ndpi_serialize_string_string(serializer, "answer", flow->host_server_name);
ndpi_serialize_end_of_block(serializer);
break;
@@ -1207,7 +1246,7 @@ int ndpi_dpi2json(struct ndpi_detection_module_struct *ndpi_struct,
case NDPI_PROTOCOL_HTTP:
ndpi_serialize_start_of_block(serializer, "http");
if(flow->host_server_name[0] != '\0')
ndpi_serialize_string_string(serializer, "hostname", (const char*)flow->host_server_name);
ndpi_serialize_string_string(serializer, "hostname", flow->host_server_name);
if(flow->http.url != NULL){
ndpi_serialize_string_string(serializer, "url", flow->http.url);
ndpi_serialize_string_uint32(serializer, "code", flow->http.response_status_code);
@@ -1219,52 +1258,56 @@ int ndpi_dpi2json(struct ndpi_detection_module_struct *ndpi_struct,
case NDPI_PROTOCOL_QUIC:
ndpi_serialize_start_of_block(serializer, "quic");
if(flow->protos.tls_quic_stun.tls_quic.client_requested_server_name[0] != '\0')
if(flow->host_server_name[0] != '\0')
ndpi_serialize_string_string(serializer, "client_requested_server_name",
flow->protos.tls_quic_stun.tls_quic.client_requested_server_name);
flow->host_server_name);
if(flow->protos.tls_quic.server_names)
ndpi_serialize_string_string(serializer, "server_names", flow->protos.tls_quic.server_names);
if(flow->http.user_agent)
ndpi_serialize_string_string(serializer, "user_agent", flow->http.user_agent);
if(flow->protos.tls_quic_stun.tls_quic.ssl_version) {
if(flow->protos.tls_quic.ssl_version) {
u_int8_t unknown_tls_version;
char *version = ndpi_ssl_version2str(flow, flow->protos.tls_quic_stun.tls_quic.ssl_version, &unknown_tls_version);
char version[16];
ndpi_ssl_version2str(version, sizeof(version), flow->protos.tls_quic.ssl_version, &unknown_tls_version);
if(!unknown_tls_version)
ndpi_serialize_string_string(serializer, "version", version);
if(flow->protos.tls_quic_stun.tls_quic.alpn)
ndpi_serialize_string_string(serializer, "alpn", flow->protos.tls_quic_stun.tls_quic.alpn);
ndpi_serialize_string_string(serializer, "ja3", flow->protos.tls_quic_stun.tls_quic.ja3_client);
if(flow->protos.tls_quic_stun.tls_quic.tls_supported_versions)
ndpi_serialize_string_string(serializer, "tls_supported_versions", flow->protos.tls_quic_stun.tls_quic.tls_supported_versions);
if(flow->protos.tls_quic.alpn)
ndpi_serialize_string_string(serializer, "alpn", flow->protos.tls_quic.alpn);
ndpi_serialize_string_string(serializer, "ja3", flow->protos.tls_quic.ja3_client);
if(flow->protos.tls_quic.tls_supported_versions)
ndpi_serialize_string_string(serializer, "tls_supported_versions", flow->protos.tls_quic.tls_supported_versions);
}
ndpi_serialize_end_of_block(serializer);
break;
case NDPI_PROTOCOL_MAIL_IMAP:
ndpi_serialize_start_of_block(serializer, "imap");
ndpi_serialize_string_string(serializer, "user", flow->protos.ftp_imap_pop_smtp.username);
ndpi_serialize_string_string(serializer, "password", flow->protos.ftp_imap_pop_smtp.password);
ndpi_serialize_string_string(serializer, "user", flow->l4.tcp.ftp_imap_pop_smtp.username);
ndpi_serialize_string_string(serializer, "password", flow->l4.tcp.ftp_imap_pop_smtp.password);
ndpi_serialize_end_of_block(serializer);
break;
case NDPI_PROTOCOL_MAIL_POP:
ndpi_serialize_start_of_block(serializer, "pop");
ndpi_serialize_string_string(serializer, "user", flow->protos.ftp_imap_pop_smtp.username);
ndpi_serialize_string_string(serializer, "password", flow->protos.ftp_imap_pop_smtp.password);
ndpi_serialize_string_string(serializer, "user", flow->l4.tcp.ftp_imap_pop_smtp.username);
ndpi_serialize_string_string(serializer, "password", flow->l4.tcp.ftp_imap_pop_smtp.password);
ndpi_serialize_end_of_block(serializer);
break;
case NDPI_PROTOCOL_MAIL_SMTP:
ndpi_serialize_start_of_block(serializer, "smtp");
ndpi_serialize_string_string(serializer, "user", flow->protos.ftp_imap_pop_smtp.username);
ndpi_serialize_string_string(serializer, "password", flow->protos.ftp_imap_pop_smtp.password);
ndpi_serialize_string_string(serializer, "user", flow->l4.tcp.ftp_imap_pop_smtp.username);
ndpi_serialize_string_string(serializer, "password", flow->l4.tcp.ftp_imap_pop_smtp.password);
ndpi_serialize_end_of_block(serializer);
break;
case NDPI_PROTOCOL_FTP_CONTROL:
ndpi_serialize_start_of_block(serializer, "ftp");
ndpi_serialize_string_string(serializer, "user", flow->protos.ftp_imap_pop_smtp.username);
ndpi_serialize_string_string(serializer, "password", flow->protos.ftp_imap_pop_smtp.password);
ndpi_serialize_string_uint32(serializer, "auth_failed", flow->protos.ftp_imap_pop_smtp.auth_failed);
ndpi_serialize_string_string(serializer, "user", flow->l4.tcp.ftp_imap_pop_smtp.username);
ndpi_serialize_string_string(serializer, "password", flow->l4.tcp.ftp_imap_pop_smtp.password);
ndpi_serialize_string_uint32(serializer, "auth_failed", flow->l4.tcp.ftp_imap_pop_smtp.auth_failed);
ndpi_serialize_end_of_block(serializer);
break;
@@ -1279,25 +1322,27 @@ int ndpi_dpi2json(struct ndpi_detection_module_struct *ndpi_struct,
case NDPI_PROTOCOL_TLS:
case NDPI_PROTOCOL_DTLS:
if(flow->protos.tls_quic_stun.tls_quic.ssl_version) {
if(flow->protos.tls_quic.ssl_version) {
char notBefore[32], notAfter[32];
struct tm a, b, *before = NULL, *after = NULL;
u_int i, off;
u_int8_t unknown_tls_version;
char *version = ndpi_ssl_version2str(flow, flow->protos.tls_quic_stun.tls_quic.ssl_version, &unknown_tls_version);
char version[16];
if(flow->protos.tls_quic_stun.tls_quic.notBefore)
before = gmtime_r((const time_t *)&flow->protos.tls_quic_stun.tls_quic.notBefore, &a);
if(flow->protos.tls_quic_stun.tls_quic.notAfter)
after = gmtime_r((const time_t *)&flow->protos.tls_quic_stun.tls_quic.notAfter, &b);
ndpi_ssl_version2str(version, sizeof(version), flow->protos.tls_quic.ssl_version, &unknown_tls_version);
if(flow->protos.tls_quic.notBefore)
before = gmtime_r((const time_t *)&flow->protos.tls_quic.notBefore, &a);
if(flow->protos.tls_quic.notAfter)
after = gmtime_r((const time_t *)&flow->protos.tls_quic.notAfter, &b);
if(!unknown_tls_version) {
ndpi_serialize_start_of_block(serializer, "tls");
ndpi_serialize_string_string(serializer, "version", version);
ndpi_serialize_string_string(serializer, "client_requested_server_name",
flow->protos.tls_quic_stun.tls_quic.client_requested_server_name);
if(flow->protos.tls_quic_stun.tls_quic.server_names)
ndpi_serialize_string_string(serializer, "server_names", flow->protos.tls_quic_stun.tls_quic.server_names);
flow->host_server_name);
if(flow->protos.tls_quic.server_names)
ndpi_serialize_string_string(serializer, "server_names", flow->protos.tls_quic.server_names);
if(before) {
strftime(notBefore, sizeof(notBefore), "%Y-%m-%d %H:%M:%S", before);
@@ -1308,27 +1353,27 @@ int ndpi_dpi2json(struct ndpi_detection_module_struct *ndpi_struct,
strftime(notAfter, sizeof(notAfter), "%Y-%m-%d %H:%M:%S", after);
ndpi_serialize_string_string(serializer, "notafter", notAfter);
}
ndpi_serialize_string_string(serializer, "ja3", flow->protos.tls_quic_stun.tls_quic.ja3_client);
ndpi_serialize_string_string(serializer, "ja3s", flow->protos.tls_quic_stun.tls_quic.ja3_server);
ndpi_serialize_string_uint32(serializer, "unsafe_cipher", flow->protos.tls_quic_stun.tls_quic.server_unsafe_cipher);
ndpi_serialize_string_string(serializer, "cipher", ndpi_cipher2str(flow->protos.tls_quic_stun.tls_quic.server_cipher));
ndpi_serialize_string_string(serializer, "ja3", flow->protos.tls_quic.ja3_client);
ndpi_serialize_string_string(serializer, "ja3s", flow->protos.tls_quic.ja3_server);
ndpi_serialize_string_uint32(serializer, "unsafe_cipher", flow->protos.tls_quic.server_unsafe_cipher);
ndpi_serialize_string_string(serializer, "cipher", ndpi_cipher2str(flow->protos.tls_quic.server_cipher));
if(flow->protos.tls_quic_stun.tls_quic.issuerDN)
ndpi_serialize_string_string(serializer, "issuerDN", flow->protos.tls_quic_stun.tls_quic.issuerDN);
if(flow->protos.tls_quic.issuerDN)
ndpi_serialize_string_string(serializer, "issuerDN", flow->protos.tls_quic.issuerDN);
if(flow->protos.tls_quic_stun.tls_quic.subjectDN)
ndpi_serialize_string_string(serializer, "issuerDN", flow->protos.tls_quic_stun.tls_quic.subjectDN);
if(flow->protos.tls_quic.subjectDN)
ndpi_serialize_string_string(serializer, "subjectDN", flow->protos.tls_quic.subjectDN);
if(flow->protos.tls_quic_stun.tls_quic.alpn)
ndpi_serialize_string_string(serializer, "alpn", flow->protos.tls_quic_stun.tls_quic.alpn);
if(flow->protos.tls_quic.alpn)
ndpi_serialize_string_string(serializer, "alpn", flow->protos.tls_quic.alpn);
if(flow->protos.tls_quic_stun.tls_quic.tls_supported_versions)
ndpi_serialize_string_string(serializer, "tls_supported_versions", flow->protos.tls_quic_stun.tls_quic.tls_supported_versions);
if(flow->protos.tls_quic.tls_supported_versions)
ndpi_serialize_string_string(serializer, "tls_supported_versions", flow->protos.tls_quic.tls_supported_versions);
if(flow->protos.tls_quic_stun.tls_quic.sha1_certificate_fingerprint[0] != '\0') {
if(flow->protos.tls_quic.sha1_certificate_fingerprint[0] != '\0') {
for(i=0, off=0; i<20; i++) {
int rc = snprintf(&buf[off], sizeof(buf)-off,"%s%02X", (i > 0) ? ":" : "",
flow->protos.tls_quic_stun.tls_quic.sha1_certificate_fingerprint[i] & 0xFF);
flow->protos.tls_quic.sha1_certificate_fingerprint[i] & 0xFF);
if(rc <= 0) break; else off += rc;
}
@@ -1351,7 +1396,7 @@ int ndpi_dpi2json(struct ndpi_detection_module_struct *ndpi_struct,
int ndpi_flow2json(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow,
u_int8_t ip_version,
u_int8_t l4_protocol, u_int16_t vlan_id,
u_int8_t l4_protocol,
u_int32_t src_v4, u_int32_t dst_v4,
struct ndpi_in6_addr *src_v6, struct ndpi_in6_addr *dst_v6,
u_int16_t src_port, u_int16_t dst_port,
@@ -1567,7 +1612,7 @@ static int ndpi_is_rce_injection(char* query) {
size_t ushlen = sizeof(ush_commands) / sizeof(ush_commands[0]);
for(int i = 0; i < ushlen; i++) {
for(unsigned long i = 0; i < ushlen; i++) {
if(strstr(query, ush_commands[i]) != NULL) {
return 1;
}
@@ -1575,7 +1620,7 @@ static int ndpi_is_rce_injection(char* query) {
size_t pwshlen = sizeof(pwsh_commands) / sizeof(pwsh_commands[0]);
for(int i = 0; i < pwshlen; i++) {
for(unsigned long i = 0; i < pwshlen; i++) {
if(strstr(query, pwsh_commands[i]) != NULL) {
return 1;
}
@@ -1676,28 +1721,28 @@ const char* ndpi_risk2str(ndpi_risk_enum risk) {
switch(risk) {
case NDPI_URL_POSSIBLE_XSS:
return("XSS attack");
return("XSS Attack");
case NDPI_URL_POSSIBLE_SQL_INJECTION:
return("SQL injection");
return("SQL Injection");
case NDPI_URL_POSSIBLE_RCE_INJECTION:
return("RCE injection");
return("RCE Injection");
case NDPI_BINARY_APPLICATION_TRANSFER:
return("Binary application transfer");
return("Binary Application Transfer");
case NDPI_KNOWN_PROTOCOL_ON_NON_STANDARD_PORT:
return("Known protocol on non standard port");
return("Known Protocol on Non Standard Port");
case NDPI_TLS_SELFSIGNED_CERTIFICATE:
return("Self-signed Certificate");
case NDPI_TLS_OBSOLETE_VERSION:
return("Obsolete TLS version (< 1.1)");
return("Obsolete TLS Version (1.1 or older)");
case NDPI_TLS_WEAK_CIPHER:
return("Weak TLS cipher");
return("Weak TLS Cipher");
case NDPI_TLS_CERTIFICATE_EXPIRED:
return("TLS Expired Certificate");
@@ -1718,13 +1763,13 @@ const char* ndpi_risk2str(ndpi_risk_enum risk) {
return("HTTP Suspicious Header");
case NDPI_TLS_NOT_CARRYING_HTTPS:
return("TLS (probably) not carrying HTTPS");
return("TLS (probably) Not Carrying HTTPS");
case NDPI_SUSPICIOUS_DGA_DOMAIN:
return("Suspicious DGA domain name");
return("Suspicious DGA Domain name");
case NDPI_MALFORMED_PACKET:
return("Malformed packet");
return("Malformed Packet");
case NDPI_SSH_OBSOLETE_CLIENT_VERSION_OR_CIPHER:
return("SSH Obsolete Client Version/Cipher");
@@ -1742,19 +1787,19 @@ const char* ndpi_risk2str(ndpi_risk_enum risk) {
return("Unsafe Protocol");
case NDPI_DNS_SUSPICIOUS_TRAFFIC:
return("Suspicious DNS traffic"); /* Exfiltration ? */
return("Suspicious DNS Traffic"); /* Exfiltration ? */
case NDPI_TLS_MISSING_SNI:
return("SNI TLS extension was missing");
return("Missing SNI TLS Extension");
case NDPI_HTTP_SUSPICIOUS_CONTENT:
return("HTTP suspicious content");
return("HTTP Suspicious Content");
case NDPI_RISKY_ASN:
return("Risky ASN");
case NDPI_RISKY_DOMAIN:
return("Risky domain name");
return("Risky Domain Name");
case NDPI_MALICIOUS_JA3:
return("Possibly Malicious JA3 Fingerprint");
@@ -1769,11 +1814,37 @@ const char* ndpi_risk2str(ndpi_risk_enum risk) {
return("Uncommon TLS ALPN");
case NDPI_TLS_CERT_VALIDITY_TOO_LONG:
return("TLS certificate validity longer than 13 months");
return("TLS Certificate Validity Too Long");
case NDPI_TLS_EXTENSION_SUSPICIOUS:
return("TLS extension suspicious");
case NDPI_TLS_SUSPICIOUS_EXTENSION:
return("TLS Suspicious Extension");
case NDPI_TLS_FATAL_ALERT:
return("TLS Fatal Alert");
case NDPI_SUSPICIOUS_ENTROPY:
return("Suspicious Entropy");
case NDPI_CLEAR_TEXT_CREDENTIALS:
return("Clear-Text Credentials");
case NDPI_DNS_LARGE_PACKET:
return("DNS Packet Larger Than 512 bytes");
case NDPI_DNS_FRAGMENTED:
return("Fragmented DNS Message");
case NDPI_INVALID_CHARACTERS:
return("Text Contains Non-Printable Characters");
case NDPI_POSSIBLE_EXPLOIT:
return("Possible Exploit Detected");
break;
case NDPI_TLS_CERTIFICATE_ABOUT_TO_EXPIRE:
return("TLS Certificate About To Expire");
break;
default:
snprintf(buf, sizeof(buf), "%d", (int)risk);
return(buf);
@@ -2084,14 +2155,13 @@ static void ndpi_handle_risk_exceptions(struct ndpi_detection_module_struct *ndp
/* TODO: add IPv6 support */
if(!flow->ip_risk_mask_evaluated) {
if(flow->packet.iph) {
struct ndpi_packet_struct *packet = &flow->packet;
if(flow->is_ipv6 == 0) {
struct in_addr pin;
pin.s_addr = packet->iph->saddr;
pin.s_addr = flow->saddr;
flow->risk_mask &= ndpi_host_ip_risk_ptree_match(ndpi_str, &pin);
pin.s_addr = packet->iph->daddr;
pin.s_addr = flow->daddr;
flow->risk_mask &= ndpi_host_ip_risk_ptree_match(ndpi_str, &pin);
}
@@ -2114,28 +2184,38 @@ void ndpi_set_risk(struct ndpi_detection_module_struct *ndpi_str,
/* ******************************************************************** */
int ndpi_is_printable_string(char const * const str, size_t len) {
for (size_t i = 0; i < len; ++i) {
if (ndpi_isprint(str[i]) == 0) {
return 0;
}
}
int ndpi_isset_risk(struct ndpi_detection_module_struct *ndpi_str,
struct ndpi_flow_struct *flow, ndpi_risk_enum r) {
ndpi_risk v = 1ull << r;
return 1;
return(((flow->risk & v) == v) ? 1 : 0);
}
/* ******************************************************************** */
float ndpi_calculate_entropy(u_int8_t const * const buf, size_t len) {
int ndpi_is_printable_string(char * const str, size_t len) {
int retval = 1;
for (size_t i = 0; i < len; ++i) {
if (ndpi_isprint(str[i]) == 0) {
str[i] = '?';
retval = 0;
}
}
return retval;
}
/* ******************************************************************** */
float ndpi_entropy(u_int8_t const * const buf, size_t len) {
float entropy = 0.0f;
u_int32_t byte_counters[256];
memset(byte_counters, 0, sizeof(byte_counters));
for (size_t i = 0; i < len; ++i) {
if (buf[i] == i) {
byte_counters[i]++;
}
byte_counters[buf[i]]++;
}
for (size_t i = 0; i < sizeof(byte_counters) / sizeof(byte_counters[0]); ++i) {
@@ -2143,11 +2223,10 @@ float ndpi_calculate_entropy(u_int8_t const * const buf, size_t len) {
continue;
}
float p = 1.0f * byte_counters[i] / len;
entropy -= p * log2f(p);
float const p = (float)byte_counters[i] / len;
entropy += p * log2f(1 / p);
}
entropy *= -1.0f;
return entropy;
}
@@ -2159,9 +2238,6 @@ char* ndpi_get_flow_name(struct ndpi_flow_struct *flow) {
if(flow->host_server_name[0] != '\0')
return((char*)flow->host_server_name);
if(flow->protos.tls_quic_stun.tls_quic.client_requested_server_name[0] != '\0')
return(flow->protos.tls_quic_stun.tls_quic.client_requested_server_name);
no_flow_info:
return((char*)"");
}
@@ -2191,6 +2267,9 @@ void load_common_alpns(struct ndpi_detection_module_struct *ndpi_str) {
"h3-fb-05", "h1q-fb",
"doq-i00",
/* ApplePush */
"apns-security-v3", "apns-pack-v1",
NULL /* end */
};
u_int i;
@@ -2226,3 +2305,33 @@ u_int8_t is_a_common_alpn(struct ndpi_detection_module_struct *ndpi_str,
return(0);
}
/* ******************************************* */
u_int8_t ndpi_is_valid_protoId(u_int16_t protoId) {
return((protoId >= NDPI_MAX_SUPPORTED_PROTOCOLS + NDPI_MAX_NUM_CUSTOM_PROTOCOLS) ? 0 : 1);
}
/* ******************************************* */
u_int8_t ndpi_is_encrypted_proto(struct ndpi_detection_module_struct *ndpi_str,
ndpi_protocol proto) {
if(proto.master_protocol == NDPI_PROTOCOL_UNKNOWN && ndpi_is_valid_protoId(proto.app_protocol)) {
return(!ndpi_str->proto_defaults[proto.app_protocol].isClearTextProto);
} else if(ndpi_is_valid_protoId(proto.master_protocol) && ndpi_is_valid_protoId(proto.app_protocol)) {
if(ndpi_str->proto_defaults[proto.master_protocol].isClearTextProto
&& (!ndpi_str->proto_defaults[proto.app_protocol].isClearTextProto))
return(0);
else
return((ndpi_str->proto_defaults[proto.master_protocol].isClearTextProto
&& ndpi_str->proto_defaults[proto.app_protocol].isClearTextProto) ? 0 : 1);
} else
return(0);
}
/* ******************************************* */
void ndpi_set_tls_cert_expire_days(struct ndpi_detection_module_struct *ndpi_str,
u_int8_t num_days) {
ndpi_str->tls_certificate_expire_in_x_days = num_days;
}
+3 -3
View File
@@ -2,7 +2,7 @@
* afp.c
*
* Copyright (C) 2009-11 by ipoque GmbH
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
@@ -37,13 +37,13 @@ struct afpHeader {
static void ndpi_int_afp_add_connection(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow)
{
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_AFP, NDPI_PROTOCOL_UNKNOWN);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_AFP, NDPI_PROTOCOL_UNKNOWN, NDPI_CONFIDENCE_DPI);
}
void ndpi_search_afp(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow)
{
struct ndpi_packet_struct *packet = &flow->packet;
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
NDPI_LOG_DBG(ndpi_struct, "search AFP\n");
+3 -3
View File
@@ -2,7 +2,7 @@
* aimini.c
*
* Copyright (C) 2009-11 - ipoque GmbH
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
@@ -33,13 +33,13 @@
static void ndpi_int_aimini_add_connection(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow/* , */
/* ndpi_protocol_type_t protocol_type */)
{
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_HTTP, NDPI_PROTOCOL_AIMINI);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_HTTP, NDPI_PROTOCOL_AIMINI, NDPI_CONFIDENCE_DPI);
}
void ndpi_search_aimini(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow)
{
struct ndpi_packet_struct *packet = &flow->packet;
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
NDPI_LOG_DBG(ndpi_struct, "search aimini\n");
+3 -5
View File
@@ -63,7 +63,7 @@ static void set_ajp_detected(struct ndpi_detection_module_struct *ndpi_struct,
/* If no custom protocol has been detected */
/* if(flow->detected_protocol_stack[0] == NDPI_PROTOCOL_UNKNOWN) */
ndpi_int_reset_protocol(flow);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_AJP, flow->guessed_host_protocol_id);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_AJP, flow->guessed_host_protocol_id, NDPI_CONFIDENCE_DPI);
}
}
@@ -73,7 +73,7 @@ static void set_ajp_detected(struct ndpi_detection_module_struct *ndpi_struct,
static void ndpi_check_ajp(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow) {
struct ajp_header ajp_hdr;
struct ndpi_packet_struct *packet = &flow->packet;
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
if (packet->payload_packet_len < sizeof(ajp_hdr)) {
NDPI_EXCLUDE_PROTO(ndpi_struct, flow);
@@ -115,15 +115,13 @@ static void ndpi_check_ajp(struct ndpi_detection_module_struct *ndpi_struct,
void ndpi_search_ajp(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow)
{
struct ndpi_packet_struct *packet = &flow->packet;
// Break after 20 packets.
if(flow->packet_counter > 20) {
NDPI_EXCLUDE_PROTO(ndpi_struct, flow);
return;
}
if(packet->detected_protocol_stack[0] != NDPI_PROTOCOL_UNKNOWN) {
if(flow->detected_protocol_stack[0] != NDPI_PROTOCOL_UNKNOWN) {
return;
}
+6 -8
View File
@@ -1,7 +1,7 @@
/*
* amazon_video.c
*
* Copyright (C) 2018-21 - ntop.org
* Copyright (C) 2018-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
@@ -29,7 +29,7 @@
static void ndpi_check_amazon_video(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow) {
struct ndpi_packet_struct *packet = &flow->packet;
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
NDPI_LOG_DBG(ndpi_struct, "search Amazon Prime\n");
@@ -40,7 +40,7 @@ static void ndpi_check_amazon_video(struct ndpi_detection_module_struct *ndpi_st
packet->payload[2] == 0xFA &&
packet->payload[3] == 0xCE)) {
NDPI_LOG_INFO(ndpi_struct, "found Amazon Video on TCP\n");
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_AMAZON_VIDEO, NDPI_PROTOCOL_UNKNOWN);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_AMAZON_VIDEO, NDPI_PROTOCOL_UNKNOWN, NDPI_CONFIDENCE_DPI);
return;
} else if((packet->udp != NULL) &&
(packet->payload[0] == 0xDE &&
@@ -48,7 +48,7 @@ static void ndpi_check_amazon_video(struct ndpi_detection_module_struct *ndpi_st
packet->payload[2] == 0xBE &&
packet->payload[3] == 0xEF)) {
NDPI_LOG_INFO(ndpi_struct, "found Amazon Video on UDP\n");
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_AMAZON_VIDEO, NDPI_PROTOCOL_UNKNOWN);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_AMAZON_VIDEO, NDPI_PROTOCOL_UNKNOWN, NDPI_CONFIDENCE_DPI);
return;
}
}
@@ -58,12 +58,10 @@ static void ndpi_check_amazon_video(struct ndpi_detection_module_struct *ndpi_st
void ndpi_search_amazon_video(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow) {
struct ndpi_packet_struct *packet = &flow->packet;
NDPI_LOG_DBG(ndpi_struct, "search amazon_video\n");
/* skip marked packets */
if(packet->detected_protocol_stack[0] != NDPI_PROTOCOL_AMAZON_VIDEO)
if(flow->detected_protocol_stack[0] != NDPI_PROTOCOL_AMAZON_VIDEO)
ndpi_check_amazon_video(ndpi_struct, flow);
}
@@ -73,7 +71,7 @@ void init_amazon_video_dissector(struct ndpi_detection_module_struct *ndpi_struc
ndpi_set_bitmask_protocol_detection("AMAZON_VIDEO", ndpi_struct, detection_bitmask, *id,
NDPI_PROTOCOL_AMAZON_VIDEO,
ndpi_search_amazon_video,
NDPI_SELECTION_BITMASK_PROTOCOL_TCP_OR_UDP_WITH_PAYLOAD,
NDPI_SELECTION_BITMASK_PROTOCOL_V4_V6_TCP_OR_UDP_WITH_PAYLOAD,
SAVE_DETECTION_BITMASK_AS_UNKNOWN,
ADD_TO_DETECTION_BITMASK);
*id += 1;
+2 -2
View File
@@ -27,13 +27,13 @@
static void ndpi_int_among_us_add_connection(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow)
{
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_AMONG_US, NDPI_PROTOCOL_UNKNOWN);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_AMONG_US, NDPI_PROTOCOL_UNKNOWN, NDPI_CONFIDENCE_DPI);
}
void ndpi_search_among_us(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow)
{
struct ndpi_packet_struct * const packet = &flow->packet;
struct ndpi_packet_struct * const packet = &ndpi_struct->packet;
/* handshake packet */
if (packet->payload_packet_len > 9 &&
+3 -3
View File
@@ -1,7 +1,7 @@
/*
* amqp.c
*
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* nDPI is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published by
@@ -36,11 +36,11 @@ struct amqp_header {
static void ndpi_int_amqp_add_connection(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow/* , */
/* ndpi_protocol_type_t protocol_type */) {
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_AMQP, NDPI_PROTOCOL_UNKNOWN);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_AMQP, NDPI_PROTOCOL_UNKNOWN, NDPI_CONFIDENCE_DPI);
}
void ndpi_search_amqp(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow) {
struct ndpi_packet_struct *packet = &flow->packet;
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
NDPI_LOG_DBG(ndpi_struct, "search amqp\n");
+50 -20
View File
@@ -27,26 +27,58 @@
#include "ndpi_api.h"
static int is_apple_push_addr(const struct ndpi_packet_struct *packet)
{
if(packet->iph) {
/* 17.0.0.0/8 */
if(((ntohl(packet->iph->saddr) & 0xFF000000 /* 255.0.0.0 */) == 0x11000000) ||
((ntohl(packet->iph->daddr) & 0xFF000000 /* 255.0.0.0 */) == 0x11000000))
return 1;
} else if(packet->iphv6) {
/* 2620:149:a44::/48 */
if(((packet->iphv6->ip6_src.u6_addr.u6_addr32[0] == ntohl(0x26200149)) &&
((packet->iphv6->ip6_src.u6_addr.u6_addr32[1] & htonl (0xffff0000)) == ntohl(0x0a440000))) ||
((packet->iphv6->ip6_dst.u6_addr.u6_addr32[0] == ntohl(0x26200149)) &&
((packet->iphv6->ip6_dst.u6_addr.u6_addr32[1] & htonl (0xffff0000)) == ntohl(0x0a440000))))
return 1;
/* 2403:300:a42::/48 */
if(((packet->iphv6->ip6_src.u6_addr.u6_addr32[0] == ntohl(0x24030300)) &&
((packet->iphv6->ip6_src.u6_addr.u6_addr32[1] & htonl (0xffff0000)) == ntohl(0x0a420000))) ||
((packet->iphv6->ip6_dst.u6_addr.u6_addr32[0] == ntohl(0x24030300)) &&
((packet->iphv6->ip6_dst.u6_addr.u6_addr32[1] & htonl (0xffff0000)) == ntohl(0x0a420000))))
return 1;
/* 2403:300:a51::/48 */
if(((packet->iphv6->ip6_src.u6_addr.u6_addr32[0] == ntohl(0x24030300)) &&
((packet->iphv6->ip6_src.u6_addr.u6_addr32[1] & htonl (0xffff0000)) == ntohl(0x0a510000))) ||
((packet->iphv6->ip6_dst.u6_addr.u6_addr32[0] == ntohl(0x24030300)) &&
((packet->iphv6->ip6_dst.u6_addr.u6_addr32[1] & htonl (0xffff0000)) == ntohl(0x0a510000))))
return 1;
/* 2a01:b740:a42::/48 */
if(((packet->iphv6->ip6_src.u6_addr.u6_addr32[0] == ntohl(0x2a0ab740)) &&
((packet->iphv6->ip6_src.u6_addr.u6_addr32[1] & htonl (0xffff0000)) == ntohl(0x0a420000))) ||
((packet->iphv6->ip6_dst.u6_addr.u6_addr32[0] == ntohl(0x2a0ab740)) &&
((packet->iphv6->ip6_dst.u6_addr.u6_addr32[1] & htonl (0xffff0000)) == ntohl(0x0a420000))))
return 1;
}
return 0;
}
static void ndpi_check_apple_push(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow) {
struct ndpi_packet_struct *packet = &flow->packet;
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
if(packet->iph) {
/* https://support.apple.com/en-us/HT203609 */
if(((ntohl(packet->iph->saddr) & 0xFF000000 /* 255.0.0.0 */) == 0x11000000 /* 17.0.0.0/8 */)
|| ((ntohl(packet->iph->daddr) & 0xFF000000 /* 255.0.0.0 */) == 0x11000000 /* 17.0.0.0/8 */)) {
u_int16_t apple_push_port = ntohs(5223);
u_int16_t notification_apn_port = ntohs(2195);
u_int16_t apn_feedback_port = ntohs(2196);
/* https://support.apple.com/en-us/HT203609 */
if(is_apple_push_addr(packet)) {
u_int16_t apple_push_port = ntohs(5223);
u_int16_t notification_apn_port = ntohs(2197);
if(((packet->tcp->source == apple_push_port) || (packet->tcp->dest == apple_push_port))
|| ((packet->tcp->source == notification_apn_port) || (packet->tcp->dest == notification_apn_port))
|| ((packet->tcp->source == apn_feedback_port) || (packet->tcp->dest == apn_feedback_port))
) {
NDPI_LOG_INFO(ndpi_struct, "found apple_push\n");
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_APPLE_PUSH, NDPI_PROTOCOL_UNKNOWN);
return;
}
if((packet->tcp->source == apple_push_port) || (packet->tcp->dest == apple_push_port) ||
(packet->tcp->source == notification_apn_port) || (packet->tcp->dest == notification_apn_port)) {
NDPI_LOG_INFO(ndpi_struct, "found apple_push\n");
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_APPLE_PUSH, NDPI_PROTOCOL_UNKNOWN, NDPI_CONFIDENCE_DPI);
return;
}
}
@@ -55,12 +87,10 @@ static void ndpi_check_apple_push(struct ndpi_detection_module_struct *ndpi_stru
void ndpi_search_apple_push(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow)
{
struct ndpi_packet_struct *packet = &flow->packet;
NDPI_LOG_DBG(ndpi_struct, "search apple_push\n");
/* skip marked packets */
if(packet->detected_protocol_stack[0] != NDPI_PROTOCOL_APPLE_PUSH)
if(flow->detected_protocol_stack[0] != NDPI_PROTOCOL_APPLE_PUSH)
ndpi_check_apple_push(ndpi_struct, flow);
}
@@ -70,7 +100,7 @@ void init_apple_push_dissector(struct ndpi_detection_module_struct *ndpi_struct,
ndpi_set_bitmask_protocol_detection("APPLE_PUSH", ndpi_struct, detection_bitmask, *id,
NDPI_PROTOCOL_APPLE_PUSH,
ndpi_search_apple_push,
NDPI_SELECTION_BITMASK_PROTOCOL_TCP,
NDPI_SELECTION_BITMASK_PROTOCOL_V4_V6_TCP_WITH_PAYLOAD,
SAVE_DETECTION_BITMASK_AS_UNKNOWN,
ADD_TO_DETECTION_BITMASK);
*id += 1;
+3 -3
View File
@@ -2,7 +2,7 @@
* applejuice.c
*
* Copyright (C) 2009-11 - ipoque GmbH
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
@@ -32,13 +32,13 @@
static void ndpi_int_applejuice_add_connection(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow)
{
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_APPLEJUICE, NDPI_PROTOCOL_UNKNOWN);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_APPLEJUICE, NDPI_PROTOCOL_UNKNOWN, NDPI_CONFIDENCE_DPI);
}
void ndpi_search_applejuice_tcp(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow)
{
struct ndpi_packet_struct *packet = &flow->packet;
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
NDPI_LOG_DBG(ndpi_struct, "search applejuice\n");
+3 -3
View File
@@ -2,7 +2,7 @@
* armagetron.c
*
* Copyright (C) 2009-11 - ipoque GmbH
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
@@ -32,12 +32,12 @@
static void ndpi_int_armagetron_add_connection(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow)
{
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_ARMAGETRON, NDPI_PROTOCOL_UNKNOWN);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_ARMAGETRON, NDPI_PROTOCOL_UNKNOWN, NDPI_CONFIDENCE_DPI);
}
void ndpi_search_armagetron_udp(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow)
{
struct ndpi_packet_struct *packet = &flow->packet;
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
NDPI_LOG_DBG(ndpi_struct, "search armagetron\n");
+1 -1
View File
@@ -28,7 +28,7 @@
static void ndpi_int_flash_add_connection(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow)
{
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_FLASH);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_FLASH, NDPI_CONFIDENCE_DPI);
}
void ndpi_search_flash(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow)
+5 -5
View File
@@ -31,7 +31,7 @@
static void ndpi_int_ftp_add_connection(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow)
{
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_FTP);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_FTP, NDPI_CONFIDENCE_DPI);
}
/**
@@ -407,7 +407,7 @@ void ndpi_search_ftp_tcp(struct ndpi_detection_module_struct *ndpi_struct, struc
if (src != NULL && ndpi_packet_dst_ip_eql(packet, &src->ftp_ip)
&& packet->tcp->syn != 0 && packet->tcp->ack == 0
&& packet->detected_protocol_stack[0] == NDPI_PROTOCOL_UNKNOWN
&& flow->detected_protocol_stack[0] == NDPI_PROTOCOL_UNKNOWN
&& NDPI_COMPARE_PROTOCOL_TO_BITMASK(src->detected_protocol_bitmask,
NDPI_PROTOCOL_FTP) != 0 && src->ftp_timer_set != 0) {
NDPI_LOG(NDPI_PROTOCOL_FTP, ndpi_struct, NDPI_LOG_DEBUG, "possible ftp data, src!= 0.\n");
@@ -425,7 +425,7 @@ void ndpi_search_ftp_tcp(struct ndpi_detection_module_struct *ndpi_struct, struc
if (dst != NULL && ndpi_packet_src_ip_eql(packet, &dst->ftp_ip)
&& packet->tcp->syn != 0 && packet->tcp->ack == 0
&& packet->detected_protocol_stack[0] == NDPI_PROTOCOL_UNKNOWN
&& flow->detected_protocol_stack[0] == NDPI_PROTOCOL_UNKNOWN
&& NDPI_COMPARE_PROTOCOL_TO_BITMASK(dst->detected_protocol_bitmask,
NDPI_PROTOCOL_FTP) != 0 && dst->ftp_timer_set != 0) {
NDPI_LOG(NDPI_PROTOCOL_FTP, ndpi_struct, NDPI_LOG_DEBUG, "possible ftp data; dst!= 0.\n");
@@ -453,7 +453,7 @@ void ndpi_search_ftp_tcp(struct ndpi_detection_module_struct *ndpi_struct, struc
/* skip excluded connections */
// we test for FTP connection and search for passive mode
if (packet->detected_protocol_stack[0] == NDPI_PROTOCOL_FTP) {
if (flow->detected_protocol_stack[0] == NDPI_PROTOCOL_FTP) {
NDPI_LOG(NDPI_PROTOCOL_FTP, ndpi_struct, NDPI_LOG_DEBUG,
"detected ftp command mode. going to test data mode.\n");
search_passive_ftp_mode(ndpi_struct, flow);
@@ -463,7 +463,7 @@ void ndpi_search_ftp_tcp(struct ndpi_detection_module_struct *ndpi_struct, struc
}
if (packet->detected_protocol_stack[0] == NDPI_PROTOCOL_UNKNOWN && search_ftp(ndpi_struct, flow) != 0) {
if (flow->detected_protocol_stack[0] == NDPI_PROTOCOL_UNKNOWN && search_ftp(ndpi_struct, flow) != 0) {
NDPI_LOG(NDPI_PROTOCOL_FTP, ndpi_struct, NDPI_LOG_DEBUG, "unknown. need next packet.\n");
return;
+1 -1
View File
@@ -35,7 +35,7 @@ static void ndpi_int_manolito_add_connection(struct
struct ndpi_id_struct *src = flow->src;
struct ndpi_id_struct *dst = flow->dst;
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_MANOLITO);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_MANOLITO, NDPI_CONFIDENCE_DPI);
if (src != NULL) {
+1 -1
View File
@@ -29,7 +29,7 @@
static void ndpi_int_popo_add_connection(struct ndpi_detection_module_struct
*ndpi_struct, struct ndpi_flow_struct *flow)
{
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_POPO);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_POPO, NDPI_CONFIDENCE_DPI);
}
void ndpi_search_popo_tcp_udp(struct ndpi_detection_module_struct
+1 -1
View File
@@ -30,7 +30,7 @@ static void ndpi_int_secondlife_add_connection(struct ndpi_detection_module_stru
struct ndpi_flow_struct *flow/* , */
/* ndpi_protocol_type_t protocol_type */)
{
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_SECONDLIFE, protocol_type);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_SECONDLIFE, protocol_type, NDPI_CONFIDENCE_DPI);
}
void ndpi_search_secondlife(struct ndpi_detection_module_struct
+4 -4
View File
@@ -1,7 +1,7 @@
/*
* avast.c
*
* Copyright (C) 2012-21 - ntop.org
* Copyright (C) 2012-22 - ntop.org
*
* This module is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published by
@@ -28,13 +28,13 @@
static void ndpi_int_avast_securedns_add_connection(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow)
{
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_AVAST_SECUREDNS, NDPI_PROTOCOL_UNKNOWN);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_AVAST_SECUREDNS, NDPI_PROTOCOL_UNKNOWN, NDPI_CONFIDENCE_DPI);
}
static void ndpi_search_avast_securedns(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow)
{
struct ndpi_packet_struct * packet = &flow->packet;
struct ndpi_packet_struct * packet = &ndpi_struct->packet;
if (packet->payload_packet_len < 34 ||
ntohl(get_u_int32_t(packet->payload, 11)) != 0x00013209 ||
@@ -60,7 +60,7 @@ void init_avast_securedns_dissector(struct ndpi_detection_module_struct *ndpi_st
ndpi_struct, detection_bitmask, *id,
NDPI_PROTOCOL_AVAST_SECUREDNS,
ndpi_search_avast_securedns,
NDPI_SELECTION_BITMASK_PROTOCOL_UDP_WITH_PAYLOAD,
NDPI_SELECTION_BITMASK_PROTOCOL_V4_V6_UDP_WITH_PAYLOAD,
SAVE_DETECTION_BITMASK_AS_UNKNOWN,
ADD_TO_DETECTION_BITMASK);
+6 -6
View File
@@ -1,7 +1,7 @@
/*
* ayiya.c
*
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
@@ -42,11 +42,11 @@ struct ayiya {
void ndpi_search_ayiya(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow)
{
struct ndpi_packet_struct *packet = &flow->packet;
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
NDPI_LOG_DBG(ndpi_struct, "search AYIYA\n");
if(packet->udp && (packet->detected_protocol_stack[0] == NDPI_PROTOCOL_UNKNOWN)) {
if(packet->udp && (flow->detected_protocol_stack[0] == NDPI_PROTOCOL_UNKNOWN)) {
/* Ayiya is udp based, port 5072 */
if ((packet->udp->source == htons(5072) || packet->udp->dest == htons(5072))
/* check for ayiya new packet */
@@ -57,11 +57,11 @@ void ndpi_search_ayiya(struct ndpi_detection_module_struct *ndpi_struct, struct
u_int32_t epoch = ntohl(a->epoch), now;
u_int32_t fiveyears = 86400 * 365 * 5;
now = flow->packet.current_time_ms;
now = packet->current_time_ms;
if((epoch >= (now - fiveyears)) && (epoch <= (now+86400 /* 1 day */))) {
NDPI_LOG_INFO(ndpi_struct, "found AYIYA\n");
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_AYIYA, NDPI_PROTOCOL_UNKNOWN);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_AYIYA, NDPI_PROTOCOL_UNKNOWN, NDPI_CONFIDENCE_DPI);
}
return;
@@ -77,7 +77,7 @@ void init_ayiya_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_in
ndpi_set_bitmask_protocol_detection("Ayiya", ndpi_struct, detection_bitmask, *id,
NDPI_PROTOCOL_AYIYA,
ndpi_search_ayiya,
NDPI_SELECTION_BITMASK_PROTOCOL_UDP_WITH_PAYLOAD,
NDPI_SELECTION_BITMASK_PROTOCOL_V4_V6_UDP_WITH_PAYLOAD,
SAVE_DETECTION_BITMASK_AS_UNKNOWN,
ADD_TO_DETECTION_BITMASK);
+3 -3
View File
@@ -1,7 +1,7 @@
/*
* bgp.c
*
* Copyright (C) 2016-21 - ntop.org
* Copyright (C) 2016-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
@@ -31,7 +31,7 @@
/* this detection also works asymmetrically */
void ndpi_search_bgp(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow)
{
struct ndpi_packet_struct *packet = &flow->packet;
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
u_int16_t bgp_port = htons(179);
NDPI_LOG_DBG(ndpi_struct, "search BGP\n");
@@ -45,7 +45,7 @@ void ndpi_search_bgp(struct ndpi_detection_module_struct *ndpi_struct, struct nd
&& (ntohs(get_u_int16_t(packet->payload, 16)) <= packet->payload_packet_len)) {
NDPI_LOG_INFO(ndpi_struct, "found BGP\n");
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_BGP, NDPI_PROTOCOL_UNKNOWN);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_BGP, NDPI_PROTOCOL_UNKNOWN, NDPI_CONFIDENCE_DPI);
return;
}
}
+261 -142
View File
@@ -2,7 +2,7 @@
* bittorrent.c
*
* Copyright (C) 2009-11 - ipoque GmbH
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
@@ -35,6 +35,9 @@
#define NDPI_PROTOCOL_PLAIN_DETECTION 0
#define NDPI_PROTOCOL_WEBSEED_DETECTION 2
#define BITTORRENT_PROTO_STRING "BitTorrent protocol"
// #define BITTORRENT_CACHE_DEBUG 1
struct ndpi_utp_hdr {
u_int8_t h_version:4, h_type:4, next_extension;
@@ -43,6 +46,25 @@ struct ndpi_utp_hdr {
u_int16_t sequence_nr, ack_nr;
};
/* Forward declaration */
static void ndpi_search_bittorrent(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow);
static void ndpi_search_bittorrent_hash(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow, int bt_offset);
/* *********************************************** */
static int search_bittorrent_again(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow) {
ndpi_search_bittorrent(ndpi_struct, flow);
ndpi_search_bittorrent_hash(ndpi_struct, flow, -1);
/* Possibly more processing */
return(1);
}
/* *********************************************** */
static u_int8_t is_utpv1_pkt(const u_int8_t *payload, u_int payload_len) {
struct ndpi_utp_hdr *h = (struct ndpi_utp_hdr*)payload;
@@ -54,38 +76,101 @@ static u_int8_t is_utpv1_pkt(const u_int8_t *payload, u_int payload_len) {
if((h->window_size == 0) && (payload_len != sizeof(struct ndpi_utp_hdr)))
return(0);
return(1);
}
/* *********************************************** */
static void ndpi_search_bittorrent_hash(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow, int bt_offset) {
const char *bt_hash = NULL; /* 20 bytes long */
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
if(bt_offset == -1) {
const char *bt_magic = ndpi_strnstr((const char *)packet->payload,
BITTORRENT_PROTO_STRING, packet->payload_packet_len);
if(bt_magic) {
if(bt_magic == (const char*)&packet->payload[1])
bt_hash = (const char*)&packet->payload[28];
else
bt_hash = &bt_magic[19];
}
} else
bt_hash = (const char*)&packet->payload[28];
if(bt_hash && (packet->payload_packet_len >= (20 + (bt_hash-(const char*)packet->payload))))
memcpy(flow->protos.bittorrent.hash, bt_hash, 20);
}
/* *********************************************** */
static void ndpi_add_connection_as_bittorrent(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow,
int bt_offset, int check_hash,
const u_int8_t save_detection, const u_int8_t encrypted_connection)
{
if(check_hash) {
const char *bt_hash = NULL; /* 20 bytes long */
const u_int8_t save_detection,
const u_int8_t encrypted_connection) {
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
if(check_hash)
ndpi_search_bittorrent_hash(ndpi_struct, flow, bt_offset);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_BITTORRENT, NDPI_PROTOCOL_UNKNOWN, NDPI_CONFIDENCE_DPI_CACHE);
if(flow->protos.bittorrent.hash[0] == '\0') {
/* This is necessary to inform the core to call this dissector again */
flow->check_extra_packets = 1;
if(bt_offset == -1) {
const char *bt_magic = ndpi_strnstr((const char *)flow->packet.payload,
"BitTorrent protocol", flow->packet.payload_packet_len);
if(bt_magic)
bt_hash = &bt_magic[19];
} else
bt_hash = (const char*)&flow->packet.payload[28];
if(bt_hash && (flow->packet.payload_packet_len >= (20 + (bt_hash-(const char*)flow->packet.payload))))
memcpy(flow->protos.bittorrent.hash, bt_hash, 20);
/* Don't use just 1 as in TCP DNS more packets could be returned (e.g. ACK). */
flow->max_extra_packets_to_check = 3;
flow->extra_packets_func = search_bittorrent_again;
}
if(ndpi_struct->bittorrent_cache == NULL)
ndpi_struct->bittorrent_cache = ndpi_lru_cache_init(32768);
ndpi_int_change_protocol(ndpi_struct, flow, NDPI_PROTOCOL_BITTORRENT, NDPI_PROTOCOL_UNKNOWN);
if(ndpi_struct->bittorrent_cache && packet->iph) {
u_int32_t key1, key2, i;
if(packet->udp)
key1 = ndpi_ip_port_hash_funct(packet->iph->saddr, packet->udp->source), key2 = ndpi_ip_port_hash_funct(packet->iph->daddr, packet->udp->dest);
else
key1 = ndpi_ip_port_hash_funct(packet->iph->saddr, packet->tcp->source), key2 = ndpi_ip_port_hash_funct(packet->iph->daddr, packet->tcp->dest);
ndpi_lru_add_to_cache(ndpi_struct->bittorrent_cache, key1, NDPI_PROTOCOL_BITTORRENT);
ndpi_lru_add_to_cache(ndpi_struct->bittorrent_cache, key2, NDPI_PROTOCOL_BITTORRENT);
/* Now add hosts as twins */
ndpi_lru_add_to_cache(ndpi_struct->bittorrent_cache,
packet->iph->saddr + packet->iph->daddr,
NDPI_PROTOCOL_BITTORRENT);
/* Also add +2 ports of the sender in order to catch additional sockets open by the same client */
for(i=0; i<2; i++) {
if(packet->udp)
key1 = ndpi_ip_port_hash_funct(packet->iph->saddr, htons(ntohs(packet->udp->source)+1));
else
key1 = ndpi_ip_port_hash_funct(packet->iph->saddr, htons(ntohs(packet->tcp->source)+1));
ndpi_lru_add_to_cache(ndpi_struct->bittorrent_cache, key1, NDPI_PROTOCOL_BITTORRENT);
}
#ifdef BITTORRENT_CACHE_DEBUG
if(packet->udp)
printf("[BitTorrent] [UDP] *** ADDED ports %u / %u [%u][%u]\n", ntohs(packet->udp->source), ntohs(packet->udp->dest), key1, key2);
else
printf("[BitTorrent] [TCP] *** ADDED ports %u / %u [%u][%u]\n", ntohs(packet->tcp->source), ntohs(packet->tcp->dest), key1, key2);
#endif
}
}
/* ************************************* */
static u_int8_t ndpi_int_search_bittorrent_tcp_zero(struct ndpi_detection_module_struct
*ndpi_struct, struct ndpi_flow_struct *flow)
{
struct ndpi_packet_struct *packet = &flow->packet;
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
u_int16_t a = 0;
if(packet->payload_packet_len == 1 && packet->payload[0] == 0x13) {
@@ -95,21 +180,21 @@ static u_int8_t ndpi_int_search_bittorrent_tcp_zero(struct ndpi_detection_module
}
if(flow->packet_counter == 2 && packet->payload_packet_len > 20) {
if(memcmp(&packet->payload[0], "BitTorrent protocol", 19) == 0) {
if(memcmp(&packet->payload[0], BITTORRENT_PROTO_STRING, 19) == 0) {
NDPI_LOG_INFO(ndpi_struct, "found BT: plain\n");
ndpi_add_connection_as_bittorrent(ndpi_struct, flow, 19, 1,
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
return 1;
}
}
if(packet->payload_packet_len > 20) {
/* test for match 0x13+"BitTorrent protocol" */
/* test for match 0x13+BITTORRENT_PROTO_STRING */
if(packet->payload[0] == 0x13) {
if(memcmp(&packet->payload[1], "BitTorrent protocol", 19) == 0) {
if(memcmp(&packet->payload[1], BITTORRENT_PROTO_STRING, 19) == 0) {
NDPI_LOG_INFO(ndpi_struct, "found BT: plain\n");
ndpi_add_connection_as_bittorrent(ndpi_struct, flow, 20, 1,
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
return 1;
}
}
@@ -118,37 +203,36 @@ static u_int8_t ndpi_int_search_bittorrent_tcp_zero(struct ndpi_detection_module
if(packet->payload_packet_len > 23 && memcmp(packet->payload, "GET /webseed?info_hash=", 23) == 0) {
NDPI_LOG_INFO(ndpi_struct, "found BT: plain webseed\n");
ndpi_add_connection_as_bittorrent(ndpi_struct, flow, -1, 1,
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_WEBSEED_DETECTION);
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_WEBSEED_DETECTION);
return 1;
}
/* seen Azureus as server for webseed, possibly other servers existing, to implement */
/* is Server: hypertracker Bittorrent? */
/* no asymmetric detection possible for answer of pattern "GET /data?fid=". */
if(packet->payload_packet_len > 60
&& memcmp(packet->payload, "GET /data?fid=", 14) == 0 && memcmp(&packet->payload[54], "&size=", 6) == 0) {
&& memcmp(packet->payload, "GET /data?fid=", 14) == 0 && memcmp(&packet->payload[54], "&size=", 6) == 0) {
NDPI_LOG_INFO(ndpi_struct, "found BT: plain Bitcomet persistent seed\n");
ndpi_add_connection_as_bittorrent(ndpi_struct, flow, -1, 1,
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_WEBSEED_DETECTION);
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_WEBSEED_DETECTION);
return 1;
}
if(packet->payload_packet_len > 90 && (memcmp(packet->payload, "GET ", 4) == 0
|| memcmp(packet->payload, "POST ", 5) == 0)) {
|| memcmp(packet->payload, "POST ", 5) == 0)) {
const u_int8_t *ptr = &packet->payload[4];
u_int16_t len = packet->payload_packet_len - 4;
/* parse complete get packet here into line structure elements */
ndpi_parse_packet_line_info(ndpi_struct, flow);
/* answer to this pattern is HTTP....Server: hypertracker */
if(packet->user_agent_line.ptr != NULL
&& ((packet->user_agent_line.len > 8 && memcmp(packet->user_agent_line.ptr, "Azureus ", 8) == 0)
|| (packet->user_agent_line.len >= 10 && memcmp(packet->user_agent_line.ptr, "BitTorrent", 10) == 0)
|| (packet->user_agent_line.len >= 11 && memcmp(packet->user_agent_line.ptr, "BTWebClient", 11) == 0))) {
&& ((packet->user_agent_line.len > 8 && memcmp(packet->user_agent_line.ptr, "Azureus ", 8) == 0)
|| (packet->user_agent_line.len >= 10 && memcmp(packet->user_agent_line.ptr, "BitTorrent", 10) == 0)
|| (packet->user_agent_line.len >= 11 && memcmp(packet->user_agent_line.ptr, "BTWebClient", 11) == 0))) {
NDPI_LOG_INFO(ndpi_struct, "found BT: Azureus /Bittorrent user agent\n");
ndpi_add_connection_as_bittorrent(ndpi_struct, flow, -1, 1,
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_WEBSEED_DETECTION);
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_WEBSEED_DETECTION);
return 1;
}
@@ -158,87 +242,87 @@ static u_int8_t ndpi_int_search_bittorrent_tcp_zero(struct ndpi_detection_module
&& packet->line[8].len >= 9 && memcmp(packet->line[8].ptr, "X-Queue: ", 9) == 0)) {
NDPI_LOG_INFO(ndpi_struct, "found BT: Shareaza detected\n");
ndpi_add_connection_as_bittorrent(ndpi_struct, flow, -1, 1,
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_WEBSEED_DETECTION);
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_WEBSEED_DETECTION);
return 1;
}
/* this is a self built client, not possible to catch asymmetrically */
if((packet->parsed_lines == 10 || (packet->parsed_lines == 11 && packet->line[11].len == 0))
&& packet->user_agent_line.ptr != NULL
&& packet->user_agent_line.len > 12
&& memcmp(packet->user_agent_line.ptr, "Mozilla/4.0 ",
12) == 0
&& packet->host_line.ptr != NULL
&& packet->host_line.len >= 7
&& packet->line[2].ptr != NULL
&& packet->line[2].len > 14
&& memcmp(packet->line[2].ptr, "Keep-Alive: 300", 15) == 0
&& packet->line[3].ptr != NULL
&& packet->line[3].len > 21
&& memcmp(packet->line[3].ptr, "Connection: Keep-alive", 22) == 0
&& packet->line[4].ptr != NULL
&& packet->line[4].len > 10
&& (memcmp(packet->line[4].ptr, "Accpet: */*", 11) == 0
|| memcmp(packet->line[4].ptr, "Accept: */*", 11) == 0)
if((packet->parsed_lines == 10 || (packet->parsed_lines == 11 && packet->line[10].len == 0))
&& packet->user_agent_line.ptr != NULL
&& packet->user_agent_line.len > 12
&& memcmp(packet->user_agent_line.ptr, "Mozilla/4.0 ",
12) == 0
&& packet->host_line.ptr != NULL
&& packet->host_line.len >= 7
&& packet->line[2].ptr != NULL
&& packet->line[2].len > 14
&& memcmp(packet->line[2].ptr, "Keep-Alive: 300", 15) == 0
&& packet->line[3].ptr != NULL
&& packet->line[3].len > 21
&& memcmp(packet->line[3].ptr, "Connection: Keep-alive", 22) == 0
&& packet->line[4].ptr != NULL
&& packet->line[4].len > 10
&& (memcmp(packet->line[4].ptr, "Accpet: */*", 11) == 0
|| memcmp(packet->line[4].ptr, "Accept: */*", 11) == 0)
&& packet->line[5].ptr != NULL
&& packet->line[5].len > 12
&& memcmp(packet->line[5].ptr, "Range: bytes=", 13) == 0
&& packet->line[7].ptr != NULL
&& packet->line[7].len > 15
&& memcmp(packet->line[7].ptr, "Pragma: no-cache", 16) == 0
&& packet->line[8].ptr != NULL
&& packet->line[8].len > 22 && memcmp(packet->line[8].ptr, "Cache-Control: no-cache", 23) == 0) {
&& packet->line[5].ptr != NULL
&& packet->line[5].len > 12
&& memcmp(packet->line[5].ptr, "Range: bytes=", 13) == 0
&& packet->line[7].ptr != NULL
&& packet->line[7].len > 15
&& memcmp(packet->line[7].ptr, "Pragma: no-cache", 16) == 0
&& packet->line[8].ptr != NULL
&& packet->line[8].len > 22 && memcmp(packet->line[8].ptr, "Cache-Control: no-cache", 23) == 0) {
NDPI_LOG_INFO(ndpi_struct, "found BT: Bitcomet LTS\n");
ndpi_add_connection_as_bittorrent(ndpi_struct, flow, -1, 1,
NDPI_PROTOCOL_UNSAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
NDPI_PROTOCOL_UNSAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
return 1;
}
/* FlashGet pattern */
if(packet->parsed_lines == 8
&& packet->user_agent_line.ptr != NULL
&& packet->user_agent_line.len > (sizeof("Mozilla/4.0 (compatible; MSIE 6.0;") - 1)
&& memcmp(packet->user_agent_line.ptr, "Mozilla/4.0 (compatible; MSIE 6.0;",
sizeof("Mozilla/4.0 (compatible; MSIE 6.0;") - 1) == 0
&& packet->host_line.ptr != NULL
&& packet->host_line.len >= 7
&& packet->line[2].ptr != NULL
&& packet->line[2].len == 11
&& memcmp(packet->line[2].ptr, "Accept: */*", 11) == 0
&& packet->line[3].ptr != NULL && packet->line[3].len >= (sizeof("Referer: ") - 1)
&& memcmp(packet->line[3].ptr, "Referer: ", sizeof("Referer: ") - 1) == 0
&& packet->line[5].ptr != NULL
&& packet->line[5].len > 13
&& memcmp(packet->line[5].ptr, "Range: bytes=", 13) == 0
&& packet->line[6].ptr != NULL
&& packet->line[6].len > 21 && memcmp(packet->line[6].ptr, "Connection: Keep-Alive", 22) == 0) {
&& packet->user_agent_line.ptr != NULL
&& packet->user_agent_line.len > (sizeof("Mozilla/4.0 (compatible; MSIE 6.0;") - 1)
&& memcmp(packet->user_agent_line.ptr, "Mozilla/4.0 (compatible; MSIE 6.0;",
sizeof("Mozilla/4.0 (compatible; MSIE 6.0;") - 1) == 0
&& packet->host_line.ptr != NULL
&& packet->host_line.len >= 7
&& packet->line[2].ptr != NULL
&& packet->line[2].len == 11
&& memcmp(packet->line[2].ptr, "Accept: */*", 11) == 0
&& packet->line[3].ptr != NULL && packet->line[3].len >= (sizeof("Referer: ") - 1)
&& memcmp(packet->line[3].ptr, "Referer: ", sizeof("Referer: ") - 1) == 0
&& packet->line[5].ptr != NULL
&& packet->line[5].len > 13
&& memcmp(packet->line[5].ptr, "Range: bytes=", 13) == 0
&& packet->line[6].ptr != NULL
&& packet->line[6].len > 21 && memcmp(packet->line[6].ptr, "Connection: Keep-Alive", 22) == 0) {
NDPI_LOG_INFO(ndpi_struct, "found BT: FlashGet\n");
ndpi_add_connection_as_bittorrent(ndpi_struct, flow, -1, 1,
NDPI_PROTOCOL_UNSAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
NDPI_PROTOCOL_UNSAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
return 1;
}
if(packet->parsed_lines == 7
&& packet->user_agent_line.ptr != NULL
&& packet->user_agent_line.len > (sizeof("Mozilla/4.0 (compatible; MSIE 6.0;") - 1)
&& memcmp(packet->user_agent_line.ptr, "Mozilla/4.0 (compatible; MSIE 6.0;",
sizeof("Mozilla/4.0 (compatible; MSIE 6.0;") - 1) == 0
&& packet->host_line.ptr != NULL
&& packet->host_line.len >= 7
&& packet->line[2].ptr != NULL
&& packet->line[2].len == 11
&& memcmp(packet->line[2].ptr, "Accept: */*", 11) == 0
&& packet->line[3].ptr != NULL && packet->line[3].len >= (sizeof("Referer: ") - 1)
&& memcmp(packet->line[3].ptr, "Referer: ", sizeof("Referer: ") - 1) == 0
&& packet->line[5].ptr != NULL
&& packet->line[5].len > 21 && memcmp(packet->line[5].ptr, "Connection: Keep-Alive", 22) == 0) {
&& packet->user_agent_line.ptr != NULL
&& packet->user_agent_line.len > (sizeof("Mozilla/4.0 (compatible; MSIE 6.0;") - 1)
&& memcmp(packet->user_agent_line.ptr, "Mozilla/4.0 (compatible; MSIE 6.0;",
sizeof("Mozilla/4.0 (compatible; MSIE 6.0;") - 1) == 0
&& packet->host_line.ptr != NULL
&& packet->host_line.len >= 7
&& packet->line[2].ptr != NULL
&& packet->line[2].len == 11
&& memcmp(packet->line[2].ptr, "Accept: */*", 11) == 0
&& packet->line[3].ptr != NULL && packet->line[3].len >= (sizeof("Referer: ") - 1)
&& memcmp(packet->line[3].ptr, "Referer: ", sizeof("Referer: ") - 1) == 0
&& packet->line[5].ptr != NULL
&& packet->line[5].len > 21 && memcmp(packet->line[5].ptr, "Connection: Keep-Alive", 22) == 0) {
NDPI_LOG_INFO(ndpi_struct, "found BT: FlashGet\n");
ndpi_add_connection_as_bittorrent(ndpi_struct, flow, -1, 1,
NDPI_PROTOCOL_UNSAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
NDPI_PROTOCOL_UNSAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
return 1;
}
@@ -305,7 +389,7 @@ static u_int8_t ndpi_int_search_bittorrent_tcp_zero(struct ndpi_detection_module
NDPI_LOG_INFO(ndpi_struct, "found BT: tracker info hash parsed\n");
ndpi_add_connection_as_bittorrent(ndpi_struct, flow, -1, 1,
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
return 1;
}
@@ -330,10 +414,10 @@ static u_int8_t ndpi_int_search_bittorrent_tcp_zero(struct ndpi_detection_module
/* did not see this pattern anywhere */
if((memcmp(&packet->payload[0], pattern_20_bytes, 20) == 0)
&& (memcmp(&packet->payload[52], pattern_12_bytes, 12) == 0)) {
&& (memcmp(&packet->payload[52], pattern_12_bytes, 12) == 0)) {
NDPI_LOG_INFO(ndpi_struct, "found BT: Warez - Plain\n");
ndpi_add_connection_as_bittorrent(ndpi_struct, flow, -1, 1,
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
return 1;
}
}
@@ -344,10 +428,10 @@ static u_int8_t ndpi_int_search_bittorrent_tcp_zero(struct ndpi_detection_module
ndpi_parse_packet_line_info(ndpi_struct, flow);
/* haven't fount this pattern anywhere */
if(packet->host_line.ptr != NULL
&& packet->host_line.len >= 9 && memcmp(packet->host_line.ptr, "ip2p.com:", 9) == 0) {
&& packet->host_line.len >= 9 && memcmp(packet->host_line.ptr, "ip2p.com:", 9) == 0) {
NDPI_LOG_INFO(ndpi_struct, "found BT: Warez - Plain Host: ip2p.com: pattern\n");
ndpi_add_connection_as_bittorrent(ndpi_struct, flow, -1, 1,
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_WEBSEED_DETECTION);
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_WEBSEED_DETECTION);
return 1;
}
}
@@ -355,11 +439,12 @@ static u_int8_t ndpi_int_search_bittorrent_tcp_zero(struct ndpi_detection_module
return 0;
}
/* ************************************* */
/*Search for BitTorrent commands*/
/* Search for BitTorrent commands */
static void ndpi_int_search_bittorrent_tcp(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow)
{
struct ndpi_packet_struct *packet = &flow->packet;
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
if(packet->payload_packet_len == 0) {
return;
@@ -378,25 +463,48 @@ static void ndpi_int_search_bittorrent_tcp(struct ndpi_detection_module_struct *
return;
}
/* ************************************* */
static u_int8_t is_port(u_int16_t a, u_int16_t b, u_int16_t what) {
return(((what == a) || (what == b)) ? 1 : 0);
}
/* ************************************* */
static void ndpi_skip_bittorrent(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow,
struct ndpi_packet_struct *packet) {
u_int16_t sport, dport;
if(packet->udp)
sport = packet->udp->source, dport = packet->udp->dest;
else
sport = packet->tcp->source, dport = packet->tcp->dest;
void ndpi_search_bittorrent(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow)
{
struct ndpi_packet_struct *packet = &flow->packet;
if(packet->iph && ndpi_search_into_bittorrent_cache(ndpi_struct, flow, packet->iph->saddr, sport, packet->iph->daddr, dport))
ndpi_add_connection_as_bittorrent(ndpi_struct, flow, -1, 0,
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
else
NDPI_EXCLUDE_PROTO(ndpi_struct, flow);
}
/* ************************************* */
static void ndpi_search_bittorrent(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow) {
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
char *bt_proto = NULL;
/* This is broadcast */
if(packet->iph) {
if((packet->iph->saddr == 0xFFFFFFFF) || (packet->iph->daddr == 0xFFFFFFFF))
goto exclude_bt;
if(packet->udp) {
u_int16_t sport = ntohs(packet->udp->source), dport = ntohs(packet->udp->dest);
if(is_port(sport, dport, 3544) /* teredo */
|| is_port(sport, dport, 5246) || is_port(sport, dport, 5247)/* CAPWAP */) {
|| is_port(sport, dport, 5246) || is_port(sport, dport, 5247) /* CAPWAP */) {
exclude_bt:
NDPI_EXCLUDE_PROTO(ndpi_struct, flow);
return;
@@ -404,22 +512,22 @@ void ndpi_search_bittorrent(struct ndpi_detection_module_struct *ndpi_struct, st
}
}
if(packet->detected_protocol_stack[0] != NDPI_PROTOCOL_BITTORRENT) {
if(flow->detected_protocol_stack[0] != NDPI_PROTOCOL_BITTORRENT) {
/* check for tcp retransmission here */
if((packet->tcp != NULL)
&& (packet->tcp_retransmission == 0 || packet->num_retried_bytes)) {
if(packet->tcp != NULL) {
ndpi_int_search_bittorrent_tcp(ndpi_struct, flow);
} else if(packet->udp != NULL) {
/* UDP */
char *bt_search = "BT-SEARCH * HTTP/1.1\r\n";
const char *bt_search = "BT-SEARCH * HTTP/1.1\r\n";
const char *bt_search1 = "d1:ad2:id20:";
if((ntohs(packet->udp->source) < 1024)
|| (ntohs(packet->udp->dest) < 1024) /* High ports only */) {
NDPI_EXCLUDE_PROTO(ndpi_struct, flow);
ndpi_skip_bittorrent(ndpi_struct, flow, packet);
return;
}
/*
Check for uTP http://www.bittorrent.org/beps/bep_0029.html
@@ -427,7 +535,10 @@ void ndpi_search_bittorrent(struct ndpi_detection_module_struct *ndpi_struct, st
*/
if(packet->payload_packet_len >= 23 /* min header size */) {
if(strncmp((const char*)packet->payload, bt_search, strlen(bt_search)) == 0) {
if(
(strncmp((const char*)packet->payload, bt_search, strlen(bt_search)) == 0)
|| (strncmp((const char*)packet->payload, bt_search1, strlen(bt_search1)) == 0)
) {
ndpi_add_connection_as_bittorrent(ndpi_struct, flow, -1, 1,
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
return;
@@ -437,15 +548,15 @@ void ndpi_search_bittorrent(struct ndpi_detection_module_struct *ndpi_struct, st
u_int8_t v0_flags = packet->payload[18];
if(is_utpv1_pkt(packet->payload, packet->payload_packet_len)) {
bt_proto = ndpi_strnstr((const char *)&packet->payload[20], "BitTorrent protocol", packet->payload_packet_len-20);
bt_proto = ndpi_strnstr((const char *)&packet->payload[20], BITTORRENT_PROTO_STRING, packet->payload_packet_len-20);
goto bittorrent_found;
} else if((packet->payload[0]== 0x60)
&& (packet->payload[1]== 0x0)
&& (packet->payload[2]== 0x0)
&& (packet->payload[3]== 0x0)
&& (packet->payload[4]== 0x0)) {
&& (packet->payload[1]== 0x0)
&& (packet->payload[2]== 0x0)
&& (packet->payload[3]== 0x0)
&& (packet->payload[4]== 0x0)) {
/* Heuristic */
bt_proto = ndpi_strnstr((const char *)&packet->payload[20], "BitTorrent protocol", packet->payload_packet_len-20);
bt_proto = ndpi_strnstr((const char *)&packet->payload[20], BITTORRENT_PROTO_STRING, packet->payload_packet_len-20);
goto bittorrent_found;
/* CSGO/DOTA conflict */
} else if((v0_flags < 6 /* ST_NUM_STATES */) && (v0_extension < 3 /* EXT_NUM_EXT */)) {
@@ -455,58 +566,66 @@ void ndpi_search_bittorrent(struct ndpi_detection_module_struct *ndpi_struct, st
now = (u_int32_t)time(NULL);
if((ts < (now+86400)) && (ts > (now-86400))) {
bt_proto = ndpi_strnstr((const char *)&packet->payload[20], "BitTorrent protocol", packet->payload_packet_len-20);
bt_proto = ndpi_strnstr((const char *)&packet->payload[20], BITTORRENT_PROTO_STRING, packet->payload_packet_len-20);
goto bittorrent_found;
}
} else if(ndpi_strnstr((const char *)&packet->payload[20], BITTORRENT_PROTO_STRING, packet->payload_packet_len-20)
) {
goto bittorrent_found;
}
}
}
flow->bittorrent_stage++;
if(flow->bittorrent_stage < 5) {
/* We have detected bittorrent but we need to wait until we get a hash */
/* We have detected bittorrent but we need to wait until we get a hash */
if(packet->payload_packet_len > 19 /* min size */) {
if(ndpi_strnstr((const char *)packet->payload, ":target20:", packet->payload_packet_len)
|| ndpi_strnstr((const char *)packet->payload, ":find_node1:", packet->payload_packet_len)
|| ndpi_strnstr((const char *)packet->payload, "d1:ad2:id20:", packet->payload_packet_len)
|| ndpi_strnstr((const char *)packet->payload, ":info_hash20:", packet->payload_packet_len)
|| ndpi_strnstr((const char *)packet->payload, ":filter64", packet->payload_packet_len)
|| ndpi_strnstr((const char *)packet->payload, "d1:rd2:id20:", packet->payload_packet_len)
|| (bt_proto = ndpi_strnstr((const char *)packet->payload, "BitTorrent protocol", packet->payload_packet_len))
) {
bittorrent_found:
if(bt_proto && (packet->payload_packet_len > 47))
memcpy(flow->protos.bittorrent.hash, &bt_proto[27], 20);
NDPI_LOG_INFO(ndpi_struct, "found BT: plain\n");
ndpi_add_connection_as_bittorrent(ndpi_struct, flow, -1, 0,
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
return;
if(packet->payload_packet_len > 19 /* min size */) {
if(ndpi_strnstr((const char *)packet->payload, ":target20:", packet->payload_packet_len)
|| ndpi_strnstr((const char *)packet->payload, ":find_node1:", packet->payload_packet_len)
|| ndpi_strnstr((const char *)packet->payload, "d1:ad2:id20:", packet->payload_packet_len)
|| ndpi_strnstr((const char *)packet->payload, ":info_hash20:", packet->payload_packet_len)
|| ndpi_strnstr((const char *)packet->payload, ":filter64", packet->payload_packet_len)
|| ndpi_strnstr((const char *)packet->payload, "d1:rd2:id20:", packet->payload_packet_len)
|| (bt_proto = ndpi_strnstr((const char *)packet->payload, BITTORRENT_PROTO_STRING, packet->payload_packet_len))
) {
bittorrent_found:
if(bt_proto != NULL && ((u_int8_t *)&bt_proto[27] - packet->payload +
sizeof(flow->protos.bittorrent.hash)) < packet->payload_packet_len) {
memcpy(flow->protos.bittorrent.hash, &bt_proto[27], sizeof(flow->protos.bittorrent.hash));
flow->extra_packets_func = NULL; /* Nothing else to do */
}
NDPI_LOG_INFO(ndpi_struct, "found BT: plain\n");
ndpi_add_connection_as_bittorrent(ndpi_struct, flow, -1, 0,
NDPI_PROTOCOL_SAFE_DETECTION, NDPI_PROTOCOL_PLAIN_DETECTION);
return;
}
}
return;
}
NDPI_EXCLUDE_PROTO(ndpi_struct, flow);
ndpi_skip_bittorrent(ndpi_struct, flow, packet);
}
}
if(flow->packet_counter > 8) {
NDPI_EXCLUDE_PROTO(ndpi_struct, flow);
return;
}
ndpi_skip_bittorrent(ndpi_struct, flow, packet);
}
}
/* ************************************* */
void init_bittorrent_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask)
void init_bittorrent_dissector(struct ndpi_detection_module_struct *ndpi_struct,
u_int32_t *id, NDPI_PROTOCOL_BITMASK *detection_bitmask)
{
ndpi_set_bitmask_protocol_detection("BitTorrent", ndpi_struct, detection_bitmask, *id,
NDPI_PROTOCOL_BITTORRENT,
ndpi_search_bittorrent,
NDPI_SELECTION_BITMASK_PROTOCOL_TCP_OR_UDP,
NDPI_SELECTION_BITMASK_PROTOCOL_V4_V6_TCP_OR_UDP_WITH_PAYLOAD_WITHOUT_RETRANSMISSION,
SAVE_DETECTION_BITMASK_AS_UNKNOWN,
ADD_TO_DETECTION_BITMASK);
*id += 1;
+5 -9
View File
@@ -8,13 +8,13 @@ static void ndpi_int_bjnp_add_connection(struct ndpi_detection_module_struct *nd
struct ndpi_flow_struct *flow,
u_int8_t due_to_correlation) {
ndpi_set_detected_protocol(ndpi_struct, flow,
NDPI_PROTOCOL_BJNP, NDPI_PROTOCOL_UNKNOWN);
NDPI_PROTOCOL_BJNP, NDPI_PROTOCOL_UNKNOWN, NDPI_CONFIDENCE_DPI);
}
static void ndpi_check_bjnp(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow)
{
struct ndpi_packet_struct *packet = &flow->packet;
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
u_int32_t payload_len = packet->payload_packet_len;
if(packet->udp != NULL) {
@@ -36,15 +36,11 @@ static void ndpi_check_bjnp(struct ndpi_detection_module_struct *ndpi_struct, st
void ndpi_search_bjnp(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow)
{
struct ndpi_packet_struct *packet = &flow->packet;
NDPI_LOG_DBG(ndpi_struct, "search bjnp\n");
/* skip marked packets */
if (packet->detected_protocol_stack[0] != NDPI_PROTOCOL_BJNP) {
if (packet->tcp_retransmission == 0) {
ndpi_check_bjnp(ndpi_struct, flow);
}
if (flow->detected_protocol_stack[0] != NDPI_PROTOCOL_BJNP) {
ndpi_check_bjnp(ndpi_struct, flow);
}
}
@@ -54,7 +50,7 @@ void init_bjnp_dissector(struct ndpi_detection_module_struct *ndpi_struct, u_int
ndpi_set_bitmask_protocol_detection("BJNP", ndpi_struct, detection_bitmask, *id,
NDPI_PROTOCOL_BJNP,
ndpi_search_bjnp,
NDPI_SELECTION_BITMASK_PROTOCOL_UDP_WITH_PAYLOAD,
NDPI_SELECTION_BITMASK_PROTOCOL_V4_V6_UDP_WITH_PAYLOAD,
SAVE_DETECTION_BITMASK_AS_UNKNOWN,
ADD_TO_DETECTION_BITMASK);
*id += 1;
+2 -2
View File
@@ -1,7 +1,7 @@
/*
* btlib.c
*
* Copyright (C) 2011-21 - ntop.org
* Copyright (C) 2011-22 - ntop.org
* Contributed by Vitaly Lavrov <vel21ripn@gmail.com>
*
* This file is part of nDPI, an open source deep packet inspection
@@ -419,7 +419,7 @@ const u_int8_t *bt_decode(const u_int8_t *b, size_t *l, int *ret, bt_parse_data_
if(c != ':') goto bad_data;
break;
}
if(d > *l) goto bad_data;
if((size_t)d > *l) goto bad_data;
cbd->t = 2;
cbd->v.s.s = b;
cbd->v.s.l = d;
+31 -12
View File
@@ -31,25 +31,44 @@
static void ndpi_int_capwap_add_connection(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow) {
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_CAPWAP, NDPI_PROTOCOL_UNKNOWN);
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_CAPWAP, NDPI_PROTOCOL_UNKNOWN, NDPI_CONFIDENCE_DPI);
}
static int is_capwap_multicast(const struct ndpi_packet_struct *packet)
{
/* RFC 5115 Sec 3.3
"The WTP MUST send the Discovery Request
message to either the limited broadcast IP address (255.255.255.255),
the well-known CAPWAP multicast address (224.0.1.140), or to the
unicast IP address of the AC. For IPv6 networks, since broadcast
does not exist, the use of "All ACs multicast address" (FF0X:0:0:0:0:
0:0:18C) is used instead.
*/
if(packet->iph) {
if((packet->iph->daddr == 0xFFFFFFFF) ||
(ntohl(packet->iph->daddr) == 0XE000018C))
return 1;
} else if(packet->iphv6) {
if(((ntohl(packet->iphv6->ip6_dst.u6_addr.u6_addr32[0] & 0xFFF0FFFF) == 0xFF000000)) &&
(packet->iphv6->ip6_dst.u6_addr.u6_addr32[1] == 0) &&
(packet->iphv6->ip6_dst.u6_addr.u6_addr32[2] == 0) &&
(ntohl(packet->iphv6->ip6_dst.u6_addr.u6_addr32[3] == 0x0000018C)))
return 1;
}
return 0;
}
/* ************************************************** */
static void ndpi_search_setup_capwap(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow) {
struct ndpi_packet_struct *packet = &flow->packet;
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
u_int16_t sport, dport;
if(!packet->iph) {
NDPI_EXCLUDE_PROTO(ndpi_struct, flow);
return;
}
sport = ntohs(packet->udp->source), dport = ntohs(packet->udp->dest);
if((dport == NDPI_CAPWAP_CONTROL_PORT)
&& (packet->iph->daddr == 0xFFFFFFFF)
&& (is_capwap_multicast(packet))
&& (packet->payload_packet_len >= 16)
&& (packet->payload[0] == 0x0)
&& (packet->payload[8] == 6 /* Mac len */)
@@ -75,7 +94,7 @@ static void ndpi_search_setup_capwap(struct ndpi_detection_module_struct *ndpi_s
}
if(
(((dport == NDPI_CAPWAP_DATA_PORT) && (packet->iph->daddr != 0xFFFFFFFF)) || (sport == NDPI_CAPWAP_DATA_PORT))
(((dport == NDPI_CAPWAP_DATA_PORT) && (!is_capwap_multicast(packet))) || (sport == NDPI_CAPWAP_DATA_PORT))
&& (packet->payload_packet_len >= 16)
&& (packet->payload[0] == 0x0)
) {
@@ -104,9 +123,9 @@ static void ndpi_search_setup_capwap(struct ndpi_detection_module_struct *ndpi_s
void ndpi_search_capwap(struct ndpi_detection_module_struct *ndpi_struct, struct ndpi_flow_struct *flow)
{
struct ndpi_packet_struct *packet = &flow->packet;
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
if(packet->udp && (packet->detected_protocol_stack[0] == NDPI_PROTOCOL_UNKNOWN))
if(packet->udp && (flow->detected_protocol_stack[0] == NDPI_PROTOCOL_UNKNOWN))
ndpi_search_setup_capwap(ndpi_struct, flow);
}
@@ -117,7 +136,7 @@ void init_capwap_dissector(struct ndpi_detection_module_struct *ndpi_struct,
ndpi_set_bitmask_protocol_detection("CAPWAP", ndpi_struct, detection_bitmask, *id,
NDPI_PROTOCOL_CAPWAP,
ndpi_search_capwap,
NDPI_SELECTION_BITMASK_PROTOCOL_UDP_WITH_PAYLOAD,
NDPI_SELECTION_BITMASK_PROTOCOL_V4_V6_UDP_WITH_PAYLOAD,
SAVE_DETECTION_BITMASK_AS_UNKNOWN,
ADD_TO_DETECTION_BITMASK);
+145
View File
@@ -0,0 +1,145 @@
/*
* cassandra.c
*
* Copyright (C) 2021 by Lucas Santos <lfneiva.santos@gmail.com>
*
* This file is part of nDPI, an open source deep packet inspection
* library based on the OpenDPI and PACE technology by ipoque GmbH
*
* nDPI is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* nDPI is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public License
* along with nDPI. If not, see <http://www.gnu.org/licenses/>.
*/
#include <stdbool.h>
#include "ndpi_protocol_ids.h"
#define NDPI_CURRENT_PROTO NDPI_PROTOCOL_CASSANDRA
#include "ndpi_api.h"
#define CASSANDRA_HEADER_LEN 9
#define CASSANDRA_MAX_BODY_SIZE 268435456 //256MB (256 * 1024^2)
enum cassandra_version
{
CASSANDRA_V1_REQUEST = 0x01,
CASSANDRA_V1_RESPONSE = 0x81,
CASSANDRA_V2_REQUEST = 0x02,
CASSANDRA_V2_RESPONSE = 0x82,
CASSANDRA_V3_REQUEST = 0x03,
CASSANDRA_V3_RESPONSE = 0x83,
CASSANDRA_V4_REQUEST = 0x04,
CASSANDRA_V4_RESPONSE = 0x84
};
enum cassandra_opcode
{
CASSANDRA_ERROR = 0x00,
CASSANDRA_STARTUP = 0x01,
CASSANDRA_READY = 0x02,
CASSANDRA_AUTHENTICATE = 0x03,
CASSANDRA_OPTIONS = 0x05,
CASSANDRA_SUPPORTED = 0x06,
CASSANDRA_QUERY = 0x07,
CASSANDRA_RESULT = 0x08,
CASSANDRA_PREPARE = 0x09,
CASSANDRA_EXECUTE = 0x0A,
CASSANDRA_REGISTER = 0x0B,
CASSANDRA_EVENT = 0x0C,
CASSANDRA_BATCH = 0x0D,
CASSANDRA_AUTH_CHALLENGE = 0x0E,
CASSANDRA_AUTH_RESPONSE = 0x0F,
CASSANDRA_AUTH_SUCCESS = 0x10
};
static bool ndpi_check_valid_cassandra_version(uint8_t version)
{
switch(version) {
case CASSANDRA_V1_REQUEST:
case CASSANDRA_V1_RESPONSE:
case CASSANDRA_V2_REQUEST:
case CASSANDRA_V2_RESPONSE:
case CASSANDRA_V3_REQUEST:
case CASSANDRA_V3_RESPONSE:
case CASSANDRA_V4_REQUEST:
case CASSANDRA_V4_RESPONSE:
return true;
}
return false;
}
static bool ndpi_check_valid_cassandra_opcode(uint8_t opcode)
{
switch (opcode) {
case CASSANDRA_ERROR:
case CASSANDRA_STARTUP:
case CASSANDRA_READY:
case CASSANDRA_AUTHENTICATE:
case CASSANDRA_OPTIONS:
case CASSANDRA_SUPPORTED:
case CASSANDRA_QUERY:
case CASSANDRA_RESULT:
case CASSANDRA_PREPARE:
case CASSANDRA_EXECUTE:
case CASSANDRA_REGISTER:
case CASSANDRA_EVENT:
case CASSANDRA_BATCH:
case CASSANDRA_AUTH_CHALLENGE:
case CASSANDRA_AUTH_RESPONSE:
case CASSANDRA_AUTH_SUCCESS:
return true;
}
return false;
}
static bool ndpi_check_valid_cassandra_flags(uint8_t flags)
{
return (flags & 0xF0) == 0;
}
void ndpi_search_cassandra(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow)
{
struct ndpi_packet_struct *packet = &ndpi_struct->packet;
if (packet->tcp) {
if (packet->payload_packet_len >= CASSANDRA_HEADER_LEN &&
ndpi_check_valid_cassandra_version(get_u_int8_t(packet->payload, 0)) &&
ndpi_check_valid_cassandra_flags(get_u_int8_t(packet->payload, 1)) &&
ndpi_check_valid_cassandra_opcode(get_u_int8_t(packet->payload, 4)) &&
le32toh(get_u_int32_t(packet->payload, 5)) <= CASSANDRA_MAX_BODY_SIZE &&
le32toh(get_u_int32_t(packet->payload, 5)) >= (uint32_t) (packet->payload_packet_len - CASSANDRA_HEADER_LEN)) {
ndpi_set_detected_protocol(ndpi_struct, flow, NDPI_PROTOCOL_CASSANDRA, NDPI_PROTOCOL_UNKNOWN, NDPI_CONFIDENCE_DPI);
return;
}
}
NDPI_EXCLUDE_PROTO(ndpi_struct, flow);
}
/* ********************************* */
void init_cassandra_dissector(struct ndpi_detection_module_struct *ndpi_struct,
u_int32_t *id,
NDPI_PROTOCOL_BITMASK *detection_bitmask) {
ndpi_set_bitmask_protocol_detection("Cassandra",
ndpi_struct, detection_bitmask,
*id,
NDPI_PROTOCOL_CASSANDRA,
ndpi_search_cassandra,
NDPI_SELECTION_BITMASK_PROTOCOL_V4_V6_TCP_WITH_PAYLOAD,
SAVE_DETECTION_BITMASK_AS_UNKNOWN,
ADD_TO_DETECTION_BITMASK);
*id += 1;
}

Some files were not shown because too many files have changed in this diff Show More