Commit Graph
61 Commits
Author SHA1 Message Date
Nardi Ivan d72a760ac3 New API for library configuration
This is the first step into providing (more) configuration options in nDPI.

The idea is to have a simple way to configure (most of) nDPI: only one
function (`ndpi_set_config()`) to set any configuration parameters
(in the present or on in the future) and we try to keep this function
prototype as agnostic as possible.

You can configure the library:
* via API, using `ndpi_set_config()`
* via a configuration file, in a text format

This way, anytime we need to add a new configuration parameter:
* we don't need to add two public functions (a getter and a setter)
* we don't break API/ABI compatibility of the library; even changing
the parameter type (from integer to a list of integer, for example)
doesn't break the compatibility.

The complete list of configuration options is provided in
`doc/configuration_parameters.md`.

As a first example, two configuration knobs are provided:
* the ability to enable/disable the extraction of the sha1 fingerprint of
the TLS certificates.
* the upper limit on the number of packets per flow that will be subject
to inspection
2024-01-18 10:21:24 +01:00
Toni 6c3d162cd6 Add realtime protocol output to ndpiReader. (#2197)
* support for using a new flow callback invoked before the flow memory is free'd
 * minor fixes

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2024-01-09 00:39:59 +01:00
Toni 32b0aeea59 Fixed OpenWRT arm related build issues. (#2104)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-16 19:49:50 +02:00
Luca Deri d9b0ba1d14 Windows warning checks 2023-10-16 16:17:41 +02:00
Ivan Nardi 560280e6f0 fuzz: add fuzzer testing nDPI (initial) configurations (#1830)
The goal of this fuzzer is to test init and deinit of the library, with
different configurations. In details:
* random memory allocation failures, even during init phase
* random `ndpi_init_prefs` parameter of `ndpi_init_detection_module()`
* random LRU caches sizes
* random bitmask of enabled protocols
* random parameters of `ndpi_set_detection_preferences()`
* random initialization of opportunistic TLS
* random load/don't load of configuration files

This new fuzzer is a C++ file, because it uses `FuzzedDataProvider`
class (see
https://github.com/google/fuzzing/blob/master/docs/split-inputs.md).
Note that the (existing) fuzzers need to be linked with C++ compiler
anyway, so this new fuzzer doesn't add any new requirements.
2022-12-23 19:07:13 +01:00
Ivan Nardi ad6bfbad4d Add protocol disabling feature (#1808)
The application may enable only some protocols.
Disabling a protocol means:
*) don't register/use the protocol dissector code (if any)
*) disable classification by-port for such a protocol
*) disable string matchings for domains/certificates involving this protocol
*) disable subprotocol registration (if any)

This feature can be tested with `ndpiReader -B list_of_protocols_to_disable`.

Custom protocols are always enabled.

Technically speaking, this commit doesn't introduce any API/ABI
incompatibility. However, calling `ndpi_set_protocol_detection_bitmask2()`
is now mandatory, just after having called `ndpi_init_detection_module()`.

Most of the diffs (and all the diffs in `/src/lib/protocols/`) are due to
the removing of some function parameters.

Fix the low level macro `NDPI_LOG`. This issue hasn't been detected
sooner simply because almost all the code uses only the helpers `NDPI_LOG_*`
2022-12-18 08:10:57 +00:00
Toni Uhlig d27fb1e73e Moved NDPI_API_VERSION to src/include/ndpi_define.h.in where it belongs.
That way, we can make `src/include/ndpi_api.h` independent of autoconf.
MSBuild will like that.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-21 18:03:22 +02:00
Luca Deri 48d00eae35 Added MAX_NBPF_CUSTOM_PROTO define 2022-09-17 11:04:40 +02:00
Luca Deri 9fb11c909e Win fixes 2022-08-05 11:17:28 +02:00
Toni ed4f106a0d Add Softether dissector. (#1679)
Signed-off-by: lns <matzeton@googlemail.com>
2022-07-29 19:29:54 +02:00
Ivan Nardi d8d525fff2 Update the protocol bitmask for some protocols (#1675)
Tcp retransmissions should be ignored.

Remove some unused protocol bitmasks.

Update script to download Whatsapp IP list.
2022-07-27 11:46:45 +02:00
Nardi Ivan d66aa49787 DTLS: fix exclusion of DTLS protocol
Add an helper to exclude a generic protocol
2022-07-20 19:16:18 +02:00
Toni 175f863665 Label SMTP w/ STARTTLS as SMTPS *and* dissect TLS clho. (#1639)
* Label SMTP w/ STARTTLS as SMTPS *and* dissect TLS clho.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Revert "SMTP with STARTTLS is now identified as SMTPS"

This reverts commit 52d987b603.

* Revert "Compilation fix"

This reverts commit c019946f60.

* Sync unit tests.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-07-06 12:40:25 +02:00
Luca Deri c019946f60 Compilation fix 2022-07-05 17:38:32 +02:00
Ivan Nardi e2cc08bfe5 Add support for PIM (Protocol Indipendent Multicast) protocol (#1599)
Close #1598
2022-06-15 12:25:26 +02:00
Luca Deri 11babc7ea3 Added Pragmatic General Multicast (PGM) protocol detection 2022-06-08 09:11:22 +02:00
Toni 7419cfee64 Added RSH dissector. Fixes #202. (#1581)
- added syslog false-positive pcap that was missing in 09fbe0a64a
 - added NDPI_ARRAY_LENGTH() macro, usable on `type var[]` declarations

Signed-off-by: lns <matzeton@googlemail.com>
2022-06-04 19:12:53 +02:00
Toni Uhlig c3df3a12aa Fixed msys2 build warnings and re-activated CI Mingw64 build.
* Removed Visual Studio leftovers. Maintaining an autotools project with VS integration requires some additional overhead.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
2022-04-14 19:17:48 +02:00
Toni 95a3d4fffe Added autoconf option `--enable-tls-sigs'. (#1471)
* Testing more code in CI environments.
 * Added strict option checking for `./configure' in CI environments.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-03 10:16:08 +01:00
Toni 4da1453d90 Added configureable ndpi packet processing limit. (#1466)
* The current behaviour ignores any user preferences
   and was also incorrectly implemented, because the
   flow->num_processed_pkts wraps every 65535 and nDPI
   will process packets again until
   NDPI_MAX_NUM_PKTS_PER_FLOW_TO_DISSECT reached.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-03-02 12:02:09 +01:00
Ivan Nardi e193f3713d Removed some unused fields (#1461) 2022-02-28 15:03:49 +01:00
Luca Deri 533b3ab0e3 FreeBSD fixes 2022-01-13 21:15:21 +01:00
Ivan Nardi 8db8c8ae34 Fix QUIC log and remove SoulSeek leftovers after b97dc6ba (#1351)
Update .gitignore file
Fix a function prototype
Close #1349
2021-10-19 21:42:55 +02:00
Luca Deri 960e8ab22a Win fix 2021-10-11 20:59:28 +02:00
Luca Deri 246d115d7b Cleaned up code moving specific includes in files their are using it. Thi prevents ndpi_config.h to be included everywhere in apps using nDPI that might leade to #define redefinitions after the latest changes 2021-10-11 11:45:45 +02:00
eac7024489 Fix armhf (#1315)
* Fix unaligned memory accesses with get_u_int64_t at armhf

see: https://bugs.debian.org/993627

* Use get_u_int64_t to avoid unaligned memory access at armhf

see: https://bugs.debian.org/993627

* Update src/include/ndpi_define.h.in

Drop const type from get_u_int64_t, from lnslbrty

Co-authored-by: Bernhard Übelacker <bernhardu@mailbox.org>
Co-authored-by: Toni <matzeton@googlemail.com>
2021-09-26 23:53:00 +02:00
Ivan Nardi 7eb73bc92a QUIC: fix old GQUIC versions on big-endian machines (#1313) 2021-09-23 13:24:49 +02:00
Ivan Nardi 227e586e63 A (tiny) effort to reduce memory usage per flow (#1304)
Follow-up of 22241a1d
Only trivial changes:
* remove completely unused fields
* remove fields only written (but never read)
* CSGO protocol only handles UDP traffic
2021-09-18 09:29:45 +02:00
Luca Deri 36f08b8d2e Win fix 2021-09-07 22:37:22 +02:00
Luca Deri 821f4c9249 Fixed some invalid TLS guesses 2021-08-17 19:04:07 +02:00
Alfredo Cardigliano 846b546dbc Fix typo 2021-08-10 11:43:32 +02:00
Luca Deri c1acb6d801 Typo 2021-08-09 18:36:36 +02:00
Luca Deri 0fbe895103 Addef ETH_ARP definition 2021-08-09 18:33:33 +02:00
Luca Deri 05492ec8e8 Defined more standard ethernet protocol types 2021-08-09 17:01:10 +02:00
Luca Deri 58ca7b5a7f Added protocol defines 2021-08-09 16:45:03 +02:00
Ivan Nardi 57b8969a3d Fix setting of flow risks on 32 bit machines (#1251)
Since 19a29e1e (NDPI_TLS_CERT_VALIDITY_TOO_LONG is 32), unit tests are
failing on 32 bit machines (i.e Raspberry 4)
2021-07-19 16:22:39 +02:00
Luca Deri b0b3e1bc6c Added bitmap boundary checks 2021-06-17 10:49:11 +02:00
Luca Deri 1fb1848586 Added chec for avoiding doung ntop_config.h include 2020-12-01 08:57:09 +00:00
Luca Deri cbf1be5789 Macro redefinition 2020-09-22 12:23:44 +02:00
Adrian Zgorzałek 56a3a33bc0 OpenBSD: Do not redefine __LITTLE_ENDIAN__
Will silence omnipresent compiler warnings when building ntopng.
2020-08-09 14:30:12 +01:00
Luca Deri a828ac0191 Tiny changes for TLS block lenght dissection 2020-07-29 22:36:27 +02:00
Luca Deri 32bd3d7a59 TLS dissection improvements 2020-07-28 01:06:38 +02:00
Luca Deri 95263fc32e Added parentheses to avoid issues with macro expansion 2020-07-25 18:59:48 +02:00
Luca Deri db707e0829 Merge pull request #932 from IvanNardi/log
Log
2020-07-07 14:43:32 +02:00
Nardi Ivan 317d3ffd3e Fix undefined behaviour in internal tests
Error messages:
ndpiReader.c:3211:2: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'
ndpiReader.c:3207:5: runtime error: left shift of 1 by 31 places cannot be represented in type 'int'

The errors started popping up since 3d9285f1
2020-06-28 12:13:21 +02:00
Nardi Ivan 56d87186f7 Fix compilation with --enable-debug-messages flag
NDPI_LOG* macros dereference ndpi_detection_module_struct object which is
private to ndpi library (via NDPI_LIB_COMPILATION define). So we can't use
them outside the library itself, i.e. in ndpiReader code
Therefore, in files in example/, convert all (rare) uses of NDPI_LOG* macros
to a new very simple macro, private to ndpiReader program. If necessary,
such macro may be improved.

According to a comment in ndpi_define.h, each dissector must define its own
NDPI_CURRENT_PROTO macro before including ndpi_api.h file
2020-06-26 12:04:02 +02:00
Luca Deri 60aaa80570 Added HyperLogLog cardinality estimator API calls
/* Memory lifecycle */
int ndpi_hll_init(struct ndpi_hll *hll, u_int8_t bits);
void ndpi_hll_destroy(struct ndpi_hll *hll);

/* Add values */
void ndpi_hll_add(struct ndpi_hll *hll, const char *data, size_t data_len);
void ndpi_hll_add_number(struct ndpi_hll *hll, u_int32_t value) ;

/* Get cardinality estimation */
double ndpi_hll_count(struct ndpi_hll *hll);
2020-06-10 23:43:35 +02:00
Luca Deri 0b9e46e65a Updated (C) 2020-06-08 23:11:20 +02:00
Luca Deri 64fab4a478 Win fixes 2020-06-08 23:08:12 +02:00
Luca Deri 3d9285f1be Added check for invalid HTTP URLs 2020-05-16 00:10:35 +02:00