Commit Graph
59 Commits
Author SHA1 Message Date
Luca Deri 8dd7716ae5 Implemented more efficient and memory savvy RSI 2021-02-05 12:38:41 +01:00
Luca Deri 60b58dbd67 RSI enhancements 2021-02-05 10:59:09 +01:00
Luca Deri 1eedf734be Implemented API for computing RSI (Relative Strenght Index)
void  ndpi_init_rsi(struct ndpi_rsi_struct *s, u_int16_t num_learning_values);
void  ndpi_free_rsi(struct ndpi_rsi_struct *s);
float ndpi_rsi_add_value(struct ndpi_rsi_struct *s, const u_int32_t value);
2021-02-04 23:52:33 +01:00
Luca Deri 3e5e9569ff Added simple hash implementation to the nDPI API 2021-01-20 21:30:19 +01:00
Toni 53ee6db795 Added a new API function `ndpi_free_flow_data' which free's all members of ndpi_flow_struct but not the struct itself. (#1101)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2021-01-04 12:47:05 +01:00
Luca Deri 32f0446c9c Updated ndpi_ptree_match_addr() prototype 2020-12-30 18:23:40 +01:00
Luca Deri dc401f8a74 Split ptree user data in 32 and 64 bit entries 2020-12-30 12:12:33 +01:00
Luca Deri cf00ffed32 Removed test code 2020-12-26 18:13:10 +01:00
Luca Deri df1b3367a9 Type change to avoid Windows compilation issues 2020-12-17 12:58:12 +01:00
Luca Deri eb689b2069 nDPI rules (work in progress) implementation 2020-11-30 22:01:49 +01:00
Alfredo Cardigliano 031cd04b7c Add serialization of values list in TLV 2020-10-15 10:45:43 +02:00
Alfredo Cardigliano c92e0d13fd Serialized doxygen doc 2020-10-05 11:25:48 +02:00
Luca 05d93790e4 Added ndpi_quick_16_byte_hash
Warning fix
2020-10-05 08:26:24 +02:00
Luca Deri d0dbcf6dd6 Added back ndpi_check_flow_func (correct) prototype 2020-09-25 10:39:33 +02:00
Luca Deri 204f3ddff5 Compilation fix 2020-09-25 09:36:10 +02:00
Luca Deri 7086197047 Added extension to detect nested subdomains as used in Browsertunnel attack tool
https://github.com/veggiedefender/browsertunnel
2020-09-09 23:25:19 +02:00
Alfredo Cardigliano 842c00e681 Add missing low-level serializer calls to the API 2020-09-09 17:09:01 +02:00
Simone Mainardi f0325f713a Passes method_len param to ndpi_http_str2method 2020-08-27 10:05:38 +02:00
Luca Deri 2e8aa9f08e Added ndpi_http_method ndpi_http_str2method(const char* method) API call 2020-08-26 23:17:19 +02:00
Luca Deri 487d915262 Added ndpi_http_method2str() API call 2020-08-26 12:25:28 +02:00
Nardi Ivan d62ae567d1 Add (optional) dependency on external libraries: libgcrypt and libgpg-error
To support QUIC payload and header decryption, it is necessary to choose an
external crypto library to handle the low-level crypto stuff. Since we will
use some Wireshark code, it is quite natural to choose the same library used
by Wireshark itself: libgcrypt.

More precisely, we will use libgcrypt and libgpg-error.
Both libraries have LGPL license, so there should be no issue from this point
of view.

These libraries are not required to build nDPI, and their usage is optional:
nDPI will keep working (and compiling) even if they are not available.
However, without them, QUIC sub-classification is next to impossible.
The configure flag "--disable-gcrypt" forces the build system to ignore these
libraries.

libgpg-error is only used for debug to have meaningful error messages and its
usage is trivial.
The same cannot be said for libgcrypt because its initialization is a significant
issue.

The rest of this commit message try explaining how libgcrypt is
initialized.

According to the documentation
   https://gnupg.org/documentation/manuals/gcrypt/Initializing-the-library.html
   https://gnupg.org/documentation/manuals/gcrypt/Multi_002dThreading.html#Multi_002dThreading
libgcrypt must be initialized before using it, but such initialization should
be performed by the actual application and not by any library.

Forcing the users to proper initialize libgcrypt in their own code seems
unreasonable: most people using nDPI might be complete unaware of any crypto
stuff and update each and every one application linking to nDPI with specific
libgcrypt code should be out of question, anyway.

Fortunately, it seems a workaround exists to initialize libgcrypt in a library
    https://lists.gnupg.org/pipermail/gcrypt-devel/2003-August/000458.html

Therefore, we could provide a wrapper to this initialization stuff in a nDPI
function. Unfortunately nDPI API lacks a global init function that must be
called only once, before any other functions. We could add it, but that would
be a major API break.
AFAIK, ndpi_init_detection_module() might be called multiple times, for example
to create multiple independent dpi engines in the same program.

The proposed solution is to (optionally) initialize libgcrypt in
ndpi_init_detection_module() anyway:
* if the actual application doesn't directly use libgcrypt and only calls
  ndpi_init_detection_module() once, everything is formally correct and it
  should work out of the box [by far the most common user case];
* if the actual application already uses libgcrypt directly, it already
  performs the required initialization. In this case the ndpi_prefs.ndpi_dont_init_libgcrypt
  flag should be passed to ndpi_init_detection_module() to avoid further
  initializations.

The only scenario not supported by this solution is when the application is
unaware of libgcrypt and calls ndpi_init_detection_module() multiple times
concurrently. But this scenario should be uncommon.

A completely different option should be to switch to another crypto library,
with a huge impact on the QUIC dissector code.

Bottom line: crypto is hard, using libgcrypt is complex and the proposed
initialization, even if not perfect, should cover the most frequent user
cases and should work, for the time being.
If anyone has some suggestions...
2020-08-21 22:04:55 +02:00
Nardi Ivan 23ec82b59d Major rework of QUIC dissector
Improve support for GQUIC (up to Q046) and add support for Q050 and (IETF-)QUIC
Still no sub-classification for Q050 and QUIC
2020-08-21 22:04:55 +02:00
Luca Deri c2156a5161 Added note on memory management 2020-08-06 09:19:04 +02:00
Luca Deri 5b6ffad278 Added new ndpi_string_sha1_hash API call 2020-08-05 21:45:38 +02:00
Luca Deri 64fb40d21e Added ndpi_reset_data_analysis() API call 2020-07-17 13:16:01 +02:00
Luca Deri acd05b171e Added ndpi_data_last() API call 2020-07-16 11:22:06 +02:00
Luca Deri 46e97d5da3 Added ndpi_data_window_variance() and ndpi_data_window_stddev() API calls 2020-07-15 18:13:06 +02:00
Luca Deri 9f3e3e8456 Add ndpi_hll_reset() API call
Fixes bug in ndpi_data_window_average() with zero points
2020-07-15 10:30:45 +02:00
Toni Uhlig 23c0721538 Fixed race condition in ndpi_ssl_version2str() caused by static qualifier in the version string buffer.
* added also GREASE supported tls versions as specified in
   https://tools.ietf.org/html/draft-davidben-tls-grease-01#page-4

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-07-11 01:05:39 +02:00
Luca Deri 014fdd9a02 Various fixes in bins implementation
Added -b flag in ndpiReader to test bins
2020-07-09 17:28:02 +02:00
Luca Deri d658ac9e61 Added -b to ndpiReader to evaluate bins 2020-07-09 00:15:21 +02:00
Luca Deri 1c60c22893 Added ndpi_cluster_bins() for clustering bins and ancillary functions for bins manipulation 2020-07-07 15:10:51 +02:00
Luca Deri db707e0829 Merge pull request #932 from IvanNardi/log
Log
2020-07-07 14:43:32 +02:00
Luca Deri 392ce44573 Added ndpi_print_bin() API call 2020-06-29 22:36:46 +02:00
Luca Deri 2c263bc726 Added ndpi_bin_similarity() for computing bin similarity 2020-06-29 19:09:38 +02:00
Luca Deri 8de62c6d34 Values stored in patricia tree are now 32 bit (they used to be 16 bit) long 2020-06-26 17:49:45 +02:00
Nardi Ivan 56d87186f7 Fix compilation with --enable-debug-messages flag
NDPI_LOG* macros dereference ndpi_detection_module_struct object which is
private to ndpi library (via NDPI_LIB_COMPILATION define). So we can't use
them outside the library itself, i.e. in ndpiReader code
Therefore, in files in example/, convert all (rare) uses of NDPI_LOG* macros
to a new very simple macro, private to ndpiReader program. If necessary,
such macro may be improved.

According to a comment in ndpi_define.h, each dissector must define its own
NDPI_CURRENT_PROTO macro before including ndpi_api.h file
2020-06-26 12:04:02 +02:00
Luca Deri 1a62f4c799 Added ndpi_bin_XXX API
Added packet lenght distribution bins
2020-06-22 01:02:54 +02:00
Luca Deri 63670927e7 Fixed API documentation: packet tiestamp is expressed in milliseconds 2020-06-18 14:15:18 +02:00
Alfredo Cardigliano 073e3a027b Add API ndpi_serializer_get_format 2020-06-16 18:40:17 +02:00
Luca Deri 55364ef0b4 Added DGA risk for names that look like a DGA 2020-06-11 18:51:53 +02:00
Luca Deri 60aaa80570 Added HyperLogLog cardinality estimator API calls
/* Memory lifecycle */
int ndpi_hll_init(struct ndpi_hll *hll, u_int8_t bits);
void ndpi_hll_destroy(struct ndpi_hll *hll);

/* Add values */
void ndpi_hll_add(struct ndpi_hll *hll, const char *data, size_t data_len);
void ndpi_hll_add_number(struct ndpi_hll *hll, u_int32_t value) ;

/* Get cardinality estimation */
double ndpi_hll_count(struct ndpi_hll *hll);
2020-06-10 23:43:35 +02:00
Alfredo Cardigliano 67c72a8cb5 Add ndpi_serializer_get_header API (CSV only) 2020-06-05 16:52:26 +02:00
Luca Deri 07c54f9992 Added ndpi_dpi2json() API call 2020-06-05 16:08:23 +02:00
Luca Deri dc5b4cbde0 Reworked ndpi_strncasestr 2020-05-31 18:50:25 +02:00
Luca Deri f6038c358a Added ndpi_serialize_risk() API function 2020-05-27 17:35:11 +02:00
Alfredo Cardigliano 2dce6cd525 Add ndpi_serialize_start_of_list/ndpi_serialize_end_of_list to serialize simple lists in JSON 2020-05-25 16:37:00 +02:00
Toni Uhlig c5e16b4ff7 Fixed docstring typos for ndpi_finalize_initalization
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-05-17 16:59:57 +02:00
Luca Deri 3d9285f1be Added check for invalid HTTP URLs 2020-05-16 00:10:35 +02:00
Luca Deri e5e69d0f7a Added the ability to detect when a known protocol is using a non-standard port
Added check to spot executables exchanged via HTTP
2020-05-10 21:25:38 +02:00