Commit Graph
4301 Commits
Author SHA1 Message Date
Ivan Nardi c5cd4df056 Add ndpi_domain_classify_finalize() function (#2084)
The "domain classify" data structure is immutable, since it uses
"bitmap64".
Allow to finalize it before starting to process packets (i.e. before
calling `ndpi_domain_classify_contains()`) to avoid, in the data-path,
all the memory allocations due to compression.

Calling `ndpi_domain_classify_finalize()` is optional.
2023-09-12 19:47:14 +02:00
Thomas Winter 5bcf3c2ddd tftp: update pcap results
The two malformed TFTP packets are no longer considered as risk
and instead match by port only.
This is because the TFTP detection was rather sparse so could
match on several other protocols if the first two opcode bytes
happened to match.
2023-09-12 13:12:14 +02:00
Thomas Winter a5f0cb7e6c tftp: check incrementation for DATA and ACK packets
The 2 bytes following the opcode for DATA and ACK packets are the
block number and this should be incrementing every packet.
We should check to see that this is occurring otherwise false matches
can occur, eg L2TPv3 over UDP matches the DATA opcode but the next two
bytes are always zero.

Remove the DATA max block size assumption since this can be false if
the blksize option is used to increase it.

Fixes #2070
2023-09-12 13:12:14 +02:00
Thomas Winter 7263562514 tftp: check for Option Acknowledgements
TFTP Option Acknowledgement packets were being excluded.
When a read/write request contains options, an Option Acknowledgement
is returned that contains the option strings that the transaction
will use.
The options sent in the request are not compared with what was
acknowledged.
2023-09-12 13:12:14 +02:00
Thomas Winter 1e9517ab74 tftp: rework request checking to account for options
Read and write requests were not handling options.
The existing code inspecting request messages assumed that the string
before the end of the payload was the mode and that the filename
length depended on the mode length.
However the first two strings are the filename and mode which can be
followed by any number of option strings.

Rework the checking of the filename and mode to just search the first
two strings which should always exist. Any options after that are
ignored.

Absence of the filename or mode is now excludes the TFTP protocol.
Absence of the filename no longer is considered malformed packet
because it can match other protocols falsely.
2023-09-12 13:12:14 +02:00
Luca Deri 1bf7e5face Fixes matches with domain name strings that start with a dot 2023-09-11 22:50:03 +02:00
Ivan Nardi 8e6500a0da fuzz: extend fuzzing coverage (#2083) 2023-09-10 21:30:26 +02:00
Ivan Nardi ef6085370f fuzz: add fuzzers to test bitmap64 and domain_classify data structures (#2082) 2023-09-10 18:44:50 +02:00
Ivan Nardi 2c5e22123e Update protocols documentation (#2081) 2023-09-10 15:23:34 +02:00
Ivan Nardi 2a0052f25e fuzz: add fuzzers to test reader_util code (#2080) 2023-09-10 15:07:52 +02:00
Ivan Nardi 0aa1cf7245 Update every ip lists (#2079) 2023-09-10 12:13:20 +02:00
Ivan Nardi 2b883b93be Fix some errors found by fuzzers (#2078)
Fix compilation on Windows.
"dirent.h" file has been taken from https://github.com/tronkko/dirent/

Fix Python bindings

Fix some warnings with x86_64-w64-mingw32-gcc:
```
protocols/dns.c: In function ‘ndpi_search_dns’:
protocols/dns.c:775:41: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
  775 |       unsigned long first_element_len = (unsigned long)dot - (unsigned long)_hostname;
      |                                         ^
protocols/dns.c:775:62: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
  775 |       unsigned long first_element_len = (unsigned long)dot - (unsigned long)_hostname;
      |
```
```
In file included from ndpi_bitmap64.c:31:
third_party/include/binaryfusefilter.h: In function ‘binary_fuse8_hash’:
third_party/include/binaryfusefilter.h:160:32: error: left shift count >= width of type [-Werror=shift-count-overflow]
  160 |     uint64_t hh = hash & ((1UL << 36) - 1);
```
```
In function ‘ndpi_match_custom_category’,
    inlined from ‘ndpi_fill_protocol_category.part.0’ at ndpi_main.c:7056:16:
ndpi_main.c:3419:3: error: ‘strncpy’ specified bound depends on the length of the source argument [-Werror=stringop-overflow=]
 3419 |   strncpy(buf, name, name_len);
```
2023-09-10 11:09:59 +02:00
Luca Deri 805df2e5ce Reworked initialization 2023-09-10 09:36:54 +02:00
Luca Deri 63977459b6 Added OperaVPN detection 2023-09-09 18:17:26 +02:00
Luca Deri 076edeab54 Enhance DNS risk for long hostnames (> 32) 2023-09-09 18:09:57 +02:00
Luca Deri f50a4d7e85 Improved detection of invalid chars in DNS names 2023-09-09 17:46:13 +02:00
Luca Deri 1a797d7b74 Win fix 2023-09-08 12:24:27 +02:00
Luca Deri 770e460ba9 Added NDPI_TLS_ALPN_SNI_MISMATCH flow risk 2023-09-07 23:42:42 +02:00
Luca 47fb5e9f3d Cleanup 2023-09-07 11:12:37 +02:00
Luca Deri 2bd986a00a Disabled debug message 2023-09-05 18:39:26 +02:00
Luca Deri ad9ed4f954 Minor warning fixes 2023-09-05 18:08:16 +02:00
Luca Deri 978df906b3 Improved classification further reducing memory used 2023-09-05 17:03:20 +02:00
Luca Deri 0080d0d092 Compilation fixes for old compilers 2023-09-05 10:43:36 +02:00
Luca Deri feaaf83bed Added sub-domain classification fix 2023-09-05 07:21:48 +02:00
Luca Deri cf67f483f4 Classification fixes 2023-09-05 01:17:02 +02:00
Luca Deri c5ebd84fd8 Added ndpi_bitmap64 support 2023-09-05 00:21:16 +02:00
Luca Deri 4f2ce2d43b Added ndpi_murmur_hash to the nDPI API 2023-09-04 12:53:42 +02:00
Luca Deri f0dc3347ec Merged new and old version of ndpi_domain_classify.c code 2023-09-02 19:27:19 +02:00
Luca Deri 1d480c18e3 Reworked domain classification based on binary filters 2023-09-02 19:16:40 +02:00
Luca Deri 854c2d80f1 Improvement for reducing false positives 2023-09-01 10:26:07 +02:00
Luca Deri 74fdac7d77 Code cleanup 2023-09-01 08:50:55 +02:00
Luca Deri 133f45f3f3 Added ndpi_binary_bitmap datastruture
It is similar to ndpi_filter but based on binary search and with
the ability to store a category per value (as ndpi_domain_classify)
2023-08-31 23:51:50 +02:00
Luca Deri 19381f330a Fixed heap-buffer-overflow issue 2023-08-31 18:35:49 +02:00
Luca Deri 16b0ce3710 Code cleanup 2023-08-31 09:14:17 +02:00
Luca Deri f824939662 Added comment 2023-08-31 09:00:36 +02:00
Luca Deri 3c7b2a714e Added missing lists files 2023-08-29 19:21:57 +02:00
Luca Deri c5c610c941 Fixes dist 2023-08-29 18:49:43 +02:00
Luca Deri 65142b2a53 Added -G flag 2023-08-29 18:32:57 +02:00
Luca Deri 1de7fd9bf0 Warning fixes 2023-08-29 18:08:56 +02:00
Luca Deri 3dbc4f65fc Added README file 2023-08-29 17:45:36 +02:00
Luca Deri 36abf06c6f Swap from Aho-Corasick to an experimental/home-grown algorithm that uses a probabilistic
approach for handling Internet domain names.

For switching back to Aho-Corasick it is necessary to edit
ndpi-typedefs.h and uncomment the line
// #define USE_LEGACY_AHO_CORASICK

[1] With Aho-Corasick
$ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory
nDPI Memory statistics:
nDPI Memory (once):      37.34 KB
Flow Memory (per flow):  960 B
Actual Memory:           33.09 MB
Peak Memory:             33.09 MB

[2] With the new algorithm
$ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory
nDPI Memory statistics:
nDPI Memory (once):      37.31 KB
Flow Memory (per flow):  960 B
Actual Memory:           7.42 MB
Peak Memory:             7.42 MB

In essence from ~33 MB to ~7 MB

This new algorithm will enable larger lists to be loaded (e.g. top 1M domans
https://s3-us-west-1.amazonaws.com/umbrella-static/index.html)

In ./lists there are file names that are named as <category>_<string>.list
With -G ndpiReader can load all of them at startup
2023-08-29 17:34:04 +02:00
Toni Uhlig 1f693c3f5a Added lists/gambling.list to extra dist.
* make dist in `./fuzz`: fixed inconsistent `*.dict` file pattern

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-27 15:53:31 +02:00
Toni Uhlig afd250f5d9 fix compilation and symbol check
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-27 15:32:42 +02:00
Toni Uhlig 25f3a7cd1d fix compilation
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-27 15:17:56 +02:00
Luca Deri 34986de297 Search fixes 2023-08-26 19:47:50 +02:00
Luca Deri eeeee46b1e Changes for supporinng more efficient sub-string matching 2023-08-26 17:55:50 +02:00
Luca Deri 4ca94369e1 Improved domain search tet unit 2023-08-26 10:43:19 +02:00
Luca Deri ac00542df2 LEak fix 2023-08-26 10:10:49 +02:00
Luca Deri 2c565c77c9 Added ndpi_domain_classify_XXX(0 API 2023-08-26 00:24:33 +02:00
Luca Deri 42fcb011ad Warning fix 2023-08-25 19:29:43 +02:00