Commit Graph
4295 Commits
Author SHA1 Message Date
Ivan Nardi 8e6500a0da fuzz: extend fuzzing coverage (#2083) 2023-09-10 21:30:26 +02:00
Ivan Nardi ef6085370f fuzz: add fuzzers to test bitmap64 and domain_classify data structures (#2082) 2023-09-10 18:44:50 +02:00
Ivan Nardi 2c5e22123e Update protocols documentation (#2081) 2023-09-10 15:23:34 +02:00
Ivan Nardi 2a0052f25e fuzz: add fuzzers to test reader_util code (#2080) 2023-09-10 15:07:52 +02:00
Ivan Nardi 0aa1cf7245 Update every ip lists (#2079) 2023-09-10 12:13:20 +02:00
Ivan Nardi 2b883b93be Fix some errors found by fuzzers (#2078)
Fix compilation on Windows.
"dirent.h" file has been taken from https://github.com/tronkko/dirent/

Fix Python bindings

Fix some warnings with x86_64-w64-mingw32-gcc:
```
protocols/dns.c: In function ‘ndpi_search_dns’:
protocols/dns.c:775:41: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
  775 |       unsigned long first_element_len = (unsigned long)dot - (unsigned long)_hostname;
      |                                         ^
protocols/dns.c:775:62: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
  775 |       unsigned long first_element_len = (unsigned long)dot - (unsigned long)_hostname;
      |
```
```
In file included from ndpi_bitmap64.c:31:
third_party/include/binaryfusefilter.h: In function ‘binary_fuse8_hash’:
third_party/include/binaryfusefilter.h:160:32: error: left shift count >= width of type [-Werror=shift-count-overflow]
  160 |     uint64_t hh = hash & ((1UL << 36) - 1);
```
```
In function ‘ndpi_match_custom_category’,
    inlined from ‘ndpi_fill_protocol_category.part.0’ at ndpi_main.c:7056:16:
ndpi_main.c:3419:3: error: ‘strncpy’ specified bound depends on the length of the source argument [-Werror=stringop-overflow=]
 3419 |   strncpy(buf, name, name_len);
```
2023-09-10 11:09:59 +02:00
Luca Deri 805df2e5ce Reworked initialization 2023-09-10 09:36:54 +02:00
Luca Deri 63977459b6 Added OperaVPN detection 2023-09-09 18:17:26 +02:00
Luca Deri 076edeab54 Enhance DNS risk for long hostnames (> 32) 2023-09-09 18:09:57 +02:00
Luca Deri f50a4d7e85 Improved detection of invalid chars in DNS names 2023-09-09 17:46:13 +02:00
Luca Deri 1a797d7b74 Win fix 2023-09-08 12:24:27 +02:00
Luca Deri 770e460ba9 Added NDPI_TLS_ALPN_SNI_MISMATCH flow risk 2023-09-07 23:42:42 +02:00
Luca 47fb5e9f3d Cleanup 2023-09-07 11:12:37 +02:00
Luca Deri 2bd986a00a Disabled debug message 2023-09-05 18:39:26 +02:00
Luca Deri ad9ed4f954 Minor warning fixes 2023-09-05 18:08:16 +02:00
Luca Deri 978df906b3 Improved classification further reducing memory used 2023-09-05 17:03:20 +02:00
Luca Deri 0080d0d092 Compilation fixes for old compilers 2023-09-05 10:43:36 +02:00
Luca Deri feaaf83bed Added sub-domain classification fix 2023-09-05 07:21:48 +02:00
Luca Deri cf67f483f4 Classification fixes 2023-09-05 01:17:02 +02:00
Luca Deri c5ebd84fd8 Added ndpi_bitmap64 support 2023-09-05 00:21:16 +02:00
Luca Deri 4f2ce2d43b Added ndpi_murmur_hash to the nDPI API 2023-09-04 12:53:42 +02:00
Luca Deri f0dc3347ec Merged new and old version of ndpi_domain_classify.c code 2023-09-02 19:27:19 +02:00
Luca Deri 1d480c18e3 Reworked domain classification based on binary filters 2023-09-02 19:16:40 +02:00
Luca Deri 854c2d80f1 Improvement for reducing false positives 2023-09-01 10:26:07 +02:00
Luca Deri 74fdac7d77 Code cleanup 2023-09-01 08:50:55 +02:00
Luca Deri 133f45f3f3 Added ndpi_binary_bitmap datastruture
It is similar to ndpi_filter but based on binary search and with
the ability to store a category per value (as ndpi_domain_classify)
2023-08-31 23:51:50 +02:00
Luca Deri 19381f330a Fixed heap-buffer-overflow issue 2023-08-31 18:35:49 +02:00
Luca Deri 16b0ce3710 Code cleanup 2023-08-31 09:14:17 +02:00
Luca Deri f824939662 Added comment 2023-08-31 09:00:36 +02:00
Luca Deri 3c7b2a714e Added missing lists files 2023-08-29 19:21:57 +02:00
Luca Deri c5c610c941 Fixes dist 2023-08-29 18:49:43 +02:00
Luca Deri 65142b2a53 Added -G flag 2023-08-29 18:32:57 +02:00
Luca Deri 1de7fd9bf0 Warning fixes 2023-08-29 18:08:56 +02:00
Luca Deri 3dbc4f65fc Added README file 2023-08-29 17:45:36 +02:00
Luca Deri 36abf06c6f Swap from Aho-Corasick to an experimental/home-grown algorithm that uses a probabilistic
approach for handling Internet domain names.

For switching back to Aho-Corasick it is necessary to edit
ndpi-typedefs.h and uncomment the line
// #define USE_LEGACY_AHO_CORASICK

[1] With Aho-Corasick
$ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory
nDPI Memory statistics:
nDPI Memory (once):      37.34 KB
Flow Memory (per flow):  960 B
Actual Memory:           33.09 MB
Peak Memory:             33.09 MB

[2] With the new algorithm
$ ./example/ndpiReader -G ./lists/ -i tests/pcap/ookla.pcap | grep Memory
nDPI Memory statistics:
nDPI Memory (once):      37.31 KB
Flow Memory (per flow):  960 B
Actual Memory:           7.42 MB
Peak Memory:             7.42 MB

In essence from ~33 MB to ~7 MB

This new algorithm will enable larger lists to be loaded (e.g. top 1M domans
https://s3-us-west-1.amazonaws.com/umbrella-static/index.html)

In ./lists there are file names that are named as <category>_<string>.list
With -G ndpiReader can load all of them at startup
2023-08-29 17:34:04 +02:00
Toni Uhlig 1f693c3f5a Added lists/gambling.list to extra dist.
* make dist in `./fuzz`: fixed inconsistent `*.dict` file pattern

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-27 15:53:31 +02:00
Toni Uhlig afd250f5d9 fix compilation and symbol check
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-27 15:32:42 +02:00
Toni Uhlig 25f3a7cd1d fix compilation
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-27 15:17:56 +02:00
Luca Deri 34986de297 Search fixes 2023-08-26 19:47:50 +02:00
Luca Deri eeeee46b1e Changes for supporinng more efficient sub-string matching 2023-08-26 17:55:50 +02:00
Luca Deri 4ca94369e1 Improved domain search tet unit 2023-08-26 10:43:19 +02:00
Luca Deri ac00542df2 LEak fix 2023-08-26 10:10:49 +02:00
Luca Deri 2c565c77c9 Added ndpi_domain_classify_XXX(0 API 2023-08-26 00:24:33 +02:00
Luca Deri 42fcb011ad Warning fix 2023-08-25 19:29:43 +02:00
Luca Deri eb9932164f added bimap and/or with allocation 2023-08-24 07:45:38 +02:00
Luca Deri ff4f291afe Minor improvements 2023-08-23 11:04:31 +02:00
Luca 95fcd08687 Added ndpi_bitmap_is_empty() and ndpi_bitmap_optimize() API calls 2023-08-23 11:03:59 +02:00
Luca 9933b5910d Boundary check 2023-08-21 23:19:57 +02:00
Luca bd25df3334 Improved Wireguard detection 2023-08-21 13:40:42 +02:00
Luca 9e4bcc7771 Removed empty line 2023-08-21 13:40:30 +02:00