Commit Graph
2753 Commits
Author SHA1 Message Date
Toni Uhlig afd250f5d9 fix compilation and symbol check
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-27 15:32:42 +02:00
Toni Uhlig 25f3a7cd1d fix compilation
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-27 15:17:56 +02:00
Luca Deri 34986de297 Search fixes 2023-08-26 19:47:50 +02:00
Luca Deri eeeee46b1e Changes for supporinng more efficient sub-string matching 2023-08-26 17:55:50 +02:00
Luca Deri ac00542df2 LEak fix 2023-08-26 10:10:49 +02:00
Luca Deri 2c565c77c9 Added ndpi_domain_classify_XXX(0 API 2023-08-26 00:24:33 +02:00
Luca Deri 42fcb011ad Warning fix 2023-08-25 19:29:43 +02:00
Luca Deri eb9932164f added bimap and/or with allocation 2023-08-24 07:45:38 +02:00
Luca Deri ff4f291afe Minor improvements 2023-08-23 11:04:31 +02:00
Luca 95fcd08687 Added ndpi_bitmap_is_empty() and ndpi_bitmap_optimize() API calls 2023-08-23 11:03:59 +02:00
Luca 9933b5910d Boundary check 2023-08-21 23:19:57 +02:00
Luca bd25df3334 Improved Wireguard detection 2023-08-21 13:40:42 +02:00
Luca 9e4bcc7771 Removed empty line 2023-08-21 13:40:30 +02:00
Luca 6e5466f9e5 Added ndpi_bitmap_andnot API call 2023-08-21 13:26:38 +02:00
Ivan Nardi cc4461f424 fuzz: extend coverage (#2073) 2023-08-20 15:18:19 +02:00
Ivan Nardi 5867f43fae Fix compilation on Windows (#2072) 2023-08-14 22:51:09 +02:00
Luca Deri cfbb7a423e Replaces free() with ndpi_free() 2023-08-14 18:51:11 +02:00
Luca Deri 049fbf819c Reworked ndpi_filter_xxx implementation using compressed bitmaps 2023-08-14 12:50:54 +02:00
Luca Deri 3b34941be1 Typo 2023-08-12 11:39:19 +02:00
Luca Deri 9a44954365 ARM compilation fix 2023-08-12 07:39:35 +02:00
Luca Deri fe634b8d55 Fixed to address issue
Run ./utils/check_symbols.sh || { FAILED=$?; echo "::error file=${NDPI_LIB}::Unwanted libc symbols found: ${FAILED}. Please make sure to use only ndpi_malloc/ndpi_calloc/ndpi_realloc/ndpi_free wrapper instead of malloc/calloc/realloc/free."; false; }
[ndpi_filter.o]: calloc
Unwanted symbols found: 1
Please make sure to use only ndpi_malloc/ndpi_calloc/ndpi_realloc/ndpi_free wrapper instead of malloc/calloc/realloc/free
Error: Unwanted libc symbols found: 1. Please make sure to use only ndpi_malloc/ndpi_calloc/ndpi_realloc/ndpi_free wrapper instead of
2023-08-12 00:27:37 +02:00
Luca Deri 8e17f50853 Added ndpi_filter_add_multi() API call 2023-08-11 23:55:44 +02:00
Luca Deri 890dcf91d9 Minor initialization fixes 2023-08-11 23:51:16 +02:00
Luca Deri ec7adc212e Added new API calls for implementing Bloom-filter like data structures
ndpi_filter* ndpi_filter_alloc(uint32_t elements_number);
bool         ndpi_filter_add(ndpi_filter *f, uint64_t value);
bool         ndpi_filter_contains(ndpi_filter *f, uint64_t value);
void         ndpi_filter_free(ndpi_filter *f);
2023-08-11 22:49:31 +02:00
Luca 5fad72ebd9 Typo fix 2023-08-07 10:20:21 +02:00
Luca 33e9dbc24d Added ndpi_bitmap_copy() API call 2023-08-06 09:35:56 +02:00
snicket2100 1fbe8a2385 Mullvad VPN service added (based on entry node IP addresses) (#2062) 2023-08-02 19:44:16 +02:00
Luca Deri 196395398e Compilation fixes for older C compilers 2023-08-01 15:35:33 +02:00
Toni e4d3d619bc Add Service Location Protocol dissector. (#2036)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-01 08:50:46 +02:00
Toni 083275f617 Added slackb.com SNI. (#2067)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-08-01 08:39:10 +02:00
Ivan Nardi 5019022e13 DNS: extract geolocation information, if available (#2065)
The option NSID (RFC5001) is used by Google DNS to report the
airport code of the metro where the DNS query is handled.

This option is quite rare, but the added overhead in DNS code is pretty
much zero for "normal" DNS traffic
2023-07-31 07:44:43 +02:00
snicket2100 d477da997f Typo in a comment fixed (#2063) 2023-07-28 17:58:25 +02:00
Ivan Nardi bc91192aca ProtonVPN: split the ip list (#2060)
Use two separate lists:
* one for the ingress nodes, which triggers a ProtonVPN classification
* one for the egress nodes, which triggers the
`NDPI_ANONYMOUS_SUBSCRIBER` risk

Add a command line option (to `ndpiReader`) to easily test IP/port
matching.

Add another example of custom rule.
2023-07-27 09:05:22 +02:00
Ivan Nardi 3326fa258e Add an heuristic to detect fully encrypted flows (#2058)
A fully encrypted session is a flow where every bytes of the
payload is encrypted in an attempt to “look like nothing”.
The heuristic needs only the very first packet of the flow.
See: https://www.usenix.org/system/files/sec23fall-prepub-234-wu-mingshi.pdf

A basic, but generic, inplementation of the popcpunt alg has been added
2023-07-26 09:09:12 +02:00
Toni 2b230e28e0 Allow init of app protocols w/o any hostnames set. (#2057) 2023-07-22 14:51:33 +02:00
Toni 88a757c4cd Fix string truncation. (#2056)
ndpi_main.c: In function ‘ndpi_load_ip_category’:
ndpi_main.c:6598:3: warning: ‘strncpy’ specified bound 64 equals destination size [-Wstringop-truncation]
 6598 |   strncpy(ipbuf, ip_address_and_mask, sizeof(ipbuf));
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-07-22 12:45:45 +02:00
Ivan Nardi 7a1509bf60 zabbix: improve detection (#2055) 2023-07-21 03:42:36 +02:00
Ivan Nardi c85f2fb0f4 TLS: add basic, basic, detection of Encrypted ClientHello (#2053) 2023-07-21 03:41:43 +02:00
Ivan Nardi fa0bd515b5 Add detection of Roblox games (#2054) 2023-07-21 03:39:40 +02:00
Ivan Nardi 3edfad01a1 fuzz: extend fuzzing coverage (#2052)
Added/merged some traces.
Improved Socks identification
2023-07-18 07:41:56 +02:00
Ivan Nardi 5267858b3d HTTP: fix another memory access error (#2049)
```
=================================================================
==199079==ERROR: AddressSanitizer: negative-size-param: (size=-1)
    #0 0x559a2a6efd4f in strncpy (/home/ivan/svnrepos/nDPI/fuzz/fuzz_ndpi_reader+0x94ad4f) (BuildId: 34aaabba403c6bc5482553ef355360fd2762a157)
    #1 0x559a2a9890f0 in ndpi_http_check_content /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:300:8
    #2 0x559a2a9812c0 in check_content_type_and_change_protocol /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:910:46
    #3 0x559a2a978fee in process_response /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1289:3
    #4 0x559a2a97622f in ndpi_check_http_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1382:9
    #5 0x559a2a975d95 in ndpi_search_http_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1468:3
    #6 0x559a2a864970 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5948:4
    #7 0x559a2a8660df in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6013:12
    #8 0x559a2a865d7f in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6032:12
    #9 0x559a2a876fd6 in ndpi_internal_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7038:15
    #10 0x559a2a87311f in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7205:22
    #11 0x559a2a77381e in packet_processing /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:1710:31
    #12 0x559a2a77381e in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/fuzz/../example/reader_util.c:2427:10
[...]
```

Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=60605
2023-07-15 22:23:52 +02:00
Ivan Nardi 5811a5613b Add a configuration knob to enable/disable loading of gambling list (#2047) 2023-07-14 23:19:30 +02:00
Luca Deri fea09e825b Fixes risk mask exception handling while improving the overall performance 2023-07-14 19:52:34 +02:00
snicket2100 abee1a2a6f Included Gambling website data from the Polish hazard.mf.gov.pl list (#2041)
* Refreshed the Belgium Gambling Site list data

Unfortunately some hostnames have been removed from that list,
which means they are disappearing from the `ndpi_gambling_match.c.inc`
file as well.

* build: added `libxml2-utils` (for `xmllint`)

* Included Gambling website data from the Polish `hazard.mf.gov.pl` list

The list contains over 30k gambling website hostnames as of today.
2023-07-14 09:55:46 +02:00
Ivan Nardi 0dd1ee5656 HTTP: fix extraction of filename (#2046) 2023-07-14 01:33:00 +02:00
Luca Deri 33c42f9830 Cosmetic fixes 2023-07-13 23:26:35 +02:00
Luca Deri 1f55dc511f Implemented Count-Min Sketch [count how many times a value has been observed]
- ndpi_cm_sketch_init()
- ndpi_cm_sketch_add()
- ndpi_cm_sketch_count()
- ndpi_cm_sketch_destroy()
2023-07-13 21:54:51 +02:00
Ivan Nardi bd0fcb2e62 HTTP: fix heap-buffer-overflow (#2044)
```
=================================================================
==673246==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x5080000002fd at pc 0x555c6a22eca2 bp 0x7ffe1db9b290 sp 0x7ffe1db9aa48
READ of size 1 at 0x5080000002fd thread T0
    #0 0x555c6a22eca1 in strncpy (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet+0x53aca1) (BuildId: 464e7474cde3de91c01588c7ea0161a05ddd6b63)
    #1 0x555c6a462a9d in ndpi_http_check_content /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:309:8
    #2 0x555c6a45a9a0 in check_content_type_and_change_protocol /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:901:46
    #3 0x555c6a4526ce in process_response /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1280:3
    #4 0x555c6a44f90f in ndpi_check_http_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1373:9
    #5 0x555c6a44f475 in ndpi_search_http_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1459:3
    #6 0x555c6a348520 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5949:6
    #7 0x555c6a348f2f in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5997:12
    #8 0x555c6a348bcf in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6016:12
    #9 0x555c6a359e26 in ndpi_internal_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7017:15
    #10 0x555c6a355f6f in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:7184:22
    #11 0x555c6a284958 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:24:5
```

Found by oss-fuzz.
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=60557
Fix: 0b0f255cc
2023-07-13 10:50:59 +02:00
Ivan Nardi 0c5a17accb RDP: improve detection over UDP (#2043) 2023-07-13 10:50:13 +02:00
Alfredo Cardigliano 669a02cd76 Add support for multiline json 2023-07-12 10:47:49 +02:00