Commit Graph
3915 Commits
Author SHA1 Message Date
Ivan Nardi e674a8ae37 Sync unit tests results (#1783) 2022-10-22 12:57:06 +02:00
Luca Deri 24cc949f14 Enhanced HTTP numeric IP check 2022-10-22 10:06:09 +02:00
Toni c521595383 Add Elasticsearch protocol dissector. (#1782)
* all credits goes to @verzulli

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-10-21 20:01:54 +02:00
Toni Uhlig 29242cbcb6 Add Munin protocol dissector.
* all credits goes to @verzulli

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-10-20 22:10:35 +02:00
Nardi Ivan 9c0caa5362 TLS: allow sub-classification via ALPN
In some rare cases, it is possible to sub-classify the flow via ALPN
matching. This is particularly usefull for asymmetric traffic where the
Client Hello doens't have the SNI.

For the time being there is only one rule, about ANYDESK.
2022-10-20 22:00:12 +02:00
Nardi Ivan c1b6aa9483 Sync unit tests results 2022-10-19 13:01:34 +02:00
Luca Deri 9a34d870e9 Updated FortiClient default ports 2022-10-19 11:25:19 +02:00
Luca Deri df0dd76ed2 Reworked severities adding critical and emergency, thus shifting values down 2022-10-19 10:53:11 +02:00
Ivan Nardi 35fdbc8148 TLS: explicit ignore client certificate (#1776)
TLS classification usually stops after processing *server* certificates
(if any). That means, that *client* certificate, if present, is usually
ignored.
However in some corner cases (i.e. unidirectional traffic) we might end
up processing client certificate and exposing its metadata: the issue is
that the application will think that this metadata are about the server
and not about the client.
So, for the time being, always ignore client certificate processing.

As a future work, we might find an efficient way to process and export both
certificates.
2022-10-18 16:40:15 +02:00
Nardi Ivan 223a6fb9f7 Fix a use-of-uninitialized-value error on PCRE code
This is likely a false positive, triggered by the fact that libpcre is
usually compiled without MASAN support.
It it was a real error, ASAN would complain loudly with a invalid-free
error at the end of the same function.

```
==83793==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x562296111174 in ndpi_compile_rce_regex /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1631:3
    #1 0x5622960e3e4a in ndpi_is_rce_injection /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1636:5
    #2 0x5622960de7cd in ndpi_validate_url /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1741:12
    #3 0x5622960dae45 in ndpi_dpi2json /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1362:29
    #4 0x5622960e2751 in ndpi_flow2json /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1512:10
    #5 0x562296033b0f in process_ndpi_collected_info /home/ivan/svnrepos/nDPI/example/reader_util.c:1310:9
    #6 0x5622960501f9 in packet_processing /home/ivan/svnrepos/nDPI/example/reader_util.c:1659:2
    #7 0x562296045aef in ndpi_workflow_process_packet /home/ivan/svnrepos/nDPI/example/reader_util.c:2202:10
    #8 0x562295e85374 in ndpi_process_packet /home/ivan/svnrepos/nDPI/example/ndpiReader.c:3937:7
    #9 0x7f1235053466  (/lib/x86_64-linux-gnu/libpcap.so.0.8+0x23466) (BuildId: b84c893ea2516d6fb2c1c6726b1fe93b3be78f61)
    #10 0x7f1235041f67 in pcap_loop (/lib/x86_64-linux-gnu/libpcap.so.0.8+0x11f67) (BuildId: b84c893ea2516d6fb2c1c6726b1fe93b3be78f61)
    #11 0x562295e53139 in runPcapLoop /home/ivan/svnrepos/nDPI/example/ndpiReader.c:4060:15
    #12 0x562295e51e7f in processing_thread /home/ivan/svnrepos/nDPI/example/ndpiReader.c:4130:3
    #13 0x7f1234e53608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
    #14 0x7f1234d2f132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

  Uninitialized value was created by an allocation of 'pcreErrorStr' in the stack frame
    #0 0x5622961108a6 in ndpi_compile_rce_regex /home/ivan/svnrepos/nDPI/src/lib/ndpi_utils.c:1603:3
```
2022-10-17 23:15:07 +02:00
Nardi Ivan 3d0c36cf22 Add some popular streaming applications
A simply back-porting from https://github.com/vel21ripn/nDPI/pull/142;
full credits to @ChrisNelson-CyberReef.

Close #1716
2022-10-17 23:08:25 +02:00
Toni 01622c4c63 Updated protocol dissector serialisation code. (#1767)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-10-15 12:58:44 +02:00
Ivan Nardi 2d153fba45 HTTP: fix stack-buffer-overflow (#1768)
```
==24879==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fa085b31e60 at pc 0x55cc63f203e2 bp 0x7ffc9ec91b10 sp 0x7ffc9ec91298
READ of size 17 at 0x7fa085b31e60 thread T0
    #0 0x55cc63f203e1 in printf_common(void*, char const*, __va_list_tag*) asan_interceptors.cpp.o
    #1 0x55cc63f20769 in vsnprintf (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x50e769) (BuildId: cce2b6b1344bfd0bdc9626fef604c2b3caad485b)
    #2 0x55cc63f22210 in __interceptor_snprintf (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x510210) (BuildId: cce2b6b1344bfd0bdc9626fef604c2b3caad485b)
    #3 0x55cc6420fc76 in ndpi_check_http_server /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:668:4
    #4 0x55cc6420344b in check_content_type_and_change_protocol /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:742:5
    #5 0x55cc642031ce in check_content_type_and_change_protocol /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:737:7
    #6 0x55cc641fac9f in ndpi_check_http_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1352:4
    #7 0x55cc641f2fd5 in ndpi_search_http_tcp /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:1461:3
    #8 0x55cc64085275 in check_ndpi_detection_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5580:6
    #9 0x55cc64085c87 in check_ndpi_tcp_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5628:12
    #10 0x55cc64085927 in ndpi_check_flow_func /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:5647:12
    #11 0x55cc64095fcb in ndpi_detection_process_packet /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:6458:15
    #12 0x55cc63fd08b4 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:29:5
    #13 0x55cc63fd09f7 in main /home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet.c:101:17
    #14 0x7fa0880fb082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
    #15 0x55cc63efb45d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_process_packet_with_main+0x4e945d) (BuildId: cce2b6b1344bfd0bdc9626fef604c2b3caad485b)

Address 0x7fa085b31e60 is located in stack of thread T0 at offset 96 in frame
    #0 0x55cc6420f1bf in ndpi_check_http_server /home/ivan/svnrepos/nDPI/src/lib/protocols/http.c:644

  This frame has 5 object(s):
    [32, 36) 'a' (line 653)
    [48, 52) 'b' (line 653)
    [64, 68) 'c' (line 653)
    [80, 96) 'buf' (line 654)
    [112, 176) 'msg' (line 662) <== Memory access at offset 96 partially underflows this variable

```
Found by oss-fuzzer
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=52229
2022-10-14 18:59:33 +00:00
Ivan Nardi 6c84ce85e4 ndpiReader: fix help message. There isn't a 'J' option (#1770) 2022-10-14 20:16:47 +02:00
Ivan Nardi 8c4ece941e Sync utests results (#1774) 2022-10-14 20:16:29 +02:00
Luca Deri 3bb6bf628d Removed gettimeofday as win files has located in windows/src/ 2022-10-14 19:01:41 +02:00
Luca Deri 1fecf69359 Added DNS error code mapping number -> string 2022-10-14 09:39:15 +02:00
Luca Deri cd76bacd4a Added detection of URLs starting with a dot (.) 2022-10-10 22:34:26 +02:00
Ivan Nardi 1796a1d814 LINE_CALL: add detection of LINE voip calls (#1761)
These flows are classifed as `LINE_CALL`; another option was
`RTP/LINE_CALL`. No sure about the best solution...

Extend LINE domains list.

Remove RTP dead code.
2022-10-06 17:09:26 +02:00
Luca Deri e4dcec560e Windows changes 2022-10-06 11:58:16 +02:00
Ivan Nardi c4fb2a590e irc: fix heap-buffer-overflow (#1765) 2022-10-06 09:25:21 +00:00
Luca Deri 3c0021d603 Fixes Stack-buffer-overflow READ {*} · check_content_type_and_change_protocol 2022-10-06 09:16:02 +02:00
Luca Deri fba926d6a4 Updated IRC result 2022-10-05 23:02:02 +02:00
Luca Deri 199c86e3df Improved credentials check in IRC 2022-10-05 22:51:20 +02:00
Luca 37c88b129f Added new flow risk NDPI_HTTP_OBSOLETE_SERVER. Currently Apache and nginx are supported 2022-10-04 22:32:45 +02:00
Nardi Ivan cca585053e Fix compilation and sync utests results 2022-10-04 22:17:05 +02:00
Luca 131d53896f Improved HTTP payload handling 2022-10-04 14:03:39 +02:00
Luca 3fa9416981 Improved detection of WebShell and PHP code in HTTP URLs that is reported via flow risk 2022-10-04 11:26:44 +02:00
Luca de59eb8237 Added the ability to track the payload via -E and via the new option 'ndpi_track_flow_payload' 2022-10-04 11:26:44 +02:00
Toni 66a8010ef7 Fix ndpi_timeval_to_milliseconds/microseconds for platforms with tv_usec is an unsigned int. (#1762)
* implicit conversion to an 32 bit unsigned int leads to an overflow

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-10-02 12:48:24 +02:00
Toni c83e0b3c8d Restore Confidence enum constants to ascending values. (#1760)
* NDPI_CONFIDENCE_MAX is useless otherwise since it is
   supposed the be the highest value w/o any gaps
 * for client applications, it makes sense to use that
   the numbers directly w/o any additional mapping

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-10-02 12:47:56 +02:00
sharonenochandSharon Enoch 503aac70bc Line app support (#1759)
* Standard support for LINE app

* Added test pcap for LINE app

* make check result for LINE app

* Make check success as 1kxun has LINE packets

* Added the ASN inc file for LINE

* Removed extra lines as its effecting make check

* Editing the SNI required a new pcap output file for TLS.Line format

* Run Configure with --with-pcre --with-maxminddb to enable the generation of h323-overflow.pcap.out

Co-authored-by: Sharon Enoch <sharone@amzetta.com>
2022-10-01 12:01:41 +02:00
Nardi Ivan f3a74d97d8 TLS/DTLS: we process certificate for UDP flows, too
Note that current code access `certificate_processed` state even before
setting the protocol classification, so this piece of information can't
be saved in `flow->protos` union.
2022-09-30 06:53:29 +02:00
Luca fbf30d1aad Fixed invalid unidirectional traffic alert for unidirectional protocols (e.g. sFlow) 2022-09-29 18:15:37 +02:00
Nardi Ivan a27538b764 Remove unused code
LRU callbacks have been added in 460ff3c7a, but they have never been
used and they have never been extended to the other LRU caches.

`ndpi_search_tcp_or_udp()` basically returns the classification by port/ip
of the flow; calling it from the dissector is useless.
The same for TOR detection: ips are checked in the generic code
2022-09-28 21:09:27 +02:00
Nardi Ivan ac7a3669da Fix value of ndpi_protocol->protocol_by_ip
Fix: a7c2734b
2022-09-25 13:31:57 +02:00
Nardi Ivan 39f9b8cadf DNS: change category of DNS flows
DNS flows should have `NDPI_PROTOCOL_CATEGORY_NETWORK` as category,
regardless of the subprotocol (if any).
Follow-up of 83de3e47
2022-09-25 13:31:04 +02:00
Toni Uhlig b7b9577254 Changed some protocol categories to align with others.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-23 20:35:55 +02:00
Nardi Ivan 1f345b311f Sizes of LRU caches are now configurable
0 as size value disable the cache.

The diffs in unit tests are due to the fact that some lookups are
performed before the first insert: before this change these lookups
weren't counted because the cache was not yet initialized, now they are.
2022-09-23 18:33:48 +02:00
Nardi Ivan 83de3e4716 DNS: change category of DNS flows
DNS flows should have `NDPI_PROTOCOL_CATEGORY_NETWORK` as category,
regardless of the subprotocol (if any).
2022-09-22 13:23:55 +02:00
Nardi Ivan b250f9d86c Extend content match lists 2022-09-22 09:30:09 +02:00
Nardi Ivan c6201284d7 NAT-PMP: fix metadata extraction 2022-09-21 20:02:23 +02:00
Toni 644ad34962 Improved NATPMP dissection. (#1745)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-21 18:24:04 +02:00
Toni Uhlig d6701e8979 Build ndpiReader and run regression tests.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
Signed-off-by: lns <matzeton@googlemail.com>
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-21 18:03:22 +02:00
segfault 3db6413ab0 Fixed VS2019 build.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-21 18:03:22 +02:00
Toni Uhlig 50ee6768b7 Provide extra ndpi_define.h for Visual Studio.
* Caution: file needs to keep in sync with `src/include/ndpi_define.h.in` manually

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-21 18:03:22 +02:00
Toni Uhlig d27fb1e73e Moved NDPI_API_VERSION to src/include/ndpi_define.h.in where it belongs.
That way, we can make `src/include/ndpi_api.h` independent of autoconf.
MSBuild will like that.

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-21 18:03:22 +02:00
segfault 817a7c616d Retargeted to Windows 10.
* disable Spectre Mitigation (may change)
 * remove autoconf generated header files
2022-09-21 18:03:22 +02:00
mark andrews 5d5b46e514 Add proj and sln for compile of dynamic x64 lib under Visual Studio 2019.
* add CI support via MSBuild

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2022-09-21 18:03:22 +02:00
Ivan Nardi 03d217eae6 Fix CI after nBPF integration (#1746)
Add one CI job testing nBPF
2022-09-21 16:49:51 +02:00