Commit Graph
4370 Commits
Author SHA1 Message Date
Ivan Nardi bdb73db1a4 IP lists: aggregate addresses wherever possible (#2152)
See #2150
2023-11-17 12:26:23 +01:00
Ivan Nardi 6c9571d9a9 Remove duplicate addreess list (#2151)
We are loading the same AS list as GOTO
See #2150
2023-11-16 20:08:06 +01:00
Maatuq 344ac69468 protocols/ethereum: enhance node discovery protocol detection. (#2145)
*  packet-type == 0x0 shouldn't be allowed.
* allow more packet-type e.g. ENRRequest, ENRResponse.

Signed-off-by: Mahmoud Maatuq <mahmoudmatook.mm@gmail.com>
2023-11-15 19:45:48 +01:00
Toni 38f9a74713 Added TeslaServices and improved TikTok host names. Fixes #2140. (#2144)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-10 11:25:10 +01:00
Toni 0cb6f4cb75 Fixed hash buffer size in ndpiSimpleIntegration. (#2143)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-10 10:23:37 +01:00
Ivan Nardi b08c787fe2 Have a clear distinction between public and private/internal API (#2137)
1) Public API/headers in `src/include/` [as it has always been]
2) Private API/headers in `src/lib/`

Try to keep the "ndpi_" prefix only for the public functions
2023-11-09 10:50:59 +01:00
Toni 0673da54b5 Fixed implicit u32 cast in ndpi_data_min() / ndpi_data_max(). (#2139)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-09 10:16:57 +01:00
Ivan Nardi b539b0d090 fuzz: improve coverage and remove dead code (#2135)
We are not able to remove custom rules: remove the empty stubs (which
originate from the original OpenDPI code).

`ndpi_guess_protocol_id()` is only called on the first packet of the
flow, so the bitmask `flow->excluded_protocol_bitmask` is always empty,
since we didn't call any dissectors yet.

Move another hash function to the dedicated source file.
2023-11-07 17:46:29 +01:00
Ivan Nardi 58a9e2d967 Remove struct ndpi_packet_struct from the public API (#2129)
Right now, the only instance of `struct ndpi_packet_struct` is embedded
into `struct ndpi_detection_module_struct`. Since the latter is a
private structure (because of `NDPI_LIB_COMPILATION` ) there is no way for
the application to get a pointer to `ndpi_struct->packet`.

Bottom line: the application can't use any API functions having `struct
ndpi_packet_struct *` as parameter. Remove them all (since they are
completly unused and unusable).

There are no public helper functions to initialize/populate/deinit a
`struct ndpi_packet_struct` object, so the application can't neither
create its own instance of this object.

Protect `struct ndpi_packet_struct` via the same define
`NDPI_LIB_COMPILATION`.
2023-11-07 15:30:56 +01:00
Luca Deri 0db12b1390 In case a pattern is redefined (for instance with protos.txt), the custom definition overwrites the previous one (so far it was ignored) 2023-11-02 20:46:14 +01:00
Toni 6dcecd73d3 Added malicious sites from the polish cert. (#2121)
* added handling of parsing errors

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-11-02 09:04:04 +01:00
Christian Marangi d5c9a16a48 Move from PCRE to PCRE2 (#2134)
Move from PCRE to PCRE2. PCRE is EOL and won't receive any security
updates anymore. Convert to PCRE2 by converting any function PCRE2 new
API.

Also update every entry in github workflows and README to point to the
new configure flag. (--with-pcre2)

Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
2023-11-01 14:16:49 +01:00
Ivan KapranovandIvan Kapranov 5a2666ce9e Fix proto_name and proto_id missmatch for Sina and SinaWeibo (#2131)
* minor fixes

fixed 'handle leak' in ndpi_load_malicious_sha1_file and removed the redundant comparison ndpi_search_eaq

* fix Stack overflow caused by invalid write in ndpi_automa_match_string_subprotocol

* fix compile errors

* fix

* Fix name missmatch for Sina and Sina Weibo

* fix

* add Sina Weibo to doc

* fix

* add Sina Weibo to doc

---------

Co-authored-by: Ivan Kapranov <i.kapranov@securitycode.ru>
2023-11-01 14:12:49 +01:00
Ivan Nardi cd59485f7e Custom rules: add missing check (#2132)
Fix an use-of-uninitialized-value error

```
==28646==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55a9b7a3dea2 in ndpi_patricia_lookup /home/ivan/svnrepos/nDPI/src/lib/third_party/src/ndpi_patricia.c:739:8
    #1 0x55a9b7442cac in add_to_ptree /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:2588:10
    #2 0x55a9b7469290 in ndpi_add_ip_risk_mask /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:4161:12
    #3 0x55a9b746cf14 in ndpi_handle_rule /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:4297:11
    #4 0x55a9b74842ae in ndpi_load_protocols_file_fd /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:4944:8
    #5 0x55a9b7424706 in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_filecfg_protocols.c:38:3
[...]
```

Found by oss-fuzz
See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=63754
2023-10-31 20:02:37 +01:00
Ivan Nardi 50006516b3 Custom rules: fixes for Windows (#2130)
Even after adding `WSAStartup()` call, the behavior with IPv6 addresses
was different on Windows; it is the same issue reported on:
https://stackoverflow.com/questions/66755681/getaddrinfo-c-on-windows-not-handling-ipv6-correctly-returning-error-code-1

Fix bitmasks.
2023-10-31 16:04:30 +01:00
Luca Deri 773e9b8170 Added pointer check 2023-10-31 15:48:06 +01:00
Luca Deri f49cedc1e8 Added missing freeaddrinfo() call 2023-10-30 22:58:17 +01:00
Luca Deri 5e9f10f820 Replaced gethostbyname2 with getaddrinfo (portable code) 2023-10-30 22:55:50 +01:00
Luca Deri c6af36990e Warning fixes 2023-10-30 13:10:28 +01:00
Ivan Nardi 42d24f8799 STUN: major code rework (#2116)
Try to have a faster classification, on first packet; use standard extra
dissection data path for sub-classification, metadata extraction and
monitoring.

STUN caches:
* use the proper confidence value
* lookup into the caches only once per flow, after having found a proper
STUN classification

Add identification of Telegram VoIP calls.
2023-10-30 10:28:19 +01:00
Ivan Nardi e399bd7e3d Custom rules: fix a stack overflow (#2128)
```
==19255==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f515bb3bf80 at pc 0x55796e01394a bp 0x7fff4fb5c050 sp 0x7fff4fb5b7e0
WRITE of size 58 at 0x7f515bb3bf80 thread T0
    #0 0x55796e013949 in scanf_common(void*, int, bool, char const*, __va_list_tag*) asan_interceptors.cpp.o
    #1 0x55796e0147df in __isoc99_sscanf (/home/ivan/svnrepos/nDPI/fuzz/fuzz_filecfg_protocols+0x77f7df) (BuildId: a88601afb2c538ead3968648f39b9aa4da53427c)
    #2 0x55796e0fc74a in ndpi_add_host_ip_subprotocol /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:2771:13
    #3 0x55796e0fb029 in ndpi_handle_rule /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:4411:16
    #4 0x55796e103738 in ndpi_load_protocols_file_fd /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:4901:8
    #5 0x55796e0ca96d in LLVMFuzzerTestOneInput /home/ivan/svnrepos/nDPI/fuzz/fuzz_filecfg_protocols.c:38:3
    #6 0x55796dfd78e0 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_filecfg_protocols+0x7428e0) (BuildId: a88601afb2c538ead3968648f39b9aa4da53427c)
    #7 0x55796dfc0e93 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_filecfg_protocols+0x72be93) (BuildId: a88601afb2c538ead3968648f39b9aa4da53427c)
    #8 0x55796dfc6d96 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/ivan/svnrepos/nDPI/fuzz/fuzz_filecfg_protocols+0x731d96) (BuildId: a88601afb2c538ead3968648f39b9aa4da53427c)
    #9 0x55796dff1672 in main (/home/ivan/svnrepos/nDPI/fuzz/fuzz_filecfg_protocols+0x75c672) (BuildId: a88601afb2c538ead3968648f39b9aa4da53427c)
    #10 0x7f515df19082 in __libc_start_main /build/glibc-BHL3KM/glibc-2.31/csu/../csu/libc-start.c:308:16
    #11 0x55796dfbbb0d in _start (/home/ivan/svnrepos/nDPI/fuzz/fuzz_filecfg_protocols+0x726b0d) (BuildId: a88601afb2c538ead3968648f39b9aa4da53427c)

Address 0x7f515bb3bf80 is located in stack of thread T0 at offset 128 in frame
    #0 0x55796e0fb977 in ndpi_add_host_ip_subprotocol /home/ivan/svnrepos/nDPI/src/lib/ndpi_main.c:2703

  This frame has 4 object(s):
    [32, 36) 'pin' (line 2705)
    [48, 64) 'pin6' (line 2706)
    [80, 96) 'd' (line 2769)
    [112, 128) 'tail' (line 2770) <== Memory access at offset 128 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow asan_interceptors.cpp.o in scanf_common(void*, int, bool, char const*, __va_list_tag*)
Shadow bytes around the buggy address:

```
2023-10-30 09:21:57 +01:00
Luca Deri 8769a2282b Disbling AES-NI on FreeBSD as the feature checke does not work well
and it enables it by mistake on IntelB. Coreb" i3-3240 Processor
2023-10-30 00:05:25 +01:00
Luca Deri 9fb7a63576 FreeBSD compilation fix (missingcvg #include <netdb.h>) 2023-10-29 23:18:17 +01:00
Luca Deri 76829b413f Implements support for symbolic host names (#2123) 2023-10-29 22:54:45 +01:00
Luca Deri b8e0602bfd Added check 2023-10-29 22:54:45 +01:00
Ivan Nardi 2c2eaf8e79 Rename some functions with more useful/clear names (#2127) 2023-10-29 20:21:24 +01:00
Ivan Nardi 03fd155ae3 IPv6: add support for custom categories (#2126) 2023-10-29 12:56:44 +01:00
Ivan Nardi 32b50f5aa4 IPv6: add support for IPv6 risk exceptions (#2122) 2023-10-29 12:14:20 +01:00
Ivan Nardi c711251578 IPv6: add support for custom rules (#2120) 2023-10-29 11:26:35 +01:00
Ivan Nardi e8e4b9e8ff IPv6: add support for IPv6 risk tree (#2118)
Fix the script to download crawler addressess
2023-10-27 13:58:15 +02:00
Toni ed17f4d658 Improved Protobuf dissector. (#2119)
* tag extraction/validation was done wrong

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-27 10:19:47 +02:00
Ivan Nardi 8b07be4b9f Jabber: remove support for UDP (#2115)
Jabber/XMPP is only over TCP (even the name `ndpi_search_jabber_tcp`
suggests that...).

Bug introduced in 5266c726f
2023-10-26 20:16:27 +02:00
Ivan Nardi 611c3b66f0 ipv6: add support for ipv6 addresses lists (#2113) 2023-10-26 20:15:44 +02:00
Luca Deri 1832d247b3 Tool for creating bitcoing IP files 2023-10-26 00:01:44 +02:00
Toni 9509cd507a Fixed endian issue while DEBUG_PROTOBUF is enabled. (#2112)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-25 13:11:17 +02:00
Maatuq 4a8e7105b2 add ethereum protocol dissector. (#2111)
as explained here for bitcoin https://www.ntop.org/guides/nDPI/protocols.html#ndpi-protocol-bitcoin
the same is applicable for ethereum.
ethereum detection was removed from mining protocol and is now handled separately.

Signed-off-by: Mahmoud Maatuq <mahmoudmatook.mm@gmail.com>
2023-10-25 12:44:33 +02:00
Toni e70333de87 Added generic Google Protobuf dissector. (#2109)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-24 12:18:31 +02:00
Luca Deri 71d768d036 Improved detection as non DGA for hostnames belnging to a CDN (#2068) 2023-10-23 17:49:50 +02:00
Toni Uhlig a443bba0dd Add CAN over Ethernet dissector.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-23 13:45:56 +02:00
Toni Uhlig 25c54dd6d7 Improved CryNetwork protocol dissector.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-23 13:44:34 +02:00
Luca Deri 7122599287 Added CLA section 2023-10-23 12:05:56 +02:00
Luca Deri 23759c6d2d Moved to 4.9 2023-10-23 11:46:04 +02:00
Luca Deri 2a99e16728 Added 4.8 changeling 2023-10-23 11:38:46 +02:00
Toni Uhlig f69909d49b Add Remote Management Control Protocol (RMCP).
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-19 19:50:57 +02:00
Alfredo Cardigliano c59fe0b2b5 PR template 2023-10-19 09:29:38 +02:00
Toni 01f384f7ff Improved Steam detection by adding steamdiscover pattern. (#2105)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-17 22:13:02 +02:00
Toni 32b0aeea59 Fixed OpenWRT arm related build issues. (#2104)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-16 19:49:50 +02:00
Toni d75a8d339a Fixed two OpenWRT arm related build issues. (#2103)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2023-10-16 18:14:28 +02:00
Luca Deri d9b0ba1d14 Windows warning checks 2023-10-16 16:17:41 +02:00
Nardi Ivan 16b4913be6 fuzz: extend fuzzing coverage 2023-10-15 12:00:26 +02:00