Ivan Nardi ebcea42e2b fuzz: improve fuzzers using pl7m (#2486)
For some unclear reasons, fuzzers using pl7m create huge corpus,
triggering OOM in oss-fuzz runs (where the memory RSS limit is set to
2560Mb). Example:

```
==25340== ERROR: libFuzzer: out-of-memory (used: 2564Mb; limit: 2560Mb)
To change the out-of-memory limit use -rss_limit_mb=<N>

Live Heap Allocations: 2364004039 bytes in 133791 chunks; quarantined: 60662293 bytes in 3664 chunks; 176432 other chunks; total chunks: 313887; showing top 95% (at most 8 unique contexts)
1285841683 byte(s) (54%) in 2956 allocation(s)
    #0 0x56f814ef4bde in __interceptor_malloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3
    #1 0x56f814e04416 in operator new(unsigned long) cxa_noexception.cpp:0
    #2 0x56f814de6b2d in assign<unsigned char *, 0> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1443:3
    #3 0x56f814de6b2d in operator= /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1412:9
    #4 0x56f814de6b2d in fuzzer::InputCorpus::AddToCorpus(std::__Fuzzer::vector<unsigned char, std::__Fuzzer::allocator<unsigned char>> const&, unsigned long, bool, bool, bool, std::__Fuzzer::chrono::duration<long long, std::__Fuzzer::ratio<1l, 1000000l>>, std::__Fuzzer::vector<unsigned int, std::__Fuzzer::allocator<unsigned int>> const&, fuzzer::DataFlowTrace const&, fuzzer::InputInfo const*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerCorpus.h:221:10
    #5 0x56f814de60e5 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:539:16
    #6 0x56f814de7df2 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:829:7
    #7 0x56f814de8127 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:867:3
    #8 0x56f814dd6736 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:914:6
    #9 0x56f814e02c62 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #10 0x7fa11e2c3082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16

1031350683 byte(s) (43%) in 2468 allocation(s)
    #0 0x56f814ef4bde in __interceptor_malloc /src/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3
    #1 0x56f814e04416 in operator new(unsigned long) cxa_noexception.cpp:0
    #2 0x56f814de6b2d in assign<unsigned char *, 0> /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1443:3
    #3 0x56f814de6b2d in operator= /work/llvm-stage2/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/vector:1412:9
    #4 0x56f814de6b2d in fuzzer::InputCorpus::AddToCorpus(std::__Fuzzer::vector<unsigned char, std::__Fuzzer::allocator<unsigned char>> const&, unsigned long, bool, bool, bool, std::__Fuzzer::chrono::duration<long long, std::__Fuzzer::ratio<1l, 1000000l>>, std::__Fuzzer::vector<unsigned int, std::__Fuzzer::allocator<unsigned int>> const&, fuzzer::DataFlowTrace const&, fuzzer::InputInfo const*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerCorpus.h:221:10
    #5 0x56f814de60e5 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:539:16
    #6 0x56f814de7635 in fuzzer::Fuzzer::MutateAndTestOne() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:760:19
    #7 0x56f814de8425 in fuzzer::Fuzzer::Loop(std::__Fuzzer::vector<fuzzer::SizedFile, std::__Fuzzer::allocator<fuzzer::SizedFile>>&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:905:5
    #8 0x56f814dd6736 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:914:6
    #9 0x56f814e02c62 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #10 0x7fa11e2c3082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16
```

See: https://oss-fuzz.com/testcase-detail/4717811415449600
See: https://oss-fuzz.com/testcase-detail/6164130982068224

Let's *try* the following workaround: set the parameter `-max-len` to
512K, to force the engine to not genereate inputs (i.e. pcap files...)
larger than 512K. Right now the value used is 1MB, i.e the default,
because we have file larger than 1MB in the initial seeds (i.e.
`/tests/pcaps/*`).

Let's hope than having smaller files lead to smaller corpus...

Update pl7m code (fix a Use-of-uninitialized-value error)
2024-06-29 18:01:05 +02:00
2024-06-05 11:56:03 +02:00
2024-01-15 19:03:46 +01:00
2022-09-21 08:24:37 +02:00
2024-05-21 13:08:07 +02:00
2024-04-04 20:06:47 +02:00
2023-10-23 12:05:56 +02:00
2015-04-19 07:25:59 +02:00
2015-04-19 07:25:59 +02:00

nDPI

Build Status Fuzzing Status

What is nDPI ?

nDPI® is an open source LGPLv3 library for deep-packet inspection. Based on OpenDPI it includes ntop extensions. We have tried to push them into the OpenDPI source tree but nobody answered emails so we have decided to create our own source tree

A generic FAQ about nDPI® is available here

How To Compile nDPI

In order to compile this project do

  • ./autogen.sh
  • make

To compile the library w/o any tools or tests:

  • ./autogen.sh --with-only-libndpi
  • make

To run tests do additionally:

  • ./tests/do.sh # Generate and check for diff's in PCAP files
  • ./tests/do-unit.sh # Run unit tests
  • ./tests/do-dga.sh # Run DGA detection test

or run all with: make check

Please note that the (minimal) pre-requisites for compilation include:

  • GNU tools (autoconf automake libtool pkg-config gettext flex bison)
  • GNU C compiler (gcc) or Clang

On Debian/Ubuntu systems do:

  • sudo apt-get install build-essential git gettext flex bison libtool autoconf automake pkg-config libpcap-dev libjson-c-dev libnuma-dev libpcre2-dev libmaxminddb-dev librrd-dev

On Arch Linux:

  • sudo pacman -S gcc git gettext flex bison libtool autoconf automake pkg-config libpcap json-c numactl pcre2 libmaxminddb rrdtool

On FreeBSD:

  • sudo pkg install gcc git gettext flex bison libtool autoconf automake devel/pkgconf gmake libpcap json-c pcre2 libmaxminddb rrdtool

Remember to use gmake and not make on FreeBSD

On MacOS:

  • brew install coreutils gcc git gettext flex bison libtool autoconf automake pkg-config libpcap json-c pcre2 libmaxminddb rrdtool

On Windows:

There are three supported ways to build nDPI:

  1. MSYS2 (assuming MSYS2 already installed):
  • msys2 -c "pacman --noconfirm -S --needed --overwrite '*' git mingw-w64-x86_64-toolchain automake1.16 automake-wrapper autoconf libtool make mingw-w64-x86_64-json-c mingw-w64-x86_64-crt-git mingw-w64-x86_64-pcre2 mingw-w64-x86_64-libpcap"
  1. Mingw-w64

  2. Visual Studio (see windows/nDPI.sln)

Note: All Windows versions require npcap with WinPcap compatibility mode enabled.

How To Build The Documentation

  • pip install --upgrade pip
  • pip install -r doc/requirements.txt
  • make doc

Use the builtin python3 webserver to view documentation:

  • make doc-view

How To Add A New Protocol Dissector

The entire procedure of adding new protocols in detail:

  1. Add new protocol together with its unique ID to: src/include/ndpi_protocol_ids.h
  2. Create a new protocol in: src/lib/protocols/
  3. Variables to be kept for the duration of the entire flow (as state variables) need to be placed in: src/include/ndpi_typedefs.h in ndpi_flow_tcp_struct (for TCP only), ndpi_flow_udp_struct (for UDP only), or ndpi_flow_struct (for both).
  4. Add a new entry for the search function for the new protocol in: src/include/ndpi_protocols.h
  5. Choose (do not change anything) a selection bitmask from: src/include/ndpi_define.h
  6. Set protocol default ports in ndpi_init_protocol_defaults in: src/lib/ndpi_main.c
  7. Be sure to have nBPF support, cloning PF_RING in the same directory where you cloned nDPI: git clone https://github.com/ntop/PF_RING/ && cd PF_RING/userland/nbpf && ./configure && make. You can ignore the /bin/sh: 1: ../lib/pfring_config: not found error
  8. From the nDPI root directory, ./autogen.sh --with-pcre2 (nBPF and PCRE2 are usually optional, but they are needed to run/update all the unit tests)
  9. make
  10. make check
  11. Update the documentation, adding this new protocol to doc/protocols.rst
  12. Update the Windows Visual Studio configuration, adding the new c file in windows/nDPI.vcxproj

How to use nDPI to Block Selected Traffic

You can use nDPI to selectively block selected Internet traffic by embedding it onto an application (remember that nDPI is just a library). Both ntopng and nProbe cento can do this.

nDPI Paper Citation

Videos and Presentations

DISCLAIMER

While we do our best to detect network protocols, we cannot guarantee that our software is error free and 100% accurate in protocol detection. Please make sure that you respect the privacy of users and you have proper authorization to listen, capture and inspect network traffic.

nDPI is a registered trademark in the US and EU.

S
Description
No description provided
Readme
226 MiB
Languages
C 91.8%
Lua 2.3%
C++ 2%
Makefile 1.3%
Shell 1.1%
Other 1.5%