ahoсorasick. Code review. Part 2. (#1236)

Simplified the process of adding lines to AC_AUTOMATA_t.
Use the ndpi_string_to_automa() function to add patterns with domain names.
For other cases can use ndpi_add_string_value_to_automa().

ac_automata_feature(ac_automa, AC_FEATURE_LC) allows adding
and compare data in a case insensitive manner. For mandatory pattern comparison
from the end of the line, the "ac_pattern.rep.at_end=1" flag is used.
This eliminated unnecessary conversions to lowercase and adding "$" for
end-of-line matching in domain name patterns.

ac_match_handler() has been renamed ac_domain_match_handler() and has been greatly simplified.
ac_domain_match_handler() looks for the template with the highest domain level.
For special cases it is possible to manually specify the domain level.
Added test for checking ambiguous domain names like:
 - short.weixin.qq.com is QQ, not Wechat
 - instagram.faae1-1.fna.fbcdn.net is Instagram, not Facebook

If you specify a NULL handler when creating the AC_AUTOMATA_t structure,
then a pattern with the maximum length that satisfies the search conditions will be found
(exact match, from the beginning of the string, from the end of the string, or a substring).

Added debugging for ac_automata_search.
To do this, you need to enable debugging globally using ac_automata_enable_debug(1) and
enable debugging in the AC_AUTOMATA_t structure using ac_automata_name("name", AC_FEATURE_DEBUG).
The search will display "name" and a list of matching patterns.
Running "AHO_DEBUG=1 ndpiReader ..." will show the lines that were searched for templates
and which templates were found.

The ac_automata_dump() prototype has been changed. Now it outputs data to a file.
If it is specified as NULL, then the output will be directed to stdout.
If you need to get data as a string, then use open_memstream().

Added the ability to run individual tests via the do.sh script
This commit is contained in:
Vitaly Lavrov
2021-07-12 17:39:43 +02:00
committed by GitHub
parent 78b1295dc1
commit c418b7110b
10 changed files with 341 additions and 295 deletions
+4 -1
View File
@@ -49,6 +49,7 @@
#include <math.h>
#include "ndpi_api.h"
#include "../src/lib/third_party/include/uthash.h"
#include "../src/lib/third_party/include/ahocorasick.h"
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>
@@ -1188,7 +1189,7 @@ void print_bin(FILE *fout, const char *label, struct ndpi_bin *b) {
/**
* @brief Print the flow
*/
static void printFlow(u_int16_t id, struct ndpi_flow_info *flow, u_int16_t thread_id) {
static void printFlow(u_int32_t id, struct ndpi_flow_info *flow, u_int16_t thread_id) {
FILE *out = results_file ? results_file : stdout;
u_int8_t known_tls;
char buf[32], buf1[64];
@@ -4326,6 +4327,8 @@ int original_main(int argc, char **argv) {
gettimeofday(&startup_time, NULL);
memset(ndpi_thread_info, 0, sizeof(ndpi_thread_info));
if(getenv("AHO_DEBUG"))
ac_automata_enable_debug(1);
parseOptions(argc, argv);
ndpi_info_mod = ndpi_init_detection_module(enable_ja3_plus ? ndpi_enable_ja3_plus : ndpi_no_prefs);