Commit Graph
290 Commits
Author SHA1 Message Date
Luca Deri 6b7e5fa8d2 Added legacy whatsapp rules used to match old pcaps that need to be removed in the future 2021-07-31 23:49:48 +02:00
Fábio Depin ce597b4806 Updated WhatsAPP and Instagram addresses (#1264)
* Updated WhatsAPP addresses

* Added Instagram addresses
2021-07-31 23:30:34 +02:00
Luca Deri a6de94a930 hulu and netflix dissection improvements 2021-07-14 12:53:03 +02:00
Vitaly Lavrov c418b7110b ahoсorasick. Code review. Part 2. (#1236)
Simplified the process of adding lines to AC_AUTOMATA_t.
Use the ndpi_string_to_automa() function to add patterns with domain names.
For other cases can use ndpi_add_string_value_to_automa().

ac_automata_feature(ac_automa, AC_FEATURE_LC) allows adding
and compare data in a case insensitive manner. For mandatory pattern comparison
from the end of the line, the "ac_pattern.rep.at_end=1" flag is used.
This eliminated unnecessary conversions to lowercase and adding "$" for
end-of-line matching in domain name patterns.

ac_match_handler() has been renamed ac_domain_match_handler() and has been greatly simplified.
ac_domain_match_handler() looks for the template with the highest domain level.
For special cases it is possible to manually specify the domain level.
Added test for checking ambiguous domain names like:
 - short.weixin.qq.com is QQ, not Wechat
 - instagram.faae1-1.fna.fbcdn.net is Instagram, not Facebook

If you specify a NULL handler when creating the AC_AUTOMATA_t structure,
then a pattern with the maximum length that satisfies the search conditions will be found
(exact match, from the beginning of the string, from the end of the string, or a substring).

Added debugging for ac_automata_search.
To do this, you need to enable debugging globally using ac_automata_enable_debug(1) and
enable debugging in the AC_AUTOMATA_t structure using ac_automata_name("name", AC_FEATURE_DEBUG).
The search will display "name" and a list of matching patterns.
Running "AHO_DEBUG=1 ndpiReader ..." will show the lines that were searched for templates
and which templates were found.

The ac_automata_dump() prototype has been changed. Now it outputs data to a file.
If it is specified as NULL, then the output will be directed to stdout.
If you need to get data as a string, then use open_memstream().

Added the ability to run individual tests via the do.sh script
2021-07-12 17:39:43 +02:00
Ivan Nardi 1116d0e4b1 Improve content match lists (#1226)
Add support for Likee app (https://likee.video/) and GitLab
Improve detection of Google domains ("gtv1.com" added in 6dd42d19f was
likely a typo)
Improve Snapchat and Tiktok detection
2021-07-05 09:42:11 +02:00
Luca Deri f19937c8c9 Addesses partial application matching (#1207) 2021-06-21 21:59:15 +02:00
Luca Deri 8761ac1887 Renamed Skyp in Skype_Teams as the protocol is now shared across these apps 2021-06-02 21:47:01 +02:00
Luca Deri abd6bce6f9 Added TLS certifiacate caching
Added Fortigate protocol
2021-05-15 10:52:16 +02:00
Luca Deri 66ebe444ae Added (partial) Activision protocol support (based on tencent cloud) 2021-05-10 15:41:52 +02:00
Luca 808050e635 Add extraction for TLS signature algorithms 2021-05-06 08:56:01 +02:00
Ivan Nardi d9278d55c3 Improve classification of Outlook/MicrosoftMail traffic (#1167)
See #1148
2021-04-18 21:37:26 +02:00
Luca Deri 0f8a994841 Improved DGA detection
Before
Accuracy 66%, Precision 86%, Recall 38%

After
Accuracy 71%, Precision 89%, Recall 49%
2021-03-03 19:30:01 +01:00
Luca Deri 56bfb439f8 Improved DGA detection with trigrams. Disadvantage: slower startup time
Reworked Tor dissector embedded in TLS (fixes #1141)
Removed false positive on HTTP User-Agent
2021-03-03 00:41:07 +01:00
Ivan Nardi c12a697f2d TOR: update node list (#1144)
https://panwdbl.appspot.com/lists/ettor.txt list is no more available

Close #1141
2021-03-02 21:14:36 +01:00
Luca Deri fc3db8f169 Implemented TLS Certificate Sibject matching
Improved AnyDesk detection
2021-02-22 22:37:33 +01:00
Luca Deri a2c5adc374 Improved nDPI string matching algorithm 2021-02-18 21:47:09 +01:00
Luca Deri d9da752aa8 Updated skype addresses 2021-02-07 19:05:56 +01:00
Luca Deri 3ecda45bc4 IP address matching update 2021-02-07 17:42:31 +01:00
Toni edf3a57a6a Added `fast.com' host pattern to Netflix detection. #1080 (#1084)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-12-11 21:01:28 +01:00
Luca Deri 38984af03e Updated disney definition 2020-11-22 18:49:28 +01:00
Luca Deri 21a6112b91 Added 129.205.94.0/23 to Facebook network list 2020-11-22 11:33:31 +01:00
Luca Deri acffd7bd1e Compilation fix 2020-11-16 21:22:54 +01:00
Zied AouiniandLuca Deri bfabb0ddf4 Add Virtual Asssitant (Alexa, Siri) support. (#1057)
* Add AmazonAlexa protocol.

* Add AmazonAlexa test file and result.

* Include pcapng as file format.

* Rename Category to VirtualAssistant.

* Add AppleSiri virtual assistant.

* Fix pcapng test files format support.

Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2020-11-16 21:19:38 +01:00
Oleksandr Natalenko 5c8e125f06 lib/ndpi_content_match: update TOR IPs (#1063)
Signed-off-by: Oleksandr Natalenko <oleksandr@natalenko.name>
2020-11-16 21:18:17 +01:00
Zied Aouini 76bb83085b Improve subprotocols detection. (#1062)
* Improve Spotify detection.

* Improve Skype detection.

* Improve Microsoft detection.

* Fix Microsoft detection categories.

* Improve Waze detection.

* Improve Apple detection.

* Improve WindowsUpdate detection.

* Improve TikTok detection.

* Improve Teams detection.

* Improve Youtube detection.

* Improve Messenger detection.

* Improve Twitch detection.

* Improve Hulu detection.

* Improve Facebook detection.

* Improve AmazonVideo detection.
2020-11-16 21:16:38 +01:00
Zied AouiniandLuca Deri 3529268df8 Add Tumblr support. (#1061)
* Add Tumblr protocol.

* Add Tumblr test file and result.

Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2020-11-16 21:14:06 +01:00
Zied AouiniandLuca Deri 22780da8d5 Add Reddit support. (#1060)
* Add Reddit protocol.

* Add Reddit test file and result.

Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2020-11-16 21:13:01 +01:00
Zied AouiniandLuca Deri 13dab51cc7 Add Pinterest support. (#1059)
* Add Pinterest protocol.

* Add Pinterest test file and result.

Co-authored-by: Luca Deri <lucaderi@users.noreply.github.com>
2020-11-16 21:11:43 +01:00
Zied Aouini f35fad13c3 Add DisneyPlus protocol. (#1058) 2020-11-16 21:10:11 +01:00
DouglasandDouglas Wordell efb712c2c0 Added new IPs for TeamViewer (#1040)
Co-authored-by: Douglas Wordell <douglas.wordell@seti.com.br>
2020-10-21 15:09:14 +02:00
Toni Uhlig 41d35fc7d0 Updated DoT/DoH provider.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-09-07 20:22:45 +02:00
Luca Deri 98a9afc40c Added support for discord 2020-08-16 10:01:40 +02:00
Luca Deri d5cac570d6 Improved DGA detection algoritm 2020-08-11 17:13:40 +02:00
Luca Deri e16675b700 Added new traffic category for connectivity check detection 2020-08-04 18:09:13 +02:00
Luca Deri 879cec94b2 User agent detection improvements 2020-07-21 12:06:34 +02:00
Toni Uhlig 8f53c231df Improved TeamViewer IP range (fixes #944)
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-07-02 18:11:39 +02:00
Toni Uhlig 2f5766d2f5 Added hulustream SNI pattern. Fixes #904.
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-06-23 17:30:37 +02:00
Luca Deri c9d8efc2a4 Warning fix 2020-06-22 10:20:05 +02:00
Luca Deri b2c24558c5 DGA detection improvements 2020-06-18 00:17:30 +02:00
Toni Uhlig da37f2444f Implemented proprietary AnyDesk protocol
Signed-off-by: Toni Uhlig <matzeton@googlemail.com>
2020-06-17 01:23:03 +02:00
Luca Deri 55364ef0b4 Added DGA risk for names that look like a DGA 2020-06-11 18:51:53 +02:00
Luca Deri 16ca69733a Improved teamviewer detection 2020-06-03 10:14:27 +02:00
Luca Deri 5afa3ad818 Improved spotify detection 2020-06-02 15:00:14 +02:00
Luca Deri 4ceff1dc77 Fixes for https://github.com/ntop/nDPI/pull/911
Added code for dumping invalid HTTP header
2020-05-29 21:23:46 +02:00
Luca Deri 030e9dddb7 Extended the cache for services that need to be reconciled such as Microsoft Teams
Added JSON-formatted Microsoft list of IP/services
2020-05-27 12:40:35 +02:00
Luca Deri 07d9e4f9ba Fixed valse positive whatsapp detection
Cleaned Microsoft IP addresses list
2020-05-20 23:28:21 +02:00
Luca Deri adfe6b763c Gotomeeting address range fix 2020-05-15 11:20:21 +02:00
Luca Deri 9ed94a722c Improvements on GotoMeeting
Added pcap for testing malware
2020-05-15 10:52:23 +02:00
Luca Deri 2ccd2c204b API cleanup for indetifying explicitly in automa's what we're searching (protocol or category)
Removed hyperscan support that is apperently unused
2020-05-06 23:19:59 +02:00
Luca Deri 4148c5e065 Removed now obsolete MSN protocol
Added nats.io protocol dissector
2020-05-03 18:20:21 +02:00