Novel Techniques for Detecting Tor Botnets

Abstract

Botnets are collections of infected computers that are controlled centrally by a botmaster, often for sending spam or launching denial of service attacks. The task of taking down these botnets is often a cat-and-mouse game, with operators frequently changing domains for their control infrastructure. More recently, some operators have moved to use Tor, a pseudo- anonymous network for hosting services, whereby identification is difficult. Additionally, because connections to the Tor network are encrypted, we cannot use traditional methods or protocol-reliant techniques like Domain Name System (DNS) and traffic signatures to detect infected hosts. This research reveals that the long-known behaviour of periodic communications is not alien to Tor botnets. This Thesis introduces a novel technique for finding and extracting periodicity-based features from Tor applications and botnets. It is specifically developed for Tor traffic and can detect up to 100% accuracy even in the midst of noise and some disrupted traffic. Interval Miner is then used to investigate the feature of periodicity in Tor applications and botnets. Our findings show that Tor botnets also produce unique characteristics like cell counts. This thesis then further introduces TorBot Stalker: the first mechanism for detecting, de-anonymising, and destroying Tor botnets. It uses machine learning to analyse and fingerprint the timings and frequency of Tor network cell/circuit data when routing botnet traffic. Several machine learning algorithms are experimented with for building a model and validation. The Random Forest and J.48 (C4.5) algorithms are chosen to develop a model capable of classifying Tor traffic into TCP botnet, telnet botnet, HTTP botnet, IRC chat, or Web traffic. It is also capable of classifying Tor traffic into legitimate traffic, botnet traffic or IRC traffic. Finally, it can reveal if a node in a botnet is a client or a hidden service. A detection mechanism that is able to identify infected hosts at the Tor network border, in real-time, while preserving the privacy of legitimate users is then developed. Experimental data demonstrate an accuracy of 99% with few false positives. We then apply the technique at the entry to the Tor network to measure the fraction of traffic which is for botnet. We observed that Torbot Stalker is able to de-anonymise real botnets in the Tor network and further identify infected hosts and control servers.

Date of Award	13 Dec 2023
Original language	English
Awarding Institution	University of Portsmouth
Supervisor	Ella Haig (Supervisor) & Stavros Shiaeles (Supervisor)