Milosh V. Ivanovich & Chathuranga Widanapathirana

The Automation of Network Diagnostics: Making Statistical TCP Analysis and Machine Learning Play Along Nicely

In today's complex networks, timely identification and resolution of performance problems is extremely challenging, and has long been a source of pain and cost for network providers.

A very powerful approach which has traditionally been applied in this context, is the analysis of Transmission Control Protocol (TCP) traces. This is because TCP, as the most common transport layer protocol, dominates the Internet with >90% of the carried traffic and sits in a unique position near the top of the TCP/IP protocol stack. This combination offers us valuable insight into behaviour across all protocol layers, which is impossible to observe from any other single vantage point. Problems in the network impact typical TCP behaviour and embed unique "artefacts" in TCP packet streams. These artefacts can therefore be used as an excellent source of information to remotely diagnose a range of network performance issues, from the rather obvious to the very subtle. The fact that TCP-based analysis and inference is a sophisticated approach able to deal with extremely complex network problems, unfortunately comes at a price. As a cognitive process which strongly depends on a human investigator's experience, it can be said to be 'human intensive' and is thus a costly and unscalable exercise for network providers.

Enter: Machine Learning. In our work, we turn to this scientific discipline concerned with the design and development of algorithms that allow computers to evolve behaviours based on training data, and use it to automate TCP-based analysis and inference. Our ultimate goal is a fully automated and scalable system, capable of identifying complex network problems and their root causes, in a rapid and accurate manner. To this end, we propose several new techniques for the characterisation of what we define as "network soft failures", which combine together to give our "Intelligent Automated Network Diagnostic" (IAND) system. IAND has the capability to handle both known and unknown network performance problems, through the application of supervised and unsupervised learning respectively. Furthermore, as we will show, it is able to achieve detection accuracies up to 95% while achieving micro-second diagnosis times.


Milosh Ivanovich (IEEE SM '06) fills the role of Principal Domain Expert in Wireless Traffic Management, at Telstra's Wireless Networks Development & Innovation group, and is an Honorary Research Fellow at Monash University in Australia. A Senior Member of IEEE, Milosh's interests lie in queuing theory, teletraffic modeling, performance analysis of wireless networks, and the study and enhancement of TCP/IP in hybrid fixed/wireless environments. Milosh obtained a B.E. (1st class Hons) in Electrical and Computer Systems Engineering (1995), a Master of Computing (1996) and a Ph.D. in Information Technology (1998), all at Monash University Australia. He is the author of several edited book chapters, a patent, and over 50 international journal and conference publications.

Chath Widanapathirana is a Data Scientist and currently holds the position of Manager, Learning Analytics and Data Sciences in the Learning Technology Products division at Open Universities Australia. He is a former student of Monash University Ph.D. program in the Department of Electrical and Computer Systems Engineering and has submitted his thesis, currently awaiting results. Chath's interests lie in Machine Learning, Artificial Intelligence, Big Data, Performance Diagnostics and their industrial applications. Chath obtained his B.Eng (1st class Hons) in Telecommunications Engineering in 2009 from Multimedia University, Malaysia and submitted his Ph.D. thesis on "Intelligent Automated Network Diagnostics" in 2015. Chath has extensive experience in applying research on Machine Learning and Big Data into industry based products, product life cycle management and commercialization.He is also the author of number of international journal and conference publications.