Unit:
Department of Informatics and TelecommunicationsΠληροφορική
Author:
KALOPISIS IOANNIS
Supervisors info:
ΝΤΟΥΛΑΣ ΑΛΕΞΑΝΔΡΟΣ, ΕΠΙΚΟΥΡΟΣ ΚΑΘΗΓΗΤΗΣ, ΤΜΗΜΑ ΠΛΗΡΟΦΟΡΙΚΗΣ ΚΑΙ ΤΗΛΕΠΙΚΟΙΝΩΝΙΩΝ, ΕΘΝΙΚΟ ΚΑΙ ΚΑΠΟΔΙΣΤΡΙΑΚΟ ΠΑΝΕΠΙΣΤΗΜΙΟ ΑΘΗΝΩΝ
Original Title:
Adjustable Publisher/Subscriber system with Machine Learning
Translated title:
Adjustable Publisher/Subscriber system with Machine Learning
Summary:
The rapid development of the Internet of Things-IoT leads to the development of many
distributed systems and smart applications. These applications generate and demand
huge amounts of data every day. It is therefore easily understood that a system is needed
to transfer this data. In order not to limit the development of large-scale applications, this
system should both be independent and have a decentralized character. This transfer
is undertaken by Publisher/Subscriber type messaging systems, such as Apache Kafka.
This system functions as the intermediate link between a producer and a consumer for
the transmission of messages.
These systems can be hosted on server clusters scattered around the world, depending on the size of the application they serve and the size of the data stream. We can understand that these are huge systems that adapt to the needs of the user. Therefore, the system parameters must be adjusted each time according to their application, use, type and data flow. However, apart from the tedious and time consuming process, the result does not always lead to optimal system performance.
In this project we present an attempt to automate the process of automatically optimizing
system performance for pub/sub systems using ML. By using algorithms and Machine
Learning techniques such as regression and classification, we try to predict the parameters of the Kafka Publisher/Subscriber system, aiming at specific system requirements.
You can find the code for this thesis, as well as the data, images, and results at the following link: https://github.com/GiannisKalopisis/Adjustable-pub-sub-system.
Main subject category:
Technology - Computer science
Keywords:
Machine Learning, Kafka, Publisher/Subscriber, Regression, Classification
