Speaker: Thomas Plagemann (University of Oslo)

Date: 5th of April, 2024

Time: 12:00pm

Place: Room A5.4

Presentation title: "Semantic and utility aware operator migration for Distributed Stream Processing"

Short bio:

Thomas Plagemann received the Dr. Sc. degree in computer science from the Swiss Federal Institute of Technology (ETH), Zurich, Switzerland, in 1994, with the Medal of ETH Zurich in 1995. He has been a Professor with Department of Informatics at the University of Oslo, Norway, since 1996. He has published over 200 papers in peer-reviewed journals, conferences, and workshops in his field. His research interests include multimodal sensor systems, distributed data stream processing, and mHealth. He is a member of the Association for Computing Machinery (ACM) and Associate Editor for the ACM Transactions on Multimedia Computing, Communication, and Applications. For 13 years he served as Editor-in-Chief for Multimedia Systems (Springer).

Abstract:

Operator migration is an essential adaptation mechanism in distributed stream processing systems. Its main challenge is to minimize the experienced latency for the data consumers and to avoid data loss. To improve on the state-of-the-art, we introduce an operator semantic aware migration mechanism, called Lazy Migration. The semantics of operators determines how their state is managed and partitioned, e.g., window extents for aggregation and tuple lists for join. There are two modes in Lazy Migration to migrate such state partitions: (1) for the latency mode we exploit, for example, the window semantics to migrate the parts exactly when the operators need them to minimize the output latency of tuples, and (2) in the utility mode each partial state is assigned a utility value based on its anticipated future demand. The utility mode is designed for situations in which it might be impossible to migrate the entire state in time to ensure that always the state parts with the highest utility are at the new host. Finally, we demonstrate join and aggregation scenarios where Lazy Migration excels, in addition to pros and cons of the other migration mechanisms.