I gave a talk at the Olympia Research Summit

I gave a talk at the Olympia Research Summit in Athens, Greece.

Towards adaptive and resource-efficient data stream processing systems

Abstract: Streaming dataflow systems, like Apache Flink, have recently gained widespread adoption in industry and academia, while all major cloud providers offer data stream analytics as managed services. However, the increasing diversity of streaming applications and the availability of compute-enabled edge devices, hardware accelerators, and new cloud computing paradigms, is pushing existing dataflow systems to their limits. In this talk, I will describe our ongoing research towards addressing three major shortcomings of the streaming platforms status quo: (i) rigid, monolithic designs, inherited from MapReduce-style predecessors, (ii) homogeneous resource assumptions that hinder the deployment of emerging use cases, like machine learning pipelines and edge analytics, and (iii) notoriously complex to configure infrastructure.

I will present our recent research results on adaptive reconfiguration and resource-conscious optimizations for stream processing, such as workload-aware state management, task placement, and energy-performance tuning. Finally, I will share our vision to develop a novel fully-disaggregated streaming runtime architecture and discuss open challenges towards making data stream analytics accessible to domain experts, scientists, and small organizations.