Self-managed streaming systems

Stream processing research has come a long way and streaming systems have matured considerably since their invention, almost 3 decades ago. What will the next generation of stream processing systems look like? This is the question I have been working on answering with my most recent and ongoing work. We envision a next-generation of stream processing systems that are not only scalable and reliable, but also capable of self-management and automatic reconfiguration without downtime.

Performance analysis and modeling

As stream processing applications are long-running, it is a matter of time before any initial configuration becomes out-of-tune. If the system is not capable of adapting to workload changes, this might lead to data loss, idle resources, and SLO violations. To avoid such situations, we need to continuously monitor system operation, identify changing conditions, and react. See our recent work on understanding the performance of streaming dataflows by generalizing online critical path analysis.

Automatic optimization and reconfiguration

One of the biggest challenges streaming systems face is dealing with variance in input workloads. Contrary to batch processing and database management systems, a stream processing system has no control over the stream arrival rate (or order). How can streaming systems continuously satisfy QoS without wasting resources? Instead of using simplified predictive system models and externally observed noisy metrics, we proposed a white-box approach. DS2 is an automatic scaling controller that leverages system instrumentation and operator dependencies to extract accurate metrics and provide automatic elasticity with accuracy, stability, performance, and safety.

Workload-aware state management

Distributed instrumentation enables a set of reconfiguration decisions and enables stream processors to manage their resource allocation. This model relies on an external controller that continuously monitors the system’s performance and sends control commands to the stream processor’s cluster manager module. There are, however, additional optimizaiton opportunities if we look at how stream processing systems perform fundamental internal operations, such as state management. As streaming dataflow operators are instantiated once and are long-running, their access patterns and state size bounds are largely known in advance (or can be learned). We recently proposed workload-aware state management: using configurable state stores with support for different layouts and data types, and leveraging knowledge about operator state characteristics.

Graph streaming systems

Continuous analysis of graph streams is an emerging application area, where events indicate edge and vertex additions, deletions, and modifications. Even though there exist many specialized systems for dynamic graph processing, modern stream processors do not inherently support graph streaming use-cases. Graph streaming computations are challenging to implement with today’s stream processors because graph streaming algorithms do not nicely fit the dataflow model. Dataflows pushe data through a series of operators that apply transformations until they produce the end result. Such a model is not suitable for graph computations which instead require multiple passes over the graph state. This challenge can be addressed by either using a cyclic dataflow model or by implementing single-pass graph streaming algorithms. See our recent survey preprint for an overview of the area.

The field of graph streaming systems is still at its infancy and there exist many fundamental problems to solve. While there is abundant work on graph streaming algorithms, most of this theoretical work is not readily applicable to the streaming model used in practice by streaming systems. See our recent experimental study on streaming graph partitioning algorithms for an example.

For a full list of publications, visit my Google Scholar profile.

Conferences, workshops, competitions

VLDB 2022, HAOC'21 (PC Member), GRADES-NDA 2021 (co-Chair), ACM DEBS 2021 (PC Member), USENIX ATC 2021 (PC Member), ACM/IFIP Middleware 2021 (PC Member), IEEE ICDE 2021 (PC Member), ACM SIGMOD 2020 Student Research Competition (Judge), EuroSys 2021 (AMA Session coordinator), ACM DEBS 2020 (PC Member), ICDE 2020 (Demonstration Track), EDBT 2020 (Demonstration Track), CCGrid 2019 (Applications and Data Science track co-Chair), OPODIS 2018 (PC member), Middleware Doctoral Symposium (ACM/IFIP Middleware 2020), GRADES-NDA 2020 (co-located with SIGMOD 2020), USENIX HotStorage 2020, DBPL 2019 (co-located with PLDI 2019), GRADES-NDA 2019 (co-located with SIGMOD 2019), DBTest 2018 (co-located with SIGMOD 2018), GRADES-NDA 2018 (co-located with SIGMOD 2018), GABB 2018 (co-located with IPDPS 2018), GABB 2017 (co-located with IPDPS 2017), DEEM 2017 (co-located with SIGMOD 2017).

Journals

IEEE Transactions on Parallel and Distributed Systems, VLDB Journal, IEEE Transactions on Knowledge and Data Engineering, IEEE Transactions on Computers

Industrial Conferences

Flink Forward Berlin 2019, Flink Forward San Francisco 2017, Berlin Buzzwords 2017, Flink Forward Berlin 2016, Berlin Buzzwords 2016