University of Illinois Urbana-Champaign

Benchmarking modern distributed stream processing systems with customizable synthetic workloads

Gupta, Srujun Thanmay

Content Files
ECE499-Sp2018-gupta.pdf

Permalink

https://hdl.handle.net/2142/99995

Description

Title
Benchmarking modern distributed stream processing systems with customizable synthetic workloads
Author(s)
Gupta, Srujun Thanmay
Contributor(s)
Gupta, Indranil
Issue Date
2018-05
Keyword(s)
  • stream processing
  • analytics
  • benchmarking
Date of Ingest
2018-05-23T18:05:37Z
Abstract
Real-time analysis of continuous data streams using distributed systems is an emerging class of data analytics problems that require systems with high throughput and low latency to efficiently analyze high velocity data. As stream processing applications become increasingly popular, many frameworks used to build clusters to process this data have emerged in recent years. These include frameworks like Samza, Storm, Heron, Spark Streaming, Flink, and Apex. For system administrators and developers, there is great value in understanding the capabilities and performance of their stream processing workloads, given the various frameworks running on their cluster configuration. In this thesis, we present Finch, a new benchmarking tool that can be used to create synthetic stream processing workloads. Finch generates metrics that system administrators and developers can use to benchmark their stream processing applications. To achieve this, Finch provides a flexible and easy way to define arbitrary workloads using tunable operators. It then translates these workloads into applications that are run by the target system. To describe Finch's design, we investigate what parameters affect workload performance, and present studies on fault tolerance and system scalability. We then use Finch to understand and compare two popular stream processing frameworks, Samza and Heron.
Type of Resource
text
Genre of Resource
Other
Language
en
Permalink
http://hdl.handle.net/2142/99995

Owning Collections

Senior Theses - Electrical and Computer Engineering PRIMARY
The best of ECE undergraduate research