Reactive resource provisioning heuristics for dynamic dataflows on cloud infrastructure

Kumbhare, AG, Simmhan, Y, Frincu, M ORCID: https://orcid.org/0000-0003-1034-8409 and Prasanna, VK, 2015. Reactive resource provisioning heuristics for dynamic dataflows on cloud infrastructure. IEEE Transactions on Cloud Computing, 3 (2), pp. 105-118. ISSN 2168-7161

Preview

Text 1393281_Frincu.pdf - Post-print Download (2MB) | Preview

Abstract

The need for low latency analysis over high-velocity data streams motivates the need for distributed continuous dataflow systems. Contemporary stream processing systems use simple techniques to scale on elastic cloud resources to handle variable data rates. However, application QoS is also impacted by variability in resource performance exhibited by clouds and hence necessitates autonomic methods of provisioning elastic resources to support such applications on cloud infrastructure. We develop the concept of “dynamic dataflows” which utilize alternate tasks as additional control over the dataflow's cost and QoS. Further, we formalize an optimization problem to represent deployment and runtime resource provisioning that allows us to balance the application's QoS, value, and the resource cost. We propose two greedy heuristics, centralized and sharded, based on the variable-sized bin packing algorithm and compare against a Genetic Algorithm (GA) based heuristic that gives a near-optimal solution. A large-scale simulation study, using the linear road benchmark and VM performance traces from the AWS public cloud, shows that while GA-based heuristic provides a better quality schedule, the greedy heuristics are more practical, and can intelligently utilize cloud elasticity to mitigate the effect of variability, both in input data rates and cloud resource performance, to meet the QoS of fast data applications.

Item Type:	Journal article
Publication Title:	IEEE Transactions on Cloud Computing
Creators:	Kumbhare, A.G., Simmhan, Y., Frincu, M. and Prasanna, V.K.
Publisher:	Institute of Electrical and Electronics Engineers
Date:	1 April 2015
Volume:	3
Number:	2
ISSN:	2168-7161
Identifiers:	Number Type 10.1109/tcc.2015.2394316 DOI 1393281 Other
Divisions:	Schools > School of Science and Technology
Record created by:	Jonathan Gallacher
Date Added:	09 Dec 2020 10:00
Last Modified:	31 May 2021 15:10
URI:	https://irep.ntu.ac.uk/id/eprint/41814

Actions (login required)

Edit View

Statistics

Views

Downloads