Adaptive workflow processing and execution in Pegasus

Lee, K ORCID logoORCID: https://orcid.org/0000-0002-2730-9150, Paton, NW, Sakellariou, R, Deelman, E, Fernandes, AAA and Mehta, G, 2009. Adaptive workflow processing and execution in Pegasus. Concurrency and Computation: Practice and Experience, 21 (16), pp. 1965-1981. ISSN 1532-0626

[thumbnail of 221167_PubSub2780_Lee_K.pdf]
Preview
Text
221167_PubSub2780_Lee_K.pdf

Download (328kB) | Preview

Abstract

Workflows are widely used in applications that require coordinated use of computational resources. Workflow definition languages typically abstract over some aspects of the way in which a workflow is to be executed, such as the level of parallelism to be used or the physical resources to be deployed. As a result, a workflow management system has responsibility for establishing how best to execute a workflow given the available resources. The Pegasus workflow management system compiles abstract workflows into concrete execution plans, and has been widely used in large-scale e-Science applications. This paper describes an extension to Pegasus whereby resource allocation decisions are revised during workflow evaluation, in the light of feedback on the performance of jobs at runtime. The contributions of this paper include: (i) a description of how adaptive processing has been retrofitted to an existing workflow management system; (ii) a scheduling algorithm that allocates resources based on runtime performance; and (iii) an experimental evaluation of the resulting infrastructure using grid middleware over clusters.

Item Type: Journal article
Description: Special issue: 3rd International Workshop on Workflow Management and Applications in Grid Environments (WaGe2008).
Publication Title: Concurrency and Computation: Practice and Experience
Creators: Lee, K., Paton, N.W., Sakellariou, R., Deelman, E., Fernandes, A.A.A. and Mehta, G.
Publisher: John Wiley & Sons Ltd.
Date: 2009
Volume: 21
Number: 16
ISSN: 1532-0626
Identifiers:
Number
Type
10.1002/cpe.1446
DOI
Rights: Copyright © 2008 John Wiley & Sons, Ltd.
Divisions: Schools > School of Science and Technology
Record created by: EPrints Services
Date Added: 09 Oct 2015 10:14
Last Modified: 09 Jun 2017 13:23
URI: https://irep.ntu.ac.uk/id/eprint/9893

Actions (login required)

Edit View Edit View

Statistics

Views

Views per month over past year

Downloads

Downloads per month over past year