ML p(r)ior | Ensemble Toolkit: Scalable and Flexible Execution of Ensembles of Tasks

Ensemble Toolkit: Scalable and Flexible Execution of Ensembles of Tasks

2016-02-01
There are many science applications that require scalable task-level parallelism and support for flexible execution and coupling of ensembles of simulations. Most high-performance system software and middleware, however, are designed to support the execution and optimization of single tasks. Motivated by the missing capabilities of these computing systems and the increasing importance of task-level parallelism, we introduce the Ensemble toolkit which has the following application development features: (i) abstractions that enable the expression of ensembles as primary entities, and (ii) support for ensemble-based execution patterns that capture the majority of application scenarios. Ensemble toolkit uses a scalable pilot-based runtime system that decouples workload execution and resource management details from the expression of the application, and enables the efficient and dynamic execution of ensembles on heterogeneous computing resources. We investigate three execution patterns and characterize the scalability and overhead of Ensemble toolkit for these patterns. We investigate scaling properties for up to O(1000) concurrent ensembles and O(1000) cores and find linear weak and strong scaling behaviour.
PDF

Highlights - Most important sentences from the article

Login to like/save this paper, take notes and configure your recommendations

Related Articles

2017-12-16

The next generation of AI applications will continuously interact with the environment and learn fro… show more
PDF

Highlights - Most important sentences from the article

2019-05-24

The growing popularity of workflows in the cloud domain promoted the development of sophisticated au… show more
PDF

Highlights - Most important sentences from the article

2016-09-12

We suggest there is a need for a fresh perspective on the design and development of workflow systems… show more
PDF

Highlights - Most important sentences from the article

2019-05-06

High-level programming languages such as Python are increasingly used to provide intuitive interface… show more
PDF

Highlights - Most important sentences from the article

2019-03-24

This paper describes a building blocks approach to the design of scientific workflow systems. We dis… show more
PDF

Highlights - Most important sentences from the article

2019-04-05

RADICAL-Cybertools (RCT) are a set of software systems that serve as middleware to develop efficient… show more
PDF

Highlights - Most important sentences from the article

2019-05-23

Ecological sciences are using imagery from a variety of sources to monitor and survey populations an… show more
PDF

Highlights - Most important sentences from the article

2019-04-09

The present volume contains the proceedings of RTEST WiP 2018, chaired by Marco Caccamo, University … show more
PDF

Highlights - Most important sentences from the article

2018-10-23

Linear algebra operations are widely used in scientific computing and machine learning applications.… show more
PDF

Highlights - Most important sentences from the article

2018-10-17

Despite advancements in the areas of parallel and distributed computing, the complexity of programmi… show more
PDF

Highlights - Most important sentences from the article

2018-11-02

High performance computing (HPC) systems underwent a significant increase in their processing capabi… show more
PDF

Highlights - Most important sentences from the article

2019-05-28
1905.11863 | cs.SE

Workflows in biomolecular science are very important as they are intricately intertwined with the sc… show more
PDF

Highlights - Most important sentences from the article

2019-04-26

As dataset sizes increase, data analysis tasks in high performance computing (HPC) are increasingly … show more
PDF

Highlights - Most important sentences from the article

2018-01-03

The efficacy of drug treatments depends on how tightly small molecules bind to their target proteins… show more
PDF

Highlights - Most important sentences from the article

2018-08-02

Motivated by the need to emulate workload execution characteristics on high-performance and distribu… show more
PDF

Highlights - Most important sentences from the article