Agent-based hierarchical approach for executing bag-of-tasks in clouds

Abstract

Numerous unrelated, independent (no inter-task communication) tasks called “bag-oftasks”
(BoTs) compared with message passing applications can be highly parallelised and
executed in any acceptable order. A common practice when executing bag-of-tasks applications
(BoT) is to exploit the master-slave topology. Cloud environments offer some features
that facilitate executing BoT applications. One of the approaches to control cloud resources
is to use agents that can flexibly act in a dynamic environment. Given these assumptions we
designed a combination of these approaches, which can be classified as: a distributed, hierarchical
solution to the issue of scalable executing of bag-of-tasks. The concept of our system
relates to a project that is focused on processing huge quantities of data incoming from a
network of sensors by the Internet. Our aim is to create a mechanism for processing such data
as a system which executes jobs while exploiting load balancing for cloud resources using,
e.g., Eucalyptus. The idea is to create a hybrid architecture which takes advantage of some
centralized parts of the system and full distributedness in other parts. On the other hand
we balance dependencies between the system components using a hierarchic master-slave
structure.