Estratégias de escalonamento para ambientes de execução guiada por fluxo de dados

Abstract

Sucuri is a minimalistic Python library that provides dataflow programming through a reasonably simple syntax. It allows transparent execution on computer clusters and natural exploitation of parallelism. In Sucuri, programmers instantiate a dataflow graph, where each node is assigned to a function and edges represent data dependencies between nodes. The original implementation of Sucuri adopts a centralized scheduler, which incurs high communication overheads, specially in clusters with a large number of machines. In this paper we modify Sucuri so that each machine in a cluster will have its own scheduler. Before execution, the dataflow graph is partitioned, so that nodes can be distributed among the machines of the cluster. In runtime, idle workers will grab tasks from a ready queue in their local scheduler. When a machine finishes running their list of nodes, it will request the other machines, which are overloaded, the tasks of their ready queue. Experimental results confirm that the solution can reduce communication overheads, improving performance in larger clusters