Distributed Data Processing on Microcomputers with Ascheduler and Apache Spark

Vladimir Korkhov, Ivan Gankevich, Oleg Iakushkin, Dmitry Gushchanskiy, Dmitry Khmel, Andrey Ivashchenko, Alexander Pyayt, Sergey Zobnin, Alexander Loginov

Research outputpeer-review

Abstract

Modern architectures of data acquisition and processing often consider low-cost and low-power devices that can be bound together to form a distributed infrastructure. In this paper we overview possibilities to organize a distributed computing testbed based on microcomputers similar to Raspberry Pi and Intel Edison. The goal of the research is to investigate and develop a scheduler for orchestrating distributed data processing and general purpose computations on such unreliable and resource-constrained hardware. Also we consider integration of the scheduler with well-known distributed data processing framework Apache Spark. We outline the project carried out in collaboration with Siemens LLC to compare different configurations of the hardware and software deployment and evaluate performance and applicability of the tools to the testbed.
Original languageEnglish
Title of host publicationComputational Science and Its Applications – ICCSA 2017
Subtitle of host publication17th International Conference, Trieste, Italy, July 3-6, 2017, Proceedings, Part V
PublisherSpringer Nature
Pages387-398
ISBN (Electronic)978-3-319-62404-4
ISBN (Print)978-3-319-62403-7
DOIs
Publication statusPublished - 2017
Event17th International Conference on Computational Science and Its Applications, ICCSA 2017 - Trieste
Duration: 2 Jul 20175 Jul 2017

Publication series

NameLecture Notes in Computer Science
PublisherSpringer Nature
Volume10408
ISSN (Print)0302-9743

Conference

Conference17th International Conference on Computational Science and Its Applications, ICCSA 2017
CountryItaly
CityTrieste
Period2/07/175/07/17

Fingerprint Dive into the research topics of 'Distributed Data Processing on Microcomputers with Ascheduler and Apache Spark'. Together they form a unique fingerprint.

Cite this