DAPHNE Runtime: Harnessing Parallelism for Integrated Data Analysis Pipelines

Aristotelis Vontzalidis; Stratos Psomadakis; Constantinos Bitsakos; Mark Dokter; Kevin Innerebner; Patrick Damme; Matthias Boehm; Florina Ciorba; Ahmed Eleliemy; Vasileios Karakostas; Aleš Zamuda; Dimitrios Tsoumakos

doi:10.1007/978-3-031-48803-0_25

DAPHNE Runtime: Harnessing Parallelism for Integrated Data Analysis Pipelines

Aristotelis Vontzalidis, Stratos Psomadakis, Constantinos Bitsakos, Mark Dokter, Kevin Innerebner, Patrick Damme, Matthias Boehm, Florina Ciorba, Ahmed Eleliemy, Vasileios Karakostas, Aleš Zamuda, Dimitrios Tsoumakos^*

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

Abstract

Integrated data analysis pipelines combine rigorous data management and processing, high-performance computing and machine learning tasks. While these systems and operations share many compilation and runtime techniques, data analysts and scientists are currently dealing with multiple systems for each stage of their pipeline. DAPHNE is an open and extensible system infrastructure for such pipelines, including language abstractions, compilation and runtime techniques, multi-level scheduling, hardware accelerators and computational storage. In this demonstration, we focus on the DAPHNE runtime that provides the implementation of kernels for local, distributed and accelerator-enhanced operations, vectorized execution, integration with existing frameworks and libraries for productivity and interoperability, as well as efficient I/O and communication primitives.

Original language	English
Title of host publication	Euro-Par 2023
Subtitle of host publication	Parallel Processing Workshops - Euro-Par 2023 International Workshops, 2023, Revised Selected Papers
Editors	Demetris Zeinalipour, Dora Blanco Heras, George Pallis, Herodotos Herodotou, Demetris Trihinas, Daniel Balouek, Patrick Diehl, Terry Cojean, Karl Fürlinger, Maja Hanne Kirkeby, Matteo Nardelli, Pierangelo Di Sanzo
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	242-246
Number of pages	5
ISBN (Print)	9783031488023
DOIs	https://doi.org/10.1007/978-3-031-48803-0_25
Publication status	Published - 2024
Event	29th International Conference on Parallel and Distributed Computing: Euro-Par 2023 - Limassol, Cyprus Duration: 28 Aug 2023 → 1 Sept 2023

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	14352 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	29th International Conference on Parallel and Distributed Computing
Abbreviated title	Euro-Par 2023
Country/Territory	Cyprus
City	Limassol
Period	28/08/23 → 1/09/23

Keywords

Distributed Systems
High Performance Computing
Machine Learning Systems
Vectorized Execution

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-031-48803-0_25

Cite this

Vontzalidis, A., Psomadakis, S., Bitsakos, C., Dokter, M., Innerebner, K., Damme, P., Boehm, M., Ciorba, F., Eleliemy, A., Karakostas, V., Zamuda, A., & Tsoumakos, D. (2024). DAPHNE Runtime: Harnessing Parallelism for Integrated Data Analysis Pipelines. In D. Zeinalipour, D. Blanco Heras, G. Pallis, H. Herodotou, D. Trihinas, D. Balouek, P. Diehl, T. Cojean, K. Fürlinger, M. H. Kirkeby, M. Nardelli, & P. Di Sanzo (Eds.), Euro-Par 2023: Parallel Processing Workshops - Euro-Par 2023 International Workshops, 2023, Revised Selected Papers (pp. 242-246). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14352 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-48803-0_25

DAPHNE Runtime: Harnessing Parallelism for Integrated Data Analysis Pipelines. / Vontzalidis, Aristotelis; Psomadakis, Stratos; Bitsakos, Constantinos et al.
Euro-Par 2023: Parallel Processing Workshops - Euro-Par 2023 International Workshops, 2023, Revised Selected Papers. ed. / Demetris Zeinalipour; Dora Blanco Heras; George Pallis; Herodotos Herodotou; Demetris Trihinas; Daniel Balouek; Patrick Diehl; Terry Cojean; Karl Fürlinger; Maja Hanne Kirkeby; Matteo Nardelli; Pierangelo Di Sanzo. Springer Science and Business Media Deutschland GmbH, 2024. p. 242-246 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14352 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

Vontzalidis, A, Psomadakis, S, Bitsakos, C, Dokter, M, Innerebner, K, Damme, P, Boehm, M, Ciorba, F, Eleliemy, A, Karakostas, V, Zamuda, A & Tsoumakos, D 2024, DAPHNE Runtime: Harnessing Parallelism for Integrated Data Analysis Pipelines. in D Zeinalipour, D Blanco Heras, G Pallis, H Herodotou, D Trihinas, D Balouek, P Diehl, T Cojean, K Fürlinger, MH Kirkeby, M Nardelli & P Di Sanzo (eds), Euro-Par 2023: Parallel Processing Workshops - Euro-Par 2023 International Workshops, 2023, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14352 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 242-246, 29th International Conference on Parallel and Distributed Computing, Limassol, Cyprus, 28/08/23. https://doi.org/10.1007/978-3-031-48803-0_25

Vontzalidis A, Psomadakis S, Bitsakos C, Dokter M, Innerebner K, Damme P et al. DAPHNE Runtime: Harnessing Parallelism for Integrated Data Analysis Pipelines. In Zeinalipour D, Blanco Heras D, Pallis G, Herodotou H, Trihinas D, Balouek D, Diehl P, Cojean T, Fürlinger K, Kirkeby MH, Nardelli M, Di Sanzo P, editors, Euro-Par 2023: Parallel Processing Workshops - Euro-Par 2023 International Workshops, 2023, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2024. p. 242-246. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-48803-0_25

Vontzalidis, Aristotelis ; Psomadakis, Stratos ; Bitsakos, Constantinos et al. / DAPHNE Runtime : Harnessing Parallelism for Integrated Data Analysis Pipelines. Euro-Par 2023: Parallel Processing Workshops - Euro-Par 2023 International Workshops, 2023, Revised Selected Papers. editor / Demetris Zeinalipour ; Dora Blanco Heras ; George Pallis ; Herodotos Herodotou ; Demetris Trihinas ; Daniel Balouek ; Patrick Diehl ; Terry Cojean ; Karl Fürlinger ; Maja Hanne Kirkeby ; Matteo Nardelli ; Pierangelo Di Sanzo. Springer Science and Business Media Deutschland GmbH, 2024. pp. 242-246 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{00a20eba55074efa9a1082f7c537e834,

title = "DAPHNE Runtime: Harnessing Parallelism for Integrated Data Analysis Pipelines",

abstract = "Integrated data analysis pipelines combine rigorous data management and processing, high-performance computing and machine learning tasks. While these systems and operations share many compilation and runtime techniques, data analysts and scientists are currently dealing with multiple systems for each stage of their pipeline. DAPHNE is an open and extensible system infrastructure for such pipelines, including language abstractions, compilation and runtime techniques, multi-level scheduling, hardware accelerators and computational storage. In this demonstration, we focus on the DAPHNE runtime that provides the implementation of kernels for local, distributed and accelerator-enhanced operations, vectorized execution, integration with existing frameworks and libraries for productivity and interoperability, as well as efficient I/O and communication primitives.",

keywords = "Distributed Systems, High Performance Computing, Machine Learning Systems, Vectorized Execution",

author = "Aristotelis Vontzalidis and Stratos Psomadakis and Constantinos Bitsakos and Mark Dokter and Kevin Innerebner and Patrick Damme and Matthias Boehm and Florina Ciorba and Ahmed Eleliemy and Vasileios Karakostas and Ale{\v s} Zamuda and Dimitrios Tsoumakos",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.; 29th International Conference on Parallel and Distributed Computing : Euro-Par 2023, Euro-Par 2023 ; Conference date: 28-08-2023 Through 01-09-2023",

year = "2024",

doi = "10.1007/978-3-031-48803-0_25",

language = "English",

isbn = "9783031488023",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "242--246",

editor = "Demetris Zeinalipour and {Blanco Heras}, Dora and George Pallis and Herodotos Herodotou and Demetris Trihinas and Daniel Balouek and Patrick Diehl and Terry Cojean and Karl F{\"u}rlinger and Kirkeby, {Maja Hanne} and Matteo Nardelli and {Di Sanzo}, Pierangelo",

booktitle = "Euro-Par 2023",

address = "Germany",

}

TY - GEN

T1 - DAPHNE Runtime

T2 - 29th International Conference on Parallel and Distributed Computing

AU - Vontzalidis, Aristotelis

AU - Psomadakis, Stratos

AU - Bitsakos, Constantinos

AU - Dokter, Mark

AU - Innerebner, Kevin

AU - Damme, Patrick

AU - Boehm, Matthias

AU - Ciorba, Florina

AU - Eleliemy, Ahmed

AU - Karakostas, Vasileios

AU - Zamuda, Aleš

AU - Tsoumakos, Dimitrios

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.

PY - 2024

Y1 - 2024

N2 - Integrated data analysis pipelines combine rigorous data management and processing, high-performance computing and machine learning tasks. While these systems and operations share many compilation and runtime techniques, data analysts and scientists are currently dealing with multiple systems for each stage of their pipeline. DAPHNE is an open and extensible system infrastructure for such pipelines, including language abstractions, compilation and runtime techniques, multi-level scheduling, hardware accelerators and computational storage. In this demonstration, we focus on the DAPHNE runtime that provides the implementation of kernels for local, distributed and accelerator-enhanced operations, vectorized execution, integration with existing frameworks and libraries for productivity and interoperability, as well as efficient I/O and communication primitives.

AB - Integrated data analysis pipelines combine rigorous data management and processing, high-performance computing and machine learning tasks. While these systems and operations share many compilation and runtime techniques, data analysts and scientists are currently dealing with multiple systems for each stage of their pipeline. DAPHNE is an open and extensible system infrastructure for such pipelines, including language abstractions, compilation and runtime techniques, multi-level scheduling, hardware accelerators and computational storage. In this demonstration, we focus on the DAPHNE runtime that provides the implementation of kernels for local, distributed and accelerator-enhanced operations, vectorized execution, integration with existing frameworks and libraries for productivity and interoperability, as well as efficient I/O and communication primitives.

KW - Distributed Systems

KW - High Performance Computing

KW - Machine Learning Systems

KW - Vectorized Execution

UR - http://www.scopus.com/inward/record.url?scp=85190993519&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-48803-0_25

DO - 10.1007/978-3-031-48803-0_25

M3 - Conference paper

AN - SCOPUS:85190993519

SN - 9783031488023

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 242

EP - 246

BT - Euro-Par 2023

A2 - Zeinalipour, Demetris

A2 - Blanco Heras, Dora

A2 - Pallis, George

A2 - Herodotou, Herodotos

A2 - Trihinas, Demetris

A2 - Balouek, Daniel

A2 - Diehl, Patrick

A2 - Cojean, Terry

A2 - Fürlinger, Karl

A2 - Kirkeby, Maja Hanne

A2 - Nardelli, Matteo

A2 - Di Sanzo, Pierangelo

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 28 August 2023 through 1 September 2023

ER -