A C++ Data Stream Processing Parallel Library for Multicores and GPUs
Overview
WindFlow is a library for parallel data stream processing on shared-memory systems and NVIDIA GPUs, with distributed support currently under development. Data stream processing is a popular computing paradigm supported by several open-source frameworks (e.g., Apache Storm, Apache Flink, and Spark Streaming). However, existing solutions primarily target distributed systems (e.g., clusters of server machines) and rely on the Java Virtual Machine (JVM) to simplify the development of streaming applications for distributed architectures. As recognized by recent publications [1,2], these frameworks are not well suited to fully exploit the capabilities of scale-up architectures, such as single machines equipped with multiple multi-core CPUs and co-processors like GPUs and FPGAs.
The library presented on this webpage aims to address this gap by providing an easy-to-use tool with the following distinctive features:
- it provides a clear API that is fully compliant with the C++17 standard;
- as in most existing systems, applications are represented as data-flow graphs of interconnected operators. The library offers common streaming operators such as map, filter, flatmap, and many others. To instantiate operators, the programmer must supply the business-logic code required for processing (provided via lambda expressions, functor objects, or plain functions). Consequently, operators can execute arbitrary user-defined functions, not only predefined transformations (such as the relational algebra operators found in traditional Data Stream Management Systems);
- the library provides window-based operators with a rich parallel semantics to accelerate continuous analytics on data streams;
- operators can be composed using the MultiPipe programming construct, which provides a compositional interface for building applications (i.e., creating operators and connecting them).
The library is built on top of FastFlow [3] (versions ≥ 3.0), a parallel programming framework developed since 2010 by the Parallel Programming Models (PPMs) group at the Department of Computer Science, University of Pisa, Italy. The framework enables programmers to construct concurrency graphs composed of execution entities that exchange data references through efficient lock-free, single-producer/single-consumer queues. Further details about FastFlow are available on the project’s webpage.
The figure below summarizes the software layers of the FastFlow ecosystem, with WindFlow built on top of FastFlow’s building blocks, alongside the high-level HPC parallel patterns provided by the framework (e.g., parallel-for, map-reduce, stencil-reduce, divide-and-conquer, and other patterns common in High-Performance Computing workloads).
How to Cite
Below is a list of the major scientific publications related to WindFlow:
- G. Mencagli, M. Torquati, A. Cardaci, A. Fais, L. Rinaldi and M. Danelutto. WindFlow: High-Speed Continuous Stream Processing With Parallel Building Blocks. In IEEE Transactions on Parallel and Distributed Systems, vol. 32, no. 11, pp. 2748-2763, 1 Nov. 2021, doi: 10.1109/TPDS.2021.3073970
- G. Mencagli, M. Torquati, D. Griebler, A. Fais, and M. Danelutto. General-purpose data stream processing on heterogeneous architectures with WindFlow. Journal of Parallel and Distributed Computing (JPDC). 2024, Elsevier. ISSN: 0743-7315, DOI: https://doi.org/10.1016/j.jpdc.2023.104782.
References
Additional useful references:
- Steffen Zeuch, Bonaventura Del Monte, Jeyhun Karimov, Clemens Lutz, Manuel Renz, Jonas Traub, Sebastian Breß, Tilmann Rabl, and Volker Markl. 2019. Analyzing efficient stream processing on modern hardware. Proc. VLDB Endow. 12, 5 (January 2019), 516-530. DOI: https://doi.org/10.14778/3303753.3303758
- S. Zhang, B. He, D. Dahlmeier, A. C. Zhou and T. Heinze. Revisiting the Design of Data Stream Processing Systems on Multi-Core Processors. 2017 IEEE 33rd International Conference on Data Engineering (ICDE), San Diego, CA, 2017, pp. 659-670
- Aldinucci, M. , Danelutto, M. , Kilpatrick, P. and Torquati, M. (2017). Fastflow: High‐Level and Efficient Streaming on Multicore. In Programming multi‐core and many‐core computing systems (eds S. Pllana and F. Xhafa). doi:10.1002/9781119332015.ch13