Apache Beam

Unified programming model

Official website

Apache Beam®

Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service). Beam also brings DSL in different languages, allowing users to easily implement their data integration processes.

beam.apache.org →

Link to the official site · 15,524 chars · not written by Vinony

Source code

github.com →

Apache Beam is a unified model for defining both batch and streaming data-parallel processing pipelines, as well as a set of language-specific SDKs for constructing pipelines and Runners for executing them on distributed processing backends, including Apache Flink, Apache Spark, Google Cloud Dataflow, and Hazelcast Jet. 2. Run your first example: Minimal WordCount example (available in this repository) Beam provides a general approach to expressing embarrassingly parallel data processing pipelines and supports three categories of users, each of which have relatively disparate backgrounds and needs. The model behind Beam evolved from several internal Google data processing projects, including MapReduce, FlumeJava, and Millwheel. This model was originally known as the “Dataflow Model”. To learn more about the Beam Model (though still under the original name of Dataflow), see the World Beyond Batch: Streaming 101 and Streaming 102 posts on O’Reilly’s Radar site, and the VLDB 2015 paper. Beam supports multiple language-specific SDKs for writing pipelines against the Beam Model. Beam supports executing programs on multiple distributed processing backends through PipelineRunners. Currently, the following PipelineRunners are available: The DirectRunner runs the pipeline on your local machine. The PrismRunner runs the pipeline on your local machine using Beam Portability. The DataflowRunner submits the pipeline to the Google Cloud Dataflow. The FlinkRunner runs the pipeline on an Apache Flink cluster. The code has been donated from dataArtisans/flink-dataflow and is now part of Beam. The SparkRunner runs the pipeline on an Apache Spark cluster. The JetRunner runs the pipeline on a Hazelcast Jet cluster. The code has been donated from hazelcast/hazelcast-jet and is now part of Beam. The Twister2Runner runs the pipeline on a Twister2 cluster. The code has been donated from DSC-SPIDAL/twister2 and is now part of Beam. Instructions for building and testing Beam itself are in the contribution guide. Apache Beam Website Official website with documentation, concepts, and guides for Apache Beam. Tour of Beam A comprehensive, interactive learning experience covering Beam concepts in depth. Beam Quest A certification granted by Google Cloud, certifying proficiency in Beam.