https://cloud.google.com/composer/
Cloud Composer is a fully managed workflow orchestration service that empowers you to author, schedule, and monitor pipelines that span across clouds and on-premises data centers. Built on the popular Apache Airflow open source project and operated using the Python programming language, Cloud Composer is free from lock-in and easy to use.
Cloud Composer pipelines are configured as directed acyclic graphs (DAGs) using Python, making it easy for users of any experience level to author and schedule a workflow. One-click deployment yields instant access to a rich library of connectors and multiple graphical representations of your workflow in action, increasing pipeline reliability by making troubleshooting easy. Automatic synchronization of your directed acyclic graphs ensures your jobs stay on schedule.
Cloud Composer is deeply integrated within the Google Cloud Platform, giving users the ability to orchestrate their full pipeline. Cloud Composer has robust, built-in integration with many products, including Google BigQuery, Cloud Dataflow, Cloud Dataproc, Cloud Datastore, Cloud Storage, Cloud Pub/Sub, and Cloud ML Engine.
Cloud Composer gives you the ability to connect your pipeline through a single orchestration tool whether your workflow lives on-premises, in multiple clouds, or fully within GCP. The ability to author, schedule, and monitor your workflows in a unified manner means you can break down the silos in your environment and focus less on infrastructure.
Cloud Composer is built on Apache Airflow, the popular open source orchestration tool. This open source project, which Google is contributing back into, provides freedom from lock in for customers as well as integration with a broad number of platforms, which will only expand as the Airflow community grows.
Multi-cloud Create workflows that connect data, processing, and services across clouds, giving you a unified data environment. Ease your transition to the cloud or maintain a hybrid data environment by orchestrating workflows that cross between on-premises and the public cloud. Python Programming Language Leverage existing Python skills to dynamically author and schedule workflows within Cloud Composer. Fully Managed Cloud Composer's managed nature allows you to focus on authoring, scheduling, and monitoring your workflows as opposed to provisioning resources.
Cloud Composer is built upon Apache Airflow, giving users freedom from lock-in and portability. Integrated Built-in integration with BigQuery, Dataflow, Dataproc, Datastore, Cloud Storage, Pub/Sub, Cloud ML Engine, and more, giving you the ability to orchestrate end-to-end GCP workloads. Reliability Increase reliability of your workflows through easy-to-use charts for monitoring and troubleshooting the root cause of an issue.
"At Blue Apron, our data workflows need to operate flawlessly to enable on-time delivery of perishable food. Cloud Composer helps us orchestrate our pipelines more reliably by allowing us to author, schedule, and monitor our workflows from one place using Python."
— Michael Collis, Staff Software Engineer, Blue Apron
“Cloud Composer gives us the flexibility we need in workflow orchestration. The ability to define our workflows in Python and use a range of graphs to monitor them makes the challenges of data processing easier for us.”