Google Cloud Composer: Overcoming The Short-living Tasks Problem

Introduction

Running short-living tasks

echo task information from Airflow UI
# time echo my-awesome-string

real 0m0.000s
user 0m0.000s
sys 0m0.000s
# date +"%9N"; echo my-awesome-string ; date +"%9N"
571368370
my-awesome-string
573152179

A more realistic, yet simple pipeline

  1. Run a query on BigQuery using the BigQuery operator.
  2. Export the data to GCS.
  3. Compose the exported file into a single file.
  4. Copy the composed file to S3.
Sample Pipeline

The Chain Airflow plugin

Conclusions

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store