Airflow In Docker

Posted onby
Airflow In Docker
  1. Docker Airflow Image
  2. Airflow Docker Compose
  3. Run Airflow In Docker

We will learn how to set up airflow environment using Docker

The GitHub links for this tutorial

Docker Airflow Image

To configure Airflow to use Celery, we need to specify some variables. A simple and convenient way to do this is by attaching the envfile field in docker-compose.You can specify these settings in airflow.cfg and mount that file, but Airflow has variable hierarchy. Environmental variables are more important than variables from the airflow.cfg file, so they will be overwritten by the default. In order to quick change airflow config inside a docker container,There are many ways. Instead of change airflow.cfg, you can change environment variable directly. In docker container, it can very easy to revise in docker-compose.yml directly. And you can just restart the docker-compose quickly. Here is some common configuration variable. In this video, we will learn how to set up airflow environment using Docker.🔥 Want to master SQL? Get the full SQL course: Subscrib. Anaconda / packages / airflow-with-docker 1.10.12. 0 Airflow is a platform to programmatically author, schedule and monitor workflows Conda. Running Airflow in Docker This quick-start guide will allow you to quickly start Airflow with CeleryExecutor in Docker. This is the fastest way to start Airflow.

Airflow problem

  • open source software grows at an overwhelming pace
  • Airflow is built to integrate with all databases, system, cloud environments, …
    • Managing and maintaining all of the dependencies changes will be really difficult.
    • Takes lots of time to set up, and config Airflow env.
    • How to share development and production environments for all developers.

Docker overview

  • Docker is an open platform for developing, shipping, and running applications.
  • Docker provides the ability to package and run an application in a loosely isolated environment called a container. The isolation and security allow you to run many containers simultaneously on a given host, regardless of its operating system: Mac, Windows, PC, cloud, data center, …

  • Containers are lightweight because they don’t need the extra load of a hypervisor, but run directly within the host machine’s kernel. This means you can run more containers on a given hardware combination than if you were using virtual machines.

Benefits of using Docker

Airflow Docker Compose

Airflow
  • Docker is freeing us from the task of managing, maintaining all of the Airflow dependencies, and deployment.
  • Easy to share and deploy different versions and environments.
  • Keep track through Github tags and releases.
  • Ease of deployment from testing to production environment.

Airflow docker image

Run Airflow In Docker

Airflow docker operator environment variables

Fork and clone the below Github repo and follow the instruction to set up

Run airflow in docker
  • airflow-tutorial: https://github.com/tuanavu/airflow-tutorial
Setup airflow in docker

Prerequisites

  • Install Docker
  • Install Docker Compose
  • Following the Airflow release from Python Package Index

Getting Started

  • Clone the repo
  • Install the prerequisites
  • Run the service
  • Check http://localhost:8080
  • Done!