Установка AirFlow

Делаю:
2023.06.30

https://github.com/apache/airflow

$ sudo vi /etc/profile.d/airflow.sh

#### AIRFLOW ########################

export AIRFLOW_HOME=/home/marley/projects/dev/mlops/airflow
export AIRFLOW_CONFIG=${AIRFLOW_HOME}/airflow.cfg

#### AIRFLOW ########################

$ sudo chmod +x /etc/profile.d/airflow.sh
$ source /etc/profile.d/airflow.sh

$ pip install 'apache-airflow==2.9.2' \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.9.2/constraints-3.8.txt"

// sqlite
$ airflow db init

// admin / admin
$ airflow users create \
        --username admin \
        --firstname admin \
        --lastname admin \
        --role Admin \
        --email [email protected]

$ mkdir -p /home/marley/projects/dev/mlops/airflow/dags

$ vi ${AIRFLOW_HOME}/dags/mlops_dag_1.py

from airflow import DAG
from datetime import datetime
from airflow.operators.python_operator import PythonOperator

def print_hello():
    print("Hello World")

with DAG(
    dag_id="hello_world",
    start_date=datetime(2023,1,1),
    schedule_interval="@once",
    catchup=False) as dag:

    hello_task = PythonOperator(
        task_id="hello_operator",
        python_callable=print_hello
    )

    hello_task

$ airflow config get-value core DAGS_FOLDER
/home/marley/projects/dev/mlops/airflow/dags

$ airflow dags list

$ cd /home/marley/.local/lib/python3.10/site-packages/airflow/example_dags
$ rm -rf *

$ airflow scheduler

$ airflow dags list

$ airflow dags test hello_world

$ airflow webserver -p 8080

// admin / admin
http://localhost:8080/

Если нужно включить test_connection

$ vi $AIRFLOW_HOME/airflow.cfg

test_connection = Disabled
меняю на
test_connection = Enabled

Running Airflow in Docker

https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html