#Airflow docker run parameters how to

Im not very happy of this, I would rather trigger this script only if there is a failure, even better if I could detect whether the failure was due to the name conflict. To solve this I created a script that runs on the host every minute: for i in $(docker ps -aq -filter status=exited) do This makes its name already assigned and therefore when Airflow runs the task again it throws an error 'a container with this name already exists'. The test works, but it never actually gets run when I start airflow. I am not sure if this is the right way for this, but it works airflow test dagid executiondate.

In case of script failure the container would still exists in exited status, it is not removed. I am using the PythonOperator with a module that loads a library that I wrote.

There is a problem in case the script fails and airflow retries the run: So Ive used t1 to execute the first python script and t2 to execute the second. Ive got two python scripts, the first one takes in the data using an API call, and the second one pushes the data into google sheets. Optimize efficiency and unlock the potential of these. Part I of our comprehensive tutorial series.

#Airflow docker run parameters how to

I am trying to run a simple DAG on Airflow running on Docker. Learn how to build DAGs and run Airflow in Docker with our step-by-step guide for beginners. If the script runs successfully the container is removed automatically. Airflow DAG runs successfully but Tasks fail. The name for the container is dynamically created with some parameters which make it meaningful to me if I list the containers to see via terminal what's going on.įor instance the container name is called my_script_daily_A I use the Docker Operator to create a number of container that run a script.