Kubernetes using a common helm chart: essentially a repeat of our Airflow v1ĪrgoCD Workflows: we already utilize Argo to manage various CI/CD pipelines, so this seemed like an easy pivot to make Our team looked at three different migration paths:Īmazon Managed Workflows for Apache Airflow (MWAA): a fully managed service offering by AWS You may even look for a different way to solve the same problem. When coming across big migration tasks such as these, it’s always a good idea to take a look at other alternatives which may be more suitable or easier to implement. Our database needed to be migrated to a new version which is not supported by our current implementation of Airflow. Several operators and utility functions that our team defined used features that were refactored or no longer supported. However, back in 2020 as the new Airflow v2 was released our team began facing problems trying to migrate to the newest version: We self-host the tool through Kubernetes. Here at Sumo, our team has been using this technology for several years to manage various jobs relating to our organization’s Global Intelligence, Global Confidence, and other data science initiatives. Apache Airflow is an open-source orchestration platform that enables the development, scheduling and monitoring of tasks using directed acyclic graphs (DAGs).

Aws managed airflow 2.0