The plugins file is a ZIP file containing the plugins used by my DAGs.Optionally, I can specify a plugins file and a requirements file: The bucket name must start with airflow. Then, I select the S3 bucket and the folder to load my DAG code. I give the environment a name and select the Airflow version to use. In the Amazon MWAA console, I click on Create environment. How to Create an Airflow Environment Using Amazon MWAA Run your DAGs in Airflow – Run your DAGs from the Airflow UI or command line interface (CLI) and monitor your environment with CloudWatch.Upload your DAGs and plugins to S3 – Amazon MWAA loads the code into Airflow automatically.Create an environment – Each environment contains your Airflow cluster, including your scheduler, workers, and web server.You can use Amazon MWAA with these three steps: Amazon MWAA provides automatic minor version upgrades and patches by default, with an option to designate a maintenance window in which these upgrades are performed. To improve observability, Airflow metrics can be published as CloudWatch Metrics, and logs can be sent to CloudWatch Logs. Workflows in Airflow are authored as Directed Acyclic Graphs (DAGs) using the Python programming language.Ī key benefit of Airflow is its open extensibility through plugins which allows you to create tasks that interact with AWS or on-premise resources required for your workflows including AWS Batch, Amazon CloudWatch, Amazon DynamoDB, AWS DataSync, Amazon Elastic Container Service (Amazon ECS) and AWS Fargate, Amazon Elastic Kubernetes Service (Amazon EKS), Amazon Kinesis Firehose, AWS Glue, AWS Lambda, Amazon Redshift, Amazon Simple Queue Service (Amazon SQS), and Amazon Simple Notification Service (Amazon SNS). However, manually installing, maintaining, and scaling Airflow, and at the same time handling security, authentication, and authorization for its users takes much of the time you’d rather use to focus on solving actual business problems.įor these reasons, I am happy to announce the availability of Amazon Managed Workflows for Apache Airflow (MWAA), a fully managed service that makes it easy to run open-source versions of Apache Airflow on AWS, and to build workflows to execute your extract-transform-load (ETL) jobs and data pipelines.Īirflow workflows retrieve input from sources like Amazon Simple Storage Service (Amazon S3) using Amazon Athena queries, perform transformations on Amazon EMR clusters, and can use the resulting data to train machine learning models on Amazon SageMaker. With Airflow you can manage workflows as scripts, monitor them via the user interface (UI), and extend their functionality through a set of powerful plugins.

To do so, many developers and data engineers use Apache Airflow, a platform created by the community to programmatically author, schedule, and monitor workflows. As the volume and complexity of your data processing pipelines increase, you can simplify the overall process by decomposing it into a series of smaller tasks and coordinate the execution of these tasks as part of a workflow.

Aws apache airflow