It is responsible for triggering the DAGs as well as the tasks according to their scheduled time and dependencies. Scheduler: As a workflow management platform, the core component that sits at the heart of airflow is a Scheduler. You’ll get a deep understanding of how each executor works, which is going to help you decide which one best suits your needs. We’ll start of with a generic Airflow architecture proceeding with what type of executors Apache Airflow provides with the detailed architecture view of each of them.īy the end of this tutorial, you’ll know how airflow manages and executes workflow pipelines using single node architecture executors as well as multi-nodes. This Exploration will lead us to know, how the various components of Apache Airflow actually interact with each other. Today in the second part of our series we’re going to explore and understand the core components and the modular design of Apache Airflow Architecture. In Airflow terms, a workflow is a DAG consisting of tasks. A Workflow is defined as sets of tasks arranged in specific order to deliver a specific business need. Before we proceed further, familiarize yourself with the term workflow, which we will be using quiet often in this article. Make sure you haven’t missed the first part of series Introduction to Airflow, in which we talked about definition, why and when its needed and core terminologies like DAG, Tasks, task dependencies etc. Welcome to the second tutorial of the series Apache Airflow for beginners, in which we’ll be diving deep into the Apache Airflow Architecture.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |