Airflow has a very powerful UI and is written on Python and is developer friendly. hence It is extremely easy to create new workflow based on DAG. It provides both CLI and UI that allows users to visualize dependencies, progress, logs, related code, and when various tasks are completed. It has many features like Centralized logging, Great automation, scheduling options. It has Auto Retry failed tasks which depends on configuration. It is modular, scalable, and highly extensible. In Airflow, we have Parameterizing scripts as built in using Jinja templating engine. it provides analytics on search ranking and sessionization information to track users’ clickstream and time spent.
GUI:UI and Python
Tools compatable:Hive, Presto, MySQL, HDFS, Postgres, or S3
Event driven:Supports Event Based Scheduling
Programmability:UI (Need simple pyhton coding)
Oozie is an Open Source Java Web-Application. It provides out-of-the-box support for mapreduce, Pig, Hive, Sqoop, and Distcp, as well as jobs that are system-specific. It has features like Scalable, reliable, extensible and High availability. Using Oozie, Identical workflows can be parameterized to run concurrently. It allows for bulk kill, suspend, or resume jobs. Multiple coordinator and workflow jobs can be packaged and managed together through Oozie Bundle.
GUI:Command line, Java API and web browser as well
Tools compatable:mapreduce, Pig, Hive, Sqoop, Distcp and jobs that are system-specific
Event driven:supports both time-based and input-data based scheduling
Programmability:XML -- Easy to use
Azkaban is another open-source workflow manager.Simple web and HTTP workflow uploads can be done in Azkaban. It is Modular and pluginable for each Hadoop ecosystem.It has features like Tracking user actions, authentication, and authorization.It Provides a separate workspace for each new project and also Provides email alerts on SLAs, failures, and successes. It allows users to retry failed jobs.
GUI: Web browser only
Tools compatable:Compatible with any version of Hadoop
Event driven:Azkaban supports only time based scheduling
Programmability: Java
0 Comments