Встал вопрос выбора аркистратора для ELT.
Что посоветуете?

данные лежат hadoop + oracle, но берем мы их отовсюду, все написанно на bash скриптах.
Ищем тулу которая бы позволяла удобно упарвять именно джобами, и все что с этим связанно (перезапуск с места падения, перезапуск по иерархии для какой-то таблицы, мониторинг процесса), поддержка запуска 1000 jobs

System main characteristics:

Environment – Hadoop(CDH 5.7), Oracle DB, ODI
Current number of configured jobs – ~ 5000
Current max number of simultaneously active jobs - ~600

Scheduler main characteristics:

- Ability to connect to heterogeneous environments:
Secure Storage for connecting info(login/password)
Ability to use “connect” templates with diff. variables(i.e. connect to diff. hosts with same characteristics)
- Ability to configure thousands of jobs(~ 10000) and more;
- Ability to simultaneously run of ~1000 jobs and more;
- Ability to combine jobs into sub-groups and sub-groups into groups;
- Ability to management of complex dependencies and helps users quickly resolve workflow bottlenecks with detailed analytics;
- Ability to configure execution rules and dependencies on group level, sub-group level, single job level(group/sub-group/job dependencies, time dependencies, variable dependencies);(limited)
- Ability to build graphical workflow based on dependencies(Predecessors, Successors);
- Ability to flexible manage job, sub-groups, groups(Need to be tested):
Hold/Stop, Release/Resume, Abort/Cancel, Rerun/Rerun with dependencies(ability to run workflow from any step with all following dependencies) and etc.;
- Fault tolerance and high availability;
- Ability to configure flexible actions for special events, i.e. job failed – send email and run special job and etc.;
- Intuitive User Interface for monitoring/configuring/creating jobs and etc.;

Apache Airflow - знаем но хотим что-то посерьезнее