Apache Flink Series 9 - How Flink & Standalone Cluster Setup Work?

  • |
  • 04 May 2020
Post image

In this post, I am going to explain, how Flink starts itself, and what happens when you submit your job to the Standalone Cluster setup

Standalone Cluster

  • Consists of at least one master process and at least one TaskManager process that run on one or more machines.
  • All processes run as regular Java JVM process.
  • The master process runs a Dispatcher and a ResourceManager in separate threads.
  • Once they start running, the TaskManagers register themselves at the ResourceManager

What happens when you submit your job?

  • A client submits a job to the Dispatcher, which internally starts a JobManager thread and provides the JobGraph for execution.
  • The JobManager requests the necessary processing slots from the ResourceManager.
  • ResourceManager requests slot(s) from TaskManagers
  • TaskManager offers their free slots to the JobManager
  • JobManager executes the tasks.
  • In a standalone deployment, the master and workers are not automatically restarted in the case of failure.
  • A job can recover from a worker failure if a sufficient #processing slots is available. This can be ensured by running one or more standby workers

Last but not least wait for the next post…

You May Also Like