In this post, I am going to explain “what is state backend”, “which options do we have for state backend” , “how to configure state backend for your …
In this post, I am going to explain, how Flink starts itself, and what happens when you submit your job to the Standalone Cluster setup
- Consists of at least one master process and at least one TaskManager process that run on one or more machines.
- All processes run as regular Java JVM process.
- The master process runs a Dispatcher and a ResourceManager in separate threads.
- Once they start running, the TaskManagers register themselves at the ResourceManager
What happens when you submit your job?
- A client submits a job to the Dispatcher, which internally starts a JobManager thread and provides the JobGraph for execution.
- The JobManager requests the necessary processing slots from the ResourceManager.
- ResourceManager requests slot(s) from TaskManagers
- TaskManager offers their free slots to the JobManager
- JobManager executes the tasks.
- In a standalone deployment, the master and workers are not automatically restarted in the case of failure.
- A job can recover from a worker failure if a sufficient #processing slots is available. This can be ensured by running one or more standby workers
Last but not least wait for the next post…