Graceful Shutdown in Spring Boot and multithread environment

Spring provides reliable shutdown of the application using feature called Graceful Shutdown. With Graceful shutdown feature we can close the application while allowing any ongoing tasks to be completed, releases resources and ensuring data integrity.

We definetely need graceful shutdown feature, because while we are deploying new application, we have to wait ongoing threads to be completed otherwise some operations, such as withdraw money, can be lost. And most probably we should never understand what was wrong!!

Github Link

If you only need to see the code, here is the github link

Enable Graceful shutdown in Spring Boot

To enable graceful shutdown feature, please update your properties (or yaml file):

server.shutdown=graceful

server:
  shutdown: "graceful"

Right now if spring boot application is closed, Tomcat or Netty will not accept new requests at network layer. And as far as I know using only server.shutdown property means that server gets shutdown immediately.

However, if you want your server to wait n amount of time before shutdown, you may use property called spring.lifecycle.timeout-per-shutdown-phase. For instance if we add:

spring.lifecycle.timeout-per-shutdown-phase=60s

Then server will wait 60 seconds to complete active requests(requests were send before shutdown operation) to finish their jobs

Note: You may not test very well graceful shutdown feature in your IDE. Because your spring application needs a SIGTERM signal to apply graceful shutdown operation.

SIGTERM is the signal that is typically used to administratively terminate a process. For instance when you use $ kill 1234 command in your terminal, you are actually sending SIGTERM signal to specific process to terminate.

Let’s learn how to use graceful shutdown feature in spring boot by working on simple project. Please create the simple spring boot project with the following dependency:

Note: I am going to use spring boot version 3.1.5

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

And also create dummy controller to simulate shutdown feature:

@RestController
public class UserController {

    @GetMapping("/hello")
    public String hello() {
        return "Hello world";
    }
}

Then let’s configure graceful shutdown feature by updating the application.properties file:

# enable graceful shutdown
server.shutdown=graceful
# configure the timeout period
spring.lifecycle.timeout-per-shutdown-phase=20s

Let’s also assume that our “Hello world” message takes 7s before returning the response:

@RestController
public class UserController {

    @GetMapping("/hello")
    public String hello() throws InterruptedException {
        log.info("Waiting for 7 seconds");
        Thread.sleep(7000);
        log.info("Returning response");
        return "Hello world";
    }
}

Now while the application runs and after we send the request to “/hello” endpoint, if someone kills the application in the meantime, the request will also be killed. However because we have enabled the graceful shutdown feature, spring boot will wait at most 20 seconds to allow current request to be completed.

After run the application, trigger the endpoint, then before getting response, kill the application

$ curl -X GET http://localhost:8080/hello

Here are the log statements in my application:

...  INFO 115261 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms
...  INFO 115261 --- [nio-8080-exec-1] c.m.g.controller.UserController          : Waiting for 7 seconds
...  INFO 115261 --- [ionShutdownHook] o.s.b.w.e.tomcat.GracefulShutdown        : Commencing graceful shutdown. Waiting for active requests to complete
...  INFO 115261 --- [nio-8080-exec-1] c.m.g.controller.UserController          : Returning response
...  INFO 115261 --- [tomcat-shutdown] o.s.b.w.e.tomcat.GracefulShutdown        : Graceful shutdown complete

Now as you can see, the moment I shutdown the application, spring boot was print log statements “Commencing graceful shutdown …”

If you want to see in more detail, let’s also add @PreDestory hook into our controller:

@Slf4j
@RestController
public class UserController {

    @GetMapping("/hello")
    public String hello() throws InterruptedException {
        // ...
    }

    @PreDestroy
    public void destroy() {
        log.info("This will be called after graceful shutdown");
    }
}

...  INFO 115885 --- [nio-8080-exec-1] c.m.g.controller.UserController          : Waiting for 7 seconds
...  INFO 115885 --- [ionShutdownHook] o.s.b.w.e.tomcat.GracefulShutdown        : Commencing graceful shutdown. Waiting for active requests to complete
...  INFO 115885 --- [nio-8080-exec-1] c.m.g.controller.UserController          : Returning response
...  INFO 115885 --- [tomcat-shutdown] o.s.b.w.e.tomcat.GracefulShutdown        : Graceful shutdown complete
...  INFO 115885 --- [ionShutdownHook] c.m.g.controller.UserController          : This will be called after graceful shutdown

Right now, everything works as expected, but when we are in the multi-thread environment (most probably you will use many threads in your spring boot application), things get a little be confused

Graceful shutdown in multithread environment

First let’s update the our controller to use multithread and one of threads will call the Thread.sleep method:

public class UserController {
    ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(4);

    @GetMapping("/hello")
    public String hello() throws InterruptedException {
        log.info("Waiting for 7 seconds");
        executor.execute(() -> {
            try {
                Thread.sleep(7000);
                log.info("Completed 7seconds in thread");
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }
        });

        return "Hello world";
    }

    @PreDestroy
    public void destroy() {
        log.info("This will be called after graceful shutdown");
    }
}

Run the application and send the same curl request:

$ curl -X GET http://localhost:8080/hello
Hello world

As you can see, request will send the response immediately. And if you don’t kill the application immediately, here are the log statements you will see:

...  INFO 117405 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms
...  INFO 117405 --- [nio-8080-exec-1] c.m.g.controller.UserController          : Waiting for 7 seconds
...  INFO 117405 --- [nio-8080-exec-1] c.m.g.controller.UserController          : Returning response
...  INFO 117405 --- [pool-2-thread-1] c.m.g.controller.UserController          : Completed 7seconds in thread

It is clear that “Completed 7 seconds” statement will be seen after the response.

Now the question is: what would happen if someone kills the application within 7 seconds? Will graceful shutdown feature wait for another thread to be completed ? The answer is no. If you kill the application within 7 seconds:

...  INFO 117126 --- [nio-8080-exec-1] c.m.g.controller.UserController          : Waiting for 7 seconds
...  INFO 117126 --- [ionShutdownHook] o.s.b.w.e.tomcat.GracefulShutdown        : Commencing graceful shutdown. Waiting for active requests to complete
...  INFO 117126 --- [tomcat-shutdown] o.s.b.w.e.tomcat.GracefulShutdown        : Graceful shutdown complete
...  INFO 117126 --- [ionShutdownHook] c.m.g.controller.UserController          : This will be called after graceful shutdown

(Important) process inside the another thread will be lost. Actually neither JWV nor tomcat has aware of child thread.

To solve that issue we can leverage the shutdown hook (@PreDestory). Inside the hook we can wait until there is no active thread. Here is the new pre-destroy version:

@Slf4j
@RestController
public class UserController {
    ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(4);

    // ...

    @PreDestroy
    public void destroy() {
        log.info("This will be called after graceful shutdown");

        // while there is an active thread
        while (executor.getActiveCount() > 0) {
            try {
                // wait another 2 seconds and check again
                Thread.sleep(2000);
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }
        }
        log.info("Definitely destroy all active threads");
    }
}

Now, if you shutdown your application within 7 seconds:

...  INFO 118358 --- [nio-8080-exec-1] c.m.g.controller.UserController          : Waiting for 7 seconds
...  INFO 118358 --- [nio-8080-exec-1] c.m.g.controller.UserController          : Returning response
...  INFO 118358 --- [ionShutdownHook] o.s.b.w.e.tomcat.GracefulShutdown        : Commencing graceful shutdown. Waiting for active requests to complete
...  INFO 118358 --- [tomcat-shutdown] o.s.b.w.e.tomcat.GracefulShutdown        : Graceful shutdown complete
...  INFO 118358 --- [ionShutdownHook] c.m.g.controller.UserController          : This will be called after graceful shutdown
...  INFO 118358 --- [pool-2-thread-1] c.m.g.controller.UserController          : Completed 7 seconds in thread
...  INFO 118358 --- [ionShutdownHook] c.m.g.controller.UserController          : Definitely destroy all active threads

With leverage of the preDestory method, we can wait until there is no active thread in the executor. Even it isn’t most the reliable way for the multithread application, you can use the PreDestory hook.

In conclusion:

Graceful shutdown is supported with all four embedded web servers (Jetty, Reactor Netty, Tomcat, and Undertow).
Graceful shutdown feature allows us to wait uncompleted task to be completed within n amount of time.
With using Graceful shutdown, we can ensure the data integrity.
However using graceful feature as it is, isn’t enought especially in multithread environment(for instance multithreaded-monolith applications). To be able to wait all threads before shutting down the application, we can leverage the @PreDestroy hook.