Async request processing in Spring MVC - Part 2

Blog Image-05.svg

Asynchronous request processing can provide performance benefits and enable new use-cases like streaming data. Learn how this is handled in the background by Spring MVC for better understanding and future troubleshooting!

Async request processing in Spring MVC - Part 2

As we have seen in the previous part, the servlet handles the http request processing by directly operating on the http servlet request and response objects. However, wouldn't it be convenient if it also could

figure out which http request has to go to which controller method
convert the payload to the corresponding DTO
resolve path and request variables
create the appropriate serialized response

Sounds like a lot of tedious work. Luckily, Spring already does that for us by providing a single servlet called the Dispatcher Servlet!

Dispatcher Servlet

SpringMVC.png

Imagine you have a Spring Application with some defined Controllers that provide request mappings (e.g. methods annotated with @RequestMapping). In this case, whenever a http request reaches our application the Dispatcher Servlet will invoke specific handlers that

look at the request URI, extract request parameters, and route to the corresponding controller method
automatically convert the request body into the Java DTO specified in your controller method
serialize whatever your controller method returns into a transport-friendly format such as JSON and hand it over to the servlet container

For all of this to happen, Spring provides a bunch of HandlerAdapters. In the context of asynchronous request processing, we'll have a look specifically at ReturnValueHandlers which - you guessed it - are selected based on what kind of object you return from your controller method.

Synchronous Request Processing

When doing synchronous processing, we only return from our request handler (i.e. your controller method annotated with @RequestMapping) after the application has done its work, blocking the servlet worker thread in the process.

SynchronousRequest.png

In this mode, the amount of work our application can do in parallel is limited by the tomcat worker thread pool size.

Now, if our application logic processes some I/O heavy workload or delegates work to another thread pool (for resource management or other reasons) our servlet thread is effectively just sitting idle but still consuming resources instead of working on the next request.

Wouldn't it be nice if there was a way to use our threads more efficiently?

Asynchronous Request Processing

And of course, there is! Spring provides several mechanisms to defer the result of an http request and return it once it's ready. One example would be the very fitting DeferredResult.

Here we would submit a long-running task to an executor service and set the result once we are done sleeping.

              @GetMapping("/deferred")
DeferredResult<String> deferredResult() {
    var result = new DeferredResult();
    var executor = Executors.newSingleThreadExecutor();

    executor.submit(() -> {
        Thread.sleep(1000);
        result.setResult("This is the result");
    });

    return result;
}


            

Comparing our pseudo code to the figure we saw for synchronous request processing, the async processing would look like this:

DeferredResult.png

Provided we have a long-running workload that is delegated to a separate application thread pool we could just leverage this thread to return our result. This enables Tomcat to reuse the worker thread for other maybe more fast-running requests instead of blocking it for the whole duration.

The workflow looks as follows:

Like before, our request handler is invoked by the Dispatcher Servlet
The handler method delegates the task to an application-managed thread pool, passing a DeferredResult object
The handler method returns the same DeferredResult object. This invokes a special ReturnValueHandler that puts the request into async mode (see below)
At some point the application thread computes the result and passes it to the DeferredResult
This causes a new dispatch event in the servlet returning the readily available response

As shown in the figure above, this process only blocks the servlet worker thread for a minimal time. First, it sets up the async request processing by using the DeferredResult and later another worker thread fetches the readily computed result and returns it to the caller.

Now what does "putting a request into async mode" actually mean?

AsyncDispatch.png

When returning a DeferredResult from a controller method, the corresponding ReturnValueHandler calls startAsync() on the http request before returning it. This signals the Servlet Container to release the worker thread but to keep this request open as the processing is not done yet.

All of this happens inside the initial servlet dispatch (REQUEST) as shown in this pseudo code

              public void handle(DeferredResult deferredResult, HttpServletRequest request) {
    var asyncContext = request.startAsync();
    deferredResult.setCompletionCallback(result -> {
        AsyncManager.setResultForRequest(asyncContext, result);
        asyncContext.dispatch();
    });
}

            

At some later point, when the actual result is passed to the DeferredResult we invoke the completion callback which triggers a second dispatch (ASYNC).

              public void setResult(Object result) {
    completionCallback.accept(result);
}

            

The async dispatch just retrieves whatever was stored by the completion callback and returns it to the caller in the same way as a regular synchronous request would. From the caller's perspective, both synchronous and the asynchronous workflow are the same.

              protected void doGet(HttpServletRequest httpRequest, HttpServletResponse httpResponse) {
    if (httpRequest.getDispatcherType() == DispatcherType.ASYNC) {
        var asyncResult = AsyncManager.getResultForRequest(httpRequest);
        httpResponse.getOutputStream().write(asyncResult);
        return;
    }
    ...
}

            

One thing to consider is that your request passes the filter chain twice (once for each dispatch) in case you do any fancy custom processing.

Streaming Data

Not having to wait for the full response to be complete also provides another benefit - we can even send intermediate results when they are ready by streaming data!

This can be achieved in multiple ways, one of them being using the ResponseBodyEmitter. There are specialized subclasses of it like the SseEmitter that can be used for streaming mechanisms like server-sent events (SSE).

ServerSentEvents.png

Similar to the DeferredResult, the request handler returns immediately after creating the ResopnseBodyEmitter and frees up the servlet worker thread. Only this time the application logic can publish multiple results which can be immediately flushed to the client without closing the connection. This is usually wrapped in the SSE format mentioned above or simply using newline delimited json objects.

The pseudo-code for our ReturnValueHandler in spring would look like this.

              public void handle(SseEmitter emitter, HttpServletRequest request,
                   HttpServletResponse response) {
    response.setContentType("text/event-stream"); // server sent events format
    var asyncContext = request.startAsync();
    emitter.initialize(asyncContext);
}

            

We set the content-type of the response to a streaming format - in this example SSE - and put the request into async mode.

The emitter itself would provide the following two (oversimplified) methods:

              public void send(String value) {
    var payload = "data: " + value + "\n\n"; // SSE format
    asyncContext.getResponse().getOutputStream().write(payload.getBytes());
    asyncContext.getResponse().flushBuffer();
}

public void complete() {
    asyncContext.dispatch();
}

            

Every time the application code wants to send an object it is converted to the streaming format (here SSE) and flushed to the response output stream. Once the request is completed, an ASYNC dispatch is triggered, cleaning up the request.

Another way to stream data would be making use of the reactive type Flux which internally works slightly differently than our ResponseBodyEmitter but will produce the same result.

Async request processing in Spring MVC - Part 2