Introduction

How it started

Some time ago, when a developer wanted to execute certain tasks in an application concurrently, they needed to create sufficient number of threads and manage them on their own. This adds another level of complexity to the app, since concurrency opens gate for various kinds of race conditions, such as dead locks, memory leaks, lost writes or outdated reads.

Developers quickly realized that it would have been better if we had moved thread management into a separate API, so that this functionality could have been developed independently with required cautiousness, being even more performant - and this is what thread pools do.

What is a thread pool?

Thread pool is a pool of threads that can be reused to execute tasks. Instead of creating a new thread manually, tasks are submitted to a thread pool which manages the execution of given task. What it brings?

• Easier to work with: Hard thread management process is delegated out, developers can focus on writing business logic.
• Performance boost: Creating, deleting and switching between threads is not a lightweight task for an OS. Having threads prepared earlier for the specific use-case allows to get better throughput and responsiveness.

Threads of a thread pool are often called worker threads. Their purpose is to simply execute given work. The pool usually contains a task queue, which is a queue to which new tasks are added, and then fetched by worker threads.

Workflow of a thread pool can be described in the following steps:

• Pool preparation: Creating and preparing a pool, a queue and worker threads to accept the work.
• Task acceptance: Taking a task and adding it to a queue.
• Task assignment: Taking a task from a queue and assigning it to a worker thread.
• Task execution: Executing a task by a worker thread.
• Thread freeing: Putting the thread back to a pool of available worker threads.

Thread pools available in Java

Java provides API, called as Java Concurrency API, that besides sharing other useful classees for concurrent applications, allows to create various kinds of thread pools. Good starting point is Executors class, which shares a couple of static factory methods that return an instance of ExecutorService interface, to create various kinds of thread pools. I try to explain them below.

Fixed-size thread pool

Very simple pool that initiates with a number of worker threads that remains constant throughout the pool's lifecycle. Useful when you know exactly how many threads you need to perform certain work (e.g., batch processing of data which amount is rather fixed but still big enough to split it onto multiple threads).

Methods

• Executors.newFixedThreadPool(int numberOfThreads) - creates a new fixed-size thread pool with given number of threads.

Example

public class Main {
    public static void main(String[] args) {
        ExecutorService executorService = Executors.newFixedThreadPool(3);
        executorService.submit(newTask(1, 100L));
        executorService.submit(newTask(2, 100L));
        executorService.submit(newTask(3, 100L));
        executorService.submit(newTask(4, 100L));
        executorService.close();
    }

    private static Runnable newTask(int taskNo, long processingTime) {
        return () -> {
            try {
                System.out.println("Started processing task #" + taskNo);
                Thread.sleep(processingTime);
                System.out.println("Finished processing task #" + taskNo);
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }
        };
    }
}

Result

Started processing task #2
Started processing task #1
Started processing task #3
Finished processing task #3
Finished processing task #2
Finished processing task #1
Started processing task #4
Finished processing task #4

As you can see, since I initialzied the pool with 3 threads, the task #4 started it's execution once after previous tasks finished, since they occupied all threads available in the pool.

Single-threaded pool

A thread pool that contains only a single thread to execute tasks. The pool just collects tasks in a queue and then runs them sequentially on a single thread. Even though this pool may seem to be overly simplistic, it can thrive in cases where tasks don't take much computation time and we really want to avoid any race conditions by simply having just a single thread. For example, in memory databases like Redis are based on single-threaded model, which allowed it to be remarkably performant. This pool works like a fixed-size thread pool with only 1 thread.

Methods

• Executors.newSingleThreadExecutor() - creates a new single-threaded pool.

Example

public class Main {
    public static void main(String[] args) {
        ExecutorService executorService = Executors.newSingleThreadExecutor();
        executorService.submit(newTask(1, 100L));
        executorService.submit(newTask(2, 100L));
        executorService.submit(newTask(3, 100L));
        executorService.submit(newTask(4, 100L));
        executorService.close();
    }

    private static Runnable newTask(int taskNo, long processingTime) {
        return () -> {
            try {
                System.out.println("Started processing task #" + taskNo);
                Thread.sleep(processingTime);
                System.out.println("Finished processing task #" + taskNo);
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }
        };
    }
}

Result

Started processing task #1
Finished processing task #1
Started processing task #2
Finished processing task #2
Started processing task #3
Finished processing task #3
Started processing task #4
Finished processing task #4

As mentioned, tasked are executed one-by-one.

Cached thread pool

Creates new worker threads when needed, but can reuse previously created threads if they are available. A used thread, after it finished its work, will stay available only for a certain amount of time (60 secs for Java 24) - once this time passes, the thread is being removed from the pool. This pool may be useful in case of varying workload, especially for short-lived tasks.

Methods

• Executors.newCachedThreadPool() - creates a new cached thread pool.

Example

public class Main {
    public static void main(String[] args) throws InterruptedException {
        ExecutorService executorService = Executors.newCachedThreadPool();
        executorService.submit(newTask(1, 100L));
        executorService.submit(newTask(2, 100L));
        executorService.submit(newTask(3, 100L));
        Thread.sleep(200L);
        executorService.submit(newTask(4, 100L));
        executorService.submit(newTask(5, 100L));
        executorService.submit(newTask(6, 100L));
        executorService.submit(newTask(7, 100L));
        executorService.close();
    }

    private static Runnable newTask(int taskNo, long processingTime) {
        return () -> {
            try {
                System.out.println("Started processing task #" + taskNo + ", by " + Thread.currentThread().getName());
                Thread.sleep(processingTime);
                System.out.println("Finished processing task #" + taskNo);
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }
        };
    }
}

Result

Started processing task #1, by pool-1-thread-1
Started processing task #2, by pool-1-thread-2
Started processing task #3, by pool-1-thread-3
Finished processing task #1
Finished processing task #2
Finished processing task #3
Started processing task #4, by pool-1-thread-3
Started processing task #5, by pool-1-thread-2
Started processing task #6, by pool-1-thread-1
Started processing task #7, by pool-1-thread-4
Finished processing task #5
Finished processing task #6
Finished processing task #4
Finished processing task #7

As you can see, for task #4, #5, and #6 - the pool reused previously created threads, but for task #7, it needed to create a new thread.

Scheduled thread pool

This is actually quite a different pool. It returns an instance of ScheduledExecutorService, which is designed to schedule tasks at intervals or with delays. Example of use-cases are batch jobs that should run once a day, real-time updates sent to users on intervals like live event updates and so on.

Methods

• Executors.newScheduledThreadPool(int corePoolSize) - creates a pool with given number of threads, that are kept in the pool even if they are inactive.
• Executors.newSingleThreadScheduledExecutor() - single-threaded variant, same as newScheduledThreadPool(1).

New interface shares new schedule methods, allowing to define the mentioned interval or delay.

Example

public class Main {
    public static void main(String[] args) {
        ScheduledExecutorService executorService = Executors.newScheduledThreadPool(3);
        executorService.schedule(newTask(1, 100L), 130L, TimeUnit.MILLISECONDS);
        executorService.schedule(newTask(2, 100L), 120L, TimeUnit.MILLISECONDS);
        executorService.schedule(newTask(3, 100L), 110L, TimeUnit.MILLISECONDS);
        executorService.schedule(newTask(4, 100L), 100L, TimeUnit.MILLISECONDS);
        executorService.close();
    }

    private static Runnable newTask(int taskNo, long processingTime) {
        return () -> {
            try {
                System.out.println("Started processing task #" + taskNo);
                Thread.sleep(processingTime);
                System.out.println("Finished processing task #" + taskNo);
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }
        };
    }
}

Result

Started processing task #4
Started processing task #3
Started processing task #2
Finished processing task #4
Started processing task #1
Finished processing task #3
Finished processing task #2
Finished processing task #1

As you can see, task #4 is executed first, as expected by the defined delay. However, it's worth noting that since the pool contains only 3 threads, task #1 needs to wait for other tasks to finish because there are no threads available for it, which means that it started later comparing to what's defined in the scheduled method (planned to start after about 130ms, but started after about 200ms). Remember to create a scheduler pool with sufficient amount of threads, unless you can accept the increased delay.

Work stealing pool

This pool is also different, since it's based on work stealing algorithm, which is an algoritm that allows for more sophisticated work distribution between the threads.

Usually, there is only one task queue for the entire thread pool. However, pushing and pulling objects from a queue are blocking operations, meaning that only one thread at a time can push/pull objects, but those methods do not block each other (e.g., when one thread is pushing new objects, another thread can pull objects from the other end of the same queue), which is why this data structure is often used in concurrent environments. In case where objects are being intensively pushed and pulled (e.g., small tasks, like event-processing tasks), you may end up in a situation known as threads contention, where two or more threads often block each other, meaning that they are not utilized efficiently.

Work stealing pool addresses this issue by creating more than one queue, each often assigned to specific worker threads. New tasks are pushed to the queue on which it's most probable to execute the task first. Additionally, threads that are free and don't have any tasks pending on their queues, are allowed to take tasks from other queues, which is called as work stealing, hence why this pool is named as work stealing pool. This allow to signicantly reduce the thread contention, and make the pool more performant.

Methods

• Executors.newWorkStealingPool(int parallelism) - creates a new work stealing pool with given parallelism level, which is nothing but the target number of worker threads that would be created if needed, or removed if no longer being actively used.
• Executors.newWorkStealingPool() - creates a new work stealing pool with parallelism level set as number of available processors.

Example

public class Main {
    public static void main(String[] args) {
        ExecutorService fixedThreadPool = Executors.newFixedThreadPool(20);
        executeTasks(fixedThreadPool);

        ExecutorService workStealingPool = Executors.newWorkStealingPool(20);
        executeTasks(workStealingPool);
    }

    private static void executeTasks(ExecutorService executorService) {
        long startTime = System.nanoTime();

        for (int i = 0; i < 5; i++) {
            executorService.submit(newTask(500L));
        }
        executorService.close();

        long endTime = System.nanoTime();
        System.out.println("Total processing time: " + (endTime - startTime) / 1_000_000L);
    }

    private static Runnable newTask(long proceessingTime) {
        return () -> {
            try {
                Thread.sleep(proceessingTime);
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }
        };
    }
}

Result

Total processing time: 509
Total processing time: 501

As you can see, in case where there is thread contention, work stealing pool can perform better than fixed-size thread pool, even though the latter has its threads already prepared before processing. The example here is not as realistic as it could be on production environment, but the bigger the scale, the bigger the difference in performance could be.

Additionally, it's worth to mention that work stealing pool is based on ForkJoinPool implementation, which interface additionally allows to submit tasks that can be divided into independent subtasks, processed by other threads, and their results can be joined later once all those divided tasks are completed. For such resursive work, the algorithm performs even better. However, it's a topic for another post, but if you are intersted, I encourage you to review this class as well.

Thread per task executor

This type uses provided ThreadFactory to spawn a new thread every time a new task comes in. In certain scenarios it might be sufficient, but often lead to inaccurate use of your processing resourses. However, instead of spawning platform threads you can spawn virtual threads, where using this executor can make much more sense.

Methods

• Executors.newThreadPerTaskExecutor(ThreadFactory threadFactory) - creates a new executor that spawns new threads using given thread factory.
• Executors.newVirtualThreadPerTaskExecutor() - same, but spawns virtual threads.

Example

public class Main {
    public static void main(String[] args) {
        ExecutorService platformThreadPerTaskExecutor = Executors.newThreadPerTaskExecutor(Thread.ofPlatform().factory());
        executeTasks(platformThreadPerTaskExecutor);

        ExecutorService virtualThreadPerTaskExecutor = Executors.newVirtualThreadPerTaskExecutor();
        executeTasks(virtualThreadPerTaskExecutor);
    }

    private static void executeTasks(ExecutorService executorService) {
        long startTime = System.nanoTime();

        for (int i = 0; i < 1000; i++) {
            executorService.submit(newTask(500L));
        }
        executorService.close();

        long endTime = System.nanoTime();
        System.out.println("Total processing time: " + (endTime - startTime) / 1_000_000L);
    }

    private static Runnable newTask(long proceessingTime) {
        return () -> {
            try {
                Thread.sleep(proceessingTime);
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }
        };
    }
}

Result

Total processing time: 576
Total processing time: 515

As you can see, virtual threads can perform better in such scenario.

Can I use virtual threads in other pools?

Yes! It's possible to use virtual thread factory to spawn threads in almost all mentioned pools. There is usually a variant that accepts custom ThreadFactory, where we could use Thread.ofVirtual().factory() to use virtual threads, instead of platform threads.

For instance, in case of cached thread pool, instead of Executors.newCachedThreadPool(), we can call Executors.newCachedThreadPool(Thread.ofVirtual().factory()). You can find more examples by reviewing documentation of Executors class.

Summary

As you can see, there are a couple types of thread pools, each addressing different group of use-cases. Thankfully, with new Java Concurrency API, it became much easier to create and work on those pools. Start by reviewing static factory methods provided by Executors class, they are often sufficient to address various needs. Once you become more and more familiar with them, try to understand implementation details under the hood and how you can leverage them to create even more sophisticated pools, more suited to what your application currently needs.

Thanks for reading, hopefully I managed to teach you something :)