Last month, I tried out using Bun as a custom JavaScript Lambda runtime and was surprised by the performance difference between Bun and AWS's managed Node.js runtime. The Node runtime had a shorter cold start time, and the invocations had a shorter duration. The cold start times weren't surprising given AWS's caching and performance optimizations, but the difference in the invocation durations was surprising.

Since then, I have been working on a better benchmark to compare performance between AWS's Node.js runtime and the custom Bun and Deno runtimes.

The Setup & Comparison

I originally generated a JWT as a simple test of the runtime's compute performance. For this new comparison, I continued to use hashing but instead generated 50 SHA3-512 hashes per invocation. I felt that generating a single hash was too small a test, and generating 50 hashes would result in a better comparison.

For Bun and Deno, I wrote the handler in TypeScript, taking advantage of the runtime's capabilities. For the Node.js handler, I used plain JavaScript. If the runtime had a hashing function built in, I used that, hoping for the best performance. For Bun, I used the CryptoHasher function. For Deno and Node.js, I used the Node.js crypto package and the createHash function. The source code for the handler functions is below.

I set up the Bun custom runtime using Lambda Layers, like I had previously. The Deno custom runtime is now container-based; it used Lambda Layers when I last used it. I used the Dockerfile that is available in Deno's documentation and only modified the filename for the handler.

//
// bun-handler.ts
//
const HASH_NUMBER = 50;

export default {
  async fetch(): Promise<Response> {
    const output = [];

    for (let i = 0; i < HASH_NUMBER; i++) {
      const hasher = new Bun.CryptoHasher('sha3-512');
      hasher.update(
        `${new Date().toISOString()}-${Math.random()}-${Math.random()}`,
      );
      output.push(hasher.digest('hex'));
    }

    return new Response(JSON.stringify(output));
  },
};

//
// deno-handler.ts
//
import { createHash } from 'node:crypto';

const HASH_NUMBER = 50;

Deno.serve(() => {
  const output = [];

  for (let i = 0; i < HASH_NUMBER; i++) {
    const hasher = createHash('sha3-512');
    hasher.update(
      `${new Date().toISOString()}-${Math.random()}-${Math.random()}`,
    );
    output.push(hasher.digest('hex'));
  }

  return new Response(JSON.stringify(output));
});

//
// node-handler.js
//
const { createHash } = require('node:crypto');

const HASH_NUMBER = 50;

export async function handler() {
  const output = [];

  for (let i = 0; i < HASH_NUMBER; i++) {
    const hasher = createHash('sha3-512');
    hasher.update(
      `${new Date().toISOString()}-${Math.random()}-${Math.random()}`,
    );
    output.push(hasher.digest('hex'));
  }

  return output;
}

I built an AWS CDK project to deploy the different Lambda functions, the infrastructure necessary to trigger the functions, and the dashboard for the aggregated results. I've created a GitHub repository with everything if you want to try out the experiment or see how I used the custom runtimes.

I used an event-based approach to invoke the Lambda functions and run the hashing code. Each Lambda was triggered based on messages being added to an SQS queue. I spaced out when messages would become available in the queue using the delay property to avoid having a surge of cold starts and parallel invocations. I wanted to be sure I got invocations that used an already running instance of the function. After all, having a readily available function and everything cached is often when you have your best performance.

Every couple of hours, a scheduled Lambda function added 30 messages to each SQS queue with a steadily increasing delay until the max delay of 5 minutes was reached.

I enabled JSON-formatted logs for the functions; I found I was able to parse out the metrics I needed easily in CloudWatch. I relied on the metrics reported by Lambda in the platform.report log entry for this analysis.

The Results

I used the duration and initialization duration measurements provided by the Lambda service for these results. Both numbers are in milliseconds. Remember that 1000 milliseconds (ms) is 1 second. The duration is how long it took the function to process the event. The initialization duration is how long it took the function's Init phase to complete. This happens when a new execution environment is being set up. Technically, this may not include the time required to download the source code, but that is difficult to measure. I'm simplifying things and considering the initialization duration and the cold start duration equivalent for this comparison.

I had 1326 invocations per runtime, totaling 3978 invocations overall. Each runtime experienced around 70 cold starts.

ℹ️ Percentiles are values below which a given percentage of all the values in the dataset exist. So if 15.190 is our value for the 50th percentile, then 50% of our values are below 15.190. You can learn more on Wikipedia and Amazon CloudWatch's documentation. In the tables below percentiles are written as "p10", "p25", etc. Where "p10" means "the 10th percentile", "p25" means "the 25th percentile", and so on.

Duration

When it comes to the invocation durations, Deno had the best averages, followed by the Node.js runtime. On average, Bun took more than twice the time to generate the hashes.

Looking across the percentiles, Deno and Node were pretty close all the way out to the 75th percentile. Even at the 75th percentile, Deno was about 5ms faster, which you probably wouldn't notice. Looking at Bun, up until about the 50th percentile, it wasn't drastically slower. You're not going to notice 4ms, or even 10ms. Let's be honest, you're probably not going to notice a 50ms difference. To put that in perspective, a human blink takes between 100ms and 400ms. But, with AWS, you're billed for that time, and it can add up at the end of your billing cycle.

Initialization Duration

Looking at the initialization durations, the Node runtime was the fastest on average. I don't find this surprising, since I imagine AWS has put a lot of effort into minimizing cold start times. Bun was the slowest on average; I expect that is because it uses Lambda Layers, where Deno uses a container.

Across the percentiles, there is a very small gap in the values for the Node runtime. To me, this indicates AWS has ensured these initialization durations for their Node runtime are fairly consistent. The difference between the 10th percentile and the 90th percentile is about 14ms. That's astoundingly consistent. The difference for Deno is about 30ms, and for Bun is 56ms. The initialization duration for the Bun runtime is consistently more than half a second. For Deno, it's under a quarter of a second more than 75% of the time.

In Summary

AWS's Node.js runtime for Lambda is very performant, and I am sure AWS has put a lot of work into ensuring that is the case. It has relatively consistent initialization durations and can generally be quite fast.

Deno's container-based custom runtime is currently more performant than Bun's Lambda Layer-based approach, for both initialization and invocation. That said, Bun's approach is easier to work with and deploy. Deno's approach means you may end up building a container for every Lambda function, or you end up with larger Lambda functions where you use a fraction of the files included. I'm impressed by how performant the Node.js runtime is, when Bun and Deno are meant to be more performant JavaScript runtimes.

These runtimes all perform well, especially if your application rarely has cold starts. If you need to gain every last bit of performance out of each Lambda invocation, your choice may matter, but if you're building a fairly basic API, I think you could choose any of these runtimes. Focus on the features you need and the developer experience you want.