HTTP compression spends CPU to save bytes

Compressing the 502,941-byte plaintext of HTTP’s own specification with gzip level 6 produced 139,862 bytes. Raising gzip to level 9 saved only another 960 bytes. On the same input, the median encoding time rose from 20.8 ms to 42.2 ms.

The obvious policy is to select the smallest output. That ignores whether the server compresses once before publishing or again while each request waits. On this input, zstd 3 matches gzip 6’s size about nine times faster: 2.3 ms versus 20.8 ms.

We’ll follow content negotiation, then measure where smaller responses justify encoding time. In a simple buffered model, request-time gzip 6 costs more time than it saves above about 140 Mbit/s. These are in-memory encoding measurements on RFC text, not browser load times, streaming tests or a representative sample of web traffic.

Negotiate the coding, not the media type#

Simply gzipping every response breaks the agreement with the receiver. RFC 9110’s Accept-Encoding rules let a client announce acceptable content codings, reversible transformations applied to response bytes. A quality value of zero rejects a coding. An empty field asks for no content coding.

The response’s Content-Encoding identifies the transformation actually applied. Content-Type still describes the decoded document, while Content-Length counts the compressed bytes. RFC 9110 Section 8.4 defines how content codings affect this metadata.

Compare the lengths in these illustrative response headers for the measured RFC file. Both responses decode to the same document.

HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 502941
Vary: Accept-Encoding

HTTP/1.1 200 OK
Content-Type: text/plain
Content-Encoding: gzip
Content-Length: 139862
Vary: Accept-Encoding

The gzip response has a smaller Content-Length with the same Content-Type. Both variants carry Vary because the request’s encoding preferences influenced selection. RFC 9110 Section 12.5.5 explains how Vary expands the cache key.

Use Next to reach the second message below and check which coding was actually selected. The initial step advertises gzip and Brotli as equally acceptable. The origin chooses an available gzip variant.

The second request rejects gzip, so the origin sends the uncompressed response. The figure shows two independent requests and omits caches.

Gzip wraps DEFLATE with a 10-byte fixed header, optional header fields, and an 8-byte trailer holding CRC-32 and size modulo 2³². The HTTP deflate label means DEFLATE inside a zlib wrapper, not raw DEFLATE.

Higher levels buy diminishing size reductions#

Brotli’s format combines LZ77 and Huffman coding, with context modeling and a static dictionary. Zstandard defines its own frame format and registers the zstd content coding. We’ll compare each codec separately on the same input.

The experiment uses the RFC Editor’s exact rfc9110.txt, downloaded on 2026-11-03 without text normalization. Check its SHA-256 before comparing measurements:

21c1cdce6ab0e5509b04d84a28000836c7a087cf786efe6f04877ebfff47232a

The checksum identifies the exact input, including its line endings. The table selects five profiles from the 22 measured settings and includes an uncompressed baseline. Measurements use Python 3.12.3, zlib 1.3, Brotli 1.1.0 and libzstd 1.5.5, with medians from seven runs.

Gzip 6 removes 502,941 - 139,862 = 363,079 bytes, or 72.19%. Brotli 11 removes another 139,862 - 108,502 = 31,360 bytes, but its median encoding time is about 43 times gzip 6’s. The uncompressed baseline has no encoding time because it does not call a compressor.

Within gzip, follow the curve from level 6 to level 9 and notice how little the output size changes.

The first five level increases save 173,551 - 139,862 = 33,689 bytes. The final three save 960 bytes, less than 0.2% of the original file.

Now compare all 22 settings by measured time rather than their level numbers. A point farther left costs less encoding time, and a point lower down leaves fewer bytes to transmit.

Zstd 3 produced 139,228 bytes in a median 2.3 ms, compared with gzip 6’s 139,862 bytes in 20.8 ms. Brotli 11 produced the fewest bytes but took the longest to encode. Compare size and time together. Codec level numbers are not a common scale across implementations.

The same level saves different byte counts#

The break-even rate derived in the next section depends on bytes saved per millisecond of encoding, so response size matters alongside the compression level. To see how sizes vary within RFC text, compare these 20 files compressed separately at gzip level 6. Each histogram entry counts one file.

RFC 768 shrinks from 5,896 to 1,924 bytes, while RFC 9110 shrinks from 502,941 to 139,862 bytes. The same level gives very different absolute savings across response sizes. A request-time policy needs the encoding cost beside those savings to decide whether compression pays off.

Pay once when bytes can be reused#

Model a buffered response by encoding it completely, then transmitting at $B$ bytes per second, ignoring client decoding and costs shared by both variants.

Here $U$ is the uncompressed size and $C$ is the compressed size. At 10,000,000 bytes/s, gzip 6’s 363,079 saved bytes avoid about 36.3 ms of transfer time. Subtracting its measured 20.8 ms encoding cost leaves about 15.6 ms.

For gzip 6, (502,941 - 139,862) / 0.020750 gives about 17.50 million bytes/s, or 140 Mbit/s. Above that rate, saved transfer time is smaller than encoding time. This threshold comes from the buffered model. Overlapping compression with transmission changes the model.

For an unchanged static asset, precompression moves encoding off the request path. The asset still needs negotiation and a correctly labeled body. When a runtime response already has valid gzip bytes, reuse them:

const coded = precompressedGzip;

This avoids request-time encoding while precompressedGzip matches the current source and selected coding.

For reusable public assets, measure smaller responses at build time and serve only codings the client accepts. For request-time encoding, measure the actual response mix and keep encoding time beside output size. The opening 960-byte improvement is real, but paying another 21 ms for it is a separate decision.

import ctypes as c
import ctypes.util
import gzip
import hashlib
import json
import pathlib
import platform
import statistics
import time
import urllib.request
import zlib

root = pathlib.Path("http-compression-data")
root.mkdir(exist_ok=True)
numbers = [768, 791, 1950, 1951, 1952, 3986, 6298, 7541, 7932, 8200,
           8259, 8446, 8878, 9000, 9110, 9111, 9112, 9113, 9114, 9293]
files = {}
for number in numbers:
    path = root / f"rfc{number}.txt"
    if not path.exists():
        path.write_bytes(urllib.request.urlopen(
            f"https://www.rfc-editor.org/rfc/rfc{number}.txt").read())
    files[path.name] = path.read_bytes()

enc = c.CDLL(ctypes.util.find_library("brotlienc"))
dec = c.CDLL(ctypes.util.find_library("brotlidec"))
zs = c.CDLL(ctypes.util.find_library("zstd"))
enc.BrotliEncoderVersion.restype = c.c_uint32
enc.BrotliEncoderMaxCompressedSize.argtypes = [c.c_size_t]
enc.BrotliEncoderMaxCompressedSize.restype = c.c_size_t
enc.BrotliEncoderCompress.argtypes = [c.c_int, c.c_int, c.c_int, c.c_size_t,
                                    c.c_void_p, c.POINTER(c.c_size_t), c.c_void_p]
dec.BrotliDecoderDecompress.argtypes = [c.c_size_t, c.c_void_p,
                                      c.POINTER(c.c_size_t), c.c_void_p]
zs.ZSTD_versionString.restype = c.c_char_p
zs.ZSTD_compressBound.argtypes = [c.c_size_t]
zs.ZSTD_compressBound.restype = c.c_size_t
zs.ZSTD_compress.argtypes = [c.c_void_p, c.c_size_t, c.c_void_p,
                            c.c_size_t, c.c_int]
zs.ZSTD_compress.restype = c.c_size_t
zs.ZSTD_isError.argtypes = [c.c_size_t]
zs.ZSTD_decompress.argtypes = [c.c_void_p, c.c_size_t, c.c_void_p, c.c_size_t]
zs.ZSTD_decompress.restype = c.c_size_t

def compress(data, codec, level):
    if codec == "gzip":
        return gzip.compress(data, compresslevel=level, mtime=0)
    if codec == "brotli":
        capacity = enc.BrotliEncoderMaxCompressedSize(len(data))
        output = c.create_string_buffer(capacity)
        size = c.c_size_t(capacity)
        if enc.BrotliEncoderCompress(level, 22, 0, len(data), data,
                                     c.byref(size), output) != 1:
            raise RuntimeError("Brotli compression failed")
        return output.raw[:size.value]
    output = c.create_string_buffer(zs.ZSTD_compressBound(len(data)))
    size = zs.ZSTD_compress(output, len(output), data, len(data), level)
    if zs.ZSTD_isError(size):
        raise RuntimeError("Zstandard compression failed")
    return output.raw[:size]

def verify(encoded, original, codec):
    if codec == "gzip":
        assert gzip.decompress(encoded) == original
        return
    output = c.create_string_buffer(len(original))
    if codec == "brotli":
        size = c.c_size_t(len(original))
        assert dec.BrotliDecoderDecompress(len(encoded), encoded,
                                          c.byref(size), output) == 1
        assert size.value == len(original)
    else:
        assert zs.ZSTD_decompress(output, len(output), encoded,
                                  len(encoded)) == len(original)
    assert output.raw == original

data = files["rfc9110.txt"]
rows = []
for codec, levels in [("gzip", range(1, 10)), ("brotli", [1, 3, 5, 7, 9, 11]),
                       ("zstd", [1, 3, 5, 7, 9, 15, 19])]:
    for level in levels:
        compress(data, codec, level)
        times = []
        for run in range(7):
            start = time.perf_counter_ns()
            encoded = compress(data, codec, level)
            times.append((time.perf_counter_ns() - start) / 1e6)
            verify(encoded, data, codec)
        rows.append(dict(codec=codec, level=level, bytes=len(encoded),
                         ms=round(statistics.median(times), 3), runs=times))

corpus = [dict(file=name, raw=len(value), gzip=len(compress(value, "gzip", 6)),
               sha256=hashlib.sha256(value).hexdigest())
          for name, value in files.items()]
result = dict(python=platform.python_version(), kernel=platform.release(),
              zlib=zlib.ZLIB_RUNTIME_VERSION,
              brotli=hex(enc.BrotliEncoderVersion()),
              zstd=zs.ZSTD_versionString().decode(), input=len(data),
              rows=rows, corpus=corpus)
print(json.dumps(result, indent=2))
(root / "results.json").write_text(json.dumps(result, indent=2) + "\n")

python3 compression-bench.py > compression-results.json