Validating File Uploads With Magic Bytes

A filename is a label the caller chose and a content type is a header the caller sent. Neither says anything about the bytes, which is why extension checks have never stopped anyone determined. This guide implements content-based validation properly: bounding the request before it is read, detecting the type from the leading bytes, allowlisting narrowly per feature, and re-encoding accepted files so that anything hiding alongside the real content does not survive.

It is part of the File Upload & Deserialization Security guide within Vulnerability Patterns & Web Mitigation Strategies. Validation is only the first of the four links in the upload chain — the storage and serving controls in serving user uploads from a sandboxed origin matter just as much.

Prerequisites

  • A signature detection library — libmagic bindings, file-type, or an equivalent that reads content
  • An image processing library you are prepared to keep patched
  • The ability to set a maximum body size at the reverse proxy or edge, not only in application code
  • A per-feature list of the types the product genuinely needs to accept

Expected Outcomes

  • Requests over the size limit rejected before the application allocates memory for them
  • Type decided from content and compared with a narrow per-feature allowlist
  • Accepted images re-encoded, stripping appended data and metadata
  • The detected type recorded alongside the stored object for use when serving it back

Step 1: Bound the Request Before You Read It

Validation that runs after the whole body is in memory has already lost the resource argument. Set the ceiling at the edge, then stream.

# nginx: refuse oversized bodies before any application process is involved.
client_max_body_size 10m;
client_body_buffer_size 128k;
// Node: stream to a temporary sink, counting bytes; abort as soon as the cap is passed.
const MAX = 10 * 1024 * 1024;

async function readBounded(req: Request): Promise<Buffer> {
  const chunks: Buffer[] = [];
  let total = 0;
  for await (const chunk of req) {
    total += chunk.length;
    if (total > MAX) {
      req.destroy();                       // stop reading; do not finish the upload
      throw new HttpError(413, 'file too large');
    }
    chunks.push(chunk);
  }
  return Buffer.concat(chunks);
}

Two independent limits are deliberate. The edge limit protects the process from ever seeing the bytes; the application limit protects against a misconfigured edge and documents the intent in code where reviewers will see it.

Three Limits, Three Different Exhaustion Attacks A byte limit at the edge stops an oversized body reaching the process at all. A byte limit in the application documents the intent and covers a misconfigured edge. A pixel limit stops a small file whose declared dimensions expand into gigabytes of buffer — an exhaustion attack that never needed a large upload. Edge byte limit — refuses before the process sees anything the cheapest place to say no, and the only one that costs no memory Application byte limit — documents intent, covers a misconfigured edge reviewers see it in the code path they are reading Pixel limit before decoding — the one teams forget a one-megabyte image can declare dimensions that allocate gigabytes

Step 2: Detect the Type From the Leading Bytes

Read a header window, ask the detector what it is, and compare against the allowlist for this feature — not a global list, because a profile photo endpoint has no reason to accept the same types as a document import.

import { fileTypeFromBuffer } from 'file-type';

const ALLOWED: Record<string, Record<string, string>> = {
  avatar:      { 'image/png': '.png', 'image/jpeg': '.jpg', 'image/webp': '.webp' },
  attachment:  { 'image/png': '.png', 'image/jpeg': '.jpg', 'application/pdf': '.pdf' },
};

export async function detect(feature: keyof typeof ALLOWED, bytes: Buffer) {
  const head = bytes.subarray(0, 8192);
  const found = await fileTypeFromBuffer(head);        // reads the signature
  if (!found) throw new HttpError(400, 'unrecognised file type');

  const allowed = ALLOWED[feature];
  if (!(found.mime in allowed)) {
    throw new HttpError(400, `${found.mime} is not accepted for ${feature}`);
  }
  return { mime: found.mime, ext: allowed[found.mime] };
}
Three Claims, One Piece of Evidence The same upload carries three assertions about its type. The filename says it is an image and is chosen entirely by the caller. The declared content type says the same and is a header the caller sent. The detected signature reads the actual leading bytes and reveals a script, which is the only one of the three the server has any reason to believe. One uploaded file, three claims about what it is Filename: holiday.png chosen by the caller, meaningless as evidence a rename is not a conversion Declared type: image/png a header the client set, equally free to invent trivially forged with any HTTP client Detected signature: text/x-php read from the leading bytes on the server the only claim with evidence behind it — reject

Step 3: Re-Encode Accepted Images

Detection identifies the format at the front of the file. It says nothing about what has been appended to the back, embedded in a comment field, or tucked into metadata. Decoding and rewriting removes all three, because the output contains only what the decoder understood.

from PIL import Image, ImageFile
import io

ImageFile.LOAD_TRUNCATED_IMAGES = False          # a truncated file is a rejection, not a repair job
MAX_PIXELS = 50_000_000                          # decompression bound, before allocation
Image.MAX_IMAGE_PIXELS = MAX_PIXELS

def reencode(data: bytes, mime: str) -> bytes:
    with Image.open(io.BytesIO(data)) as img:
        img.verify()                             # structural check, cheap
    with Image.open(io.BytesIO(data)) as img:    # verify() consumes the file object
        img = img.convert("RGB") if mime == "image/jpeg" else img.convert("RGBA")
        out = io.BytesIO()
        img.save(out, format="JPEG" if mime == "image/jpeg" else "PNG", optimize=True)
        return out.getvalue()                    # no appended archive, no metadata, no comments

The pixel ceiling matters as much as the byte ceiling. A one-megabyte image can declare dimensions that expand to gigabytes of pixel buffer, and the resulting allocation is a denial of service that never needed a large upload at all.

What Re-Encoding Removes That Validation Does Not Type detection identifies the format at the front of the file and says nothing about what follows it. Re-encoding rewrites the image from a decoded pixel buffer, so appended archives, comment-field payloads, trailing data and metadata all disappear — because none of them were part of what the decoder understood. An archive appended after the image data detection sees a valid image; re-encoding drops everything after it A payload hidden in a comment or metadata field never reaches the pixel buffer, so it never reaches the output Location and device metadata the uploader forgot about a privacy improvement that arrives for free with the same step What survives re-encoding the pixels, and nothing else — which is the whole intent of accepting an image

Verification

Prove the defences with files that are deliberately dishonest.

# 1. A script wearing an image name and content type.
printf '<?php system($_GET["c"]); ?>' > /tmp/shell.php.png
curl -s -o /dev/null -w '%{http_code}\n' -F 'file=@/tmp/shell.php.png;type=image/png' \
  -H "Cookie: $SESSION" http://localhost:3000/api/uploads
# Expected: 400 — the signature is not an accepted image type

# 2. A real image with an archive appended: accepted, but the archive must not survive.
cat real.png payload.zip > /tmp/polyglot.png
URL=$(curl -s -F 'file=@/tmp/polyglot.png' -H "Cookie: $SESSION" \
  http://localhost:3000/api/uploads | jq -r .url)
curl -s "$URL" | tail -c 200 | grep -c 'PK' || echo "archive stripped as expected"

# 3. A pixel bomb: small on the wire, enormous when decoded.
curl -s -o /dev/null -w '%{http_code}\n' -F 'file=@/tmp/50000x50000.png' \
  -H "Cookie: $SESSION" http://localhost:3000/api/uploads
# Expected: 400, and no memory spike on the server

A passing run means: dishonest types are refused, honest-but-loaded files are normalised, and decode limits hold before allocation rather than after.

Troubleshooting

Symptom Likely cause Fix
Legitimate uploads rejected as unrecognised Detector needs more context than the window provides, e.g. container formats Increase the header window to 8 kilobytes and detect container subtypes explicitly
Re-encoding rotates or flips photographs Orientation metadata is dropped by the rewrite Read the orientation, apply the transform, then write without the metadata
Memory spikes on large images Byte limit enforced but pixel limit missing Set a maximum pixel count before decoding and reject beyond it
Files accepted by one instance and rejected by another Different library versions across the fleet Pin the detector and image library versions and verify them in the health endpoint
PDFs pass validation but rendering hangs Parser fetching remote resources during processing Disable network access in the conversion sandbox and set a hard timeout

Common Implementation Mistakes

Frequently Asked Questions

How many bytes do I need to detect a type?

Most signatures sit within the first 16 bytes, but container formats need more: an office document is a zip archive whose real type depends on entries inside it, and some media containers place identifying boxes several kilobytes in. An 8-kilobyte window covers the practical cases without buffering whole files, and is small enough to hold in memory safely while the remainder of the body streams to storage.

Should I reject or re-encode a file that fails a strict check?

Reject anything whose detected type falls outside the allowlist, and re-encode everything inside it. Re-encoding is not a rescue operation for suspicious files — it is normalisation of accepted ones. A valid image that also carries an appended archive stops carrying it once it has been decoded to pixels and written back out, which is why re-encoding is the highest-value step in the pipeline.

What about formats that cannot be re-encoded, like PDF?

Accept them only where the feature genuinely needs them, parse them with the network disabled and a hard timeout, and serve them from the isolated content origin with an attachment disposition so the browser downloads rather than renders them in your security context. Where the product only needs a preview, render one server-side into an image, serve that, and keep the original strictly as a download.