Writing

Technology · 6 min read

Making Immich faster with NVMe

I moved Immich’s database, model cache, and previews to NVMe, kept originals on HDD, and measured the difference.

In this article4 sections

I started with Immich on hard disks. Loading the library and opening photos felt slow, and I use it often enough that the waiting bothered me. My first guess was RAM. Then I looked at the price and wondered how much of the waiting came from storage. The NVMe drive I was considering was a little cheaper, so I tried that first.

I tried moving the media to NVMe, then started looking at which parts actually benefited. I ended up with the database, model cache, thumbnails, and previews on NVMe, and originals on HDD. Finding that I could choose where each part lived made the upgrade more useful. I could put the smaller drive to work without fitting my whole library on it.

What I moved in Docker

DataLocation
PostgreSQL databaseNVMe
Machine-learning model cacheNVMe
Thumbnails and larger previewsNVMe
Original photos, uploads, and encoded videosHDD

The change came down to a few Docker mounts. This is the media part of my Compose configuration.

services:
  immich-server:
    volumes:
      - /volume1/immich:/data
      - /volume2/appdata/immich-media/thumbs:/data/thumbs

PostgreSQL and the machine-learning service have their own NVMe mounts. Docker’s engine data directory is still on HDD.

Immich supports separate locations for thumbnails, encoded videos, and other files. Small thumbnails and larger previews share the thumbs directory. You can start with just the database and model cache, add previews, or put the whole library on NVMe if the capacity makes sense. Immich’s custom storage guide

What the benchmarks showed

I tested Immich 3.4.1 on my UGREEN DXP4800 Plus, comparing a Btrfs HDD mirror with a single 1 TB ext4 NVMe volume. Each comparison used ten alternating pairs with the same files, model revision, or database snapshot.

Reading previews

I started with preview reads because browsing was what felt slow. In the latest media run, reading 32 previews took 32.6 ms on HDD and 10.8 ms on NVMe at the median. Eight original photos also read faster.

HDD versus NVMeLocal file reads · median
Static paired comparison. Gap labels show the difference; all exact values are in the accompanying table.21.8 ms saved33.4 ms saved
Static paired comparison. Gap labels show the difference; all exact values are in the accompanying table.21.8 ms saved33.4 ms saved
HDDNVMe
Exact comparison values; change is relative to HDD.
DatasetHDD (ms)NVMe (ms)Change
32 previews32.610.8-66.9%21.8 ms saved
8 photos42.08.5-79.6%33.4 ms saved

Tap a point or use arrow keys to inspect values.

Ten batches per volume. File-cache eviction requested before each first pass. Lower is faster.

The preview mean was 96.9 ms on HDD and 10.8 ms on NVMe. A slow HDD batch pulled the average up. Its p95 was 553.3 ms, compared with 16.0 ms on NVMe. With ten observations, p95 is the slowest sample.

I wanted to see whether that held up across runs. The earlier ten-pair session had preview medians of 28.5 ms and 9.8 ms. The timings moved around, but both sessions gave me a reason to keep previews on NVMe.

Loading the database and model

I also wanted to know what happened when Immich had to load data for the first time. The first timeline database query fell from 843 ms to 41 ms at the median. The first Smart Search text prediction, including loading the model, fell from 1.44 seconds to 424 ms.

HDD versus NVMeFirst request · median
Static paired comparison. Gap labels show the difference; all exact values are in the accompanying table.802.3 ms saved1,017.2 ms saved
Static paired comparison. Gap labels show the difference; all exact values are in the accompanying table.802.3 ms saved1,017.2 ms saved
HDDNVMe
Exact comparison values; change is relative to HDD.
DatasetHDD (ms)NVMe (ms)Change
Timeline query842.940.6-95.2%802.3 ms saved
Text prediction1,441.1423.9-70.6%1,017.2 ms saved

Tap a point or use arrow keys to inspect values.

Database and model measurements from the earlier ten-pair session. Lower is faster.

For PostgreSQL, I copied the same stopped database to each volume, requested eviction of its data files from the OS cache, and started a new server for each leg. The query read metadata for 250 timeline items. For the model, I prepared the graph before timing and used a fresh process for each first prediction.

Once the data was in memory

Then I repeated the requests in the same process. Would the faster drive still make much difference once the data was loaded? The query stayed near 11 ms, and the model stayed near 40 ms on either drive.

HDD versus NVMeWarm requests · median
Static paired comparison. Gap labels show the difference; all exact values are in the accompanying table.
Static paired comparison. Gap labels show the difference; all exact values are in the accompanying table.
HDDNVMe
Exact comparison values; change is relative to HDD.
DatasetHDD (ms)NVMe (ms)Change
Timeline query11.310.8-4.2%
Text prediction40.040.4+0.9%

Tap a point or use arrow keys to inspect values.

100 repeat observations per volume for each workload. Lower is faster.

That brought me back to my original thought about RAM. Keeping data in memory clearly mattered in these tests. I kept the installed RAM fixed, so I still don’t know how much a memory upgrade would help. I do have a better idea of which waits the NVMe can shorten.

Why I kept originals on HDD

The remaining question was whether the originals needed the same treatment. They take up most of the space. Reading chunks from ten original videos reached a median 207 MiB/s on HDD and 872 MiB/s on NVMe.

HDD versus NVMeOriginal-video reads · median
Static paired comparison. Gap labels show the difference; all exact values are in the accompanying table.
Static paired comparison. Gap labels show the difference; all exact values are in the accompanying table.
HDDNVMe
Exact comparison values; change is relative to HDD.
DatasetHDD (MiB/s)NVMe (MiB/s)Change
64 MiB chunks206.5871.9+322.2%

Tap a point or use arrow keys to inspect values.

Ten local reads. Higher is faster. The 1 Gb/s network has a raw ceiling of about 119 MiB/s.

The NVMe result looks like a good argument for moving everything. Then I compared it with my network. Even the slowest HDD sample reached 138 MiB/s, above the 1 Gb/s link’s raw ceiling of about 119 MiB/s. For the large reads I sampled, the hard drives could already feed a single transfer as fast as that connection could carry it.

That was enough for me to put the originals back on HDD. I’d revisit it with a faster network or several readers competing for the disks. For my current setup, the previews, database, and model cache make better use of the NVMe space.

How to read these numbers

These tests measured work on the NAS. Opening a photo in the browser also involves network transfer and image decoding. A complete browser timeline load remains unmeasured.

I retained the per-run data, checked sampled media hashes, and calculated means, medians, and nearest-rank p95s. Linux file-cache eviction was requested for first reads, and block-input counters were recorded. The media charts use the latest repeat; the database and model charts use the earlier matched session.

During migration, I retained the source copies and checked the running container mounts and file access. Moving the database also means including its new location in the backup plan alongside the original media. Immich’s backup guide

Was this useful?

What was missing?

Thanks for the feedback.