Recovering a SelfPrivacy server when Kanidm fails to start after an upgrade (`AccessDenied` / "phase 8 - delete UUIDS")

Troubleshooting
1

This guide covers a specific failure mode where Kanidm breaks itself during its own internal database migration (domain level 13 → 14, Kanidm 1.10.2). The daemon refuses to start, which takes down single sign-on and every service that depends on it (Nextcloud login, mail, the app’s user list, SSH key management, etc.).

It was first diagnosed and resolved on a real user server in June 2026. The fix involves surgically editing the Kanidm database backup, which is inherently risky — read the whole guide before touching anything, and do not skip the backup steps.

At this moment we are not sure what exactly causes it and why it affects a small fraction of servers. We didn’t observe this problem on our own instances of SelfPrivacy yet.

1. Symptoms — how to recognise this issue

You are likely looking at this problem (and not something else) if several of the following are true:

  • In the app: the Users tab loads forever; creating SSH keys or opening a user shows “couldn’t load user”. The app cannot manage anything that lives in Kanidm.
  • In Server → System logs (or journalctl) you see a repeating line from kanidm.service:

    Tried for at least 30 seconds, giving up...

  • kanidm.service is in a failed state.
  • Other services fail as a consequence — most visibly postfix-setup.service, because the mail server can’t read a token that Kanidm never generated:

    cat: /run/keys/dovecot2/kanidm-service-account-token: No such file or directory

2. The root cause (what’s actually happening)

During startup, Kanidm runs an internal schema migration, migrate_domain_13_to_14. One phase of that migration (“phase 8 - delete UUIDS”) tries to delete some now-obsolete built-in schema objects — in this case the built-in attribute type id_verification_eckey, whose reserved UUID is:

00000000-0000-0000-0000-ffff00000134

The problem: in the affected database this object already exists as a tombstone (a soft-deleted, replication-tracked entry). Kanidm’s replication state machine refuses to hard-delete a tombstone, treating it as a possible attempt to bypass replication:

WARN   Refusing to delete entries which may be an attempt to bypass replication state machine.
ERROR   | err: AccessDenied | msg: "phase 8 - delete UUIDS"
ERROR  Unable to setup query server or idm server -> AccessDenied
ERROR  Failed to start server core!

Because the migration can’t complete, the whole server core aborts, the daemon exits 1/FAILURE, and Kanidm never opens its socket. Everything downstream then fails.

This is a Kanidm bug — the upgrade is not resilient to this state, and it leaves the operator with no legitimate in-product way to recover. The fix below removes the stray tombstone from a database backup and restores the cleaned copy.

3. Diagnosis — confirm before you operate

Get a root shell on the server (see Section 4 if you don’t have SSH access yet), then collect evidence.

a. Which services failed:

systemctl status --failed

Expect kanidm.service failed, and typically postfix-setup.service failed with the missing-token error above.

b. The Kanidm error itself (newest lines first):

journalctl -u kanidm --no-pager

Look for the four-line signature: Refusing to delete entries… → AccessDenied | msg: "phase 8 - delete UUIDS" → Unable to setup query server → Failed to start server core!, all under a migrate_domain_13_to_14 span.

Viewing logs from a phone (Termux): in the terminal pager, > characters at the right edge mean the line is cut off. Maximise the terminal window (or rotate to landscape) so the actual error text is visible, not just the timestamps. Scroll with arrows / PgUp / PgDn, quit with q.

c. (Optional) Confirm the socket is absent:

kanidmd domain upgrade-check

If Kanidm is down this reports Unable to connect to socket path … /run/kanidmd/sock … No such file or directory, confirming the daemon never came up. This command is only informative here; it does not fix anything.

kanidmd database verify will likely report “Verification passed!” even on the broken database. That is expected — the on-disk data is structurally valid; the problem is the migration step, not corruption. Don’t let a passing verify talk you out of this diagnosis.

4. Prerequisite: get root SSH access

The fix runs entirely over SSH as root. Two situations:

4a. You already have root SSH

Follow the standard guide to connect: https://selfprivacy.org/docs/how-to-guides/root_ssh/. Skip to Section 5.

4b. You cannot add an SSH key through the app

This is the common case: because Kanidm is down, the app can’t load users, so it can’t create SSH keys for you. You must inject a key using your provider’s rescue system. Instructions below are for Hetzner; other providers (DigitalOcean, etc.) have an equivalent “recovery console / rescue mode” — the idea is identical: boot a rescue OS, mount the system disk, append your public key to root’s authorized_keys, reboot.

Hetzner rescue procedure:

  1. In the Hetzner Cloud console for the server, open Rescue → Enable rescue & power cycle, and choose linux64. Leave the SSH-key field empty to get a one-time root password (this only affects the rescue OS, not your server). Copy the password.
  2. SSH into the server as root with that password (or use the web console).
  3. Mount the system partition and open root’s authorized keys:
    mount /dev/sda1 /mnt
nano /mnt/root/.ssh/authorized_keys

    Use sda1 (the system partition), not sda. If unsure, run lsblk and pick the partition that’s large enough to be the OS disk.

  4. Paste your public key on its own line. A valid public key is a single line with three space-separated fields: type key comment (e.g. ssh-ed25519 AAAA… you@mac). If you don’t have one, generate it per the SSH guide above.
  5. Save (Ctrl+X, then y, then Enter) and reboot back into the normal system:
    reboot
  6. Wait a couple of minutes, then SSH in normally with your key.

Expected host-key warning: after switching from the rescue OS back to your real system, SSH will warn loudly that the host key changed. This is normal — the rescue OS and your server have different fingerprints. Remove the stale line(s) for your server’s IP/domain from ~/.ssh/known_hosts on your own machine and reconnect. If it’s easier and you don’t mind re-trusting other hosts later, you can delete the file entirely (rm ~/.ssh/known_hosts). This only forgets server fingerprints; it never touches your own keys.

5. The fix — remove the stray tombstone and restore

Before you start: you are about to edit an identity database by hand. Follow every step in order and do not proceed past any count/verify check that doesn’t match. Keep every intermediate file — they are your safety net.

Step 1 — Stop Kanidm

systemctl stop kanidm
systemctl status kanidm --no-pager

It should show inactive/failed and not be actively running. (Since it fails to start anyway, stopping is instant.)

Step 2 — Make two independent safety copies

Copy the entire data directory (raw on-disk state):

cp -a /var/lib/kanidm /root/var-lib-kanidm

Then export a logical database backup:

kanidmd database backup /root/kanidm.back

Step 3 — Make sure the backup is plain JSON

In the original case the backup came out gzip-compressed, which made jq and restore fail with serde_json error … "expected value". Check the file type:

nix run nixpkgs#file -- /root/kanidm.back
  • If it says JSON/ASCII text, continue.
  • If it says gzip compressed data, decompress it in place:
    mv /root/kanidm.back /root/kanidm.back.gz
gzip --decompress /root/kanidm.back.gz
    This leaves you with a plain-JSON /root/kanidm.back.

Step 4 — Locate the offending tombstone

Confirm the bad UUID exists and find its entry index:

jq -c 'paths(scalars) as $p | select(getpath($p) == "00000000-0000-0000-0000-ffff00000134") | $p' /root/kanidm.back

Expected output is a path ending in the entry index, e.g.:

["entries",109,"ent","V3","attrs","uuid","UU",0]

Read that entry (substitute your own index if different) and confirm it is a tombstone:

jq -c '.entries[109]' /root/kanidm.back

You should see "class":{"I8":["object","tombstone"]} and a "changestate":{"V1Tombstone": …}. That confirms it’s the soft-deleted object the migration is choking on.

Step 5 — Produce a cleaned backup with the tombstone removed

Keep an untouched original, then filter it out by UUID (index-independent, so it’s safe even if your index differed):

cp -a /root/kanidm.back /root/kanidm.back.orig

# how many entries before?
jq '.entries | length' /root/kanidm.back.orig

# write a copy with the tombstone removed
jq '.entries |= map(select(.ent.V3.attrs.uuid.UU[0] != "00000000-0000-0000-0000-ffff00000134"))' \
  /root/kanidm.back.orig > /root/kanidm.back.fixed

# how many entries after? MUST be exactly one fewer
jq '.entries | length' /root/kanidm.back.fixed

# confirm the UUID is gone — MUST print nothing
jq -c 'paths(scalars) as $p | select(getpath($p) == "00000000-0000-0000-0000-ffff00000134") | $p' /root/kanidm.back.fixed

Stop and re-check if: the count didn’t drop by exactly one, or the last command printed anything. Do not restore a file you’re unsure about.

Step 6 — Restore the cleaned database

kanidmd database restore /root/kanidm.back.fixed

A successful run reindexes all entries, raises the domain level to 14, and ends with ✅ Restore Success!. This time the migration’s delete step finds nothing to delete and only logs harmless warnings, e.g.:

WARN  delete: no candidates match filter
WARN  Attribute id_verification_eckey was not found in schema during replication request
WARN  Domain level has been raised to 14
INFO  ✅ Restore Success!

Those warnings are expected and fine — they mean the obsolete object is simply gone.

Step 7 — Start Kanidm and watch it come up

systemctl start kanidm && journalctl -u kanidm -f

Healthy output shows real request/search/LDAP activity and Started kanidm identity management daemon. — for example service accounts binding over LDAP (Nextcloud, mailserver) with ✅ LDAP Bind success. Once it clearly looks alive with no start-up failure, stop following logs with Ctrl+C.

Step 8 — Reboot into a clean state

reboot

You do not need to run an upgrade from the app afterwards — the server is already on the latest version; the database was the only problem.

6. Validation — confirm the fix

After the reboot:

  • systemctl status --failed — should be empty (no kanidm.service, no postfix-setup.service).
  • journalctl -u kanidm -b — no AccessDenied / Failed to start server core!.
  • In the app: the Users tab loads; you can open users and manage SSH keys again.
  • End-user login works: the Kanidm auth page (https://auth.<your-domain>/) loads, and logging into a dependent service (e.g. Nextcloud) via SSO succeeds.

In the original case, all of the above passed and the user could log into Nextcloud and the Kanidm auth page immediately after reboot.