# Rollback and recovery

Rollback is emergency recovery, not a routine alternative to a complete Arch upgrade. Stop and coordinate through the operator whenever possible.

## New kernel fails

1. Use the independent console or bootloader menu.
2. Select the previously verified kernel or fallback initramfs entry recorded before reboot.
3. Do not repeatedly boot the failed entry or regenerate boot artifacts blindly.
4. After the previous kernel boots, record `uname -a`, boot journal, failed units, DKMS state, `/boot` contents, and bootloader entries.
5. Keep production stopped or degraded only as required for safety; obtain explicit approval before rebuilding initramfs, changing boot defaults, or removing a kernel.

## Package recovery

The first recovery source is an exact, validated package archive still present in `/var/cache/pacman/pkg`. Arch Linux Archive is the fallback source when the exact archive is unavailable.

Emergency package restoration must:

- use the exact before-inventory and transaction log;
- restore a coherent dependency set, not one guessed library;
- verify package signatures and checksums;
- preserve full command output;
- be approved as emergency recovery;
- be followed by `pacman -Dk`, package/file consistency checks, initramfs/DKMS/bootloader validation, and production health tests.

Do not improvise a broad downgrade, use `--overwrite` casually, disable signature checking, delete the package database lock without proving no Pacman process exists, or mix repository dates. Arch does not support partial upgrades; a downgrade can require an internally consistent dated repository snapshot.

## SSH recovery

- Expect the OpenSSH hook to restart a running `sshd.service` during the transaction.
- Keep the active session open and confirm a second session before reboot when the maintenance plan allows.
- If SSH fails, use the independent console. Inspect `sshd -t`, service status/journal, host keys, PAM, `sshd_config`, and any new `.pacnew` without replacing configuration automatically.
- Do not weaken authentication, expose a new port, copy keys, or disable the firewall as a shortcut.

## Docker and service recovery

- Compare against `running-containers-before.txt` and `containers-before.txt`.
- Inspect Docker daemon and container logs, mounts, networks, health checks, and storage availability before restarting anything.
- Restart only an identified failed service with operator approval; do not restart all services or recreate containers/volumes indiscriminately.
- Never delete Docker data, volumes, databases, images, or production configuration as recovery.
- Database recovery follows the service-specific backup/runbook and requires explicit approval before any mutation.

## Mandatory stop conditions

Stop rather than improvise on:

- package signature/keyring/database errors;
- interrupted Pacman transaction or uncertain package state;
- failed kernel, initramfs, DKMS, or bootloader generation;
- missing previous-kernel boot option or loss of console access;
- SSH/PAM/network/firewall configuration ambiguity;
- new unexplained `.pacnew`/`.pacsave` for critical configuration;
- storage, filesystem, database, Docker-volume, or backup inconsistency;
- failed production database, proxy, mail, Forgejo, Docker, or public endpoint;
- any proposed action that would modify production data/configuration, systemd, nftables, credentials, or package history beyond the approved recovery scope.

Rootless-Docker provisioning and contained Codex setup are never part of recovery. Resume them only after the host is healthy and a later phase is explicitly approved.

All recovery evidence remains in the root-only unique preflight, maintenance, and post-reboot record directories. It may contain operationally sensitive metadata and must not be committed or copied into a user-readable workspace.