From 7f4233402aee6e5ec9183d2db945415d81e8d767 Mon Sep 17 00:00:00 2001
From: server <server@jakubkadlec.dev>
Date: Tue, 14 Apr 2026 09:03:08 +0200
Subject: [PATCH] docs: expand operational runbooks

---
 docs/README.md             |  11 ++-
 docs/config-and-secrets.md |  56 +++++++++++++
 docs/database-bootstrap.md |  69 ++++++++++++++++
 docs/deploy-workflow.md    | 165 +++++++++++++++++++++++++++++++++++++
 docs/rollback.md           |  97 ++++++++++++++++++++++
 5 files changed, 394 insertions(+), 4 deletions(-)
 create mode 100644 docs/config-and-secrets.md
 create mode 100644 docs/database-bootstrap.md
 create mode 100644 docs/deploy-workflow.md
 create mode 100644 docs/rollback.md

diff --git a/docs/README.md b/docs/README.md
index 3c533a6..2599e41 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -6,10 +6,13 @@ Current documents:
 
 - [Debian Runtime](debian-runtime.md)
 - [Healthchecks](healthchecks.md)
+- [Deploy Workflow](deploy-workflow.md)
+- [Rollback](rollback.md)
+- [Database Bootstrap](database-bootstrap.md)
+- [Config And Secrets](config-and-secrets.md)
 
 Suggested next additions:
 
-- database/bootstrap notes
-- deployment workflow
-- rollback procedure
-- configuration/secrets contract
+- client connection notes
+- incident response checklist
+- runtime log map
diff --git a/docs/config-and-secrets.md b/docs/config-and-secrets.md
new file mode 100644
index 0000000..7ec9edd
--- /dev/null
+++ b/docs/config-and-secrets.md
@@ -0,0 +1,56 @@
+# Config And Secrets
+
+This document describes the current config layout and the intended secret-handling boundary for the Debian deployment.
+
+## Current Config Files In Repo
+
+Main runtime config examples live under:
+
+- `share/conf/db.txt`
+- `share/conf/game.txt`
+
+These files are part of the runtime tree and are required by the legacy server layout.
+
+## Important Security Boundary
+
+The repository currently contains literal config values for legacy compatibility. They must be treated as bootstrap/default values, not as a safe long-term secret store.
+
+Practical rule:
+
+- do not treat git-tracked config files as the final production secret source of truth
+
+## What Should Not Live In Git
+
+Do not commit:
+
+- production DB passwords
+- real admin page passwords
+- host-specific private tokens
+- private SSH material
+- per-environment override files with live secrets
+
+## Current Operational Model
+
+For the current Debian VPS:
+
+- root-only operational wrappers may inject short-lived values locally
+- headless login healthcheck uses a temporary password via environment, not a command-line literal
+- the installed wrapper is root-only and not network-facing
+
+## Admin Page Password
+
+The source repository now supports hardening for the admin page password. The long-term goal should be:
+
+- no implicit production default
+- host-local secret injection
+- explicit runtime validation
+
+## Recommended Direction
+
+The Debian deployment should eventually move to a clearer contract such as:
+
+- git-tracked template/default files
+- host-local env file or secret file owned by root
+- documented override points
+
+Until that is done, keep all real secret rotation and secret overrides on the host, not in commits.
diff --git a/docs/database-bootstrap.md b/docs/database-bootstrap.md
new file mode 100644
index 0000000..6b9d0be
--- /dev/null
+++ b/docs/database-bootstrap.md
@@ -0,0 +1,69 @@
+# Database Bootstrap
+
+This document describes the database files included in the repository and the current bootstrap model used for the Debian VPS.
+
+## Databases
+
+The current server layout expects these databases:
+
+- `account`
+- `player`
+- `common`
+- `log`
+- `hotbackup`
+
+## Repository SQL Files
+
+Relevant files:
+
+- `sql/database_create/database_create.sql`
+- `sql/account.sql`
+- `sql/player.sql`
+- `sql/common.sql`
+- `sql/log.sql`
+- `sql/hotbackup_(empty).sql`
+- `sql/my.cnf`
+- `sql/my.cnf.bak_for_mysql5.6`
+
+## Practical Import Order
+
+Typical bootstrap order:
+
+1. create databases and database user
+2. import `account.sql`
+3. import `player.sql`
+4. import `common.sql`
+5. import `log.sql`
+6. import `hotbackup_(empty).sql`
+
+## Notes About The Included SQL Config Files
+
+`sql/my.cnf` and related files are legacy reference material. They are not the authoritative production configuration for the Debian VPS.
+
+The Debian runtime currently uses MariaDB and system-level service management rather than a repo-owned database daemon.
+
+## Current Runtime Expectation
+
+Server config files reference:
+
+- `DB_ADDR: 127.0.0.1`
+- `DB_PORT: 9000`
+
+This means:
+
+- the game/auth processes connect to the DB proxy/cache process on `127.0.0.1:9000`
+- that DB process then talks to MariaDB using the configured SQL credentials
+
+## Production Reality
+
+The current production VPS already has the databases imported and in use. The headless login healthcheck proves that `account` and `player` reads/writes are working end-to-end.
+
+## Recommended Follow-Up
+
+This repo should eventually gain a Debian-first documented bootstrap script or runbook that covers:
+
+- MariaDB package installation
+- SQL mode compatibility
+- database/user creation
+- import commands
+- post-import verification
diff --git a/docs/deploy-workflow.md b/docs/deploy-workflow.md
new file mode 100644
index 0000000..f1705ea
--- /dev/null
+++ b/docs/deploy-workflow.md
@@ -0,0 +1,165 @@
+# Deploy Workflow
+
+This document describes the current Debian deployment workflow used on the VPS.
+
+## Repositories And Responsibilities
+
+There are two separate repositories involved in a production deploy:
+
+- `m2dev-server-src`: source code, build system, smoke tests, login smoke utility
+- `m2dev-server`: runtime files, configs, quests, systemd deployment assets
+
+In practice:
+
+- source changes are built from `m2dev-server-src`
+- runtime files are synchronized from `m2dev-server`
+- the live instance runs from `~/metin/runtime/server`
+
+## Current VPS Helper Scripts
+
+The current production VPS also has local host-side helper scripts under:
+
+```text
+/home/mt2.jakubkadlec.dev/metin/deploy/
+```
+
+Current helpers:
+
+- `build-server.sh`
+- `sync-runtime.sh`
+- `compile-quests.sh`
+- `install-runtime.sh`
+
+These are currently VPS-local operational wrappers. They are useful, but they are not yet versioned in this repository.
+
+## Common Deploy Paths
+
+Source repository:
+
+```text
+/home/mt2.jakubkadlec.dev/metin/repos/m2dev-server-src
+```
+
+Runtime repository:
+
+```text
+/home/mt2.jakubkadlec.dev/metin/repos/m2dev-server
+```
+
+Build directory:
+
+```text
+/home/mt2.jakubkadlec.dev/metin/build/server-src
+```
+
+Live runtime:
+
+```text
+/home/mt2.jakubkadlec.dev/metin/runtime/server
+```
+
+## Standard Deploy Sequence
+
+### 1. Update repositories
+
+```bash
+ssh mt2
+sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server-src pull --ff-only
+sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server pull --ff-only
+```
+
+### 2. Build source
+
+Equivalent manual build:
+
+```bash
+sudo -iu mt2.jakubkadlec.dev \
+  cmake -S ~/metin/repos/m2dev-server-src \
+        -B ~/metin/build/server-src \
+        -G Ninja
+
+sudo -iu mt2.jakubkadlec.dev \
+  cmake --build ~/metin/build/server-src --parallel "$(nproc)"
+```
+
+Current VPS wrapper:
+
+```bash
+sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/build-server.sh
+```
+
+### 3. Synchronize runtime files
+
+Equivalent manual sync:
+
+- copy runtime tree from `repos/m2dev-server`
+- exclude generated runtime state
+- install built `game`, `db`, `qc` into the live runtime
+
+Current VPS wrapper:
+
+```bash
+sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/sync-runtime.sh
+```
+
+### 4. Compile quests
+
+```bash
+sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/compile-quests.sh
+```
+
+This runs:
+
+```bash
+cd ~/metin/runtime/server/share/locale/english/quest
+python3 make.py
+```
+
+### 5. Restart services
+
+```bash
+systemctl restart metin-server.service
+```
+
+### 6. Verify runtime
+
+Recommended checks:
+
+```bash
+systemctl status metin-server.service --no-pager
+ss -ltnp | rg ':(9000|11000|11011|11012|11013|11991) '
+/usr/local/sbin/metin-login-healthcheck
+```
+
+## When A Full Rebuild Is Needed
+
+Rebuild `m2dev-server-src` when:
+
+- C++ code changed
+- CMake changed
+- smoke/login test utility changed
+
+Runtime sync only may be enough when:
+
+- configs changed
+- quest files changed
+- locale data changed
+- runtime assets changed
+
+## When systemd Must Be Reinstalled
+
+Re-run the systemd installer when files under `deploy/systemd/` change:
+
+```bash
+python3 deploy/systemd/install_systemd.py \
+  --user mt2.jakubkadlec.dev \
+  --group mt2.jakubkadlec.dev \
+  --runtime-root /home/mt2.jakubkadlec.dev/metin/runtime/server \
+  --channel 1 \
+  --channel 99 \
+  --restart
+```
+
+## Recommended Future Improvement
+
+The current VPS-local helper scripts should eventually be moved into a versioned, reviewable deploy workflow in the repository, or replaced with a single documented deployment entrypoint.
diff --git a/docs/rollback.md b/docs/rollback.md
new file mode 100644
index 0000000..3b156c1
--- /dev/null
+++ b/docs/rollback.md
@@ -0,0 +1,97 @@
+# Rollback
+
+This document describes the practical rollback workflow for the current Debian VPS.
+
+## Important Principle
+
+There are two rollback surfaces:
+
+- source rollback (`m2dev-server-src`)
+- runtime-file rollback (`m2dev-server`)
+
+They are related, but they are not the same repository and do not always need to move together.
+
+## Fast Safety Rule
+
+Before a risky deploy, record both current commits:
+
+```bash
+ssh mt2
+sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server-src rev-parse --short HEAD
+sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server rev-parse --short HEAD
+```
+
+If the deploy fails, rollback to those exact commits.
+
+## Source Rollback
+
+Use this when the regression is in:
+
+- server binaries
+- network code
+- DB logic
+- C++ runtime behavior
+
+Example:
+
+```bash
+ssh mt2
+sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server-src checkout <good_commit>
+sudo -iu mt2.jakubkadlec.dev cmake --build ~/metin/build/server-src --parallel "$(nproc)"
+sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/sync-runtime.sh
+systemctl restart metin-server.service
+/usr/local/sbin/metin-login-healthcheck
+```
+
+## Runtime Rollback
+
+Use this when the regression is in:
+
+- configs
+- quests
+- locale files
+- runtime-only scripts
+
+Example:
+
+```bash
+ssh mt2
+sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server checkout <good_commit>
+sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/sync-runtime.sh
+sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/compile-quests.sh
+systemctl restart metin-server.service
+/usr/local/sbin/metin-login-healthcheck
+```
+
+## Combined Rollback
+
+Use this when the regression could be caused by an interaction between:
+
+- new binaries
+- new runtime files
+
+Rollback both repositories together and rebuild/redeploy.
+
+## systemd Rollback
+
+If the regression is in `deploy/systemd/`, rollback the runtime repository to a known good commit and re-run:
+
+```bash
+python3 deploy/systemd/install_systemd.py ... --restart
+```
+
+## Verification After Rollback
+
+Minimum verification:
+
+```bash
+systemctl status metin-server.service --no-pager
+ss -ltnp | rg ':(9000|11000|11011|11012|11013|11991) '
+/usr/local/sbin/metin-login-healthcheck
+```
+
+## What Not To Do
+
+- do not use `git reset --hard` blindly on a dirty VPS checkout
+- do not rollback only one repository if the deploy changed both and the fault domain is unclear
+- do not close the current SSH session before the healthcheck passes