docs: expand operational runbooks
This commit is contained in:
server
@@ -6,10 +6,13 @@ Current documents:
|
||||
|
||||
- [Debian Runtime](debian-runtime.md)
|
||||
- [Healthchecks](healthchecks.md)
|
||||
- [Deploy Workflow](deploy-workflow.md)
|
||||
- [Rollback](rollback.md)
|
||||
- [Database Bootstrap](database-bootstrap.md)
|
||||
- [Config And Secrets](config-and-secrets.md)
|
||||
|
||||
Suggested next additions:
|
||||
|
||||
- database/bootstrap notes
|
||||
- deployment workflow
|
||||
- rollback procedure
|
||||
- configuration/secrets contract
|
||||
- client connection notes
|
||||
- incident response checklist
|
||||
- runtime log map
|
||||
|
||||
56
docs/config-and-secrets.md
Normal file
56
docs/config-and-secrets.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# Config And Secrets
|
||||
|
||||
This document describes the current config layout and the intended secret-handling boundary for the Debian deployment.
|
||||
|
||||
## Current Config Files In Repo
|
||||
|
||||
Main runtime config examples live under:
|
||||
|
||||
- `share/conf/db.txt`
|
||||
- `share/conf/game.txt`
|
||||
|
||||
These files are part of the runtime tree and are required by the legacy server layout.
|
||||
|
||||
## Important Security Boundary
|
||||
|
||||
The repository currently contains literal config values for legacy compatibility. They must be treated as bootstrap/default values, not as a safe long-term secret store.
|
||||
|
||||
Practical rule:
|
||||
|
||||
- do not treat git-tracked config files as the final production secret source of truth
|
||||
|
||||
## What Should Not Live In Git
|
||||
|
||||
Do not commit:
|
||||
|
||||
- production DB passwords
|
||||
- real admin page passwords
|
||||
- host-specific private tokens
|
||||
- private SSH material
|
||||
- per-environment override files with live secrets
|
||||
|
||||
## Current Operational Model
|
||||
|
||||
For the current Debian VPS:
|
||||
|
||||
- root-only operational wrappers may inject short-lived values locally
|
||||
- headless login healthcheck uses a temporary password via environment, not a command-line literal
|
||||
- the installed wrapper is root-only and not network-facing
|
||||
|
||||
## Admin Page Password
|
||||
|
||||
The source repository now supports hardening for the admin page password. The long-term goal should be:
|
||||
|
||||
- no implicit production default
|
||||
- host-local secret injection
|
||||
- explicit runtime validation
|
||||
|
||||
## Recommended Direction
|
||||
|
||||
The Debian deployment should eventually move to a clearer contract such as:
|
||||
|
||||
- git-tracked template/default files
|
||||
- host-local env file or secret file owned by root
|
||||
- documented override points
|
||||
|
||||
Until that is done, keep all real secret rotation and secret overrides on the host, not in commits.
|
||||
69
docs/database-bootstrap.md
Normal file
69
docs/database-bootstrap.md
Normal file
@@ -0,0 +1,69 @@
|
||||
# Database Bootstrap
|
||||
|
||||
This document describes the database files included in the repository and the current bootstrap model used for the Debian VPS.
|
||||
|
||||
## Databases
|
||||
|
||||
The current server layout expects these databases:
|
||||
|
||||
- `account`
|
||||
- `player`
|
||||
- `common`
|
||||
- `log`
|
||||
- `hotbackup`
|
||||
|
||||
## Repository SQL Files
|
||||
|
||||
Relevant files:
|
||||
|
||||
- `sql/database_create/database_create.sql`
|
||||
- `sql/account.sql`
|
||||
- `sql/player.sql`
|
||||
- `sql/common.sql`
|
||||
- `sql/log.sql`
|
||||
- `sql/hotbackup_(empty).sql`
|
||||
- `sql/my.cnf`
|
||||
- `sql/my.cnf.bak_for_mysql5.6`
|
||||
|
||||
## Practical Import Order
|
||||
|
||||
Typical bootstrap order:
|
||||
|
||||
1. create databases and database user
|
||||
2. import `account.sql`
|
||||
3. import `player.sql`
|
||||
4. import `common.sql`
|
||||
5. import `log.sql`
|
||||
6. import `hotbackup_(empty).sql`
|
||||
|
||||
## Notes About The Included SQL Config Files
|
||||
|
||||
`sql/my.cnf` and related files are legacy reference material. They are not the authoritative production configuration for the Debian VPS.
|
||||
|
||||
The Debian runtime currently uses MariaDB and system-level service management rather than a repo-owned database daemon.
|
||||
|
||||
## Current Runtime Expectation
|
||||
|
||||
Server config files reference:
|
||||
|
||||
- `DB_ADDR: 127.0.0.1`
|
||||
- `DB_PORT: 9000`
|
||||
|
||||
This means:
|
||||
|
||||
- the game/auth processes connect to the DB proxy/cache process on `127.0.0.1:9000`
|
||||
- that DB process then talks to MariaDB using the configured SQL credentials
|
||||
|
||||
## Production Reality
|
||||
|
||||
The current production VPS already has the databases imported and in use. The headless login healthcheck proves that `account` and `player` reads/writes are working end-to-end.
|
||||
|
||||
## Recommended Follow-Up
|
||||
|
||||
This repo should eventually gain a Debian-first documented bootstrap script or runbook that covers:
|
||||
|
||||
- MariaDB package installation
|
||||
- SQL mode compatibility
|
||||
- database/user creation
|
||||
- import commands
|
||||
- post-import verification
|
||||
165
docs/deploy-workflow.md
Normal file
165
docs/deploy-workflow.md
Normal file
@@ -0,0 +1,165 @@
|
||||
# Deploy Workflow
|
||||
|
||||
This document describes the current Debian deployment workflow used on the VPS.
|
||||
|
||||
## Repositories And Responsibilities
|
||||
|
||||
There are two separate repositories involved in a production deploy:
|
||||
|
||||
- `m2dev-server-src`: source code, build system, smoke tests, login smoke utility
|
||||
- `m2dev-server`: runtime files, configs, quests, systemd deployment assets
|
||||
|
||||
In practice:
|
||||
|
||||
- source changes are built from `m2dev-server-src`
|
||||
- runtime files are synchronized from `m2dev-server`
|
||||
- the live instance runs from `~/metin/runtime/server`
|
||||
|
||||
## Current VPS Helper Scripts
|
||||
|
||||
The current production VPS also has local host-side helper scripts under:
|
||||
|
||||
```text
|
||||
/home/mt2.jakubkadlec.dev/metin/deploy/
|
||||
```
|
||||
|
||||
Current helpers:
|
||||
|
||||
- `build-server.sh`
|
||||
- `sync-runtime.sh`
|
||||
- `compile-quests.sh`
|
||||
- `install-runtime.sh`
|
||||
|
||||
These are currently VPS-local operational wrappers. They are useful, but they are not yet versioned in this repository.
|
||||
|
||||
## Common Deploy Paths
|
||||
|
||||
Source repository:
|
||||
|
||||
```text
|
||||
/home/mt2.jakubkadlec.dev/metin/repos/m2dev-server-src
|
||||
```
|
||||
|
||||
Runtime repository:
|
||||
|
||||
```text
|
||||
/home/mt2.jakubkadlec.dev/metin/repos/m2dev-server
|
||||
```
|
||||
|
||||
Build directory:
|
||||
|
||||
```text
|
||||
/home/mt2.jakubkadlec.dev/metin/build/server-src
|
||||
```
|
||||
|
||||
Live runtime:
|
||||
|
||||
```text
|
||||
/home/mt2.jakubkadlec.dev/metin/runtime/server
|
||||
```
|
||||
|
||||
## Standard Deploy Sequence
|
||||
|
||||
### 1. Update repositories
|
||||
|
||||
```bash
|
||||
ssh mt2
|
||||
sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server-src pull --ff-only
|
||||
sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server pull --ff-only
|
||||
```
|
||||
|
||||
### 2. Build source
|
||||
|
||||
Equivalent manual build:
|
||||
|
||||
```bash
|
||||
sudo -iu mt2.jakubkadlec.dev \
|
||||
cmake -S ~/metin/repos/m2dev-server-src \
|
||||
-B ~/metin/build/server-src \
|
||||
-G Ninja
|
||||
|
||||
sudo -iu mt2.jakubkadlec.dev \
|
||||
cmake --build ~/metin/build/server-src --parallel "$(nproc)"
|
||||
```
|
||||
|
||||
Current VPS wrapper:
|
||||
|
||||
```bash
|
||||
sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/build-server.sh
|
||||
```
|
||||
|
||||
### 3. Synchronize runtime files
|
||||
|
||||
Equivalent manual sync:
|
||||
|
||||
- copy runtime tree from `repos/m2dev-server`
|
||||
- exclude generated runtime state
|
||||
- install built `game`, `db`, `qc` into the live runtime
|
||||
|
||||
Current VPS wrapper:
|
||||
|
||||
```bash
|
||||
sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/sync-runtime.sh
|
||||
```
|
||||
|
||||
### 4. Compile quests
|
||||
|
||||
```bash
|
||||
sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/compile-quests.sh
|
||||
```
|
||||
|
||||
This runs:
|
||||
|
||||
```bash
|
||||
cd ~/metin/runtime/server/share/locale/english/quest
|
||||
python3 make.py
|
||||
```
|
||||
|
||||
### 5. Restart services
|
||||
|
||||
```bash
|
||||
systemctl restart metin-server.service
|
||||
```
|
||||
|
||||
### 6. Verify runtime
|
||||
|
||||
Recommended checks:
|
||||
|
||||
```bash
|
||||
systemctl status metin-server.service --no-pager
|
||||
ss -ltnp | rg ':(9000|11000|11011|11012|11013|11991) '
|
||||
/usr/local/sbin/metin-login-healthcheck
|
||||
```
|
||||
|
||||
## When A Full Rebuild Is Needed
|
||||
|
||||
Rebuild `m2dev-server-src` when:
|
||||
|
||||
- C++ code changed
|
||||
- CMake changed
|
||||
- smoke/login test utility changed
|
||||
|
||||
Runtime sync only may be enough when:
|
||||
|
||||
- configs changed
|
||||
- quest files changed
|
||||
- locale data changed
|
||||
- runtime assets changed
|
||||
|
||||
## When systemd Must Be Reinstalled
|
||||
|
||||
Re-run the systemd installer when files under `deploy/systemd/` change:
|
||||
|
||||
```bash
|
||||
python3 deploy/systemd/install_systemd.py \
|
||||
--user mt2.jakubkadlec.dev \
|
||||
--group mt2.jakubkadlec.dev \
|
||||
--runtime-root /home/mt2.jakubkadlec.dev/metin/runtime/server \
|
||||
--channel 1 \
|
||||
--channel 99 \
|
||||
--restart
|
||||
```
|
||||
|
||||
## Recommended Future Improvement
|
||||
|
||||
The current VPS-local helper scripts should eventually be moved into a versioned, reviewable deploy workflow in the repository, or replaced with a single documented deployment entrypoint.
|
||||
97
docs/rollback.md
Normal file
97
docs/rollback.md
Normal file
@@ -0,0 +1,97 @@
|
||||
# Rollback
|
||||
|
||||
This document describes the practical rollback workflow for the current Debian VPS.
|
||||
|
||||
## Important Principle
|
||||
|
||||
There are two rollback surfaces:
|
||||
|
||||
- source rollback (`m2dev-server-src`)
|
||||
- runtime-file rollback (`m2dev-server`)
|
||||
|
||||
They are related, but they are not the same repository and do not always need to move together.
|
||||
|
||||
## Fast Safety Rule
|
||||
|
||||
Before a risky deploy, record both current commits:
|
||||
|
||||
```bash
|
||||
ssh mt2
|
||||
sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server-src rev-parse --short HEAD
|
||||
sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server rev-parse --short HEAD
|
||||
```
|
||||
|
||||
If the deploy fails, rollback to those exact commits.
|
||||
|
||||
## Source Rollback
|
||||
|
||||
Use this when the regression is in:
|
||||
|
||||
- server binaries
|
||||
- network code
|
||||
- DB logic
|
||||
- C++ runtime behavior
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
ssh mt2
|
||||
sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server-src checkout <good_commit>
|
||||
sudo -iu mt2.jakubkadlec.dev cmake --build ~/metin/build/server-src --parallel "$(nproc)"
|
||||
sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/sync-runtime.sh
|
||||
systemctl restart metin-server.service
|
||||
/usr/local/sbin/metin-login-healthcheck
|
||||
```
|
||||
|
||||
## Runtime Rollback
|
||||
|
||||
Use this when the regression is in:
|
||||
|
||||
- configs
|
||||
- quests
|
||||
- locale files
|
||||
- runtime-only scripts
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
ssh mt2
|
||||
sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server checkout <good_commit>
|
||||
sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/sync-runtime.sh
|
||||
sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/compile-quests.sh
|
||||
systemctl restart metin-server.service
|
||||
/usr/local/sbin/metin-login-healthcheck
|
||||
```
|
||||
|
||||
## Combined Rollback
|
||||
|
||||
Use this when the regression could be caused by an interaction between:
|
||||
|
||||
- new binaries
|
||||
- new runtime files
|
||||
|
||||
Rollback both repositories together and rebuild/redeploy.
|
||||
|
||||
## systemd Rollback
|
||||
|
||||
If the regression is in `deploy/systemd/`, rollback the runtime repository to a known good commit and re-run:
|
||||
|
||||
```bash
|
||||
python3 deploy/systemd/install_systemd.py ... --restart
|
||||
```
|
||||
|
||||
## Verification After Rollback
|
||||
|
||||
Minimum verification:
|
||||
|
||||
```bash
|
||||
systemctl status metin-server.service --no-pager
|
||||
ss -ltnp | rg ':(9000|11000|11011|11012|11013|11991) '
|
||||
/usr/local/sbin/metin-login-healthcheck
|
||||
```
|
||||
|
||||
## What Not To Do
|
||||
|
||||
- do not use `git reset --hard` blindly on a dirty VPS checkout
|
||||
- do not rollback only one repository if the deploy changed both and the fault domain is unclear
|
||||
- do not close the current SSH session before the healthcheck passes
|
||||
Reference in New Issue
Block a user