docs: expand operational runbooks
This commit is contained in:
server
@@ -6,10 +6,13 @@ Current documents:
|
|||||||
|
|
||||||
- [Debian Runtime](debian-runtime.md)
|
- [Debian Runtime](debian-runtime.md)
|
||||||
- [Healthchecks](healthchecks.md)
|
- [Healthchecks](healthchecks.md)
|
||||||
|
- [Deploy Workflow](deploy-workflow.md)
|
||||||
|
- [Rollback](rollback.md)
|
||||||
|
- [Database Bootstrap](database-bootstrap.md)
|
||||||
|
- [Config And Secrets](config-and-secrets.md)
|
||||||
|
|
||||||
Suggested next additions:
|
Suggested next additions:
|
||||||
|
|
||||||
- database/bootstrap notes
|
- client connection notes
|
||||||
- deployment workflow
|
- incident response checklist
|
||||||
- rollback procedure
|
- runtime log map
|
||||||
- configuration/secrets contract
|
|
||||||
|
|||||||
56
docs/config-and-secrets.md
Normal file
56
docs/config-and-secrets.md
Normal file
@@ -0,0 +1,56 @@
|
|||||||
|
# Config And Secrets
|
||||||
|
|
||||||
|
This document describes the current config layout and the intended secret-handling boundary for the Debian deployment.
|
||||||
|
|
||||||
|
## Current Config Files In Repo
|
||||||
|
|
||||||
|
Main runtime config examples live under:
|
||||||
|
|
||||||
|
- `share/conf/db.txt`
|
||||||
|
- `share/conf/game.txt`
|
||||||
|
|
||||||
|
These files are part of the runtime tree and are required by the legacy server layout.
|
||||||
|
|
||||||
|
## Important Security Boundary
|
||||||
|
|
||||||
|
The repository currently contains literal config values for legacy compatibility. They must be treated as bootstrap/default values, not as a safe long-term secret store.
|
||||||
|
|
||||||
|
Practical rule:
|
||||||
|
|
||||||
|
- do not treat git-tracked config files as the final production secret source of truth
|
||||||
|
|
||||||
|
## What Should Not Live In Git
|
||||||
|
|
||||||
|
Do not commit:
|
||||||
|
|
||||||
|
- production DB passwords
|
||||||
|
- real admin page passwords
|
||||||
|
- host-specific private tokens
|
||||||
|
- private SSH material
|
||||||
|
- per-environment override files with live secrets
|
||||||
|
|
||||||
|
## Current Operational Model
|
||||||
|
|
||||||
|
For the current Debian VPS:
|
||||||
|
|
||||||
|
- root-only operational wrappers may inject short-lived values locally
|
||||||
|
- headless login healthcheck uses a temporary password via environment, not a command-line literal
|
||||||
|
- the installed wrapper is root-only and not network-facing
|
||||||
|
|
||||||
|
## Admin Page Password
|
||||||
|
|
||||||
|
The source repository now supports hardening for the admin page password. The long-term goal should be:
|
||||||
|
|
||||||
|
- no implicit production default
|
||||||
|
- host-local secret injection
|
||||||
|
- explicit runtime validation
|
||||||
|
|
||||||
|
## Recommended Direction
|
||||||
|
|
||||||
|
The Debian deployment should eventually move to a clearer contract such as:
|
||||||
|
|
||||||
|
- git-tracked template/default files
|
||||||
|
- host-local env file or secret file owned by root
|
||||||
|
- documented override points
|
||||||
|
|
||||||
|
Until that is done, keep all real secret rotation and secret overrides on the host, not in commits.
|
||||||
69
docs/database-bootstrap.md
Normal file
69
docs/database-bootstrap.md
Normal file
@@ -0,0 +1,69 @@
|
|||||||
|
# Database Bootstrap
|
||||||
|
|
||||||
|
This document describes the database files included in the repository and the current bootstrap model used for the Debian VPS.
|
||||||
|
|
||||||
|
## Databases
|
||||||
|
|
||||||
|
The current server layout expects these databases:
|
||||||
|
|
||||||
|
- `account`
|
||||||
|
- `player`
|
||||||
|
- `common`
|
||||||
|
- `log`
|
||||||
|
- `hotbackup`
|
||||||
|
|
||||||
|
## Repository SQL Files
|
||||||
|
|
||||||
|
Relevant files:
|
||||||
|
|
||||||
|
- `sql/database_create/database_create.sql`
|
||||||
|
- `sql/account.sql`
|
||||||
|
- `sql/player.sql`
|
||||||
|
- `sql/common.sql`
|
||||||
|
- `sql/log.sql`
|
||||||
|
- `sql/hotbackup_(empty).sql`
|
||||||
|
- `sql/my.cnf`
|
||||||
|
- `sql/my.cnf.bak_for_mysql5.6`
|
||||||
|
|
||||||
|
## Practical Import Order
|
||||||
|
|
||||||
|
Typical bootstrap order:
|
||||||
|
|
||||||
|
1. create databases and database user
|
||||||
|
2. import `account.sql`
|
||||||
|
3. import `player.sql`
|
||||||
|
4. import `common.sql`
|
||||||
|
5. import `log.sql`
|
||||||
|
6. import `hotbackup_(empty).sql`
|
||||||
|
|
||||||
|
## Notes About The Included SQL Config Files
|
||||||
|
|
||||||
|
`sql/my.cnf` and related files are legacy reference material. They are not the authoritative production configuration for the Debian VPS.
|
||||||
|
|
||||||
|
The Debian runtime currently uses MariaDB and system-level service management rather than a repo-owned database daemon.
|
||||||
|
|
||||||
|
## Current Runtime Expectation
|
||||||
|
|
||||||
|
Server config files reference:
|
||||||
|
|
||||||
|
- `DB_ADDR: 127.0.0.1`
|
||||||
|
- `DB_PORT: 9000`
|
||||||
|
|
||||||
|
This means:
|
||||||
|
|
||||||
|
- the game/auth processes connect to the DB proxy/cache process on `127.0.0.1:9000`
|
||||||
|
- that DB process then talks to MariaDB using the configured SQL credentials
|
||||||
|
|
||||||
|
## Production Reality
|
||||||
|
|
||||||
|
The current production VPS already has the databases imported and in use. The headless login healthcheck proves that `account` and `player` reads/writes are working end-to-end.
|
||||||
|
|
||||||
|
## Recommended Follow-Up
|
||||||
|
|
||||||
|
This repo should eventually gain a Debian-first documented bootstrap script or runbook that covers:
|
||||||
|
|
||||||
|
- MariaDB package installation
|
||||||
|
- SQL mode compatibility
|
||||||
|
- database/user creation
|
||||||
|
- import commands
|
||||||
|
- post-import verification
|
||||||
165
docs/deploy-workflow.md
Normal file
165
docs/deploy-workflow.md
Normal file
@@ -0,0 +1,165 @@
|
|||||||
|
# Deploy Workflow
|
||||||
|
|
||||||
|
This document describes the current Debian deployment workflow used on the VPS.
|
||||||
|
|
||||||
|
## Repositories And Responsibilities
|
||||||
|
|
||||||
|
There are two separate repositories involved in a production deploy:
|
||||||
|
|
||||||
|
- `m2dev-server-src`: source code, build system, smoke tests, login smoke utility
|
||||||
|
- `m2dev-server`: runtime files, configs, quests, systemd deployment assets
|
||||||
|
|
||||||
|
In practice:
|
||||||
|
|
||||||
|
- source changes are built from `m2dev-server-src`
|
||||||
|
- runtime files are synchronized from `m2dev-server`
|
||||||
|
- the live instance runs from `~/metin/runtime/server`
|
||||||
|
|
||||||
|
## Current VPS Helper Scripts
|
||||||
|
|
||||||
|
The current production VPS also has local host-side helper scripts under:
|
||||||
|
|
||||||
|
```text
|
||||||
|
/home/mt2.jakubkadlec.dev/metin/deploy/
|
||||||
|
```
|
||||||
|
|
||||||
|
Current helpers:
|
||||||
|
|
||||||
|
- `build-server.sh`
|
||||||
|
- `sync-runtime.sh`
|
||||||
|
- `compile-quests.sh`
|
||||||
|
- `install-runtime.sh`
|
||||||
|
|
||||||
|
These are currently VPS-local operational wrappers. They are useful, but they are not yet versioned in this repository.
|
||||||
|
|
||||||
|
## Common Deploy Paths
|
||||||
|
|
||||||
|
Source repository:
|
||||||
|
|
||||||
|
```text
|
||||||
|
/home/mt2.jakubkadlec.dev/metin/repos/m2dev-server-src
|
||||||
|
```
|
||||||
|
|
||||||
|
Runtime repository:
|
||||||
|
|
||||||
|
```text
|
||||||
|
/home/mt2.jakubkadlec.dev/metin/repos/m2dev-server
|
||||||
|
```
|
||||||
|
|
||||||
|
Build directory:
|
||||||
|
|
||||||
|
```text
|
||||||
|
/home/mt2.jakubkadlec.dev/metin/build/server-src
|
||||||
|
```
|
||||||
|
|
||||||
|
Live runtime:
|
||||||
|
|
||||||
|
```text
|
||||||
|
/home/mt2.jakubkadlec.dev/metin/runtime/server
|
||||||
|
```
|
||||||
|
|
||||||
|
## Standard Deploy Sequence
|
||||||
|
|
||||||
|
### 1. Update repositories
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh mt2
|
||||||
|
sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server-src pull --ff-only
|
||||||
|
sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server pull --ff-only
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Build source
|
||||||
|
|
||||||
|
Equivalent manual build:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo -iu mt2.jakubkadlec.dev \
|
||||||
|
cmake -S ~/metin/repos/m2dev-server-src \
|
||||||
|
-B ~/metin/build/server-src \
|
||||||
|
-G Ninja
|
||||||
|
|
||||||
|
sudo -iu mt2.jakubkadlec.dev \
|
||||||
|
cmake --build ~/metin/build/server-src --parallel "$(nproc)"
|
||||||
|
```
|
||||||
|
|
||||||
|
Current VPS wrapper:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/build-server.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Synchronize runtime files
|
||||||
|
|
||||||
|
Equivalent manual sync:
|
||||||
|
|
||||||
|
- copy runtime tree from `repos/m2dev-server`
|
||||||
|
- exclude generated runtime state
|
||||||
|
- install built `game`, `db`, `qc` into the live runtime
|
||||||
|
|
||||||
|
Current VPS wrapper:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/sync-runtime.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Compile quests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/compile-quests.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
This runs:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/metin/runtime/server/share/locale/english/quest
|
||||||
|
python3 make.py
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Restart services
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl restart metin-server.service
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6. Verify runtime
|
||||||
|
|
||||||
|
Recommended checks:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl status metin-server.service --no-pager
|
||||||
|
ss -ltnp | rg ':(9000|11000|11011|11012|11013|11991) '
|
||||||
|
/usr/local/sbin/metin-login-healthcheck
|
||||||
|
```
|
||||||
|
|
||||||
|
## When A Full Rebuild Is Needed
|
||||||
|
|
||||||
|
Rebuild `m2dev-server-src` when:
|
||||||
|
|
||||||
|
- C++ code changed
|
||||||
|
- CMake changed
|
||||||
|
- smoke/login test utility changed
|
||||||
|
|
||||||
|
Runtime sync only may be enough when:
|
||||||
|
|
||||||
|
- configs changed
|
||||||
|
- quest files changed
|
||||||
|
- locale data changed
|
||||||
|
- runtime assets changed
|
||||||
|
|
||||||
|
## When systemd Must Be Reinstalled
|
||||||
|
|
||||||
|
Re-run the systemd installer when files under `deploy/systemd/` change:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 deploy/systemd/install_systemd.py \
|
||||||
|
--user mt2.jakubkadlec.dev \
|
||||||
|
--group mt2.jakubkadlec.dev \
|
||||||
|
--runtime-root /home/mt2.jakubkadlec.dev/metin/runtime/server \
|
||||||
|
--channel 1 \
|
||||||
|
--channel 99 \
|
||||||
|
--restart
|
||||||
|
```
|
||||||
|
|
||||||
|
## Recommended Future Improvement
|
||||||
|
|
||||||
|
The current VPS-local helper scripts should eventually be moved into a versioned, reviewable deploy workflow in the repository, or replaced with a single documented deployment entrypoint.
|
||||||
97
docs/rollback.md
Normal file
97
docs/rollback.md
Normal file
@@ -0,0 +1,97 @@
|
|||||||
|
# Rollback
|
||||||
|
|
||||||
|
This document describes the practical rollback workflow for the current Debian VPS.
|
||||||
|
|
||||||
|
## Important Principle
|
||||||
|
|
||||||
|
There are two rollback surfaces:
|
||||||
|
|
||||||
|
- source rollback (`m2dev-server-src`)
|
||||||
|
- runtime-file rollback (`m2dev-server`)
|
||||||
|
|
||||||
|
They are related, but they are not the same repository and do not always need to move together.
|
||||||
|
|
||||||
|
## Fast Safety Rule
|
||||||
|
|
||||||
|
Before a risky deploy, record both current commits:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh mt2
|
||||||
|
sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server-src rev-parse --short HEAD
|
||||||
|
sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server rev-parse --short HEAD
|
||||||
|
```
|
||||||
|
|
||||||
|
If the deploy fails, rollback to those exact commits.
|
||||||
|
|
||||||
|
## Source Rollback
|
||||||
|
|
||||||
|
Use this when the regression is in:
|
||||||
|
|
||||||
|
- server binaries
|
||||||
|
- network code
|
||||||
|
- DB logic
|
||||||
|
- C++ runtime behavior
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh mt2
|
||||||
|
sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server-src checkout <good_commit>
|
||||||
|
sudo -iu mt2.jakubkadlec.dev cmake --build ~/metin/build/server-src --parallel "$(nproc)"
|
||||||
|
sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/sync-runtime.sh
|
||||||
|
systemctl restart metin-server.service
|
||||||
|
/usr/local/sbin/metin-login-healthcheck
|
||||||
|
```
|
||||||
|
|
||||||
|
## Runtime Rollback
|
||||||
|
|
||||||
|
Use this when the regression is in:
|
||||||
|
|
||||||
|
- configs
|
||||||
|
- quests
|
||||||
|
- locale files
|
||||||
|
- runtime-only scripts
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh mt2
|
||||||
|
sudo -iu mt2.jakubkadlec.dev git -C ~/metin/repos/m2dev-server checkout <good_commit>
|
||||||
|
sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/sync-runtime.sh
|
||||||
|
sudo -iu mt2.jakubkadlec.dev ~/metin/deploy/compile-quests.sh
|
||||||
|
systemctl restart metin-server.service
|
||||||
|
/usr/local/sbin/metin-login-healthcheck
|
||||||
|
```
|
||||||
|
|
||||||
|
## Combined Rollback
|
||||||
|
|
||||||
|
Use this when the regression could be caused by an interaction between:
|
||||||
|
|
||||||
|
- new binaries
|
||||||
|
- new runtime files
|
||||||
|
|
||||||
|
Rollback both repositories together and rebuild/redeploy.
|
||||||
|
|
||||||
|
## systemd Rollback
|
||||||
|
|
||||||
|
If the regression is in `deploy/systemd/`, rollback the runtime repository to a known good commit and re-run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 deploy/systemd/install_systemd.py ... --restart
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verification After Rollback
|
||||||
|
|
||||||
|
Minimum verification:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl status metin-server.service --no-pager
|
||||||
|
ss -ltnp | rg ':(9000|11000|11011|11012|11013|11991) '
|
||||||
|
/usr/local/sbin/metin-login-healthcheck
|
||||||
|
```
|
||||||
|
|
||||||
|
## What Not To Do
|
||||||
|
|
||||||
|
- do not use `git reset --hard` blindly on a dirty VPS checkout
|
||||||
|
- do not rollback only one repository if the deploy changed both and the fault domain is unclear
|
||||||
|
- do not close the current SSH session before the healthcheck passes
|
||||||
Reference in New Issue
Block a user