# Server Management

This document describes the current Debian-side control plane for the production Metin runtime.

## Inventory

The channel topology now lives in one versioned file:

- `deploy/channel-inventory.json`

It defines:

- auth and DB listener ports
- channel ids
- per-core public ports and P2P ports
- whether a channel is public/client-visible
- whether a special channel should always be included by management tooling

This inventory is now the source used by:

- `channel_inventory.py`
- `channels.py` compatibility exports
- `install.py`
- `deploy/systemd/install_systemd.py`
- `metinctl`

## metinctl

The Debian deployment installs:

- `/usr/local/bin/metinctl`

`metinctl` is a lightweight operational CLI for:

- showing an operational summary
- showing recent auth success/failure activity
- showing auth activity grouped by source IP
- showing recent `syserr.log` entries
- summarizing recurring `syserr.log` entries
- viewing inventory
- listing managed units
- checking service status
- listing declared ports
- listing recent auth failures
- listing recent login sessions
- listing stale open sessions without logout
- restarting the whole stack or specific channels/instances
- viewing logs
- listing core files in the runtime tree
- generating a backtrace for the newest or selected core file
- collecting incident bundles
- running the root-only headless healthcheck
- waiting for login-ready state after restart

## Examples

Show inventory:

```bash
metinctl inventory
```

Show current unit state:

```bash
metinctl status
```

Show a quick operational summary:

```bash
metinctl summary
```

Show declared ports and whether they are currently listening:

```bash
metinctl ports --live
```

Show recent real auth failures and skip smoke-test logins:

```bash
metinctl auth-failures
```

Show recent auth success/failure flow:

```bash
metinctl auth-activity
```

Show only recent auth failures including smoke tests:

```bash
metinctl auth-activity --status failure --include-smoke
```

Show auth activity grouped by IP:

```bash
metinctl auth-ips
```

Show the latest runtime errors collected from all `syserr.log` files:

```bash
metinctl recent-errors
```

Show the most repeated runtime errors in the last 24 hours:

```bash
metinctl error-summary
```

Include smoke-test failures too:

```bash
metinctl auth-failures --include-smoke
```

Show recent login sessions from `log.loginlog2`:

```bash
metinctl sessions
```

Show only sessions that still have no recorded logout:

```bash
metinctl sessions --active-only
```

Show stale open sessions older than 30 minutes:

```bash
metinctl session-audit
```

Use a different stale threshold:

```bash
metinctl session-audit --stale-minutes 10
```

Restart only channel 1 cores:

```bash
metinctl restart channel:1
```

Restart one specific game instance:

```bash
metinctl restart instance:channel1_core2
```

Tail auth logs:

```bash
metinctl logs auth -n 200 -f
```

Run the deeper end-to-end healthcheck:

```bash
metinctl healthcheck --mode full
```

Run the lighter readiness probe:

```bash
metinctl healthcheck --mode ready
```

Wait until a restarted stack is login-ready:

```bash
metinctl wait-ready
```

List core files currently present in the runtime tree:

```bash
metinctl cores
```

Generate a backtrace for the newest core file:

```bash
metinctl backtrace
```

Generate a backtrace for one specific core file:

```bash
metinctl backtrace --core channels/channel1/core1/core.2255450
```

Collect an incident bundle with logs, unit status, port state and repository revisions:

```bash
metinctl incident-collect --tag auth-timeout --since "-20 minutes"
```

List the most recent incident bundles:

```bash
metinctl incidents
```

## systemd installer behavior

`deploy/systemd/install_systemd.py` now uses the same inventory and installs `metinctl`.

It also reconciles enabled game instance units against the selected channels:

- selected game units are enabled
- stale game units are disabled
- if `--restart` is passed, stale game units are disabled with `--now`

This makes channel enablement declarative instead of depending on whatever happened to be enabled previously.

## Crash / Incident Pipeline

The Debian deployment now also installs:

- `/usr/local/sbin/metin-collect-incident`
- `/usr/local/sbin/metin-core-backtrace`

The collector creates a timestamped bundle under:

- `/var/lib/metin/incidents`

Each bundle contains:

- repo revisions for `m2dev-server` and `m2dev-server-src`
- `systemctl status` for the whole stack
- recent `journalctl` output per unit
- listener state from `ss -ltnp`
- tailed runtime `syslog.log` and `syserr.log` files
- metadata for any `core*` files found under `runtime/server/channels`

If you call it with `--include-cores`, matching core files are copied into the bundle as well.

The runtime units now also declare `LimitCORE=infinity`, so after the next service restart the processes are allowed to emit core dumps when the host kernel/core policy permits it.

For quick manual crash triage outside the incident bundle flow, use:

```bash
metinctl backtrace
```

It defaults to the newest core file under the runtime tree, infers the executable path, and uses `gdb` or `lldb` when present on the host. If no supported debugger is installed, it still prints file/readelf metadata for the core and executable.