Networking Overhaul: Modern packets, buffers, handshake, dispatch & security hardening

See Readme

README.md

@@ -55,13 +55,199 @@ The entire legacy encryption system has been replaced with [libsodium](https://d
 * **libsodium-based pack encryption** — `PackLib` now uses XChaCha20-Poly1305 for pack file encryption, replacing the legacy Camellia/XTEA system
 * **Secure key derivation** — Pack encryption keys are derived using `crypto_pwhash` (Argon2id)
 
-<br>
-<br>
+---
+
+### Networking Modernization Roadmap
+
+A 5-phase modernization of the entire client/server networking stack — packet format, buffer management, handshake protocol, connection architecture, and packet dispatching. Every phase is complete and verified on both client and server.
 
 ---
 
-<br>
-<br>
+#### Phase 1 — Packet Format + Buffer System + Memory Safety
+
+Replaced the legacy 1-byte packet headers and raw C-style buffers with a modern, uniform protocol.
+
+##### What changed
+* **2-byte headers + 2-byte length prefix** — All packet types (`CG::`, `GC::`, `GG::`, `GD::`, `DG::`) now use `uint16_t` header + `uint16_t` length. This increases the addressable packet space from 256 to 65,535 unique packet types and enables safe variable-length parsing
+* **Namespaced packet headers** — All headers moved from flat `HEADER_CG_*` defines to C++ namespaces: `CG::MOVE`, `GC::PING`, `GG::LOGIN`, `GD::PLAYER_SAVE`, `DG::BOOT`. Subheaders similarly namespaced: `GuildSub::GC::LOGIN`, `ShopSub::CG::BUY`, etc.
+* **RAII RingBuffer** — All raw `buffer_t` / `LPBUF` / `new[]`/`delete[]` patterns replaced with a single `RingBuffer` class featuring lazy compaction at 50% read position, exponential growth, and inlined accessors
+* **PacketReader / PacketWriter** — Type-safe helpers that wrap buffer access with bounds checking, eliminating raw pointer arithmetic throughout the codebase
+* **Sequence system modernized** — Packet sequence tracking retained for debugging but fixed to byte offset 4 (after header + length)
+* **SecureCipher** — XChaCha20-Poly1305 stream cipher for all post-handshake traffic (see Encryption section above)
+
+##### What was removed
+* `buffer.h` / `buffer.cpp` (legacy C buffer library)
+* All `LPBUF`, `buffer_new()`, `buffer_delete()`, `buffer_read()`, `buffer_write()` calls
+* Raw `new[]`/`delete[]` buffer allocations in DESC classes
+* 1-byte header constants (`HEADER_CG_*`, `HEADER_GC_*`, etc.)
+
+##### Why
+The legacy 1-byte header system limited the protocol to 256 packet types (already exhausted), raw C buffers had no bounds checking and were prone to buffer overflows, and the flat namespace caused header collisions between subsystems.
+
+---
+
+#### Phase 2 — Modern Buffer System *(merged into Phase 1)*
+
+All connection types now use `RingBuffer` uniformly.
+
+##### What changed
+* **All DESC types** (`DESC`, `CLIENT_DESC`, `DESC_P2P`) use `RingBuffer` for `m_inputBuffer`, `m_outputBuffer`, and `m_bufferedOutputBuffer`
+* **PeerBase** (db layer) ported to `RingBuffer`
+* **TEMP_BUFFER** (local utility for building packets) backed by `RingBuffer`
+
+##### What was removed
+* `libthecore/buffer.h` and `libthecore/buffer.cpp` — the entire legacy buffer library
+
+##### Why
+The legacy buffer system used separate implementations across different connection types, had no RAII semantics (manual malloc/free), and offered no protection against buffer overflows.
+
+---
+
+#### Phase 3 — Simplified Handshake
+
+Replaced the legacy multi-step handshake (4+ round trips with time synchronization and UDP binding) with a streamlined 1.5 round-trip flow.
+
+##### What changed
+* **1.5 round-trip handshake** — Server sends `GC::KEY_CHALLENGE` (with embedded time sync), client responds with `CG::KEY_RESPONSE`, server confirms with `GC::KEY_COMPLETE`. Session is encrypted from that point forward
+* **Time sync embedded** — Initial time synchronization folded into `GC::KEY_CHALLENGE`; periodic time sync handled by `GC::PING` / `CG::PONG`
+* **Handshake timeout** — 5-second expiry on handshake phase; stale connections are automatically cleaned up in `DestroyClosed()`
+
+##### What was removed
+* **6 dead packet types**: `CG_HANDSHAKE`, `GC_HANDSHAKE`, `CG_TIME_SYNC`, `GC_TIME_SYNC`, `GC_HANDSHAKE_OK`, `GC_BINDUDP`
+* **Server functions**: `StartHandshake()`, `SendHandshake()`, `HandshakeProcess()`, `CreateHandshake()`, `FindByHandshake()`, `m_map_handshake`
+* **Client functions**: `RecvHandshakePacket()`, `RecvHandshakeOKPacket()`, `m_HandshakeData`, `SendHandshakePacket()`
+* ~12 server files and ~10 client files modified
+
+##### Why
+The original handshake required 4+ round trips, included dead UDP binding steps, had no timeout protection (stale connections could linger indefinitely), and the time sync was a separate multi-step sub-protocol that added latency to every new connection.
+
+---
+
+#### Phase 4 — Unified Connection (Client-Side Deduplication)
+
+Consolidated duplicated connection logic into the base `CNetworkStream` class.
+
+##### What changed
+* **Key exchange** (`RecvKeyChallenge` / `RecvKeyComplete`) moved from 4 separate implementations to `CNetworkStream` base class
+* **Ping/pong** (`RecvPingPacket` / `SendPongPacket`) moved from 3 separate implementations to `CNetworkStream` base class
+* **CPythonNetworkStream** overrides `RecvKeyChallenge` only for time sync, delegates all crypto to base
+* **CGuildMarkDownloader/Uploader** — `RecvKeyCompleteAndLogin` wraps base + sends `CG::MARK_LOGIN`
+* **CAccountConnector** — Fixed raw `crypto_aead` bug (now uses base class `cipher.DecryptToken`)
+* **Control-plane structs** extracted to `EterLib/ControlPackets.h` (Phase, Ping, Pong, KeyChallenge, KeyResponse, KeyComplete)
+* **CGuildMarkUploader** — `m_pbySymbolBuf` migrated from `new[]`/`delete[]` to `std::vector<uint8_t>`
+
+##### What was removed
+* ~200 lines of duplicated code across `CAccountConnector`, `CGuildMarkDownloader`, `CGuildMarkUploader`, and `CPythonNetworkStream`
+
+##### Why
+The same key exchange and ping/pong logic was copy-pasted across 3-4 connection subclasses, leading to inconsistent behavior (the `CAccountConnector` had a raw crypto bug), difficult maintenance, and unnecessary code volume.
+
+---
+
+#### Phase 5 — Packet Handler Registration / Dispatcher
+
+Replaced giant `switch` statements with `std::unordered_map` dispatch tables for O(1) packet routing.
+
+##### What changed
+
+**Client:**
+* `CPythonNetworkStream` — Phase-specific handler maps for Game, Loading, Login, Select, and Handshake phases
+* Registration pattern: `m_gameHandlers[GC::MOVE] = &CPythonNetworkStream::RecvCharacterMovePacket;`
+* Dispatch: `DispatchPacket(m_gameHandlers)` — reads header, looks up handler, calls it
+
+**Server:**
+* `CInputMain`, `CInputDead`, `CInputAuth`, `CInputLogin`, `CInputP2P`, `CInputHandshake`, `CInputDB` — all converted to dispatch tables
+* `CInputDB` uses 3 template adapters (`DataHandler`, `DescHandler`, `TypedHandler`) + 14 custom adapters for the diverse DB callback signatures
+
+##### What was removed
+* All `switch (header)` blocks across 7 server input processors and 5 client phase handlers
+* ~3,000 lines of switch/case boilerplate
+
+##### Why
+The original dispatch used switch statements with 50-100+ cases each. Adding a new packet required modifying a massive switch block, which was error-prone and caused merge conflicts. The table-driven approach enables O(1) lookup, self-documenting handler registration, and trivial addition of new packet types.
+
+---
+
+### Post-Phase 5 Cleanup
+
+Follow-up tasks after the core roadmap was complete.
+
+#### One-Liner Adapter Reformat
+* 24 adapter methods across 4 server files (`input_main.cpp`, `input_p2p.cpp`, `input_auth.cpp`, `input_login.cpp`) reformatted from single-line to multi-line for readability
+
+#### MAIN_CHARACTER Packet Merge
+* **4 mutually exclusive packets** (`GC::MAIN_CHARACTER`, `MAIN_CHARACTER2_EMPIRE`, `MAIN_CHARACTER3_BGM`, `MAIN_CHARACTER4_BGM_VOL`) merged into a single unified `GC::MAIN_CHARACTER` packet
+* Single struct always includes BGM fields (zero when unused — 29 extra bytes on a one-time-per-load packet)
+* 4 nearly identical client handlers merged into 1
+* 3 redundant server send paths merged into 1
+
+#### UDP Leftover Removal
+* **7 client files deleted**: `NetDatagram.h/.cpp`, `NetDatagramReceiver.h/.cpp`, `NetDatagramSender.h/.cpp`, `PythonNetworkDatagramModule.cpp`
+* **8 files edited**: Removed dead stubs (`PushUDPState`, `initudp`, `netSetUDPRecvBufferSize`, `netConnectUDP`), declarations, and Python method table entries
+* **Server**: Removed `socket_udp_read()`, `socket_udp_bind()`, `__UDP_BLOCK__` define
+
+#### Subheader Dispatch
+* Extended the Phase 5 table-driven pattern to subheader switches with 8+ cases
+* **Client**: Guild (19 sub-handlers), Shop (10), Exchange (8) in `PythonNetworkStreamPhaseGame.cpp`
+* **Server**: Guild (15 sub-handlers) in `input_main.cpp`
+* Small switches intentionally kept as-is: Messenger (5), Fishing (6), Dungeon (2), Server Shop (4)
+
+---
+
+### Performance Audit & Optimization
+
+Comprehensive audit of all Phase 1-5 changes to identify and eliminate performance overhead.
+
+#### Debug Logging Cleanup
+* **Removed all hot-path `TraceError`/`Tracef`/`sys_log`** from networking code on both client and server
+* Client: `NetStream.cpp`, `SecureCipher.cpp`, `PythonNetworkStream*.cpp` — eliminated per-frame and per-packet traces that caused disk I/O every frame
+* Server: `desc.cpp`, `input.cpp`, `input_login.cpp`, `input_auth.cpp`, `SecureCipher.cpp` — eliminated `[SEND]`, `[RECV]`, `[CIPHER]` logs that fired on every packet
+
+#### Packet Processing Throughput
+* **`MAX_RECV_COUNT` 4 → 32** — Game phase now processes up to 32 packets per frame (was 4, severely limiting entity spawning on map entry)
+* **Loading phase while-loop** — Changed from processing 1 packet per frame to draining all available packets, making phase transitions near-instant
+
+#### Flood Check Optimization
+* **Replaced `get_dword_time()` with `thecore_pulse()`** in `DESC::CheckPacketFlood()` — eliminates a `gettimeofday()` syscall on every single packet received. `thecore_pulse()` is cached once per game-loop iteration
+
+#### Flood Protection
+* **Per-IP connection limits** — Configurable maximum connections per IP address (`flood_max_connections_per_ip`, default: 10)
+* **Global connection limits** — Configurable maximum total connections (`flood_max_global_connections`, default: 8192)
+* **Per-second packet rate limiting** — Connections exceeding `flood_max_packets_per_sec` (default: 300) are automatically disconnected
+* **Handshake timeout** — 5-second expiry prevents connection slot exhaustion from incomplete handshakes
+
+---
+
+### Pre-Phase 3 Cleanup
+
+Preparatory cleanup performed before the handshake simplification.
+
+* **File consolidation** — Merged scattered packet definitions into centralized header files
+* **Alias removal** — Removed legacy `#define` aliases that mapped old names to new identifiers
+* **Monarch system removal** — Completely removed the unused Monarch (emperor) system from both client and server, including all related packets, commands, quest functions, and UI code
+* **TrafficProfiler removal** — Removed the `TrafficProfiler` class and all references (unnecessary runtime overhead)
+* **Quest management stub removal** — Removed empty `questlua_mgmt.cpp` (monarch-era placeholder with no functions)
+
+---
+
+### Summary of Removed Legacy Systems
+
+A consolidated reference of all legacy systems, files, and dead code removed across the entire modernization effort.
+
+| System | What was removed | Replaced by |
+|--------|-----------------|-------------|
+| **Legacy C buffer** | `buffer.h`, `buffer.cpp`, all `LPBUF`/`buffer_new()`/`buffer_delete()` calls, raw `new[]`/`delete[]` buffer allocations | RAII `RingBuffer` class |
+| **1-byte packet headers** | All `HEADER_CG_*`, `HEADER_GC_*`, `HEADER_GG_*`, `HEADER_GD_*`, `HEADER_DG_*` defines | 2-byte namespaced headers (`CG::`, `GC::`, `GG::`, `GD::`, `DG::`) |
+| **Old handshake protocol** | 6 packet types (`CG_HANDSHAKE`, `GC_HANDSHAKE`, `CG_TIME_SYNC`, `GC_TIME_SYNC`, `GC_HANDSHAKE_OK`, `GC_BINDUDP`), all handshake functions and state | 1.5 round-trip key exchange (`KEY_CHALLENGE`/`KEY_RESPONSE`/`KEY_COMPLETE`) |
+| **UDP networking** | 7 client files (`NetDatagram*.h/.cpp`, `PythonNetworkDatagramModule.cpp`), server `socket_udp_read()`/`socket_udp_bind()`/`__UDP_BLOCK__` | Removed entirely (game is TCP-only) |
+| **Old sequence system** | `m_seq`, `SetSequence()`, `GetSequence()`, old sequence variables | Modernized sequence at fixed byte offset 4 |
+| **TrafficProfiler** | `TrafficProfiler` class and all references | Removed entirely |
+| **Monarch system** | All monarch/emperor packets, commands (`do_monarch_*`), quest functions (`questlua_monarch.cpp`, `questlua_mgmt.cpp`), UI code, GM commands | Removed entirely (unused feature) |
+| **Legacy crypto** | Crypto++, Panama cipher, TEA, DH2, Camellia, XTEA, `adwClientKey[4]`, `LSS_SECURITY_KEY` | libsodium (X25519 + XChaCha20-Poly1305) |
+| **Switch-based dispatch** | Giant `switch (header)` blocks (50-100+ cases each) across 7 server input processors and 5 client phase handlers | `std::unordered_map` dispatch tables |
+| **Duplicated connection code** | Key exchange and ping/pong copy-pasted across 3-4 client subclasses | Consolidated in `CNetworkStream` base class |
+
+---
 
 # Installation/Configuration
 This is the third part of the entire project and it's about the client binary, the executable of the game.