mirror of
https://github.com/Corfirean/coa-server-manager.git
synced 2026-11-21 03:25:56 +00:00
feat(portable): offline importer into a stopped realm, with journaled recovery (Phase 3)
Imports a canonical snapshot into a stopped disposable realm through the existing Db abstraction, in one MySQL transaction: guids allocated inside the transaction from the realm's own tables and every reference remapped, orphan cleanup, name and start-position handling, self-asserting result, and a marker row. Untrusted strings and blobs reach SQL only as hex literals from a typed encoder. Manager mappings are recorded only after the realm commit, in one local transaction; an import journal (schema 3) plus the marker recover a crash between the two commits. Preflight refuses wrong ruleset, running realm, missing account/content and more before anything is written. Stock character_talent rows are never written: a real worldserver asserts on a non-talent row. Item identity no longer includes the crafter (v2). Production import remains the designed in-core PortableImportService (docs/PORTABLE_IMPORT_SERVICE.md); nothing of it is implemented. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d971fd5e28
commit
e8f99c16b7
21 files changed
+2796
-23
No files matched your search
@@ -642,6 +642,12 @@ D1-D13 were approved as recommended in section 15, with these amendments:
|
||||
| D13 checkpoints | keyed **internally by the local guid**: checkpoints arrive as `(server_id, local_guid)` and are resolved to the portable character through `Store::find_by_local` |
|
||||
| item mapping | `UNIQUE(character_id, server_id, local_item_guid)` in addition to the primary key (implemented in Phase 1) |
|
||||
|
||||
**Phase 2.1 / Phase 3 amendments (2026-10-06):** `character_settings` carry list tightened to ten strict key patterns, everything
|
||||
else quarantined (`PORTABLE_EXPORT.md`); Phase 3 is an offline importer into a stopped disposable realm that keeps the existing
|
||||
`Db` abstraction, writes untrusted text only as hex literals from a typed encoder, imports in one MySQL transaction with
|
||||
journaled crash recovery (`PORTABLE_IMPORT.md`); the production importer is an in-core `PortableImportService` triggered over RA by
|
||||
an opaque job id (`PORTABLE_IMPORT_SERVICE.md`, design only).
|
||||
|
||||
---------------------------------------------------------------------------------------------------------
|
||||
|
||||
## Appendix A. Phase 1 - exact proposed plan (not started)
|
||||
|
||||
@@ -76,7 +76,7 @@ Account-wide sets of numeric ids (`coa:wardrobe`, `coa:vanity`, ...), stored per
|
||||
## Store (`%LOCALAPPDATA%\CoAServerManager\portable\portable.db`)
|
||||
|
||||
SQLite, WAL, `synchronous=FULL`, `foreign_keys=ON`, `application_id = 0x434F4150`, forward-only migrations tracked in
|
||||
`PRAGMA user_version` (currently 2). A database from a newer Manager, or a foreign SQLite file, is refused and left untouched.
|
||||
`PRAGMA user_version` (currently 3). A database from a newer Manager, or a foreign SQLite file, is refused and left untouched.
|
||||
|
||||
| Table | Purpose |
|
||||
|---|---|
|
||||
@@ -85,6 +85,7 @@ SQLite, WAL, `synchronous=FULL`, `foreign_keys=ON`, `application_id = 0x434F4150
|
||||
| `snapshot` | `(character_id, revision)` -> compressed canonical payload + hash |
|
||||
| `character_server_mapping` | `(character_id, server_id)` -> local guid; `UNIQUE(server_id, local_guid)` |
|
||||
| `item_mapping` | one row per mapping with a lifecycle (schema 2, see below) |
|
||||
| `import_journal` | schema 3: one row per import into a realm (`prepared` / `committed` / `aborted` / `needs_attention`) with the nonce/marker and the plan; see `PORTABLE_IMPORT.md` |
|
||||
| `collection` | per profile and kind: revision, hash, encoded set |
|
||||
| `setting` | `history_keep` |
|
||||
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
# Offline import into a stopped realm (Phase 3)
|
||||
|
||||
Code: `crates/coa-core/src/portable/realm/{sqlenc,plan,import}.rs`, journal in `portable/store/journal.rs`.
|
||||
|
||||
This is the **offline** importer: it writes a canonical snapshot into a **stopped** realm's characters database. It proves
|
||||
the model end to end and serves realms that are stopped. It is **not** the production path for a running public host:
|
||||
that is the in-core `PortableImportService` ([`PORTABLE_IMPORT_SERVICE.md`](PORTABLE_IMPORT_SERVICE.md)), which is designed but
|
||||
not implemented.
|
||||
|
||||
## Flow
|
||||
|
||||
```text
|
||||
preflight (read-only snapshot) -> journal "prepared" -> ONE realm transaction -> mappings + journal "committed"
|
||||
(marker inside it) (ONE local transaction)
|
||||
```
|
||||
|
||||
1. **Preflight** (`preflight`): a read-only consistent snapshot of the realm. Refuses (`ImportRefused`, nothing written,
|
||||
no journal entry) when the character belongs to another ruleset (CoA and Wildcard never mix), an id is of a namespace
|
||||
the importer does not apply, the realm is running (a session of the game server's database user exists), characters are
|
||||
marked online, the account does not exist / is a bot or Manager account / is full, the realm has no start position for the
|
||||
race and class, or `item_template` / `creature_template` do not know an entry the character carries (the realm would
|
||||
delete such items on load).
|
||||
2. **Journal**: `begin_import` writes an `import_journal` row `prepared` (nonce, the plan: portable ids, entries,
|
||||
identities) *before* anything happens in the realm.
|
||||
3. **Realm transaction**: one `mysql` session, one `START TRANSACTION ... COMMIT`, generated by `plan.rs`.
|
||||
4. **Local commit**: `finish_import` records the realm binding, the item mappings and the journal state in **one** local
|
||||
transaction. Manager mappings exist only after the realm has committed.
|
||||
|
||||
No driver is added; everything goes through the existing `Db` abstraction.
|
||||
|
||||
## The typed encoder (`sqlenc.rs`)
|
||||
|
||||
Untrusted strings and blobs (names, item texts, pet names, settings, macro blobs) can only be `Val::Text` / `Val::Bytes` and
|
||||
are rendered **only** as hex literals (`_utf8mb4 0x...` / `0x...`). Nothing is ever escaped or interpolated: a quote,
|
||||
backslash, NUL, `;` or `--` in any input cannot reach the SQL text. Numbers come from typed integers; the only free-form SQL
|
||||
is `Val::Expr(&'static str)`, i.e. literals in this crate's source. Tests prove this by building the script for a hostile
|
||||
character and a quiet one and checking that the scaffolds are identical once hex literals are removed, and on a real MySQL by
|
||||
importing hostile text and reading it back byte for byte.
|
||||
|
||||
## What the transaction does
|
||||
|
||||
* takes the named lock `coa_portable_import`; asserts the realm is stopped (no online character, no session of the
|
||||
game server's database user), the account exists and is not internal, and has room;
|
||||
* allocates `@char`, `@item_base`, `@pet_base` **inside the transaction from the realm's own tables**: above the highest
|
||||
character guid, above every item guid that *anything* still names (inventory, mail, auction, guild bank, gifts, refund and
|
||||
trade data, bank, loot, ...), above every pet number;
|
||||
* remaps every reference through them: item *i* of the plan is `@item_base + i` in the item row, the inventory row and as a
|
||||
bag's contents' `bag`; pet *i* is `@pet_base + i`; no guid of the source realm or of the snapshot is ever written
|
||||
(crafter guids are zeroed);
|
||||
* reads the **start position from the realm's own `playercreateinfo`** (map, zone, position, homebind);
|
||||
* checks the name inside the transaction (case/accent-insensitively against characters and `reserved_name`); a taken or
|
||||
reserved name becomes `<name><HEX(guid)>` with `AT_LOGIN_RENAME`, as the core's `.pdump load` does;
|
||||
* deletes **leftovers of a deleted character that had this guid** from a curated list of per-character tables (the realm's
|
||||
own delete does not clean module tables);
|
||||
* inserts the character and, for the approved v1 scope, its inventory/items/gifts, spells, skills, glyphs, reputation, quests,
|
||||
action bars, pets (+ spells, declined names), carried settings and macros;
|
||||
* **asserts its own result** (`DO IF(count = expected, 1, <error>)` for every table, every inventory row resolves to an item of
|
||||
this character, every bag to an inventory row of this character); a failed assertion is an SQL error, so there is no
|
||||
`COMMIT` and everything rolls back;
|
||||
* writes the **marker** `character_settings (@char, 'coa.portable.import', '<nonce words> <revision> ')` in the same
|
||||
transaction, then commits and reports `@char`, the bases and the final name.
|
||||
|
||||
A fresh arrival is not online, has the intro marked as seen, no first-login flag, full health, mana and energy (the realm
|
||||
clamps to the maximum).
|
||||
|
||||
### What is not applied
|
||||
|
||||
* **Stock talents (`character_talent`) are never written.** Found with a real worldserver: a row naming a spell that is not a
|
||||
talent of the realm's DBC makes `Player::_LoadTalents` `ASSERT` and **kills the whole worldserver**; nothing in the
|
||||
database can tell whether a spell id is a talent. CoA keeps talents as spells and builds in `character_settings`; real CoA
|
||||
characters have no `character_talent` rows. The rows stay in the snapshot and are reported as not applied.
|
||||
* Settings outside the carry list (policy v2) and all extensions stay in the snapshot only.
|
||||
* Equipment sets, per-character wardrobe, personal bank, collections: deferred by decision.
|
||||
|
||||
## Crash recovery (the gap between the two commits)
|
||||
|
||||
The realm and the local store cannot share a transaction, so the **marker** (inside the realm transaction) decides what
|
||||
happened:
|
||||
|
||||
| Crash point | What exists | Recovery (`recover_imports`) |
|
||||
|---|---|---|
|
||||
| after `begin_import`, before the realm commit | journal `prepared`, no marker | marker absent -> `aborted` (after a grace period and after taking the import lock, so an orphaned client process that is still finishing the import is waited for) |
|
||||
| after the realm commit, before `finish_import` | journal `prepared`, marker present | marker present -> item/pet bases and the guid are read from the realm, counts and contiguity are checked against the journal's plan, then `finish_import` records the mappings -> `committed` |
|
||||
| realm does not match the plan | marker present, rows missing | `needs_attention`; **no mapping is recorded on a guess** |
|
||||
| a lost answer (error after commit) | same as above | `import_character` itself resolves through the marker before reporting |
|
||||
|
||||
`finish_import` is idempotent. The marker stays in the realm after a successful import; the export policy drops
|
||||
`coa.portable.*` so it is never carried.
|
||||
|
||||
## Verified against a real server
|
||||
|
||||
On two disposable MySQL servers (source A with synthetic characters, destination B with the real CoA world data) and a
|
||||
**real `worldserver.exe`** started on B: the six characters were imported while the realm was stopped; the server then started
|
||||
(`Loaded Character Infos For 39 Characters`), four of them were logged in through the real `HandlePlayerLoginFromDB` /
|
||||
`Player::LoadFromDB` (`botcmd spawnbot`), saved, and read back.
|
||||
|
||||
Findings, all expected consequences of the realm's own rules, none of them import bugs:
|
||||
|
||||
* the two characters imported under a temporary name (`<name><HEX>`, rename-at-login) cannot log in without a client: the
|
||||
realm demands the rename first;
|
||||
* on first login the realm **adds** defaults (class spells and skills, taxi mask, default reputations, `core.ascension_*`
|
||||
settings), **resets** daily honor and an unknown chosen title, **removes** what its DBC does not know (action buttons and
|
||||
spells of the synthetic fixture), and **mails** items it cannot place (31 of 60 synthetic items, whose entries do not fit the
|
||||
slots they were put in);
|
||||
* the import marker survives the realm's save.
|
||||
|
||||
These are exactly the changes Phase 4's reconciliation has to treat as the realm's own, not as the player's.
|
||||
|
||||
A real client login (character select, entering the world) was **not** performed: no game client is available to the
|
||||
automated tests.
|
||||
|
||||
## Tests
|
||||
|
||||
* `realm/import_tests.rs` (no database): one transaction, hex-only text, remapping, counts and assertions, settings policy,
|
||||
marker, guards, name logic, orphan cleanup list, refusals, the report parser.
|
||||
* `store` tests: journal begin/finish/abort/flag, idempotence, rollback of a failing local step.
|
||||
* `realm/live_import.rs` (`#[ignore]`, two real MySQL servers): a trip through a second realm for six characters with
|
||||
read-back equality; GUID remapping and orphan cleanup; failure injected in the middle (all tables' row counts unchanged,
|
||||
nothing mapped, retry works); every refusal; crash after the realm commit / before it / lock held / realm tampered;
|
||||
hostile text; the realm's own start data. Steps `server_check_*` drive the real worldserver.
|
||||
@@ -0,0 +1,77 @@
|
||||
# PortableImportService (production importer): design
|
||||
|
||||
Status: **design only, approved, not implemented.** It is a core-side workstream (the fork's `src/server/game/Tools/`, next to
|
||||
`PlayerDump`). The offline importer ([`PORTABLE_IMPORT.md`](PORTABLE_IMPORT.md)) stays the tool for stopped realms and for
|
||||
tests.
|
||||
|
||||
## Why in-core
|
||||
|
||||
A running worldserver owns the id counters, the character name cache and the realm character count in memory
|
||||
(`ObjectMgr::SetHighestGuids`, `CharacterCache`, `World::UpdateRealmCharCount`). Anything written from outside collides with
|
||||
ids the server is about to hand out and is invisible to name lookups until a restart. `PlayerDumpReader::LoadDump`
|
||||
(`PlayerDump.cpp:764`) already solves this inside the server; the service follows its runtime invariants and adds the CoA
|
||||
tables and the portable model.
|
||||
|
||||
## Trigger: an opaque job id, never a payload
|
||||
|
||||
```text
|
||||
Manager worldserver
|
||||
1. writes <JobDir>/<job_id>.job (atomically)
|
||||
2. RA: `.portable import <job_id>` -------------> validates job_id, reads ONLY that file from the fixed JobDir
|
||||
runs the import, writes <JobDir>/<job_id>.result (atomically)
|
||||
3. RA reply: "OK <job_id>" / "ERR <code>" <--------
|
||||
4. reads the .result file, deletes both files
|
||||
```
|
||||
|
||||
* `PortableImport.JobDir` is a worldserver setting (default `<realm>/PortableImport/`). The command takes **only** `job_id`, a
|
||||
UUIDv7 string matched against `^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$`; the file name is
|
||||
built by the service as `<JobDir>/<job_id>.job`. No path, no separator, no payload, no SQL travels through RA, and RA's
|
||||
typed-operation rule in `ra.rs` is kept (one new typed operation, `import_job(job_id)`).
|
||||
* Console only, with its own RBAC permission; one job at a time; the reply never contains character data.
|
||||
* The job file is the snapshot envelope of `PORTABLE_FORMAT.md` (compressed canonical JSON + SHA-256) plus a small header:
|
||||
`{job_id, character_id, revision, account, nonce, expected_content_hash, ruleset}`. Size limits and the decompression cap are
|
||||
enforced **before** parsing; the decoder is strict (unknown fields refused, bounded lists) and validates the same limits as
|
||||
`PortableCharacter::validate`.
|
||||
|
||||
## The runtime invariants (taken from `PlayerDumpReader`)
|
||||
|
||||
All of these happen **inside the server, in one `CharacterDatabaseTransaction`**, on the world thread's DB path:
|
||||
|
||||
1. Refuse when the account does not exist, is a bot/Manager account, is full (`CharactersPerAccount`), or a character with this
|
||||
marker already exists (idempotency, below).
|
||||
2. Allocate ids from the **live generators**: `GetGenerator<HighGuid::Player>().GetNextAfterMaxUsed()`, the item generator,
|
||||
`_hiPetNumber`; remap every reference through those maps (the same remap rules as the offline importer).
|
||||
3. Name check with `normalizePlayerName` / `CheckPlayerName` and the name cache; a taken or invalid name gets the temporary
|
||||
name and `AT_LOGIN_RENAME` (as `.pdump load`).
|
||||
4. Validate against the realm's *own* data, which the offline importer cannot see: `sObjectMgr->GetItemTemplate`, creature
|
||||
templates, `sSpellMgr`, **`sTalentStore` (so stock talents can be validated instead of skipped)**, `playercreateinfo`.
|
||||
5. Write with **prepared statements** (`CharacterDatabasePreparedStatement::SetData`), never string-built SQL; all rows of the
|
||||
character (including the marker `coa.portable.import`) in the one transaction.
|
||||
6. `CharacterDatabase.CommitTransaction(trans)`; **after** a successful commit only:
|
||||
`sCharacterCache->AddCharacterCacheEntry(...)`, advance the generators/`_hiPetNumber` exactly as `LoadDump` does,
|
||||
`sWorld->UpdateRealmCharCount(account)`.
|
||||
7. Never touch a character that is online; never overwrite an existing one (replacing is the separate, backed-up operation of
|
||||
Phase 4).
|
||||
|
||||
## Idempotency and crash recovery
|
||||
|
||||
The marker row is written in the same transaction as the character, with the job's nonce. Re-running the same job id finds the
|
||||
marker, writes the same `.result` and does nothing else. The Manager keeps the same `import_journal` as the offline path, so
|
||||
the recovery table of `PORTABLE_IMPORT.md` applies unchanged; only the "who writes the realm side" part differs.
|
||||
|
||||
## Result file
|
||||
|
||||
`<job_id>.result` (JSON, written to a temp name and renamed): `{status, local_guid, item_base, pet_base, renamed, final_name,
|
||||
counts, not_applied[], warnings[]}` or `{status:"refused", problems[]}` with the same problem codes as the offline preflight. The
|
||||
Manager turns it into `finish_import`.
|
||||
|
||||
## What it must also cover (beyond the offline importer)
|
||||
|
||||
Online accounts (the character list on a connected client is stale until relog), the cache refresh above, per-character module
|
||||
state through the module extension API of Phase 7, collections of the account (Phase 6, keyed by the destination account), and
|
||||
the level-cap projection hooks of Phase 8.
|
||||
|
||||
## Not part of this design
|
||||
|
||||
Exports from a running realm (checkpoints need a per-character save command, `.character save <name>`, tracked separately),
|
||||
a payload over RA, a path argument, a remote caller of any kind.
|
||||
Reference in new issue
Block a user