feat(portable): offline importer into a stopped realm, with journaled recovery (Phase 3)

Imports a canonical snapshot into a stopped disposable realm through the existing
Db abstraction, in one MySQL transaction: guids allocated inside the transaction
from the realm's own tables and every reference remapped, orphan cleanup, name
and start-position handling, self-asserting result, and a marker row. Untrusted
strings and blobs reach SQL only as hex literals from a typed encoder.

Manager mappings are recorded only after the realm commit, in one local
transaction; an import journal (schema 3) plus the marker recover a crash
between the two commits. Preflight refuses wrong ruleset, running realm,
missing account/content and more before anything is written.

Stock character_talent rows are never written: a real worldserver asserts on a
non-talent row. Item identity no longer includes the crafter (v2).

Production import remains the designed in-core PortableImportService
(docs/PORTABLE_IMPORT_SERVICE.md); nothing of it is implemented.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
ДмитрийandClaude Sonnet 5.5 committed 2026-10-06 09:02:21 +05:00
1 parent d971fd5e28
commit e8f99c16b7
21 files changed
+2796 -23

No files matched your search

+6
View File
@@ -642,6 +642,12 @@ D1-D13 were approved as recommended in section 15, with these amendments:
| D13 checkpoints | keyed **internally by the local guid**: checkpoints arrive as `(server_id, local_guid)` and are resolved to the portable character through `Store::find_by_local` |
| item mapping | `UNIQUE(character_id, server_id, local_item_guid)` in addition to the primary key (implemented in Phase 1) |
**Phase 2.1 / Phase 3 amendments (2026-10-06):** `character_settings` carry list tightened to ten strict key patterns, everything
else quarantined (`PORTABLE_EXPORT.md`); Phase 3 is an offline importer into a stopped disposable realm that keeps the existing
`Db` abstraction, writes untrusted text only as hex literals from a typed encoder, imports in one MySQL transaction with
journaled crash recovery (`PORTABLE_IMPORT.md`); the production importer is an in-core `PortableImportService` triggered over RA by
an opaque job id (`PORTABLE_IMPORT_SERVICE.md`, design only).
---------------------------------------------------------------------------------------------------------
## Appendix A. Phase 1 - exact proposed plan (not started)
+2 -1
View File
@@ -76,7 +76,7 @@ Account-wide sets of numeric ids (`coa:wardrobe`, `coa:vanity`, ...), stored per
## Store (`%LOCALAPPDATA%\CoAServerManager\portable\portable.db`)
SQLite, WAL, `synchronous=FULL`, `foreign_keys=ON`, `application_id = 0x434F4150`, forward-only migrations tracked in
`PRAGMA user_version` (currently 2). A database from a newer Manager, or a foreign SQLite file, is refused and left untouched.
`PRAGMA user_version` (currently 3). A database from a newer Manager, or a foreign SQLite file, is refused and left untouched.
| Table | Purpose |
|---|---|
@@ -85,6 +85,7 @@ SQLite, WAL, `synchronous=FULL`, `foreign_keys=ON`, `application_id = 0x434F4150
| `snapshot` | `(character_id, revision)` -> compressed canonical payload + hash |
| `character_server_mapping` | `(character_id, server_id)` -> local guid; `UNIQUE(server_id, local_guid)` |
| `item_mapping` | one row per mapping with a lifecycle (schema 2, see below) |
| `import_journal` | schema 3: one row per import into a realm (`prepared` / `committed` / `aborted` / `needs_attention`) with the nonce/marker and the plan; see `PORTABLE_IMPORT.md` |
| `collection` | per profile and kind: revision, hash, encoded set |
| `setting` | `history_keep` |
+120
View File
@@ -0,0 +1,120 @@
# Offline import into a stopped realm (Phase 3)
Code: `crates/coa-core/src/portable/realm/{sqlenc,plan,import}.rs`, journal in `portable/store/journal.rs`.
This is the **offline** importer: it writes a canonical snapshot into a **stopped** realm's characters database. It proves
the model end to end and serves realms that are stopped. It is **not** the production path for a running public host:
that is the in-core `PortableImportService` ([`PORTABLE_IMPORT_SERVICE.md`](PORTABLE_IMPORT_SERVICE.md)), which is designed but
not implemented.
## Flow
```text
preflight (read-only snapshot) -> journal "prepared" -> ONE realm transaction -> mappings + journal "committed"
(marker inside it) (ONE local transaction)
```
1. **Preflight** (`preflight`): a read-only consistent snapshot of the realm. Refuses (`ImportRefused`, nothing written,
no journal entry) when the character belongs to another ruleset (CoA and Wildcard never mix), an id is of a namespace
the importer does not apply, the realm is running (a session of the game server's database user exists), characters are
marked online, the account does not exist / is a bot or Manager account / is full, the realm has no start position for the
race and class, or `item_template` / `creature_template` do not know an entry the character carries (the realm would
delete such items on load).
2. **Journal**: `begin_import` writes an `import_journal` row `prepared` (nonce, the plan: portable ids, entries,
identities) *before* anything happens in the realm.
3. **Realm transaction**: one `mysql` session, one `START TRANSACTION ... COMMIT`, generated by `plan.rs`.
4. **Local commit**: `finish_import` records the realm binding, the item mappings and the journal state in **one** local
transaction. Manager mappings exist only after the realm has committed.
No driver is added; everything goes through the existing `Db` abstraction.
## The typed encoder (`sqlenc.rs`)
Untrusted strings and blobs (names, item texts, pet names, settings, macro blobs) can only be `Val::Text` / `Val::Bytes` and
are rendered **only** as hex literals (`_utf8mb4 0x...` / `0x...`). Nothing is ever escaped or interpolated: a quote,
backslash, NUL, `;` or `--` in any input cannot reach the SQL text. Numbers come from typed integers; the only free-form SQL
is `Val::Expr(&'static str)`, i.e. literals in this crate's source. Tests prove this by building the script for a hostile
character and a quiet one and checking that the scaffolds are identical once hex literals are removed, and on a real MySQL by
importing hostile text and reading it back byte for byte.
## What the transaction does
* takes the named lock `coa_portable_import`; asserts the realm is stopped (no online character, no session of the
game server's database user), the account exists and is not internal, and has room;
* allocates `@char`, `@item_base`, `@pet_base` **inside the transaction from the realm's own tables**: above the highest
character guid, above every item guid that *anything* still names (inventory, mail, auction, guild bank, gifts, refund and
trade data, bank, loot, ...), above every pet number;
* remaps every reference through them: item *i* of the plan is `@item_base + i` in the item row, the inventory row and as a
bag's contents' `bag`; pet *i* is `@pet_base + i`; no guid of the source realm or of the snapshot is ever written
(crafter guids are zeroed);
* reads the **start position from the realm's own `playercreateinfo`** (map, zone, position, homebind);
* checks the name inside the transaction (case/accent-insensitively against characters and `reserved_name`); a taken or
reserved name becomes `<name><HEX(guid)>` with `AT_LOGIN_RENAME`, as the core's `.pdump load` does;
* deletes **leftovers of a deleted character that had this guid** from a curated list of per-character tables (the realm's
own delete does not clean module tables);
* inserts the character and, for the approved v1 scope, its inventory/items/gifts, spells, skills, glyphs, reputation, quests,
action bars, pets (+ spells, declined names), carried settings and macros;
* **asserts its own result** (`DO IF(count = expected, 1, <error>)` for every table, every inventory row resolves to an item of
this character, every bag to an inventory row of this character); a failed assertion is an SQL error, so there is no
`COMMIT` and everything rolls back;
* writes the **marker** `character_settings (@char, 'coa.portable.import', '<nonce words> <revision> ')` in the same
transaction, then commits and reports `@char`, the bases and the final name.
A fresh arrival is not online, has the intro marked as seen, no first-login flag, full health, mana and energy (the realm
clamps to the maximum).
### What is not applied
* **Stock talents (`character_talent`) are never written.** Found with a real worldserver: a row naming a spell that is not a
talent of the realm's DBC makes `Player::_LoadTalents` `ASSERT` and **kills the whole worldserver**; nothing in the
database can tell whether a spell id is a talent. CoA keeps talents as spells and builds in `character_settings`; real CoA
characters have no `character_talent` rows. The rows stay in the snapshot and are reported as not applied.
* Settings outside the carry list (policy v2) and all extensions stay in the snapshot only.
* Equipment sets, per-character wardrobe, personal bank, collections: deferred by decision.
## Crash recovery (the gap between the two commits)
The realm and the local store cannot share a transaction, so the **marker** (inside the realm transaction) decides what
happened:
| Crash point | What exists | Recovery (`recover_imports`) |
|---|---|---|
| after `begin_import`, before the realm commit | journal `prepared`, no marker | marker absent -> `aborted` (after a grace period and after taking the import lock, so an orphaned client process that is still finishing the import is waited for) |
| after the realm commit, before `finish_import` | journal `prepared`, marker present | marker present -> item/pet bases and the guid are read from the realm, counts and contiguity are checked against the journal's plan, then `finish_import` records the mappings -> `committed` |
| realm does not match the plan | marker present, rows missing | `needs_attention`; **no mapping is recorded on a guess** |
| a lost answer (error after commit) | same as above | `import_character` itself resolves through the marker before reporting |
`finish_import` is idempotent. The marker stays in the realm after a successful import; the export policy drops
`coa.portable.*` so it is never carried.
## Verified against a real server
On two disposable MySQL servers (source A with synthetic characters, destination B with the real CoA world data) and a
**real `worldserver.exe`** started on B: the six characters were imported while the realm was stopped; the server then started
(`Loaded Character Infos For 39 Characters`), four of them were logged in through the real `HandlePlayerLoginFromDB` /
`Player::LoadFromDB` (`botcmd spawnbot`), saved, and read back.
Findings, all expected consequences of the realm's own rules, none of them import bugs:
* the two characters imported under a temporary name (`<name><HEX>`, rename-at-login) cannot log in without a client: the
realm demands the rename first;
* on first login the realm **adds** defaults (class spells and skills, taxi mask, default reputations, `core.ascension_*`
settings), **resets** daily honor and an unknown chosen title, **removes** what its DBC does not know (action buttons and
spells of the synthetic fixture), and **mails** items it cannot place (31 of 60 synthetic items, whose entries do not fit the
slots they were put in);
* the import marker survives the realm's save.
These are exactly the changes Phase 4's reconciliation has to treat as the realm's own, not as the player's.
A real client login (character select, entering the world) was **not** performed: no game client is available to the
automated tests.
## Tests
* `realm/import_tests.rs` (no database): one transaction, hex-only text, remapping, counts and assertions, settings policy,
marker, guards, name logic, orphan cleanup list, refusals, the report parser.
* `store` tests: journal begin/finish/abort/flag, idempotence, rollback of a failing local step.
* `realm/live_import.rs` (`#[ignore]`, two real MySQL servers): a trip through a second realm for six characters with
read-back equality; GUID remapping and orphan cleanup; failure injected in the middle (all tables' row counts unchanged,
nothing mapped, retry works); every refusal; crash after the realm commit / before it / lock held / realm tampered;
hostile text; the realm's own start data. Steps `server_check_*` drive the real worldserver.
+77
View File
@@ -0,0 +1,77 @@
# PortableImportService (production importer): design
Status: **design only, approved, not implemented.** It is a core-side workstream (the fork's `src/server/game/Tools/`, next to
`PlayerDump`). The offline importer ([`PORTABLE_IMPORT.md`](PORTABLE_IMPORT.md)) stays the tool for stopped realms and for
tests.
## Why in-core
A running worldserver owns the id counters, the character name cache and the realm character count in memory
(`ObjectMgr::SetHighestGuids`, `CharacterCache`, `World::UpdateRealmCharCount`). Anything written from outside collides with
ids the server is about to hand out and is invisible to name lookups until a restart. `PlayerDumpReader::LoadDump`
(`PlayerDump.cpp:764`) already solves this inside the server; the service follows its runtime invariants and adds the CoA
tables and the portable model.
## Trigger: an opaque job id, never a payload
```text
Manager worldserver
1. writes <JobDir>/<job_id>.job (atomically)
2. RA: `.portable import <job_id>` -------------> validates job_id, reads ONLY that file from the fixed JobDir
runs the import, writes <JobDir>/<job_id>.result (atomically)
3. RA reply: "OK <job_id>" / "ERR <code>" <--------
4. reads the .result file, deletes both files
```
* `PortableImport.JobDir` is a worldserver setting (default `<realm>/PortableImport/`). The command takes **only** `job_id`, a
UUIDv7 string matched against `^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$`; the file name is
built by the service as `<JobDir>/<job_id>.job`. No path, no separator, no payload, no SQL travels through RA, and RA's
typed-operation rule in `ra.rs` is kept (one new typed operation, `import_job(job_id)`).
* Console only, with its own RBAC permission; one job at a time; the reply never contains character data.
* The job file is the snapshot envelope of `PORTABLE_FORMAT.md` (compressed canonical JSON + SHA-256) plus a small header:
`{job_id, character_id, revision, account, nonce, expected_content_hash, ruleset}`. Size limits and the decompression cap are
enforced **before** parsing; the decoder is strict (unknown fields refused, bounded lists) and validates the same limits as
`PortableCharacter::validate`.
## The runtime invariants (taken from `PlayerDumpReader`)
All of these happen **inside the server, in one `CharacterDatabaseTransaction`**, on the world thread's DB path:
1. Refuse when the account does not exist, is a bot/Manager account, is full (`CharactersPerAccount`), or a character with this
marker already exists (idempotency, below).
2. Allocate ids from the **live generators**: `GetGenerator<HighGuid::Player>().GetNextAfterMaxUsed()`, the item generator,
`_hiPetNumber`; remap every reference through those maps (the same remap rules as the offline importer).
3. Name check with `normalizePlayerName` / `CheckPlayerName` and the name cache; a taken or invalid name gets the temporary
name and `AT_LOGIN_RENAME` (as `.pdump load`).
4. Validate against the realm's *own* data, which the offline importer cannot see: `sObjectMgr->GetItemTemplate`, creature
templates, `sSpellMgr`, **`sTalentStore` (so stock talents can be validated instead of skipped)**, `playercreateinfo`.
5. Write with **prepared statements** (`CharacterDatabasePreparedStatement::SetData`), never string-built SQL; all rows of the
character (including the marker `coa.portable.import`) in the one transaction.
6. `CharacterDatabase.CommitTransaction(trans)`; **after** a successful commit only:
`sCharacterCache->AddCharacterCacheEntry(...)`, advance the generators/`_hiPetNumber` exactly as `LoadDump` does,
`sWorld->UpdateRealmCharCount(account)`.
7. Never touch a character that is online; never overwrite an existing one (replacing is the separate, backed-up operation of
Phase 4).
## Idempotency and crash recovery
The marker row is written in the same transaction as the character, with the job's nonce. Re-running the same job id finds the
marker, writes the same `.result` and does nothing else. The Manager keeps the same `import_journal` as the offline path, so
the recovery table of `PORTABLE_IMPORT.md` applies unchanged; only the "who writes the realm side" part differs.
## Result file
`<job_id>.result` (JSON, written to a temp name and renamed): `{status, local_guid, item_base, pet_base, renamed, final_name,
counts, not_applied[], warnings[]}` or `{status:"refused", problems[]}` with the same problem codes as the offline preflight. The
Manager turns it into `finish_import`.
## What it must also cover (beyond the offline importer)
Online accounts (the character list on a connected client is stale until relog), the cache refresh above, per-character module
state through the module extension API of Phase 7, collections of the account (Phase 6, keyed by the destination account), and
the level-cap projection hooks of Phase 8.
## Not part of this design
Exports from a running realm (checkpoints need a per-character save command, `.character save <name>`, tracked separately),
a payload over RA, a path argument, a remote caller of any kind.