diff --git a/Cargo.lock b/Cargo.lock index 58c533363e..811ba2040b 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -6846,13 +6846,16 @@ dependencies = [ name = "wordcraft-cli" version = "0.15.1" dependencies = [ + "image", "serde_json", "sys-locale", "wordcraft-doc", "wordcraft-engine", + "wordcraft-fonts", "wordcraft-mcp", "wordcraft-render", "wordcraft-zotero", + "zip 3.0.0", ] [[package]] diff --git a/GLOSSARY.md b/GLOSSARY.md index e17f1e76d6..9d1ec531ed 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -48,3 +48,21 @@ _Avoid_: User dictionary (in the interface) A folder of Hunspell dictionaries you supply, used before the system's (an installed Language pack comes after it). _Avoid_: Custom dictionaries + +## Export + +**Strict export**: +An export that writes nothing, and reports why, when a character or picture the output is meant +to show would be missing from it. Text the output doesn't show (hidden text, deletions without +markup) isn't checked. +_Avoid_: Safe mode, validated export + +**Missing glyph**: +A character the output draws that no face in the font fallback chain can draw, so it would appear +as an empty box. +_Avoid_: Tofu, box, notdef character + +**Missing picture**: +A picture the document refers to that the output cannot include: its part is absent, its data +cannot be decoded, its format is unsupported, or it is linked from outside the file. +_Avoid_: Broken image, dropped image diff --git a/README.md b/README.md index ba5081af35..66a7b7776a 100644 --- a/README.md +++ b/README.md @@ -157,6 +157,7 @@ Command line: ```sh wordcraft-cli convert report.docx report.pdf # docx, pdf, odt, rtf, html, md, tex, txt, png +wordcraft-cli convert report.docx report.pdf --strict # exit 3, no file, if a character or picture would be left out wordcraft-cli text report.docx # plain text wordcraft-cli inspect report.docx # structure as JSON wordcraft-cli run --template sample \ @@ -164,6 +165,40 @@ wordcraft-cli run --template sample \ wordcraft-cli run --script job.json # the same from a file: commands, then save to docx and pdf ``` +`convert --strict` (a `.pdf` or `.png` output only) and `render --strict` (a PNG of one page) refuse a silently incomplete render: they exit 3 +and write nothing if the output would draw a character no font has (an empty box) or leave out a +picture. Exit codes are 0 (written), 1 (unreadable input, usage error, or `--strict` with another +output type, checked before the input is read) and 3 (strict refusal). On exit 3, stderr has one +`wordcraft-cli: strict: …` line per kind, then one JSON line: + +```sh +wordcraft-cli convert in.docx out.pdf --strict +# wordcraft-cli: strict: 1 character cannot be drawn: U+2D30 (×3) +# wordcraft-cli: strict: 1 picture(s) left out: word/media/image2.bin (header2.xml): unsupported format +# {"schema":1,"strict":"failed","missing_glyphs":[{"cp":"U+2D30","count":3}],"missing_pictures":[{"part":"word/media/image2.bin","source_part":"word/header2.xml","reason":"unsupported_format"}],"truncated":false,"total_missing_glyphs":1,"total_missing_pictures":1,"glyphs_truncated":false,"pictures_truncated":false} +``` + +The JSON line (schema 1, also what `file.strictReport` returns) has `schema`, `strict` +(`"passed"` or `"failed"`), `missing_glyphs` (entries `{"cp", "count"}`, sorted by code point), +`missing_pictures` (entries `{"part", "source_part", "reason"}`: the picture's part or target, the +story part it sits in such as `word/header2.xml`, and a reason code; `source_part` is also +`background` (the page picture), `body`, `header`, `footer`, `footnote`, `endnote`, `comment` or +`text box` where the part name is not known (a document not opened from a DOCX), or a chunk path such as +`word/afchunk1.docx > word/header1.xml` for a picture inside an embedded DOCX chunk), `truncated`, the two totals +`total_missing_glyphs` and `total_missing_pictures`, and `glyphs_truncated` and +`pictures_truncated` for each list. The reason codes are `missing_part`, `external_target`, +`link_only`, `undecodable` and `unsupported_format`; the stderr lines use plain words for them. When a list is cut, the stderr line gives the total +(`300 characters cannot be drawn (first 256 listed)`). +Each list stops at 256 entries separately; `truncated` is true when either was cut, and the totals +count every distinct character and picture found, those past the cap included. Only characters the output draws count (not hidden text, or deleted text +without `--markup`), and a character a fallback font draws is fine. Text inside a WMF or EMF picture counts too, shaped in the font the picture names with no fallback. A Private Use Area character +fails only if no installed font maps it. The result depends on the fonts installed on the host +(see #5624 for a fixed font set). An external picture target is never fetched: it is reported, not +loaded. Pictures dropped when a DOCX is opened (a missing part, an external target, or a `link only` picture with no embedded copy) are reported for DOCX input only. They have no page, so a PNG of any page is also refused for a missing or external picture anywhere in the document, unused headers included. `render --strict` checks only the page it +writes (`--page`; `--scale` doesn't matter), prints nothing on stdout when it refuses, and reports the +same way. The engine takes `strict: true` on `file.save`, `file.saveAs`, `file.exportPdf` and +`file.exportPng`, and `file.strictReport {markup?, page?}` returns the report without writing. + Web: `cd apps/wordcraft-web && trunk serve`, then open . ## For agents: CLI and MCP diff --git a/ROADMAP.md b/ROADMAP.md index ed2884d695..d4c951f995 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -8,7 +8,7 @@ **Stage: alpha** · next: beta, ~15 points (60% → 75% ready for real work) and ~150–230 h away -> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-11-08 · **Change:** trivial (progress log: ODT columns stating only their shares); previously trivial (progress log: ODT font size for every script); previously trivial (progress log: ODT text turning off what its style turns on); previously trivial (progress log: edited ODT paragraphs keep custom styles with an outline level); previously trivial (progress log: ODT pictures keep the size their file gives); previously trivial (progress log: ODT frames keep the page they are anchored to); previously trivial (pointer zoom stretches page rasters until it stops); previously trivial (dark-theme interface text on Linux no longer looks semi-bold); previously trivial (Control/MCP #2596 review: partial-write outcomes, path-bearing image exports, accept-thread startup cleanup and malformed UTF-8 handling); previously trivial (progress log: a turned inline picture whose file keeps no room for the turn keeps its line to its frame); previously trivial (progress log: the Arabic interface's Styles gallery reads from the right (#5566)); previously trivial (progress log: ODT pictures offered in several formats); previously trivial (progress log: ODT heading text of another size); previously trivial (Protected typing boundary lookup counts preceding objects once and indexes editable permission IDs (#3614); caret movement and enforcement policy are unchanged; previously trivial: progress log: Custom document properties dialog, #2982; previously trivial: progress log: View › New Window; previously trivial: progress log: interface translations come from storytold/craft-i18n through the `craft-i18n` crate; previously trivial: progress log: disabled-control reasons translated in every interface language, #2678; previously trivial: progress log: mail merge record navigation and record ranges for printing, #3028; previously trivial: selection bar for right-to-left paragraphs and table rows; previously trivial: group connector rerouting preserves rotated and flipped free or unresolved endpoints under nonuniform group scaling; readiness estimates unchanged; previously trivial: progress log: a word with no break opportunity in it no longer makes the layout quadratic, #3412; previously trivial: progress log: a relationship the markup names without the prefix is read as it wrote, and an ordinary `id` names no part; previously trivial: progress log: a document with many relationships opens in the time its part takes, and an id the text only mentions is not a reference, #3411; previously trivial: progress log: read only and comments-only protection grey out edits; previously trivial: progress log: an empty paragraph with a hidden mark shows nothing, not even its list number, #3982; previously trivial: progress log: "NaN" in the Font dialog's Size box no longer closes the app, #3663; previously trivial: progress log: a table cell span from a `.json` file no longer overflows, #3637; previously trivial: progress log: a click in the footer while editing the header moves there; previously trivial: explicit documents keep the editor when recovery discovery finishes; readable recovery controls; previously trivial: Slovenian interface catalog, 2,614 entries; previously trivial: Insert Table of Figures dialog for any caption label, #3013; previously trivial: progress log: Home › Underline drop-down, #2738; previously trivial: personal templates in File › New and Save as Word Template, #2852; previously trivial: progress log: the scrollbar page tip stays on one line, #3341; previously trivial: progress log: forms protection greys out the control buttons; previously trivial: progress log: a document with many content controls saves in the time the controls take, #13; previously trivial: progress log: Word signature lines kept on save; previously trivial: a Thai run with Latin finds one installed font covering it all on Linux too, #1549; previously trivial: progress log: closing the header returns to the body selection, #3331; previously trivial: progress log: saving to a NAS share mounted by GVFS, #3003; previously trivial: progress log: a picture dragged near the bottom of a page lands there, #3169; previously trivial: centred list labels after leading page breaks, #2964; previously trivial: progress log: inline pictures drag to an insertion cursor and stay inline, #2233, #2234; previously trivial: progress log: a text file saved over itself closes without asking, #2376; previously trivial: progress log: a text frame in the paragraph defaults frames nothing, #2360; previously trivial: fixed font inputs in four layout controls; previously trivial: typing no longer starts render threads on every repaint, #3027; previously trivial: a right, centre or decimal tab whose text doesn't fit, or a left tab past the right indent, goes to the next line with its text, #1840; previously trivial: progress log: banded rows shade as in Word, #3415; previously trivial: progress log: the table handles show while the caret is in the table, #2428; previously trivial: progress log: a click below a floating table reaches the text after it, #2038; previously trivial: progress log: Delete on selected table cells leaves one empty paragraph in each, #2802; previously trivial: progress log: lam-alef in exported PDFs extracts whole with fonts that draw it as two glyphs, #2310; previously trivial: Bullets and Numbering toggle off style-inherited lists at their effective level, including basedOn chains and numbered ListParagraph styles; other paragraphs and list definitions stay unchanged (related to #3459); previously trivial (progress log: a shape dropped just after a page break inside a paragraph stays on the new page, #4262); previously trivial (progress log: a float anchored on a later page of a long paragraph is drawn there, #4250); previously minor (progress log: `file.closeDocument` over the control channel and MCP bridge is refused while the document it names has unsaved changes; previously trivial (progress log: ODT element audit, ODT checked against 220 files from other applications, ODT display styles); previously trivial (progress log: Shape Format › Size in the interface unit, #4196); previously trivial (progress log: typed n-ary operators such as ∫ and ∑ wait for their limits, observed in Word for Windows on #4938 (#4917)); previously trivial (progress log: a paragraph naming an undefined style is formatted as Normal, #5077); previously trivial (progress log: justified and distributed lines keep the width of the spaces before their first character); previously trivial (print helpers erase loaded plaintext inputs before driver work, #5710); previously trivial (Windows print submissions use isolated helpers so a cancelled driver cannot block a retry, #5710); previously trivial (print cancellation remains available without a document; sending-boundary limitation clarified, #5710); previously trivial (progress log: a print job waiting on its driver can be cancelled, and quitting doesn't wait for it, #5710); previously trivial (progress log: the Print panel's printer list no longer waits on printer drivers, #5710); previously trivial (Page drawing indexes text-box warp lookups per paragraph (#3614)); previously trivial (built-in citation source lookups, #4888; current-main progress records retained); previously trivial (progress log: Times and Courier take their Windows counterparts' line spacing; previously trivial (progress log: the status bar's count can be words, characters or characters without spaces, #2797); previously trivial (Linux display server setting in File › Options › Advanced, #2184); previously trivial (progress log: Linux Read Aloud through Speech Dispatcher, #4576); previously trivial (progress log: a click just outside a floating table by turned cell text, #4181; previously minor: progress log: Spanish proofing, #3360, #3553; previously trivial: progress log: Insert › Table's grid previews the hovered table on the page, #3931; previously trivial (progress log: MHT quoted MIME parameters); previously trivial (progress log: CLI convert format options); previously trivial (MATH construction reuse, #3630); previously trivial (DOCX `w:adjustRightInd` is preserved through style resolution and ordered DOCX saves); previously trivial (per-page ink object lookups, #3614); previously trivial (progress log: ODT display settings removed); previously trivial (progress log: dialogs are modal except the five Word keeps modeless, and a press beside one shakes it, #3859); previously trivial (progress log: picture brightness, contrast and transparency, ODT element audit, ODT checked against 220 files from other applications, ODT display styles); previously trivial (progress log: the dialogs of the mirrored Arabic interface read right to left: label-and-field rows, tables (list columns, the Models table), tile galleries and the Add/Remove arrows follow the reading direction, through one `rtl::Table` helper that draws exactly as before in English; no ranked gap closed, readiness estimates unchanged); previously trivial (progress log: fonts that aren't installed are flagged in the font box and status bar); previously trivial (progress log: ODT sections inside text boxes); previously trivial (progress log: ODT definitions sharing a name, sections around tables, stacking order); previously trivial (progress log: ODT forms and indexes written as read); previously trivial (progress log: ODT table styles saved, WordCraft-origin documents checked in LibreOffice); previously trivial (progress log: ODT interpretation profile); previously trivial (progress log: ODT controls, custom shapes, in-line placement); previously trivial (progress log: ODT charts kept, frame outlines, header height); previously trivial (progress log: ODT header rows); previously trivial (progress log: ODT layout setting, padding, columns); previously trivial (progress log: ODT tables, header spacing, picture paragraphs); previously trivial (progress log: ODT round trips keep their page count); previously trivial (progress log: emphasis marks, SET fields, ODT page backgrounds and tables); previously trivial (progress log: ODT checked against LibreOffice's templates); previously trivial (progress log: ODT at ~90%); previously trivial (progress log: ODT read and written nearly whole); previously trivial (progress log: ODT text boxes); previously trivial (progress log: ODT layout settings); previously trivial (progress log: ODT character styles); previously trivial (progress log: ODT custom styles); previously trivial (progress log: ODT equations saved); previously trivial (progress log: ODT floating pictures); previously trivial (progress log: ODT paragraph borders); previously trivial (progress log: ODT heading numbering); previously trivial (progress log: ODT table borders); previously trivial (progress log: ODT heading styles); previously trivial (progress log: ODT tab stops); previously trivial (progress log: ODT character formatting); previously trivial (progress log: ODT lists); previously trivial (progress log: ODT comments); previously trivial (progress log: ODT footnotes and endnotes); previously trivial (progress log: ODT headers, footers and fields); previously trivial (progress log: ODT paragraph spacing); previously trivial (progress log: Options › Language redesign, CP4.3); previously trivial (progress log: the missing-pack bar on a fresh install, the caret's typing language, and the spelling menu heading in the interface language, CP4.3); previously trivial (progress log: Language pack languages in every language list, CP4.3); previously trivial (progress log: craft-proof pin, French elision and Valencian, CP4.3); previously trivial (progress log: a long comment's card and balloon stop at ten lines, #3382); previously trivial (progress log: a pasted text box story is charged for the formatting it copies); previously trivial (progress log: a .doc table row keeps one override per cell); previously trivial (progress log: a .doc file's sections find their paragraphs by index); previously minor (Arabic interface language with mirrored UI, #1195); previously trivial (progress log: Replace and Remove Table of Contents with Word's title above a section break, #4041); previously trivial (progress log: a password-protected file named on the command line asks for its password after the startup document is replaced, #5299); previously trivial (progress log: language-packs-2 published and pinned, 75 Language packs); previously trivial (progress log: language-packs-1 published and pinned, CP4.4); previously trivial (progress log: Language packs, CP4);; previously trivial (DOCX table rows and cells inside selected alternate-content branches survive open and repeated saves (#5219); broader corpus qualification and readiness estimates unchanged); previously trivial (progress log: a list inside a Markdown block quote stays quoted); previously trivial (progress log: arrowheads keep their kind, width and length and are drawn at Word's sizes); previously trivial (progress log: the line between columns for selected text, #5178); previously trivial (progress log: .doc standard font, #5066); previously trivial (progress log: a table row doesn't split off a picture that doesn't fit, #4272); previously trivial (progress log: a middle or bottom cell of a row split across pages lines up on the first page); previously trivial (progress log: tab leader dots spaced and placed as in Word); previously trivial (progress log: a list level above counts its first number at the first deeper item); previously trivial (progress log: paragraph side borders lie outside the text, #5036); previously trivial (progress log: TOC page numbers as PAGEREF fields, #5097); previously trivial (progress log: Backstage keytips, #4960); previously trivial (Progress log: only the printer's PDF leaves out links; Save these pages as PDF keeps them, #4734); previously trivial (progress log: Chinese and Japanese text counts a word per character, #4547; previously trivial: aligned-copy review fixes retain the document axis through scrolling and include secondary selections (#4764)); previously trivial (connector rerouting indexes laid-out object positions once per batch (#3631); previously trivial (progress log: corporate authors in bibliographies, #4848); previously trivial (custom-property bookmark-link preservation, #4767); previously trivial (progress log: Table Properties follows the selection, #4857); previously trivial (Relative font-size commands preserve mixed sizes in the selection (#4903); existing button steps and one-point shortcuts are unchanged; readiness estimates unchanged); previously trivial (Arabic Backspace removes one combining mark and keeps its base letter (#4757); previously trivial (note conversion walks reference offsets and formatting runs together, #4750); previously trivial (DOCX comment-ID save lookups); previously trivial (progress log: RTF pictures stored as Windows or Enhanced Metafiles are imported, #4259); previously trivial (#4658 bounds nested Markdown reference-label work); previously trivial (HTML import keeps preformatted lines with CR or CRLF input newlines, #4810); previously trivial (progress log: Insert › Shapes icon grid, #4349); previously trivial (progress log: AutoCorrect Options with no document open, #4585); previously trivial (progress log: window fallback geometry and equation symbol tooltips); previously trivial (progress log: window titles omit NUL characters, #3634); previously trivial (Recipient labels continue across product-sized sheets instead of stopping after the first sheet (#4880)); previously trivial (progress log: Custom document properties dialog, #2982; previously trivial: Russian sorting keeps ё with е and recognises decomposed ё/й; previously trivial: progress log: a text box that keeps the keyboard no longer resets the input method every frame, #2986; previously trivial: progress log: recovery copies and .json files store pictures as base64, #586 follow-up; previously trivial: progress log: legacy menu keys, #1697; previously trivial: progress log: shape gradients and transparency, #1500; previously trivial: progress log: Focus fills the screen, and Full Screen View shows the pages alone, #1678; previously trivial: progress log: text above a float's anchor wraps around it, #2048; previously trivial: progress log: File › Recover follows the app theme, #3243; previously trivial: progress log: formatting toolbar with the text context menu, #2959; previously trivial: progress log: caret inside ligatures, #1974; previously trivial: text typed into a content control takes the control's formatting; localized placeholder styles recognised; previously trivial: forms protection keeps its content controls: no inserting, Properties or Remove Content Control; previously trivial: long navigation headings wrap onto further lines within the pane, #3015; previously trivial: clearing a no-match font query keeps the recent heading visible, #2068; previously trivial: a word after a slash wraps whole, like Word, #2963; previously trivial: progress log: a header-cell picture watermark sits behind the text, #1775; previously trivial: symbols keep their font when the text font changes, #2036; previously trivial: opening a DOCX no longer copies its pictures; previously trivial: Add Font imports every selected font file, #1789); previously trivial (RTF export writes sections (gap 19); previously trivial: ODT export writes sections (gap 19); previously trivial: Read Aloud speaks each sentence in its own language, #1843; previously trivial: hint-reliant CJK outlines; previously trivial: Ctrl+wheel zoom no longer scrolls the document, #1743; previously trivial: progress log: Turkish case mapping in AutoCorrect, #1923; previously trivial: a mouse drag on a table leaves the view where it is, #1738; previously trivial: progress log: Hebrew field results and smart quotes, #1670; previously trivial: Backspace removes a selected table, rows or columns, #1844; previously trivial: progress log: mixed Hebrew/Latin text keeps its order in Word, #1543; previously trivial: progress log: web typing in non-Latin text no longer repeats, #1480; previously trivial: progress log: font family caches bounded and math font lookups follow font changes, #1354; previously trivial: `-->` AutoCorrect replacement, #2223; previously trivial: numbered headings, #1793; previously trivial: Navigation Find exposes Results, #2096; previously trivial: progress log: Columns on selected text, #2024; previously trivial: progress log: legacy outline, shadow, emboss and imprint text effects drawn, #2534; previously trivial: progress log: Indonesian proofing, #498; previously trivial: progress log: DOCX glossary documents (building blocks); previously trivial: progress log: Read Mode keeps the keyboard and shows whole, unobscured columns, #1772); previously trivial: progress log: font list entries of tall-ascent fonts no longer cut off, #2570); previously trivial: progress log: Split Cells and Convert Text to Table dialogs follow the Word check, #2592; previously trivial: a new source's Tag is made like Word's, three letters of the author and two digits of the year with a number added when taken, and an edit keeps the Tag, #2094; previously trivial: Web Layout fills the window width, #1876; previously trivial: progress log: Different First Page headers reachable from page 1, #2826; previously trivial: Language dialog keeps inherited proofing language and no-proof choices (#2717); previously trivial: progress log: web preferences kept between visits, #2132; previously trivial: progress log: undrawable private-use list bullets show the level's default bullet, #1683; previously trivial: progress log: `w:eastAsianLayout` kept on save; previously trivial: table style editor formats every part, #714); previously trivial: VML freeforms read and kept on save; previously trivial: progress log: consistent preferred cell widths; previously trivial: multi-paragraph field instructions no longer add blank lines; previously trivial: page-break-only list paragraphs hide their labels; previously trivial: progress log: grouped `flowChartOr` shapes draw their circle and crossing spokes; previously trivial: progress log: copies keep their formatting in other programs, #1523; previously trivial: progress log: privacy options on save, #1062; previously trivial: progress log: equation pictures in theme colours on Windows, #1002; previously trivial: symbol fonts, #776; previously trivial: the save prompt answers from the keyboard, #861; previously trivial: table column and row markers on the rulers, #700; previously trivial: table borders and the resize handle drag on the page, #217; previously trivial: formatting revisions landed, #41; previously trivial: Draw tab ink, #307, and Draw Table and Eraser, #303, landed: catalog 402/434; previously minor: readiness table with hours per audience; full number back to the additive weighted sum, 60%; mainstream and essentials numbers added; alpha gate checked; previously major: full re-measure against Word for Mac 16.113.4; progress docs restructured); previously trivial (progress log: ODT documents keep their language); previously trivial (progress log: ODT table and row backgrounds) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4 +> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-11-08 · **Change:** trivial (progress log: `convert --strict` and `render --strict` refuse a silently incomplete PDF or PNG (#6005, #6008)); ; trivial (progress log: ODT columns stating only their shares); previously trivial (progress log: ODT font size for every script); previously trivial (progress log: ODT text turning off what its style turns on); previously trivial (progress log: edited ODT paragraphs keep custom styles with an outline level); previously trivial (progress log: ODT pictures keep the size their file gives); previously trivial (progress log: ODT frames keep the page they are anchored to); previously trivial (pointer zoom stretches page rasters until it stops); previously trivial (dark-theme interface text on Linux no longer looks semi-bold); previously trivial (Control/MCP #2596 review: partial-write outcomes, path-bearing image exports, accept-thread startup cleanup and malformed UTF-8 handling); previously trivial (progress log: a turned inline picture whose file keeps no room for the turn keeps its line to its frame); previously trivial (progress log: the Arabic interface's Styles gallery reads from the right (#5566)); previously trivial (progress log: ODT pictures offered in several formats); previously trivial (progress log: ODT heading text of another size); previously trivial (Protected typing boundary lookup counts preceding objects once and indexes editable permission IDs (#3614); caret movement and enforcement policy are unchanged; previously trivial: progress log: Custom document properties dialog, #2982; previously trivial: progress log: View › New Window; previously trivial: progress log: interface translations come from storytold/craft-i18n through the `craft-i18n` crate; previously trivial: progress log: disabled-control reasons translated in every interface language, #2678; previously trivial: progress log: mail merge record navigation and record ranges for printing, #3028; previously trivial: selection bar for right-to-left paragraphs and table rows; previously trivial: group connector rerouting preserves rotated and flipped free or unresolved endpoints under nonuniform group scaling; readiness estimates unchanged; previously trivial: progress log: a word with no break opportunity in it no longer makes the layout quadratic, #3412; previously trivial: progress log: a relationship the markup names without the prefix is read as it wrote, and an ordinary `id` names no part; previously trivial: progress log: a document with many relationships opens in the time its part takes, and an id the text only mentions is not a reference, #3411; previously trivial: progress log: read only and comments-only protection grey out edits; previously trivial: progress log: an empty paragraph with a hidden mark shows nothing, not even its list number, #3982; previously trivial: progress log: "NaN" in the Font dialog's Size box no longer closes the app, #3663; previously trivial: progress log: a table cell span from a `.json` file no longer overflows, #3637; previously trivial: progress log: a click in the footer while editing the header moves there; previously trivial: explicit documents keep the editor when recovery discovery finishes; readable recovery controls; previously trivial: Slovenian interface catalog, 2,614 entries; previously trivial: Insert Table of Figures dialog for any caption label, #3013; previously trivial: progress log: Home › Underline drop-down, #2738; previously trivial: personal templates in File › New and Save as Word Template, #2852; previously trivial: progress log: the scrollbar page tip stays on one line, #3341; previously trivial: progress log: forms protection greys out the control buttons; previously trivial: progress log: a document with many content controls saves in the time the controls take, #13; previously trivial: progress log: Word signature lines kept on save; previously trivial: a Thai run with Latin finds one installed font covering it all on Linux too, #1549; previously trivial: progress log: closing the header returns to the body selection, #3331; previously trivial: progress log: saving to a NAS share mounted by GVFS, #3003; previously trivial: progress log: a picture dragged near the bottom of a page lands there, #3169; previously trivial: centred list labels after leading page breaks, #2964; previously trivial: progress log: inline pictures drag to an insertion cursor and stay inline, #2233, #2234; previously trivial: progress log: a text file saved over itself closes without asking, #2376; previously trivial: progress log: a text frame in the paragraph defaults frames nothing, #2360; previously trivial: fixed font inputs in four layout controls; previously trivial: typing no longer starts render threads on every repaint, #3027; previously trivial: a right, centre or decimal tab whose text doesn't fit, or a left tab past the right indent, goes to the next line with its text, #1840; previously trivial: progress log: banded rows shade as in Word, #3415; previously trivial: progress log: the table handles show while the caret is in the table, #2428; previously trivial: progress log: a click below a floating table reaches the text after it, #2038; previously trivial: progress log: Delete on selected table cells leaves one empty paragraph in each, #2802; previously trivial: progress log: lam-alef in exported PDFs extracts whole with fonts that draw it as two glyphs, #2310; previously trivial: Bullets and Numbering toggle off style-inherited lists at their effective level, including basedOn chains and numbered ListParagraph styles; other paragraphs and list definitions stay unchanged (related to #3459); previously trivial (progress log: a shape dropped just after a page break inside a paragraph stays on the new page, #4262); previously trivial (progress log: a float anchored on a later page of a long paragraph is drawn there, #4250); previously minor (progress log: `file.closeDocument` over the control channel and MCP bridge is refused while the document it names has unsaved changes; previously trivial (progress log: ODT element audit, ODT checked against 220 files from other applications, ODT display styles); previously trivial (progress log: Shape Format › Size in the interface unit, #4196); previously trivial (progress log: typed n-ary operators such as ∫ and ∑ wait for their limits, observed in Word for Windows on #4938 (#4917)); previously trivial (progress log: a paragraph naming an undefined style is formatted as Normal, #5077); previously trivial (progress log: justified and distributed lines keep the width of the spaces before their first character); previously trivial (print helpers erase loaded plaintext inputs before driver work, #5710); previously trivial (Windows print submissions use isolated helpers so a cancelled driver cannot block a retry, #5710); previously trivial (print cancellation remains available without a document; sending-boundary limitation clarified, #5710); previously trivial (progress log: a print job waiting on its driver can be cancelled, and quitting doesn't wait for it, #5710); previously trivial (progress log: the Print panel's printer list no longer waits on printer drivers, #5710); previously trivial (Page drawing indexes text-box warp lookups per paragraph (#3614)); previously trivial (built-in citation source lookups, #4888; current-main progress records retained); previously trivial (progress log: Times and Courier take their Windows counterparts' line spacing; previously trivial (progress log: the status bar's count can be words, characters or characters without spaces, #2797); previously trivial (Linux display server setting in File › Options › Advanced, #2184); previously trivial (progress log: Linux Read Aloud through Speech Dispatcher, #4576); previously trivial (progress log: a click just outside a floating table by turned cell text, #4181; previously minor: progress log: Spanish proofing, #3360, #3553; previously trivial: progress log: Insert › Table's grid previews the hovered table on the page, #3931; previously trivial (progress log: MHT quoted MIME parameters); previously trivial (progress log: CLI convert format options); previously trivial (MATH construction reuse, #3630); previously trivial (DOCX `w:adjustRightInd` is preserved through style resolution and ordered DOCX saves); previously trivial (per-page ink object lookups, #3614); previously trivial (progress log: ODT display settings removed); previously trivial (progress log: dialogs are modal except the five Word keeps modeless, and a press beside one shakes it, #3859); previously trivial (progress log: picture brightness, contrast and transparency, ODT element audit, ODT checked against 220 files from other applications, ODT display styles); previously trivial (progress log: the dialogs of the mirrored Arabic interface read right to left: label-and-field rows, tables (list columns, the Models table), tile galleries and the Add/Remove arrows follow the reading direction, through one `rtl::Table` helper that draws exactly as before in English; no ranked gap closed, readiness estimates unchanged); previously trivial (progress log: fonts that aren't installed are flagged in the font box and status bar); previously trivial (progress log: ODT sections inside text boxes); previously trivial (progress log: ODT definitions sharing a name, sections around tables, stacking order); previously trivial (progress log: ODT forms and indexes written as read); previously trivial (progress log: ODT table styles saved, WordCraft-origin documents checked in LibreOffice); previously trivial (progress log: ODT interpretation profile); previously trivial (progress log: ODT controls, custom shapes, in-line placement); previously trivial (progress log: ODT charts kept, frame outlines, header height); previously trivial (progress log: ODT header rows); previously trivial (progress log: ODT layout setting, padding, columns); previously trivial (progress log: ODT tables, header spacing, picture paragraphs); previously trivial (progress log: ODT round trips keep their page count); previously trivial (progress log: emphasis marks, SET fields, ODT page backgrounds and tables); previously trivial (progress log: ODT checked against LibreOffice's templates); previously trivial (progress log: ODT at ~90%); previously trivial (progress log: ODT read and written nearly whole); previously trivial (progress log: ODT text boxes); previously trivial (progress log: ODT layout settings); previously trivial (progress log: ODT character styles); previously trivial (progress log: ODT custom styles); previously trivial (progress log: ODT equations saved); previously trivial (progress log: ODT floating pictures); previously trivial (progress log: ODT paragraph borders); previously trivial (progress log: ODT heading numbering); previously trivial (progress log: ODT table borders); previously trivial (progress log: ODT heading styles); previously trivial (progress log: ODT tab stops); previously trivial (progress log: ODT character formatting); previously trivial (progress log: ODT lists); previously trivial (progress log: ODT comments); previously trivial (progress log: ODT footnotes and endnotes); previously trivial (progress log: ODT headers, footers and fields); previously trivial (progress log: ODT paragraph spacing); previously trivial (progress log: Options › Language redesign, CP4.3); previously trivial (progress log: the missing-pack bar on a fresh install, the caret's typing language, and the spelling menu heading in the interface language, CP4.3); previously trivial (progress log: Language pack languages in every language list, CP4.3); previously trivial (progress log: craft-proof pin, French elision and Valencian, CP4.3); previously trivial (progress log: a long comment's card and balloon stop at ten lines, #3382); previously trivial (progress log: a pasted text box story is charged for the formatting it copies); previously trivial (progress log: a .doc table row keeps one override per cell); previously trivial (progress log: a .doc file's sections find their paragraphs by index); previously minor (Arabic interface language with mirrored UI, #1195); previously trivial (progress log: Replace and Remove Table of Contents with Word's title above a section break, #4041); previously trivial (progress log: a password-protected file named on the command line asks for its password after the startup document is replaced, #5299); previously trivial (progress log: language-packs-2 published and pinned, 75 Language packs); previously trivial (progress log: language-packs-1 published and pinned, CP4.4); previously trivial (progress log: Language packs, CP4);; previously trivial (DOCX table rows and cells inside selected alternate-content branches survive open and repeated saves (#5219); broader corpus qualification and readiness estimates unchanged); previously trivial (progress log: a list inside a Markdown block quote stays quoted); previously trivial (progress log: arrowheads keep their kind, width and length and are drawn at Word's sizes); previously trivial (progress log: the line between columns for selected text, #5178); previously trivial (progress log: .doc standard font, #5066); previously trivial (progress log: a table row doesn't split off a picture that doesn't fit, #4272); previously trivial (progress log: a middle or bottom cell of a row split across pages lines up on the first page); previously trivial (progress log: tab leader dots spaced and placed as in Word); previously trivial (progress log: a list level above counts its first number at the first deeper item); previously trivial (progress log: paragraph side borders lie outside the text, #5036); previously trivial (progress log: TOC page numbers as PAGEREF fields, #5097); previously trivial (progress log: Backstage keytips, #4960); previously trivial (Progress log: only the printer's PDF leaves out links; Save these pages as PDF keeps them, #4734); previously trivial (progress log: Chinese and Japanese text counts a word per character, #4547; previously trivial: aligned-copy review fixes retain the document axis through scrolling and include secondary selections (#4764)); previously trivial (connector rerouting indexes laid-out object positions once per batch (#3631); previously trivial (progress log: corporate authors in bibliographies, #4848); previously trivial (custom-property bookmark-link preservation, #4767); previously trivial (progress log: Table Properties follows the selection, #4857); previously trivial (Relative font-size commands preserve mixed sizes in the selection (#4903); existing button steps and one-point shortcuts are unchanged; readiness estimates unchanged); previously trivial (Arabic Backspace removes one combining mark and keeps its base letter (#4757); previously trivial (note conversion walks reference offsets and formatting runs together, #4750); previously trivial (DOCX comment-ID save lookups); previously trivial (progress log: RTF pictures stored as Windows or Enhanced Metafiles are imported, #4259); previously trivial (#4658 bounds nested Markdown reference-label work); previously trivial (HTML import keeps preformatted lines with CR or CRLF input newlines, #4810); previously trivial (progress log: Insert › Shapes icon grid, #4349); previously trivial (progress log: AutoCorrect Options with no document open, #4585); previously trivial (progress log: window fallback geometry and equation symbol tooltips); previously trivial (progress log: window titles omit NUL characters, #3634); previously trivial (Recipient labels continue across product-sized sheets instead of stopping after the first sheet (#4880)); previously trivial (progress log: Custom document properties dialog, #2982; previously trivial: Russian sorting keeps ё with е and recognises decomposed ё/й; previously trivial: progress log: a text box that keeps the keyboard no longer resets the input method every frame, #2986; previously trivial: progress log: recovery copies and .json files store pictures as base64, #586 follow-up; previously trivial: progress log: legacy menu keys, #1697; previously trivial: progress log: shape gradients and transparency, #1500; previously trivial: progress log: Focus fills the screen, and Full Screen View shows the pages alone, #1678; previously trivial: progress log: text above a float's anchor wraps around it, #2048; previously trivial: progress log: File › Recover follows the app theme, #3243; previously trivial: progress log: formatting toolbar with the text context menu, #2959; previously trivial: progress log: caret inside ligatures, #1974; previously trivial: text typed into a content control takes the control's formatting; localized placeholder styles recognised; previously trivial: forms protection keeps its content controls: no inserting, Properties or Remove Content Control; previously trivial: long navigation headings wrap onto further lines within the pane, #3015; previously trivial: clearing a no-match font query keeps the recent heading visible, #2068; previously trivial: a word after a slash wraps whole, like Word, #2963; previously trivial: progress log: a header-cell picture watermark sits behind the text, #1775; previously trivial: symbols keep their font when the text font changes, #2036; previously trivial: opening a DOCX no longer copies its pictures; previously trivial: Add Font imports every selected font file, #1789); previously trivial (RTF export writes sections (gap 19); previously trivial: ODT export writes sections (gap 19); previously trivial: Read Aloud speaks each sentence in its own language, #1843; previously trivial: hint-reliant CJK outlines; previously trivial: Ctrl+wheel zoom no longer scrolls the document, #1743; previously trivial: progress log: Turkish case mapping in AutoCorrect, #1923; previously trivial: a mouse drag on a table leaves the view where it is, #1738; previously trivial: progress log: Hebrew field results and smart quotes, #1670; previously trivial: Backspace removes a selected table, rows or columns, #1844; previously trivial: progress log: mixed Hebrew/Latin text keeps its order in Word, #1543; previously trivial: progress log: web typing in non-Latin text no longer repeats, #1480; previously trivial: progress log: font family caches bounded and math font lookups follow font changes, #1354; previously trivial: `-->` AutoCorrect replacement, #2223; previously trivial: numbered headings, #1793; previously trivial: Navigation Find exposes Results, #2096; previously trivial: progress log: Columns on selected text, #2024; previously trivial: progress log: legacy outline, shadow, emboss and imprint text effects drawn, #2534; previously trivial: progress log: Indonesian proofing, #498; previously trivial: progress log: DOCX glossary documents (building blocks); previously trivial: progress log: Read Mode keeps the keyboard and shows whole, unobscured columns, #1772); previously trivial: progress log: font list entries of tall-ascent fonts no longer cut off, #2570); previously trivial: progress log: Split Cells and Convert Text to Table dialogs follow the Word check, #2592; previously trivial: a new source's Tag is made like Word's, three letters of the author and two digits of the year with a number added when taken, and an edit keeps the Tag, #2094; previously trivial: Web Layout fills the window width, #1876; previously trivial: progress log: Different First Page headers reachable from page 1, #2826; previously trivial: Language dialog keeps inherited proofing language and no-proof choices (#2717); previously trivial: progress log: web preferences kept between visits, #2132; previously trivial: progress log: undrawable private-use list bullets show the level's default bullet, #1683; previously trivial: progress log: `w:eastAsianLayout` kept on save; previously trivial: table style editor formats every part, #714); previously trivial: VML freeforms read and kept on save; previously trivial: progress log: consistent preferred cell widths; previously trivial: multi-paragraph field instructions no longer add blank lines; previously trivial: page-break-only list paragraphs hide their labels; previously trivial: progress log: grouped `flowChartOr` shapes draw their circle and crossing spokes; previously trivial: progress log: copies keep their formatting in other programs, #1523; previously trivial: progress log: privacy options on save, #1062; previously trivial: progress log: equation pictures in theme colours on Windows, #1002; previously trivial: symbol fonts, #776; previously trivial: the save prompt answers from the keyboard, #861; previously trivial: table column and row markers on the rulers, #700; previously trivial: table borders and the resize handle drag on the page, #217; previously trivial: formatting revisions landed, #41; previously trivial: Draw tab ink, #307, and Draw Table and Eraser, #303, landed: catalog 402/434; previously minor: readiness table with hours per audience; full number back to the additive weighted sum, 60%; mainstream and essentials numbers added; alpha gate checked; previously major: full re-measure against Word for Mac 16.113.4; progress docs restructured); previously trivial (progress log: ODT documents keep their language); previously trivial (progress log: ODT table and row backgrounds) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4 > > **Recipient CSV update:** 2026-10-27 · Complete quoted header records choose the delimiter (#4249); detection and regressions follow the current mailings module split. Recipient limits and readiness estimates are unchanged. > @@ -150,6 +150,7 @@ Ranked, to beta (detail and remaining milestones in [docs/roadmap.md](docs/roadm ## Progress log +- 2026-11-08: `wordcraft-cli convert --strict` (#6005) and `render --strict` (#6008): a PDF or PNG export (render: the page it writes) refuses, exit 3 and nothing written, when it would draw a character no font has or leave out a picture (missing part, external target, link only, undecodable or unsupported format); the report is on stderr with one JSON line. Engine: `strict: true` on `file.save`, `file.saveAs`, `file.exportPdf` and `file.exportPng`, and the pure `file.strictReport`. External picture targets are never fetched. The result depends on the host's installed fonts (#5624); readiness estimates unchanged. - 2026-11-05: ODT tables with a usable width keep their column proportions when the file gives only shares (gap 19): LibreOffice's CV template opened and saved with equal columns. - 2026-11-05: ODT font sizes reach Asian and complex-script text (gap 19): Chinese text set to 24 pt stayed at the default size in LibreOffice. - 2026-11-05: ODT text can turn off what its style turns on (gap 19): a word not bold in a bold heading showed bold, and came back bold in LibreOffice after a save. @@ -882,6 +883,8 @@ MATH glyphs that reference the same construction now share its immutable variant ## Revision history +> 2026-11-08 · Progress log: `convert --strict` refuses a PDF or PNG that would silently lack a character or a picture (#6005). +> > 2026-11-08 · Progress log: turned inline pictures without a recorded effect extent keep their line to the frame, as in Word. > > 2026-11-05 · Linux self-update ignores another program's AppImage variables (#5653); recorded in the progress log, readiness unchanged. diff --git a/apps/wordcraft-cli/Cargo.toml b/apps/wordcraft-cli/Cargo.toml index 7f9b25dfa7..da51fba676 100644 --- a/apps/wordcraft-cli/Cargo.toml +++ b/apps/wordcraft-cli/Cargo.toml @@ -21,3 +21,8 @@ wordcraft-mcp = { workspace = true } wordcraft-zotero = { workspace = true } serde_json = { workspace = true } sys-locale = "0.3" + +[dev-dependencies] +wordcraft-fonts = { workspace = true } +image = { workspace = true } +zip = { workspace = true } diff --git a/apps/wordcraft-cli/src/main.rs b/apps/wordcraft-cli/src/main.rs index 3d4235786d..903ba69d5c 100644 --- a/apps/wordcraft-cli/src/main.rs +++ b/apps/wordcraft-cli/src/main.rs @@ -17,17 +17,23 @@ const USAGE: &str = "\ wordcraft-cli — WordCraft from the command line USAGE: - wordcraft-cli convert [--markup] [--update-fields] [--format wordml2003|flatopc] + wordcraft-cli convert [--markup] [--update-fields] [--strict] [--format wordml2003|flatopc] convert between formats (docx, doc [read], pdf, odt, rtf, html, md, tex, txt, json, png, xml); --update-fields refreshes unlocked fields before saving; an .xml file is a Word XML Document (Flat OPC), or with --format wordml2003 a Word 2003 XML Document; --markup (Print Markup): comments beside each page and - tracked changes in pdf/png, a comment list in html/md/txt + tracked changes in pdf/png, a comment list in html/md/txt; + --strict (pdf or png output only) writes nothing and exits 3 + if a character can't be drawn or a picture is missing or + undecodable; the report is on stderr, with one JSON line wordcraft-cli info pages, words, paragraphs, properties (JSON) wordcraft-cli text plain text wordcraft-cli inspect document structure (JSON) - wordcraft-cli render [--page N] [--scale S] + wordcraft-cli render [--page N] [--scale S] [--strict] + --strict writes nothing and exits 3 if a character on the + page can't be drawn or a picture is missing or undecodable + (report on stderr, as for convert --strict) wordcraft-cli run [--file F | --template T] [--script FILE|-] --cmd 'id={json}' [--cmd …] [--save OUT] [--print] [--author NAME] run commands headlessly, then save (--author names @@ -77,6 +83,14 @@ fn wants_help(cmd: &str, rest: &[String]) -> bool { SUBCOMMANDS.contains(&cmd) && rest.iter().any(|a| a == "--help" || a == "-h") } +/// A failure whose message may carry text from a file or from the engine's reading of one (a +/// picture's part name, a parse error quoting the input): control, bidi and invisible characters +/// are escaped, so it can't forge or spoof a line (the CLI's own messages, which have line breaks +/// on purpose, are not routed through here). Backslashes stay: messages quote Windows paths. +fn engine_error(e: impl fmt::Display) -> Failure { + Failure::Error(escape_text(&e.to_string(), false)) +} + /// The error every subcommand gives when a required path is not there. fn missing_argument() -> Failure { format!("missing argument\n\n{USAGE}").into() @@ -85,11 +99,11 @@ fn missing_argument() -> Failure { /// Open a document for a subcommand. A password-protected Word file needs its password /// ([`password_from`]). As in the app, a document opened with one is saved with it again, so /// `convert` does not quietly write a copy the password no longer protects. -fn open(path: &Path, password: Option<&str>) -> Result { +fn open(path: &Path, password: Option<&str>) -> Result { // A file that gives no page size (HTML, Markdown, plain text) opens on the system region's // paper, as in the app: A4 in Italy, Letter in the US (#1381). let region = system_region(); - let (doc, encrypted) = wordcraft_engine::io::open_path_in(path, password, ®ion)?; + let (doc, encrypted) = wordcraft_engine::io::open_path_in(path, password, ®ion).map_err(engine_error)?; let mut s = session(doc); s.path = Some(path.into()); if encrypted { @@ -198,7 +212,7 @@ fn listed_text(doc: &wordcraft_doc::Document) -> Result { /// The command also checks whether the open document is protected: this name determines which /// named editing exceptions the session can use, so the flag must not bypass that check. fn set_author(s: &mut Session, name: &str) -> Result<(), String> { - s.run("file.setAuthor", &json!({"name": name})).map(|_| ()).map_err(|e| format!("--author: {e}")) + s.run("file.setAuthor", &json!({"name": name})).map(|_| ()).map_err(|e| escape_text(&format!("--author: {e}"), false)) } fn arg_value(args: &[String], name: &str) -> Option { @@ -310,10 +324,12 @@ const CONVERT_OPTIONS: &[OptionSpec] = &[ OptionSpec { name: "--markup", takes_value: false }, OptionSpec { name: "--format", takes_value: true }, OptionSpec { name: "--update-fields", takes_value: false }, + OptionSpec { name: "--strict", takes_value: false }, OptionSpec { name: "--password", takes_value: true }, OptionSpec { name: "--password-file", takes_value: true }, ]; const RENDER_OPTIONS: &[OptionSpec] = &[ + OptionSpec { name: "--strict", takes_value: false }, OptionSpec { name: "--page", takes_value: true }, OptionSpec { name: "--scale", takes_value: true }, OptionSpec { name: "--password", takes_value: true }, @@ -350,6 +366,9 @@ enum Failure { /// Whoever read stdout closed it (`wordcraft-cli commands --json | head`): there is nobody left /// to write to, so stop quietly and exit 0 like other command-line tools. StdoutClosed, + /// `convert --strict` refused: the text for stderr (the report's lines, then its JSON line). + /// Exit status 3. + Strict(String), } impl fmt::Display for Failure { @@ -357,6 +376,7 @@ impl fmt::Display for Failure { match self { Failure::Error(e) => f.write_str(e), Failure::StdoutClosed => f.write_str("stdout closed"), + Failure::Strict(report) => f.write_str(report), } } } @@ -402,11 +422,80 @@ fn valid_connect_addr(addr: &str) -> bool { !host.is_empty() && port.parse::().is_ok() } +/// A failed `--strict` export: a strict refusal becomes [`Failure::Strict`] carrying what stderr +/// shows (each line of the report as `wordcraft-cli: …`, then its JSON on one line, from +/// `file.strictReport` for `page` of a page image, else the PDF); any other error is an ordinary +/// failure. +fn refusal(s: &mut Session, e: wordcraft_engine::CmdError, markup: bool, page: Option) -> Failure { + if !matches!(&e, wordcraft_engine::CmdError::Failed(m) if m.starts_with("strict: ")) { + return engine_error(e); + } + let mut params = json!({"markup": markup}); + if let Some(page) = page { + params["page"] = json!(page); + } + Failure::Strict(match s.run("file.strictReport", ¶ms).map_err(|e| e.to_string()) { + Ok(json) => { + let mut text = String::new(); + for line in report_lines(&json) { + text += &format!("wordcraft-cli: {}\n", escape_peer(&line)); + } + text + &json_line_safe(&json.to_string()) + } + Err(e) => format!("wordcraft-cli: strict: refused, and the report could not be made: {}", escape_text(&e, false)), + }) +} + +/// The report's human lines, from its JSON (the engine's `StrictReport::lines`). +fn report_lines(json: &Value) -> Vec { + wordcraft_engine::strict::lines_from_json(json) +} + +/// `convert `: `rest` is the command line after `convert`, `open` opens the input. +fn run_convert(rest: &[String], open: impl Fn(&str, Option<&str>) -> Result) -> Result<(), Failure> { + validate_options("convert", rest, CONVERT_OPTIONS)?; + // `--format wordml2003` (or `flatopc`) picks the kind of `.xml` file; its value isn't a path. + let format = match find_option_arg(rest, "--format") { + OptionArg::Absent => None, + OptionArg::Duplicate => return Err("convert: duplicate --format option".into()), + OptionArg::MissingValue => return Err("convert: --format requires a value".into()), + OptionArg::Value(value) => Some(value), + }; + let paths = positionals(rest, CONVERT_OPTIONS); + let (i, o) = (paths.first().ok_or_else(missing_argument)?.to_string(), paths.get(1).ok_or_else(missing_argument)?.to_string()); + let markup = rest.iter().any(|a| a == "--markup"); + let strict = rest.iter().any(|a| a == "--strict"); + // Before the input is opened: a wrong output type is a usage error, not a file error. + let to_png = o.to_ascii_lowercase().ends_with(".png"); + if strict && !(to_png || o.to_ascii_lowercase().ends_with(".pdf")) { + return Err("convert: --strict needs a .pdf or .png output".into()); + } + let password = password_from(rest)?; + let mut s = open(&i, password.as_deref())?; + if rest.iter().any(|a| a == "--update-fields") { + s.run("references.updateFields", &json!({})).map_err(engine_error)?; + } + let r = if to_png { + s.run("file.exportPng", &json!({"path": o, "page": 1, "markup": markup, "strict": strict})) + } else { + let mut params = json!({"path": o, "markup": markup, "strict": strict}); + if let Some(f) = format { + params["format"] = json!(f); + } + s.run("file.save", ¶ms) + }; + if let Err(e) = r { + return Err(if strict { refusal(&mut s, e, markup, to_png.then_some(1)) } else { engine_error(e) }); + } + eprintln!("wrote {o}"); + Ok(()) +} + fn load_run_script(rest: &[String], script_path: Option<&Path>) -> Result, Failure> { match arg_value(rest, "--script") { Some(path) if !path.starts_with("--") => { let source = script_path.unwrap_or(Path::new(&path)); - Ok(Some(script::load(source)?)) + Ok(Some(script::load(source).map_err(engine_error)?)) } _ if rest.iter().any(|arg| arg == "--script") => Err("run: --script requires a value".into()), _ => Ok(None), @@ -438,42 +527,13 @@ fn run>(args: &[S]) -> Result<(), Failure> { return write_stdout(|o| write!(o, "{USAGE}")); } match cmd { - "convert" => { - validate_options("convert", &rest, CONVERT_OPTIONS)?; - // `--format wordml2003` (or `flatopc`) picks the kind of `.xml` file; its value isn't a path. - let format = match find_option_arg(&rest, "--format") { - OptionArg::Absent => None, - OptionArg::Duplicate => return Err("convert: duplicate --format option".into()), - OptionArg::MissingValue => return Err("convert: --format requires a value".into()), - OptionArg::Value(value) => Some(value), - }; - let paths = positionals(&rest, CONVERT_OPTIONS); - let (i, o) = (paths.first().ok_or_else(missing_argument)?.to_string(), paths.get(1).ok_or_else(missing_argument)?.to_string()); - let markup = rest.iter().any(|a| a == "--markup"); - let password = password_from(&rest)?; - let mut s = open(&i, password.as_deref())?; - if rest.iter().any(|a| a == "--update-fields") { - s.run("references.updateFields", &json!({})).map_err(|e| e.to_string())?; - } - let r = if o.to_ascii_lowercase().ends_with(".png") { - s.run("file.exportPng", &json!({"path": o, "page": 1, "markup": markup})) - } else { - let mut params = json!({"path": o, "markup": markup}); - if let Some(f) = format { - params["format"] = json!(f); - } - s.run("file.save", ¶ms) - }; - r.map_err(|e| e.to_string())?; - eprintln!("wrote {o}"); - Ok(()) - } + "convert" => run_convert(&rest, open), "info" => { validate_options("info", &rest, PASSWORD_OPTIONS)?; let paths = positionals(&rest, PASSWORD_OPTIONS); let password = password_from(&rest)?; let mut s = open(paths.first().ok_or_else(missing_argument)?, password.as_deref())?; - let info = serde_json::to_string_pretty(&s.run("file.info", &json!({})).map_err(|e| e.to_string())?).unwrap_or_default(); + let info = serde_json::to_string_pretty(&s.run("file.info", &json!({})).map_err(engine_error)?).unwrap_or_default(); write_stdout(|o| writeln!(o, "{info}")) } "text" => { @@ -489,7 +549,7 @@ fn run>(args: &[S]) -> Result<(), Failure> { let paths = positionals(&rest, PASSWORD_OPTIONS); let password = password_from(&rest)?; let mut s = open(paths.first().ok_or_else(missing_argument)?, password.as_deref())?; - let tree = serde_json::to_string_pretty(&s.run("document.inspect", &json!({})).map_err(|e| e.to_string())?).unwrap_or_default(); + let tree = serde_json::to_string_pretty(&s.run("document.inspect", &json!({})).map_err(engine_error)?).unwrap_or_default(); write_stdout(|o| writeln!(o, "{tree}")) } "render" => { @@ -503,7 +563,17 @@ fn run>(args: &[S]) -> Result<(), Failure> { let page = parse_page_arg("render", &rest)?.unwrap_or(1); let scale = parse_scale_arg("render", &rest)?.unwrap_or(2.0); let mut s = open(&i, password.as_deref())?; - let r = s.run("file.exportPng", &json!({"path": o, "page": page, "scale": scale})).map_err(|e| e.to_string())?; + let strict = rest.iter().any(|a| a == "--strict"); + let mut params = json!({"path": o, "page": page, "scale": scale, "strict": strict}); + if strict { + // The check and its report must see the page as it is written. + params["markup"] = json!(false); + } + let r = match s.run("file.exportPng", ¶ms) { + Ok(r) => r, + Err(e) if strict => return Err(refusal(&mut s, e, false, Some(page))), + Err(e) => return Err(engine_error(e)), + }; write_stdout(|o| writeln!(o, "{r}")) } "run" => { @@ -522,8 +592,8 @@ fn run>(args: &[S]) -> Result<(), Failure> { let mut s = session(wordcraft_doc::Document::new()); let first = s.document_id(); let params = t.map_or_else(|| json!({}), |t| json!({"template": t})); - s.run("file.new", ¶ms).map_err(|e| e.to_string())?; - s.close_document(first).map_err(|e| e.to_string())?; + s.run("file.new", ¶ms).map_err(engine_error)?; + s.close_document(first).map_err(engine_error)?; s } }; @@ -532,7 +602,7 @@ fn run>(args: &[S]) -> Result<(), Failure> { } let print = rest.iter().any(|a| a == "--print"); for (n, (id, params)) in script.iter().flat_map(|c| c.commands.iter()).enumerate() { - let r = s.run(id, params).map_err(|e| format!("command {} ({id}): {e}", n + 1))?; + let r = s.run(id, params).map_err(|e| engine_error(format!("command {} ({id}): {e}", n + 1)))?; if print { write_stdout(|o| writeln!(o, "{id}: {r}"))?; } @@ -542,10 +612,12 @@ fn run>(args: &[S]) -> Result<(), Failure> { if rest.get(i).map(String::as_str) == Some("--cmd") { let spec = rest.get(i + 1).ok_or("--cmd needs a value")?; let (id, params) = match spec.split_once('=') { - Some((id, p)) => (id.to_string(), serde_json::from_str::(p).map_err(|e| format!("{id}: bad JSON params: {e}"))?), + Some((id, p)) => { + (id.to_string(), serde_json::from_str::(p).map_err(|e| engine_error(format!("{id}: bad JSON params: {e}")))?) + } None => (spec.clone(), json!({})), }; - let r = s.run(&id, ¶ms).map_err(|e| format!("{id}: {e}"))?; + let r = s.run(&id, ¶ms).map_err(|e| engine_error(format!("{id}: {e}")))?; if print { write_stdout(|o| writeln!(o, "{id}: {r}"))?; } @@ -565,7 +637,7 @@ fn run>(args: &[S]) -> Result<(), Failure> { return Err(format!("{out}: is a folder, not a file").into()); } if !out.to_ascii_lowercase().ends_with(".png") { - wordcraft_engine::io::save_bytes(out, &s.doc)?; + wordcraft_engine::io::save_bytes(out, &s.doc).map_err(engine_error)?; } } for out in outputs { @@ -581,7 +653,7 @@ fn run>(args: &[S]) -> Result<(), Failure> { } else { s.run("file.save", &json!({"path": out})) }; - r.map_err(|e| e.to_string())?; + r.map_err(engine_error)?; eprintln!("wrote {out}"); } Ok(()) @@ -680,10 +752,12 @@ fn run>(args: &[S]) -> Result<(), Failure> { let mut s = open(file, None)?; for spec in rest.iter().zip(rest.iter().skip(1)).filter(|(a, _)| *a == "--cmd").map(|(_, v)| v) { let (id, params) = match spec.split_once('=') { - Some((id, p)) => (id.to_string(), serde_json::from_str::(p).map_err(|e| format!("{id}: bad JSON params: {e}"))?), + Some((id, p)) => { + (id.to_string(), serde_json::from_str::(p).map_err(|e| engine_error(format!("{id}: bad JSON params: {e}")))?) + } None => (spec.clone(), json!({})), }; - s.run(&id, ¶ms).map_err(|e| format!("{id}: {e}"))?; + s.run(&id, ¶ms).map_err(|e| engine_error(format!("{id}: {e}")))?; } let mut opts = wordcraft_zotero::client::Options::default(); if let Some(p) = arg_value(&rest, "--port") { @@ -700,13 +774,13 @@ fn run>(args: &[S]) -> Result<(), Failure> { } r }) - .map_err(|e| e.to_string())?; + .map_err(engine_error)?; for a in &host.alerts { eprintln!("Zotero: {}", escape_peer(a)); } eprintln!("{} calls, {}", out.calls, if out.completed { "completed" } else { "ended without completing (cancelled?)" }); if let Some(o) = arg_value(&rest, "--save") { - s.run("file.save", &json!({"path": o})).map_err(|e| e.to_string())?; + s.run("file.save", &json!({"path": o})).map_err(engine_error)?; eprintln!("wrote {o}"); } Ok(()) @@ -730,22 +804,62 @@ fn clip(s: &str) -> String { /// overrides are shown as escapes (`\\`, `\n`, `\u{1b}`), so it can't end its line, drive the /// terminal or pass for other output. fn escape_peer(s: &str) -> String { + escape_text(s, true) +} + +/// Whether `c` must not reach a terminal as itself: a control character, a line or paragraph +/// separator, a bidi control, or one that shows nothing (soft hyphen, zero-width characters, word +/// joiners, the byte order mark, tag characters), which can hide or reorder what a line says. +fn is_unsafe_char(c: char) -> bool { + c.is_control() + || matches!( + c, + '\u{00ad}' + | '\u{061c}' + | '\u{200b}'..='\u{200f}' + | '\u{2028}'..='\u{202e}' + | '\u{2060}'..='\u{2064}' + | '\u{2066}'..='\u{2069}' + | '\u{feff}' + | '\u{e0000}'..='\u{e007f}' + ) +} + +/// [`escape_peer`], with backslashes escaped only when `backslash` (error messages quote Windows +/// paths, and a backslash can't forge or hide anything). +fn escape_text(s: &str, backslash: bool) -> String { let mut out = String::with_capacity(s.len()); for c in s.chars() { match c { - '\\' => out.push_str("\\\\"), + '\\' if backslash => out.push_str("\\\\"), '\n' => out.push_str("\\n"), '\r' => out.push_str("\\r"), '\t' => out.push_str("\\t"), - c if c.is_control() || matches!(c, '\u{061c}' | '\u{200e}' | '\u{200f}' | '\u{2028}'..='\u{202e}' | '\u{2066}'..='\u{2069}') => { - out.extend(c.escape_unicode()) - } + c if is_unsafe_char(c) => out.extend(c.escape_unicode()), c => out.push(c), } } out } +/// A JSON line safe to print: serde escapes control characters below U+0020 but not the other +/// characters [`is_unsafe_char`] names, which a crafted file could put in a report. They become +/// `\uXXXX` escapes (a surrogate pair above U+FFFF), so the line parses to the same value and +/// can't drive or spoof a terminal. +fn json_line_safe(json: &str) -> String { + let mut out = String::with_capacity(json.len()); + for c in json.chars() { + if is_unsafe_char(c) { + for unit in c.encode_utf16(&mut [0u16; 2]) { + out.push_str(&format!("\\u{unit:04x}")); + } + } else { + out.push(c); + } + } + out +} + fn parity_markdown(p: &Value) -> String { let mut s = String::from( "# WordCraft feature parity\n\nGenerated by `cargo xtask parity` (`wordcraft-cli parity --markdown`): the word-processor feature catalog (`crates/engine/src/catalog.rs`) compared with the live command registry.\n\n", @@ -786,6 +900,10 @@ fn main() -> ExitCode { eprintln!("wordcraft-cli: {e}"); ExitCode::FAILURE } + Err(Failure::Strict(report)) => { + eprintln!("{report}"); + ExitCode::from(3) + } } } diff --git a/apps/wordcraft-cli/src/tests.rs b/apps/wordcraft-cli/src/tests.rs index 844dda5689..c44143fb41 100644 --- a/apps/wordcraft-cli/src/tests.rs +++ b/apps/wordcraft-cli/src/tests.rs @@ -564,3 +564,19 @@ fn script_parse_resolves_paths_against_the_base() { assert_eq!(s.save, vec!["o.docx".to_string()]); assert_eq!(s.commands.len(), 1); } + +#[test] +fn invisible_characters_are_escaped_and_json_lines_still_parse_back() { + let hidden = "a\u{ad}b\u{200b}c\u{200d}d\u{2060}e\u{2064}f\u{feff}g\u{e0041}h\u{e007f}i\u{85}j\u{202e}k"; + let shown = escape_peer(hidden); + assert!(!shown.chars().any(is_unsafe_char), "{shown}"); + assert!(shown.contains("\\u{e0041}") && shown.contains("\\u{ad}"), "{shown}"); + // Errors keep a Windows path readable. + assert_eq!(escape_text("C:\\dir\\a.docx", false), "C:\\dir\\a.docx"); + + let value = json!({"part": hidden, "plain": "ünïcode ok"}); + let line = json_line_safe(&value.to_string()); + assert!(!line.chars().any(is_unsafe_char), "{line}"); + assert!(line.contains("\\ud800") || line.contains("\\udb40"), "{line}"); + assert_eq!(serde_json::from_str::(&line).unwrap(), value); +} diff --git a/apps/wordcraft-cli/tests/it/main.rs b/apps/wordcraft-cli/tests/it/main.rs index 61a27c8129..5aaf93cca5 100644 --- a/apps/wordcraft-cli/tests/it/main.rs +++ b/apps/wordcraft-cli/tests/it/main.rs @@ -6,4 +6,5 @@ mod help; #[cfg(all(unix, not(target_os = "macos")))] mod non_utf8_args; mod picture_then_typing; +mod strict; mod text_tracked; diff --git a/apps/wordcraft-cli/tests/it/strict.rs b/apps/wordcraft-cli/tests/it/strict.rs new file mode 100644 index 0000000000..399fc09927 --- /dev/null +++ b/apps/wordcraft-cli/tests/it/strict.rs @@ -0,0 +1,495 @@ +//! `convert --strict` (#6005): exit 3 and a report when a PDF or PNG would silently lack a +//! character or a picture, nothing written; exit 0 otherwise. +use std::io::Write; +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +use wordcraft_doc::Document; +use wordcraft_fonts::FontDb; + +const NS: &str = r#"xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main" xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships" xmlns:wp="http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing" xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main" xmlns:pic="http://schemas.openxmlformats.org/drawingml/2006/picture""#; +const REL: &str = "http://schemas.openxmlformats.org/officeDocument/2006/relationships"; +const RELS_NS: &str = "http://schemas.openxmlformats.org/package/2006/relationships"; + +fn dir(name: &str) -> PathBuf { + let d = std::env::temp_dir().join(format!("wc-cli-strict-{name}-{}", std::process::id())); + std::fs::create_dir_all(&d).unwrap(); + d +} + +fn convert(input: &Path, output: &Path, strict: bool) -> Output { + let mut cmd = Command::new(env!("CARGO_BIN_EXE_wordcraft-cli")); + cmd.arg("convert").arg(input).arg(output); + if strict { + cmd.arg("--strict"); + } + cmd.output().unwrap() +} + +fn stderr(o: &Output) -> String { + String::from_utf8_lossy(&o.stderr).into_owned() +} + +/// The report's JSON line: the last line of stderr. +fn report_json(o: &Output) -> serde_json::Value { + let text = stderr(o); + let last = text.lines().last().unwrap_or_default(); + serde_json::from_str(last).unwrap_or_else(|e| panic!("last stderr line is not JSON ({e}): {text}")) +} + +fn json_input(d: &Path, text: &str) -> PathBuf { + let path = d.join("in.json"); + std::fs::write(&path, serde_json::to_vec(&Document::from_text(text)).unwrap()).unwrap(); + path +} + +/// Built without craft-fonts, the host's own fonts decide what is drawable: skip, as the layout +/// tests do. +fn without_fonts() -> bool { + let missing = option_env!("CRAFT_FONTS_DIR").is_none_or(str::is_empty); + if missing { + eprintln!("skipped: built without craft-fonts (set CRAFT_FONTS_DIR to a checkout)"); + } + missing +} + +/// A code point in unassigned plane 4 that no face of the host covers. +fn uncovered() -> char { + let db = FontDb::global(); + let primary = wordcraft_fonts::word::resolve(&Document::default().settings.minor_font, false, false).face; + (0x40000u32..0x4FFF0) + .filter_map(char::from_u32) + .find(|c| !primary.covers(*c) && db.fallback_styled(*c, primary.id(), false, false).is_none() && db.fallback_for(*c, primary.id()).is_none()) + .expect("the host has faces covering all of plane 4") +} + +fn png() -> Vec { + let mut out = Vec::new(); + image::RgbaImage::from_pixel(4, 4, image::Rgba([20, 90, 200, 255])) + .write_to(&mut std::io::Cursor::new(&mut out), image::ImageFormat::Png) + .unwrap(); + out +} + +fn pic(rid: &str) -> String { + format!( + r#""# + ) +} + +/// A DOCX whose header holds a picture of `header_media` (`word/media/image2.bin`). +fn docx_with_header_picture(header_media: &[u8]) -> Vec { + let doc = format!( + r#"text"# + ); + let drels = format!(r#""#); + let hrels = format!(r#""#); + let hdr = format!(r#"{}"#, pic("rIdI")); + package( + &doc, + &drels, + &[("word/header2.xml", hdr.as_bytes()), ("word/_rels/header2.xml.rels", hrels.as_bytes()), ("word/media/image2.bin", header_media)], + ) +} + +/// A DOCX whose body picture points outside the file, at `target`. +fn docx_with_external_picture(target: &str) -> Vec { + let doc = format!(r#"{}text"#, pic("rIdX")); + let drels = format!( + r#""# + ); + package(&doc, &drels, &[]) +} + +fn package(document: &str, document_rels: &str, extra: &[(&str, &[u8])]) -> Vec { + let rels = format!( + r#""# + ); + let types = r#""#; + let mut zip = zip::ZipWriter::new(std::io::Cursor::new(Vec::new())); + let mut parts: Vec<(&str, &[u8])> = vec![ + ("[Content_Types].xml", types.as_bytes()), + ("_rels/.rels", rels.as_bytes()), + ("word/document.xml", document.as_bytes()), + ("word/_rels/document.xml.rels", document_rels.as_bytes()), + ]; + parts.extend_from_slice(extra); + for (name, bytes) in parts { + zip.start_file(name, zip::write::SimpleFileOptions::default()).unwrap(); + zip.write_all(bytes).unwrap(); + } + zip.finish().unwrap().into_inner() +} + +#[test] +fn multilingual_names_pass_strict() { + if without_fonts() { + return; + } + let names = "Łukasz Żmijewski\nGheorghe Ștefănescu\nKrejčí\nŞahın Gül\nLlŷr Grŵn\n王小明\n김민수\nสมชาย ใจดี\nराम शर्मा"; + let d = dir("names"); + let input = json_input(&d, names); + for out in ["names.pdf", "names.png"] { + let out = d.join(out); + let r = convert(&input, &out, true); + assert_eq!(r.status.code(), Some(0), "{}", stderr(&r)); + assert!(out.exists()); + } + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn an_uncovered_code_point_fails_strict_with_a_report_and_no_file() { + let c = uncovered(); + let cp = format!("U+{:04X}", c as u32); + let d = dir("uncovered"); + let input = json_input(&d, &format!("a{c}b{c}{c}")); + for out in ["out.pdf", "out.png"] { + let out = d.join(out); + let r = convert(&input, &out, true); + assert_eq!(r.status.code(), Some(3), "{}", stderr(&r)); + let text = stderr(&r); + assert!(text.contains(&format!("wordcraft-cli: strict: 1 character cannot be drawn: {cp} (×3)")), "{text}"); + let json = report_json(&r); + assert_eq!(json["schema"], 1); + assert_eq!(json["strict"], "failed"); + assert_eq!(json["missing_glyphs"][0]["cp"], cp.as_str()); + assert_eq!(json["missing_glyphs"][0]["count"], 3); + assert!(!out.exists(), "{} was written", out.display()); + eprintln!("{text}"); + // Without --strict the same input converts, as before. + let r = convert(&input, &out, false); + assert_eq!(r.status.code(), Some(0), "{}", stderr(&r)); + assert!(out.exists()); + std::fs::remove_file(&out).unwrap(); + } + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn a_real_png_in_a_header_passes_strict() { + let d = dir("header-png"); + let input = d.join("in.docx"); + std::fs::write(&input, docx_with_header_picture(&png())).unwrap(); + for out in ["out.pdf", "out.png"] { + let out = d.join(out); + let r = convert(&input, &out, true); + assert_eq!(r.status.code(), Some(0), "{}", stderr(&r)); + assert!(out.exists()); + } + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn a_corrupt_header_picture_fails_strict_naming_the_part_and_the_header() { + let d = dir("header-corrupt"); + let input = d.join("in.docx"); + std::fs::write(&input, docx_with_header_picture(b"corrupt bytes, not a picture")).unwrap(); + for out in ["out.pdf", "out.png"] { + let out = d.join(out); + let r = convert(&input, &out, true); + assert_eq!(r.status.code(), Some(3), "{}", stderr(&r)); + let text = stderr(&r); + assert!(text.contains("word/media/image2.bin (header2.xml): unsupported format"), "{text}"); + let json = report_json(&r); + assert_eq!(json["strict"], "failed"); + assert_eq!( + json["missing_pictures"][0], + serde_json::json!({"part": "word/media/image2.bin", "source_part": "word/header2.xml", "reason": "unsupported_format"}) + ); + assert!(!out.exists()); + eprintln!("{text}"); + } + // Without --strict the grey box is drawn, as before. + let out = d.join("lenient.pdf"); + assert_eq!(convert(&input, &out, false).status.code(), Some(0)); + assert!(out.exists()); + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn an_external_picture_target_is_never_fetched_and_is_reported_under_strict() { + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + listener.set_nonblocking(true).unwrap(); + let target = format!("http://{}/canary.png", listener.local_addr().unwrap()); + let d = dir("external"); + let input = d.join("in.docx"); + std::fs::write(&input, docx_with_external_picture(&target)).unwrap(); + + let lenient = convert(&input, &d.join("lenient.pdf"), false); + assert_eq!(lenient.status.code(), Some(0), "{}", stderr(&lenient)); + let out = d.join("strict.pdf"); + let strict = convert(&input, &out, true); + assert_eq!(strict.status.code(), Some(3), "{}", stderr(&strict)); + let json = report_json(&strict); + assert_eq!(json["missing_pictures"][0]["reason"], "external_target"); + assert!(stderr(&strict).contains("external target")); + assert!(!out.exists()); + + match listener.accept() { + Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => {} + Ok((_, from)) => panic!("the converter connected to the external target from {from}"), + Err(e) => panic!("{e}"), + } + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn bad_input_exits_1_with_the_usual_message_with_and_without_strict() { + let d = dir("bad-input"); + let good = docx_with_header_picture(&png()); + let truncated = d.join("truncated.docx"); + std::fs::write(&truncated, &good[..good.len() / 2]).unwrap(); + let not_zip = d.join("not-zip.docx"); + std::fs::write(¬_zip, b"this is plainly not a zip archive").unwrap(); + for input in [&truncated, ¬_zip] { + let plain = convert(input, &d.join("a.pdf"), false); + let strict = convert(input, &d.join("b.pdf"), true); + assert_eq!(plain.status.code(), Some(1), "{}", stderr(&plain)); + assert_eq!(strict.status.code(), Some(1), "{}", stderr(&strict)); + assert_eq!(stderr(&plain), stderr(&strict)); + assert!(stderr(&plain).starts_with("wordcraft-cli: "), "{}", stderr(&plain)); + assert!(!d.join("a.pdf").exists() && !d.join("b.pdf").exists()); + } + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn strict_with_another_output_type_exits_1_before_the_input_is_read() { + let d = dir("wrong-output"); + let missing = d.join("does-not-exist.docx"); + for out in ["out.docx", "out.txt", "out.odt"] { + let r = convert(&missing, &d.join(out), true); + assert_eq!(r.status.code(), Some(1)); + let text = stderr(&r); + assert!(text.contains("--strict needs a .pdf or .png output"), "{text}"); + assert!(!d.join(out).exists()); + } + // Without --strict the same command fails on the missing input. + let r = convert(&missing, &d.join("out.docx"), false); + assert_eq!(r.status.code(), Some(1)); + assert!(!stderr(&r).contains("--strict")); + std::fs::remove_dir_all(d).unwrap(); +} + +fn render(input: &Path, output: &Path, args: &[&str]) -> Output { + Command::new(env!("CARGO_BIN_EXE_wordcraft-cli")).arg("render").arg(input).arg(output).args(args).output().unwrap() +} + +/// Page 1 has `a{c}b`, page 2 plain text. +fn two_page_input(d: &Path, text: &str) -> PathBuf { + let mut doc = Document::from_text(&format!("{text}\nSecond page")); + doc.para_mut(wordcraft_doc::StoryRef::Body, &wordcraft_doc::Path::top(1)).unwrap().props.page_break_before = Some(true); + let path = d.join("two.json"); + std::fs::write(&path, serde_json::to_vec(&doc).unwrap()).unwrap(); + path +} + +#[test] +fn render_strict_refuses_an_uncovered_character_on_the_page_with_no_stdout_or_file() { + let c = uncovered(); + let cp = format!("U+{:04X}", c as u32); + let d = dir("render-glyph"); + let input = two_page_input(&d, &format!("a{c}b")); + let out = d.join("p1.png"); + let r = render(&input, &out, &["--strict"]); + assert_eq!(r.status.code(), Some(3), "{}", stderr(&r)); + assert!(r.stdout.is_empty(), "{}", String::from_utf8_lossy(&r.stdout)); + assert!(!out.exists()); + let text = stderr(&r); + assert!(text.contains(&format!("wordcraft-cli: strict: 1 character cannot be drawn: {cp} (×1)")), "{text}"); + assert_eq!(report_json(&r)["missing_glyphs"][0]["cp"], cp.as_str()); + eprintln!("{text}"); + + // The character is not on page 2, and --scale does not matter to the check. + let out2 = d.join("p2.png"); + let r = render(&input, &out2, &["--page", "2", "--strict", "--scale", "0.5"]); + assert_eq!(r.status.code(), Some(0), "{}", stderr(&r)); + assert!(out2.exists()); + assert!(String::from_utf8_lossy(&r.stdout).contains("\"path\"")); + // And the same page 1 without --strict renders, with the usual stdout. + let out3 = d.join("lenient.png"); + let r = render(&input, &out3, &[]); + assert_eq!(r.status.code(), Some(0), "{}", stderr(&r)); + assert!(out3.exists()); + let json: serde_json::Value = serde_json::from_slice(&r.stdout).unwrap(); + assert_eq!(json["path"], out3.to_string_lossy().as_ref()); + assert!(json["width"].as_u64().is_some() && json["bytes"].as_u64().is_some()); + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn render_strict_refuses_a_corrupt_header_picture_and_bad_input_still_exits_1() { + let d = dir("render-pic"); + let input = d.join("in.docx"); + std::fs::write(&input, docx_with_header_picture(b"corrupt bytes, not a picture")).unwrap(); + let out = d.join("out.png"); + let r = render(&input, &out, &["--strict"]); + assert_eq!(r.status.code(), Some(3), "{}", stderr(&r)); + assert!(r.stdout.is_empty()); + assert!(stderr(&r).contains("word/media/image2.bin (header2.xml)"), "{}", stderr(&r)); + assert!(!out.exists()); + + let bad = d.join("bad.docx"); + std::fs::write(&bad, b"this is plainly not a zip archive").unwrap(); + let (plain, strict) = (render(&bad, &out, &[]), render(&bad, &out, &["--strict"])); + assert_eq!((plain.status.code(), strict.status.code()), (Some(1), Some(1))); + assert_eq!(stderr(&plain), stderr(&strict)); + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn a_picture_target_with_a_newline_and_a_bidi_override_cannot_forge_a_report_line() { + let d = dir("escape"); + let input = d.join("in.docx"); + // XML character references put a newline and U+202E in the target. + std::fs::write(&input, docx_with_external_picture("http://example.invalid/a wordcraft-cli: strict: forged‮gnp.png")).unwrap(); + let out = d.join("out.pdf"); + let r = convert(&input, &out, true); + assert_eq!(r.status.code(), Some(3), "{}", stderr(&r)); + let text = stderr(&r); + let lines: Vec<&str> = text.lines().collect(); + // One human line for the picture, then the JSON line. + assert_eq!(lines.len(), 2, "{text}"); + assert!(lines[0].starts_with("wordcraft-cli: strict: 1 picture(s) left out: "), "{text}"); + assert!(lines[0].contains("\\nwordcraft-cli: strict: forged\\u{202e}gnp.png"), "{text}"); + assert!(!text.contains('\u{202e}'), "the raw bidi override reached the terminal"); + // The JSON keeps the raw value, escaped by the JSON encoder. + let json = report_json(&r); + let part = json["missing_pictures"][0]["part"].as_str().unwrap(); + assert!(part.contains('\n') && part.contains('\u{202e}'), "{part:?}"); + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn run_with_strict_cannot_forge_an_error_line_either() { + let d = dir("run-escape"); + let input = d.join("in.docx"); + std::fs::write(&input, docx_with_external_picture("http://example.invalid/a wordcraft-cli: strict: forged‮gnp.png")).unwrap(); + let out = d.join("out.pdf"); + let cmd = format!("file.exportPdf={}", serde_json::json!({"path": out, "strict": true})); + let r = Command::new(env!("CARGO_BIN_EXE_wordcraft-cli")).arg("run").arg("--file").arg(&input).arg("--cmd").arg(cmd).output().unwrap(); + assert_eq!(r.status.code(), Some(1)); + let text = stderr(&r); + assert_eq!(text.lines().count(), 1, "{text}"); + assert!(text.contains("\\u{1b}") || text.contains("\\u{1b}[2J"), "{text}"); + assert!(!text.contains('\u{1b}') && !text.contains('\u{202e}'), "{text:?}"); + assert!(!out.exists()); + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn usage_errors_keep_their_line_breaks() { + // Errors go to stderr with the usage after a blank line (`missing argument`, `unknown command`). + for args in [&["frobnicate"][..], &["convert"][..]] { + let r = Command::new(env!("CARGO_BIN_EXE_wordcraft-cli")).args(args).output().unwrap(); + let text = stderr(&r); + assert_eq!(r.status.code(), Some(1), "{args:?}: {text}"); + assert!(text.lines().count() > 3, "{args:?}: {text}"); + assert!(text.lines().any(|l| l == "USAGE:"), "{args:?}: {text}"); + assert!(!text.contains("\\n"), "{args:?}: {text}"); + } + // No arguments at all prints the usage as help, on stdout. + let r = Command::new(env!("CARGO_BIN_EXE_wordcraft-cli")).output().unwrap(); + let all = format!("{}{}", String::from_utf8_lossy(&r.stdout), stderr(&r)); + assert!(all.lines().count() > 3 && all.lines().any(|l| l == "USAGE:"), "{all}"); +} + +/// A minimal EMF (88-byte header, 100 mm square frame) with one EMR_EXTTEXTOUTW record showing +/// `text` in the default font. +fn emf_with_text(text: &str) -> Vec { + let units: Vec = text.encode_utf16().collect(); + let mut rec = vec![0u8; 76]; + rec[0..4].copy_from_slice(&84u32.to_le_bytes()); + rec[36..40].copy_from_slice(&10i32.to_le_bytes()); + rec[40..44].copy_from_slice(&40i32.to_le_bytes()); + rec[44..48].copy_from_slice(&(units.len() as u32).to_le_bytes()); + rec[48..52].copy_from_slice(&76u32.to_le_bytes()); + rec.extend(units.iter().flat_map(|u| u.to_le_bytes())); + while !rec.len().is_multiple_of(4) { + rec.push(0); + } + let len = rec.len() as u32; + rec[4..8].copy_from_slice(&len.to_le_bytes()); + let eof: Vec = [14u32.to_le_bytes(), 20u32.to_le_bytes(), [0; 4], [0; 4], 20u32.to_le_bytes()].concat(); + let total = 88 + rec.len() + eof.len(); + let mut h = vec![0u8; 88]; + h[0..4].copy_from_slice(&1u32.to_le_bytes()); + h[4..8].copy_from_slice(&88u32.to_le_bytes()); + h[8..24].copy_from_slice(&[0i32, 0, 100, 100].iter().flat_map(|v| v.to_le_bytes()).collect::>()); + h[24..40].copy_from_slice(&[0i32, 0, 10000, 10000].iter().flat_map(|v| v.to_le_bytes()).collect::>()); + h[40..44].copy_from_slice(&0x464D_4520u32.to_le_bytes()); + h[44..48].copy_from_slice(&0x0001_0000u32.to_le_bytes()); + h[48..52].copy_from_slice(&(total as u32).to_le_bytes()); + h[52..56].copy_from_slice(&2u32.to_le_bytes()); + h[56..58].copy_from_slice(&1u16.to_le_bytes()); + h[72..76].copy_from_slice(&1000i32.to_le_bytes()); + h[76..80].copy_from_slice(&1000i32.to_le_bytes()); + h[80..84].copy_from_slice(&250i32.to_le_bytes()); + h[84..88].copy_from_slice(&250i32.to_le_bytes()); + [h, rec, eof].concat() +} + +/// The text a metafile picture draws is checked like the page's own: a character no face covers +/// refuses the export from `convert` and `render`, naming the code point, and nothing is written. +#[test] +fn a_character_in_a_metafile_pictures_text_fails_strict_like_the_pages_own() { + if without_fonts() { + return; + } + let c = uncovered(); + let cp = format!("U+{:04X}", c as u32); + let d = dir("emf-text"); + let emf = emf_with_text(&format!("ab{c}")); + let input = d.join("in.docx"); + std::fs::write(&input, docx_with_header_picture(&emf)).unwrap(); + for out in ["out.pdf", "out.png"] { + let out = d.join(out); + let r = convert(&input, &out, true); + assert_eq!(r.status.code(), Some(3), "{}", stderr(&r)); + assert_eq!(report_json(&r)["missing_glyphs"][0]["cp"], cp.as_str(), "{}", stderr(&r)); + assert!(!out.exists(), "{} was written", out.display()); + // Without --strict it converts, the character a hollow box. + let r = convert(&input, &out, false); + assert_eq!(r.status.code(), Some(0), "{}", stderr(&r)); + std::fs::remove_file(&out).unwrap(); + } + let out = d.join("p1.png"); + let r = render(&input, &out, &["--strict"]); + assert_eq!(r.status.code(), Some(3), "{}", stderr(&r)); + assert!(r.stdout.is_empty() && !out.exists()); + assert_eq!(report_json(&r)["missing_glyphs"][0]["cp"], cp.as_str()); + // Text the faces cover passes. + std::fs::write(&input, docx_with_header_picture(&emf_with_text("abc"))).unwrap(); + let r = convert(&input, &d.join("ok.pdf"), true); + assert_eq!(r.status.code(), Some(0), "{}", stderr(&r)); + std::fs::remove_dir_all(d).unwrap(); +} + +/// The page raster leaves out a metafile page background (it only draws raster ones), so a PNG +/// would silently lack it: strict refuses that output, and a PDF, which draws it as vector, passes. +#[test] +fn a_metafile_page_background_fails_strict_for_png_but_not_for_pdf() { + if without_fonts() { + return; + } + let d = dir("emf-background"); + let mut doc = Document::from_text("text"); + doc.media.insert("bg.emf".into(), std::sync::Arc::new(emf_with_text("abc"))); + doc.settings.page_picture = Some(wordcraft_doc::PagePicture { media: "bg.emf".into(), fill_type: "frame".into(), title: String::new() }); + let input = d.join("in.json"); + std::fs::write(&input, serde_json::to_vec(&doc).unwrap()).unwrap(); + let png = d.join("out.png"); + let r = convert(&input, &png, true); + assert_eq!(r.status.code(), Some(3), "{}", stderr(&r)); + let json = report_json(&r); + assert_eq!(json["missing_pictures"][0]["part"], "bg.emf", "{}", stderr(&r)); + assert_eq!(json["missing_pictures"][0]["source_part"], "background"); + assert_eq!(json["missing_pictures"][0]["reason"], "unsupported_format"); + assert!(!png.exists()); + let pdf = d.join("out.pdf"); + let r = convert(&input, &pdf, true); + assert_eq!(r.status.code(), Some(0), "{}", stderr(&r)); + assert!(pdf.exists()); + std::fs::remove_dir_all(d).unwrap(); +} diff --git a/contributors/people.toml b/contributors/people.toml index 1f5825f1ad..d1129e53f8 100644 --- a/contributors/people.toml +++ b/contributors/people.toml @@ -204,3 +204,6 @@ real_name = "Helen Kwok" [people.ashokram-si] real_name = "Ashok Ram" + +[people.benandez] +sync_github_name = true diff --git a/crates/doc/src/document.rs b/crates/doc/src/document.rs index 2c08391ff2..87b922b0c0 100644 --- a/crates/doc/src/document.rs +++ b/crates/doc/src/document.rs @@ -77,6 +77,7 @@ impl Document { glossary: Vec::new(), media_index: MediaIndex::default(), kept: Default::default(), + missing_pictures: Default::default(), } } @@ -154,6 +155,7 @@ impl Document { glossary, media_index: _, kept: _, + missing_pictures: _, } = self; let Document { body: other_body, @@ -174,6 +176,7 @@ impl Document { glossary: other_glossary, media_index: _, kept: _, + missing_pictures: _, } = other; same_blocks(body, other_body) && same_parts(parts, other_parts) diff --git a/crates/doc/src/lib.rs b/crates/doc/src/lib.rs index d6ee0a1a74..e8b84b17eb 100644 --- a/crates/doc/src/lib.rs +++ b/crates/doc/src/lib.rs @@ -75,6 +75,7 @@ mod block; mod compare; mod document; mod error; +mod missing; mod parts; mod position; mod settings; @@ -84,6 +85,7 @@ pub use block::{Block, Blocks, BoxBudget, MAX_TEXT_BOX_DEPTH}; pub use compare::para_block; use document::MediaIndex; pub use error::{DocError, Result}; +pub use missing::{MAX_MISSING_PICTURES, MissingPicture, MissingPictureReason, MissingPictures, picture_hash}; pub use parts::{MAX_LINKED_BOXES, MAX_TEXT_BOX_INSET, NoteSeparators, Part, PartKind, PermRange, TEXT_BOX_INSET, TextBody, free_id}; pub use position::{Path, Pos, StoryRef}; pub use settings::{ @@ -138,6 +140,10 @@ pub struct Document { /// [`kept`]); never saved, ignored by `==`. #[serde(skip)] pub kept: kept::Kept, + /// Pictures a reader dropped because their target is missing, external or link-only (strict + /// export reports them); never saved, never drawn, ignored by `==`. + #[serde(skip)] + pub missing_pictures: MissingPictures, } #[cfg(test)] diff --git a/crates/doc/src/missing.rs b/crates/doc/src/missing.rs new file mode 100644 index 0000000000..d60d93c141 --- /dev/null +++ b/crates/doc/src/missing.rs @@ -0,0 +1,143 @@ +//! Pictures a reader could not bring in or an exporter could not draw, for strict export (#6005). +//! Kept beside the document while it is open; never saved, never drawn, and not part of the +//! document's content (`==` ignores it). + +use serde::{Deserialize, Serialize}; + +/// Most pictures remembered per document; past it [`MissingPictures::truncated`] is set. +pub const MAX_MISSING_PICTURES: usize = 256; + +/// Most characters of a picture's part or story name a record keeps (names come from the file). +pub const MAX_PICTURE_NAME_CHARS: usize = 1024; + +/// Why a picture is not in the document or not drawn. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub enum MissingPictureReason { + /// The relationship names a part the file does not contain, or no relationship at all. + MissingPart, + /// The target is outside the file (`TargetMode="External"`); it is never fetched. + ExternalTarget, + /// The picture only links to a file (`r:link`) and holds no embedded copy. + LinkOnly, + /// The bytes are a picture format we know but they do not decode. + Undecodable, + /// The bytes are not a picture format we can draw. + UnsupportedFormat, +} + +impl MissingPictureReason { + /// The words used in reports (`"missing part"`, `"external target"`, `"link only"`, + /// `"undecodable"`, `"unsupported format"`). + pub fn label(self) -> &'static str { + match self { + Self::MissingPart => "missing part", + Self::ExternalTarget => "external target", + Self::LinkOnly => "link only", + Self::Undecodable => "undecodable", + Self::UnsupportedFormat => "unsupported format", + } + } + + /// The reason a [`code`](Self::code) names. + pub fn from_code(code: &str) -> Option { + [Self::MissingPart, Self::ExternalTarget, Self::LinkOnly, Self::Undecodable, Self::UnsupportedFormat].into_iter().find(|r| r.code() == code) + } + + /// The stable snake_case code the JSON report carries (`"missing_part"`, `"external_target"`, + /// `"link_only"`, `"undecodable"`, `"unsupported_format"`); never reworded. + pub fn code(self) -> &'static str { + match self { + Self::MissingPart => "missing_part", + Self::ExternalTarget => "external_target", + Self::LinkOnly => "link_only", + Self::Undecodable => "undecodable", + Self::UnsupportedFormat => "unsupported_format", + } + } +} + +/// One picture that is missing. +#[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)] +pub struct MissingPicture { + /// The target as written, or the part name it resolved to (a media key for a decoded one). + pub part: String, + /// The story part it is in (`word/document.xml`, `word/header2.xml`), as the source file + /// names it; empty when not known. + pub story: String, + pub reason: MissingPictureReason, +} + +/// A capped list of [`MissingPicture`]s, with the source names a report needs to say where a +/// picture is (a reader fills them; a document made or opened another way leaves them empty). +#[derive(Clone, Debug, Default)] +pub struct MissingPictures { + pub list: Vec, + /// More than [`MAX_MISSING_PICTURES`] were found; the rest are not listed. + pub truncated: bool, + /// Pictures found, those past the cap included (saturating). Each distinct one counts once. + pub total: u64, + /// The main story's part name in the source file (`word/document.xml`). + pub main: String, + /// A header, footer, note or comment story's part id ([`crate::Document::parts`]) to the + /// part it was read from (`word/header2.xml`). Ids freed and reused after opening may name + /// the old part. + pub part_names: std::collections::BTreeMap>, + /// A [`crate::Document::media`] key to the part its bytes were read from + /// (`word/media/image2.bin`). + pub media_parts: std::collections::BTreeMap, + /// What `list` holds, for an O(1) test of "already there" (a hostile file has many). + seen: std::collections::HashSet, + /// Hashes of the distinct pictures past the cap (8 bytes each), so the total counts them once. + unlisted: std::collections::HashSet, + /// The chunk parts whose unlisted pictures [`Self::add_unlisted`] has counted, so a chunk + /// imported twice counts them once. + counted_chunks: std::collections::HashSet, +} + +impl MissingPictures { + /// Add `n` pictures of the chunk part `chunk` that its reader counted but did not list. The + /// same chunk part counts once however often it is imported (its listed pictures dedupe + /// through [`Self::push`]). + pub fn add_unlisted(&mut self, chunk: &str, n: u64) { + if n > 0 && self.counted_chunks.insert(chunk.to_string()) { + self.total = self.total.saturating_add(n); + } + } + + /// Add one (the same part, story and reason once). + pub fn push(&mut self, mut item: MissingPicture) { + for name in [&mut item.part, &mut item.story] { + if name.len() > MAX_PICTURE_NAME_CHARS { + *name = name.chars().take(MAX_PICTURE_NAME_CHARS).collect(); + } + } + if self.seen.contains(&item) { + return; + } + if self.list.len() >= MAX_MISSING_PICTURES { + self.truncated = true; + if self.unlisted.insert(picture_hash(&item)) { + self.total = self.total.saturating_add(1); + } + return; + } + self.total = self.total.saturating_add(1); + self.seen.insert(item.clone()); + self.list.push(item); + } +} + +/// A 64-bit hash of a record, to tell the ones past a cap apart without keeping them. +pub fn picture_hash(item: &MissingPicture) -> u64 { + use std::hash::{Hash, Hasher}; + let mut h = std::collections::hash_map::DefaultHasher::new(); + item.hash(&mut h); + h.finish() +} + +impl PartialEq for MissingPictures { + /// Not part of the document's content. + fn eq(&self, _: &Self) -> bool { + true + } +} diff --git a/crates/doc/src/tests.rs b/crates/doc/src/tests.rs index 828996d060..088bd3bd2c 100644 --- a/crates/doc/src/tests.rs +++ b/crates/doc/src/tests.rs @@ -786,3 +786,31 @@ fn lone_table_open_markers_mixed_tables_and_paired() { assert!(res1.controls.open.is_empty()); assert_eq!(res1.controls.close, 0); } + +#[test] +fn missing_picture_names_are_capped_at_a_char_boundary() { + let mut d = Document::new(); + let long: String = "é".repeat(5000); + d.missing_pictures.push(MissingPicture { part: long.clone(), story: long, reason: MissingPictureReason::MissingPart }); + let item = &d.missing_pictures.list[0]; + assert_eq!((item.part.chars().count(), item.story.chars().count()), (1024, 1024)); +} + +#[test] +fn missing_pictures_are_capped_deduplicated_and_not_content() { + let mut d = Document::new(); + let item = |n: usize| MissingPicture { part: format!("p{n}"), story: "word/document.xml".into(), reason: MissingPictureReason::MissingPart }; + d.missing_pictures.push(item(0)); + d.missing_pictures.push(item(0)); + assert_eq!(d.missing_pictures.list.len(), 1); + for n in 1..MAX_MISSING_PICTURES + 3 { + d.missing_pictures.push(item(n)); + } + assert_eq!(d.missing_pictures.list.len(), MAX_MISSING_PICTURES); + assert!(d.missing_pictures.truncated); + let e = Document::new(); + assert_eq!(d, e); + assert!(d.same_content(&e)); + assert!(!serde_json::to_string(&d).unwrap().contains("p0")); + assert_eq!(MissingPictureReason::ExternalTarget.label(), "external target"); +} diff --git a/crates/docx/src/read.rs b/crates/docx/src/read.rs index 4c6b1cb01a..f8570c1089 100644 --- a/crates/docx/src/read.rs +++ b/crates/docx/src/read.rs @@ -27,7 +27,8 @@ use wordcraft_doc::props::VAlign; use wordcraft_doc::section::NumFormat; use wordcraft_doc::styles::{PartFormat, Style, StyleKind, StyleSheet, TablePart, TableStyleParts}; use wordcraft_doc::{ - Block, Blocks, Comment, Document, InlineObject, PagePicture, Paragraph, PartKind, Revision, RevisionKind, Zoom, ZoomFit, para_block, + Block, Blocks, Comment, Document, InlineObject, MissingPicture, MissingPictureReason, PagePicture, Paragraph, PartKind, Revision, RevisionKind, + Zoom, ZoomFit, para_block, }; use crate::DocxError; @@ -103,6 +104,8 @@ pub(crate) struct Reader<'p> { pub shape_ids: bool, /// The main document part's name. main: String, + /// The part being read (`word/header2.xml`), for naming where a dropped picture was. + story: Arc, /// The package's content types, read when first needed. content_types: Option, /// Parts kept for charts, diagrams and OLE objects (see `embed`). @@ -183,6 +186,7 @@ fn read_opened_in(pkg: &Package, chunks: ChunkState) -> Result<(Document, usize, glued: HashSet::new(), shape_ids: false, main: main.clone(), + story: main.as_str().into(), content_types: None, embedded: EmbeddedManifest::default(), txbx_links: Vec::new(), @@ -190,6 +194,7 @@ fn read_opened_in(pkg: &Package, chunks: ChunkState) -> Result<(Document, usize, group_fill: None, chunks, }; + r.doc.missing_pictures.main = main.clone(); r.pc.major_font = r.doc.settings.major_font.clone(); r.pc.minor_font = r.doc.settings.minor_font.clone(); let part = |t: &str| rels.by_type(t).filter(|x| !x.external).map(|x| x.target.clone()); @@ -230,14 +235,18 @@ fn read_opened_in(pkg: &Package, chunks: ChunkState) -> Result<(Document, usize, let ex = part(rt::COMMENTS_EX); let ids = part(rt::COMMENTS_IDS); let cex = part(rt::COMMENTS_EXTENSIBLE); + r.story = p.as_str().into(); lenient("comments", r.read_comments(&p, ex.as_deref(), ids.as_deref(), cex.as_deref())); } if let Some(p) = part(rt::FOOTNOTES) { + r.story = p.as_str().into(); lenient("footnotes", r.read_notes(&p, true)); } if let Some(p) = part(rt::ENDNOTES) { + r.story = p.as_str().into(); lenient("endnotes", r.read_notes(&p, false)); } + r.story = main.as_str().into(); // Bibliography sources live in a custom XML data store (`b:Sources`), related from the // package root or the main part; Word's own files sometimes leave the relationship out, so // the conventional `customXml/itemN.xml` names are tried too. Each part is read at most once. diff --git a/crates/docx/src/read/chunk.rs b/crates/docx/src/read/chunk.rs index b97aa4b47c..937d8c615f 100644 --- a/crates/docx/src/read/chunk.rs +++ b/crates/docx/src/read/chunk.rs @@ -80,6 +80,21 @@ impl Reader<'_> { // `w:matchSrc`: a Word document's content keeps its own styles (other formats' formatting // is their own already). let keep = is_docx && e.child("w:altChunkPr").and_then(|p| p.child("w:matchSrc")).is_some_and(crate::units::on_off); + // The pictures the chunk's own reader dropped are the file's too (strict export, #6005), + // each named by the chunk and its part in it (`word/afchunk1.docx > word/header1.xml`). + // A chunk in another format brings none: those importers record nothing yet (#6007). + let nested = std::mem::take(&mut doc.missing_pictures); + let list_len = nested.list.len() as u64; + for mut item in nested.list { + let inner = if item.story.is_empty() { nested.main.as_str() } else { item.story.as_str() }; + item.story = format!("{part} > {inner}"); + self.doc.missing_pictures.push(item); + } + // The nested records past its list are the file's too; their stories carry the chunk's + // part, so none is one the outer list already counted. + let unlisted = nested.total.saturating_sub(list_len); + self.doc.missing_pictures.add_unlisted(&part, unlisted); + self.doc.missing_pictures.truncated |= nested.truncated; let mut blocks = self.doc.adopt_blocks(doc, keep, &mut self.next_perm); // Bookmarks and comment marks read before the chunk stay before it. self.flush_pending_before(sc, out, &mut blocks); diff --git a/crates/docx/src/read/glossary.rs b/crates/docx/src/read/glossary.rs index 0383f11b75..bc6b8dd54a 100644 --- a/crates/docx/src/read/glossary.rs +++ b/crates/docx/src/read/glossary.rs @@ -53,7 +53,10 @@ impl Reader<'_> { // Without content types (a converted package) the root element alone decides. if let Some(root) = root.filter(|r| r.name == "w:glossaryDocument" && ct.as_deref().is_none_or(|t| t == GLOSSARY_CONTENT_TYPE)) { let rels = self.pkg.rels(&rel.target); + // Pictures in a building block belong to the glossary part, not the main story. + let outer = std::mem::replace(&mut self.story, rel.target.as_str().into()); entries = self.doc_parts(&root, &rels); + self.story = outer; } } self.doc.glossary = entries; diff --git a/crates/docx/src/read/meta.rs b/crates/docx/src/read/meta.rs index 7f6e93fc32..eb97fe2282 100644 --- a/crates/docx/src/read/meta.rs +++ b/crates/docx/src/read/meta.rs @@ -117,6 +117,7 @@ impl Reader<'_> { // Reserve the id first: the note's own reference mark points at it. let (pk, kind) = if foot { (PartKind::Footnote, NoteKind::Footnote) } else { (PartKind::Endnote, NoteKind::Endnote) }; let id = self.doc.add_part(pk, Blocks::new()); + self.doc.missing_pictures.part_names.insert(id, self.story.clone()); let mut sc = StoryCtx::default(); let mut blocks = Blocks::new(); self.current_note = Some((kind, id)); @@ -214,6 +215,7 @@ impl Reader<'_> { self.read_blocks(&mut sc, c, &rels, &mut blocks, self.chunks.depth); self.flush_pending(&mut sc, &mut blocks); let part = self.doc.add_part(PartKind::Comment, blocks); + self.doc.missing_pictures.part_names.insert(part, self.story.clone()); if let Some(pid) = c.children("w:p").last().and_then(|p| p.attr("w14:paraId")) { by_para.insert(pid.to_string(), id); } diff --git a/crates/docx/src/read/parts.rs b/crates/docx/src/read/parts.rs index ce4184dc7d..565e65c2a5 100644 --- a/crates/docx/src/read/parts.rs +++ b/crates/docx/src/read/parts.rs @@ -1,6 +1,6 @@ use super::{ - Arc, Blocks, ContentTypes, DocxError, El, MAX_FONT_NAME, MAX_HF_DEPTH, MAX_PARTS, MAX_TABLE_FONTS, MAX_VBA_RELATED, PartKind, Reader, Rels, - Revision, RevisionKind, Rgb, StoryCtx, VBA_PROJECT_PART, VBA_RELATED, props, shape_style, + Arc, Blocks, ContentTypes, DocxError, El, MAX_FONT_NAME, MAX_HF_DEPTH, MAX_PARTS, MAX_TABLE_FONTS, MAX_VBA_RELATED, MissingPicture, + MissingPictureReason, PartKind, Reader, Rels, Revision, RevisionKind, Rgb, StoryCtx, VBA_PROJECT_PART, VBA_RELATED, props, shape_style, }; impl Reader<'_> { @@ -91,14 +91,22 @@ impl Reader<'_> { /// Media key for an image relationship (loads the bytes once). pub fn media_for(&mut self, rels: &Rels, id: &str) -> Option { - let rel = rels.by_id(id)?; + let Some(rel) = rels.by_id(id) else { + self.note_missing_picture(id, MissingPictureReason::MissingPart); + return None; + }; if rel.external { + // Never fetched: a picture whose target is outside the file is dropped (and reported). + self.note_missing_picture(&rel.target, MissingPictureReason::ExternalTarget); return None; } if let Some(k) = self.media_by_path.get(&rel.target) { return Some(k.clone()); } - let bytes = self.pkg.get_shared(&rel.target)?; + let Some(bytes) = self.pkg.get_shared(&rel.target) else { + self.note_missing_picture(&rel.target, MissingPictureReason::MissingPart); + return None; + }; let base: String = rel.target.rsplit('/').next().unwrap_or("image").chars().filter(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '_' | '-')).collect(); let base = if base.is_empty() || base.starts_with('.') { format!("image{base}") } else { base }; @@ -112,10 +120,18 @@ impl Reader<'_> { }; } self.doc.media.insert(key.clone(), bytes.clone()); + self.doc.missing_pictures.media_parts.insert(key.clone(), rel.target.clone()); self.media_by_path.insert(rel.target.clone(), key.clone()); Some(key) } + /// Remember a picture that was dropped at open, in the story being read. + pub(super) fn note_missing_picture(&mut self, part: &str, reason: MissingPictureReason) { + let part: String = part.chars().take(1024).collect(); + let story = self.story.to_string(); + self.doc.missing_pictures.push(MissingPicture { part, story, reason }); + } + /// A chart or diagram part, parsed once however often it's referenced (`None` when unreadable). pub(super) fn graphic_part(&mut self, path: &str) -> Option> { if let Some(p) = self.graphic_parts.get(path) { @@ -137,6 +153,8 @@ impl Reader<'_> { let kind = if footer { PartKind::Footer } else { PartKind::Header }; let id = self.doc.add_part(kind, Blocks::new()); self.hf_by_path.insert(path.to_string(), id); + let name: Arc = path.into(); + self.doc.missing_pictures.part_names.insert(id, Arc::clone(&name)); let root = match self.xml(path) { Ok(Some(r)) => r, _ => return Some(id), @@ -146,7 +164,9 @@ impl Reader<'_> { sc.in_hf = true; let mut blocks = Blocks::new(); self.hf_depth += 1; + let outer = std::mem::replace(&mut self.story, name); self.read_blocks(&mut sc, &root, &rels, &mut blocks, self.chunks.depth); + self.story = outer; self.hf_depth = self.hf_depth.saturating_sub(1); self.flush_pending(&mut sc, &mut blocks); if let Some(p) = self.doc.parts.get_mut(&id) diff --git a/crates/docx/src/read/story/drawing.rs b/crates/docx/src/read/story/drawing.rs index 75435a22a7..9edf2743fa 100644 --- a/crates/docx/src/read/story/drawing.rs +++ b/crates/docx/src/read/story/drawing.rs @@ -279,7 +279,15 @@ impl Reader<'_> { fn read_pic(&mut self, pic: &El, rels: &Rels, w: f32, h: f32, alt: String, mut float: Float) -> Option { let blip = pic.find("a:blip")?; float.set_spin(xfrm_spin(pic.find("pic:spPr"))); - let media = blip.attr("r:embed").and_then(|id| self.media_for(rels, id))?; + let Some(embed) = blip.attr("r:embed") else { + // A linked picture holds no copy: nothing to draw, and nothing is fetched. + if let Some(link) = blip.attr("r:link") { + let target = rels.by_id(link).map_or_else(|| link.to_string(), |r| r.target.clone()); + self.note_missing_picture(&target, wordcraft_doc::MissingPictureReason::LinkOnly); + } + return None; + }; + let media = self.media_for(rels, embed)?; let mut crop = [0.0f32; 4]; if let Some(sr) = pic.find("a:srcRect") { for (i, n) in ["l", "t", "r", "b"].iter().enumerate() { diff --git a/crates/docx/src/read/story/vml.rs b/crates/docx/src/read/story/vml.rs index e40d38b3a7..11e7e754aa 100644 --- a/crates/docx/src/read/story/vml.rs +++ b/crates/docx/src/read/story/vml.rs @@ -131,7 +131,14 @@ impl Reader<'_> { None => vml_float(shape, style, sc.in_hf), }; if let Some(img) = shape.find("v:imagedata") { - let id = img.attr("r:id").or_else(|| img.attr("r:pict"))?; + let Some(id) = img.attr("r:id").or_else(|| img.attr("r:pict")) else { + // `r:href` names a file outside the package: never fetched. + if let Some(href) = img.attr("r:href") { + let target = rels.by_id(href).map_or_else(|| href.to_string(), |r| r.target.clone()); + self.note_missing_picture(&target, wordcraft_doc::MissingPictureReason::LinkOnly); + } + return None; + }; let media = self.media_for(rels, id)?; let alt = shape.attr("alt").or_else(|| img.attr("o:title")).unwrap_or("").to_string(); // Brightness and contrast (Word's picture watermark "Washout" sets them, #1711). diff --git a/crates/docx/tests/it/alt_chunks.rs b/crates/docx/tests/it/alt_chunks.rs index af2404329f..5b3ab7e136 100644 --- a/crates/docx/tests/it/alt_chunks.rs +++ b/crates/docx/tests/it/alt_chunks.rs @@ -372,3 +372,74 @@ fn hostile_chunks_never_panic() { let bytes = package(&format!(r#"{}"#, p("y")), None); assert_eq!(body_lines(&read(&bytes).unwrap()), ["y"]); } + +/// A DOCX whose only picture names a part the package lacks (`media/gone.png`), or sits outside it. +fn inner_with_dropped_picture(external: bool) -> Vec { + inner_with_dropped_pictures(1, external) +} + +/// As [`inner_with_dropped_picture`], with `n` pictures each naming its own missing part. +fn inner_with_dropped_pictures(n: usize, external: bool) -> Vec { + let drawing = r#""#; + let body: String = (0..n).map(|i| drawing.replace("{i}", &i.to_string())).collect(); + let doc = format!(r#"{body}"#); + let mode = if external { r#" TargetMode="External""# } else { "" }; + let image = |i: usize| { + let target = if external { format!("http://127.0.0.1:9/never-fetched{i}.png") } else { format!("media/gone{i}.png") }; + format!( + r#""# + ) + }; + let rels = format!( + r#"{}"#, + (0..n).map(image).collect::() + ); + let types = r#""#; + let mut zw = zip::ZipWriter::new(std::io::Cursor::new(Vec::new())); + for (name, bytes) in [ + ("[Content_Types].xml", types.as_bytes()), + ("_rels/.rels", ROOT_RELS.as_bytes()), + ("word/document.xml", doc.as_bytes()), + ("word/_rels/document.xml.rels", rels.as_bytes()), + ] { + zw.start_file(name, zip::write::SimpleFileOptions::default()).unwrap(); + zw.write_all(bytes).unwrap(); + } + zw.finish().unwrap().into_inner() +} + +/// A picture the chunk's own reader dropped is the file's too (#6005), named by the chunk part. +#[test] +fn a_picture_dropped_inside_a_chunk_is_recorded_against_the_chunk() { + for (external, reason) in [(false, wordcraft_doc::MissingPictureReason::MissingPart), (true, wordcraft_doc::MissingPictureReason::ExternalTarget)] + { + let inner = inner_with_dropped_picture(external); + let bytes = package(&format!("{}{CHUNK}", p("Before")), Some(("docx", DOCX_MAIN, &inner))); + let d = read(&bytes).unwrap(); + let got: Vec<(&str, &str, _)> = d.missing_pictures.list.iter().map(|m| (m.part.as_str(), m.story.as_str(), m.reason)).collect(); + assert_eq!(got.len(), 1, "{got:?}"); + assert_eq!(got[0].1, "word/afchunk.docx > word/document.xml"); + assert_eq!(got[0].2, reason); + } +} + +/// A chunk with more dropped pictures than the list holds still counts them all (#6005). +#[test] +fn a_chunks_total_of_dropped_pictures_counts_past_the_list() { + let n = wordcraft_doc::MAX_MISSING_PICTURES + 44; + let inner = inner_with_dropped_pictures(n, false); + let d = read(&package(&format!("{}{CHUNK}", p("Before")), Some(("docx", DOCX_MAIN, &inner)))).unwrap(); + assert_eq!(d.missing_pictures.list.len(), wordcraft_doc::MAX_MISSING_PICTURES); + assert!(d.missing_pictures.truncated); + assert_eq!(d.missing_pictures.total, n as u64); +} + +/// The same chunk imported twice counts its unlisted pictures once, as it dedupes its listed ones. +#[test] +fn a_chunk_imported_twice_counts_its_dropped_pictures_past_the_list_once() { + let n = wordcraft_doc::MAX_MISSING_PICTURES + 44; + let inner = inner_with_dropped_pictures(n, false); + let d = read(&package(&format!("{}{CHUNK}{CHUNK}", p("Before")), Some(("docx", DOCX_MAIN, &inner)))).unwrap(); + assert_eq!(d.missing_pictures.list.len(), wordcraft_doc::MAX_MISSING_PICTURES); + assert_eq!(d.missing_pictures.total, n as u64); +} diff --git a/crates/docx/tests/it/glossary.rs b/crates/docx/tests/it/glossary.rs index 7b234542a0..40e6ea4913 100644 --- a/crates/docx/tests/it/glossary.rs +++ b/crates/docx/tests/it/glossary.rs @@ -560,3 +560,23 @@ fn a_private_save_keeps_character_references_in_attribute_values_as_written() { let again = wordcraft_docx::read(&saved).unwrap(); assert_eq!(again.glossary.first().map(|e| e.description.as_str()), Some(before.as_str())); } + +/// A picture in a building block that names a part the package lacks is recorded against the +/// glossary part, not the main document (#6005). +#[test] +fn a_missing_picture_in_a_building_block_names_the_glossary_part() { + let drawing = r#""#; + let part = + format!(r#"{drawing}"#); + let bytes = package( + &[("rId7", GLOSSARY_REL, "glossary/document.xml")], + &[ + ("[Content_Types].xml", content_types(&[("/word/glossary/document.xml", GLOSSARY_CT)])), + ("word/glossary/document.xml", glossary_xml(&part)), + ("word/glossary/_rels/document.xml.rels", rels(&[("rIdPic", &format!("{REL}/image"), "media/gone.png")])), + ], + ); + let d = wordcraft_docx::read(&bytes).unwrap(); + let stories: Vec<&str> = d.missing_pictures.list.iter().map(|m| m.story.as_str()).collect(); + assert_eq!(stories, ["word/glossary/document.xml"], "{:?}", d.missing_pictures.list); +} diff --git a/crates/docx/tests/it/main.rs b/crates/docx/tests/it/main.rs index df7e28fdff..910106646a 100644 --- a/crates/docx/tests/it/main.rs +++ b/crates/docx/tests/it/main.rs @@ -21,6 +21,7 @@ mod heading_levels; mod legacy_forms; mod ligatures; mod malformed; +mod missing_pictures; mod nested_block_controls; mod note_comments; mod passthrough; diff --git a/crates/docx/tests/it/missing_pictures.rs b/crates/docx/tests/it/missing_pictures.rs new file mode 100644 index 0000000000..a7b065c71c --- /dev/null +++ b/crates/docx/tests/it/missing_pictures.rs @@ -0,0 +1,163 @@ +//! Pictures dropped at open are recorded for strict export (#6005): a missing part, an external +//! target and a link-only picture (ECMA-376 Part 1 §20.1.8.13 `a:blip` `r:embed`/`r:link`). +//! Original synthetic packages; nothing is fetched, the reader has no network code. +use std::io::Write; +use wordcraft_doc::{Document, InlineObject, MissingPictureReason}; + +const NS: &str = r#"xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main" xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships" xmlns:wp="http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing" xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main" xmlns:pic="http://schemas.openxmlformats.org/drawingml/2006/picture" xmlns:v="urn:schemas-microsoft-com:vml""#; +const REL: &str = "http://schemas.openxmlformats.org/officeDocument/2006/relationships"; + +fn png() -> Vec { + let mut out = Vec::new(); + image::RgbaImage::from_pixel(2, 2, image::Rgba([20, 90, 200, 255])) + .write_to(&mut std::io::Cursor::new(&mut out), image::ImageFormat::Png) + .unwrap(); + out +} + +fn blip(attrs: &str) -> String { + format!( + r#""# + ) +} + +/// A package: `body` in `word/document.xml` (whose relationships are `doc_rels`), an optional +/// header with `header` as its content (relationships `header_rels`) and `extra` parts. +fn package(body: &str, doc_rels: &str, header: Option<(&str, &str)>, extra: &[(&str, Vec)]) -> Vec { + let hdr_rel = if header.is_some() { format!(r#""#) } else { String::new() }; + let sect = if header.is_some() { r#""# } else { "" }; + let doc = format!(r#"{body}{sect}"#); + let rels = format!( + r#""# + ); + let drels = format!(r#"{doc_rels}{hdr_rel}"#); + let types = r#""#; + let mut zip = zip::ZipWriter::new(std::io::Cursor::new(Vec::new())); + let mut put = |name: &str, bytes: &[u8]| { + zip.start_file(name, zip::write::SimpleFileOptions::default()).unwrap(); + zip.write_all(bytes).unwrap(); + }; + put("[Content_Types].xml", types.as_bytes()); + put("_rels/.rels", rels.as_bytes()); + put("word/document.xml", doc.as_bytes()); + put("word/_rels/document.xml.rels", drels.as_bytes()); + if let Some((content, hrels)) = header { + put("word/header2.xml", format!(r#"{content}"#).as_bytes()); + put( + "word/_rels/header2.xml.rels", + format!(r#"{hrels}"#).as_bytes(), + ); + } + for (n, b) in extra { + put(n, b); + } + zip.finish().unwrap().into_inner() +} + +fn image_rel(id: &str, target: &str, external: bool) -> String { + let mode = if external { r#" TargetMode="External""# } else { "" }; + format!(r#""#) +} + +fn missing(d: &Document) -> Vec<(String, String, MissingPictureReason)> { + d.missing_pictures.list.iter().map(|m| (m.part.clone(), m.story.clone(), m.reason)).collect() +} + +fn pictures(d: &Document) -> usize { + d.body[0].as_para().unwrap().objects.iter().filter(|o| matches!(o, InlineObject::Image { .. })).count() +} + +#[test] +fn external_missing_and_link_only_pictures_are_recorded_and_still_dropped() { + let body = [blip(r#"r:embed="rIdExt""#), blip(r#"r:embed="rIdGone""#), blip(r#"r:link="rIdLink""#), blip(r#"r:embed="rIdNone""#)].concat(); + let rels = [ + image_rel("rIdExt", "http://127.0.0.1:9/never-fetched.png", true), + image_rel("rIdGone", "media/absent.png", false), + image_rel("rIdLink", "file:///C:/pics/linked.png", true), + ] + .concat(); + let d = wordcraft_docx::read(&package(&body, &rels, None, &[])).unwrap(); + assert_eq!(pictures(&d), 0, "nothing is drawn for them, as before"); + assert_eq!( + missing(&d), + [ + ("http://127.0.0.1:9/never-fetched.png".into(), "word/document.xml".into(), MissingPictureReason::ExternalTarget), + ("word/media/absent.png".into(), "word/document.xml".into(), MissingPictureReason::MissingPart), + ("file:///C:/pics/linked.png".into(), "word/document.xml".into(), MissingPictureReason::LinkOnly), + ("rIdNone".into(), "word/document.xml".into(), MissingPictureReason::MissingPart), + ] + ); + assert!(!d.missing_pictures.truncated); +} + +#[test] +fn a_header_pictures_story_is_the_header_part() { + let rels = [image_rel("rIdHx", "https://example.invalid/logo.png", true), image_rel("rIdOk", "media/ok.png", false)].concat(); + let header = [blip(r#"r:embed="rIdHx""#), blip(r#"r:embed="rIdOk""#)].concat(); + let bytes = package("", "", Some((&header, &rels)), &[("word/media/ok.png", png())]); + let d = wordcraft_docx::read(&bytes).unwrap(); + assert_eq!(missing(&d), [("https://example.invalid/logo.png".into(), "word/header2.xml".into(), MissingPictureReason::ExternalTarget)]); + assert_eq!(d.missing_pictures.main, "word/document.xml"); + assert!(d.missing_pictures.part_names.values().any(|n| &**n == "word/header2.xml")); + assert_eq!(d.missing_pictures.media_parts.get("ok.png").map(String::as_str), Some("word/media/ok.png")); +} + +#[test] +fn a_picture_that_is_there_is_not_recorded() { + let body = blip(r#"r:embed="rIdOk""#); + let d = wordcraft_docx::read(&package(&body, &image_rel("rIdOk", "media/ok.png", false), None, &[("word/media/ok.png", png())])).unwrap(); + assert_eq!(pictures(&d), 1); + assert!(d.missing_pictures.list.is_empty()); +} + +#[test] +fn vml_image_data_links_are_recorded() { + let pict = + |attrs: &str| format!(r#""#); + let body = [pict(r#"r:id="rIdExt""#), pict(r#"r:href="rIdLink""#), pict(r#"r:id="rIdGone""#)].concat(); + let rels = [ + image_rel("rIdExt", "https://example.invalid/a.png", true), + image_rel("rIdLink", "https://example.invalid/b.png", true), + image_rel("rIdGone", "media/none.png", false), + ] + .concat(); + let d = wordcraft_docx::read(&package(&body, &rels, None, &[])).unwrap(); + let got: Vec<_> = missing(&d).into_iter().map(|(p, _, r)| (p, r)).collect(); + assert_eq!( + got, + [ + ("https://example.invalid/a.png".to_string(), MissingPictureReason::ExternalTarget), + ("https://example.invalid/b.png".to_string(), MissingPictureReason::LinkOnly), + ("word/media/none.png".to_string(), MissingPictureReason::MissingPart), + ] + ); +} + +#[test] +fn the_list_is_capped_and_the_document_is_still_equal() { + let n = wordcraft_doc::MAX_MISSING_PICTURES + 5; + let body: String = (0..n).map(|i| blip(&format!(r#"r:embed="r{i}""#))).collect(); + let rels: String = (0..n).map(|i| image_rel(&format!("r{i}"), &format!("media/gone{i}.png"), false)).collect(); + let d = wordcraft_docx::read(&package(&body, &rels, None, &[])).unwrap(); + assert_eq!(d.missing_pictures.list.len(), wordcraft_doc::MAX_MISSING_PICTURES); + assert!(d.missing_pictures.truncated); + // Not part of the document: `==` and the clone ignore it, and a save doesn't carry it. + let mut e = d.clone(); + e.missing_pictures = Default::default(); + assert_eq!(d, e); + assert!(!serde_json::to_string(&d).unwrap().contains("gone")); +} + +/// Every note of a part shares the one allocation of its source name, so a file with many notes +/// in a part with a long name does not keep a copy of the name per note. +#[test] +fn the_notes_of_a_part_share_one_copy_of_its_name() { + let notes: String = (1..=3).map(|i| format!(r#"n{i}"#)).collect(); + let rels = format!(r#""#); + let part = format!(r#"{notes}"#); + let d = wordcraft_docx::read(&package("x", &rels, None, &[("word/footnotes.xml", part.into_bytes())])).unwrap(); + let names: Vec<_> = d.missing_pictures.part_names.values().collect(); + assert!(names.len() >= 3, "{names:?}"); + assert!(names.iter().all(|n| &***n == "word/footnotes.xml")); + assert!(names.windows(2).all(|w| std::sync::Arc::ptr_eq(w[0], w[1])), "each note holds its own copy of the part name"); +} diff --git a/crates/engine/src/cmd/file.rs b/crates/engine/src/cmd/file.rs index c9d523c5d6..c676e2c802 100644 --- a/crates/engine/src/cmd/file.rs +++ b/crates/engine/src/cmd/file.rs @@ -94,9 +94,9 @@ pub fn specs() -> Vec { .pure(), CommandSpec::new("file.save", "Save", "File", save) .key("Mod+S / Shift+F12 / Alt+Shift+F2") - .params(r#"{"path"?: string, "password"?: string (encrypt this and later saves with it, as file.encrypt does), "markup"?: bool (Print Markup for a PDF, PNG, HTML, Markdown or text copy: comments shown; default file.printMarkup), "format"?: "wordml2003" (an .xml file as a Word 2003 XML Document) | "flatopc" (as a Word XML Document, the default; saving over a Word 2003 XML file keeps that format)}"#) + .params(r#"{"path"?: string, "password"?: string (encrypt this and later saves with it, as file.encrypt does), "markup"?: bool (Print Markup for a PDF, PNG, HTML, Markdown or text copy: comments shown; default file.printMarkup), "format"?: "wordml2003" (an .xml file as a Word 2003 XML Document) | "flatopc" (as a Word XML Document, the default; saving over a Word 2003 XML file keeps that format), "strict"?: bool (a .pdf or .png only: refuse, writing nothing, if a character can't be drawn or a picture is missing; see file.strictReport)}"#) .pure(), - CommandSpec::new("file.saveAs", "Save As", "File", save_as).key(super::mac_or("F12 / Mod+Shift+S", "F12 / Alt+F2")).params(r#"{"path": string, "password"?: string, "markup"?: bool, "format"?: string (as file.save)}"#).pure(), + CommandSpec::new("file.saveAs", "Save As", "File", save_as).key(super::mac_or("F12 / Mod+Shift+S", "F12 / Alt+F2")).params(r#"{"path": string, "password"?: string, "markup"?: bool, "format"?: string (as file.save), "strict"?: bool (as file.save)}"#).pure(), // Pure: the password must not land in Repeat or a recorded macro, and it isn't an edit // to undo; it marks the document as changed itself. CommandSpec::new("file.encrypt", "Encrypt with Password", "File › Info › Protect Document", encrypt) @@ -114,10 +114,13 @@ pub fn specs() -> Vec { crate::io::save_path_with(std::path::Path::new(&path), &s.doc, None, &opts).map_err(CmdError::Failed)?; Ok(json!({"path": path, "markup": opts.markup})) }) - .params(r#"{"path": string, "markup"?: bool (Print Markup: tracked changes marked up and each page scaled down beside the comments in balloons; default file.printMarkup)}"#) + .params(r#"{"path": string, "markup"?: bool (Print Markup: tracked changes marked up and each page scaled down beside the comments in balloons; default file.printMarkup), "strict"?: bool (refuse, writing nothing, if a character can't be drawn or a picture is missing; see file.strictReport)}"#) .pure(), CommandSpec::new("file.exportPng", "Export Page as PNG", "File › Export", export_png) - .params(r#"{"path": string, "page"?: n (1-based), "scale"?: px per pt, "markup"?: bool (the page with its comments in balloons beside it; default file.printMarkup)}"#) + .params(r#"{"path": string, "page"?: n (1-based), "scale"?: px per pt, "markup"?: bool (the page with its comments in balloons beside it; default file.printMarkup), "strict"?: bool (refuse, writing nothing, if a character on the page can't be drawn or a picture is missing; see file.strictReport)}"#) + .pure(), + CommandSpec::new("file.strictReport", "Strict Export Report", "File › Export", strict_report) + .params(r#"{"markup"?: bool (as file.exportPdf), "page"?: n (1-based: check that page of the page image file.exportPng writes; omit for every page of a PDF)} → {"schema": 1, "strict": "passed" | "failed", "missing_glyphs": [{"cp": "U+2D30", "count": n}], "missing_pictures": [{"part", "source_part": the story part it is in (word/header2.xml; background, body, header, footer, text box or "" where unknown), "reason": "missing_part" | "external_target" | "link_only" | "undecodable" | "unsupported_format"}], "truncated": bool (either list was cut at 256 entries), "total_missing_glyphs": n (distinct, past the cap too), "total_missing_pictures": n (past the cap too), "glyphs_truncated": bool, "pictures_truncated": bool}"#) .pure(), // A print setting, like Word's: not part of the document, so no undo step. CommandSpec::new("file.printMarkup", "Print Markup", "File › Print", |s, v| { @@ -369,6 +372,7 @@ fn save(s: &mut Session, v: &Value) -> CmdResult { return Ok(json!({"saved": false})); } }; + check_strict_output(p::bool(v, "strict").unwrap_or(false), &path.to_string_lossy())?; if let Some(pw) = v.get("password").filter(|pw| !pw.is_null()) { let Some(pw) = pw.as_str().filter(|pw| !pw.is_empty()) else { return Err(CmdError::Params("`password` must be a non-empty string (file.encrypt removes a password)".into())); @@ -450,9 +454,39 @@ pub fn export_options(s: &Session, v: &Value) -> crate::io::ExportOptions { comments: markup && s.prefs.markup.comments, word2003: p::str(v, "format") == Some(FORMAT_WORD_2003), numeral: s.view.numeral.resolve(crate::session::system_locale()), + strict: p::bool(v, "strict").unwrap_or(false), } } +/// `strict: true` is for a PDF or a page image only: any other output is a parameter error. +fn check_strict_output(strict: bool, path: &str) -> Result<(), CmdError> { + if strict && !crate::strict::is_strict_output(path) { + return Err(CmdError::Params("`strict` applies to a .pdf or .png output only".into())); + } + Ok(()) +} + +/// The strict report of a PDF export of the document (every page), or with `page` of the page +/// image `file.exportPng` writes. Pure: it reads the document and writes nothing. +fn strict_report(s: &mut Session, v: &Value) -> CmdResult { + let opts = export_options(s, v); + let report = match p::u64(v, "page") { + Some(n) => { + let page = crate::p::page_index(n); + let l = s.export_layout_with(opts.markup); + if l.pages.get(page).is_none() { + return Err(CmdError::Params(format!("no page {}", page + 1))); + } + crate::strict::strict_report(&s.doc, &l, &crate::strict::png_display(), Some(page..=page), wordcraft_layout::StrictOutput::Png) + } + None => { + let out = super::references::output_fields(&s.doc); + crate::strict::pdf_report(out.as_ref().unwrap_or(&s.doc), &opts) + } + }; + Ok(report.to_json()) +} + fn export_png(s: &mut Session, v: &Value) -> CmdResult { // No path: the app asks where to save, as file.exportPdf does (#2313). let Some(path) = p::str(v, "path") else { @@ -460,16 +494,22 @@ fn export_png(s: &mut Session, v: &Value) -> CmdResult { return Ok(json!({"exported": false})); }; let page = crate::p::page_index(p::u64(v, "page").unwrap_or(1)); + let strict = p::bool(v, "strict").unwrap_or(false); let scale = p::f32(v, "scale").unwrap_or(2.0).clamp(0.1, 8.0); let l = s.export_layout_with(export_options(s, v).markup); let pg = l.pages.get(page).ok_or_else(|| CmdError::Params(format!("no page {}", page + 1)))?; // A page image is the printed page: headers and footers at full strength. Dimming them is // the on-screen cue while editing the body, not part of the page. - let mut opts = wordcraft_render::RenderOptions::default(); - opts.display.dim_header = false; + let opts = wordcraft_render::RenderOptions { display: crate::strict::png_display(), ..Default::default() }; if !path.to_ascii_lowercase().ends_with(".png") { return Err(CmdError::Params("path must end in .png".into())); } + if strict { + let report = crate::strict::strict_report(&s.doc, &l, &opts.display, Some(page..=page), wordcraft_layout::StrictOutput::Png); + if !report.passed() { + return Err(CmdError::Failed(report.refusal())); + } + } let img = wordcraft_render::render_page(&s.doc, pg, scale, &opts); let png = img.to_png(); crate::io::write_file(path, &png).map_err(CmdError::Failed)?; diff --git a/crates/engine/src/io.rs b/crates/engine/src/io.rs index 583da36e89..6f196c7e4f 100644 --- a/crates/engine/src/io.rs +++ b/crates/engine/src/io.rs @@ -66,6 +66,10 @@ pub struct ExportOptions { pub word2003: bool, /// How digits look in PDFs and page images (the Numeral setting, `System` resolved). pub numeral: wordcraft_layout::NumeralMode, + /// A PDF or page image is written only if nothing in it is silently missing (a character no + /// face can draw, a picture that is missing or can't be decoded); else the save fails with + /// `strict: …` (#6005). Ignored for other formats. + pub strict: bool, } /// A document password: wiped from memory when dropped, never printed (`Debug` shows `***`). diff --git a/crates/engine/src/io_ext.rs b/crates/engine/src/io_ext.rs index e506ac510a..16c8136f8f 100644 --- a/crates/engine/src/io_ext.rs +++ b/crates/engine/src/io_ext.rs @@ -65,13 +65,7 @@ pub fn save(ext: &str, doc: &Document, opts: &crate::io::ExportOptions) -> Optio // Output shows the fields Word fills in as it prints (see `output_fields`). let out = crate::cmd::references::output_fields(doc); let doc = out.as_ref().unwrap_or(doc); - let pdf = wordcraft_pdf::PdfOptions { - include_markup: opts.markup, - comments: opts.markup && opts.comments, - numeral: opts.numeral, - ..Default::default() - }; - Some(wordcraft_pdf::export(doc, &pdf).map_err(|e| e.to_string())) + Some(crate::strict::export_pdf(doc, opts)) } other => wordcraft_formats::export_with(other, doc, &wordcraft_formats::ExportOptions { comments: opts.markup && opts.comments }), } @@ -85,6 +79,12 @@ pub fn render_png(doc: &Document, page: usize, scale: f32, opts: &crate::io::Exp let lo = wordcraft_layout::LayoutOptions { comment_balloons: opts.markup && opts.comments, numeral: opts.numeral, ..Default::default() }; let l = wordcraft_layout::layout(doc, &mut wordcraft_layout::LayoutCache::new(), &lo); let p = l.pages.get(page).ok_or_else(|| format!("no page {}", page + 1))?; + if opts.strict { + let report = crate::strict::strict_report(doc, &l, &crate::strict::png_display(), Some(page..=page), wordcraft_layout::StrictOutput::Png); + if !report.passed() { + return Err(report.refusal()); + } + } let img = wordcraft_render::render_page(doc, p, scale, &Default::default()); Ok(img.to_png()) } diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rs index 60728898b3..e5795e534e 100644 --- a/crates/engine/src/lib.rs +++ b/crates/engine/src/lib.rs @@ -25,6 +25,7 @@ pub mod sample; mod session; pub(crate) mod snapshot; pub mod speech; +pub mod strict; pub(crate) mod view; use std::collections::HashMap; diff --git a/crates/engine/src/strict.rs b/crates/engine/src/strict.rs new file mode 100644 index 0000000000..223366b81e --- /dev/null +++ b/crates/engine/src/strict.rs @@ -0,0 +1,233 @@ +//! Strict export (#6005): the report of what a PDF or PNG would leave out silently (characters no +//! face can draw, pictures that are missing or can't be decoded), and the refusal built from it. +//! +//! The check runs on the layout and display options the export itself uses, so what it reports is +//! what the output lacks. + +use std::ops::RangeInclusive; + +use serde_json::{Value, json}; +use wordcraft_doc::{Document, MAX_MISSING_PICTURES, MissingPicture}; +use wordcraft_layout::display::DisplayOptions; +use wordcraft_layout::{DocLayout, LayoutCache, StrictOutput}; + +use crate::io::ExportOptions; + +/// The report's JSON schema version; fields may be added without raising it. +pub const SCHEMA: u32 = 1; + +/// What a strict export found missing. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct StrictReport { + /// (code point, times drawn), sorted by code point. + pub glyphs: Vec<(char, u64)>, + pub pictures: Vec, + /// More distinct characters were missing than are listed. + pub glyphs_truncated: bool, + /// More pictures were missing than are listed. + pub pictures_truncated: bool, + /// Distinct characters found missing, those past the list's cap included. + pub total_glyphs: usize, + /// Pictures found missing, those past the list's cap included. + pub total_pictures: u64, +} + +impl StrictReport { + /// Nothing is missing: the strict export passes. + pub fn passed(&self) -> bool { + self.glyphs.is_empty() && self.pictures.is_empty() && !self.truncated() + } + + /// Either list was cut at its cap. + pub fn truncated(&self) -> bool { + self.glyphs_truncated || self.pictures_truncated + } + + /// The schema-1 JSON of the report. + pub fn to_json(&self) -> Value { + json!({ + "schema": SCHEMA, + "strict": if self.passed() { "passed" } else { "failed" }, + "missing_glyphs": self.glyphs.iter().map(|(c, n)| json!({"cp": code_point(*c), "count": n})).collect::>(), + "missing_pictures": self.pictures.iter().map(|p| json!({"part": p.part, "source_part": p.story, "reason": p.reason.code()})).collect::>(), + "truncated": self.truncated(), + "total_missing_glyphs": self.total_glyphs, + "total_missing_pictures": self.total_pictures, + "glyphs_truncated": self.glyphs_truncated, + "pictures_truncated": self.pictures_truncated, + }) + } + + /// The human lines (without the `strict: ` prefix): one for the characters, one for the + /// pictures; none when nothing is missing. + fn parts(&self) -> Vec { + let glyphs: Vec<(String, u64)> = self.glyphs.iter().map(|(c, n)| (code_point(*c), *n)).collect(); + let pictures: Vec<(String, String, String)> = + self.pictures.iter().map(|p| (p.part.clone(), story_label(&p.story).to_string(), p.reason.label().to_string())).collect(); + human_parts(&glyphs, &pictures, (self.glyphs_truncated, self.pictures_truncated), (self.total_glyphs as u64, self.total_pictures)) + } + + /// The human lines, each `strict: …`. + pub fn lines(&self) -> Vec { + self.parts().into_iter().map(|p| format!("strict: {p}")).collect() + } + + /// The refusal's message: `strict: …`, the lines joined. + pub fn refusal(&self) -> String { + format!("strict: {}", self.parts().join("; ")) + } +} + +/// The human text of a report: (code point text, count) and (part, story label, reason words). +fn human_parts(glyphs: &[(String, u64)], pictures: &[(String, String, String)], cut: (bool, bool), totals: (u64, u64)) -> Vec { + let (glyphs_cut, pictures_cut) = cut; + let mut out = Vec::new(); + if !glyphs.is_empty() { + let list: Vec = glyphs.iter().map(|(cp, n)| format!("{cp} (×{n})")).collect(); + let more = if glyphs_cut { ", …" } else { "" }; + let listed = glyphs.len() as u64; + let n = if glyphs_cut { totals.0.max(listed) } else { listed }; + let first = if n > listed { format!(" (first {listed} listed)") } else { String::new() }; + out.push(format!("{n} character{} cannot be drawn{first}: {}{more}", if n == 1 { "" } else { "s" }, list.join(", "))); + } + if !pictures.is_empty() { + let list: Vec = pictures + .iter() + .map(|(part, at, reason)| if at.is_empty() { format!("{part}: {reason}") } else { format!("{part} ({at}): {reason}") }) + .collect(); + let more = if pictures_cut { "; …" } else { "" }; + let listed = pictures.len() as u64; + let n = if pictures_cut { totals.1.max(listed) } else { listed }; + let first = if n > listed { format!(" (first {listed} listed)") } else { String::new() }; + out.push(format!("{n} picture(s) left out{first}: {}{more}", list.join("; "))); + } + out +} + +/// The `strict: …` human lines of a report in its JSON form (what `file.strictReport` returns). +/// Anything the JSON lacks is left out. +pub fn lines_from_json(report: &Value) -> Vec { + let text = |v: &Value, k: &str| v.get(k).and_then(Value::as_str).unwrap_or_default().to_string(); + let glyphs: Vec<(String, u64)> = report + .get("missing_glyphs") + .and_then(Value::as_array) + .map(|a| a.iter().map(|g| (text(g, "cp"), g.get("count").and_then(Value::as_u64).unwrap_or(0))).collect()) + .unwrap_or_default(); + let pictures: Vec<(String, String, String)> = report + .get("missing_pictures") + .and_then(Value::as_array) + .map(|a| a.iter().map(|p| (text(p, "part"), story_label(&text(p, "source_part")).to_string(), reason_label(&text(p, "reason")))).collect()) + .unwrap_or_default(); + let flag = |k: &str| report.get(k).and_then(Value::as_bool).unwrap_or(false); + // A report without the per-list flags (the top-level one alone) marks both lists. + let (glyphs_cut, pictures_cut) = match (report.get("glyphs_truncated"), report.get("pictures_truncated")) { + (None, None) => (flag("truncated"), flag("truncated")), + _ => (flag("glyphs_truncated"), flag("pictures_truncated")), + }; + let total = |k: &str, listed: usize| report.get(k).and_then(Value::as_u64).unwrap_or(listed as u64); + let totals = (total("total_missing_glyphs", glyphs.len()), total("total_missing_pictures", pictures.len())); + human_parts(&glyphs, &pictures, (glyphs_cut, pictures_cut), totals).into_iter().map(|p| format!("strict: {p}")).collect() +} + +/// The words for a JSON reason code; a code this version does not know is shown as it is. +fn reason_label(code: &str) -> String { + wordcraft_doc::MissingPictureReason::from_code(code).map_or_else(|| code.to_string(), |r| r.label().to_string()) +} + +/// `U+2D30`: upper-case hex, at least four digits. +fn code_point(c: char) -> String { + format!("U+{:04X}", c as u32) +} + +/// The part a picture is in, as the report names it: `header2.xml` for `word/header2.xml`. +fn story_label(story: &str) -> &str { + story.strip_prefix("word/").unwrap_or(story) +} + +/// The report of `doc` as drawn by `lay` with `display` into `output`, on `pages` (0-based; `None` +/// is every page). The watermark is checked with the page, and the pictures dropped when the +/// document was opened are added to those that can't be decoded now, capped like each. +pub fn strict_report( + doc: &Document, + lay: &DocLayout, + display: &DisplayOptions, + pages: Option>, + output: StrictOutput, +) -> StrictReport { + let mut missing = lay.missing_glyphs(doc, display, pages.clone()); + let any_page = lay.pages.iter().enumerate().any(|(n, _)| pages.as_ref().is_none_or(|r| r.contains(&n))); + if any_page && let Some(wm) = &doc.settings.watermark { + let max = match output { + StrictOutput::Pdf => wordcraft_pdf::WATERMARK_MAX_CHARS, + StrictOutput::Png => usize::MAX, + }; + missing.merge(wordcraft_layout::missing_watermark_glyphs(wm, max)); + } + let (undecodable, mut pictures_truncated, undecodable_total) = wordcraft_layout::undecodable_pictures(doc, lay, pages, display, output); + pictures_truncated |= doc.missing_pictures.truncated; + // The two lists overlap where the same picture is in both; those are counted once. + let mut overlap: u64 = 0; + let mut pictures: Vec = Vec::new(); + for p in doc.missing_pictures.list.iter().cloned().chain(undecodable) { + if pictures.contains(&p) { + overlap = overlap.saturating_add(1); + continue; + } + if pictures.len() >= MAX_MISSING_PICTURES { + pictures_truncated = true; + break; + } + pictures.push(p); + } + let total_pictures = doc.missing_pictures.total.saturating_add(undecodable_total).saturating_sub(overlap).max(pictures.len() as u64); + StrictReport { + glyphs: missing.chars, + pictures, + glyphs_truncated: missing.truncated, + pictures_truncated, + total_glyphs: missing.total, + total_pictures, + } +} + +/// The PDF export's options for `opts`: the one place they are built, shared by the export and +/// its strict check. +pub(crate) fn pdf_options(opts: &ExportOptions) -> wordcraft_pdf::PdfOptions { + wordcraft_pdf::PdfOptions { include_markup: opts.markup, comments: opts.markup && opts.comments, numeral: opts.numeral, ..Default::default() } +} + +/// What a page image shows: the display options `file.exportPng` renders with. +pub(crate) fn png_display() -> DisplayOptions { + // A page image is the printed page: headers and footers at full strength. + let mut opts = wordcraft_render::RenderOptions::default(); + opts.display.dim_header = false; + opts.display +} + +/// The strict report of a PDF export of `doc`: every page, laid out as the export lays it out. +pub(crate) fn pdf_report(doc: &Document, opts: &ExportOptions) -> StrictReport { + let pdf = pdf_options(opts); + let lay = wordcraft_layout::layout(doc, &mut LayoutCache::new(), &pdf.layout_options()); + strict_report(doc, &lay, &pdf.display_options(), None, StrictOutput::Pdf) +} + +/// The PDF of `doc`; with `opts.strict`, only if nothing is missing, laid out once for the check +/// and the export. +pub(crate) fn export_pdf(doc: &Document, opts: &ExportOptions) -> Result, String> { + let pdf = pdf_options(opts); + if !opts.strict { + return wordcraft_pdf::export(doc, &pdf).map_err(|e| e.to_string()); + } + let lay = wordcraft_layout::layout(doc, &mut LayoutCache::new(), &pdf.layout_options()); + let report = strict_report(doc, &lay, &pdf.display_options(), None, StrictOutput::Pdf); + if !report.passed() { + return Err(report.refusal()); + } + wordcraft_pdf::export_layout(doc, &lay, &pdf).map_err(|e| e.to_string()) +} + +/// Is `path` a PDF or PNG output (the only outputs a strict export applies to)? +pub(crate) fn is_strict_output(path: &str) -> bool { + let lower = path.to_ascii_lowercase(); + lower.ends_with(".pdf") || lower.ends_with(".png") +} diff --git a/crates/engine/src/tests.rs b/crates/engine/src/tests.rs index 79e6558de2..6439b3de9e 100644 --- a/crates/engine/src/tests.rs +++ b/crates/engine/src/tests.rs @@ -4683,7 +4683,7 @@ fn typed_paragraphs_are_capped_for_every_caller() { assert!(!body_texts(&s).iter().any(|p| p.contains('y'))); } -fn hostile_junk() -> [serde_json::Value; 8] { +fn hostile_junk() -> [serde_json::Value; 10] { [ json!(null), json!({}), @@ -4693,6 +4693,9 @@ fn hostile_junk() -> [serde_json::Value; 8] { json!({"page": 0, "x": 1e30, "y": -1e30, "extend": 1}), json!({"width": 1e-9, "height": 1e30, "top": -1e9, "left": 1e9}), json!({"section": {"pageW": -1e308, "pageH": 0, "marginTop": 1e30, "header": -5, "columns": {"count": 1000000, "space": -1e300, "widths": [[0, 0]]}}}), + // Strict export (#6005): the flag, and file.strictReport's `page`, with junk. + json!({"strict": true, "page": 0, "scale": 0, "markup": true}), + json!({"strict": "x", "page": 1e30, "scale": 1e-9, "markup": "x"}), ] } @@ -4789,6 +4792,9 @@ fn hostile_file_params_never_panic() { json!({"path": path, "password": "x", "format": "wordml2003", "markup": "x", "confirmDamaged": 1}), json!({"path": path, "password": 5, "format": 5, "markup": true}), json!({"path": path, "page": 0, "scale": 0, "width": -1e308, "alt": 5}), + json!({"path": path, "strict": true, "page": 1, "scale": 0.1}), + json!({"path": path, "strict": true, "page": 1e30, "markup": true, "format": "flatopc"}), + json!({"path": path, "strict": 5, "page": -1}), json!({"path": path, "data": "not base64!"}), json!({"path": path, "data": "AAAA"}), json!({"path": path, "pagesPerSheet": 7, "range": "x", "pages": "9-1,,", "parity": "x", "copies": 3, "paper": "x"}), diff --git a/crates/engine/tests/strict_export.rs b/crates/engine/tests/strict_export.rs new file mode 100644 index 0000000000..819085b1ec --- /dev/null +++ b/crates/engine/tests/strict_export.rs @@ -0,0 +1,322 @@ +//! Strict export (#6005): `strict: true` on the PDF and PNG exports refuses an output that would +//! silently lack something, before any byte is written; `file.strictReport` says what. +use serde_json::{Value, json}; +use wordcraft_doc::Document; +use wordcraft_engine::{CmdError, Session}; +use wordcraft_fonts::FontDb; + +/// Code points in unassigned plane 4 that no face of the host covers (as the layout tests find them). +fn uncovered_many(n: usize) -> Vec { + let db = FontDb::global(); + let primary = wordcraft_fonts::word::resolve(&Document::default().settings.minor_font, false, false).face; + let out: Vec = (0x40000u32..0x4FFF0) + .filter_map(char::from_u32) + .filter(|c| { + !primary.covers(*c) && db.fallback_styled(*c, primary.id(), false, false).is_none() && db.fallback_for(*c, primary.id()).is_none() + }) + .take(n) + .collect(); + assert_eq!(out.len(), n, "the host has faces covering all of plane 4"); + out +} + +fn uncovered() -> char { + uncovered_many(1)[0] +} + +fn dir(name: &str) -> std::path::PathBuf { + let d = std::env::temp_dir().join(format!("wc-strict-export-{name}-{}", std::process::id())); + std::fs::create_dir_all(&d).unwrap(); + d +} + +fn refused(r: Result) -> String { + match r { + Err(CmdError::Failed(m)) => { + assert!(m.starts_with("strict: "), "{m}"); + m + } + other => panic!("expected a strict refusal, got {other:?}"), + } +} + +#[test] +fn strict_refuses_an_uncovered_character_and_writes_nothing() { + let c = uncovered(); + let d = dir("refuse"); + let mut s = Session::new(Document::from_text(&format!("a{c}b"))); + let pdf = d.join("out.pdf"); + let png = d.join("out.png"); + let msg = refused(s.run("file.exportPdf", &json!({"path": pdf, "strict": true}))); + assert!(msg.contains(&format!("U+{:04X} (×1)", c as u32)), "{msg}"); + refused(s.run("file.save", &json!({"path": pdf, "strict": true}))); + refused(s.run("file.saveAs", &json!({"path": pdf, "strict": true}))); + refused(s.run("file.exportPng", &json!({"path": png, "strict": true}))); + refused(s.run("file.save", &json!({"path": png, "strict": true}))); + assert!(!pdf.exists() && !png.exists()); + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn a_refusal_leaves_an_existing_output_untouched() { + let c = uncovered(); + let d = dir("keep"); + let pdf = d.join("out.pdf"); + std::fs::write(&pdf, b"previous").unwrap(); + let mut s = Session::new(Document::from_text(&format!("a{c}"))); + refused(s.run("file.exportPdf", &json!({"path": pdf, "strict": true}))); + assert_eq!(std::fs::read(&pdf).unwrap(), b"previous"); + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn a_refused_save_leaves_the_stamps_path_and_dirty_flag_alone() { + let c = uncovered(); + let d = dir("stamp"); + let mut s = Session::new(Document::from_text(&format!("a{c}"))); + s.dirty = true; + let before = + (s.doc.core.modified.clone(), s.doc.core.created.clone(), s.doc.core.revision, s.doc.core.last_modified_by.clone(), s.doc.settings.zoom); + refused(s.run("file.save", &json!({"path": d.join("out.pdf"), "strict": true}))); + let after = + (s.doc.core.modified.clone(), s.doc.core.created.clone(), s.doc.core.revision, s.doc.core.last_modified_by.clone(), s.doc.settings.zoom); + assert_eq!(before, after); + assert!(s.dirty); + assert_eq!(s.path, None); + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn strict_passes_and_writes_when_nothing_is_missing() { + let d = dir("pass"); + let mut s = Session::new(Document::from_text("Hello, world.")); + for (cmd, file, magic) in [ + ("file.exportPdf", "a.pdf", &b"%PDF-"[..]), + ("file.save", "b.pdf", &b"%PDF-"[..]), + ("file.exportPng", "a.png", &b"\x89PNG"[..]), + ("file.save", "b.png", &b"\x89PNG"[..]), + ] { + let path = d.join(file); + s.run(cmd, &json!({"path": path, "strict": true})).unwrap(); + assert!(std::fs::read(&path).unwrap().starts_with(magic), "{file}"); + } + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn strict_passes_a_document_the_same_pdf_as_without_it() { + let d = dir("pdfsame"); + let mut s = Session::new(Document::from_text("Plain text")); + s.run("file.exportPdf", &json!({"path": d.join("a.pdf")})).unwrap(); + s.run("file.exportPdf", &json!({"path": d.join("b.pdf"), "strict": true})).unwrap(); + let (a, b) = (std::fs::read(d.join("a.pdf")).unwrap(), std::fs::read(d.join("b.pdf")).unwrap()); + assert_eq!(a.len(), b.len()); + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn strict_is_for_pdf_and_png_outputs_only() { + let d = dir("other"); + let mut s = Session::new(Document::from_text("Hello")); + for file in ["a.docx", "a.txt", "a.odt", "a.html"] { + let path = d.join(file); + let r = s.run("file.save", &json!({"path": path, "strict": true})); + assert!(matches!(r, Err(CmdError::Params(_))), "{file}: {r:?}"); + assert!(!path.exists(), "{file}"); + } + let r = s.run("file.exportPng", &json!({"path": d.join("a.jpg"), "strict": true})); + assert!(matches!(r, Err(CmdError::Params(_))), "{r:?}"); + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn strict_report_has_the_schema_1_shape() { + let c = uncovered(); + let mut s = Session::new(Document::from_text(&format!("a{c}b{c}"))); + let r = s.run("file.strictReport", &json!({})).unwrap(); + assert_eq!( + r, + json!({ + "schema": 1, + "strict": "failed", + "missing_glyphs": [{"cp": format!("U+{:04X}", c as u32), "count": 2}], + "missing_pictures": [], + "truncated": false, + "total_missing_glyphs": 1, + "total_missing_pictures": 0, + "glyphs_truncated": false, + "pictures_truncated": false, + }) + ); + let png = s.run("file.strictReport", &json!({"page": 1})).unwrap(); + assert_eq!(png, r); + assert!(matches!(s.run("file.strictReport", &json!({"page": 99})), Err(CmdError::Params(_)))); + + let mut ok = Session::new(Document::from_text("fine")); + let r = ok.run("file.strictReport", &json!({})).unwrap(); + assert_eq!( + r, + json!({"schema": 1, "strict": "passed", "missing_glyphs": [], "missing_pictures": [], "truncated": false, "total_missing_glyphs": 0, "total_missing_pictures": 0, "glyphs_truncated": false, "pictures_truncated": false}) + ); +} + +#[test] +fn strict_report_changes_nothing() { + let c = uncovered(); + let mut s = Session::new(Document::from_text(&format!("a{c}"))); + let before = serde_json::to_string(&s.doc).unwrap(); + s.run("file.strictReport", &json!({})).unwrap(); + assert_eq!(serde_json::to_string(&s.doc).unwrap(), before); + assert!(!s.dirty); +} + +fn garbage_background(d: &mut Document) { + d.media.insert("bg.bin".into(), std::sync::Arc::new(b"this is not a picture".to_vec())); + d.settings.page_picture = Some(wordcraft_doc::PagePicture { media: "bg.bin".into(), fill_type: "stretch".into(), title: String::new() }); +} + +#[test] +fn an_undecodable_page_background_picture_is_refused_for_pdf_and_png() { + let d = dir("background"); + let mut doc = Document::from_text("Hello"); + garbage_background(&mut doc); + let mut s = Session::new(doc); + let r = s.run("file.strictReport", &json!({})).unwrap(); + assert_eq!(r["strict"], "failed"); + assert_eq!(r["missing_pictures"][0], json!({"part": "bg.bin", "source_part": "background", "reason": "unsupported_format"})); + let msg = refused(s.run("file.exportPdf", &json!({"path": d.join("a.pdf"), "strict": true}))); + assert!(msg.contains("bg.bin (background)"), "{msg}"); + refused(s.run("file.exportPng", &json!({"path": d.join("a.png"), "strict": true}))); + refused(s.run("file.save", &json!({"path": d.join("b.png"), "strict": true}))); + assert!(!d.join("a.pdf").exists() && !d.join("a.png").exists() && !d.join("b.png").exists()); + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn a_decodable_page_background_picture_passes() { + let mut png = Vec::new(); + image::RgbaImage::from_pixel(4, 4, image::Rgba([200, 200, 255, 255])) + .write_to(&mut std::io::Cursor::new(&mut png), image::ImageFormat::Png) + .unwrap(); + let mut doc = Document::from_text("Hello"); + doc.media.insert("bg.png".into(), std::sync::Arc::new(png)); + doc.settings.page_picture = Some(wordcraft_doc::PagePicture { media: "bg.png".into(), fill_type: "stretch".into(), title: String::new() }); + let mut s = Session::new(doc); + assert_eq!(s.run("file.strictReport", &json!({})).unwrap()["strict"], "passed"); +} + +#[test] +fn a_picture_wider_than_the_pdf_draws_is_refused_for_pdf_only() { + // 40000 x 10: past the PDF's decode limit (30000), within what the page raster draws. + let mut png = Vec::new(); + image::RgbaImage::from_pixel(40_000, 10, image::Rgba([9, 9, 9, 255])) + .write_to(&mut std::io::Cursor::new(&mut png), image::ImageFormat::Png) + .unwrap(); + let mut doc = Document::from_text("Hello"); + doc.media.insert("wide.png".into(), std::sync::Arc::new(png)); + doc.settings.page_picture = Some(wordcraft_doc::PagePicture { media: "wide.png".into(), fill_type: "stretch".into(), title: String::new() }); + let mut s = Session::new(doc); + assert_eq!(s.run("file.strictReport", &json!({})).unwrap()["strict"], "failed", "PDF"); + assert_eq!(s.run("file.strictReport", &json!({"page": 1})).unwrap()["strict"], "passed", "PNG"); +} + +fn watermark(text: &str) -> Document { + let mut doc = Document::from_text("Hello"); + doc.settings.watermark = Some(wordcraft_doc::Watermark { text: text.into(), ..Default::default() }); + doc +} + +#[test] +fn watermark_text_no_font_can_draw_is_refused_and_latin_passes() { + let cs = uncovered_many(2); + let d = dir("watermark"); + let mut s = Session::new(watermark(&cs.iter().collect::())); + let r = s.run("file.strictReport", &json!({})).unwrap(); + assert_eq!(r["strict"], "failed"); + let cps: Vec = cs.iter().map(|c| format!("U+{:04X}", *c as u32)).collect(); + let got: Vec = r["missing_glyphs"].as_array().unwrap().iter().map(|g| g["cp"].as_str().unwrap().to_string()).collect(); + assert_eq!(got, cps); + refused(s.run("file.exportPdf", &json!({"path": d.join("w.pdf"), "strict": true}))); + refused(s.run("file.exportPng", &json!({"path": d.join("w.png"), "strict": true}))); + assert!(!d.join("w.pdf").exists() && !d.join("w.png").exists()); + + let mut ok = Session::new(watermark("CONFIDENTIAL")); + assert_eq!(ok.run("file.strictReport", &json!({})).unwrap()["strict"], "passed"); + ok.run("file.exportPdf", &json!({"path": d.join("ok.pdf"), "strict": true})).unwrap(); + ok.run("file.exportPng", &json!({"path": d.join("ok.png"), "strict": true})).unwrap(); + std::fs::remove_dir_all(d).unwrap(); +} + +#[test] +fn wordart_text_effects_keep_their_glyphs_in_the_check() { + let c = uncovered(); + let props = wordcraft_doc::CharProps { shadow: Some(true), outline: Some(true), ..Default::default() }; + let mut doc = Document::from_text(""); + doc.body = vec![wordcraft_doc::para_block(wordcraft_doc::Paragraph::with_text(&format!("a{c}b"), props))].into_iter().collect(); + let mut s = Session::new(doc); + let r = s.run("file.strictReport", &json!({})).unwrap(); + assert_eq!(r["missing_glyphs"][0]["cp"], format!("U+{:04X}", c as u32), "{r}"); +} + +#[test] +fn each_list_says_for_itself_that_it_was_cut() { + let many = uncovered_many(300); + let mut doc = Document::from_text(&many.iter().collect::()); + garbage_background(&mut doc); + let mut s = Session::new(doc); + let r = s.run("file.strictReport", &json!({})).unwrap(); + assert_eq!(r["missing_glyphs"].as_array().unwrap().len(), 256); + assert_eq!((r["truncated"].clone(), r["glyphs_truncated"].clone(), r["pictures_truncated"].clone()), (json!(true), json!(true), json!(false))); + let lines = wordcraft_engine::strict::lines_from_json(&r); + assert!(lines[0].ends_with(", …"), "{}", lines[0]); + assert!(lines[0].starts_with("strict: 300 characters cannot be drawn (first 256 listed): "), "{}", lines[0]); + assert!(!lines[1].contains('…'), "{}", lines[1]); + assert_eq!((r["total_missing_glyphs"].clone(), r["total_missing_pictures"].clone()), (json!(300), json!(1))); +} + +#[test] +fn a_truncated_report_carries_totals_past_the_cap() { + let many = uncovered_many(300); + let mut doc = Document::from_text(&many.iter().collect::()); + for n in 0..300 { + doc.missing_pictures.push(wordcraft_doc::MissingPicture { + part: format!("word/media/gone{n}.png"), + story: "word/document.xml".into(), + reason: wordcraft_doc::MissingPictureReason::MissingPart, + }); + // The same record again is one picture. + doc.missing_pictures.push(wordcraft_doc::MissingPicture { + part: format!("word/media/gone{n}.png"), + story: "word/document.xml".into(), + reason: wordcraft_doc::MissingPictureReason::MissingPart, + }); + } + let mut s = Session::new(doc); + let r = s.run("file.strictReport", &json!({})).unwrap(); + assert_eq!(r["missing_glyphs"].as_array().unwrap().len(), 256); + assert_eq!(r["missing_pictures"].as_array().unwrap().len(), 256); + assert_eq!(r["truncated"], true); + assert_eq!(r["total_missing_glyphs"], 300); + assert_eq!(r["total_missing_pictures"], 300); + assert_eq!(r["missing_pictures"][0]["reason"], "missing_part"); + assert_eq!(r["missing_pictures"][0]["source_part"], "word/document.xml"); +} + +#[test] +fn strict_off_gives_the_same_png_as_before_even_with_a_missing_character() { + let c = uncovered(); + let d = dir("pngsame"); + let mut s = Session::new(Document::from_text(&format!("Hello {c}"))); + let png: Vec> = [json!({}), json!({"strict": false})] + .into_iter() + .enumerate() + .map(|(i, mut p)| { + p["path"] = json!(d.join(format!("{i}.png"))); + s.run("file.exportPng", &p).unwrap(); + std::fs::read(d.join(format!("{i}.png"))).unwrap() + }) + .collect(); + assert!(png[0].starts_with(b"\x89PNG")); + assert_eq!(png[0], png[1]); + std::fs::remove_dir_all(d).unwrap(); +} diff --git a/crates/engine/tests/strict_pictures.rs b/crates/engine/tests/strict_pictures.rs new file mode 100644 index 0000000000..5a3df44299 --- /dev/null +++ b/crates/engine/tests/strict_pictures.rs @@ -0,0 +1,100 @@ +//! Pictures a layout draws that the exporters can't decode (#6005): a DOCX opened, laid out, and +//! asked which of its pictures would be grey boxes, and in which story. +use std::io::Write; +use wordcraft_doc::{Document, MissingPictureReason}; +use wordcraft_layout::display::DisplayOptions; +use wordcraft_layout::{LayoutCache, LayoutOptions, StrictOutput, layout, media_check, undecodable_pictures}; + +const NS: &str = r#"xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main" xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships" xmlns:wp="http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing" xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main" xmlns:pic="http://schemas.openxmlformats.org/drawingml/2006/picture""#; +const REL: &str = "http://schemas.openxmlformats.org/officeDocument/2006/relationships"; + +fn png() -> Vec { + let mut out = Vec::new(); + image::RgbaImage::from_pixel(4, 4, image::Rgba([20, 90, 200, 255])) + .write_to(&mut std::io::Cursor::new(&mut out), image::ImageFormat::Png) + .unwrap(); + out +} + +fn pic(rid: &str) -> String { + format!( + r#""# + ) +} + +/// A document with one header picture (`header_media`) and one body picture (`body_media`), the +/// two parts' bytes as given. +fn docx(header_media: &[u8], body_media: &[u8]) -> Vec { + let doc = format!( + r#"{}text"#, + pic("rIdB") + ); + let rels = format!( + r#""# + ); + let drels = format!( + r#""# + ); + let hrels = format!( + r#""# + ); + let types = r#""#; + let mut zip = zip::ZipWriter::new(std::io::Cursor::new(Vec::new())); + let hdr = format!(r#"{}"#, pic("rIdI")); + for (name, bytes) in [ + ("[Content_Types].xml", types.as_bytes()), + ("_rels/.rels", rels.as_bytes()), + ("word/document.xml", doc.as_bytes()), + ("word/_rels/document.xml.rels", drels.as_bytes()), + ("word/header2.xml", hdr.as_bytes()), + ("word/_rels/header2.xml.rels", hrels.as_bytes()), + ("word/media/image2.bin", header_media), + ("word/media/body.png", body_media), + ] { + zip.start_file(name, zip::write::SimpleFileOptions::default()).unwrap(); + zip.write_all(bytes).unwrap(); + } + zip.finish().unwrap().into_inner() +} + +fn report(d: &Document, pages: Option>) -> Vec<(String, String, MissingPictureReason)> { + let l = layout(d, &mut LayoutCache::new(), &LayoutOptions::default()); + let (list, truncated, total) = undecodable_pictures(d, &l, pages, &DisplayOptions::default(), StrictOutput::Pdf); + assert!(!truncated); + assert_eq!(total, list.len() as u64); + list.into_iter().map(|m| (m.part, m.story, m.reason)).collect() +} + +#[test] +fn a_header_picture_of_corrupt_bytes_is_reported_with_its_part_and_story() { + let d = wordcraft_docx::read(&docx(b"corrupt bytes, not a picture", &png())).unwrap(); + assert_eq!(report(&d, None), [("word/media/image2.bin".into(), "word/header2.xml".into(), MissingPictureReason::UnsupportedFormat)]); +} + +#[test] +fn a_truncated_picture_is_undecodable_not_unsupported() { + let mut cut = png(); + cut.truncate(cut.len() - 20); + let d = wordcraft_docx::read(&docx(&png(), &cut)).unwrap(); + assert_eq!(report(&d, None), [("word/media/body.png".into(), "word/document.xml".into(), MissingPictureReason::Undecodable)]); +} + +#[test] +fn a_real_png_in_a_header_is_not_reported() { + let d = wordcraft_docx::read(&docx(&png(), &png())).unwrap(); + assert_eq!(report(&d, None), []); +} + +#[test] +fn only_the_selected_pages_are_checked() { + let d = wordcraft_docx::read(&docx(b"corrupt", &png())).unwrap(); + assert_eq!(report(&d, Some(0..=0)).len(), 1); + assert_eq!(report(&d, Some(5..=9)), []); +} + +#[test] +fn the_check_names_what_the_exporters_draw() { + assert_eq!(media_check(&png(), StrictOutput::Pdf), Ok(())); + assert!(media_check(b"", StrictOutput::Pdf).is_err()); + assert!(media_check(&[1, 0, 0, 0, 88, 0, 0, 0], StrictOutput::Pdf).is_err(), "a metafile that doesn't parse is a grey box"); +} diff --git a/crates/geom/src/picture.rs b/crates/geom/src/picture.rs index 25bc311970..1d05eaa868 100644 --- a/crates/geom/src/picture.rs +++ b/crates/geom/src/picture.rs @@ -194,6 +194,87 @@ pub fn decode(bytes: &[u8], budget: u64) -> Result { probe(bytes)?.decode(budget) } +/// Why a picture can't be drawn, as strict export reports it (#6005). +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum DrawFailure { + /// A picture format we know, but it doesn't decode (damaged, truncated, too large). + Undecodable, + /// Bytes that are not a picture format we draw. + UnsupportedFormat, +} + +impl DrawFailure { + /// The words reports use. + pub fn label(self) -> &'static str { + match self { + DrawFailure::Undecodable => "undecodable", + DrawFailure::UnsupportedFormat => "unsupported format", + } + } +} + +impl From<&Refusal> for DrawFailure { + fn from(r: &Refusal) -> Self { + match r { + Refusal::NotAPicture | Refusal::Unsupported(_) => DrawFailure::UnsupportedFormat, + Refusal::TooLarge { .. } | Refusal::Broken(_) => DrawFailure::Undecodable, + } + } +} + +/// Decode a raster picture for drawing: the one decision the PDF export, the raster renderer and +/// strict export's check all take. The decoder is held to `max_side` pixels a side and the header +/// is charged against `budget` bytes before anything is allocated. A picture with no pixels or a +/// side past `max_side` is refused. +pub fn decode_for_drawing(bytes: &[u8], budget: u64, max_side: u32) -> Result { + let mut limits = image::Limits::default(); + limits.max_image_width = Some(max_side); + limits.max_image_height = Some(max_side); + let reader = || { + let mut r = image::ImageReader::new(Cursor::new(bytes)).with_guessed_format().map_err(|_| Refusal::NotAPicture)?; + r.limits(limits.clone()); + Ok::<_, Refusal>(r) + }; + // Header-only size check before anything is allocated: the decoder's buffer at its own pixel + // size (8 bytes for 16-bit RGBA) and the RGBA conversion are charged. + let (hw, hh, native) = { + let d = reader()?.into_decoder().map_err(broken)?; + let (w, h) = d.dimensions(); + (w, h, u64::from(d.color_type().bytes_per_pixel())) + }; + if !crate::decode_peak_fits(hw, hh, native, budget) { + return Err(Refusal::TooLarge { width: hw, height: hh }); + } + let img = reader()?.decode().map_err(broken)?; + let (w, h) = (img.width(), img.height()); + if w == 0 || h == 0 || w > max_side || h > max_side { + return Err(Refusal::Broken("it has no pixels or a side past the limit".into())); + } + Ok(img) +} + +/// Largest page-background picture the page raster draws, in pixels (hostile files). +pub const MAX_BACKGROUND_PIXELS: u64 = 64 * 1024 * 1024; + +/// The pixel size of a page-background picture the page raster draws (Design > Page Color > Fill +/// Effects), or None when it leaves the background out: not a raster picture it can read (a +/// metafile is not) or one with no pixels or more than [`MAX_BACKGROUND_PIXELS`]. The raster and +/// the strict check both ask this, so they can't drift. +pub fn background_size(bytes: &[u8]) -> Option<(u32, u32)> { + let (w, h) = image::ImageReader::new(Cursor::new(bytes)).with_guessed_format().ok()?.into_dimensions().ok()?; + (w > 0 && h > 0 && u64::from(w) * u64::from(h) <= MAX_BACKGROUND_PIXELS).then_some((w, h)) +} + +/// Whether a raster picture decodes for drawing, by the rule the exporters use ([`probe`] within +/// [`crate::MAX_DECODED_BYTES`], then [`decode_for_drawing`] with `max_side`). +pub fn check_for_drawing(bytes: &[u8], max_side: u32) -> Result<(), DrawFailure> { + let fail = |r: Refusal| DrawFailure::from(&r); + if !probe(bytes).map_err(fail)?.fits(crate::MAX_DECODED_BYTES) { + return Err(DrawFailure::Undecodable); + } + decode_for_drawing(bytes, crate::MAX_DECODED_BYTES, max_side).map(|_| ()).map_err(fail) +} + /// Bytes decoding a `width` × `height` picture holds at its peak: the decoder's output (`native` /// bytes a pixel) beside what the decoder keeps while it decodes (`held`), then the output beside /// its RGBA8 conversion, then that beside a premultiplied copy. None on overflow. @@ -340,6 +421,18 @@ mod tests { out } + #[test] + fn drawing_check_tells_unsupported_from_undecodable() { + let png = encode(image::RgbaImage::new(4, 4), ImageFormat::Png); + assert_eq!(check_for_drawing(&png, 100), Ok(())); + assert_eq!(check_for_drawing(b"junk", 100), Err(DrawFailure::UnsupportedFormat)); + assert_eq!(check_for_drawing(&png[..png.len() - 20], 100), Err(DrawFailure::Undecodable)); + // A side past the limit is refused, by the same function the exporters draw with. + assert_eq!(check_for_drawing(&png, 3), Err(DrawFailure::Undecodable)); + assert!(decode_for_drawing(&png, crate::MAX_DECODED_BYTES, 100).is_ok()); + assert_eq!(DrawFailure::Undecodable.label(), "undecodable"); + } + /// A PNG whose header declares `w` × `h` pixels, with no pixel data. fn png_header(w: u32, h: u32) -> Vec { let mut png = encode(image::RgbaImage::new(1, 1), ImageFormat::Png); diff --git a/crates/layout/src/display/lines.rs b/crates/layout/src/display/lines.rs index c1a319a757..3fabfbfb01 100644 --- a/crates/layout/src/display/lines.rs +++ b/crates/layout/src/display/lines.rs @@ -848,7 +848,7 @@ fn section_break_mark(start: SectionStart, x0: f32, x1: f32, base: f32, size: f3 fn equation(items: &[MItem], x: f32, base: f32, alpha: f32, screen: bool, out: &mut Vec) { for it in items { match it { - MItem::Glyphs { face, size, color, synth_bold, synth_italic, glyphs, text } => out.push(Draw::Glyphs { + MItem::Glyphs { face, size, color, synth_bold, synth_italic, glyphs, text, ranges } => out.push(Draw::Glyphs { face: *face, size: *size, glyphs: glyphs.iter().map(|(g, gx, gy)| (*g, x + gx, base - gy)).collect(), @@ -858,8 +858,7 @@ fn equation(items: &[MItem], x: f32, base: f32, alpha: f32, screen: bool, out: & synth_italic: *synth_italic, text: text.clone(), link: None, - // Equation text maps back through the font (no per-glyph ranges from the math layout). - ranges: Vec::new(), + ranges: ranges.clone(), text_bg: None, x_scale: 1.0, }), diff --git a/crates/layout/src/display/objects.rs b/crates/layout/src/display/objects.rs index 89211cf41a..ee574e735e 100644 --- a/crates/layout/src/display/objects.rs +++ b/crates/layout/src/display/objects.rs @@ -307,6 +307,7 @@ fn text_glyphs(doc: &Document, r: Rect, it: &GraphicItem, alpha: f32, k: f32) -> let mut out = Vec::with_capacity(runs.len()); for (face, s, shaped) in runs { let k = scale(face); + let ranges = crate::missing::cluster_ranges(s, &shaped); let mut glyphs = Vec::with_capacity(shaped.len()); for g in &shaped { glyphs.push((g.gid, pen + g.x_offset as f32 * k, baseline - g.y_offset as f32 * k)); @@ -324,7 +325,7 @@ fn text_glyphs(doc: &Document, r: Rect, it: &GraphicItem, alpha: f32, k: f32) -> synth_italic: false, text: s.to_string(), link: None, - ranges: Vec::new(), + ranges, text_bg: None, x_scale: 1.0, }); diff --git a/crates/layout/src/lib.rs b/crates/layout/src/lib.rs index 018e06b485..86e2664285 100644 --- a/crates/layout/src/lib.rs +++ b/crates/layout/src/lib.rs @@ -58,10 +58,12 @@ mod ctx; mod floats; mod flow; mod headers; +mod missing; mod notes; mod page; mod pass; mod placed; +mod strict; mod substituted; mod table_flow; @@ -82,6 +84,7 @@ use headers::MAX_HEADER_ITEMS; #[cfg(test)] use headers::placed_cost; use headers::placed_cost_by; +pub use missing::{MAX_MISSING_GLYPHS, MissingGlyphs, metafile_text_run, missing_watermark_glyphs, watermark_run}; pub use notes::DECOR_STORY; pub(crate) use notes::NOTE_SEP_W; use notes::{MAX_NOTE_PASSES, footnote_pages}; @@ -91,6 +94,7 @@ use page::item_bottom; use pass::layout_pass; pub use pass::now_ms; pub use placed::{HfSlot, Placed, turn_point, turn_rect, unturn_point}; +pub use strict::{MAX_PDF_PICTURE_SIDE, MAX_PNG_PICTURE_SIDE, StrictOutput, media_check, undecodable_pictures}; pub use substituted::MAX_SUBSTITUTED; /// How the document is viewed. diff --git a/crates/layout/src/math.rs b/crates/layout/src/math.rs index b5ba3d8897..d5b086f67a 100644 --- a/crates/layout/src/math.rs +++ b/crates/layout/src/math.rs @@ -41,6 +41,8 @@ pub enum MItem { /// Glyph id, x, y (baseline, up). glyphs: Vec<(u32, f32, f32)>, text: String, + /// Byte range of `text` each glyph shows (empty when `text` is: a glyph picked by id). + ranges: Vec>, }, /// A filled rectangle; `y` is its bottom edge. Rect { diff --git a/crates/layout/src/math/glyphs.rs b/crates/layout/src/math/glyphs.rs index 25dfe57b0d..6c6f45c5af 100644 --- a/crates/layout/src/math/glyphs.rs +++ b/crates/layout/src/math/glyphs.rs @@ -103,7 +103,16 @@ impl Ctx { b.glyph = Some((face, g, attach)); } if !glyphs.is_empty() { - b.items.push(MItem::Glyphs { face, size, color, synth_bold: synth.0, synth_italic: synth.1, glyphs, text: text.to_string() }); + b.items.push(MItem::Glyphs { + face, + size, + color, + synth_bold: synth.0, + synth_italic: synth.1, + glyphs, + text: text.to_string(), + ranges: crate::missing::cluster_ranges(text, &shaped), + }); } b } @@ -132,7 +141,16 @@ impl Ctx { if let Some(t) = &self.t { b.ic = t.italics.get(&gid).copied().unwrap_or(0.0) * k; } - b.items.push(MItem::Glyphs { face, size, color, synth_bold: false, synth_italic: false, glyphs: vec![(gid, 0.0, 0.0)], text: String::new() }); + b.items.push(MItem::Glyphs { + face, + size, + color, + synth_bold: false, + synth_italic: false, + glyphs: vec![(gid, 0.0, 0.0)], + text: String::new(), + ranges: Vec::new(), + }); b } @@ -227,7 +245,16 @@ impl Ctx { } else { b.w = pos * k; } - b.items.push(MItem::Glyphs { face: self.face, size, color: self.color, synth_bold: false, synth_italic: false, glyphs, text: String::new() }); + b.items.push(MItem::Glyphs { + face: self.face, + size, + color: self.color, + synth_bold: false, + synth_italic: false, + glyphs, + text: String::new(), + ranges: Vec::new(), + }); Some(b) } diff --git a/crates/layout/src/missing.rs b/crates/layout/src/missing.rs new file mode 100644 index 0000000000..6ea5c3bc20 --- /dev/null +++ b/crates/layout/src/missing.rs @@ -0,0 +1,309 @@ +//! The characters the output would draw as an empty box ([`DocLayout::missing_glyphs`]): glyph +//! id 0 in a [`Draw::Glyphs`] of the display list, traced back to its character through the +//! run's `text` and per-glyph `ranges`. Backs strict export (#6005). + +use std::collections::{BTreeMap, HashMap}; +use std::ops::{Range, RangeInclusive}; + +use wordcraft_doc::Document; +use wordcraft_fonts::ShapedGlyph; + +use crate::DocLayout; +use crate::display::{DisplayOptions, Draw, page_display}; +use crate::para::draws_something; + +/// Most distinct code points [`DocLayout::missing_glyphs`] lists: a hostile file full of unmapped +/// characters can't make the report unbounded. +pub const MAX_MISSING_GLYPHS: usize = 256; + +/// The missing glyphs of a render: characters it draws that no face in the font fallback chain can +/// draw (glyph id 0, an empty box). +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct MissingGlyphs { + /// (code point, times drawn), sorted by code point, at most [`MAX_MISSING_GLYPHS`] long. + pub chars: Vec<(char, u64)>, + /// More than [`MAX_MISSING_GLYPHS`] distinct code points were missing: the rest aren't listed. + pub truncated: bool, + /// How many distinct code points were missing, those past the cap included. + pub total: usize, + /// Every distinct code point found, kept only once the cap was passed (136 KiB), so merging + /// the watermark's can still count the distinct ones. + pub(crate) seen: Option>, +} + +impl MissingGlyphs { + /// Nothing is missing. + pub fn is_empty(&self) -> bool { + self.chars.is_empty() && !self.truncated + } + + /// A list of `chars` that fits under the cap: its total is its length. + pub fn from_listed(chars: Vec<(char, u64)>) -> Self { + let total = chars.len(); + Self { chars, truncated: false, total, seen: None } + } + + /// Add `other` (the watermark's characters): counts of a shared code point are summed, new + /// ones are listed while the cap allows, and the total counts each distinct one once. + pub fn merge(&mut self, other: MissingGlyphs) { + let mut all: BTreeMap = std::mem::take(&mut self.chars).into_iter().collect(); + let mut seen = self.seen.take(); + for (c, n) in other.chars { + if let Some(count) = all.get_mut(&c) { + *count = count.saturating_add(n); + } else if let Some(set) = seen.as_mut() { + set.insert(c); + } else if all.len() < MAX_MISSING_GLYPHS { + all.insert(c, n); + } else { + let mut set = CodePointSet::of(all.keys().copied()); + set.insert(c); + seen = Some(Box::new(set)); + } + } + if let Some(theirs) = other.seen { + seen.get_or_insert_with(|| Box::new(CodePointSet::of(all.keys().copied()))).union(&theirs); + } + self.truncated |= other.truncated || seen.is_some(); + self.total = seen.as_ref().map_or(all.len(), |s| s.len()); + self.chars = all.into_iter().collect(); + self.seen = seen; + } +} + +/// A set of code points as a bitset over the whole code point space: bounded at 136 KiB however +/// many distinct characters a hostile file holds. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct CodePointSet(Vec); + +impl CodePointSet { + fn empty() -> Self { + Self(vec![0; 0x11_0000 / 64]) + } + + fn of(chars: impl Iterator) -> Self { + let mut set = Self::empty(); + for c in chars { + set.insert(c); + } + set + } + + fn insert(&mut self, c: char) { + let n = c as usize; + if let Some(word) = self.0.get_mut(n / 64) { + *word |= 1 << (n % 64); + } + } + + fn union(&mut self, other: &Self) { + for (a, b) in self.0.iter_mut().zip(&other.0) { + *a |= b; + } + } + + fn len(&self) -> usize { + self.0.iter().map(|w| w.count_ones() as usize).sum() + } +} + +impl DocLayout { + /// The characters drawn as glyph id 0 on `pages` (0-based, inclusive; `None` is every page), + /// from the display list the PDF and PNG exports draw with `opts`: body, headers, footers, + /// text boxes, notes, table cells, charts, diagrams and equations. A character counts once per + /// glyph drawn for it. Default-ignorable and control characters never count (they draw + /// nothing), and a Private Use Area character counts only if it really drew as glyph 0. + /// Pages past the end are ignored. + pub fn missing_glyphs(&self, doc: &Document, opts: &DisplayOptions, pages: Option>) -> MissingGlyphs { + let mut found = Found::default(); + for (i, page) in self.pages.iter().enumerate() { + if pages.as_ref().is_some_and(|r| !r.contains(&i)) { + continue; + } + for d in page_display(doc, page, opts) { + found.walk(doc, &d); + } + // The page picture is drawn behind every page, outside the display list. + if let Some(pic) = &doc.settings.page_picture { + found.metafile_text(doc, &pic.media); + } + } + found.finish() + } +} + +/// How the watermark text is shaped for output, as the PDF export and the page raster both do: +/// its first `max_chars` characters in the font its name resolves to (no fallback face), so a +/// character that font lacks shapes to glyph id 0. Strict export checks these same glyphs +/// ([`missing_watermark_glyphs`]), so the check and the exporters can't drift. `None` for a +/// blank watermark. +pub fn watermark_run(wm: &wordcraft_doc::Watermark, max_chars: usize) -> Option<(wordcraft_fonts::FaceRef, String, Vec)> { + if wm.text.trim().is_empty() { + return None; + } + let face = wordcraft_fonts::word::resolve(&wm.font, false, false).face; + let text: String = wm.text.chars().take(max_chars).collect(); + let shaped = wordcraft_fonts::shape(&face, &text, &[], |c| c); + Some((face, text, shaped)) +} + +/// How a metafile text run is shaped for output, as the PDF export and the page raster both do: +/// in the face its font name resolves to (no fallback face), so a character that face lacks shapes +/// to glyph id 0. Strict export checks these same glyphs, so the check and the exporters can't +/// drift. +pub fn metafile_text_run(t: &wordcraft_metafile::Text) -> (wordcraft_fonts::FaceRef, Vec) { + let face = wordcraft_fonts::word::resolve(&t.font, t.bold, t.italic).face; + let shaped = wordcraft_fonts::shape(&face, &t.text, &[], |c| c); + (face, shaped) +} + +/// The characters of the watermark that draw as an empty box (glyph id 0), counted as +/// [`DocLayout::missing_glyphs`] counts them; sorted by code point. The watermark is on every +/// page, so it counts once. +pub fn missing_watermark_glyphs(wm: &wordcraft_doc::Watermark, max_chars: usize) -> MissingGlyphs { + let mut found = Found::default(); + if let Some((face, text, shaped)) = watermark_run(wm, max_chars) { + let ranges = cluster_ranges(&text, &shaped); + found.run(&face, shaped.iter().map(|g| g.gid), &text, &ranges); + } + found.finish() +} + +#[derive(Default)] +struct Found { + /// The text runs of each metafile picture met, by media key, shaped once: (face, text, ranges, glyph ids). + metafiles: HashMap>, + counts: BTreeMap, + truncated: bool, + /// Every distinct code point, started when the list passes its cap. + over: Option>, +} + +/// A metafile text run as drawn: its face, text, per-glyph ranges and glyph ids. +struct MetaRun { + face: wordcraft_fonts::FaceRef, + text: String, + ranges: Vec>, + gids: Vec, +} + +impl Found { + /// The text of the metafile picture `key` (nothing for a picture that is not a metafile, does + /// not parse or has no size: the exporters draw none of it), counted once per call. + fn metafile_text(&mut self, doc: &Document, key: &str) { + if !self.metafiles.contains_key(key) { + let runs = doc.media.get(key).map(|bytes| meta_runs(bytes)).unwrap_or_default(); + self.metafiles.insert(key.to_string(), runs); + } + let Some(runs) = self.metafiles.remove(key) else { return }; + for r in &runs { + self.run(&r.face, r.gids.iter().copied(), &r.text, &r.ranges); + } + self.metafiles.insert(key.to_string(), runs); + } + + fn finish(self) -> MissingGlyphs { + let total = self.over.as_ref().map_or(self.counts.len(), |s| s.len()); + MissingGlyphs { chars: self.counts.into_iter().collect(), truncated: self.truncated, total, seen: self.over } + } + + fn walk(&mut self, doc: &Document, d: &Draw) { + match d { + Draw::Glyphs { face, glyphs, text, ranges, .. } => self.run(face, glyphs.iter().map(|g| g.0), text, ranges), + Draw::Image { media, .. } => self.metafile_text(doc, media), + Draw::Figure { draws: items, .. } | Draw::Turned { items, .. } | Draw::Rotated { items, .. } | Draw::Clip { items, .. } => { + for d in items { + self.walk(doc, d); + } + } + _ => {} + } + } + + /// A run of glyphs of `face` (`gids`) showing `text`, glyph `k` showing `ranges[k]`. + fn run(&mut self, face: &wordcraft_fonts::FaceRef, gids: impl Iterator, text: &str, ranges: &[Range]) { + let glyph_count = gids.size_hint().1.unwrap_or(0); + let one_each = ranges.is_empty() && text.chars().count() == glyph_count; + let mut chars = text.chars(); + for (k, gid) in gids.enumerate() { + // A run without ranges (a list label) maps one character to one glyph, when the + // counts agree; else it can't be traced and isn't counted. + let own = chars.next(); + if gid != 0 { + continue; + } + let shown: Vec = if one_each { + own.into_iter().collect() + } else { + ranges.get(k).and_then(|r| text.get(r.clone())).map(|t| t.chars().collect()).unwrap_or_default() + }; + let drawn: Vec = shown.into_iter().filter(|c| draws_something(*c)).collect(); + // A cluster of several characters drew one empty box: it is the one the face + // lacks, not the letter beside it. + let lacking: Vec = drawn.iter().copied().filter(|c| !face.covers(*c)).collect(); + for c in if lacking.is_empty() { drawn } else { lacking } { + self.count(c); + } + } + } + + fn count(&mut self, c: char) { + if let Some(n) = self.counts.get_mut(&c) { + *n = n.saturating_add(1); + } else if let Some(set) = self.over.as_mut() { + set.insert(c); + } else if self.counts.len() < MAX_MISSING_GLYPHS { + self.counts.insert(c, 1); + } else { + self.truncated = true; + let mut set = CodePointSet::of(self.counts.keys().copied()); + set.insert(c); + self.over = Some(Box::new(set)); + } + } +} + +/// The text runs of a metafile picture that the exporters draw (a run whose layout fails is not). +fn meta_runs(bytes: &[u8]) -> Vec { + if !wordcraft_metafile::is_metafile(bytes) { + return Vec::new(); + } + let Ok(pic) = wordcraft_metafile::parse(bytes) else { return Vec::new() }; + if !pic.has_size() { + return Vec::new(); + } + let mut out = Vec::new(); + for item in &pic.items { + let wordcraft_metafile::Item::Text(t) = item else { continue }; + let (face, shaped) = metafile_text_run(t); + let glyphs: Vec<_> = shaped.iter().map(|g| (g.cluster, f64::from(g.x_advance), f64::from(g.x_offset), f64::from(g.y_offset))).collect(); + if t.layout(face.upem, face.ascent, face.descent, &glyphs).is_none() { + continue; + } + let ranges = cluster_ranges(&t.text, &shaped); + out.push(MetaRun { gids: shaped.iter().map(|g| g.gid).collect(), text: t.text.clone(), ranges, face }); + } + out +} + +/// The byte range of `text` each of `shaped` shows: from its cluster to the next cluster start in +/// the text (or its end), whichever order the glyphs are in. A ligature's glyph covers all its +/// characters; the glyphs of one cluster share its range. Ranges always lie on character +/// boundaries of `text`. +pub(crate) fn cluster_ranges(text: &str, shaped: &[ShapedGlyph]) -> Vec> { + let mut starts: Vec = shaped.iter().map(|g| g.cluster.min(text.len())).filter(|b| text.is_char_boundary(*b)).collect(); + starts.sort_unstable(); + starts.dedup(); + shaped + .iter() + .map(|g| { + let from = g.cluster.min(text.len()); + let from = if text.is_char_boundary(from) { from } else { (0..from).rev().find(|b| text.is_char_boundary(*b)).unwrap_or(0) }; + let to = match starts.binary_search(&from) { + Ok(i) => starts.get(i + 1).copied().unwrap_or(text.len()), + Err(_) => text.len(), + }; + from..to.max(from) + }) + .collect() +} diff --git a/crates/layout/src/strict.rs b/crates/layout/src/strict.rs new file mode 100644 index 0000000000..c9dcfec389 --- /dev/null +++ b/crates/layout/src/strict.rs @@ -0,0 +1,201 @@ +//! Pictures a layout draws that the exporters cannot decode, for strict export (#6005). +//! +//! The PDF export, the raster renderer and this check decide whether a picture is drawn the same +//! way (`wordcraft_geom::picture::decode_for_drawing`, and the metafile parser): what fails here +//! is what shows as a grey box there. + +use std::collections::{HashMap, HashSet}; +use std::ops::RangeInclusive; + +use wordcraft_doc::{Document, MAX_MISSING_PICTURES, MissingPicture, MissingPictureReason, StoryRef, picture_hash}; +use wordcraft_geom::picture::DrawFailure; + +use crate::display::{DisplayOptions, Draw, ItemDisplay}; +use crate::{DocLayout, Placed}; + +/// The widest or tallest picture the PDF export draws, pixels (its decoder limit). +pub const MAX_PDF_PICTURE_SIDE: u32 = 30_000; + +/// The widest or tallest picture the page raster draws, pixels (what a `Pixmap` holds). +pub const MAX_PNG_PICTURE_SIDE: u32 = u16::MAX as u32; + +/// The output a strict export checks, which decides what its exporter can draw. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum StrictOutput { + Pdf, + Png, +} + +impl StrictOutput { + /// The widest or tallest picture this output decodes. + pub fn max_picture_side(self) -> u32 { + match self { + StrictOutput::Pdf => MAX_PDF_PICTURE_SIDE, + StrictOutput::Png => MAX_PNG_PICTURE_SIDE, + } + } +} + +/// Whether `bytes` are a picture `output` draws: a metafile that parses, or a raster picture that +/// decodes within the decode budget. The same decision `wordcraft-pdf` and `wordcraft-render` +/// take, so a picture that fails here is drawn as a grey box (or not at all) by that output. +pub fn media_check(bytes: &[u8], output: StrictOutput) -> Result<(), DrawFailure> { + if wordcraft_metafile::is_metafile(bytes) && wordcraft_metafile::parse(bytes).is_ok_and(|p| p.has_size()) { + return Ok(()); + } + wordcraft_geom::picture::check_for_drawing(bytes, output.max_picture_side()) +} + +/// The pictures drawn on `pages` (0-based, `None` for every page) that fail [`media_check`] for +/// `output`, the page's background picture included (named `background`), one +/// per picture and story, at most [`MAX_MISSING_PICTURES`]; the flag says more were found, and the +/// count is how many distinct ones there were in all (saturating). +/// +/// `part` is the part the bytes were read from when the document knows it (a DOCX opened by the +/// reader), else the media key. `story` is the source part of the header, footer, note or comment +/// the picture is in (`word/header2.xml`), `word/document.xml` for the body; where the document +/// holds no source names (a document not opened from a DOCX) it is `header`, `footer`, +/// `footnote`, `endnote`, `comment`, `text box` or `body`. A picture in a text box or a shape is +/// named for the header, footer or body area it is placed in, not for the box. +pub fn undecodable_pictures( + doc: &Document, + layout: &DocLayout, + pages: Option>, + opts: &DisplayOptions, + output: StrictOutput, +) -> (Vec, bool, u64) { + let mut out: Vec = Vec::new(); + let mut truncated = false; + let mut total: u64 = 0; + // Hashes of the distinct ones past the cap, so the total counts each once. + let mut unlisted: HashSet = HashSet::new(); + // What `out` holds, for an O(1) test of "already listed". + let mut listed: HashSet = HashSet::new(); + let mut verdicts: HashMap> = HashMap::new(); + let range = pages.unwrap_or(0..=usize::MAX); + for (n, page) in layout.pages.iter().enumerate() { + if !range.contains(&n) { + continue; + } + let shown = ItemDisplay::new(doc, page); + let regions: [(&[Placed], Region); 3] = + [(&page.header, Region::Header(page.header_story)), (&page.footer, Region::Footer(page.footer_story)), (&page.items, Region::Body)]; + for (items, region) in regions { + for it in items { + let mut media: Vec = Vec::new(); + let draws = shown.draws(it, opts); + collect_media(&draws, &mut media); + for key in media { + let verdict = *verdicts.entry(key.clone()).or_insert_with(|| match doc.media.get(&key) { + Some(bytes) => media_check(bytes, output).err(), + None => Some(DrawFailure::Undecodable), + }); + let Some(failure) = verdict else { continue }; + let reason = match failure { + DrawFailure::Undecodable => MissingPictureReason::Undecodable, + DrawFailure::UnsupportedFormat => MissingPictureReason::UnsupportedFormat, + }; + let part = doc.missing_pictures.media_parts.get(&key).cloned().unwrap_or(key); + let item = MissingPicture { part, story: story_name(doc, region, it), reason }; + if listed.contains(&item) { + continue; + } + if out.len() >= MAX_MISSING_PICTURES { + truncated = true; + if unlisted.insert(picture_hash(&item)) { + total = total.saturating_add(1); + } + continue; + } + total = total.saturating_add(1); + listed.insert(item.clone()); + out.push(item); + } + } + } + } + // The page picture is drawn behind every page, outside the display list. + if let Some(pic) = &doc.settings.page_picture + && layout.pages.iter().enumerate().any(|(n, _)| range.contains(&n)) + { + // The page raster leaves out a background it can't read as a raster picture (a metafile), + // which the PDF draws as vector. + let failure = match doc.media.get(&pic.media) { + Some(bytes) => media_check(bytes, output).err().or_else(|| { + (output == StrictOutput::Png && wordcraft_geom::picture::background_size(bytes).is_none()).then_some(DrawFailure::UnsupportedFormat) + }), + None => Some(DrawFailure::Undecodable), + }; + if let Some(failure) = failure { + let reason = match failure { + DrawFailure::Undecodable => MissingPictureReason::Undecodable, + DrawFailure::UnsupportedFormat => MissingPictureReason::UnsupportedFormat, + }; + let part = doc.missing_pictures.media_parts.get(&pic.media).cloned().unwrap_or_else(|| pic.media.clone()); + let item = MissingPicture { part, story: "background".into(), reason }; + if !listed.contains(&item) { + if out.len() >= MAX_MISSING_PICTURES { + truncated = true; + if unlisted.insert(picture_hash(&item)) { + total = total.saturating_add(1); + } + } else { + total = total.saturating_add(1); + listed.insert(item.clone()); + out.push(item); + } + } + } + } + (out, truncated, total) +} + +#[derive(Clone, Copy)] +enum Region { + Header(Option), + Footer(Option), + Body, +} + +fn collect_media(draws: &[Draw], out: &mut Vec) { + for d in draws { + match d { + Draw::Image { media, .. } => out.push(media.clone()), + Draw::Figure { draws, .. } => collect_media(draws, out), + Draw::Turned { items, .. } | Draw::Rotated { items, .. } | Draw::Clip { items, .. } => collect_media(items, out), + _ => {} + } + } +} + +/// The story a placed item is in, named as [`undecodable_pictures`] says. +fn story_name(doc: &Document, region: Region, it: &Placed) -> String { + let names = &doc.missing_pictures; + let known = |id: Option| id.and_then(|id| names.part_names.get(&id)).map(|n| n.to_string()); + match region { + Region::Header(id) => known(id).unwrap_or_else(|| "header".into()), + Region::Footer(id) => known(id).unwrap_or_else(|| "footer".into()), + Region::Body => { + let story = match it { + Placed::Lines { story, .. } | Placed::Image { story, .. } | Placed::Graphic { story, .. } => *story, + _ => StoryRef::Body, + }; + match story { + StoryRef::Body if names.main.is_empty() => "body".into(), + StoryRef::Body => names.main.clone(), + StoryRef::Part(id) => known(Some(id)).unwrap_or_else(|| { + use wordcraft_doc::PartKind::*; + match doc.parts.get(&id).map(|p| p.kind) { + Some(Header) => "header", + Some(Footer) => "footer", + Some(Footnote) => "footnote", + Some(Endnote) => "endnote", + Some(Comment) => "comment", + Some(TextBox) | None => "text box", + } + .into() + }), + } + } + } +} diff --git a/crates/layout/src/tests.rs b/crates/layout/src/tests.rs index 6db3eaa84c..98a75854ad 100644 --- a/crates/layout/src/tests.rs +++ b/crates/layout/src/tests.rs @@ -13,6 +13,7 @@ mod float_top; mod grid; mod ligature; pub(crate) mod linear; +mod missing_glyphs; mod nbsp; mod nested_cant_split; mod numeral; diff --git a/crates/layout/src/tests/missing_glyphs.rs b/crates/layout/src/tests/missing_glyphs.rs new file mode 100644 index 0000000000..1a9444f346 --- /dev/null +++ b/crates/layout/src/tests/missing_glyphs.rs @@ -0,0 +1,244 @@ +//! [`DocLayout::missing_glyphs`] (strict export, #6005): glyph id 0 in the display list, named by +//! its character, wherever the text is drawn. + +use std::sync::Arc; + +use wordcraft_doc::graphic::{Graphic, GraphicItem, GraphicKind, TextAlign}; +use wordcraft_doc::para::InlineObject; +use wordcraft_doc::props::Rgb; +use wordcraft_doc::{Block, Document, Path, Pos, StoryRef, Table}; +use wordcraft_fonts::FontDb; + +use super::{footnote, lay, page_float, text_box_at}; +use crate::display::DisplayOptions; +use crate::{DocLayout, MAX_MISSING_GLYPHS, MissingGlyphs}; + +/// Code points in unassigned plane 4 that no face of the host covers, checked against the body +/// font and the whole fallback chain (so the test never guesses). +fn uncovered(n: usize) -> Vec { + crate::test_fonts(); + let db = FontDb::global(); + let primary = wordcraft_fonts::word::resolve(&Document::default().settings.minor_font, false, false).face; + let out: Vec = (0x40000u32..0x4FFF0) + .filter_map(char::from_u32) + .filter(|c| { + !primary.covers(*c) && db.fallback_styled(*c, primary.id(), false, false).is_none() && db.fallback_for(*c, primary.id()).is_none() + }) + .take(n) + .collect(); + assert_eq!(out.len(), n, "the host has faces covering all of plane 4"); + out +} + +fn missing(d: &Document, l: &DocLayout) -> MissingGlyphs { + l.missing_glyphs(d, &DisplayOptions::default(), None) +} + +fn chars(m: &MissingGlyphs) -> Vec { + m.chars.iter().map(|(c, _)| *c).collect() +} + +fn hf_part(d: &mut Document, kind: wordcraft_doc::PartKind, text: &str) -> u32 { + d.add_part(kind, vec![wordcraft_doc::para_block(wordcraft_doc::Paragraph::with_text(text, Default::default()))]) +} + +fn chart_with(d: &mut Document, label: &str) { + let items = vec![GraphicItem::Text { + rect: [0.0, 100.0, 216.0, 20.0], + text: label.into(), + size: 10.0, + color: Rgb::BLACK, + bold: false, + align: TextAlign::Center, + font: None, + }]; + let graphic = Graphic { kind: GraphicKind::Chart, items, ..Default::default() }; + let obj = InlineObject::Graphic { w: 216.0, h: 144.0, alt: String::new(), float: Default::default(), graphic: Arc::new(graphic) }; + d.insert_object(&Pos::body(0, 0), obj, &Default::default()).unwrap(); +} + +fn equation_with(d: &mut Document, linear: &str) { + let math = wordcraft_doc::math::Math { nodes: wordcraft_doc::math::parse_linear(linear), ..Default::default() }; + let n = d.para_mut(StoryRef::Body, &Path::top(0)).unwrap().len(); + d.para_mut(StoryRef::Body, &Path::top(0)) + .unwrap() + .insert_object(n, InlineObject::Equation { linear: linear.into(), display: false, math }, &Default::default()) + .unwrap(); +} + +#[test] +fn nothing_is_missing_in_plain_text() { + let d = Document::from_text("Hello, world."); + assert!(missing(&d, &lay(&d)).is_empty()); +} + +#[test] +fn a_character_no_face_covers_is_counted_once_per_occurrence_in_the_body() { + let c = uncovered(1)[0]; + let d = Document::from_text(&format!("a{c}b{c}{c}c")); + let m = missing(&d, &lay(&d)); + assert_eq!(m, MissingGlyphs::from_listed(vec![(c, 3)])); +} + +#[test] +fn a_missing_glyph_is_found_in_a_header() { + let c = uncovered(1)[0]; + let mut d = Document::from_text("Body"); + d.last_section.headers.default = Some(hf_part(&mut d, wordcraft_doc::PartKind::Header, &format!("H{c}{c}"))); + assert_eq!(missing(&d, &lay(&d)).chars, vec![(c, 2)]); +} + +#[test] +fn a_missing_glyph_is_found_in_a_footer() { + let c = uncovered(1)[0]; + let mut d = Document::from_text("Body"); + d.last_section.footers.default = Some(hf_part(&mut d, wordcraft_doc::PartKind::Footer, &format!("F{c}"))); + assert_eq!(missing(&d, &lay(&d)).chars, vec![(c, 1)]); +} + +#[test] +fn a_missing_glyph_is_found_in_a_footnote() { + let c = uncovered(1)[0]; + let mut d = Document::from_text("Body text."); + footnote(&mut d, &Pos::body(0, 4), &format!("Note {c}")); + assert_eq!(missing(&d, &lay(&d)).chars, vec![(c, 1)]); +} + +#[test] +fn a_missing_glyph_is_found_in_a_text_box() { + let c = uncovered(1)[0]; + let mut d = Document::from_text("Body text."); + text_box_at(&mut d, &Pos::body(0, 0), &format!("Box {c}{c}{c}"), 120.0, 40.0, page_float(300.0, 300.0)); + assert_eq!(missing(&d, &lay(&d)).chars, vec![(c, 3)]); +} + +#[test] +fn a_missing_glyph_is_found_in_a_table_cell() { + let c = uncovered(1)[0]; + let mut d = Document::from_text("before"); + let mut t = Table::new(1, 2, 300.0); + t.rows[0].cells[1].blocks = vec![wordcraft_doc::para_block(wordcraft_doc::Paragraph::with_text(&format!("cell {c}"), Default::default()))]; + d.insert_block(StoryRef::Body, &Path::top(1), Block::Table(t)).unwrap(); + assert_eq!(missing(&d, &lay(&d)).chars, vec![(c, 1)]); +} + +#[test] +fn a_missing_glyph_is_found_in_a_chart_label() { + let c = uncovered(1)[0]; + let mut d = Document::from_text("Chart"); + chart_with(&mut d, &format!("Q{c}Q{c}")); + assert_eq!(missing(&d, &lay(&d)).chars, vec![(c, 2)]); +} + +#[test] +fn a_chart_label_run_names_each_glyphs_characters() { + use crate::display::Draw; + let mut d = Document::from_text("Chart"); + chart_with(&mut d, "Q1 total"); + let l = lay(&d); + fn runs(d: &Draw, out: &mut Vec<(String, usize, usize)>) { + match d { + Draw::Glyphs { text, glyphs, ranges, .. } if text == "Q1 total" || !ranges.is_empty() => { + out.push((text.clone(), glyphs.len(), ranges.len())); + assert!(ranges.iter().all(|r| text.get(r.clone()).is_some()), "ranges on character boundaries"); + } + Draw::Figure { draws, .. } => draws.iter().for_each(|x| runs(x, out)), + _ => {} + } + } + let mut found = Vec::new(); + for x in crate::display::page_display(&d, &l.pages[0], &DisplayOptions::default()) { + runs(&x, &mut found); + } + assert!(found.iter().any(|(t, g, r)| t == "Q1 total" && g == r), "{found:?}"); +} + +#[test] +fn a_missing_glyph_is_found_in_an_equation() { + let c = uncovered(1)[0]; + let mut d = Document::from_text("Eq: "); + equation_with(&mut d, &format!("x+{c}")); + let m = missing(&d, &lay(&d)); + assert_eq!(m.chars, vec![(c, 1)], "{m:?}"); +} + +#[test] +fn joiners_and_variation_selectors_are_never_counted() { + let d = Document::from_text("a\u{200D}b\u{200C}c\u{FE0F}d\u{FE0E}e\u{00AD}f\u{200E}g\u{2028}h\ti"); + assert!(missing(&d, &lay(&d)).is_empty()); + // A ZWJ emoji sequence: the joiners and selectors in it never appear, whatever the host's + // emoji face covers. + let d = Document::from_text("\u{1F469}\u{200D}\u{1F4BB} \u{2764}\u{FE0F} \u{1F44D}\u{1F3FD}"); + let m = missing(&d, &lay(&d)); + assert!(!chars(&m).iter().any(|c| matches!(c, '\u{200D}' | '\u{FE0F}' | '\u{FE0E}')), "{m:?}"); +} + +#[test] +fn a_selector_beside_a_missing_character_is_not_counted_with_it() { + let c = uncovered(1)[0]; + let d = Document::from_text(&format!("{c}\u{FE0F}")); + assert_eq!(chars(&missing(&d, &lay(&d))), vec![c]); +} + +#[test] +fn the_list_is_sorted_and_capped_at_256_distinct_code_points() { + let all = uncovered(MAX_MISSING_GLYPHS + 44); + let text: String = all.iter().rev().collect(); + let d = Document::from_text(&text); + let m = missing(&d, &lay(&d)); + assert!(m.truncated); + assert_eq!(m.chars.len(), MAX_MISSING_GLYPHS); + assert_eq!(m.total, MAX_MISSING_GLYPHS + 44, "the total counts the unlisted ones too"); + assert!(m.chars.windows(2).all(|w| w[0].0 < w[1].0), "sorted by code point"); + // Exactly the cap's worth is no truncation. + let d = Document::from_text(&all.iter().take(MAX_MISSING_GLYPHS).collect::()); + let m = missing(&d, &lay(&d)); + assert!(!m.truncated); + assert_eq!(m.chars.len(), MAX_MISSING_GLYPHS); + assert_eq!(m.total, MAX_MISSING_GLYPHS); +} + +#[test] +fn the_page_filter_selects_pages() { + let [x, y] = <[char; 2]>::try_from(uncovered(2)).unwrap(); + let d = Document::from_text(&format!("first {x}\n{}last {y}", "filler line\n".repeat(150))); + let l = lay(&d); + let last = l.pages.len() - 1; + assert!(last >= 1, "the document runs over pages"); + let on = |r| l.missing_glyphs(&d, &DisplayOptions::default(), r); + assert_eq!(on(None).chars, vec![(x.min(y), 1), (x.max(y), 1)]); + assert_eq!(on(Some(0..=0)).chars, vec![(x, 1)]); + assert_eq!(on(Some(last..=last)).chars, vec![(y, 1)]); + assert!(on(Some(last + 1..=last + 9)).is_empty()); + #[allow(clippy::reversed_empty_ranges)] + let backwards = Some(1..=0); + assert!(on(backwards).is_empty()); +} + +#[test] +fn an_equation_run_names_each_glyphs_characters() { + use crate::display::Draw; + let mut d = Document::from_text("Eq: "); + equation_with(&mut d, "x+ab"); + let l = lay(&d); + fn runs(d: &Draw, out: &mut Vec<(String, usize, usize)>) { + match d { + Draw::Glyphs { text, glyphs, ranges, .. } if !text.is_empty() => { + assert!(ranges.iter().all(|r| text.get(r.clone()).is_some()), "ranges on character boundaries"); + out.push((text.clone(), glyphs.len(), ranges.len())); + } + Draw::Figure { draws, .. } | Draw::Clip { items: draws, .. } | Draw::Rotated { items: draws, .. } => { + draws.iter().for_each(|x| runs(x, out)) + } + _ => {} + } + } + let mut found = Vec::new(); + for x in crate::display::page_display(&d, &l.pages[0], &DisplayOptions::default()) { + runs(&x, &mut found); + } + // The equation's runs (the body text "Eq: " is a run too) each carry one range per glyph. + // The letters are the math italic ones where the font has them, else plain. + assert!(found.iter().any(|(t, g, _)| (t.contains('\u{1D44E}') || t == "ab") && *g == 2), "{found:?}"); + assert!(found.iter().all(|(_, g, r)| g == r), "{found:?}"); +} diff --git a/crates/pdf/src/export.rs b/crates/pdf/src/export.rs index 78e575c5b5..81b7b94d20 100644 --- a/crates/pdf/src/export.rs +++ b/crates/pdf/src/export.rs @@ -3,8 +3,8 @@ use super::metadata::metadata; use super::options::{PdfError, PdfOptions}; use super::paint::{clamp_side, ok, rect_path}; use super::{ - DocLayout, Document, FillRule, HashMap, HashSet, KrillaError, LayoutCache, LayoutOptions, Page, PageSettings, Rect, Size, Transform, WriteError, - Writer, imposition, krilla, layout, write, + DocLayout, Document, FillRule, HashMap, HashSet, KrillaError, LayoutCache, Page, PageSettings, Rect, Size, Transform, WriteError, Writer, + imposition, krilla, layout, write, }; /// Lay out `doc` and write it as PDF. Without markup the layout is the final text: tracked @@ -12,8 +12,7 @@ use super::{ /// markup area with the comment balloons. pub fn export(doc: &Document, opts: &PdfOptions) -> Result, PdfError> { write::guard(|| { - let lo = LayoutOptions { hide_deleted: !opts.include_markup, comment_balloons: opts.comments, numeral: opts.numeral, ..Default::default() }; - let lay = layout(doc, &mut LayoutCache::new(), &lo); + let lay = layout(doc, &mut LayoutCache::new(), &opts.layout_options()); export_layout_inner(doc, &lay, opts) }) } diff --git a/crates/pdf/src/exporter.rs b/crates/pdf/src/exporter.rs index 67737b3394..77fe80c348 100644 --- a/crates/pdf/src/exporter.rs +++ b/crates/pdf/src/exporter.rs @@ -5,10 +5,10 @@ use super::paint::{append_path, fill, grown, norm, ok, paint_fill, pdf_line_cap, use super::picture::{Vector, build_vector, load_image, paint_of, vector_picture}; use super::tags::{Role, TagEntry, figure_alt, graphic_alt, holds_picture, holds_text}; use super::{ - Annotation, Arc, Artifact, ArtifactType, ClKind, ContentTag, DisplayOptions, DocLayout, DocPath, Document, Draw, FaceRef, Fill, FillRule, Font, - HashMap, HashSet, Identifier, Image, InlineObject, ItemDisplay, KrillaGlyph, LineCap, LineEnd, LineJoin, LineStyle, LinkAnnotation, Node, - NonZeroU16, Outline, OutlineNode, Page, PathBuilder, Placed, Point, Rect, Rgb, ShapeKind, Size, SpanTag, StoryRef, Stroke, StrokeDash, Surface, - Tag, TagGroup, TagKind, TagTree, Target, Transform, WriteError, XyzDestination, krilla, rgb, write, + Annotation, Arc, Artifact, ArtifactType, ClKind, ContentTag, DocLayout, DocPath, Document, Draw, FaceRef, Fill, FillRule, Font, HashMap, HashSet, + Identifier, Image, InlineObject, ItemDisplay, KrillaGlyph, LineCap, LineEnd, LineJoin, LineStyle, LinkAnnotation, Node, NonZeroU16, Outline, + OutlineNode, Page, PathBuilder, Placed, Point, Rect, Rgb, ShapeKind, Size, SpanTag, StoryRef, Stroke, StrokeDash, Surface, Tag, TagGroup, + TagKind, TagTree, Target, Transform, WriteError, XyzDestination, krilla, rgb, write, }; use wordcraft_doc::picfx::PicFx; @@ -240,14 +240,7 @@ impl Exporter<'_> { let area = page.body; self.tagged(s, Role::Artifact(ArtifactType::Watermark), None, |me, s| me.watermark(s, &wm, w, h, area)); } - let dopts = DisplayOptions { - marks: false, - dim_header: false, - dim_body: false, - markup: self.opts.include_markup, - placeholders: false, - ..Default::default() - }; + let dopts = self.opts.display_options(); // What the items share (the authors' colours, the clipped cells) is worked out once per page. let shown = ItemDisplay::new(self.doc, page); for it in page.header.iter().chain(page.footer.iter()) { @@ -356,10 +349,7 @@ impl Exporter<'_> { /// the baseline) on the box's bottom; the box centred on the margins (`area`, else the page) /// and, when diagonal, turned 45° counterclockwise. fn watermark(&mut self, s: &mut Surface, wm: &wordcraft_doc::Watermark, w: f32, h: f32, area: Rect) { - let r = wordcraft_fonts::word::resolve(&wm.font, false, false); - let face = r.face; - let text: String = wm.text.chars().take(200).collect(); - let shaped = wordcraft_fonts::shape(&face, &text, &[], |c| c); + let Some((face, text, shaped)) = wordcraft_layout::watermark_run(wm, super::WATERMARK_MAX_CHARS) else { return }; let upem = face.upem.max(1.0) as f32; let raw_w: f32 = shaped.iter().map(|g| g.x_advance as f32).sum::() / upem; if raw_w <= 0.0 || !raw_w.is_finite() { diff --git a/crates/pdf/src/exporter/text.rs b/crates/pdf/src/exporter/text.rs index 74c15fc763..d1eeeb74b3 100644 --- a/crates/pdf/src/exporter/text.rs +++ b/crates/pdf/src/exporter/text.rs @@ -183,8 +183,7 @@ impl Exporter<'_> { /// Draws a metafile text run as PDF text (outlines when the font cannot be embedded). `tf` maps /// the run's own space (see [`PlacedItem::Text`]) to the page. pub(crate) fn vector_text(&mut self, s: &mut Surface, t: &wordcraft_metafile::Text, tf: kurbo::Affine) { - let face = wordcraft_fonts::word::resolve(&t.font, t.bold, t.italic).face; - let shaped = wordcraft_fonts::shape(&face, &t.text, &[], |c| c); + let (face, shaped) = wordcraft_layout::metafile_text_run(t); let glyphs: Vec<_> = shaped.iter().map(|g| (g.cluster, f64::from(g.x_advance), f64::from(g.x_offset), f64::from(g.y_offset))).collect(); let Some(lay) = t.layout(face.upem, face.ascent, face.descent, &glyphs) else { return }; let font = self.font(&face); diff --git a/crates/pdf/src/lib.rs b/crates/pdf/src/lib.rs index 404f097774..de43694a7f 100644 --- a/crates/pdf/src/lib.rs +++ b/crates/pdf/src/lib.rs @@ -49,9 +49,9 @@ use wordcraft_doc::{Document, Path as DocPath, Rgb, StoryRef}; use wordcraft_fonts::FaceRef; use wordcraft_geom::Rect; use wordcraft_layout::display::Paint; -use wordcraft_layout::display::{DisplayOptions, Draw, ItemDisplay, Stroke as LineStyle}; +use wordcraft_layout::display::{Draw, ItemDisplay, Stroke as LineStyle}; use wordcraft_layout::para::{ClKind, LineEnd}; -use wordcraft_layout::{DocLayout, LayoutCache, LayoutOptions, NumeralMode, Page, Placed, layout}; +use wordcraft_layout::{DocLayout, LayoutCache, NumeralMode, Page, Placed, layout}; use wordcraft_metafile::{Picture, PlacedItem}; mod export; @@ -72,7 +72,7 @@ use exporter::Exporter; use glyphs::{baseline, cluster_ends, left_to_right, split_clusters, visual_order}; #[cfg(test)] use metadata::{civil, parse_iso}; -pub use options::{PdfError, PdfOptions}; +pub use options::{PdfError, PdfOptions, WATERMARK_MAX_CHARS}; #[cfg(test)] use paint::shape_path; #[cfg(test)] diff --git a/crates/pdf/src/options.rs b/crates/pdf/src/options.rs index f8cef7ba04..7a0f57d8da 100644 --- a/crates/pdf/src/options.rs +++ b/crates/pdf/src/options.rs @@ -52,6 +52,35 @@ pub struct PdfOptions { pub numeral: NumeralMode, } +/// How many characters of the watermark text the PDF draws. +pub const WATERMARK_MAX_CHARS: usize = 200; + +impl PdfOptions { + /// What a page shows when drawn into the PDF: no marks or dimming, tracked changes as markup + /// only with `include_markup`. The one place that says so; strict export (#6005) checks the + /// same display list. + pub fn display_options(&self) -> wordcraft_layout::display::DisplayOptions { + wordcraft_layout::display::DisplayOptions { + marks: false, + dim_header: false, + dim_body: false, + markup: self.include_markup, + placeholders: false, + ..Default::default() + } + } + + /// How [`crate::export`] lays a document out for these options. + pub fn layout_options(&self) -> wordcraft_layout::LayoutOptions { + wordcraft_layout::LayoutOptions { + hide_deleted: !self.include_markup, + comment_balloons: self.comments, + numeral: self.numeral, + ..Default::default() + } + } +} + impl Default for PdfOptions { /// Keep ordinary document export unchanged unless a print layout is requested. fn default() -> Self { diff --git a/crates/pdf/src/picture.rs b/crates/pdf/src/picture.rs index e2b0502111..66b9938a0b 100644 --- a/crates/pdf/src/picture.rs +++ b/crates/pdf/src/picture.rs @@ -116,32 +116,11 @@ pub(crate) fn load_image(data: &Arc>, fx: PicFx) -> Option { /// [`load_image`] with an explicit RGBA byte budget (a small one in tests). pub(crate) fn load_image_within(data: &Arc>, fx: PicFx, budget: u64) -> Option { - use image::ImageDecoder; let fmt = image::guess_format(data).ok()?; - let mut limits = image::Limits::default(); - limits.max_image_width = Some(MAX_IMAGE_SIDE); - limits.max_image_height = Some(MAX_IMAGE_SIDE); - let reader = || { - let mut r = image::ImageReader::new(std::io::Cursor::new(data.as_slice())).with_guessed_format().ok()?; - r.limits(limits.clone()); - Some(r) - }; - // Header-only size check before anything is allocated (the decoder's buffer at its own pixel - // size, 8 bytes for 16-bit RGBA, plus the RGBA conversion are charged); the side limits and - // the default 512 MB allocation cap apply to the header read and the decode alike. - let (hw, hh, native) = { - let d = reader()?.into_decoder().ok()?; - let (w, h) = d.dimensions(); - (w, h, u64::from(d.color_type().bytes_per_pixel())) - }; - if !wordcraft_geom::decode_peak_fits(hw, hh, native, budget) { - return None; - } - let dec = reader()?.decode().ok()?; + // The decision to draw or not is the shared one (`wordcraft_geom::picture`), the one strict + // export checks with. + let dec = wordcraft_geom::picture::decode_for_drawing(data, budget, MAX_IMAGE_SIDE).ok()?; let (w, h) = (dec.width(), dec.height()); - if w == 0 || h == 0 || w > MAX_IMAGE_SIDE || h > MAX_IMAGE_SIDE { - return None; - } if !fx.changes_pixels() && fmt == image::ImageFormat::Jpeg && matches!(dec.color(), image::ColorType::L8 | image::ColorType::Rgb8) { return Image::from_jpeg(data.clone().into(), true).ok(); } diff --git a/crates/pdf/src/tests.rs b/crates/pdf/src/tests.rs index c155cddee9..d767b7b644 100644 --- a/crates/pdf/src/tests.rs +++ b/crates/pdf/src/tests.rs @@ -6,6 +6,8 @@ use wordcraft_doc::props::NumRef; use wordcraft_doc::{Block, CharProps, Document, ListKind, Paragraph, Table, Watermark, para_block}; use crate::*; +use wordcraft_layout::LayoutOptions; +use wordcraft_layout::display::DisplayOptions; /// Verify preset lines and connectors emit the requested PDF line-cap operator. #[test] @@ -2912,3 +2914,89 @@ fn the_author_is_left_out_when_personal_information_is_removed() { d.settings.remove_personal_info = true; assert!(!raw(&d).contains("Ada Writer"), "the creator stays out of the PDF"); } + +/// A page with text and three pictures that don't draw: garbage, a PNG cut off in its data and a +/// metafile that is only its header. The export is the same bytes as before strict export +/// existed (#6005): each is a grey box, and the rest of the page is untouched. +#[test] +fn undecodable_pictures_keep_their_grey_box_bytes() { + let mut whole = png(); + let cut = whole.len() - 24; + whole.truncate(cut); + let mut d = Document::new(); + d.core.created = "2024-01-02T03:04:05Z".into(); + let mut p = Paragraph::new(); + for (bytes, ext) in [(b"this is not a picture at all".to_vec(), "bin"), (whole, "png"), (vec![1, 0, 0, 0, 88, 0, 0, 0], "emf")] { + let media = d.add_media(bytes, ext); + let n = p.len(); + let obj = InlineObject::Image { + media, + w: 60.0, + h: 40.0, + alt: String::new(), + float: Default::default(), + crop: [0.0; 4], + fx: Default::default(), + ole: None, + }; + p.insert_object(n, obj, &CharProps::default()).unwrap(); + } + d.body = vec![para_block(p)]; + let opts = PdfOptions { compress: false, ..Default::default() }; + let a = export(&d, &opts).unwrap(); + assert_eq!(a, export(&d, &opts).unwrap(), "the export must be deterministic for the golden hash to mean anything"); + let text = String::from_utf8_lossy(&a); + assert!(text.contains("0.8157 0.8157 0.8157") || text.contains(".81"), "the grey box is drawn: {}", text.len()); + // FNV-1a of the file, recorded from the export before the shared decode check existed. + let h = a.iter().fold(0xcbf29ce484222325u64, |h, b| (h ^ u64::from(*b)).wrapping_mul(0x100000001b3)); + assert_eq!(h, 0x1edbbffab353d744, "{h:#x}"); +} + +/// A page with a chart label and an equation, exported without compression. Chart and equation +/// runs carry the text each glyph shows since strict export (#6005); this pins the PDF they +/// produce, ToUnicode maps included, to what they were before. +#[test] +fn chart_and_equation_text_keep_their_pdf_bytes() { + use wordcraft_doc::graphic::{Graphic, GraphicItem, GraphicKind, TextAlign}; + let mut d = Document::from_text("Body text"); + d.core.created = "2024-01-02T03:04:05Z".into(); + let items = vec![GraphicItem::Text { + rect: [0.0, 100.0, 216.0, 20.0], + text: "Revenue 2024".into(), + size: 10.0, + color: wordcraft_doc::props::Rgb::BLACK, + bold: false, + align: TextAlign::Center, + font: None, + }]; + let graphic = Graphic { kind: GraphicKind::Chart, items, ..Default::default() }; + let obj = InlineObject::Graphic { w: 216.0, h: 144.0, alt: String::new(), float: Default::default(), graphic: Arc::new(graphic) }; + d.insert_object(&wordcraft_doc::Pos::body(0, 0), obj, &CharProps::default()).unwrap(); + let linear = "x^2+1/2"; + let math = wordcraft_doc::math::Math { nodes: wordcraft_doc::math::parse_linear(linear), ..Default::default() }; + let n = d.para_mut(wordcraft_doc::StoryRef::Body, &wordcraft_doc::Path::top(0)).unwrap().len(); + d.para_mut(wordcraft_doc::StoryRef::Body, &wordcraft_doc::Path::top(0)) + .unwrap() + .insert_object(n, InlineObject::Equation { linear: linear.into(), display: false, math }, &CharProps::default()) + .unwrap(); + let opts = PdfOptions { compress: false, ..Default::default() }; + let a = export(&d, &opts).unwrap(); + assert_eq!(a, export(&d, &opts).unwrap(), "the export must be deterministic for the golden hash to mean anything"); + // What the ToUnicode maps say each glyph shows, across all fonts: the text of the body, the + // chart label and the equation (a mathematical italic x is one character beyond the BMP). + // Recorded on the code before strict export: the maps are the same. (The whole file's hash + // is not pinned: it follows the fonts the host resolves, which differ between runs.) + let text = String::from_utf8_lossy(&a).into_owned(); + let mut shown = std::collections::BTreeSet::new(); + for block in text.split("beginbfchar").skip(1) { + for line in block.split("endbfchar").next().unwrap_or("").lines() { + let Some((_, to)) = line.trim().split_once("> <") else { continue }; + let units: Vec = + to.trim_end_matches('>').as_bytes().chunks(4).filter_map(|c| u16::from_str_radix(std::str::from_utf8(c).ok()?, 16).ok()).collect(); + shown.insert(String::from_utf16_lossy(&units)); + } + } + let expected: std::collections::BTreeSet = + ["B", "o", "d", "y", " ", "t", "e", "x", "R", "v", "n", "u", "2", "0", "4", "+", "1", "\u{1D465}"].into_iter().map(String::from).collect(); + assert_eq!(shown, expected); +} diff --git a/crates/render/src/lib.rs b/crates/render/src/lib.rs index 3bfb27f8ce..8e57e07713 100644 --- a/crates/render/src/lib.rs +++ b/crates/render/src/lib.rs @@ -461,9 +461,7 @@ pub fn render_text_line(face: &wordcraft_fonts::FontFace, text: &str, size: f32, } fn draw_watermark(ctx: &mut RenderContext, view: Affine, page: &Page, wm: &wordcraft_doc::Watermark, opts: &RenderOptions) { - let r = wordcraft_fonts::word::resolve(&wm.font, false, false); - let face = r.face; - let glyphs = wordcraft_fonts::shape(&face, &wm.text, &[], |c| c); + let Some((face, _, glyphs)) = wordcraft_layout::watermark_run(wm, usize::MAX) else { return }; let upem = face.upem.max(1.0); let raw_w: f64 = glyphs.iter().map(|g| g.x_advance as f64).sum::() / upem; if raw_w <= 0.0 { diff --git a/crates/render/src/media.rs b/crates/render/src/media.rs index 34c91aea92..2dbf1faf88 100644 --- a/crates/render/src/media.rs +++ b/crates/render/src/media.rs @@ -82,30 +82,14 @@ pub fn decode_pixmap_fx(bytes: &[u8], fx: wordcraft_doc::picfx::PicFx) -> Option /// [`decode_pixmap_fx`] with an explicit RGBA byte budget (a small one in tests). pub(crate) fn decode_pixmap_within(bytes: &[u8], fx: wordcraft_doc::picfx::PicFx, budget: u64) -> Option { - use image::ImageDecoder; - let mut limits = image::Limits::default(); - limits.max_image_width = Some(MAX_DECODE_SIDE); - limits.max_image_height = Some(MAX_DECODE_SIDE); - let reader = || { - let mut r = image::ImageReader::new(std::io::Cursor::new(bytes)).with_guessed_format().ok()?; - r.limits(limits.clone()); - Some(r) - }; - // The header gives the size and the decoder's pixel format (16-bit RGBA takes 8 bytes a pixel - // before its RGBA8 copy); refuse before the decoder (and the RGBA copy) allocate anything. - let (hw, hh, native) = { - let d = reader()?.into_decoder().ok()?; - let (w, h) = d.dimensions(); - (w, h, u64::from(d.color_type().bytes_per_pixel())) - }; - if !wordcraft_geom::decode_peak_fits(hw, hh, native, budget) { + // The decision to draw or not is the shared one (`wordcraft_geom::picture`), the one strict + // export checks with. + let decoded = wordcraft_geom::picture::decode_for_drawing(bytes, budget, MAX_DECODE_SIDE); + if let Err(wordcraft_geom::picture::Refusal::TooLarge { width, height }) = &decoded { // Logged once a picture: its refusal is cached with it (`pixmap_for`). - log::warn!( - "a {hw}×{hh} picture of {native} bytes a pixel needs more than {budget} bytes to decode: it is not drawn (in the text, a grey box takes its place)" - ); - return None; + log::warn!("a {width}×{height} picture needs more than {budget} bytes to decode: it is not drawn (in the text, a grey box takes its place)"); } - let mut img = reader()?.decode().ok()?.into_rgba8(); + let mut img = decoded.ok()?.into_rgba8(); let (w, h) = img.dimensions(); fx.apply_rgba(&mut img); premul_pixmap(w, h, img.as_raw()) @@ -270,8 +254,7 @@ pub(crate) fn draw_vector(ctx: &mut RenderContext, v: &Vector, r: &kurbo::Rect, /// Draws a metafile text run: `tf` maps the run's own space (see [`PlacedItem::Text`]) to the device. fn draw_vector_text(ctx: &mut RenderContext, t: &wordcraft_metafile::Text, tf: Affine) { - let face = wordcraft_fonts::word::resolve(&t.font, t.bold, t.italic).face; - let shaped = wordcraft_fonts::shape(&face, &t.text, &[], |c| c); + let (face, shaped) = wordcraft_layout::metafile_text_run(t); let glyphs: Vec<_> = shaped.iter().map(|g| (g.cluster, f64::from(g.x_advance), f64::from(g.x_offset), f64::from(g.y_offset))).collect(); let Some(lay) = t.layout(face.upem, face.ascent, face.descent, &glyphs) else { return }; let k = lay.em / face.upem.max(1.0); @@ -326,18 +309,12 @@ fn device_pixel(view: Affine) -> f64 { if scale.is_finite() && scale > 1e-9 { 1.0 / scale } else { 1.0 } } -/// Largest page-background picture decoded, in pixels (hostile files). -const MAX_BACKGROUND_PIXELS: u64 = 64 * 1024 * 1024; - /// Draw the page's picture or texture fill (Design › Page Color › Fill Effects) behind the content: /// tiled at its natural size (96 dpi) for a `tile` fill, stretched over the page otherwise. pub(crate) fn draw_page_picture(ctx: &mut RenderContext, doc: &Document, page: &Page, view: Affine) { let Some(pic) = &doc.settings.page_picture else { return }; let Some(bytes) = doc.media.get(&pic.media) else { return }; - let (nw, nh) = match image_size(bytes) { - Some((w, h)) if w > 0 && h > 0 && (w as u64) * (h as u64) <= MAX_BACKGROUND_PIXELS => (w as f64, h as f64), - _ => return, - }; + let Some((nw, nh)) = wordcraft_geom::picture::background_size(bytes).map(|(w, h)| (f64::from(w), f64::from(h))) else { return }; let Some(pm) = pixmap_for(doc, &pic.media, wordcraft_doc::picfx::PicFx::default()) else { return }; let (pw, ph) = (pm.width().max(1) as f64, pm.height().max(1) as f64); let (w, h) = (page.w as f64, (page.h as f64).min(1e6)); diff --git a/crates/render/src/tests.rs b/crates/render/src/tests.rs index 2a42178c13..c18f75790d 100644 --- a/crates/render/src/tests.rs +++ b/crates/render/src/tests.rs @@ -1487,3 +1487,41 @@ fn paths_far_past_every_canvas_are_not_flattened() { assert!(!path_extent_ok(Affine::scale(1e12), &line(10.0))); assert!(!path_extent_ok(Affine::IDENTITY, &line(f64::NAN))); } + +/// The same three pictures that don't draw as in the PDF test: each stays a grey box, and the page's +/// pixels are what they were before strict export existed (#6005). +#[test] +fn undecodable_pictures_keep_their_grey_box_pixels() { + let mut whole = Vec::new(); + image::RgbaImage::from_fn(16, 8, |x, y| image::Rgba([(x * 15) as u8, (y * 30) as u8, 200, 255])) + .write_to(&mut std::io::Cursor::new(&mut whole), image::ImageFormat::Png) + .unwrap(); + let cut = whole.len() - 24; + whole.truncate(cut); + let mut d = Document::new(); + let mut p = wordcraft_doc::Paragraph::new(); + for (bytes, ext) in [(b"this is not a picture at all".to_vec(), "bin"), (whole, "png"), (vec![1, 0, 0, 0, 88, 0, 0, 0], "emf")] { + let media = d.add_media(bytes, ext); + let n = p.len(); + let obj = wordcraft_doc::para::InlineObject::Image { + media, + w: 60.0, + h: 40.0, + alt: String::new(), + float: Default::default(), + crop: [0.0; 4], + fx: Default::default(), + ole: None, + }; + p.insert_object(n, obj, &Default::default()).unwrap(); + } + d.body = vec![wordcraft_doc::para_block(p)]; + let l = layout(&d, &mut LayoutCache::new(), &LayoutOptions::default()); + let opts = RenderOptions::default(); + let img = render_page(&d, &l.pages[0], 1.0, &opts); + let boxes = page_display(&d, &l.pages[0], &opts.display).iter().filter(|i| matches!(i, Draw::Image { .. })).count(); + assert_eq!(boxes, 3); + // FNV-1a of the pixels, recorded before the shared decode check existed. + let h = img.pixels.iter().fold(0xcbf29ce484222325u64, |h, b| (h ^ u64::from(*b)).wrapping_mul(0x100000001b3)); + assert_eq!(h, 0x79f9f1d3602a9925, "{h:#x}"); +} diff --git a/docs/file-format-parity.md b/docs/file-format-parity.md index 075031fbfa..f834b858cd 100644 --- a/docs/file-format-parity.md +++ b/docs/file-format-parity.md @@ -1,4 +1,6 @@ # File-format parity with Microsoft Word + +> **2026-11-08:** `wordcraft-cli convert --strict` refuses a PDF or PNG that would draw an empty box for a character or leave out a picture, including those a DOCX names outside the file or not at all (#6005); never fetched. > **2026-11-05:** A Word template opened from the file manager starts a new document based on it, and is never saved over (#3433). > **2026-11-05:** Strict Open XML documents open with their shapes, text boxes, groups and canvases (#3548). @@ -321,6 +323,8 @@ For field results, the engine parser treats a backslash in quoted arguments as e ## Revision history +> 2026-11-08 · `convert --strict` for PDF and PNG output, and the pictures a DOCX open leaves out are recorded (#6005); recorded in the progress log, readiness unchanged. + > 2026-11-07 · VML text boxes: `mso-layout-flow-alt:bottom-to-top` reads as `vert270` with any `layout-flow`, and `layout-flow:vertical` or `vertical-ideographic` otherwise as `vert`, so a compatibility-mode header's turned text box is drawn turned (#5193, `it::text_boxes::vml_text_boxes_keep_their_inset_fit_and_anchor`). > 2026-11-03 · Section `w:chapStyle` values preserve the full unsigned-byte range, including 10, while values above 255 are rejected.