Mirrors/pdfcraft
1
0 Fork 0
mirror of https://github.com/storytold/pdfcraft.git synced 2026-11-12 17:31:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/main' into HEAD

Browse Source
This commit is contained in:
mtlcl410 committed 2026-11-07 18:00:33 +08:00
2 parents 88289a5f30 f42fda7342
commit 9dfd49cc94
148 files changed
+24768 -743
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/house-rules.yml
+28
View File
@@ -0,0 +1,28 @@
name: House rules
on:
push:
branches: [main]
paths:
- AGENTS.md
- CLAUDE.md
- .github/workflows/house-rules.yml
pull_request:
paths:
- AGENTS.md
- CLAUDE.md
- .github/workflows/house-rules.yml
workflow_dispatch:
permissions:
contents: read
concurrency:
group: house-rules-${{ github.ref }}
cancel-in-progress: true
jobs:
house-rules:
uses: storytold/craft-common-actions/.github/workflows/house-rules.yml@main
with:
rust: true
AGENTS.md
+67 -6
View File
@@ -1,6 +1,59 @@
<!-- craft-house-rules:start core source=dde09ea31e2e457f5b4653a6e76f7cdf55944bdf sha256=540e6cea75d12d0710c3dca89658328e81e508ecef102446ca5bb95b7c0b92bb -->
## House rules
These apply in every Craft repository. A later rule may be stricter, not weaker. The source is `standards/house-rules-core.md` in `storytold/craft-contrib`. Propose changes there, not in a copy.
1. **A person is accountable.** Whoever runs an agent answers for what it opens and for its review comments.
2. **Check, then claim.** Before writing code, read the issue comments, search pull requests for `#N` and the keywords, and check recent commits on `main`. Comment on the issue first. Release a claim you will not finish.
3. **Verify, don't guess.** Open a definition before using it. Confirm the cause in the code, even when someone names a location.
4. **Keep fixes narrow and tested.** Fix what the issue describes, where the bug is. Add a test that fails without the fix.
5. **Review the diff before opening a pull request.** Identifiers exist as used, the test fails without the fix, nothing unrelated changed, and the description is true.
6. **Look for a better pull request.** If another fixes the same thing, close yours when it is better or equal, and say why.
7. **Report only what ran.** Say what you tested and what you could not run. Without a local build or test, say so and rely on CI before marking ready.
8. **`main` changes through a pull request.** Do not push to it. Branch protection enforces this. There is no exception.
9. **Stop and ask only about** licensing outside the allowlist, publishing, releases, a new remote, secrets, spending money, destructive git, and anything else hard to undo. Otherwise keep working. `plan/` is a gitignored notebook, not rules. Without it, do not ask for it and do not stop.
10. **Keep secrets and personal details out** of the repository, logs and requests.
11. **CI stays shared.** Workflows call `storytold/craft-common-actions`. Do not change CI in a feature or fix pull request. If CI cannot cover a change, open an issue there.
12. **Guards say why.** A check that looks removable says what it prevents. Read that before changing it.
13. **Instructions must be readable by everyone.** No notes about one machine, and no links to files a reader cannot open. `craftrules` is retired. Do not link to it.
14. **Keep pull requests moving.** Concurrent pull requests are fine; unanswered actionable feedback is not. Respond in 7 days, a reminder at 14, closure eligible at 30, from when an actionable request waits on the author. Suggestions and unconfirmed bot findings do not start the clock. A fix, explanation or agreed plan resets it; unrelated activity does not. Waiting on a reviewer or maintainer pauses it and does not count against the author. A maintainer decides closure; it is never automatic. Reuse search results, do not poll, and stop at a rate limit.
15. **No agent attribution.** A pull request does not say an agent wrote it. A repository rule may not require that.
<!-- craft-house-rules:end -->
<!-- craft-house-rules:start rust source=dde09ea31e2e457f5b4653a6e76f7cdf55944bdf sha256=387e0ccb4b49e567f60951d7d6fff7305054c89f43ad812b9993dc7c38528013 -->
## Rust Craft repositories
These apply to native Craft apps and to shared Rust libraries. Select them by repository type. A repository that merely contains some Rust does not take this block. Rules later in a repository's file may be stricter, but not weaker. The source is `standards/house-rules-rust.md` in [storytold/craft-contrib](https://github.com/storytold/craft-contrib).
1. **Never crash.** Non-test code does not panic and has no `unsafe`, except where the repository file names an exception. Return errors, treat input as hostile, and bound recursion and allocations. New production crate roots copy the `deny` attribute from an existing crate: `clippy::unwrap_used`, `clippy::expect_used`, `clippy::panic`, `clippy::unimplemented`, `clippy::todo`, `clippy::unreachable`. Every crash fix comes with a test that panicked before the fix.
2. **Clean room.** Learn from observed behaviour and public specifications. Do not copy another product's code, assets, strings or artwork. Check a dependency's licence before adding it.
3. **Assets and fonts.** Every asset is original or suitably licensed, and is recorded in `ATTRIBUTION.md` in the same change. Font files live in [storytold/craft-fonts](https://github.com/storytold/craft-fonts), not in the app repository.
4. **Rust only for native Craft apps.** No handwritten JavaScript or TypeScript, and no Tauri, Electron or webview shell, in those apps. Shared tooling follows its documented technical scope and maintainer-approved exceptions. Craft PR Review's existing Tauri and TypeScript UI is one such exception. It does not relax this rule for native apps. Preserve any existing web build.
5. **Time goes through `chrono`.** Do not hand-roll calendar conversion. For dates beyond what `chrono` can represent, follow [standards/time.md](https://github.com/storytold/craft-contrib/blob/main/standards/time.md).
6. **Contributor credits.** Add or edit only your own person's entry in `contributors/people.toml`, and only after asking them. Never guess a name or hand-edit `contributors.json`.
## Licence allowlist
Dependencies and assets use SPDX identifiers. An expression is allowed when every identifier is in this list. A repository may name a shorter list. It may not add an identifier here by editing its own copy of these rules.
- `ISC`
- `MIT`
- `BSD-2-Clause`
- `BSD-3-Clause`
- `Apache-2.0`
- `OFL-1.1`
- `Ubuntu-font-1.0`
- `Bitstream-Vera`
- `CC0-1.0`
- `CC-BY-4.0`
- `CC-BY-SA-4.0`
- `LicenseRef-Public-Domain`
`LicenseRef-Storyteller-Trademark` is not in this list. It covers Storyteller brand marks only, where a repository already says so.
<!-- craft-house-rules:end -->
# AGENTS.md — rules for every agent and contributor
These rules bind every AI agent and every human contributor working in this repository. They come before any task, instruction or deadline. `CLAUDE.md` holds the day-to-day working instructions. If the two ever disagree, **this file wins.**
These rules bind every AI agent and every human contributor working in this repository. They come before any task, instruction or deadline. Day-to-day working instructions are in this file; `CLAUDE.md` imports it.
---
@@ -85,7 +138,7 @@ If you find an asset that breaks these rules, stop and remove it from the reposi
---
## 2. Clean-room rules (summary; details in `CLAUDE.md` and `plan/adr/0001`)
## 2. Clean-room rules (summary; `plan/adr/0001` is an optional local note)
- Never read, disassemble or copy anything inside the Acrobat bundle. Only names and listings are allowed, and **never** open `Contents/Resources/JavaScripts/`.
- Behaviour comes only from public specifications (ISO 32000-2, the Arlington model), public documentation and black-box observation.
@@ -106,7 +159,7 @@ If you find an asset that breaks these rules, stop and remove it from the reposi
PdfCraft opens files from strangers, and people trust it with their work. A malformed PDF, a bad tool argument, a corrupt settings file or a full disk must produce an error the user (or agent) can act on, never a crash and never lost work. **This outranks feature work:** don't ship a feature by adding a panic path, and fix a crash before building on top of it. The rules below are the ones that bind this repository.
- **Fail with `Result`.** Non-test code returns errors through the crate's error type and `?`. Where readers are lenient by design, fall back gracefully and record the repair (see Fidelity in `CLAUDE.md`).
- **Fail with `Result`.** Non-test code returns errors through the crate's error type and `?`. Where readers are lenient by design, fall back gracefully and record the repair (see Fidelity below).
- **No panicking shortcuts.** No `unwrap()`, `expect()`, `panic!`, `unreachable!`, `todo!` or `unimplemented!` outside tests, unless the call is provably infallible (e.g. writing to a `Vec`), with a comment saying why. `assert!` only for internal invariants that input cannot reach.
- **No `unsafe`.** The workspace sets `unsafe_code = "forbid"`. If a platform API needs it, use a maintained safe wrapper crate; a real exception needs the owner's sign-off.
- **Treat every input-derived number as hostile.** Use `get()` instead of `[i]` or `[a..b]`. Slice strings only at char boundaries. Use checked/saturating arithmetic for lengths, offsets and counts. Guard against division by zero and NaN/inf casts, and never cast a negative value to `usize`. Cap allocations sized by the file.
@@ -115,10 +168,18 @@ PdfCraft opens files from strangers, and people trust it with their work. A malf
- **Prove it.** Every crash fix comes with a small synthetic regression test that panicked before the fix. Turn fuzzer findings (`cargo xtask fuzz`) into such tests before fixing them.
- **Enforced by clippy.** Root `clippy.toml` lets tests use `unwrap`/`expect`/`panic!`. A crate that is clean carries `#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]`; new crates start with it.
## 5. Everything else
## 5. Page interaction, fidelity and gates
Follow `CLAUDE.md`. It covers quality gates, layering, commit rules and how to run and look at the app.
- **Layering.** Nothing below L7 depends on egui/winit/eframe/rfd. `cos`, `filters`, `crypt` and `arlington` stay standalone.
- **Page input.** Page modules read `PagePointer` (`crates/ui-egui/src/page_pointer.rs`), never egui's raw pointer state. `page_pointer::tests::page_modules_read_the_pointer_only_through_page_pointer` enforces it.
- **Page cursor.** Page modules claim the cursor for what is under the pointer (`view.cursor.claim`, `crates/ui-egui/src/canvas/page_cursor.rs`) and never set it themselves. `page_pointer::tests::page_modules_claim_the_cursor_and_never_set_it` enforces it.
- **Page presses.** A page module decides what is under a point in one hit test (`Hit` in `crates/ui-egui/src/canvas/page_router.rs`) and acts on a press only through that router. `page_pointer::tests::page_modules_act_on_a_press_only_through_the_router` enforces it.
- **Fidelity.** The PDF object graph is the model. Preserve unknown data. Saves are incremental unless a full rewrite is required. Never silently drop or repair data without recording it.
- **Fonts** are the optional build input `CRAFT_FONTS_DIR`. Code using `pdfcraft_fonts::CRAFT_FONTS` must work when it is empty.
- **Quality gates** before every commit: `cargo fmt --check`, clippy with `-D warnings`, the tests, `cargo xtask layers`, `cargo xtask assets`, and the wasm check. New workspace crates include `workspace = true` under `[lints]`. Scope `cargo clippy -p <crate> --all-targets -- -D warnings` and `cargo test -p <crate>` to the crates you touched; the workspace sweep is CI's job.
- **Review at four zooms** as you go: the clause, the function, the module, and the project (layering, wasm, undo, save fidelity, a matching automation tool, an honest `parity/` entry). Prove a test fails before the fix by reverting the fix once.
- `plan/` is an optional gitignored notebook. Do not ask for it and do not stop when it is absent. Public status is [`ROADMAP.md`](ROADMAP.md), [`docs/gaps.md`](docs/gaps.md) and [`docs/roadmap.md`](docs/roadmap.md). "Shipped" in `parity/` means exists and tested, not parity with Acrobat.
- `cargo run -p pdfcraft -- <file.pdf>` opens the app. `pdfcraft-cli run` drives the engine headlessly. `cargo xtask fuzz` findings stay in `fuzz-out/` and become synthetic tests. For UI work, look at a screenshot from `--control` or the `shot` example.
## Contributor credits (About window)
ATTRIBUTION.md
+2 -1
View File
@@ -4,10 +4,11 @@
Every asset PdfCraft includes, bundles or uses to build its published material, with its author, source and licence. The policy is in [AGENTS.md](AGENTS.md) §1. The machine-readable list, with the SHA-256 of every asset, is [ATTRIBUTION.toml](ATTRIBUTION.toml). Licence texts are kept beside the assets and summarised in [NOTICE](NOTICE).
## In this repository (257)
## In this repository (258)
| Asset | Title | Author | Licence | Source | Used for |
|---|---|---|---|---|---|
| `docs/images/pdfcraft-ocr-verify.png` | PdfCraft screenshot: OCR Verify screen | PdfCraft contributors (generated by `cargo xtask screenshots`) | MIT OR Apache-2.0 | this repository | README: OCR review. Shows only PdfCraft UI, its Lucide icons and a synthetic scanned page built from the embedded fonts, with a real recognition (the OCR models) loaded into the Verify screen |
| `assets/icons/accessibility.svg` | Lucide icon "accessibility" | Lucide Contributors (portions Cole Bemis, Feather) | ISC AND MIT | https://github.com/lucide-icons/lucide (npm lucide-static 1.49.0, icons/accessibility.svg) | UI icon, embedded via crates/ui-egui/src/icon_data.rs |
| `assets/icons/align-left.svg` | Lucide icon "align-left" | Lucide Contributors (portions Cole Bemis, Feather) | ISC AND MIT | https://github.com/lucide-icons/lucide (npm lucide-static 1.49.0, icons/align-left.svg) | UI icon, embedded via crates/ui-egui/src/icon_data.rs |
| `assets/icons/align-center.svg` | Lucide icon "align-center" | Lucide Contributors (portions Cole Bemis, Feather) | ISC AND MIT | https://github.com/lucide-icons/lucide (npm lucide-static 1.49.0, icons/align-center.svg) | UI icon, embedded via crates/ui-egui/src/icon_data.rs |
ATTRIBUTION.toml
+11
View File
@@ -10,6 +10,17 @@
# licence: SPDX expression from the allowlist in AGENTS.md §1.2. adobe_data = true marks the closed list of
# Adobe-authored non-visual technical data allowed by AGENTS.md §1.1.
[[asset]]
path = "docs/images/pdfcraft-ocr-verify.png"
title = "PdfCraft screenshot: OCR Verify screen"
author = "PdfCraft contributors (generated by `cargo xtask screenshots`)"
source = "this repository"
licence = "MIT OR Apache-2.0"
licence_file = "LICENSE-MIT"
kind = "image"
usage = "README: OCR review. Shows only PdfCraft UI, its Lucide icons and a synthetic scanned page built from the embedded fonts, with a real recognition (the OCR models) loaded into the Verify screen"
sha256 = "ea7485b151e27ad3fd2a2dac079436ab11745d42032201a029bb20f819c6af8f"
[[asset]]
path = "assets/icons/accessibility.svg"
title = "Lucide icon \"accessibility\""
CLAUDE.md
+1 -65
View File
@@ -1,65 +1 @@
# PdfCraft — instructions for agents
PdfCraft is a clean-room, open-source, Rust-native PDF application targeting Adobe Acrobat Pro parity. It runs natively on macOS, Windows, Linux and FreeBSD, and on the web via WASM. It is the sibling of `../photocraft` (a Photoshop-class editor) and follows the same conventions.
## Start every session here
1. Read `plan/STATUS.md`: the current milestone, the next unchecked task and any blockers. `ROADMAP.md` is the one-page summary (stage, numbers, progress log) and `docs/roadmap.md` the milestone table; at the end of every session update the milestone row and add a line to the ROADMAP.md progress log.
- Read `docs/gaps.md` (the ranked gaps: Acrobat interop, the open-issue backlog, the renderer and a fidelity harness, editing existing content, Pro workflows, 1.0 polish) and `docs/target-app-parity.md` (the numbers and how they are measured) before choosing work. Prefer the gaps over new P2/P3 features, and keep both documents true when things change (`docs/` progress docs carry a status line and a revision history).
- "Shipped" in `parity/` means "exists and tested", not "as good as Acrobat". Don't mark a feature shipped on a generic test, and say in its notes what is still missing.
2. Read that task in `plan/execution-plan.md` §3, the relevant section of `plan/architecture.md`, and the README of the crate you're touching.
3. Follow the **autonomous operation protocol** in `plan/execution-plan.md` §7 (orient → plan → implement + test → verify → record → commit). Don't stop to ask unless §7 lists the decision as the user's.
`plan/` is gitignored (local-only, like PhotoCraft). The machine-readable parity checklist lives in `parity/` (committed).
## Non-negotiables
- **Assets: read `AGENTS.md` §1 before adding or showing any icon, image, font or document.** No assets from Adobe products, ever. Only openly licensed or contributor-original assets are allowed, each with an entry in `ATTRIBUTION.toml`. `cargo xtask assets` enforces this. `AGENTS.md` overrides this file.
- **Fonts live in [storytold/craft-fonts](https://github.com/storytold/craft-fonts).** Never commit font files here (`AGENTS.md` §1.4). It is the optional build input `CRAFT_FONTS_DIR`: `git clone https://github.com/storytold/craft-fonts ../craft-fonts && CRAFT_FONTS_DIR=../craft-fonts cargo test --workspace` embeds the Japanese fonts (UI fallback, Japanese text in edited PDFs) and runs their tests, which otherwise skip. Code using `pdfcraft_fonts::CRAFT_FONTS` must work when it is empty.
- **Clean-room.**
- Never read, disassemble or copy anything inside the Acrobat bundle (names and listings only). **Never open `Contents/Resources/JavaScripts/`.**
- Behaviour comes from public docs, specs (ISO 32000-2, the Arlington model) and black-box observation (`plan/acrobat/`).
- Never copy GPL/AGPL code. MuPDF, Ghostscript, Poppler, veraPDF and DSS run only as external oracle processes.
- See `plan/README.md` §Clean-room and `plan/adr/0001`.
- **Privacy.** When observing Acrobat, use synthetic fixtures only. Never capture the Home view, recent files or account info. Capture by window id (`plan/acrobat/tools/`). Never commit Acrobat outputs, corpus files or personal data.
- **Layering.** Nothing below L7 depends on egui/winit/eframe/rfd (`plan/architecture.md` §3). `cos`/`filters`/`crypt`/`arlington` stay standalone.
- **Page input.** Page modules read `PagePointer` (`crates/ui-egui/src/page_pointer.rs`), never egui's raw pointer state (`hover_pos`, `interact_pos`, `latest_pos`, `press_origin`, button reads); `page_pointer::tests::page_modules_read_the_pointer_only_through_page_pointer` enforces it.
- **Page cursor.** Page modules claim the cursor for what is under the pointer (`view.cursor.claim(Layer::…, icon)`, `crates/ui-egui/src/canvas/page_cursor.rs`), never set it (`set_cursor_icon`, `cursor_icon =`, `on_hover_cursor`); `document_area` sets it once, from the claim on the highest layer the active tool reaches. `page_pointer::tests::page_modules_claim_the_cursor_and_never_set_it` enforces it.
- **Page presses.** A page module decides what of its own is under a point in one hit test, which returns a `Hit` (owner, layer, cursor, tip; `crates/ui-egui/src/canvas/page_router.rs`) and gives its cursor, its tip and its press. It acts on a press only when the press is its own or, for the active tool, nobody's: the page view tells it which (`PagePress`: its own, nobody's, or another page's), and it never tests the press point again or asks another module whether it wants it. Escape cancels any press the pages hold (`cancel_on_escape`), and that press is ignored until the button is up. `page_pointer::tests::page_modules_act_on_a_press_only_through_the_router` enforces it.
- **Fidelity.** The PDF object graph is the model. Preserve unknown data. Saves are incremental unless a full rewrite is required. Never silently drop or repair data without recording it.
- **Never panic.** Every PDF, script, MCP call, settings file and keystroke is untrusted input, and none of it may crash the app or lose the user's work. This outranks feature work. Return `Result` (the crate's error enum) and propagate with `?`, or fall back leniently and record it. In non-test code:
- No `unwrap`/`expect`/`panic!`/`unreachable!`/`todo!` unless it is provably infallible, with a comment saying why.
- No indexing or slicing with input-derived positions (use `get`, and slice strings only at char boundaries).
- Use checked or saturating arithmetic on input-derived numbers, and guard against division by zero and NaN/inf casts.
- Cap allocations sized by input, and bound recursion with depth limits or seen-sets.
- Handle lock poisoning.
- No `unsafe` (`unsafe_code = "forbid"`).
- Every crash fix gets a synthetic regression test. See `AGENTS.md` §4.
- **Rust only** in the product and build (`xtask`). No handwritten JS/TS.
- **Time goes through `chrono`** (workspace dependency). Never hand-roll UTC/calendar conversion (days-to-date algorithms, leap-year math); a crate owns only its output format string. The one exception is range extension: past the years chrono can represent, shift the date by whole 400-year Gregorian cycles (146,097 days, after which the calendar repeats), let chrono compute the month and day, and add the cycles back to the year (splitting seconds into hours, minutes and seconds is plain arithmetic).
- **Review every update as you make it, at four zooms.** Not an end-of-task pass — a per-update habit, so a defect never gets built on. **Clause:** checked arithmetic, `get()` not `[i]`, char-boundary slicing, the right error through `?`, `<` against `<=`, the 1-based/0-based seam, capped allocations. **Function:** is it total — name an input that panics; walk empty, zero, max, malformed, cyclic; is the return type honest. **File/module:** two sources of truth, dead or duplicated code, is the public surface still smallest and the doc comment still true. **Project:** layering, wasm, undo/redo, save fidelity, encryption and permissions, a matching `automation` tool or registry entry, and whether a `parity/` status or note just became untrue. Scale the zoom to the change, and fix what a level turns up before widening.
- Tests mirror the zooms: property → unit → crate integration → end-to-end.
- **Prove a test fails before the fix by actually reverting the fix.** A test nobody has seen fail is not evidence. This has caught two worthless tests in this repo: one confinement test passed with path resolution switched off entirely, because the tools failed on a nonexistent document id before reaching the path.
- **Rebasing is a review step, not a mechanical one.** Read every hunk you resolve, and the auto-merged hunks in the files you touched too — `main` moves fast enough that a file can change by hundreds of lines under a branch.
- **External PRs are held to safety, security and completeness before style.** Provenance first (no Adobe-derived asset, no committed font, no copied GPL/AGPL code, behaviour from the spec and not from reading another implementation), then no new panic path or `unsafe`, then `--root` confinement on every path including ones reached by nested dispatch, opt-in-only MCP and control channel, no new outbound network, then a headless `automation` tool, tests at the right zoom and an honest `parity/` entry. Provenance is the part a contributor cannot be expected to know, and the only part that cannot be undone after merge.
- **Quality gates** before every commit: `cargo fmt --check`, clippy with `-D warnings`, the tests, `cargo xtask layers`, `cargo xtask assets`, and the wasm check once `xtask ci` exists (M0).
- New workspace crates must include `[lints] workspace = true` in `Cargo.toml` to inherit `[workspace.lints]`. Production crate roots must declare the required crate-level lint attributes before modules or other crate items; crate-level `//!` documentation may precede them.
- **Scope builds and tests to the crates you touched** — `cargo clippy -p <crate> -j 2 --all-targets` and `cargo test -p <crate> -j 2`, under a `timeout`. A full `--workspace` build has locked up a development machine's CPU; it is CI's job and CI runs it on every PR. Run the workspace sweep locally only when you have a reason to and the machine is free.
- **Check for existing work before starting a fix or feature.** Search the issues and PRs, open and closed, with `gh issue list -R storytold/pdfcraft --state all --search "<terms>"` and the same with `gh pr list` (try a few phrasings), and check whether a merged fix is already on `main` (`git merge-base --is-ancestor <commit> origin/main`). If one is, the report is likely from an older build: say so rather than fixing it again. If an open PR covers it, review or build on that PR instead of opening a competing one. If an issue exists, reference it in the PR (`Fixes #123`).
- **Commits and PRs:** `main` is the only integration branch and is **never pushed to directly**. Branch from an up-to-date `main` as `<type>/<slug>` (`feat/`, `fix/`, `docs/`, `chore/`, `refactor/`), one task id per commit (e.g. `M1.4: xref stream reader`), commit only green states, and open a PR into `main` using `.github/pull_request_template.md`. Merge only when CI is green.
- **`main` moves fast enough that PRs rot.** It took 396 commits in the three days to 2026-10-12, and most open PRs are conflicting at any given time. Rebase your own branch the same day `main` moves under it, and ask the author to rebase theirs — never rewrite a contributor's branch. Conflict risk tracks how *contested* the files are, not how large the change is: tooling, tests and docs are nearly free, while `crates/fonts`, `ui-egui` and the i18n catalogues are where several people collide at once.
## Running and looking at the app
- `cargo run -p pdfcraft -- <file.pdf>` opens the desktop app.
- `cargo run -p pdfcraft-cli -- run --script steps.json --root DIR` drives the engine headlessly through the automation tools (`pdfcraft-cli tools` lists them). Use it, together with `page_render`, to check engine changes. `pdfcraft-cli mcp` is the opt-in MCP server (AGENTS.md §3).
- `cargo xtask fuzz --time 300` mutation-fuzzes open/render/edit/save in child processes. Findings land in `fuzz-out/findings/` (git-ignored; never commit corpus-derived files). Turn every real finding into a small synthetic regression test before fixing it.
- `cargo xtask parity [--partial]` reports Acrobat-parity progress from `parity/acrobat-features.toml`. Update the entry when a feature ships.
- `cargo xtask demo-pdf` builds `dist/demo/pdfcraft-showcase.pdf` (needs Chrome) for visual checks.
- For UI work, **look at the result**. Either launch `pdfcraft --control FILE doc.pdf` and use `pdfcraft-cli ui --control FILE screenshot --out x.png` (plus `inspect`, `click`, `key`, `type`, `command`, `set`), or take a headless shot with `cargo run -p pdfcraft-ui-egui --example shot`. Compare against `plan/acrobat/02-ui-ux.md`. Control-channel tests use kittest (`crates/ui-egui/tests/control.rs`).
- Parallel agents: use a separate `CARGO_TARGET_DIR` per agent and separate git worktrees.
## Current bootstrap debt (tracked in STATUS.md)
- `pdfcraft-render` renders through the `hayro` crate directly and inspects documents through `lopdf`. Both get replaced by `cos` / `model` / the DisplayList device (M1–M2, ADR-0004).
## CI workflows
- Workflows in `.github/workflows/` call the shared workflows in [`storytold/craft-common-actions`](https://github.com/storytold/craft-common-actions) at `@main`, with their inputs and secrets. No other jobs: if no shared workflow does what's needed, it doesn't go here.
- Don't add or change CI in a feature or fix PR. If the existing workflows can't build or test a change, link an existing craft-common-actions issue or PR, or open a new issue, or a PR only for a small fix that is safe for every app and does not increase CI time.
- Some workflow files here still run their own jobs. Don't add to them. Moving a job into craft-common-actions is its own task, never part of another PR, and the job stays here until a shared workflow runs it.
@AGENTS.md
Cargo.lock
Generated
+16
View File
@@ -143,6 +143,15 @@ dependencies = [
"zerocopy",
]
[[package]]
name = "aho-corasick"
version = "1.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba"
dependencies = [
"memchr",
]
[[package]]
name = "aligned-vec"
version = "0.6.4"
@@ -4333,6 +4342,7 @@ version = "0.18.1"
dependencies = [
"chrono",
"hayro-syntax",
"js-sys",
"pdfcraft-crypt",
"pdfcraft-filters",
"pdfcraft-oracle",
@@ -4340,6 +4350,7 @@ dependencies = [
"proptest",
"serde_json",
"thiserror 2.0.21",
"wasm-bindgen",
]
[[package]]
@@ -4516,6 +4527,7 @@ version = "0.18.1"
dependencies = [
"ocrs",
"pdfcraft-fonts",
"png 0.18.1",
"rten",
"tempfile",
"thiserror 2.0.21",
@@ -4607,6 +4619,7 @@ dependencies = [
name = "pdfcraft-redact"
version = "0.18.1"
dependencies = [
"aho-corasick",
"pdfcraft-annot",
"pdfcraft-content",
"pdfcraft-cos",
@@ -4615,8 +4628,10 @@ dependencies = [
"pdfcraft-fonts",
"pdfcraft-forms",
"pdfcraft-model",
"pdfcraft-render",
"pdfcraft-testkit",
"pdfcraft-xfa",
"sha2 0.11.1",
"thiserror 2.0.21",
]
@@ -4711,6 +4726,7 @@ dependencies = [
"pdfcraft-engine",
"pdfcraft-fonts",
"pdfcraft-model",
"pdfcraft-ocr",
"pdfcraft-platform",
"pdfcraft-render",
"pdfcraft-xfa",
README.md
+12
View File
@@ -253,6 +253,18 @@ Documents restricted by their author show a clear notice, and PdfCraft honours t
</tr>
</table>
## Scan pages, then read the words back before trusting them
**Scan & OCR ▸ Recognize text** turns a scan into searchable text — the current page, a range, or a folder of files written as searchable copies — with optional cleanup before reading (auto-rotate, deskew, denoise, binarize; word boxes map back to the original scan) and a second engine: an installed Tesseract joins the built-in one, merged per your strategy. A searchable result applies as one undoable step; the **Editable text** output opens the **OCR Verify screen** instead, where each word carries the engine's confidence as a colour, suspects are one keypress away, and corrections, a user dictionary and Re-OCR of a region or a page happen before Accept writes the new document. **Scan & OCR ▸ Correct recognized text** reviews any recognition there before it is applied.
<p align="center">
<img src="docs/images/pdfcraft-ocr-verify.png" alt="The OCR Verify screen: a scanned note with each recognized word boxed in its confidence colour, the info strip reporting a 95% average confidence and one low-confidence word, and the confidence legend with its thresholds below" width="100%">
<br>
<sub>The OCR Verify screen: words coloured by confidence — the legend spells the thresholds out — reviewed and corrected before anything is applied.</sub>
</p>
---
## Every tool, one keystroke away
Press <kbd>⌘K</kbd> to search every tool and command, or browse the **All tools** catalogue. Tools that are still in development are marked with the milestone that will ship them.
ROADMAP.md
+4
View File
@@ -329,6 +329,7 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md
- **2026-10-20 (M11, #3507):** Image merging keeps trailing comments, split operands and graphics state across content streams. Synthetic tool tests check saved text, rendered regions and exact image rows.
- **2026-10-17 (M8, gap 6, #2662):** A filled cover over a long line counts when it hides half of any one word, not only 40% of the line's words or box, so a black box over one number in a long line — or an unapplied redaction mark over it — no longer audits clean while the number is still extractable; a sliver under half of every word it touches and under 40% of the line is still not flagged
- **2026-10-13 (M10, OCR review, batch and two engines):** Scan & OCR gains a review step, a second engine and Unicode output. Recognition can clean the raster first (opt-in auto-rotate, deskew, denoise, binarize; word boxes map back to the scan) and read with the built-in ocrs engine, an installed tesseract program run as a fenced external process (a validated location that fails closed instead of falling back to the PATH, allow-listed languages, a private temp directory, timeout and output caps), or both: Automatic asks tesseract when the ocrs reading is weak or of unknown confidence, and a ROVER merge keeps the more confident word (an unscored ocrs word yields only to a tesseract word of 70 or more). The OCR Verify screen (Scan & OCR ▸ Correct recognized text) reviews a recognition before it is applied: confidence-coloured word boxes, corrections, a per-language user dictionary (a plain-text file in the preferences folder, so a word added from any document stays on disk after it closes), Re-OCR of a page or a region, Accept/Reject per page or whole. Pages accepted one by one each apply once; a region re-read moves the review marks of the words it keeps, and a whole-page re-read clears that page's marks (undoable in the panel). Output is searchable (an invisible layer, one undo step) or a NEW editable document of visible text. Words beyond WinAnsi are drawn in Type3 faces from craft-fonts with a ToUnicode map in the editable output (characters no bundled face has appear as '?'); the searchable layer writes them as Type3 fonts with empty character programs and a ToUnicode map, so extraction reads them back with no fonts and no substitutions; redaction removes that layer like any other text (tests in pdfcraft-redact; the end-to-end engine test needs the craft-fonts build input and is skipped without it). Batch runs skip pages that already have text unless forced, and skip whole files that have text only with the opt-in skip-files flag (off by default; Force OCR overrides both); they refuse two inputs of the same file name and existing outputs before reading anything, and report low-confidence pages at the settings' threshold. Headless: ocr_recognize (engine, strategy, preprocessing, region; the editable output goes to an `out` that must not exist yet, and `out` is refused for a searchable run), ocr_words, ocr_review_apply, ocr_dictionary, ocr_status and ocr_recognize_files. All 21 interface catalogs carry the new strings; the Lithuanian ones were added without native review. Reading with every cleaning step off passes the render through untouched, as on main (the 300 dpi normalization runs only when a cleaning step is on: enlarging a 72 dpi render 4x on its own made the models read "due: €250" as "ue: ?250"). Still missing: the deep scan clean-up options (descreen, background and edge-shadow removal), scan images in the editable output, matching the scanned typeface, and recognition quality beyond the tested English set.
- **2026-10-13 (M10, #1757):** Office export keeps each dotted-leader contents entry in one paragraph, with its page number, instead of inferring table cells from its separate text objects.
- **2026-10-22 (M6, calculation scripts):** The Calculate tab and a rename read simplified field notation only from a comment, where Acrobat writes it: a string that spells `BVCALC … EVCALC` is no longer taken for the notation, or rewritten by a rename. A `/` after the condition of `if`, `while`, `for` or `with` starts a regular expression, so a rename no longer rewrites a name inside one, and a quote inside one no longer hides the `getField` calls after it. Both found in review of #3550. A `//` comment ends at `\r`, U+2028 or U+2029 as well as `\n`, so in a script with `\r` line endings the code after one is no longer hidden from a rename, or taken for the notation. Estimates unchanged.
- **2026-10-20 (M6, calculation order page):** Prepare a form ▸ Calculation order, a page of its own, lists the calculated fields in the order they calculate with their formulas and values, moved with Up/Down arrows (as Acrobat's Set Field Calculation Order), Alt+↑/↓ or by dragging. A card says what is wrong with the order and holds Fix order; a loop has a card of its own that opens one of its formulas. On the page, each calculated field shows its step, and arrows show what the field in focus reads and what reads it. Estimates unchanged.
@@ -339,6 +340,7 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md
- **2026-10-20 (M12, #3673):** Character encoding accepts Identity-H/V fonts with Adobe CJK collections and recognizes predefined non-Identity CMaps independently. Unknown CMaps and foreign registries remain refused; synthetic checker and read-only automation regressions passed after failing before the fix. Per-glyph mapping validation and embedded-font extraction remain separate. Estimates unchanged.
- **2026-10-19 (M6, renames keep formulas):** Renaming a field renames it in every calculation that reads it, in the same undo step: `AFSimple_Calculate` field lists, simplified field notation (the comment and the JavaScript after it) and `getField` calls naming it in one string literal; the rest of each script is kept as written. A sum used to drop the renamed field, notation read it as 0 and a script kept a stale value. Field Properties says how many formulas changed and `form_set_props` lists them as `formulas_updated`. Names built at run time and other events' scripts are not rewritten. Estimates unchanged.
- **2026-10-19 (M6, text field table):** Prepare a form ▸ Text field table drags out an area and adds rows and columns of named text fields as one undoable edit, laid out on the page as displayed, with a live preview, per-column name patterns and widths, and refusal before anything is added when a cell would be too small or a name taken; `form_add_field_table` does the same headlessly. `form.multiple-copies` is partial: copying an existing field into a grid is still missing. Estimates unchanged.
- **2026-10-17 (research):** Added a [PDF workflow research reference](docs/research/pdf-workflows/README.md): dated public-source locators, 20 workflow research domains, a pinned feature mapping and interaction evidence. Reference documentation; no feature statuses, milestone priorities or estimates changed.
- **2026-10-16 (M7, #2516):** A paragraph number from `text_paragraphs` keeps naming the paragraph it listed after edits regroup the page: `text_edit`, `text_edit_layout` and `page_render`'s `hide_text_block` find it again by its text and box, and refuse one that was merged into a neighbour (the edited paragraph included) instead of rewriting the paragraph that took its number (`stale_paragraphs` lists the numbers gone). The inline editor's switch-paragraph click holds the same key. No stable identity across save and reopen yet (#2508).
- **2026-10-19 (M7, #3591):** A background added to a page that already has one is painted over the old one instead of going under it and staying hidden. Behind the content, backgrounds come first, in the order added, and then behind-content watermarks, so a watermark added after a background no longer disappears. Synthetic regressions in `crates/edit` check the order. Estimates unchanged.
@@ -355,6 +357,8 @@ Ranked; detail and the full milestone table in [docs/roadmap.md](docs/roadmap.md
- **2026-10-20 (M5, arrow keys move comments):** With a comment selected, the arrow keys move it a screen pixel and ⌘/Ctrl or Shift and an arrow ten, instead of scrolling the view or turning the page; up is up on screen at any zoom and rotation, and each press is one undo step. Text markup and locked comments stay put. Estimates unchanged.
- **2026-10-20 (M8, #3647):** Apply prunes fully removed tiling patterns, then checks retained pattern resources for text, images and unreadable content before reporting success. Shared patterns, Form-owned and inherited patterns, and cells kept by another page cause an atomic refusal when safe removal cannot be shown. Synthetic saved-file and automation regressions cover refusal, preserved marks and undo, and vector-only controls. Estimates unchanged.
- **2026-10-20 (M1, PR #1143 review):** Repair-aware dictionary/preflight edits and snapshot restoration preserve both accepted generations of a damaged live slot. Such edits use the existing full-save fallback; classic and packed rewrites coalesce aliases so page-tree parent links retain one object identity. Explicit generation replacement remains single-generation. Materialized xref diagnostics inspect retained primary and hybrid rows before folding, retaining the 256-object cap plus summary and discarding abandoned-index notes. Synthetic red/green controls cover preflight, undo restoration, equal values, explicit narrowing, encryption packing and four save variants. Estimates unchanged.
- **2026-10-19 (M14, #3373):** The main menu and its long submenus scroll in short windows, keeping their commands reachable. Estimates unchanged.
apps/pdfcraft-cli/tests/it/batch_exit.rs
+111
View File
@@ -0,0 +1,111 @@
//! The exit codes the CLI has today: `ExitCode::SUCCESS` (0) or `ExitCode::FAILURE` (1).
//!
//! Pinned here so that the exit-code contract is a visible, reviewed change when it comes: a
//! failed file, an outside-root path and a usage error all exit 1 today.
use serde_json::Value;
use std::path::{Path, PathBuf};
use std::process::{Command, Output};
const BIN: &str = env!("CARGO_BIN_EXE_pdfcraft-cli");
/// A fresh folder for one test, so the tests can run in parallel and leave nothing behind.
fn workdir(name: &str) -> PathBuf {
let dir = std::env::temp_dir().join(format!("pdfcraft-cli-exit-{}-{name}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
dir
}
/// `batch` with `args`, run from the working directory `dir`.
fn batch(dir: &Path, args: &[&str]) -> Output {
Command::new(BIN).current_dir(dir).arg("batch").args(args).output().unwrap()
}
/// `run --script --report` from the working directory `dir`: one JSON document for the run.
fn run_report(dir: &Path, script: &Path) -> Output {
Command::new(BIN).current_dir(dir).args(["run", "--script"]).arg(script).arg("--report").output().unwrap()
}
/// The one JSON document a command prints on stdout.
fn reported(output: &Output) -> Value {
serde_json::from_slice(&output.stdout).unwrap()
}
#[test]
fn batch_exits_0_when_every_file_succeeds() {
let dir = workdir("batch-ok");
std::fs::write(dir.join("page.txt"), "Exit code page").unwrap();
let output = batch(&dir, &["page.txt", "--steps", "convert", "--out-dir", "out"]);
assert_eq!(output.status.code(), Some(0), "{}", String::from_utf8_lossy(&output.stderr));
let report = reported(&output);
assert_eq!(report["completed"], 1, "{report}");
assert_eq!(report["failed"], 0, "{report}");
assert!(dir.join("out/page.pdf").exists(), "the converted page is written");
std::fs::remove_dir_all(&dir).unwrap();
}
#[test]
fn batch_exits_1_when_one_file_fails() {
let dir = workdir("batch-fail");
std::fs::write(dir.join("bad.pdf"), "broken").unwrap();
let output = batch(&dir, &["bad.pdf", "--steps", "convert", "--out-dir", "out"]);
assert_eq!(output.status.code(), Some(1), "{}", String::from_utf8_lossy(&output.stderr));
let report = reported(&output);
assert_eq!(report["failed"], 1, "{report}");
std::fs::remove_dir_all(&dir).unwrap();
}
#[test]
fn batch_usage_error_exits_1_and_creates_nothing() {
let dir = workdir("batch-usage");
std::fs::write(dir.join("page.txt"), "Exit code page").unwrap();
// Baseline: a usage error exits 1 today. A later change to the exit-code contract will
// update this assertion on purpose.
let output = batch(&dir, &["page.txt", "--out-dir", "out"]);
assert_eq!(output.status.code(), Some(1), "{}", String::from_utf8_lossy(&output.stderr));
assert!(output.stdout.is_empty(), "a usage error prints to stderr, not a report on stdout");
assert!(!output.stderr.is_empty(), "a usage error says what is wrong");
assert!(!dir.join("out").exists(), "a refused batch creates no output");
std::fs::remove_dir_all(&dir).unwrap();
}
#[test]
fn batch_outside_root_exits_1_and_creates_nothing() {
let dir = workdir("batch-outside-root");
let output = Command::new(BIN)
.current_dir(&dir)
.args(["batch", "../outside.pdf", "--steps", "convert", "--out-dir", "out", "--root"])
.arg(&dir)
.output()
.unwrap();
assert_eq!(output.status.code(), Some(1), "{}", String::from_utf8_lossy(&output.stderr));
let stderr = String::from_utf8_lossy(&output.stderr);
assert!(stderr.contains("is outside the allowed directory"), "the input is refused by the root: {stderr}");
assert!(output.stdout.is_empty(), "a refused batch prints no report");
assert!(!dir.join("out").exists(), "a refused batch creates no output");
std::fs::remove_dir_all(&dir).unwrap();
}
#[test]
fn run_script_report_exits_0_when_ok_and_1_when_a_step_fails() {
let dir = workdir("run-report");
let script = dir.join("steps.json");
let blank = serde_json::json!({ "tool": "doc_create", "args": { "from": "blank", "width": 100, "height": 100 } });
let steps = serde_json::json!([blank.clone(), { "tool": "no_such_tool", "args": {} }]);
std::fs::write(&script, steps.to_string()).unwrap();
let failing = run_report(&dir, &script);
assert_eq!(failing.status.code(), Some(1), "{}", String::from_utf8_lossy(&failing.stderr));
let report = reported(&failing);
assert_eq!(report["ok"], false, "{report}");
assert_eq!(report["steps"][1]["status"], "error", "{report}");
std::fs::write(&script, serde_json::json!([blank]).to_string()).unwrap();
let good = run_report(&dir, &script);
assert_eq!(good.status.code(), Some(0), "{}", String::from_utf8_lossy(&good.stderr));
let report = reported(&good);
assert_eq!(report["ok"], true, "{report}");
assert_eq!(report["steps"][0]["status"], "ok", "{report}");
std::fs::remove_dir_all(&dir).unwrap();
}
apps/pdfcraft-cli/tests/it/main.rs
+1
View File
@@ -3,6 +3,7 @@
mod audit;
mod automation_roots;
mod batch;
mod batch_exit;
mod check;
mod edit;
mod extract;
apps/pdfcraft/src/main.rs
+2
View File
@@ -439,6 +439,8 @@ fn app_creator<'a>(
if let Some(dir) = pdfcraft_ui_egui::RecoveryStore::default_dir() {
app.enable_recovery(pdfcraft_ui_egui::RecoveryStore::new(dir));
}
// The OCR Verify screen stores its per-language dictionary beside recovery data.
app.ocr_verify.dictionary_dir = pdfcraft_ui_egui::default_dictionary_dir();
if let Some(state) = &cc.wgpu_render_state {
let adapter = state.adapter.get_info();
let max_texture_dimension_2d = state.device.limits().max_texture_dimension_2d;
crates/automation/README.md
+23
View File
@@ -281,3 +281,26 @@ navigation-history tool; use the opt-in UI control channel to navigate the displ
(`#RRGGBB`, default black) for `text` and `date`. The colour is saved in the
FreeText appearance and survives reopening. Other mark/signature types reject
`color`; their appearance controls are separate.
## Reviewing OCR headlessly
`ocr_words {doc, retain_review: true}` reads words without editing the document and returns
`review`, plus pages and word boxes/confidences. Page and word positions are 1-based.
`region` can restrict recognition to one named page, using displayed-page points. Without
`retain_review`, word inspection remains stateless.
Apply reviewed text with `ocr_review_apply {doc, review, corrections: [{page: 1, word: 1,
text: "Corrected"}]}`. Omitted words keep their text; empty text removes a word. Geometry
comes from the retained recognition, never caller input. The captured document guard rejects
stale/foreign results. Successful application consumes the handle and is one undoable step.
Use `output_mode: "editable", out: "reviewed.pdf"` to write a separate editable document,
or `discard: true` to release the result without applying it. Failed validation preserves the
review for correction or discard. A new retained recognition replaces that document's prior
handle; closing the document releases it. At most eight documents may retain reviews, with
65,536-word/4-MiB input bounds and a 65,536-page cap.
`ocr_dictionary {path: "ocr-dictionary-en.txt", add: ["Quartz"], remove: ["old"]}` atomically
updates the same one-word-per-UTF-8-line format OCR Verify uses. Omit add/remove to read it.
The explicit path honors `--root`, including symlink confinement; choose the appropriate
per-language file used by the UI. Entries are case-insensitive and duplicate-free. Invalid
or oversized inputs fail before writing (65,536 words, 4 MiB, at most 1,024 bytes per word).
crates/automation/src/a11y/ocr.rs
+472 -36
View File
@@ -1,98 +1,534 @@
use super::{Args, Automation, Result, ToolError, Value, child, failed, json, write_atomic};
impl Automation {
/// Recognition settings from the arguments: `dpi` (clamped to 72–600), `language` (refused
/// unless the models read it) and `skip_text_pages`, each left at its default when absent.
/// The settings every OCR tool shares: dpi, language, the engine/strategy choice, the four
/// preprocessing flags, the skip flags and the output mode — each named exactly as its tool
/// schema says. An unknown engine, strategy or mode is an honest error, never a silent
/// fallback: a tool run must not quietly do something other than what it was asked.
pub(crate) fn ocr_settings(&self, a: &Args) -> Result<pdfcraft_engine::ocr::OcrSettings> {
use pdfcraft_engine::ocr::{EngineChoice, MergeStrategy, OutputMode};
let mut settings = pdfcraft_engine::ocr::OcrSettings::default();
if let Some(d) = a.opt_num("dpi")? {
settings.dpi = d.clamp(72.0, 600.0) as f32;
}
if let Some(l) = a.opt_str("language")? {
if !pdfcraft_engine::ocr::LANGUAGES.iter().any(|x| x.0 == l) {
return Err(ToolError::InvalidArgs(format!("unsupported language {l:?}")));
return Err(ToolError::InvalidArgs(format!(
"unsupported language {l:?} (supported: {})",
pdfcraft_engine::ocr::LANGUAGES.iter().map(|(c, _)| *c).collect::<Vec<_>>().join(", ")
)));
}
settings.language = l.into();
}
if let Some(s) = a.opt_bool("skip_text_pages")? {
settings.skip_text_pages = s;
if let Some(e) = a.opt_str("engine")? {
settings.engine = match e.trim().to_ascii_lowercase().as_str() {
"ocrs" => EngineChoice::Ocrs,
"tesseract" => EngineChoice::Tesseract,
"auto" | "automatic" | "ensemble" => EngineChoice::Auto,
other => return Err(ToolError::InvalidArgs(format!("unsupported engine {other:?} (ocrs, tesseract or auto)"))),
};
}
settings.strategy = match a.opt_str("strategy")? {
Some(s) => match s.trim().to_ascii_lowercase().as_str() {
"primary" | "primary-only" | "primary_only" => MergeStrategy::PrimaryOnly,
"confidence" | "confidence-weighted" | "confidence_weighted" => MergeStrategy::ConfidenceWeighted,
"rover" | "rover-vote" | "rover_vote" | "vote" => MergeStrategy::RoverVote,
other => return Err(ToolError::InvalidArgs(format!("unsupported strategy {other:?} (primary, confidence or rover)"))),
},
// Unset: the engine choice decides (auto brings in the second engine when the
// first reading is weak; a named engine reads alone).
None => settings.engine.default_strategy(),
};
if let Some(m) = a.opt_str("output_mode")? {
settings.output_mode = match m.trim().to_ascii_lowercase().as_str() {
"searchable" | "searchable-image" | "searchable_image" => OutputMode::Searchable,
"editable" | "editable-text" | "editable_text" => OutputMode::EditableText,
other => return Err(ToolError::InvalidArgs(format!("unsupported output mode {other:?} (searchable or editable)"))),
};
}
for (key, flag) in [
("auto_rotate", &mut settings.auto_rotate),
("deskew", &mut settings.deskew),
("denoise", &mut settings.denoise),
("binarize", &mut settings.binarize),
] {
if let Some(on) = a.opt_bool(key)? {
*flag = on;
}
}
for (key, flag) in [
("skip_text_pages", &mut settings.skip_text_pages),
("skip_text_files", &mut settings.skip_text_files),
("force_ocr", &mut settings.force_ocr),
] {
if let Some(on) = a.opt_bool(key)? {
*flag = on;
}
}
Ok(settings)
}
/// Recognizes text in each of `paths` and writes the result under the same file name in
/// `folder`, reporting per file its output, word count and skipped pages, or its error. A
/// path outside the root refuses the whole call before any work starts.
/// The optional `region` argument ([x0, y0, x1, y1] in points from the top-left of the
/// displayed page) as the job's pixel crop, at the resolution that page is really read at.
/// A region reads exactly one page.
fn ocr_region(
&self,
a: &Args,
id: pdfcraft_engine::DocId,
page: usize,
settings: &pdfcraft_engine::ocr::OcrSettings,
) -> Result<Option<[f32; 4]>> {
const BAD: &str = "region must be [x0, y0, x1, y1] in points";
let Some(v) = a.get("region") else { return Ok(None) };
let arr = v.as_array().ok_or_else(|| ToolError::InvalidArgs(BAD.into()))?;
if arr.len() != 4 {
return Err(ToolError::InvalidArgs(BAD.into()));
}
let mut pts = [0.0f64; 4];
for (i, x) in arr.iter().enumerate() {
pts[i] = x.as_f64().filter(|f| f.is_finite()).ok_or_else(|| ToolError::InvalidArgs(BAD.into()))?;
}
let scale = f64::from(
self.session.ocr_job(id, &[page], settings.clone()).ok_or_else(|| failed("the document could not be read"))?.render_scale(page),
);
let px: Vec<f32> = pts.iter().map(|p| scale * p).map(|p| if p.is_finite() { p } else { 0.0 }).map(|p| p as f32).collect();
Ok(Some([px[0], px[1], px[2], px[3]]))
}
/// The per-page result entries the OCR tools return: recognised text or why the page was
/// passed over. The text itself is only echoed when the document allows copying; the
/// counts and metadata always are.
fn ocr_pages_json(found: &[pdfcraft_engine::ocr::OcrPage], include_text: bool) -> Vec<Value> {
found
.iter()
.map(|p| match &p.skipped {
Some(why) => json!({ "page": p.page + 1, "skipped": why }),
None => {
let mut page = json!({ "page": p.page + 1, "words": p.words.len(), "already_on_page": p.already_on_page });
// The editable output's substitution warnings (characters no font could
// show) are part of the result: reported per page, never dropped.
if !p.notes.is_empty() {
page["notes"] = json!(p.notes);
}
if include_text {
page["text"] = json!(p.text());
}
page
}
})
.collect()
}
pub(crate) fn ocr_recognize_files(&mut self, a: &Args) -> Result<Value> {
let settings = self.ocr_settings(a)?;
let folder = self.resolve(a.str("folder")?, true)?;
// A path that leaves the root refuses the whole call before any work starts, like the
// other tools (the write check passes missing files, which stay per-file errors below).
// A path that leaves the read root refuses the whole call before any work starts;
// a file that is merely missing stays a per-file error below.
let paths = a.strs("paths")?;
for p in &paths {
self.resolve_for(p, crate::file_access::Access::Read, true)?;
}
// Two inputs that would write the same output file overwrite each other while both report
// success, so the whole batch is refused here: before the folder is created, anything is
// read, or the OCR engines load.
// Two inputs of one file name (from different folders) would write the same output:
// the second would replace the first while both report success. The whole batch is
// refused up front — before the folder is created, anything is read or the engines
// load — naming the file, the way the default filesystem compares names.
let targets: Vec<std::path::PathBuf> = paths.iter().map(|p| child(&folder, &input_file_name(p))).collect();
let shared = shared_file_names(&targets);
if !shared.is_empty() {
return Err(ToolError::InvalidArgs(format!(
"more than one input would write to the same output file, so each would replace the one before it. Colliding file names: {}. Give each input a different file name, or run them in separate batches with different folders.",
"several paths share a file name, so their outputs in folder would overwrite each other (rename or run them separately): {}",
shared.join(", ")
)));
}
std::fs::create_dir_all(&folder).map_err(|e| failed(e.to_string()))?;
let ocr = pdfcraft_engine::ocr::engine().map_err(failed)?;
let names: Vec<String> = paths.iter().map(|p| input_file_name(p)).collect();
// Collision pre-flight: a searchable copy never silently overwrites. Every target is
// checked before the first page is read; a collision aborts the whole batch with the
// names in the error.
let mut collisions = std::collections::BTreeSet::new();
for name in names {
if child(&folder, &name).exists() {
collisions.insert(name);
}
}
if !collisions.is_empty() {
let mut list: Vec<String> = collisions.iter().take(5).cloned().collect();
if collisions.len() > list.len() {
list.push(format!("and {} more", collisions.len() - list.len()));
}
return Err(failed(format!(
"the output folder already holds {} (delete or move them, or choose another folder): {}",
if collisions.len() == 1 { "a file of that name" } else { "files of those names" },
list.join(", ")
)));
}
// The engines are built (and named if they cannot be) before the first file is read.
let recognizers = pdfcraft_engine::ocr::recognizers(&settings).map_err(failed)?;
let mut out = Vec::new();
let (mut ok, mut skipped_files, mut words) = (0usize, 0usize, 0usize);
for p in paths {
let name = input_file_name(p);
let result =
self.resolve(p, false).map_err(|e| e.to_string()).and_then(|src| crate::read_input_bytes(&src).map_err(|e| e.to_string())).and_then(
|bytes| pdfcraft_engine::ocr::recognize_file(&name, std::sync::Arc::new(bytes), None, settings.clone(), &ocr, |_, _| true),
|bytes| {
pdfcraft_engine::ocr::recognize_file(&name, std::sync::Arc::new(bytes), None, settings.clone(), &recognizers, |_, _| true)
},
);
out.push(match result {
match result {
Ok(r) => {
let skipped_pages: Vec<usize> = r.pages.iter().filter(|p| p.skipped.is_some()).map(|p| p.page + 1).collect();
if r.skipped() {
// A file passed over by the skip-files option is its own bucket: its
// original is left alone and nothing is written.
skipped_files += 1;
out.push(json!({ "path": p, "skipped": r.file_skipped, "skipped_pages": skipped_pages }));
continue;
}
let target = child(&folder, &name);
write_atomic(&target, &r.bytes)?;
let skipped: Vec<usize> = r.pages.iter().filter(|p| p.skipped.is_some()).map(|p| p.page + 1).collect();
json!({ "path": p, "output": target.to_string_lossy(), "words": r.words(), "skipped_pages": skipped })
ok += 1;
words += r.words();
let mut entry = json!({ "path": p, "output": target.to_string_lossy(), "words": r.words(), "skipped_pages": skipped_pages });
// The batch's low-confidence note: at or under the settings' threshold (60
// by default), on 1-based sorted pages.
let threshold = settings.low_confidence_threshold;
if let Some((count, pages)) = pdfcraft_engine::ocr::low_confidence_pages(&r.pages, threshold) {
entry["low_confidence"] = json!({ "count": count, "pages": pages, "threshold": threshold });
}
out.push(entry);
}
Err(e) => json!({ "path": p, "error": e }),
});
Err(e) => out.push(json!({ "path": p, "error": e })),
}
}
Ok(json!({ "files": out }))
Ok(json!({ "files": out, "recognized": ok, "words": words, "skipped_files": skipped_files }))
}
/// Recognizes text on the open document's `pages` (every page when absent) and adds it,
/// reporting the total word count and, per page, why it was skipped or what it found.
pub(crate) fn ocr_recognize(&mut self, a: &Args) -> Result<Value> {
let settings = self.ocr_settings(a)?;
use pdfcraft_engine::ocr::OutputMode;
let mut settings = self.ocr_settings(a)?;
let pages = if a.opt_ints("pages")?.is_some() { self.pages(a, "pages")? } else { Vec::new() };
let doc = self.doc(a)?;
// Recognizing needs only permission to change the document; the words it found are
// returned only when copying is allowed too.
let (id, copy) = (doc.id, doc.allows_copying());
let found = self.session.recognize_text(id, &pages, settings).map_err(failed)?;
let (id, copy) = {
let doc = self.doc(a)?;
(doc.id, doc.allows_copying())
};
// The optional region reads exactly one page.
if a.get("region").is_some() {
if pages.len() != 1 {
return Err(ToolError::InvalidArgs("region reads one page: name it in pages".into()));
}
settings.region = self.ocr_region(a, id, pages[0], &settings)?;
}
// The editable output's destination is checked before anything runs: a NEW file, never
// the open source and never an existing file replaced silently.
let out = match settings.output_mode {
OutputMode::Searchable => {
// `out` belongs to the editable output; a searchable run writes into the open
// document, so an `out` here would be silently ignored (and never checked
// against the root).
if a.get("out").is_some() {
return Err(ToolError::InvalidArgs("out is only used with output_mode editable; remove it, or choose editable".into()));
}
None
}
OutputMode::EditableText => {
// The editable output writes the recognized text out of the document: a
// document that forbids copying refuses here, before anything runs.
self.doc_to_copy(a)?;
// A write: the file does not exist yet, and write_atomic makes its folder.
let out = self.resolve(a.str("out")?, true)?;
if let Some(source) = self.session.get(id).and_then(|d| d.path.clone())
&& let (Ok(a), Ok(b)) = (std::fs::canonicalize(&out), std::fs::canonicalize(&source))
&& a == b
{
return Err(failed("out is the source document; the editable output is a new file, so choose another path"));
}
if out.exists() {
return Err(failed(format!(
"out already exists ({}); the editable output is a new file, so choose a path that does not exist yet",
out.to_string_lossy()
)));
}
Some(out)
}
};
// Honest up front: the engines are built (and named if they cannot be) before anything
// is read or written.
let recognizers = pdfcraft_engine::ocr::recognizers(&settings).map_err(failed)?;
match out {
None => {
let found = self.session.recognize_text(id, &pages, settings).map_err(failed)?;
let words = found.iter().map(|p| p.words.len()).sum::<usize>();
Ok(json!({ "words": words, "pages": Self::ocr_pages_json(&found, copy) }))
}
Some(out) => {
let job = self.session.ocr_job(id, &pages, settings).ok_or_else(|| failed("the document could not be read"))?;
let found = job.run(&recognizers, |_, _| true);
let bytes = self.session.editable_ocr_bytes(id, &found).map_err(failed)?;
// recognition can take minutes: a file created meanwhile is refused, not replaced
if let Some(folder) = out.parent() {
std::fs::create_dir_all(folder).map_err(failed)?;
}
crate::bates::publish_new(&out, &bytes)?;
Ok(json!({
"out": out.to_string_lossy(),
"words": found.iter().map(|p| p.words.len()).sum::<usize>(),
"pages": Self::ocr_pages_json(&found, copy),
}))
}
}
}
/// The words of a recognition, for reviewing without the GUI: text, box (points from the
/// top-left of the displayed page), confidence, band, and which words are suspects.
/// Changes nothing; needs the engines (ocr_status).
pub(crate) fn ocr_words(&mut self, a: &Args) -> Result<Value> {
let mut settings = self.ocr_settings(a)?;
let pages = if a.opt_ints("pages")?.is_some() { self.pages(a, "pages")? } else { Vec::new() };
// The word list is the document's text, word by word: refused when the document
// forbids copying it out.
let id = self.doc_to_copy(a)?.id;
if a.get("region").is_some() {
if pages.len() != 1 {
return Err(ToolError::InvalidArgs("region reads one page: name it in pages".into()));
}
settings.region = self.ocr_region(a, id, pages[0], &settings)?;
}
let retain = a.opt_bool("retain_review")?.unwrap_or(false);
if retain && !self.ocr_reviews.contains_key(&id) && self.ocr_reviews.len() >= 8 {
return Err(failed("at most 8 OCR reviews may be retained; discard one first"));
}
let recognizers = pdfcraft_engine::ocr::recognizers(&settings).map_err(failed)?;
let job = self.session.ocr_job(id, &pages, settings.clone()).ok_or_else(|| failed("the document could not be read"))?;
let infos: std::collections::HashMap<usize, pdfcraft_render::PageInfo> =
self.session.get(id).map(|d| d.info.pages.clone()).unwrap_or_default().into_iter().enumerate().collect();
let guard = job.guard.clone();
let found = job.run(&recognizers, |_, _| true);
if found.len() > 65536 {
return Err(failed("at most 65536 OCR review pages are allowed"));
}
pdfcraft_engine::ocr::validate_ocr_words(found.iter().flat_map(|p| &p.words)).map_err(failed)?;
let band_name = |band: pdfcraft_engine::ocr::ConfidenceBand| match band {
pdfcraft_engine::ocr::ConfidenceBand::High => "high",
pdfcraft_engine::ocr::ConfidenceBand::Medium => "medium",
pdfcraft_engine::ocr::ConfidenceBand::Low => "low",
};
let mut suspects = 0usize;
let list: Vec<Value> = found
.iter()
.map(|p| match &p.skipped {
Some(why) => json!({ "page": p.page + 1, "skipped": why }),
None if copy => json!({ "page": p.page + 1, "words": p.words.len(), "already_on_page": p.already_on_page, "text": p.text() }),
None => json!({ "page": p.page + 1, "words": p.words.len(), "already_on_page": p.already_on_page }),
.map(|p| {
let info = infos.get(&p.page);
let words: Vec<Value> = p
.words
.iter()
.map(|w| {
let suspect = pdfcraft_engine::ocr::is_suspect(w.confidence);
if suspect {
suspects += 1;
}
// The word's corners in view space, as the box [x, y, w, h] in points
// from the top-left of the displayed page.
let view = |pt: [f64; 2]| info.map(|i| i.user_to_view(pt[0] as f32, pt[1] as f32));
let corners = [
view(w.origin),
view([w.origin[0] + w.across[0], w.origin[1] + w.across[1]]),
view([w.origin[0] + w.up[0], w.origin[1] + w.up[1]]),
view([w.origin[0] + w.across[0] + w.up[0], w.origin[1] + w.across[1] + w.up[1]]),
];
let xs: Vec<f32> = corners.iter().flatten().map(|v| v[0]).collect();
let ys: Vec<f32> = corners.iter().flatten().map(|v| v[1]).collect();
let (min_x, min_y) = (xs.iter().cloned().fold(f32::INFINITY, f32::min), ys.iter().cloned().fold(f32::INFINITY, f32::min));
let (w_pt, h_pt) = (
xs.iter().cloned().fold(f32::NEG_INFINITY, f32::max) - min_x,
ys.iter().cloned().fold(f32::NEG_INFINITY, f32::max) - min_y,
);
json!({
"text": w.text,
"box": [min_x, min_y, w_pt, h_pt],
"confidence": w.confidence.map(|c| (c * 10.0).round() / 10.0),
"band": w.confidence.map(|c| band_name(pdfcraft_engine::ocr::band_for(c))),
"suspect": suspect,
"source": w.source,
})
})
.collect();
let mut page =
json!({ "page": p.page + 1, "already_on_page": p.already_on_page, "words": words, "mean_confidence": p.mean_confidence().map(|c| (c * 10.0).round() / 10.0) });
if let Some(why) = &p.skipped {
page["skipped"] = json!(why);
}
page
})
.collect();
Ok(json!({ "words": found.iter().map(|p| p.words.len()).sum::<usize>(), "pages": list }))
let mut out = json!({ "words": found.iter().map(|p| p.words.len()).sum::<usize>(), "suspects": suspects, "pages": list });
if retain {
self.next_ocr_review =
self.next_ocr_review.checked_add(1).filter(|n| *n <= i64::MAX as u64).ok_or_else(|| failed("OCR review handles exhausted"))?;
out["review"] = json!(self.next_ocr_review);
self.ocr_reviews.insert(id, crate::OcrReview { handle: self.next_ocr_review, guard, pages: found });
}
Ok(out)
}
/// Apply only caller corrections to retained original boxes, through the stored guard.
pub(crate) fn ocr_review_apply(&mut self, a: &Args) -> Result<Value> {
let id = self.doc(a)?.id;
let handle = a.int("review")?;
let review = self
.ocr_reviews
.get(&id)
.filter(|r| i64::try_from(r.handle).ok() == Some(handle))
.ok_or_else(|| failed("unknown OCR review for this document; call ocr_words with retain_review"))?;
if a.opt_bool("discard")?.unwrap_or(false) {
if a.get("corrections").is_some() {
return Err(ToolError::InvalidArgs("discard cannot include corrections".into()));
}
self.ocr_reviews.remove(&id);
return Ok(json!({ "discarded": handle }));
}
if let Some(why) = self.session.stale_reason(&review.guard) {
return Err(failed(why));
}
let corrections = a.get("corrections").map(|v| v.as_array().ok_or_else(|| Args::wrong("corrections", "an array"))).transpose()?;
if corrections.is_some_and(|c| c.len() > 65536) {
return Err(failed("at most 65536 OCR corrections are allowed"));
}
// Prevalidate and cap every replacement before cloning the retained result.
let mut edits = Vec::new();
let mut bytes = 0usize;
let mut seen = std::collections::BTreeSet::new();
let page_indices: std::collections::BTreeMap<_, _> = review.pages.iter().enumerate().map(|(i, p)| (p.page, i)).collect();
for item in corrections.into_iter().flatten() {
if !item.is_object() || item.as_object().is_some_and(|o| o.keys().any(|k| !matches!(k.as_str(), "page" | "word" | "text"))) {
return Err(ToolError::InvalidArgs("corrections accept only page, word and text".into()));
}
let c = Args(item);
let page =
usize::try_from(c.int("page")?).ok().and_then(|p| p.checked_sub(1)).ok_or_else(|| Args::wrong("page", "a 1-based position"))?;
let word =
usize::try_from(c.int("word")?).ok().and_then(|w| w.checked_sub(1)).ok_or_else(|| Args::wrong("word", "a 1-based position"))?;
let text = c.str("text")?;
bytes = bytes.checked_add(text.len()).filter(|n| *n <= 4 * 1024 * 1024).ok_or_else(|| failed("OCR corrections exceed 4 MiB"))?;
if !seen.insert((page, word)) {
return Err(failed("duplicate OCR word correction"));
}
let pi = page_indices.get(&page).copied().ok_or_else(|| failed("page is not in this OCR review"))?;
if review.pages.get(pi).and_then(|p| p.words.get(word)).is_none() {
return Err(failed("word is not in this OCR review"));
}
edits.push((pi, word, text.to_owned()));
}
let mut pages = review.pages.clone();
for (pi, wi, text) in edits {
if let Some(w) = pages.get_mut(pi).and_then(|p| p.words.get_mut(wi)) {
w.text = text;
}
}
pdfcraft_engine::ocr::validate_ocr_words(pages.iter().flat_map(|p| &p.words)).map_err(failed)?;
for page in &mut pages {
page.words.retain(|w| !w.text.trim().is_empty());
}
if pages.iter().all(|p| p.words.is_empty()) {
return Err(failed("the OCR review has no words to apply"));
}
let out = match a.opt_str("output_mode")?.unwrap_or("searchable") {
"searchable" => {
if a.get("out").is_some() {
return Err(ToolError::InvalidArgs("out is only for editable output".into()));
}
let count = self.session.apply_ocr_checked(id, &pages, &review.guard).map_err(failed)?;
json!({ "applied": count, "review": handle })
}
"editable" => {
// The editable output writes the review's words out of the document: a
// document that forbids copying refuses here (the review is kept).
self.doc_to_copy(a)?;
let target = self.resolve(a.str("out")?, true)?;
if self
.session
.get(id)
.and_then(|d| d.path.as_deref())
.and_then(|p| std::fs::canonicalize(p).ok())
.as_ref()
.is_some_and(|source| std::fs::canonicalize(&target).as_ref().ok() == Some(source))
{
return Err(failed("out is the source document; choose a new file"));
}
// A NEW file: an existing one is never replaced silently (the review is kept).
if target.exists() {
return Err(failed(format!("out already exists ({}); choose a path that does not exist yet", target.to_string_lossy())));
}
let bytes = self.session.editable_ocr_bytes(id, &pages).map_err(failed)?;
write_atomic(&target, &bytes)?;
json!({ "out": target.to_string_lossy(), "review": handle, "words": pages.iter().map(|p| p.words.len()).sum::<usize>() })
}
_ => return Err(ToolError::InvalidArgs("output_mode must be searchable or editable".into())),
};
self.ocr_reviews.remove(&id);
Ok(out)
}
/// Read/update the same per-language dictionary file used by OCR Verify.
pub(crate) fn ocr_dictionary(&mut self, a: &Args) -> Result<Value> {
use pdfcraft_engine::ocr::dictionary;
use std::io::Read as _;
for key in ["add", "remove"] {
if let Some(value) = a.get(key) {
let values = value.as_array().ok_or_else(|| Args::wrong(key, "an array of strings"))?;
if values.len() > 65536 {
return Err(failed("at most 65536 dictionary words are allowed"));
}
}
}
let add = if a.get("add").is_some() { a.strs("add")? } else { Vec::new() };
let remove = if a.get("remove").is_some() { a.strs("remove")? } else { Vec::new() };
let writing = a.get("add").is_some() || a.get("remove").is_some();
// Updating sorts, lower-cases and rewrites the file, so this tool names dictionary files only:
// pointed at `.env` or a README it would corrupt it.
let file = std::path::Path::new(a.str("path")?).file_name().and_then(|n| n.to_str()).unwrap_or("");
let plain = file.chars().all(|c| c.is_ascii_alphanumeric() || matches!(c, '-' | '_' | '.'));
if file.len() > 64 || !plain || !file.starts_with("ocr-dictionary") || !file.ends_with(".txt") {
return Err(failed("path must name a dictionary file: ocr-dictionary-<language>.txt (letters, digits, '-', '_' and '.' only)"));
}
let path = self.resolve(a.str("path")?, writing)?;
let mut input = String::new();
match std::fs::File::open(&path) {
Ok(file) => {
file.take((dictionary::MAX_BYTES + 1) as u64).read_to_string(&mut input).map_err(|e| failed(format!("OCR dictionary: {e}")))?;
}
Err(e) if e.kind() == std::io::ErrorKind::NotFound && writing => {}
Err(e) => return Err(failed(format!("OCR dictionary: {e}"))),
}
let text = dictionary::update(&input, &add, &remove).map_err(failed)?;
if writing {
write_atomic(&path, text.as_bytes())?;
}
let words = dictionary::words(&text).map_err(failed)?;
Ok(json!({ "path": path.to_string_lossy(), "count": words.len(), "words": words }))
}
/// Whether text recognition's models are installed, the directories searched for them (in
/// order), and the languages it reads, each as a code and a name.
pub(crate) fn ocr_status(&self) -> Result<Value> {
use pdfcraft_engine::ocr;
let dirs: Vec<String> = ocr::Models::search_dirs().iter().map(|d| d.to_string_lossy().into_owned()).collect();
let langs: Vec<Value> = ocr::LANGUAGES.iter().map(|(c, n)| json!({ "code": c, "name": n })).collect();
Ok(json!({ "available": ocr::available(), "search_dirs": dirs, "languages": langs }))
let engines: Vec<Value> = ocr::engines_status()
.iter()
.map(|e| {
let mut v = json!({ "id": e.id, "available": e.available });
if let Some(p) = &e.program {
v["path"] = json!(p);
}
if let Some(vv) = &e.version {
v["version"] = json!(vv);
}
if let Some(why) = &e.why_not {
v["unavailable"] = json!(why);
}
v
})
.collect();
Ok(json!({ "available": ocr::available(), "search_dirs": dirs, "languages": langs, "engines": engines }))
}
}
crates/automation/src/batch.rs
+2 -2
View File
@@ -84,8 +84,8 @@ impl<'a> Recipe<'a> {
match *step {
"convert" => bytes = session.convert_to_pdf(name, &bytes)?.1,
"ocr" => {
let engine = ocr::engine().map_err(failed)?;
let result = ocr::recognize_file(name, bytes, None, self.ocr.clone(), &engine, |_, _| true).map_err(failed)?;
let recognizers = ocr::recognizers(&self.ocr).map_err(failed)?;
let result = ocr::recognize_file(name, bytes, None, self.ocr.clone(), &recognizers, |_, _| true).map_err(failed)?;
log.push(
json!({"step":"ocr","words":result.words(),"skipped_pages":result.pages.iter().filter(|p| p.skipped.is_some()).count()}),
);
crates/automation/src/bates.rs
+1 -1
View File
@@ -199,7 +199,7 @@ impl Automation {
}
/// Publish a PDF atomically while refusing a target created by another process after preflight.
fn publish_new(path: &Path, bytes: &[u8]) -> Result<()> {
pub(super) fn publish_new(path: &Path, bytes: &[u8]) -> Result<()> {
let parent = path.parent().ok_or_else(|| failed("output has no parent directory"))?;
let mut temp = tempfile::NamedTempFile::new_in(parent).map_err(failed)?;
temp.write_all(bytes).map_err(failed)?;
crates/automation/src/dispatch.rs
+3
View File
@@ -812,6 +812,9 @@ impl Automation {
"form_set_actions" => self.form_set_actions(&a)?,
"ocr_status" => self.ocr_status()?,
"ocr_recognize_files" => self.ocr_recognize_files(&a)?,
"ocr_words" => self.ocr_words(&a)?,
"ocr_review_apply" => self.ocr_review_apply(&a)?,
"ocr_dictionary" => self.ocr_dictionary(&a)?,
"accessibility_report" => self.a11y_report(&a)?,
"accessibility_review_record" => self.a11y_review_record(&a)?,
"accessibility_review_save" => self.a11y_review_save(&a)?,
crates/automation/src/documents.rs
+1
View File
@@ -120,6 +120,7 @@ impl Automation {
self.texts.remove(&id);
self.listed.retain(|(d, _), _| *d != id);
self.evicted.remove(&id);
self.ocr_reviews.remove(&id);
Ok(json!({ "closed": id.0 }))
}
crates/automation/src/lib.rs
+11
View File
@@ -76,6 +76,12 @@ pub(crate) use paths::{read_input, read_input_bytes};
#[cfg(test)]
use paths::{write_atomic_synced, write_atomic_with};
struct OcrReview {
handle: u64,
guard: pdfcraft_engine::ocr::OcrGuard,
pages: Vec<pdfcraft_engine::ocr::OcrPage>,
}
/// A headless PdfCraft session driven by tool calls.
pub struct Automation {
session: Session,
@@ -92,6 +98,9 @@ pub struct Automation {
/// Documents that lost a listing to the cap. A page of one with no listing is refused rather
/// than read as "a number is a position", which the page may no longer match.
evicted: HashSet<DocId>,
/// Review sessions held for `ocr_words` (a guard and the pages read), by document.
ocr_reviews: HashMap<DocId, OcrReview>,
next_ocr_review: u64,
}
impl Default for Automation {
@@ -110,6 +119,8 @@ impl Automation {
texts: TextCache::default(),
listed: HashMap::new(),
evicted: HashSet::new(),
ocr_reviews: HashMap::new(),
next_ocr_review: 0,
}
}
crates/automation/src/tools.rs
+71 -9
View File
@@ -5,6 +5,8 @@
use serde_json::{Value, json};
use pdfcraft_engine::REDACT_PATTERNS;
use crate::ToolError;
#[derive(Clone, Debug)]
@@ -30,6 +32,12 @@ fn pages(what: &str) -> Value {
json!({ "type": "array", "items": { "type": "integer", "minimum": 1 }, "minItems": 1, "description": format!("1-based page numbers {what}.") })
}
/// The Search & Redact pattern ids, from the engine's registry, so the advertised schema cannot
/// drift from the patterns the runtime accepts.
fn pattern_enum() -> Value {
json!({ "type": "string", "enum": REDACT_PATTERNS.iter().map(|p| p.id()).collect::<Vec<&str>>() })
}
fn path(what: &str) -> Value {
json!({ "type": "array", "items": { "type": "integer", "minimum": 1 }, "description": format!("{what}: 1-based positions from the top level, e.g. [2, 1] = the first child of the second bookmark.") })
}
@@ -77,6 +85,12 @@ fn schema(props: Value, required: &[&str]) -> Value {
json!({ "type": "object", "properties": props, "required": required, "additionalProperties": false })
}
/// The language codes the engines read, as the `language` enum — from the engine's own table,
/// so the schema and the validator can never disagree.
fn language_enum() -> Value {
Value::Array(pdfcraft_engine::ocr::LANGUAGES.iter().map(|(code, _)| json!(code)).collect())
}
struct T {
name: &'static str,
title: &'static str,
@@ -605,7 +619,7 @@ pub fn tools() -> Vec<ToolDef> {
t(
"redact_mark",
"Mark for redaction",
"Mark content for redaction (nothing is removed until redact_apply). One of: rect [x0, y0, x1, y1] (points from the top-left of the displayed page) with page; find (text, every match); words (a list of words or phrases, every match of each; the result's matched_words says how many marks each made); pattern (phone, email, credit-card, ssn, date: every match, Acrobat's Search & Redact patterns); whole_pages: true. find and words accept case_sensitive and whole_words (both default false). Select one page with page or multiple with pages (default all); do not pass both. rect requires page. overlay: text shown on the box once applied, or code_set (foia: U.S. FOIA (b)(1)(A)…(b)(9); privacy-act: U.S. Privacy Act (d)(5), (j)(1)…(k)(7)) with codes, shown as \"(b)(6), (b)(7)(C)\"; fill: box colour (default black). Undoable.",
"Mark content for redaction (nothing is removed until redact_apply). One of: rect [x0, y0, x1, y1] (points from the top-left of the displayed page) with page; find (text, every match); words (a list of words or phrases, every match of each; the result's matched_words says how many marks each made); pattern (phone, email, credit-card, ssn, date, iban: every match, Acrobat's Search & Redact patterns); whole_pages: true. find and words accept case_sensitive and whole_words (both default false). Select one page with page or multiple with pages (default all); do not pass both. rect requires page. overlay: text shown on the box once applied, or code_set (foia: U.S. FOIA (b)(1)(A)…(b)(9); privacy-act: U.S. Privacy Act (d)(5), (j)(1)…(k)(7)) with codes, shown as \"(b)(6), (b)(7)(C)\"; fill: box colour (default black). Undoable.",
)
.with(schema(
json!({
@@ -616,7 +630,7 @@ pub fn tools() -> Vec<ToolDef> {
"words": { "type": "array", "items": { "type": "string", "minLength": 1 }, "minItems": 1, "maxItems": 1000 },
"case_sensitive": { "type": "boolean", "description": "Match letter case for find or words (default false)." },
"whole_words": { "type": "boolean", "description": "Match whole words for find or words (default false)." },
"pattern": { "type": "string", "enum": ["phone", "email", "credit-card", "ssn", "date"] },
"pattern": pattern_enum(),
"whole_pages": { "type": "boolean" },
"pages": pages("to search or mark (default: all); mutually exclusive with page, and not valid with rect"),
"overlay": { "type": "string" },
@@ -1365,31 +1379,79 @@ pub fn tools() -> Vec<ToolDef> {
)),
t("js_enabled", "JavaScript on or off", "Preferences ▸ JavaScript ▸ Enable Acrobat JavaScript: set it with `enabled`, or read it. With JavaScript off, field scripts other than Acrobat's AF calls don't run.")
.with(schema(json!({ "enabled": { "type": "boolean" } }), &[])),
t("ocr_recognize", "Recognize text (OCR)", "Scan & OCR ▸ Recognize text: render pages, read the words in them and add them as invisible text over the page image, so scanned pages become searchable and selectable (a searchable image; the image is not changed). Pages that already have text are skipped unless skip_text_pages is false; then pictures on those pages are read too, and recognised words that the page's own text already has (an earlier OCR layer included) are left out and counted as already_on_page. Returns each page's recognised text (left out when the document's security doesn't allow copying), word count or why it was skipped. Needs the OCR models (ocr_status). Undoable as one step.")
t("ocr_recognize", "Recognize text (OCR)", "Scan & OCR ▸ Recognize text: render pages, read the words in them and add them as invisible text over the page image, so scanned pages become searchable and selectable (a searchable image; the image is not changed). Pages that already have text are skipped unless skip_text_pages or force_ocr say otherwise; words overlapping existing text are omitted and counted as already_on_page. With output_mode \"editable\" nothing is applied: a NEW document of visible text at the recognised boxes is written to out (which must not be the source; empty pages stay blank). engine and strategy choose the readers (an engine that cannot run is an error named up front); the four preprocessing flags clean the raster first. region reads just that part of one page ([x0, y0, x1, y1] in points from the top-left of the displayed page). Returns each page's recognised text, word count or why it was skipped. Needs the engines (ocr_status). Searchable output is undoable as one step.")
.cmd("ocr.recognize")
.with(schema(
json!({
"doc": doc(),
"pages": pages("to recognise (default: all)"),
"dpi": { "type": "number", "minimum": 72, "maximum": 600, "description": "Resolution pages are read at (default 300; a scanned page is read at most at its own resolution)." },
"language": { "type": "string", "enum": ["en"], "description": "Document language (default en)." },
"skip_text_pages": { "type": "boolean", "description": "Leave pages that already contain text alone (default true). false also reads pictures on such pages, without adding their text again." },
"language": { "type": "string", "enum": language_enum(), "description": "Document language (default en)." },
"engine": { "type": "string", "enum": ["auto", "ocrs", "tesseract"], "description": "Which engine reads the pages (default ocrs). auto: ocrs first and tesseract as the second engine, combined by strategy (only the engines that can run take part)." },
"strategy": { "type": "string", "enum": ["primary", "confidence", "rover"], "description": "How two engines' readings combine: primary = the first engine alone; confidence = the second engine joins when the first reading's mean confidence is under 70 or unknown; rover = both always read, merged. Default: confidence with engine auto, primary otherwise." },
"output_mode": { "type": "string", "enum": ["searchable", "editable"], "description": "searchable (default): invisible text over the pages of the open document. editable: a NEW <document>_ocr-style file of visible text written to out; the source is not touched." },
"out": { "type": "string", "description": "Where the editable output is written (required with output_mode editable; a new file: never the source path, never an existing file)." },
"region": { "type": "array", "items": { "type": "number" }, "minItems": 4, "maxItems": 4, "description": "[x0, y0, x1, y1] in points from the top-left of the displayed page: read only that part. Needs exactly one page." },
"skip_text_pages": { "type": "boolean", "description": "Leave pages that already contain text alone (default true)." },
"skip_text_files": { "type": "boolean", "description": "Batch: leave whole files that already contain text alone (they come back skipped, not read)." },
"force_ocr": { "type": "boolean", "description": "Read pages and files even when the skip options would leave them alone (default false)." },
"auto_rotate": { "type": "boolean", "description": "Turn pages scanned sideways or upside down upright before reading (default false)." },
"deskew": { "type": "boolean", "description": "Straighten skewed text lines before reading (default false)." },
"denoise": { "type": "boolean", "description": "A 3×3 median filter against scan speckle (default false)." },
"binarize": { "type": "boolean", "description": "Sauvola binarization: ink to black, paper to white (default false)." },
}),
&["doc"],
)),
t("ocr_recognize_files", "Recognize text in multiple files", "Scan & OCR ▸ Recognize text ▸ In multiple files: read every page of each PDF in paths and write the searchable result into folder under the same name (pages that already have text are left alone unless skip_text_pages is false, as in ocr_recognize). Two inputs that would write the same output file are refused before anything is read or written. Returns, per file, the output path, word count and skipped pages, or the error.")
t("ocr_words", "Words of a recognition (for review)", "The words a recognition finds, for inspection without the GUI: per page, each word's text, its box ([x, y, w, h] in points from the top-left of the displayed page), its confidence and band (high >= 90, medium 70–89, low < 70), which engine read it, and whether it is a suspect (low band). suspects counts the words a reviewer should look at first. Changes nothing on the document. retain_review returns a session-local handle for ocr_review_apply (one per document, at most eight; a new retained recognition replaces the old one). Needs the engines (ocr_status).")
.ro()
.with(schema(
json!({
"doc": doc(),
"retain_review": { "type": "boolean" },
"region": { "type": "array", "items": { "type": "number" }, "minItems": 4, "maxItems": 4 },
"pages": pages("to read (default: all)"),
"dpi": { "type": "number", "minimum": 72, "maximum": 600 },
"language": { "type": "string", "enum": language_enum() },
"engine": { "type": "string", "enum": ["auto", "ocrs", "tesseract"] },
"strategy": { "type": "string", "enum": ["primary", "confidence", "rover"] },
"auto_rotate": { "type": "boolean" },
"deskew": { "type": "boolean" },
"denoise": { "type": "boolean" },
"binarize": { "type": "boolean" },
"skip_text_pages": { "type": "boolean", "description": "Leave pages that already contain text alone (default true)." },
"force_ocr": { "type": "boolean", "description": "Read pages even when the skip options would leave them alone (default false)." },
}),
&["doc"],
)),
t("ocr_review_apply", "Apply or discard reviewed OCR words", "Accept corrected recognition from ocr_words retain_review. Corrections use 1-based page/word positions and text only; empty text removes a word, omitted words keep their original text and boxes. The stored document guard refuses stale, foreign or consumed reviews. Searchable output is one undoable step; editable requires out and writes a new document (out must not exist yet; the review is kept when it does). discard releases the review without applying it. Input is capped at 65536 words and 4 MiB, never truncated.")
.cmd("ocr.correct")
.with(schema(json!({ "doc": doc(), "review": { "type": "integer", "minimum": 1 }, "discard": { "type": "boolean" }, "output_mode": { "type": "string", "enum": ["searchable", "editable"] }, "out": { "type": "string" },
"corrections": { "type": "array", "maxItems": 65536, "items": { "type": "object", "additionalProperties": false, "properties": { "page": { "type": "integer", "minimum": 1 }, "word": { "type": "integer", "minimum": 1 }, "text": { "type": "string" } }, "required": ["page", "word", "text"] } } }), &["doc", "review"])),
t("ocr_dictionary", "Read or update an OCR review dictionary", "Read or add/remove words in an OCR Verify dictionary file (one case-insensitive word per UTF-8 line). Choose its per-language ocr-dictionary-<language>.txt path. Paths honor root; updates are atomic. Refuses invalid words, over 65536 entries or over 4 MiB. No document changes.")
.destructive()
.cmd("ocr.correct")
.with(schema(json!({ "path": { "type": "string" }, "add": { "type": "array", "maxItems": 65536, "items": { "type": "string" } }, "remove": { "type": "array", "maxItems": 65536, "items": { "type": "string" } } }), &["path"])),
t("ocr_recognize_files", "Recognize text in multiple files", "Scan & OCR ▸ Recognize text ▸ In multiple files: read every page of each PDF in paths and write the searchable result into folder under the same name. skip_text_files leaves whole files that already contain text alone (they are their own skipped bucket, nothing is written); skip_text_pages leaves single text pages alone; force_ocr reads regardless of both. Every output name is checked before the first page is read: two paths sharing a file name (from different folders) are refused, and a file of that name already in folder aborts the batch, with the names in the error. Each written file carries a low_confidence note when words at or under 60 confidence sit on it (1-based sorted pages). Returns, per file, the output path, word count, skipped pages or the error, plus recognized/words/skipped_files totals. Needs the engines (ocr_status).")
.cmd("ocr.recognize_batch")
.with(schema(
json!({
"paths": { "type": "array", "items": { "type": "string" }, "minItems": 1 },
"folder": { "type": "string" },
"dpi": { "type": "number", "minimum": 72, "maximum": 600 },
"language": { "type": "string", "enum": ["en"] },
"skip_text_pages": { "type": "boolean", "description": "Leave pages that already contain text alone (default true); see ocr_recognize." },
"language": { "type": "string", "enum": language_enum() },
"engine": { "type": "string", "enum": ["auto", "ocrs", "tesseract"] },
"strategy": { "type": "string", "enum": ["primary", "confidence", "rover"] },
"skip_text_pages": { "type": "boolean", "description": "Leave pages that already contain text alone (default true)." },
"skip_text_files": { "type": "boolean", "description": "Leave whole files that already contain text alone (default false)." },
"force_ocr": { "type": "boolean", "description": "Read even where the skip options would leave things alone (default false)." },
"auto_rotate": { "type": "boolean" },
"deskew": { "type": "boolean" },
"denoise": { "type": "boolean" },
"binarize": { "type": "boolean" },
}),
&["paths", "folder"],
)),
t("ocr_status", "OCR status", "Whether text recognition is available (its models are installed: release packages include them; a source build fetches them with `cargo xtask models`; PDFCRAFT_MODELS overrides), where it looks for them, and the languages it reads.")
t("ocr_status", "OCR status", "Whether text recognition is available, where it looks for models (release packages include them; a source build fetches them with `cargo xtask models`; PDFCRAFT_MODELS overrides), its languages, and the engines: ocrs (built in, models) and tesseract (an installed program, with its path).")
.ro()
.cmd("ocr.recognize")
.with(schema(json!({}), &[])),
crates/automation/tests/it/action_failures.rs
+66
View File
@@ -0,0 +1,66 @@
//! A step that fails fails its file: the file is reported with an error and nothing is written
//! for it. A step that finds nothing to do is still not a failure.
use std::sync::Arc;
use pdfcraft_automation::{Automation, Content};
use pdfcraft_engine::Session;
use pdfcraft_engine::actions::{Action, Step, run_on};
use serde_json::json;
fn pdf() -> Vec<u8> {
Session::new().create_from_text("Original", "Original text").unwrap().as_ref().clone()
}
fn one_step(step: Step) -> Action {
Action { name: "Custom".into(), description: String::new(), steps: vec![step], builtin: false }
}
/// The standard fonts can't draw the character, so the watermark step fails and no file is produced.
#[test]
fn failing_watermark_step_fails_the_file_and_publishes_nothing() {
let r = run_on(&one_step(Step::AddWatermark("中".into())), "one.pdf", Arc::new(pdf()), |_, _| {});
assert!(r.is_err(), "a failed step must fail its file: {r:?}");
}
/// A comment step on a document with no comments has nothing to do, which is not a failure.
#[test]
fn nothing_to_do_step_still_succeeds() {
let r = run_on(&one_step(Step::FlattenComments), "one.pdf", Arc::new(pdf()), |_, _| {});
assert!(r.is_ok(), "a step with nothing to do is not a failure: {r:?}");
}
/// Through the tool: the failing file gets an error entry and no output, and the folder stays empty.
#[test]
fn action_run_marks_the_failing_file_as_error_without_output() {
let dir = tempfile::tempdir().unwrap();
let input = dir.path().join("in");
std::fs::create_dir_all(&input).unwrap();
let in_file = input.join("one.pdf");
std::fs::write(&in_file, pdf()).unwrap();
let out = dir.path().join("out");
let mut auto = Automation::new().with_root(dir.path()).unwrap();
let reply = auto.call(
"action_run",
&json!({
"paths": [in_file.to_string_lossy()],
"folder": out.to_string_lossy(),
"steps": [{ "step": "add_watermark", "arg": "中" }],
}),
);
let report = match reply.unwrap().remove(0) {
Content::Json(value) => value,
other => panic!("expected a report: {other:?}"),
};
let entry = &report["files"][0];
assert!(entry["error"].is_string(), "{report}");
assert!(entry.get("output").is_none(), "{report}");
assert_eq!(std::fs::read_dir(&out).unwrap().count(), 0, "nothing is published for a failed file");
}
/// A removal step with nothing to remove keeps its existing note and succeeds.
#[test]
fn removal_step_with_nothing_to_remove_still_succeeds() {
let r = run_on(&one_step(Step::RemoveHiddenInformation), "one.pdf", Arc::new(pdf()), |_, _| {});
assert!(r.is_ok(), "nothing to remove is not a failure: {r:?}");
}
crates/automation/tests/it/automation.rs
+330 -14
View File
@@ -843,10 +843,11 @@ fn arabic_ligature_actual_text_is_one_editable_word_through_save_reopen() {
}
/// Each of these writes to its required `path` and replaces an existing file there, so the MCP
/// annotations must not tell clients the call is read-only or harmless (#130).
/// annotations must not tell clients the call is read-only or harmless (#130). The OCR
/// dictionary's updates rewrite the file in place (lower-cased and sorted), so it counts too.
#[test]
fn file_writing_tools_are_not_read_only() {
for name in ["doc_export_data", "accessibility_report", "image_save"] {
for name in ["doc_export_data", "accessibility_report", "image_save", "ocr_dictionary"] {
let t = tools().into_iter().find(|t| t.name == name).unwrap();
assert!(!t.read_only, "{name} writes a file but advertises read-only");
assert!(t.destructive, "{name} overwrites its path but advertises non-destructive");
@@ -1715,6 +1716,25 @@ fn mcp_default_lists_every_tool_and_has_no_meta_tools() {
assert_eq!(rpc(&mut s, 3, "tools/call", json!({ "name": "tool_search" }))["error"]["code"], -32602);
}
#[cfg(feature = "mcp")]
#[test]
/// `tools/list` advertises exactly the Search & Redact patterns the registry supports, in `pattern`
/// and by name in the description, so a schema-obeying client can select every one of them.
fn the_advertised_redact_mark_schema_offers_every_pattern() {
use pdfcraft_engine::REDACT_PATTERNS;
let registry: Vec<&str> = REDACT_PATTERNS.iter().map(|p| p.id()).collect();
let mut s = McpServer::new(Automation::new());
let list = rpc(&mut s, 1, "tools/list", json!({}));
let mark = list["result"]["tools"].as_array().unwrap().iter().find(|t| t["name"] == "redact_mark").expect("redact_mark is listed");
let advertised: Vec<&str> =
mark["inputSchema"]["properties"]["pattern"]["enum"].as_array().unwrap().iter().map(|v| v.as_str().unwrap()).collect();
assert_eq!(advertised, registry, "the advertised pattern enum must be the registry's ids");
let description = mark["description"].as_str().unwrap();
for id in &registry {
assert!(description.contains(id), "the description must name {id}: {description}");
}
}
#[cfg(feature = "mcp")]
#[test]
fn mcp_compact_lists_the_core_tools_and_two_meta_tools() {
@@ -2561,6 +2581,16 @@ fn copy_restricted_text_stays_in_the_document() {
}
let restyled = ok(&mut a, "text_edit", json!({ "doc": re, "page": 1, "paragraph": 1, "size": 30 }));
assert!(!restyled.to_string().contains("Page"), "{restyled}");
// A path that would carry the page's words OUT of the document — the editable output, or
// the review word list — is refused on a copy-restricted document (no engines needed).
match a.call("ocr_recognize", &json!({ "doc": re, "output_mode": "editable", "out": "copy.pdf" })) {
Err(ToolError::Failed(m)) => assert!(m.contains("don't allow copying"), "{m}"),
other => panic!("the editable output is refused: {other:?}"),
}
match a.call("ocr_words", &json!({ "doc": re })) {
Err(ToolError::Failed(m)) => assert!(m.contains("don't allow copying"), "{m}"),
other => panic!("ocr_words is refused: {other:?}"),
}
if ok(&mut a, "ocr_status", json!({}))["available"] != true {
eprintln!("skipped the OCR part: OCR models not installed");
return;
@@ -4809,6 +4839,35 @@ fn redacting_italian_international_slash_numbers_through_tools() {
std::fs::remove_dir_all(dir).unwrap();
}
#[test]
/// A client obeying the advertised schema can select iban, and the mark made through the tool
/// covers the spaced number and nothing else.
fn redacting_an_iban_through_tools() {
let table = tools();
let advertised: Vec<&str> = table.iter().find(|t| t.name == "redact_mark").unwrap().input_schema["properties"]["pattern"]["enum"]
.as_array()
.unwrap()
.iter()
.map(|v| v.as_str().unwrap())
.collect();
assert!(advertised.contains(&"iban"), "the advertised redact_mark schema must offer iban: {advertised:?}");
let dir = workdir("redact-iban");
std::fs::write(dir.join("invoice.txt"), "Pay DE89 3704 0044 0532 0130 00 today\nAccount 12345678 is ours\nTotal due").unwrap();
let mut a = auto(&dir);
let doc = ok(&mut a, "doc_create", json!({ "from": "text", "path": "invoice.txt" }))["doc"].as_u64().unwrap();
let marked = ok(&mut a, "redact_mark", json!({ "doc": doc, "pattern": "iban" }));
assert_eq!(marked["marked"], 1, "the spaced German sample IBAN needs one mark");
assert!(page_text(&mut a, doc)[0].contains("DE89 3704 0044 0532 0130 00"), "marks alone remove nothing");
assert_eq!(ok(&mut a, "redact_apply", json!({ "doc": doc }))["applied"], 1);
ok(&mut a, "doc_save", json!({ "doc": doc, "path": "redacted.pdf" }));
let reopened = ok(&mut a, "doc_open", json!({ "path": "redacted.pdf" }))["doc"].as_u64().unwrap();
let text = &page_text(&mut a, reopened)[0];
assert!(!text.contains("DE89") && !text.contains("0532"), "{text}");
assert!(text.lines().any(|line| line.trim() == "Pay"), "only the label before the number remains: {text}");
assert!(text.contains("Account 12345678") && text.contains("Total due"), "text without an IBAN remains: {text}");
std::fs::remove_dir_all(dir).unwrap();
}
/// One page with the text "Secret 1234" and a Redact annotation that has neither a usable
/// QuadPoints array nor a Rect, so there is no area to redact.
fn redact_pdf_with_unreadable_mark() -> Vec<u8> {
@@ -7454,6 +7513,8 @@ fn ocr_tools_make_a_scan_searchable() {
let mut a = auto(&dir);
let status = ok(&mut a, "ocr_status", json!({}));
assert_eq!(status["languages"][0]["code"], "en");
assert_eq!(status["engines"][0]["id"], "ocrs", "{status}");
assert!(status["engines"][1]["id"] == "tesseract" && status["engines"][1]["available"].is_boolean(), "{status}");
if status["available"] != true {
eprintln!("skipped: OCR models not installed");
return;
@@ -7468,6 +7529,64 @@ fn ocr_tools_make_a_scan_searchable() {
let again = ok(&mut a, "ocr_recognize", json!({ "doc": scan, "pages": [1] }));
assert!(again["pages"][0]["skipped"].is_string(), "{again}");
assert!(matches!(a.call("ocr_recognize", &json!({ "doc": scan, "language": "xx" })), Err(ToolError::InvalidArgs(_))));
// An engine that cannot run is named up front, before anything is read.
if status["engines"][1]["available"] != true {
let why = a.call("ocr_recognize", &json!({ "doc": scan, "engine": "tesseract", "skip_text_pages": false })).unwrap_err();
assert!(matches!(why, ToolError::Failed(_)), "{why}");
assert!(why.to_string().contains("tesseract"), "{why}");
}
// ocr_words: the recognition's words with confidence, band and suspects, changing nothing.
// ocrs reports no confidences, so `auto` + a confidence merge is asked for when the second
// engine is there to supply them; without it confidence and band are honestly null.
ok(&mut a, "doc_close", json!({ "doc": scan, "discard_changes": true }));
let scan = ok(&mut a, "doc_create", json!({ "from": "images", "paths": [png.to_string_lossy()] }))["doc"].as_u64().unwrap();
let mut args = json!({ "doc": scan });
if status["engines"][1]["available"] == true {
args["engine"] = json!("auto");
args["strategy"] = json!("confidence");
}
let words = ok(&mut a, "ocr_words", args);
assert!(words["words"].as_u64().unwrap() >= 4, "{words}");
let first = &words["pages"][0]["words"][0];
assert!(first["text"].is_string() && first["box"].as_array().is_some_and(|b| b.len() == 4), "{words}");
if status["engines"][1]["available"] == true {
let with_confidence = words["pages"][0]["words"]
.as_array()
.expect("words")
.iter()
.filter(|w| w["confidence"].is_number() && w["band"].as_str().is_some())
.count();
assert!(with_confidence >= 4, "the second engine's confidences arrive: {words}");
} else {
assert!(first["confidence"].is_null() && first["band"].is_null() && first["suspect"] == false, "{words}");
}
assert!(a.call("edit_undo", &json!({ "doc": scan })).is_err(), "ocr_words changes nothing, so there is no undo");
}
/// An editable-text recognition writes a NEW file of visible text, never the source.
#[test]
fn ocr_recognize_editable_output_writes_visible_text() {
let dir = workdir("ocr-editable");
let mut a = auto(&dir);
if ok(&mut a, "ocr_status", json!({}))["available"] != true {
eprintln!("skipped: OCR models not installed");
return;
}
let text = ok(&mut a, "doc_create", json!({ "from": "text", "text": "Editable output words" }))["doc"].as_u64().unwrap();
ok(&mut a, "doc_export_images", json!({ "doc": text, "folder": "scan", "dpi": 150, "pages": [1] }));
let png = std::fs::read_dir(dir.join("scan")).unwrap().next().unwrap().unwrap().path();
let scan = ok(&mut a, "doc_create", json!({ "from": "images", "paths": [png.to_string_lossy()] }))["doc"].as_u64().unwrap();
ok(&mut a, "doc_save", json!({ "doc": scan, "path": "src/scan.pdf" }));
// The output never lands on the source.
let why = a.call("ocr_recognize", &json!({ "doc": scan, "pages": [1], "output_mode": "editable", "out": "src/scan.pdf" })).unwrap_err();
assert!(why.to_string().contains("source"), "{why}");
let r = ok(&mut a, "ocr_recognize", json!({ "doc": scan, "pages": [1], "output_mode": "editable", "out": "out/ocr.pdf" }));
assert!(r["words"].as_u64().unwrap() >= 3, "{r}");
let ocr = ok(&mut a, "doc_open", json!({ "path": "out/ocr.pdf" }))["doc"].as_u64().unwrap();
// The text is visible (the engine may misread a letter at this dpi — look for a word that
// survives).
let found = ok(&mut a, "text_find", json!({ "doc": ocr, "query": "output" }));
assert!(found["count"].as_u64() >= Some(1), "the new file's text is visible: {found}");
}
/// #1028: `skip_text_pages: false` reads a picture on a page that has its own text, and the
@@ -7510,6 +7629,46 @@ fn ocr_recognize_files_refuses_paths_outside_the_root() {
assert!(!dir.join("out").exists(), "nothing was started");
}
/// Two inputs of one file name from different folders would write the same output in the
/// batch folder: the batch is refused up front, with or without the OCR engines, naming the
/// file, and nothing is written.
#[test]
fn ocr_recognize_files_refuses_inputs_that_share_a_file_name() {
let dir = workdir("ocr-files-same-name");
let mut a = auto(&dir);
let doc = ok(&mut a, "doc_open", json!({ "path": "a.pdf" }))["doc"].as_u64().unwrap();
ok(&mut a, "doc_save", json!({ "doc": doc, "path": "one/scan.pdf" }));
ok(&mut a, "doc_save", json!({ "doc": doc, "path": "two/scan.pdf" }));
let r = a.call("ocr_recognize_files", &json!({ "paths": ["one/scan.pdf", "two/scan.pdf"], "folder": "out" }));
assert!(matches!(&r, Err(ToolError::InvalidArgs(m)) if m.contains("scan.pdf") && m.contains("overwrite")), "{r:?}");
assert!(std::fs::read_dir(dir.join("out")).map_or(true, |mut d| d.next().is_none()), "nothing was written");
}
/// `out` belongs to the editable output. A searchable run (the default) writes into the open
/// document, so an `out` is refused rather than silently ignored, with or without the engines.
#[test]
fn ocr_recognize_refuses_an_out_that_nothing_would_use() {
let dir = workdir("ocr-out-ignored");
let mut a = auto(&dir);
let doc = ok(&mut a, "doc_open", json!({ "path": "a.pdf" }))["doc"].as_u64().unwrap();
let r = a.call("ocr_recognize", &json!({ "doc": doc, "out": "../outside.pdf" }));
assert!(matches!(&r, Err(ToolError::InvalidArgs(m)) if m.contains("editable")), "{r:?}");
assert!(!dir.join("../outside.pdf").exists());
}
/// The editable output is a NEW file: an existing `out` is refused before anything runs (with
/// or without the engines), and the file is left as it was.
#[test]
fn ocr_recognize_editable_output_never_replaces_an_existing_file() {
let dir = workdir("ocr-editable-exists");
let mut a = auto(&dir);
let doc = ok(&mut a, "doc_open", json!({ "path": "a.pdf" }))["doc"].as_u64().unwrap();
std::fs::write(dir.join("taken.pdf"), b"keep me").unwrap();
let r = a.call("ocr_recognize", &json!({ "doc": doc, "output_mode": "editable", "out": "taken.pdf" }));
assert!(matches!(&r, Err(e) if e.to_string().contains("already exists")), "{r:?}");
assert_eq!(std::fs::read(dir.join("taken.pdf")).unwrap(), b"keep me");
}
#[test]
fn ocr_recognize_files_writes_searchable_copies() {
let dir = workdir("ocr-files");
@@ -7523,9 +7682,22 @@ fn ocr_recognize_files_writes_searchable_copies() {
let png = std::fs::read_dir(dir.join("scan")).unwrap().next().unwrap().unwrap().path();
let scan = ok(&mut a, "doc_create", json!({ "from": "images", "paths": [png.to_string_lossy()] }))["doc"].as_u64().unwrap();
ok(&mut a, "doc_save", json!({ "doc": scan, "path": "in/scan.pdf" }));
// A file that already has text: the skip-files option passes it over as its own bucket.
let typed = ok(&mut a, "doc_create", json!({ "from": "text", "text": "Already typed" }))["doc"].as_u64().unwrap();
ok(&mut a, "doc_save", json!({ "doc": typed, "path": "in/typed.pdf" }));
// Collision pre-flight: the existing out/scan.pdf (written below) stops a rerun.
let r = ok(&mut a, "ocr_recognize_files", json!({ "paths": ["in/scan.pdf", "missing.pdf"], "folder": "out" }));
assert!(r["files"][0]["words"].as_u64().unwrap() >= 3, "{r}");
assert!(r["files"][1]["error"].is_string(), "{r}");
let why = a.call("ocr_recognize_files", &json!({ "paths": ["in/scan.pdf"], "folder": "out" })).unwrap_err();
assert!(matches!(why, ToolError::Failed(_)), "{why}");
assert!(why.to_string().contains("scan.pdf"), "{why}");
// The skip-files bucket and the low-confidence note.
let r = ok(&mut a, "ocr_recognize_files", json!({ "paths": ["in/typed.pdf", "in/scan.pdf"], "folder": "out2", "skip_text_files": true }));
assert_eq!(r["skipped_files"].as_u64(), Some(1), "{r}");
assert!(r["files"][0]["skipped"].is_string(), "{r}");
assert_eq!(r["recognized"].as_u64(), Some(1), "{r}");
assert!(r["files"][1]["output"].is_string(), "{r}");
let out = ok(&mut a, "doc_open", json!({ "path": "out/scan.pdf" }))["doc"].as_u64().unwrap();
let found = ok(&mut a, "text_find", json!({ "doc": out, "query": "recognition" }));
assert!(found.to_string().contains("\"page\""), "{found}");
@@ -7535,18 +7707,6 @@ fn ocr_recognize_files_writes_searchable_copies() {
/// the second output replaces the first while both results report success. The batch is refused
/// up front, naming the file, before the folder is created, anything is read or the OCR engines
/// load, so the outcome is the same with or without the models installed.
#[test]
fn ocr_recognize_files_refuses_inputs_that_share_a_file_name() {
let dir = workdir("ocr-files-same-name");
let mut a = auto(&dir);
let doc = ok(&mut a, "doc_open", json!({ "path": "a.pdf" }))["doc"].as_u64().unwrap();
ok(&mut a, "doc_save", json!({ "doc": doc, "path": "one/scan.pdf" }));
ok(&mut a, "doc_save", json!({ "doc": doc, "path": "two/scan.pdf" }));
let r = a.call("ocr_recognize_files", &json!({ "paths": ["one/scan.pdf", "two/scan.pdf"], "folder": "out" }));
assert!(matches!(&r, Err(ToolError::InvalidArgs(m)) if m.contains("scan.pdf")), "{r:?}");
assert!(!dir.join("out").exists(), "nothing was started: the output folder was not created");
}
/// File names are compared without regard to case on Windows and macOS (`scan.pdf` and `Scan.PDF`
/// are one file there), and exactly elsewhere, where they are two files and the batch is not
/// refused.
@@ -10722,6 +10882,162 @@ fn simple_rule_scan_drawn(content: &[u8]) -> Vec<u8> {
write_full(&doc, &SaveOptions::default()).unwrap()
}
/// Reviewing headlessly: corrections keep the boxes, a stale or foreign review is refused, and
/// applying consumes the handle (the pages already accepted were applied once).
#[test]
fn ocr_review_corrections_keep_boxes_are_guarded_and_apply_once() {
fn render(a: &mut Automation, doc: u64) -> Vec<u8> {
let mut result = a.call("page_render", &json!({ "doc": doc, "page": 1, "dpi": 72 })).unwrap();
match result.remove(0) {
Content::Png { data, .. } => data,
other => panic!("expected a rendered page, got {other:?}"),
}
}
let dir = workdir("ocr-review");
let mut a = auto(&dir);
if ok(&mut a, "ocr_status", json!({}))["available"] != true {
eprintln!("skipped: OCR models not installed");
return;
}
let text = ok(&mut a, "doc_create", json!({ "from": "text", "text": "Review these original words" }))["doc"].as_u64().unwrap();
ok(&mut a, "doc_export_images", json!({ "doc": text, "folder": "scan", "dpi": 150, "pages": [1] }));
let png = std::fs::read_dir(dir.join("scan")).unwrap().next().unwrap().unwrap().path();
let scan = ok(&mut a, "doc_create", json!({ "from": "images", "paths": [png.to_string_lossy()] }))["doc"].as_u64().unwrap();
let before = render(&mut a, scan);
let found = ok(&mut a, "ocr_words", json!({ "doc": scan, "retain_review": true }));
assert!(found["words"].as_u64().unwrap() >= 3, "{found}");
let review = found["review"].as_u64().unwrap();
assert!(page_text(&mut a, scan).iter().all(String::is_empty));
for args in [
json!({ "doc": text, "review": review }),
json!({ "doc": scan, "review": review, "corrections": [{"page":1,"word":999999,"text":"bad"}] }),
json!({ "doc": scan, "review": review, "corrections": [{"page":1,"word":1,"text":"bad","origin":[0,0]}] }),
json!({ "doc": scan, "review": review, "corrections": [{"page":1,"word":1,"text":"a".repeat(4*1024*1024+1)}] }),
json!({ "doc": scan, "review": review, "corrections": [{"page":1,"word":1,"text":"a"},{"page":1,"word":1,"text":"b"}] }),
] {
assert!(a.call("ocr_review_apply", &args).is_err());
assert!(page_text(&mut a, scan).iter().all(String::is_empty));
}
let result = ok(
&mut a,
"ocr_review_apply",
json!({ "doc": scan, "review": review, "corrections": [{"page":1,"word":1,"text":"日本語"},{"page":1,"word":2,"text":""}] }),
);
assert!(result["applied"].as_u64().unwrap() >= 2);
let extracted = page_text(&mut a, scan).join("\n");
assert!(extracted.contains("日本語"), "{extracted}");
assert_eq!(render(&mut a, scan), before, "corrected text remains invisible at its original geometry");
assert!(a.call("ocr_review_apply", &json!({ "doc": scan, "review": review })).is_err(), "consumed reviews cannot be replayed");
ok(&mut a, "edit_undo", json!({ "doc": scan }));
assert!(page_text(&mut a, scan).iter().all(String::is_empty));
assert!(a.call("edit_undo", &json!({ "doc": scan })).is_err(), "review acceptance is one undo step");
let found = ok(&mut a, "ocr_words", json!({ "doc": scan, "retain_review": true }));
let stale = found["review"].as_u64().unwrap();
ok(&mut a, "page_rotate", json!({ "doc": scan, "pages": [1], "degrees": 90 }));
let error = a.call("ocr_review_apply", &json!({ "doc": scan, "review": stale })).unwrap_err().to_string();
assert!(error.contains("changed since recognition"), "{error}");
assert!(page_text(&mut a, scan).iter().all(String::is_empty));
ok(&mut a, "ocr_review_apply", json!({ "doc": scan, "review": stale, "discard": true }));
assert!(a.call("ocr_review_apply", &json!({ "doc": scan, "review": stale })).is_err());
ok(&mut a, "edit_undo", json!({ "doc": scan }));
let first = ok(&mut a, "ocr_words", json!({ "doc": scan, "retain_review": true }))["review"].as_u64().unwrap();
let second = ok(&mut a, "ocr_words", json!({ "doc": scan, "retain_review": true }))["review"].as_u64().unwrap();
assert_ne!(first, second);
assert!(a.call("ocr_review_apply", &json!({ "doc": scan, "review": first })).is_err());
ok(
&mut a,
"ocr_review_apply",
json!({ "doc": scan, "review": second, "output_mode": "editable", "out": "reviewed.pdf", "corrections": [{"page":1,"word":1,"text":"Corrected"}] }),
);
assert!(page_text(&mut a, scan).iter().all(String::is_empty));
let output = ok(&mut a, "doc_open", json!({ "path": "reviewed.pdf" }))["doc"].as_u64().unwrap();
assert!(page_text(&mut a, output).join("\n").contains("Corrected"));
// The editable output is a new file: an existing one is refused and the review is kept.
let third = ok(&mut a, "ocr_words", json!({ "doc": scan, "retain_review": true }))["review"].as_u64().unwrap();
let again = a.call("ocr_review_apply", &json!({ "doc": scan, "review": third, "output_mode": "editable", "out": "reviewed.pdf" }));
assert!(matches!(&again, Err(e) if e.to_string().contains("already exists")), "{again:?}");
ok(&mut a, "ocr_review_apply", json!({ "doc": scan, "review": third, "discard": true }));
}
#[test]
fn ocr_dictionary_updates_are_atomic_bounded_and_confined() {
let dir = workdir("ocr-dictionary");
let mut a = auto(&dir);
let path = "ocr-dictionary-en.txt";
let out = ok(&mut a, "ocr_dictionary", json!({ "path": path, "add": ["Quartz", "quartz", "日本語"] }));
assert_eq!(out["count"], 2);
assert_eq!(pdfcraft_engine::ocr::dictionary::words(&std::fs::read_to_string(dir.join(path)).unwrap()).unwrap().len(), 2);
let before = std::fs::read(dir.join(path)).unwrap();
for args in [
json!({ "path": path, "add": ["a\nb"] }),
json!({ "path": path, "add": ["x".repeat(1025)] }),
json!({ "path": path, "add": ["İ".repeat(512)] }),
json!({ "path": path, "add": vec!["x"; 65537] }),
json!({ "path": "../outside-ocr-dictionary.txt", "add": ["x"] }),
] {
assert!(a.call("ocr_dictionary", &args).is_err());
}
assert_eq!(std::fs::read(dir.join(path)).unwrap(), before);
assert_eq!(ok(&mut a, "ocr_dictionary", json!({ "path": path, "remove": ["QUARTZ"] }))["words"], json!(["日本語"]));
assert_eq!(ok(&mut a, "ocr_dictionary", json!({ "path": path }))["count"], 1);
std::fs::write(dir.join("ocr-dictionary-large.txt"), vec![b'a'; 4 * 1024 * 1024 + 1]).unwrap();
let too_big = a.call("ocr_dictionary", &json!({ "path": "ocr-dictionary-large.txt", "add": ["x"] })).unwrap_err().to_string();
assert!(!too_big.contains("must name a dictionary file"), "refused for its size, not its name: {too_big}");
assert_eq!(std::fs::metadata(dir.join("ocr-dictionary-large.txt")).unwrap().len(), 4 * 1024 * 1024 + 1);
}
/// The tool rewrites what it is pointed at (lower-cased, sorted), so it only takes dictionary file
/// names: another small text file under the root is neither read nor changed.
#[test]
fn ocr_dictionary_refuses_files_that_are_not_dictionaries() {
let dir = workdir("ocr-dictionary-names");
let mut a = auto(&dir);
let env = b"TOKEN=Zebra
APPLE=1
";
for name in [".env", "notes.txt", "ocr-dictionary-en.md", "ocr-dictionary-e n.txt", "sub/readme.txt"] {
if let Some(parent) = std::path::Path::new(name).parent() {
std::fs::create_dir_all(dir.join(parent)).unwrap();
}
std::fs::write(dir.join(name), env).unwrap();
for args in [json!({ "path": name }), json!({ "path": name, "add": [] }), json!({ "path": name, "add": ["x"] })] {
let err = a.call("ocr_dictionary", &args).unwrap_err().to_string();
assert!(err.contains("must name a dictionary file"), "{name} {args}: {err}");
}
assert_eq!(std::fs::read(dir.join(name)).unwrap(), env, "{name} is untouched");
}
}
#[test]
fn ocr_retained_reviews_have_eight_document_cap_and_close_releases_slot() {
let dir = workdir("ocr-review-cap");
let mut a = auto(&dir);
if ok(&mut a, "ocr_status", json!({}))["available"] != true {
eprintln!("skipped: OCR models not installed");
return;
}
let mut docs = Vec::new();
let mut handles = Vec::new();
for _ in 0..9 {
let doc = ok(&mut a, "doc_create", json!({"from":"text", "text":"Already recognized text"}))["doc"].as_u64().unwrap();
docs.push(doc);
}
for doc in docs.iter().take(8) {
handles.push(ok(&mut a, "ocr_words", json!({"doc":doc,"retain_review":true}))["review"].as_u64().unwrap());
}
let error = a.call("ocr_words", &json!({"doc":docs[8],"retain_review":true})).unwrap_err().to_string();
assert!(error.contains("at most 8"), "{error}");
let replacement = ok(&mut a, "ocr_words", json!({"doc":docs[0],"retain_review":true}))["review"].as_u64().unwrap();
assert_ne!(replacement, handles[0]);
assert!(a.call("ocr_review_apply", &json!({"doc":docs[0],"review":handles[0],"discard":true})).is_err());
ok(&mut a, "doc_close", json!({"doc":docs[0],"discard_changes":true}));
assert!(a.call("ocr_review_apply", &json!({"doc":docs[0],"review":replacement})).is_err());
let final_handle = ok(&mut a, "ocr_words", json!({"doc":docs[8],"retain_review":true}))["review"].as_u64().unwrap();
ok(&mut a, "ocr_review_apply", json!({"doc":docs[8],"review":final_handle,"discard":true}));
}
#[test]
fn editor_crop_group_move_and_arrange_survive_added_item_deletion_through_tools() {
let dir = workdir("editor-crop-group-arrange");
crates/automation/tests/it/batch_failures.rs
+89
View File
@@ -0,0 +1,89 @@
//! A batch run that fails a file publishes only the completed outputs, and a
//! publish the file system refuses publishes nothing.
use pdfcraft_automation::{Automation, Content};
use pdfcraft_engine::Session;
use serde_json::{Value, json};
fn result(auto: &mut Automation, args: Value) -> Value {
match auto.call("doc_batch", &args).unwrap().remove(0) {
Content::Json(value) => value,
_ => panic!("expected a report"),
}
}
fn pdf() -> Vec<u8> {
Session::new().create_from_text("Original", "Original text").unwrap().as_ref().clone()
}
/// Sets a folder's permission bits.
#[cfg(unix)]
fn set_mode(path: &std::path::Path, mode: u32) {
use std::os::unix::fs::PermissionsExt;
std::fs::set_permissions(path, std::fs::Permissions::from_mode(mode)).unwrap();
}
/// Puts back the write permission the temporary folder needs before it is removed.
#[cfg(unix)]
struct RestoreMode(std::path::PathBuf);
#[cfg(unix)]
impl Drop for RestoreMode {
fn drop(&mut self) {
use std::os::unix::fs::PermissionsExt;
let _ = std::fs::set_permissions(&self.0, std::fs::Permissions::from_mode(0o755));
}
}
#[test]
fn failed_inputs_publish_only_completed_outputs() {
let dir = tempfile::tempdir().unwrap();
std::fs::create_dir(dir.path().join("input")).unwrap();
std::fs::write(dir.path().join("input/good.pdf"), pdf()).unwrap();
std::fs::write(dir.path().join("input/bad.pdf"), "not a pdf").unwrap();
let mut auto = Automation::new().with_root(dir.path()).unwrap();
let report = result(&mut auto, json!({"paths":["input"],"steps":["optimize"],"out_dir":"out","recursive":false}));
assert_eq!(report["completed"], 1, "{report}");
assert_eq!(report["failed"], 1, "{report}");
let files = report["files"].as_array().unwrap();
assert_eq!(files.len(), 2, "{report}");
let bad = files.iter().find(|file| file["path"].as_str().is_some_and(|path| path.ends_with("bad.pdf"))).unwrap();
assert_eq!(bad["status"], "error", "{report}");
assert!(bad["output"].is_null(), "a failed file reports no output: {report}");
let names: Vec<String> =
std::fs::read_dir(dir.path().join("out")).unwrap().map(|entry| entry.unwrap().file_name().to_string_lossy().into_owned()).collect();
// Publish stages its bytes under a dot-prefixed name in the output folder, so
// no entry carrying that prefix may survive the run.
for name in &names {
assert!(!name.starts_with('.'), "temporary name left behind: {name} ({names:?})");
}
assert_eq!(names, vec!["good.pdf".to_string()], "only the completed output is published");
}
#[cfg(unix)]
#[test]
fn read_only_output_folder_publishes_nothing() {
let dir = tempfile::tempdir().unwrap();
std::fs::create_dir(dir.path().join("input")).unwrap();
std::fs::write(dir.path().join("input/good.pdf"), pdf()).unwrap();
let out = dir.path().join("out");
std::fs::create_dir(&out).unwrap();
set_mode(&out, 0o555);
let _restore = RestoreMode(out.clone());
// A root process ignores the mode bits, and then the assertions below would
// pass without testing the refusal. Fail loudly instead of skipping.
assert!(std::fs::write(out.join("probe"), b"").is_err(), "run as a non-root user: the read-only output folder accepted a new file");
let mut auto = Automation::new().with_root(dir.path()).unwrap();
let report = result(&mut auto, json!({"paths":["input"],"steps":["optimize"],"out_dir":"out","recursive":false}));
assert_eq!(report["completed"], 0, "{report}");
assert_eq!(report["failed"], 1, "{report}");
let files = report["files"].as_array().unwrap();
assert_eq!(files.len(), 1, "{report}");
// Publish cannot create its staging file in a folder it may not write, so
// the entry carries the system's error and no output.
assert_eq!(files[0]["status"], "error", "{report}");
assert!(files[0]["output"].is_null(), "a refused publish reports no output: {report}");
let error = files[0]["error"].as_str().unwrap();
assert!(!error.is_empty(), "{report}");
assert_eq!(std::fs::read_dir(&out).unwrap().count(), 0, "the refused publish writes nothing into the read-only folder: {report}");
}
crates/automation/tests/it/batch_secrets.rs
+84
View File
@@ -0,0 +1,84 @@
//! Tool results and errors from the folder tools carry no known test password.
//!
//! The passwords are set on the document and passed to the tool, and must never come back out: a
//! result that echoes one has leaked it into logs, transcripts and MCP callers.
use pdfcraft_automation::{Automation, Content};
use pdfcraft_engine::{Changes, Edit, Protection, Session};
use serde_json::{Value, json};
/// The known test passwords; neither may appear in a result or an error.
const OPEN_PASSWORD: &str = "PLANT-OPEN-5f2c9a";
const OWNER_PASSWORD: &str = "PLANT-OWNER-71de0b";
/// A one-page PDF locked with the known test passwords.
///
/// The open password is the only way in. The batch bates tool rewrites page content, and
/// `Changes::None` forbids that, so this fixture permits every change except extracting pages;
/// the restrictions stay real, since the open password never lifts them.
fn protected_pdf() -> Vec<u8> {
let mut session = Session::new();
let bytes = session.create_from_text("Batch secrets", "Planted password test").unwrap();
let id = session.open("locked.pdf", None, bytes, None).unwrap();
session
.apply(
id,
Edit::Protect(Protection {
open_password: Some(OPEN_PASSWORD.into()),
permissions_password: Some(OWNER_PASSWORD.into()),
changes: Changes::AnyExceptExtract,
..Default::default()
}),
)
.unwrap();
session.save_bytes(id).unwrap().as_ref().clone()
}
fn json_result(a: &mut Automation, tool: &str, args: Value) -> Value {
match a.call(tool, &args) {
Ok(mut content) => match content.remove(0) {
Content::Json(value) => value,
other => panic!("{tool}: expected JSON, got {other:?}"),
},
Err(error) => panic!("{tool} {args}: {error}"),
}
}
#[test]
fn results_and_errors_hold_no_planted_password_for_bates_or_batch() {
let dir = tempfile::tempdir().unwrap();
std::fs::write(dir.path().join("locked.pdf"), protected_pdf()).unwrap();
let mut a = Automation::new().with_root(dir.path()).unwrap();
// The right open password: numbering succeeds and reports the numbers only.
let numbered =
json_result(&mut a, "doc_bates_multi", json!({"files":[{"input":"locked.pdf","output":"locked-numbered.pdf","password":OPEN_PASSWORD}]}));
assert_eq!(numbered["completed"], 1, "{numbered}");
assert_eq!(numbered["files"][0]["status"], "ok", "{numbered}");
// The report carries the per-file entry the Bates tool builds, so a field added to that entry
// lands in the text checked below.
assert_eq!(numbered["files"][0]["first"], 1, "{numbered}");
assert!(numbered["files"][0]["output"].as_str().is_some_and(|path| path.ends_with("locked-numbered.pdf")), "{numbered}");
assert!(dir.path().join("locked-numbered.pdf").exists(), "the numbered copy is written");
// The wrong open password: the tool refuses the file with an error.
let refused = match a
.call("doc_bates_multi", &json!({"files":[{"input":"locked.pdf","output":"locked-refused.pdf","password":"not-the-open-password"}]}))
{
Err(error) => error.to_string(),
Ok(value) => panic!("a wrong open password must be refused: {value:?}"),
};
// The folder job refuses the encrypted file without a password, in its per-file report.
let batch = json_result(&mut a, "doc_batch", json!({"paths":["locked.pdf"],"steps":["optimize"],"out_dir":"out"}));
assert_eq!(batch["failed"], 1, "{batch}");
assert_eq!(batch["files"][0]["status"], "error", "{batch}");
assert!(batch["files"][0]["error"].as_str().is_some_and(|text| !text.is_empty()), "{batch}");
assert!(!dir.path().join("out/locked.pdf").exists(), "a refused file publishes nothing");
for report in [serde_json::to_string(&numbered).unwrap(), refused, serde_json::to_string(&batch).unwrap()] {
for secret in [OPEN_PASSWORD, OWNER_PASSWORD] {
assert!(!report.contains(secret), "a result or error echoed a planted password: {report}");
}
}
}
crates/automation/tests/it/main.rs
+3
View File
@@ -1,10 +1,13 @@
//! The crate's integration tests, linked into one test binary.
mod action_failures;
mod attachment_key;
mod automation;
mod automation_roots;
mod backdrop;
mod batch;
mod batch_failures;
mod batch_secrets;
mod button_caption;
mod form_bulk;
mod hostile_args;
crates/automation/tests/pattern_redaction.rs
+146
View File
@@ -0,0 +1,146 @@
//! Rectangle redaction must not report a pattern-cell secret safely removed (#3647).
use std::sync::Arc;
use pdfcraft_automation::{Automation, Content};
use pdfcraft_cos::{Dict, Document, Object, Stream};
use pdfcraft_engine::DocId;
use serde_json::{Value, json};
fn tool(automation: &mut Automation, name: &str, args: Value) -> Value {
match automation.call(name, &args).unwrap().remove(0) {
Content::Json(value) => value,
other => panic!("expected JSON: {other:?}"),
}
}
/// `retained`: paint with the pattern once more, away from the mark (x 290..340 lies right of it).
/// A path the mark hits is removed whole (F-008) and a pattern nothing surviving paints with is
/// pruned, so only a pattern that some surviving paint still uses reaches the guard.
fn fixture(cell: &[u8], partial: bool, compressed: bool, retained: bool) -> Vec<u8> {
let mut pattern = Dict::from_iter([
(b"Type".to_vec(), Object::name("Pattern")),
(b"PatternType".to_vec(), Object::Int(1)),
(b"PaintType".to_vec(), Object::Int(1)),
(b"TilingType".to_vec(), Object::Int(1)),
(b"BBox".to_vec(), Object::Array(vec![0.into(), 0.into(), 500.into(), 300.into()])),
(b"XStep".to_vec(), Object::Int(500)),
(b"YStep".to_vec(), Object::Int(300)),
]);
pattern.set(
b"Resources".to_vec(),
Object::Dict(Dict::from_iter([(
b"Font".to_vec(),
Object::Dict(Dict::from_iter([(b"F1".to_vec(), Object::Ref(pdfcraft_cos::ObjRef::new(4, 0)))])),
)])),
);
let width = if partial { 400 } else { 170 };
let survivor = if retained { " 290 115 50 30 re f" } else { "" };
let content = format!("BT /F1 20 Tf 40 250 Td (PUBLIC) Tj ET /Pattern cs /P0 scn 35 110 {width} 35 re f{survivor}");
let mut page_stream = Vec::new();
pdfcraft_cos::serialize(&Object::Stream(Stream::from_raw(Dict::new(), content.into_bytes())), &mut page_stream);
let mut pattern_stream = Vec::new();
let stream = if compressed { Stream::flate(pattern, cell) } else { Stream::from_raw(pattern, cell.to_vec()) };
pdfcraft_cos::serialize(&Object::Stream(stream), &mut pattern_stream);
let objects = [
b"<< /Type /Catalog /Pages 2 0 R >>".to_vec(),
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".to_vec(),
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 500 300] /Contents 5 0 R /Resources << /Font << /F1 4 0 R >> /Pattern << /P0 6 0 R >> >> >>"
.to_vec(),
b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>".to_vec(),
page_stream,
pattern_stream,
];
let mut bytes = b"%PDF-1.7\n".to_vec();
let mut offsets = Vec::new();
for (index, object) in objects.iter().enumerate() {
offsets.push(bytes.len());
bytes.extend(format!("{} 0 obj\n", index + 1).as_bytes());
bytes.extend(object);
bytes.extend(b"\nendobj\n");
}
let xref = bytes.len();
bytes.extend(format!("xref\n0 {}\n0000000000 65535 f \n", objects.len() + 1).as_bytes());
for offset in offsets {
bytes.extend(format!("{offset:010} 00000 n \n").as_bytes());
}
bytes.extend(format!("trailer\n<< /Size {} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n", objects.len() + 1).as_bytes());
bytes
}
fn saved_has_secret(bytes: Vec<u8>) -> bool {
let document = Document::open(Arc::new(bytes)).unwrap();
document.object_numbers().into_iter().any(|num| {
let object = document.try_get(num).unwrap();
matches!(&*object, Object::Stream(stream) if stream.decoded().unwrap().windows(10).any(|part| part == b"CELLSECRET"))
})
}
#[test]
fn pattern_redaction_tools_remove_or_refuse_without_leaking_or_changing_undo() {
let dir = tempfile::tempdir().unwrap();
for (partial, retained) in [(false, false), (true, false), (true, true)] {
for compressed in [false, true] {
let cell = b"BT /F1 20 Tf 40 120 Td (CELLSECRET) Tj ET BT /F1 20 Tf 300 120 Td (SAFE) Tj ET";
let bytes = fixture(cell, partial, compressed, retained);
assert!(saved_has_secret(bytes.clone()));
std::fs::write(dir.path().join("input.pdf"), bytes).unwrap();
let mut automation = Automation::new().with_root(dir.path()).unwrap().with_clock(|| 1_700_000_000);
let id = tool(&mut automation, "doc_open", json!({"path": "input.pdf"}))["doc"].as_u64().unwrap();
tool(&mut automation, "redact_mark", json!({"doc": id, "page": 1, "rect": [30, 150, 215, 195]}));
let before = automation.session().get(DocId(id)).unwrap().bytes.clone();
let pixels = automation.render_pixels(id, 1, 1.0).unwrap();
match automation.call("redact_apply", &json!({"doc": id})) {
Ok(_) => {
assert!(!retained, "a text-bearing cell still in use must not be reported safe (partial: {partial})");
tool(&mut automation, "doc_save", json!({"doc": id, "path": "redacted.pdf"}));
assert!(
!saved_has_secret(std::fs::read(dir.path().join("redacted.pdf")).unwrap()),
"successful apply/save left the pattern-cell text recoverable"
);
}
Err(error) => {
assert!(retained, "a pattern nothing surviving paints with should apply (partial: {partial}): {error}");
assert!(error.to_string().contains("tiling pattern"), "{error}");
assert!(!error.to_string().contains("CELLSECRET"));
let document = automation.session().get(DocId(id)).unwrap();
assert_eq!(document.bytes, before);
assert_eq!(document.redaction_marks(), 1);
assert_eq!(document.can_undo(), Some("Add redaction mark"));
assert_eq!(automation.render_pixels(id, 1, 1.0).unwrap(), pixels);
tool(&mut automation, "edit_undo", json!({"doc": id}));
assert_eq!(automation.session().get(DocId(id)).unwrap().redaction_marks(), 0);
}
}
}
}
}
#[test]
fn pattern_redaction_vector_control_saves_reopens_and_preserves_outside_pixels() {
let dir = tempfile::tempdir().unwrap();
// The guard's control: the cell is plain vectors and a paint outside the mark still uses it,
// so it is inspected and kept, not refused, and what that paint draws is unchanged.
let bytes = fixture(b"0.2 0.4 0.6 rg 40 120 15 15 re f 300 120 30 20 re f", true, true, true);
std::fs::write(dir.path().join("vectors.pdf"), bytes).unwrap();
let mut automation = Automation::new().with_root(dir.path()).unwrap().with_clock(|| 1_700_000_000);
let id = tool(&mut automation, "doc_open", json!({"path": "vectors.pdf"}))["doc"].as_u64().unwrap();
let (width, height, original) = automation.render_pixels(id, 1, 1.0).unwrap();
tool(&mut automation, "redact_mark", json!({"doc": id, "page": 1, "rect": [30, 150, 215, 195]}));
let applied = tool(&mut automation, "redact_apply", json!({"doc": id}));
assert_eq!(applied["applied"], 1);
let save = tool(&mut automation, "doc_save", json!({"doc": id, "path": "vectors-redacted.pdf"}));
assert_eq!(save["incremental"], false);
let reopened = tool(&mut automation, "doc_open", json!({"path": "vectors-redacted.pdf"}))["doc"].as_u64().unwrap();
let (new_width, new_height, actual) = automation.render_pixels(reopened, 1, 1.0).unwrap();
assert_eq!((new_width, new_height), (width, height));
for (index, (before, after)) in original.as_chunks::<4>().0.iter().zip(actual.as_chunks::<4>().0).enumerate() {
let (x, y) = (index % width as usize, index / width as usize);
if x >= 280 {
assert_eq!(before, after, "unmarked pattern pixels changed at {x},{y}");
}
if (35..205).contains(&x) && (155..190).contains(&y) {
assert_eq!(after, &[0, 0, 0, 255], "marked interior must be black");
}
}
}
crates/compare/src/lib.rs
+14
View File
@@ -41,6 +41,8 @@ pub struct Side {
/// The page the change is on (for a pure insertion or deletion, where the other side's
/// words would have been).
pub page: usize,
/// The page its last word is on: words changed across a page break are one change.
pub last_page: usize,
pub rects: Vec<[f64; 4]>,
/// For an empty side (the other side's words were inserted or deleted): the box of the
/// word next to where they would be.
@@ -245,6 +247,7 @@ fn side(words: &[&Word], neighbour: Option<&Word>) -> Side {
Side {
text: words.iter().map(|w| w.text.as_str()).collect::<Vec<_>>().join(" "),
page: words.first().or(neighbour.as_ref()).map_or(0, |w| w.page),
last_page: words.last().or(neighbour.as_ref()).map_or(0, |w| w.page),
rects: rects(words),
near: if words.is_empty() { neighbour.map(|w| w.rect) } else { None },
}
@@ -349,6 +352,17 @@ mod tests {
assert_eq!(c.count(Kind::Inserted), 2);
}
#[test]
fn a_change_across_a_page_break_knows_both_pages() {
let mut old = words("alpha beta", 0);
old.extend(words("gamma delta", 1));
let mut new = words("alpha one", 0);
new.extend(words("two delta", 1));
let c = compare(&old, &new);
assert_eq!(c.changes.len(), 1);
assert_eq!((c.changes[0].new.page, c.changes[0].new.last_page), (0, 1));
}
#[test]
fn deletions_across_pages_keep_their_pages() {
let mut old = words("alpha beta gamma", 0);
crates/cos/Cargo.toml
+7
View File
@@ -15,6 +15,13 @@ thiserror = { workspace = true }
# PDF date strings (`pdf_date`) go through chrono (no hand-rolled calendar code).
chrono = { workspace = true }
# On the web, a rewrite's fresh file identifier takes its entropy from the browser
# (`crypto.getRandomValues`). Both crates are already in the dependency tree of the web shell
# (`js-sys` 0.3 / `wasm-bindgen` 0.2); native builds compile neither.
[target.'cfg(target_arch = "wasm32")'.dependencies]
js-sys = "0.3"
wasm-bindgen = "0.2"
[dev-dependencies]
pdfcraft-oracle = { workspace = true }
pdfcraft-testkit = { workspace = true }
crates/cos/README.md
+11 -1
View File
@@ -62,7 +62,9 @@ another. Edits never touch the original bytes.
not expand compact xref ranges; the winning xref entry remains authoritative for free
slots. Compressed objects require generation zero. Explicit `set` overlays accept only the
supplied generation; `set_existing` and `update_dict` retain both accepted generations of a
damaged live slot, including through snapshots, and require full rewrite. Headers beyond
damaged live slot, including through snapshots, and require full rewrite.
`accepted_generations(num)` lists the generations a slot accepts, so code comparing two
revisions can tell whether a reference still reaches the same slot. Headers beyond
this bounded probe are not additionally recovered. Materialized primary and hybrid rows are
diagnosed before folding, so a newer row does not hide a retained disagreement.
- **Tolerant, never fatal.** The lexer works on a byte slice and a position; it allocates nothing
@@ -89,6 +91,14 @@ into compressed object streams with a cross-reference stream (PDF 1.5+); turning
classic table any reader accepts. Both forms map accepted xref/header aliases to one output
object, retaining shared identity such as page-tree parent links; unrelated generations stay null.
A full save keeps the file's `/ID` as it is, unless the document asked for a new identifier with
`Document::require_full_save_with_new_id` (redaction and sanitizing do). Then both strings are
regenerated, and `SaveOptions::id_entropy` can supply entropy for reproducible output (tests pin
output this way). Without it the identifier mixes in fresh entropy from the host — the browser's
`crypto.getRandomValues` on the web, the operating system's random hash seed on the desktop — so a
fresh copy of the program cannot reproduce it. An encrypted file keeps the first string, because its
key is derived from it.
Dictionaries keep their key order and streams keep their *encoded* bytes, so rewritten objects stay
recognisable in a diff and byte-stable when re-serialized.
crates/cos/src/document.rs
+3
View File
@@ -71,6 +71,9 @@ pub struct Document {
encryption_changed: bool,
/// Set by edits that must not leave earlier revisions in the file (redaction).
full_save: bool,
/// The next full save also gives the file a new identifier (`/ID`), see
/// [`Document::require_full_save_with_new_id`].
new_id: bool,
/// The handler and `/Encrypt` object number that saves use, when encryption changed.
out_security: Option<Arc<pdfcraft_crypt::SecurityHandler>>,
out_encrypt_num: Option<u32>,
crates/cos/src/document/objects.rs
+24
View File
@@ -175,6 +175,30 @@ impl Document {
}
}
/// The generations a reference to `num` may carry and still resolve (see [`Self::get`]),
/// sorted: none for a free or unknown slot, else the slot's generation, and for a damaged
/// file slot (or a repair-aware edit of one) also the generation its object header carries.
/// Code comparing revisions (signature validation) uses it to tell whether the references
/// that reached an object in one revision still reach the same slot in another.
pub fn accepted_generations(&self, num: u32) -> Vec<u16> {
// Mirrors `reference_matches`, which stays allocation-free for every lookup.
let mut generations = match self.overlay.get(&num) {
Some(Slot::Set(generation, _)) => vec![*generation],
Some(Slot::RepairedSet(generation, alias, _)) => vec![*generation, *alias],
Some(Slot::Freed(_)) => Vec::new(),
None => match self.entries.lookup(num, &self.data) {
Some(XrefEntry::InFile { offset, generation }) => {
std::iter::once(generation).chain(self.indexed_header_reference(num, offset).map(|h| h.generation)).collect()
}
Some(XrefEntry::InStream { .. }) => vec![0],
None | Some(XrefEntry::Free { .. }) => Vec::new(),
},
};
generations.sort_unstable();
generations.dedup();
generations
}
pub(super) fn try_get_reference(&self, r: ObjRef) -> Result<Arc<Object>, CosError> {
if !self.reference_matches(r) {
return Ok(Arc::new(Object::Null));
crates/cos/src/document/open.rs
+15
View File
@@ -35,6 +35,7 @@ impl Document {
unfound: Arc::default(),
encryption_changed: false,
full_save: false,
new_id: false,
out_security: None,
out_encrypt_num: None,
};
@@ -121,6 +122,7 @@ impl Document {
unfound: Arc::default(),
encryption_changed: false,
full_save: false,
new_id: false,
out_security: None,
out_encrypt_num: None,
};
@@ -290,12 +292,25 @@ impl Document {
self.full_save = true;
}
/// Like [`Document::require_full_save`], and the rewritten file also gets a new file identifier
/// (`/ID`), so that a redacted or sanitized copy never claims to be its source (ISO 32000-2
/// §14.4). An encrypted file keeps the first string of its identifier, which its key depends on.
pub fn require_full_save_with_new_id(&mut self) {
self.full_save = true;
self.new_id = true;
}
/// The next save must rewrite the whole file (see [`Document::require_full_save`]).
/// Repair-aware edits retaining two accepted generations cannot use one incremental slot.
pub fn full_save_required(&self) -> bool {
self.full_save || self.overlay.values().any(|slot| matches!(slot, Slot::RepairedSet(..)))
}
/// The next full save gives the file a new identifier (see [`Document::require_full_save_with_new_id`]).
pub fn new_id_required(&self) -> bool {
self.new_id
}
/// Security was set or removed since the document was opened (the next save applies it).
pub fn encryption_changed(&self) -> bool {
self.encryption_changed
crates/cos/src/lib.rs
+1 -1
View File
@@ -30,7 +30,7 @@ pub use parser::{Lexer, is_whitespace, parse_indirect};
pub use pdfcraft_crypt::{
Algorithm, Auth, Method as CryptMethod, NewEncryption, Permissions, SecurityHandler, effective_password, pdfdoc_byte, pdfdoc_char,
};
pub use writer::{SaveOptions, pdf_date, serialize, write_full, write_incremental};
pub use writer::{SaveOptions, pdf_date, reachable_objects, serialize, write_full, write_incremental};
#[derive(Debug, thiserror::Error, Clone, PartialEq)]
pub enum CosError {
crates/cos/src/writer.rs
+246 -31
View File
@@ -167,11 +167,17 @@ pub struct SaveOptions {
/// before adding it, with [`Document::free_unlinked_additions`]. Full saves always leave
/// these out.
pub drop_unlinked: bool,
/// Entropy for the file identifier of a rewrite that asks for a new one (see
/// [`Document::require_full_save_with_new_id`]). Given, the identifier depends only on this and
/// on the written bytes, so the same save gives the same file; tests use it to pin output.
/// Without it, fresh entropy from the host (the operating system on native targets, the
/// browser on WebAssembly) is mixed in.
pub id_entropy: Option<[u8; 16]>,
}
impl Default for SaveOptions {
fn default() -> Self {
Self { mod_date: None, id_seed: 0x5052_494E_5443_5241, object_streams: true, drop_unlinked: false }
Self { mod_date: None, id_seed: 0x5052_494E_5443_5241, object_streams: true, drop_unlinked: false, id_entropy: None }
}
}
@@ -267,16 +273,12 @@ pub fn write_incremental(doc: &Document, opts: &SaveOptions) -> Result<Vec<u8>,
Ok(file)
}
/// Write only reachable objects, renumbered from 1, with a table or cross-reference stream.
pub fn write_full(doc: &Document, opts: &SaveOptions) -> Result<Vec<u8>, CosError> {
if doc.stream_limited() {
return Err(CosError::ReadOnlyLimit);
}
// A full traversal must not fill the editor's shared caches (which undo snapshots also
// own). Parsed objects and decoded object streams are temporary working data here.
let mut reader = doc.object_reader();
stamp_mod_date(&mut reader.document, opts);
let trailer_in = reader.document.trailer().clone();
/// The objects [`write_full`] writes: everything reachable from the trailer's /Root, /Info and
/// /Encrypt and from its unknown entries, breadth-first, with the new number each gets
/// (aliases of one slot included) and, in visit order, whether it is a stream.
type SaveOrder = (HashMap<ObjRef, u32>, Vec<(ObjRef, bool)>);
fn full_save_order(reader: &mut ObjectReader, trailer_in: &Dict) -> Result<SaveOrder, CosError> {
// Breadth-first walk from the trailer, assigning new numbers in visit order.
let mut map: HashMap<ObjRef, u32> = HashMap::new();
// Record the type while traversing so classifying packed objects needs no second parse.
@@ -320,6 +322,30 @@ pub fn write_full(doc: &Document, opts: &SaveOptions) -> Result<Vec<u8>, CosErro
order.push((canonical, matches!(*o, Object::Stream(_))));
visit_refs(&o, &mut queue);
}
Ok((map, order))
}
/// The object numbers a full save of `doc` would write: exactly the objects [`write_full`]
/// reaches from the trailer, object streams included. For a check that something removed from a
/// document cannot reach the saved file through a reference nobody looked for.
pub fn reachable_objects(doc: &Document) -> Result<HashSet<u32>, CosError> {
let mut reader = doc.object_reader();
let trailer_in = reader.document.trailer().clone();
let (_, order) = full_save_order(&mut reader, &trailer_in)?;
Ok(order.into_iter().map(|(r, _)| r.num).collect())
}
/// Write only reachable objects, renumbered from 1, with a table or cross-reference stream.
pub fn write_full(doc: &Document, opts: &SaveOptions) -> Result<Vec<u8>, CosError> {
if doc.stream_limited() {
return Err(CosError::ReadOnlyLimit);
}
// A full traversal must not fill the editor's shared caches (which undo snapshots also
// own). Parsed objects and decoded object streams are temporary working data here.
let mut reader = doc.object_reader();
stamp_mod_date(&mut reader.document, opts);
let trailer_in = reader.document.trailer().clone();
let (map, order) = full_save_order(&mut reader, &trailer_in)?;
let version = reader.document.version().chars().take(3).collect::<String>();
let version = if opts.object_streams && version.as_str() < "1.5" { "1.5".to_string() } else { version };
let mut out = Vec::new();
@@ -383,26 +409,7 @@ pub fn write_full(doc: &Document, opts: &SaveOptions) -> Result<Vec<u8>, CosErro
}
}
batch.flush(&reader.document, &mut next, &mut rows, &mut out)?;
let mut trailer = Dict::new();
for key in [&b"Root"[..], b"Info", b"Encrypt"] {
if let Some(r) = trailer_in.reference(key).and_then(|r| map.get(&r)) {
trailer.set(key.to_vec(), Object::Ref(ObjRef::new(*r, 0)));
}
}
if let Some(encrypt @ Object::Dict(_)) = trailer_in.get(b"Encrypt") {
trailer.set(b"Encrypt".to_vec(), renumber(encrypt, &map));
}
if let Some(id) = trailer_in.get(b"ID") {
trailer.set(b"ID".to_vec(), direct_id(&reader.document, id));
}
for (key, value) in trailer_in.iter() {
if !trailer_key_rewritten(key) {
trailer.set(key.clone(), renumber(value, &map));
}
}
if reader.document.output_security().0.is_none() {
ensure_id(&mut trailer, opts, reader.document.bytes());
}
let mut trailer = full_save_trailer(&reader.document, &trailer_in, &map, opts, &out);
if opts.object_streams {
write_xref_stream(&mut out, 0, &mut trailer, rows, next);
} else {
@@ -421,6 +428,34 @@ pub fn write_full(doc: &Document, opts: &SaveOptions) -> Result<Vec<u8>, CosErro
Ok(out)
}
/// Assemble the full rewrite's trailer and apply its identifier policy.
fn full_save_trailer(doc: &Document, trailer_in: &Dict, map: &HashMap<ObjRef, u32>, opts: &SaveOptions, out: &[u8]) -> Dict {
let mut trailer = Dict::new();
for key in [&b"Root"[..], b"Info", b"Encrypt"] {
if let Some(r) = trailer_in.reference(key).and_then(|r| map.get(&r)) {
trailer.set(key.to_vec(), Object::Ref(ObjRef::new(*r, 0)));
}
}
if let Some(encrypt @ Object::Dict(_)) = trailer_in.get(b"Encrypt") {
trailer.set(b"Encrypt".to_vec(), renumber(encrypt, map));
}
if let Some(id) = trailer_in.get(b"ID") {
trailer.set(b"ID".to_vec(), direct_id(doc, id));
}
for (key, value) in trailer_in.iter() {
if !trailer_key_rewritten(key) {
trailer.set(key.clone(), renumber(value, map));
}
}
if doc.output_security().0.is_none() {
ensure_id(&mut trailer, opts, doc.bytes());
}
if doc.new_id_required() {
regenerate_id(&mut trailer, opts, out, doc.output_security().0.is_some());
}
trailer
}
/// Object `r` read again for writing. The traversal read it once and only kept its number, so
/// a second read that fails, finds nothing or finds a different kind of object must fail the
/// save rather than write `null` in its place.
@@ -669,6 +704,136 @@ fn ensure_id(trailer: &mut Dict, opts: &SaveOptions, content: &[u8]) {
trailer.set(b"ID".to_vec(), Object::Array(vec![s.clone(), s]));
}
/// Give a rewritten file a new identifier (§14.4), so that a redacted or sanitized copy does not
/// claim to be its source. An encrypted file keeps the first string, which its key is derived from,
/// and gets a new second one. When there is no such string (no `/ID`, one entry, or a first entry
/// that is not a string) the reader takes the empty string as the key input (`Document::first_id`),
/// so the empty string is written first and the key stays valid.
fn regenerate_id(trailer: &mut Dict, opts: &SaveOptions, written: &[u8], encrypted: bool) {
let fresh = Object::String(PdfString { bytes: new_id_bytes(opts, written), hex: true });
let pair = if encrypted {
let first = match trailer.get(b"ID") {
Some(Object::Array(a)) => match a.first() {
Some(Object::String(s)) => Object::String(s.clone()),
_ => Object::String(PdfString { bytes: Vec::new(), hex: true }),
},
_ => Object::String(PdfString { bytes: Vec::new(), hex: true }),
};
vec![first, fresh]
} else {
vec![fresh.clone(), fresh]
};
trailer.set(b"ID".to_vec(), Object::Array(pair));
}
/// FNV-1a over 64 bits, with its published offset basis and prime. The identifier is built from it
/// rather than from `std`'s `DefaultHasher`, whose algorithm the standard library does not specify.
const FNV_OFFSET: u64 = 0xcbf2_9ce4_8422_2325;
const FNV_PRIME: u64 = 0x0000_0100_0000_01b3;
fn fnv1a(state: u64, bytes: &[u8]) -> u64 {
bytes.iter().fold(state, |h, b| (h ^ u64::from(*b)).wrapping_mul(FNV_PRIME))
}
/// Sixteen bytes for a file identifier, one 8-byte half per lane. The bytes are a pure function of
/// `SaveOptions::id_entropy`, `id_seed` and the written file: the hash is FNV-1a (above) and every
/// integer is written as a fixed-width big-endian value, so the same input gives the same identifier
/// on every platform and toolchain. Without entropy only the salt changes (see `id_salt`). FNV is
/// not collision-resistant against chosen content; the identifier only has to be unique, not secret.
/// Every written byte is hashed, once, in a single traversal of the output.
fn new_id_bytes(opts: &SaveOptions, written: &[u8]) -> Vec<u8> {
let salt = id_salt(opts);
let length = u64::try_from(written.len()).unwrap_or(u64::MAX);
let mut halves = [0u8, 1].map(|lane| {
let mut h = fnv1a(FNV_OFFSET, &salt.to_be_bytes());
h = fnv1a(h, &[lane]);
h = fnv1a(h, &opts.id_seed.to_be_bytes());
fnv1a(h, &length.to_be_bytes())
});
for chunk in written.chunks(4096) {
for h in &mut halves {
*h = fnv1a(*h, chunk);
}
}
halves.iter().flat_map(|h| h.to_be_bytes()).collect()
}
/// The salt of an identifier. With caller entropy it is a fixed hash of that entropy (deterministic,
/// for tests and callers who pin output). Without it the salt is fresh: on WebAssembly from the
/// browser ([`browser_salt`], so two fresh web apps that write the same bytes cannot share an
/// identifier), and elsewhere from the standard library's random hash keys — seeded by the operating
/// system, so separate processes differ — mixed with a process-wide counter, which separates
/// successive calls in one process.
fn id_salt(opts: &SaveOptions) -> u64 {
use std::hash::BuildHasher;
use std::sync::atomic::{AtomicU64, Ordering};
static COUNTER: AtomicU64 = AtomicU64::new(0);
match &opts.id_entropy {
Some(entropy) => fnv1a(FNV_OFFSET, entropy),
None => {
#[cfg(target_arch = "wasm32")]
if let Some(salt) = browser_salt() {
return salt;
}
std::collections::hash_map::RandomState::new().hash_one(COUNTER.fetch_add(1, Ordering::Relaxed))
}
}
}
/// One byte from one `Math.random` draw: the draw (in [0, 1)) scaled to [0, 256), integer part
/// taken. The quotient is in [0, 256), so a saturating cast never wraps; a NaN or infinite draw
/// (which a conforming engine never returns) folds in as 0. No input can panic here.
#[cfg(any(target_arch = "wasm32", test))]
fn draw_byte(random: f64) -> u8 {
(random * 256.0).floor().rem_euclid(256.0) as u8
}
/// The `Math.random` fallback's mixing: one salt byte per draw (see [`draw_byte`]), folded through
/// FNV-1a exactly like the Web Crypto bytes, so all sixteen bytes of the fold carry entropy from
/// the draws. Pure and platform-independent, so the native test target can exercise it; on wasm
/// each draw comes from `Math.random`.
#[cfg(any(target_arch = "wasm32", test))]
fn salt_from_draws(draws: &[f64; 16]) -> u64 {
let mut bytes = [0u8; 16];
for (byte, draw) in bytes.iter_mut().zip(draws) {
*byte = draw_byte(*draw);
}
fnv1a(FNV_OFFSET, &bytes)
}
/// Fresh salt entropy from the browser, on WebAssembly. `crypto.getRandomValues` (Web Crypto, the
/// browser's secure generator, available in windows and workers alike) gives sixteen bytes, folded
/// into one salt. When that API is missing or refuses the call, `Math.random` is mixed instead
/// ([`salt_from_draws`]: sixteen draws, so the salt's full width carries entropy) — browsers seed
/// it afresh for every document, so a fresh instance still cannot reproduce another's salt. Only
/// where neither is reachable does the caller fall back to its `RandomState` salt, whose keys are
/// fixed at compile time on `wasm32-unknown-unknown` — the state before browser entropy was wired
/// in.
#[cfg(target_arch = "wasm32")]
fn browser_salt() -> Option<u64> {
use wasm_bindgen::JsCast;
let crypto = js_sys::Reflect::get(&js_sys::global(), &wasm_bindgen::JsValue::from_str("crypto"))
.ok()
.filter(|crypto| !crypto.is_undefined() && !crypto.is_null());
if let Some(crypto) = crypto {
let fill = js_sys::Reflect::get(&crypto, &wasm_bindgen::JsValue::from_str("getRandomValues"))
.ok()
.and_then(|fill| fill.dyn_into::<js_sys::Function>().ok());
if let Some(fill) = fill {
let bytes = js_sys::Uint8Array::new_with_length(16);
if fill.call1(&crypto, &bytes).is_ok() {
return Some(fnv1a(FNV_OFFSET, &bytes.to_vec()));
}
}
}
// Web Crypto is missing or refused the call: degrade to `Math.random` rather than giving up.
let mut draws = [0.0; 16];
for draw in &mut draws {
*draw = js_sys::Math::random();
}
Some(salt_from_draws(&draws))
}
/// A PDF date string for "now" given seconds since the Unix epoch (UTC).
pub fn pdf_date(unix_secs: i64) -> String {
let (days, secs) = (unix_secs.div_euclid(86_400), unix_secs.rem_euclid(86_400));
@@ -711,6 +876,56 @@ mod tests {
use proptest::prelude::*;
#[test]
fn the_same_entropy_and_bytes_give_the_same_identifier_on_every_platform() {
// Computed from the formula outside this crate: fixed-width big-endian fields and FNV-1a.
let opts = super::SaveOptions { id_entropy: Some([0x42; 16]), id_seed: 7, ..super::SaveOptions::default() };
let want = [0x96, 0xf8, 0xcb, 0x7d, 0x5b, 0xb0, 0x09, 0xc8, 0x92, 0xc9, 0xb7, 0x51, 0xdf, 0xe6, 0xab, 0xd1];
assert_eq!(super::new_id_bytes(&opts, b"%PDF-1.7\n"), want);
}
#[test]
fn fnv1a_is_the_published_64_bit_algorithm() {
// Published FNV-1a (64 bit) test vectors, so the identifier can be reproduced outside this crate.
assert_eq!(super::fnv1a(super::FNV_OFFSET, b""), 0xcbf2_9ce4_8422_2325);
assert_eq!(super::fnv1a(super::FNV_OFFSET, b"a"), 0xaf63_dc4c_8601_ec8c);
assert_eq!(super::fnv1a(super::FNV_OFFSET, b"foobar"), 0x8594_4171_f739_67e8);
}
#[test]
fn math_random_draws_fill_the_whole_salt_width() {
// Draw i = (i + 1/2)/256 scales to the integer i, so the fold's byte i proves draw i landed
// there: all sixteen bytes carry their own draw's entropy (the fallback once shifted four
// draws into one word, leaving the other four bytes of the fold constant).
let draws: [f64; 16] = std::array::from_fn(|i| (i as f64 + 0.5) / 256.0);
let bytes: Vec<u8> = (0..16).collect();
assert_eq!(super::salt_from_draws(&draws), super::fnv1a(super::FNV_OFFSET, &bytes));
}
#[test]
fn every_math_random_draw_reaches_the_salt() {
let base = [0.5; 16];
let baseline = super::salt_from_draws(&base);
for i in 0..16 {
let mut draws = base;
draws[i] = 0.75;
assert_ne!(super::salt_from_draws(&draws), baseline, "draw {i} did not reach the salt");
}
}
#[test]
fn a_math_random_draw_outside_its_range_still_mixes_without_panicking() {
// A conforming `Math.random` stays in [0, 1), but the mixing must not panic outside it.
assert_eq!(super::draw_byte(0.0), 0);
assert_eq!(super::draw_byte(0.5), 128);
assert_eq!(super::draw_byte(255.5 / 256.0), 255);
assert_eq!(super::draw_byte(-0.5 / 256.0), 255);
assert_eq!(super::draw_byte(-1.0), 0);
assert_eq!(super::draw_byte(2.0), 0);
assert_eq!(super::draw_byte(f64::NAN), 0);
assert_eq!(super::draw_byte(f64::INFINITY), 0);
}
use super::*;
use crate::Lexer;
crates/cos/tests/file_identifier.rs
+276
View File
@@ -0,0 +1,276 @@
//! The file identifier (`/ID`, ISO 32000-2 §14.4) of a full save. An ordinary save keeps the
//! identifier the file has and writes the same bytes twice. A rewrite the document asked for (after
//! redaction or sanitizing) gives the file a new pair. An encrypted file keeps its first string,
//! because its file key is derived from it.
use std::sync::Arc;
use pdfcraft_cos::{Algorithm, Document, NewEncryption, Object, PdfString, SaveOptions, write_full, write_incremental};
use pdfcraft_testkit::Pdf;
const ID_HEX: &str = "/ID [<00112233445566778899AABBCCDDEEFF> <00112233445566778899AABBCCDDEEFF>]";
const ID_BYTES: [u8; 16] = [0x00, 0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF];
/// One page whose content holds a secret, with `entry` added to the trailer (for example an `/ID`).
fn source_with(entry: Option<&str>) -> Vec<u8> {
let pdf = Pdf::new()
.obj("<< /Type /Catalog /Pages 2 0 R >>")
.obj("<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>")
.obj("<< /Type /Page /Parent 2 0 R /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >>")
.stream("", "BT /F1 12 Tf 20 100 Td (TOPSECRET) Tj ET")
.obj("<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>");
match entry {
Some(entry) => pdf.trailer_entry(entry).build(),
None => pdf.build(),
}
}
/// [`source_with`] with the standard identifier, or none.
fn source(with_id: bool) -> Vec<u8> {
source_with(with_id.then_some(ID_HEX))
}
/// A one-page file whose 70,000-byte content stream ends in `last`. Two of these built with
/// different last bytes have the same length and differ only past the first 64 KiB.
fn large_source(last: u8) -> Vec<u8> {
let mut content = vec![b' '; 70_000];
content.push(last);
Pdf::new()
.obj("<< /Type /Catalog /Pages 2 0 R >>")
.obj("<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>")
.obj("<< /Type /Page /Parent 2 0 R /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> >> >>")
.stream("", content)
.obj("<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>")
.build()
}
fn reopen(bytes: Vec<u8>) -> Document {
Document::open(Arc::new(bytes)).unwrap()
}
/// The strings of the trailer's `/ID` (empty when there is none).
fn id_strings(doc: &Document) -> Vec<Vec<u8>> {
match doc.trailer().get(b"ID") {
Some(Object::Array(a)) => a.iter().map(|s| s.as_string().map(|s| s.bytes.clone()).unwrap_or_default()).collect(),
_ => Vec::new(),
}
}
#[test]
fn an_ordinary_save_keeps_the_identifier_and_writes_the_same_bytes() {
let doc = reopen(source(true));
let opts = SaveOptions::default();
let saved = write_full(&doc, &opts).unwrap();
assert_eq!(saved, write_full(&doc, &opts).unwrap(), "unchanged inputs produce identical bytes");
assert_eq!(id_strings(&reopen(saved)), vec![ID_BYTES.to_vec(), ID_BYTES.to_vec()]);
let appended = reopen(write_incremental(&doc, &opts).unwrap());
assert_eq!(id_strings(&appended), vec![ID_BYTES.to_vec(), ID_BYTES.to_vec()]);
}
#[test]
fn an_ordinary_save_of_a_file_without_an_identifier_creates_the_same_one_every_time() {
let doc = reopen(source(false));
let opts = SaveOptions::default();
let saved = write_full(&doc, &opts).unwrap();
assert_eq!(saved, write_full(&doc, &opts).unwrap(), "unchanged inputs produce identical bytes");
let ids = id_strings(&reopen(saved));
assert_eq!(ids.len(), 2);
assert!(ids.iter().all(|s| s.len() == 16), "{ids:?}");
}
#[test]
fn a_rewrite_the_document_asked_for_gets_a_new_identifier() {
let mut doc = reopen(source(true));
doc.require_full_save_with_new_id();
assert!(doc.full_save_required() && doc.new_id_required());
let opts = SaveOptions::default();
for saved in [write_incremental(&doc, &opts).unwrap(), write_full(&doc, &opts).unwrap()] {
let ids = id_strings(&reopen(saved));
assert_eq!(ids.len(), 2);
for id in &ids {
assert_eq!(id.len(), 16);
assert_ne!(id.as_slice(), ID_BYTES.as_slice(), "the copy must not claim the source's identifier");
}
}
// Every save is a new identity, not the same one again.
let first = id_strings(&reopen(write_full(&doc, &opts).unwrap()));
let second = id_strings(&reopen(write_full(&doc, &opts).unwrap()));
assert_ne!(first, second);
}
#[test]
fn a_rewrite_without_an_identifier_still_gets_one() {
let mut doc = reopen(source(false));
doc.require_full_save_with_new_id();
let ids = id_strings(&reopen(write_full(&doc, &SaveOptions::default()).unwrap()));
assert_eq!(ids.len(), 2);
assert!(ids.iter().all(|s| s.len() == 16), "{ids:?}");
}
#[test]
fn caller_entropy_makes_the_new_identifier_reproducible() {
let mut doc = reopen(source(true));
doc.require_full_save_with_new_id();
let with = |entropy: u8| SaveOptions { id_entropy: Some([entropy; 16]), ..SaveOptions::default() };
let first = write_full(&doc, &with(1)).unwrap();
assert_eq!(first, write_full(&doc, &with(1)).unwrap(), "the same entropy gives the same bytes");
let other = write_full(&doc, &with(2)).unwrap();
assert_ne!(id_strings(&reopen(first)), id_strings(&reopen(other)), "different entropy gives a different identifier");
}
#[test]
fn an_encrypted_rewrite_keeps_the_first_identifier_string_its_key_depends_on() {
for algorithm in [Algorithm::Rc4_128, Algorithm::Aes128, Algorithm::Aes256] {
let mut doc = reopen(source(true));
doc.set_encryption(&NewEncryption {
algorithm,
user_password: "user",
owner_password: "owner",
permissions: -1,
encrypt_metadata: true,
seed: [5; 32],
})
.unwrap();
let protected = write_full(&doc, &SaveOptions::default()).unwrap();
let before = Document::open_with_password(Arc::new(protected), Some("user")).unwrap();
let mut doc = before.clone();
doc.require_full_save_with_new_id();
let out = write_incremental(&doc, &SaveOptions::default()).unwrap();
// The output opens with the same password only if its first identifier string is unchanged.
let back = Document::open_with_password(Arc::new(out), Some("user")).unwrap();
let (old, new) = (id_strings(&before), id_strings(&back));
assert_eq!(new.len(), 2, "{algorithm:?}");
assert_eq!(new.first(), old.first(), "{algorithm:?}: the key depends on the first string");
assert_ne!(new.get(1), old.get(1), "{algorithm:?}: the second string is new");
}
}
/// The `/ID` shapes whose key the reader takes as the empty string: none at all, one entry, and a
/// first entry that is not a string.
fn unusable_ids() -> Vec<(&'static str, Option<Object>)> {
let empty = Object::String(PdfString::literal(Vec::new()));
vec![
("absent", None),
("one entry", Some(Object::Array(vec![empty.clone()]))),
("first entry not a string", Some(Object::Array(vec![Object::Int(7), empty]))),
]
}
/// An encrypted file whose key is derived from the empty string, with its trailer `/ID` replaced by `id`.
fn protected_with_id(algorithm: Algorithm, id: Option<Object>) -> Vec<u8> {
let mut doc = reopen(source_with(Some("/ID [<> <>]")));
doc.set_encryption(&NewEncryption {
algorithm,
user_password: "user",
owner_password: "owner",
permissions: -1,
encrypt_metadata: true,
seed: [5; 32],
})
.unwrap();
match id {
Some(id) => doc.trailer_mut().set(b"ID".to_vec(), id),
None => {
doc.trailer_mut().remove(b"ID");
}
}
write_full(&doc, &SaveOptions::default()).unwrap()
}
#[test]
fn an_encrypted_rewrite_without_a_usable_identifier_still_gets_a_new_one() {
// The key is derived from the first string of /ID, and a missing /ID, a one-entry /ID and a
// first entry that is not a string all count as the empty string. The rewrite keeps that
// (empty) first string, so the file still opens with its password, and writes a new second one.
for algorithm in [Algorithm::Rc4_128, Algorithm::Aes128, Algorithm::Aes256] {
for (shape, id) in unusable_ids() {
let before = Document::open_with_password(Arc::new(protected_with_id(algorithm, id)), Some("user")).unwrap();
let mut doc = before.clone();
doc.require_full_save_with_new_id();
let out = write_incremental(&doc, &SaveOptions::default()).unwrap();
let back = Document::open_with_password(Arc::new(out), Some("user")).unwrap();
let ids = id_strings(&back);
let previous = id_strings(&before);
assert_eq!(ids.len(), 2, "{algorithm:?}, {shape}: the rewrite writes a pair");
assert!(ids.first().is_some_and(|s| s.is_empty()), "{algorithm:?}, {shape}: the first string is the one the key uses");
assert_eq!(ids.get(1).map(Vec::len), Some(16), "{algorithm:?}, {shape}");
assert_ne!(ids.get(1), previous.get(1), "{algorithm:?}, {shape}: the second string is new");
}
}
}
/// The standard fixture rewritten for a new identifier, with `id_entropy` of 0x42.
fn fixed_entropy_rewrite() -> Vec<u8> {
let mut doc = reopen(source(true));
doc.require_full_save_with_new_id();
write_full(&doc, &SaveOptions { id_entropy: Some([0x42; 16]), ..SaveOptions::default() }).unwrap()
}
#[test]
fn a_fixed_entropy_gives_the_same_identifier_every_time() {
// The formula itself is pinned on fixed bytes in the writer's tests. The written file is not the
// same on every platform, so no identifier of a rewrite can be pinned here.
let first = id_strings(&reopen(fixed_entropy_rewrite()));
assert_eq!(first.len(), 2);
assert!(first.iter().all(|s| s.len() == 16), "{first:?}");
assert_eq!(first, id_strings(&reopen(fixed_entropy_rewrite())));
}
#[test]
fn the_identifier_depends_on_the_whole_rewritten_file() {
// Same caller entropy and same length; the two rewrites differ only in one byte of a stream
// that lies past the first 64 KiB of the output. Their identifiers must still differ.
let opts = SaveOptions { id_entropy: Some([9; 16]), ..SaveOptions::default() };
let rewrite = |last: u8| {
let mut doc = reopen(large_source(last));
doc.require_full_save_with_new_id();
write_full(&doc, &opts).unwrap()
};
let (a, b) = (rewrite(b'A'), rewrite(b'B'));
assert_eq!(a.len(), b.len(), "the two rewrites have the same length");
let first_difference = a.iter().zip(&b).position(|(x, y)| x != y).unwrap_or(usize::MAX);
assert!(first_difference > 64 * 1024, "the rewrites first differ at byte {first_difference}, inside the first 64 KiB");
assert_ne!(id_strings(&reopen(a)), id_strings(&reopen(b)), "content past 64 KiB must change the identifier");
}
/// Where [`stand_in_for_a_fresh_process`] writes its rewrite.
const ID_PROBE: &str = "PDFCRAFT_TEST_ID_PROBE";
/// The child half of [`separate_fresh_processes_get_different_identifiers`]: it runs alone, in its
/// own process, and writes a default-path rewrite of the standard fixture to the file the parent
/// named. Without the variable it is a plain no-op.
#[test]
fn stand_in_for_a_fresh_process() {
let Some(path) = std::env::var_os(ID_PROBE) else { return };
let mut doc = reopen(source(true));
doc.require_full_save_with_new_id();
std::fs::write(path, write_full(&doc, &SaveOptions::default()).unwrap()).unwrap();
}
#[test]
fn separate_fresh_processes_get_different_identifiers() {
// The default identifier is meant to differ between fresh instances of the whole save pipeline,
// not only between two calls in one process (which the counter already separates). On the native
// test target a fresh instance is a fresh process, whose hash keys the operating system seeds;
// the fix under test runs on WebAssembly, where no browser test harness exists — there the
// browser's `crypto.getRandomValues` stands in for this seeding, and this test cannot see it.
let exe = std::env::current_exe().unwrap();
let probe = |name: &str| {
let path = std::env::temp_dir().join(format!("pdfcraft-id-probe-{}-{name}.pdf", std::process::id()));
let out = std::process::Command::new(&exe)
.args(["--exact", "stand_in_for_a_fresh_process", "--nocapture", "--"])
.env(ID_PROBE, &path)
.output()
.expect("run this test's own binary as a child");
assert!(out.status.success(), "{name}: the child failed: {}", String::from_utf8_lossy(&out.stderr));
let saved = std::fs::read(&path).unwrap_or_else(|e| panic!("{name}: the child wrote its file: {e}"));
let ids = id_strings(&reopen(saved));
assert_eq!(ids.len(), 2, "{name}");
assert!(ids.iter().all(|s| s.len() == 16), "{name}: {ids:?}");
let _ = std::fs::remove_file(&path);
ids
};
let (first, second) = (probe("a"), probe("b"));
assert_ne!(first, second, "two fresh processes must not share an identifier");
}
crates/cos/tests/it/generations.rs
+35
View File
@@ -473,3 +473,38 @@ fn restoring_explicit_sets_removes_old_generation_aliases() {
}
}
}
/// `accepted_generations` lists exactly the generations `get` resolves, for every kind of slot:
/// a clean file slot, a damaged one (row and header disagree), a repair-aware edit, a plain
/// edit, a freed slot, a free row, an object stream member and an unknown number.
#[test]
fn accepted_generations_are_the_generations_get_resolves() {
let agrees = |doc: &Document, num: u32, expected: &[u16]| {
assert_eq!(doc.accepted_generations(num), expected, "object {num}");
for generation in 0..8 {
let live = !matches!(*doc.get(ObjRef::new(num, generation)), Object::Null);
assert_eq!(live, expected.contains(&generation), "object {num} generation {generation}");
}
};
let doc = Document::open(contradictory_entry(false)).unwrap();
agrees(&doc, 1, &[0]);
agrees(&doc, 3, &[0, 1]);
agrees(&doc, 9, &[]);
let mut repaired = doc.clone();
repaired.set_existing(ObjRef::new(3, 0), Object::Int(77)).unwrap();
agrees(&repaired, 3, &[0, 1]);
let mut narrowed = doc.clone();
narrowed.set(ObjRef::new(3, 2), Object::Int(99));
agrees(&narrowed, 3, &[2]);
let mut freed = doc.clone();
freed.free(ObjRef::new(3, 0));
agrees(&freed, 3, &[]);
agrees(&Document::open(contradictory_entry(true)).unwrap(), 3, &[]);
let mut doc =
Document::open(object_file(&[(1, 0, "<< /Type /Catalog /Pages 2 0 R >>"), (2, 0, "<< /Type /Pages /Kids [] /Count 0 >>")])).unwrap();
let member = doc.add(Object::Dict(Default::default()));
doc.update_dict(ObjRef::new(1, 0), |c| c.set(b"Extra".to_vec(), Object::Ref(member))).unwrap();
let packed = Document::open(Arc::new(write_full(&doc, &SaveOptions { object_streams: true, ..SaveOptions::default() }).unwrap())).unwrap();
assert!(matches!(packed.xref_entry(member.num), Some(XrefEntry::InStream { .. })), "the member is in an object stream");
agrees(&packed, member.num, &[0]);
}
crates/create/src/lib.rs
+153 -1
View File
@@ -92,6 +92,26 @@ pub fn blank(width: f64, height: f64, pages: usize) -> Result<Document, CreateEr
Ok(doc)
}
/// A document whose pages hold ready-made content: one `(width, height, resources, content)`
/// tuple per page, in page order. Empty content vectors make blank pages. The OCR Verify
/// screen's editable-text output builds its pages this way.
pub fn from_contents(pages: &[(f64, f64, Dict, Vec<u8>)]) -> Result<Document, CreateError> {
if pages.is_empty() || pages.len() > 10_000 {
return Err(CreateError::Invalid("invalid page count".into()));
}
if pages.iter().any(|(w, h, _, _)| !(w.is_finite() && h.is_finite() && (3.0..=MAX_SIDE).contains(w) && (3.0..=MAX_SIDE).contains(h))) {
return Err(CreateError::Invalid("invalid page size".into()));
}
let mut doc = Document::new_empty();
let mut tree = Pages::of(&doc)?;
for (w, h, resources, content) in pages {
let content = if content.is_empty() { None } else { Some(content.clone()) };
tree.add(&mut doc, *w, *h, resources.clone(), content);
}
tree.finish(&mut doc)?;
Ok(doc)
}
// ── images ──────────────────────────────────────────────────────────────────────────────────
/// An image ready to embed: its XObject dictionary, encoded data, optional soft mask, the ICC
@@ -496,6 +516,129 @@ fn decoded(name: &str, bytes: &[u8], format: image::ImageFormat) -> Result<Embed
Ok(rgba_image(rgba.as_raw(), rgba.dimensions(), (72.0, 72.0)))
}
fn le16(b: &[u8], at: usize) -> Option<u16> {
b.get(at..at + 2).map(|x| u16::from_le_bytes([x[0], x[1]]))
}
/// How a PCX image holds its pixels: bits per plane and planes.
#[derive(Clone, Copy, PartialEq)]
enum PcxLayout {
/// One byte per pixel per colour: each scanline has a red, a green and a blue row.
Rgb,
/// One byte per pixel, an index into the palette after the image (grey without one).
Indexed,
/// One bit per pixel in each of 1 to 4 planes, an index into the header's palette.
Bits(usize),
}
impl PcxLayout {
fn of(bits: u8, planes: usize) -> Option<PcxLayout> {
match (bits, planes) {
(8, 3) => Some(PcxLayout::Rgb),
(8, 1) => Some(PcxLayout::Indexed),
(1, 1..=4) => Some(PcxLayout::Bits(planes)),
_ => None,
}
}
}
/// The palette of a PCX image, and where its run-length coded scanlines end. An 8-bit image ends
/// with a 0x0C byte and 256 colours; a one-plane bit image is black and white whatever its header
/// holds; other bit images use the 16 colours at byte 16.
fn pcx_palette(bytes: &[u8], layout: PcxLayout) -> (Vec<u8>, usize) {
match layout {
PcxLayout::Rgb => (Vec::new(), bytes.len()),
PcxLayout::Indexed => match bytes.len().checked_sub(769).filter(|at| *at >= 128 && bytes.get(*at) == Some(&0x0C)) {
Some(at) => (bytes.get(at + 1..).unwrap_or_default().to_vec(), at),
None => ((0..=255u8).flat_map(|v| [v, v, v]).collect(), bytes.len()),
},
PcxLayout::Bits(1) => (vec![0, 0, 0, 255, 255, 255], bytes.len()),
PcxLayout::Bits(_) => (bytes.get(16..64).unwrap_or_default().to_vec(), bytes.len()),
}
}
/// The `rows` scanlines of `line` bytes each, decoded. A run may cross from one plane's row into the
/// next, but not out of its scanline.
fn pcx_scanlines(mut body: &[u8], rows: usize, line: usize) -> Option<Vec<u8>> {
let mut out = Vec::new();
out.try_reserve_exact(rows.checked_mul(line)?).ok()?;
for row in 1..=rows {
let end = row * line;
while out.len() < end {
let (&c, rest) = body.split_first()?;
body = rest;
if c < 0xC0 {
out.push(c);
continue;
}
let (&value, rest) = body.split_first()?;
body = rest;
let count = usize::from(c & 0x3F);
if count > end - out.len() {
return None;
}
out.resize(out.len() + count, value);
}
}
Some(out)
}
/// The RGB samples of the `w` pixels of scanline `row` (planes `stride` bytes apart).
fn pcx_row(out: &mut Vec<u8>, row: &[u8], w: usize, stride: usize, layout: PcxLayout, palette: &[u8]) {
let at = |i: usize| row.get(i).copied().unwrap_or(0);
for x in 0..w {
let index = match layout {
PcxLayout::Rgb => {
out.extend([at(x), at(stride + x), at(2 * stride + x)]);
continue;
}
PcxLayout::Indexed => usize::from(at(x)),
PcxLayout::Bits(planes) => (0..planes).fold(0, |index, k| index | (usize::from(at(k * stride + x / 8) >> (7 - x % 8) & 1) << k)),
};
out.extend((0..3).map(|c| palette.get(index * 3 + c).copied().unwrap_or(0)));
}
}
/// A PCX image: a 128-byte header (size at 4-11, resolution at 12 and 14, plane count at 65, bytes
/// per plane row at 66), then run-length coded scanlines, each holding one row of every plane.
fn pcx(name: &str, bytes: &[u8]) -> Result<Embedded, CreateError> {
let bad = |m: &str| CreateError::Image(name.into(), m.into());
let header = bytes.get(..128).ok_or_else(|| bad("a damaged PCX header"))?;
let word = |at: usize| le16(header, at).unwrap_or(0);
let byte = |at: usize| header.get(at).copied().unwrap_or(0);
let (bits, planes, stride) = (byte(3), usize::from(byte(65)), usize::from(word(66)));
let (Some(w), Some(h)) = (word(8).checked_sub(word(4)), word(10).checked_sub(word(6))) else {
return Err(bad("a damaged PCX header"));
};
let (w, h) = (usize::from(w) + 1, usize::from(h) + 1);
if planes == 0 || stride == 0 {
return Err(bad("a damaged PCX header"));
}
let layout = PcxLayout::of(bits, planes).ok_or_else(|| bad("this PCX plane layout is not supported"))?;
if stride < (w * usize::from(bits)).div_ceil(8) {
return Err(bad("a damaged PCX header"));
}
let line = planes * stride;
if line.saturating_mul(h) > MAX_DECODED_BYTES || w.saturating_mul(h).saturating_mul(3) > MAX_DECODED_BYTES {
return Err(bad("unusual PCX image size"));
}
let (palette, end) = pcx_palette(bytes, layout);
let body = bytes.get(128..end).unwrap_or_default();
let scanlines = pcx_scanlines(body, h, line).ok_or_else(|| bad("a damaged PCX body"))?;
let mut data = Vec::new();
data.try_reserve_exact(w * h * 3).map_err(|_| bad("not enough memory for this image"))?;
for row in scanlines.chunks_exact(line) {
pcx_row(&mut data, row, w, stride, layout, &palette);
}
let dpi = (f64::from(word(12)), f64::from(word(14)));
let dpi = if dpi.0 > 0.0 && dpi.1 > 0.0 { dpi } else { (72.0, 72.0) };
let mut d = Dict::new();
d.set(b"ColorSpace".to_vec(), Object::name("DeviceRGB"));
d.set(b"BitsPerComponent".to_vec(), Object::Int(8));
let px = (u32::try_from(w).unwrap_or(0), u32::try_from(h).unwrap_or(0));
Ok(Embedded { dict: d, data, filtered: false, smask: None, icc: None, px, dpi })
}
/// Largest GIF logical screen, in pixels (4096×4096).
const MAX_GIF_PIXELS: u64 = 1 << 24;
@@ -603,7 +746,13 @@ pub enum SourceKind {
}
/// File extensions Create converts to PDF (images and plain text).
pub const CONVERTIBLE: [&str; 12] = ["png", "jpg", "jpeg", "tif", "tiff", "gif", "bmp", "jp2", "j2k", "jpx", "txt", "text"];
pub const CONVERTIBLE: [&str; 13] = ["png", "jpg", "jpeg", "tif", "tiff", "gif", "bmp", "jp2", "j2k", "jpx", "pcx", "txt", "text"];
/// A PCX file: its manufacturer byte, a version (1 was never used), run-length coding and 1 or 8
/// bits per plane. `pcx` says when the rest of the header isn't one it can read.
fn is_pcx(bytes: &[u8]) -> bool {
bytes.first() == Some(&0x0A) && matches!(bytes.get(1), Some(0 | 2..=5)) && bytes.get(2) == Some(&1) && matches!(bytes.get(3), Some(1 | 8))
}
/// A classic TIFF or a BigTIFF (version 43), in either byte order.
fn is_tiff(bytes: &[u8]) -> bool {
@@ -620,6 +769,7 @@ fn is_image(bytes: &[u8]) -> bool {
|| bytes.starts_with(b"\x89PNG\r\n\x1a\n")
|| is_tiff(bytes)
|| is_gif(bytes)
|| is_pcx(bytes)
// JPEG 2000: a JP2 file or a raw codestream.
|| bytes.starts_with(JP2_SIGNATURE)
|| bytes.starts_with(&[0xFF, 0x4F, 0xFF, 0x51])
@@ -655,6 +805,8 @@ fn embed(name: &str, bytes: &[u8], wanted: usize) -> Result<Vec<Embedded>, Creat
Ok(vec![decoded(name, &pdfcraft_xfa::image::bmp_with_pixel_offset(bytes), image::ImageFormat::Bmp)?])
} else if is_gif(bytes) {
Ok(vec![decoded(name, bytes, image::ImageFormat::Gif)?])
} else if is_pcx(bytes) {
Ok(vec![pcx(name, bytes)?])
} else if bytes.starts_with(JP2_SIGNATURE) || bytes.starts_with(&[0xFF, 0x4F, 0xFF, 0x51]) {
Ok(vec![jpx(name, bytes)?])
} else {
crates/create/src/tests.rs
+76
View File
@@ -485,6 +485,82 @@ fn bigtiff_images_become_pages() {
assert_eq!(media(&pages(&doc)[0]), [0.0, 0.0, 2.0, 1.0], "4×2 px at 144 dpi");
}
/// A PCX header for a `w` by `h` image at 72 dpi.
fn pcx_header(version: u8, bits: u8, planes: u8, stride: u16, w: u16, h: u16) -> Vec<u8> {
let mut pcx = vec![0u8; 128];
pcx[..4].copy_from_slice(&[0x0A, version, 1, bits]);
pcx[8..10].copy_from_slice(&(w - 1).to_le_bytes());
pcx[10..12].copy_from_slice(&(h - 1).to_le_bytes());
pcx[12..14].copy_from_slice(&72u16.to_le_bytes());
pcx[14..16].copy_from_slice(&72u16.to_le_bytes());
pcx[65] = planes;
pcx[66..68].copy_from_slice(&stride.to_le_bytes());
pcx
}
/// A 24-bit PCX: each scanline holds a red, a green and a blue row.
#[test]
fn pcx_images_become_pages() {
let mut pcx = pcx_header(5, 8, 3, 2, 2, 2);
pcx.extend_from_slice(&[10, 11, 20, 21, 30, 31]);
pcx.extend_from_slice(&[40, 41, 50, 51, 60, 61]);
let embedded = super::pcx("a.pcx", &pcx).unwrap();
assert_eq!(embedded.data, [10, 20, 30, 11, 21, 31, 40, 50, 60, 41, 51, 61]);
let doc = reopen(&from_images(&[("a.pcx".into(), pcx)]).unwrap());
assert_eq!(pages(&doc).len(), 1);
assert_eq!(media(&pages(&doc)[0]), [0.0, 0.0, 2.0, 2.0]);
assert_eq!(image_of(&doc, 0).name(b"ColorSpace"), Some(&b"DeviceRGB"[..]));
}
#[test]
fn a_pcx_run_may_cross_planes_but_not_scanlines() {
// 0xC4 repeats 7 four times: the red row and half of the green one
let mut pcx = pcx_header(5, 8, 3, 2, 2, 1);
pcx.extend_from_slice(&[0xC4, 7, 8, 9]);
assert_eq!(super::pcx("a.pcx", &pcx).unwrap().data[..], [7, 7, 8, 7, 7, 9]);
let mut pcx = pcx_header(5, 8, 3, 2, 2, 2);
pcx.extend_from_slice(&[0xC8, 7, 1, 2, 3, 4]);
assert!(super::pcx("a.pcx", &pcx).is_err(), "eight bytes in a six-byte scanline");
}
#[test]
fn an_8_bit_pcx_uses_its_palette_and_is_grey_without_one() {
let mut pcx = pcx_header(5, 8, 1, 2, 2, 1);
pcx.extend_from_slice(&[1, 2]);
assert_eq!(super::pcx("a.pcx", &pcx).unwrap().data[..], [1, 1, 1, 2, 2, 2], "no palette: grey");
pcx.push(0x0C);
let mut palette = vec![0u8; 768];
palette[3..9].copy_from_slice(&[9, 8, 7, 6, 5, 4]);
pcx.extend_from_slice(&palette);
assert_eq!(super::pcx("a.pcx", &pcx).unwrap().data[..], [9, 8, 7, 6, 5, 4]);
}
#[test]
fn bit_plane_pcx_images_read_their_palette() {
// version 2, one plane: black and white
let mut pcx = pcx_header(2, 1, 1, 2, 2, 1);
pcx.extend_from_slice(&[0b1000_0000, 0]);
assert!(from_images(&[("a.pcx".into(), pcx.clone())]).is_ok(), "an old version is still a PCX");
assert_eq!(super::pcx("a.pcx", &pcx).unwrap().data[..], [255, 255, 255, 0, 0, 0]);
// two planes: colour 3 is at byte 16 + 9
let mut pcx = pcx_header(5, 1, 2, 1, 1, 1);
pcx[25..28].copy_from_slice(&[1, 2, 3]);
pcx.extend_from_slice(&[0x80, 0x80]);
assert_eq!(super::pcx("a.pcx", &pcx).unwrap().data[..], [1, 2, 3]);
}
#[test]
fn damaged_pcx_headers_and_bodies_are_refused() {
let no_planes = pcx_header(5, 8, 0, 2, 2, 1);
let no_stride = pcx_header(5, 8, 3, 0, 2, 1);
let short_stride = pcx_header(5, 8, 3, 1, 2, 1);
let short_body = pcx_header(5, 8, 3, 2, 2, 1);
for bytes in [no_planes, no_stride, short_stride, short_body] {
assert!(super::pcx("a.pcx", &bytes).is_err());
}
assert!(super::pcx("a.pcx", &[0x0A, 5, 1, 8]).is_err(), "no header");
}
/// A synthetic ICC profile: a 128-byte header (size, device class, colour space, `acsp`) and
/// some payload. Only the header is read.
fn icc_profile(space: &[u8; 4]) -> Vec<u8> {
crates/crypt/src/lib.rs
+19 -7
View File
@@ -82,17 +82,24 @@ pub enum Auth {
User,
}
/// Permission bits from `/P` (§7.6.4.2, Table 22). Owners have every permission.
/// Permission flags from `/P`, read the way the handler's revision reads them (ISO 32000-1:2008,
/// Table 22). Owners have every permission.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct Permissions {
pub bits: i32,
pub owner: bool,
/// The `/R` value of the handler that read `bits`.
pub revision: i64,
}
impl Permissions {
fn bit(&self, n: u32) -> bool {
self.owner || (self.bits >> (n - 1)) & 1 == 1
}
/// Revision 3 and later read the full flag list of Table 22; revision 2 has its own.
fn is_revision_3_or_later(&self) -> bool {
self.revision >= 3
}
pub fn print(&self) -> bool {
self.bit(3)
}
@@ -107,17 +114,22 @@ impl Permissions {
self.bit(6)
}
pub fn fill_forms(&self) -> bool {
self.bit(9) || self.bit(6)
if self.is_revision_3_or_later() { self.bit(9) || self.bit(6) } else { self.bit(6) }
}
/// Revision 2 covers accessibility with the copy flag (bit 5); revision 3 and later with bit 10.
pub fn extract_for_accessibility(&self) -> bool {
self.bit(10)
if self.is_revision_3_or_later() { self.bit(10) } else { self.bit(5) }
}
/// Insert, rotate or delete pages; create bookmarks and thumbnails.
/// Insert, rotate or delete pages; create bookmarks and thumbnails. Revision 3 and later grant
/// it by bit 11 alone. Table 22 defines bit 11 for those revisions only, so at revision 2 the
/// modify flag (bit 4) covers it.
pub fn assemble(&self) -> bool {
self.bit(11) || self.bit(4)
if self.is_revision_3_or_later() { self.bit(11) } else { self.bit(4) }
}
/// Revision 3 and later need bit 3 as well as bit 12. Table 22 defines bit 12 for those
/// revisions only: at revision 2 the print flag (bit 3) is all there is.
pub fn print_high_quality(&self) -> bool {
self.bit(12)
if self.is_revision_3_or_later() { self.print() && self.bit(12) } else { self.print() }
}
}
@@ -250,7 +262,7 @@ impl SecurityHandler {
/// edited `/P` grants nothing extra. Without a readable `/Perms`, `/P` applies as before.
pub fn permissions(&self) -> Permissions {
let bits = self.perms_p().map_or(self.dict.p, |p| self.dict.p & p);
Permissions { bits, owner: self.auth == Auth::Owner }
Permissions { bits, owner: self.auth == Auth::Owner, revision: self.dict.r }
}
/// `/P` as `/Perms` holds it: AES-256 ECB of one block (= CBC with a zero IV), marked `adb`.
crates/crypt/src/tests.rs
+3 -3
View File
@@ -240,11 +240,11 @@ fn unsupported_handlers_are_reported() {
#[test]
fn permission_bits_follow_table_22() {
let p = Permissions { bits: -3904, owner: false }; // 0xFFFFF0C0: only reserved bits set
let p = Permissions { bits: -3904, owner: false, revision: 3 }; // 0xFFFFF0C0: only reserved bits set
assert!(!p.print() && !p.modify() && !p.copy() && !p.annotate() && !p.assemble());
let p = Permissions { bits: -1, owner: false };
let p = Permissions { bits: -1, owner: false, revision: 3 };
assert!(p.print() && p.modify() && p.copy() && p.annotate() && p.assemble() && p.fill_forms());
let p = Permissions { bits: 0, owner: true };
let p = Permissions { bits: 0, owner: true, revision: 3 };
assert!(p.modify(), "owners ignore /P");
}
crates/crypt/tests/it/main.rs
+1
View File
@@ -1,4 +1,5 @@
//! The crate's integration tests, linked into one test binary.
mod permissions_revision;
mod proptests;
mod qpdf_literals;
crates/crypt/tests/it/permissions_revision.rs
+126
View File
@@ -0,0 +1,126 @@
//! The permission accessors of `pdfcraft_crypt::Permissions`, checked against the flag table of
//! ISO 32000-1:2008 (Table 22). Expected values come from that table only.
//!
//! Bit n has the mask `1 << (n - 1)`. Readers use bits 3 to 6 and 9 to 12 and ignore the rest.
use pdfcraft_crypt::{Algorithm, Auth, NewEncryption, Permissions, SecurityHandler, create};
const ID: &[u8] = b"0123456789abcdef";
const T: bool = true;
const F: bool = false;
/// The eight accessors in this order: print, high-quality print, modify, copy, accessibility,
/// annotate, fill forms, assemble.
fn cells(p: &Permissions) -> [bool; 8] {
[p.print(), p.print_high_quality(), p.modify(), p.copy(), p.extract_for_accessibility(), p.annotate(), p.fill_forms(), p.assemble()]
}
/// Creates a file with `flags` at revision `revision`, opens it with the user password, and
/// returns the user's permissions.
fn user_view(alg: Algorithm, revision: i64, flags: i32) -> Permissions {
let made = create(
&NewEncryption { algorithm: alg, user_password: "user", owner_password: "owner", permissions: flags, encrypt_metadata: true, seed: [42; 32] },
ID,
)
.expect("the handler is created");
assert_eq!(made.dict().r, revision, "the algorithm writes revision {revision}");
let opened = SecurityHandler::open(made.dict().clone(), ID, Some("user")).expect("the user password opens the file");
assert_eq!(opened.auth(), Auth::User, "the user password gives user access");
opened.permissions()
}
/// Revision 3 and later: each row is a flag word and the expected eight cells.
const REVISION_3_ROWS: &[(&str, i32, [bool; 8])] = &[
("every meaningful flag set, ignored flags clear", -196, [T, T, T, T, T, T, T, T]),
("every flag set, ignored flags set too", -1, [T, T, T, T, T, T, T, T]),
("reserved flags only", -4096, [F, F, F, F, F, F, F, F]),
("ignored flags set, meaningful flags clear", -3901, [F, F, F, F, F, F, F, F]),
("modify flag clear", -9, [T, T, F, T, T, T, T, T]),
("copy flag clear", -17, [T, T, T, F, T, T, T, T]),
("annotate flag clear", -33, [T, T, T, T, T, F, T, T]),
("annotate and fill-existing flags clear", -289, [T, T, T, T, T, F, F, T]),
("annotate flag set, fill-existing flag clear", -257, [T, T, T, T, T, T, T, T]),
("accessibility flag clear", -513, [T, T, T, T, F, T, T, T]),
("high-quality flag clear, print flag set", -2049, [T, F, T, T, T, T, T, T]),
("bits 3 to 12 set, reserved bits clear", 4092, [T, T, T, T, T, T, T, T]),
("positive flag word, bit 32 clear", 2_147_483_647, [T, T, T, T, T, T, T, T]),
("revision 2 example read at revision 3", -44, [T, T, F, T, T, F, T, T]),
];
#[test]
fn revision_three_rows_hold_through_a_created_and_reopened_file() {
for (name, bits, want) in REVISION_3_ROWS {
assert_eq!(cells(&user_view(Algorithm::Rc4_128, 3, *bits)), *want, "{name}: flag word {bits:#010x}");
}
}
#[test]
fn owner_access_gives_every_flag() {
let made = create(
&NewEncryption {
algorithm: Algorithm::Rc4_128,
user_password: "user",
owner_password: "owner",
permissions: 0,
encrypt_metadata: true,
seed: [42; 32],
},
ID,
)
.expect("the handler is created");
assert_eq!(made.auth(), Auth::Owner, "the owner password gives owner access");
assert_eq!(cells(&made.permissions()), [T; 8]);
}
#[test]
fn revision_two_print_modify_copy_and_annotate_follow_the_table() {
for (bits, want) in [(-4, [T, T, T, T]), (-44, [T, F, T, F]), (-17, [T, T, F, T])] {
let p = user_view(Algorithm::Rc4_40, 2, bits);
assert_eq!([p.print(), p.modify(), p.copy(), p.annotate()], want, "revision 2, flag word {bits}");
}
}
#[test]
fn revision_two_with_every_flag_set_grants_accessibility_and_form_filling() {
let p = user_view(Algorithm::Rc4_40, 2, -4);
assert!(p.extract_for_accessibility(), "bit 5 is set, and at revision 2 it covers accessibility");
assert!(p.fill_forms(), "bit 6 is set");
}
#[test]
fn revision_three_assembly_needs_its_own_flag() {
let p = user_view(Algorithm::Rc4_128, 3, -1025);
assert!(p.modify(), "bit 4 is set");
assert!(!p.assemble(), "assembly needs bit 11; bit 4 alone does not grant it");
}
#[test]
fn revision_three_high_quality_print_needs_the_print_flag() {
let p = user_view(Algorithm::Rc4_128, 3, -5);
assert!(!p.print(), "bit 3 is clear");
assert!(!p.print_high_quality(), "high-quality print needs bit 3 and bit 12");
}
#[test]
fn revision_two_accessibility_follows_the_copy_flag() {
let p = user_view(Algorithm::Rc4_40, 2, -17);
assert!(!p.copy(), "bit 5 is clear");
assert!(!p.extract_for_accessibility(), "at revision 2 accessibility is bit 5");
}
#[test]
fn revision_two_form_filling_ignores_bit_nine() {
let p = user_view(Algorithm::Rc4_40, 2, -44);
assert!(!p.annotate(), "bit 6 is clear");
assert!(!p.fill_forms(), "at revision 2 form filling is bit 6 alone");
}
#[test]
fn revision_two_ignores_the_flags_only_later_revisions_define() {
// Bit 4 is clear and bit 11 set: bit 11 means nothing at revision 2, so assembly stays refused.
let p = user_view(Algorithm::Rc4_40, 2, -44);
assert!(!p.assemble(), "at revision 2 assembly is bit 4 alone");
// Bit 3 is set and bit 12 clear: printing at revision 2 has one quality.
let p = user_view(Algorithm::Rc4_40, 2, -2049);
assert!(p.print_high_quality(), "at revision 2 high-quality print is bit 3 alone");
}
crates/edit/src/lib.rs
+1 -1
View File
@@ -78,7 +78,7 @@ pub(crate) use place::undrawable;
#[cfg(test)]
use place::{ISOLATION_MAX_DEPTH, OLD_WRAP_CLOSE, WRAP_CLOSE, WRAP_OPEN, is_wrap_close, page_fonts, stream_bytes, stream_tag, tagged, wrapper_role};
use place::{Reach, check, check_on_top, contents, draw_on_top, lift_on_top, page_list, place_tagged, revealed_under_paint};
pub use place::{stamp, undrawable_message};
pub use place::{add_type3_fonts, stamp, stamp_with_fonts, undrawable_message};
#[cfg(test)]
mod marked_isolation_tests;
crates/edit/src/place.rs
+65
View File
@@ -478,14 +478,79 @@ pub(crate) fn place_inside_wrapper(doc: &mut Document, page: &pdfcraft_model::Pa
/// it away); the original content is wrapped in q/Q first. Used for applied redaction boxes.
/// Returns the new content stream.
pub fn stamp(doc: &mut Document, page: usize, tag: &str, content: Vec<u8>) -> Result<ObjRef, EditError> {
stamp_with_fonts(doc, page, tag, &mut pdfcraft_fonts::TextPlan::default(), |_| content)
}
/// Draw the content `make` builds on top of a page, with standard Helvetica as `/PCHelv` and
/// the plan's Type3 fonts added to the page's resources first (so the content can reference
/// them by the names the plan carries). The original content is wrapped in q/Q first; the
/// stream is tagged `tag` (not a mark kind, so Remove never takes it away). This is how the
/// OCR text layers get their Unicode faces (see [`add_type3_fonts`]). Returns the new content
/// stream.
pub fn stamp_with_fonts(
doc: &mut Document,
page: usize,
tag: &str,
plan: &mut pdfcraft_fonts::TextPlan,
make: impl FnOnce(&pdfcraft_fonts::TextPlan) -> Vec<u8>,
) -> Result<ObjRef, EditError> {
let all = page_list(doc);
check(&[page], all.len())?;
check_on_top(doc, &all[page])?;
add_type3_fonts(doc, page, plan)?;
let content = make(plan);
let all = page_list(doc);
add_resources(doc, &all[page], None, Some(&content))?;
let p = page_list(doc).swap_remove(page);
place_tagged(doc, &p, tag, content, false)
}
/// Add the plan's Type3 fonts to a page's (0-based) resources, building each font once and
/// sharing the object with every page that uses it. Fills in each planned font's resource name
/// (`PCOcr`, `PCOcr1`, … — a name free on the page being registered: a font built for an
/// earlier page keeps its name only where that name is free or already that font) and object;
/// the content generators read the names from the plan, so this must run before they are
/// called for each page.
pub fn add_type3_fonts(doc: &mut Document, page: usize, plan: &mut pdfcraft_fonts::TextPlan) -> Result<(), EditError> {
if plan.fonts.is_empty() {
return Ok(());
}
let all = page_list(doc);
check(&[page], all.len())?;
let p = &all[page];
let mut res = p.dict.get(b"Resources").map(|r| doc.resolve(r)).and_then(|r| r.as_dict().cloned()).unwrap_or_default();
let mut fonts = res.get(b"Font").map(|f| doc.resolve(f)).and_then(|f| f.as_dict().cloned()).unwrap_or_default();
for i in 0..plan.fonts.len() {
let object = match plan.fonts[i].object {
Some(r) => {
// Built for an earlier page under a name that was free THERE. On this page the
// name may already be another font (the page's own, or an earlier OCR layer's):
// then this page gets a free name of its own, and the plan carries it for the
// content written next.
let taken = fonts.get(plan.fonts[i].name.as_bytes()).is_some_and(|o| o.as_ref() != Some(r));
if taken {
plan.fonts[i].name = super::text::unique_font_name(&fonts, "PCOcr");
}
r
}
None => {
let glyphs = plan.fonts[i].glyphs.iter().map(|(ch, g)| (*ch, super::text::glyph_path(g), g.width)).collect::<Vec<_>>();
let dict = super::text::type3_font_dict(doc, &glyphs)?;
let r = doc.add(Object::Dict(dict));
plan.fonts[i].name = super::text::unique_font_name(&fonts, "PCOcr");
plan.fonts[i].object = Some(r);
r
}
};
fonts.set(plan.fonts[i].name.as_bytes().to_vec(), Object::Ref(object));
}
res.set(b"Font".to_vec(), Object::Dict(fonts));
// The page gets its own (possibly copied) resources; shared dictionaries stay untouched.
let obj = p.obj;
doc.update_dict(obj, |d| d.set(b"Resources".to_vec(), Object::Dict(res)))?;
Ok(())
}
pub(crate) fn begin(kind: MarkKind, subtype: &str, matrix: [f64; 6]) -> String {
format!(
"q\n/Artifact <</Type /Pagination /Subtype /{subtype} /PCMark /{}>> BDC\n{} {} {} {} {} {} cm\n",
crates/edit/src/tests.rs
+111 -3
View File
@@ -3514,7 +3514,10 @@ fn page_content_bytes(doc: &Document, page: usize) -> Vec<u8> {
}
}
/// A page drawing image XObject /Im0 at 100,100 size 200 × 100 (pixels 4 × 2).
/// A page drawing image XObject /Im0 at 100,100 size 200 × 100 (pixels 4 × 2), on a real optional
/// content layer: the catalog's `/OCProperties` declares object 7 as the only OCG and the page
/// resources' `/Properties` maps `/L0` to it, so `/OC /L0 BDC` marks a valid optional-content
/// member rather than a dangling name.
fn image_page() -> Document {
image_page_with("q 200 0 0 100 100 100 cm /Im0 Do Q BT /F1 12 Tf 72 700 Td (Caption) Tj ET")
}
@@ -3534,12 +3537,13 @@ fn remove_crop(doc: &mut Document, page: usize, index: usize) -> Result<(), Edit
fn image_page_with(content: &str) -> Document {
let objs: Vec<String> = vec![
"<< /Type /Catalog /Pages 2 0 R >>".into(),
"<< /Type /Catalog /Pages 2 0 R /OCProperties << /OCGs [7 0 R] /D << /Order [7 0 R] >> >> >>".into(),
"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".into(),
"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 600 800] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> /XObject << /Im0 6 0 R >> /ExtGState << /GS0 << /Type /ExtGState /ca 0.5 /CA 0.5 >> /GSalpha << /Type /ExtGState /ca 0.6 >> /GSblend << /Type /ExtGState /BM /Multiply >> /GS_translucent << /Type /ExtGState /ca 0.5 >> >> >> >>".into(),
"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 600 800] /Contents 4 0 R /Resources << /Font << /F1 5 0 R >> /XObject << /Im0 6 0 R >> /ExtGState << /GS0 << /Type /ExtGState /ca 0.5 /CA 0.5 >> /GSalpha << /Type /ExtGState /ca 0.6 >> /GSblend << /Type /ExtGState /BM /Multiply >> /GS_translucent << /Type /ExtGState /ca 0.5 >> >> /Properties << /L0 7 0 R >> >> >>".into(),
format!("<< /Length {} >>\nstream\n{content}\nendstream", content.len()),
"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>".into(),
"<< /Type /XObject /Subtype /Image /Width 4 /Height 2 /ColorSpace /DeviceGray /BitsPerComponent 8 /Length 8 >>\nstream\n\u{0}\u{0}\u{0}\u{0}\u{0}\u{0}\u{0}\u{0}\nendstream".into(),
"<< /Type /OCG /Name (Layer 0) >>".into(),
];
let mut out = b"%PDF-1.7\n".to_vec();
let mut offs = Vec::new();
@@ -3753,6 +3757,30 @@ fn repeated_group_moves_of_a_cropped_image_change_one_clip_transform() {
assert_eq!(caption, ["Caption"]);
}
#[test]
fn moving_an_image_inside_marked_content_leaves_every_sequence_intact() {
// `translate_images` changes the image's own `cm`; its `Do` stays where it was, so a moved
// image stays in the sequence that tags it (an image on a layer sits in `/OC … BDC`).
for content in [
"/P <</MCID 0>> BDC q 100 0 0 100 50 50 cm /Im0 Do Q EMC /P <</MCID 1>> BDC BT /F1 12 Tf 72 700 Td (Text) Tj ET EMC",
"/OC /L0 BDC q 100 0 0 100 50 50 cm /Im0 Do Q EMC /P <</MCID 1>> BDC BT /F1 12 Tf 72 700 Td (Text) Tj ET EMC",
] {
let mut doc = image_page_with(content);
let before = images::page_images(&doc, 0).unwrap()[0].rect;
images::translate_images(&mut doc, 0, &[0], [10.0, 0.0]).unwrap();
let after = images::page_images(&doc, 0).unwrap()[0].rect;
assert!(close(after, [before[0] + 10.0, before[1], before[2] + 10.0, before[3]]), "{content}: {after:?}");
let ops = pdfcraft_content::parse(&page_content_bytes(&doc, 0)).ops;
let count = |name: &str| ops.iter().filter(|op| op.is(name)).count();
assert_eq!((count("BDC"), count("EMC")), (2, 2), "{content}: both sequences remain");
let first = ops.iter().position(|op| op.is("BDC")).unwrap();
let end = ops.iter().position(|op| op.is("EMC")).unwrap();
let image = ops.iter().position(|op| op.is("Do")).unwrap();
assert!(first < image && image < end, "{content}: the image is still inside its own sequence");
assert!(!ops[end..].iter().any(|op| op.is("Do")), "{content}: the other sequence gained no image");
}
}
#[test]
fn page_images_move_turn_replace_and_delete() {
let mut doc = image_page();
@@ -10229,6 +10257,86 @@ fn styled_existing_justification_is_refused_instead_of_silently_left_aligning()
assert_eq!(page_content_bytes(&reopen(&doc), 0), before);
}
/// The OCR stamp path registers the plan's Type3 faces (a free `PCOcr…` name per stamp, so a
/// page that is OCR'd twice keeps both layers' fonts), and the stamped codes extract back
/// through the font's ToUnicode CMap.
#[test]
fn stamp_with_fonts_registers_the_planned_type3_faces() {
if without_craft_fonts("stamp_with_fonts_registers_the_planned_type3_faces") {
return;
}
let mut doc = text_page("BT /F2 12 Tf 72 700 Td (ab) Tj ET");
let mut plan = pdfcraft_fonts::TextPlan::for_words(["日本"]);
assert_eq!(plan.missing(), 0);
stamp_with_fonts(&mut doc, 0, "OCR", &mut plan, |p| {
let codes: Vec<u8> = p.encoding(0).iter().map(|&(_, c)| c).collect();
let mut c = format!("BT 3 Tr /PCHelv 1 Tf\n/{} 1 Tf 100 700 Td ", p.fonts[0].name).into_bytes();
c.extend_from_slice(&pdfcraft_fonts::literal(&codes));
c.extend_from_slice(b" Tj ET");
c
})
.unwrap();
assert!(plan.fonts[0].name.starts_with("PCOcr"), "{:?}", plan.fonts[0].name);
// The registered font carries a ToUnicode CMap; the engine's own reader decodes the run.
let reopened = Document::open(Arc::new(write_incremental(&doc, &SaveOptions::default()).unwrap())).unwrap();
let lines = text::text_lines(&reopened, 0).unwrap();
assert!(lines.iter().any(|l| l.text.contains("日本")), "{lines:?}");
// A second stamp on the same page takes the next free name; the first layer's font stands.
let mut second = pdfcraft_fonts::TextPlan::for_words(["語"]);
stamp_with_fonts(&mut doc, 0, "OCR", &mut second, |p| format!("BT 3 Tr /{} 1 Tf 100 600 Td (\u{1}) Tj ET", p.fonts[0].name).into_bytes())
.unwrap();
assert_eq!(second.fonts[0].name, format!("{}1", plan.fonts[0].name));
}
/// A Type3 face built for page 1 is reused on page 2, but page 2 already has a font under the
/// name page 1 gave it: page 2's own font keeps its name, and the shared face is registered on
/// page 2 under a free one that the plan carries for page 2's content. (The plan is written out
/// by hand, so this runs without the craft-fonts build input.)
#[test]
fn a_reused_type3_face_never_replaces_a_later_page_s_font_of_the_same_name() {
let mut doc = build(&[
"<< /Type /Catalog /Pages 2 0 R >>",
"<< /Type /Pages /Kids [3 0 R 4 0 R] /Count 2 >>",
"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 600 800] /Contents 5 0 R >>",
"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 600 800] /Contents 6 0 R /Resources << /Font << /PCOcr 7 0 R >> >> >>",
&stream("", ""),
&stream("", "BT /PCOcr 12 Tf 72 700 Td (own) Tj ET"),
"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>",
]);
let glyph = pdfcraft_fonts::GlyphOutline { contours: vec![vec![[0.1, 0.0], [0.9, 0.0], [0.5, 0.8]]], width: 1.0, bbox: [0.1, 0.0, 0.9, 0.8] };
let mut plan = pdfcraft_fonts::TextPlan::default();
plan.fonts.push(pdfcraft_fonts::PlannedType3 { family: "Test", glyphs: vec![('日', glyph)], name: String::new(), object: None });
let draw = |p: &pdfcraft_fonts::TextPlan| format!("BT 3 Tr /{} 1 Tf 100 600 Td (\u{1}) Tj ET", p.fonts[0].name).into_bytes();
stamp_with_fonts(&mut doc, 0, "OCR", &mut plan, draw).unwrap();
assert_eq!(plan.fonts[0].name, "PCOcr", "free on page 1");
let face = plan.fonts[0].object.expect("built once");
stamp_with_fonts(&mut doc, 1, "OCR", &mut plan, draw).unwrap();
assert_ne!(plan.fonts[0].name, "PCOcr", "page 2 already has a PCOcr");
let page2 = &pdfcraft_model::pages(&doc)[1];
let res = doc.resolve(page2.dict.get(b"Resources").unwrap());
let fonts = doc.resolve(res.as_dict().unwrap().get(b"Font").unwrap());
let fonts = fonts.as_dict().unwrap();
assert_eq!(fonts.get(b"PCOcr").and_then(|o| o.as_ref()), Some(pdfcraft_cos::ObjRef::new(7, 0)), "page 2's own font keeps its name");
assert_eq!(fonts.get(plan.fonts[0].name.as_bytes()).and_then(|o| o.as_ref()), Some(face), "the shared face, under a free name");
assert_eq!(plan.fonts[0].object, Some(face), "still built once");
}
#[test]
fn planned_type3_charproc_has_valid_metrics_and_encloses_its_ink() {
let glyph = pdfcraft_fonts::GlyphOutline {
contours: vec![vec![[-0.1001, -0.2001], [0.9001, 1.1001]]],
width: 0.8,
bbox: [-0.1001, -0.2001, 0.9001, 1.1001],
};
let parsed = pdfcraft_content::parse(&text::glyph_path(&glyph));
assert_eq!(parsed.skipped, 0);
let metrics = &parsed.ops[0];
assert!(metrics.is("d1"));
assert_eq!(metrics.operands.len(), 6);
assert_eq!((metrics.num(0), metrics.num(1)), (Some(800.0), Some(0.0)));
assert_eq!((metrics.num(2), metrics.num(3), metrics.num(4), metrics.num(5)), (Some(-101.0), Some(-201.0), Some(901.0), Some(1101.0)));
}
#[test]
fn image_arrange_refuses_crossed_scope_boundaries_without_changes() {
for content in ["/P <</MCID 0>> BDC q 10 0 0 10 0 0 cm /Im0 Do EMC Q /Im0 Do", "q /P <</MCID 0>> BDC /Im0 Do Q EMC /Im0 Do"] {
crates/edit/src/text.rs
+1
View File
@@ -52,6 +52,7 @@ use fallback::{
};
#[cfg(test)]
use fallback::{add_type3_font, is_type3_fallback};
pub(crate) use fallback::{glyph_path, type3_font_dict, unique_font_name};
use interp::{Origin, Paint, union};
use lines::page_dict;
pub(crate) use lines::{FormBudget, content_streams, content_streams_within, dict_in_place, form_call, resource_dict, rewritten_contents, splice};
crates/edit/src/text/fallback.rs
+86
View File
@@ -493,6 +493,92 @@ pub(super) fn add_type3_font(doc: &mut Document, fonts_res: &mut Dict, face: &Cr
name
}
/// A glyph's CharProc body at 1000 units per em, with a six-operand `d1` and its actual ink box.
pub(crate) fn glyph_path(glyph: &pdfcraft_fonts::GlyphOutline) -> Vec<u8> {
let scale = 1000.0;
let b = if glyph.contours.is_empty() { [0.0; 4] } else { glyph.bbox };
let mut out = format!(
"{} 0 {} {} {} {} d1\n",
pdf_num(glyph.width * scale),
pdf_num((b[0] * scale).floor()),
pdf_num((b[1] * scale).floor()),
pdf_num((b[2] * scale).ceil()),
pdf_num((b[3] * scale).ceil())
)
.into_bytes();
for contour in &glyph.contours {
let Some(first) = contour.first() else { continue };
out.extend_from_slice(format!("{} {} m\n", pdf_num(first[0] * scale), pdf_num(first[1] * scale)).as_bytes());
for p in contour.iter().skip(1) {
out.extend_from_slice(format!("{} {} l\n", pdf_num(p[0] * scale), pdf_num(p[1] * scale)).as_bytes());
}
out.extend_from_slice(b"h\n");
}
out.extend_from_slice(b"f\n");
out
}
/// A resource name free in `fonts_res`: `base`, then `base1`, `base2`, … (a page that was
/// OCR'd twice keeps both layers' fonts distinct).
pub(crate) fn unique_font_name(fonts_res: &Dict, base: &str) -> String {
let mut name = String::from(base);
let mut suffix = 0usize;
while fonts_res.contains(name.as_bytes()) {
suffix = suffix.saturating_add(1);
name = format!("{base}{suffix}");
}
name
}
/// The Type3 font dictionary for `(character, CharProc body, advance in em)` glyphs, codes from
/// 1: one CharProc per code, `/Differences` glyph names, real `/Widths`, and a `/ToUnicode` CMap
/// so the text extracts back (this is what makes the output Unicode-correct, not just visible).
pub(crate) fn type3_font_dict(doc: &mut Document, glyphs: &[(char, Vec<u8>, f64)]) -> Result<Dict, EditError> {
let mut charprocs = Dict::new();
let mut differences = vec![Object::Int(1)];
let mut widths = Vec::with_capacity(glyphs.len());
let mut cmap = String::from(
"/CIDInit /ProcSet findresource begin\n12 dict begin\nbegincmap\n/CMapType 2 def\n1 begincodespacerange\n<01> <FF>\nendcodespacerange\n",
);
cmap.push_str(&format!("{} beginbfchar\n", glyphs.len()));
let mut codes = Vec::with_capacity(glyphs.len());
for (i, (ch, path, width)) in glyphs.iter().enumerate() {
let code = u8::try_from(i + 1).map_err(|_| EditError::Invalid("too many unique characters for one font".into()))?;
let glyph_name = format!("g{code:02X}");
let mut pd = Dict::new();
pd.set(b"Length".to_vec(), path.len() as i64);
let proc_ref = doc.add(Object::Stream(Stream::from_raw(pd, path.clone())));
charprocs.set(glyph_name.as_bytes().to_vec(), Object::Ref(proc_ref));
differences.push(Object::name(&glyph_name));
widths.push(Object::Real((width * 1000.0).round()));
cmap.push_str(&format!("<{code:02X}> <{}>\n", unicode_hex(*ch)));
codes.push((*ch, code, *width));
}
cmap.push_str("endbfchar\nendcmap\nCMapName currentdict /CMap defineresource pop\nend\nend\n");
let mut cmap_dict = Dict::new();
cmap_dict.set(b"Length".to_vec(), cmap.len() as i64);
let cmap_ref = doc.add(Object::Stream(Stream::from_raw(cmap_dict, cmap.into_bytes())));
let mut encoding = Dict::new();
encoding.set(b"Type".to_vec(), Object::name("Encoding"));
encoding.set(b"Differences".to_vec(), Object::Array(differences));
let mut font = Dict::new();
font.set(b"Type".to_vec(), Object::name("Font"));
font.set(b"Subtype".to_vec(), Object::name("Type3"));
font.set(b"Name".to_vec(), Object::name("PCFallback"));
font.set(b"FontBBox".to_vec(), Object::Array(vec![Object::Int(0), Object::Int(-300), Object::Int(1000), Object::Int(1000)]));
font.set(
b"FontMatrix".to_vec(),
Object::Array(vec![Object::Real(0.001), Object::Int(0), Object::Int(0), Object::Real(0.001), Object::Int(0), Object::Int(0)]),
);
font.set(b"FirstChar".to_vec(), Object::Int(1));
font.set(b"LastChar".to_vec(), Object::Int(codes.len() as i64));
font.set(b"Widths".to_vec(), Object::Array(widths));
font.set(b"Encoding".to_vec(), Object::Dict(encoding));
font.set(b"CharProcs".to_vec(), Object::Dict(charprocs));
font.set(b"ToUnicode".to_vec(), Object::Ref(cmap_ref));
Ok(font)
}
/// The runs that show `text` in the fallback: consecutive characters in one font form one run, so
/// a font is switched only where the text moves to another font. `None` if a character has no code.
pub(super) fn type3_encode(fallback: &Type3Fallback, text: &str) -> Option<Vec<(String, Vec<u8>)>> {
crates/engine/src/actions.rs
+4 -4
View File
@@ -197,14 +197,14 @@ fn apply_step(s: &mut Session, id: DocId, step: &Step, log: &mut Vec<String>) ->
return Ok(None);
}
};
// Steps that find nothing to do (no comments to flatten, …) aren't failures. A removal step
// that didn't remove is: the file would be passed on as cleaned with the data still in it.
// A step that finds nothing to do is not a failure. Any other error fails the file, so nothing is
// published for it: the file would otherwise be passed on without the step it was asked to apply.
let removes = matches!(step, Step::RemoveHiddenInformation | Step::Sanitize);
match s.apply(id, edit) {
Ok(()) => log.push(step.label().to_string()),
Err(EditError::Redact(pdfcraft_redact::RedactError::NothingToApply)) if removes => log.push(format!("{}: nothing to remove", step.label())),
Err(e) if removes => return Err(e),
Err(e) => log.push(format!("{}: {e}", step.label())),
Err(EditError::Redact(pdfcraft_redact::RedactError::NothingToApply)) => log.push(format!("{}: nothing to do", step.label())),
Err(e) => return Err(e),
}
Ok(None)
}
crates/engine/src/catalog.rs
+1 -1
View File
@@ -267,7 +267,7 @@ pub static TOOL_GROUPS: &[ToolGroup] = &[
item("In multiple files", "files", "ocr.recognize_batch", Ready),
item("Enhance scanned file", "sparkles", "ocr.enhance", Ready),
item("Enhance camera image", "camera", "ocr.enhance_camera", Ready),
item("Correct recognized text", "text-select", "ocr.correct", Planned("M10")),
item("Correct recognized text", "text-select", "ocr.correct", Ready),
],
}],
},
crates/engine/src/commands.rs
+1
View File
@@ -361,6 +361,7 @@ pub const COMMANDS: &[CommandSpec] = &[
c("ocr.enhance_camera", "Enhance camera image", None, None, Modification, "camera"),
c("ocr.enhance", "Enhance scanned file…", None, None, Modification, "sparkles"),
c("ocr.recognize_batch", "Recognize text in multiple files…", None, None, Nothing, "files"),
c("ocr.correct", "Correct recognized text…", None, None, Document, "text-select"),
c("a11y.check", "Check for accessibility…", None, None, Document, "accessibility"),
c("a11y.find_untagged", "Find untagged content", None, None, Document, "search"),
c("a11y.autotag", "Automatically tag PDF", None, None, Document, "tag"),
crates/engine/src/compare.rs
+21 -2
View File
@@ -63,6 +63,15 @@ impl Session {
Ok(pdfcraft_compare::compare(&a.words(), &b.words()))
}
/// [`Session::compare`], keeping the changes on `pages` of `new` (0-based; empty: every page).
pub fn compare_pages(&self, old: DocId, new: DocId, pages: &[usize]) -> Result<Comparison, EditError> {
let mut c = self.compare(old, new)?;
if !pages.is_empty() {
c.changes.retain(|ch| pages.iter().any(|page| (ch.new.page..=ch.new.last_page).contains(page)));
}
Ok(c)
}
/// Visual compare: regions where page n of `new` looks different from page n of `old`
/// (rendered at `dpi`), as (page, user-space box in `new`).
pub fn compare_visual(&self, old: DocId, new: DocId, dpi: f32) -> Result<Vec<(usize, [f64; 4])>, EditError> {
@@ -95,7 +104,12 @@ impl Session {
/// The compare report as a new PDF (not opened).
pub fn compare_report(&self, old: DocId, new: DocId) -> Result<Arc<Vec<u8>>, EditError> {
let c = self.compare(old, new)?;
self.compare_report_of(old, new, &[])
}
/// [`Session::compare_report`] of the changes on `pages` ([`Session::compare_pages`]).
pub fn compare_report_of(&self, old: DocId, new: DocId, pages: &[usize]) -> Result<Arc<Vec<u8>>, EditError> {
let c = self.compare_pages(old, new, pages)?;
let name = |id| self.get(id).map(|d| d.name.clone()).unwrap_or_default();
self.create_report("Compare Report", &pdfcraft_compare::report(&c, &name(old), &name(new)))
}
@@ -104,7 +118,12 @@ impl Session {
/// (blue, green) and a note where text was deleted (red), authored "Compare". One undoable
/// step; returns how many comments were added.
pub fn mark_differences(&mut self, old: DocId, new: DocId) -> Result<usize, EditError> {
let c = self.compare(old, new)?;
self.mark_differences_of(old, new, &[])
}
/// [`Session::mark_differences`] of the changes on `pages` ([`Session::compare_pages`]).
pub fn mark_differences_of(&mut self, old: DocId, new: DocId, pages: &[usize]) -> Result<usize, EditError> {
let c = self.compare_pages(old, new, pages)?;
let mut edits = Vec::new();
for ch in &c.changes {
let color = colour(ch.kind);
crates/engine/src/document.rs
+1 -1
View File
@@ -539,7 +539,7 @@ impl Document {
_ => "Attachments only",
};
let pending = editor.cos.encryption_changed();
let permissions = if pending { pdfcraft_cos::Permissions { bits: h.permissions().bits, owner: false } } else { h.permissions() };
let permissions = if pending { pdfcraft_cos::Permissions { owner: false, ..h.permissions() } } else { h.permissions() };
Some(SecuritySummary { method: method.into(), owner: h.auth() == pdfcraft_cos::Auth::Owner && !pending, permissions, pending })
}
crates/engine/src/edit/run.rs
+1 -1
View File
@@ -718,7 +718,7 @@ pub(crate) fn run_edit(doc: &mut pdfcraft_cos::Document, edit: &Edit, cx: &mut E
Edit::RemoveStandardId { standard } => pdfcraft_organize::remove_standard_id(doc, *standard)?,
Edit::SetDocumentScript { name, script } => pdfcraft_forms::set_document_script(doc, name, script.as_deref())?,
Edit::AddOcrText { page, words } => {
pdfcraft_edit::stamp(doc, *page, "OCR", pdfcraft_ocr::text_layer(words))?;
crate::ocr::layer::add_text(doc, *page, words)?;
}
Edit::EnhanceScan { images } => crate::scan_clean::write_back(doc, images)?,
Edit::EditPageImage { page, index, change } => run_edit_page_image(doc, *page, *index, change)?,
crates/engine/src/ocr.rs
+1741 -68
View File
File diff suppressed because it is too large. Load diff
crates/engine/src/ocr/dictionary.rs
+80
View File
@@ -0,0 +1,80 @@
//! The UI dictionary format: one case-insensitive word per UTF-8 line.
use std::collections::BTreeSet;
pub const MAX_BYTES: usize = 4 * 1024 * 1024;
const MAX_WORDS: usize = 65536;
fn word(value: &str) -> Result<String, String> {
let value = value.trim();
if value.is_empty() || value.len() > 1024 || value.chars().any(char::is_control) {
return Err("dictionary words must be nonempty single lines of at most 1024 bytes".into());
}
let normalized = value.to_lowercase();
if normalized.len() > 1024 {
return Err("normalized dictionary words must be at most 1024 bytes".into());
}
Ok(normalized)
}
pub fn words(text: &str) -> Result<BTreeSet<String>, String> {
if text.len() > MAX_BYTES {
return Err("OCR dictionary exceeds 4 MiB".into());
}
let mut words = BTreeSet::new();
for line in text.lines().filter(|line| !line.trim().is_empty()) {
words.insert(word(line)?);
if words.len() > MAX_WORDS {
return Err("OCR dictionary exceeds 65536 words".into());
}
}
Ok(words)
}
/// Normalize, update and bound the complete dictionary before the caller writes it.
pub fn update(text: &str, add: &[&str], remove: &[&str]) -> Result<String, String> {
if add.len() > MAX_WORDS || remove.len() > MAX_WORDS {
return Err("OCR dictionary exceeds 65536 words".into());
}
let mut entries = words(text)?;
for value in remove {
entries.remove(&word(value)?);
}
for value in add {
entries.insert(word(value)?);
if entries.len() > MAX_WORDS {
return Err("OCR dictionary exceeds 65536 words".into());
}
}
let mut out = String::new();
for entry in entries {
if out.len().checked_add(entry.len()).and_then(|n| n.checked_add(1)).is_none_or(|n| n > MAX_BYTES) {
return Err("OCR dictionary exceeds 4 MiB".into());
}
out.push_str(&entry);
out.push('\n');
}
Ok(out)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn dictionary_updates_round_trip_and_refuse_expanding_normalization() {
let text = update("Quartz\n日本語\n", &["QUARTZ", "extra"], &["日本語"]).unwrap();
assert_eq!(text, "extra\nquartz\n");
assert_eq!(words(&text).unwrap(), BTreeSet::from(["extra".into(), "quartz".into()]));
assert!(update("original\n", &[&"İ".repeat(512)], &[]).unwrap_err().contains("normalized"));
assert!(update("original\n", &["two\nlines"], &[]).is_err());
}
#[test]
fn dictionary_limits_cover_existing_and_added_entries() {
let full: String = (0..65536).map(|i| format!("word{i}\n")).collect();
assert_eq!(words(&full).unwrap().len(), 65536);
assert!(update(&full, &["extra"], &[]).is_err());
assert_eq!(update(&full, &["extra"], &["word0"]).unwrap().lines().count(), 65536);
assert!(words(&"a".repeat(MAX_BYTES + 1)).is_err());
}
}
crates/engine/src/ocr/layer.rs
+145
View File
@@ -0,0 +1,145 @@
//! Unicode search text has no visible glyphs. A generated Type 3 font supplies advance widths
//! and a ToUnicode map, with empty character programs; no display font is needed or embedded.
//! Adapted from the searchable-layer implementation in #433 (32e09fbe).
use std::collections::BTreeMap;
use pdfcraft_cos::{Dict, Document, Object, Stream};
use pdfcraft_edit::EditError;
use pdfcraft_ocr::PlacedWord;
/// Refuse oversized text or invalid geometry before cloning words or allocating fonts.
/// These limits apply to the complete input, including all editable export pages.
pub(crate) fn validate_words<'a>(words: impl IntoIterator<Item = &'a PlacedWord>) -> Result<usize, EditError> {
let mut count = 0usize;
let mut length = 0usize;
for w in words {
count += 1;
length = length.checked_add(w.text.len()).ok_or_else(|| EditError::Invalid("OCR text layer is too large".into()))?;
if count > 65536 || length > 4 * 1024 * 1024 {
return Err(EditError::Invalid("OCR text layer is too large".into()));
}
if !w.origin.iter().chain(&w.across).chain(&w.up).all(|v| v.is_finite())
|| !w.across[0].hypot(w.across[1]).is_finite()
|| !w.up[0].hypot(w.up[1]).is_finite()
|| (0..2).any(|i| !(w.origin[i] + 0.2 * w.up[i]).is_finite() || !(w.origin[i] + w.across[i] + 0.2 * w.up[i]).is_finite())
{
return Err(EditError::Invalid("OCR text coordinates must be finite".into()));
}
}
Ok(count)
}
pub(crate) fn add_text(doc: &mut Document, page: usize, words: &[PlacedWord]) -> Result<(), EditError> {
validate_words(words)?;
// Keep the existing Latin layout, including Helvetica's proportional advance widths.
if words.iter().all(|w| {
w.text
.chars()
.all(|c| matches!(c, '\u{20}'..='\u{7e}' | '\u{a0}'..='\u{ff}' | '€' | '‚' | '„' | '…' | '‘' | '’' | '“' | '”' | '•' | '–' | '—' | '™'))
}) {
pdfcraft_edit::stamp(doc, page, "OCR", pdfcraft_ocr::text_layer(words, &pdfcraft_fonts::TextPlan::default()))?;
return Ok(());
}
let p = pdfcraft_model::pages(doc).get(page).cloned().ok_or_else(|| EditError::Invalid("OCR page does not exist".into()))?;
let mut res = p.dict.get(b"Resources").map(|r| doc.resolve(r)).and_then(|r| r.as_dict().cloned()).unwrap_or_default();
let mut fonts = res.get(b"Font").map(|f| doc.resolve(f)).and_then(|f| f.as_dict().cloned()).unwrap_or_default();
let chars: Vec<char> = words.iter().flat_map(|w| w.text.chars()).collect::<std::collections::BTreeSet<_>>().into_iter().collect();
// Match the extractor's em box (-200..800). Baselines below are offset by 0.2 em,
// so search/selection rectangles span the detected image box.
let blank = doc.add(Object::Stream(Stream::from_raw(Dict::new(), b"1000 0 0 -200 1000 800 d1\n".to_vec())));
let mut codes = BTreeMap::new();
let mut names = Vec::new();
for (index, chunk) in chars.chunks(255).enumerate() {
let mut name = format!("PCOcr{index}");
while fonts.contains(name.as_bytes()) {
name.push('x');
}
let mut differences = vec![Object::Int(1)];
let mut procs = Dict::new();
let mut cmap = String::from(
"/CIDInit /ProcSet findresource begin\n12 dict begin\nbegincmap\n/CIDSystemInfo << /Registry (PdfCraft) /Ordering (Unicode) /Supplement 0 >> def\n/CMapName /PdfCraftOCR def\n/CMapType 2 def\n1 begincodespacerange\n<01> <FF>\nendcodespacerange\n",
);
for (batch, part) in chunk.chunks(100).enumerate() {
cmap.push_str(&format!("{} beginbfchar\n", part.len()));
for (i, ch) in part.iter().enumerate() {
let code = (batch * 100 + i + 1) as u8;
let glyph = format!("g{code:02X}");
differences.push(Object::name(&glyph));
procs.set(glyph.into_bytes(), Object::Ref(blank));
let mut units = [0u16; 2];
let hex: String = ch.encode_utf16(&mut units).iter().map(|u| format!("{u:04X}")).collect();
cmap.push_str(&format!("<{code:02X}> <{hex}>\n"));
codes.insert(*ch, (index, code));
}
cmap.push_str("endbfchar\n");
}
cmap.push_str("endcmap\nCMapName currentdict /CMap defineresource pop\nend\nend\n");
let map = doc.add(Object::Stream(Stream::flate(Dict::new(), cmap.as_bytes())));
let mut encoding = Dict::new();
encoding.set(b"Type".to_vec(), Object::name("Encoding"));
encoding.set(b"Differences".to_vec(), Object::Array(differences));
let mut font = Dict::new();
font.set(b"Type".to_vec(), Object::name("Font"));
font.set(b"Subtype".to_vec(), Object::name("Type3"));
font.set(b"FontBBox".to_vec(), Object::Array(vec![0.into(), (-200).into(), 1000.into(), 800.into()]));
font.set(b"FontMatrix".to_vec(), Object::Array(vec![Object::Real(0.001), 0.into(), 0.into(), Object::Real(0.001), 0.into(), 0.into()]));
font.set(b"FirstChar".to_vec(), Object::Int(1));
font.set(b"LastChar".to_vec(), Object::Int(chunk.len() as i64));
font.set(b"Widths".to_vec(), Object::Array(vec![Object::Int(1000); chunk.len()]));
font.set(b"Encoding".to_vec(), Object::Dict(encoding));
font.set(b"CharProcs".to_vec(), Object::Dict(procs));
font.set(b"Resources".to_vec(), Object::Dict(Dict::new()));
font.set(b"ToUnicode".to_vec(), Object::Ref(map));
fonts.set(name.as_bytes().to_vec(), Object::Ref(doc.add(Object::Dict(font))));
names.push(name);
}
res.set(b"Font".to_vec(), Object::Dict(fonts));
doc.update_dict(p.obj, |d| d.set(b"Resources".to_vec(), Object::Dict(res)))?;
// `3 Tr` and the font are graphics state, which `ET` doesn't reset: like
// `pdfcraft_ocr::text_layer` (#3029), the text is drawn inside `q … Q` so the invisible
// rendering mode cannot leak into later content streams on the page.
let mut content = String::from("/OCR BMC\nq\nBT\n3 Tr\n");
for w in words {
let count = w.text.chars().count();
if count == 0 || w.up[0].hypot(w.up[1]) <= 0.0 || w.across[0].hypot(w.across[1]) <= 0.0 {
continue;
}
let mut start = 0;
let mut run = Vec::new();
let mut font_index = None;
let emit = |content: &mut String, font: usize, start: usize, run: &[u8]| {
let n = count as f64;
let at = start as f64 / n;
let hex: String = run.iter().map(|b| format!("{b:02X}")).collect();
content.push_str(&format!(
"/{} 1 Tf\n{} {} {} {} {} {} Tm <{hex}> Tj\n",
names[font],
w.across[0] / n,
w.across[1] / n,
w.up[0],
w.up[1],
w.origin[0] + at * w.across[0] + 0.2 * w.up[0],
w.origin[1] + at * w.across[1] + 0.2 * w.up[1]
));
};
for (i, ch) in w.text.chars().enumerate() {
let Some(&(font, code)) = codes.get(&ch) else { continue };
if let Some(previous) = font_index
&& previous != font
{
emit(&mut content, previous, start, &run);
run.clear();
start = i;
}
font_index = Some(font);
run.push(code);
}
if let Some(font) = font_index {
emit(&mut content, font, start, &run);
}
}
content.push_str("ET\nQ\nEMC\n");
pdfcraft_edit::stamp(doc, page, "OCR", content.into_bytes())?;
Ok(())
}
crates/engine/src/scan_fields.rs
+6 -2
View File
@@ -186,10 +186,14 @@ impl Session {
if !ocr::available() {
return Vec::new();
}
let (Ok(engine), Some(job)) = (ocr::engine(), self.ocr_job(id, &[page], ocr::OcrSettings::default())) else {
let settings = ocr::OcrSettings::default();
let Some(job) = self.ocr_job(id, &[page], settings.clone()) else {
return Vec::new();
};
job.run(&engine, |_, _| true)
let Ok(recognizers) = ocr::recognizers(&settings) else {
return Vec::new();
};
job.run(&recognizers, |_, _| true)
.into_iter()
.flat_map(|p| p.words)
.filter_map(|w| {
crates/engine/src/session/files.rs
+1 -1
View File
@@ -397,7 +397,7 @@ impl Session {
/// A new file (Combine, Extract, Split, Create) as bytes, checked like a save before anyone
/// writes it (#1698, #2166): it reopens, and shows the pages it was built from.
fn write_new(&self, doc: &pdfcraft_cos::Document) -> Result<Arc<Vec<u8>>, EditError> {
pub(crate) fn write_new(&self, doc: &pdfcraft_cos::Document) -> Result<Arc<Vec<u8>>, EditError> {
let bytes = self.write_unchecked(doc)?;
let expected = pdfcraft_organize::page_count(doc)?;
pdfcraft_cos::Document::open(bytes.clone()).map_err(|e| EditError::Reopen(e.to_string()))?;
crates/engine/src/tests.rs
+15 -8
View File
@@ -623,12 +623,12 @@ fn a_3d_model_needs_the_comments_permission_and_changes_the_comments() {
activation: Activation3d::default(),
name: String::new(),
};
let p = pdfcraft_cos::Permissions { bits: 0, owner: false };
let p = pdfcraft_cos::Permissions { bits: 0, owner: false, revision: 3 };
assert_eq!(check_permission(&edit, &p), Err(EditError::NotPermitted("comments")));
assert_eq!(crate::history::scope_of(&edit), crate::history::Scope::Comments);
}
fn page_texts(s: &Session, id: DocId) -> Vec<String> {
pub(crate) fn page_texts(s: &Session, id: DocId) -> Vec<String> {
let doc = s.get(id).unwrap();
let config = pdfcraft_render::RenderConfig { password: doc.password.as_deref().map(Arc::from), ..Default::default() };
let mut r = pdfcraft_render::PageRenderer::new(doc.bytes.clone(), config);
@@ -5841,6 +5841,10 @@ fn comparing_two_versions_of_a_document() {
let got: Vec<(compare::Kind, &str, &str)> = c.changes.iter().map(|x| (x.kind, x.old.text.as_str(), x.new.text.as_str())).collect();
assert_eq!(got, [(compare::Kind::Replaced, "Monday.", "Tuesday."), (compare::Kind::Replaced, "days.", "days of invoice.")], "{got:?}");
assert!(s.compare(old, old).unwrap().identical());
// limited to pages of the new file: both changes are on the first
assert_eq!(s.compare_pages(old, new, &[]).unwrap().changes.len(), 2);
assert_eq!(s.compare_pages(old, new, &[0]).unwrap().changes.len(), 2);
assert!(s.compare_pages(old, new, &[1]).unwrap().identical());
let report = s.compare_report(old, new).unwrap();
let rid = s.open("report.pdf", None, report, None).unwrap();
@@ -6048,7 +6052,7 @@ fn measurement_edits_are_transactional_and_honour_annotation_permissions() {
assert!(session.get(id).unwrap().measurements().unwrap().measurements.is_empty());
session.redo(id).unwrap();
assert_eq!(session.get(id).unwrap().bytes, before);
let p = pdfcraft_cos::Permissions { bits: 0, owner: false };
let p = pdfcraft_cos::Permissions { bits: 0, owner: false, revision: 3 };
assert_eq!(check_permission(&Edit::AddMeasurement(m), &p), Err(EditError::NotPermitted("comments")));
assert_eq!(
check_permission(
@@ -7322,12 +7326,13 @@ fn datasets_streams(bytes: &Arc<Vec<u8>>) -> usize {
/// and noted.
#[test]
fn hostile_xfa_open_scripts_are_merged_capped_kept_from_printing_and_noted() {
// On open: a calculate setting its own value 50 000 times, an initialize script showing a
// thousand messages and asking to print, open a link and save.
// On open: a calculate setting its own value 20 000 times (50 000 sat right at the 1 s
// unasked-script timeout and abandoned whenever the test machine was loaded), an
// initialize script showing a thousand messages and asking to print, open a link and save.
let tpl = pdfcraft_xfa::fixtures::scripted_template()
.replace(
r#"<calculate><script contentType="application/x-javascript">qty.rawValue * price.rawValue</script></calculate>"#,
r#"<calculate><script contentType="application/x-javascript">for (var i = 0; i &lt; 50000; i++) this.rawValue = i;</script></calculate>"#,
r#"<calculate><script contentType="application/x-javascript">for (var i = 0; i &lt; 20000; i++) this.rawValue = i;</script></calculate>"#,
)
.replace(
r#"if (qty.rawValue === null) qty.rawValue = 2;"#,
@@ -7337,9 +7342,11 @@ fn hostile_xfa_open_scripts_are_merged_capped_kept_from_printing_and_noted() {
let mut s = Session::new().with_clock(|| 1_700_000_000);
let started = std::time::Instant::now();
let id = s.open("hostile.pdf", None, bytes, None).expect("opens");
assert!(started.elapsed() < std::time::Duration::from_secs(5), "{:?}", started.elapsed());
// 30 s and not 5 s: the run is wall-clock timed and a loaded test machine (all suites in
// parallel) can take several seconds for the 20 000 iterations; a real hang still fails.
assert!(started.elapsed() < std::time::Duration::from_secs(30), "{:?}", started.elapsed());
let doc = s.get(id).unwrap();
assert_eq!(doc.form.iter().find(|f| f.name == "total").unwrap().value, vec!["49999".to_string()]);
assert_eq!(doc.form.iter().find(|f| f.name == "total").unwrap().value, vec!["19999".to_string()]);
// One datasets stream for the scripts' revision, not one per value set.
assert!(datasets_streams(&doc.bytes) <= 3, "{} datasets streams", datasets_streams(&doc.bytes));
// What opening changed is noted with the form's other XFA notes.
crates/engine/tests/it/ocr_edit.rs
+17 -3
View File
@@ -12,7 +12,14 @@ const WORD: [u32; 4] = [40, 40, 100, 60];
/// A 200 × 100 pt "scan" on tinted paper, with OCR text over its word.
fn scanned() -> (Session, pdfcraft_engine::DocId) {
scanned_with(vec![PlacedWord { text: "John".into(), origin: [40.0, 40.0], across: [60.0, 0.0], up: [0.0, 20.0] }])
scanned_with(vec![PlacedWord {
text: "John".into(),
origin: [40.0, 40.0],
across: [60.0, 0.0],
up: [0.0, 20.0],
confidence: None,
source: "ocrs".into(),
}])
}
fn scanned_with(words: Vec<PlacedWord>) -> (Session, pdfcraft_engine::DocId) {
@@ -86,11 +93,18 @@ fn an_explicit_colour_is_kept_for_the_new_text() {
/// descender line when it has descenders, up to the cap height.
fn scanned_word(text: &str, x0: f64, x1: f64, baseline: f64, size: f64) -> PlacedWord {
let low = if text.chars().any(|c| "gjpqy".contains(c)) { 0.21 * size } else { 0.0 };
PlacedWord { text: text.into(), origin: [x0, baseline - low], across: [x1 - x0, 0.0], up: [0.0, 0.718 * size + low] }
PlacedWord {
text: text.into(),
origin: [x0, baseline - low],
across: [x1 - x0, 0.0],
up: [0.0, 0.718 * size + low],
confidence: None,
source: "ocrs".into(),
}
}
fn word(text: &str, x0: f64, x1: f64, baseline: f64) -> PlacedWord {
PlacedWord { text: text.into(), origin: [x0, baseline], across: [x1 - x0, 0.0], up: [0.0, 10.0] }
PlacedWord { text: text.into(), origin: [x0, baseline], across: [x1 - x0, 0.0], up: [0.0, 10.0], confidence: None, source: "ocrs".into() }
}
/// OCR words keep the scan's layout: a phrase is one paragraph, a value in the next table cell and
crates/fonts/README.md
+5
View File
@@ -59,7 +59,12 @@ set). Without that variable `CRAFT_FONTS` is empty and everything below copes:
and shaped Thai Add Text content. Harfrust positions stacked vowel/tone marks and decomposes
Sara Am; cluster ToUnicode entries preserve the original text. The shared Type 3 writer
embeds outlines, so reopening does not depend on installed fonts.
- `craft_glyph(face, ch)` and `face_has(bytes, ch)`: bounded outlines and glyph coverage for any bundled face.
- `SHIPPORI_MINCHO`: Shippori Mincho's bytes, or `None`.
- `TextPlan::for_words`: how the OCR output writes text beyond WinAnsi — per word the first
face (document face first) whose glyphs cover it, planned once per page into Type 3 fonts
(codes, advances, `?` stand-ins) that `pdfcraft-edit` registers and the `pdfcraft-ocr` layers
draw with; characters no face has are counted (`missing`) so the output reports them.
Generated Japanese Type 3 fonts record the fallback face's family and style in an indirect
`FontDescriptor`. `pdf::Metrics::base_font` uses its `FontName` when the Type 3 dictionary has
crates/fonts/src/lib.rs
+14 -1
View File
@@ -14,6 +14,7 @@ pub mod embedding;
mod encodings;
pub mod fallback;
pub mod pdf;
pub mod plan;
mod script;
mod shaping;
pub mod source_shaping;
@@ -24,8 +25,10 @@ pub use craft::{
document_japanese_font_for_style, document_japanese_fonts_for_style, document_latin_fonts, document_thai_font, ui_arabic_fonts, ui_chinese_fonts,
ui_cjk_fonts, ui_devanagari_fonts, ui_hebrew_fonts, ui_japanese_fonts, ui_telugu_fonts, ui_thai_fonts,
};
pub use plan::{MAX_PLAN_GLYPHS, MAX_TYPE3_GLYPHS, PlannedType3, TextPlan};
pub use script::{
GlyphError, GlyphOutline, INTER_REGULAR, MAX_SIGNATURE_CHARS, ScriptOutline, japanese_glyph, japanese_glyph_from, script_missing, script_outline,
GlyphError, GlyphOutline, INTER_REGULAR, MAX_SIGNATURE_CHARS, ScriptOutline, craft_glyph, face_has, japanese_glyph, japanese_glyph_from,
script_missing, script_outline,
};
pub use shaping::{
ShapedCluster, ShapedScript, arabic_glyph, arabic_has, hebrew_has, shape_arabic, shape_hebrew, shape_thai, shaped_glyph, thai_has,
@@ -273,6 +276,16 @@ pub fn set_undrawn_mark(d: &mut pdfcraft_cos::Dict, c: Option<char>) {
}
}
/// The byte used by Unicode text plans, from the shared WinAnsi table (tabs become spaces).
pub fn win_ansi_char(c: char) -> Option<u8> {
if c == '\t' { Some(b' ') } else { win_ansi_byte(c) }
}
/// Whether every character can be represented without substitution in WinAnsi.
pub fn fits_win_ansi(s: &str) -> bool {
s.chars().all(win_ansi_encodable)
}
/// Bytes as a PDF literal string, `(` … `)`, with delimiters escaped.
pub fn literal(bytes: &[u8]) -> Vec<u8> {
let mut out = Vec::with_capacity(bytes.len() + 2);
crates/fonts/src/plan.rs
+338
View File
@@ -0,0 +1,338 @@
//! Planning visible Unicode text output for OCR: which characters the standard WinAnsi
//! fonts show as is, which need a Type3 font drawn from the craft-fonts faces, and which have
//! no glyph anywhere (written as `?` — and counted, never silently dropped).
//!
//! The plan is a pure function of the words ([`TextPlan::for_words`]). The content generators
//! (`pdfcraft_ocr::text_layer` / `visible_text_layer`) read the per-character codes and
//! advances from it, and whoever registers the fonts (`pdfcraft_edit::add_type3_fonts`) fills
//! in each face's resource name and object. The build must work without craft-fonts too: there
//! [`CRAFT_FONTS`] is empty, nothing can be planned, and every non-WinAnsi character is
//! reported as a substitution instead.
use std::collections::HashMap;
use pdfcraft_cos::ObjRef;
use crate::{CraftFont, GlyphOutline, craft_glyph, document_japanese_font, face_has, fits_win_ansi, win_ansi_char};
/// Unique characters per Type3 font: the same bound the text-editing fallback uses, so one
/// hostile word cannot balloon into thousands of glyph streams. A face with more characters
/// simply gets another font.
pub const MAX_TYPE3_GLYPHS: usize = 240;
/// Planned glyphs per plan in total (10 full fonts). A page's recognition cannot realistically
/// exceed it; hostile input beyond it degrades to `?` substitutions, which are counted.
pub const MAX_PLAN_GLYPHS: usize = 2400;
/// One planned Type3 font: the glyphs of one craft-fonts face, one character per code
/// (`code = position + 1`, as in the text-editing fallback). `name` and `object` are filled in
/// when the font is added to a page's resources; until then the content generators fall back to
/// the WinAnsi path for the words that reference it.
pub struct PlannedType3 {
/// The face family the glyphs are drawn from ("Shippori Mincho").
pub family: &'static str,
/// (character, outline with its advance) per code, in code order.
pub glyphs: Vec<(char, GlyphOutline)>,
/// The resource name the content references (`PCOcr`, `PCOcr1`, …); empty until registered.
pub name: String,
/// The built font object, shared by every page that uses the font; set at registration.
pub object: Option<ObjRef>,
}
impl PlannedType3 {
fn new(family: &'static str) -> Self {
PlannedType3 { family, glyphs: Vec::new(), name: String::new(), object: None }
}
fn push(&mut self, ch: char, glyph: GlyphOutline) -> u8 {
self.glyphs.push((ch, glyph));
self.glyphs.len() as u8
}
}
/// How the text layers write a plan's characters: each planned character is a
/// `(font index, code)` pair in its font's encoding; a character no face had a glyph for is
/// either the font's `?` code or absent from the run (both counted in [`TextPlan::missing`]).
#[derive(Default)]
pub struct TextPlan {
/// One planned font per face chunk, in first-use order.
pub fonts: Vec<PlannedType3>,
/// Per input word: `(font, code)` per character; empty for a word the standard WinAnsi
/// fonts write as is (or for which nothing could be planned).
words: Vec<Vec<(usize, u8)>>,
/// character → where its glyph lives, across all fonts (codes are font-local).
index: HashMap<char, (usize, u8)>,
/// Characters with no code in their word's font: they appear as `?` (or, when not even a
/// `?` glyph exists, are left out) and are reported, never silently lost.
missing: usize,
}
impl TextPlan {
/// Plan the output for `words`, in order: characters the WinAnsi fonts can't show get codes
/// in Type3 fonts built from the craft-fonts faces (the document face preferred, then the
/// rest of the build input in manifest order); a word whose face can't show a character
/// substitutes its `?`. Faces are chosen per word, so a word of one script stays one face;
/// characters already planned for an earlier word keep their code. Built without
/// craft-fonts, the plan reports every non-WinAnsi character as a substitution.
pub fn for_words<'a>(words: impl IntoIterator<Item = &'a str>) -> TextPlan {
let faces = document_faces();
let mut coverage = Coverage::default();
let mut plan = TextPlan::default();
for word in words {
if fits_win_ansi(word) {
// The WinAnsi path substitutes '?' behind the caller's back; count it.
plan.missing += word.chars().filter(|&c| win_ansi_char(c).is_none()).count();
plan.words.push(Vec::new());
continue;
}
let Some(face) = choose_face(&faces, word, &mut coverage) else {
// No face for this word's script: the WinAnsi path writes '?' for what it
// can't encode.
plan.missing += word.chars().filter(|&c| win_ansi_char(c).is_none()).count();
plan.words.push(Vec::new());
continue;
};
let mut enc: Vec<(usize, u8)> = Vec::with_capacity(word.len());
let mut run: Option<usize> = None;
for ch in word.chars() {
if let Some(&where_) = plan.index.get(&ch) {
enc.push(where_);
run = Some(where_.0);
continue;
}
if let Ok(glyph) = craft_glyph(face, ch)
&& plan.planned_glyphs() < MAX_PLAN_GLYPHS
&& let Some((fi, code)) = plan.add_glyph(face, ch, glyph)
{
plan.index.insert(ch, (fi, code));
enc.push((fi, code));
run = Some(fi);
continue;
}
// The face has no glyph for the character (or the plan is full): substitute
// the font's '?', in the font the surrounding run is set in.
if let Some((fi, code)) = plan.question_mark(face, run.filter(|&f| plan.fonts[f].family == face.family)) {
enc.push((fi, code));
run = Some(fi);
}
plan.missing += 1;
}
plan.words.push(enc);
}
plan
}
/// The encoding of word `i` (parallel to the slice [`Self::for_words`] planned): a list of
/// `(font, code)` per character, or an empty slice for a word the WinAnsi fonts write.
/// Out-of-range indexes degrade to the empty slice rather than panicking — callers pass the
/// same word list they planned with.
pub fn encoding(&self, word: usize) -> &[(usize, u8)] {
self.words.get(word).map(Vec::as_slice).unwrap_or_default()
}
/// How many characters no font could show (they appear as `?`, or are left out when the
/// word's face has no `?` glyph either).
pub fn missing(&self) -> usize {
self.missing
}
/// The advance of one planned code, in em units (the Type3 glyphs' own widths). `0.0` for
/// anything out of range.
pub fn advance(&self, font: usize, code: u8) -> f64 {
self.fonts.get(font).and_then(|f| f.glyphs.get(usize::from(code).checked_sub(1)?)).map_or(0.0, |(_, g)| g.width)
}
/// The sum of the advances of a word's planned codes, in em units.
pub fn advance_of(&self, encoding: &[(usize, u8)]) -> f64 {
encoding.iter().map(|&(f, c)| self.advance(f, c)).sum()
}
/// Whether every font the encoding references has been registered (has a resource name).
/// Unregistered plans fall back to the WinAnsi path, the same output as before the
/// Unicode mapping existed.
pub fn registered(&self, encoding: &[(usize, u8)]) -> bool {
encoding.iter().all(|&(f, _)| self.fonts.get(f).is_some_and(|font| !font.name.is_empty()))
}
/// Characters with a planned glyph so far.
fn planned_glyphs(&self) -> usize {
self.fonts.iter().map(|f| f.glyphs.len()).sum()
}
/// Add a glyph to the face's current font (a fresh one when that font is full).
fn add_glyph(&mut self, face: &CraftFont, ch: char, glyph: GlyphOutline) -> Option<(usize, u8)> {
let fi = self.font_of_face(face.family)?;
let code = self.fonts[fi].push(ch, glyph);
Some((fi, code))
}
/// The index of the face's open font, or a fresh one. `None` without craft-fonts (the
/// face itself comes from there, so this cannot happen for a planned face).
fn font_of_face(&mut self, family: &'static str) -> Option<usize> {
if let Some(fi) = self.fonts.iter().rposition(|f| f.family == family && f.glyphs.len() < MAX_TYPE3_GLYPHS) {
return Some(fi);
}
if self.fonts.len() >= MAX_PLAN_GLYPHS / MAX_TYPE3_GLYPHS {
return None;
}
self.fonts.push(PlannedType3::new(family));
Some(self.fonts.len() - 1)
}
/// The code of the font's `?` stand-in, adding the glyph when the font lacks it.
/// `prefer` is the run's current font; `None` without craft-fonts.
fn question_mark(&mut self, face: &CraftFont, prefer: Option<usize>) -> Option<(usize, u8)> {
if let Some(fi) = prefer
&& let Some(code) = self.fonts[fi].glyphs.iter().position(|(c, _)| *c == '?')
{
return Some((fi, code as u8 + 1));
}
let fi = self.font_of_face(face.family)?;
if let Some(code) = self.fonts[fi].glyphs.iter().position(|(c, _)| *c == '?') {
return Some((fi, code as u8 + 1));
}
let glyph = craft_glyph(face, '?').ok()?;
let code = self.fonts[fi].push('?', glyph);
Some((fi, code))
}
}
/// The faces that can draw document text, in preference order: the Japanese document face
/// (serif document text) first, then the rest of the build input in manifest order.
fn document_faces() -> Vec<&'static CraftFont> {
let mut out = Vec::new();
if let Some(d) = document_japanese_font() {
out.push(d);
}
for f in crate::CRAFT_FONTS {
if !out.iter().any(|d| std::ptr::eq(*d, f)) {
out.push(f);
}
}
out
}
/// The face for a word: the first (in preference order) whose glyphs cover every character,
/// else the one covering the most characters (ties go to the earlier face). `None` when no
/// face covers any character of the word.
fn choose_face<'f>(faces: &[&'f CraftFont], word: &str, coverage: &mut Coverage) -> Option<&'f CraftFont> {
if let Some(f) = faces.iter().enumerate().find(|(i, f)| word.chars().all(|c| coverage.has(*i, f, c))).map(|(_, f)| *f) {
return Some(f);
}
faces
.iter()
.enumerate()
.map(|(i, f)| (word.chars().filter(|&c| coverage.has(i, f, c)).count(), std::cmp::Reverse(i), *f))
.max_by(|a, b| (a.0, a.1).cmp(&(b.0, b.1)))
.filter(|(covered, _, _)| *covered > 0)
.map(|(_, _, f)| f)
}
/// Which faces have a glyph for which characters, remembered for one plan: a page of words
/// asks the same (face, character) pairs over and over, and each fresh answer parses the font.
#[derive(Default)]
struct Coverage {
known: std::collections::HashMap<(usize, char), bool>,
lookups: usize,
}
impl Coverage {
/// Whether face number `i` (`face`) has `ch`; asks the font once per pair.
fn has(&mut self, i: usize, face: &CraftFont, ch: char) -> bool {
*self.known.entry((i, ch)).or_insert_with(|| {
self.lookups += 1;
face_has(face.bytes, ch)
})
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Face coverage is asked once per (face, character): repeating words cost no new font
/// lookups.
#[test]
fn face_coverage_is_remembered_per_face_and_character() {
let faces = document_faces();
let Some(face) = faces.first() else {
eprintln!("skipping: built without craft-fonts (set CRAFT_FONTS_DIR to run it)");
return;
};
let mut coverage = Coverage::default();
let first = (0..50).map(|_| choose_face(&faces, "日本語", &mut coverage)).last().flatten();
assert!(first.is_some());
assert!(coverage.lookups <= faces.len() * 3, "{} lookups for 3 characters", coverage.lookups);
assert_eq!(coverage.has(0, face, '日'), face_has(face.bytes, '日'));
}
/// A page's vocabulary: German umlauts are WinAnsi, and a character no bundled face has
/// (here an emoji) is counted wherever it appears.
#[test]
fn win_ansi_words_get_no_encoding_and_count_only_real_losses() {
let plan = TextPlan::for_words(["Zürcher", "café", "size"]);
assert!(plan.words.iter().all(Vec::is_empty), "all WinAnsi: no Type3 fonts needed");
assert!(plan.fonts.is_empty());
assert_eq!(plan.missing(), 0);
let none = TextPlan::for_words(["plain text"]);
assert_eq!(none.missing(), 0);
// U+1F600 has no WinAnsi code and no glyph in any bundled face (with craft-fonts the
// JIS faces even cover the snowman): counted, shown as '?'.
let emoji = TextPlan::for_words(["\u{1f600}!"]);
assert_eq!(emoji.missing(), 1);
}
#[test]
fn without_craft_fonts_every_non_win_ansi_character_is_reported() {
if !crate::CRAFT_FONTS.is_empty() {
eprintln!("built with craft-fonts: the planning tests with faces cover this instead");
return;
}
let plan = TextPlan::for_words(["Привет", "日本", "abc"]);
assert!(plan.words.iter().all(Vec::is_empty));
assert!(plan.fonts.is_empty());
assert_eq!(plan.missing(), 8, "six Cyrillic + two CJK characters, each reported");
}
#[test]
fn encodings_are_aligned_and_total_the_advances() {
if crate::CRAFT_FONTS.is_empty() {
eprintln!("skipping: built without craft-fonts (set CRAFT_FONTS_DIR to run it)");
assert!(TextPlan::for_words(["日本"]).fonts.is_empty());
return;
}
let plan = TextPlan::for_words(["日本語", "語"]);
assert_eq!(plan.encoding(0).len(), 3);
assert_eq!(plan.encoding(1).len(), 1);
// 語 planned once keeps its code (and font) in both words.
let (f, c) = plan.encoding(1)[0];
assert_eq!(plan.encoding(0)[2], (f, c), "the repeated character keeps its code");
assert!(plan.advance(f, c) > 0.0);
assert_eq!(plan.advance(f, 0), 0.0, "codes start at 1");
assert_eq!(plan.encoding(7), &[], "out of range degrades to the WinAnsi path");
assert!(plan.advance_of(plan.encoding(0)) > plan.advance_of(plan.encoding(1)));
}
#[test]
fn hostile_input_stays_bounded() {
// Thousands of DISTINCT characters: the covered ones fill fonts up to the caps, the
// rest are reported substitutions.
let word: String = (0..4000u32).filter_map(|i| char::from_u32(0x4e00 + i)).collect();
let plan = TextPlan::for_words([word.as_str()]);
assert!(plan.planned_glyphs() <= MAX_PLAN_GLYPHS);
assert!(plan.fonts.iter().all(|f| f.glyphs.len() <= MAX_TYPE3_GLYPHS));
assert!(plan.missing() > 0, "characters beyond the caps are reported, never silently dropped");
}
#[test]
fn substitution_and_unknown_scripts_come_out_counted() {
if crate::CRAFT_FONTS.is_empty() {
eprintln!("skipping: built without craft-fonts (set CRAFT_FONTS_DIR to run it)");
return;
}
let plan = TextPlan::for_words(["日本\u{1f4a9}語"]);
// The emoji has no glyph in any bundled face; the kana/kanji do.
assert_eq!(plan.missing(), 1);
assert_eq!(plan.encoding(0).len(), 4, "the emoji still has an encoding: the font's ?");
}
}
crates/fonts/src/script.rs
+16 -1
View File
@@ -134,6 +134,17 @@ impl OutlinePen for Flatten {
}
}
/// Whether the face has a glyph for `ch` (a cheap character-map probe, no outline work).
pub fn face_has(bytes: &[u8], ch: char) -> bool {
FontRef::new(bytes).is_ok_and(|font| font.charmap().map(ch).is_some())
}
/// Return one glyph of `face` (a craft-fonts face), bounded so hostile replacement text cannot
/// allocate unbounded outline data.
pub fn craft_glyph(face: &crate::CraftFont, ch: char) -> Result<GlyphOutline, GlyphError> {
glyph_in_face(face.bytes, ch)
}
/// Return one glyph of the Japanese document face ([`crate::document_japanese_font`], Shippori
/// Mincho from craft-fonts), bounded so hostile replacement text cannot allocate unbounded
/// outline data. [`GlyphError::NoFont`] when the build has no Japanese face.
@@ -145,7 +156,11 @@ pub fn japanese_glyph(ch: char) -> Result<GlyphOutline, GlyphError> {
/// The bounded outline and advance from the explicitly selected craft-fonts face.
/// Uses the same complexity and missing-glyph checks as [`japanese_glyph`].
pub fn japanese_glyph_from(face: &crate::CraftFont, ch: char) -> Result<GlyphOutline, GlyphError> {
let Ok(font) = FontRef::new(face.bytes) else { return Err(GlyphError::Missing) };
glyph_in_face(face.bytes, ch)
}
fn glyph_in_face(bytes: &[u8], ch: char) -> Result<GlyphOutline, GlyphError> {
let Ok(font) = FontRef::new(bytes) else { return Err(GlyphError::Missing) };
let loc = LocationRef::default();
let metrics = font.metrics(Size::unscaled(), loc);
let scale = 1.0 / metrics.units_per_em.max(1) as f64;
crates/js/fuzz/Cargo.toml
+24
View File
@@ -0,0 +1,24 @@
[package]
name = "pdfcraft-js-fuzz"
version = "0.0.0"
publish = false
edition = "2024"
license = "MIT OR Apache-2.0"
[package.metadata]
cargo-fuzz = true
# Detached from the PdfCraft workspace: run with `cargo +nightly fuzz run run_script`
# from crates/js/.
[workspace]
[dependencies]
libfuzzer-sys = "0.4"
pdfcraft-js = { path = ".." }
[[bin]]
name = "run_script"
path = "fuzz_targets/run_script.rs"
test = false
doc = false
bench = false
crates/js/fuzz/fuzz_targets/run_script.rs
+26
View File
@@ -0,0 +1,26 @@
//! Feeds arbitrary bytes, read as a script, to the JavaScript run entry with fixed event and document values.
//!
//! The run's result is discarded: the target only checks that no input panics or hangs past the limits.
#![no_main]
use libfuzzer_sys::fuzz_target;
use pdfcraft_js::{DocInfo, Event, Limits, run};
/// The largest script the target runs (64 KiB); larger inputs are skipped.
const MAX_INPUT: usize = 64 << 10;
fuzz_target!(|data: &[u8]| {
if data.len() > MAX_INPUT {
return;
}
let script = String::from_utf8_lossy(data).into_owned();
let event = Event {
name: "Keystroke".into(),
kind: "Field".into(),
target: Some("field1".into()),
value: "value".into(),
change: String::new(),
will_commit: true,
};
let _outcome = run(&script, &event, &DocInfo::default(), &[], &[], Limits::default());
});
crates/ocr/Cargo.toml
+4
View File
@@ -16,5 +16,9 @@ ocrs = { version = "0.13.1", default-features = false, features = ["rten"] }
rten = { version = "0.26.0", default-features = false, features = ["onnx_format", "rten_format"] }
thiserror.workspace = true
[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
# The page image is handed to the external tesseract process as a PNG.
png = "0.18"
[dev-dependencies]
tempfile = "3.27"
crates/ocr/README.md
+104
View File
@@ -0,0 +1,104 @@
# pdfcraft-ocr
Layer L4 (may use `fonts` for the text layers' metrics): Scan & OCR — recognising the words in
a page raster and turning them back into page content (execution plan M10).
```rust
// The caller renders a page to pixels and hands them over:
let image = OcrImage::new(w, h, rgba)?; // validated by construction
let lines = recognizer.recognize(&image, &options)?; // Vec<Line>: words in reading order
let placed: Vec<PlacedWord> = …; // PlacedWord::place maps boxes to user space
// The editable output draws real glyphs: plan the words, register the display faces into the
// plan (`pdfcraft_edit::add_type3_fonts`), then generate the layer. An unregistered plan falls
// back to WinAnsi, and characters no registered face has become '?'.
let plan = TextPlan::for_words(placed.iter().map(|w| w.text.as_str()));
add_type3_fonts(&mut doc, page, &mut plan)?; // names the plan's fonts on the page
let layer = visible_text_layer(&placed, &plan); // visible text: the editable output
// The searchable layer needs no display fonts: the engine's own layer (engine::ocr::layer)
// keeps every Unicode character with empty Type3 programs and a ToUnicode map beyond WinAnsi.
```
- **Engines behind one trait.** A [`Recognizer`] reports its `id`, whether it is `available`,
the languages it reads, and reads `Vec<Line>` from an image. Two exist: the pure-Rust
`OcrsRecognizer` (the `ocrs` crate, MIT/Apache-2.0, with its pre-trained models,
CC-BY-SA-4.0, fetched by `cargo xtask models`; found via `$PDFCRAFT_MODELS` or `models/`
beside the executable — see ATTRIBUTION.toml), and — off the web build —
`tesseract::TesseractCli`, which drives an installed `tesseract` binary as an external
process: never linked, bundled or downloaded. Implementations are shared across threads
(`Send + Sync`); nothing spawns on wasm32.
- **The tesseract process is fenced in.** The program comes from a user-configured location or
the PATH, never from document data (the engine validates a configured path as an absolute
path to an existing executable file before it is used). The language code is allow-listed
(`tesseract::language_code`, 12 codes) before anything reaches a process; the page image
goes to a private temp file (a fresh 0700 directory holding a 0600 file on Unix) that is
deleted afterwards; the run has a 120 s timeout, a 64 MiB output cap and a 50 000-word
reading cap (an incomplete reading is rejected rather than applied); every failure comes back through `OcrError` with its cause named. A configured
location that is invalid fails closed (it is never replaced by the PATH search). Availability
is probed with `--list-langs` (its list is read from stdout, or from stderr where older
versions print it); a failed probe is remembered for 30 s, so an install that appears later
is found without a restart. Known limit: at the timeout or the output cap only the direct
child is killed; a wrapper script's own children may outlive it.
- **Confidence drives review.** One classifier (`band_for`) puts a word in the High
(≥ 90), Medium (70–89) or Low (< 70) band — the review overlay's colours, the legend and the
counts; the batch flag rule (`low_confidence`, default threshold 60) is deliberately a
separate threshold. The `ocrs` models report no confidence (its words carry `None`);
a reading whose confidence is unknown is treated as unknown, never as good.
- **Two engines can be merged (ROVER).** Strategies: `PrimaryOnly` (the default), and two
ensembles — `ConfidenceWeighted` (the secondary engine joins when the primary's mean
confidence is under 70 or unknown; an empty primary keeps the fast path) and `RoverVote`
(both engines always run). The merge lets every primary word adopt the best-overlapping
unused secondary word at IoU ≥ 0.5; the higher confidence wins text, box and confidence (a
tie keeps the primary; against a primary word of unknown confidence — ocrs — the secondary
wins only at 70 or more, `UNKNOWN_CONFIDENCE_BAR`), the survivor is named `"ROVER"`, and
unclaimed secondary words are
appended under their engine's name. A merge is capped at 100 M word-pair comparisons (over
it, the primary reading stands alone), and a failing partner degrades to the primary with a
recorded note — a page is never skipped because a partner could not read it.
- **Preprocessing between the raster and recognition**, a pure function in fixed order: DPI
normalization to 300 (only when one of the steps below is on, since they are calibrated for
it; with all of them off the raster is read exactly as rendered, and a scan at or above 300
is left alone), then the opt-in steps —
orientation (90°/180°/270° from projection-profile banding), deskew (± 5°, shear-scored,
below 0.1° is a no-op), denoise (3×3 median) and binarization (Sauvola, window 8, k 0.34).
Every geometric step composes one exact affine `InverseTransform`, and callers map every
word box back to the original raster before placing it. The heuristics are conservative and
panic-free: too small to judge, blank, or no clear winner is a no-op, never a guess.
- **Searchable output preserves Unicode without display fonts.** The engine uses empty
Type3 character programs and `/ToUnicode` maps for non-WinAnsi words, leaving the scan
unchanged. WinAnsi words use `/PCHelv`; the `/OCR` layer is fitted to each word's box.
- **Editable output uses display glyphs.** `visible_text_layer` writes visible text whose
size comes from the box height and whose horizontal scaling uses real font advances.
Beyond WinAnsi, attributed craft-fonts faces supply outline Type3 glyphs with `/ToUnicode`.
Uncovered characters become `?` and are reported in editable output notes; searchable
output retains them. The engine refuses nonfinite geometry or oversized input before
planning fonts: 4 MiB of text and 65,536 words, across all pages for editable export.
Empty geometry is skipped; rotated words carry `across`/`up` vectors and retain direction.
- **Hostile input is rejected, not panicked at.** `OcrImage` holds private, validated state:
non-zero sides, a 10 000-pixel side cap, a 64-megapixel area cap (checked arithmetic — the
counts come from documents, so they are untrusted), and an exact buffer length.
## Where it is wired
The engine's `ocr` module renders pages into `OcrImage`s, runs jobs (`OcrJob::run`) and applies
the words: searchable-image mode draws the invisible layer over the page's own image and leaves
that image untouched; editable mode writes a NEW `<name>_ocr.pdf` of visible text (the scan
image is not carried over, the source is never overwritten). Skip probes keep pages — and
whole files — that already contain text out of the work unless Force OCR overrides them. A
searchable result applies as one undoable step; the Editable text output opens the OCR Verify
screen first, and Scan & OCR ▸ Correct recognized text reviews any recognition there before it
is applied (confidence-coloured boxes, corrections, a per-language user dictionary — a
plain-text file in the preferences folder, so words added from a document persist on disk
outside it until removed — Re-OCR of a region or a page, Accept/Reject), a
multiple-file run writes searchable copies as it goes, and the automation tools `ocr_recognize`,
`ocr_words`, `ocr_status` and `ocr_recognize_files` expose recognition headlessly.
`ocr_words` with `retain_review` returns a bounded session-local handle; `ocr_review_apply`
applies text corrections at the stored original boxes through its stale-document guard,
or discards the result. Empty corrections remove words. Searchable acceptance is one undo
step; editable acceptance writes a separate file. `ocr_dictionary` reads or atomically
updates the UI's per-language dictionary format through an explicitly root-confined path.
Not yet: languages beyond the tested set (the `ocrs` models read only unaccented Latin; the
Tesseract side is bounded by its allow-list and whatever its installed language packs cover),
OCR in the browser (the tesseract module is compiled out for wasm32), the deep scan-cleanup
options (descreen, background removal, edge-shadow removal), and editable glyph coverage
beyond the available attributed faces.
crates/ocr/src/confidence.rs
+106
View File
@@ -0,0 +1,106 @@
//! Confidence: one classifier ([`band_for`]) drives the review overlay, the legend, the counts
//! and the uncertain-word navigation; the batch rule ([`low_confidence`]) is a separate
//! threshold with its own range check. The two rules are deliberately never merged.
/// The three confidence bands (the review overlay's green / yellow / red).
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum ConfidenceBand {
Low,
Medium,
High,
}
impl ConfidenceBand {
/// The legend line for this band, so the meaning never rests on colour alone.
pub fn legend(self) -> &'static str {
match self {
ConfidenceBand::High => "HIGH (>= 90%)",
ConfidenceBand::Medium => "MEDIUM (70-89%)",
ConfidenceBand::Low => "LOW (< 70%)",
}
}
}
/// The one classifier: `>= 90` is High, `70..=89` Medium, `< 70` Low. Out-of-range values
/// clamp (negative → Low, above 100 → High); NaN is Low.
pub fn band_for(confidence: f32) -> ConfidenceBand {
if confidence >= 90.0 {
ConfidenceBand::High
} else if confidence >= 70.0 {
ConfidenceBand::Medium
} else {
ConfidenceBand::Low
}
}
/// The batch rule, separate from [`band_for`]: a word is flagged when `0 <= conf <= threshold`
/// (default 60). Negative or above-threshold values are not flagged; NaN is not.
pub fn low_confidence(confidence: f32, threshold: f32) -> bool {
(0.0..=threshold).contains(&confidence)
}
/// A word whose confidence lands in the Low band — a suspect for review. A word without a
/// reported confidence has no signal and is never a suspect.
pub fn is_suspect(confidence: Option<f32>) -> bool {
confidence.is_some_and(|c| band_for(c) == ConfidenceBand::Low)
}
#[cfg(test)]
mod tests {
use super::*;
/// The boundary matrix: 89 and 90, 69 and 70 straddle the bands.
#[test]
fn bands_straddle_the_boundaries() {
assert_eq!(band_for(89.0), ConfidenceBand::Medium);
assert_eq!(band_for(90.0), ConfidenceBand::High);
assert_eq!(band_for(69.0), ConfidenceBand::Low);
assert_eq!(band_for(70.0), ConfidenceBand::Medium);
assert_eq!(band_for(0.0), ConfidenceBand::Low);
assert_eq!(band_for(100.0), ConfidenceBand::High);
}
/// Out-of-range values clamp, and NaN lands in the Low band instead of comparing equal to
/// nothing.
#[test]
fn out_of_range_confidences_clamp() {
assert_eq!(band_for(-0.5), ConfidenceBand::Low, "negative → Low");
assert_eq!(band_for(1000.0), ConfidenceBand::High, "above 100 → High");
assert_eq!(band_for(f32::NAN), ConfidenceBand::Low);
assert_eq!(band_for(f32::NEG_INFINITY), ConfidenceBand::Low);
assert_eq!(band_for(f32::INFINITY), ConfidenceBand::High);
}
/// The batch threshold is its own rule: everything in `0..=threshold` flags, including
/// words band_for calls Medium (60 < 70), and negatives never do.
#[test]
fn batch_threshold_is_separate_from_the_bands() {
assert!(low_confidence(60.0, 60.0), "at the threshold it flags");
assert!(low_confidence(0.0, 60.0));
assert!(!low_confidence(60.5, 60.0));
assert!(!low_confidence(-1.0, 60.0), "negative confidences are out of the rule's range");
assert!(!low_confidence(f32::NAN, 60.0));
// The rules disagree on real values: 65 is inside the Low band yet outside the batch
// rule with the default threshold (60), which is why the two must not be merged.
assert!(!low_confidence(65.0, 60.0));
assert_eq!(band_for(65.0), ConfidenceBand::Low);
}
/// Unconfident words are not suspects; Low-band words are.
#[test]
fn suspects_need_a_low_band_confidence() {
assert!(!is_suspect(None));
assert!(!is_suspect(Some(95.0)));
assert!(!is_suspect(Some(70.0)));
assert!(is_suspect(Some(69.9)));
assert!(is_suspect(Some(0.0)));
}
/// The legend names the thresholds, so the overlay never rests on colour alone.
#[test]
fn legend_text_matches_the_classifier() {
assert_eq!(ConfidenceBand::High.legend(), "HIGH (>= 90%)");
assert_eq!(ConfidenceBand::Medium.legend(), "MEDIUM (70-89%)");
assert_eq!(ConfidenceBand::Low.legend(), "LOW (< 70%)");
}
}
crates/ocr/src/lib.rs
+603 -192
View File
@@ -1,35 +1,50 @@
//! pdfcraft-ocr — Scan & OCR ▸ Recognize text (L4).
//!
//! Recognition runs the ocrs engine (MIT/Apache-2.0) with its pre-trained models (CC-BY-SA-4.0,
//! fetched by `cargo xtask models`; see ATTRIBUTION.toml). The caller renders a page to pixels;
//! [`Ocr::recognize`] finds the words in them, and [`text_layer`] turns words placed in user space
//! into page content: invisible text (rendering mode 3) over each word, so the page becomes a
//! The caller renders a page to pixels ([`OcrImage`]) and hands it to a [`Recognizer`]. Two
//! engines exist: the pure-Rust `OcrsRecognizer` (ocrs, MIT/Apache-2.0, with its pre-trained
//! models, CC-BY-SA-4.0, fetched by `cargo xtask models`; see ATTRIBUTION.toml) and — off the
//! web — `tesseract::TesseractCli`, which drives an installed `tesseract` binary as an external
//! process. `merge` combines two engines' readings (ROVER), `preprocess` cleans the raster up
//! between rendering and recognition, and [`text_layer`] turns words placed in user space into
//! page content: invisible text (rendering mode 3) over each word, so the page becomes a
//! searchable image (Acrobat's "Searchable Image (Exact)": the image is left untouched).
//!
//! The models read the Latin alphabet (English and other languages written without accents).
//! Visible editable output uses [`pdfcraft_fonts::TextPlan`] and optional craft-fonts glyph
//! outlines; unsupported visible characters become `?` and are counted. The engine's invisible
//! searchable layer uses empty Type3 programs and ToUnicode maps, preserving Unicode without
//! display fonts. WinAnsi-only invisible text keeps the standard Helvetica layout here.
//!
//! The ocrs models read the Latin alphabet (English and other languages written without
//! accents); Tesseract (the `tesseract` module, off the web build) reads whatever its installed
//! language packs cover.
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
use std::path::{Path, PathBuf};
pub mod confidence;
pub mod merge;
mod ocrs;
pub mod preprocess;
#[cfg(not(target_arch = "wasm32"))]
pub mod tesseract;
pub use confidence::{ConfidenceBand, band_for, is_suspect, low_confidence};
pub use merge::{MergeStrategy, StrategyOutcome, mean_confidence, recognize_with_strategy, rover_merge};
pub use ocrs::{ALPHABET, DETECTION_MODEL, LANGUAGES, Models, Ocr, OcrsRecognizer, RECOGNITION_MODEL};
pub use pdfcraft_fonts::TextPlan;
pub use pdfcraft_fonts::helvetica_width;
/// The model files, as named in ATTRIBUTION.toml.
pub const DETECTION_MODEL: &str = "text-detection.onnx";
pub const RECOGNITION_MODEL: &str = "text-recognition.onnx";
/// Hard cap on either side of an image any code in this crate will touch, in pixels; larger
/// images are rejected with [`OcrError::ImageTooLarge`] instead of being processed.
pub const MAX_SIDE: u32 = 10_000;
/// The recognition model's alphabet, as it was trained (ocrs-models' `DEFAULT_ALPHABET`): its
/// output classes follow this order. The ocrs crate's own copy has an `E` where the training had
/// `€`, which made a scanned "€250" read as "E250".
pub const ALPHABET: &str = " 0123456789!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~€ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz";
/// The languages the models read (ISO 639-1); all use the Latin alphabet without accents.
pub const LANGUAGES: &[(&str, &str)] = &[("en", "English")];
/// Hard cap on an image's TOTAL area, in pixels (64 MP): sides under [`MAX_SIDE`] can still
/// multiply out to a raster no engine should have to read; larger images are rejected with
/// [`OcrError::ImageTooLarge`] instead of being processed.
pub const MAX_PIXELS: u32 = 64_000_000;
#[derive(Debug, thiserror::Error)]
pub enum OcrError {
/// Shown to end users (release packages ship the models; a source build fetches them with
/// `cargo xtask models`). The UI catalogs key this exact text.
#[error("the text recognition models are not installed (reinstall PdfCraft, or set PDFCRAFT_MODELS to the folder that holds them)")]
NoModels,
#[error("loading {0}: {1}")]
@@ -38,52 +53,93 @@ pub enum OcrError {
Recognize(String),
#[error("the image is empty")]
EmptyImage,
#[error("the image is larger than the {MAX_SIDE}-pixel side or {MAX_PIXELS}-pixel limit")]
ImageTooLarge,
#[error("the image buffer holds {got} bytes, expected {want} for {width}x{height} RGBA")]
ImageSize { got: usize, want: usize, width: u32, height: u32 },
#[error("the tesseract program is not available: {0}")]
NoTesseract(String),
#[error("tesseract did not finish within {0} seconds")]
Timeout(u64),
#[error("tesseract produced more than {0} bytes of output")]
OutputTooLarge(u64),
#[error("tesseract failed: {0}")]
Process(String),
}
/// Where the two model files are.
#[derive(Clone, Debug, PartialEq)]
pub struct Models {
pub detection: PathBuf,
pub recognition: PathBuf,
/// An RGBA8 raster, row-major, top-left origin: what the engines read. The fields are private
/// so the invariants (non-zero, capped, exactly `width * height * 4` bytes) hold by
/// construction: an [`OcrImage`] in hand has always been validated, whatever produced it.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct OcrImage {
width: u32,
height: u32,
rgba: Vec<u8>,
}
impl Models {
/// A complete model pair in `dir`, preferring current ONNX files over legacy RTEN files.
/// An incomplete update never mixes formats or hides an intact legacy pair.
pub fn in_dir(dir: &Path) -> Option<Models> {
[(DETECTION_MODEL, RECOGNITION_MODEL), ("text-detection.rten", "text-recognition.rten")].into_iter().find_map(|(detection, recognition)| {
let m = Models { detection: dir.join(detection), recognition: dir.join(recognition) };
(m.detection.is_file() && m.recognition.is_file()).then_some(m)
})
}
/// Look for the models: `$PDFCRAFT_MODELS`, then where the release packages install them
/// beside the executable ([`Models::dirs_beside_exe`]), then the source tree's `assets/models/`.
pub fn find() -> Option<Models> {
Self::search_dirs().iter().find_map(|d| Self::in_dir(d))
}
/// The directories [`Models::find`] looks in, in order.
pub fn search_dirs() -> Vec<PathBuf> {
let mut dirs = Vec::new();
if let Some(d) = std::env::var_os("PDFCRAFT_MODELS") {
dirs.push(PathBuf::from(d));
impl OcrImage {
/// Validate the sides ([`MAX_SIDE`]) and total area ([`MAX_PIXELS`]) and that `rgba` holds
/// exactly `width * height * 4` bytes (checked arithmetic; the counts come from documents,
/// so they are untrusted).
pub fn new(width: u32, height: u32, rgba: Vec<u8>) -> Result<OcrImage, OcrError> {
if width == 0 || height == 0 {
return Err(OcrError::EmptyImage);
}
if let Some(exe) = std::env::current_exe().ok().and_then(|e| e.parent().map(Path::to_path_buf)) {
dirs.extend(Self::dirs_beside_exe(&exe));
if width > MAX_SIDE || height > MAX_SIDE {
return Err(OcrError::ImageTooLarge);
}
let pixels = (width as usize).checked_mul(height as usize).ok_or(OcrError::ImageTooLarge)?;
if pixels > MAX_PIXELS as usize {
return Err(OcrError::ImageTooLarge);
}
match pixels.checked_mul(4) {
Some(want) if rgba.len() == want => Ok(OcrImage { width, height, rgba }),
Some(want) => Err(OcrError::ImageSize { got: rgba.len(), want, width, height }),
None => Err(OcrError::ImageTooLarge),
}
dirs.push(Path::new(env!("CARGO_MANIFEST_DIR")).join("../../assets/models"));
dirs
}
/// Where the release packages put the models, relative to the directory holding the
/// executable (packaging/*):
/// - `models/` beside it: the Windows MSI and portable zip;
/// - `../Resources/models`: the macOS app bundle (`Contents/MacOS` → `Contents/Resources`);
/// - `../share/pdfcraft/models`: the FHS-style trees (`bin/` → `share/`) of the Linux
/// deb/rpm/tar.gz/AppImage/Flatpak and the FreeBSD tarball, wherever they are installed.
pub fn dirs_beside_exe(exe_dir: &Path) -> Vec<PathBuf> {
vec![exe_dir.join("models"), exe_dir.join("../Resources/models"), exe_dir.join("../share/pdfcraft/models")]
/// The image's width, in pixels.
pub fn width(&self) -> u32 {
self.width
}
/// The image's height, in pixels.
pub fn height(&self) -> u32 {
self.height
}
/// The RGBA bytes, row-major, exactly `width * height * 4` of them.
pub fn rgba(&self) -> &[u8] {
&self.rgba
}
/// The smallest side, in pixels.
pub fn min_side(&self) -> u32 {
self.width.min(self.height)
}
/// One RGBA byte at integer pixel (`x`, `y`), edges clamped; 255 (white) outside. Only
/// whole-pixel positions and channel 0–3 make sense; anything else reads clamped.
pub fn byte(&self, x: f64, y: f64, ch: usize) -> u8 {
let stride = self.width as usize * 4;
if stride == 0 || self.height == 0 || ch > 3 {
return 255;
}
let xi = (x.round().max(0.0) as usize).min(self.width as usize - 1);
let yi = (y.round().max(0.0) as usize).min(self.height as usize - 1);
self.rgba.get(yi * stride + xi * 4 + ch).copied().unwrap_or(255)
}
/// The grayscale value of the pixel at (`x`, `y`), 0 (ink) … 255 (paper), with the borders
/// clamped (edges replicate). `None` only when the image is empty.
pub fn gray(&self, x: i64, y: i64) -> Option<u8> {
let x = x.clamp(0, self.width as i64 - 1);
let y = y.clamp(0, self.height as i64 - 1);
let row = y as usize * self.width as usize * 4;
let o = row + x as usize * 4;
let px = self.rgba.get(o..o + 4)?;
Some(((299 * px[0] as u32 + 587 * px[1] as u32 + 114 * px[2] as u32) / 1000) as u8)
}
}
@@ -92,6 +148,26 @@ impl Models {
pub struct Word {
pub text: String,
pub rect: [f32; 4],
/// How sure the engine is, 0–100, when it reports a confidence at all (`None`: the ocrs
/// engine does not).
pub confidence: Option<f32>,
/// Which engine read this word: the recogniser's [`Recognizer::id`], or `"ROVER"` when two
/// engines were merged and the surviving text is the merged choice.
pub source: String,
}
impl Word {
pub fn new(text: impl Into<String>, rect: [f32; 4], source: &str) -> Word {
Word { text: text.into(), rect, confidence: None, source: source.into() }
}
/// Set the confidence (builder style), clamped to the documented 0–100 scale (a negative
/// value is 0, anything over 100 is 100). NaN means no confidence was reported (`None`),
/// the same unknown the engines that report nothing carry.
pub fn with_confidence(mut self, confidence: f32) -> Word {
self.confidence = if confidence.is_nan() { None } else { Some(confidence.clamp(0.0, 100.0)) };
self
}
}
/// A recognised line of words, in reading order.
@@ -106,63 +182,34 @@ impl Line {
}
}
/// A loaded recogniser. Loading takes a moment; keep one and reuse it.
pub struct Ocr {
engine: ocrs::OcrEngine,
/// What a recogniser needs to know beyond the pixels.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RecognizeOptions {
/// Language code from [`LANGUAGES`] (`"en"`, `"de"`, …). Engines map it to their own codes
/// (Tesseract: `tesseract::language_code`, allow-listed before anything reaches a process);
/// unknown codes fall back to the engine's default.
pub language: String,
}
impl Ocr {
pub fn load(models: &Models) -> Result<Ocr, OcrError> {
let load = |p: &Path| rten::Model::load_file(p).map_err(|e| OcrError::Load(p.display().to_string(), e.to_string()));
let params = ocrs::OcrEngineParams {
detection_model: Some(load(&models.detection)?),
recognition_model: Some(load(&models.recognition)?),
alphabet: Some(ALPHABET.into()),
..Default::default()
};
let engine = ocrs::OcrEngine::new(params).map_err(|e| OcrError::Load("ocr engine".into(), e.to_string()))?;
Ok(Ocr { engine })
impl Default for RecognizeOptions {
fn default() -> Self {
RecognizeOptions { language: "en".into() }
}
}
/// Load the models found by [`Models::find`].
pub fn find() -> Result<Ocr, OcrError> {
Self::load(&Models::find().ok_or(OcrError::NoModels)?)
}
/// Recognise the text in an RGBA (or RGB, or grey) image, `width` × `height` pixels.
pub fn recognize(&self, pixels: &[u8], width: u32, height: u32) -> Result<Vec<Line>, OcrError> {
if width == 0 || height == 0 || pixels.is_empty() {
return Err(OcrError::EmptyImage);
}
// ocrs wants 1 or 3 channels.
let rgb: std::borrow::Cow<[u8]> = if pixels.len() == (width * height * 4) as usize {
pixels.as_chunks::<4>().0.iter().flat_map(|p| [p[0], p[1], p[2]]).collect::<Vec<u8>>().into()
} else {
pixels.into()
};
let err = |e: &dyn std::fmt::Display| OcrError::Recognize(e.to_string());
let source = ocrs::ImageSource::from_bytes(&rgb, (width, height)).map_err(|e| err(&e))?;
let input = self.engine.prepare_input(source).map_err(|e| err(&e))?;
let found = self.engine.detect_words(&input).map_err(|e| err(&e))?;
let lines = self.engine.find_text_lines(&input, &found);
let read = self.engine.recognize_text(&input, &lines).map_err(|e| err(&e))?;
use ocrs::TextItem;
Ok(read
.into_iter()
.flatten()
.map(|line| Line {
words: line
.words()
.map(|w| {
let r = w.bounding_rect();
Word { text: w.to_string(), rect: [r.left() as f32, r.top() as f32, r.right() as f32, r.bottom() as f32] }
})
.filter(|w| !w.text.trim().is_empty())
.collect(),
})
.filter(|l| !l.words.is_empty())
.collect())
}
/// One text-recognition engine. Implementations are shared across threads
/// (`Arc<dyn Recognizer>`); `available` and `recognize` may spawn processes only off the web
/// build (see the `tesseract` module).
pub trait Recognizer: Send + Sync {
/// The engine's short name (`"ocrs"`, `"tesseract"`): what [`Word::source`] carries and
/// what settings and tools name it by.
fn id(&self) -> &'static str;
/// Whether this engine can run right now (models installed, binary found).
fn available(&self) -> bool;
/// The languages it reads, as `(code, name)` pairs (`"en"`, `"English"`).
fn languages(&self) -> Vec<(String, String)>;
/// Read the words in `image`, in reading order.
fn recognize(&self, image: &OcrImage, options: &RecognizeOptions) -> Result<Vec<Line>, OcrError>;
}
/// A word placed on a page, in PDF user space: the bottom-left corner of its box, and the
@@ -173,6 +220,10 @@ pub struct PlacedWord {
pub origin: [f64; 2],
pub across: [f64; 2],
pub up: [f64; 2],
/// Confidence 0–100, when the engine reported one.
pub confidence: Option<f32>,
/// Which engine read this word (see [`Word::source`]).
pub source: String,
}
impl PlacedWord {
@@ -183,7 +234,14 @@ impl PlacedWord {
let o = to_user(l, b);
let br = to_user(r, b);
let tl = to_user(l, t);
PlacedWord { text: word.text.clone(), origin: o, across: [br[0] - o[0], br[1] - o[1]], up: [tl[0] - o[0], tl[1] - o[1]] }
PlacedWord {
text: word.text.clone(),
origin: o,
across: [br[0] - o[0], br[1] - o[1]],
up: [tl[0] - o[0], tl[1] - o[1]],
confidence: word.confidence,
source: word.source.clone(),
}
}
}
@@ -214,32 +272,193 @@ fn num(v: f64) -> String {
if s == "-0" { "0".into() } else { s.into() }
}
/// Page content that writes `words` as invisible text in `/PCHelv` (standard Helvetica,
/// WinAnsiEncoding), each word stretched to its box so selection and search highlight the
/// right place. Marked content `/OCR` so it can be told apart from the page's own text.
pub fn text_layer(words: &[PlacedWord]) -> Vec<u8> {
/// One run of a word's planned characters in one Type3 font: the resource name, the codes to
/// show (one byte per character), and the run's start as an offset in em units from the word's
/// origin (the accumulated advances of the runs before it).
struct PlannedRun {
font: String,
codes: Vec<u8>,
start: f64,
}
/// The planned runs of word `i` (the same index [`pdfcraft_fonts::TextPlan::for_words`] planned
/// the words with), or `None` when the word is drawn in Helvetica: every character has a
/// WinAnsi code, or nothing could be planned/registered for it (the pre-Unicode output).
fn planned_runs(plan: &pdfcraft_fonts::TextPlan, word: usize) -> Option<(Vec<PlannedRun>, f64)> {
let enc = plan.encoding(word);
if enc.is_empty() || !plan.registered(enc) {
return None;
}
let mut runs: Vec<PlannedRun> = Vec::new();
let mut total = 0.0;
for &(f, code) in enc {
let font = plan.fonts[f].name.as_str();
match runs.last_mut() {
Some(last) if last.font == font => last.codes.push(code),
_ => runs.push(PlannedRun { font: font.to_string(), codes: vec![code], start: total }),
}
total += plan.advance(f, code);
}
(total.is_finite() && total > 0.0).then_some((runs, total))
}
/// Page content that writes `words` as VISIBLE text at their original boxes: the
/// editable-text output's page content. WinAnsi words are set in `/PCHelv` (standard
/// Helvetica); words with characters beyond WinAnsi are set in the plan's Type3 fonts, one run
/// per font. Each word fills its box — the font size comes from the box height and the
/// horizontal scaling is computed from the real advances of the font that draws the word
/// (Helvetica's approximated, the Type3 faces' exact) and emitted as an explicit `Tz`
/// percentage. Words whose box is empty (or whose text has no measurable width) are skipped; a
/// page that ends up with nothing simply shows nothing.
pub fn visible_text_layer(words: &[PlacedWord], plan: &pdfcraft_fonts::TextPlan) -> Vec<u8> {
visible_text_layer_at(words, plan, 0)
}
/// Visible text for words starting at `word_offset` in a shared document plan.
pub fn visible_text_layer_at(words: &[PlacedWord], plan: &pdfcraft_fonts::TextPlan, word_offset: usize) -> Vec<u8> {
// The rendering mode and the font are graphics state, which `ET` doesn't reset: without
// `q … Q` they would stay set for every later content stream on the page, so text added
// after OCR (a header or footer) drew invisibly (#3029).
let mut out = b"/OCR BMC\nq\nBT\n0 Tr\n/PCHelv 1 Tf\n".to_vec();
// The font and size the text state is set in, so a Tf is written only when they change.
// (The header's size-1 Helvetica covers layers of WinAnsi words only.)
let mut current: Option<(String, f64)> = Some(("PCHelv".to_string(), 1.0));
for (i, w) in words.iter().enumerate() {
let height = w.up[0].hypot(w.up[1]);
let across = w.across[0].hypot(w.across[1]);
let (ux, uy) = (w.across[0] / across, w.across[1] / across);
let (vx, vy) = (w.up[0] / height, w.up[1] / height);
if !across.is_finite()
|| !height.is_finite()
|| across <= 0.0
|| height <= 0.0
|| !ux.is_finite()
|| !vx.is_finite()
|| !w.origin.iter().all(|v| v.is_finite())
{
continue;
}
let size = height / BOX_EM;
let runs = planned_runs(plan, word_offset.saturating_add(i));
// The word's natural advance at this size: Helvetica's approximation, or the exact sum
// of the planned glyphs' widths.
let natural = runs.as_ref().map_or_else(|| helvetica_width(&w.text, size), |(_, total)| total * size);
if natural.is_nan() || natural <= 0.0 {
continue;
}
// Horizontal scaling as a percentage of the natural advance at this size, so the word
// lands exactly on its box (100 = no scaling; a long box on a short word stretches it).
let tz = (across / natural * 100.0).max(1.0);
// The baseline sits above the box's bottom edge by the descender depth.
let o = [w.origin[0] + vx * DESCENT * size, w.origin[1] + vy * DESCENT * size];
out.extend_from_slice(format!("{} Tz ", num(tz)).as_bytes());
match &runs {
None => {
if current.as_ref().is_some_and(|(f, s)| f != "PCHelv" || *s != size) {
out.extend_from_slice(format!("/PCHelv {} Tf ", num(size)).as_bytes());
current = Some(("PCHelv".to_string(), size));
}
out.extend_from_slice(format!("{} {} {} {} {} {} Tm ", num(ux), num(uy), num(vx), num(vy), num(o[0]), num(o[1])).as_bytes());
out.extend(pdfcraft_fonts::literal(&pdfcraft_fonts::win_ansi(&w.text)));
out.extend_from_slice(b" Tj\n");
}
Some((planned, total)) => {
for run in planned {
if current.as_ref().is_some_and(|(f, s)| *f != run.font || *s != size) {
out.extend_from_slice(format!("/{} {} Tf ", run.font, num(size)).as_bytes());
current = Some((run.font.clone(), size));
}
// The run's origin: the word's baseline plus the across vector scaled by
// the advances before this run (the stretch applies to runs alike).
let frac = run.start / total;
out.extend_from_slice(
format!(
"{} {} {} {} {} {} Tm ",
num(ux),
num(uy),
num(vx),
num(vy),
num(o[0] + w.across[0] * frac),
num(o[1] + w.across[1] * frac)
)
.as_bytes(),
);
out.extend(pdfcraft_fonts::literal(&run.codes));
out.extend_from_slice(b" Tj\n");
}
}
}
}
out.extend_from_slice(b"ET\nQ\nEMC\n");
out
}
/// Page content that writes `words` as invisible text (rendering mode 3), each word stretched
/// to its box so selection and search highlight the right place. WinAnsi words are set in
/// `/PCHelv` (standard Helvetica); words with characters beyond WinAnsi are set in the plan's
/// Type3 fonts, one run per font, with the same exact box fit. Marked content `/OCR` so it can
/// be told apart from the page's own text.
pub fn text_layer(words: &[PlacedWord], plan: &pdfcraft_fonts::TextPlan) -> Vec<u8> {
// `3 Tr` and the font are graphics state, which `ET` doesn't reset: without `q … Q` they
// would stay set for every later content stream on the page, so text added after OCR (a
// header or footer) drew invisibly (#3029).
let mut out = b"/OCR BMC\nq\nBT\n3 Tr\n/PCHelv 1 Tf\n".to_vec();
for w in words {
let width = helvetica_width(&w.text, 1.0);
let mut current: Option<(String, f64)> = Some(("PCHelv".to_string(), 1.0));
for (i, w) in words.iter().enumerate() {
let runs = planned_runs(plan, i);
let width = runs.as_ref().map_or_else(|| helvetica_width(&w.text, 1.0), |(_, total)| *total);
let height = w.up[0].hypot(w.up[1]);
if width <= 0.0 || height <= 0.0 {
let across = w.across[0].hypot(w.across[1]);
// Only positive, finite geometry reaches the content stream: a zero-sized box cannot
// be drawn, and a NaN or infinite one would put non-finite numbers on the page. (NaN
// fails every comparison, so the positivity checks already refuse it; the finiteness
// checks keep that property explicit.)
if width <= 0.0 || !(height > 0.0 && height.is_finite()) || !(across > 0.0 && across.is_finite()) || !w.origin.iter().all(|v| v.is_finite()) {
continue;
}
// The em square: the word's own ink (cap height or x-height, descenders or not) spans the
// box, so the text is as tall as the scanned word; its width is stretched to fit the word.
let (top, bottom) = ink_extent(&w.text);
// The em square. Helvetica words: the word's own ink (cap height or x-height,
// descenders or not) spans the box, so the text is as tall as the scanned word. Planned
// (Type3) words: those Helvetica letter heights do not describe the face, so the box spans
// the ascender-descender height. Either way the width is stretched to fit the word.
let (top, bottom) = if runs.is_some() { (BOX_EM - DESCENT, -DESCENT) } else { ink_extent(&w.text) };
let em = [w.up[0] / (top - bottom), w.up[1] / (top - bottom)];
let ax = [w.across[0] / width, w.across[1] / width];
let o = [w.origin[0] - em[0] * bottom, w.origin[1] - em[1] * bottom];
let m = [ax[0], ax[1], em[0], em[1], o[0], o[1]];
let nums: Vec<String> = m.iter().map(|v| num(*v)).collect();
out.extend_from_slice(nums.join(" ").as_bytes());
out.extend_from_slice(b" Tm ");
out.extend(pdfcraft_fonts::literal(&pdfcraft_fonts::win_ansi(&w.text)));
out.extend_from_slice(b" Tj\n");
match &runs {
None => {
let ax = [w.across[0] / width, w.across[1] / width];
if current.as_ref().is_some_and(|(f, s)| f != "PCHelv" || *s != 1.0) {
out.extend_from_slice(b"/PCHelv 1 Tf ");
current = Some(("PCHelv".to_string(), 1.0));
}
let m = [ax[0], ax[1], em[0], em[1], o[0], o[1]];
let nums: Vec<String> = m.iter().map(|v| num(*v)).collect();
out.extend_from_slice(nums.join(" ").as_bytes());
out.extend_from_slice(b" Tm ");
out.extend(pdfcraft_fonts::literal(&pdfcraft_fonts::win_ansi(&w.text)));
out.extend_from_slice(b" Tj\n");
}
Some((planned, total)) => {
for run in planned {
if current.as_ref().is_some_and(|(f, s)| *f != run.font || *s != 1.0) {
out.extend_from_slice(format!("/{} 1 Tf ", run.font).as_bytes());
current = Some((run.font.clone(), 1.0));
}
let m = [
w.across[0] / total,
w.across[1] / total,
em[0],
em[1],
o[0] + w.across[0] * run.start / total,
o[1] + w.across[1] * run.start / total,
];
let nums: Vec<String> = m.iter().map(|v| num(*v)).collect();
out.extend_from_slice(nums.join(" ").as_bytes());
out.extend_from_slice(b" Tm ");
out.extend(pdfcraft_fonts::literal(&run.codes));
out.extend_from_slice(b" Tj\n");
}
}
}
}
out.extend_from_slice(b"ET\nQ\nEMC\n");
out
@@ -253,10 +472,22 @@ mod tests {
fn places_words_through_the_mapping() {
// 2 pixels per point, page 612 × 792, y flipped.
let to_user = |x: f32, y: f32| [x as f64 / 2.0, 792.0 - y as f64 / 2.0];
let w = PlacedWord::place(&Word { text: "Hello".into(), rect: [100.0, 200.0, 300.0, 240.0] }, to_user);
let w = PlacedWord::place(&Word::new("Hello", [100.0, 200.0, 300.0, 240.0], "ocrs"), to_user);
assert_eq!(w.origin, [50.0, 672.0]);
assert_eq!(w.across, [100.0, 0.0]);
assert_eq!(w.up, [0.0, 20.0]);
assert_eq!(w.source, "ocrs");
assert_eq!(w.confidence, None);
}
/// Confidence and source survive placement.
#[test]
fn placed_words_carry_confidence_and_source() {
let to_user = |x: f32, y: f32| [x as f64, y as f64];
let w = Word::new("Hi", [1.0, 2.0, 3.0, 4.0], "tesseract").with_confidence(87.5);
let p = PlacedWord::place(&w, to_user);
assert_eq!(p.confidence, Some(87.5));
assert_eq!(p.source, "tesseract");
}
/// The text layer is Helvetica in WinAnsiEncoding: every character the model can recognise
@@ -267,10 +498,27 @@ mod tests {
assert_eq!(pdfcraft_fonts::first_undrawable(ALPHABET), None);
}
/// `with_confidence` is clamped to the documented 0–100 scale: negatives (and −inf) are 0,
/// anything over 100 (and +inf) is 100, in-scale values pass through, and NaN means no
/// confidence was reported — the same `None` unknown the unreporting engines carry.
#[test]
fn with_confidence_clamps_and_treats_nan_as_unknown() {
let c = |v: f32| Word::new("w", [0.0; 4], "ocrs").with_confidence(v).confidence;
assert_eq!(c(0.0), Some(0.0));
assert_eq!(c(50.0), Some(50.0));
assert_eq!(c(100.0), Some(100.0));
assert_eq!(c(-1.0), Some(0.0), "negatives clamp to 0");
assert_eq!(c(f32::NEG_INFINITY), Some(0.0));
assert_eq!(c(150.0), Some(100.0), "over 100 clamps to 100");
assert_eq!(c(f32::INFINITY), Some(100.0));
assert_eq!(c(f32::NAN), None, "NaN is unknown, not a confidence");
assert_eq!(c(f32::EPSILON), Some(f32::EPSILON), "in-scale values are untouched");
}
#[test]
fn text_layer_is_invisible_text_fitted_to_each_box() {
let w = PlacedWord { text: "Hi".into(), origin: [10.0, 20.0], across: [30.0, 0.0], up: [0.0, 9.3] };
let s = String::from_utf8(text_layer(&[w])).unwrap();
let w = PlacedWord { text: "Hi".into(), origin: [10.0, 20.0], across: [30.0, 0.0], up: [0.0, 9.3], confidence: None, source: "ocrs".into() };
let s = String::from_utf8(text_layer(&[w], &TextPlan::default())).unwrap();
assert!(s.contains("3 Tr") && s.contains("/PCHelv 1 Tf"), "{s}");
let width = helvetica_width("Hi", 1.0);
// "Hi" has no descender: its box spans the cap height, from the baseline up.
@@ -283,8 +531,8 @@ mod tests {
/// after it on the page starts from the graphics state it had before.
#[test]
fn the_layer_restores_the_graphics_state_it_changes() {
let w = PlacedWord { text: "Hi".into(), origin: [10.0, 20.0], across: [30.0, 0.0], up: [0.0, 9.3] };
let s = String::from_utf8(text_layer(&[w])).unwrap();
let w = PlacedWord { text: "Hi".into(), origin: [10.0, 20.0], across: [30.0, 0.0], up: [0.0, 9.3], confidence: None, source: "ocrs".into() };
let s = String::from_utf8(text_layer(&[w], &TextPlan::default())).unwrap();
let mut depth = 0_i32;
for token in s.split_ascii_whitespace() {
match token {
@@ -309,78 +557,200 @@ mod tests {
assert_eq!(ink_extent("Gallus.Ze@camtel.cm"), (CAP_HEIGHT, 0.0));
assert_eq!(ink_extent("Corporation"), (CAP_HEIGHT, -DESCENT));
assert_eq!(ink_extent("--"), (BOX_EM - DESCENT, -DESCENT));
let w = PlacedWord { text: "gap".into(), origin: [0.0, 100.0], across: [20.0, 0.0], up: [0.0, 7.3] };
let s = String::from_utf8(text_layer(&[w])).unwrap();
let w = PlacedWord { text: "gap".into(), origin: [0.0, 100.0], across: [20.0, 0.0], up: [0.0, 7.3], confidence: None, source: "ocrs".into() };
let s = String::from_utf8(text_layer(&[w], &TextPlan::default())).unwrap();
let em = 7.3 / (X_HEIGHT + DESCENT);
assert!(s.contains(&format!(" 0 0 {} 0 {} Tm (gap) Tj", num(em), num(100.0 + em * DESCENT))), "{s}");
}
/// A word whose box is not positive, finite geometry never reaches the content stream:
/// NaN, infinite and zero-sized boxes are dropped, so the layer never carries non-finite
/// numbers.
#[test]
fn text_layer_refuses_non_finite_and_degenerate_boxes() {
let word = |across: [f64; 2], up: [f64; 2], origin: [f64; 2]| PlacedWord {
text: "Hi".into(),
origin,
across,
up,
confidence: Some(90.0),
source: "tesseract".into(),
};
for (why, w) in [
("NaN across", word([f64::NAN, 0.0], [0.0, 9.3], [0.0; 2])),
("NaN up", word([30.0, 0.0], [f64::NAN, 1.0], [0.0; 2])),
("inf up", word([30.0, 0.0], [0.0, f64::INFINITY], [0.0; 2])),
("inf origin", word([30.0, 0.0], [0.0, 9.3], [0.0, f64::INFINITY])),
("zero across", word([0.0, 0.0], [0.0, 9.3], [0.0; 2])),
("NaN origin", word([30.0, 0.0], [0.0, 9.3], [0.0, f64::NAN])),
] {
let s = String::from_utf8(text_layer(&[w], &TextPlan::default())).unwrap();
assert_eq!(s, "/OCR BMC\nq\nBT\n3 Tr\n/PCHelv 1 Tf\nET\nQ\nEMC\n", "{why}: {s}");
assert!(!s.contains("NaN") && !s.contains("inf"), "{why}: {s}");
}
}
/// The editable-text layer draws visible text whose horizontal scaling is an explicit
/// percentage computed from real Helvetica metrics: `Tz` × natural advance = box width.
#[test]
fn visible_text_layer_scales_by_a_percentage_of_real_metrics() {
let w = PlacedWord { text: "Hi".into(), origin: [10.0, 20.0], across: [40.0, 0.0], up: [0.0, 9.3], confidence: None, source: "ocrs".into() };
let s = String::from_utf8(visible_text_layer(&[w], &TextPlan::default())).unwrap();
assert!(s.contains("0 Tr"), "visible, not invisible: {s}");
let size = 9.3 / BOX_EM;
let natural = helvetica_width("Hi", size);
let tz = 40.0 / natural * 100.0;
assert!(s.contains(&format!("{} Tz", num(tz))), "{s}");
assert!(s.contains("(Hi) Tj"), "{s}");
assert!(s.starts_with("/OCR BMC") && s.ends_with("EMC\n"));
}
/// Degenerate boxes and words with no measurable width are skipped, never divide by zero.
#[test]
fn visible_text_layer_skips_degenerate_words() {
let word = |across: [f64; 2], up: [f64; 2], text: &str| PlacedWord {
text: text.into(),
origin: [0.0; 2],
across,
up,
confidence: None,
source: "ocrs".into(),
};
let words = vec![
word([0.0, 0.0], [0.0, 9.0], "flat"),
word([10.0, 0.0], [0.0, 0.0], "short"),
word([f64::NAN, 0.0], [0.0, 9.0], "nan"),
word([10.0, 0.0], [0.0, 9.0], ""),
word([10.0, 0.0], [0.0, 9.0], "Hi"),
];
let s = String::from_utf8(visible_text_layer(&words, &TextPlan::default())).unwrap();
assert_eq!(s.matches("Tj").count(), 1, "only the one drawable word is written: {s}");
}
/// A word whose ORIGIN is not finite never reaches the visible layer either: a NaN or
/// infinite origin would put "NaN"/"inf" into the content stream.
#[test]
fn visible_text_layer_refuses_non_finite_origins() {
let word =
|origin: [f64; 2]| PlacedWord { text: "Hi".into(), origin, across: [10.0, 0.0], up: [0.0, 9.0], confidence: None, source: "ocrs".into() };
for (why, origin) in [("NaN x", [f64::NAN, 100.0]), ("inf y", [10.0, f64::INFINITY]), ("NaN both", [f64::NAN; 2])] {
let s = String::from_utf8(visible_text_layer(&[word(origin)], &TextPlan::default())).unwrap();
assert!(!s.contains("NaN") && !s.contains("inf"), "{why}: {s}");
assert_eq!(s.matches("Tj").count(), 0, "{why}: {s}");
}
}
/// After a Type3 word the text state goes back to Helvetica: the Latin words around a
/// planned word must not inherit its font.
#[test]
fn helvetica_is_restored_after_a_planned_word() {
if pdfcraft_fonts::CRAFT_FONTS.is_empty() {
eprintln!("skipping: built without craft-fonts (set CRAFT_FONTS_DIR to run it)");
return;
}
let word = |text: &str| PlacedWord {
text: text.into(),
origin: [10.0, 20.0],
across: [60.0, 0.0],
up: [0.0, 9.3],
confidence: None,
source: "ocrs".into(),
};
let words = [word("Hi"), word("日本"), word("Ho")];
let mut plan = TextPlan::for_words(["Hi", "日本", "Ho"]);
for (i, font) in plan.fonts.iter_mut().enumerate() {
font.name = format!("PCOcr{i}");
}
let s = String::from_utf8(text_layer(&words, &plan)).unwrap();
assert!(s.contains("/PCOcr0 1 Tf"), "{s}");
// The last Helvetica selection comes after the Type3 run, and the Latin word after it
// is drawn in Helvetica, not left in the planned font.
assert!(s.rfind("/PCOcr0 1 Tf").unwrap_or(0) < s.rfind("/PCHelv 1 Tf").unwrap_or(usize::MAX), "{s}");
// The `Q` closes the `q … Q` scope that keeps `3 Tr` from leaking into later content.
assert!(s.ends_with("Tm (Ho) Tj\nET\nQ\nEMC\n"), "{s}");
// The visible layer restores Helvetica with the word's size.
let v = String::from_utf8(visible_text_layer(&words, &plan)).unwrap();
assert!(v.rfind("/PCOcr0").unwrap_or(0) < v.rfind("/PCHelv").unwrap_or(usize::MAX), "{v}");
assert!(v.contains("(Ho) Tj"), "{v}");
}
/// Words with characters beyond WinAnsi draw in the plan's Type3 font, one run, with the
/// same exact box fit as the Helvetica words (the matrix x-axis spans across / advance).
#[test]
fn planned_words_draw_in_their_font_with_the_same_exact_fit() {
if pdfcraft_fonts::CRAFT_FONTS.is_empty() {
eprintln!("skipping: built without craft-fonts (set CRAFT_FONTS_DIR to run it)");
return;
}
let w =
PlacedWord { text: "日本".into(), origin: [10.0, 20.0], across: [60.0, 0.0], up: [0.0, 9.3], confidence: None, source: "ocrs".into() };
let mut plan = TextPlan::for_words(["日本"]);
// The names the registrar (pdfcraft_edit::add_type3_fonts) fills in — this crate
// cannot reach it, so the test sets them where it reads them.
for (i, font) in plan.fonts.iter_mut().enumerate() {
font.name = format!("PCOcr{i}");
}
let codes: Vec<u8> = plan.encoding(0).iter().map(|&(_, c)| c).collect();
assert_eq!(codes.len(), 2, "{codes:?}");
let total = plan.advance_of(plan.encoding(0));
let s = String::from_utf8(text_layer(std::slice::from_ref(&w), &plan)).unwrap();
assert!(s.contains("/PCOcr0 1 Tf"), "{s}");
assert!(s.contains(&format!("{} 0 0 10 10 22.1 Tm ", num(60.0 / total))), "{s}");
assert!(s.contains(&format!("Tm {} Tj\n", String::from_utf8_lossy(&pdfcraft_fonts::literal(&codes)))), "{s}");
// The visible layer: size from the box height, Tz from the Type3's own advance.
let v = String::from_utf8(visible_text_layer(std::slice::from_ref(&w), &plan)).unwrap();
let size = 9.3 / BOX_EM;
assert!(v.contains(&format!("/PCOcr0 {} Tf ", num(size))), "{v}");
assert!(v.contains(&format!("{} Tz", num(60.0 / (total * size) * 100.0))), "{v}");
}
/// Without planned characters (an empty plan: built without craft-fonts, or before the
/// fonts are registered) the WinAnsi path stands, '?' included.
#[test]
fn unplanned_words_fall_back_to_win_ansi_question_marks() {
let w =
PlacedWord { text: "日本".into(), origin: [10.0, 20.0], across: [60.0, 0.0], up: [0.0, 9.3], confidence: None, source: "ocrs".into() };
let s = String::from_utf8(text_layer(&[w], &TextPlan::default())).unwrap();
assert!(s.contains("(??) Tj"), "{s}");
assert!(!s.contains("PCOcr"), "{s}");
}
/// A character no face has a glyph for becomes the font's '?', still in the run, counted
/// by the plan.
#[test]
fn substitutions_are_written_as_the_font_s_question_mark() {
if pdfcraft_fonts::CRAFT_FONTS.is_empty() {
eprintln!("skipping: built without craft-fonts (set CRAFT_FONTS_DIR to run it)");
return;
}
let w = PlacedWord {
text: "日\u{1f4a9}語".into(),
origin: [0.0, 0.0],
across: [60.0, 0.0],
up: [0.0, 9.3],
confidence: None,
source: "ocrs".into(),
};
let mut plan = TextPlan::for_words(["日\u{1f4a9}語"]);
for (i, font) in plan.fonts.iter_mut().enumerate() {
font.name = format!("PCOcr{i}");
}
assert_eq!(plan.missing(), 1, "the emoji has no glyph anywhere");
assert_eq!(plan.encoding(0).len(), 3, "it still gets an encoding: the font's ?");
let codes: Vec<u8> = plan.encoding(0).iter().map(|&(_, c)| c).collect();
let s = String::from_utf8(text_layer(&[w], &plan)).unwrap();
assert!(s.contains(&format!("Tm {} Tj\n", String::from_utf8_lossy(&pdfcraft_fonts::literal(&codes)))), "{s}");
}
#[test]
fn rotated_pages_keep_the_reading_direction() {
// A page turned 90°: image x runs up the page.
let to_user = |x: f32, y: f32| [y as f64, x as f64];
let w = PlacedWord::place(&Word { text: "Up".into(), rect: [0.0, 0.0, 50.0, 10.0] }, to_user);
let w = PlacedWord::place(&Word::new("Up", [0.0, 0.0, 50.0, 10.0], "ocrs"), to_user);
assert_eq!(w.across, [0.0, 50.0]);
assert_eq!(w.up, [-10.0, 0.0]);
}
/// The current ONNX pair wins when an installation also retains both legacy models.
#[test]
fn model_discovery_prefers_the_complete_onnx_pair() {
let dir = tempfile::tempdir().unwrap();
for file in ["text-detection.onnx", "text-recognition.onnx"] {
std::fs::write(dir.path().join(file), b"model").unwrap();
}
let expected = Some(Models { detection: dir.path().join("text-detection.onnx"), recognition: dir.path().join("text-recognition.onnx") });
assert_eq!(Models::in_dir(dir.path()), expected, "a complete ONNX pair is sufficient");
for file in ["text-detection.rten", "text-recognition.rten"] {
std::fs::write(dir.path().join(file), b"model").unwrap();
}
assert_eq!(Models::in_dir(dir.path()), expected, "the complete ONNX pair takes precedence over legacy models");
}
/// An interrupted ONNX update still permits an intact legacy pair in the same directory.
#[test]
fn model_discovery_falls_back_to_legacy_when_the_onnx_pair_is_incomplete() {
for current in ["text-detection.onnx", "text-recognition.onnx"] {
let dir = tempfile::tempdir().unwrap();
for file in [current, "text-detection.rten", "text-recognition.rten"] {
std::fs::write(dir.path().join(file), b"model").unwrap();
}
assert_eq!(
Models::in_dir(dir.path()),
Some(Models { detection: dir.path().join("text-detection.rten"), recognition: dir.path().join("text-recognition.rten") }),
"{current} alone must not hide a complete legacy pair"
);
}
}
/// Detection and recognition must come from one complete format pair, never a mixed update.
#[test]
fn model_discovery_refuses_mixed_format_pairs() {
for pair in [["text-detection.onnx", "text-recognition.rten"], ["text-detection.rten", "text-recognition.onnx"]] {
let dir = tempfile::tempdir().unwrap();
for file in pair {
std::fs::write(dir.path().join(file), b"model").unwrap();
}
assert_eq!(Models::in_dir(dir.path()), None, "{pair:?} is not a complete model pair");
}
}
/// Existing source builds and PDFCRAFT_MODELS overrides retain their complete RTEN pair.
#[test]
fn model_discovery_keeps_legacy_only_installations() {
let dir = tempfile::tempdir().unwrap();
assert_eq!(Models::in_dir(dir.path()), None);
std::fs::write(dir.path().join("text-detection.rten"), b"model").unwrap();
assert_eq!(Models::in_dir(dir.path()), None, "one legacy model is incomplete");
std::fs::write(dir.path().join("text-recognition.rten"), b"model").unwrap();
assert_eq!(
Models::in_dir(dir.path()),
Some(Models { detection: dir.path().join("text-detection.rten"), recognition: dir.path().join("text-recognition.rten") })
);
}
/// #103: every release package's layout puts the models where the installed app looks.
/// Mirrors packaging/{windows/package.ps1,macos/package.sh,linux/package.sh,freebsd/package.sh}.
#[test]
@@ -443,6 +813,47 @@ mod tests {
assert!(word.rect[0] >= 10.0 && word.rect[0] < 60.0, "{word:?}");
}
/// The recognised words name their engine.
#[test]
fn ocrs_recognizer_names_its_words() {
let Ok(rec) = OcrsRecognizer::find() else {
eprintln!("skipped: OCR models not installed");
return;
};
assert_eq!(rec.id(), "ocrs");
assert!(rec.languages().iter().any(|(c, n)| c == "en" && n == "English"), "{:?}", rec.languages());
let (w, h, px) = test_image::hello();
let image = OcrImage::new(w, h, px).unwrap();
let lines = rec.recognize(&image, &RecognizeOptions::default()).unwrap();
let words: Vec<&Word> = lines.iter().flat_map(|l| &l.words).collect();
assert!(!words.is_empty());
assert!(words.iter().all(|w| w.source == "ocrs"), "{words:?}");
}
/// Hostile images are rejected with errors, never panics: zero sides, sides over the cap,
/// totals over the area cap (even with legal sides), buffers that do not fit, counts that
/// overflow.
#[test]
fn hostile_image_dimensions_are_rejected_not_panicked_at() {
assert!(matches!(OcrImage::new(0, 10, vec![]), Err(OcrError::EmptyImage)));
assert!(matches!(OcrImage::new(10, 0, vec![]), Err(OcrError::EmptyImage)));
assert!(matches!(OcrImage::new(u32::MAX, u32::MAX, vec![]), Err(OcrError::ImageTooLarge)));
assert!(matches!(OcrImage::new(MAX_SIDE + 1, 1, vec![0; 4]), Err(OcrError::ImageTooLarge)));
// 10 000 × 7 000 = 70 MP: legal on both sides, over the 64 MP area cap (no allocation
// is needed to reject it — the buffer is never inspected).
assert!(matches!(OcrImage::new(MAX_SIDE, 7000, vec![]), Err(OcrError::ImageTooLarge)));
// 8 000 × 8 000 = exactly the area cap: accepted shape (the short buffer proves the
// cap check passed), while one more row is refused.
assert!(matches!(OcrImage::new(8000, 8000, vec![0; 16]), Err(OcrError::ImageSize { .. })));
assert!(matches!(OcrImage::new(8000, 8001, vec![0; 16]), Err(OcrError::ImageTooLarge)));
assert!(matches!(OcrImage::new(2, 2, vec![0; 15]), Err(OcrError::ImageSize { .. })));
let ok = OcrImage::new(2, 2, vec![255; 16]).unwrap();
assert_eq!((ok.width(), ok.height()), (2, 2));
assert_eq!(ok.rgba().len(), 16);
assert_eq!(ok.gray(5, 5), Some(255), "clamped reads stay inside");
assert_eq!(ok.gray(-1, -1), Some(255));
}
mod test_image {
/// "HELLO WORLD" drawn with blocky 5×7 letters, 4 pixels per dot, black on white.
pub fn hello() -> (u32, u32, Vec<u8>) {
crates/ocr/src/merge.rs
+583
View File
@@ -0,0 +1,583 @@
//! Combining engines. "ROVER" merges two engines' readings of one page: every primary word
//! adopts the best-overlapping unused secondary word when their boxes agree enough (IoU, below);
//! the higher confidence wins the text, box and confidence (a tie keeps the primary; against a
//! primary of unknown confidence the secondary must reach [`UNKNOWN_CONFIDENCE_BAR`]) and the
//! surviving word is named `"ROVER"`; secondary words nothing claimed are appended, keeping
//! their own engine's name.
use crate::{Line, OcrError, OcrImage, RecognizeOptions, Recognizer, Word};
/// How the chosen engines combine. The default is [`MergeStrategy::PrimaryOnly`] with the
/// default engine (ocrs), which is today's behavior.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)]
pub enum MergeStrategy {
/// Only the primary engine runs.
#[default]
PrimaryOnly,
/// The secondary engine joins only when the primary's reading looks weak: its mean word
/// confidence is under 70, or its confidence is unknown (words were read but none reports
/// a confidence — the ocrs engine). An unknown reading is never counted as 100; an empty
/// primary keeps the fast path (no words, nothing to double-check).
ConfidenceWeighted,
/// Both engines always run and their readings merge (with no secondary engine this
/// degrades to primary only).
RoverVote,
}
/// The mean of the confidences the words report, 0–100; `None` when the reading's confidence
/// is UNKNOWN — words were read but none reports a confidence (the ocrs engine's words).
/// Unknown is never counted as 100: there is no low-confidence signal, but a reading whose
/// confidence nobody knows is not therefore confident, so a confidence-weighted strategy
/// answers it by asking the secondary engine. An empty reading is `Some(100.0)`: no words
/// were read, so there is nothing to double-check and the fast path stands.
pub fn mean_confidence(lines: &[Line]) -> Option<f32> {
let mut total = 0usize;
let mut reported = (0.0f64, 0usize);
for word in lines.iter().flat_map(|l| &l.words) {
total += 1;
if let Some(c) = word.confidence {
reported.0 += c as f64;
reported.1 += 1;
}
}
match (total, reported.1) {
(0, _) => Some(100.0),
(_, 0) => None,
(_, n) => Some((reported.0 / n as f64) as f32),
}
}
/// Intersection over union of two `[left, top, right, bottom]` boxes. An empty intersection is
/// 0; degenerate or NaN boxes give 0 rather than panicking or poisoning the comparison.
pub fn iou(a: &[f32; 4], b: &[f32; 4]) -> f64 {
let l = a[0].max(b[0]);
let t = a[1].max(b[1]);
let r = a[2].min(b[2]);
let btm = a[3].min(b[3]);
if !(r > l && btm > t) {
return 0.0; // disjoint, touching, inverted, or NaN (all comparisons false)
}
let inter = (r - l) as f64 * (btm - t) as f64;
let area = |x: &[f32; 4]| ((x[2] - x[0]).max(0.0) as f64) * ((x[3] - x[1]).max(0.0) as f64);
let union = area(a) + area(b) - inter;
if union <= 0.0 {
return 0.0;
}
inter / union
}
/// The most primary×secondary word comparisons one ROVER merge will attempt. Known limit: a
/// merge over this cap is skipped — the primary reading stands alone, and the skip is noted,
/// never silent — so two enormous readings cannot turn the merge into quadratic work without
/// end. A real page is a few hundred words on each side, orders of magnitude under the cap.
pub const ROVER_MAX_COMPARISONS: usize = 100_000_000;
/// Whether the two readings are too large to merge: their word counts multiply past
/// [`ROVER_MAX_COMPARISONS`]. Shared by [`rover_merge`] (which then keeps the primary) and
/// [`recognize_with_strategy`] (which then notes the skip).
fn over_cap(primary: &[Line], secondary: &[Line]) -> bool {
let primary_words: usize = primary.iter().map(|l| l.words.len()).sum();
let secondary_words: usize = secondary.iter().map(|l| l.words.len()).sum();
primary_words.saturating_mul(secondary_words) > ROVER_MAX_COMPARISONS
}
/// The confidence a secondary word needs to replace a primary word whose confidence is
/// unknown (the ocrs engine reports none). It is the bar under which a reading counts as weak
/// ([`MergeStrategy::ConfidenceWeighted`] asks the second engine below it): a word the second
/// engine itself calls weak does not displace a reading nobody scored.
pub const UNKNOWN_CONFIDENCE_BAR: f32 = 70.0;
/// True when the secondary word's confidence `a` beats the primary's `b`. Both known: the
/// strictly higher one wins (a tie keeps the primary). A known confidence against an unknown
/// primary wins only at or above [`UNKNOWN_CONFIDENCE_BAR`]: an unknown confidence is neither
/// 0 nor 100, so a 5% reading must not replace it, while a confident one may. NaN never wins,
/// and with no confidence on the secondary side the primary stands.
fn higher(a: Option<f32>, b: Option<f32>) -> bool {
match (a, b) {
(Some(a), Some(b)) => a.partial_cmp(&b) == Some(std::cmp::Ordering::Greater),
(Some(a), None) => a >= UNKNOWN_CONFIDENCE_BAR,
_ => false,
}
}
/// Merge the two engines' readings. Primary words keep their order and lines; a word is
/// replaced by its overlapping secondary (IoU >= 0.5) only when that secondary is more
/// confident, and either way the merged word's [`Word::source`] is `"ROVER"`. Secondary words
/// no primary claimed are appended, in their own reading order and lines, under their own
/// engine's name (`secondary_id`). Known limit: over [`ROVER_MAX_COMPARISONS`] comparisons
/// (word counts multiplied) the primary reading is kept as is, unmerged; the strategy runner
/// names that skip in a note.
pub fn rover_merge(primary: Vec<Line>, secondary: Vec<Line>, secondary_id: &str) -> Vec<Line> {
if over_cap(&primary, &secondary) {
return primary; // over the comparison cap: the primary stands alone
}
let mut primary = primary;
// The secondary's words, marked `None` once a primary word claimed them, grouped by their
// own lines so the unclaimed ones can be appended in order.
let mut unused: Vec<Vec<Option<Word>>> = secondary.into_iter().map(|l| l.words.into_iter().map(Some).collect()).collect();
for line in &mut primary {
for word in &mut line.words {
// The best-overlapping unused secondary word, if any beats IoU 0.5.
let mut best: Option<(f64, usize, usize)> = None;
for (li, words) in unused.iter().enumerate() {
for (wi, slot) in words.iter().enumerate() {
let Some(w) = slot else { continue };
let score = iou(&word.rect, &w.rect);
if score >= 0.5 && best.is_none_or(|(b, _, _)| score > b) {
best = Some((score, li, wi));
}
}
}
let Some((_, li, wi)) = best else { continue };
let claimed = unused.get_mut(li).and_then(|l| l.get_mut(wi)).and_then(|slot| slot.take());
let Some(claim) = claimed else { continue };
if higher(claim.confidence, word.confidence) {
*word = Word { source: "ROVER".into(), ..claim };
} else {
// Tie (or the primary wins): keep the primary's text, box and confidence, but
// record that this reading is the engines' agreed one.
word.source = "ROVER".into();
}
}
}
// Append the secondary words nothing claimed, line by line, under their engine's name
// (`secondary_id` backfills words that name no source of their own).
let mut merged = primary;
for line in unused {
let words: Vec<Word> = line
.into_iter()
.flatten()
.map(|mut w| {
if w.source.is_empty() {
w.source = secondary_id.into();
}
w
})
.collect();
if !words.is_empty() {
merged.push(Line { words });
}
}
merged
}
/// What a strategy run produced: the merged (or primary-only) reading, whether the secondary
/// engine's reading is part of it, and the notes recorded along the way.
#[derive(Clone, Debug, PartialEq, Default)]
pub struct StrategyOutcome {
pub lines: Vec<Line>,
/// Whether the secondary engine's reading is part of `lines`: true only when the merge
/// actually happened. A secondary that failed, or whose reading was discarded over the
/// comparison cap, is `false`, with a note in [`StrategyOutcome::notes`] saying why.
pub secondary_ran: bool,
/// Why the page's reading is partial, if it is. The secondary engine is an enhancement,
/// never a gate: when it is configured but fails, the primary reading is kept and the
/// failure is noted here — a page is never skipped because a partner could not read it.
/// Only a primary failure is an [`OcrError`].
pub notes: Vec<String>,
}
/// The note recorded when an ensemble partner fails: the page keeps the primary reading and
/// the cause is named for the report.
fn degraded_note(id: &str, error: &OcrError) -> String {
format!("the {id} engine failed ({error}) — the page keeps the primary reading alone")
}
/// The note recorded when the two readings are too large to merge: the page keeps the primary
/// reading alone and the discarded partner reading is named, never silent.
fn cap_note(id: &str, primary_words: usize, secondary_words: usize) -> String {
format!(
"the readings were too large to merge ({primary_words} × {secondary_words} comparisons, over the cap of {ROVER_MAX_COMPARISONS}) — the page keeps the primary reading alone and the {id} reading is unused"
)
}
/// Run the engines per `strategy` and merge. See [`StrategyOutcome`] for what comes back.
pub fn recognize_with_strategy(
primary: &dyn Recognizer,
secondary: Option<&dyn Recognizer>,
strategy: MergeStrategy,
image: &OcrImage,
options: &RecognizeOptions,
) -> Result<StrategyOutcome, OcrError> {
// The shared shape of the ensemble strategies: the primary has read, the secondary joins
// or fails (a failure degrades to the primary, with a note).
let join = |p: Vec<Line>, sec: &dyn Recognizer| match sec.recognize(image, options) {
// Over the comparison cap the merge is skipped: the secondary's reading was read but
// is not part of the result, so the outcome says so — primary only, honestly not
// merged, with the skip named.
Ok(s) if over_cap(&p, &s) => {
let (primary_words, secondary_words) = (p.iter().map(|l| l.words.len()).sum::<usize>(), s.iter().map(|l| l.words.len()).sum::<usize>());
StrategyOutcome { lines: p, secondary_ran: false, notes: vec![cap_note(sec.id(), primary_words, secondary_words)] }
}
Ok(s) => StrategyOutcome { lines: rover_merge(p, s, sec.id()), secondary_ran: true, notes: Vec::new() },
Err(e) => StrategyOutcome { lines: p, secondary_ran: false, notes: vec![degraded_note(sec.id(), &e)] },
};
match strategy {
MergeStrategy::PrimaryOnly => Ok(StrategyOutcome { lines: primary.recognize(image, options)?, secondary_ran: false, notes: Vec::new() }),
MergeStrategy::RoverVote => match secondary {
None => Ok(StrategyOutcome { lines: primary.recognize(image, options)?, secondary_ran: false, notes: Vec::new() }),
Some(sec) => {
let p = primary.recognize(image, options)?;
Ok(join(p, sec))
}
},
MergeStrategy::ConfidenceWeighted => {
let p = primary.recognize(image, options)?;
match mean_confidence(&p) {
// Confident enough — or an empty reading: the primary stands alone.
Some(mean) if mean >= UNKNOWN_CONFIDENCE_BAR => Ok(StrategyOutcome { lines: p, secondary_ran: false, notes: Vec::new() }),
// A weak mean, or an unknown confidence (words, none reported): the secondary
// runs.
_ => match secondary {
None => Ok(StrategyOutcome { lines: p, secondary_ran: false, notes: Vec::new() }),
Some(sec) => Ok(join(p, sec)),
},
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::atomic::{AtomicUsize, Ordering};
/// A fake engine: returns canned lines, counts recognize calls, never available.
struct Fake {
id: &'static str,
lines: Vec<Line>,
calls: AtomicUsize,
fail: bool,
}
impl Fake {
fn new(id: &'static str, words: &[(&str, [f32; 4], f32)]) -> Fake {
Fake {
id,
lines: vec![Line { words: words.iter().map(|(t, r, c)| Word::new(*t, *r, id).with_confidence(*c)).collect() }],
calls: AtomicUsize::new(0),
fail: false,
}
}
fn empty(id: &'static str) -> Fake {
Fake { id, lines: Vec::new(), calls: AtomicUsize::new(0), fail: false }
}
/// An engine like ocrs: words, but no confidence reported on any of them.
fn unknown(id: &'static str, n: usize) -> Fake {
Fake {
id,
lines: vec![Line {
words: (0..n).map(|i| Word::new(format!("w{i}"), [i as f32 * 10.0, 0.0, i as f32 * 10.0 + 8.0, 10.0], id)).collect(),
}],
calls: AtomicUsize::new(0),
fail: false,
}
}
}
impl Recognizer for Fake {
fn id(&self) -> &'static str {
self.id
}
fn available(&self) -> bool {
!self.fail
}
fn languages(&self) -> Vec<(String, String)> {
vec![]
}
fn recognize(&self, _image: &OcrImage, _options: &RecognizeOptions) -> Result<Vec<Line>, OcrError> {
self.calls.fetch_add(1, Ordering::SeqCst);
if self.fail {
return Err(OcrError::Recognize("the fake engine failed".into()));
}
Ok(self.lines.clone())
}
}
fn image() -> OcrImage {
OcrImage::new(4, 4, vec![255; 64]).unwrap()
}
fn words(lines: &[Line]) -> Vec<&Word> {
lines.iter().flat_map(|l| &l.words).collect()
}
#[test]
fn iou_of_disjoint_touching_and_nan_boxes_is_zero() {
let a = [0.0, 0.0, 10.0, 10.0];
assert_eq!(iou(&a, &[20.0, 0.0, 30.0, 10.0]), 0.0, "disjoint");
assert_eq!(iou(&a, &[10.0, 0.0, 20.0, 10.0]), 0.0, "touching edges are an empty intersection");
assert_eq!(iou(&a, &[f32::NAN, 0.0, 10.0, 10.0]), 0.0, "NaN boxes score 0, not a panic");
assert_eq!(iou(&a, &a), 1.0, "identical boxes");
assert!((iou(&a, &[5.0, 0.0, 15.0, 10.0]) - 1.0 / 3.0).abs() < 1e-9, "half overlap is IoU 1/3");
}
/// With nothing on the primary side, every secondary word is appended under the secondary
/// engine's own name.
#[test]
fn rover_appends_everything_when_the_primary_is_empty() {
let primary = Vec::new();
let secondary = vec![Line { words: vec![Word::new("sole", [0.0, 0.0, 9.0, 9.0], "tesseract").with_confidence(80.0)] }];
let merged = rover_merge(primary, secondary, "tesseract");
assert_eq!(words(&merged).len(), 1);
assert_eq!(merged[0].words[0].text, "sole");
assert_eq!(merged[0].words[0].source, "tesseract", "unclaimed words keep their own engine");
}
/// Non-overlapping secondary words never displace primary words; they are appended.
#[test]
fn rover_appends_non_overlapping_secondary_words() {
let primary = vec![Line { words: vec![Word::new("kept", [0.0, 0.0, 10.0, 10.0], "ocrs").with_confidence(50.0)] }];
let secondary = vec![Line { words: vec![Word::new("extra", [100.0, 100.0, 120.0, 110.0], "tesseract").with_confidence(99.0)] }];
let merged = rover_merge(primary, secondary, "tesseract");
let w = words(&merged);
assert_eq!(w.len(), 2);
assert_eq!((w[0].text.as_str(), w[0].source.as_str()), ("kept", "ocrs"), "the more confident but disjoint word does not win");
assert_eq!((w[1].text.as_str(), w[1].source.as_str()), ("extra", "tesseract"));
}
/// The higher-confidence word of an overlapping pair wins text, box and confidence, and is
/// named ROVER.
#[test]
fn rover_higher_confidence_wins_and_is_named_rover() {
let primary = vec![Line { words: vec![Word::new("helo", [0.0, 0.0, 10.0, 10.0], "ocrs").with_confidence(60.0)] }];
let secondary = vec![Line { words: vec![Word::new("hello", [1.0, 0.0, 11.0, 10.0], "tesseract").with_confidence(95.0)] }];
let merged = rover_merge(primary, secondary, "tesseract");
let w = words(&merged);
assert_eq!(w.len(), 1, "the secondary word was claimed, not appended");
assert_eq!(w[0].text, "hello");
assert_eq!(w[0].rect, [1.0, 0.0, 11.0, 10.0]);
assert_eq!(w[0].confidence, Some(95.0));
assert_eq!(w[0].source, "ROVER");
}
/// A tie keeps the primary word (still named ROVER, as an agreed reading).
#[test]
fn rover_tie_keeps_the_primary() {
let primary = vec![Line { words: vec![Word::new("primary", [0.0, 0.0, 10.0, 10.0], "ocrs").with_confidence(80.0)] }];
let secondary = vec![Line { words: vec![Word::new("secondary", [0.0, 0.0, 10.0, 10.0], "tesseract").with_confidence(80.0)] }];
let merged = rover_merge(primary, secondary, "tesseract");
let w = words(&merged);
assert_eq!(w.len(), 1);
assert_eq!(w[0].text, "primary");
assert_eq!(w[0].source, "ROVER");
}
/// Boxes under the 0.5 IoU bar never merge, even when the secondary is more confident.
#[test]
fn rover_below_half_iou_does_not_merge() {
let primary = vec![Line { words: vec![Word::new("kept", [0.0, 0.0, 10.0, 10.0], "ocrs").with_confidence(50.0)] }];
// IoU 10·10 / (100 + 169 - 25) = 100/244 ≈ 0.41 < 0.5.
let secondary = vec![Line { words: vec![Word::new("other", [5.0, 0.0, 23.0, 20.0], "tesseract").with_confidence(99.0)] }];
let merged = rover_merge(primary, secondary, "tesseract");
let w = words(&merged);
assert_eq!(w.len(), 2);
assert_eq!(w[0].text, "kept");
assert_eq!(w[0].source, "ocrs");
}
/// Merged words keep their line; the merge never panics on NaN boxes.
#[test]
fn rover_survives_nan_boxes() {
let primary = vec![Line { words: vec![Word::new("nan", [f32::NAN, 0.0, 10.0, 10.0], "ocrs").with_confidence(50.0)] }];
let secondary = vec![Line { words: vec![Word::new("real", [0.0, 0.0, 10.0, 10.0], "tesseract").with_confidence(90.0)] }];
let merged = rover_merge(primary, secondary, "tesseract");
let w = words(&merged);
assert_eq!(w.len(), 2, "a NaN primary box merges with nothing");
assert_eq!(w[0].text, "nan");
assert_eq!(w[1].text, "real");
}
/// Only reported confidences count. An empty reading is Some(100) (nothing to
/// double-check); words without a single confidence are UNKNOWN (`None`), never 100.
#[test]
fn mean_confidence_distinguishes_empty_from_unknown() {
assert_eq!(mean_confidence(&[]), Some(100.0));
assert_eq!(mean_confidence(&[Line { words: vec![Word::new("a", [0.0; 4], "ocrs")] }]), None, "words, no confidences: unknown");
let lines = vec![Line {
words: vec![
Word::new("a", [0.0; 4], "e").with_confidence(40.0),
Word::new("b", [0.0; 4], "e").with_confidence(80.0),
Word::new("c", [0.0; 4], "e"),
],
}];
assert_eq!(mean_confidence(&lines), Some(60.0), "reported confidences are averaged");
}
/// ROVER's preference needs a meaningful comparison: a known confidence beats an unknown
/// one (the ocrs primary yields to a confident tesseract reading), and None vs None keeps
/// the primary.
#[test]
fn rover_known_confidence_beats_unknown_and_none_vs_none_keeps_the_primary() {
let primary = vec![Line { words: vec![Word::new("helo", [0.0, 0.0, 10.0, 10.0], "ocrs")] }];
let secondary = vec![Line { words: vec![Word::new("hello", [0.0, 0.0, 10.0, 10.0], "tesseract").with_confidence(95.0)] }];
let merged = rover_merge(primary, secondary, "tesseract");
assert_eq!(
(merged[0].words[0].text.as_str(), merged[0].words[0].confidence, merged[0].words[0].source.as_str()),
("hello", Some(95.0), "ROVER")
);
let primary = vec![Line { words: vec![Word::new("primary", [0.0, 0.0, 10.0, 10.0], "ocrs")] }];
let secondary = vec![Line { words: vec![Word::new("secondary", [0.0, 0.0, 10.0, 10.0], "tesseract")] }];
let merged = rover_merge(primary, secondary, "tesseract");
assert_eq!(
(merged[0].words[0].text.as_str(), merged[0].words[0].source.as_str()),
("primary", "ROVER"),
"unknown vs unknown is not a comparison: the primary stands"
);
}
/// Over the comparison cap the merge is skipped and the primary reading stands alone (a
/// documented known limit, bounding the merge's quadratic core).
#[test]
fn rover_over_the_comparison_cap_keeps_the_primary() {
// 11 000 × 11 000 = 121 M comparisons > the 100 M cap.
let words = |id: &str| (0..11_000).map(|i| Word::new(format!("w{i}"), [i as f32, 0.0, i as f32 + 5.0, 10.0], id)).collect::<Vec<_>>();
let primary = vec![Line { words: words("ocrs") }];
let secondary = vec![Line { words: words("tesseract") }];
let merged = rover_merge(primary, secondary, "tesseract");
assert_eq!(merged.len(), 1);
assert!(merged[0].words.iter().all(|w| w.source == "ocrs"), "every primary word stands, unmerged");
assert_eq!(merged[0].words.len(), 11_000);
// Under the cap, merging still happens (a small overlap is replaced and named ROVER).
let small_primary = vec![Line { words: vec![Word::new("kept", [0.0, 0.0, 10.0, 10.0], "ocrs").with_confidence(50.0)] }];
let small_secondary = vec![Line { words: vec![Word::new("wins", [0.0, 0.0, 10.0, 10.0], "tesseract").with_confidence(95.0)] }];
let merged = rover_merge(small_primary, small_secondary, "tesseract");
assert_eq!((merged[0].words[0].text.as_str(), merged[0].words[0].source.as_str()), ("wins", "ROVER"));
}
/// Over the comparison cap the merge is skipped honestly at the strategy level: the page
/// keeps the primary reading alone, the skip is named in a note, and `secondary_ran` does
/// not claim the discarded reading is part of the result (though the secondary really did
/// run — its work was spent, its reading unused).
#[test]
fn rover_cap_hit_reports_primary_only_with_a_note_and_an_honest_flag() {
let mut primary = Fake::empty("ocrs");
primary.lines =
vec![Line { words: (0..11_000).map(|i| Word::new(format!("w{i}"), [i as f32, 0.0, i as f32 + 5.0, 10.0], "ocrs")).collect() }];
let mut secondary = Fake::empty("tesseract");
secondary.lines =
vec![Line { words: (0..11_000).map(|i| Word::new(format!("w{i}"), [i as f32, 0.0, i as f32 + 5.0, 10.0], "tesseract")).collect() }];
let outcome = recognize_with_strategy(&primary, Some(&secondary), MergeStrategy::RoverVote, &image(), &RecognizeOptions::default()).unwrap();
assert_eq!(secondary.calls.load(Ordering::SeqCst), 1, "the secondary really ran");
assert_eq!(outcome.lines.len(), 1);
assert_eq!(outcome.lines[0].words.len(), 11_000, "primary only");
assert!(outcome.lines[0].words.iter().all(|w| w.source == "ocrs"), "every primary word stands, unmerged");
assert!(!outcome.secondary_ran, "the secondary's reading is not part of the result");
assert_eq!(outcome.notes.len(), 1, "{:?}", outcome.notes);
assert!(outcome.notes[0].contains("merge"), "{:?}", outcome.notes[0]);
assert!(outcome.notes[0].contains("tesseract"), "{:?}", outcome.notes[0]);
// Under the cap the same strategy merges and reports the secondary honestly.
let primary = Fake::new("ocrs", &[("kept", [0.0, 0.0, 10.0, 10.0], 50.0)]);
let secondary = Fake::new("tesseract", &[("wins", [0.0, 0.0, 10.0, 10.0], 95.0)]);
let outcome = recognize_with_strategy(&primary, Some(&secondary), MergeStrategy::RoverVote, &image(), &RecognizeOptions::default()).unwrap();
assert!(outcome.secondary_ran && outcome.notes.is_empty());
assert_eq!((outcome.lines[0].words[0].text.as_str(), outcome.lines[0].words[0].source.as_str()), ("wins", "ROVER"));
}
#[test]
fn primary_only_never_runs_the_secondary() {
let primary = Fake::new("ocrs", &[("w", [0.0; 4], 90.0)]);
let secondary = Fake::empty("tesseract");
let outcome =
recognize_with_strategy(&primary, Some(&secondary), MergeStrategy::PrimaryOnly, &image(), &RecognizeOptions::default()).unwrap();
assert_eq!(primary.calls.load(Ordering::SeqCst), 1);
assert_eq!(secondary.calls.load(Ordering::SeqCst), 0);
assert!(!outcome.secondary_ran);
assert!(outcome.notes.is_empty());
assert_eq!(words(&outcome.lines).len(), 1);
}
#[test]
fn confidence_weighted_runs_the_secondary_only_on_a_weak_primary() {
let weak = Fake::new("ocrs", &[("w", [0.0; 4], 40.0)]);
let strong = Fake::new("ocrs", &[("w", [0.0; 4], 95.0)]);
let secondary = Fake::empty("tesseract");
let outcome =
recognize_with_strategy(&weak, Some(&secondary), MergeStrategy::ConfidenceWeighted, &image(), &RecognizeOptions::default()).unwrap();
assert!(outcome.secondary_ran, "mean 40 < 70 → the secondary joins");
let outcome =
recognize_with_strategy(&strong, Some(&secondary), MergeStrategy::ConfidenceWeighted, &image(), &RecognizeOptions::default()).unwrap();
assert!(!outcome.secondary_ran, "mean 95 >= 70 → primary alone");
// An empty primary counts as 100: no signal, no secondary run.
let empty = Fake::empty("ocrs");
let outcome =
recognize_with_strategy(&empty, Some(&secondary), MergeStrategy::ConfidenceWeighted, &image(), &RecognizeOptions::default()).unwrap();
assert!(!outcome.secondary_ran);
assert_eq!(secondary.calls.load(Ordering::SeqCst), 1, "only the weak page ran the secondary");
}
/// The ocrs engine reports no confidences: unknown is not 100, so the secondary runs —
/// and its known reading wins the overlapping word (the ensemble's whole point).
#[test]
fn confidence_weighted_runs_the_secondary_on_an_unknown_primary() {
let ocrs_like = Fake::unknown("ocrs", 1);
let secondary = Fake::new("tesseract", &[("hello", [0.0, 0.0, 10.0, 10.0], 95.0)]);
let outcome =
recognize_with_strategy(&ocrs_like, Some(&secondary), MergeStrategy::ConfidenceWeighted, &image(), &RecognizeOptions::default()).unwrap();
assert!(outcome.secondary_ran, "an unknown confidence asks the secondary");
assert_eq!((outcome.lines[0].words[0].text.as_str(), outcome.lines[0].words[0].confidence), ("hello", Some(95.0)));
assert_eq!(outcome.lines[0].words[0].source, "ROVER");
assert_eq!(secondary.calls.load(Ordering::SeqCst), 1);
}
/// Against a primary word of unknown confidence, a secondary word wins only when it is
/// confident (at or above the weak-reading bar): a 5% reading does not replace a word nobody
/// scored, a 70% one does. The merged word is named ROVER either way.
#[test]
fn an_unknown_primary_word_yields_only_to_a_confident_secondary() {
let merge = |conf: f32| {
let secondary = vec![Line { words: vec![Word::new("other", [0.0, 0.0, 10.0, 10.0], "tesseract").with_confidence(conf)] }];
let primary = Fake::unknown("ocrs", 1).lines;
rover_merge(primary, secondary, "tesseract").remove(0).words.remove(0)
};
let weak = merge(5.0);
assert_eq!((weak.text.as_str(), weak.confidence, weak.source.as_str()), ("w0", None, "ROVER"), "a 5% word keeps the primary");
let below = merge(UNKNOWN_CONFIDENCE_BAR - 0.5);
assert_eq!(below.text, "w0");
let at_bar = merge(UNKNOWN_CONFIDENCE_BAR);
assert_eq!((at_bar.text.as_str(), at_bar.confidence), ("other", Some(UNKNOWN_CONFIDENCE_BAR)), "a confident word wins");
let nan = merge(f32::NAN);
assert_eq!(nan.text, "w0", "NaN (unknown after clamping) never wins");
}
#[test]
fn rover_vote_always_runs_both_and_degrades_without_a_secondary() {
let primary = Fake::new("ocrs", &[("w", [0.0; 4], 95.0)]);
let secondary = Fake::empty("tesseract");
let outcome = recognize_with_strategy(&primary, Some(&secondary), MergeStrategy::RoverVote, &image(), &RecognizeOptions::default()).unwrap();
assert!(outcome.secondary_ran);
assert_eq!(secondary.calls.load(Ordering::SeqCst), 1);
// No secondary: primary only.
let outcome = recognize_with_strategy(&primary, None, MergeStrategy::RoverVote, &image(), &RecognizeOptions::default()).unwrap();
assert!(!outcome.secondary_ran);
}
/// A configured secondary that fails degrades to the primary reading, with the failure
/// recorded as a note — never a skipped page, never a silently partial reading without a
/// trace. Both ensemble strategies degrade; a primary failure is still an error.
#[test]
fn a_failing_secondary_degrades_to_the_primary_with_a_note() {
let mut secondary = Fake::empty("tesseract");
secondary.fail = true;
let primary = Fake::new("ocrs", &[("w", [0.0; 4], 95.0)]);
let outcome = recognize_with_strategy(&primary, Some(&secondary), MergeStrategy::RoverVote, &image(), &RecognizeOptions::default()).unwrap();
assert_eq!(words(&outcome.lines).len(), 1, "the primary reading stands");
assert_eq!(outcome.lines[0].words[0].text, "w");
assert!(!outcome.secondary_ran);
assert_eq!(outcome.notes.len(), 1);
assert!(outcome.notes[0].contains("tesseract"), "{:?}", outcome.notes[0]);
// The confidence-weighted strategy degrades the same way on its weak path.
let weak = Fake::new("ocrs", &[("w", [0.0; 4], 40.0)]);
let outcome =
recognize_with_strategy(&weak, Some(&secondary), MergeStrategy::ConfidenceWeighted, &image(), &RecognizeOptions::default()).unwrap();
assert_eq!(words(&outcome.lines).len(), 1);
assert_eq!(outcome.notes.len(), 1, "{:?}", outcome.notes);
// A failing primary is still an error: there is nothing to degrade to.
let mut failing_primary = Fake::new("ocrs", &[("w", [0.0; 4], 95.0)]);
failing_primary.fail = true;
let r = recognize_with_strategy(&failing_primary, Some(&secondary), MergeStrategy::RoverVote, &image(), &RecognizeOptions::default());
assert!(r.is_err());
}
}
crates/ocr/src/ocrs.rs
+251
View File
@@ -0,0 +1,251 @@
//! The pure-Rust engine: the ocrs crate with its pre-trained models, wrapped as a
//! [`Recognizer`] (`OcrsRecognizer`). Reading is offline and local; the models do not report a
//! confidence, so its words have `confidence: None`.
use std::path::{Path, PathBuf};
use super::{Line, MAX_PIXELS, MAX_SIDE, OcrError, OcrImage, RecognizeOptions, Recognizer, Word};
/// The validated byte length of a `width` × `height` RGBA raster, with the same caps
/// [`OcrImage::new`] applies: zero sides are empty, sides over [`MAX_SIDE`] or totals over
/// [`MAX_PIXELS`] are too large. `Ocr::recognize` takes raw dimensions (they come from
/// callers, ultimately from documents, so they are untrusted), and the naive
/// `width * height * 4` overflows `u32` for absurd values — the caps are checked first, in
/// `usize`, so nothing overflows and nothing huge is ever touched.
fn checked_len(width: u32, height: u32) -> Result<usize, OcrError> {
if width == 0 || height == 0 {
return Err(OcrError::EmptyImage);
}
if width > MAX_SIDE || height > MAX_SIDE {
return Err(OcrError::ImageTooLarge);
}
let pixels = (width as usize).checked_mul(height as usize).ok_or(OcrError::ImageTooLarge)?;
if pixels > MAX_PIXELS as usize {
return Err(OcrError::ImageTooLarge);
}
pixels.checked_mul(4).ok_or(OcrError::ImageTooLarge)
}
/// The languages the models read (ISO 639-1); all use the Latin alphabet without accents.
pub const LANGUAGES: &[(&str, &str)] = &[("en", "English")];
/// The model files, as named in ATTRIBUTION.toml.
pub const DETECTION_MODEL: &str = "text-detection.onnx";
pub const RECOGNITION_MODEL: &str = "text-recognition.onnx";
pub const ALPHABET: &str = " 0123456789!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~€ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz";
/// Where the two model files are.
#[derive(Clone, Debug, PartialEq)]
pub struct Models {
pub detection: PathBuf,
pub recognition: PathBuf,
}
impl Models {
pub fn dirs_beside_exe(exe_dir: &Path) -> Vec<PathBuf> {
vec![exe_dir.join("models"), exe_dir.join("../Resources/models"), exe_dir.join("../share/pdfcraft/models")]
}
/// The models in `dir`, preferring current ONNX files over legacy RTEN files. An incomplete
/// update never mixes formats or hides an intact legacy pair.
pub fn in_dir(dir: &Path) -> Option<Models> {
[(DETECTION_MODEL, RECOGNITION_MODEL), ("text-detection.rten", "text-recognition.rten")].into_iter().find_map(|(detection, recognition)| {
let m = Models { detection: dir.join(detection), recognition: dir.join(recognition) };
(m.detection.is_file() && m.recognition.is_file()).then_some(m)
})
}
/// Look for the models: `$PDFCRAFT_MODELS`, then `models/` beside the executable (and
/// `Resources/models` in a macOS bundle), then the source tree's `assets/models/`.
pub fn find() -> Option<Models> {
Self::search_dirs().iter().find_map(|d| Self::in_dir(d))
}
/// The directories [`Models::find`] looks in, in order.
pub fn search_dirs() -> Vec<PathBuf> {
let mut dirs = Vec::new();
if let Some(d) = std::env::var_os("PDFCRAFT_MODELS") {
dirs.push(PathBuf::from(d));
}
if let Some(exe) = std::env::current_exe().ok().and_then(|e| e.parent().map(Path::to_path_buf)) {
dirs.extend(Self::dirs_beside_exe(&exe));
}
dirs.push(Path::new(env!("CARGO_MANIFEST_DIR")).join("../../assets/models"));
dirs
}
}
/// A loaded recogniser. Loading takes a moment; keep one and reuse it.
pub struct Ocr {
engine: ocrs::OcrEngine,
}
impl Ocr {
pub fn load(models: &Models) -> Result<Ocr, OcrError> {
let load = |p: &Path| rten::Model::load_file(p).map_err(|e| OcrError::Load(p.display().to_string(), e.to_string()));
let params = ocrs::OcrEngineParams {
detection_model: Some(load(&models.detection)?),
recognition_model: Some(load(&models.recognition)?),
alphabet: Some(ALPHABET.into()),
..Default::default()
};
let engine = ocrs::OcrEngine::new(params).map_err(|e| OcrError::Load("ocr engine".into(), e.to_string()))?;
Ok(Ocr { engine })
}
/// Load the models found by [`Models::find`].
pub fn find() -> Result<Ocr, OcrError> {
Self::load(&Models::find().ok_or(OcrError::NoModels)?)
}
/// Recognise the text in an RGBA (or RGB, or grey) image, `width` × `height` pixels.
pub fn recognize(&self, pixels: &[u8], width: u32, height: u32) -> Result<Vec<Line>, OcrError> {
if pixels.is_empty() {
return Err(OcrError::EmptyImage);
}
let want = checked_len(width, height)?;
// ocrs wants 1 or 3 channels.
let rgb: std::borrow::Cow<[u8]> = if pixels.len() == want {
pixels.as_chunks::<4>().0.iter().flat_map(|p| [p[0], p[1], p[2]]).collect::<Vec<u8>>().into()
} else {
pixels.into()
};
let err = |e: &dyn std::fmt::Display| OcrError::Recognize(e.to_string());
let source = ocrs::ImageSource::from_bytes(&rgb, (width, height)).map_err(|e| err(&e))?;
let input = self.engine.prepare_input(source).map_err(|e| err(&e))?;
let found = self.engine.detect_words(&input).map_err(|e| err(&e))?;
let lines = self.engine.find_text_lines(&input, &found);
let read = self.engine.recognize_text(&input, &lines).map_err(|e| err(&e))?;
use ocrs::TextItem;
Ok(read
.into_iter()
.flatten()
.map(|line| Line {
words: line
.words()
.map(|w| {
let r = w.bounding_rect();
Word::new(w.to_string(), [r.left() as f32, r.top() as f32, r.right() as f32, r.bottom() as f32], "ocrs")
})
.filter(|w| !w.text.trim().is_empty())
.collect(),
})
.filter(|l| !l.words.is_empty())
.collect())
}
}
/// The ocrs engine as a [`Recognizer`] (the default engine).
pub struct OcrsRecognizer {
engine: Ocr,
}
impl OcrsRecognizer {
/// Load the models from `models`.
pub fn load(models: &Models) -> Result<OcrsRecognizer, OcrError> {
Ok(OcrsRecognizer { engine: Ocr::load(models)? })
}
/// Load the models found by [`Models::find`].
pub fn find() -> Result<OcrsRecognizer, OcrError> {
Self::load(&Models::find().ok_or(OcrError::NoModels)?)
}
}
impl Recognizer for OcrsRecognizer {
fn id(&self) -> &'static str {
"ocrs"
}
fn available(&self) -> bool {
Models::find().is_some()
}
fn languages(&self) -> Vec<(String, String)> {
LANGUAGES.iter().map(|(c, n)| ((*c).to_string(), (*n).to_string())).collect()
}
fn recognize(&self, image: &OcrImage, _options: &RecognizeOptions) -> Result<Vec<Line>, OcrError> {
self.engine.recognize(image.rgba(), image.width(), image.height())
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Raw dimensions are untrusted (they come from callers, ultimately from documents): zero
/// sides, sides or pixel totals over the caps, and products that would overflow the naive
/// `width * height * 4` are errors, never panics, before anything is touched.
#[test]
fn raw_dimensions_are_capped_and_checked_not_panicked_at() {
assert!(matches!(checked_len(0, 10), Err(OcrError::EmptyImage)));
assert!(matches!(checked_len(10, 0), Err(OcrError::EmptyImage)));
assert!(matches!(checked_len(MAX_SIDE + 1, 1), Err(OcrError::ImageTooLarge)));
// 65536² would wrap u32 in the old `width * height * 4`; the side cap refuses it first.
assert!(matches!(checked_len(u32::MAX, u32::MAX), Err(OcrError::ImageTooLarge)));
// Legal sides whose product is over the 64 MP area cap.
assert!(matches!(checked_len(MAX_SIDE, 7000), Err(OcrError::ImageTooLarge)));
assert_eq!(checked_len(2, 2).unwrap(), 16);
// Exactly the area cap fits: 8000 × 8000 = 64 MP.
assert_eq!(checked_len(8000, 8000).unwrap(), 8000usize * 8000 * 4);
}
/// The current ONNX pair wins when an installation also retains both legacy models.
#[test]
fn model_discovery_prefers_the_complete_onnx_pair() {
let dir = tempfile::tempdir().unwrap();
for file in ["text-detection.onnx", "text-recognition.onnx"] {
std::fs::write(dir.path().join(file), b"model").unwrap();
}
let expected = Some(Models { detection: dir.path().join("text-detection.onnx"), recognition: dir.path().join("text-recognition.onnx") });
assert_eq!(Models::in_dir(dir.path()), expected, "a complete ONNX pair is sufficient");
for file in ["text-detection.rten", "text-recognition.rten"] {
std::fs::write(dir.path().join(file), b"model").unwrap();
}
assert_eq!(Models::in_dir(dir.path()), expected, "the complete ONNX pair takes precedence over legacy models");
}
/// An interrupted ONNX update still permits an intact legacy pair in the same directory.
#[test]
fn model_discovery_falls_back_to_legacy_when_the_onnx_pair_is_incomplete() {
for current in ["text-detection.onnx", "text-recognition.onnx"] {
let dir = tempfile::tempdir().unwrap();
for file in [current, "text-detection.rten", "text-recognition.rten"] {
std::fs::write(dir.path().join(file), b"model").unwrap();
}
assert_eq!(
Models::in_dir(dir.path()),
Some(Models { detection: dir.path().join("text-detection.rten"), recognition: dir.path().join("text-recognition.rten") }),
"{current} alone must not hide a complete legacy pair"
);
}
}
/// Detection and recognition must come from one complete format pair, never a mixed update.
#[test]
fn model_discovery_refuses_mixed_format_pairs() {
for pair in [["text-detection.onnx", "text-recognition.rten"], ["text-detection.rten", "text-recognition.onnx"]] {
let dir = tempfile::tempdir().unwrap();
for file in pair {
std::fs::write(dir.path().join(file), b"model").unwrap();
}
assert_eq!(Models::in_dir(dir.path()), None, "{pair:?} is not a complete model pair");
}
}
/// Existing source builds and PDFCRAFT_MODELS overrides retain their complete RTEN pair.
#[test]
fn model_discovery_keeps_legacy_only_installations() {
let dir = tempfile::tempdir().unwrap();
assert_eq!(Models::in_dir(dir.path()), None);
std::fs::write(dir.path().join("text-detection.rten"), b"model").unwrap();
assert_eq!(Models::in_dir(dir.path()), None, "one legacy model is incomplete");
std::fs::write(dir.path().join("text-recognition.rten"), b"model").unwrap();
assert_eq!(
Models::in_dir(dir.path()),
Some(Models { detection: dir.path().join("text-detection.rten"), recognition: dir.path().join("text-recognition.rten") })
);
}
}
crates/ocr/src/preprocess.rs
+973
View File
@@ -0,0 +1,973 @@
//! Preprocessing: a pure function between the page raster and recognition, in fixed order —
//! (a) DPI normalization, then the four opt-in steps (b) orientation, (c) deskew, (d) denoise,
//! (e) binarization. Normalization exists for those steps (their windows and thresholds are
//! calibrated for 300 dpi), so it runs only when at least one of them is on: with every step
//! off the raster is handed to the engines exactly as it was rendered. (Enlarging a render
//! on its own changes what the models read: a 72 dpi scan of "due: €250" read as "ue: ?250"
//! after a 4× bilinear enlargement.) Everything is plain Rust on the RGBA raster: no
//! Leptonica, no process, no new dependencies.
//!
//! Every geometric step records its inverse; the result carries one exact
//! [`InverseTransform`] (an affine map, composed in f64), and callers map every word box
//! through it back to the *original* raster coordinates before placing the words.
//!
//! The heuristics are deliberately conservative and panic-free: an image too small to judge,
//! a blank page, or a score without a clear margin is a no-op, never a guess.
use crate::{MAX_PIXELS, MAX_SIDE, OcrError, OcrImage};
/// The four opt-in steps. All off by default: a scan is passed through as rendered.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub struct PreprocessOptions {
/// Detect pages scanned at 90°/180°/270° and turn them upright.
pub auto_rotate: bool,
/// Straighten skewed text lines.
pub deskew: bool,
/// 3×3 median filter (salt-and-pepper noise).
pub denoise: bool,
/// Sauvola binarization (window 8, k 0.34).
pub binarize: bool,
}
impl PreprocessOptions {
/// Whether any step is on (and so the raster is normalized to 300 dpi first).
pub fn any(&self) -> bool {
self.auto_rotate || self.deskew || self.denoise || self.binarize
}
}
/// A preprocessed raster: the image to recognize, the exact way back to the input raster, and
/// the resolution the image now stands for.
#[derive(Debug)]
pub struct Preprocessed {
pub image: OcrImage,
/// Maps pixel coordinates in [`Preprocessed::image`] back to the input raster's.
pub inverse_transform: InverseTransform,
/// The image's resolution in dots per inch: 300 after normalization (only when a step is
/// on), or the source's own
/// when it already rendered at 300 or more.
pub dpi: f32,
}
/// Preprocess `image` (rendered at `source_dpi`) per `options`, in the fixed order. The image
/// is consumed: the common path (every step off, or a no-op) hands the raster back without
/// copying it, and a step that rewrites pixels replaces it.
pub fn preprocess(image: OcrImage, source_dpi: f32, options: &PreprocessOptions) -> Result<Preprocessed, OcrError> {
let mut current = image;
let mut inverse = InverseTransform::identity();
// (a) DPI normalization: for the steps below, a scan under 300 dpi is enlarged by 300/src so
// they see letter-sized strokes. With every step off nothing is resampled. A broken dpi
// (zero, negative, NaN) is left alone.
let mut dpi = if source_dpi.is_finite() && source_dpi >= 1.0 { source_dpi } else { 300.0 };
if options.any() && dpi < 300.0 {
let scale = f64::from(300.0 / dpi);
current = resize_bilinear(&current, scale)?;
inverse = compose(&inverse, &InverseTransform::scale(1.0 / scale));
dpi = 300.0;
}
// (b) Orientation: turn 90/180/270 scans upright (opt-in, skipped for small images).
if options.auto_rotate {
let (w, h) = (current.width, current.height);
let turns = detect_orientation(&current);
if turns != 0 {
current = rotate_quarter_turns(&current, turns);
inverse = compose(&inverse, &inverse_quarter_turns(turns, w, h));
}
}
// (c) Deskew: straighten skewed lines (opt-in; no-op under 64 px, blank, or without a
// clear margin or below 0.1°).
if options.deskew
&& let Some(angle) = estimate_skew(&current)
{
let (straightened, back) = deskew_rotate(&current, angle);
current = straightened;
inverse = compose(&inverse, &back);
}
// (d) Denoise: 3×3 median with edge clamping (opt-in; dimensions never change).
if options.denoise {
current = median3(&current);
}
// (e) Binarize: Sauvola (window 8, k 0.34) via an integral image (opt-in); dark ink stays
// dark and paper stays light.
if options.binarize {
current = sauvola(&current);
}
Ok(Preprocessed { image: current, inverse_transform: inverse, dpi })
}
/// An affine map `(x, y) → (a·x + c·y + e, b·x + d·y + f)`, composed in f64 so mapping a
/// word box back is exact up to float rounding.
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct InverseTransform {
a: f64,
b: f64,
c: f64,
d: f64,
e: f64,
f: f64,
}
impl InverseTransform {
pub fn identity() -> InverseTransform {
InverseTransform { a: 1.0, b: 0.0, c: 0.0, d: 1.0, e: 0.0, f: 0.0 }
}
fn scale(s: f64) -> InverseTransform {
InverseTransform { a: s, b: 0.0, c: 0.0, d: s, e: 0.0, f: 0.0 }
}
/// `compose(outer, inner)` maps `p` to `outer(inner(p))`: the accumulated inverse of the
/// earlier steps is the outer map (their spaces come later in the round trip), so each
/// step calls `compose(&inverse, &this_step_inverse)`. The scale step, for instance,
/// happens first, so its inverse is applied last: `scale⁻¹(rotate⁻¹(p))`.
fn compose_(outer: &InverseTransform, inner: &InverseTransform) -> InverseTransform {
InverseTransform {
a: outer.a * inner.a + outer.c * inner.b,
b: outer.b * inner.a + outer.d * inner.b,
c: outer.a * inner.c + outer.c * inner.d,
d: outer.b * inner.c + outer.d * inner.d,
e: outer.a * inner.e + outer.c * inner.f + outer.e,
f: outer.b * inner.e + outer.d * inner.f + outer.f,
}
}
/// Map a point from the processed image back to the input raster.
pub fn apply(&self, x: f64, y: f64) -> (f64, f64) {
(self.a * x + self.c * y + self.e, self.b * x + self.d * y + self.f)
}
/// Map a word box `[left, top, right, bottom]` back to the input raster: the four corners
/// go through [`Self::apply`] and their bounding box is rounded outward to whole pixels.
/// A box with no finite corners comes back unchanged (garbage in, garbage out — never a
/// panic).
pub fn map_rect(&self, rect: [f32; 4]) -> [f32; 4] {
let corners = [(rect[0], rect[1]), (rect[2], rect[1]), (rect[0], rect[3]), (rect[2], rect[3])];
let mut lo = (f64::INFINITY, f64::INFINITY);
let mut hi = (f64::NEG_INFINITY, f64::NEG_INFINITY);
let mut finite = false;
for (x, y) in corners {
if !(x.is_finite() && y.is_finite()) {
continue;
}
let (mx, my) = self.apply(f64::from(x), f64::from(y));
if mx.is_finite() && my.is_finite() {
finite = true;
lo = (lo.0.min(mx), lo.1.min(my));
hi = (hi.0.max(mx), hi.1.max(my));
}
}
if !finite {
return rect;
}
[lo.0.floor() as f32, lo.1.floor() as f32, hi.0.ceil() as f32, hi.1.ceil() as f32]
}
}
/// The step inverses compose in pipeline order: each new step's inverse is applied *after*
/// the inverses accumulated so far.
fn compose(outer: &InverseTransform, inner: &InverseTransform) -> InverseTransform {
InverseTransform::compose_(outer, inner)
}
/// The size `width` x `height` becomes under `scale`, refused (before any buffer exists) when
/// a side is under one pixel or over [`MAX_SIDE`], or the area is over [`MAX_PIXELS`]: two
/// sides under the side cap can still ask for hundreds of megabytes, which `OcrImage::new`
/// would only refuse after they were allocated.
fn scaled_size(width: u32, height: u32, scale: f64) -> Result<(u32, u32), OcrError> {
let new_w = (f64::from(width) * scale).ceil();
let new_h = (f64::from(height) * scale).ceil();
if !(1.0..=f64::from(MAX_SIDE)).contains(&new_w) || !(1.0..=f64::from(MAX_SIDE)).contains(&new_h) {
return Err(OcrError::ImageTooLarge);
}
let (nw, nh) = (new_w as u32, new_h as u32);
if (nw as usize).checked_mul(nh as usize).is_none_or(|pixels| pixels > MAX_PIXELS as usize) {
return Err(OcrError::ImageTooLarge);
}
Ok((nw, nh))
}
/// Bilinear resize by `scale`, clamped at the edges; the result stays inside [`scaled_size`]'s
/// limits (the side cap and the area cap).
fn resize_bilinear(image: &OcrImage, scale: f64) -> Result<OcrImage, OcrError> {
let (nw, nh) = scaled_size(image.width, image.height, scale)?;
let mut out = vec![255u8; (nw as usize).checked_mul(nh as usize).and_then(|p| p.checked_mul(4)).ok_or(OcrError::ImageTooLarge)?];
for (y, row) in out.chunks_exact_mut(4 * nw as usize).enumerate() {
for (x, px) in row.as_chunks_mut::<4>().0.iter_mut().enumerate() {
// The source position of the destination pixel's centre, and the four
// neighbours bilinear interpolation blends (positions clamp at the edges).
let sx = (x as f64 + 0.5) / scale - 0.5;
let sy = (y as f64 + 0.5) / scale - 0.5;
let (x0, y0) = (sx.floor().max(0.0), sy.floor().max(0.0));
let (fx, fy) = (sx - x0, sy - y0);
for (ch, o) in px.iter_mut().enumerate() {
let blend = |dx: f64, dy: f64| f64::from(image.byte(x0 + dx, y0 + dy, ch));
let v = blend(0.0, 0.0) * (1.0 - fx) * (1.0 - fy)
+ blend(1.0, 0.0) * fx * (1.0 - fy)
+ blend(0.0, 1.0) * (1.0 - fx) * fy
+ blend(1.0, 1.0) * fx * fy;
*o = v.round().clamp(0.0, 255.0) as u8;
}
}
}
OcrImage::new(nw, nh, out)
}
/// Grayscale (Rec.601 luma) of every pixel.
fn grayscale(image: &OcrImage) -> Vec<u8> {
image.rgba.as_chunks::<4>().0.iter().map(|p| ((299 * p[0] as u32 + 587 * p[1] as u32 + 114 * p[2] as u32) / 1000).min(255) as u8).collect()
}
/// How the page must be turned (quarter turns, clockwise) to read upright.
type QuarterTurns = u32;
/// Detect the correction for pages scanned sideways or upside down. Method (documented,
/// panic-free, no OCR): ink is everything darker than the midpoint between the darkest and
/// lightest pixel; for each candidate correction (0/90/180/270 clockwise) the ink's row
/// profile is taken and scored by how sharply it bands (Σ Δ², the projection-profile score:
/// upright text lines band sharply, turned text does not). The best-banding axis wins when it
/// beats the other axis by a factor of 1.3; within the winning axis the line bands'
/// bottom-heaviness (real text carries more ink below each line's middle — baselines and
/// descenders) picks the quarter turn, with a 4%-of-ink margin. Anything less clear is a
/// no-op (`0`), and images under 32 px are never judged.
fn detect_orientation(image: &OcrImage) -> QuarterTurns {
if image.min_side() < 32 {
return 0;
}
let gray = grayscale(image);
let (Some(lo), Some(hi)) = (gray.iter().min(), gray.iter().max()) else { return 0 };
let t = (*lo as u32 + *hi as u32) / 2;
let ink: Vec<bool> = gray.iter().map(|&g| (g as u32) < t).collect();
let (w, h) = (image.width as usize, image.height as usize);
let total: u64 = ink.iter().filter(|i| **i).count() as u64;
if total == 0 {
return 0;
}
// The ink grid rotated by k quarter turns (clockwise), as row profiles.
let mut scores = [0.0f64; 4];
let mut asyms = [0.0f64; 4];
for k in 0u32..4 {
let k = k as usize;
let profile = rotated_row_profile(&ink, w, h, k as u32);
scores[k] = profile.windows(2).map(|p| (p[1] as f64 - p[0] as f64).powi(2)).sum();
asyms[k] = band_bottom_heaviness(&profile);
}
// 0 and 180 share one profile shape (reversed), as do 90 and 270: pick the axis first.
let (vertical_pair, horizontal_pair) = ((scores[0], scores[2]), (scores[1], scores[3]));
let row_axis = vertical_pair.0.max(vertical_pair.1);
let column_axis = horizontal_pair.0.max(horizontal_pair.1);
if row_axis >= column_axis {
if row_axis < column_axis * 1.3 {
return 0; // the axes are too close to call
}
if asyms[0] > 0.15 * total as f64 {
0
} else if asyms[2] > 0.15 * total as f64 {
2
} else {
0 // the bands are not clearly bottom-heavy either way: no-op
}
} else {
if column_axis < row_axis * 1.3 {
return 0;
}
if asyms[1] > 0.15 * total as f64 {
1
} else if asyms[3] > 0.15 * total as f64 {
3
} else {
0
}
}
}
/// The row profile of the ink grid after `k` quarter turns clockwise.
fn rotated_row_profile(ink: &[bool], w: usize, h: usize, k: u32) -> Vec<u64> {
// The rotated image is `w` pixels tall for odd k and `h` for even k; its row index is
// always the new y of the source pixel.
let len = if k.is_multiple_of(2) { h } else { w };
let mut profile = vec![0u64; len];
for (y, row) in ink.chunks_exact(w).enumerate() {
for (x, on) in row.iter().enumerate() {
if !on {
continue;
}
// Where does (x, y) land after k quarter turns clockwise?
let ny = match k {
1 => x,
2 => h - 1 - y,
3 => w - 1 - x,
_ => y,
};
if let Some(p) = profile.get_mut(ny) {
*p += 1;
}
}
}
profile
}
/// Sum over text-line bands of (ink in the band's lower half − ink in its upper half):
/// positive when the bands carry more ink near their bottoms, as real text does.
fn band_bottom_heaviness(profile: &[u64]) -> f64 {
let mut asym = 0.0;
let mut start: Option<usize> = None;
for (i, &p) in profile.iter().chain(std::iter::once(&0)).enumerate() {
match (start, p > 0) {
(None, true) => start = Some(i),
(Some(s), false) => {
let mid = (s + i) / 2;
let lower: u64 = profile.get(s..i).map_or(0, |b| b[mid.saturating_sub(s)..].iter().sum());
let upper: u64 = profile.get(s..i).map_or(0, |b| b[..mid.saturating_sub(s)].iter().sum());
asym += lower as f64 - upper as f64; // signed: a top-heavy band scores below zero
start = None;
}
_ => {}
}
}
asym
}
/// Rotate the raster `k` quarter turns clockwise (an exact pixel permutation).
fn rotate_quarter_turns(image: &OcrImage, k: QuarterTurns) -> OcrImage {
let (w, h) = (image.width as usize, image.height as usize);
let (nw, nh) = if k.is_multiple_of(2) { (image.width, image.height) } else { (image.height, image.width) };
let mut out = vec![255u8; nw as usize * nh as usize * 4];
for (y, row) in image.rgba.chunks_exact(w * 4).enumerate() {
for (x, px) in row.as_chunks::<4>().0.iter().enumerate() {
let (nx, ny) = match k {
1 => (h - 1 - y, x),
2 => (w - 1 - x, h - 1 - y),
3 => (y, w - 1 - x),
_ => (x, y),
};
if let Some(dst) = out.get_mut((ny * nw as usize + nx) * 4..(ny * nw as usize + nx) * 4 + 4) {
dst.copy_from_slice(px);
}
}
}
OcrImage::new(nw, nh, out).unwrap_or_else(|_| image.clone())
}
/// The inverse of a `k`-quarter-turn correction: where each corrected pixel came from.
fn inverse_quarter_turns(k: QuarterTurns, old_w: u32, old_h: u32) -> InverseTransform {
let (w, h) = (f64::from(old_w), f64::from(old_h));
match k {
// Forward (content moves): (x, y) ↦ (h−1−y, x) for one turn clockwise.
1 => InverseTransform { a: 0.0, b: -1.0, c: 1.0, d: 0.0, e: 0.0, f: h - 1.0 },
2 => InverseTransform { a: -1.0, b: 0.0, c: 0.0, d: -1.0, e: w - 1.0, f: h - 1.0 },
3 => InverseTransform { a: 0.0, b: 1.0, c: -1.0, d: 0.0, e: w - 1.0, f: 0.0 },
_ => InverseTransform::identity(),
}
}
/// Estimate the skew of the text lines, in degrees (−5…5, positive: lines run downhill to the
/// right). Method (documented, panic-free): for each candidate angle the ink is sheared into a
/// row profile (`r = y − (x − cx)·tan φ`); the projection-profile score (Σ profile²) peaks
/// where the shear realigns the lines. `None` — leave the page alone — when the image is under
/// 64 px on a side, blank, the winner is under 0.1°, or the winner does not clearly beat
/// leaving the page as is (score < 1.02 × the 0° score).
fn estimate_skew(image: &OcrImage) -> Option<f64> {
if image.min_side() < 64 {
return None;
}
let gray = grayscale(image);
let (Some(lo), Some(hi)) = (gray.iter().min(), gray.iter().max()) else { return None };
let t = (*lo as u32 + *hi as u32) / 2;
let ink: Vec<bool> = gray.iter().map(|&g| (g as u32) < t).collect();
if !ink.iter().any(|i| *i) {
return None; // a blank page has no skew
}
let (w, h) = (image.width, image.height);
let cx = f64::from(w) / 2.0;
let max_shift = (f64::from(w) / 2.0 * 5f64.to_radians().tan()).ceil() as i64 + 2;
let score = |phi_deg: f64| -> f64 {
let slope = phi_deg.to_radians().tan();
let mut profile = vec![0u64; (h as i64 + 2 * max_shift + 1) as usize];
for (y, row) in ink.chunks_exact(w as usize).enumerate() {
for (x, on) in row.iter().enumerate() {
if !on {
continue;
}
let r = y as f64 - (x as f64 - cx) * slope;
let idx = r.round() as i64 + max_shift;
if let Some(p) = profile.get_mut(idx as usize) {
*p += 1;
}
}
}
profile.iter().map(|p| (*p as f64).powi(2)).sum()
};
let mut best = (0.0f64, score(0.0));
let mut phi = -5.0;
while phi <= 5.0 + 1e-9 {
let s = score(phi);
if s > best.1 {
best = (phi, s);
}
phi += 0.25;
}
let (angle, best_score) = best;
if angle.abs() < 0.1 || best_score < 1.02 * score(0.0) {
return None; // straight enough, or no clear winner: no-op
}
Some(angle)
}
/// Rotate the content by `−angle` about the centre (the deskew correction), nearest-neighbour
/// with white fill; also returns the exact dest→src map (the correction's inverse).
fn deskew_rotate(image: &OcrImage, angle_deg: f64) -> (OcrImage, InverseTransform) {
let phi = angle_deg.to_radians();
let (cos, sin) = (phi.cos(), phi.sin());
let (w, h) = (image.width, image.height);
let (cx, cy) = (f64::from(w) / 2.0, f64::from(h) / 2.0);
let back = InverseTransform { a: cos, b: sin, c: -sin, d: cos, e: cx - cos * cx + sin * cy, f: cy - sin * cx - cos * cy };
let mut out = vec![255u8; w as usize * h as usize * 4];
for (y, row) in out.chunks_exact_mut(w as usize * 4).enumerate() {
for (x, px) in row.as_chunks_mut::<4>().0.iter_mut().enumerate() {
let (sx, sy) = back.apply(x as f64, y as f64);
let (xi, yi) = (sx.round() as i64, sy.round() as i64);
if xi < 0 || yi < 0 || xi >= w as i64 || yi >= h as i64 {
continue; // outside the scan: white fill
}
if let Some(src) = image.rgba.get((yi as usize * w as usize + xi as usize) * 4..(yi as usize * w as usize + xi as usize) * 4 + 4) {
px.copy_from_slice(src);
}
}
}
(OcrImage::new(w, h, out).unwrap_or_else(|_| image.clone()), back)
}
/// 3×3 median filter on each colour channel, edges clamped (border pixels replicate);
/// dimensions and alpha are unchanged.
fn median3(image: &OcrImage) -> OcrImage {
let (w, h) = (image.width as usize, image.height as usize);
let mut out = image.rgba.clone();
for (i, dst) in out.as_chunks_mut::<4>().0.iter_mut().enumerate() {
let (x, y) = (i % w, i / w);
let mut rgba = [255u8; 3];
for (ch, v) in rgba.iter_mut().enumerate() {
let mut window = [0u8; 9];
for (n, (dx, dy)) in [(-1i64, -1i64), (0, -1), (1, -1), (-1, 0), (0, 0), (1, 0), (-1, 1), (0, 1), (1, 1)].iter().enumerate() {
let nx = ((x as i64) + dx).clamp(0, w as i64 - 1) as usize;
let ny = ((y as i64) + dy).clamp(0, h as i64 - 1) as usize;
window[n] = image.byte(nx as f64, ny as f64, ch);
}
window.sort_unstable();
*v = window[4];
}
let alpha = dst[3];
dst.copy_from_slice(&[rgba[0], rgba[1], rgba[2], alpha]);
}
OcrImage::new(image.width, image.height, out).unwrap_or_else(|_| image.clone())
}
/// The Sauvola ink mask: local thresholding with the documented window — 8 pixels wide, the
/// 8 pixels from −4 to +3 (the end is exclusive, so `x0..x1` is exactly `x−4..=x+3`), k 0.34 —
/// through an integral image. A pixel is ink when its grey is at or under the local
/// threshold.
fn sauvola_mask(image: &OcrImage) -> Vec<bool> {
let (w, h) = (image.width as usize, image.height as usize);
let gray = grayscale(image);
// Integral sums and square sums, (w+1)×(h+1), so any window sum is four lookups.
let stride = w + 1;
let mut sums = vec![0u64; stride * (h + 1)];
let mut squares = vec![0u64; stride * (h + 1)];
for (y, row) in gray.chunks_exact(w).enumerate() {
let (mut row_sum, mut row_sq) = (0u64, 0u64);
for (x, &g) in row.iter().enumerate() {
row_sum += u64::from(g);
row_sq += u64::from(g) * u64::from(g);
let above = sums.get(y * stride + x + 1).copied().unwrap_or(0);
let above_sq = squares.get(y * stride + x + 1).copied().unwrap_or(0);
if let (Some(s), Some(sq)) = (sums.get_mut((y + 1) * stride + x + 1), squares.get_mut((y + 1) * stride + x + 1)) {
*s = above + row_sum;
*sq = above_sq + row_sq;
}
}
}
let (k, r) = (0.34f64, 128.0f64);
let half = 4usize;
let mut ink = vec![false; w * h];
for (y, row) in ink.chunks_exact_mut(w).enumerate() {
for (x, dst) in row.iter_mut().enumerate() {
let (x0, y0) = (x.saturating_sub(half), y.saturating_sub(half));
// Exclusive end: the window covers −4..=+3, 8 pixels on a side (an end of
// `x + half + 1` would make it 9 and disagree with the documented window).
let (x1, y1) = ((x + half).min(w), (y + half).min(h));
let n = ((x1 - x0) * (y1 - y0)) as f64;
let sum = rect_sum(&sums, stride, x0, y0, x1, y1);
let sq = rect_sum(&squares, stride, x0, y0, x1, y1);
let mean = sum / n;
let var = (sq / n - mean * mean).max(0.0);
let threshold = mean * (1.0 + k * (var.sqrt() / r - 1.0));
*dst = matches!(gray.get(y * w + x), Some(&g) if f64::from(g) <= threshold);
}
}
ink
}
/// Sauvola binarization through [`sauvola_mask`], followed by a one-pixel dilation of the ink:
/// ink goes to black, paper to white, alpha preserved.
fn sauvola(image: &OcrImage) -> OcrImage {
let (w, h) = (image.width as usize, image.height as usize);
// One dilation keeps anti-aliased strokes whole: a thresholded edge pixel that flaked off
// rejoins its stroke (slightly bold beats broken, for reading).
let ink = dilate1(&sauvola_mask(image), w, h);
let mut out = image.rgba.clone();
for (i, px) in out.as_chunks_mut::<4>().0.iter_mut().enumerate() {
let v = if ink.get(i).copied().unwrap_or(false) { 0u8 } else { 255u8 };
if let Some(dst) = px.get_mut(..3) {
dst.copy_from_slice(&[v, v, v]);
}
}
OcrImage::new(image.width, image.height, out).unwrap_or_else(|_| image.clone())
}
/// Grow the ink mask by one pixel (8-neighbourhood, edges clamped by the bounds checks).
fn dilate1(ink: &[bool], w: usize, h: usize) -> Vec<bool> {
let mut out = ink.to_vec();
for y in 0..h {
for x in 0..w {
if !ink.get(y * w + x).copied().unwrap_or(false) {
continue;
}
for dy in [-1isize, 0, 1] {
for dx in [-1isize, 0, 1] {
let (nx, ny) = (x as isize + dx, y as isize + dy);
if nx < 0 || ny < 0 || nx >= w as isize || ny >= h as isize {
continue;
}
if let Some(o) = out.get_mut(ny as usize * w + nx as usize) {
*o = true;
}
}
}
}
}
out
}
fn rect_sum(table: &[u64], stride: usize, x0: usize, y0: usize, x1: usize, y1: usize) -> f64 {
let a = table.get(y1 * stride + x1).copied().unwrap_or(0);
let b = table.get(y0 * stride + x1).copied().unwrap_or(0);
let c = table.get(y1 * stride + x0).copied().unwrap_or(0);
let d = table.get(y0 * stride + x0).copied().unwrap_or(0);
(a + d - b - c) as f64
}
#[cfg(test)]
mod tests {
use super::*;
/// Sides under MAX_SIDE whose area passes MAX_PIXELS (9500 × 9500, 90 MP) are refused
/// when the size is worked out, before the 360 MB buffer is allocated rather than after.
#[test]
fn an_upscale_over_the_area_cap_is_refused_before_allocating() {
assert!(matches!(scaled_size(1000, 1000, 9.5), Err(OcrError::ImageTooLarge)));
assert_eq!(scaled_size(1000, 1000, 2.0).unwrap(), (2000, 2000), "under the caps it still scales");
assert_eq!(scaled_size(8000, 8000, 1.0).unwrap(), (8000, 8000), "exactly 64 MP is allowed");
assert!(matches!(scaled_size(8001, 8000, 1.0), Err(OcrError::ImageTooLarge)), "one row over is not");
assert!(matches!(scaled_size(1000, 1000, 10.5), Err(OcrError::ImageTooLarge)), "a side over MAX_SIDE");
assert!(matches!(scaled_size(1000, 1000, 0.0), Err(OcrError::ImageTooLarge)), "a side under one pixel");
assert!(matches!(scaled_size(1000, 1000, f64::NAN), Err(OcrError::ImageTooLarge)));
let small = OcrImage::new(100, 100, vec![255; 100 * 100 * 4]).unwrap();
assert!(matches!(resize_bilinear(&small, 100.5), Err(OcrError::ImageTooLarge)));
assert_eq!(resize_bilinear(&small, 2.0).unwrap().width, 200);
}
fn white(w: u32, h: u32) -> OcrImage {
OcrImage::new(w, h, vec![255; (w * h * 4) as usize]).unwrap()
}
/// Ink bounding box of a raster (None when blank).
fn ink_bbox(image: &OcrImage) -> Option<[f32; 4]> {
let gray = grayscale(image);
let (Some(lo), Some(hi)) = (gray.iter().min(), gray.iter().max()) else { return None };
let t = (*lo as u32 + *hi as u32) / 2;
let (mut l, mut tp, mut r, mut b) = (u32::MAX, u32::MAX, 0u32, 0u32);
let mut any = false;
for (i, &g) in gray.iter().enumerate() {
if (g as u32) >= t {
continue;
}
any = true;
let (x, y) = ((i as u32) % image.width, (i as u32) / image.width);
l = l.min(x);
r = r.max(x);
tp = tp.min(y);
b = b.max(y);
}
any.then_some([l as f32, tp as f32, r as f32, b as f32])
}
/// A horizontal line of three bottom-heavy block glyphs (L shapes) on white — the fixture
/// for orientation detection (240 × 64 at 300 dpi).
fn l_line() -> OcrImage {
let (w, h) = (240u32, 64u32);
let mut img = white(w, h);
for gx in [20u32, 100, 180] {
fill(&mut img, gx, 12, 12, 40);
fill(&mut img, gx, 40, 36, 12);
}
img
}
fn fill(img: &mut OcrImage, x0: u32, y0: u32, w: u32, h: u32) {
for y in y0..y0 + h {
for x in x0..x0 + w {
let o = ((y * img.width + x) * 4) as usize;
if let Some(p) = img.rgba.get_mut(o..o + 3) {
p.copy_from_slice(&[0, 0, 0]);
}
}
}
}
/// The same fixture pre-rotated by k quarter turns clockwise, built independently of the
/// crate's rotation (so the test cannot pass by a shared bug).
fn rotated(k: u32) -> OcrImage {
let src = l_line();
let (w, h) = (src.width as usize, src.height as usize);
let (nw, nh) = if k.is_multiple_of(2) { (src.width, src.height) } else { (src.height, src.width) };
let mut out = white(nw, nh);
for (y, row) in src.rgba.chunks_exact(w * 4).enumerate() {
for (x, px) in row.as_chunks::<4>().0.iter().enumerate() {
if px[0] != 0 {
continue;
}
let (nx, ny) = match k {
1 => (h - 1 - y, x),
2 => (w - 1 - x, h - 1 - y),
3 => (y, w - 1 - x),
_ => (x, y),
};
fill(&mut out, nx as u32, ny as u32, 1, 1);
}
}
out
}
/// A ruled fixture sheared by `degrees` (lines running downhill to the right for positive
/// angles): four 8-px bars across a 300 × 120 page.
fn sheared(degrees: f64) -> OcrImage {
let (w, h) = (300u32, 120u32);
let mut img = white(w, h);
let slope = degrees.to_radians().tan();
for (base, height) in [(20u32, 8u32), (45, 8), (70, 8), (95, 8)] {
for x in 20..280u32 {
let y = (base as f64 + (x as f64 - 150.0) * slope).round().clamp(0.0, f64::from(h - height - 1)) as u32;
fill(&mut img, x, y, 1, height);
}
}
img
}
/// With the four opt-in flags off and a dpi at or above 300, the raster passes through
/// untouched and the inverse transform is exact.
#[test]
fn flags_off_and_adequate_dpi_is_an_identity_pass_through() {
let src = l_line();
let p = preprocess(src.clone(), 300.0, &PreprocessOptions::default()).unwrap();
assert_eq!(p.image.rgba, src.rgba);
assert_eq!(p.dpi, 300.0);
assert_eq!(p.inverse_transform, InverseTransform::identity());
assert_eq!(p.inverse_transform.map_rect([3.0, 7.0, 41.0, 19.0]), [3.0, 7.0, 41.0, 19.0]);
}
/// With a step on, a scan below 300 dpi is enlarged by 300/src first, and its boxes map
/// back to the original raster.
#[test]
fn low_dpi_scans_are_upscaled_for_the_steps_and_boxes_map_back() {
let src = white(100, 50);
let on = PreprocessOptions { denoise: true, ..Default::default() };
let p = preprocess(src.clone(), 150.0, &on).unwrap();
assert_eq!((p.image.width, p.image.height), (200, 100));
assert_eq!(p.dpi, 300.0);
assert_eq!(p.inverse_transform.map_rect([0.0, 0.0, 200.0, 100.0]), [0.0, 0.0, 100.0, 50.0]);
assert_eq!(p.inverse_transform.apply(2.0, 4.0), (1.0, 2.0));
// 300 and 600 dpi are not resampled; a broken dpi is left alone.
assert_eq!(preprocess(src.clone(), 600.0, &on).unwrap().image.width, 100);
assert_eq!(preprocess(src, f32::NAN, &on).unwrap().image.width, 100);
}
/// With every step off a low-dpi render is read exactly as rendered: not resampled, with
/// the identity as its way back. (Reading a 72 dpi scan through a 4x enlargement changed
/// what the models read: "due: \€250" came back as "ue: ?250".)
#[test]
fn with_every_step_off_a_low_dpi_render_is_not_resampled() {
let src = l_line();
let p = preprocess(src.clone(), 72.0, &PreprocessOptions::default()).unwrap();
assert_eq!(p.image.rgba, src.rgba);
assert_eq!((p.image.width, p.image.height), (src.width, src.height));
assert_eq!(p.dpi, 72.0, "the render's own resolution");
assert_eq!(p.inverse_transform, InverseTransform::identity());
}
/// The upscale refuses to break the side cap.
#[test]
fn upscaling_respects_the_side_cap() {
let src = white(9999, 10);
let r = preprocess(src, 72.0, &PreprocessOptions { denoise: true, ..Default::default() });
assert!(matches!(r, Err(OcrError::ImageTooLarge)), "upscale past the cap: {r:?}");
}
/// An upright page is left alone by the orientation detector.
#[test]
fn upright_pages_are_not_rotated() {
let p = preprocess(l_line(), 300.0, &PreprocessOptions { auto_rotate: true, ..Default::default() }).unwrap();
assert_eq!(p.inverse_transform, InverseTransform::identity());
}
/// Pages scanned at 90/180/270 are detected, turned upright, and their word boxes map back
/// onto the scanned raster's ink.
#[test]
fn quarter_turns_are_detected_corrected_and_mapped_back() {
for k in 1..=3 {
let scanned = rotated(k);
let p = preprocess(scanned.clone(), 300.0, &PreprocessOptions { auto_rotate: true, ..Default::default() }).unwrap();
let corrected = ink_bbox(&p.image).unwrap_or_else(|| panic!("k={k}: the corrected page still has ink"));
// The corrected reading is a horizontal line again: wider than tall.
assert!(corrected[2] - corrected[0] > corrected[3] - corrected[1], "k={k}: {corrected:?}");
// And the box maps back onto the scanned raster's own ink box.
let back = p.inverse_transform.map_rect(corrected);
let original = ink_bbox(&scanned).unwrap();
for i in 0..4 {
assert!((back[i] - original[i]).abs() <= 2.0, "k={k}: {back:?} vs {original:?}");
}
}
}
/// Images under 32 px on a side are never judged; the step is a no-op.
#[test]
fn tiny_images_skip_orientation() {
let src = white(31, 31);
let p = preprocess(src, 300.0, &PreprocessOptions { auto_rotate: true, ..Default::default() }).unwrap();
assert_eq!(p.inverse_transform, InverseTransform::identity());
}
/// Blank pages are inconclusive: every step leaves them alone.
#[test]
fn blank_pages_are_a_noop() {
let src = white(200, 200);
let p = preprocess(src, 300.0, &PreprocessOptions { auto_rotate: true, deskew: true, denoise: true, binarize: true }).unwrap();
assert_eq!(p.inverse_transform, InverseTransform::identity());
}
/// The skew estimator sees a 3° shear (either direction) and leaves a straight page alone.
#[test]
fn skew_is_estimated_and_straight_pages_are_left_alone() {
for degrees in [3.0, -3.0] {
let angle = estimate_skew(&sheared(degrees)).unwrap_or_else(|| panic!("{degrees}° should be found"));
assert!((angle - degrees).abs() <= 0.5, "{degrees}°: got {angle}");
}
assert!(estimate_skew(&sheared(0.0)).is_none(), "a straight page is a no-op");
assert!(estimate_skew(&l_line()).is_none(), "glyph lines without a shear are a no-op");
}
/// Deskew straightens ±3° lines (the vertical span tightens) and the centre maps back to
/// itself.
#[test]
fn deskew_corrects_plus_and_minus_three_degrees() {
for degrees in [3.0, -3.0] {
let src = sheared(degrees);
let before = ink_bbox(&src).unwrap();
let p = preprocess(src.clone(), 300.0, &PreprocessOptions { deskew: true, ..Default::default() }).unwrap();
let after = ink_bbox(&p.image).unwrap();
assert!(after[3] - after[1] < before[3] - before[1], "{degrees}°: {before:?} → {after:?}");
// The exact inverse maps the corrected centre back to the scanned centre.
let (cx, cy) = (f64::from(src.width) / 2.0, f64::from(src.height) / 2.0);
let (bx, by) = p.inverse_transform.apply(cx, cy);
assert!((bx - cx).abs() < 1.0 && (by - cy).abs() < 1.0, "{degrees}°: ({bx}, {by})");
}
}
/// Deskew no-ops on small images (under 64 px) — the detector is never consulted.
#[test]
fn small_images_skip_deskew() {
let src = white(63, 63);
let p = preprocess(src, 300.0, &PreprocessOptions { deskew: true, ..Default::default() }).unwrap();
assert_eq!(p.inverse_transform, InverseTransform::identity());
}
/// The median filter removes isolated specks, keeps solid strokes, and never changes the
/// dimensions.
#[test]
fn denoise_removes_specks_and_keeps_dimensions() {
let mut src = white(32, 32);
fill(&mut src, 15, 15, 1, 1); // salt
fill(&mut src, 4, 4, 5, 5); // a solid stroke
let p = preprocess(src, 300.0, &PreprocessOptions { denoise: true, ..Default::default() }).unwrap();
assert_eq!((p.image.width, p.image.height), (32, 32));
assert_eq!(p.image.byte(15.0, 15.0, 0), 255, "an isolated speck is gone");
assert_eq!(p.image.byte(6.0, 6.0, 0), 0, "the inside of a solid block stays");
assert_eq!(p.inverse_transform, InverseTransform::identity());
}
/// Sauvola keeps the polarity (dark ink stays dark, paper stays light) and the metadata.
/// The fixture is a thin grey stroke: its windows mix ink and paper, so the local
/// threshold separates them (a perfectly flat mid-tone area has no contrast and whitens,
/// which is Sauvola working as designed).
#[test]
fn binarize_keeps_polarity_and_metadata() {
let (w, h) = (64u32, 64u32);
let mut src = white(w, h);
for y in 26..30 {
for x in 8..56 {
let o = ((y * w + x) * 4) as usize;
if let Some(p) = src.rgba.get_mut(o..o + 3) {
p.copy_from_slice(&[70, 70, 70]); // a grey scan's ink, far below paper
}
}
}
let p = preprocess(src, 300.0, &PreprocessOptions { binarize: true, ..Default::default() }).unwrap();
assert_eq!((p.image.width, p.image.height), (w, h));
assert_eq!(p.dpi, 300.0);
assert_eq!(p.image.byte(30.0, 28.0, 0), 0, "ink stays dark");
assert_eq!(p.image.byte(2.0, 2.0, 0), 255, "paper stays light");
for px in p.image.rgba.as_chunks::<4>().0.iter() {
assert!((px[0] == 0 || px[0] == 255) && px[0] == px[1] && px[1] == px[2], "{px:?}");
}
}
/// The inverse transform composes: an upscaled, turned, deskewed page maps corrected
/// points back inside the original raster.
#[test]
fn inverse_transform_maps_points_back_through_every_step() {
let scanned = rotated(1); // a page photographed on its side
let p = preprocess(scanned.clone(), 150.0, &PreprocessOptions { auto_rotate: true, deskew: true, ..Default::default() }).unwrap();
let (w, h) = (f64::from(p.image.width), f64::from(p.image.height));
for (x, y) in [(0.0, 0.0), (w - 1.0, 0.0), (0.0, h - 1.0), (w - 1.0, h - 1.0), (w / 2.0, h / 2.0)] {
let (bx, by) = p.inverse_transform.apply(x, y);
assert!((-2.0..=f64::from(scanned.width) + 2.0).contains(&bx), "x {x} → {bx}");
assert!((-2.0..=f64::from(scanned.height) + 2.0).contains(&by), "y {y} → {by}");
}
// The scale step, alone, is exact.
assert_eq!(InverseTransform::scale(0.5).apply(4.0, 10.0), (2.0, 5.0));
}
/// A realistic letter page (one line of type on white, at 300 dpi) reads as upright: the
/// row axis bands decisively sharper than the column axis, and the near-balanced
/// bottom-heaviness of real mixed-case text is below the 15% flip bar, so the page is left
/// alone rather than risk flipping a good scan upside down.
#[test]
fn realistic_letter_pages_are_left_upright() {
let (w, h) = (1200u32, 1650u32);
let mut img = white(w, h);
let mut x = 120u32;
let base = 300u32;
let mut seed = 7u64;
let mut rnd = || {
seed = seed.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407);
(seed >> 33) as u32
};
while x < 1000 {
let word = rnd() % 5 + 2;
for _ in 0..word {
let kind = rnd() % 4;
let wl = rnd() % 8 + 3;
match kind {
0 => fill(&mut img, x, base - 20, wl, 20), // x-height
1 => fill(&mut img, x, base - 34, wl, 34), // ascender
2 => fill(&mut img, x, base - 20, wl, 28), // descender
_ => {
fill(&mut img, x, base - 20, 2, 20);
fill(&mut img, x + 3, base - 14, 2, 14); // stems
}
}
x += wl + 2;
}
x += 12; // space
}
assert_eq!(detect_orientation(&img), 0);
let p = preprocess(img, 300.0, &PreprocessOptions { auto_rotate: true, ..Default::default() }).unwrap();
assert_eq!(p.inverse_transform, InverseTransform::identity());
}
/// Hostile inputs are errors or no-ops, never panics: 1×1 rasters, all-black pages, NaN
/// boxes through the inverse map.
#[test]
fn hostile_inputs_never_panic() {
let tiny = white(1, 1);
let all = PreprocessOptions { auto_rotate: true, deskew: true, denoise: true, binarize: true };
let p = preprocess(tiny, 300.0, &all).unwrap();
assert_eq!((p.image.width, p.image.height), (1, 1));
// All-black: no paper, nothing to judge.
let mut black = white(80, 80);
for px in black.rgba.as_chunks_mut::<4>().0 {
px.copy_from_slice(&[0, 0, 0, 255]);
}
let p = preprocess(black, 300.0, &all).unwrap();
assert_eq!(p.inverse_transform, InverseTransform::identity(), "all-black is inconclusive");
// NaN boxes never poison downstream math: the finite corners bound the mapped box,
// and a box with no finite corners comes back unchanged.
let mapped = p.inverse_transform.map_rect([f32::NAN, 0.0, 5.0, 5.0]);
assert!(mapped.iter().all(|v| v.is_finite()), "{mapped:?}");
let unchanged = InverseTransform::identity().map_rect([f32::NAN; 4]);
assert!(unchanged.iter().all(|v| v.is_nan()), "{unchanged:?}");
}
/// The one-pixel dilation is load-bearing, pinned with a fixture that fragments without
/// it: a thin anti-aliased-looking stroke on a grey scan (alternating dark and light
/// pixels) — the raw Sauvola mask drops the light runs (the local window sits on a grey
/// page, so the threshold falls below them), while the binarized image keeps the stroke
/// whole.
#[test]
fn binarize_without_the_dilation_the_stroke_fragments_and_with_it_stays_whole() {
let (w, h) = (64u32, 64u32);
let mut src = white(w, h);
let set = |img: &mut OcrImage, x: u32, y: u32, v: u8| {
let o = ((y * w + x) * 4) as usize;
if let Some(p) = img.rgba.get_mut(o..o + 3) {
p.copy_from_slice(&[v, v, v]);
}
};
for px in src.rgba.as_chunks_mut::<4>().0.iter_mut() {
px[..3].copy_from_slice(&[210, 210, 210]); // a grey scan's paper
}
// The stroke: one pixel tall, alternating dark (reads as ink) and light (an
// anti-aliased shoulder the threshold alone drops) in runs of two.
for x in 8..56u32 {
let v = if (x / 2) % 2 == 0 { 110 } else { 170 };
set(&mut src, x, 32, v);
}
let mask = sauvola_mask(&src);
let at = |x: u32, y: u32| mask[y as usize * w as usize + x as usize];
let dropped: Vec<u32> = (12..52).filter(|&x| !at(x, 32)).collect();
assert!(!dropped.is_empty(), "without the dilation the stroke fragments: these stroke pixels fall under the threshold at {dropped:?}");
let p = preprocess(src, 300.0, &PreprocessOptions { binarize: true, ..Default::default() }).unwrap();
// With the dilation the stroke is whole: every position on it has ink within its
// 8-neighbourhood.
for x in 12..=51u32 {
let ink_near = (-1i32..=1).any(|dy| {
(-1i32..=1).any(|dx| {
let (nx, ny) = (x as i32 + dx, 32 + dy);
p.image.byte(f64::from(nx), f64::from(ny), 0) == 0
})
});
assert!(ink_near, "the stroke is broken at x={x}");
}
}
/// The deskew inverse, checked at OFF-CENTRE points (the centre maps to itself under any
/// rotation, so it cannot catch a wrong sign): a point on a straightened line maps back
/// onto the sheared source line, uphill and downhill.
#[test]
fn deskew_inverse_maps_off_centre_points_back_onto_the_source_line() {
for degrees in [3.0, -3.0] {
let src = sheared(degrees);
let p = preprocess(src, 300.0, &PreprocessOptions { deskew: true, ..Default::default() }).unwrap();
let slope = degrees.to_radians().tan();
// The corrected first line sits at y = 20 (the fixture's base); the same ink was
// at 20 + (x − 150)·slope in the source. Well off centre on purpose. The
// rotation moves x a little too (it mixes in y: at most sin(5°)·40 ≈ 3.5 px
// here), so the line is checked at the mapped x — a wrong sign still misses it
// by ±(x−150)·slope, 6+ px at these offsets.
for x in [30.0f64, 150.0, 270.0] {
let (sx, sy) = p.inverse_transform.apply(x, 20.0);
assert!((sx - x).abs() < 4.0, "{degrees}° x={x}: mapped to ({sx}, {sy})");
let expected = 20.0 + (sx - 150.0) * slope;
assert!((sy - expected).abs() < 2.0, "{degrees}° x={x}: ({sx}, {sy}), the source line is at {expected}");
}
}
}
}
crates/ocr/src/tesseract.rs
+1022
View File
File diff suppressed because it is too large. Load diff
crates/preflight/fuzz/Cargo.toml
+32
View File
@@ -0,0 +1,32 @@
[package]
name = "pdfcraft-preflight-fuzz"
version = "0.0.0"
publish = false
edition = "2024"
license = "MIT OR Apache-2.0"
[package.metadata]
cargo-fuzz = true
# Detached from the PdfCraft workspace: run with `cargo +nightly fuzz run verify`
# from crates/preflight/.
[workspace]
[dependencies]
libfuzzer-sys = "0.4"
pdfcraft-cos = { path = "../../cos" }
pdfcraft-preflight = { path = ".." }
[[bin]]
name = "verify"
path = "fuzz_targets/verify.rs"
test = false
doc = false
bench = false
[[bin]]
name = "convert"
path = "fuzz_targets/convert.rs"
test = false
doc = false
bench = false
crates/preflight/fuzz/fuzz_targets/convert.rs
+21
View File
@@ -0,0 +1,21 @@
//! Feeds arbitrary bytes to `convert` at level A2b; the report it returns is discarded.
#![no_main]
use std::sync::Arc;
use libfuzzer_sys::fuzz_target;
use pdfcraft_cos::Document;
use pdfcraft_preflight::{Level, convert};
/// Largest input the target parses; a longer input is dropped before `Document::open` copies it.
const MAX_INPUT: usize = 16 << 20;
fuzz_target!(|data: &[u8]| {
if data.len() > MAX_INPUT {
return;
}
let Ok(mut doc) = Document::open(Arc::new(data.to_vec())) else {
return;
};
let _ = convert(&mut doc, Level::A2b);
});
crates/preflight/fuzz/fuzz_targets/verify.rs
+22
View File
@@ -0,0 +1,22 @@
//! Feeds arbitrary bytes to `verify` at both levels; the issue list it returns is discarded.
#![no_main]
use std::sync::Arc;
use libfuzzer_sys::fuzz_target;
use pdfcraft_cos::Document;
use pdfcraft_preflight::{Level, verify};
/// Largest input the target parses; a longer input is dropped before `Document::open` copies it.
const MAX_INPUT: usize = 16 << 20;
fuzz_target!(|data: &[u8]| {
if data.len() > MAX_INPUT {
return;
}
let Ok(doc) = Document::open(Arc::new(data.to_vec())) else {
return;
};
let _a2b = verify(&doc, Level::A2b);
let _a3b = verify(&doc, Level::A3b);
});
crates/preflight/src/lib.rs
+3
View File
@@ -877,6 +877,9 @@ pub fn convert(doc: &mut Document, level: Level) -> Result<Report, pdfcraft_cos:
doc.update_dict(p.obj, |pd| pd.set(b"Annots".to_vec(), Object::Array(kept)))?;
}
}
// The object stays in the file unless it is replaced: verify scans every object, so a
// forbidden annotation left behind would still be reported (clause 6.3.1).
doc.set(r, Object::Null);
removed += 1;
continue;
}
crates/preflight/tests/convert_annotations.rs
+57
View File
@@ -0,0 +1,57 @@
use pdfcraft_cos::{Document, ObjRef, Object};
use pdfcraft_preflight::{Level, convert, verify};
fn build(objs: &[&str], trailer_extra: &str) -> Document {
let mut out = b"%PDF-1.7\n".to_vec();
let mut offsets = Vec::new();
for (i, o) in objs.iter().enumerate() {
offsets.push(out.len());
out.extend_from_slice(format!("{} 0 obj\n{o}\nendobj\n", i + 1).as_bytes());
}
let xref = out.len();
out.extend_from_slice(format!("xref\n0 {}\n0000000000 65535 f \n", objs.len() + 1).as_bytes());
for o in offsets {
out.extend_from_slice(format!("{o:010} 00000 n \n").as_bytes());
}
out.extend_from_slice(format!("trailer\n<< /Size {} /Root 1 0 R {trailer_extra} >>\nstartxref\n{xref}\n%%EOF\n", objs.len() + 1).as_bytes());
Document::open(std::sync::Arc::new(out)).unwrap()
}
/// One page whose `/Annots` lists a Sound annotation (forbidden by clause 6.3.1) and a link.
fn page_with_a_forbidden_sound_annotation() -> Document {
build(
&[
"<< /Type /Catalog /Pages 2 0 R >>",
"<< /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >>",
"<< /Type /Page /Parent 2 0 R /Annots [4 0 R 5 0 R] >>",
"<< /Type /Annot /Subtype /Sound /Rect [10 10 30 30] /P 3 0 R /F 4 >>",
"<< /Type /Annot /Subtype /Link /Rect [40 40 60 60] /P 3 0 R /F 4 /Border [0 0 0] >>",
],
"",
)
}
/// The forbidden annotation is reported before `convert` and by neither `convert` nor a later
/// `verify` afterwards: the object it took off the page is no longer an annotation.
#[test]
fn convert_removes_the_forbidden_annotation_clause_6_3_1() {
let mut doc = page_with_a_forbidden_sound_annotation();
let before = verify(&doc, Level::A2b);
assert!(before.iter().any(|i| i.clause == "6.3.1"), "the fixture starts with the finding: {before:#?}");
let report = convert(&mut doc, Level::A2b).unwrap();
assert!(!report.remaining.iter().any(|i| i.clause == "6.3.1"), "6.3.1 remains after convert: {:#?}", report.remaining);
let after = verify(&doc, Level::A2b);
assert!(!after.iter().any(|i| i.clause == "6.3.1"), "6.3.1 is reported after convert: {after:#?}");
}
/// The page keeps its other annotation: only the forbidden one leaves `/Annots`.
#[test]
fn convert_keeps_the_other_annotations_on_the_page() {
let mut doc = page_with_a_forbidden_sound_annotation();
convert(&mut doc, Level::A2b).unwrap();
let page = doc.get(ObjRef::new(3, 0));
let dict = page.as_dict().expect("the page is a dictionary");
let annots = doc.resolve(dict.get(b"Annots").expect("the page has /Annots"));
let kept: Vec<ObjRef> = annots.as_array().expect("the page's /Annots is an array").iter().filter_map(Object::as_ref).collect();
assert_eq!(kept, vec![ObjRef::new(5, 0)]);
}
crates/preflight/tests/known_answers.rs
+403
View File
@@ -0,0 +1,403 @@
//! What `verify` answers for each clause it implements, pinned as a known answer.
//!
//! Each test builds a small file by hand, from the shape the clause describes rather than from
//! anything this repository saves, so a bug shared between the verifier and the writer cannot make a
//! case pass. Two things in the file are not hand-written: the output intent profiles come from
//! `icc::srgb()`, the crate's own sRGB generator, and the encrypted document in the 6.1.3 case is
//! encrypted by the object layer, because writing an encryption dictionary by hand would restate the
//! rules under test.
//!
//! The clause strings are the verifier's own citations, pinned here as `lib.rs` writes them. They
//! have not been compared with the text of the profile yet, so a number that does not name the rule
//! it is attached to is a finding about the verifier and not about this file.
//!
//! The positive case expects no issue under the clause and the negative case expects one; both are
//! asserted by clause string, so a later change that alters an answer fails here first. Cases where
//! the verifier is known to disagree with the standard are left out on purpose and recorded as gaps
//! where the change that adds them is reviewed. Nothing here is a conformance claim.
use pdfcraft_cos::{Algorithm, Document, NewEncryption};
use pdfcraft_preflight::{Level, convert, declared, icc, verify};
/// A document from hand-written objects and extra trailer entries (copied from the crate's unit
/// tests).
fn build(objs: &[&str], trailer_extra: &str) -> Document {
let mut out = b"%PDF-1.7\n".to_vec();
let mut offsets = Vec::new();
for (i, o) in objs.iter().enumerate() {
offsets.push(out.len());
out.extend_from_slice(format!("{} 0 obj\n{o}\nendobj\n", i + 1).as_bytes());
}
let xref = out.len();
out.extend_from_slice(format!("xref\n0 {}\n0000000000 65535 f \n", objs.len() + 1).as_bytes());
for o in offsets {
out.extend_from_slice(format!("{o:010} 00000 n \n").as_bytes());
}
out.extend_from_slice(format!("trailer\n<< /Size {} /Root 1 0 R {trailer_extra} >>\nstartxref\n{xref}\n%%EOF\n", objs.len() + 1).as_bytes());
Document::open(std::sync::Arc::new(out)).unwrap()
}
/// The byte-order mark an XMP packet's `xpacket` instruction names.
const BOM: char = '\u{feff}';
/// The XMP packet identifying a file as `part`/`conformance`, with `title` as its `dc:title`.
fn xmp(part: u8, conformance: &str, title: Option<&str>) -> String {
let title = match title {
Some(t) => format!(" <dc:title><rdf:Alt><rdf:li xml:lang=\"x-default\">{t}</rdf:li></rdf:Alt></dc:title>\n"),
None => String::new(),
};
format!(
r#"<?xpacket begin="{BOM}" id="W5M0MpCehiHzreSzNTczkc9d"?>
<x:xmpmeta xmlns:x="adobe:ns:meta/">
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about="" xmlns:pdfaid="http://www.aiim.org/pdfa/ns/id/" xmlns:dc="http://purl.org/dc/elements/1.1/">
<pdfaid:part>{part}</pdfaid:part>
<pdfaid:conformance>{conformance}</pdfaid:conformance>
{title} </rdf:Description>
</rdf:RDF>
</x:xmpmeta>
<?xpacket end="w"?>"#
)
}
/// The XMP packet of a PDF/A-2b file, with `title` as its title when one is given.
fn a2b_packet(title: Option<&str>) -> String {
xmp(2, "B", title)
}
/// The catalog's `/Metadata` stream object holding `packet`.
fn xmp_object(packet: &str) -> String {
format!("<< /Type /Metadata /Subtype /XML /Length {} >>\nstream\n{packet}\nendstream", packet.len())
}
/// A content stream object holding `content`.
fn content(content: &str) -> String {
format!("<< /Length {} >>\nstream\n{content}\nendstream", content.len())
}
/// An ASCII-hex stream object holding `bytes`, with `dict` as its other entries.
fn hex_stream(dict: &str, bytes: &[u8]) -> String {
let hex: String = bytes.iter().map(|b| format!("{b:02X}")).collect();
let data = format!("{hex}>");
format!("<< /Filter /ASCIIHexDecode {dict} /Length {} >>\nstream\n{data}\nendstream", data.len())
}
/// A 128-byte ICC header for a four-component profile, with no `/N` so the header decides.
fn cmyk_profile_header() -> Vec<u8> {
let mut header = vec![0u8; 128];
header[0..4].copy_from_slice(&128u32.to_be_bytes());
header[16..20].copy_from_slice(b"CMYK");
header[20..24].copy_from_slice(b"XYZ ");
header[36..40].copy_from_slice(b"acsp");
header
}
/// The catalog's `/OutputIntents`: a `GTS_PDFA1` intent naming the profile in object `n`.
fn output_intent(n: usize) -> String {
let condition = "/OutputConditionIdentifier (sRGB IEC61966-2.1)";
let intent = format!("<< /Type /OutputIntent /S /GTS_PDFA1 {condition} /DestOutputProfile {n} 0 R >>");
format!("/OutputIntents [{intent}]")
}
/// An embedded font program stream object, compressed with `filter` when one is given. Its bytes
/// are not read by any rule: a font program is data, not content.
fn font_program(filter: Option<&str>) -> String {
let filter = match filter {
Some(filter) => format!("/Filter {filter} "),
None => String::new(),
};
format!("<< /Length1 4 {filter}/Length 4 >>\nstream\nabcd\nendstream")
}
/// An image XObject that asks to be interpolated, or does not.
fn image(interpolate: bool) -> String {
format!(
"<< /Type /XObject /Subtype /Image /Width 1 /Height 1 /ColorSpace /DeviceGray /BitsPerComponent 8 /Interpolate {interpolate} /Length 1 >>\nstream\n\u{0}\nendstream"
)
}
/// A document whose catalog (object 1) carries `catalog_extra` and, when `packet` is given, the
/// metadata object 3; whose page tree (object 2) holds the single page `page` (object 4); and whose
/// objects 5 onwards are `more`, which `page` and `catalog_extra` refer to by number. An absent
/// `packet` leaves object 3 empty. The page must resolve, so a case that loses it fails instead of
/// passing because no rule ever reached it.
fn page_document(page: &str, packet: Option<&str>, catalog_extra: &str, more: &[&str], trailer_extra: &str) -> Document {
let metadata = match packet {
Some(packet) => xmp_object(packet),
None => "null".to_owned(),
};
let catalog = match packet {
Some(_) => format!("<< /Type /Catalog /Pages 2 0 R /Metadata 3 0 R {catalog_extra} >>"),
None => format!("<< /Type /Catalog /Pages 2 0 R {catalog_extra} >>"),
};
let pages = "<< /Type /Pages /Kids [4 0 R] /Count 1 /MediaBox [0 0 200 200] >>";
let mut objs = vec![catalog.as_str(), pages, metadata.as_str(), page];
objs.extend_from_slice(more);
let doc = build(&objs, trailer_extra);
assert_eq!(pdfcraft_model::pages(&doc).len(), 1, "the page of this case does not resolve");
doc
}
/// Asserts that `verify` reports `clause` for `doc`.
fn assert_reported(doc: &Document, level: Level, clause: &str) {
let issues = verify(doc, level);
assert!(issues.iter().any(|i| i.clause == clause), "no {clause} issue in {issues:#?}");
}
/// Asserts that `verify` reports nothing under `clause` for `doc`.
fn assert_not_reported(doc: &Document, level: Level, clause: &str) {
let found: Vec<_> = verify(doc, level).into_iter().filter(|i| i.clause == clause).collect();
assert!(found.is_empty(), "{clause} was reported: {found:#?}");
}
#[test]
fn xmp_metadata_is_required_clause_6_6_2_1() {
let page = "<< /Type /Page /Parent 2 0 R >>";
let packet = a2b_packet(None);
let identified = page_document(page, Some(packet.as_str()), "", &[], "");
assert_not_reported(&identified, Level::A2b, "6.6.2.1");
assert_eq!(declared(&identified).pdfa, Some((2, "B".into())));
let bare = page_document(page, None, "", &[], "");
assert_reported(&bare, Level::A2b, "6.6.2.1");
}
#[test]
fn the_declared_part_and_conformance_must_be_the_level_clause_6_6_4() {
let page = "<< /Type /Page /Parent 2 0 R >>";
let matching = a2b_packet(None);
let doc = page_document(page, Some(matching.as_str()), "", &[], "");
assert_not_reported(&doc, Level::A2b, "6.6.4");
for (part, conformance) in [(3, "B"), (2, "A")] {
let packet = xmp(part, conformance, None);
let doc = page_document(page, Some(packet.as_str()), "", &[], "");
assert_reported(&doc, Level::A2b, "6.6.4");
assert_eq!(declared(&doc).pdfa, Some((part, conformance.to_owned())));
}
}
#[test]
fn the_xmp_title_must_match_the_information_dictionary_clause_6_6_3() {
let page = "<< /Type /Page /Parent 2 0 R >>";
let info = "/Info << /Title (Report) >>";
let matching = a2b_packet(Some("Report"));
let doc = page_document(page, Some(matching.as_str()), "", &[], info);
assert_not_reported(&doc, Level::A2b, "6.6.3");
let differing = a2b_packet(Some("Draft"));
let doc = page_document(page, Some(differing.as_str()), "", &[], info);
assert_reported(&doc, Level::A2b, "6.6.3");
}
#[test]
fn the_trailer_must_not_name_encryption_clause_6_1_3() {
let page = "<< /Type /Page /Parent 2 0 R >>";
let packet = a2b_packet(None);
let plain = page_document(page, Some(packet.as_str()), "", &[], "");
assert_not_reported(&plain, Level::A2b, "6.1.3");
// An encrypted file's keys are not written out by hand here, so the /Encrypt dictionary comes
// from the object layer; the rule under test only asks whether the trailer names one.
let mut encrypted = page_document(page, Some(packet.as_str()), "", &[], "");
let params = NewEncryption {
algorithm: Algorithm::Aes256,
user_password: "",
owner_password: "owner",
permissions: -4,
encrypt_metadata: true,
seed: [7; 32],
};
encrypted.set_encryption(&params).unwrap();
assert_reported(&encrypted, Level::A2b, "6.1.3");
let report = convert(&mut encrypted, Level::A2b).unwrap();
assert!(!report.remaining.iter().any(|i| i.clause == "6.1.3"), "{:#?}", report.remaining);
}
#[test]
fn device_colour_needs_an_output_intent_clause_6_2_3() {
let page = "<< /Type /Page /Parent 2 0 R /Contents 5 0 R /Resources << >> >>";
let packet = a2b_packet(None);
let rgb = content("1 0 0 rg 10 10 20 20 re f");
let doc = page_document(page, Some(packet.as_str()), "", &[rgb.as_str()], "");
assert_reported(&doc, Level::A2b, "6.2.3");
let none = content("q Q");
let doc = page_document(page, Some(packet.as_str()), "", &[none.as_str()], "");
assert_not_reported(&doc, Level::A2b, "6.2.3");
}
#[test]
fn device_rgb_needs_an_rgb_output_intent_clause_6_2_4_3() {
let page = "<< /Type /Page /Parent 2 0 R /Contents 5 0 R /Resources << >> >>";
let packet = a2b_packet(None);
let rgb = content("1 0 0 rg 10 10 20 20 re f");
let srgb = hex_stream("/N 3", &icc::srgb());
let doc = page_document(page, Some(packet.as_str()), &output_intent(6), &[rgb.as_str(), srgb.as_str()], "");
assert_not_reported(&doc, Level::A2b, "6.2.4.3");
// The intent is there, so the neighbour rule about device colour without one stays quiet.
assert_not_reported(&doc, Level::A2b, "6.2.3");
let cmyk = hex_stream("", &cmyk_profile_header());
let doc = page_document(page, Some(packet.as_str()), &output_intent(6), &[rgb.as_str(), cmyk.as_str()], "");
assert_reported(&doc, Level::A2b, "6.2.4.3");
}
#[test]
fn the_catalog_must_not_have_additional_actions_clause_6_5_2() {
let page = "<< /Type /Page /Parent 2 0 R >>";
let packet = a2b_packet(None);
let plain = page_document(page, Some(packet.as_str()), "", &[], "");
assert_not_reported(&plain, Level::A2b, "6.5.2");
let acting = page_document(page, Some(packet.as_str()), "/AA << /WC << /S /GoTo >> >>", &[], "");
assert_reported(&acting, Level::A2b, "6.5.2");
}
#[test]
fn the_catalog_must_not_have_document_javascript_clause_6_5_1() {
let page = "<< /Type /Page /Parent 2 0 R >>";
let packet = a2b_packet(None);
let plain = page_document(page, Some(packet.as_str()), "", &[], "");
assert_not_reported(&plain, Level::A2b, "6.5.1");
// An empty name tree is enough: an action object here would report the same clause, and this
// case is about the catalog's own entry.
let tree = "/Names << /JavaScript << /Names [] >> >>";
let scripted = page_document(page, Some(packet.as_str()), tree, &[], "");
assert_reported(&scripted, Level::A2b, "6.5.1");
}
#[test]
fn the_form_must_not_ask_viewers_for_appearances_clause_6_4_1() {
let page = "<< /Type /Page /Parent 2 0 R >>";
let packet = a2b_packet(None);
let plain = page_document(page, Some(packet.as_str()), "/AcroForm << /Fields [] >>", &[], "");
assert_not_reported(&plain, Level::A2b, "6.4.1");
let asking = "/AcroForm << /Fields [] /NeedAppearances true >>";
let doc = page_document(page, Some(packet.as_str()), asking, &[], "");
assert_reported(&doc, Level::A2b, "6.4.1");
}
#[test]
fn forbidden_annotation_types_are_reported_clause_6_3_1() {
let page = "<< /Type /Page /Parent 2 0 R /Annots [5 0 R] >>";
let packet = a2b_packet(None);
let link = "<< /Type /Annot /Subtype /Link /Rect [10 10 30 30] /P 4 0 R /F 4 >>";
let doc = page_document(page, Some(packet.as_str()), "", &[link], "");
assert_not_reported(&doc, Level::A2b, "6.3.1");
let sound = "<< /Type /Annot /Subtype /Sound /Rect [10 10 30 30] /P 4 0 R /F 4 >>";
let doc = page_document(page, Some(packet.as_str()), "", &[sound], "");
assert_reported(&doc, Level::A2b, "6.3.1");
}
#[test]
fn annotations_must_print_and_not_hide_clause_6_3_2() {
let page = "<< /Type /Page /Parent 2 0 R /Annots [5 0 R] >>";
let packet = a2b_packet(None);
let appearance = "<< /Type /XObject /Subtype /Form /BBox [0 0 20 20] /Length 0 >>\nstream\n\nendstream";
let printing = "<< /Type /Annot /Subtype /Square /Rect [10 10 30 30] /P 4 0 R /F 4 /AP << /N 6 0 R >> >>";
let doc = page_document(page, Some(packet.as_str()), "", &[printing, appearance], "");
assert_not_reported(&doc, Level::A2b, "6.3.2");
let unprinted = "<< /Type /Annot /Subtype /Square /Rect [10 10 30 30] /P 4 0 R /F 0 /AP << /N 6 0 R >> >>";
let doc = page_document(page, Some(packet.as_str()), "", &[unprinted, appearance], "");
assert_reported(&doc, Level::A2b, "6.3.2");
let hidden = "<< /Type /Annot /Subtype /Square /Rect [10 10 30 30] /P 4 0 R /F 6 /AP << /N 6 0 R >> >>";
let doc = page_document(page, Some(packet.as_str()), "", &[hidden, appearance], "");
assert_reported(&doc, Level::A2b, "6.3.2");
}
#[test]
fn annotations_need_an_appearance_stream_clause_6_3_3() {
let page = "<< /Type /Page /Parent 2 0 R /Annots [5 0 R] >>";
let packet = a2b_packet(None);
let appearance = "<< /Type /XObject /Subtype /Form /BBox [0 0 20 20] /Length 0 >>\nstream\n\nendstream";
let with = "<< /Type /Annot /Subtype /Square /Rect [10 10 30 30] /P 4 0 R /F 4 /AP << /N 6 0 R >> >>";
let doc = page_document(page, Some(packet.as_str()), "", &[with, appearance], "");
assert_not_reported(&doc, Level::A2b, "6.3.3");
let without = "<< /Type /Annot /Subtype /Square /Rect [10 10 30 30] /P 4 0 R /F 4 >>";
let doc = page_document(page, Some(packet.as_str()), "", &[without], "");
assert_reported(&doc, Level::A2b, "6.3.3");
}
#[test]
fn lzw_compressed_streams_are_reported_clause_6_1_7_2() {
let page = "<< /Type /Page /Parent 2 0 R /Contents 5 0 R /Resources << /Font << /F1 6 0 R >> >> >>";
let packet = a2b_packet(None);
let text = content("BT /F1 12 Tf 10 10 Td (Hi) Tj ET");
let font = "<< /Type /Font /Subtype /TrueType /BaseFont /Sans /FirstChar 72 /LastChar 105 /Widths [500 500] /FontDescriptor 7 0 R >>";
let descriptor = "<< /Type /FontDescriptor /FontName /Sans /Flags 32 /FontFile2 8 0 R >>";
let flate = font_program(Some("/FlateDecode"));
let doc = page_document(page, Some(packet.as_str()), "", &[text.as_str(), font, descriptor, flate.as_str()], "");
assert_not_reported(&doc, Level::A2b, "6.1.7.2");
// The neighbour rule holds: the font still reads as embedded.
assert_not_reported(&doc, Level::A2b, "6.2.11.4");
let lzw = font_program(Some("/LZWDecode"));
let doc = page_document(page, Some(packet.as_str()), "", &[text.as_str(), font, descriptor, lzw.as_str()], "");
assert_reported(&doc, Level::A2b, "6.1.7.2");
}
#[test]
fn streams_must_not_read_external_files_clause_6_1_7_1() {
let page = "<< /Type /Page /Parent 2 0 R >>";
let packet = a2b_packet(None);
let held = "<< /Length 4 >>\nstream\nabcd\nendstream";
let doc = page_document(page, Some(packet.as_str()), "", &[held], "");
assert_not_reported(&doc, Level::A2b, "6.1.7.1");
// A stream object nothing refers to: the rule looks at every stream in the file.
let external = "<< /Length 4 /F (data.bin) >>\nstream\nabcd\nendstream";
let doc = page_document(page, Some(packet.as_str()), "", &[external], "");
assert_reported(&doc, Level::A2b, "6.1.7.1");
}
#[test]
fn images_must_not_be_interpolated_clause_6_2_8() {
let page = "<< /Type /Page /Parent 2 0 R /Contents 6 0 R /Resources << /XObject << /Im0 5 0 R >> >> >>";
let packet = a2b_packet(None);
let drawing = content("q 10 0 0 10 0 0 cm /Im0 Do Q");
let smooth = image(true);
let doc = page_document(page, Some(packet.as_str()), "", &[smooth.as_str(), drawing.as_str()], "");
assert_reported(&doc, Level::A2b, "6.2.8");
let crisp = image(false);
let doc = page_document(page, Some(packet.as_str()), "", &[crisp.as_str(), drawing.as_str()], "");
assert_not_reported(&doc, Level::A2b, "6.2.8");
}
#[test]
fn postscript_xobjects_are_reported_clause_6_2_9() {
let plain = "<< /Type /Page /Parent 2 0 R /Contents 5 0 R >>";
let with = "<< /Type /Page /Parent 2 0 R /Contents 5 0 R /Resources << /XObject << /PS0 6 0 R >> >> >>";
let packet = a2b_packet(None);
let drawing = content("q Q");
let doc = page_document(plain, Some(packet.as_str()), "", &[drawing.as_str()], "");
assert_not_reported(&doc, Level::A2b, "6.2.9");
let postscript = "<< /Type /XObject /Subtype /PS /Length 0 >>\nstream\n\nendstream";
let doc = page_document(with, Some(packet.as_str()), "", &[drawing.as_str(), postscript], "");
assert_reported(&doc, Level::A2b, "6.2.9");
}
#[test]
fn fonts_must_be_embedded_clause_6_2_11_4() {
let page = "<< /Type /Page /Parent 2 0 R /Resources << /Font << /F1 5 0 R >> >> >>";
let packet = a2b_packet(None);
let embedded = "<< /Type /Font /Subtype /TrueType /BaseFont /Sans /FontDescriptor 6 0 R >>";
let descriptor = "<< /Type /FontDescriptor /FontName /Sans /Flags 32 /FontFile2 7 0 R >>";
let program = font_program(None);
let doc = page_document(page, Some(packet.as_str()), "", &[embedded, descriptor, program.as_str()], "");
assert_not_reported(&doc, Level::A2b, "6.2.11.4");
let missing = "<< /Type /Font /Subtype /TrueType /BaseFont /Sans >>";
let doc = page_document(page, Some(packet.as_str()), "", &[missing], "");
assert_reported(&doc, Level::A2b, "6.2.11.4");
}
/// A file built by hand, apart from its output profile, with everything the verifier asks for: an
/// XMP packet naming PDF/A-2b with a title that matches the document information, an sRGB output
/// intent and a blank page. It checks that the rules agree with each other, which is not by itself a
/// conformance claim.
#[test]
fn minimal_document_has_no_verifier_issues() {
let page = "<< /Type /Page /Parent 2 0 R >>";
let packet = a2b_packet(Some("Report"));
let profile = hex_stream("/N 3", &icc::srgb());
let info = "/Info << /Title (Report) >>";
let mut doc = page_document(page, Some(packet.as_str()), &output_intent(5), &[profile.as_str()], info);
let issues = verify(&doc, Level::A2b);
assert!(issues.is_empty(), "{issues:#?}");
assert_eq!(declared(&doc).pdfa, Some((2, "B".into())));
assert_eq!(declared(&doc).output_intents, ["sRGB IEC61966-2.1"]);
let report = convert(&mut doc, Level::A2b).unwrap();
assert!(report.remaining.is_empty(), "{:#?}", report.remaining);
assert_eq!(declared(&doc).pdfa, Some((2, "B".into())));
}
crates/redact/Cargo.toml
+5
View File
@@ -9,6 +9,7 @@ license.workspace = true
workspace = true
[dependencies]
aho-corasick = "1"
pdfcraft-cos = { workspace = true }
pdfcraft-content = { workspace = true }
pdfcraft-fonts = { workspace = true }
@@ -20,6 +21,10 @@ pdfcraft-xfa = { workspace = true }
# CCITT Group 4 and JBIG2 encoding for cleared one-bit images.
pdfcraft-filters = { workspace = true }
thiserror = { workspace = true }
sha2 = "0.11"
[dev-dependencies]
pdfcraft-testkit = { workspace = true }
# The redacted page is rendered in `tests::a_page_whose_pruned_pattern_is_still_selected_renders`:
# layer 3 from layer 4, a dev dependency only.
pdfcraft-render = { workspace = true }
crates/redact/README.md
+9
View File
@@ -60,6 +60,15 @@ resume after it would be a guess. A Redact annotation with neither a usable `/Qu
`/Rect` makes it fail with `RedactError::UnreadableMark` before anything changes, instead of
skipping that mark. The count covers the marks on the pages being applied.
Tiling-pattern cells are separate streams. Applying a mark removes a pattern resource when no
surviving paint uses it. If a cell remains reachable after rewriting the selected pages and may
contain text, images, replacement descriptions, or unreadable content, Apply refuses with a
`RedactError::Pattern` and restores the document and pending marks. This includes patterns owned
by Forms and patterns inherited by Forms. Vector-only retained cells remain supported. The
bounded check is conservative: an unrelated retained pattern on a selected page can also cause
refusal when its content cannot be proven safe. If a selected page had a pattern resource, the
check covers all pages after rewriting so another page cannot keep the same cell in the saved PDF.
Before changing any page, `apply` prepares the redaction appearances with a shared 8 MiB
budget for their uncompressed PDF content across the selected pages. It bounds repeated-text
counts and working line buffers before allocating them, and measures and encodes each repeated
crates/redact/fuzz/Cargo.toml
+25
View File
@@ -0,0 +1,25 @@
[package]
name = "pdfcraft-redact-fuzz"
version = "0.0.0"
publish = false
edition = "2024"
license = "MIT OR Apache-2.0"
[package.metadata]
cargo-fuzz = true
# Detached from the PdfCraft workspace: run with `cargo +nightly fuzz run apply`
# from crates/redact/.
[workspace]
[dependencies]
libfuzzer-sys = "0.4"
pdfcraft-cos = { path = "../../cos" }
pdfcraft-redact = { path = ".." }
[[bin]]
name = "apply"
path = "fuzz_targets/apply.rs"
test = false
doc = false
bench = false
crates/redact/fuzz/fuzz_targets/apply.rs
+24
View File
@@ -0,0 +1,24 @@
//! Feeds arbitrary bytes, read as a PDF, to the redaction marks and the apply entry.
//!
//! Results are discarded: the target checks that no input panics.
#![no_main]
use std::sync::Arc;
use libfuzzer_sys::fuzz_target;
use pdfcraft_cos::Document;
use pdfcraft_redact::{apply, marks};
/// The largest input the target opens (16 MiB); larger inputs are skipped.
const MAX_INPUT: usize = 16 << 20;
fuzz_target!(|data: &[u8]| {
if data.len() > MAX_INPUT {
return;
}
let Ok(mut doc) = Document::open(Arc::new(data.to_vec())) else {
return;
};
let _count = marks(&doc).len();
let _result = apply(&mut doc, None);
});
crates/redact/src/image.rs
+1 -1
View File
@@ -12,7 +12,7 @@ use pdfcraft_cos::{Dict, Document, ImageBudget, Object, Stream};
/// Most pixels an image may have to be cleared in place (larger ones are removed), or to be stored
/// again without shared `/JBIG2Globals` (see [`without_globals`]).
const MAX_PIXELS: u64 = pdfcraft_cos::MAX_REDACTED_IMAGE_PIXELS;
pub(crate) const MAX_PIXELS: u64 = pdfcraft_cos::MAX_REDACTED_IMAGE_PIXELS;
fn components(doc: &Document, s: &Stream) -> Option<usize> {
let cs = doc.resolve(s.dict.get(b"ColorSpace")?);
crates/redact/src/interp.rs
+213 -14
View File
@@ -31,7 +31,7 @@ mod prune;
use layers::strip_hidden_layers;
use prune::{clip_out, extgstate_font, prune_form_resources};
pub(crate) use prune::{prune, prune_properties, resource_ops, unlink_doomed};
pub(crate) use prune::{prune, prune_patterns, prune_properties, resource_ops, unlink_doomed, unlink_pruned_patterns};
/// How deep this pass follows form XObjects. A form past it is removed rather than walked, which
/// is the fail-safe choice for a redaction -- marked content inside a form we cannot read would
@@ -53,6 +53,23 @@ pub(crate) enum Mode {
Verify,
}
/// A run of glyphs cut out under a region, kept for the redaction proof
/// ([`Snapshot::capture`](crate::verify::Snapshot::capture)).
pub(crate) struct Removed {
/// Index of the region (in `Scope::rects`) the run lay under.
pub(crate) rect: usize,
/// The character codes as they stood in the string.
pub(crate) raw: Vec<u8>,
/// What the font says they showed (codes without a known meaning are left out).
pub(crate) text: String,
/// How many character codes the run holds (a composite font's codes are several bytes).
pub(crate) codes: usize,
/// Which text operator showed it, counted from the start of the page.
pub(crate) op: usize,
/// Where in that operator it starts: the operand's place in a `TJ` array, and the byte in it.
pub(crate) at: (usize, usize),
}
#[derive(Clone)]
struct Gs {
ctm: Matrix,
@@ -68,6 +85,17 @@ struct Gs {
scale: f64,
leading: f64,
rise: f64,
/// The tiling or shading pattern selected as the fill colour (`/Name scn`), and as the stroke
/// colour (`/Name SCN`). Part of the graphics state, so `q`/`Q` save and restore them with
/// everything else. A pattern's cell is a content stream of its own and this pass does not
/// enter it, so a pattern whose only painting was removed has to leave the page's resources
/// or its drawing stays in the file (#3647).
fill_pattern: Option<Vec<u8>>,
stroke_pattern: Option<Vec<u8>>,
/// The current font is a Type 3 font without `/Resources`: its glyph descriptions look their
/// names up in this scope's resources (ISO 32000-2 §9.6.4), so a shown glyph may paint with
/// any of this scope's patterns.
type3_inherits: bool,
}
/// What processing a scope produced.
@@ -85,6 +113,16 @@ pub(crate) struct Output {
/// dictionaries may name them too (an inherited or shared /Resources, or the caller's when a
/// form has none of its own): [`unlink_doomed`] removes those entries.
pub doomed: Vec<ObjRef>,
/// Pattern resource names a painting operator that survived the rewrite still uses. A name
/// this scope saw selected but never painted with afterwards is pruned from the page's
/// `/Resources /Pattern`, so a cell whose only drawing was redacted does not stay in the
/// saved file (#3647).
pub patterns_seen: Vec<Vec<u8>>,
pub patterns_painted: Vec<Vec<u8>>,
/// A Type 3 glyph without resources of its own was shown in this scope or a nested form,
/// through a font set by `Tf` or an ExtGState: which of the page's patterns it paints with is
/// not tracked, so the page keeps them all.
pub keeps_all_patterns: bool,
/// Original property values retired by this scope or a nested form. Other resource holders
/// may still name them; unlink only entries no surviving content uses.
pub retired_properties: Vec<Object>,
@@ -124,6 +162,12 @@ pub(crate) struct Scope<'a> {
xobject_oc_cache: HashMap<ObjRef, bool>,
/// Content removed from hidden layers (blocks and XObjects).
pub layer_blocks: usize,
/// Collects the glyph runs removed under a region in apply mode (`None` = not collected).
/// The redaction proof's snapshot is what sets it: it must know what applying is about to
/// remove, by the interpreter's own judgement, without trusting anything else to it.
pub removed: Option<Vec<Removed>>,
/// How many text operators have been shown, for [`Removed::op`].
shown: usize,
fonts: HashMap<Vec<u8>, Rc<Metrics>>,
used_names: Vec<Vec<u8>>,
depth: usize,
@@ -149,6 +193,8 @@ impl<'a> Scope<'a> {
ocmd_cache: HashMap::new(),
xobject_oc_cache: HashMap::new(),
layer_blocks: 0,
removed: None,
shown: 0,
fonts: HashMap::new(),
used_names: Vec::new(),
depth: 0,
@@ -169,21 +215,33 @@ impl<'a> Scope<'a> {
self.rects.iter().any(|r| contains(*r, b, 0.01))
}
/// Does a glyph box count as under a region? Any real overlap does (more than a fifth of
/// Which regions a glyph box counts as under? Any real overlap does (more than a fifth of
/// the glyph or a point, whichever is less, in both directions), so neighbours that only
/// touch an edge survive while partly covered glyphs go. Degenerate boxes (zero-size text)
/// count when their origin is inside.
fn glyph_hit(&self, b: [f64; 4], origin: (f64, f64)) -> bool {
/// count when their origin is inside. Empty = under no region (a glyph kept). A glyph may
/// lie under several regions at once (overlapping marks); each is reported.
fn glyph_rects(&self, b: [f64; 4], origin: (f64, f64)) -> Vec<usize> {
let (gw, gh) = (b[2] - b[0], b[3] - b[1]);
if gw < 0.01 || gh < 0.01 {
return self.rects.iter().any(|r| origin.0 >= r[0] && origin.0 <= r[2] && origin.1 >= r[1] && origin.1 <= r[3]);
return self
.rects
.iter()
.enumerate()
.filter(|(_, r)| origin.0 >= r[0] && origin.0 <= r[2] && origin.1 >= r[1] && origin.1 <= r[3])
.map(|(i, _)| i)
.collect();
}
let (mx, my) = ((gw * 0.2).min(1.0), (gh * 0.2).min(1.0));
self.rects.iter().any(|r| {
let ox = b[2].min(r[2]) - b[0].max(r[0]);
let oy = b[3].min(r[3]) - b[1].max(r[1]);
ox > mx && oy > my
})
self.rects
.iter()
.enumerate()
.filter(|(_, r)| {
let ox = b[2].min(r[2]) - b[0].max(r[0]);
let oy = b[3].min(r[3]) - b[1].max(r[1]);
ox > mx && oy > my
})
.map(|(i, _)| i)
.collect()
}
}
@@ -223,6 +281,24 @@ struct Glyph {
/// Advance in unscaled text space (before `Th`).
advance: f64,
hit: bool,
/// The regions it lies under (empty for hidden text).
rects: Vec<usize>,
}
/// Record the patterns an operator paints with, so [`prune_patterns`] keeps them. Only path
/// painting knows exactly what it draws and whether the paint survived; text, images and forms
/// are recorded as they are met, which can keep a pattern whose paint was in fact all redacted,
/// and with it the pattern's cell. A cell that carries text or images is then refused by
/// `pattern_guard`; a vector-only cell stays in the file though nothing shows it
/// (`a_pattern_painted_only_by_redacted_text_keeps_its_vector_cell`). Dropping a pattern something
/// still draws with would lose paint from the page instead.
fn uses_patterns(out: &mut Output, gs: &Gs, fills: bool, strokes: bool) {
if fills && let Some(n) = &gs.fill_pattern {
out.patterns_painted.push(n.clone());
}
if strokes && let Some(n) = &gs.stroke_pattern {
out.patterns_painted.push(n.clone());
}
}
/// Process a scope's content streams (state flows from one stream to the next).
@@ -238,6 +314,9 @@ pub(crate) fn process(doc: &mut Document, scope: &mut Scope<'_>, streams: &[Vec<
scale: 1.0,
leading: 0.0,
rise: 0.0,
fill_pattern: None,
stroke_pattern: None,
type3_inherits: false,
};
process_from(doc, scope, streams, resources, gs)
}
@@ -338,6 +417,8 @@ struct Walk {
fonts_res: Dict,
ext_gstates: Dict,
xobjects: Dict,
/// This scope's pattern names, which a Type 3 glyph without resources may paint with.
patterns: Vec<Vec<u8>>,
fallback: Rc<Metrics>,
/// The current path: its operators, its bounding box in user space, and whether it clips.
path: Vec<Op>,
@@ -356,6 +437,7 @@ impl Walk {
fonts_res: res_dict(doc, resources, b"Font"),
ext_gstates: res_dict(doc, resources, b"ExtGState"),
xobjects: res_dict(doc, resources, b"XObject"),
patterns: res_dict(doc, resources, b"Pattern").iter().map(|(n, _)| n.to_vec()).collect(),
fallback: Rc::new(Metrics::fallback()),
path: Vec::new(),
path_box: None,
@@ -394,6 +476,39 @@ impl Walk {
b"BI" => Self::inline_image(&self.gs, scope, &mut self.out, op, piece),
b"sh" if scope.mode == Mode::Apply => Self::shading(&self.gs, scope, op, piece),
b"Do" => self.draw_xobject(doc, scope, resources, op, piece),
// `/Name scn` selects a pattern as the fill colour; numbers select a plain colour
// and clear it. `cs` with a space other than /Pattern clears it too.
b"scn" | b"SCN" => {
let fill = o == b"scn";
let picked = op.operands.last().and_then(Object::as_name).map(<[u8]>::to_vec);
if let Some(name) = &picked {
self.out.patterns_seen.push(name.clone());
}
if fill {
self.gs.fill_pattern = picked;
} else {
self.gs.stroke_pattern = picked;
}
Some(op)
}
b"cs" | b"CS" => {
if op.name(0) != Some(b"Pattern") {
if o == b"cs" {
self.gs.fill_pattern = None;
} else {
self.gs.stroke_pattern = None;
}
}
Some(op)
}
b"g" | b"rg" | b"k" => {
self.gs.fill_pattern = None;
Some(op)
}
b"G" | b"RG" | b"K" => {
self.gs.stroke_pattern = None;
Some(op)
}
_ => Some(op),
};
piece.ops.extend(kept);
@@ -425,6 +540,12 @@ impl Walk {
let clips = std::mem::replace(&mut self.clip, false);
let hit = scope.mode == Mode::Apply && op.op.as_slice() != b"n" && bbox.is_some_and(|b| scope.hits(b));
if !hit {
// The paint survives, so the pattern it draws with is still in use. `n` paints
// nothing, so it uses none. A hit path is removed whole (F-008), pattern and all.
let o = op.op.as_slice();
let strokes = matches!(o, b"S" | b"s" | b"B" | b"B*" | b"b" | b"b*");
let fills = matches!(o, b"f" | b"F" | b"f*" | b"B" | b"B*" | b"b" | b"b*");
uses_patterns(&mut self.out, &self.gs, fills, strokes);
piece.ops.extend(body);
piece.ops.push(op);
return;
@@ -458,6 +579,8 @@ impl Walk {
_ => {
if let Some((font, vertical, size)) = op.name(0).and_then(|n| extgstate_font(doc, scope, &self.ext_gstates, n, &self.fallback)) {
(self.gs.font, self.gs.vertical, self.gs.size) = (font, vertical, size);
// As for `Tf`: a Type 3 font without /Resources paints through this scope's.
self.gs.type3_inherits = op.name(0).is_some_and(|n| extgstate_type3_inherits(doc, &self.ext_gstates, n));
}
}
}
@@ -468,6 +591,7 @@ impl Walk {
let Some(name) = op.name(0) else { return };
let font_dict = self.fonts_res.get(name).and_then(|f| doc.resolve(f).as_dict().cloned());
self.gs.vertical = font_dict.as_ref().is_some_and(|d| is_vertical(doc, d));
self.gs.type3_inherits = font_dict.as_ref().is_some_and(|d| type3_without_resources(doc, d));
self.gs.font = match scope.fonts.get(name) {
Some(f) => f.clone(),
None => {
@@ -516,6 +640,16 @@ impl Walk {
/// `Tj`, `TJ`, `'` and `"`: glyphs under a region are cut out (or counted, when verifying).
fn show_text(&mut self, scope: &mut Scope<'_>, op: Op, piece: &mut Piece) -> Option<Op> {
// Text paints with the fill pattern, and with the stroke pattern in the stroking render
// modes. Mode 3 paints nothing and mode 7 only clips.
let mode = self.gs.render_mode;
uses_patterns(&mut self.out, &self.gs, matches!(mode, 0 | 2 | 4 | 6), matches!(mode, 1 | 2 | 5 | 6));
// A Type 3 glyph without resources of its own paints through this scope's: keep every
// pattern it could select, as with a form that has no /Resources.
if self.gs.type3_inherits {
self.out.patterns_painted.extend(self.patterns.iter().cloned());
self.out.keeps_all_patterns = true;
}
let o = op.op.as_slice();
if o == b"\"" {
self.gs.word_spacing = op.num(0).unwrap_or(self.gs.word_spacing);
@@ -569,6 +703,8 @@ impl Walk {
/// An inline image under a region is removed (or counted, when verifying).
fn inline_image(gs: &Gs, scope: &mut Scope<'_>, out: &mut Output, op: Op, piece: &mut Piece) -> Option<Op> {
// An image mask paints with the fill colour, pattern included.
uses_patterns(out, gs, true, false);
if !scope.hits(gs.ctm.bbox([0.0, 0.0, 1.0, 1.0])) {
return Some(op);
}
@@ -593,6 +729,9 @@ impl Walk {
}
fn draw_xobject(&mut self, doc: &mut Document, scope: &mut Scope<'_>, resources: &Dict, op: Op, piece: &mut Piece) -> Option<Op> {
// An image mask paints with the fill colour, and a form's content is interpreted with a
// graphics state of its own, so a pattern it inherits from here is never recorded inside it.
uses_patterns(&mut self.out, &self.gs, true, true);
let Some(new) = xobject(doc, scope, &op, &self.gs, resources, &mut self.xobjects, &mut self.out) else { return Some(op) };
piece.changed = true;
piece.ops.extend(new);
@@ -644,11 +783,14 @@ fn clean_replacements(doc: &Document, resources: &Dict, original: &[Op], process
/// Lay out a shown string (or `TJ` array) glyph by glyph, advancing `tm`. Returns the `TJ`
/// items with the glyphs under a region replaced by displacements, and how many were removed.
fn show(scope: &Scope<'_>, gs: &Gs, tm: &mut Matrix, items: &[Object]) -> (Vec<Object>, usize) {
/// In apply mode the runs removed under each region are also handed to `scope.removed`.
fn show(scope: &mut Scope<'_>, gs: &Gs, tm: &mut Matrix, items: &[Object]) -> (Vec<Object>, usize) {
let f = &gs.font;
let size = gs.size;
let mut out: Vec<Object> = Vec::new();
let mut removed = 0;
scope.shown += 1;
let op = scope.shown;
let push_num = |out: &mut Vec<Object>, v: f64| {
if v == 0.0 {
return;
@@ -661,7 +803,7 @@ fn show(scope: &Scope<'_>, gs: &Gs, tm: &mut Matrix, items: &[Object]) -> (Vec<O
}
out.push(num(v));
};
for item in items {
for (place, item) in items.iter().enumerate() {
match item {
Object::String(s) => {
let bytes = &s.bytes;
@@ -674,18 +816,50 @@ fn show(scope: &Scope<'_>, gs: &Gs, tm: &mut Matrix, items: &[Object]) -> (Vec<O
let b = trm.bbox([0.0, f.descent, w0, f.ascent]);
let origin = trm.apply(0.0, 0.0);
let advance = w0 * size + spacing;
let mut rects = Vec::new();
let hit = match scope.hidden_text {
// Invisible (Tr 3) and clip-only (Tr 7) text, or text wholly off the page.
Some(page) => {
let degenerate = b[2] - b[0] < 0.01 && b[3] - b[1] < 0.01;
matches!(gs.render_mode, 3 | 7) || !(overlaps(page, b, 0.0) || degenerate)
}
None => scope.glyph_hit(b, origin),
None => {
rects = scope.glyph_rects(b, origin);
!rects.is_empty()
}
};
glyphs.push(Glyph { bytes: pos..pos + len, advance, hit });
glyphs.push(Glyph { bytes: pos..pos + len, advance, hit, rects });
*tm = Matrix::translate(advance * gs.scale, 0.0).then(tm);
pos += len;
}
if scope.mode == Mode::Apply
&& let Some(sink) = scope.removed.as_mut()
{
// Consecutive removed glyphs under one region are one run, recorded for
// every region the glyphs lie under: overlapping marks each account for
// their own share of what was cut.
let mut under: Vec<usize> = glyphs.iter().flat_map(|g| g.rects.iter().copied()).collect();
under.sort_unstable();
under.dedup();
for r in under {
let mut k = 0;
while let Some(g) = glyphs.get(k) {
if !g.rects.contains(&r) {
k += 1;
continue;
}
let mut end = k;
while glyphs.get(end + 1).is_some_and(|n| n.rects.contains(&r)) {
end += 1;
}
let span = g.bytes.start..glyphs.get(end).map_or(g.bytes.end, |l| l.bytes.end);
let at = (place, span.start);
let raw = bytes.get(span).unwrap_or_default().to_vec();
sink.push(Removed { rect: r, text: f.decode(&raw), raw, codes: end - k + 1, op, at });
k = end + 1;
}
}
}
let mut run: Vec<u8> = Vec::new();
for g in &glyphs {
if g.hit {
@@ -850,8 +1024,18 @@ fn xobject(
out.ambiguous |= inner.ambiguous;
out.doomed.extend_from_slice(&inner.doomed);
out.retired_properties.extend(inner.retired_properties);
// A Type 3 glyph without resources shown inside the form may paint with the page's
// patterns (the renderer resolves its names there), whether or not the form has
// /Resources of its own.
out.keeps_all_patterns |= inner.keeps_all_patterns;
out.tag_changes.extend(inner.tag_changes);
out.error = inner.error.or(out.error.take());
// A form without /Resources of its own paints with its caller's patterns, so what it
// still paints keeps the caller's entry. Only the painted side is passed up: a name
// the form selected but no longer paints with is left for the caller to keep.
if own.is_none() {
out.patterns_painted.extend(inner.patterns_painted);
}
if scope.mode == Mode::Verify {
return None;
}
@@ -918,3 +1102,18 @@ fn xobject(
_ => None,
}
}
/// Whether the ExtGState `name` sets a Type 3 font that has no `/Resources` of its own.
fn extgstate_type3_inherits(doc: &Document, ext_gstates: &Dict, name: &[u8]) -> bool {
let Some(state) = ext_gstates.get(name).map(|s| doc.resolve(s)) else { return false };
let Some(font) = state.as_dict().and_then(|d| d.get(b"Font")).map(|f| doc.resolve(f)) else { return false };
let Some(reference) = font.as_array().and_then(|a| a.first()) else { return false };
let dict = doc.resolve(reference);
dict.as_dict().is_some_and(|d| type3_without_resources(doc, d))
}
/// A Type 3 font whose glyph descriptions use the resources of whatever shows them: no
/// `/Resources`, or `/Resources null`, which is the same thing.
fn type3_without_resources(doc: &Document, font: &Dict) -> bool {
font.name(b"Subtype") == Some(b"Type3") && font.get(b"Resources").is_none_or(|r| matches!(&*doc.resolve(r), Object::Null))
}
crates/redact/src/interp/prune.rs
+136
View File
@@ -355,3 +355,139 @@ pub(super) fn extgstate_font(
};
Some((metrics, vertical, size))
}
/// Drop every `/Pattern` entry this scope saw selected but no surviving painting operator drew
/// with. The `scn` that selected it stays in the content, so the page names a pattern its
/// resources no longer hold; nothing surviving paints with that colour, and
/// `a_page_whose_pruned_pattern_is_still_selected_renders` pins that the page still renders and
/// that operators after the stale selector still draw. A pattern's cell is a content stream of its own which
/// this pass does not enter, so a pattern whose only painting was redacted would otherwise keep
/// its drawing -- text included -- in the saved file (#3647). A pattern some surviving paint
/// still uses, or one the scope never saw, is left alone.
/// `/Pattern` may be an indirect object like any other value, so it is resolved: reading the
/// reference itself as a dictionary found none and pruned nothing, leaving the cell in the file.
/// What is written back is a dictionary of this page's own, so pruning never reaches through a
/// `/Pattern` dictionary another page shares. The pattern objects dropped are added to `pruned`,
/// for [`unlink_pruned_patterns`].
pub(crate) fn prune_patterns(doc: &Document, res: &mut Dict, seen: &[Vec<u8>], painted: &[Vec<u8>], pruned: &mut Vec<ObjRef>) -> usize {
let Some(mut pat) = res.get(b"Pattern").map(|p| doc.resolve(p)).and_then(|p| p.as_dict().cloned()) else { return 0 };
let mut dropped = 0;
for n in seen {
if !painted.contains(n)
&& let Some(value) = pat.get(n)
{
// Every object along the reference chain: a name can lead to a reference-valued
// object whose value is the cell, and the cell is what must not stay reachable.
let mut next = value.as_ref();
for _ in 0..MAX_DEPTH {
let Some(reference) = next else { break };
pruned.push(reference);
next = match &*doc.get(reference) {
Object::Ref(r) => Some(*r),
_ => None,
};
}
pat.remove(n);
dropped += 1;
}
}
if dropped > 0 {
if pat.is_empty() {
res.remove(b"Pattern");
} else {
res.set(b"Pattern".to_vec(), Object::Dict(pat));
}
}
dropped
}
/// Unlink the patterns a page's rewrite pruned (`pruned`) from page-tree nodes whose
/// `/Resources` no page inherits any more: a full save keeps whatever Root still reaches, so a
/// `/Pages` node above the page that still names the cell would keep it in the file. That happens
/// when the page's nearest holder was retired but a holder further up, which it shadowed, still
/// names the pattern. A holder some page still inherits from is left alone; whatever stays
/// reachable is checked by `pattern_guard`, which refuses a retained cell it cannot prove safe.
/// When any node's `/Parent` disagrees with where the `/Kids` tree puts it (or the root has
/// one), which holder a page inherits from is ambiguous and a node may be reached only through
/// `/Parent`, so nothing is unlinked and the guard, which follows `/Parent` too, decides.
pub(crate) fn unlink_pruned_patterns(doc: &mut Document, pruned: &[ObjRef]) -> Result<(), RedactError> {
if pruned.is_empty() {
return Ok(());
}
let pruned: HashSet<ObjRef> = pruned.iter().copied().collect();
let Some(top) = doc.root().and_then(|root| doc.get(root).as_dict().and_then(|d| d.reference(b"Pages"))) else { return Ok(()) };
// Every node with a /Resources of its own, and whether any page inherits from it (it is the
// nearest holder above a page, or the page itself). Cycles are cut as in `unlink_doomed`.
let mut holders: Vec<(ObjRef, bool)> = Vec::new();
let mut seen = HashSet::new();
let mut consistent = true;
let mut stack: Vec<(ObjRef, Option<ObjRef>, Option<usize>)> = vec![(top, None, None)];
seen.insert(top);
while let Some((node, parent, mut holder)) = stack.pop() {
let obj = doc.get(node);
let Some(d) = obj.as_dict() else { continue };
// Every node's /Parent must be the node /Kids reached it from, and the root has none.
// A node reached only through a /Parent link is then impossible here; when the links
// disagree, the guard inspects every node they reach instead. A /Parent that is not a
// reference (a direct dictionary, written inline by a full save) never agrees.
let agrees = match d.get(b"Parent") {
None => parent.is_none(),
Some(Object::Ref(r)) => Some(*r) == parent,
Some(_) => false,
};
if !agrees {
consistent = false;
break;
}
if d.contains(b"Resources") {
holders.push((node, false));
holder = Some(holders.len() - 1);
}
let is_pages = d.name(b"Type") == Some(b"Pages") || (d.contains(b"Kids") && d.name(b"Type") != Some(b"Page"));
if is_pages {
// Borrowed, not copied: a /Kids array can be shared by many hostile nodes.
let kids_object = d.get(b"Kids").map(|k| doc.resolve(k));
for kid in kids_object.as_deref().and_then(Object::as_array).into_iter().flatten().filter_map(Object::as_ref) {
// A node reached twice (a shared /Kids entry or a cycle) has a /Parent that agrees
// with at most one of the nodes above it.
if !seen.insert(kid) || seen.len() > 1_000_000 {
consistent = false;
break;
}
stack.push((kid, Some(node), holder));
}
if !consistent {
break;
}
} else if let Some(h) = holder.and_then(|i| holders.get_mut(i)) {
h.1 = true;
}
}
if !consistent {
return Ok(());
}
for (node, inherited) in holders {
if inherited {
continue;
}
let Some(mut res) = doc.get(node).as_dict().and_then(|d| d.get(b"Resources")).and_then(|r| doc.resolve(r).as_dict().cloned()) else {
continue;
};
let mut pat = res_dict(doc, &res, b"Pattern");
let names: Vec<Vec<u8>> = pat.iter().filter(|(_, v)| v.as_ref().is_some_and(|r| pruned.contains(&r))).map(|(n, _)| n.to_vec()).collect();
if names.is_empty() {
continue;
}
for n in &names {
pat.remove(n);
}
if pat.is_empty() {
res.remove(b"Pattern");
} else {
res.set(b"Pattern".to_vec(), Object::Dict(pat));
}
// A private copy on this node: the dictionary object may be shared with other nodes.
doc.update_dict(node, |d| d.set(b"Resources".to_vec(), Object::Dict(res)))?;
}
Ok(())
}
crates/redact/src/lib.rs
+147 -6
View File
@@ -22,15 +22,21 @@ pub mod codes;
mod image;
mod interp;
mod overlay;
mod pattern_guard;
pub mod patterns;
pub mod sanitize;
mod tags;
#[cfg(test)]
mod tests;
#[cfg(test)]
mod tests_verify;
pub(crate) mod type3;
pub(crate) mod type3_tables;
pub mod verify;
use interp::{Mode, Scope, process};
pub use pattern_guard::PatternProblem;
pub use verify::{Proof, ProofOptions, Snapshot};
pub type Rgb = [f64; 3];
@@ -49,6 +55,8 @@ pub enum RedactError {
#[error("redaction could not be verified: {0} item(s) were still found under the marks, so nothing was changed")]
Residue(usize),
/// A mark's overlay text has a character Helvetica can't draw; nothing was redacted.
#[error("page {page} cannot be redacted safely: {reason}; nothing was changed")]
Pattern { page: usize, reason: PatternProblem },
#[error("{0}")]
Undrawable(String),
#[error(
@@ -59,6 +67,11 @@ pub enum RedactError {
"redaction appearance has invalid geometry or exceeds the 8 MiB budget; reduce the marked area, turn off repeated text or apply fewer pages at once (nothing was redacted)"
)]
OverlayLimit,
/// A stream the redaction created to draw the marks' boxes and labels can't be read back, so
/// the proof can't tell it apart from content that was there before and nothing can be
/// exempted from the sweep.
#[error("a stream the redaction created can't be read back, so the result can't be proven")]
OverlayUnreadable,
/// The structure tree couldn't be cleaned completely (#3302): an element written in place that
/// would need changing, an element whose inherited page can't be found, or a tree too large to
/// walk. Tags that kept the removed text or annotation would undo the redaction.
@@ -137,6 +150,8 @@ fn overlay_look(doc: &Document, d: &Dict) -> pdfcraft_annot::OverlayLook {
/// What [`apply`] removed.
#[derive(Clone, Debug, Default, PartialEq)]
pub struct Report {
/// Patterns dropped from a page's resources because nothing painted with them any more.
pub patterns_removed: usize,
pub marks: usize,
pub pages: usize,
pub glyphs: usize,
@@ -164,7 +179,7 @@ pub struct Report {
pub overlays: Vec<ObjRef>,
}
fn rect_of(doc: &Document, o: Option<&Object>) -> Option<[f64; 4]> {
pub(crate) fn rect_of(doc: &Document, o: Option<&Object>) -> Option<[f64; 4]> {
let v: Vec<f64> = doc.resolve(o?).as_array()?.iter().filter_map(|x| doc.resolve(x).as_f64()).collect();
(v.len() == 4 && v.iter().all(|x| x.is_finite())).then(|| [v[0].min(v[2]), v[1].min(v[3]), v[0].max(v[2]), v[1].max(v[3])])
}
@@ -388,6 +403,38 @@ pub fn apply(doc: &mut Document, pages: Option<&[usize]>) -> Result<Report, Reda
Ok(report)
}
// Shared appearance stream references can keep a removed annotation's pattern cell reachable
// from an unselected page. Collect only the appearance map and its state streams, not unrelated
// annotation data such as fonts or metadata.
fn appearance_refs(doc: &Document, annotation: &Dict) -> Option<HashSet<ObjRef>> {
const MAX_REFERENCES: usize = 16_384;
let mut references = HashSet::new();
let Some(ap) = annotation.get(b"AP") else { return Some(references) };
if let Some(reference) = ap.as_ref() {
references.insert(reference);
}
let resolved_ap = doc.resolve(ap);
let Some(appearances) = resolved_ap.as_dict() else { return Some(references) };
for key in [b"N".as_slice(), b"R", b"D"] {
let Some(value) = appearances.get(key) else { continue };
if let Some(reference) = value.as_ref() {
references.insert(reference);
}
let resolved = doc.resolve(value);
if let Object::Dict(states) = &*resolved {
for (_, state) in states.iter() {
if let Some(reference) = state.as_ref() {
references.insert(reference);
if references.len() > MAX_REFERENCES {
return None;
}
}
}
}
}
Some(references)
}
/// [`apply`] without the copy: an error can leave `doc` part redacted.
fn apply_to(doc: &mut Document, pages: Option<&[usize]>) -> Result<Report, RedactError> {
let images = pdfcraft_cos::ImageBudget::new(doc);
@@ -420,6 +467,16 @@ fn apply_within(doc: &mut Document, pages: Option<&[usize]>, images: pdfcraft_co
// content rewriting and verification instead of scanning all marks for every page.
let groups: Vec<_> = chosen.chunk_by(|a, b| a.page == b.page).filter_map(|group| group.first().map(|mark| (mark.page, group))).collect();
let by_page: Vec<_> = groups.iter().map(|(page, _)| *page).collect();
// A pattern pruned from a selected page can remain reachable through another page's
// resources. When a selected page had patterns, inspect every page after the rewrite.
let source_pages = pdfcraft_model::pages(doc);
let shared_pattern_possible = by_page.iter().any(|&pi| {
source_pages
.get(pi)
.and_then(|page| page.dict.get(b"Resources"))
.map(|resources| doc.resolve(resources))
.is_some_and(|resources| resources.as_dict().is_some_and(|dict| dict.get(b"Pattern").is_some()))
});
// Budget failures must leave even a direct library caller's document untouched.
let mut remaining = overlay::MAX_BYTES;
let overlays = groups
@@ -439,6 +496,9 @@ fn apply_within(doc: &mut Document, pages: Option<&[usize]>, images: pdfcraft_co
widget_owner,
removed_everywhere: HashSet::new(),
images,
removed_appearances: HashSet::new(),
appearance_scan_limit: false,
pruned_patterns: Vec::new(),
};
for &(pi, page_marks) in &groups {
let page = pdfcraft_model::pages(doc).swap_remove(pi);
@@ -449,7 +509,18 @@ fn apply_within(doc: &mut Document, pages: Option<&[usize]>, images: pdfcraft_co
if applying.images.refused() > 0 {
return Err(RedactError::ImagesPastAllowance);
}
let Applying { mut report, doomed_fields, doomed_xobjects, retired_properties, mut removed_everywhere, mut images, .. } = applying;
let Applying {
mut report,
doomed_fields,
doomed_xobjects,
retired_properties,
mut removed_everywhere,
mut images,
mut removed_appearances,
mut appearance_scan_limit,
pruned_patterns,
..
} = applying;
// 2b. Replies and pop-ups of a removed annotation can be listed on another page than their
// parent: their `/IRT` or `/Parent` would keep the removed comment reachable.
@@ -472,6 +543,19 @@ fn apply_within(doc: &mut Document, pages: Option<&[usize]>, images: pdfcraft_co
d.get(b"IRT").and_then(Object::as_ref).is_some_and(|x| removed_everywhere.contains(&x)) && d.name(b"Subtype") != Some(b"Widget");
let is_popup = d.get(b"Parent").and_then(Object::as_ref).is_some_and(|x| removed_everywhere.contains(&x));
if removed_everywhere.contains(&r) || is_reply || is_popup {
// A reply or pop-up removed here can share its appearance with an annotation
// that stays on an unselected page, like one removed under a mark: the
// all-pages pattern check must see it.
if !appearance_scan_limit {
match appearance_refs(doc, d) {
Some(references) => removed_appearances.extend(references),
None => appearance_scan_limit = true,
}
if removed_appearances.len() > 16_384 {
appearance_scan_limit = true;
removed_appearances.clear();
}
}
if removed_everywhere.insert(r) {
grew = true;
if is_reply {
@@ -532,6 +616,8 @@ fn apply_within(doc: &mut Document, pages: Option<&[usize]>, images: pdfcraft_co
// A removed image or form must not stay reachable through another resource dictionary (an
// inherited or shared /Resources): a full save keeps whatever is still referenced.
interp::unlink_doomed(doc, &doomed_xobjects, &retired_properties)?;
// Likewise a pruned pattern: a /Pages node above the page can still name its cell.
interp::unlink_pruned_patterns(doc, &pruned_patterns)?;
report.images_restored = restore_shared_jbig2(doc, &doomed_xobjects, &mut images)?;
// 5. Verify: nothing readable may remain under a region.
@@ -554,8 +640,32 @@ fn apply_within(doc: &mut Document, pages: Option<&[usize]>, images: pdfcraft_co
if residue > 0 {
return Err(RedactError::Residue(residue));
}
// The previous revision still holds the removed content: the next save must rewrite.
doc.require_full_save();
// The interpreter can remove a pattern's paint while another use keeps the cell reachable.
// Inspect the finished selected pages, including rewritten Forms, and all pages if a removed
// pattern or appearance may still be shared. Retained unsafe cells refuse the redaction;
// `apply` runs on a copy, so the refusal leaves the caller's document as it was.
let shared_appearance_possible = appearance_scan_limit
|| !removed_appearances.is_empty()
&& pdfcraft_model::pages(doc).iter().enumerate().any(|(index, page)| {
!by_page.contains(&index)
&& annots_of(doc, &page.dict).iter().any(|annotation| {
doc.resolve(annotation).as_dict().is_some_and(|dict| {
appearance_refs(doc, dict).is_none_or(|refs| refs.iter().any(|reference| removed_appearances.contains(reference)))
})
})
});
let first_page = by_page.first().map_or(1, |p| p + 1);
let check_pages =
if shared_pattern_possible || shared_appearance_possible { (0..pdfcraft_model::pages(doc).len()).collect::<Vec<_>>() } else { by_page };
// A pruned pattern may also stay named by a /Pages node no page inherits from, or by one a
// page with a broken /Parent link still reaches: then every page-tree node is inspected too.
pattern_guard::check(doc, &check_pages, shared_pattern_possible || !pruned_patterns.is_empty())?;
// Whatever reference keeps a pruned pattern in the file -- one no walk above follows -- the
// full save would write it: prove it safe or refuse.
pattern_guard::check_pruned_unreachable(doc, &pruned_patterns, first_page)?;
// The previous revision still holds the removed content: the next save must rewrite, and the
// rewritten file is a different file from its source, so it gets a new identifier.
doc.require_full_save_with_new_id();
Ok(report)
}
@@ -640,6 +750,13 @@ struct Applying {
removed_everywhere: HashSet<ObjRef>,
/// What clearing images may spend, across the run's pages.
images: pdfcraft_cos::ImageBudget,
/// Appearance streams of the annotations removed, on any page: one shared with a kept
/// annotation keeps a pruned pattern's cell reachable (see `pattern_guard`).
removed_appearances: HashSet<ObjRef>,
/// Too many appearance references to track: the pattern guard inspects every page.
appearance_scan_limit: bool,
/// Pattern objects pruned from a page's resources, which page-tree nodes may still name.
pruned_patterns: Vec<ObjRef>,
}
/// Step 1 for one page: rewrites its content so nothing readable stays under a mark, and cleans the tags it no longer shows.
@@ -650,7 +767,7 @@ fn redact_page_content(
page: &pdfcraft_model::Page,
rects: &[[f64; 4]],
) -> Result<(), RedactError> {
let Applying { report, doomed_xobjects, retired_properties, images, .. } = applying;
let Applying { report, doomed_xobjects, retired_properties, images, pruned_patterns, .. } = applying;
// 1. Content.
let mut budget = pdfcraft_cos::DecodeBudget::page();
let (list, data) = page_streams_within(doc, &page.dict, pi, &mut budget)?;
@@ -723,6 +840,11 @@ fn redact_page_content(
},
pi,
)?;
// A pattern this page selected but no surviving paint draws leaves its resources, so
// its cell -- a content stream this pass does not enter -- cannot stay in the file.
// A Type 3 glyph without resources paints with patterns nobody tracked: keep them all.
let painted = if out.keeps_all_patterns { &out.patterns_seen } else { &out.patterns_painted };
report.patterns_removed += interp::prune_patterns(doc, &mut res, &out.patterns_seen, painted, pruned_patterns);
retired_properties.extend(interp::prune_properties(doc, &mut res, &cleaned_data)?);
type3::prune_type3_from_resources(doc, &mut res, &cleaned_data)?;
doc.update_dict(page.obj, |d| {
@@ -751,7 +873,7 @@ fn remove_page_annotations(
rects: &[[f64; 4]],
page_marks: &[&Mark],
) -> Result<(), RedactError> {
let Applying { report, doomed_fields, widget_owner, removed_everywhere, .. } = applying;
let Applying { report, doomed_fields, widget_owner, removed_everywhere, removed_appearances, appearance_scan_limit, .. } = applying;
// 2. Annotations: the marks, and whatever lies under them (with their pop-ups).
let annots = annots_of(doc, &doc.get(page.obj).as_dict().cloned().unwrap_or_default());
let mut removed: Vec<ObjRef> = Vec::new();
@@ -809,6 +931,25 @@ fn remove_page_annotations(
}
}
removed_everywhere.extend(removed.iter().copied());
// A removed annotation's appearance streams can be shared with a kept one on another page,
// which would keep a pruned pattern's cell reachable: remember them for the pattern guard.
for reference in &removed {
if *appearance_scan_limit {
break;
}
if let Some(annotation) = doc.get(*reference).as_dict() {
match appearance_refs(doc, annotation) {
Some(references) => {
removed_appearances.extend(references);
if removed_appearances.len() > 16_384 {
*appearance_scan_limit = true;
removed_appearances.clear();
}
}
None => *appearance_scan_limit = true,
}
}
}
// Pop-ups dropped with their parent below count too: a reply on another page may answer one.
for a in &annots {
let Some(r) = a.as_ref() else { continue };
crates/redact/src/pattern_guard.rs
+496
View File
@@ -0,0 +1,496 @@
//! Refuse pattern content that remains reachable after redaction (#3647).
//!
//! Pattern cells paint through a path, not `Do`, so clipping/removing the path does not remove
//! their bytes. Inspect the rewritten pages' resource graphs before committing the
//! working copy. This is deliberately conservative: an unused text/image-bearing pattern is
//! refused too. A plain vector pattern remains supported. The bounded read-only pass follows
//! the fail-closed approach proposed in PR #162.
use std::collections::HashSet;
use pdfcraft_cos::{Dict, Document, ObjRef, Object, Stream};
use crate::RedactError;
/// How deep the walk may recurse. Every step down (a reference, an array, a dictionary, a stream,
/// a resources map) is one frame, and a nested form costs about four of them: the 50-deep chain
/// in `a_deeply_nested_form_keeps_the_text_nobody_marked` peaks at 200. The interpreter follows
/// forms as deep as the renderer and Edit text read (`interp::MAX_DEPTH`, #3645), so this must
/// reach at least that far, or the guard would refuse a document the redaction itself handles.
/// Total work is bounded separately by `MAX_OBJECTS`, and 512 frames stay far inside a thread's
/// stack.
const MAX_DEPTH: usize = 512;
const _: () = assert!(MAX_DEPTH >= 8 * crate::interp::MAX_DEPTH, "the guard must walk past the deepest form the interpreter follows");
const MAX_OBJECTS: usize = 16_384;
const MAX_STREAM_BYTES: usize = 1 << 20;
const MAX_TOTAL_BYTES: usize = 8 << 20;
const MAX_FILTERS: usize = 16;
#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)]
pub enum PatternProblem {
#[error("a tiling pattern carries text or images that cannot be removed safely")]
Content,
#[error("pattern resources or content cannot be read completely")]
Unreadable,
#[error("pattern inspection exceeded its resource, depth or decoded-content budget")]
Limit,
}
struct Guard {
left: usize,
bytes_left: usize,
seen: HashSet<(ObjRef, bool)>,
active: HashSet<(ObjRef, bool)>,
}
impl Guard {
fn new() -> Self {
Self { left: MAX_OBJECTS, bytes_left: MAX_TOTAL_BYTES, seen: HashSet::new(), active: HashSet::new() }
}
fn charge(&mut self, depth: usize) -> Result<(), PatternProblem> {
self.left = self.left.checked_sub(1).ok_or(PatternProblem::Limit)?;
if depth > MAX_DEPTH {
return Err(PatternProblem::Limit);
}
Ok(())
}
fn walk(&mut self, doc: &Document, object: &Object, inside: bool, depth: usize) -> Result<(), PatternProblem> {
self.charge(depth)?;
match object {
Object::Ref(reference) => {
let key = (*reference, inside);
if inside && self.active.contains(&key) {
return Err(PatternProblem::Unreadable);
}
if !self.seen.insert(key) {
return Ok(());
}
let value = doc.try_get(reference.num).map_err(|_| PatternProblem::Unreadable)?;
self.active.insert(key);
let result = self.walk(doc, &value, inside, depth + 1);
self.active.remove(&key);
result
}
Object::Array(values) => {
for value in values {
self.walk(doc, value, inside, depth + 1)?;
}
Ok(())
}
Object::Dict(dict) => self.dictionary(doc, dict, inside, false, depth + 1),
Object::Stream(stream) => {
if inside {
let subtype = resolved(doc, stream.dict.get(b"Subtype"))?;
let pattern_type = resolved(doc, stream.dict.get(b"PatternType"))?;
if subtype.as_name() == Some(b"Image") {
return Err(PatternProblem::Content);
}
if subtype.as_name() == Some(b"Form") {
if resolved(doc, stream.dict.get(b"Subtype2"))? != Object::Null || resolved(doc, stream.dict.get(b"Ref"))? != Object::Null {
return Err(PatternProblem::Unreadable);
}
self.content(doc, stream)?;
} else if pattern_type.as_int() == Some(1) {
self.content(doc, stream)?;
}
}
self.dictionary(doc, &stream.dict, inside, true, depth + 1)
}
_ => Ok(()),
}
}
fn dictionary(&mut self, doc: &Document, dict: &Dict, inside: bool, stream: bool, depth: usize) -> Result<(), PatternProblem> {
// Stream dictionaries include appearance and soft-mask streams whose consumers
// may infer Form semantics without /Subtype. Resource-map names, in contrast,
// are arbitrary and must not be mistaken for dictionary field names.
let subtype = resolved(doc, dict.get(b"Subtype"))?;
let pattern_type = resolved(doc, dict.get(b"PatternType"))?;
let owns_resources = stream || matches!(subtype.as_name(), Some(b"Form" | b"Type3")) || pattern_type.as_int() == Some(1);
for (key, value) in dict.iter() {
if owns_resources && key.as_slice() == b"Resources" {
self.resources(doc, value, inside, depth + 1)?;
} else if inside || !owns_resources {
self.walk(doc, value, inside, depth + 1)?;
}
}
Ok(())
}
fn resources(&mut self, doc: &Document, object: &Object, inside: bool, depth: usize) -> Result<(), PatternProblem> {
self.charge(depth)?;
let object = resolved(doc, Some(object))?;
// A Form's null Resources is equivalent to an absent entry: it uses the already
// inspected caller resources. Tiling-pattern resources are required separately.
if object == Object::Null {
return Ok(());
}
let Some(resources) = object.as_dict() else { return Err(PatternProblem::Unreadable) };
for (category, value) in resources.iter() {
// Resource dictionaries may include large embedded fonts and colour tables. Only
// structures that can own another pattern cell need recursive inspection.
if !matches!(category.as_slice(), b"Pattern" | b"XObject" | b"Font" | b"Properties" | b"ExtGState") {
continue;
}
if category.as_slice() == b"Font" {
let fonts = resolved(doc, Some(value))?;
let Some(fonts) = fonts.as_dict() else { return Err(PatternProblem::Unreadable) };
for (_, font) in fonts.iter() {
self.charge(depth)?;
let font = resolved(doc, Some(font))?;
let Some(dict) = font.as_dict() else { return Err(PatternProblem::Unreadable) };
if resolved(doc, dict.get(b"Subtype"))?.as_name() == Some(b"Type3")
&& let Some(resources) = dict.get(b"Resources")
{
self.resources(doc, resources, inside, depth + 1)?;
}
}
continue;
}
if matches!(category.as_slice(), b"Pattern" | b"XObject" | b"Properties" | b"ExtGState") && resolved(doc, Some(value))? == Object::Null {
continue;
}
if category.as_slice() == b"Pattern" {
let object = resolved(doc, Some(value))?;
let Some(patterns) = object.as_dict() else { return Err(PatternProblem::Unreadable) };
for (_, pattern) in patterns.iter() {
self.charge(depth)?;
let object = resolved(doc, Some(pattern))?;
if object == Object::Null {
continue;
}
let Some(dict) = object.as_dict() else { return Err(PatternProblem::Unreadable) };
let kind = resolved(doc, dict.get(b"PatternType"))?;
match (kind.as_int(), &object) {
(Some(1), Object::Stream(_)) => {
// Do not guess a fallback for a malformed cell, which different
// readers may resolve differently. Pattern resources are mandatory.
if resolved(doc, dict.get(b"Resources"))?.as_dict().is_none() {
return Err(PatternProblem::Unreadable);
}
}
(Some(2), Object::Dict(_)) => {}
_ => return Err(PatternProblem::Unreadable),
}
self.walk(doc, pattern, true, depth + 1)?;
}
} else {
if (inside && matches!(category.as_slice(), b"XObject" | b"Properties")) || category.as_slice() == b"ExtGState" {
let object = resolved(doc, Some(value))?;
let Some(entries) = object.as_dict() else { return Err(PatternProblem::Unreadable) };
for (_, entry) in entries.iter() {
self.charge(depth)?;
let object = resolved(doc, Some(entry))?;
if object == Object::Null {
continue;
}
if category.as_slice() == b"Properties" {
let Some(properties) = object.as_dict() else { return Err(PatternProblem::Unreadable) };
replacement_text(doc, properties)?;
} else if category.as_slice() == b"ExtGState" {
let Some(state) = object.as_dict() else { return Err(PatternProblem::Unreadable) };
match resolved(doc, state.get(b"SMask"))? {
Object::Null => {}
Object::Name(name) if name == b"None" => {}
Object::Dict(mask) => {
let group = resolved(doc, mask.get(b"G"))?;
let Object::Stream(group) = group else { return Err(PatternProblem::Unreadable) };
readable_form(doc, &group)?;
if let Some(value) = mask.get(b"G") {
self.walk(doc, value, inside, depth + 1)?;
}
}
_ => return Err(PatternProblem::Unreadable),
}
} else {
let Object::Stream(stream) = object else { return Err(PatternProblem::Unreadable) };
let subtype = resolved(doc, stream.dict.get(b"Subtype"))?;
if subtype.as_name() == Some(b"Image") {
return Err(PatternProblem::Content);
}
readable_form(doc, &stream)?;
}
}
}
if category.as_slice() == b"XObject" {
let entries = resolved(doc, Some(value))?;
let Some(entries) = entries.as_dict() else { return Err(PatternProblem::Unreadable) };
for (_, entry) in entries.iter() {
self.charge(depth)?;
let object = resolved(doc, Some(entry))?;
if let Object::Stream(stream) = object
&& resolved(doc, stream.dict.get(b"Subtype"))?.as_name() == Some(b"Form")
{
self.walk(doc, entry, inside, depth + 1)?;
}
}
}
}
}
Ok(())
}
fn content(&mut self, doc: &Document, stream: &Stream) -> Result<(), PatternProblem> {
// Strict decoding with a small bound, unlike the tolerant general reader. Normalize
// indirect filter declarations before using Stream::filters, which accepts direct data.
let normalized = normalized_stream(doc, stream)?;
let chain = normalized.filters();
let max = MAX_STREAM_BYTES.min(self.bytes_left);
if chain.is_empty() && normalized.raw.len() > max {
return Err(PatternProblem::Limit);
}
let data = pdfcraft_filters::decode(&chain, &normalized.raw, max).map_err(|error| match error {
pdfcraft_filters::FilterError::LimitExceeded(_) => PatternProblem::Limit,
_ => PatternProblem::Unreadable,
})?;
self.bytes_left = self.bytes_left.checked_sub(data.len()).ok_or(PatternProblem::Limit)?;
let parsed = pdfcraft_content::parse(&data);
if parsed.skipped != 0 {
return Err(PatternProblem::Unreadable);
}
for op in parsed.ops {
self.charge(0)?;
if matches!(op.op.as_slice(), b"BDC" | b"DP") {
for operand in &op.operands {
if let Some(properties) = resolved(doc, Some(operand))?.as_dict() {
replacement_text(doc, properties)?;
}
}
}
match op.op.as_slice() {
b"Tj" | b"TJ" | b"'" | b"\"" | b"BI" => return Err(PatternProblem::Content),
// Unknown operators cannot establish that no hidden content was skipped.
b"q" | b"Q" | b"cm" | b"w" | b"J" | b"j" | b"M" | b"d" | b"ri" | b"i" | b"gs" | b"m" | b"l" | b"c" | b"v" | b"y" | b"h" | b"re"
| b"S" | b"s" | b"f" | b"F" | b"f*" | b"B" | b"B*" | b"b" | b"b*" | b"n" | b"W" | b"W*" | b"BT" | b"ET" | b"Tc" | b"Tw" | b"Tz"
| b"TL" | b"Tf" | b"Tr" | b"Ts" | b"Td" | b"TD" | b"Tm" | b"T*" | b"d0" | b"d1" | b"CS" | b"cs" | b"SC" | b"SCN" | b"sc" | b"scn"
| b"G" | b"g" | b"RG" | b"rg" | b"K" | b"k" | b"sh" | b"Do" | b"MP" | b"DP" | b"BMC" | b"BDC" | b"EMC" | b"BX" | b"EX" => {}
_ => return Err(PatternProblem::Unreadable),
}
}
Ok(())
}
}
fn readable_form(doc: &Document, stream: &Stream) -> Result<(), PatternProblem> {
// PS, Form/Subtype2 PS, reference XObjects and unknown/missing subtypes cannot be
// proven free of recoverable pattern content by the PDF content parser.
if resolved(doc, stream.dict.get(b"Subtype"))?.as_name() != Some(b"Form")
|| resolved(doc, stream.dict.get(b"Subtype2"))? != Object::Null
|| resolved(doc, stream.dict.get(b"Ref"))? != Object::Null
{
return Err(PatternProblem::Unreadable);
}
Ok(())
}
fn replacement_text(doc: &Document, properties: &Dict) -> Result<(), PatternProblem> {
for key in [b"ActualText".as_slice(), b"Alt", b"E"] {
if resolved(doc, properties.get(key))? != Object::Null {
return Err(PatternProblem::Content);
}
}
Ok(())
}
fn resolved(doc: &Document, value: Option<&Object>) -> Result<Object, PatternProblem> {
let mut value = value.cloned().unwrap_or(Object::Null);
for _ in 0..MAX_DEPTH {
let Object::Ref(reference) = value else { return Ok(value) };
value = (*doc.try_get(reference.num).map_err(|_| PatternProblem::Unreadable)?).clone();
}
Err(PatternProblem::Limit)
}
fn normalized_stream(doc: &Document, stream: &Stream) -> Result<Stream, PatternProblem> {
if stream.dict.contains(b"F") || stream.dict.contains(b"FFilter") || stream.dict.contains(b"FDecodeParms") || stream.dict.contains(b"DP") {
return Err(PatternProblem::Unreadable);
}
let mut normalized = stream.clone();
let filter = resolved(doc, stream.dict.get(b"Filter"))?;
let names = match &filter {
Object::Null => Vec::new(),
Object::Name(_) => vec![filter],
Object::Array(values) if values.len() <= MAX_FILTERS => values.iter().map(|v| resolved(doc, Some(v))).collect::<Result<Vec<_>, _>>()?,
_ => return Err(PatternProblem::Unreadable),
};
if names.iter().any(|name| name.as_name().is_none()) {
return Err(PatternProblem::Unreadable);
}
let parms = resolved(doc, stream.dict.get(b"DecodeParms"))?;
let parms = match parms {
Object::Null => vec![Object::Null; names.len()],
Object::Dict(_) if names.len() == 1 => vec![parms],
Object::Array(values) if values.len() == names.len() => values,
_ => return Err(PatternProblem::Unreadable),
};
let parms = parms
.iter()
.map(|value| {
let value = resolved(doc, Some(value))?;
let Object::Dict(mut dict) = value else {
return if value == Object::Null { Ok(value) } else { Err(PatternProblem::Unreadable) };
};
for key in [b"Predictor".as_slice(), b"Colors", b"BitsPerComponent", b"Columns", b"EarlyChange"] {
if let Some(value) = dict.get(key) {
let value = resolved(doc, Some(value))?;
if value.as_int().is_none() {
return Err(PatternProblem::Unreadable);
}
dict.set(key.to_vec(), value);
}
}
if !matches!(dict.int(b"Predictor").unwrap_or(1), 1 | 2 | 10..=15)
|| dict.int(b"Colors").unwrap_or(1) <= 0
|| dict.int(b"Columns").unwrap_or(1) <= 0
|| !matches!(dict.int(b"BitsPerComponent").unwrap_or(8), 1 | 2 | 4 | 8 | 16)
|| !matches!(dict.int(b"EarlyChange").unwrap_or(1), 0 | 1)
{
return Err(PatternProblem::Unreadable);
}
Ok(Object::Dict(dict))
})
.collect::<Result<Vec<_>, _>>()?;
normalized.dict.set(b"Filter".to_vec(), Object::Array(names));
normalized.dict.set(b"DecodeParms".to_vec(), Object::Array(parms));
Ok(normalized)
}
/// Page-tree nodes a walk of the tree may visit, pages included, before it refuses: the page tree
/// is walked whole elsewhere too (`pdfcraft_model::pages`, `unlink_doomed`), so a long document
/// must not refuse here.
const MAX_TREE_NODES: usize = 1_000_000;
/// Every node a full save reaches through the page tree -- down `/Kids` and up the `/Parent` of
/// every node visited, page or `/Pages`, so a node reached only through a `/Parent` link counts
/// too -- that has `/Resources` of its own and is not one of the document's pages: those are
/// walked with their effective resources. Cycles are cut by the seen-set. Holders are bounded by
/// `MAX_OBJECTS`, pages are not.
fn tree_holders(doc: &Document) -> Result<Vec<Object>, PatternProblem> {
let Some(top) = doc.root().and_then(|root| doc.get(root).as_dict().and_then(|d| d.reference(b"Pages"))) else { return Ok(Vec::new()) };
let pages: HashSet<ObjRef> = pdfcraft_model::pages(doc).iter().map(|p| p.obj).collect();
// The `/Resources` values to inspect. A direct `/Parent` dictionary is written inline by a
// full save, so it is a node like any other: its resources, `/Parent` and `/Kids` count.
let mut holders = Vec::new();
let mut seen = HashSet::new();
seen.insert(top);
let mut stack = vec![top];
let mut inline: Vec<Dict> = Vec::new();
// One node: queue what it links to that was not seen yet (so a /Kids array shared by many
// nodes costs its length once per node, never a copy), and keep its resources.
let mut visit =
|dict: &Dict, is_page: bool, stack: &mut Vec<ObjRef>, inline: &mut Vec<Dict>, holders: &mut Vec<Object>| -> Result<(), PatternProblem> {
match dict.get(b"Parent") {
Some(Object::Ref(parent)) => {
if seen.insert(*parent) {
stack.push(*parent);
}
}
Some(Object::Dict(parent)) => {
if inline.len() >= MAX_DEPTH {
return Err(PatternProblem::Limit);
}
inline.push(parent.clone());
}
_ => {}
}
if let Some(resources) = dict.get(b"Resources")
&& !is_page
{
holders.push(resources.clone());
if holders.len() > MAX_OBJECTS {
return Err(PatternProblem::Limit);
}
}
let is_pages = dict.name(b"Type") == Some(b"Pages") || (dict.contains(b"Kids") && dict.name(b"Type") != Some(b"Page"));
if is_pages {
let kids_object = match dict.get(b"Kids") {
Some(Object::Ref(reference)) => Some(doc.try_get(reference.num).map_err(|_| PatternProblem::Unreadable)?),
_ => None,
};
let kids = match &kids_object {
Some(object) => object.as_array(),
None => dict.get(b"Kids").and_then(Object::as_array),
};
for kid in kids.into_iter().flatten().filter_map(Object::as_ref) {
if seen.insert(kid) {
stack.push(kid);
}
}
}
if seen.len() > MAX_TREE_NODES {
return Err(PatternProblem::Limit);
}
Ok(())
};
loop {
if let Some(node) = stack.pop() {
let object = doc.try_get(node.num).map_err(|_| PatternProblem::Unreadable)?;
if let Some(dict) = object.as_dict() {
visit(dict, pages.contains(&node), &mut stack, &mut inline, &mut holders)?;
}
} else if let Some(dict) = inline.pop() {
visit(&dict, false, &mut stack, &mut inline, &mut holders)?;
} else {
break;
}
}
Ok(holders)
}
/// Validate the requested pages before `apply` commits its working copy. With `tree`, the
/// `/Resources` of every page-tree node above the pages is inspected as well: a node no page
/// inherits from, or one a page with a broken `/Parent` link still reaches, can keep a pruned
/// cell in the saved file.
pub(crate) fn check(doc: &Document, selected: &[usize], tree: bool) -> Result<(), RedactError> {
if tree && let Some(&first) = selected.first() {
let page = first.saturating_add(1);
let holders = tree_holders(doc).map_err(|reason| RedactError::Pattern { page, reason })?;
for resources in holders {
let mut guard = Guard::new();
guard.resources(doc, &resources, false, 0).map_err(|reason| RedactError::Pattern { page, reason })?;
}
}
let pages = pdfcraft_model::pages(doc);
for &index in selected {
let Some(page) = pages.get(index) else { return Err(RedactError::Unreadable(index.saturating_add(1))) };
let mut guard = Guard::new();
let checked = (|| {
if let Some(resources) = page.dict.get(b"Resources") {
guard.resources(doc, resources, false, 0)?;
}
// Kept appearance streams may own their own pattern resources.
for annotation in crate::annots_of(doc, &page.dict) {
let object = resolved(doc, Some(&annotation))?;
if let Some(appearance) = object.as_dict().and_then(|dict| dict.get(b"AP")) {
guard.walk(doc, appearance, false, 0)?;
}
}
Ok(())
})();
checked.map_err(|reason| RedactError::Pattern { page: index.saturating_add(1), reason })?;
}
Ok(())
}
/// The backstop after every other check: a full save writes whatever the trailer reaches, by any
/// reference at all -- a stray resource category, a key on a page, a direct /Parent, `/AcroForm
/// /DR`, `/PieceInfo` -- and the walks above follow only the references they know. A pattern
/// pruned from a page that a full save would still write must therefore be proven safe like any
/// retained cell (vector-only), or the redaction is refused. Costs one walk of the reachable
/// objects, and only when a pattern was pruned.
pub(crate) fn check_pruned_unreachable(doc: &Document, pruned: &[ObjRef], page: usize) -> Result<(), RedactError> {
if pruned.is_empty() {
return Ok(());
}
let reachable = pdfcraft_cos::reachable_objects(doc)?;
let mut guard = Guard::new();
let mut checked = HashSet::new();
for &pattern in pruned {
if reachable.contains(&pattern.num) && checked.insert(pattern.num) {
guard.walk(doc, &Object::Ref(pattern), true, 0).map_err(|reason| RedactError::Pattern { page, reason })?;
}
}
Ok(())
}
crates/redact/src/patterns.rs
+356 -2
View File
@@ -1,5 +1,5 @@
//! Search & Redact patterns (Acrobat's Find Text ▸ Patterns): phone numbers, email addresses,
//! credit card numbers, US Social Security numbers and dates. Each matcher works on a page's
//! credit card numbers, US Social Security numbers, dates and IBANs. Each matcher works on a page's
//! text as characters and returns character ranges; callers map them to glyphs and areas.
use std::ops::Range;
@@ -11,9 +11,10 @@ pub enum Pattern {
CreditCard,
Ssn,
Date,
Iban,
}
pub const PATTERNS: [Pattern; 5] = [Pattern::Phone, Pattern::Email, Pattern::CreditCard, Pattern::Ssn, Pattern::Date];
pub const PATTERNS: [Pattern; 6] = [Pattern::Phone, Pattern::Email, Pattern::CreditCard, Pattern::Ssn, Pattern::Date, Pattern::Iban];
impl Pattern {
pub fn label(self) -> &'static str {
@@ -23,6 +24,7 @@ impl Pattern {
Pattern::CreditCard => "Credit Cards",
Pattern::Ssn => "Social Security Numbers",
Pattern::Date => "Dates",
Pattern::Iban => "IBAN Numbers",
}
}
@@ -33,6 +35,7 @@ impl Pattern {
Pattern::CreditCard => "credit-card",
Pattern::Ssn => "ssn",
Pattern::Date => "date",
Pattern::Iban => "iban",
}
}
@@ -397,6 +400,173 @@ fn date(s: &[char], i: usize) -> Option<usize> {
Some(k2 + y - i)
}
/// An IBAN (ISO 13616): a two-letter country code, two check digits and the rest
/// of the country's exact length in letters and digits, either compact or in the
/// printed groups of four. A space is accepted only after a full group, so an
/// ordinary word after a code ("BA37 leaves at noon") ends the run. Valid iff the
/// number is one modulo 97 after moving the first four characters to the end. The
/// country code must be in the registry, and an alphanumeric character glued to
/// the end rejects the match.
fn iban(s: &[char], i: usize) -> Option<usize> {
if !(s.get(i).copied().is_some_and(|c| c.is_ascii_alphabetic())
&& s.get(i + 1).copied().is_some_and(|c| c.is_ascii_alphabetic())
&& s.get(i + 2).copied().is_some_and(|c| c.is_ascii_digit())
&& s.get(i + 3).copied().is_some_and(|c| c.is_ascii_digit()))
{
return None;
}
// Not the tail of a longer alphanumeric run.
if s.get(i.wrapping_sub(1)).is_some_and(|c| c.is_alphanumeric()) {
return None;
}
let country: String = s.iter().skip(i).take(2).map(char::to_ascii_uppercase).collect();
let len = iban_length(&country)?;
// The IBAN's characters uppercased, read up to the country's length.
let mut digits = Vec::with_capacity(len);
let mut j = i;
let mut after_space = false;
while digits.len() < len {
match s.get(j) {
Some(c) if c.is_ascii_alphanumeric() => {
digits.push(c.to_ascii_uppercase());
after_space = false;
}
// A single space between two groups of four.
Some(&' ') if digits.len() % 4 == 0 && !after_space => after_space = true,
_ => return None,
}
j += 1;
}
// Reject an IBAN glued directly to trailing alphanumerics ("...00ABC").
// A space-separated suffix ("...00 123") is not part of the IBAN.
if s.get(j).is_some_and(|c| c.is_alphanumeric()) {
return None;
}
iban_check(&digits).then_some(j - i)
}
/// Whether `digits` is a valid IBAN: move the first four characters to the end,
/// replace letters by two digits (A = 10, ..., Z = 35), and the number must be
/// one modulo 97.
fn iban_check(digits: &[char]) -> bool {
let mut rem = 0u32;
for c in digits.iter().skip(4).chain(digits.iter().take(4)).copied() {
let value = iban_value(c);
// A letter stands for two decimal digits (A = 10, ..., Z = 35),
// a digit for one.
rem = if c.is_ascii_digit() { (rem * 10 + value) % 97 } else { (rem * 100 + value) % 97 };
}
rem == 1
}
/// The value of an IBAN character: a digit is itself, a letter is 10 (A) to 35 (Z).
fn iban_value(c: char) -> u32 {
match c {
'0'..='9' => c as u32 - '0' as u32,
_ => c as u32 - 'A' as u32 + 10,
}
}
/// The exact length of an IBAN for each country code of the SWIFT IBAN registry
/// (release 99, December 2024), counting letters and digits only. A country code
/// that is not listed has no IBANs, so nothing starting with it is matched.
const IBAN_LENGTHS: [(&str, usize); 89] = [
("AD", 24),
("AE", 23),
("AL", 28),
("AT", 20),
("AZ", 28),
("BA", 20),
("BE", 16),
("BG", 22),
("BH", 22),
("BI", 27),
("BR", 29),
("BY", 28),
("CH", 21),
("CR", 22),
("CY", 28),
("CZ", 24),
("DE", 22),
("DJ", 27),
("DK", 18),
("DO", 28),
("EE", 20),
("EG", 29),
("ES", 24),
("FI", 18),
("FK", 18),
("FO", 18),
("FR", 27),
("GB", 22),
("GE", 22),
("GI", 23),
("GL", 18),
("GR", 27),
("GT", 28),
("HN", 28),
("HR", 21),
("HU", 28),
("IE", 22),
("IL", 23),
("IQ", 23),
("IS", 26),
("IT", 27),
("JO", 30),
("KW", 30),
("KZ", 20),
("LB", 28),
("LC", 32),
("LI", 21),
("LT", 20),
("LU", 20),
("LV", 21),
("LY", 25),
("MC", 27),
("MD", 24),
("ME", 22),
("MK", 19),
("MN", 20),
("MR", 27),
("MT", 31),
("MU", 30),
("NI", 28),
("NL", 18),
("NO", 15),
("OM", 23),
("PK", 24),
("PL", 28),
("PS", 29),
("PT", 25),
("QA", 29),
("RO", 24),
("RS", 22),
("RU", 33),
("SA", 24),
("SC", 31),
("SD", 18),
("SE", 24),
("SI", 19),
("SK", 24),
("SM", 27),
("SO", 23),
("ST", 25),
("SV", 28),
("TL", 23),
("TN", 24),
("TR", 26),
("UA", 29),
("VA", 22),
("VG", 24),
("XK", 20),
("YE", 30),
];
/// The length of an IBAN of `country`, if the registry lists it.
fn iban_length(country: &str) -> Option<usize> {
IBAN_LENGTHS.iter().find(|(code, _)| *code == country).map(|&(_, len)| len)
}
/// Character ranges of `text` matching `pattern`.
pub fn find(pattern: Pattern, text: &[char]) -> Vec<Range<usize>> {
let f: fn(&[char], usize) -> Option<usize> = match pattern {
@@ -405,6 +575,7 @@ pub fn find(pattern: Pattern, text: &[char]) -> Vec<Range<usize>> {
Pattern::CreditCard => credit_card,
Pattern::Ssn => ssn,
Pattern::Date => date,
Pattern::Iban => iban,
};
let mut out = Vec::new();
let mut i = 0;
@@ -665,4 +836,187 @@ mod tests {
let s: Vec<char> = "\u{130}\u{131}I\u{15E}".chars().collect();
assert_eq!(s.iter().copied().map(fold).collect::<String>(), "iii\u{15F}");
}
#[test]
fn iban_matches_compact_spaced_and_rejects_near_matches() {
assert_eq!(Pattern::from_id("iban").map(Pattern::id), Some("iban"));
// Spaced and compact IBANs match, a wrong check digit is rejected ("DE89
// 3704 0044 0532 0130 00" and "DE89370400440532013000" match, "...01"
// does not).
if let Some(iban_pattern) = Pattern::from_id("iban") {
// Spaced IBAN (valid German IBAN: DE89 3704 0044 0532 0130 00)
assert_eq!(found(iban_pattern, "DE89 3704 0044 0532 0130 00"), vec!["DE89 3704 0044 0532 0130 00"]);
// Compact IBAN (same number, no spaces)
assert_eq!(found(iban_pattern, "DE89370400440532013000"), vec!["DE89370400440532013000"]);
// Wrong check digit should not match (DE89...01 instead of DE89...00)
assert_eq!(found(iban_pattern, "DE89 3704 0044 0532 0130 01"), Vec::<String>::new());
// Near match: IBAN followed by extra digits should match only the IBAN
assert_eq!(found(iban_pattern, "DE89 3704 0044 0532 0130 00 123"), vec!["DE89 3704 0044 0532 0130 00"]);
}
}
#[test]
fn iban_rejects_unknown_country_code_wrong_length_and_glued_tail() {
if let Some(iban_pattern) = Pattern::from_id("iban") {
// Unknown country code: the check digits pass mod-97, but ISO 13616 defines no length for XX.
assert_eq!(found(iban_pattern, "XX65000000000000000000"), Vec::<String>::new());
// DE is 22 characters long. This 15-character number passes mod-97 but has
// the wrong national length, so it must be rejected.
assert_eq!(found(iban_pattern, "DE8300000000000"), Vec::<String>::new());
// A valid IBAN glued directly to trailing alphanumerics is not an IBAN.
assert_eq!(found(iban_pattern, "DE89370400440532013000ABC"), Vec::<String>::new());
assert_eq!(found(iban_pattern, "ABCDE89370400440532013000"), Vec::<String>::new());
// Letters in place of the two check digits.
assert_eq!(found(iban_pattern, "DEAB 3704 0044 0532 0130 00"), Vec::<String>::new());
// A space-separated suffix does not glue to the IBAN.
assert_eq!(found(iban_pattern, "DE89 3704 0044 0532 0130 00 123"), vec!["DE89 3704 0044 0532 0130 00"]);
}
}
#[test]
fn iban_accepts_spaces_only_between_groups_of_four() {
// A code followed by ordinary words: the words end the run, so nothing is marked.
for s in [
"Flight BA37 leaves at noon from gate four",
"Flight BA30 leaves at noon",
"Gate BA45 leaves at noon",
"BA72 have water are word",
"GB00 their an her make that",
"SI30 write can all more",
] {
assert_eq!(found(Pattern::Iban, s), Vec::<String>::new(), "{s}");
}
// A valid IBAN with a space in the wrong place, or two spaces, is not marked.
for s in ["DE89 3704 0044 0532 013 000", "DE89 3704 0044 0532 0 13000", "DE8 9370 4004 4053 2013 000", "DE89 3704 0044 0532 0130 00"] {
assert_eq!(found(Pattern::Iban, s), Vec::<String>::new(), "{s}");
}
// Any group boundary may be a space or not, and the IBAN inside a sentence is marked alone.
assert_eq!(found(Pattern::Iban, "DE893704 0044 0532013000"), vec!["DE893704 0044 0532013000"]);
assert_eq!(found(Pattern::Iban, "Pay to DE89 3704 0044 0532 0130 00 today"), vec!["DE89 3704 0044 0532 0130 00"]);
}
/// One example IBAN per country of the SWIFT IBAN registry (release 99).
const IBAN_REGISTRY_EXAMPLES: [&str; IBAN_LENGTHS.len()] = [
"AD1200012030200359100100",
"AE070331234567890123456",
"AL47212110090000000235698741",
"AT611904300234573201",
"AZ21NABZ00000000137010001944",
"BA391290079401028494",
"BE68539007547034",
"BG80BNBG96611020345678",
"BH67BMAG00001299123456",
"BI4210000100010000332045181",
"BR1800360305000010009795493C1",
"BY13NBRB3600900000002Z00AB00",
"CH9300762011623852957",
"CR05015202001026284066",
"CY17002001280000001200527600",
"CZ6508000000192000145399",
"DE89370400440532013000",
"DJ2100010000000154000100186",
"DK5000400440116243",
"DO28BAGR00000001212453611324",
"EE382200221020145685",
"EG380019000500000000263180002",
"ES9121000418450200051332",
"FI2112345600000785",
"FK88SC123456789012",
"FO6264600001631634",
"FR1420041010050500013M02606",
"GB29NWBK60161331926819",
"GE29NB0000000101904917",
"GI75NWBK000000007099453",
"GL8964710001000206",
"GR1601101250000000012300695",
"GT82TRAJ01020000001210029690",
"HN88CABF00000000000250005469",
"HR1210010051863000160",
"HU42117730161111101800000000",
"IE29AIBK93115212345678",
"IL620108000000099999999",
"IQ98NBIQ850123456789012",
"IS140159260076545510730339",
"IT60X0542811101000000123456",
"JO94CBJO0010000000000131000302",
"KW81CBKU0000000000001234560101",
"KZ86125KZT5004100100",
"LB62099900000001001901229114",
"LC55HEMM000100010012001200023015",
"LI21088100002324013AA",
"LT121000011101001000",
"LU280019400644750000",
"LV80BANK0000435195001",
"LY83002048000020100120361",
"MC5811222000010123456789030",
"MD24AG000225100013104168",
"ME25505000012345678951",
"MK07250120000058984",
"MN121234123456789123",
"MR1300020001010000123456753",
"MT84MALT011000012345MTLCAST001S",
"MU17BOMM0101101030300200000MUR",
"NI45BAPR00000013000003558124",
"NL91ABNA0417164300",
"NO9386011117947",
"OM810180000001299123456",
"PK36SCBL0000001123456702",
"PL61109010140000071219812874",
"PS92PALS000000000400123456702",
"PT50000201231234567890154",
"QA58DOHB00001234567890ABCDEFG",
"RO49AAAA1B31007593840000",
"RS35260005601001611379",
"RU0304452522540817810538091310419",
"SA0380000000608010167519",
"SC18SSCB11010000000000001497USD",
"SD2129010501234001",
"SE4550000000058398257466",
"SI56263300012039086",
"SK3112000000198742637541",
"SM86U0322509800000000270100",
"SO211000001001000100141",
"ST68000100010051845310112",
"SV62CENR00000000000000700025",
"TL380080012345678910157",
"TN5910006035183598478831",
"TR330006100519786457841326",
"UA213223130000026007233566001",
"VA59001123000012345678",
"VG96VPVG0000012345678901",
"XK051212012345678906",
"YE15CBYE0001018861234567891234",
];
/// `iban` in the printed form: groups of four separated by single spaces.
fn printed(iban: &str) -> String {
iban.chars().collect::<Vec<_>>().chunks(4).map(|group| group.iter().collect::<String>()).collect::<Vec<_>>().join(" ")
}
#[test]
fn iban_registry_examples_cover_every_country_and_length() {
let from_examples: Vec<(&str, usize)> = IBAN_REGISTRY_EXAMPLES.iter().map(|e| (&e[..2], e.len())).collect();
assert_eq!(from_examples.as_slice(), IBAN_LENGTHS.as_slice());
}
#[test]
fn iban_matches_the_registry_example_of_every_country() {
for example in IBAN_REGISTRY_EXAMPLES {
assert_eq!(found(Pattern::Iban, example), vec![example], "compact {example}");
let spaced = printed(example);
assert_eq!(found(Pattern::Iban, &spaced), vec![spaced.clone()], "printed {spaced}");
}
}
#[test]
fn iban_rejects_a_registry_example_with_one_digit_changed() {
for example in IBAN_REGISTRY_EXAMPLES {
// The last digit becomes the next digit, so the country, the length and the layout stay the same.
let Some(at) = example.rfind(|c: char| c.is_ascii_digit()) else { continue };
let digit = example[at..=at].parse::<u32>().unwrap_or(0);
let changed = format!("{}{}{}", &example[..at], (digit + 1) % 10, &example[at + 1..]);
assert_eq!(found(Pattern::Iban, &changed), Vec::<String>::new(), "{changed}");
}
}
}
crates/redact/src/sanitize.rs
+8 -3
View File
@@ -674,16 +674,21 @@ pub fn remove_hidden(doc: &mut Document, which: &[Hidden]) -> Result<Vec<(Hidden
if done.is_empty() {
return Err(RedactError::NothingToApply);
}
doc.require_full_save();
// The rewrite drops the revisions that held the removed content, so it is a different file from
// its source: it gets a new identifier, whoever asked for the removal (the remove-hidden edit
// and the optimizer's discard option call this function directly, not through [`sanitize`]).
doc.require_full_save_with_new_id();
Ok(done)
}
/// Sanitize Document: remove every category of hidden information.
pub fn sanitize(doc: &mut Document) -> Result<Vec<(Hidden, usize)>, RedactError> {
match remove_hidden(doc, &HIDDEN) {
// Nothing hidden is fine for Sanitize: the full rewrite still drops old revisions.
// Nothing hidden is fine for Sanitize: the full rewrite still drops old revisions. The
// rewrite is a new file either way, so it asks for a new identifier itself (when something
// was removed, `remove_hidden` has already asked).
Err(RedactError::NothingToApply) => {
doc.require_full_save();
doc.require_full_save_with_new_id();
Ok(Vec::new())
}
r => r,
crates/redact/src/tests.rs
+310
View File
@@ -290,6 +290,200 @@ fn vectors_are_removed_or_clipped_and_inline_images_go() {
assert!(c.contains("150 150 m"), "the line elsewhere stays");
}
/// A tiling pattern's cell is a content stream of its own and can draw text. Redacting a shape
/// filled with such a pattern removed the fill from the page but left the pattern in the page's
/// resources, so its cell -- with the text that was under the mark -- stayed in the saved file
/// (#3647). A pattern nothing paints with any more now leaves the resources; one some surviving
/// paint still uses stays.
#[test]
fn a_pattern_whose_only_painting_was_redacted_leaves_the_page() {
fn page_with(fills: &str) -> Document {
let cell = stream(
"/Type /Pattern /PatternType 1 /PaintType 1 /TilingType 1 /BBox [0 0 300 300] /XStep 300 /YStep 300 /Resources << /Font << /F1 5 0 R >> >>",
b"BT /F1 10 Tf 10 150 Td (SECRET) Tj ET",
);
let content = format!("BT /F1 10 Tf 10 280 Td (PUBLIC) Tj ET {fills}");
one_page(content.as_bytes(), "/Pattern << /P0 7 0 R >>", vec![cell])
}
let has_pattern = |doc: &Document| {
pdfcraft_model::pages(doc)[0]
.dict
.get(b"Resources")
.and_then(|r| doc.resolve(r).as_dict().cloned())
.is_some_and(|res| res.get(b"Pattern").is_some())
};
// One box, marked: the pattern is no longer painted, so it goes with the fill.
let mut doc = page_with("/Pattern cs /P0 scn 20 140 100 30 re f");
mark(&mut doc, 0, &[[15.0, 135.0, 130.0, 175.0]], "");
let report = apply(&mut doc, None).unwrap();
assert_eq!(report.patterns_removed, 1, "the pattern left the page");
assert!(!has_pattern(&doc), "the page has no /Pattern left");
let saved = write_incremental(&doc, &SaveOptions::default()).unwrap();
assert!(!saved.windows(8).any(|w| w == b"(SECRET)"), "the cell's text is gone from the saved file");
// Two boxes share a text-bearing cell. The second still needs its pattern, but it is not
// safe to infer that it visibly displays every byte of the cell.
let mut doc = page_with("/Pattern cs /P0 scn 20 140 100 30 re f /Pattern cs /P0 scn 20 40 100 30 re f");
mark(&mut doc, 0, &[[15.0, 135.0, 130.0, 175.0]], "");
let before = write_full(&doc, &SaveOptions::default()).unwrap();
assert!(matches!(apply(&mut doc, None), Err(RedactError::Pattern { reason: PatternProblem::Content, .. })));
assert_eq!(write_full(&doc, &SaveOptions::default()).unwrap(), before, "refusal keeps the pending mark and original bytes");
// Paint is not only path painting: text shows with the fill pattern too, and the text here
// survives the mark. Dropping the pattern would take the fill away from text still on the
// page, so a pattern is kept for every operator that can paint with it.
let mut doc = page_with("/Pattern cs /P0 scn 20 140 100 30 re f BT /F1 10 Tf 10 240 Td (KEEP) Tj ET");
mark(&mut doc, 0, &[[15.0, 135.0, 130.0, 175.0]], "");
assert!(matches!(apply(&mut doc, None), Err(RedactError::Pattern { reason: PatternProblem::Content, .. })));
assert!(has_pattern(&doc), "the refused operation keeps the original pattern");
assert!(content(&doc, 0).contains("(KEEP)"), "the text is still there: {}", content(&doc, 0));
// Text in render mode 3 paints nothing, so it keeps no pattern: the unpainted pattern goes.
let mut doc = page_with("/Pattern cs /P0 scn 20 140 100 30 re f BT 3 Tr /F1 10 Tf 10 240 Td (KEEP) Tj ET");
mark(&mut doc, 0, &[[15.0, 135.0, 130.0, 175.0]], "");
assert_eq!(apply(&mut doc, None).unwrap().patterns_removed, 1, "invisible text paints no pattern");
// `/Pattern` may be an indirect object, like any value. Reading the reference itself as a
// dictionary found none, so nothing was pruned and the cell stayed in the saved file.
let cell = stream(
"/Type /Pattern /PatternType 1 /PaintType 1 /TilingType 1 /BBox [0 0 300 300] /XStep 300 /YStep 300 /Resources << /Font << /F1 5 0 R >> >>",
b"BT /F1 10 Tf 10 150 Td (SECRET) Tj ET",
);
let mut doc = one_page(b"/Pattern cs /P0 scn 20 140 100 30 re f", "/Pattern 8 0 R", vec![cell, b"<< /P0 7 0 R >>".to_vec()]);
mark(&mut doc, 0, &[[15.0, 135.0, 130.0, 175.0]], "");
assert_eq!(apply(&mut doc, None).unwrap().patterns_removed, 1, "an indirect /Pattern is resolved before it is pruned");
assert!(!has_pattern(&doc), "the page has no /Pattern left");
let saved = write_incremental(&doc, &SaveOptions::default()).unwrap();
assert!(!saved.windows(8).any(|w| w == b"(SECRET)"), "the cell's text is gone from the saved file");
}
/// Pruning takes the pattern out of `/Resources` but leaves the `/P0 scn` that selected it, so
/// the saved page names a pattern that is no longer there. ISO 32000-1 (8.6.6.2) says that
/// operand is a key of the `/Pattern` subdictionary, so this is what a reader is handed: the
/// question is whether it stops reading the stream there. Asked in the review of #3647.
#[test]
fn a_page_whose_pruned_pattern_is_still_selected_renders() {
let cell = stream(
"/Type /Pattern /PatternType 1 /PaintType 1 /TilingType 1 /BBox [0 0 300 300] /XStep 300 /YStep 300 /Resources << /Font << /F1 5 0 R >> >>",
b"BT /F1 10 Tf 10 150 Td (SECRET) Tj ET",
);
// `0 g` after the fill puts the text back on a plain colour: with the pattern still selected
// the text would paint with it, and a pattern something still paints with is never pruned.
// The text is *after* the selector on purpose -- that is what a reader would lose if it gave
// up on the stale `/P0`.
let content_stream = b"/Pattern cs /P0 scn 20 140 100 30 re f 0 g BT /F1 24 Tf 10 40 Td (PUBLIC) Tj ET";
let mut doc = one_page(content_stream, "/Pattern << /P0 7 0 R >>", vec![cell]);
mark(&mut doc, 0, &[[15.0, 135.0, 130.0, 175.0]], "");
assert_eq!(apply(&mut doc, None).unwrap().patterns_removed, 1, "the pattern is pruned");
// The state this test is about: the selector outlived the resource it names.
let after = content(&doc, 0);
assert!(after.contains("/P0 scn"), "the selector is still in the content: {after}");
assert!(!after.contains("20 140 100 30 re"), "the fill that used it is gone: {after}");
let bytes = write_full(&doc, &SaveOptions::default()).unwrap();
assert!(!bytes.windows(8).any(|w| w == b"(SECRET)"), "the cell's text is gone from the saved file");
let mut r = pdfcraft_render::PageRenderer::new(std::sync::Arc::new(bytes), pdfcraft_render::RenderConfig::default());
let out = r.render(pdfcraft_render::RenderRequest { page: 0, scale: 1.0, ..Default::default() });
assert!(out.error.is_none(), "the page still renders: {:?}, warnings {:?}", out.error, out.warnings);
assert_eq!((out.width, out.height), (300, 300), "the page box is unchanged");
// The text after the stale selector is still drawn: 24 pt at y 40, so the top-left origin
// puts it in the rows around 240..265. Without it the reader gave up at the bad operand.
let painted = |rows: std::ops::Range<u32>, cols: std::ops::Range<u32>| {
rows.flat_map(|y| cols.clone().map(move |x| (y, x))).any(|(y, x)| {
let i = ((y * out.width + x) * 4) as usize;
out.rgba[i..i + 3] != [255, 255, 255]
})
};
assert!(painted(236..268, 8..120), "the text after the stale selector is still painted");
// The mark itself is drawn over the area the pattern filled -- `apply` writes a black
// overlay there (`q 0 0 0 rg 15 135 115 40 re f Q`), so the cell cannot show through it
// even if a reader kept the stale selector.
let black = |y: u32, x: u32| {
let i = ((y * out.width + x) * 4) as usize;
out.rgba[i..i + 3] == [0, 0, 0]
};
assert!(black(145, 70), "the redaction overlay is painted over the pattern's box");
}
#[test]
fn redaction_cannot_leave_hidden_cell_text_when_pattern_is_used_elsewhere() {
let cell = stream(
"/Type /Pattern /PatternType 1 /PaintType 1 /TilingType 1 /BBox [0 0 300 300] /XStep 300 /YStep 300 /Resources << /Font << /F1 5 0 R >> >>",
b"0.9 g 0 0 300 300 re f 0 g BT /F1 10 Tf 30 150 Td (SECRET) Tj ET",
);
let content = b"/Pattern cs /P0 scn 20 140 100 30 re f 20 40 100 30 re f";
let mut doc = one_page(content, "/Pattern << /P0 7 0 R >>", vec![cell]);
mark(&mut doc, 0, &[[15.0, 135.0, 130.0, 175.0]], "");
let before = write_full(&doc, &SaveOptions::default()).unwrap();
assert!(matches!(apply(&mut doc, None), Err(RedactError::Pattern { reason: PatternProblem::Content, .. })));
assert_eq!(write_full(&doc, &SaveOptions::default()).unwrap(), before, "refusal must be atomic");
}
#[test]
fn redaction_cannot_leave_hidden_cell_text_on_an_unselected_page() {
let cell = stream(
"/Type /Pattern /PatternType 1 /PaintType 1 /TilingType 1 /BBox [0 0 300 300] /XStep 300 /YStep 300 /Resources << /Font << /F1 5 0 R >> >>",
b"BT /F1 10 Tf 30 150 Td (SECRET) Tj ET",
);
let mut doc = pdf(vec![
b"<< /Type /Catalog /Pages 2 0 R >>".to_vec(),
b"<< /Type /Pages /Kids [3 0 R 8 0 R] /Count 2 >>".to_vec(),
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 4 0 R /Resources << /Pattern << /P0 7 0 R >> >> >>".to_vec(),
stream("", b"/Pattern cs /P0 scn 20 140 100 30 re f"),
FONT.replace("95 0 R", "6 0 R").into_bytes(),
widths(),
cell,
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 9 0 R /Resources << /Pattern << /P0 7 0 R >> >> >>".to_vec(),
stream("", b"/Pattern cs /P0 scn 20 40 100 30 re f"),
]);
mark(&mut doc, 0, &[[15.0, 135.0, 130.0, 175.0]], "");
let before = write_full(&doc, &SaveOptions::default()).unwrap();
match apply(&mut doc, Some(&[0])) {
Ok(_) => {
let saved = write_full(&doc, &SaveOptions::default()).unwrap();
assert!(!saved.windows(8).any(|w| w == b"(SECRET)"), "the unselected page kept the hidden cell text");
}
Err(RedactError::Pattern { reason: PatternProblem::Content, .. }) => {
assert_eq!(write_full(&doc, &SaveOptions::default()).unwrap(), before, "refusal must keep both pages and the mark");
}
Err(error) => panic!("unexpected refusal: {error}"),
}
}
#[test]
fn redacted_form_must_not_retain_its_own_pattern_cell_text() {
let cell = stream(
"/Type /Pattern /PatternType 1 /PaintType 1 /TilingType 1 /BBox [0 0 300 300] /XStep 300 /YStep 300 /Resources << /Font << /F1 5 0 R >> >>",
b"BT /F1 10 Tf 30 150 Td (SECRET) Tj ET",
);
let form = stream(
"/Type /XObject /Subtype /Form /BBox [0 0 300 300] /Resources << /Pattern << /P0 7 0 R >> >>",
b"/Pattern cs /P0 scn 20 140 100 30 re f",
);
let mut doc = one_page(b"/Fm Do", "/XObject << /Fm 8 0 R >>", vec![cell, form]);
mark(&mut doc, 0, &[[15.0, 135.0, 130.0, 175.0]], "");
let before = write_full(&doc, &SaveOptions::default()).unwrap();
assert!(matches!(apply(&mut doc, None), Err(RedactError::Pattern { reason: PatternProblem::Content, .. })));
assert_eq!(write_full(&doc, &SaveOptions::default()).unwrap(), before, "form-owned pattern refusal must be atomic");
}
#[test]
fn redacted_form_must_not_retain_inherited_pattern_cell_text() {
let cell = stream(
"/Type /Pattern /PatternType 1 /PaintType 1 /TilingType 1 /BBox [0 0 300 300] /XStep 300 /YStep 300 /Resources << /Font << /F1 5 0 R >> >>",
b"BT /F1 10 Tf 30 150 Td (SECRET) Tj ET",
);
let form = stream("/Type /XObject /Subtype /Form /BBox [0 0 300 300]", b"20 140 100 30 re f");
let mut doc = one_page(b"/Pattern cs /P0 scn /Fm Do", "/Pattern << /P0 7 0 R >> /XObject << /Fm 8 0 R >>", vec![cell, form]);
mark(&mut doc, 0, &[[15.0, 135.0, 130.0, 175.0]], "");
let before = write_full(&doc, &SaveOptions::default()).unwrap();
assert!(matches!(apply(&mut doc, None), Err(RedactError::Pattern { reason: PatternProblem::Content, .. })));
assert_eq!(write_full(&doc, &SaveOptions::default()).unwrap(), before, "inherited pattern refusal must be atomic");
}
#[test]
fn partially_covered_vector_paths_do_not_leak_drawing_commands() {
// A vector path with detailed drawing commands (e.g. outlined text or confidential geometry)
@@ -2272,6 +2466,87 @@ fn inline_outline_children_count_towards_the_total_budget() {
assert_eq!(scan(&doc).iter().find(|c| c.0 == Hidden::Bookmarks), Some(&(Hidden::Bookmarks, 100_000)));
}
/// A redacted copy and a sanitized copy are different files from their source: neither may keep
/// the source's file identifier (ISO 32000-2 §14.4). A save without the rewrite keeps it.
#[test]
fn a_redacted_or_sanitized_copy_gets_a_new_identifier() {
fn with_id(mut doc: Document) -> Document {
let id = Object::Array(vec![
Object::String(pdfcraft_cos::PdfString::literal(vec![0x11; 16])),
Object::String(pdfcraft_cos::PdfString::literal(vec![0x22; 16])),
]);
doc.trailer_mut().set(b"ID".to_vec(), id);
doc
}
fn saved_ids(doc: &Document) -> Vec<Vec<u8>> {
let saved = Document::open(Arc::new(write_full(doc, &SaveOptions::default()).unwrap())).unwrap();
match saved.trailer().get(b"ID") {
Some(Object::Array(a)) => a.iter().map(|s| s.as_string().map(|s| s.bytes.clone()).unwrap_or_default()).collect(),
_ => Vec::new(),
}
}
let source = vec![vec![0x11; 16], vec![0x22; 16]];
let assert_new = |ids: &[Vec<u8>], what: &str| {
assert_eq!(ids.len(), 2, "{what}");
for (new, old) in ids.iter().zip(&source) {
assert_eq!(new.len(), 16, "{what}");
assert_ne!(new, old, "{what}: the copy keeps the source's identifier");
}
};
// Redaction: the same save keeps the identifier until the redaction asks for a new one.
let mut doc = with_id(one_page(b"BT /F1 10 Tf 10 100 Td (Secret) Tj ET", "", vec![]));
mark(&mut doc, 0, &[[0.0, 95.0, 300.0, 110.0]], "");
assert_eq!(saved_ids(&doc), source, "before the redaction");
apply(&mut doc, None).unwrap();
assert!(doc.new_id_required());
assert_new(&saved_ids(&doc), "redaction");
// Sanitizing with hidden information to remove.
let mut doc = with_id(hidden_fixture());
crate::sanitize::sanitize(&mut doc).unwrap();
assert!(doc.new_id_required());
assert_new(&saved_ids(&doc), "sanitizing");
// Sanitizing a file with nothing hidden still rewrites it, so it gets a new identifier too.
let mut doc = with_id(one_page(b"BT ET", "", vec![]));
assert!(crate::sanitize::sanitize(&mut doc).unwrap().is_empty());
assert_new(&saved_ids(&doc), "sanitizing without hidden information");
}
/// A copy that lost hidden information through `remove_hidden` itself is a different file from its
/// source, so it does not keep the source's identifier either: the remove-hidden edit and the
/// optimizer's discard option call `remove_hidden` directly, not through `sanitize`.
#[test]
fn a_remove_hidden_copy_gets_a_new_identifier() {
use crate::sanitize::{Hidden, remove_hidden};
let with_id = |mut doc: Document| {
doc.trailer_mut().set(
b"ID".to_vec(),
Object::Array(vec![
Object::String(pdfcraft_cos::PdfString::literal(vec![0x11; 16])),
Object::String(pdfcraft_cos::PdfString::literal(vec![0x22; 16])),
]),
);
doc
};
let saved_ids = |doc: &Document| {
let saved = Document::open(Arc::new(write_full(doc, &SaveOptions::default()).unwrap())).unwrap();
match saved.trailer().get(b"ID") {
Some(Object::Array(a)) => a.iter().map(|s| s.as_string().map(|s| s.bytes.clone()).unwrap_or_default()).collect::<Vec<_>>(),
_ => Vec::new(),
}
};
let mut doc = with_id(hidden_fixture());
assert_eq!(remove_hidden(&mut doc, &[Hidden::Metadata]).unwrap().len(), 1, "the fixture has metadata to remove");
assert!(doc.new_id_required());
let ids = saved_ids(&doc);
assert_eq!(ids.len(), 2);
assert_ne!(ids[0], vec![0x11; 16], "the copy keeps the source's first identifier string");
assert_ne!(ids[1], vec![0x22; 16], "the copy keeps the source's second identifier string");
}
/// Issue #2447: in a tagged PDF the structure tree points at the link and the comment it tags
/// (`/OBJR`). Removing them must drop those references too, or the full rewrite keeps the
/// annotations, the URL and the comment text included, while the page no longer lists them.
@@ -3872,6 +4147,41 @@ fn a_reply_on_another_page_to_a_removed_popup_goes_too() {
}
}
/// An invisible OCR-style text layer drawn in a Type3 font (what the OCR output writes for
/// characters beyond WinAnsi) is redacted like any other text: its glyphs are hit-tested by
/// their real advances, removed, and the verifier passes.
#[test]
fn a_type3_text_layer_is_redacted_like_any_other() {
// One glyph (code 1): a filled box, 500 units wide, with a ToUnicode CMap that reads it
// back as U+3042.
let t3 = b"<< /Type /Font /Subtype /Type3 /Name /PCOcr /FontBBox [0 0 700 700] /FontMatrix [0.001 0 0 0.001 0 0] \
/FirstChar 1 /LastChar 1 /Widths [500] /Encoding << /Type /Encoding /Differences [1 /g01] >> \
/CharProcs << /g01 8 0 R >> /ToUnicode 9 0 R >>";
let charproc = b"500 0 0 0 0 700 700 d1\n0 0 700 700 re f\n";
let cmap = b"/CIDInit /ProcSet findresource begin\n12 dict begin\nbegincmap\n/CMapType 2 def\n1 begincodespacerange\n<01> <01>\nendcodespacerange\n1 beginbfchar\n<01> <3042>\nendbfchar\nendcmap\nCMapName currentdict /CMap defineresource pop\nend\nend\n";
let layer = b"BT 3 Tr /T3 10 Tf 100 100 Td (\x01) Tj ET";
let mut doc = pdf(vec![
b"<< /Type /Catalog /Pages 2 0 R >>".to_vec(),
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".to_vec(),
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 4 0 R /Resources << /Font << /T3 5 0 R >> >> >>".to_vec(),
stream("", layer),
t3.to_vec(),
FONT.replace("95 0 R", "6 0 R").into_bytes(),
widths(),
stream("", charproc),
stream("", cmap),
]);
// The glyph box at size 10: 5 pt wide, from the font's descent to its ascent.
assert_eq!(under(&mut doc, 0, &[[98.0, 96.0, 112.0, 110.0]]), 1, "the Type3 glyph is found under the mark");
assert_eq!(under(&mut doc, 0, &[[200.0, 200.0, 250.0, 250.0]]), 0, "elsewhere the page is clean");
mark(&mut doc, 0, &[[98.0, 96.0, 112.0, 110.0]], "");
let report = apply(&mut doc, None).unwrap();
assert_eq!(report.glyphs, 1, "{report:?}");
// The verified apply left no text-showing operator: the re-read page is empty of them.
assert!(!content(&doc, 0).contains("Tj"), "{}", content(&doc, 0));
assert_eq!(under(&mut doc, 0, &[[98.0, 96.0, 112.0, 110.0]]), 0, "nothing is left under the mark");
}
#[test]
fn a_rewritten_form_xobject_drops_the_source_text_of_added_content() {
// A form drawing invisible text whose dictionary repeats it in /PCAdded.
crates/redact/src/tests_verify.rs
+1380
View File
File diff suppressed because it is too large. Load diff
crates/redact/src/verify/extract.rs
+711
View File
@@ -0,0 +1,711 @@
//! A minimal text extractor, independent of the interpreter's font metrics and hit test.
use std::collections::HashMap;
use pdfcraft_content::{Matrix, parse};
use pdfcraft_cos::{Dict, Document, ObjRef, Object, Stream};
use super::util::{Ctx, latin1, resolve_strict, squash};
use crate::annots_of;
use crate::verify::limits::{MAX_CMAP, MAX_CMAP_ENTRIES, MAX_CMAP_WORK, MAX_DEPTH};
/// Is this one of the glyph-carrying text-showing operators (`Tj`, `'`, `"`, and `TJ` with at
/// least one non-empty string)? A `TJ` of numbers only (the gap redaction leaves for a removed
/// string) moves the cursor but shows nothing, so it does not count.
pub(super) fn carries_glyphs(op: &pdfcraft_content::Op) -> bool {
let nonempty = |o: &Object| o.as_string().is_some_and(|s| !s.bytes.is_empty());
match op.op.as_slice() {
b"Tj" | b"'" | b"\"" => op.operands.last().is_some_and(nonempty),
b"TJ" => op.operands.first().and_then(Object::as_array).is_some_and(|a| a.iter().any(nonempty)),
_ => false,
}
}
/// A page's text as two whitespace-free views: decoded text, and the character codes as Latin-1
/// (what fonts without a usable mapping show).
#[derive(Default)]
pub(super) struct Extracted {
pub(super) text: String,
pub(super) raw: String,
pub(super) glyph_ops: usize,
/// Glyph-carrying operators whose text could not be decoded.
pub(super) undecodable_ops: usize,
/// Per region asked about: glyphs whose centre this extractor places inside it.
pub(super) shown_in: Vec<usize>,
/// Glyph-carrying operators drawn where this extractor can't place them (tiling patterns,
/// or an annotation appearance without a usable `/Rect`).
pub(super) unplaced: usize,
}
/// How one font turns codes into text.
#[derive(Clone, Default)]
pub(super) struct Decoder {
pub(super) map: HashMap<u32, String>,
/// Bytes per code.
pub(super) width: usize,
/// No way to read these codes as text (composite font without a mapping).
pub(super) opaque: bool,
}
impl Decoder {
fn latin1() -> Self {
Decoder { map: HashMap::new(), width: 1, opaque: false }
}
/// `Err` when the font's own data can't be read in full (a damaged or oversized CMap).
fn new(doc: &Document, font: &Dict, ctx: &Ctx) -> Result<Self, ()> {
let composite = font.name(b"Subtype") == Some(b"Type0");
let mut d = Decoder { map: HashMap::new(), width: if composite { 2 } else { 1 }, opaque: false };
let tu = font.get(b"ToUnicode").map(|t| doc.resolve(t));
if let Some(Object::Stream(s)) = tu.as_deref() {
let data = ctx.decode(s)?;
if data.len() > MAX_CMAP {
return Err(());
}
let (map, width) = parse_cmap(&data).ok_or(())?;
if !map.is_empty() {
d.map = map;
if let Some(w) = width {
d.width = w;
}
return Ok(d);
}
}
if composite {
d.opaque = true;
return Ok(d);
}
// Simple font: Latin-1, with the /Differences of the encoding.
if let Some(enc) = font.get(b"Encoding").map(|e| doc.resolve(e))
&& let Some(diff) = enc.as_dict().and_then(|e| e.get(b"Differences")).map(|x| doc.resolve(x))
&& let Some(items) = diff.as_array()
{
let mut code = 0u32;
for it in items {
match &*doc.resolve(it) {
Object::Int(i) => code = u32::try_from(*i).unwrap_or(0),
Object::Name(n) => {
if let Some(c) = std::str::from_utf8(n).ok().and_then(pdfcraft_fonts::pdf::glyph_unicode) {
d.map.insert(code, c.to_string());
}
code = code.saturating_add(1);
}
_ => {}
}
}
}
Ok(d)
}
/// Each code of a shown string, with the text it gives (`None`: no way to read it as text).
fn split(&self, s: &[u8]) -> Vec<(u32, Option<String>)> {
let w = self.width.max(1);
s.chunks(w)
.map(|c| {
let code = code_of(c);
let text = if self.opaque {
None
} else {
match self.map.get(&code) {
Some(t) => Some(t.clone()),
None if self.map.is_empty() || w == 1 && code < 256 => Some(char::from(u8::try_from(code).unwrap_or(b'?')).to_string()),
None => None,
}
};
(code, text)
})
.collect()
}
/// The text of a shown string.
fn decode(&self, s: &[u8]) -> String {
self.split(s).into_iter().filter_map(|(_, t)| t).collect()
}
}
/// A font as the geometric check sees it: its decoder and its own simple width model.
#[derive(Clone)]
pub(super) struct Font {
pub(super) dec: Decoder,
/// Advance per code, in text space units per 1 em of font size (0.5 = half an em).
pub(super) widths: HashMap<u32, f64>,
/// Advance of a code `widths` doesn't list.
pub(super) default_width: f64,
pub(super) vertical: bool,
}
/// The width a font without usable `/Widths` is assumed to have (half an em).
pub(super) const FLAT_WIDTH: f64 = 0.5;
impl Font {
fn flat() -> Self {
Font { dec: Decoder::latin1(), widths: HashMap::new(), default_width: FLAT_WIDTH, vertical: false }
}
fn new(doc: &Document, font: &Dict, ctx: &Ctx) -> Result<Self, ()> {
let dec = Decoder::new(doc, font, ctx)?;
let mut f = Font { dec, widths: HashMap::new(), default_width: FLAT_WIDTH, vertical: false };
let num = |o: &Object| doc.resolve(o).as_f64();
if font.name(b"Subtype") == Some(b"Type0") {
f.vertical = font.get(b"Encoding").map(|e| doc.resolve(e)).and_then(|e| e.as_name().map(|n| n.ends_with(b"-V"))).unwrap_or(false);
let desc = font.get(b"DescendantFonts").map(|d| doc.resolve(d)).and_then(|d| d.as_array().and_then(|a| a.first().cloned()));
let desc = desc.and_then(|d| doc.resolve(&d).as_dict().cloned());
if let Some(d) = desc {
f.default_width = d.get(b"DW").and_then(num).map_or(1.0, |w| w / 1000.0);
// The array is charged to the shared work budget as it is taken (a form
// invoked again rebuilds its fonts, so a per-font counter alone would let the
// same work be paid for many times), and every entry is paid for there too,
// whether or not it grows the map (a range may rewrite the same codes again
// and again, and the array is bounded only by the file).
let items: Vec<Object> = match d.get(b"W").map(|w| doc.resolve(w)).and_then(|w| w.as_array().cloned()) {
Some(a) => {
if !ctx.charge(a.len()) {
return Err(());
}
a
}
None => Vec::new(),
};
let mut work = 0usize;
let mut step = |n: usize| -> Result<(), ()> {
work = work.saturating_add(n);
if work <= MAX_CMAP_WORK && ctx.charge(n) { Ok(()) } else { Err(()) }
};
let mut i = 0;
while let Some(first) = items.get(i).and_then(num) {
let first = first.max(0.0) as u32;
match items.get(i + 1).map(|o| doc.resolve(o)) {
Some(a) if a.as_array().is_some() => {
step(a.as_array().map_or(0, Vec::len))?;
for (k, w) in a.as_array().into_iter().flatten().enumerate() {
if f.widths.len() >= MAX_CMAP_ENTRIES {
return Err(());
}
if let (Some(w), Ok(k)) = (num(w), u32::try_from(k)) {
f.widths.insert(first.saturating_add(k), w / 1000.0);
}
}
i += 2;
}
Some(last) => {
let (Some(last), Some(w)) = (last.as_f64(), items.get(i + 2).and_then(num)) else { break };
let last = (last.max(0.0) as u32).min(first.saturating_add(0xFFFF));
// Each code the range covers is paid for, grown or not (a span is at
// most 0x10000 codes, so the count always fits).
step(last.saturating_sub(first) as usize + 1)?;
for c in first..=last {
if f.widths.len() >= MAX_CMAP_ENTRIES {
return Err(());
}
f.widths.insert(c, w / 1000.0);
}
i += 3;
}
None => break,
}
}
}
} else {
// Glyph space units per em: 1000, or what a Type3 font's matrix says.
let scale = if font.name(b"Subtype") == Some(b"Type3") {
font.get(b"FontMatrix").map(|m| doc.resolve(m)).and_then(|m| m.as_array().and_then(|a| a.first().and_then(num))).unwrap_or(0.001)
} else {
0.001
};
let first = font.get(b"FirstChar").and_then(num).map_or(0, |v| v.max(0.0) as u32);
let missing = font
.get(b"FontDescriptor")
.map(|d| doc.resolve(d))
.and_then(|d| d.as_dict().and_then(|d| d.get(b"MissingWidth").and_then(num)))
.map(|w| w * scale);
if let Some(ws) = font.get(b"Widths").map(|w| doc.resolve(w)).and_then(|w| w.as_array().cloned()) {
if ws.len() > MAX_CMAP_ENTRIES {
return Err(());
}
for (k, w) in ws.iter().enumerate() {
if let (Some(w), Ok(k)) = (num(w), u32::try_from(k)) {
f.widths.insert(first.saturating_add(k), w * scale);
}
}
f.default_width = missing.unwrap_or(0.0);
} else if let Some(std) = pdfcraft_fonts::Std14::from_base_font(&String::from_utf8_lossy(font.name(b"BaseFont").unwrap_or(b""))) {
// No /Widths: a standard font has exact published widths, by WinAnsi code (Symbol,
// ZapfDingbats and other encodings are refused before the proof is reached).
for code in 0..=255u8 {
f.widths.insert(u32::from(code), std.width(code) / 1000.0);
}
f.default_width = if std.is_fixed_pitch() { 0.6 } else { 0.0 };
}
}
Ok(f)
}
fn width(&self, code: u32) -> f64 {
self.widths.get(&code).copied().unwrap_or(self.default_width)
}
}
pub(super) fn hex_token(tok: &[u8]) -> Option<Vec<u8>> {
let digits: Vec<u8> = tok.iter().filter_map(|c| (*c as char).to_digit(16).map(|d| d as u8)).collect();
if digits.is_empty() || !digits.len().is_multiple_of(2) {
return None;
}
Some(digits.chunks(2).map(|p| p.iter().fold(0u8, |a, d| (a << 4) | d)).collect())
}
pub(super) fn utf16_text(b: &[u8]) -> String {
if let [c] = b {
return char::from(*c).to_string();
}
let units: Vec<u16> = b.as_chunks::<2>().0.iter().map(|c| u16::from_be_bytes(*c)).collect();
String::from_utf16_lossy(&units)
}
pub(super) fn code_of(b: &[u8]) -> u32 {
b.iter().fold(0u32, |a, x| a.wrapping_shl(8) | u32::from(*x))
}
pub(super) enum Tok {
Hex(Vec<u8>),
Word(String),
Open,
Close,
}
pub(super) fn cmap_tokens(data: &[u8]) -> Vec<Tok> {
let mut out = Vec::new();
let mut i = 0;
while let Some(&c) = data.get(i) {
match c {
b'<' if data.get(i + 1) != Some(&b'<') => {
let end = data.get(i..).and_then(|d| d.iter().position(|x| *x == b'>')).map_or(data.len(), |e| i + e);
if let Some(h) = data.get(i + 1..end).and_then(hex_token) {
out.push(Tok::Hex(h));
}
i = end + 1;
}
b'[' => {
out.push(Tok::Open);
i += 1;
}
b']' => {
out.push(Tok::Close);
i += 1;
}
b'%' => {
while data.get(i).is_some_and(|x| !matches!(x, b'\n' | b'\r')) {
i += 1;
}
}
c if c.is_ascii_alphabetic() => {
let start = i;
while data.get(i).is_some_and(|x| x.is_ascii_alphanumeric()) {
i += 1;
}
out.push(Tok::Word(String::from_utf8_lossy(data.get(start..i).unwrap_or_default()).into_owned()));
}
_ => i += 1,
}
}
out
}
/// `bfchar` / `bfrange` entries of a ToUnicode CMap, and the code length of its codespace.
/// `None` when the CMap is bigger than [`MAX_CMAP_ENTRIES`] entries or costs more than
/// [`MAX_CMAP_WORK`] steps: a map that is cut short would read as "no such code".
pub(super) fn parse_cmap(data: &[u8]) -> Option<(HashMap<u32, String>, Option<usize>)> {
let toks = cmap_tokens(data);
let mut map: HashMap<u32, String> = HashMap::new();
let mut width = None;
// Every step is paid for, whether or not it grows the map (a range may rewrite the same
// codes again and again).
let mut work = 0usize;
let mut step = |n: usize| -> Option<()> {
work = work.saturating_add(n);
(work <= MAX_CMAP_WORK).then_some(())
};
let mut i = 0;
while let Some(t) = toks.get(i) {
i += 1;
let Tok::Word(w) = t else { continue };
match w.as_str() {
"begincodespacerange" => {
if let Some(Tok::Hex(h)) = toks.get(i) {
width = Some(h.len());
}
}
"beginbfchar" => {
while let (Some(Tok::Hex(src)), Some(Tok::Hex(dst))) = (toks.get(i), toks.get(i + 1)) {
step(1)?;
map.insert(code_of(src), utf16_text(dst));
if map.len() > MAX_CMAP_ENTRIES {
return None;
}
i += 2;
}
}
"beginbfrange" => {
while let (Some(Tok::Hex(lo)), Some(Tok::Hex(hi))) = (toks.get(i), toks.get(i + 1)) {
let (lo, hi) = (code_of(lo), code_of(hi));
let span = hi.saturating_sub(lo).min(0xFFFF);
match toks.get(i + 2) {
Some(Tok::Hex(dst)) => {
let base = utf16_text(dst);
step(usize::try_from(span).ok()?.saturating_add(1))?;
for k in 0..=span {
let mut chars: Vec<char> = base.chars().collect();
if let Some(last) = chars.last_mut() {
*last = char::from_u32(u32::from(*last).saturating_add(k)).unwrap_or('\u{fffd}');
}
map.insert(lo.saturating_add(k), chars.into_iter().collect());
}
if map.len() > MAX_CMAP_ENTRIES {
return None;
}
i += 3;
}
Some(Tok::Open) => {
i += 3;
let mut k = 0u32;
while let Some(Tok::Hex(d)) = toks.get(i) {
step(1)?;
map.insert(lo.saturating_add(k), utf16_text(d));
if map.len() > MAX_CMAP_ENTRIES {
return None;
}
k = k.saturating_add(1);
i += 1;
}
i += 1; // the closing bracket
}
_ => break,
}
}
}
_ => {}
}
}
Some((map, width))
}
pub(super) fn res_dict(doc: &Document, res: &Dict, key: &[u8]) -> Dict {
res.get(key).map(|o| doc.resolve(o)).and_then(|o| o.as_dict().cloned()).unwrap_or_default()
}
pub(super) fn push_squashed(into: &mut String, s: &str) {
into.extend(s.chars().filter(|c| !c.is_whitespace() && *c != '\u{feff}'));
}
/// A form's or pattern's `/Matrix`: absent is the identity, present but malformed is an error
/// (guessing the identity would put its content in the wrong place).
pub(super) fn matrix_of(doc: &Document, dict: &Dict) -> Result<Matrix, ()> {
match dict.get(b"Matrix") {
None => Ok(Matrix::IDENTITY),
Some(m) => {
let m = doc.resolve(m);
let items: Vec<Object> = m.as_array().ok_or(())?.iter().map(|x| (*doc.resolve(x)).clone()).collect();
Matrix::from_operands(&items).filter(|m| m.0.iter().all(|v| v.is_finite())).ok_or(())
}
}
}
/// A form's `/BBox` as `[x0 y0 x1 y1]`: absent is `None`, present but malformed is an error.
pub(super) fn bbox_of(doc: &Document, dict: &Dict) -> Result<Option<[f64; 4]>, ()> {
let Some(b) = dict.get(b"BBox") else { return Ok(None) };
let b = doc.resolve(b);
let v: Vec<f64> =
b.as_array().ok_or(())?.iter().map(|x| doc.resolve(x).as_f64().filter(|v| v.is_finite()).ok_or(())).collect::<Result<_, _>>()?;
match v[..] {
[a, b, c, d] => Ok(Some([a.min(c), b.min(d), a.max(c), b.max(d)])),
_ => Err(()),
}
}
/// The content streams of a page as `(raw object, decoded)`, a missing or unreadable entry an
/// error. `skip_overlay` leaves out the streams this run drew the boxes and labels with.
pub(super) fn page_content(doc: &Document, page: &Dict, ctx: &Ctx, skip_overlay: bool) -> Result<Vec<u8>, ()> {
let list: Vec<Object> = match page.get(b"Contents") {
None => Vec::new(),
Some(c) => match &*resolve_strict(doc, c)? {
Object::Array(a) => a.clone(),
_ => vec![c.clone()],
},
};
let mut data: Vec<u8> = Vec::new();
for o in &list {
match &*resolve_strict(doc, o)? {
Object::Stream(s) => {
let d = ctx.decode(s)?;
if skip_overlay && ctx.is_overlay(&d) {
continue;
}
data.extend(d);
data.push(b'\n');
}
// A null entry is an empty piece, as for the redaction itself (ISO 32000-2 7.3.9).
Object::Null => {}
_ => return Err(()),
}
}
Ok(data)
}
/// The text of a page: its content, the forms and patterns it reaches (visible or not), and the
/// appearance streams of its annotations; and which of `regions` still hold glyphs.
pub(super) fn extract_page(doc: &Document, page: &Dict, ctx: &Ctx, regions: &[[f64; 4]]) -> Result<Extracted, ()> {
let data = page_content(doc, page, ctx, true)?;
let res = page.get(b"Resources").and_then(|r| doc.resolve(r).as_dict().cloned()).unwrap_or_default();
let mut ex = Extractor { doc, ctx, regions, out: Extracted { shown_in: vec![0; regions.len()], ..Extracted::default() }, stack: Vec::new() };
ex.scope(&data, &res, Matrix::IDENTITY, 0, true)?;
for a in annots_of(doc, page) {
let Some(d) = doc.resolve(&a).as_dict().cloned() else { continue };
let Some(ap) = d.get(b"AP").and_then(|x| doc.resolve(x).as_dict().cloned()) else { continue };
let rect =
d.get(b"Rect").and_then(|r| doc.resolve(r).as_array().map(|a| a.iter().filter_map(|x| doc.resolve(x).as_f64()).collect::<Vec<_>>()));
for (_, state) in ap.iter() {
let s = doc.resolve(state);
let streams: Vec<(Option<ObjRef>, Object)> = match &*s {
Object::Stream(_) => vec![(state.as_ref(), (*s).clone())],
Object::Dict(sub) => sub.iter().map(|(_, v)| (v.as_ref(), (*doc.resolve(v)).clone())).collect(),
_ => Vec::new(),
};
for (r, o) in streams {
if let Object::Stream(st) = o {
// Where the appearance lands on the page (ISO 32000-2 §12.5.5), if it can be told.
let ctm = rect.as_deref().and_then(|r| appearance_matrix(doc, &st, r));
// A font of the appearance is looked up in the appearance's own resources
// first (what the redaction itself does), then in the page's.
let own = st.dict.get(b"Resources").and_then(|x| doc.resolve(x).as_dict().cloned());
let parent = own.as_ref().unwrap_or(&res);
ex.stream(&st, r, parent, ctm.unwrap_or_default(), 1, ctm.is_some())?;
}
}
}
}
Ok(ex.out)
}
/// The matrix that maps an appearance stream's form space onto an annotation's `/Rect`
/// (ISO 32000-2 §12.5.5): the appearance's `/Matrix`, then the scale that fits the mapped
/// `/BBox` into the rectangle. `None` when that can't be told (no usable `/Rect`, `/BBox` or
/// `/Matrix`, or a mapped box that degenerates).
pub(crate) fn appearance_matrix(doc: &Document, ap: &Stream, rect: &[f64]) -> Option<Matrix> {
let [rx0, ry0, rx1, ry1] = rect[..] else { return None };
let bbox = bbox_of(doc, &ap.dict).ok()??;
let m = matrix_of(doc, &ap.dict).ok()?;
let t = m.bbox(bbox);
let (tw, th) = (t[2] - t[0], t[3] - t[1]);
if tw <= 0.0 || th <= 0.0 {
return None;
}
let (rx0, rx1, ry0, ry1) = (rx0.min(rx1), rx0.max(rx1), ry0.min(ry1), ry0.max(ry1));
let (sx, sy) = ((rx1 - rx0) / tw, (ry1 - ry0) / th);
let a = Matrix([sx, 0.0, 0.0, sy, rx0 - t[0] * sx, ry0 - t[1] * sy]);
Some(m.then(&a))
}
/// Text-state parameters that `q`/`Q` save and restore.
#[derive(Clone)]
pub(super) struct TextGs {
pub(super) ctm: Matrix,
pub(super) tc: f64,
pub(super) tw: f64,
pub(super) th: f64,
pub(super) tl: f64,
pub(super) rise: f64,
pub(super) size: f64,
pub(super) font: Vec<u8>,
}
pub(super) struct Extractor<'a> {
pub(super) doc: &'a Document,
pub(super) ctx: &'a Ctx,
pub(super) regions: &'a [[f64; 4]],
pub(super) out: Extracted,
pub(super) stack: Vec<ObjRef>,
}
impl Extractor<'_> {
/// A form or pattern stream. `ctm` maps its space onto the page's; `placed` is false where
/// that mapping is not known.
fn stream(&mut self, s: &Stream, id: Option<ObjRef>, parent: &Dict, ctm: Matrix, depth: usize, placed: bool) -> Result<(), ()> {
if let Some(r) = id {
if self.stack.contains(&r) {
return Ok(());
}
self.stack.push(r);
}
let result = (|| {
let data = self.ctx.decode(s)?;
let own = s.dict.get(b"Resources").and_then(|r| self.doc.resolve(r).as_dict().cloned());
self.scope(&data, own.as_ref().unwrap_or(parent), ctm, depth, placed)
})();
if id.is_some() {
self.stack.pop();
}
result
}
/// Show the strings of one text-showing operator: extract their text, and place each glyph
/// to count the ones inside a region. `tm` moves along with the pen.
fn show_text(&mut self, font: &Font, op: &pdfcraft_content::Op, gs: &TextGs, tm: &mut Matrix, placed: bool) {
let o = op.op.as_slice();
// Strings and TJ adjustments, in order.
let items: Vec<&Object> = match o {
b"TJ" => op.operands.first().and_then(Object::as_array).map(|a| a.iter().collect()).unwrap_or_default(),
_ => op.operands.last().into_iter().collect(),
};
let mut any_text = false;
for it in items {
if let Some(n) = it.as_f64() {
let d = -n / 1000.0 * gs.size;
let m = if font.vertical { Matrix::translate(0.0, d) } else { Matrix::translate(d * gs.th, 0.0) };
*tm = m.then(tm);
continue;
}
let Some(s) = it.as_string() else { continue };
let t = font.dec.decode(&s.bytes);
any_text |= !squash(&t).is_empty();
push_squashed(&mut self.out.text, &t);
push_squashed(&mut self.out.raw, &latin1(&s.bytes));
for (code, text) in font.dec.split(&s.bytes) {
let w = font.width(code);
let space = text.as_deref().is_some_and(|t| t == " ");
// The glyph's centre, in text space (the box runs from 0.2 em below the
// baseline to 0.8 em above), and where the pen moves to.
let (centre, advance) = if font.vertical {
let adv = -gs.size + gs.tc + if space { gs.tw } else { 0.0 };
((0.0, gs.rise - 0.5 * gs.size), Matrix::translate(0.0, adv))
} else {
let adv = (w * gs.size + gs.tc + if space { gs.tw } else { 0.0 }) * gs.th;
((0.5 * w * gs.size * gs.th, gs.rise + 0.3 * gs.size), Matrix::translate(adv, 0.0))
};
let visible = text.as_deref().is_none_or(|t| !squash(t).is_empty());
if placed && visible && !self.regions.is_empty() {
let (x, y) = tm.then(&gs.ctm).apply(centre.0, centre.1);
for (r, n) in self.regions.iter().zip(self.out.shown_in.iter_mut()) {
if x >= r[0] && x <= r[2] && y >= r[1] && y <= r[3] {
*n += 1;
}
}
}
*tm = advance.then(tm);
}
}
if carries_glyphs(op) {
self.out.glyph_ops += 1;
if !any_text {
self.out.undecodable_ops += 1;
}
if !placed {
self.out.unplaced += 1;
}
}
}
fn scope(&mut self, data: &[u8], res: &Dict, ctm: Matrix, depth: usize, placed: bool) -> Result<(), ()> {
if depth > MAX_DEPTH {
return Err(());
}
let doc = self.doc;
let fonts = res_dict(doc, res, b"Font");
let xobjects = res_dict(doc, res, b"XObject");
let mut cache: HashMap<Vec<u8>, Font> = HashMap::new();
let mut gs = TextGs { ctm, tc: 0.0, tw: 0.0, th: 1.0, tl: 0.0, rise: 0.0, size: 0.0, font: Vec::new() };
let mut saved: Vec<TextGs> = Vec::new();
let (mut tm, mut tlm) = (Matrix::IDENTITY, Matrix::IDENTITY);
for op in parse(data).ops {
let o = op.op.as_slice();
match o {
b"q" => {
if saved.len() >= 4096 {
return Err(());
}
saved.push(gs.clone());
}
b"Q" => {
if let Some(g) = saved.pop() {
gs = g;
}
}
b"cm" => {
if let Some(m) = op.nums::<6>() {
gs.ctm = Matrix(m).then(&gs.ctm);
}
}
b"BT" => {
tm = Matrix::IDENTITY;
tlm = Matrix::IDENTITY;
}
b"Tc" => gs.tc = op.num(0).unwrap_or(0.0),
b"Tw" => gs.tw = op.num(0).unwrap_or(0.0),
b"Tz" => gs.th = op.num(0).unwrap_or(100.0) / 100.0,
b"TL" => gs.tl = op.num(0).unwrap_or(0.0),
b"Ts" => gs.rise = op.num(0).unwrap_or(0.0),
b"Tf" => {
gs.font = op.name(0).map(<[u8]>::to_vec).unwrap_or_default();
gs.size = op.num(1).unwrap_or(0.0);
}
b"Td" | b"TD" => {
if let Some([x, y]) = op.nums::<2>() {
if o == b"TD" {
gs.tl = -y;
}
tlm = Matrix::translate(x, y).then(&tlm);
tm = tlm;
}
}
b"Tm" => {
if let Some(m) = op.nums::<6>() {
tm = Matrix(m);
tlm = tm;
}
}
b"T*" => {
tlm = Matrix::translate(0.0, -gs.tl).then(&tlm);
tm = tlm;
}
b"Tj" | b"TJ" | b"'" | b"\"" => {
if matches!(o, b"'" | b"\"") {
tlm = Matrix::translate(0.0, -gs.tl).then(&tlm);
tm = tlm;
}
if o == b"\""
&& let (Some(aw), Some(ac)) = (op.num(0), op.num(1))
{
(gs.tw, gs.tc) = (aw, ac);
}
if !cache.contains_key(&gs.font) {
let f = match fonts.get(&gs.font).map(|f| doc.resolve(f)) {
Some(f) => f.as_dict().map_or_else(|| Ok(Font::flat()), |d| Font::new(doc, d, self.ctx))?,
None => Font::flat(),
};
cache.insert(gs.font.clone(), f);
}
let Some(font) = cache.get(&gs.font) else { continue };
self.show_text(font, &op, &gs, &mut tm, placed);
}
b"Do" => {
let Some(name) = op.name(0) else { continue };
let Some(entry) = xobjects.get(name) else { continue };
if let Object::Stream(s) = &*doc.resolve(entry)
&& s.dict.name(b"Subtype") == Some(b"Form")
{
let m = matrix_of(doc, &s.dict)?;
bbox_of(doc, &s.dict)?;
self.stream(s, entry.as_ref(), res, m.then(&gs.ctm), depth + 1, placed)?;
}
}
_ => {}
}
}
// Tiling patterns draw content of their own, repeated where the pattern is used: its
// glyphs can't be placed.
for (_, p) in res_dict(doc, res, b"Pattern").iter() {
if let Object::Stream(s) = &*doc.resolve(p) {
self.stream(s, p.as_ref(), res, Matrix::IDENTITY, depth + 1, false)?;
}
}
Ok(())
}
}
crates/redact/src/verify/geometry.rs
+500
View File
@@ -0,0 +1,500 @@
//! The non-text content under the regions: images, paths and shadings, and the placement of the
//! glyphs still shown, checked independently of the interpreter.
use std::collections::HashSet;
use pdfcraft_content::{Matrix, contains, overlaps, parse};
use pdfcraft_cos::{Dict, Document, ObjRef, Object, Stream};
use super::extract::{appearance_matrix, bbox_of, matrix_of, page_content, res_dict};
use super::util::{Ctx, image_codec};
use crate::verify::limits::{MAX_CLIP_SUBPATHS, MAX_DEPTH, MAX_GS_STACK, MAX_HOLES, MAX_PIXELS};
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub(super) enum Finding {
/// This many glyphs are placed inside the region.
Text(usize),
Image,
Path,
Shading,
Missing,
Unverifiable(&'static str),
}
#[derive(Clone)]
pub(super) struct Gs {
pub(super) ctm: Matrix,
/// Device-space boxes cut out of the clip (even-odd clips with a hole).
pub(super) holes: Vec<[f64; 4]>,
}
pub(super) struct Geo<'a> {
pub(super) doc: &'a Document,
pub(super) ctx: &'a Ctx,
pub(super) regions: &'a [[f64; 4]],
pub(super) found: Vec<(usize, Finding)>,
/// What `found` already holds: each finding is recorded once per region, so a page that
/// paints thousands of times cannot grow the report (or the memory) with repeats.
reported: HashSet<(usize, Finding)>,
/// Pattern, soft-mask and Type3 resources already searched for images.
pub(super) searched: HashSet<ObjRef>,
}
/// What still paints inside `regions` on a page of the output (the overlay boxes excepted):
/// images with pixels there, paths and shadings not clipped away from there.
pub(super) fn check_geometry(doc: &Document, page: &Dict, regions: &[[f64; 4]], ctx: &Ctx) -> Vec<(usize, Finding)> {
let unverifiable = |why: &'static str| (0..regions.len()).map(|i| (i, Finding::Unverifiable(why))).collect();
// The redaction boxes are painted on purpose.
let Ok(data) = page_content(doc, page, ctx, true) else { return unverifiable("the page content can't be read in full") };
let mut geo = Geo { doc, ctx, regions, found: Vec::new(), reported: HashSet::new(), searched: HashSet::new() };
let res = page.get(b"Resources").and_then(|r| doc.resolve(r).as_dict().cloned()).unwrap_or_default();
geo.run(&data, &res, Matrix::IDENTITY, Vec::new(), 0);
geo.appearances(page, &res);
geo.found
}
impl Geo<'_> {
/// Record a finding for region `i`, once: a page that paints over a region a thousand
/// times is one finding, not a thousand.
fn report(&mut self, i: usize, f: Finding) {
if self.reported.insert((i, f.clone())) {
self.found.push((i, f));
}
}
/// The appearances of the annotations still on the page, checked like the page itself: their
/// content is placed with the appearance matrix (ISO 32000-2 §12.5.5) and swept for images,
/// paths and shadings under the regions. An appearance that has content but can't be placed
/// makes every region unverifiable: the redaction refuses such documents, so one in the
/// output means what it paints can't be vouched for.
fn appearances(&mut self, page: &Dict, res: &Dict) {
let doc = self.doc;
for a in crate::annots_of(doc, page) {
// A direct dictionary has no reference of its own; it is checked like an indirect
// one (what its appearance paints does not depend on how it is stored).
let Some(d) = doc.resolve(&a).as_dict().cloned() else { continue };
// A redaction mark's appearance outlines the marked area on purpose (and applying
// removes the mark whole): it is exempt like the boxes drawn over the regions.
if d.name(b"Subtype") == Some(b"Redact") {
continue;
}
let Some(ap) = d.get(b"AP").and_then(|x| doc.resolve(x).as_dict().cloned()) else { continue };
let rect = crate::rect_of(doc, d.get(b"Rect"));
for (_, state) in ap.iter() {
let s = doc.resolve(state);
let streams: Vec<Stream> = match &*s {
Object::Stream(st) => vec![st.clone()],
Object::Dict(sub) => sub
.iter()
.filter_map(|(_, v)| match &*doc.resolve(v) {
Object::Stream(st) => Some(st.clone()),
_ => None,
})
.collect(),
_ => Vec::new(),
};
for st in streams {
// A stream shared by several annotations is checked at each annotation's
// own placement: where it lands comes from each `/Rect`, so the first
// placement tells nothing about the others. Repeated decodes spend the
// same bounded budget as everything else.
let Ok(data) = self.ctx.decode(&st) else {
self.unverifiable("an annotation appearance can't be read in full");
continue;
};
if data.is_empty() {
continue;
}
let Some(ctm) = rect.as_ref().and_then(|r| appearance_matrix(doc, &st, r)) else {
self.unverifiable("an annotation appearance can't be placed, so what it paints can't be checked");
continue;
};
let own = st.dict.get(b"Resources").and_then(|x| doc.resolve(x).as_dict().cloned());
self.run(&data, own.as_ref().unwrap_or(res), ctm, Vec::new(), 1);
}
}
}
}
fn touching(&self, b: [f64; 4]) -> Vec<usize> {
(0..self.regions.len()).filter(|&i| self.regions.get(i).is_some_and(|r| overlaps(*r, b, 0.0))).collect()
}
/// Does something painting over `within` reach region `i` through the clip? Only the part of
/// the region the painting could touch has to be cut out.
fn unclipped(&self, i: usize, within: [f64; 4], gs: &Gs) -> bool {
let Some(r) = self.regions.get(i) else { return false };
let reach = [r[0].max(within[0]), r[1].max(within[1]), r[2].min(within[2]), r[3].min(within[3])];
!gs.holes.iter().any(|h| contains(*h, reach, 0.01))
}
fn run(&mut self, data: &[u8], res: &Dict, ctm: Matrix, holes: Vec<[f64; 4]>, depth: usize) {
if depth > MAX_DEPTH {
for i in 0..self.regions.len() {
self.report(i, Finding::Unverifiable("forms are nested too deep to check"));
}
return;
}
let xobjects = res_dict(self.doc, res, b"XObject");
self.hidden_images(res, depth);
let mut gs = Gs { ctm, holes };
let mut stack: Vec<Gs> = Vec::new();
// The current path in device space: its subpaths, and the clip operator it carries.
let mut sub: Vec<Vec<(f64, f64)>> = Vec::new();
let mut clip: Option<bool> = None;
for op in parse(data).ops {
let o = op.op.as_slice();
if matches!(o, b"m" | b"l" | b"c" | b"v" | b"y" | b"h" | b"re" | b"W" | b"W*") {
extend_path(&op, &gs.ctm, &mut sub, &mut clip);
continue;
}
if matches!(o, b"S" | b"s" | b"f" | b"F" | b"f*" | b"B" | b"B*" | b"b" | b"b*" | b"n") {
self.paint_path(o, &mut gs, &sub, clip);
sub.clear();
clip = None;
continue;
}
sub.clear();
clip = None;
match o {
b"q" => {
// The saved state carries the holes with it, so an unbounded stack of them
// is unbounded memory: past the cap the walk says so instead of growing.
if stack.len() >= MAX_GS_STACK {
return self.unverifiable("the content stacks graphics states too deeply to check");
}
stack.push(gs.clone());
}
b"Q" => {
if let Some(g) = stack.pop() {
gs = g;
}
}
b"cm" => {
if let Some(m) = op.nums::<6>() {
gs.ctm = Matrix(m).then(&gs.ctm);
}
}
b"sh" => {
for i in 0..self.regions.len() {
if self.unclipped(i, [f64::MIN, f64::MIN, f64::MAX, f64::MAX], &gs) {
self.report(i, Finding::Shading);
}
}
}
b"BI" => {
for i in self.touching(gs.ctm.bbox([0.0, 0.0, 1.0, 1.0])) {
self.report(i, Finding::Image);
}
}
b"Do" => {
let Some(entry) = op.name(0).and_then(|n| xobjects.get(n)) else { continue };
let obj = self.doc.resolve(entry);
let Object::Stream(s) = &*obj else { continue };
match s.dict.name(b"Subtype") {
Some(b"Image") => {
for i in self.touching(gs.ctm.bbox([0.0, 0.0, 1.0, 1.0])) {
let Some(region) = self.regions.get(i) else { continue };
match image_cleared(self.doc, self.ctx, s, &gs.ctm, *region) {
Ok(true) => {}
Ok(false) => self.report(i, Finding::Image),
Err(()) => self.report(i, Finding::Unverifiable("an image under the region can't be inspected")),
}
}
}
Some(b"Form") => self.form(s, &gs, res, depth),
_ => {}
}
}
_ => {}
}
}
}
/// A path-painting operator `o` ends the current path: report what it paints over a region,
/// and let an even-odd clip add its holes to `gs`.
fn paint_path(&mut self, o: &[u8], gs: &mut Gs, sub: &[Vec<(f64, f64)>], clip: Option<bool>) {
let pts: Vec<(f64, f64)> = sub.iter().flatten().copied().collect();
let bbox = pts.iter().fold(None, |b: Option<[f64; 4]>, &(u, v)| {
Some(match b {
None => [u, v, u, v],
Some(b) => [b[0].min(u), b[1].min(v), b[2].max(u), b[3].max(v)],
})
});
if o != b"n"
&& let Some(b) = bbox
{
for i in self.touching(b) {
if self.unclipped(i, b, gs) {
self.report(i, Finding::Path);
}
}
}
// An even-odd clip whose subpaths are all axis-aligned rectangles lets through what an
// odd number of the inner rectangles covers. Where a region lies under the outer box
// and an odd number of inner ones, it is cut out of the clip; under an even number it
// is painted and must be reported. Anything that is not a stock of disjoint rectangles
// (a triangle, a rotated box, overlapping inners, too many of them) cannot be told
// cheaply, so no hole is cut: what the clip lets through is then reported as paint
// (fail closed). This is how redaction itself clips regions out of what it leaves.
if clip != Some(true) {
return;
}
let Some(rects) = sub.iter().map(|s| axis_aligned_rect(s)).collect::<Option<Vec<[f64; 4]>>>() else { return };
if rects.len() < 2 || rects.len() > MAX_CLIP_SUBPATHS {
return;
}
let (outer, inners) = (rects[0], &rects[1..]);
// Inner rectangles must be disjoint, or their overlaps would change the parity the
// reader draws with.
let disjoint = inners.iter().enumerate().all(|(x, a)| inners.iter().skip(x + 1).all(|b| !overlaps(*a, *b, 0.0)));
let regions: Vec<[f64; 4]> = self.regions.to_vec();
let mut cut = Vec::new();
for r in regions {
if contains(outer, r, 0.0) && disjoint && inners.iter().filter(|b| contains(**b, r, 0.0)).count() % 2 == 1 {
cut.push(r);
}
}
if gs.holes.len() + cut.len() > MAX_HOLES {
return self.unverifiable("the content cuts clips with too many pieces to check");
}
gs.holes.extend(cut);
}
fn unverifiable(&mut self, why: &'static str) {
for i in 0..self.regions.len() {
self.report(i, Finding::Unverifiable(why));
}
}
fn form(&mut self, s: &Stream, gs: &Gs, parent: &Dict, depth: usize) {
let (Ok(m), Ok(bbox)) = (matrix_of(self.doc, &s.dict), bbox_of(self.doc, &s.dict)) else {
self.unverifiable("a form under the region has a malformed /Matrix or /BBox");
return;
};
let fctm = m.then(&gs.ctm);
if let Some(bb) = bbox
&& self.touching(fctm.bbox(bb)).is_empty()
{
return;
}
let Ok(data) = self.ctx.decode(s) else {
self.unverifiable("a form under the region can't be decoded in full");
return;
};
let own = s.dict.get(b"Resources").and_then(|r| self.doc.resolve(r).as_dict().cloned());
self.run(&data, own.as_ref().unwrap_or(parent), fctm, gs.holes.clone(), depth + 1);
}
/// Images drawn from tiling patterns, soft-mask groups and Type3 glyph procedures can't be
/// placed or cleared by this check: if `res` reaches any, every region is unverifiable.
fn hidden_images(&mut self, res: &Dict, depth: usize) {
let doc = self.doc;
let mut streams: Vec<(Option<ObjRef>, Stream)> = Vec::new();
let take = |o: &Object, streams: &mut Vec<(Option<ObjRef>, Stream)>| {
if let Object::Stream(s) = &*doc.resolve(o) {
streams.push((o.as_ref(), s.clone()));
}
};
for (_, p) in res_dict(doc, res, b"Pattern").iter() {
take(p, &mut streams);
}
for (_, g) in res_dict(doc, res, b"ExtGState").iter() {
if let Some(Object::Dict(sm)) = doc.resolve(g).as_dict().and_then(|g| g.get(b"SMask")).map(|m| (*doc.resolve(m)).clone()).as_ref()
&& let Some(group) = sm.get(b"G")
{
take(group, &mut streams);
}
}
for (_, f) in res_dict(doc, res, b"Font").iter() {
let Some(font) = doc.resolve(f).as_dict().cloned() else { continue };
if font.name(b"Subtype") != Some(b"Type3") {
continue;
}
if let Some(procs) = font.get(b"CharProcs").and_then(|c| doc.resolve(c).as_dict().cloned()) {
for (_, p) in procs.iter() {
take(p, &mut streams);
}
}
}
for (id, s) in streams {
if let Some(r) = id
&& !self.searched.insert(r)
{
continue;
}
match self.has_image(&s, res, depth + 1) {
Ok(false) => {}
Ok(true) => self.unverifiable("an image inside a pattern, soft mask or Type3 glyph can't be placed or cleared"),
Err(()) => self.unverifiable("a pattern, soft mask or Type3 glyph can't be read in full"),
}
}
}
/// Does the content of `s` (forms it draws included) draw an image?
fn has_image(&mut self, s: &Stream, parent: &Dict, depth: usize) -> Result<bool, ()> {
if depth > MAX_DEPTH {
return Err(());
}
let data = self.ctx.decode(s)?;
let own = s.dict.get(b"Resources").and_then(|r| self.doc.resolve(r).as_dict().cloned());
let res = own.as_ref().unwrap_or(parent);
let xobjects = res_dict(self.doc, res, b"XObject");
for op in parse(&data).ops {
match op.op.as_slice() {
b"BI" => return Ok(true),
b"Do" => {
let Some(entry) = op.name(0).and_then(|n| xobjects.get(n)) else { continue };
let Object::Stream(x) = &*self.doc.resolve(entry) else { continue };
match x.dict.name(b"Subtype") {
Some(b"Image") => return Ok(true),
Some(b"Form") => {
if let Some(r) = entry.as_ref()
&& !self.searched.insert(r)
{
continue;
}
if self.has_image(x, res, depth + 1)? {
return Ok(true);
}
}
_ => {}
}
}
_ => {}
}
}
Ok(false)
}
}
/// Extend the current path (`sub`: its subpaths in device space; `clip`: the clip operator it
/// carries) by one construction operator.
fn extend_path(op: &pdfcraft_content::Op, ctm: &Matrix, sub: &mut Vec<Vec<(f64, f64)>>, clip: &mut Option<bool>) {
let o = op.op.as_slice();
let nums: Vec<f64> = op.operands.iter().filter_map(Object::as_f64).collect();
let at = |x: f64, y: f64| ctm.apply(x, y);
match o {
b"m" => {
if let [x, y] = nums[..] {
sub.push(vec![at(x, y)]);
}
}
b"l" | b"c" | b"v" | b"y" => {
if let Some(cur) = sub.last_mut() {
cur.extend(nums.as_chunks::<2>().0.iter().map(|p| at(p[0], p[1])));
}
}
b"re" => {
if let [x, y, w, h] = nums[..] {
sub.push(vec![at(x, y), at(x + w, y), at(x + w, y + h), at(x, y + h)]);
}
}
b"W" => *clip = Some(false),
b"W*" => *clip = Some(true),
_ => {}
}
}
/// The axis-aligned rectangle a four-corner subpath draws, in device space: `None` for anything
/// else (open paths, more or fewer corners, skewed or rotated boxes, non-finite coordinates).
fn axis_aligned_rect(pts: &[(f64, f64)]) -> Option<[f64; 4]> {
// `re` leaves its rectangle open: the four corners are the whole subpath. A rectangle drawn
// by hand closes by repeating the first point at the end. Both spell the same four corners.
let pts = match pts {
[a, _, _, _, e] if a == e => &pts[..4],
_ => pts,
};
let [p0, p1, p2, p3] = pts else { return None };
let finite = |p: &(f64, f64)| p.0.is_finite() && p.1.is_finite();
if !finite(p0) || !finite(p1) || !finite(p2) || !finite(p3) {
return None;
}
// The edges alternate between horizontal and vertical, whichever corner they start from and
// whichever way the rectangle is wound.
let flat = |a: &(f64, f64), b: &(f64, f64)| a.1 == b.1 && a.0 != b.0;
let steep = |a: &(f64, f64), b: &(f64, f64)| a.0 == b.0 && a.1 != b.1;
let even = flat(p0, p1) && steep(p1, p2) && flat(p2, p3) && steep(p3, p0);
let odd = steep(p0, p1) && flat(p1, p2) && steep(p2, p3) && flat(p3, p0);
if !even && !odd {
return None;
}
let (x0, x1) = (p0.0.min(p2.0), p0.0.max(p2.0));
let (y0, y1) = (p0.1.min(p2.1), p0.1.max(p2.1));
(x0 < x1 && y0 < y1).then_some([x0, y0, x1, y1])
}
/// Are the samples of image `s` whose centres fall in `region` all clear (zero; for image masks,
/// the sample that does not paint)? `Err` when the image can't be inspected.
#[allow(clippy::result_unit_err)]
pub(super) fn image_cleared(doc: &Document, ctx: &Ctx, s: &Stream, ctm: &Matrix, region: [f64; 4]) -> Result<bool, ()> {
if image_codec(&s.dict) {
// Still encoded pixels: nothing was cleared.
return Ok(false);
}
let (w, h) = (s.dict.int(b"Width").ok_or(())?, s.dict.int(b"Height").ok_or(())?);
if w <= 0 || h <= 0 || (w as u64).saturating_mul(h as u64) > MAX_PIXELS {
return Err(());
}
let (w, h) = (usize::try_from(w).map_err(|_| ())?, usize::try_from(h).map_err(|_| ())?);
let mask = matches!(s.dict.get(b"ImageMask"), Some(Object::Bool(true)));
let (ncomp, bpc) = if mask {
(1, 1)
} else {
let cs = doc.resolve(s.dict.get(b"ColorSpace").ok_or(())?);
let name = match &*cs {
Object::Name(n) => n.clone(),
Object::Array(a) => a.first().and_then(Object::as_name).ok_or(())?.to_vec(),
_ => return Err(()),
};
let n = match name.as_slice() {
b"DeviceGray" | b"G" | b"CalGray" | b"Indexed" | b"I" | b"Separation" => 1,
b"DeviceRGB" | b"RGB" | b"CalRGB" | b"Lab" => 3,
b"DeviceCMYK" | b"CMYK" => 4,
// One component per colorant name.
b"DeviceN" => doc.resolve(cs.as_array().and_then(|a| a.get(1)).ok_or(())?).as_array().ok_or(())?.len(),
b"ICCBased" => match &*doc.resolve(cs.as_array().and_then(|a| a.get(1)).ok_or(())?) {
Object::Stream(icc) => usize::try_from(icc.dict.int(b"N").ok_or(())?).map_err(|_| ())?,
_ => return Err(()),
},
_ => return Err(()),
};
(n, usize::try_from(s.dict.int(b"BitsPerComponent").ok_or(())?).map_err(|_| ())?)
};
if !matches!(bpc, 1 | 2 | 4 | 8 | 16) || ncomp == 0 || ncomp > 32 {
return Err(());
}
let data = ctx.decode(s)?;
let bits = ncomp.checked_mul(bpc).ok_or(())?;
let row = w.checked_mul(bits).ok_or(())?.div_ceil(8);
if data.len() < row.checked_mul(h).ok_or(())? {
return Err(());
}
let inverted = s.dict.get(b"Decode").is_some_and(|d| {
let d = doc.resolve(d);
d.as_array().and_then(|a| a.first()).is_some_and(|first| doc.resolve(first).as_f64() == Some(1.0))
});
let want = u8::from(mask && !inverted);
for j in 0..h {
let v = 1.0 - (j as f64 + 0.5) / h as f64;
// Rows that don't reach the region hold nothing to check.
let ((ax, ay), (bx, by)) = (ctm.apply(0.0, v), ctm.apply(1.0, v));
if !overlaps(region, [ax.min(bx), ay.min(by) - 0.01, ax.max(bx), ay.max(by) + 0.01], 0.0) {
continue;
}
for i in 0..w {
let (x, y) = ctm.apply((i as f64 + 0.5) / w as f64, v);
if x < region[0] || x > region[2] || y < region[1] || y > region[3] {
continue;
}
let start = j.checked_mul(row).and_then(|v| v.checked_mul(8)).and_then(|v| v.checked_add(i.checked_mul(bits)?)).ok_or(())?;
for bit in start..start.checked_add(bits).ok_or(())? {
let byte = data.get(bit / 8).ok_or(())?;
if (byte >> (7 - bit % 8)) & 1 != want {
return Ok(false);
}
}
}
}
Ok(true)
}
crates/redact/src/verify/limits.rs
+42
View File
@@ -0,0 +1,42 @@
//! Bounds of the redaction proof (`verify`). Hitting one is a failure the proof reports, never
//! a quiet skip: a proof that cannot sweep everything it must is not a passing proof.
/// Most decoded bytes of one stream the proof reads.
pub(crate) const MAX_STREAM: usize = 64 << 20;
/// Most decoded bytes the whole proof reads, over every stream and every pass.
pub(crate) const MAX_TOTAL_DECODED: u64 = 2 << 30;
/// Most units of linear search work (input, patterns and reported matches) in one proof.
pub(crate) const MAX_WORK: u64 = 4 << 30;
/// Nested attachment levels that are swept (deeper ones are reported as unswept).
pub(crate) const MAX_NESTING: usize = 4;
/// Nesting bound of object walks (deeper is reported, never skipped silently).
pub(crate) const MAX_WALK: usize = 64;
/// Deepest graphics-state stack the paint check follows (saved states carry their clip holes
/// with them, so an unbounded stack is unbounded memory). Deeper is unverifiable.
pub(crate) const MAX_GS_STACK: usize = 4096;
/// Most clip holes one walk may cut (each is kept until the state that made it is restored).
/// More is unverifiable.
pub(crate) const MAX_HOLES: usize = 4096;
/// Most subpaths of one even-odd clip the hole rule examines (it compares them pairwise).
/// More, and no hole is cut: what the clip lets through is reported as paint.
pub(crate) const MAX_CLIP_SUBPATHS: usize = 64;
/// Most decoded bytes of an embedded CMap.
pub(crate) const MAX_CMAP: usize = 4 << 20;
/// Most CMap range steps one font may cost (a range that never grows the map still costs).
pub(crate) const MAX_CMAP_WORK: usize = 4 << 20;
/// Most entries one ToUnicode CMap may define.
pub(crate) const MAX_CMAP_ENTRIES: usize = 1 << 20;
/// Needles shorter than this (in characters) are too common to look for in raw bytes.
pub(crate) const MIN_NEEDLE: usize = 3;
/// Smallest needle looked for in the raw file and in decoded streams. Those hold object syntax,
/// numbers and binary data, where three characters match by chance (`/Length 612`, a
/// `/MediaBox`, pixels); shorter needles are still swept on page text, strings, names, Info, XMP
/// and attachment text.
pub(crate) const MIN_STRUCTURED_NEEDLE: usize = 6;
/// Deepest nesting of forms, patterns and appearance streams the proof's independent walks
/// follow: the bound the interpreter itself walks forms at, so the proof reaches everything a
/// redaction could have removed and the two cannot drift apart.
pub(crate) const MAX_DEPTH: usize = crate::interp::MAX_DEPTH;
/// Most pixels an image may have for the proof to inspect it (larger ones are removed by
/// [`apply`](crate::apply), and the proof's pixel look has the same ceiling: `crate::image`).
pub(crate) const MAX_PIXELS: u64 = crate::image::MAX_PIXELS;
crates/redact/src/verify/mod.rs
+389
View File
@@ -0,0 +1,389 @@
//! An independent proof that redaction removed what it claimed to remove.
//!
//! [`apply`](crate::apply) cuts glyphs out with its own interpreter, so asking that interpreter
//! whether anything is left would only repeat its own mistakes. The proof therefore works from
//! the other end, and does not trust the interpreter's font metrics or hit test:
//!
//! 1. **Before** applying, [`Snapshot::capture`] records, per redaction region, the glyph runs the
//! interpreter is about to remove (their character codes and the text the font gives them), a
//! digest of each page's content, and the text of the whole document as a separate extractor
//! sees it. The strings the sweep looks for therefore *start* from what the interpreter
//! removed; a glyph it wrongly kept is caught by the geometric check below, not by a needle.
//! 2. **After** applying, [`Snapshot::prove`] serializes the document and sweeps the *output*
//! for those strings on every surface a reader could recover them from: the raw file, every
//! string object, every decoded stream (orphans included), the text of every page (read by a
//! minimal extractor of its own, not the interpreter's font metrics or hit test), the Info
//! dictionary, the XMP packet, embedded files (PDF attachments are swept recursively) and the
//! revision structure. Each string is looked for in every encoding a PDF can hold it in.
//! 3. It places every glyph still shown on the redacted pages with its own text-state tracking
//! and width model (the font's `/Widths`, else a flat guess) and fails a region when the
//! centre of a surviving glyph lies inside it, whatever the interpreter decided.
//! 4. It also re-checks the non-text content under each region — on the page and in the
//! appearance streams of the annotations left on it: no image still carries pixels there, and
//! no path or shading paints there unclipped.
//!
//! The proof fails closed: a survivor, but also anything it could not sweep (page content, a form
//! or an appearance stream, XMP, or an attachment that won't decode in full or can't be opened; a
//! nesting or work limit hit; an appearance with content that can't be placed on its page), makes
//! it fail. The one exception is binary data that redaction
//! never edits: a font program, colour profile, `/Indexed` lookup table, function sample table or
//! Flate/LZW image that is damaged or too long to decode is searched as stored bytes and reported
//! as `raw_only`, not as unswept. A proof that cannot fail is worthless.
//!
//! Work is bounded: every stream is decoded strictly under `limits::MAX_STREAM`, all decodes together
//! under `limits::MAX_TOTAL_DECODED`, and all searching under `limits::MAX_WORK`. A document that needs more
//! is reported as unswept, not skipped (binary data as above excepted).
//!
//! The result is a [`Proof`]: a per-region manifest (page, region, glyph counts, SHA-256 of
//! the page content after) and a verdict per surface. It holds counts and digests,
//! never the redacted text, unless [`ProofOptions::include_plaintext`] asks for it.
//!
//! Limits, stated plainly: text that exists only as pixels of an image is not text; a string
//! that also appears in content that was *not* redacted is counted, over the whole document, on
//! the page text, the raw file and the decoded streams (those hold the kept copy too) and fails
//! on a surplus; names are swept against the matches the input itself had, since a redacted word
//! that is also a `/Subtype` or a `/BaseFont` is structure, not a leftover copy, and a surplus of
//! matches still fails; every other surface is swept with no allowance. A word of six characters or more
//! that also spells a structural name (`/Filter`, `/MediaBox`) can still be reported by the raw-file and
//! decoded-stream surfaces, which search stored bytes and cannot tell a name from a copy: that is
//! over-reporting, never a pass. The raw file and the decoded
//! streams are searched for strings of at least six characters only (shorter ones match object
//! syntax and numbers by chance), skipping the trailer's file identifier and the
//! pixels of an image codec (DCT, JPX, JBIG2, CCITT), which are searched as stored bytes only; shorter strings are still found on the page text,
//! in strings, names, Info, XMP and attachment text. A region without extractable text is reported as such, and as
//! unverifiable when the page draws text this extractor cannot decode. The geometric check
//! places glyphs with the font's `/Widths` only: for a font without them the widths are a guess,
//! so a neighbouring glyph may be reported when the guess puts its centre inside a region.
//! `runs_removed` and `codes_removed` are exact counts of what was cut: a proof that is shared
//! discloses how much text sat under each region (never what it said).
use std::fmt::Write as _;
use pdfcraft_cos::ObjRef;
mod extract;
mod geometry;
pub(crate) mod limits;
#[cfg(test)]
mod search_tests;
mod snapshot;
mod sweep;
mod util;
pub use snapshot::Snapshot;
/// What the proof established for one region.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Status {
/// What was removed was swept for on every surface and is gone.
Verified,
/// The region held no extractable text: nothing was removed, and that is recorded as such.
VerifiedNoTextInRegion,
/// Neither "text was removed" nor "there was none" can be checked (the page draws text this
/// extractor can't decode, or an image under the region can't be inspected). The proof can't
/// pass while any region is unverifiable.
Unverifiable,
/// A survivor was found, or the mechanical evidence is missing.
Failed,
}
/// The places a redacted string is looked for.
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub enum Surface {
RawBytes,
ObjectStrings,
DecodedStreams,
ExtractedText,
Info,
Xmp,
EmbeddedFiles,
Revisions,
}
const SURFACES: [Surface; 8] = [
Surface::RawBytes,
Surface::ObjectStrings,
Surface::DecodedStreams,
Surface::ExtractedText,
Surface::Info,
Surface::Xmp,
Surface::EmbeddedFiles,
Surface::Revisions,
];
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Verdict {
/// Swept, nothing found.
Clean,
/// Swept, a redacted string (or an earlier revision) is still there.
Survivor,
/// Could not be swept; the proof fails.
Unswept,
/// There is nothing of this kind in the document.
Absent,
}
#[derive(Clone, Debug, PartialEq)]
pub struct SurfaceReport {
pub surface: Surface,
pub verdict: Verdict,
/// Streams, strings, pages or files looked at.
pub scanned: usize,
/// Streams scanned as stored bytes only: image-codec pixels (not text), and font, colour or
/// image streams that could not be decoded in full.
pub raw_only: usize,
pub survivors: usize,
/// Where each survivor sits ("object 12", "page 1", "attachment #1 > object 4"); never the text.
pub locations: Vec<String>,
/// Why the surface could not be swept.
pub problems: Vec<String>,
}
/// The manifest entry of one redaction region.
#[derive(Clone, Debug, PartialEq)]
pub struct Excision {
/// 0-based page.
pub page: usize,
/// The region in user space.
pub region: [f64; 4],
/// The Redact annotation it came from.
pub mark: ObjRef,
pub status: Status,
pub detail: String,
/// Glyph runs (consecutive glyphs of one string) removed under the region, and their
/// character codes.
pub runs_removed: usize,
pub codes_removed: usize,
/// SHA-256 of the page's decoded content streams, joined, after redaction. There is no digest
/// of the content before: a proof is meant to be shared, and a hash of the original page
/// would let a reader confirm a guess at what it said.
pub page_sha256_after: Option<String>,
/// Glyph-carrying text operators on the page (forms included) before and after. A `TJ` of
/// numbers only (the gap left for a removed string) carries no glyphs and does not count.
pub text_ops_before: Option<usize>,
pub text_ops_after: Option<usize>,
/// The removed text, only when [`ProofOptions::include_plaintext`] is set.
pub removed_text: Option<Vec<String>>,
}
#[derive(Clone, Debug, PartialEq)]
pub struct Proof {
pub entries: Vec<Excision>,
pub surfaces: Vec<SurfaceReport>,
/// Located, plain reasons for every failure (never the redacted text).
pub failures: Vec<String>,
}
impl Proof {
/// Survivors found across all surfaces.
pub fn survivors(&self) -> usize {
self.surfaces.iter().map(|s| s.survivors).sum()
}
/// Surfaces that could not be swept.
pub fn unswept(&self) -> usize {
self.surfaces.iter().filter(|s| s.verdict == Verdict::Unswept || !s.problems.is_empty()).count()
}
pub fn unverifiable(&self) -> usize {
self.entries.iter().filter(|e| e.status == Status::Unverifiable).count()
}
/// Pass only when nothing failed, nothing is unverifiable and no surface went unswept.
pub fn passed(&self) -> bool {
self.failures.is_empty()
&& self.survivors() == 0
&& self.unswept() == 0
&& self.entries.iter().all(|e| matches!(e.status, Status::Verified | Status::VerifiedNoTextInRegion))
}
/// A human-readable report: the verdict, the manifest and the surfaces. It names places and
/// counts, and the removed text only if it was asked for.
pub fn to_text(&self) -> String {
let mut s = String::new();
let _ = writeln!(s, "Redaction proof: {}", if self.passed() { "PASS" } else { "FAIL" });
for f in &self.failures {
let _ = writeln!(s, " failure: {f}");
}
for (i, e) in self.entries.iter().enumerate() {
let _ = writeln!(
s,
" region {} page {} {:?} {:?}: {} run(s), {} code(s) removed; text ops {:?} -> {:?}; page sha256 after {}{}",
i + 1,
e.page + 1,
e.region,
e.status,
e.runs_removed,
e.codes_removed,
e.text_ops_before,
e.text_ops_after,
e.page_sha256_after.as_deref().unwrap_or("-"),
if e.detail.is_empty() { String::new() } else { format!(" ({})", e.detail) },
);
}
for r in &self.surfaces {
let _ = writeln!(
s,
" {:?}: {:?} ({} scanned, {} raw only, {} survivor(s)){}",
r.surface,
r.verdict,
r.scanned,
r.raw_only,
r.survivors,
if r.locations.is_empty() { String::new() } else { format!(" at {}", r.locations.join(", ")) },
);
for p in &r.problems {
let _ = writeln!(s, " unswept: {p}");
}
}
let _ = writeln!(s, " Evidence, not a guarantee: text that exists only as pixels of an image is not text, and is not searched.");
s
}
}
/// What a proof may contain.
#[derive(Clone, Debug, Default)]
pub struct ProofOptions {
/// Include the removed text in the manifest. Off by default: a proof is meant to be shared,
/// and must not carry what it proves was removed.
pub include_plaintext: bool,
}
/// Handles for the tests to reach the byte sweep without a whole document.
#[cfg(test)]
pub(crate) mod tests_support {
use pdfcraft_content::parse;
use pdfcraft_cos::{Dict, Object, Stream};
use super::extract::{carries_glyphs, parse_cmap};
use super::limits::{MAX_STREAM, MAX_TOTAL_DECODED, MAX_WORK, MIN_STRUCTURED_NEEDLE};
use super::sweep::{Needle, add_needle, bytes_counts, bytes_hits, walk_strings};
use super::util::Ctx;
pub(crate) use super::util::survivor_encodings;
/// The proof's whole search-work budget (for tests that pre-spend part of it).
pub(crate) fn work_budget() -> u64 {
MAX_WORK
}
/// The byte sweep of `hay` for a bulk of `count` distinct needles, on a context already
/// `spent` units into its work budget: `Some` (with the needles that occur) while the sweep
/// stays under the budget, `None` once it ran out.
pub(crate) fn bulk_counts(hay: &[u8], text: &str, count: usize, spent: u64) -> Option<usize> {
let mut needles: Vec<Needle> = Vec::new();
for i in 0..count {
add_needle(&mut needles, false, format!("{text}{i:04}"), Some(i));
}
let ctx = Ctx::new(Default::default());
ctx.work.set(spent);
bytes_counts(&needles, hay, MIN_STRUCTURED_NEEDLE, &ctx).map(|hits| hits.len())
}
/// The glyph-carrying text-showing operators of a decoded content stream.
pub fn count_text_operators(decoded: &[u8]) -> usize {
parse(decoded).ops.iter().filter(|op| carries_glyphs(op)).count()
}
/// Does the byte sweep for `text` find it in `hay`?
pub fn found(text: &str, hay: &[u8]) -> bool {
let mut needles: Vec<Needle> = Vec::new();
add_needle(&mut needles, false, text.to_string(), Some(0));
bytes_hits(&needles, hay, 3, &Ctx::new(Default::default())).is_some_and(|h| !h.is_empty())
}
/// Does this CMap parse completely (not cut short by a bound)?
pub fn cmap_is_complete(data: &[u8]) -> bool {
parse_cmap(data).is_some()
}
/// Is an object nested `depth` levels deep walked in full?
pub fn walks_in_full(depth: usize) -> bool {
let mut o = Object::Null;
for _ in 0..depth {
o = Object::Array(vec![o]);
}
walk_strings(&o, 0, &mut |_| {})
}
/// Does decoding fail once the cumulative decode budget is used up (and not before)?
pub fn decode_budget_runs_out() -> (bool, bool) {
let ctx = Ctx::new(Default::default());
let s = Stream::from_raw(Dict::new(), vec![0u8; 100]);
let before = ctx.decode(&s).is_ok();
let spent = Ctx::with_decode_left(50);
(before, spent.decode(&s).is_err())
}
/// Does a decode that fails past the one-stream cap pay for the attempt, so the same bomb
/// cannot be decoded (and fail) again and again for free?
/// `(refused, charged, then refused for want of allowance)`.
pub fn failed_decode_charges_the_budget() -> (bool, bool, bool) {
let big = vec![0u8; MAX_STREAM + (1 << 20)];
let bomb = Stream::flate(Dict::new(), &big);
drop(big);
let small = Stream::from_raw(Dict::new(), vec![0u8; 100]);
let ctx = Ctx::with_decode_left(u64::try_from(MAX_STREAM + 50).unwrap_or(u64::MAX));
let refused = ctx.decode(&bomb).is_err();
let charged = ctx.decoded.get() == MAX_TOTAL_DECODED - 50;
let gone = ctx.decode(&small).is_err();
(refused, charged, gone)
}
/// Does the byte search report `None` once the work budget is used up (and not before)?
pub fn search_budget_runs_out() -> (bool, bool) {
let mut needles: Vec<Needle> = Vec::new();
add_needle(&mut needles, false, "secret".to_string(), Some(0));
let ctx = Ctx::new(Default::default());
let before = bytes_hits(&needles, b"some bytes", 3, &ctx).is_some();
ctx.work.set(MAX_WORK);
(before, bytes_hits(&needles, b"some bytes", 3, &ctx).is_none())
}
/// Is a survivor counted on an earlier page still reported when the work budget runs out on
/// a later one? `(survivor reported, page left unswept)`.
pub fn page_sweep_survivors_survive_the_limit() -> (bool, bool) {
use super::Surface;
use super::sweep::{Acc, sweep_pages};
let objs = vec![
b"<< /Type /Catalog /Pages 2 0 R >>".to_vec(),
b"<< /Type /Pages /Kids [3 0 R 4 0 R] /Count 2 >>".to_vec(),
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 5 0 R /Resources << /Font << /F1 6 0 R >> >> >>".to_vec(),
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 7 0 R /Resources << /Font << /F1 6 0 R >> >> >>".to_vec(),
crate::tests_verify::stream("", b"BT /F1 10 Tf 10 100 Td (SECRETVALUE) Tj ET"),
b"<< /Type /Font /Subtype /TrueType /BaseFont /Arial >>".to_vec(),
crate::tests_verify::stream("", &format!("BT /F1 10 Tf 10 100 Td ({}) Tj ET", "x".repeat(700_000)).into_bytes()),
];
let doc = pdfcraft_cos::Document::open(std::sync::Arc::new(crate::tests_verify::pdf_bytes(objs, ""))).unwrap();
let mut needles = Vec::new();
add_needle(&mut needles, false, "SECRETVALUE".to_string(), Some(0));
let mut acc = Acc::new(Ctx::new(Default::default()));
// Page 1 (the survivor's) fits in what is left; page 2 does not.
acc.ctx.work.set(MAX_WORK - 100_000);
sweep_pages(&needles, &doc, 0, "", &mut acc);
let text = acc.reps.get(&Surface::ExtractedText).unwrap();
(text.survivors > 0, !text.problems.is_empty())
}
/// Does a font's `/W` parsing answer to the shared work budget, and not only to its own
/// counter (a form invoked again rebuilds its fonts, and would pay for the same array
/// again)? `(read in full, refused on a spent budget)`.
pub fn font_widths_charge_shares_the_budget() -> (bool, bool) {
use super::extract::extract_page;
let font = b"<< /Type /Font /Subtype /Type0 /BaseFont /X /Encoding /Identity-H /DescendantFonts [<< /Type /Font /Subtype /CIDFontType2 /BaseFont /X /DW 1000 /W [1 65535 250] >>] >>".to_vec();
let objs = vec![
b"<< /Type /Catalog /Pages 2 0 R >>".to_vec(),
b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>".to_vec(),
b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 300] /Contents 4 0 R /Resources << /Font << /F2 5 0 R >> >> >>".to_vec(),
crate::tests_verify::stream("", b"BT /F2 10 Tf 10 100 Td <00410042> Tj ET"),
font,
];
let bytes = std::sync::Arc::new(crate::tests_verify::pdf_bytes(objs, ""));
let probe = |spent: u64| -> bool {
let doc = pdfcraft_cos::Document::open(bytes.clone()).unwrap();
let ctx = Ctx::new(Default::default());
ctx.work.set(spent);
let page = &pdfcraft_model::pages(&doc)[0];
extract_page(&doc, &page.dict, &ctx, &[]).is_ok()
};
(probe(0), !probe(MAX_WORK - 10_000))
}
}
crates/redact/src/verify/search_tests.rs
+34
View File
@@ -0,0 +1,34 @@
use super::{
sweep::{add_needle, bytes_counts, bytes_hits},
util::Ctx,
};
use std::collections::HashSet;
#[test]
fn a_thousand_needles_share_one_multimegabyte_scan() {
let mut needles = Vec::new();
for i in 0..1000 {
add_needle(&mut needles, false, format!("SECRET_{i:04}"), None);
}
let hay = vec![b'z'; 8 << 20];
assert_eq!(bytes_hits(&needles, &hay, 6, &Ctx::new(HashSet::new())), Some(Vec::new()));
assert_eq!(bytes_counts(&needles, &hay, 6, &Ctx::new(HashSet::new())), Some(Vec::new()));
}
#[test]
fn overlapping_encodings_count_one_occurrence() {
let mut needles = Vec::new();
add_needle(&mut needles, false, "SECRET42".into(), None);
let text: Vec<u8> = "SECRET42".encode_utf16().flat_map(u16::to_be_bytes).collect();
let mut hay = vec![0xfe, 0xff];
hay.extend(text);
assert_eq!(bytes_counts(&needles, &hay, 6, &Ctx::new(HashSet::new())), Some(vec![(0, 1)]));
}
#[test]
fn match_count_exhaustion_is_incomplete_not_a_truncated_pass() {
let mut needles = Vec::new();
add_needle(&mut needles, true, "SECRET42".into(), None);
let hay = b"SECRET42 ".repeat((1 << 16) + 1);
assert!(bytes_counts(&needles, &hay, 6, &Ctx::new(HashSet::new())).is_none());
}
crates/redact/src/verify/snapshot.rs
+532
View File
@@ -0,0 +1,532 @@
//! The snapshot taken before applying (what is about to go) and the proof run on the output.
use std::collections::{BTreeMap, HashMap, HashSet};
use std::sync::Arc;
use pdfcraft_content::Matrix;
use pdfcraft_cos::{Dict, Document, ObjRef, Object, SaveOptions, write_full};
use super::extract::extract_page;
use super::geometry::{Finding, check_geometry};
use super::sweep::{Acc, Needle, add_needle, bytes_hits, sweep_doc, walk_strings};
use super::util::{Ctx, count, latin1, sha256, squash, words};
use super::{Excision, Proof, ProofOptions, SURFACES, Status, Surface};
use crate::interp::{Mode, Removed, Scope, process};
use crate::verify::limits::MIN_NEEDLE;
use crate::{Mark, RedactError, Report, page_streams};
pub(super) struct RegionSnap {
pub(super) page: usize,
pub(super) region: [f64; 4],
pub(super) mark: ObjRef,
pub(super) runs: Vec<(Vec<u8>, String)>,
/// The runs of one text operator joined, where it split them: `[(sec) (ret)] TJ` reads as
/// `secret`, which a copy elsewhere in the file holds whole.
pub(super) joined: Vec<(Vec<u8>, String)>,
/// The character codes the runs hold (a composite font's codes are several bytes).
pub(super) codes: usize,
}
pub(super) struct PageSnap {
pub(super) sha: Option<String>,
pub(super) text_ops: Option<usize>,
pub(super) undecodable_ops: usize,
}
/// What redaction is about to remove, recorded before it does.
pub struct Snapshot {
regions: Vec<RegionSnap>,
pages: BTreeMap<usize, PageSnap>,
needles: Vec<Needle>,
/// Things that made the "before" side unreadable (reported on the text surface).
problems: Vec<String>,
/// The document's page text before redaction (decoded view, code view), for the allowance
/// of needles added later.
src: (String, String),
/// The names of the document (structure redaction never edits), for the name allowance of
/// needles added later.
names: BTreeMap<Vec<u8>, usize>,
/// Digests of the decoded content of the streams this run created to draw the redaction
/// boxes and labels (see [`Snapshot::set_overlay_streams`]).
overlay: HashSet<String>,
}
impl std::fmt::Debug for Snapshot {
// The snapshot holds what is about to be redacted: never print it.
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Snapshot").field("regions", &self.regions.len()).field("needles", &self.needles.len()).finish()
}
}
impl Snapshot {
/// Record what redacting the marks on `pages` (0-based; `None` = all) will remove. Call it
/// before [`apply`](crate::apply); it changes nothing.
pub fn capture(doc: &Document, pages: Option<&[usize]>) -> Result<Snapshot, RedactError> {
let all = crate::marks(doc);
let chosen: Vec<&Mark> = all.iter().filter(|m| pages.is_none_or(|p| p.contains(&m.page))).collect();
if chosen.is_empty() {
return Err(RedactError::NothingToApply);
}
Snapshot::capture_marks(doc, &chosen)
}
pub(crate) fn capture_marks(doc: &Document, chosen: &[&Mark]) -> Result<Snapshot, RedactError> {
let mut by_page: Vec<usize> = chosen.iter().map(|m| m.page).collect();
by_page.sort_unstable();
by_page.dedup();
let page_list = pdfcraft_model::pages(doc);
let ctx = Ctx::new(HashSet::new());
let mut regions: Vec<RegionSnap> = Vec::new();
let mut pages = BTreeMap::new();
let mut images = pdfcraft_cos::ImageBudget::new(doc);
// What was cut, each copy once, however many overlapping marks it lay under: the allowance
// for what stays is what the source holds less this.
let (mut cut_text, mut cut_raw) = (String::new(), String::new());
for &pi in &by_page {
let page = page_list.get(pi).ok_or(RedactError::Unreadable(pi + 1))?;
let page_marks: Vec<&&Mark> = chosen.iter().filter(|m| m.page == pi).collect();
let rects: Vec<[f64; 4]> = page_marks.iter().flat_map(|m| m.rects.iter().copied()).collect();
let (_, data) = page_streams(doc, &page.dict, pi)?;
let resources = page.dict.get(b"Resources").and_then(|r| doc.resolve(r).as_dict().cloned()).unwrap_or_default();
// The interpreter's own removal path, on a scratch copy: what it takes out is what
// has to be gone.
let mut scratch = doc.clone();
let mut sink = Report::default();
let removed: Vec<Removed> = {
let mut scope = Scope::new(&rects, Mode::Apply, &mut sink, pdfcraft_cos::DecodeBudget::page(), &mut images);
scope.removed = Some(Vec::new());
process(&mut scratch, &mut scope, &data, &resources, Matrix::IDENTITY);
scope.removed.take().unwrap_or_default()
};
let lines = joined(&removed);
let mut seen = HashSet::new();
for line in lines.iter().filter(|l| seen.insert((l.op, l.at, l.raw.clone()))) {
cut_text.push('\u{1}');
cut_text.push_str(&squash(&line.text));
cut_raw.push('\u{1}');
cut_raw.push_str(&squash(&latin1(&line.raw)));
}
let before = extract_page(doc, &page.dict, &ctx, &[]);
pages.insert(
pi,
PageSnap {
sha: Some(sha256(&data)),
text_ops: before.as_ref().ok().map(|e| e.glyph_ops),
undecodable_ops: before.as_ref().map_or(0, |e| e.undecodable_ops),
},
);
let mut k = 0;
for m in &page_marks {
for r in &m.rects {
let hits: Vec<&Removed> = removed.iter().filter(|x| x.rect == k).collect();
let runs = hits.iter().map(|x| (x.raw.clone(), x.text.clone())).collect();
let codes = hits.iter().map(|x| x.codes).sum();
let joined = lines.iter().filter(|l| l.rect == k && l.pieces > 1).map(|l| (l.raw.clone(), l.text.clone())).collect();
regions.push(RegionSnap { page: pi, region: *r, mark: m.obj, runs, joined, codes });
k += 1;
}
}
}
// The document's text as the independent extractor reads it, for the allowance of
// strings that legitimately remain elsewhere.
let (mut src_text, mut src_raw) = (String::new(), String::new());
let mut problems = Vec::new();
for (i, p) in page_list.iter().enumerate() {
match extract_page(doc, &p.dict, &ctx, &[]) {
Ok(e) => {
src_text.push_str(&e.text);
src_raw.push_str(&e.raw);
}
Err(()) => {
problems.push(format!("page {} of the source can't be read, so strings left on it can't be told from removed ones", i + 1))
}
}
}
let mut needles: Vec<Needle> = Vec::new();
for (ri, r) in regions.iter().enumerate() {
for (raw, text) in r.runs.iter().chain(&r.joined) {
let raw_s = latin1(raw);
for (raw_view, whole) in [(false, text.trim().to_string()), (true, raw_s)] {
let mut shown: Vec<String> = Vec::new();
if !squash(&whole).is_empty() {
shown.push(whole.clone());
shown.extend(words(&whole).into_iter().filter(|w| *w != whole).map(str::to_string));
}
for s in shown {
add_needle(&mut needles, raw_view, s, Some(ri));
}
}
}
}
for (src, cut, raw) in [(&src_text, &cut_text, false), (&src_raw, &cut_raw, true)] {
match (super::sweep::key_counts(&needles, src, raw, &ctx), super::sweep::key_counts(&needles, cut, raw, &ctx)) {
(Some(source), Some(removed)) => {
let removed: BTreeMap<usize, usize> = removed.into_iter().collect();
for (i, total) in source {
if let Some(n) = needles.get_mut(i) {
n.allowance = total.saturating_sub(removed.get(&i).copied().unwrap_or(0));
}
}
}
_ => problems.push("the source text allowances could not be counted in full".into()),
}
}
// The names the removed text legitimately sits in (a redacted word that is also a
// `/Subtype`, a `/Filter`, a `/BaseFont`): names are structure redaction never edits,
// so the output may hold the input's own matches, and only a surplus is a survivor.
let names = all_names(doc);
let name_allow: Vec<Option<usize>> = needles.iter().map(|n| count_name_matches(&names, n, &ctx)).collect();
if name_allow.iter().any(Option::is_none) {
problems.push("the name allowances could not be counted in full, so names are swept with none of them allowed".into());
}
for (n, allow) in needles.iter_mut().zip(name_allow) {
n.name_allowance = allow.unwrap_or(0);
}
Ok(Snapshot { regions, pages, needles, problems, src: (src_text, src_raw), names, overlay: HashSet::new() })
}
/// Tell the proof which streams this run created to draw the redaction boxes and labels
/// (`created`: references into `doc`, the document after `apply`). Call it before
/// [`prove`](Snapshot::prove) / [`prove_saved`](Snapshot::prove_saved).
///
/// Those streams paint over the regions on purpose and may show label text, so the proof
/// leaves them out of the text sweep and the paint check. They are recognised by the digest
/// of their decoded content (a save renumbers objects, so references would not survive it),
/// and *only* those: a stream that merely carries a `/PCMark /Redaction` key in the input is
/// not exempt. Streams that can't be read here are an error; nothing is then exempted.
pub fn set_overlay_streams(&mut self, doc: &Document, created: &[ObjRef]) -> Result<(), RedactError> {
let ctx = Ctx::new(HashSet::new());
let mut digests = HashSet::new();
for r in created {
let obj = doc.try_get(r.num).map_err(|_| RedactError::OverlayUnreadable)?;
let Object::Stream(s) = &*obj else { continue };
let data = ctx.decode(s).map_err(|_| RedactError::OverlayUnreadable)?;
digests.insert(sha256(&[data]));
}
self.overlay = digests;
Ok(())
}
/// Also look for these strings in the output: text that left the document by another route
/// than a page's glyphs (a removed annotation's `/Contents` or `/RC`, a form field's value).
/// Each is searched whole, on every surface. A string that also legitimately remains as page
/// text is counted on the page-content surfaces only (as for the removed glyph runs).
/// Strings that are blank or shorter than the minimum needle are ignored.
pub fn add_removed_strings(&mut self, strings: &[String]) {
for s in strings {
let text = s.trim();
let key = squash(text);
if key.chars().count() < MIN_NEEDLE {
continue;
}
let known = self.needles.iter().any(|n| !n.raw_view && n.key == key);
// No owner: a survivor fails the proof through its surface; no region carries it.
add_needle(&mut self.needles, false, text.to_string(), None);
if !known {
let ctx = Ctx::new(HashSet::new());
let matches = self.needles.last().map(|n| count_name_matches(&self.names, n, &ctx));
if let Some(n) = self.needles.last_mut() {
n.allowance = count(&self.src.0, &n.key);
n.name_allowance = matches.flatten().unwrap_or(0);
}
if matches == Some(None) {
self.problems.push("the name allowances could not be counted in full, so names are swept with none of them allowed".into());
}
}
}
}
/// Serialize `doc` the way a save would and sweep the result. The output is re-opened with
/// `password`, the credentials the caller opened the document with (`None` tries the empty
/// password): a save of an encrypted document is itself encrypted, and a proof that could
/// not read its own subject would prove nothing.
pub fn prove(&self, doc: &Document, password: Option<&str>, opts: &ProofOptions) -> Proof {
match write_full(doc, &SaveOptions::default()) {
Ok(bytes) => self.prove_inner(&bytes, Some(doc), password, opts),
Err(e) => {
let mut p = self.prove_inner(&[], None, password, opts);
p.failures.push(format!("the document could not be serialized for the sweep: {e}"));
p
}
}
}
/// Sweep an already saved file (for example an incrementally saved one, whose earlier
/// revision still holds the original content). The file is opened with `password`, the
/// credentials it is read with.
pub fn prove_saved(&self, bytes: &[u8], password: Option<&str>, opts: &ProofOptions) -> Proof {
self.prove_inner(bytes, None, password, opts)
}
fn prove_inner(&self, bytes: &[u8], live: Option<&Document>, password: Option<&str>, opts: &ProofOptions) -> Proof {
let mut acc = Acc::new(Ctx::new(self.overlay.clone()));
let mut failures: Vec<String> = Vec::new();
for p in &self.problems {
acc.problem(Surface::ExtractedText, p.clone());
}
// Always the caller's credentials, never unconditionally the empty password: a
// document whose user password is not empty must be provable with that password, and
// must fail closed (an unswept output, never a false pass) without it.
let parsed = if bytes.is_empty() { None } else { Document::open_with_password(Arc::new(bytes.to_vec()), password).ok() };
match &parsed {
Some(out) => sweep_doc(&self.needles, out, bytes, 0, "", &mut acc),
None => {
let why = "the output could not be read back".to_string();
for s in SURFACES {
acc.problem(s, why.clone());
}
failures.push(why);
}
}
if live.is_some_and(|d| !d.full_save_required()) {
acc.survivor(Surface::Revisions, "the document would be saved as an incremental update, keeping the original revision".into(), None);
}
// The non-text content under each region, on the output.
let (findings, after) = match &parsed {
Some(out) => self.check_output(out, &acc.ctx),
None => Default::default(),
};
let mut entries = Vec::with_capacity(self.regions.len());
for (i, r) in self.regions.iter().enumerate() {
let (entry, failure) = self.entry(i, r, &findings, &after, &acc.failed_needles, opts);
failures.extend(failure);
entries.push(entry);
}
let surfaces = acc.finish();
for s in &surfaces {
if s.survivors > 0 {
failures.push(format!("{:?}: {} survivor(s) at {}", s.surface, s.survivors, s.locations.join(", ")));
}
for p in &s.problems {
failures.push(format!("{:?} could not be swept: {p}", s.surface));
}
}
Proof { entries, surfaces, failures }
}
/// The non-text content under each region of the output (what still paints or shows there),
/// and per page the digest and glyph-operator count of what `apply` left.
fn check_output(&self, out: &Document, ctx: &Ctx) -> (HashMap<usize, Vec<Finding>>, PagesAfter) {
let mut findings: HashMap<usize, Vec<Finding>> = HashMap::new();
let mut after: PagesAfter = BTreeMap::new();
let out_pages = pdfcraft_model::pages(out);
for &pi in self.pages.keys() {
let idx: Vec<usize> = (0..self.regions.len()).filter(|&i| self.regions.get(i).is_some_and(|r| r.page == pi)).collect();
let Some(page) = out_pages.get(pi) else {
for &i in &idx {
findings.entry(i).or_default().push(Finding::Missing);
}
continue;
};
let rects: Vec<[f64; 4]> = idx.iter().filter_map(|&i| self.regions.get(i).map(|r| r.region)).collect();
let (sha, ops, shown) = page_after(out, &page.dict, ctx, &rects);
after.insert(pi, (sha, ops));
match shown {
Some(shown) => {
for (k, n) in shown.into_iter().enumerate() {
if let (Some(&i), true) = (idx.get(k), n > 0) {
findings.entry(i).or_default().push(Finding::Text(n));
}
}
}
None => {
for &i in &idx {
findings
.entry(i)
.or_default()
.push(Finding::Unverifiable("the page text can't be placed, so what still shows under the region can't be checked"));
}
}
}
for (k, f) in check_geometry(out, &page.dict, &rects, ctx) {
if let Some(&i) = idx.get(k) {
findings.entry(i).or_default().push(f);
}
}
}
(findings, after)
}
/// The manifest entry of region `i`, and why it failed when it did.
fn entry(
&self,
i: usize,
r: &RegionSnap,
findings: &HashMap<usize, Vec<Finding>>,
after: &PagesAfter,
failed_needles: &HashSet<usize>,
opts: &ProofOptions,
) -> (Excision, Option<String>) {
let mut failure = None;
let snap = self.pages.get(&r.page);
let (sha_after, ops_after) = after.get(&r.page).cloned().unwrap_or((None, None));
let sha_before = snap.and_then(|s| s.sha.clone());
let owned: Vec<&Needle> = self.needles.iter().filter(|n| n.owners.contains(&i)).collect();
let survived = self.needles.iter().enumerate().filter(|(ni, n)| n.owners.contains(&i) && failed_needles.contains(ni)).count();
let mut detail: Vec<String> = Vec::new();
let mut status = if r.runs.is_empty() { Status::VerifiedNoTextInRegion } else { Status::Verified };
let fs = findings.get(&i).map(Vec::as_slice).unwrap_or_default();
if survived > 0 {
status = Status::Failed;
detail.push(format!("{survived} removed string(s) still found"));
}
for f in fs {
match f {
Finding::Text(n) => detail.push(format!("{n} glyph(s) still show inside the region (placed independently of the redaction)")),
Finding::Image => detail.push("an image still carries pixels under the region".into()),
Finding::Path => detail.push("a path still paints inside the region".into()),
Finding::Shading => detail.push("a shading still paints inside the region".into()),
Finding::Missing => detail.push("the page is missing from the output".into()),
Finding::Unverifiable(why) => detail.push((*why).to_string()),
}
if !matches!(f, Finding::Unverifiable(_)) {
status = Status::Failed;
} else if status != Status::Failed {
status = Status::Unverifiable;
}
}
if sha_before.is_none() || sha_after.is_none() {
status = Status::Failed;
detail.push("the page content can't be read on one side, so it can't be compared".into());
} else if !r.runs.is_empty() && sha_before == sha_after {
status = Status::Failed;
detail.push("glyphs were removed but the page content is unchanged".into());
}
if r.runs.is_empty() && snap.is_some_and(|s| s.undecodable_ops > 0) && status != Status::Failed {
status = Status::Unverifiable;
detail.push("the page draws text that can't be decoded, so \"no text in the region\" can't be checked".into());
}
let ambiguous = owned.iter().filter(|n| n.allowance > 0).count();
if ambiguous > 0 {
detail.push(format!("{ambiguous} string(s) also remain in retained content; only a surplus over those copies counts"));
}
if matches!(status, Status::Failed | Status::Unverifiable) {
failure = Some(format!("page {} region {:?}: {}", r.page + 1, r.region, detail.join("; ")));
}
let entry = Excision {
page: r.page,
region: r.region,
mark: r.mark,
status,
detail: detail.join("; "),
runs_removed: r.runs.len(),
codes_removed: r.codes,
page_sha256_after: sha_after,
text_ops_before: snap.and_then(|s| s.text_ops),
text_ops_after: ops_after,
removed_text: opts.include_plaintext.then(|| r.runs.iter().map(|(_, t)| t.clone()).collect()),
};
(entry, failure)
}
}
/// Per page: the digest of its content in the output and its glyph-operator count.
type PagesAfter = BTreeMap<usize, (Option<String>, Option<usize>)>;
/// Every name of the document (the values, wherever an object holds them), with how often
/// each occurs.
fn all_names(doc: &Document) -> BTreeMap<Vec<u8>, usize> {
let mut names: BTreeMap<Vec<u8>, usize> = BTreeMap::new();
for num in doc.object_numbers() {
if let Ok(obj) = doc.try_get(num) {
walk_strings(&obj, 0, &mut |o| {
if let Object::Name(n) = o {
let seen = names.entry(n.clone()).or_default();
*seen = seen.saturating_add(1);
}
});
}
}
names
}
/// How many of `names` carry `needle`, by the same search the sweep runs on a name. `None`
/// when the search work ran out: an unknown allowance must not read as "none allowed", so the
/// caller reports that (and sweeps names with no allowance).
fn count_name_matches(names: &BTreeMap<Vec<u8>, usize>, needle: &Needle, ctx: &Ctx) -> Option<usize> {
if needle.short() {
return Some(0);
}
let mut total = 0usize;
for (name, seen) in names {
if !bytes_hits(std::slice::from_ref(needle), name, MIN_NEEDLE, ctx)?.is_empty() {
total = total.saturating_add(*seen);
}
}
Some(total)
}
/// The page's content digest and glyph-operator count in the output, and per region the glyphs
/// still placed inside it (`None` when the page's text can't be placed). The digest covers the
/// page's content as `apply` left it, the overlay streams included.
pub(super) fn page_after(doc: &Document, page: &Dict, ctx: &Ctx, regions: &[[f64; 4]]) -> (Option<String>, Option<usize>, Option<Vec<usize>>) {
let sha = page_streams(doc, page, 0).ok().map(|(_, d)| sha256(&d));
let e = extract_page(doc, page, ctx, regions).ok();
(sha, e.as_ref().map(|e| e.glyph_ops), e.map(|e| e.shown_in))
}
/// One text operator's removed runs under one region, as a reader takes them: joined.
struct Line {
rect: usize,
op: usize,
at: (usize, usize),
raw: Vec<u8>,
text: String,
pieces: usize,
}
/// The runs of `removed` joined by text operator and region, in the order they were shown.
fn joined(removed: &[Removed]) -> Vec<Line> {
let mut lines: Vec<Line> = Vec::new();
let mut found: HashMap<(usize, usize), usize> = HashMap::new();
for r in removed {
let line = found.get(&(r.op, r.rect)).copied().and_then(|i| lines.get_mut(i));
match line {
Some(l) => {
l.raw.extend_from_slice(&r.raw);
l.text.push_str(&r.text);
l.pieces += 1;
}
None => {
found.insert((r.op, r.rect), lines.len());
lines.push(Line { rect: r.rect, op: r.op, at: r.at, raw: r.raw.clone(), text: r.text.clone(), pieces: 1 });
}
}
}
lines
}
#[cfg(test)]
mod tests {
use super::*;
fn run(rect: usize, op: usize, place: usize, text: &str) -> Removed {
Removed { rect, raw: text.as_bytes().to_vec(), text: text.to_string(), codes: text.len(), op, at: (place, 0) }
}
#[test]
fn a_word_split_across_a_tj_array_is_one_line_for_each_region() {
let lines = joined(&[run(0, 1, 0, "sec"), run(1, 1, 0, "sec"), run(0, 1, 1, "ret"), run(1, 1, 1, "ret"), run(0, 2, 0, "other")]);
let shown: Vec<(usize, usize, &str, usize)> = lines.iter().map(|l| (l.rect, l.op, l.text.as_str(), l.pieces)).collect();
assert_eq!(shown, [(0, 1, "secret", 2), (1, 1, "secret", 2), (0, 2, "other", 1)]);
}
#[test]
fn overlapping_marks_are_one_copy_and_two_copies_stay_two() {
// Under two marks at once: the same operator, place and bytes, so one copy was cut.
let both = joined(&[run(0, 1, 0, "word"), run(1, 1, 0, "word")]);
let mut seen = HashSet::new();
assert_eq!(both.iter().filter(|l| seen.insert((l.op, l.at, l.raw.clone()))).count(), 1);
// The same word twice in one array, one under each mark: two copies were cut.
let apart = joined(&[run(0, 1, 0, "word"), run(1, 1, 2, "word")]);
let mut seen = HashSet::new();
assert_eq!(apart.iter().filter(|l| seen.insert((l.op, l.at, l.raw.clone()))).count(), 2);
}
#[test]
fn a_spent_decode_budget_is_told_apart() {
assert!(!Ctx::new(HashSet::new()).decode_spent());
assert!(Ctx::with_decode_left(0).decode_spent());
}
}
crates/redact/src/verify/sweep.rs
+759
View File
@@ -0,0 +1,759 @@
//! Sweeping the serialized output for the removed strings, on every surface a reader could
//! recover them from.
use std::collections::{BTreeMap, BTreeSet, HashSet};
use std::sync::Arc;
use pdfcraft_cos::{Dict, Document, ObjRef, Object, PdfString};
use super::extract::extract_page;
use super::util::{Ctx, decode_text, find, image_codec, spellings, squash, survivor_encodings, xml_unescape};
use super::{SURFACES, Surface, SurfaceReport, Verdict};
use crate::verify::limits::{MAX_NESTING, MAX_WALK, MIN_NEEDLE, MIN_STRUCTURED_NEEDLE};
/// A string the sweep looks for.
pub(super) struct Needle {
/// The code bytes as Latin-1 text (the "raw view" of a page) rather than decoded text.
pub(super) raw_view: bool,
/// As removed, whitespace included (for the byte encodings).
pub(super) shown: String,
/// `shown` without whitespace (for page text, which is compared whitespace-blind).
pub(super) key: String,
/// What the raw file may spell it as.
pub(super) bytes: Vec<Vec<u8>>,
pub(super) owners: BTreeSet<usize>,
/// Occurrences that legitimately remain in content that was not redacted. The surfaces that
/// hold page content (the page text, the raw file, decoded streams) count occurrences over
/// the whole document and fail on a surplus; every other surface (strings, Info, XMP,
/// attachments) is swept with no allowance.
pub(super) allowance: usize,
/// Names of the input that carry this needle (a redacted word that is also a `/Subtype`, a
/// `/Filter`, a `/BaseFont`). Names are structure redaction never edits, so the name surface
/// is swept against the input's own matches: the output may hold as many as the input did,
/// and only a surplus is a survivor.
pub(super) name_allowance: usize,
}
impl Needle {
pub(super) fn short(&self) -> bool {
self.key.chars().count() < MIN_NEEDLE
}
/// Long enough to be looked for in bytes (`min`: the shortest the haystack allows).
pub(super) fn at_least(&self, min: usize) -> bool {
self.key.chars().count() >= min
}
}
pub(super) fn add_needle(set: &mut Vec<Needle>, raw_view: bool, shown: String, owner: Option<usize>) {
let key = squash(&shown);
if key.is_empty() {
return;
}
if let Some(n) = set.iter_mut().find(|n| n.raw_view == raw_view && n.key == key) {
n.owners.extend(owner);
return;
}
let bytes = if raw_view {
// The code bytes themselves (Latin-1 round-trips them).
let raw: Vec<u8> = shown.chars().filter_map(|c| u8::try_from(u32::from(c)).ok()).collect();
spellings([raw])
} else {
survivor_encodings(&shown)
};
set.push(Needle { raw_view, shown, key, bytes, owners: owner.into_iter().collect(), allowance: 0, name_allowance: 0 });
}
pub(super) struct Acc {
pub(super) reps: BTreeMap<Surface, SurfaceReport>,
pub(super) failed_needles: HashSet<usize>,
pub(super) ctx: Ctx,
}
impl Acc {
pub(super) fn new(ctx: Ctx) -> Self {
let mut a = Acc { reps: BTreeMap::new(), failed_needles: HashSet::new(), ctx };
for s in SURFACES {
a.rep(s);
}
a
}
pub(super) fn rep(&mut self, surface: Surface) -> &mut SurfaceReport {
self.reps.entry(surface).or_insert_with(|| SurfaceReport {
surface,
verdict: Verdict::Absent,
scanned: 0,
raw_only: 0,
survivors: 0,
locations: Vec::new(),
problems: Vec::new(),
})
}
pub(super) fn scanned(&mut self, s: Surface) {
self.rep(s).scanned += 1;
}
pub(super) fn raw_only(&mut self, s: Surface) {
self.rep(s).raw_only += 1;
}
pub(super) fn survivor(&mut self, s: Surface, loc: String, needle: Option<usize>) {
self.survivors(s, loc, needle, 1);
}
/// `count` survivors at one place.
pub(super) fn survivors(&mut self, s: Surface, loc: String, needle: Option<usize>, count: usize) {
if let Some(n) = needle {
self.failed_needles.insert(n);
}
let r = self.rep(s);
r.survivors = r.survivors.saturating_add(count);
if !r.locations.contains(&loc) && r.locations.len() < 64 {
r.locations.push(loc);
}
}
pub(super) fn problem(&mut self, s: Surface, msg: String) {
let r = self.rep(s);
if r.problems.len() < 64 {
r.problems.push(msg);
}
}
/// The needles (at least `min` long) found in `hay`; running out of search work is a
/// problem of `surface`.
pub(super) fn bytes(&mut self, surface: Surface, needles: &[Needle], hay: &[u8], min: usize) -> Vec<usize> {
bytes_hits(needles, hay, min, &self.ctx).unwrap_or_else(|| {
self.problem(surface, "the search work limit was reached".into());
Vec::new()
})
}
/// How often each needle (at least `min` long) occurs in `hay`, as `(needle, count)`.
pub(super) fn counts(&mut self, surface: Surface, needles: &[Needle], hay: &[u8], min: usize) -> Vec<(usize, usize)> {
bytes_counts(needles, hay, min, &self.ctx).unwrap_or_else(|| {
self.problem(surface, "the search work limit was reached".into());
Vec::new()
})
}
/// Survivors and problems so far, over all surfaces.
pub(super) fn totals(&self) -> (usize, usize) {
(self.reps.values().map(|r| r.survivors).sum(), self.reps.values().map(|r| r.problems.len()).sum())
}
pub(super) fn finish(mut self) -> Vec<SurfaceReport> {
for r in self.reps.values_mut() {
r.verdict = if r.survivors > 0 {
Verdict::Survivor
} else if !r.problems.is_empty() {
Verdict::Unswept
} else if r.scanned + r.raw_only == 0 {
Verdict::Absent
} else {
Verdict::Clean
};
}
self.reps.into_values().collect()
}
}
/// Occurrences of needles on one surface, per place, so that what legitimately remains
/// (a needle's allowance) is taken off the document-wide total once, not once per place.
#[derive(Default)]
pub(super) struct Tally(BTreeMap<usize, Vec<(String, usize)>>);
impl Tally {
pub(super) fn add(&mut self, needle: usize, loc: &str, count: usize) {
if count > 0 {
self.0.entry(needle).or_default().push((loc.to_string(), count));
}
}
/// Report what is left of each needle's occurrences after its allowance (none inside an
/// attachment, where nothing legitimately remains).
pub(super) fn settle(self, surface: Surface, needles: &[Needle], depth: usize, acc: &mut Acc) {
self.settle_with(surface, needles, depth, acc, |n| n.allowance)
}
/// The name surface: the input's own name matches are the allowance (redaction never edits
/// names), so what is left is a surplus of matches over the input.
pub(super) fn settle_names(self, surface: Surface, needles: &[Needle], depth: usize, acc: &mut Acc) {
self.settle_with(surface, needles, depth, acc, |n| n.name_allowance)
}
fn settle_with(self, surface: Surface, needles: &[Needle], depth: usize, acc: &mut Acc, allow_of: impl Fn(&Needle) -> usize) {
for (ni, places) in self.0 {
let mut allow = if depth == 0 { needles.get(ni).map_or(0, &allow_of) } else { 0 };
for (loc, count) in places {
let used = count.min(allow);
allow -= used;
if count > used {
acc.survivors(surface, loc, Some(ni), count - used);
}
}
}
}
}
/// Search all encodings in one bounded pass, including overlapping matches.
fn pattern_matches<'a>(
patterns: impl IntoIterator<Item = (usize, &'a [u8])>,
hay: &[u8],
ctx: &Ctx,
mut found: impl FnMut(usize, usize, usize) -> bool,
) -> Option<()> {
const MAX_PATTERNS: usize = 100_000;
const MAX_PATTERN_BYTES: usize = 16 << 20;
const MAX_MATCHES: usize = 1 << 20;
if !ctx.charge(hay.len()) {
return None;
}
let mut map: BTreeMap<Vec<u8>, Vec<usize>> = BTreeMap::new();
let (mut bytes, mut links) = (0usize, 0usize);
for (owner, pattern) in patterns {
if pattern.is_empty() {
continue;
}
if !ctx.charge(pattern.len().saturating_add(1)) {
return None;
}
if !map.contains_key(pattern) {
bytes = bytes.checked_add(pattern.len())?;
if bytes > MAX_PATTERN_BYTES || map.len() >= MAX_PATTERNS {
return None;
}
}
let owners = map.entry(pattern.to_vec()).or_default();
if owners.last() != Some(&owner) {
links = links.checked_add(1)?;
if links > MAX_PATTERNS {
return None;
}
owners.push(owner);
}
}
if map.is_empty() {
return Some(());
}
let matcher = aho_corasick::AhoCorasick::new(map.keys()).ok()?;
let owners: Vec<Vec<usize>> = map.into_values().collect();
let mut matches = 0usize;
for hit in matcher.find_overlapping_iter(hay) {
for &owner in owners.get(hit.pattern().as_usize())? {
matches = matches.checked_add(1)?;
if matches > MAX_MATCHES || !ctx.charge(1) || !found(owner, hit.start(), hit.end()) {
return None;
}
}
}
Some(())
}
/// Count normalized text keys in one pass, with the same non-overlap rule as `str::matches`.
pub(super) fn key_counts(needles: &[Needle], hay: &str, raw: bool, ctx: &Ctx) -> Option<Vec<(usize, usize)>> {
key_counts_from(needles, hay, raw, ctx, 0)
}
/// [`key_counts`] of the needles at least `min` long, so a one-letter needle can't use up the match
/// limits of a search it takes no part in.
fn key_counts_from(needles: &[Needle], hay: &str, raw: bool, ctx: &Ctx, min: usize) -> Option<Vec<(usize, usize)>> {
let patterns = needles.iter().enumerate().filter(|(_, n)| n.raw_view == raw && n.at_least(min)).map(|(i, n)| (i, n.key.as_bytes()));
let mut spans: BTreeMap<usize, Vec<(usize, usize)>> = BTreeMap::new();
pattern_matches(patterns, hay.as_bytes(), ctx, |i, a, b| {
let hits = spans.entry(i).or_default();
if hits.len() >= MAX_COUNTED {
return false;
}
hits.push((a, b));
true
})?;
Some(
spans
.into_iter()
.map(|(i, mut hits)| {
hits.sort_unstable();
let (mut count, mut end) = (0, 0);
for (a, b) in hits {
if a >= end {
count += 1;
end = b;
}
}
(i, count)
})
.collect(),
)
}
pub(super) fn bytes_hits(needles: &[Needle], hay: &[u8], min: usize, ctx: &Ctx) -> Option<Vec<usize>> {
let patterns = needles.iter().enumerate().filter(|(_, n)| n.at_least(min)).flat_map(|(i, n)| n.bytes.iter().map(move |b| (i, b.as_slice())));
let mut found = BTreeSet::new();
pattern_matches(patterns, hay, ctx, |i, _, _| {
found.insert(i);
true
})?;
Some(found.into_iter().collect())
}
/// A larger count is incomplete evidence, never a silently truncated count.
pub(super) const MAX_COUNTED: usize = 1 << 16;
pub(super) fn bytes_counts(needles: &[Needle], hay: &[u8], min: usize, ctx: &Ctx) -> Option<Vec<(usize, usize)>> {
let patterns = needles.iter().enumerate().filter(|(_, n)| n.at_least(min)).flat_map(|(i, n)| n.bytes.iter().map(move |b| (i, b.as_slice())));
let mut spans: BTreeMap<usize, Vec<(usize, usize)>> = BTreeMap::new();
pattern_matches(patterns, hay, ctx, |i, a, b| {
let hits = spans.entry(i).or_default();
if hits.len() >= MAX_COUNTED {
return false;
}
hits.push((a, b));
true
})?;
Some(
spans
.into_iter()
.map(|(i, mut spans)| {
spans.sort_unstable();
spans.dedup();
let (mut count, mut end) = (0usize, 0usize);
for (a, b) in spans {
if count == 0 || a >= end {
count += 1;
}
end = end.max(b);
}
(i, count)
})
.collect(),
)
}
pub(super) fn string_hits(needles: &[Needle], value: &PdfString, ctx: &Ctx) -> Option<Vec<usize>> {
let text = value.to_text();
let key = squash(&text);
if !ctx.charge(value.bytes.len().saturating_add(key.len())) {
return None;
}
let mut found: BTreeSet<usize> = bytes_hits(needles, &value.bytes, MIN_NEEDLE, ctx)?.into_iter().collect();
let patterns = needles.iter().enumerate().filter(|(_, n)| !n.raw_view && !n.short()).map(|(i, n)| (i, n.key.as_bytes()));
pattern_matches(patterns, key.as_bytes(), ctx, |i, _, _| {
found.insert(i);
true
})?;
let tokens: BTreeSet<&str> = text.split(|c: char| !c.is_alphanumeric()).filter(|s| !s.is_empty()).collect();
for (i, needle) in needles.iter().enumerate().filter(|(_, n)| n.short() && n.allowance == 0 && !n.raw_view) {
if !ctx.charge(needle.shown.len().saturating_add(1)) {
return None;
}
if tokens.contains(needle.shown.as_str()) {
found.insert(i);
}
}
Some(found.into_iter().collect())
}
/// Streams whose data is binary rather than text: images, embedded font programs, colour
/// profiles, cross-reference and object streams (the objects in the latter are swept as
/// objects). Their decoded data is swept like any other stream's; the pixels of an image codec
/// stay undecoded (text that exists only as pixels is not text), and one that won't decode (or is
/// past the cap) is searched as stored bytes only. A short needle "found" in stored bytes is
/// chance.
pub(super) fn binary_stream(d: &Dict) -> bool {
let subtype = d.name(b"Subtype");
subtype == Some(b"Image".as_slice())
|| matches!(d.name(b"Type"), Some(b"XRef" | b"ObjStm"))
|| matches!(subtype, Some(b"Type1C" | b"CIDFontType0C" | b"OpenType"))
|| [b"Length1".as_slice(), b"Length2", b"Length3"].iter().any(|k| d.contains(k))
|| (d.contains(b"N") && !d.contains(b"Type") && subtype.is_none())
}
/// What a file identifier (`/ID` of the trailer) is spelled as in the file.
pub(super) fn id_spellings(doc: &Document) -> Vec<Vec<u8>> {
let Some(Object::Array(ids)) = doc.trailer().get(b"ID").map(|o| doc.resolve(o)).as_deref().cloned() else { return Vec::new() };
ids.iter().filter_map(Object::as_string).flat_map(|s| spellings([s.bytes.to_vec()])).collect()
}
/// Visit `o` and everything nested in it. False when something lay deeper than [`MAX_WALK`]
/// and was not visited: the caller reports that as unswept.
pub(super) fn walk_strings(o: &Object, depth: usize, f: &mut impl FnMut(&Object)) -> bool {
if depth > MAX_WALK {
return false;
}
f(o);
// Every child is visited even after one is too deep: nothing is skipped silently.
let mut ok = true;
match o {
Object::Array(a) => a.iter().for_each(|x| ok &= walk_strings(x, depth + 1, f)),
Object::Dict(d) => d.iter().for_each(|(_, v)| ok &= walk_strings(v, depth + 1, f)),
Object::Stream(s) => s.dict.iter().for_each(|(_, v)| ok &= walk_strings(v, depth + 1, f)),
_ => {}
}
ok
}
/// Sweep a document (the output, or an attachment) on every surface.
pub(super) fn sweep_doc(needles: &[Needle], doc: &Document, bytes: &[u8], depth: usize, tag: &str, acc: &mut Acc) {
let (embedded, overlay_raw) = sweep_objects(needles, doc, depth, tag, acc);
sweep_file(needles, doc, bytes, overlay_raw, depth, tag, acc);
sweep_pages(needles, doc, depth, tag, acc);
sweep_info(needles, doc, tag, acc);
sweep_xmp(needles, doc, tag, acc);
// Revision structure: one revision, no earlier content kept.
acc.scanned(Surface::Revisions);
let revs = doc.revisions().len();
if revs > 1 || doc.trailer().contains(b"Prev") {
acc.survivor(Surface::Revisions, format!("{tag}{revs} revisions: earlier ones keep the original content"), None);
}
sweep_attachments(needles, doc, &embedded, depth, tag, acc);
}
/// Every object: strings, names and stream data (orphans included). Returns the embedded file
/// streams found, and the raw bytes of the overlay streams (to leave out of the raw sweep).
fn sweep_objects(needles: &[Needle], doc: &Document, depth: usize, tag: &str, acc: &mut Acc) -> (BTreeSet<u32>, Vec<Vec<u8>>) {
let mut embedded: BTreeSet<u32> = BTreeSet::new();
let mut overlay_raw: Vec<Vec<u8>> = Vec::new();
let mut in_streams = Tally::default();
let mut in_names = Tally::default();
let lookup = lookup_streams(doc);
for num in doc.object_numbers() {
let loc = format!("{tag}object {num}");
let obj = match doc.try_get(num) {
Ok(o) => o,
Err(_) => {
acc.problem(Surface::ObjectStrings, format!("{loc} can't be read"));
continue;
}
};
let mut hits: BTreeSet<usize> = BTreeSet::new();
let mut spent = false;
let ctx = &acc.ctx;
// The file identifier is random bytes, never text: a short needle must not "read" it.
let without_id = match &*obj {
Object::Stream(s) if s.dict.name(b"Type") == Some(b"XRef") => {
let mut d = s.dict.clone();
d.remove(b"ID");
Some(Object::Dict(d))
}
_ => None,
};
let complete = walk_strings(without_id.as_ref().unwrap_or(&obj), 0, &mut |o| match o {
Object::String(s) => match string_hits(needles, s, ctx) {
Some(h) => hits.extend(h),
None => spent = true,
},
Object::Name(n) => match bytes_hits(needles, n, MIN_NEEDLE, ctx) {
// A name the input had its own matches in is structure, not a leftover copy:
// the matches are tallied and settled against the input's count.
Some(h) => h.into_iter().for_each(|ni| in_names.add(ni, &loc, 1)),
None => spent = true,
},
_ => {}
});
if !complete {
acc.problem(Surface::ObjectStrings, format!("{loc} is nested too deeply to sweep in full"));
}
if spent {
acc.problem(Surface::ObjectStrings, "the search work limit was reached".into());
}
acc.scanned(Surface::ObjectStrings);
for n in hits {
acc.survivor(Surface::ObjectStrings, loc.clone(), Some(n));
}
// File specifications, wherever they sit (a name tree keeps them as separate objects,
// annotations and collections may inline them).
walk_strings(&obj, 0, &mut |o| {
if let Object::Dict(d) = o
&& let Some(Object::Dict(ef)) = d.get(b"EF")
{
embedded.extend(ef.iter().filter_map(|(_, v)| v.as_ref()).map(|r| r.num));
}
});
if let Object::Stream(s) = &*obj {
if s.dict.name(b"Type") == Some(b"EmbeddedFile") {
embedded.insert(num);
}
acc.scanned(Surface::DecodedStreams);
if binary_stream(&s.dict) && image_codec(&s.dict) {
// The pixels of an image codec are still encoded and are not text: their stored
// bytes are searched for whole strings only.
acc.raw_only(Surface::DecodedStreams);
for n in acc.bytes(Surface::DecodedStreams, needles, &s.raw, MIN_STRUCTURED_NEEDLE) {
acc.survivor(Surface::DecodedStreams, format!("{loc} (raw)"), Some(n));
}
} else if binary_stream(&s.dict) {
// A compressed copy of the removed text can hide in a font program or colour
// profile (with /Length1 in its dict, its stored bytes hold no plaintext): the
// stored bytes and the decoded data are both searched. Such a stream that won't
// decode (damaged, or past the cap) is searched as stored bytes only and counted
// as raw only: it holds no page content, and redaction never touches it. A
// cross-reference or object stream that won't decode is a problem of the proof.
for n in acc.bytes(Surface::DecodedStreams, needles, &s.raw, MIN_STRUCTURED_NEEDLE) {
acc.survivor(Surface::DecodedStreams, format!("{loc} (raw)"), Some(n));
}
match acc.ctx.decode(s) {
Ok(d) => {
let loc = format!("{loc} (decoded)");
for (n, c) in acc.counts(Surface::DecodedStreams, needles, &d, MIN_STRUCTURED_NEEDLE) {
in_streams.add(n, &loc, c);
}
}
Err(()) if acc.ctx.decode_spent() || matches!(s.dict.name(b"Type"), Some(b"XRef" | b"ObjStm")) => {
acc.problem(Surface::DecodedStreams, format!("{loc} can't be decoded in full"));
}
Err(()) => acc.raw_only(Surface::DecodedStreams),
}
} else {
match acc.ctx.decode(s) {
// The boxes and labels this run drew are not text to look for.
Ok(d) if acc.ctx.is_overlay(&d) => overlay_raw.push(s.raw.to_vec()),
Ok(d) => {
let loc = format!("{loc} (decoded)");
for (n, c) in acc.counts(Surface::DecodedStreams, needles, &d, MIN_STRUCTURED_NEEDLE) {
in_streams.add(n, &loc, c);
}
}
// Colour-space lookup data and function samples are data tables, not content.
Err(()) if !acc.ctx.decode_spent() && (lookup.contains(&num) || s.dict.contains(b"FunctionType")) => {
acc.raw_only(Surface::DecodedStreams);
for n in acc.bytes(Surface::DecodedStreams, needles, &s.raw, MIN_STRUCTURED_NEEDLE) {
acc.survivor(Surface::DecodedStreams, format!("{loc} (raw)"), Some(n));
}
}
Err(()) => acc.problem(Surface::DecodedStreams, format!("{loc} can't be decoded in full")),
}
}
}
}
in_names.settle_names(Surface::ObjectStrings, needles, depth, acc);
in_streams.settle(Surface::DecodedStreams, needles, depth, acc);
(embedded, overlay_raw)
}
/// The streams that hold the lookup table of an `/Indexed` colour space (object numbers).
fn lookup_streams(doc: &Document) -> BTreeSet<u32> {
let mut out = BTreeSet::new();
for num in doc.object_numbers() {
let Ok(obj) = doc.try_get(num) else { continue };
walk_strings(&obj, 0, &mut |o| {
if let Object::Array(a) = o
&& matches!(a.first(), Some(Object::Name(n)) if n == b"Indexed" || n == b"I")
&& let Some(Object::Ref(r)) = a.get(3)
{
out.insert(r.num);
}
});
}
out
}
/// The raw bytes of the whole file, the overlay streams (stored as they are, or not at all) and
/// the file identifier (random bytes, regenerated on every write) aside.
fn sweep_file(needles: &[Needle], doc: &Document, bytes: &[u8], overlay_raw: Vec<Vec<u8>>, depth: usize, tag: &str, acc: &mut Acc) {
acc.scanned(Surface::RawBytes);
let mut mask = overlay_raw;
mask.extend(id_spellings(doc));
let masked = mask_all(bytes, &mask, &acc.ctx);
let mut in_file = Tally::default();
for (n, c) in acc.counts(Surface::RawBytes, needles, masked.as_deref().unwrap_or(bytes), MIN_STRUCTURED_NEEDLE) {
in_file.add(n, &format!("{tag}file"), c);
}
in_file.settle(Surface::RawBytes, needles, depth, acc);
if masked.is_none() && !mask.is_empty() {
acc.problem(Surface::RawBytes, "the search work limit was reached".into());
}
}
/// Embedded files, PDF attachments swept in full.
fn sweep_attachments(needles: &[Needle], doc: &Document, embedded: &BTreeSet<u32>, depth: usize, tag: &str, acc: &mut Acc) {
for &num in embedded {
let loc = format!("{tag}attachment object {num}");
acc.scanned(Surface::EmbeddedFiles);
let obj = doc.get(ObjRef::new(num, doc.generation(num)));
let Object::Stream(s) = &*obj else {
acc.problem(Surface::EmbeddedFiles, format!("{loc} is not a stream"));
continue;
};
let payload = match acc.ctx.decode(s) {
Ok(p) => p,
Err(()) => {
acc.problem(Surface::EmbeddedFiles, format!("{loc} can't be decoded in full"));
continue;
}
};
sweep_payload(needles, &payload, depth, &loc, acc);
}
}
/// `bytes` with every occurrence of `pats` zeroed (`None` when the work limit ran out, or
/// there is nothing to mask).
pub(super) fn mask_all(bytes: &[u8], pats: &[Vec<u8>], ctx: &Ctx) -> Option<Vec<u8>> {
if pats.iter().all(Vec::is_empty) {
return None;
}
let mut out = bytes.to_vec();
for p in pats.iter().filter(|p| !p.is_empty()) {
if !ctx.charge(bytes.len()) {
return None;
}
let mut at = 0;
while let Some(w) = bytes.get(at..at + p.len()) {
if w == p.as_slice() {
if let Some(z) = out.get_mut(at..at + p.len()) {
z.fill(0);
}
at += p.len();
} else {
at += 1;
}
}
}
Some(out)
}
/// One attachment: a PDF is swept as a document, text is searched, containers are reported as
/// unswept (their content can't be seen), images are searched as raw bytes only.
pub(super) fn sweep_payload(needles: &[Needle], payload: &[u8], depth: usize, loc: &str, acc: &mut Acc) {
let head = payload.get(..1024).unwrap_or(payload);
let tag = format!("{loc} > ");
if find(head, b"%PDF-") {
if depth >= MAX_NESTING {
acc.problem(Surface::EmbeddedFiles, format!("{loc} is nested too deep to sweep"));
return;
}
match Document::open(Arc::new(payload.to_vec())) {
Ok(inner) => {
let before = acc.totals();
sweep_doc(needles, &inner, payload, depth + 1, &tag, acc);
// What the attachment's own sweep found counts against the attachment too.
let after = acc.totals();
if after.0 > before.0 {
acc.survivor(Surface::EmbeddedFiles, loc.to_string(), None);
}
if after.1 > before.1 {
acc.problem(Surface::EmbeddedFiles, format!("{loc} could not be swept in full"));
}
}
Err(_) => acc.problem(Surface::EmbeddedFiles, format!("{loc} is a PDF that can't be opened")),
}
return;
}
let starts = |sig: &[u8]| payload.starts_with(sig);
let container = starts(b"PK\x03\x04")
|| starts(b"PK\x05\x06")
|| starts(&[0x1F, 0x8B])
|| starts(b"BZh")
|| starts(&[0xFD, b'7', b'z', b'X', b'Z', 0])
|| starts(&[b'7', b'z', 0xBC, 0xAF, 0x27, 0x1C])
|| starts(b"Rar!")
|| starts(&[0xD0, 0xCF, 0x11, 0xE0])
|| payload.get(257..262) == Some(b"ustar");
if container {
acc.problem(Surface::EmbeddedFiles, format!("{loc} is an archive or container that can't be opened"));
return;
}
let image = starts(&[0x89, b'P', b'N', b'G']) || starts(&[0xFF, 0xD8, 0xFF]) || starts(b"GIF8");
if image {
acc.raw_only(Surface::EmbeddedFiles);
}
let min = if image { MIN_STRUCTURED_NEEDLE } else { MIN_NEEDLE };
for n in acc.bytes(Surface::EmbeddedFiles, needles, payload, min) {
acc.survivor(Surface::EmbeddedFiles, loc.to_string(), Some(n));
}
if !image {
let key = squash(&decode_text(payload));
match key_counts_from(needles, &key, false, &acc.ctx, MIN_NEEDLE) {
Some(counts) => {
for (i, count) in counts {
if count > 0 {
acc.survivor(Surface::EmbeddedFiles, loc.to_string(), Some(i));
}
}
}
None => acc.problem(Surface::EmbeddedFiles, format!("{loc} could not be searched in full")),
}
}
}
pub(super) fn sweep_pages(needles: &[Needle], doc: &Document, depth: usize, tag: &str, acc: &mut Acc) {
let mut shown = Tally::default();
'pages: for (i, p) in pdfcraft_model::pages(doc).iter().enumerate() {
acc.scanned(Surface::ExtractedText);
let loc = format!("{tag}page {}", i + 1);
let Ok(e) = extract_page(doc, &p.dict, &acc.ctx, &[]) else {
acc.problem(Surface::ExtractedText, format!("{loc} can't be read"));
continue;
};
for (hay, raw) in [(&e.text, false), (&e.raw, true)] {
let Some(counts) = key_counts(needles, hay, raw, &acc.ctx) else {
acc.problem(Surface::ExtractedText, "the search work limit was reached".into());
break 'pages;
};
for (ni, count) in counts {
shown.add(ni, &loc, count);
}
}
}
// A surplus over what legitimately remains anywhere in the document is a survivor. The
// counts made before the work ran out are settled too: a survivor already seen is
// reported where it sits, not hidden behind the limit.
shown.settle(Surface::ExtractedText, needles, depth, acc);
}
pub(super) fn sweep_info(needles: &[Needle], doc: &Document, tag: &str, acc: &mut Acc) {
let Some(info) = doc.trailer().get(b"Info").map(|i| doc.resolve(i)) else { return };
let Some(d) = info.as_dict() else {
acc.problem(Surface::Info, format!("{tag}Info is not a dictionary"));
return;
};
for (k, v) in d.iter() {
acc.scanned(Surface::Info);
let v = doc.resolve(v);
let mut hits: BTreeSet<usize> = BTreeSet::new();
let mut spent = false;
let ctx = &acc.ctx;
let complete = walk_strings(&v, 0, &mut |o| {
if let Object::String(s) = o {
match string_hits(needles, s, ctx) {
Some(h) => hits.extend(h),
None => spent = true,
}
}
});
if !complete || spent {
acc.problem(Surface::Info, format!("{tag}Info /{} can't be swept in full", String::from_utf8_lossy(k)));
}
for n in hits {
acc.survivor(Surface::Info, format!("{tag}Info /{}", String::from_utf8_lossy(k)), Some(n));
}
}
}
pub(super) fn sweep_xmp(needles: &[Needle], doc: &Document, tag: &str, acc: &mut Acc) {
let Some(root) = doc.root() else { return };
let Some(meta) = doc.get(root).as_dict().and_then(|c| c.get(b"Metadata")).map(|m| doc.resolve(m)) else { return };
acc.scanned(Surface::Xmp);
let loc = format!("{tag}XMP metadata");
let Object::Stream(s) = &*meta else {
acc.problem(Surface::Xmp, format!("{loc} is not a stream"));
return;
};
let Ok(data) = acc.ctx.decode(s) else {
acc.problem(Surface::Xmp, format!("{loc} can't be decoded in full"));
return;
};
let mut hits: BTreeSet<usize> = acc.bytes(Surface::Xmp, needles, &data, MIN_NEEDLE).into_iter().collect();
let key = squash(&xml_unescape(&decode_text(&data)));
for (i, n) in needles.iter().enumerate() {
if n.at_least(MIN_NEEDLE) && !n.raw_view && key.contains(&n.key) {
hits.insert(i);
}
}
for n in hits {
acc.survivor(Surface::Xmp, loc.clone(), Some(n));
}
}
crates/redact/src/verify/util.rs
+301
View File
@@ -0,0 +1,301 @@
//! Shared helpers of the proof: what one proof may spend, text and byte search, and the
//! spellings a string can have in a PDF.
use std::cell::Cell;
use std::collections::HashSet;
use std::fmt::Write as _;
use std::sync::Arc;
use pdfcraft_cos::{DecodeBudget, Dict, Document, Object, Stream};
use sha2::{Digest, Sha256};
use crate::verify::limits::{MAX_STREAM, MAX_TOTAL_DECODED, MAX_WORK, MIN_NEEDLE};
/// What one proof may spend, and what it knows about the document it checks.
pub(super) struct Ctx {
/// Digests of the decoded content of the overlay streams (their boxes and labels are drawn
/// on purpose and are neither text to look for nor paint to flag).
pub(super) overlay: HashSet<String>,
/// Decoded bytes of the proof so far: one allowance for the whole proof, failures charged
/// too (see [`Ctx::decode`]), so a stream referenced many times, or one that fails after
/// decoding part of itself, cannot decode again for free.
pub(super) decoded: Cell<u64>,
pub(super) work: Cell<u64>,
}
impl Ctx {
pub(super) fn new(overlay: HashSet<String>) -> Self {
Ctx { overlay, decoded: Cell::new(0), work: Cell::new(0) }
}
/// Decode `s` strictly: a stream that is damaged, longer than [`MAX_STREAM`], or one more
/// than [`MAX_TOTAL_DECODED`] allows is an error (never silently truncated data). Every
/// attempt is charged to the cumulative allowance, failures included: a decode that fails
/// past the one-stream cap pays the cap, and one that fails mid-way pays what it decoded
/// before the error — the work of finding out is not work a repeated reference gets for free.
pub(super) fn decode(&self, s: &Stream) -> Result<Vec<u8>, ()> {
let left = MAX_TOTAL_DECODED.saturating_sub(self.decoded.get());
let cap = usize::try_from(left).unwrap_or(usize::MAX).min(MAX_STREAM);
let mut budget = DecodeBudget::new(cap, usize::MAX);
let result = budget.decode_strict(s);
// The attempt's own accounting covers a success (what it decoded), a stream cut short
// by corruption (what it decoded first) and one refused past the cap (the cap): charge
// it through, whichever way the attempt went.
let spent = cap.saturating_sub(budget.remaining());
self.decoded.set(self.decoded.get().saturating_add(spent as u64));
result.map_err(|_| ())
}
/// Is the cumulative decode allowance used up? A decode refused then says nothing about the
/// stream, so it must not pass for a stream that can't be decoded.
pub(super) fn decode_spent(&self) -> bool {
self.decoded.get() >= MAX_TOTAL_DECODED
}
/// Spend `n` units of search work; false once [`MAX_WORK`] is used up.
pub(super) fn charge(&self, n: usize) -> bool {
let used = self.work.get().saturating_add(n as u64);
self.work.set(used);
used <= MAX_WORK
}
/// Is this decoded content one of the overlay streams this run created?
pub(super) fn is_overlay(&self, data: &[u8]) -> bool {
!self.overlay.is_empty() && self.overlay.contains(&sha256(&[data]))
}
/// (tests) A context whose cumulative decode allowance has `left` bytes left.
#[cfg(test)]
pub(super) fn with_decode_left(left: u64) -> Self {
Ctx { overlay: HashSet::new(), decoded: Cell::new(MAX_TOTAL_DECODED.saturating_sub(left)), work: Cell::new(0) }
}
}
/// Follow `o` to its object; a reference that can't be loaded is an error (a missing object is
/// `Null` for [`Document::resolve`], which would read as "empty").
pub(super) fn resolve_strict(doc: &Document, o: &Object) -> Result<Arc<Object>, ()> {
match o {
Object::Ref(r) => doc.try_get(r.num).map_err(|_| ()),
other => Ok(Arc::new(other.clone())),
}
}
pub(super) fn hex(bytes: &[u8]) -> String {
bytes.iter().fold(String::with_capacity(bytes.len() * 2), |mut s, b| {
let _ = write!(s, "{b:02x}");
s
})
}
pub(super) fn sha256<T: AsRef<[u8]>>(parts: &[T]) -> String {
let mut h = Sha256::new();
for p in parts {
let p = p.as_ref();
h.update((p.len() as u64).to_be_bytes());
h.update(p);
}
hex(h.finalize().as_slice())
}
/// The text of `s` without whitespace (page text is compared whitespace-blind: a gap in a `TJ`
/// is a space in one extraction and nothing in another).
pub(super) fn squash(s: &str) -> String {
s.chars().filter(|c| !c.is_whitespace() && *c != '\u{feff}').collect()
}
pub(super) fn latin1(b: &[u8]) -> String {
b.iter().map(|&c| char::from(c)).collect()
}
/// The words of a run worth looking for on their own (a run that survives in pieces).
pub(super) fn words(s: &str) -> Vec<&str> {
s.split_whitespace().filter(|w| w.chars().count() >= MIN_NEEDLE).collect()
}
pub(super) fn count(hay: &str, needle: &str) -> usize {
if needle.is_empty() { 0 } else { hay.matches(needle).count() }
}
/// Is `pat` in `hay`?
pub(super) fn find(hay: &[u8], pat: &[u8]) -> bool {
find_from(hay, pat, 0).is_some()
}
/// Where `pat` first occurs in `hay` at or after `from`. The pattern's prefix function is
/// built once and the haystack is walked once: the search is worst-case linear, so
/// adversarial input (a needle of one letter and a haystack full of it) can neither make it
/// quadratic nor spend more comparisons than the caller charges for.
pub(super) fn find_from(hay: &[u8], pat: &[u8], from: usize) -> Option<usize> {
let m = pat.len();
if m == 0 {
return None;
}
// fail[i]: the longest proper prefix of `pat` that ends at `i` and is also a suffix there.
let mut fail = vec![0usize; m];
let mut k = 0;
for (i, &c) in pat.iter().enumerate().skip(1) {
while k > 0 && pat[k] != c {
k = fail[k - 1];
}
if pat[k] == c {
k += 1;
}
fail[i] = k;
}
let mut k = 0;
for (at, &c) in hay.iter().enumerate().skip(from) {
while k > 0 && pat[k] != c {
k = fail[k - 1];
}
if pat[k] == c {
k += 1;
if k == m {
return Some(at + 1 - m);
}
}
}
None
}
pub(super) fn escape_literal(b: &[u8], octal: bool) -> Vec<u8> {
let mut out = Vec::with_capacity(b.len() * 2);
for &c in b {
if octal {
out.extend(format!("\\{c:03o}").bytes());
continue;
}
match c {
b'(' | b')' | b'\\' => out.extend([b'\\', c]),
b'\n' => out.extend(*b"\\n"),
b'\r' => out.extend(*b"\\r"),
b'\t' => out.extend(*b"\\t"),
_ => out.push(c),
}
}
out
}
/// Every way `text` can be spelled in a PDF: UTF-8, UTF-16BE (with and without a byte order
/// mark) and UTF-16LE, each as raw bytes, as a hex string (either case) and as a backslash-
/// escaped literal string (minimal escapes, and octal for every byte).
pub(crate) fn survivor_encodings(text: &str) -> Vec<Vec<u8>> {
let utf16 = |be: bool, bom: bool| -> Vec<u8> {
let mut v = Vec::new();
if bom {
v.extend(if be { [0xFE, 0xFF] } else { [0xFF, 0xFE] });
}
for u in text.encode_utf16() {
v.extend(if be { u.to_be_bytes() } else { u.to_le_bytes() });
}
v
};
let forms = [text.as_bytes().to_vec(), utf16(true, true), utf16(true, false), utf16(false, true), utf16(false, false)];
spellings(forms)
}
/// The spellings of a byte string (see [`survivor_encodings`]).
pub(super) fn spellings(forms: impl IntoIterator<Item = Vec<u8>>) -> Vec<Vec<u8>> {
let mut out: Vec<Vec<u8>> = Vec::new();
for f in forms {
if f.is_empty() {
continue;
}
let h = hex(&f);
out.push(h.to_uppercase().into_bytes());
out.push(h.into_bytes());
out.push(escape_literal(&f, false));
out.push(escape_literal(&f, true));
out.push(f);
}
let mut seen = HashSet::new();
out.retain(|b| seen.insert(b.clone()));
out
}
/// Text of a payload: UTF-16 when it has a byte order mark, else UTF-8 (lossy).
pub(super) fn decode_text(b: &[u8]) -> String {
let units = |be: bool, rest: &[u8]| -> String {
let u: Vec<u16> = rest.as_chunks::<2>().0.iter().map(|c| if be { u16::from_be_bytes(*c) } else { u16::from_le_bytes(*c) }).collect();
String::from_utf16_lossy(&u)
};
match b {
[0xFE, 0xFF, rest @ ..] => units(true, rest),
[0xFF, 0xFE, rest @ ..] => units(false, rest),
_ => String::from_utf8_lossy(b).into_owned(),
}
}
pub(super) fn xml_unescape(s: &str) -> String {
let mut out = String::with_capacity(s.len());
let mut rest = s;
while let Some(at) = rest.find('&') {
out.push_str(rest.get(..at).unwrap_or_default());
rest = rest.get(at..).unwrap_or_default();
match xml_entity(rest) {
Some((c, len)) => {
out.push(c);
rest = rest.get(len..).unwrap_or_default();
}
None => {
out.push('&');
rest = rest.get(1..).unwrap_or_default();
}
}
}
out.push_str(rest);
out
}
/// The entity starting at `&`: the character it reads as, and how far it runs (through the
/// closing `;`). A numeric reference may carry any number of leading zeros, so its digits
/// are padded without a bound; more significant digits than a character value can hold, or
/// a missing `;`, is not an entity and the `&` stays as it is. Walking only the digits the
/// reference itself holds keeps the decode linear in the input.
fn xml_entity(s: &str) -> Option<(char, usize)> {
for (name, c) in [("&amp;", '&'), ("&lt;", '<'), ("&gt;", '>'), ("&quot;", '"'), ("&apos;", '\'')] {
if s.starts_with(name) {
return Some((c, name.len()));
}
}
let body = s.strip_prefix("&#")?;
let (radix, digits) = match body.as_bytes().first() {
Some(b'x' | b'X') => (16u32, &body[1..]),
_ => (10, body),
};
let mut value: u32 = 0;
let mut digits_held = false;
let mut end = None;
for (i, c) in digits.char_indices() {
match c.to_digit(radix) {
Some(d) => {
digits_held = true;
// Leading zeros only pad; past a character's worth of significant digits the
// multiplication overflows and this was never an entity.
value = value.checked_mul(radix)?.checked_add(d)?;
}
None => {
if c == ';' && digits_held {
end = Some(i);
}
break;
}
}
}
let end = end?;
let ch = char::from_u32(value)?;
Some((ch, 2 + usize::from(radix == 16) + end + 1))
}
/// Names of the filters of a stream dictionary.
pub(super) fn filter_names(d: &Dict) -> Vec<Vec<u8>> {
match d.get(b"Filter") {
Some(Object::Name(n)) => vec![n.clone()],
Some(Object::Array(a)) => a.iter().filter_map(|o| o.as_name().map(<[u8]>::to_vec)).collect(),
_ => Vec::new(),
}
}
/// Image codecs: their data is pixels, still encoded.
pub(super) fn image_codec(d: &Dict) -> bool {
filter_names(d).iter().any(|n| matches!(n.as_slice(), b"DCTDecode" | b"DCT" | b"JPXDecode" | b"CCITTFaxDecode" | b"CCF" | b"JBIG2Decode"))
}
Loaded 100 of 148 files, more files were not shown because too many files have changed in this diff. Show more