Lolly Beta
Launch App

Conformance and fidelity

This chapter is a draft for review, dated 2026-09-24.

Conformance is what a suite measured. Acceptance is what an authority decided about that measurement. The two answers must never be merged into one verdict (R8, D12).

This chapter specifies conformance suites, their features and the report a run emits. It then specifies the acceptance record an authority writes, the three checks that test a still-image claim, the motion checks, the editable round trips and the conformance of operations that draw nothing. It restates plan section 11 in normative form, with the corrections the review recorded as C8. The operations being measured are specified in Operations and outcomes. The authority that accepts a result is specified in Constraints, authority and local choice.

Two separate answers

A measured result and an accepted result answer different questions, so this specification keeps them in separate records (R8).

Measured conformance states the suite name and its released version, the implementation identity, the reference environment, the fixture, reference and comparator digests and the per-feature results (R8, C8). It is a fact about one run on one machine.

Contextual acceptance states the authority, the suite reference it judged and each exception it granted (R8, D12). It is a decision, and a different authority may decide otherwise about the same report.

An authority may select a suite, require stricter checks, approve references or accept an exception (R8). None of those decisions may change the meaning of a suite version already published under another identity (R8). A local waiver of a creative rule must leave measured conformance to the unmodified rule reading failed, never passed (R6, C4). The waiver is recorded as an exception on the acceptance, never on the report (R8).

A conformance suite producing a report on one side and an authority producing an acceptance on the other, with no arrow that merges the two into a single verdict.Open this in Diagram Builder

Suites and features

A conformance suite is an immutable versioned set of fixtures, references, a comparator and a reference environment (plan section 3). A suite must be released under a version, and a report must never name a branch or a commit in its place (R8).

Each suite declares named features, and each feature is marked core or extended (R8). A core feature must never be skipped (R8). An extended feature may be unsupported, and the report must list it as unsupported rather than silently omit it (R8).

Requiredness is scoped to the affected operation or output (R11). Missing motion support must never block inspection or a still export that has no motion dependency (R11), which is the rule the four gates in packages/core/src/host-v1/apis.ts and schemas/tool.schema.json already follow.

Independent authors, Lolly Work instances and local people may publish suites and feature namespaces under their own namespace and version (D10, plan section 10.2). A suite namespace carries publisher identity. What names they take, and who may publish under lolly/*, is Q5 below, decided on 2026-10-16 by accepting its default.

Fixtures

A fixture must identify its inputs, its immutable dependencies, the reference environment, the features it claims, the expected artifact or typed result and the comparison rules (D3, tests/docs-shots-vector.test.ts).

Expected trees and traces may supplement output checks (D3). They must never replace them, because a valid JSON tree is not evidence of a rendered result (D3).

A fixture must declare its comparison regions itself (R9, C8). A region discovered from the difference between a candidate and its reference is exploratory evidence, never a region a fixture may declare.

ConformanceReportV1

The report shape follows Gateway API's conformance report, which the plan consulted as outside practice (R8, plan section 15).

draft shape

  ConformanceReportV1
    suite           { name, version }  a released version, never a branch or a commit
    implementation  { project, version, shell, mode, contact }
    date
    features
      core          { result: passed | failed, statistics, failedTests[] }
      extended      { result, statistics, supportedFeatures[], unsupportedFeatures[], skippedTests[] }
    environment     { fonts, shapingEngine, rasteriser, codecs, gpu, os }
    digests         { fixture, reference, comparator }

This is a draft shape, not a frozen type. Nothing is added to packages/core or schemas/ in this wave, and the plan's section 18 orders the counterexample fixtures in Proof cases before any of it is frozen.

A skipped test must never be recorded as a pass (R8). A report whose suite version does not match the fixtures it ran is invalid (R8). A report must bind to the fixture, reference and comparator digests as well as the suite name and version (C8). Reports are data, and publishing one grants no trust (R8).

AcceptanceV1

draft shape

  AcceptanceV1
    authority   { kind: work-administrator | local-person, instance, id, name }
    subject     { definition, instance, revision, operation }
    report      { suite, version, reportDigest }
    decision    accepted | accepted-with-exceptions | rejected
    exceptions[]  { target: ruleId | featureName, reason, by, date, scope }
    date

An acceptance must name the report it judged by digest (R8, C8). An acceptance describes exactly the revision named in its subject. When that revision changes the acceptance stays as history and must never describe the new revision as previously accepted (C7). Packaging, identity and migration states the same rule for receipts and signatures. An exception must name its target, its reason and the authority that granted it (R6, R8). An acceptance must never restate a measured result as passed when the report said otherwise (R8).

The governed authority is the Lolly Work administrator for the brand, recorded with the instance they administer (D12). Otherwise it is the local person using the client (D12).

Source review, use eligibility and delivery approval

AcceptanceV1 judges a conformance report. The DAM review requires separate subjects for catalog admission, upstream creative review, contextual use eligibility and final-byte delivery approval (plan 314; R8). A source review may have no conformance report. It must keep its own evidence and must never be turned into a suite pass.

A proposed source-review record must bind the decision to the source, exact reviewed file revisions or digests, reviewer authority and scope. A proof containing several files must identify its reviewed membership and revisions. Conditional approval, such as changes still required, remains conditional. Replacing a file or changing that membership must never transfer the earlier decision silently. These shared review records are not built by this amendment. Lolly Work currently treats replacement of admitted asset bytes as curation; its contribution approval is not a general review of every replacement (lolly-work/server/src/catalog/submit.ts).

Use eligibility also needs the requested operation, channel, territory and date where the applicable evidence requires them. Missing source data leaves an unknown answer; a general approved flag does not settle those conditions (packages/core/src/rights-v1.ts, policy). Signature validity, source assertions, review and eligibility remain independently reported facts.

Fixed delivery approval must bind the final staged output digest. Lolly Work's existing optional destination approval records that subject before provider egress (lolly-work/server/src/api/app.ts, lolly-work/server/src/delivery/types.ts). Ordinary exports do not acquire the same guarantee merely by referring to the same source. Each remotely processed derivative is a new artifact with its own checks and review requirements.

Adaptive publication would instead accept a pinned source and a versioned policy for permitted transformations. This is a proposed acceptance class, not a capability supplied by the draft. It requires its own validation and delivery contract before implementation and must never be presented as approval of exact delivered bytes. Existing fixed-byte delivery keeps its current meaning (R8; plan 300).

Review purpose and production proofs

A source approval assertion needs its purpose, activation target, origin and subject, including whether a person decided or an automated rule set the state (R8, R15; plan 315). Approval for an internal library must never imply approval for public delivery. A mapped flag with unknown subject stays an incomplete upstream assertion, even when the bytes have a verified digest.

Campaign review identifies exact variant revisions and proof-set membership. A responsive proof additionally identifies the viewport, rendering environment and sampled state it shows; a screenshot proves only that view. Changed copy, crop, bytes or membership needs the relevant decision again. Successful unchanged variants keep their historical decisions; a set-level policy may still require resubmission of the set. Review routing belongs to the host workflow and does not grant source, processing or delivery access.

The native 0.1 suites

SuiteScopeState in this draft
still-2d/1PNG, SVG and PDF, including print marks, bleed and CMYK output targets.The one native suite for 0.1 (R12).
motion-2d/1Timed appearance and duration for motion output targets.Staged after plan 196's TimelineV1 exists (R12).
present/1Interactive presentation and its state transitions.Stays an adapter until Agenda's portable runtime contract is typed (R12).

The evidence folder records 25 raw manifest rows declaring video and one declaring render.portable, raw directory counts taken before content profile resolution (C9), while the frame clock exists and TimelineV1 does not (R12).

Still-image fidelity

D8 allows a very low pixel difference, difficult to notice without close inspection. The plan reads that intent as no visible loss on close inspection (plan section 2) and keeps it, tested by three checks rather than by one number (R9).

Each fixture declares the parameters of all three checks, its fuzz band, its regions and the facts to read back from the artifact, and a claim must pass all three (R9). A fixture that cannot state one of them must say why in the fixture, and a report that rests on it must name the check that did not run (C8).

  1. Whole-image fuzz band. A per-channel maximum difference and a maximum count of differing pixels, each with a lower and an upper bound, in the form web-platform-tests uses (R9). A tolerance that must accept identical output has a lower bound of zero. This catches gross drift and admits anti-aliasing.
  2. Region-scoped structural checks. Regions the fixture declares, such as a legal line, a logo or chart labels, compared by ink ratio and structural similarity inside the region (R9). The fixture declares the region (R9). The comparator must never derive it from the difference it is measuring (C8).
  3. Semantic checks on the produced artifact. Text content, page count, chart data and layer inventory, read from the artifact or from an independently extracted representation, compared through compareSources in engine/src/compare.ts, which dispatches by mode to compareText in engine/src/compare-text.ts and compareStructure in engine/src/compare-structure.ts (R9). Comparing authored text with the same authored text would miss a broken exporter, so the comparison must read the output (C8).

Pixel counts alone must never decide a still-image claim (R9).

Three boxes, a whole-image fuzz band, region checks and semantic checks, each with its own arrow into a single result box, labelled whole image, ink and SSIM, and text and pages.Open this in Diagram Builder

The evidence and what it does not settle

A comparator probe rendered a QR code with a caption through the CLI shell as render A, then derived three variants from it. B removed the caption glyphs and left every box unchanged. C shifted the image by half a pixel. D rasterised at 1023 px and resampled to 1024 px. The caption region is 6.09% of the canvas. The probe is exploratory evidence and is not published with this specification.

PairChanged pixels, whole imageSSIM, whole imageChanged pixels in the caption regionInk ratio in the regionSSIM in the region
A vs B, caption gone1.46%0.96924%0.000.43
A vs C, half-pixel shift1.25%0.9688.8%1.000.89
A vs D, resampled1.29%0.9686.9%1.000.95

The three pairs sit within a quarter of a point of each other on changed pixels and within 0.002 on SSIM. A real content loss and two harmless changes are therefore indistinguishable by a whole-image number. The region-scoped measures separate them at once. A text-presence check on the artifact would separate them with no pixels at all, by construction rather than by measurement: B was made by removing the caption glyphs from the source, and the probe ran no such check.

The probe establishes the need for complementary checks. It sets no threshold (R9, C8).

It also does not establish a validated combined comparator or cross-shell conformance (C8). Its region was discovered from the A and B difference, which a fixture must never do (C8). Its fonts were whatever the rasteriser found on one machine, and resvg-js does not report which files it resolved. The raster was composited onto white. The structural similarity is one small implementation. Nobody decoded the QR afterwards, so "nothing lost" in C and D describes the source edit only.

A second measurement, from the drawing compiler's fidelity harness in plan 295 (tests/design-draw-fidelity.browser.test.ts), uses declared regions as this chapter requires. It mounts fixture pages through the real Design tool and draws them in Chromium beside the SVG compiled from the same rows. Shapes, paths and effects give zero differing pixels. Live text differs only where runs meet, by a fraction of a pixel (0.49% of a mixed-style text box). Outlined text cannot match pixel for pixel, because the font rasteriser thickens glyph stems that a path fill does not: on macOS Chromium a page of outlined text differs from the browser's text in 5,578 pixels while every word is present and in place. The harness therefore declares one region per laid-out run, taken from the drawing operations and never from the difference, and compares the ink each side adds to the page without its words, and the centroid of that ink. Outlines measured 2.5% to 17.4% less ink with centroids at most 0.60 px apart, and on every run the harness also draws two rejected examples, a missing word and a word moved by 2 px, and fails if its limits let either pass. Those limits come from one machine, one accepted set and two rejected examples, so they are the harness's own and are not yet a number for this specification.

The same harness judges the web shell's DOM walker, the fallback for pages the drawing operations do not admit, against the same Lolly Design pages and the same regions, so the incumbent is a candidate like any other rather than the reference. The walker passes the shapes, paths, effects and text pages, and fails eight regions the drawing operations pass: a picture fitted by fill (27% of its region differs) or none (47%), because the walker writes no aspect rule for either; a picture whose fitted edge falls between pixels, which Chromium snaps and the walker does not; and a row the frame clips inside its border, which the walker clips at the frame's outer corner instead. A reference renderer is how such faults surface, and an incumbent export cannot be its own reference.

A PDF is drawn by a different rasteriser from the reference. The harness rasterises each PDF page with Poppler at one pixel per CSS pixel and compares the result with the same Lolly Design pages and regions. At the Chromium threshold, every curved region differed by about 1% for both the drawing operations and the walker, because Poppler covers an edge pixel differently. For PDF, a pixel therefore counts as different only when nothing within 1 px in the other image matches it, and a half-covered edge pixel may differ by up to 40 levels (the measured worst case was 28). A picture drawn at another size than its own is resampled by the viewer and may differ in 2% of its region (measured: 1.3% for both renderers). The harness proves those allowances still catch a fault: the same page with one rectangle moved by 2 px must fail its region, and does. With these limits the operations' PDF passes every region, and the walker's PDF still fails the clipped rows and the frame picture's corner. The numbers come from one machine and one rasteriser; they are the harness's own, not this specification's.

The Node PDF path adds artifact checks, not another fidelity reference (tests/design-ops-pdf-node.test.ts). Its tests reopen a two-page outlined RGB Lolly Design PDF and inspect the authored page sizes, metadata, output intent, page boxes and absence of artwork fonts. They verify bleed and marks on each page, and deliver a marked PDF while the CLI's browser endpoints are unavailable. The final Content Credential is read back and its data hash and signature are checked against the delivered bytes. A Unicode provenance label that Helvetica cannot encode exercises a real finishing failure: the direct Node call keeps the original WinAnsi error as its cause, a later valid export succeeds, and the CLI with --html-fallback writes neither PDF nor HTML and never escalates to a browser. Separate finishing tests decrypt the AES-256 output's strings and metadata to verify the producer, retained output intent and withheld PDF/X claim (tests/pdf-finishing.test.ts). These observations do not establish CMYK fidelity or independently certify PDF/X compliance.

Outlined strike-through adds a decorated reference page with variable weights, italic and mono runs, wrapping, colours, opacity, rotation and clipping. The same harness compares outlined SVG and Poppler-rendered PDF against the real Lolly Design HTML page. Each declared line region measures the added decoration ink against a separately rendered undecorated page, so the glyphs cannot hide a missing line. Both formats pass the existing 25% ink and 1 px centroid bands and reject lines removed or moved by 2 px. The geometry follows the pinned Chromium reference's CSS auto policy, including its local solid-line pixel rounding, with ascent read from the resolved HarfBuzz font instance. These observations qualify those fixtures on this Mac; they do not establish a universal browser policy. Pure italic or mono pages still select the web DOM walker when its vector-font registry cannot resolve the base face; mixed-base fixtures establish the admitted Node/web SVG byte parity without changing that refusal.

Outlined underline adds a second decorated page: descenders at several weights, italic and mono runs, a coloured right-aligned run on a fractional baseline, a strike and an underline on one run, a clipped translucent box, a wrapped run whose hanging space is underlined, a 15 px run with a one-pixel line, and a rotated row. The same declared line regions and undecorated grounds judge it in Chromium 153.0.8010.12 and Poppler, within the same 25% ink and 1 px centroid bands. Measured on this Mac, the SVG lines differ from the reference by at most 0.8% ink and 0.55 px, and the PDF lines by at most 13.2% and 0.56 px. Each format rejects a removed line, a line moved by 2 px and a line drawn straight through the descenders, so the check sees skip-ink and not only the line's place. The web DOM walker fails seven of the page's regions where the drawing operations pass. These observations come from one machine and one fixture page.

SVGZ tests decode the gzip bytes and compare the result with SVG from the same operations, including dimensions and metadata (tests/design-ops-svg-node.test.ts). Transport tests check that an unadmitted SVGZ page or live-text PDF makes one complete browser request, preserves the returned bytes and reports its admission reason (tests/design-session-transport.test.ts). Snapshot tests prove that a beforeExport mutation cannot rewrite the authored values passed to the bridge. Rejected integrity cases remain errors rather than admission findings. Rejected hook cases write no artifact, including an HTML substitute when --html-fallback was supplied. These are maintained implementation checks, not a released still-2d/1 suite or the full draft receipt; the reference, environment and comparator requirements above still apply to a conformance claim.

Thresholds must be calibrated on accepted and rejected examples before any number enters this specification (R9). Masks must be explicit and justified, and human review sets and reviews the bands and must never replace the comparator (R9, plan section 11.2).

The output-validation boundary

An output-validation boundary sits after rendering and before delivery of any artifact that claims acceptance (C8). An artifact that has not passed that boundary must never be delivered with an acceptance claim attached.

Comparison for a claim must run at native raster scale with pinned fonts and rasteriser, with declared colour handling and alpha treatment (R9, C8). The preview-grade comparison in engine/src/compare-visual.ts samples at most 768 px per edge with nearest-neighbour sampling on white, and it states that limit in its own result. It stays a preview tool and must never decide a conformance claim.

An undetermined result must never be a pass (R9, C8). compareSources in engine/src/compare.ts returns undetermined when a budget cut the comparison short, and engine/src/compare-budget.ts is where that budget is spent. Harmless truncation of displayed detail must be distinguished from incomplete comparison coverage (C8).

PNG alpha, PDF page boxes and colour intent must each have their own checks (C8). A white RGB fixture says nothing about CMYK print conformance, and a suite must never generalise from one to the other (C8).

A run whose text stayed a live <text> element rather than an outline must record that in its receipt, and it must never claim font-independent fidelity (R15). Evaluation and receipts specifies the receipt.

Graphics devices and numerical hosts

The first WebGPU and Rust kernel work (plan 295) supplies evidence for the fuzz band and its limits, added on 2026-10-07. It sets no threshold for any suite.

Motion fidelity

Appearance is compared at normalised progress points, and total duration and the time mapping are compared separately (D9, plan section 11.3).

Sampling must include the start, the end, scene and transition boundaries, authored events and samples immediately around discontinuities (D9). Each suite must bound its sampling interval, with denser checks for fast motion (D9).

Loop behaviour and audio timing must be tested where they are claimed (D9).

Sampling is evidence at the tested points. It does not prove the absence of a defect between them, and the suite must state that limit (D9).

Editable round trips

Visual similarity is necessary and insufficient for an interchange claim (R14). A raster fallback may claim appearance and must never claim editable-vector preservation (R14).

Each route must declare preservation separately for six dimensions: appearance, editable objects, semantic data, interactions, timing and constraints (R14, engine/src/penpot-file.ts).

A route is tested by a meaningful edit after import, followed by an export, followed by a comparison across all six dimensions (R14). The required edits are moving a layer, changing a text run and swapping a token.

The first routes are Q1 below. Andy accepted them on 2026-10-16 as the table states them.

PilotFirst routeWhat exists today
Lolly DesignTo and from Penpot.The .penpot writer in engine/src/penpot-file.ts; the editable import is parsePenpotBinfile in shells/web/src/views/design-import.ts over the pure mapper engine/src/design-map.ts (the Unpack reader in shells/web/src/views/penpot-import.ts extracts, it does not round trip).
Lolly DesignTo native PPTX and back.The deck reader readPptx in engine/src/pptx-read.ts, reached by the Rebrand source path in packages/node-shell/src/pptx.ts.
ChartTo and from CSV and JSON data, plus the SVG semantic overlay.The portable chart contract in packages/core/src/chart-v1.ts.
AgendaTo ICS, portable HTML and PPTX.The outputs listed in docs/agenda.md.

Today's composes and compose.renderUrl paths produce a flattened image (R14). That image may pass an appearance check and must fail an editable round trip (R14). Proof cases carries it as a failing case, not as a pilot.

Conformance without a picture

An operation that draws nothing is still measurable (D11, R13). The three paragraphs below state what its suite tests.

For Lolly Rebrand, compare the approved changes against the rebrand plan, the protected content against the source and the output against a reference for that rebrand plan (R13). Partial completion is read through OutcomeV1 as Operations and outcomes specifies.

For Unpack and the other route operations, compare typed results, resource inventories, declared effects and failure behaviour (plan section 11.5, R13). A fabricated preview must never stand in for the real outcome (R13).

For interactive Agenda, test search, navigation, calendar output, state transitions and accessibility (plan section 11.5).

Material that exists today

None of this is a suite yet. These are the parts a suite would be assembled from, and each states its own limit.

PieceWhat it establishesWhere
lolly smokeEvery tool in the active content profile renders at its defaults to its first Node-native format. A skip is never counted as a pass.shells/cli/src/smoke.ts
Export characterisationA fixed matrix of nine tools, chosen so every export branch is exercised, is rendered twice through the real web export path and classified as hashed or size-banded. Same machine only.scripts/characterize-export.ts
Comparison modulesBounded text and structure comparison with stable-id moves, an explicit undetermined result and a preview-grade visual mode that states its limits.engine/src/compare.ts, engine/src/compare-structure.ts, engine/src/compare-visual.ts, engine/src/compare-budget.ts
Host conformance kitA live host carries the members the contract declares, checked at run time rather than at compile time.packages/core/src/host-conformance.ts
Docs shotsA vector baseline is compared exactly, and every raster baseline carries a written reason in one allowlist that fails in both directions.tests/docs-shots-vector.test.ts

When a suite exists, its first reports come from this repository's continuous integration, where the render gate, lolly smoke and the export characterisation script already run. A page listing each shell's results is the cheapest trust asset the suite can carry (plan sections 11.1 and 18).

This draft adds no conformance or reproducibility claim beyond the ones docs/determinism.md already makes, and it states no threshold and no latency target (C8).

Open points

The other questions are carried in other chapters, each decided on 2026-10-16 (plan section 16). Q2 and Q3 are in Constraints, authority and local choice. Q4 is in Operations and outcomes. Q6 is in Status and open points.

Precedents