evaluation · grammar v2.0.0 · weights w-2026-05-001

What is measured — and what is not yet.

This page reports two different things and keeps them apart. Extraction fidelity: the extractor reads a brand's own homepage and is compared, per axis, with the hand-graded catalog entry for that same brand — a test of the parser, published below. Judgment validity: whether a score on a new page tracks what independent designers judge — a test of the judgment, pre-registered and not yet measured. Every fidelity number is reproducible from open code. No validity number is claimed.

The Golden Set — five hand-graded anchors.

These five brands define what graded means on tasteHQ. The rates below are extraction fidelity: how faithfully the extractor recovers each hand-graded axis from the brand's own homepage. They are not a measure of agreement with designers about other pages.

Anthropic

Reference

Word-level underlines replace color as the sole emphasis mechanism in display-scale headlines.

Axis Extraction vs hand grade
Highest fidelity
palette.warmth 1.00 · exact
surface.ground 1.00 · cream ✓
emphasis.mechanism 1.00 · underline ✓
Lowest fidelity
surface.radius 0.50 · pred 12–16px / actual 8–10px
type.pairing 0.00 · pred single-family / actual serif/sans
voice.archetype pending · awaits ANTHROPIC_API_KEY

Apple

Reference

Full-viewport hero: single product photo centered on white, one-line SF Pro Display name, nothing else above the fold — the page is a cinematic catalog, not a website.

Axis Extraction vs hand grade
Highest fidelity
surface.ground 1.00 · white ✓
palette.contrast 1.00 · high ✓
surface.borders ~0.92 · hairline (25-brand)
Lowest fidelity
imagery.strategy pending · n=1 only (Acne)
motion.budget pending · Playwright required
voice.archetype pending · per-brand tau to come

Stripe

Reference

Animated gradient mesh aurora runs behind the hero — brand colors signal financial infrastructure as a living, breathing system.

Axis Extraction vs hand grade
Highest fidelity
type.pairing 1.00 · sans/sans ✓
palette.contrast 1.00 · high ✓
surface.ground 1.00 · white ✓
Lowest fidelity
palette.warmth 0.00 · accent recovered, hue-bucket wrong
palette.saturation 0.50 · ordinal map needs recalibration
motion.budget pending · pervasive vs static-CSS abstention

Linear

Reference

Every action has a keyboard shortcut shown inline as a kbd-styled badge — the keyboard is treated as the primary input device, not an accelerator for power users.

Axis Extraction vs hand grade
Highest fidelity
surface.dark_mode 1.00 · dark-first ✓
surface.borders ~0.92 · hairline (25-brand)
palette.contrast 1.00 · high ✓
Lowest fidelity
voice.sentence_length~0.56 · sparse <main> on SPA shell
motion.character pending · scroll-recording analysis
voice.archetype pending · per-brand tau to come

Acne Studios

Reference

Helvetica Monospaced Pro at 9px, 0.033em tracking, caps throughout — one face, one size, one weight across the entire UI.

Axis Extraction vs hand grade
Highest fidelity
type.tracking 1.00 · loose, em-median exact
imagery.strategy 1.00 · photographic ✓ (only n=1 hit)
surface.ground 1.00 · white ✓
Lowest fidelity
emphasis.mechanism 0.00 · pred underline / actual scale (widget CSS swamp)
motion.budget abstain · 56 Shopify @keyframes vs catalog "none"
palette.warmth / saturation0.00 · no :root vars, fallback poisoned
Cosign spec

Every brand entry on tasteHQ ships with canonical: false in /.well-known/taste/<slug> — we mark ourselves a mirror. The brand's own homepage is the source of truth.

When a brand publishes the same shape at <brand>.com/.well-known/taste/<slug> with canonical: true, tasteHQ's mirror auto-marks itself derivative and links upstream. No vendor lock-in. Read the spec at /cosign.

Extraction fidelity · Golden Set
0.839
5-brand reference set, round-trip extraction vs hand grade on scored axes. Not a validity figure.
Mean fidelity · 25-brand test
0.743
Held-out catalog subset (≥8 graded axes, Surge 3b · 2026-05-17).
Axes at target
9/17 scored
Meeting the per-method RFC-001 §V threshold.

Judgment validity — agreement with independent designers

Status: not yet measured. Nothing above tests whether a score on a new page tracks what competent designers judge. That study is pre-registered here before any data is collected, so the criteria cannot be tuned to the result. No agreement number will be published without the human–human inter-rater band beside it.

Stimuli
For each Golden Set brand, 8 page variants from one scaffold: 1 faithful, 5 single-axis ablations, 2 multi-axis drifts — 40 total. Stage 1 tests axis sensitivity (discriminant validity), not ecological validity; stage 2 uses real candidate pages.
Raters
8–12 working designers, blind to the canonical variant and to the judge's output; recruited outside the Studio W230 network where possible; familiarity recorded and stratified.
Task
Pairwise "which is more on-brand for {brand}?" plus a pass / revise / reject per stimulus. Sessions capped at 20 pairs.
Reliability ceiling
Krippendorff's α on verdicts, Kendall's W on rankings — the human–human band the judge is held to.
Judge–human
Spearman ρ vs Bradley–Terry human ranking; AUROC of pass/reject vs human majority; per-axis discriminant validity (does ablating axis k move the judge's axis-k score the way humans penalised?).
Pass criterion
Judge–human ρ falls within the 95% CI of mean human–human pairwise ρ. A judge far above the human band is reported as a flag, not a success: it would suggest the stimuli are easier for a rule engine than for people.
Held-out
At least 5 catalog brands never touched by learn_weights.py are reserved as a permanent validation set.
Committed in advance
If human α is low (< 0.4), the claim becomes "more consistent than designers, and auditable", not "agrees with designers". Logged as a dated amendment to the protocol before any data was collected, so the stimuli cannot be tuned toward a flattering ρ.
Protocol
Judgment Validity Protocol v1 — pre-registered 10 Sep 2026, amended 5 Oct 2026, no data collected yet. The report will be published at docs/VALIDITY-REPORT-v1.md and linked here whatever it shows; if the judge fails the criterion, this page will say so.

Known weak axes — dated

The fidelity numbers on this page were measured on 17 May 2026 (weights w-2026-05-001; calibration note dated May) and have not been re-measured since. Six scored axes sat below 0.50. Listing them beats hiding them.

palette.warmth · 0.25
hue-bucket cutoffs — in flight since May.
whitespace.element_gap · 0.38
blocked: needs a DOM heuristic (rendered pass).
type.pairing · 0.40
Extractor fix landed 10 Sep 2026 (CSS var() resolution, body-font capture, system stacks as kind evidence). The May number is now stale in the other direction and awaits re-measurement — it is deliberately not revised by hand.
surface.shadows · 0.49 · palette.saturation · 0.50 · others
See the full per-axis table below; statuses are as of May.
Weights
w-2026-05-001 is the only weights version ever published; the self-learning loop has not yet run on real evaluation records (kendall_w is null on every axis).

Extraction fidelity — methodology, in three lines

Per-axis fidelity is the round-trip score: fetch a brand's homepage, project raw CSS/HTML signals into grammar axes, then compare against the hand-graded catalog value. Ordinal axes use normalized distance; nominal axes use binary match.

Golden Set
anthropic · linear · stripe · acne-studios · antimetal — 5 brands, fully hand-graded.
Test set
25 catalog brands with ≥8 graded axes — held out for fidelity measurement.
Target bands (RFC-001 §V)
det ≥ 0.85 mix ≥ 0.70 model ≥ 0.60

Full per-axis extraction fidelity

Reference material — the 30 axes across 8 categories used by the judge. Fidelity is the round-trip score from tools/spike_roundtrip.py on the 25-brand test set; bold scores are at-target. Axes marked pending are honest non-predictions — the extractor abstains rather than guesses. For per-brand extraction-vs-hand-grade comparisons, see the Golden Set above.

Axis Label Scale Extraction Fidelity Status
at-target meets RFC §V band in-flight close, scheduled fix blocked named external blocker pending not yet measured

Reproduce this

Every figure on this page is generated by code you can run. No closed evaluation set, no secret prompts.

  1. Read the eval pipeline spec — what is being measured, and why, in detail.
  2. Re-run round-trip on the 25-brand set: python tools/spike_roundtrip.py --weights w-2026-05-001
  3. Inspect per-brand breakdown and known blockers in the calibration note.