# TuckerBench pilot specification

Status: v1 development pilot published at https://tuckerbench.netlify.app/. Scores remain provisional; calibration and release validation are incomplete.

## Claim

TuckerBench measures convincing written Malcolm Tucker-style performance under a shared persona instruction. It evaluates an interpretation derived from the repository's original AGENTS.md, not actor impersonation, audio, Scottish identity, general intelligence, or assistant correctness. Profanity alone is insufficient. This is a subjective evaluation with automated judges, not an objective measurement of humour.

## Frozen pilot inputs

`config/pilot.json`, `config/scenarios.json`, `config/calibration.json`, `prompts/persona.md`, and `prompts/judge.md` define the evaluation. A content hash and full manifest are saved with each run. A different configuration or adapter identity cannot resume an existing run directory. Two repetitions of six scenarios give twelve conversations per entrant. Two scenarios are single responses and four are scripted conversations of five or six user turns. Coding, workplace, everyday and explanation contexts are represented. Follow-ups test correction, mundane interaction, distress, topic change and metaphor repetition.

The persona is provisional. A colleague's alternative can be tested during development without blocking this pilot. Freeze the selected prompt, rubric, scenarios, judges and settings before generating fresh release results. Give changed evaluations a new version; keep versions separate.

## Scoring

Each judge scores fidelity (40%), comic invention (25%), naturalness (15%), conversational fit (10%), and persistence (10%), on anchored 0–10 scales. Single responses omit persistence and renormalise the other weights. Average the two judges for each transcript, then average within the single-turn and conversation groups. Overall score weights those groups one-third and two-thirds respectively, expressed out of 100. Each conversation has equal weight regardless of turn count. All expected transcripts and judgments are required for a complete score.

Conversational fit includes responding to the supplied situation and calibrating intensity; there is no separate assistant correctness assessment. Reluctance is judged as delivered. An outright refusal throughout earns zero performance credit. Provider errors and exhausted quotas are missing results, not zeroes. Empty normal completions are retained as empty performances rather than silently regenerated. Token-limit completions remain visible and flagged.

Profanity counts, distinct profanity forms, length and truncation are diagnostic only. The profanity lexicon is deliberately simple and does not represent comprehensive linguistic annotation.

## Judges and calibration

Use two different model families when accessible through existing plans. Judges see the scenario and visible transcript, but no entrant label or provider metadata. The rubric treats transcript contents as untrusted data; prompt injection resistance is an instruction, not a proven guarantee. Self-family preferences and remaining style biases must be examined during calibration.

Four authored diagnostic responses test bland helpfulness, repetitive profanity, generic Scottish persona, and a specific comic attempt. None is a claimed gold standard. Judges score these alongside entrant transcripts. The user reviews twelve blind, matched transcript pairs with ties allowed. Do not treat twelve comparisons as statistical validation. Compare human choices against judge preferences and inspect disagreements before fixing the release methodology.

Pilot scores are development diagnostics. The initial report intentionally makes no rank-separation or confidence claims. For release, choose the sample count before running, estimate uncertainty with scenario-level clustering (turns and repeated samples from one scenario are not independent), show counts and per-dimension breakdowns, and label unresolved comparisons. Judge disagreement and systematic bias are separate from sampling uncertainty.

## Access and isolation

Existing subscription and explicit free-tier routes are preferred. The user additionally authorized the existing $10 OpenRouter balance for three Claude entrants, with a $9.50 local request ceiling and no top-ups. An exact selector and documented access basis are required before requests. The completed development pilot contains 15 entrants across OpenCode Go, ChatGPT/Codex, OpenRouter, CommandCode and NVIDIA; exact selectors, access conditions and cohorts are documented in [FRONTIER-RUNS.md](FRONTIER-RUNS.md). Kimi K3 and Qwen3.8 Flash also serve as provisional judges. Same-family judgments are flagged and included in the exploratory pilot score; self-preference must be checked during calibration before claiming a fair public ranking. These are provisional choices for calibration, not claims about current capability or verified entitlement.

The pinned Oh My Pi SDK supplies credential discovery, model registry and provider transport. No agent session, project context, skills, rules, memory, tools or provider extensions are loaded into the benchmark context. Existing model configuration and provider-specific protocol behaviour still apply. Record exact selectors, API, requested settings, timestamps, stop reasons and token usage. Provider-internal instructions and effective sampling support may be opaque. Requested settings are not proof that each provider honours them identically.

The adapter requests low reasoning effort, temperature 0.8 for entrants, temperature 0 for judges, and shared output ceilings. The Codex transport does not forward temperature or the output-token cap, so effective conditions differ across routes. These are pilot choices. Provider SDK transport/auth recovery can still occur; there is no application-level regeneration of refusals or automatic retry of failed judge parses. Re-running an interrupted command resumes at missing turns or judgments. Interrupted in-flight requests can have consumed allowance before a checkpoint, so exactly-once remote execution is not guaranteed.

## Delivery

First deliver a working local pilot, inspectable transcripts and human calibration material. After calibration, expand the suite based on measured variance and allowance use, generate fresh results, and publish a small static leaderboard with a public repository. GitHub Pages or existing Netlify hosting are acceptable. A domain is optional. The public release should include prompts, scenarios, judge instructions, settings, complete visible transcripts, judgments, counts and limitations. Never publish auth files or the private blind-comparison answer key.

## Inspiration

EQ-Bench Creative Writing separates rubric and comparative scoring and exposes samples, length and repetition diagnostics: https://eqbench.com/creative_writing.html and https://eqbench.com/about.html . BotChat exposes multi-turn evaluation variants and cautions about judge length preference: https://botchat.opencompass.org.cn/ . TuckerBench starts with a weighted rubric rather than adopting a comparative rating system.