Gate
A pass/fail check that rejects a candidate.
Why. Rules that can't be traded off shouldn't be scores.
Boundary. A gate never contributes to a total.
Best-of-N selection
Pick the best of several model outputs without letting one judge decide alone.
Generate, score and select the best of N candidates with pluggable code or prompt scorers, and record why the winner won.
Prevents: A failed judge scores a good candidate 0, a hard rule gets averaged away, the expensive judge reads everything, and nobody can explain the winner later.
Why it exists
The failure appears after the happy path.
Best-of-N is dependable but every project rewrites it by hand with the same bugs.
Scope. One reusable selection step and an experiments harness; not an orchestrator or model client.
Five-minute orientation
See the smallest complete path.
pip install "git+https://github.com/honeworks/hone-select"Real output
What it looks like running.
Architecture
The architecture is a set of promises.
A small engine over pydantic: generator, gates, staged scorers, aggregation, policy and fallback, with ports for judges, text clients, embedders, record sinks and score caches. Adapters for OpenAI-compatible servers, LangChain and hone-models are optional extras.
- A scorer that fails gives None with an error, never 0. Inspect test
- Gates reject; they are never averaged into scores. Design note
- A pairwise judgement counts only when both orders agree. Inspect test
- Every run can be explained from the span store alone. Inspect test
Core concepts
The few concepts you need before reading the code.
Operations
What happens after install.
Records every step to a local SQLite span store; `hone-select explain` and `hone-select dashboard` read it.
Current boundary
What this project does not solve.
Alpha. LLM judges stay noisy; pairwise agreement and cross-family judges reduce, not remove, bias. Best-of-N multiplies generation cost.
Near-term roadmap. Follow the design history in design/changes/.
What it implements
- gates
- cheap-first scoring cascades
- missing scores as None
- pairwise tie escalation in both orders
- budgets and score cache
- span records with explain
- experiments with plan and approval
- local dashboard
Engineering checklist
What the repository ships.
- Quickstart
- Present
- Success tests
- Present
- Failure tests
- Present
- Architecture notes
- Present
- Security notes
- Not published
- Runbook
- Not published
- Limitations
- Present
- Changelog
- Present
Need the control, not just the component?
Fit it to the system that has to survive.
The repository exposes the mechanism. Production work is defining the permissions, data, failure costs, evidence, and owners around it.