JonnyJr

My personal tooling: an agent bench of Claude skill specs with explicit refusal conditions, plus an Obsidian second brain.

In useContinuously updated · agent evals not yet graded

Division: Systems
Role: Solo · personal tooling

Source

Two halves. The agent bench holds domain-specialised definitions for the work I actually do — software, hardware, coursework, finance, legal review — and the interesting part is that each one declares when it must refuse. The second brain is the Obsidian vault where my notes, coursework, and project thinking live.

Claude Code Skills (SKILL.md)
Claude API · Managed Agents
Python
YAML manifests

01Problem / user need

A general-purpose assistant fails on consistency, not capability: the same question asked twice gets differently shaped answers, domain guardrails have to be restated every session, and output formats have to be re-specified each time.

The fix is to make each agent’s role, activation trigger, output format, capability boundary, and refusal conditions explicit — and version-controlled.

02Requirements

#	Requirement	Acceptance criterion	Status
R1	Guardrail enforcement	Agents refuse when their declared condition is unmet, and only then	Adversarial fixtures written; runs not yet graded
R2	Non-overlap	Overlapping triggers resolve by a declared precedence	Routing matrix written
R3	Actual use	Invocations per agent over 30 days, long tail included	Not yet measured
R4	Maintenance	Specs under version control	Met

03Architecture

┌──────────────────────────────────────────────┐
│ skills/<agent>/SKILL.md                      │
│ role · trigger · output format · capabilities│
│ REFUSAL CONDITIONS  ← the design feature     │
└───────────────────┬──────────────────────────┘
                    │ manifest.yaml (single roster)
         ┌──────────┴───────────┐
┌────────▼─────────┐   ┌────────▼──────────────┐
│ Claude Code      │   │ Managed Agents        │
│ local skills     │   │ (deploy scripts)      │
└──────────────────┘   └────────┬──────────────┘
                                │
                     ┌──────────▼───────────┐
                     │ run_evals.py         │
                     │ fixtures → recorded  │
                     │ transcripts (ungraded│
                     └──────────────────────┘

04Design decisions

Explicit refusal conditions

Chosen: Agents declare when they must not produce output. The trading agent will not treat a setup as a thesis without an invalidation level, and defaults to analysis only; supervised execution is opt-in, gated per session and per order.
Tradeoff: A risk control is encoded in the tool rather than left to the user’s discipline in the moment. Cost: a markdown spec states intent; it doesn’t enforce it. Only testing shows whether the agent actually refuses.

Scope limits in regulated domains by default

Chosen: Finance, tax, legal, and trading agents state that they are not professional advice and flag what needs a licensed professional.
Tradeoff: This is a stated constraint, not a technical guarantee — so it is described as one.

Many narrow agents instead of a few broad ones

Chosen: Narrow specs that each carry their own domain conventions.
Tradeoff: Conventions that would conflict if merged can coexist. Cost: trigger collisions and maintenance load, handled by a routing matrix with explicit precedence.

4.5Scope & role

Solo. Personal tooling: no users, not distributed, not a product. The second brain is complete as a system and updated continuously; the agent bench is in active use.

05Implementation

Markdown skill specs with YAML frontmatter, a shared commons spec for evidence and write-gate policy, and a single manifest as the roster.
Python scripts to build, upload, and deploy the skills as Managed Agents, and to export them for a second editor.
Eval fixtures per agent covering trigger, boundary, and freshness cases. The finance, legal, trading, and code-audit agents include adversarial cases.
An eval runner that executes fixtures against the deployed agents and records full transcripts. It deliberately does not grade.
Second brain: an Obsidian vault, maintained continuously. It’s private, so it isn’t linked here.

06Verification

PendingGraded results aren’t published yet. Next: score the recorded adversarial transcripts as refusal rates on should-refuse vs. should-answer inputs, per agent with a declared refusal condition. Any productivity figure (“saves N hours”) is deliberately not claimed.

07Failures / iterations

PendingNot yet documented: agents written but rarely used, duplicate-looking pairs, and specs rewritten after bad output.

08Limitations

Personal tooling, not a product. The second brain is private, so nothing about it can be verified from this page.
Specs are intent, not enforcement, until the evals are graded.
No measurement that any agent produces better output than the same model without the spec.
Inherits the underlying model’s limitations, including hallucination.
The count of specs is not a measure of capability.

09Repository / demo / report

Repositorygithub.com/jonnyterrero/workflows-and-automations/tree/main/agents/agent-team
Agent benchgithub.com/jonnyterrero/workflows-and-automations/tree/main/agents/agent-team
Eval fixturesgithub.com/jonnyterrero/workflows-and-automations/tree/main/agents/agent-team/evals
Routing matrixgithub.com/jonnyterrero/workflows-and-automations/tree/main/agents/agent-team/docs/ROUTING_MATRIX.md

← All systems