The Self-Improving Prompt Loop: How Telemetry Closes the Gap Between Good and Great

Support escalates: “the contract summarizer got worse last week.” You open the agent. The system string is a wall of prose with no version. Token spend is up. Negative feedback is anecdotal. There is nothing to query because the prompt was never an identity—just bytes that went into a completion call and vanished.

HUMΛN’s prompt path closes that gap: URI identity, call metadata threaded into provenance, feedback signals, performance snapshots, model affinity, and a Prompt Refinement Agent that proposes changes. Humans still approve what matters—especially core prompts.

Honesty: telemetry + refinement APIs and CLI flows described here ship as protocol surfaces; treat affinity-biased routing and org autoTune as governed features you enable deliberately—not as “the agent rewrote production unsupervised.”

The Observability Gap

To understand why this matters, consider what's invisible in a typical AI system:

• Which prompts are used most? No idea. They're inline strings without identity.
• Which prompts cost the most? Unknown. Token consumption isn't tracked per prompt.
• Which prompts produce the best results? Unknowable. There's no feedback mechanism.
• Which model works best for which prompt? Nobody's measuring. Models are assigned statically.
• When a prompt degrades, how do you know? You don't. Until users complain.

This isn't a tooling problem. It's an architectural problem. Without prompt identity, there's nothing to observe. Without telemetry, there's nothing to improve. Without a feedback loop, improvement is manual, ad-hoc, and slow.

Protocol-Level Prompt Telemetry

In HUMΛN's HAIO protocol, prompt telemetry is a MARA (Multi-Agent Runtime Architecture) concern. It's not a Companion feature. It's not an add-on. Every agent that uses ctx.prompts automatically participates in the telemetry pipeline.

The system captures four types of data:

1. Prompt Call Metadata

When an agent loads and composes prompts, the system generates PromptCallMetadata — a structured record of exactly which prompts were used:

interface PromptCallMetadata {
  promptUris: string[];        // Full URIs of all prompts used
  promptVersions: string[];    // Their versions
  layers?: PromptLayer[];      // Full provenance: scope, URI, version per layer
  compositionMethod: string;   // 'compose' | 'effective' | 'manual'
  variablesUsed?: string[];    // Which template variables were substituted (not values)
  estimatedTokens?: number;    // Pre-call estimate
  orgId: string;               // Org context
}

This metadata is threaded through the LLM call via ctx.llm.complete({ promptMetadata }). It travels into provenance alongside the model selection, token usage, latency, and cost. After the call, you can trace exactly which prompts contributed to any response.

2. Feedback Signals

Agents and users can report prompt effectiveness through structured feedback:

await ctx.llm.recordPromptFeedback({
  provenanceId: result.provenanceId,
  signal: 'positive',    // or 'negative', 'rephrase', 'correction'
  source: 'agent',       // or 'user', 'system'
  detail: 'High-confidence analysis with structured output',
});

Four signal types capture different quality dimensions:

Feedback signals are lightweight and can be generated automatically by agents (based on downstream quality metrics) or manually by users. The key insight: agents can evaluate their own output quality and feed that signal back to the prompt that produced it.

3. Performance Snapshots

Telemetry data is aggregated into PromptPerformanceSnapshot — a comprehensive view of how each prompt is performing:

interface PromptPerformanceSnapshot {
  promptKey: string;
  window: { start: string; end: string };
  stats: {
    totalCalls: number;
    avgTokens: number;
    avgCostUsd: number;
    avgDurationMs: number;
    positiveSignals: number;
    negativeSignals: number;
    rephrases: number;
    corrections: number;
  };
  modelBreakdown: Array<{
    modelId: string;
    callCount: number;
    avgLatencyMs: number;
    avgCostUsd: number;
    feedbackScore: number;
  }>;
  recommendedModel?: string;
}

This answers previously unanswerable questions: How many times was this prompt used last week? What did it cost? Which model performed best? Is the negative signal rate trending up?

4. Model Affinity

Here's where it gets interesting. Since HUMΛN's Model Registry dynamically selects models based on capability and cost, the same prompt might run on different models across different calls. Over time, the system accumulates data on which (prompt, model) pairs produce the best results.

interface PromptModelAffinity {
  promptUri: string;
  modelId: string;
  sampleSize: number;
  stats: {
    avgLatencyMs: number;
    avgCostUsd: number;
    successRate: number;
    positiveSignalRate: number;
    negativeSignalRate: number;
  };
  affinityScore: number;    // Quality-weighted, cost-adjusted score
}

This affinity data feeds back into model routing as a soft preference signal. When a prompt has strong affinity data showing that GPT-4o produces better results than Claude Sonnet for this specific use case, the Model Registry biases toward GPT-4o — without hard-pinning, so capability-first routing still applies.

The result is a system that gets better at matching prompts to models over time, entirely from empirical evidence.

The Prompt Refinement Agent

Data without action is just a dashboard. The Prompt Refinement Agent is what closes the loop between observation and improvement.

This is a scheduled operations agent — not a human, not a cron job, but a real agent running in the MARA runtime with its own identity, delegation scopes, and audit trail. It:

1. Monitors telemetry across all prompts in its scope
2. Identifies underperformers using configurable thresholds:
   • High negative signal rate (> threshold)
   • Low success rate compared to peer prompts
   • Cost outliers (same task type, 3x the tokens)
   • Model affinity mismatches (prompt consistently assigned to suboptimal model)
3. Generates improvement proposals (PromptChangeProposal):
   • Analyses telemetry patterns
   • Uses an LLM to draft improved prompt text
   • Calculates risk level based on prompt scope and usage volume
4. Surfaces proposals for human review with full evidence

Governance: Human Approval Is Non-Negotiable

The refinement agent follows strict governance rules:

Core prompts — the ones that define the system's foundational behaviour — never auto-apply. This is a Canon-level guarantee. AI can propose. Humans decide.

The CLI Workflow

Stewards interact with proposals through the CLI:

# Check prompt health
human prompts performance contract-analysis
# Output:
# prompt://org/acme/legal.contract-analysis@1.0.0
# Calls: 847 (last 7 days)
# Avg tokens: 1,203 | Avg cost: $0.006 | Avg latency: 1.2s
# Positive: 72% | Negative: 12% | Rephrase: 16%
# Best model: gpt-4o (0.92 affinity) > claude-sonnet (0.87)
#
# Recommendations:
# - [PROPOSAL-47] Reduce token count by restructuring output format (-18%)
# - [PROPOSAL-48] Switch recommended model to gpt-4o based on affinity data

# Review a specific proposal
human prompts proposals show PROPOSAL-47
# Shows: full diff, evidence summary, risk assessment, affected usage volume

# Accept — publishes new version
human prompts proposals accept PROPOSAL-47
# Published: legal.contract-analysis@1.1.0
# Previous active: 1.0.0 (available for rollback)

# Or reject with reason
human prompts proposals reject PROPOSAL-47 --reason "Output format change would break downstream parsing"

Every decision — accept, reject, modify — is logged to provenance with full context. The audit trail shows who approved what change, based on what evidence, at what time.

The Virtuous Cycle

Put it all together and you get a system that compounds improvements over time:

Each pass through this loop makes the system measurably better. Prompts get refined based on real usage data. Model routing can soft-bias from empirical affinity when that preference is enabled under org policy. And humans stay in control of every decision that shapes how the system behaves.

This isn't machine learning in the traditional sense. There's no gradient descent. No fine-tuning. It's evidence-based prompt engineering at scale, with the AI doing the analysis and the human making the call.

Why this is different

Most platforms stop at “deploy the prompt.” A few add A/B tests. HUMΛN treats prompts as protocol-level managed artifacts with identity, delegation, telemetry, and a refinement agent that cannot silently rewrite Canon.

• Protocol-level: Telemetry is built into ctx.prompts and ctx.llm
• Identity-first: You cannot improve what you cannot identify
• Agent-driven: Prompt Refinement Agent has delegation scopes and an audit trail—not a dashboard widget
• Human-in-the-loop: AI proposes; humans decide. Core prompts always require approval
• Cross-cutting: Every agent benefits, not just Companion

Scroll-stopper: Telemetry without a human gate is just automated drift with prettier charts.

So that…

…you can ask which prompt version burned tokens last week; accept or reject a refinement proposal with provenance; and refuse “the model improved itself” as an explanation for a behavior change.

Go deeper

• Product: What is HUMΛN, HumanOS
• Docs: Introduction, HumanOS concepts
• Community: Protocol-level prompt management, Shipping prompts, Developer guide to ctx.prompts

Series: Part 3 of 4 — Prompt Management