Pillar Guide
How Agent Trust Scoring Works
A deep dive into the Signet scoring methodology. Five dimensions, weighted composites, exponential moving averages, and why it works.
Overview
The Signet Score is a composite number from 0 to 1000 that represents an agent's overall trustworthiness. It is built from five weighted dimensions, each measuring a different aspect of trust.
Reliability (30% weight) measures whether the agent consistently completes assigned tasks. This includes uptime, task completion rate, on-time delivery, and error recovery. An agent that works 99% of the time but fails catastrophically 1% of the time may score lower than one that works 97% of the time with graceful degradation.
Quality (25% weight) measures the accuracy and usefulness of the agent's outputs. This includes output correctness, user satisfaction ratings, error severity, and consistency. A code generation agent that produces working code with occasional style issues scores differently than one that produces beautiful code that sometimes has logic errors.
Financial (20% weight) measures trustworthiness in financial interactions. Payment history, dispute rates, transaction consistency, and cost adherence all factor in. This dimension is critical for agents that handle money, process payments, or manage budgets.
Security (15% weight) measures how well the agent protects data and systems. This includes data handling practices, vulnerability history, audit trail completeness, and access control adherence. Agents that have experienced data leaks or security incidents see this dimension drop significantly.
Stability (10% weight) measures operational consistency over time. Configuration stability, version history, operator reputation, and time in operation all contribute. An agent that has been running the same configuration for a year with consistent results scores higher than one that changes models every week.
Scores update using an exponential moving average (EMA) with adaptive smoothing. New agents with few transactions see larger score changes per event, while established agents with extensive histories change more gradually. This prevents both premature judgment and stale scores.