API reference¶

Rendered from source. assaylab keeps a light base wheel; heavy surfaces (LLM providers, REST) are lazy-imported behind extras.

Models¶

models ¶

assaylab domain models, built on the agentsensory contract.

The test domain has no pixel (BBox) or time (Span) grounding, so each issue grounds itself in detail (a JSON-safe dict) carrying the signature id, affected tests, and file loci. Report / Issue subclass the agentsensory base types so every result speaks the shared pass/warn/fail + Handoff language.

Outcome ¶

Bases: str, Enum

Normalized outcome of a single test execution.

IssueKind ¶

Bases: str, Enum

assaylab issue kinds (the domain vocabulary carried in Issue.kind).

IssueSource ¶

Bases: str, Enum

Which analysis produced an issue.

TestRecord ¶

Bases: BaseModel

One test execution, normalized across ingestion formats.

This is the universal internal schema every backend maps onto: (project, commit, test_id, run_id, outcome, duration, message, stacktrace).

FailureSignature ¶

Bases: BaseModel

A cluster of failing executions that share a normalized fingerprint.

The fingerprint is a stable hash of the templated (variable-stripped) error type + message + top stack frames — so runs that differ only in addresses, line numbers, temp paths or timestamps collapse together.

to_detail ¶

to_detail() -> dict[str, object]

JSON-safe grounding payload for an Issue.detail.

Issue ¶

Bases: IssueBase

assaylab issue — an agentsensory.IssueBase with test-domain grounding in detail.

Report ¶

Bases: ReportBase

assaylab report — an agentsensory.ReportBase (verdict + issues + handoff).

Core sense¶

analyze ¶

Grade test records into an agentsensory :class:Report.

P1 verdict rule (deterministic):

no failing records -> PASS
failing records present -> FAIL, one issue per failure signature
a signature absent from a supplied baseline is marked NEW_FAILURE (higher risk) rather than a plain FAILURE_SIGNATURE.

Flaky-vs-real downgrading to WARN arrives in P2; the hook (baseline diff) is already here.

grade_records ¶

grade_records(records: list[TestRecord], *, baseline_ids: set[str] | None = None, backend: str = 'unknown') -> Report

Cluster failures and return a graded :class:Report.

analyze `async` ¶

analyze(source: str, *, backend: str | None = None, baseline: str | None = None, settings: Settings | None = None) -> Report

Ingest source (and optional baseline) and grade it.

signature ¶

Failure-signature normalization and clustering.

Given failing :class:TestRecord executions, collapse the ones that differ only in incidental detail — memory addresses, object hashes, line numbers, temp paths, timestamps, UUIDs, numbers — into a single :class:FailureSignature identified by a stable fingerprint. This is the evidence layer everything else (RCA, flaky classification, confidence bounds) grounds on.

Pure-stdlib and deterministic: the same corpus always yields the same signature ids, so verdicts are reproducible.

normalize ¶

normalize(text: str) -> str

Strip incidental variation, returning a stable template string.

fingerprint ¶

fingerprint(record: TestRecord) -> tuple[str, str, str, list[str]]

Return (signature_id, template, exception_type, files) for a failing record.

The template combines the normalized message with normalized top frames, so two failures with the same root but different incidental text collapse.

cluster ¶

cluster(records: list[TestRecord]) -> list[FailureSignature]

Cluster failing records into signatures, ordered by descending run count.

Passing/skipped records are ignored (only failures carry a signature).

Root-cause analysis¶

rootcause ¶

Root-cause categorization.

A transparent, always-available baseline: map a failure signature to a root-cause category with a confidence and the evidence that fired. This is the interpretable floor the learned model (model.py) is measured against, and what grounds every RCA verdict so a human can audit why a cause was assigned.

categorize ¶

categorize(sig: FailureSignature) -> Categorization

Assign a root-cause category to a signature with confidence + evidence.

flaky ¶

Flaky-vs-real classification.

A failure signature is flaky-leaning when its affected tests behave non-deterministically — the strongest evidence being a single commit that shows both a pass and a failure (order-agnostic flakiness, à la iDFlakies), backed by a high flip-rate across runs. Real failures are consistent at a given commit.

The heuristic here is always available; :func:flaky_probability optionally defers to a learned model when one is supplied.

flaky_probability ¶

flaky_probability(features: dict[str, float], tests: list[str], stats: dict[str, TestStats], model: object | None = None) -> FlakyVerdict

Learned probability when model is given, else the heuristic.

risk ¶

Per-test risk scoring and next-run failure forecasting.

Risk blends recency-weighted failure rate with instability (flip-rate): a test that fails often and recently is high risk; a rarely-failing stable test is low risk. forecast is the model's estimate that the test fails on its next run, used later (P3) to bound the confidence lost when a test is skipped.

rank_risk ¶

rank_risk(stats: dict[str, TestStats]) -> list[Risk]

All tests ranked by descending risk.

model ¶

A small, self-contained logistic-regression model for flaky prediction.

Pure Python (batch gradient descent) — no numpy/scikit-learn in the base wheel, and the trained model serializes to JSON, never pickle. That matters: a pickled model is arbitrary-code-execution on load, so a model pulled from a registry or CI cache would be a supply-chain hole. JSON weights load inertly.

Features are addressed by name (a dict), so callers never depend on column order. Unknown/missing features default to 0.0. Inputs are standardized using means/stds captured at fit time.

For heavier models, train externally and export coefficients into this JSON shape; prediction stays dependency-free.

train ¶

train(samples: list[tuple[dict[str, float], int]], *, feature_names: list[str] | None = None, epochs: int = 400, lr: float = 0.3, l2: float = 0.001) -> LogisticModel

Fit a logistic model on (features, label) pairs (label in {0,1}).

Deterministic: no randomness, weights start at zero, so the same data always yields the same model.

Attested selection¶

engine ¶

Test-selection engine with a verifiable confidence bound.

Each candidate test t has a detection probability q_t — the modelled chance it fails (catches a regression) on this run, estimated from history (P2 forecast). If we run a subset S and skip U = All \ S, the probability that at least one skipped test would have failed is

epsilon = 1 - prod_{t in U} (1 - q_t)

under an independence assumption. epsilon is the confidence lost by skipping — an upper-bound proxy for the chance we miss a regression the suite would have caught. Selection keeps the highest value-density tests (q_t per second) until either epsilon <= target_epsilon or the time_budget_s is exhausted; the achieved epsilon is always reported.

Honest limits (stated, not hidden): independence of failures, q_t stationarity, and coverage only of regression classes seen historically. These are the receipt's residual assumptions, not guarantees.

select ¶

select(candidates: list[Candidate], *, target_epsilon: float | None = None, time_budget_s: float | None = None) -> Selection

Choose a subset to run under a confidence target and/or a time budget.

Exactly one of target_epsilon / time_budget_s should be the primary driver; if both are given, selection stops as soon as either is satisfied (target met) or violated (budget spent).

receipt ¶

The attested receipt: a signed commitment to a test-selection outcome.

The signature (HMAC-SHA256) covers the result — the inputs hash, the selected and skipped sets (by hash + count), and the computed confidence bound epsilon — so it binds what actually happened, not merely that a run occurred. Verifying is constant-time and, given the candidate inputs, re-derivable: a consumer can recompute epsilon from the committed inputs and check it matches.

Receipt ¶

Bases: BaseModel

A signed test-selection receipt. signature is not part of the signed body.

signed_body ¶

signed_body() -> bytes

Canonical bytes covered by the signature (everything but signature).

verify ¶

verify(key: bytes) -> bool

Constant-time signature check.

hash_ids ¶

hash_ids(ids: list[str]) -> str

Stable hash of a set of test ids (order-independent).

keys ¶

Signing-key resolution — env, else a persisted per-installation random key.

Never a hardcoded/default secret (a shipped default key lets anyone forge a receipt). Resolution order:

ASSAYLAB_SIGNING_KEY — an explicitly encoded key: hex:<hex>, base64:<b64>, or raw:<utf8> (or a bare value, treated as raw). The encoding is never guessed, so one string can't resolve to two different keys. Must carry >= 16 bytes and not be a degenerate (single-byte-repeated) value.
A per-installation key persisted at <user_config>/assaylab/signing.key, created with :func:secrets.token_bytes and 0600 perms on first use.

Both paths refuse to follow symlinks and reject files not owned by the current user, closing the symlink/TOCTOU forgery vector. A malformed env key fails closed (raises) rather than silently falling back.

resolve_key ¶

resolve_key() -> bytes

Return the signing key (env override, else persisted per-installation key).

key_id ¶

key_id(key: bytes) -> str

A short, non-secret identifier for a key (first bytes of its SHA-256).

Lets a receipt name which key signed it without revealing the key.

Test generation & self-healing¶

models ¶

The Proposal: an LLM artifact assaylab never executes or applies.

Proposal ¶

Bases: BaseModel

A dry-run suggestion + its provenance and acceptance criterion.

applied is always False: assaylab produces the proposal and defines how to verify it, but a human/CI applies and runs it in their own sandbox. The verdict layer decides acceptance (see :mod:assaylab.llm.gate).

signed_body ¶

signed_body() -> bytes

Canonical bytes covered by the signature (everything but the signature).

generate ¶

Propose a regression test from a failure signature (dry-run).

gate ¶

Gate a proposal against a REAL run — acceptance flows through the verdict layer.

assaylab never runs the proposed code. Instead the user applies + runs it in their own sandbox and feeds the resulting test output back here; the proposal is accepted only if that graded run meets its acceptance criterion. This is what keeps LLM output honest: a generated test counts only if it actually reproduces the bug; a heal counts only if the flaky signature stops failing.

evaluate_proposal ¶

evaluate_proposal(proposal: Proposal, result_source: str, *, backend: str | None = None, settings: Settings | None = None) -> Evaluation

Grade the returned run and decide whether the proposal is accepted.

The proposal is treated as UNTRUSTED: its signature must verify (it was signed at generation by the tool that derived the criteria from a real signature), or we refuse to grade it. This stops a hand-crafted proposal JSON from weakening its own acceptance criteria.

API reference¶

Models¶

models ¶

Outcome ¶

IssueKind ¶

IssueSource ¶

TestRecord ¶

FailureSignature ¶

to_detail ¶

Issue ¶

Report ¶

Core sense¶

analyze ¶

grade_records ¶

analyze async ¶

signature ¶

normalize ¶

fingerprint ¶

cluster ¶

Root-cause analysis¶

rootcause ¶

categorize ¶

flaky ¶

flaky_probability ¶

risk ¶

rank_risk ¶

model ¶

train ¶

Attested selection¶

engine ¶

select ¶

receipt ¶

Receipt ¶

signed_body ¶

verify ¶

hash_ids ¶

keys ¶

resolve_key ¶

key_id ¶

Test generation & self-healing¶

models ¶

Proposal ¶

signed_body ¶

generate ¶

gate ¶

evaluate_proposal ¶

analyze `async` ¶