Compose evaluation profiles for the Swedish voice agent. Patch in the metrics you want, set each threshold on the fader, and save the rack — the runner scores exactly what you build here.
Runs score_recordings.py over the selected data — calls the judge LLM, ~30–60s. Pick a dataset to score its frozen, labelled items (unlocks the label-based metrics); a profile scores exactly that selection.
Everything below is read live from the brain. A knowledge base with no chunks retrieves nothing, and a source that produced none contributes nothing — both are silent during a call, so they are counted here instead.
What the agent can retrieve, source by source. Read-only — the brain owns ingestion; re-crawl by onboarding again and Promote. Open a source to read the chunks the retriever actually sees.
The brain embeds the query itself, with the model that embedded the chunks — so these scores are the ones a call would see. A k above the call's own top-k shows the near-misses it would drop.
A block answers instantly when a caller's question matches it — the latency and compute lever. There are two libraries here: slot blocks, composed from a crawl into an answer table the selector reads directly, and seeded blocks typed in below, which are written on the v1 company and reach calls after the next Promote. A natively onboarded tenant has only the first, and an empty seeded list says nothing about whether it answers.
A promoted v1 tenant has a corpus, a voice and a channel, and no v2 facts. This buys it the same crawl a native run buys — the reader that searches, follows and renders — and composes its answer table through an agana-blocks schema. Its facts then come from the same place every other v2 tenant's do, which is what makes the two paths comparable.
Minutes, not milliseconds — the log lands on the Run tab. Nothing is created and no id changes, so every recording made against this tenant still points at it: the same tenant, before and after. Its voice and its prompt are left alone.
One block per intent, with the phrasings that reach it and what a caller may narrow by. The sentences were written offline against the crawl's facts, so a turn looks one up instead of generating it — open a block to read them.
The real selector, on the v2 copies a call matches — a library that was never promoted matches nothing however many blocks the list shows. Scored as a first turn: greeting blocks compete here that a follow-up would exclude.
Compares the v2 blocks to each other in the space the selector matches in — no query involved. A group whose blocks answer differently is one utterance with two cached replies, and which one plays is decided by the stored confidence and the bandit's score, not by the caller.
A different library from the one above: one phrase, its triggers, no slots. Written on the v1 company, so it reaches a call after the next Promote — and a natively onboarded tenant has none of these by design.
Installing a plugin writes the real manifest the brain reads, so its tools are offered to the LLM and its callbacks are what the agent speaks. During a call an HTTP tool runs through the bridge's own dispatcher — transport, secrets and response parsing all real — answered by the plugin's golden fixtures instead of the vendor. A bridge_handler tool has no manifest transport, so it gets a canned payload. A manifest that fails validation or its own golden cases is refused here.
Sibling agana-plugins-* checkouts and the synthetic toolsets. Re-installing replaces the agent's copy, so edit a manifest and install again to iterate.
Change the base, save, then Simulate and score from the Run tab, comparing in Results — the iteration loop.
The brain only builds tonality from Intercom transcripts, so a website-only tenant has none and its agent prompt is just the opening line. Onboarding without an Intercom token already drafts one — this redrafts it from the text the crawl stored, writes it to the v1 company and promotes — the same column a real Intercom pass would fill, so that pass still wins later.
The crawler onboarding runs, on domains that have no company yet — nothing is created. Each site is ranked and scraped exactly as a bootstrap would, then checked for whether the pages it kept even contain a phone number, an address or opening hours. Those flags are a presence check over the raw text, not parsed values: they say whether extraction could work. To see what it actually extracts, open a page and press Extract — that spends one Bedrock call.
Three sites crawl at a time; a domain takes tens of seconds, most of it probing subdomains and common paths. Raising pages read costs one fetch each and is where a page that ranked low but holds the address shows up.
Its prompt, blocks and knowledge are set up in the Company tab.
An LLM plays the caller against agana-brain and the call is read from its DB into recordings.json. Simulate only generates data — score it from the Run tab (Simulated view). Needs the brain stack up · ~15–40s per scenario.
A real call on agana-brain — every turn is recorded like any other, so it shows up in Dataset and Trace too. Its prompt, blocks and knowledge are set up in the Company tab. Needs the brain stack up · ~2–8s per turn.
The whole real pipeline: your mic → Soniox STT → agana-brain → ElevenLabs → your speakers, with barge-in. The tab dials /ws/webcall on the bridge, so the bridge must be running with WEBCALL_ENABLED=true. It spends real STT/TTS credit, and the call is recorded like any other — Dataset and Trace show it. Use headphones: there is no Krisp here, so on speakers the agent hears itself.
Calls are read live from the agana-brain DB (the source of truth — nothing stored locally). Simulated calls run against that same brain, so they are in here too. Open a call, then add it to a dataset to curate + label it — the dataset keeps its own snapshot, independent of the DB.
Curated golden datasets — conversations snapshotted from the sources, each carrying a human answer key (per golden-dataset-spec.md). Label the facts an answer must convey / must not, the expected tool and outcome, then freeze and export for deepeval.
Where one turn's time actually went, from the recorded latencies — not a replay. Time zero is the caller's finalized utterance; the brain's stages run end to end from there, while the bridge's audio legs are placed at their own measured offsets, so an overlap reads as an overlap. A leg with no number is marked not measured rather than drawn as zero.
Every component below is here because traffic went through it — nothing is read from configuration. A knowledge base with no queries has no node, and a tool nobody called cannot hide. Each turn follows exactly one path, so a company's count equals its outgoing edges.
The prompt is many times the answer, so what a call costs is decided by what goes in. Every turn resends the system prompt and the conversation so far; the retrieved chunks are replaced, not added to. This splits the recorded total across those parts, so you can see which one to cut.
Replays each turn's recorded Bedrock call with the candidate base swapped in, then scores the regenerated answers. Offline — no brain, no live calls. A run is dominated by judge calls: expect minutes, not seconds.
Where the search starts. Loaded live from the brain's _base_system_prompt() — edit it here to search from something else; the edit is used for this run only and never written to the brain. A winner ships as a reviewed change to that function — it applies to every tenant.
Keep the {{today}} token: each replayed turn renders it from its own recorded date.