RESEARCH
Findings we're willing to defend
Ilmora publishes applied research from production systems: agent reliability, evaluation methodology, document intelligence, and the governance of autonomy. Numbers come from real deployments, caveats included — because research you can't reproduce is just marketing with citations.
PUBLICATIONS
Selected technical reports
Full PDFs, datasets where releasable, and reference implementations are available on request — or bundled with enterprise partnerships.
Supervision Trees for Multi-Agent Workflows: Failure Containment in Production Agent Fleets
Vogel, L., Reyes, D., et al.
Abstract. We adapt hierarchical supervision — long used in fault-tolerant telecom systems — to fleets of LLM-based agents. Across 14 months of production traces from three enterprise deployments, supervised topologies reduced unrecoverable workflow failures by 83% relative to flat orchestration, at a median latency overhead of 4.1%. We characterize which failure classes supervision cannot contain and propose escalation contracts for human-gated recovery.
Trace-Distilled Evaluation: Mining Production Logs for Regression Suites That Predict Incidents
Vogel, L., Ishikawa, M., Adeoye, T.
Abstract. Hand-curated evaluation sets systematically under-sample the failure modes that cause production incidents. We present a pipeline that distills anonymized production traces into compact regression suites, weighting by downstream error cost. Suites built this way predicted incident-causing regressions 4.2x more reliably than expert-curated sets of equal size across two contact-center deployments.
Layout Robustness in Structured Document Extraction: Measuring and Mitigating Format Drift
Ishikawa, M., Vogel, L., Okoro, C.
Abstract. Document extraction systems degrade silently when upstream forms are redesigned — a failure mode we term format drift. We introduce a drift benchmark built from 12,000 synthetically re-laid-out enterprise documents and show that layout-randomized fine-tuning recovers 91% of baseline accuracy under severe drift, versus 62% for models trained on canonical layouts alone.
Learned Cost-Quality Routing for Heterogeneous LLM Backends
Reyes, D., Adeoye, T., Vogel, L.
Abstract. We formulate model selection as a constrained routing problem: minimize inference cost subject to a per-task quality floor estimated from evaluation history. A lightweight learned router held 96.3% of frontier-model task quality at 38% of the cost across three production workloads, and degraded gracefully under distribution shift when paired with an abstention fallback.
What Would It Take to Audit an Agent? Requirements from Regulated-Industry Deployments
Vogel, L., Whitcombe, R., Mehta, S.
Abstract. Drawing on deployment reviews with compliance teams at four regulated enterprises, we catalogue the evidentiary requirements — action provenance, authorization context, replayability, and counterfactual traces — that autonomous systems must satisfy for internal audit sign-off. We propose a hash-chained action-log schema meeting these requirements and measure its overhead in a claims-processing pilot.
The Economics of Human-in-the-Loop: When Review Queues Help, and When They Hide Failure
Okoro, C., Reyes, D.
Abstract. Human review is treated as a universal safety valve for AI systems, but poorly designed queues create rubber-stamp dynamics that are worse than automation alone. Using 400,000 review decisions from two document-processing deployments, we identify queue designs that preserve reviewer vigilance — confidence-stratified sampling, seeded known-error items — and quantify the cost-accuracy frontier for each.
OPEN SOURCE
Tools we maintain in the open
The evaluation and observability tooling our engagements depend on — open-sourced because trust infrastructure works better when everyone can read it.
PRINCIPLES
How we keep the research honest
Four rules every Labs project signs up to before the first experiment runs.
01Publish the failures
Negative results are findings. Internal experiment reports record what didn't work and why, and our public write-ups keep the caveats the marketing team would rather cut.
02Production data or honest synthetics
Benchmarks built on toy datasets produce toy conclusions. We evaluate on anonymized production traces where consent allows, and on synthetics engineered to preserve the ugliness of the real thing everywhere else.
03Client confidentiality is absolute
Everything we publish is aggregated, anonymized, and released only with client review. No paper is worth a client's trust, and none has ever cost us one.
04Research serves deployment
We are an applied lab. Every research thread must name the engagement problem it serves, and every published technique ships with a reference implementation someone can actually run.
Want the full papers?
Request any technical report, dataset card, or reference implementation — or talk to the researchers directly about applying the findings to your systems.