Two weeks. One very quiet Slack channel. Here is what we ended up shipping, and what we should have shipped in the first version.
Read a post recently from someone who nearly shipped an agent with payment approval access and no audit trail. Two weeks to rebuild the flow. Genuinely disturbed a few people. The comment thread was full of the same.
Been on that channel. So, most of our team has. This is a write-up of what we ended up building, the middleware layer, not the model. The model is not a problem. The model is usually right; roughly the way GPS is usually right. The problem is what happens when it is not, and whether we can prove what it did afterwards.
At iStreet Network, India’s AI Centre of Excellence, we have seen this pattern repeat across banks, insurers, and healthcare providers. We did not set out to build a governance product. We set out to stop shipping the same near-miss twice.
The failure mode nobody wants to name
Here is the version we see most often, roughly sanitized:
@app.tool()
def approve_payment(vendor_id: str, amount: float) -> dict:
“””Approves a payment for the given vendor.”””
return payment_gateway.execute(vendor_id, amount)
The agent has access to approve_payment. The docstring is friendly. The function does what it says. Somewhere in the model context, we have written you are a helpful finance assistant. The demo works. QA passes. The stakeholder loves it.
Then we ship. Six weeks later, an auditor asks a question: on 14 March at 15:47, agent instance agt-4f2a1e approved a ₹47,00,000 payment to vendor V-8891. Who authorised that agent to approve payments up to that amount? What was the prompt context at the time? What was the model’s reasoning?
We have logs. We do not have that.
That is the gap. The technical fix is well-understood. It does not get built because nothing in the tutorial required it — and because the governance layer is an afterthought in most team roadmaps. This is precisely the problem our Integrated GRC and Agentic AI platforms were designed to close.
The four things we had to build
Skip the marketing framing. Here is the middleware that makes an agent survive an audit.
1. Explicit capability grants, that is machine-readable
The agent will not have access to approve_payment. That is not permission. That is a code path.
What we want:
agent: finance-assistant-v2
capabilities:
– action: approve_payment
max_amount_inr: 100000
vendor_whitelist: [“V-8891”, “V-2213”]
requires_secondary_approval: false
– action: approve_payment
max_amount_inr: 5000000
vendor_whitelist: “*”
requires_secondary_approval: true
approver_role: “finance_manager”
The grant is declarative, versioned, and enforced by a wrapper that sits between the agent and the tool. The agent asks; the wrapper decides whether to let it through.
This is the pattern iStreet’s Integrated GRC platform enforces at the infrastructure level, not as an add-on, but as foundational wiring. Capability grants are versioned, auditable, and scoped before an agent ever touches a tool. Boring middleware. The kind that keeps you out of a regulatory inquiry.
Most serious agent frameworks now support this pattern, LangGraph interrupts, MCP tool-level auth, Anthropic’s tool-use scoping. The frameworks make it possible. Whether we use it is on us.
2. Structured audit logs on every tool call
This is often where teams compromise, as logging can seem tedious and costly. Yet it remains essential.
For every tool call, capture:
- agent_id and agent_version
- tool_name and tool_args
- prompt_context_hash — not the whole prompt, a hash that joins back to a prompt store
- model_id, model_version, temperature, seed if available
- decision_rationale — the model’s own reasoning trace, or a follow-up call that asks it to explain
- timestamp and trace_id, OpenTelemetry-compatible
- outcome — success, denied, escalated
- reversibility — a static flag on the tool definition
This is precisely where iStreet’s AI Observability, AIOps and GenAIOps solutions deliver value. Full-stack observability across on-premises, cloud, and SaaS, structured, queryable, and surfaced through our Resiliency Operations Centre. This is not Splunk-for-alerts. It is governance-ready audit fabric that enables compliance officers to retrieve decision evidence through reporting tools, even six months later.
The cost is real, but so is the value. Assess it against the financial and operational impact of the incidents it can prevent.
3. Approval gates before anything irreversible
Every tool gets reversibility classification. Anything marked irreversible, payments, contract execution, external notifications, DB writes to production of record, routes through a gate.
The gate is a state machine, not a Slack message.
# pseudocode
agent proposes action
→ capability check (does the agent hold the grant?)
→ yes and reversible: execute
→ yes and irreversible: escalate to named human
→ human approves within TTL: execute
→ human denies or TTL expires: reject with reason
→ no: reject with reason
TTL matters. An approval that waits forever is not control. It is a backlog. Set a real time bound, log the outcome, and route stale approvals accountable.
iStreet’s Agentic AI for Enterprises platform, deployed with leading PSU banks, embeds exactly this state-machine pattern. Human-in-the-loop escalation is embedded into the decision architecture from the outset, with time-to-live enforcement and a complete audit trail of every approval, denial and timeout.
4. Post-flight sampling with rationale reconstruction
This is the one most teams skip. Post-flight review of a randomized, risk-weighted sample of agent decisions.
For each sampled decision, ask a second model (or a human, depending on risk tier) to reconstruct: given the prompt, the tool set, the grants, and the observed decision, is this reasoning defensible? Would a regulator accept the rationale?
Store the reconstruction alongside the original decision. Track the rate at which sampled decisions fail review. That number is our governance health metric. If it drifts up, something changed, the model, the prompt, the tool surface, the user population. Investigate before the auditor does.
This is the explainable AI layer in iStreet’s Resiliency Operations stack, autonomous AIOps that does not just alert, but explains, to a compliance team that can act on. When the governance health metric drifts, it surfaces before it becomes an incident.
The Defining Challenge: Governing Non-Determinism
The hardest part of all of this is not the code. It is that models are non-deterministic and traditional audit assumes reproducibility.
We will not be able to replay a decision and get the same output. Same prompt, same context, different day, different answer. Regulators are still catching up to what that means. Teams doing this well pin versions — model_version, system_prompt_hash, tool_schema_hash, so that, they can say: on this date, this agent configuration produced this decision. Here is the system’s exact state at that moment, the rationale behind its decision, and the subsequent review of that rationale.
That is not full reproducibility. It is defensibility, which is what a regulator actually needs. iStreet’s governance-by-design architecture pins all of this at the infrastructure layer, so the answer to “what did the agent do and why” is always retrievable, regardless of when the question is asked.
Why this is a middleware problem, not a model problem
A common approach is to solve governance by picking a better model, or fine-tuning for compliance, or writing tighter system prompts. None of that helps. The model is a component. Governance is the control architecture surrounding it.
We keep coming back to the database transaction analogy. Nobody argues that transactional integrity is the database’s job. It is above, application code, middleware, ORM. Same with agent governance. The model does what models do. Middleware is what makes it deployable in a regulated business.
If a team is still treating the audit layer as something to add later, that team is going to find out about it the wrong way. That is not hypothetical. That is the story we hear in a Slack DM every month, from teams building for BFSI, and insurance who caught it late, or did not catch it at all.
What iStreet built
iStreet Network is not a consultancy that hands off a design doc. We build and operate AI-native systems for banks, insurers, and healthcare providers. Where governance failures carry real operational consequences. This accountability shapes both what we build and how we prioritise it.
When we rebuilt an approval flow from scratch for a client, we stopped treating middleware as something each engagement reinvents. We pulled the whole stack, capability grants, structured audit logging, approval gates, post-flight sampling, into a shared environment that now powers the Sanjeevani of AI platform: the foundation for everything we deliver into regulated enterprises.
Sanjeevani of AI is our approach to the specific problem of deploying AI in regulated industries: not just making the model capable, but making it auditable, defensible, and sovereign. The same way a hospital must show its clinical decision trail, a bank deploying an agent on payment workflows must show, to a regulator, to an auditor, to itself, that every decision was bounded, logged, and explainable.
What that looks like in production, across banks
- Agents operate under explicit, versioned capability grants, not code access, but declared permissions, enforced by middleware, scoped to amount limits, counterparty lists, and escalation rules.
- Every tool call writes a structured audit record: agent version, prompt context hash, model configuration, reasoning trace, outcome, reversibility flag. Queryable by compliance teams, not just engineers.
- Irreversible actions route through a state-machine gate with TTL. No approval request remains pending indefinitely.
- A risk-weighted sample of decisions undergoes post-decision rationale reconstruction, in which an independent model or human reviewer validates whether the reasoning remains sound. The validation pass rate serves as a key governance health metric.
- Full-stack AI observability across on-premises, cloud, and SaaS, with explainable AI that reduces alert noise by up to 96% and surfaces what matters before an auditor asks.
iStreet’s AICoE framework integrates Generative AI, Cybersecurity, and Governance, Risk & Compliance as a single operational triad, not three separate products bolted together. Governance is not retrofitted. It is foundational. We built this capability because our clients operate in regulated industries, where our reputation is tied to every action their AI takes, even at 15:47 on a Tuesday, when no one is watching. If your team is building agents for a regulated workflow and the audit layer is still on the backlog, we have seen how that ends. Let’s talk before it does.
Suraj Varma
Suraj Varma is an LLM and Agentic Systems Engineer at iStreet Network Limited, with experience in GenAI, AIOps and production-grade LLM systems. He works on AIOps capabilities, including LLM-powered root cause analysis and agentic infrastructure for hybrid enterprise deployments across banking, finance and insurance. His work encompasses RAG pipelines, multi-agent orchestration and GPU-backed inference infrastructure supporting deployments at major public-sector banks.
