MSAIE · AI Engineering

STS Track

An agentic RAG system for tubular asset operations. Grounded answers, cited to the section. Actions gated in code.

Architecture

One Azure web service: a slim FastAPI app, an agent orchestrator, and an MCP server over stdio. The browser never talks to tools or the vector store directly.

FastAPI exposes /chat, /health, and provider controls.
The orchestrator runs the tool loop and returns answer, citations, and trace.
The MCP subprocess owns RAG retrieval and mock operational data.
The LLM is external (Azure AI Foundry by default; Groq / OpenRouter / Ollama selectable).

RAG & citations

Thirteen synthetic QMS documents for a fictional tubular services company are chunked on ## headings so every claim can point at a numbered section.

82 deterministic chunks in Chroma with fastembed (ONNX) embeddings.
Policy answers must cite document sections retrieved this turn.
Citation verification strips fabricated references the tools never fetched.
The index is a disposable cache rebuilt from corpus/ on deploy.

Tools & gates

Seven MCP tools answer one business question each. Six are read-only; one mutates mock state and is confirmation-gated in orchestrator code—not only in the prompt.

RAG: search_procedures, get_procedure_section.
Assets & inspections: lookup, history, compliance, certification checks.
create_work_order stores a pending proposal; “yes” executes that exact payload.
Structured tool errors teach the model valid enums and filters.

Agent loop

Prompt rules steer style; correctness lives in deterministic code. The loop allows up to eight tool rounds, then returns a grounded answer with a collapsible trace.

No invented asset IDs—tools must return what the model reasons about.
Compliance rules (including failed EMI) enforced inside check_compliance.
Provider registry with runtime switching; Azure Foundry is the demo default.
Degraded or incomplete generations are surfaced in the UI, not hidden.

Evaluation

A 25-question evaluation set with gold answers exercises policy lookup, multi-step asset workflows, confirmation gates, and citation honesty.

Runner in evaluation/run_eval.py with resumable JSON results.
Gates scored both ways: unconfirmed actions must not execute; confirmed ones must.
Provider, base URL, and model are recorded in results metadata.
Full methodology and findings: design-and-evaluation.md in the repo.

Deployment

Hosted on Azure App Service (Python 3.12, Basic B1) with GitHub Actions deploy and a custom domain at sts-ai-agent.stensen.no.

Bind-first startup: /health reports warming → ready | error.
Index rebuild and MCP connect run in a background warmup task.
Embeddings use ONNX (fastembed) to stay within B1 memory limits.
Portfolio-safe framing via FRAME_ANCESTORS and chat rate limits.