A Datadog alert fires and the bot in Slack reports what fired, how often and how severe. The next questions during triage are the ones it couldn't answer well: who owns this service, and what happened last time? For those, engineers left Slack and pieced it together by hand, and the assistant meant to save them the trip was sharp on one question and vague on the next.
The situation
RobinRelay started as a Slack bot that posted scheduled Datadog alert summaries. During an incident, engineers wanted operational answers instead. Who owns this service? What happened last time? Which alerts are related? Is this recurring? The data to answer those existed, but getting at it meant leaving the channel. And in SRE work a confidently wrong answer is worse than no answer, because someone acts on it mid-incident, so I treated response quality as a reliability problem.
The work described here was mine alone: the Slack heatmap path and the redesign of the backend retrieval flow, meaning intent classification, the retrieval strategy for each intent, and the preprocessing before inference. RobinRelay is a product with a founding team around it, and this write-up covers only what I owned.
What I saw
Questions went through a generic assistant flow, the obvious approach, with one retrieval path for every kind of question. Its answers were inconsistent. Some were sharp, some were vague, and some blended ownership, timelines and infrastructure context into one response. For a general chatbot, "mostly right" is acceptable. For "who handled the production outage last Tuesday?" it isn't.
Worse, a bad answer gave no clue which part had failed, because every type of question ran through the same path. An ownership question needs service and incident-owner records; a summary needs timeline and alert history. One flow can't fetch the right context for both. That pointed me at the real constraint: how the data was structured and retrieved, not the model behind it.
What I did
I split retrieval by the kind of question being asked. Each question is classified by intent first, and each intent gets its own retrieval strategy.
def handle_sre_query(user_input):
intent = classify_intent(user_input)
if intent == "OWNERSHIP":
return fetch_service_owners(user_input)
if intent == "SUMMARY":
return generate_incident_timeline(user_input)
return run_contextual_lookup(user_input, intent)
Ownership questions get service and incident-owner records, summaries get timeline and alert history, and infrastructure questions fall through to a broader lookup with tighter grounding. Datadog webhooks feed a FastAPI service that stores alert history; each path hands what it retrieves to Azure OpenAI with the question (retrieval-augmented generation, or RAG); n8n ties the automation together; the answer comes back in Slack.
Intent-based retrieval routing (ownership / summary / infrastructure)
One generic assistant flow for every question
Why: A single flow mixed ownership, summaries and infra context. It gave no signal about which stage produced a bad answer. Splitting by intent let each path use its own retrieval strategy and fail on its own. The trade-off is that intent classification becomes a new way to fail. Misroute the question and the right path never runs.
How I know it helps
Nothing here was measured: no evaluation set, no accuracy comparison. What the design gives is debuggability. Every response can be traced from the question, to the retrieved context, to the Slack reply, so weak ownership answers meant tuning one retrieval path instead of reasoning about the whole system.
The result
The assistant went from a bot that said what fired to a workflow engineers could ask who owns it and what happened last time, inside Slack. When an answer comes back wrong, the failure points at one retrieval path rather than the whole system.
Separately, I built a monthly alert heatmap in Slack Home, noisy days dark and quiet days light. FastAPI groups historical alerts by day and delivers the image through Cloudinary, so "was this week noisy?" became a glance instead of an investigation.
What's still open
- Intent misclassification. The design introduced this failure mode. A misrouted question runs a retrieval path that was never going to answer it, and the result looks like a retrieval failure, not a routing one.
- Per-intent accuracy, tracked separately. A drop in ownership answers shouldn't be able to hide behind healthy summaries.
- Grounding failures. Confidently wrong answers are what SRE work can least afford.
- Slack response latency. The workflow is only useful mid-incident if it answers fast.