RobinRelay was a Slack bot that posted scheduled Datadog alert summaries. It told you what fired; during triage engineers needed to know who owned the service and what happened last time, which meant leaving Slack. I picked up that gap as the DevOps engineer on it. The interesting finding was that the assistant's inconsistent answers weren't a model problem — ownership, summaries and infrastructure questions were all being served by one generic retrieval flow, so nothing could be tuned or debugged independently.
Ownership. Sole on the work described here: I built the Slack heatmap path and redesigned the backend retrieval flow — intent classification, the per-intent retrieval strategies, and the preprocessing ahead of inference. RobinRelay is a product with a founding team around it; this write-up covers what I owned.
From what fired to who owns it and what happened last time
How it fits together
Datadog webhooks land on a FastAPI service that persists alert history. The same service renders monthly heatmaps through Cloudinary into Slack, and routes incoming questions by operational intent into a RAG path backed by Azure OpenAI. n8n stitches the automation steps between them. Slack stays the only surface engineers touch.
Making alert noise visible
The alert data existed; the pattern didn't. Scrolling Slack for it on a calm day is fine — mid-incident it isn't.
So I built a Slack Home heatmap: a monthly grid where noisy days are dark and quiet days light. FastAPI aggregates historical alert data by day, renders the image, and delivers it through Cloudinary into Slack Home. Small piece of work, and the point was purely that "was this week noisy?" became a glance instead of an investigation. Whether the team then tuned specific surfaces was their call, and I don't have a measurement of what changed.
The part that mattered: retrieval was the constraint, not the model
Once scheduled summaries weren't enough, the hard problem was context. Engineers wanted operational answers — who owns this service, what happened last time, which alerts are related, is this recurring.
A generic assistant flow was the obvious path and it's what existed. In practice the answers were inconsistent: sometimes sharp, sometimes vague, sometimes blending ownership, timelines and infrastructure context into one response. For a general chatbot "mostly right" is acceptable. For "who handled the production outage last Tuesday?" it isn't — and worse, when an answer was bad there was no way to tell which part had failed, because every question type shared one path.
That's what pointed at the real constraint. The limitation was how data was structured and retrieved, not the model behind it.
Intent-based retrieval routing (ownership / summary / infrastructure)
One generic assistant flow for every question
Why: a single flow mixed ownership, summaries and infra context, and gave no signal about which stage produced a bad answer. Splitting by intent let each path use its own retrieval strategy and fail independently. The trade-off: intent classification becomes a new failure mode of its own — misroute the question and the right path never runs.
def handle_sre_query(user_input):
intent = classify_intent(user_input)
if intent == "OWNERSHIP":
return fetch_service_owners(user_input)
if intent == "SUMMARY":
return generate_incident_timeline(user_input)
return run_contextual_lookup(user_input, intent)
Each intent gets the context it actually needs: ownership questions need service and incident-owner records, summaries need timeline and alert history, infrastructure questions fall through to a broader lookup with tighter grounding.
The practical gain was debuggability. Weak ownership answers meant tuning one retrieval path rather than reasoning about the whole system, and every response had a traceable route from question to retrieved context to Slack reply.
What I'd watch
- Intent misclassification. This is the failure mode the design introduced. A misrouted question runs a retrieval path that was never going to answer it, and the output looks like a retrieval failure rather than a routing one.
- Per-intent accuracy, tracked separately. A regression in ownership shouldn't be able to hide behind healthy summaries.
- Grounding failures. Confidently wrong answers are the specific thing SRE work can least afford.
- Slack response latency. The workflow is only useful mid-incident if it answers fast.
Measurement
No alert-noise or answer-accuracy figures are claimed here — neither was measured, so there's no comparison window, alert count, or evaluation set behind them.
What the page claims is the mechanism: repeated noise became visible in Slack, and incident-context failures became attributable to a specific retrieval path rather than to the system as a whole. Both are verifiable from the code.