Build or Buy? SRE Tools and Slack Bots

The Broken AI-SRE Promise

Through the same 6-9 month window of time that we built Monacoder (see our last post), we experimented with a couple AI SR-type products. We found them intriguing but were unconvinced on adoption after the release of the Datadog MCP and, more recently, Claude plugin. Anecdotally, their results when investigating a monitor trigger were wrong at a high enough frequency that developers taking them at face value was more harmful than helpful. In the worst case they would waste time making developers go down the wrong investigative rabbit hole and looping in the wrong people without having verified the agents’ findings.

These vendors were integration-heavy requiring elaborate runbooks and at times even recommending direct access to our postgres database. This felt like indefinitely training and paying a Junior SRE opposed to empowering senior developers with SRE tooling. Not to mention, when we looked deeply at our usage, none of us were engaging with these AI-SRE vendors in their full-stack web applications, only with their Slack bots.

Following our Feb 11, 2026 public beta launch, we’ve had the pleasure of onboarding many customers to our platform. This has naturally led to more customer questions for post-sales, production, and engineers to collaborate on and investigate. We needed a force multiplier to scale.

The Context of Our Growth

Before our public launch, engineers were the primary beneficiaries of productivity initiatives, and workflows remained fragmented, with individuals relying on their own preferred approaches, particularly on non-engineering teams. Meanwhile, Claude Cowork and other enterprise AI offerings dramatically lowered the barrier to experimenting with AI-first ways of working.

In many ways, our strong in-person culture worked well. People aligned quickly by walking across the office and tapping a colleague on the shoulder, and the source of truth propagated through conversations or a verdict sent in Slack many hours later. But with rapid hiring, customer onboarding, and product usage came a surge of requests for ad-hoc engineering support, product clarifications, and time-sensitive hotfixes, and a handful of engineers ended up responding to the majority of them. Even after assigning dedicated Product and Engineering on-call owners, on-calls still needed to pull in domain experts to investigate bugs or evaluate feature requests, and the volume of context switching remained overwhelming. We had far more users and customers to support, but not yet the scalable processes or shared knowledge infrastructure to support them.

The Birth of Our First In-house Bot

Our first in-house tool was comically named after one of our engineers, Yunjia Xu. During one of those tedious on-call shifts, he created "YunjiaBot" in Slack as his personal assistant and one centralized interface for the requests landing on him. We had already tried adding an AI layer on top of our Pylon integration, but Pylon organizes customer-facing support workflows, while the real bottleneck was internal engineering escalation: pulling together the right code context, operational traces, product knowledge, and domain intuition to produce an actionable first pass.

YunjiaBot kept the workflow simple. Anyone could tag the bot in a Slack thread, whether for a product question, bug report, or system alert, and a Claude session would start automatically on his laptop with the relevant context collected. A custom agent would then triage the request and invoke shared skills or local workflows to inspect traces, search code, or create PRs. The name reflected what made the system work: its namesake had voluntarily distilled his own way of handling these requests, and the way Monaco engineers apply business context to technical problems, into automatable workflows and shareable company knowledge.

From the observers' perspective, the experience seemed magical. A few things stood out:

• Answers in minutes. Product questions were clarified with the latest code serving as the most up-to-date source of truth, and the domain owner only needed to confirm or augment the answer.
• Drafted PRs instead of tickets. Bugs and alerts arrived already investigated, often with justification and a fully tested fix the domain owner could approve, take over, or quickly tweak. The time to close an issue greatly decreased.
• Zero forcing functions. Within a week, people in public threads stopped tagging domain owners and started tagging YunjiaBot. It fit right into existing workflows and the quality spoke for itself.
• Surprisingly generalizable. PRs for Linear tickets, Notion documentation, hotfix commits, and whatever else people thought to try.
• It learned from its failures. When it went wrong (usually a new domain, or complex context search missing a causal connection) we could have it create a new skill for itself so the next iteration would be better.
• Vendors became plug-ins. Some products we were evaluating fit into this framework as just another connector, e.g. asking the Devin API to augment code search. People who deeply understand the organization's context and pain points can implement better frameworks than generalized solutions.
• High ROI on a short build. A few hours of development saved many hours within the first week, and the whole team was onboarded to a new mental model.

There is an important caveat, though. YunjiaBot lived entirely on Yunjia's laptop, and every action effectively ran as him, with security amounting to permission clicks, tool whitelists, and a final check of human eyes on responses. He was both the "human in the loop" and the "harness": the bot slept as soon as he did (or whenever his laptop lid closed), and it only got smarter when he sat down to keep developing it. That was only manageable tentatively during the experimentation period. The proof of concept stage stops here.

From YunjiaBot’s Ashes, HAL Rises

The grassroots initiative and rampant adoption made it clear: we needed a Slack bot with rich context on our codebase and sandboxed access to our telemetry, documentation, and systems, a high-leverage, always-up-to-date agent for product questions and investigating user issues. But we needed to avoid the pitfalls of YunjiaBot's prototype runtime, because running a widely used central service on a developer's laptop…

• … will not scale. As covered above, one engineer's approvals (and laptop lid) gated every response. We need something always on.
• … is a poor auth model. Any user in our Slack workspace can essentially trigger some action as that engineer, not as a machine user or themselves.
• … is not sufficiently monitored. We cannot reliably track uptime, network flow logs, resource contention, or commands run on a personal laptop. This is trivially easy on EC2 in the cloud.

We chose to run OpenClaw on a single VM using its Slack communication channel, with the OpenViking context engine plugin pointed at AWS Bedrock models, Titan v2 Embedding, and Nova LLM models over a VPC endpoint. We called the app HAL, an ominous reminder and nod to HAL 9000 in 2001: A Space Odyssey. HAL was given read-only access to Notion, Postgres, and Datadog, and read-write access to Linear (creating issues), Github (pushing branches up to a dozen lines for engineers to open PRs with), and obviously Slack.

If any of that scares you, good! Us too. We put a few controls in place that helped us mitigate the inherent risks:

• Field-level encryption in the database. Sensitive customer data of all kinds is envelope encrypted at the field level, so even where HAL's tooling can query operational data, customer content and credentials are only ever ciphertext to it: its EC2 IAM instance profile has no kms:Decrypt permissions, connections use short-lived IAM auth so no long-lived credentials can leak, and every query is audit logged.
• Everything is audited to Datadog. All bytes sent and received, every TCP connection established, and every single command any POSIX user runs on the VM. Should we need to look, we have a complete history of anything it or an operator has ever run.
• No public internet exposure. HAL's Slack app is configured for socket mode, so we never receive or validate HTTP requests from the public internet, and nothing has a public IP.
• Version-pinned dependencies, continuously scanned. Supply chain vulnerabilities remain the greatest threat, so OpenClaw itself and every other dependency on the machine is version-pinned, and AWS Inspector scans the node with findings flowing into our Vanta test pipeline to be remediated like any other CVEs.
• Everything is intentionally scoped to machine users. There is no attempt at any level to store users' access or refresh tokens; the agent never impersonates the requester.
• Sequestered security groups. HAL has no egress access to anything in our company network by default; it is allowlist-only, managed in Terraform.
• DNS sinkholing (planned). One further defense-in-depth is an adjacent Pi-hole service to sinkhole DNS requests for poor-reputation domains, or even scope resolution to an allowlist of (sub)domains.

This leaves us with static, centrally controlled permissions and network access, everything audited, and a minimal blast radius for leaked data or credentials. Considering HAL lives in only a few internal channels and we have no intention of distributing it, we are pretty happy with this implementation.

One of the more interesting components of HAL is the OpenViking context engine. We highly suggest checking it out (do note the project is AGPL-3.0, so not suitable for modification and monetization for proprietary purposes). Indexing our codebase and various Notion pages and their descendants with the very cheap Bedrock Titan and Nova models has been a game changer for the bot's ability to answer questions about how the product works or its capabilities, and the OpenViking team implemented the few issues we filed very quickly.

We have also begun surfacing the OpenViking server to developer laptops and Monacoder workspaces over our VPC to leverage their Claude Code context plugin for RAG. The idea is to incrementally build a hive-mind of rich developer context, shorten the cold-start agent exploration of our code for every session, and learn from our past mistakes. Admittedly, it is too early to tell how that is working and rather difficult to measure.

HAL, on the other hand, is easy to measure. Both OpenClaw and OpenViking have OpenTelemetry exporters publishing metrics to the on-instance Datadog receiver, and we have seen runaway adoption. Since we lit up full telemetry in mid-May, HAL has started nearly 6,900 agent runs across roughly 3,000 Slack threads and DMs. This is almost 700 distinct threads in the last two weeks alone and holds steady at 800–1,000 runs per week. Along the way it has made 55k model calls and 39k tool invocations, consuming 5.3 billion input tokens (roughly 70% of them served from prompt cache, which is what makes the economics work) and 18 million output tokens.

Recently, a Monaco engineer, Cem Tuncelli, extended HAL by adding another agent configuration to help our top-notch go-to-market team get less engineering-oriented and more customer-ready responses, and that agent has already produced its first two million output tokens, alongside a third configuration that triages EPD escalations. HAL motivated a new round of agentic internal tool innovations at Monaco.

A Second Look at the Promise

We should be honest about the developer experience working on HAL: it is painful. HAL has had some dropped responses, hallucinations, fights with YunjiaBot, and other issues a bit more than we would like, and chasing those issues down surfaced a few pains:

• Black boxes all the way down. When HAL gets something wrong, the fault could be in the model, the harness, the context engine, or a prompt. Each of those layers is opaque.
• Context injection needs a ton of tuning. Embedding-based retrieval works great at first, but context window decay soon exposes the need for real context management: capturing causal chains across code, traces, and conversations rather than whatever scores highest on similarity.
• Markdown-development. The OpenClaw agent SOUL.md, TOOLS.md, etc. are effectively load-bearing production code with no types and no tests.
• Slow iteration cycles. We have no evals or reproducible failure cases, and difficult metrics to define HAL’s success criteria other than a hive vibe check

All of this has bought us some real empathy for the AI-SRE vendors we wrote off at the top of this post. Everything above is their day job, except they are working with customers of different business shapes and without our shortcuts. They cannot bake in knowledge of one codebase or one Slack culture, which is exactly why their agents latch onto the first error they see and keep asking for more permissions and more runbooks. It is a genuinely hard product to build from the outside.

At the same time, going through this development cycle taught us the problem in the shape of our own business. We now know which context actually moves an investigation at Monaco, where our escalations really flow, and which failures are tolerable versus embarrassing. That understanding is something we would not have gotten from onboarding a vendor, and it is a large part of what made the in-house route tractable at all. The catch is that our own business is changing shape just as quickly, and the problem we solved for today's Monaco may not fit next year's, so there may well come a time when handing this off to a vendor is the right call. For now, the math still says run it ourselves.

In Conclusion

The agentic SRE space is moving so fast that who knows what the competitive landscape will look like in even 6 months time.

For teams weighing build versus buy, our guidance comes down to three questions. Is experimenting with the vendor actually cheap? If configuring it to work in your environment takes longer than a build would, you might as well build. Do you understand the problem you are solving? Time spent building internally is a forcing function for drawing out the real issues, while vendors can only ever solve the service layer. And can you consolidate? Before adopting point solutions everywhere, look for the chance to solve at a higher level in one framework, the way the vendors we evaluated ended up as plug-in connectors.

For companies building in the DevEx space (shout out to all of y'all trying to make our jobs easier), think clearly about your ICP. What team sizes is your solution best for? Can you work with a team's native telemetry setup and meet their security bar? How easy is it to deploy, and how quickly can a customer measure the ROI? Getting those answers right is exactly what would tip math like ours toward buy.

For Monaco, sinking one or two engineers for a week or two at a time on a project that provides equivalent value, in environments within our own purview, at ¼ to ⅓ the cost of the competition, has not felt like the wrong decision. HAL mostly only took one platform engineer, John Tracy, less than two weeks to roll out and has been pretty low-touch since. We have also made this an initiative to spark more innovative DevEx use cases internally, holding weekly DevEx office hours for people to bring and build on their ideas.

We truly do want to take these innovative projects as far as we can, as safely as we can, and are hiring to do exactly that. We have an open Developer Experience Engineer role for our Cloud Platform team and many more roles open across the rest of engineering. Check them out here!