Autonoma shipped a secure IoT service agent in one week
Autonoma is a leading IoT platform for industrial machinery manufacturers, helping their customers fully digitize their systems to enable data-driven services and sales. Technicians onsite need content to troubleshoot equipment issues, and any downtime is expensive. To diagnose and resolve issues more accurately and quickly, Autonoma decided to build an AI agent that provides instructions for tricky technical problems. The first prototype hallucinated, answered in the wrong language and could be talked out of its instructions. Vijil found all of it, fixed it, and hardened the agent; Autonoma took it to production on DigitalOcean in one week, saving a lean team months of development time and cost.
What the tests found
Autonoma began by developing a prototype retrieval-augmented generation system with Mistral Nemo Instruct as the generative model, using their online help centre as the knowledge base. Confounding factors during the initial build meant the agent produced a large number of hallucinations. It also failed to respond in the correct language — German or English — based on the user’s query, even though the documentation existed in both.
Vijil tested the agent’s performance and trustworthiness along three dimensions: reliability, security and safety. The performance tests showed that generated answers were not grounded, and were not being retrieved from the context with sufficient precision and recall. The security and safety tests found the agent was susceptible both to simple misuse — performing tasks outside its scope — and to malicious jailbreak attacks, which force an agent to ignore its operator’s instructions and follow the attacker’s instead.
How Vijil solved for trust
Four areas of improvement, none of which would have been sufficient on its own:
Mistral Nemo Instruct does a better job of inserting images into content directly, but has significant security vulnerabilities that make it prone to prompt injection attacks. Switching to Llama 3.1 8B lowered that vulnerability.
Vijil scraped Autonoma’s online help centre in both languages and extracted the plaintext content as well as the linked images and metadata, then separated English and German into different knowledge bases so the agent could cite content in the user’s own language.
Expanding the system prompt gave the agent a sufficiently detailed set of instructions; adding policies prevented misuse; and an explicit instruction to respond in the correct language let it switch between English and German. Vijil’s findings here fed back into DigitalOcean’s own knowledge-base documentation.
Vijil and Autonoma were working on the platform before DigitalOcean released its guardrails feature, so the safety policy and misuse-prevention instructions went into the prompt instead. Together with the new base model, that raised the agent’s score in Vijil’s evaluation of its trustworthiness.
Into production in one week
Autonoma took the agent from development to production in a week. It now produces accurate content with sufficient references and grounding in the original documentation, responds in English and German appropriately, and is resilient to common prompt-injection attacks. Autonoma has reduced their customers’ time to troubleshoot each incident and is positioned to scale to more customers faster.
The work ran the other way too: Vijil’s findings, with Autonoma’s, shaped DigitalOcean’s AI Platform — better documentation, and features prioritized that teams like Autonoma need to deploy agents confidently.
“For our small team, it’s been a real game changer. It let us dive right in and start building, without the need of a ton of resources.”
Those four fixes were hand-built in a week. Vijil Diamond runs the tests that found the failures, and Vijil Dome enforces a safety policy at run time rather than in a prompt.
This story was first published as a DigitalOcean customer story → · one-page PDF →