# Miru Miru provides infrastructure for configuration management tailored specifically for robotics applications. It enables teams to version, manage, and deploy application configurations at scale. The platform focuses on maintaining reliability and scalability across fleets of connected devices and robots. Its design supports complex environments where configurations must adapt to hardware variations, customer-specific deployments, and dynamic operational contexts. Miru is backed by venture funding and is structured to help robotics companies build, deploy, and update their systems efficiently. The platform differentiates itself with built-in version control, policy-based configuration overrides, and tools for seamless integration with existing CI/CD pipelines. These capabilities make Miru a foundational layer for organizations managing robotic devices or large distributed hardware systems. ## Core Products & Services ### Configuration Management Platform - **What it does:** Miru offers a robust configuration management platform that tracks, versions, and deploys configurations across large-scale robotic fleets. It ensures that every configuration change is logged, versioned, and recoverable, supporting rollback and continuous deployment needs. - **Who uses it:** Robotics companies and teams managing distributed devices or embedded systems use Miru to streamline and secure their configuration workflows. - **Key features:** - **Versioned Configuration Management:** Enables precise tracking of every configuration change with rollback capabilities. - **Dynamic Assignment:** Uses tags and metadata to assign configurations based on customer, hardware type, or location. - **Policy-Based Overrides:** Allows hierarchical configuration policies that intelligently override defaults at the fleet, customer, or device level. - **Seamless Deployment:** Integrates programmatically with CI/CD systems using the Miru SDK. - **APIs and UI Components:** Offers APIs and React components to build custom internal tools. - **Reliability:** Handles network instability, low bandwidth, and power-loss failures; the robot itself acts as the source of truth. - **Over-the-Air Deployment:** Supports OTA deployment by pushing containerized updates from a connected GitHub repository. ## Use Cases & Applications ### Robotics & Autonomous Systems - **Their needs:** Robotics organizations require consistent, scalable, and safe configuration management to keep fleets synchronized, minimize downtime, and respond to dynamic field conditions. - **How they use it:** Teams employ Miru to manage per-device or per-customer configurations, deploy updates over-the-air, and maintain synchronized system states across their devices. - **Results:** With integrated versioning, automation, and policy-based management, robotics teams improve reliability while decreasing manual deployment errors and integration overhead. ## Feature Deep Dive ### Versioned Configuration Management - **How it works:** Each configuration change is recorded using built-in version control, maintaining a full history. Teams can revert to previous versions if needed, ensuring traceability and controlled rollbacks. - **Benefits:** Enhances operational reliability and mitigates the risk of misconfigurations by maintaining recoverable configuration states. ### Dynamic Assignment - **How it works:** Configurations are tagged and associated with metadata fields like customer, hardware type, or deployment region. This allows automated assignment logic when deploying across heterogeneous fleets. - **Benefits:** Reduces manual targeting and enables consistent deployments based on contextual criteria. ### Policy-Based Overrides - **How it works:** Configuration policies cascade from defaults to more specific scopes (fleet, customer, device). Overrides are applied hierarchically, maintaining control without redundant definitions. - **Benefits:** Simplifies customization and enforces governance while preserving consistency across deployments. ### Seamless Deployment - **How it works:** Through the Miru SDK, configuration updates integrate with existing CI/CD pipelines enabling controlled, automated releases. - **Benefits:** Accelerates deployment workflows and ensures continuous delivery practices fit seamlessly with robotic system requirements. ### Over-the-Air Deployment - **How it works:** Miru supports OTA updates for containerized environments. Developers trigger updates by pushing new container versions to a linked GitHub repository, which Miru then deploys across connected devices. - **Benefits:** Enables remote, scalable updates without direct device intervention, maintaining uptime and reducing maintenance costs. ## Getting Started Organizations can begin by signing up for Miru and connecting it to their existing CI/CD pipeline using the Miru SDK. Prospective users can schedule an introductory session or book a demo through Mirus booking portal. Integration is designed to be seamless, with the SDK providing direct compatibility with CI/CD workflows for automated configuration management and deployment. ## Site Map Pages crawled from www.mirurobotics.com (1 total): - [Miru](https://www.mirurobotics.com/) --- # Miru Blog - LLMs.txt # This file provides all blog content for LLM consumption # Website Information - Site: Miru Blog - URL: https://www.mirurobotics.com/blog - Description: Robotics, AI, and software engineering insights from Miru # Blog Posts ## Threat Vectors in Robotics OTA Updates - URL: /blog/threat-vectors-in-robotics-ota-updates - Published: 2025-09-08T18:30:32.000+00:00 - Author: Vedant Nair - Tags: Insights - Summary: This post explains the series and how to help scaling robotics teams instill security into their OTA pipelines and defend themselves against imminent threats. - Excerpt: Welcome to the “OTA Security Series!”. This collection is designed to help scaling robotics teams instill security into their OTA pipelines and defend themselves against imminent threats. Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. Setting the Stage In their infancy, robotics teams use hacky measures to perform updates; to deploy software, they SSH into the robot, run git pull, and sometimes even compile code on the device! This workflow is m Welcome to the “OTA Security Series!”. This collection is designed to help scaling robotics teams instill security into their OTA pipelines and defend themselves against imminent threats. Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. Setting the Stage In their infancy, robotics teams use hacky measures to perform updates; to deploy software, they SSH into the robot, run git pull , and sometimes even compile code on the device! This workflow is manageable with one or two robots, but as the fleet grows and the product matures, it becomes untenable. Customers have an expectation of privacy for their devices. They don’t want their vendors to shell into their machines. And even if they didn’t care, the manual updating flow collapses once you’re juggling more than a handful of robots. Gone are the days of a single engineer tweaking a feature and quickly deploying it. Now, in the name of process 1 , there’s a concept of a release cycle : Software Engineers furiously program a new feature set, and when they’re finished, the code is ‘frozen’. Then, the Quality Assurance team tests the code to ensure it performs well on the robot. Finally, via an OTA update, the code is canary released to the fleet. It’s a structured and strict process with guardrails in place to promote safety and performance. The shift from hacky scripts to OTA pipelines solves the scaling problem, but creates new challenges. OTA updates open up a larger attack surface. Engineers get so focused on making deployments work at all that they often miss the security risks they’re introducing. This series is about these risks: how attackers exploit OTA pipelines, and what robotics teams can do to defend against them. In this first blog, we’ll look at the most common threat vectors, drawing lessons from past attacks in IoT and automotive. Man-in-the-Middle Attacks A man-in-the-middle (MitM) attack occurs when an attacker intercepts traffic between a robot and the update server. The attacker can observe, modify, or replace data in transit. In OTA, this can be crippling if the robot accepts a fake update with malicious software or leaks sensitive information. Attackers can: Eavesdrop: Monitor what is being sent back and forth, like software packages, logs, or data dumps. These messages can contain IP or private customer data, which the attacker exploits. Tamper: Swap in a malicious update or inject malware. Impersonate: Pretend to be either the server or the robot for later advantage. They have multiple ways to attack: DNS Spoofing Let’s say that our robot wants to talk to the URL https://cloudbucket.robot.com for its newest update. When the robot requests the DNS, an attacker can inject a fake DNS 2 , pointing the robot to a malicious server. Without strict certificate validation, the robot will download the attacker’s update without question. Compromised Wi-Fi Access Point Usually, when robots are deployed to customer sites, they are connected to a WiFi network, such as warehouse_wifi . This network will have an SSID. With a $50 Wi-Fi adapter and open-source tools 3 , an attacker can clone the SSID . If the robot trusts SSIDs blindly, it will connect to the attacker’s network. Now, the attacker is in the driver’s seat, and all the traffic will flow through their router. An attacker might not even need to create their own WiFi network to succeed. Often, in logistics and manufacturing facilities, local routers are outdated and inexpensive, making them easier to exploit. TLS Downgrade Before 2015, weak SSL/TLS made downgrade attacks common. An attacker could block HTTPS, force HTTP, and read or modify traffic in plaintext. If your robot doesn’t verify signatures on updates, it won’t notice the tampering. Today, modern TLS makes this attack rare, but it’s a reminder of why strict signature checks matter. Let’s look at a real-world example of a MitM attack, the Jeep Cherokee Hack of 2015 : In 2015, security researchers demonstrated a remote exploit against the Jeep Cherokee. The attack chain started through the car’s infotainment system, which was exposed to an internet-connected service for OTA updates, among other pipelines. The service had weak authentication and ran with high system privileges. Once compromised, the attacker could pivot from the infotainment unit onto the vehicle’s CAN bus. Over the CAN bus, they could send arbitrary messages to critical ECUs (engine, brakes, steering). Famously, the researchers cut the transmission of a Jeep while it was driving. Chrysler had to recall 1.4 million of its vehicles! Unauthorized Firmware Access If an attacker can gain the ability to read, modify, or replace firmware, they own the system. For OTA, that risk shows up in two ways: Extract: Attackers pull firmware off the device (via debug ports or external storage) to reverse-engineer it or look for vulnerabilities. Inject: Attackers load their own firmware onto your robot. Unlike application code, firmware persists across reboots and can even disable future updates. Firmware attacks can be physical or remote: Physical access is outside the scope of OTA, but teams should still be cognizant of it. Many early-stage companies start by shipping developer kit SoCs ( Jetson AGX Orin Dev Kit). These dev kits often ship with the JTAG/UART debug ports exposed. So stay vigilant! For OTA, the concern is insecure update paths. If your robots accept unsigned images or rely only on weak checks, an attacker can push malicious firmware remotely. The importance of authentication was highlighted by the D-Link Router Firmware Vulnerability in 2019 : In 2019, Fortinet researchers found that several D-Link routers gave the ability to install arbitrary system commands via a script in the router’s firmware. Because there was no authentication needed, commands could be executed as the root user! D-Link responded days after the disclosure by declaring the affected models End-Of-Life. This meant that unless users replaced their hardware, they were stuck running exploitable firmware. Downgrade / Replay Attacks A downgrade or replay attack forces a robot to install an older version of software. Because the image was once validly signed, it still passes cryptographic checks, but it may contain known vulnerabilities. If an attacker can get your robot to run on a software version with known CVEs, their job becomes easier. They don’t have to devise any zero-days. Instead, they can rely on often publicly documented details to attack. Replay attacks often piggyback on other weaknesses: Man-in-the-middle or DNS spoofing can redirect a robot to download an older package. Local caches or storage of older versions let attackers trigger a reinstall. Weak validation that checks only the signature, not the version or freshness, allows the downgrade to succeed. Let’s see how this affected Android , the biggest mobile OS in the world: A series of these exploits plagued Android devices in the 2010s. Every month, Android devices receive OTA updates. Each update would be a signed zip file that passes signature verification. However, for a long time, Android only checked that the signature was valid, not whether the update was newer than what was currently installed. Attackers could grab an older, signed update from OEM archives or a device cache and push it back onto the phone. The update would pass signature checks, but re-enable known vulnerabilities. One infamous case was the Stagefright bug (2015), where a crafted MMS could execute code on the device . Rollbacks let attackers bring that bug ‘back to life’ even after it had been patched. To remedy this, starting in Android 8.0, Google introduced rollback protection . This meant that devices kept a rollback index in a trusted, tamper-resistant storage. Each time an OTA update was installed, it would burn a higher version number into that rollback index. If you tried to downgrade it with a replay attack, the bootloader would fail to boot, even if the signature is valid. Since most robotics fleets have never gotten to real scale, they haven’t experienced this problem. They allow rollbacks if the deployed software is failing. But mature organizations don’t do this. For example, Amazon consumer devices only ‘roll forward’. This is a best practice that limits the downside of reintroducing CVEs . Conclusion We’ve looked at three of the biggest OTA threat vectors: man-in-the-middle attacks, unauthorized firmware access, and downgrade/replay exploits. Each has burned IoT and Automotive companies in the past, and while there haven’t been any high-profile exploits in robotics, we’d like it if none of our readers set a precedent! OTA updates are a crucial component of scaling a fleet of robots, but they also open the door for attacks. The best teams prioritize security best practices while building up their pipeline to mitigate potential threats. In the next blog, we’ll talk about how to do just that. We’ll cover the best practices to keep OTA secure, so your robots can scale safely. Series Index Threat Vectors Best Practices [coming soon] OTA Security Diagram [coming soon] An engineering manager’s favorite word! ↩ DNS injection is when an attacker tampers with the DNS lookup process, tricking a device into resolving a domain name to the wrong IP address. This can happen if an attacker compromises the DNS resolver itself, or slips forged DNS responses into the traffic before the legitimate ones arrive. ↩ See Aircrack-ng and hostapd . ↩ --- ## Config Wars - Chapter 5: What to Choose? - URL: /blog/config-wars-chapter-5-what-to-choose - Published: 2025-08-04T17:40:24.000+00:00 - Author: Vedant Nair - Tags: Insights - Summary: This post compares configuration schema approaches and how they fit a robotics stack. It highlights trade‑offs in validation, composition, tooling, and long‑term maintenance. - Excerpt: Welcome to “Choosing a Schema for Your Robotics Stack!” This series is designed to help scaling robotics teams understand why schemas are useful for their configuration management and navigate which schema is best suited for their stack. In this blog, we’ll wrap our deep dive into schema languages. We’ll compare JSON Schema, CUE, and Protobuf head-to-head, share some honorable mentions for schema languages, and finally, discuss which schema language Miru chose to support for our config manageme Welcome to “Choosing a Schema for Your Robotics Stack!” This series is designed to help scaling robotics teams understand why schemas are useful for their configuration management and navigate which schema is best suited for their stack. In this blog, we’ll wrap our deep dive into schema languages. We’ll compare JSON Schema, CUE, and Protobuf head-to-head, share some honorable mentions for schema languages, and finally, discuss which schema language Miru chose to support for our config management system. Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. Intro: A Quick Refresher Throughout this series, we’ve explored the importance of configuration management in robotics and why schemas are useful in helping scale these config systems: Configurations (Configs) are an important part of the infrastructure equation when scaling a fleet of robots. Configs are a set of parameters, injected at runtime, that influence our robot’s behavior. They help us adapt to different hardware versions, operating environments, and use cases without needing to modify application code. Instead, we can dynamically swap configs in and out, giving our system modularity and scalability. The ankle biter that scaling robotics teams face with configs is that they don’t use a schema. They usually use YAML or JSON files, which are error-prone and manual; it’s easy for misconfigurations to be deployed to the robot, causing the application to crash. They also lack any tooling that helps scaffold your config system as you scale. In this series, we introduced schema languages as the best way to scale and harden your config system. Schema languages provide a structure for configs: they define valid values, set constraints, and support typing. They also have a variety of validation and code generation tools that make it easy to perform DevOps functions around your configs. We spent this series carefully examining JSON Schema, CUE, and Protocol Buffers (Protobuf). JSON Schema JSON Schema is a standard for describing and validating the structure of JSON data. Think of it as a contract that defines what a valid JSON object looks like, specifying types, required fields, constraints, and structure. JSON Schema is one of the most widely adopted schema languages today. It boasts a multitude of strengths, including a mature ecosystem, robust third-party validation libraries, and comprehensive tooling support across various languages. It lacks some functionality regarding overrides, templating, and logic. Teams usually pair it with an external templating engine, such as Jsonnet or Jinja, to fill these gaps. If your team is already using JSON or YAML and wants strong validation and wide compatibility, JSON Schema is a solid choice. CUE CUE, which is short for “Configure, Unify, Execute” was developed at Google by Marcel van Lohuizen, one of the original creators of Go. The original motivation stemmed from Google's internal struggles in managing complex, large-scale configurations for infrastructure and applications. JSON and YAML were too weak for expressing constraints, logic, or composition, so CUE set out to solve that. It combines validation, logic, and composition into a single language, making it especially useful for teams managing complex and dynamic configs. CUE’s strengths come from its logic and computation. You can write conditionals, compute derived values, and merge overrides natively. In terms of logic and expressiveness, CUE offers more out of the box than any other schema language we’ve evaluated. The downside is that CUE has limited language support outside of Go. While the ecosystem is growing rapidly, this issue will persist in the short to medium term. Teams using CUE alongside other languages will need to translate CUE into JSON Schema to unlock code generation. If you’re looking for a config language that combines logic, inheritance, and validation in one place, CUE is an option worth exploring Protocol Buffers (Protobuf) Protobuf was created by Google in the early 2000s. The team needed a compact, fast, and language-agnostic way to serialize structured data. The performance offered by XML and JSON wasn’t cutting it, both in terms of size and speed. It offers best-in-class code generation and excellent support for static typing and schema evolution. With plugins like protovalidate , you can also add runtime validation to match the flexibility of other schema languages. The glaring downside with Protobuf is that it wasn’t designed for configuration. It lacks native support for overrides, templating, and logic. You’ll need to handle inheritance and composability manually or with external tooling. If you need compact, versioned, type-safe configs that compile down to a binary and integrate with firmware or message-passing systems, Protobuf is a strong choice. Before we get to the grand finale (the part where we compare these three schemas and everyone rides into the sunset), it’s worth mentioning a few tools that are worth knowing about, but just fell short of deserving a deep dive Honorable Mentions Pkl: Pkl (pronounced “Pickle”) is a modern config language developed by Apple. It combines JSON-like syntax with a real type system, a modular system, and even object-oriented features like inheritance and visibility modifiers. It stands out for its strong support for reusability and testing. You can write unit tests for your config and create base classes for components. The ergonomics feel like configs as code. It has positioned itself as a ‘configuration DSL’, with a promising direction. The problem is that it’s still young. The tooling ecosystem is limited, especially for robotics-adjacent languages. Dhall: Dhall is a functional programming language specifically designed for configs. It’s strongly typed and deterministic; you can write configs like code with imports, functions, types, and guarantees. However, it isn’t Turing-complete. One of Dhall’s core promises is safe refactoring. You can define types once, import them across files, and be sure that nothing breaks when you change things upstream. It also supports normalization and diffing, which is great for reviewing config changes in CI. Unfortunately, Dhall has a steep learning curve. Its syntax is unique: lambdas look like λ(x : Text) → x ++ "!" , and even simple expressions can feel complex. For most teams, the tradeoff to learn a foreign syntax isn’t worth it. It's a nail in the coffin that, like Pkl, it lacks codegen support for both Rust and C++. Jsonnet + Jinja: We’ve referenced these two a few times throughout the blog. They’re powerful, but are only an honorable mention because they aren’t a schema language. They’re templating engines. They’re widely used across robotics teams to generate structured configs. Jsonnet is a data templating language (surprise, surprise, created by Google!) that lets you write JSON with variables, conditionals, and functions. As we showed previously, you can write your config in Jsonnet and then validate it against JSON schema. Jinja is a string-based templating engine for Python. It’s less structured than Jsonnet in that you’re just generating text, but it’s familiar and easy to use in Python. You can loop over variables and render JSON/YAML. These tools are great for creating overrides or conditional logic, but by themselves, aren’t robust enough. You still want validation, type checking, and codegen offered by schema languages. Use them together! Putting it All Together We’ve now seen how JSON Schema, CUE, and Protobuf handle validation, codegen, overrides, and their associated DevX. Let’s look at how they stack up head-to-head. Schema Validation CUE is the only schema format that has native validation and programmable logic constraints. With the other two, you’ll have to use an external validation method. This isn’t inherently bad, but it’s much easier when you can validate configs as code using CUE. Code Generation It’s a close call here between Protobuf and JSON Schema. Protobuf wins because its codegen tooling is more robust. C++, Python, and Rust are all supported out of the box. Its outputs are also better; JSON Schema generates data models, which are useful but not as powerful as the serialization and deserialization, as well as getters and setters, that Protobuf generates for you. Developer UX While CUE has the most expressive syntax, JSON Schema may have the best UX. It’s familiar, supported by every major editor, and integrates easily into workflows already built around YAML or JSON. Protobuf is more rigid and requires a build step, and CUE’s syntax can be tough to onboard. Community Ecosystem JSON Schema has the richest community ecosystem. You can find supporting libraries across every language you can think of. It’s not just the codegen and validation use cases that we’ve mentioned. If you do some more digging, you’ll find some more esoteric examples like this reverse gen, NixOS to JSON Schema converter 🙂 Composability / Overrides Again, CUE wins here because of its native support. You can override layered config and merge config fragments using CUE’s unification model. CUE ensures that overrides don’t accidentally violate constraints, and conflicting fields will fail to unify unless they’re compatible. When to Use What CUE If you want built-in validation and native override logic, CUE is your best choice. Because CUE is self-encapsulating, you won’t have to use any third-party libraries. However, its codegen is weak for robotics-specific languages. This means you’ll be on your own to build this tooling. Protobuf If you’re already using Protobuf for telemetry or message parsing, it makes sense to use it for your config system as well. It has robust schema enforcement, versioning, and codegen. JSON Schema If you’re already familiar with YAML/JSON, JSON Schema is a good option. It has well-supported validation and tooling integration, and boasts a strong community of maintainers. Wrapping it Up In this series, we walked through configuration schemas, why they are important for scaling robotics teams, and introduced you to a few schema languages (along with some decision criteria). What language do we prefer for Miru’s config management infrastructure? Well, all of them. All the options that we’ve discussed are perfectly reasonable, and we plan to support them. Resources are limited, though, so we had to choose one to focus on initially. We decided on JSON Schema. Here’s why: It was the easiest for teams to adopt. If they were using schema-less JSON or YAML, the migration wasn’t too painful. It has a large, mature community. It supports codegen for all the languages we care about (C++, Rust, and Python) It has a litany of third-party tools from validators to templating engines and more. For our Protobuf and CUE lovers, worry not, we’ll work quickly to support them! In the meantime, give JSON Schema + Miru a whirl to see how the combination makes scaling config management painless. Happy building, and feel free to reach out with any questions! Config Wars Series Index Config Wars - Chapter 1: Intro to Schemas Config Wars - Chapter 2: JSON Schema Config Wars - Chapter 3: CUE Config Wars - Chapter 4: Protocol Buffers (Protobuf) Config Wars - Chapter 5: What to Choose? --- ## Using Unix Sockets in Robotics - URL: /blog/using-unix-sockets-in-robotics - Published: 2025-08-01T16:20:18.000+00:00 - Author: Vedant Nair - Tags: Guides - Summary: This post explains the core idea and why it matters in production. It ends with steps you can use without re‑architecting your stack. - Excerpt: Introduction When we were building the Miru Agent and SDK for our configuration management system, we hit an early design decision: how should these two components talk to each other? Both would run on our customers’ robots, the Agent as a systemd service, and the SDK integrated into their application code. That meant we had to be fast and lightweight. Our customers are running large models on constrained edge computers. They don’t have room for unnecessary CPU overhead or bloated dependencies Introduction When we were building the Miru Agent and SDK for our configuration management system, we hit an early design decision: how should these two components talk to each other? Both would run on our customers’ robots, the Agent as a systemd service, and the SDK integrated into their application code. That meant we had to be fast and lightweight. Our customers are running large models on constrained edge computers. They don’t have room for unnecessary CPU overhead or bloated dependencies. Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. We’ve spent years building SaaS, so our first instinct was familiar: spin up a REST API on localhost , expose a few endpoints, and call it a day. But when we dug deeper into the requirements, that approach didn’t hold up. We didn’t want the overhead of the full TCP/IP stack, managing ports, serialization, or localhost to monitor. And that’s how we landed on Unix sockets. In this blog, we’ll explore what Unix sockets are, how they work, and why they work wonderfully for Miru’s architecture! What is a Socket? Before diving into Unix sockets, let’s zoom out a layer above. What are sockets? A socket is an endpoint for communication between two programs. You can think of it as a virtual pipe that allows us to send and receive data, either over a network or locally. (Credit Aritra Pain, LinkedIn) At the OS level, a socket is just a file descriptor, which is the same abstraction that Unix uses for files, and standard input ( stdin )/output ( stdout ) streams. In Unix, you read() and write() to a file. You do the same with a socket. The abstractions from the programmer's perspective are similar, but the striking difference is that sockets are ‘active’. When you write to a file, it’s saved to disk. Another program can open it and read the data later. A socket, on the other hand, is a live, bidirectional channel. Both sides can send and receive data in real time. Where Are Sockets Used? Sockets are ideal when you need to live, two-way channel between ‘components’ (machines, containers, or processes). You’ll find them everywhere: Client/server architecture When your browser loads a webpage, it opens a socket to the web server. In fact, you're reading this blog through one right now! Inter-process communication (IPC) on the same device Many core Linux services, like journald and systemd talk to each other using Unix domain sockets. Remote procedure calls (RPC) between distributed systems gRPC , for example, uses sockets under the hood to call functions on remote machines as if they were local. Streaming systems Sockets are used to stream tokens from an LLM inference server to an application like ChatGPT. If their power and ubiquity haven’t convinced you yet, let’s dive into why you should be using sockets in your systems. Why Should I Be Using Sockets? Why should you? Well, If you're communicating over a network, you don't have much of a choice. Protocols like HTTPS, gRPC, and WebSockets all rely on TCP sockets under the hood, which is the default layer. But when you’re working with local communication, you do have options. Unix domain sockets are often the best choice, but there are other inter-process communication (IPC) mechanisms too. Let's walk through the main ones: Alternatives 1. Pipes Type : Unidirectional stream How it works : One process writes to one end, another reads from the other. It’s a one-way channel. Example : stdout piped into another process via | in a shell. It’s simple to set up, and great for chaining processes. 2. Message Queues (POSIX or System V) Type : FIFO messaging system. How it works : One process puts a message into the queue, and another takes it out. Messages can be tagged with IDs or metadata. It’s useful for structured communication and deferred processing. However, it requires a lot of customization: you’ll need to avoid stale messages or overflows. 3. File Watchers (e.g., inotify on Linux) Type : File system-based signaling. How it works : One process writes to a file. Another watches the file or directory for changes. These are easy to understand and set up, but not suited for real-time or high-frequency data exchange. Unix sockets hit the sweet spot. Compared to other local IPC methods, sockets: Support f ull duplex communication (vs. pipes) Require less setup and system configuration (vs. message queues) Are faster and more reliable (vs. file watchers) How Do Sockets Work? Let’s dive into the full lifecycle, from setup to close. Let’s walk through it from both the server and client perspectives. (Credit, GeeksforGeeks Socket Programming in C - GeeksforGeeks ) Server The server initially sets up the socket, announces its presence, and waits for incoming connections. socket() First, the server creates a socket using the socket() system call. This returns a file descriptor that represents the socket in future operations bind() This binds the socket to a known address. For TCP, that’s an IP address and port For Unix sockets, it’s a file path like /run/miru/miru.sock. listen() The socket is marked as passive, which means it is ready to accept incoming connections. Now, the server “starts listening.” accept() The call is blocked until a client connects. Once it does: The OS creates a new socket specifically for that connection (this is a 1:1 socket relationship) The original socket continues to listen for new clients read() / write() Now, the server can send and receive data with the client using I/O system calls. close() When the session is done, the socket is closed. Client The client initiates the connection to the server. socket() Same as the server, the client creates a socket and gets back a file descriptor. connect() Attempts to connect to the server’s address (IP + port or file path). If the server is listening, the OS establishes the connection between them. read() / write() The client can now send and receive data, using the same system calls as the server. close() Once done, it closes the socket. Whether it’s TCP or Unix, the OS is responsible for managing the actual connection. It routes data between processes, handles buffering, and enforces permissions. For TCP, the OS also manages the networking stack. Unix vs. TCP Sockets Now that we understand how sockets work, we can comb through the differences between Unix (local) and TCP (over network). TCP Sockets (Credit, Sushant Gaurav TCP Server-Client Implementation - Scaler Topics ) When a TCP connection is established: The client calls connect() and specifies the server’s IP and a port (e.g. 127.0.0.1:8080 ) The server has already called bind() and listen(), and is waiting on the specified port. The OS handles the TCP/IP Stack It performs the handshake (SYN → SYN ACK → ACK) The stack also ensures that all data (which is broken into packets) is delivered in order, without loss, and without duplication. At the application layer, all of this is abstracted into a simple file descriptor. Let’s look at how we’d initialize a basic IPv6 TCP server and client in C++ using raw socket APIs. Server #include <iostream> #include <cstring> #include <unistd.h> #include <arpa/inet.h> #include <netinet/in.h> int main() { int server_fd = socket(AF_INET6, SOCK_STREAM, 0); if (server_fd < 0) { perror("socket failed"); return 1; } sockaddr_in6 addr; std::memset(&addr, 0, sizeof(addr)); addr.sin6_family = AF_INET6; addr.sin6_addr = in6addr_any; // Listen on all IPv6 interfaces addr.sin6_port = htons(8080); // Port 8080 if (bind(server_fd, (sockaddr*)&addr, sizeof(addr)) < 0) { perror("bind failed"); return 1; } if (listen(server_fd, 5) < 0) { perror("listen failed"); return 1; } std::cout << "Server listening on port 8080 (IPv6)" << std::endl; sockaddr_in6 client_addr; socklen_t client_len = sizeof(client_addr); int client_fd = accept(server_fd, (sockaddr*)&client_addr, &client_len); if (client_fd < 0) { perror("accept failed"); return 1; } char buffer[1024] = {0}; read(client_fd, buffer, sizeof(buffer)); std::cout << "Received: " << buffer << std::endl; const char* reply = "Hello from server!"; send(client_fd, reply, strlen(reply), 0); close(client_fd); close(server_fd); return 0; } We can see that the server is using AF_INET6 to tell the OS to use IPv6 sockets and SOCK_STREAM to ensure lossless, ordered delivery. It uses bind() to associate the socket with the specified IP address and port. Then it uses listen() and accept() to prepare itself to receive connections. Client #include <iostream> #include <cstring> #include <unistd.h> #include <arpa/inet.h> #include <netinet/in.h> int main() { int sock = socket(AF_INET6, SOCK_STREAM, 0); if (sock < 0) { perror("socket failed"); return 1; } sockaddr_in6 addr; std::memset(&addr, 0, sizeof(addr)); addr.sin6_family = AF_INET6; inet_pton(AF_INET6, "::1", &addr.sin6_addr); // Connect to localhost (::1) addr.sin6_port = htons(8080); if (connect(sock, (sockaddr*)&addr, sizeof(addr)) < 0) { perror("connect failed"); return 1; } const char* msg = "Hello from client!"; send(sock, msg, strlen(msg), 0); char buffer[1024] = {0}; read(sock, buffer, sizeof(buffer)); std::cout << "Received from server: " << buffer << std::endl; close(sock); return 0; } The client uses connect() to establish a connection with the server. From there, both the client and server use send() and read() to exchange data. Finally, when they are done communicating, they close() the socket. Unix Sockets Unix sockets, also called Unix domain sockets, are used when two programs need to talk on the same machine. They’re best for fast, secure IPC. To establish a Unix socket: The server calls bind() with a file path ( /run/miru/miru.sock ). The client calls connect() using that same path. The kernel routes the data directly between the processes with no TCP/IP overhead. Unlike TCP, Unix sockets transfer data directly through memory buffers. There’s no serialization/deserialization associated with TCP/IP The socket can be permissioned using standard Unix tools like chmod or chown Data is exchanged with read() and write() Note that there are two types of Unix sockets: Stream sockets (STREAM_SOCK) behave like TCP sockets. They ensure that data is transmitted through a reliable, ordered stream. Datagram sockets (SOCK_DGRAM), like UDP, have no built-in ordering or delivery guarantees. Like TCP, we’ll use socket() , bind() , listen() , and accept() on the server side, but instead of binding to an IP and port, we bind to a file path. Server #include <iostream> #include <cstring> #include <unistd.h> #include <sys/socket.h> #include <sys/un.h> #define SOCKET_PATH "/tmp/miru.sock" int main() { int server_fd = socket(AF_UNIX, SOCK_STREAM, 0); if (server_fd < 0) { perror("socket failed"); return 1; } sockaddr_un addr; std::memset(&addr, 0, sizeof(addr)); addr.sun_family = AF_UNIX; std::strncpy(addr.sun_path, SOCKET_PATH, sizeof(addr.sun_path) - 1); unlink(SOCKET_PATH); // Delete existing socket file if present if (bind(server_fd, (sockaddr*)&addr, sizeof(addr)) < 0) { perror("bind failed"); return 1; } if (listen(server_fd, 5) < 0) { perror("listen failed"); return 1; } std::cout << "Server listening on " << SOCKET_PATH << std::endl; int client_fd = accept(server_fd, nullptr, nullptr); if (client_fd < 0) { perror("accept failed"); return 1; } char buffer[1024] = {0}; read(client_fd, buffer, sizeof(buffer)); std::cout << "Received: " << buffer << std::endl; const char* reply = "Hello from Unix socket server!"; send(client_fd, reply, strlen(reply), 0); close(client_fd); close(server_fd); unlink(SOCKET_PATH); return 0; } AF_UNIX tells the OS that this is a Unix domain socket. We’re still using a SOCK_STREAM socket type. Client #include <iostream> #include <cstring> #include <unistd.h> #include <sys/socket.h> #include <sys/un.h> #define SOCKET_PATH "/tmp/miru.sock" int main() { int sock = socket(AF_UNIX, SOCK_STREAM, 0); if (sock < 0) { perror("socket failed"); return 1; } sockaddr_un addr; std::memset(&addr, 0, sizeof(addr)); addr.sun_family = AF_UNIX; std::strncpy(addr.sun_path, SOCKET_PATH, sizeof(addr.sun_path) - 1); if (connect(sock, (sockaddr*)&addr, sizeof(addr)) < 0) { perror("connect failed"); return 1; } const char* msg = "Hello from Unix socket client!"; send(sock, msg, strlen(msg), 0); char buffer[1024] = {0}; read(sock, buffer, sizeof(buffer)); std::cout << "Received from server: " << buffer << std::endl; close(sock); return 0; } Performance is higher, and the overhead is lower because there’s no networking stack involved. But as we’ll see, there are tradeoffs between Unix and TCP sockets. Tradeoffs Between Unix and TCP Sockets From an engineer’s perspective, interacting with a Unix socket vs a TCP socket looks the same. But under the hood, they have some important differences. Speed & Performance Unix sockets are often 3-10x faster than TCP sockets because they avoid the entire networking stack. We’ve discussed the TCP/IP stack (handshakes, packetization, and congestion control) that causes overhead for sending data. Based on benchmarks, for the same connection, a Unix socket round trip takes 1-5 microseconds, whereas a TCP socket round trip takes 15-30 microseconds. Although microseconds are minuscule, with real-time systems running thousands of processes per second, like robots, these operations can add up. Security Unix sockets inherit file-level permissions from the host system: You can use chmod / chown to lock access down to a user or group. No open ports means no external actors can attack your robot If you’re an operating in a firewalled environment, you won’t face any trouble TCP sockets, on the other hand, live in the network namespace, even if bound to localhost . That means: They’re exposed to firewall rules. You must secure them with TLS or socket-level ACLs if you care about hardening. You can build in fine-grained control like mTLS Setup Effort TCP sockets require more effort to integrate. You’ll need to choose and expose a port. You’ll have to handle IP binding, firewalls, and TLS setups. With Unix, you just have to bind to a file path, and that’s it! Something we’ve seen teams do during their development cycle is to start with Unix sockets early in the dev cycle to allow for fast iteration, but once you’re ready to productionize, you can take the time to set up a TCP socket. Flexibility This is where TCP wins. Unix sockets only work within the same host. You can't use them across machines or even across containers unless you explicitly share the socket file. TCP sockets are universal. They work across: Physical hosts Containers Pods in Kubernetes clusters If you ever need remote communication, TCP is your only option. Tooling & Ecosystem TCP is the default for most networking tools and protocols. Therefore, it has a massive ecosystem. Encryption: TLS/mTLS with OpenSSL, etc. Load balancing: Built-in support with Envoy, HAProxy, and NGINX. Monitoring: Works out of the box with Wireshark, tcpdump , and Prometheus exporters. Debugging: Easy to probe with curl , telnet , netcat , etc. Unix sockets are simpler, so the tooling is simpler too. Since there’s no networking stack involved, you can debug with basic Unix tools: Use ss -x or lsof to see which processes have the socket open. Use socat or nc -U to send test messages through the socket. How Miru Uses Sockets Back to our original problem statement. When we were architecting Miru, we knew that we needed the Miru Agent and SDK to communicate with each other quickly, securely, and reliably. The Miru Agent is an open-source, systemd service that runs on the customer’s robot. It’s responsible for identifying and pulling the latest version of the config. The Miru SDK is also open-source and is currently supported in C++, with Python and Rust coming soon. It’s used in our customer’s application code. Whenever there’s a new config available, the Agent needs to notify the SDK immediately so that the application can consume it without delay. Why We Chose Unix Sockets We had three core requirements: speed, security, and simplicity. ⚡ Speed & Low Overhead Our customers run heavy workloads on resource-constrained edge computers (think large PyTorch models running on a Jetson board). They were concerned with not adding any bloat to their system. This meant that adding a localhost server or sending data through the networking stack was overhead we couldn’t afford. With Unix, we could move data with blazing speed with almost zero CPU overhead. 🔒 Security Robots are safety-critical systems. If they go ‘rouge’ they have the potential to endanger people, facilities, and cause critical damage. Unix sockets gave us a secure channel scoped only to the local machine. There are no ports open or external exposure. And with Unix permissions, we can restrict access so that Miru Agent can only read/write to certain file paths. Combine that with the fact that our Agent and SDK are fully open-source, and customers have complete visibility and confidence over what is running on their device. 🪶 Simplicity We wanted to ship fast, and that meant we wanted our setup to be as functional and straightforward as possible! With a Unix socket, we could bind to /run/miru/miru.sock . This made the implementation lightweight, minimal, and easy to reason about. A Unix socket was the right fit between our Agent and SDK. It gave us the performance to run on our customers’ devices, met their security standards, and provided the simplicity we needed to ship fast. Conclusion We’ve covered what Unix sockets are, how they work, and why they’re so powerful in robotics and embedded systems. We built the Miru Agent and SDK for fast, secure, and efficient on-device communication. If you’re building similar software with tight constraints, be sure to give Unix sockets a try! And if you’re looking for a configuration management tool built to be highly performant, check out Miru . --- ## Config Wars - Chapter 4: Protocol Buffers (Protobuf) - URL: /blog/config-wars-chapter-4-protocol-buffers - Published: 2025-07-30T16:48:58.000+00:00 - Author: Vedant Nair - Tags: Insights - Summary: This post compares configuration schema approaches and how they fit a robotics stack. It highlights trade‑offs in validation, composition, tooling, and long‑term maintenance. - Excerpt: Welcome to “Choosing a Schema for Your Robotics Stack!” This series is designed to help scaling robotics teams understand why schemas are useful for their configuration management and navigate which schema is best suited for their stack. Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. In this blog, we’ll do a deep dive into Protocol Buffers (Protobuf), a data serialization format turned popular choice for schemas in robotics. We’ll discuss its origi Welcome to “Choosing a Schema for Your Robotics Stack!” This series is designed to help scaling robotics teams understand why schemas are useful for their configuration management and navigate which schema is best suited for their stack. Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. In this blog, we’ll do a deep dive into Protocol Buffers (Protobuf), a data serialization format turned popular choice for schemas in robotics. We’ll discuss its origin, features, and some of its pros and cons. History / Motivation: Protobuf was created by Google in the early 2000s. The team needed a compact, fast, and language-agnostic way to serialize structured data. The performance offered by XML and JSON wasn’t cutting it, both in terms of size and speed. To keep up with their rocket ship growth, Google needed to efficiently pass structured messages across thousands of internal services written in various languages. Protobuf was created to solve these problems: Efficient cross-language serialization. They needed to encode and decode structured data across systems and languages (C++, Java, Python, Go, etc.) Smaller payload sizes. Protobuf produced a binary format that is smaller and more efficient than XML/JSON Schema guarantees. Define data schemas with . proto files, allowing for type-safe communication between services Forward/backward compatibility. With services constantly changing, schemas need to safely evolve over time. With Protobuf, you can add, remove, or modify fields without breaking old binaries Protobuf wasn’t designed for configuration, but over the years, it has been co-opted for this use case in environments where binary formats and strong typing matter. It’s become popular in the world of embedded systems and robotics. That’s great news for us! Usage in Practice: Protobuf operates in the infrastructure layer of many distributed systems: APIs & microservices: Especially with gRPC, which uses Protobuf as its IDL (Interface Definition Language) and serialization format. Internal service communication: For large-scale systems, Protobuf is often used to reduce payload size and parsing overhead Embedded systems: Robotics, IoT, etc., anywhere where bandwidth is constrained Protobuf wasn’t built for configuration, but many teams use .proto files as config schemas, given that they want: Typed configs and are okay with compiling the schema before runtime To control the full pipeline (generate configs at build time and deploy as binaries) Community / Ecosystem: Protobuf is still led by Google with a mature and production-grade ecosystem. It’s been around for ~15 years, so you’ll find wrappers, plugins, and tooling across every major language. GitHub Activity: Tooling: IDE Support VS Code: Extensions for syntax highlighting, autocomplete, and linting. IntelliJ, GoLand, etc: Built-in .proto support for major languages. CLI Tools protoc : the compiler that turns .proto files into code Protobuf’s CLI, buf , allows for: Linting Breaking change detections Schema Versioning Dependency management protovalidate : Adds runtime validation rules. Useful since native Protobuf doesn’t have strong validation. Feature Evaluation Validation Model: Protobuf’s native validation model is minimal. It enforces types but doesn’t support constraints like: Min/max values String length Regex patterns Cross-field logic For a real validation, most teams use protovalidate , a buf plugin that lets you define rich constraints directly in your .proto files. Example with protovalidate import "buf/validate/validate.proto"; message JointLimits { string joint_name = 1 [(buf.validate.field).string.min_len = 1]; float min_position = 2; float max_position = 3 [(buf.validate.field).float.gt = 0]; // Ensure max_position > min_position in application logic } This snippet ensures: joint_name can’t be an empty string. max_position must be > 0. You can enforce max_position > min_position at runtime via generated validation code. protovalidate allows you to validate at runtime and unlock most of the validation capability from both JSON Schema and CUE. Code Generation: Protobuf has excellent code generation. You define your schema once in a .proto file, and you can easily generate typed, structured code in your language of choice. Let’s say you have a schema for camera parameters on a robot: message CameraConfig { float focal_length = 1; int32 resolution_width = 2; int32 resolution_height = 3; bool auto_exposure = 4; } You can generate a C++ class using the  protoc  compiler: protoc --cpp_out=./generated camera.proto This generates a  camera.pb.h  file and a  camera.pb.cc  file. Here’s a snippet from the former: class CameraConfig : public ::google::protobuf::Message { public: CameraConfig(); virtual ~CameraConfig(); // Getters float focal_length() const; int resolution_width() const; int resolution_height() const; bool auto_exposure() const; // Setters void set_focal_length(float value); void set_resolution_width(int value); void set_resolution_height(int value); void set_auto_exposure(bool value); // Serialization bool SerializeToString(std::string* output) const; bool ParseFromString(const std::string& data); // ... }; This gives you: A single source of truth for shared config structures Type-safe config access across firmware, host software, and cloud services Taint tracking. If a field is added or removed, you get a compile-time error in every place it’s used Protobuf’s codegen is the best in the business. Composability / Overrides: Protobuf is not naturally composable in the way that Cue is. It’s more of a strict schema definition than a configuration language. That said, you can build some modularity using nested messages: message Resolution { int32 width = 1; int32 height = 2; } message CameraConfig { Resolution resolution = 1; float focal_length = 2; } This lets you define reusable components and assemble them into larger schemas. On the override side, oneof can give you some basic functionality. However, you can’t do hierarchical overrides (base config → fleet config → individual robot config) in the way that you might want to do once you have a fleet of production robots. You also won’t be able to implement any conditional logic. To get around this, teams either build their override logic into their application code, or use Protobuf strictly for static configs, and use another schema language for more dynamic, runtime configs. Templating / Logic / Computation: Protobuf has no support for: logic, conditionals, or computed fields. templating, variables, or macros. expressing constraints like “field A must equal field B + 1”. Which means that it’s very poor at templating! The schema is static and must be compiled ahead of time. If you want something more dynamic, you’ll need to couple it with a templating engine like Jinja to generate configs from .proto files. Self-Documentation / Human Readability: Protobuf schemas are readable if you know the syntax, but they aren’t designed to double as documentation. Comments are supported, but there’s no built-in support for markdown, structured metadata, or annotations. // Configuration for the main robot arm camera message CameraConfig { // Effective focal length in millimeters float focal_length = 1; // Image width in pixels int32 resolution_width = 2; // Image height in pixels int32 resolution_height = 3; // Whether to enable automatic exposure control bool auto_exposure = 4; } You can describe each field, but the schema can’t explain itself beyond these comments. Of course, once you serialize it into a binary blob, it is certainly not human readable. Summary Protobuf is a high-performance, cross-language serialization format that’s been widely adopted in infrastructure, embedded systems, and now, robotics. It offers best-in-class code generation and excellent support for static typing and schema evolution. With plugins like protovalidate , you can also add runtime validation to match the flexibility of other schema languages. The glaring downside with Protobuf is that it wasn’t designed for configuration. It lacks native support for overrides, templating, and logic. You’ll need to handle inheritance and composability manually or with external tooling. If you need compact, versioned, type-safe configs that compile down to a binary and integrate with firmware or message-passing systems, Protobuf is a strong choice. In the next post, we’ll wrap up the series with takeaways. We’ll compare these schema langugaes side by side, share some honorable mentions, and share some insight on which language we chose to support first at Miru! It’s going to be a fun one. See you in Chapter 5. Config Wars Series Index Config Wars - Chapter 1: Intro to Schemas Config Wars - Chapter 2: JSON Schema Config Wars - Chapter 3: CUE Config Wars - Chapter 4: Protocol Buffers (Protobuf) Config Wars - Chapter 5: What to Choose? [Coming Soon] --- ## Using K3s and ArgoCD for Robotics OTA Deployment - URL: /blog/using-k3s-and-argocd-for-robotics - Published: 2025-07-22T20:20:55.000+00:00 - Author: Vedant Nair - Tags: Guides - Summary: This post maps the main risks in robotics OTA pipelines and where they usually appear. It offers concrete safeguards—credential hygiene, signing/verification, delivery integrity, and safe recovery. - Excerpt: Intro You’re going for a hike. You wake up early, lace up your boots, and grab a banana as you walk out the door. You drive a ways out, into the thick of the woods, to find the trailhead. You hike for a few grueling hours, and finally, you get to the summit! You take the perfect picture; the sun is cresting over the ridgeline in just the perfect way. Now this is Instagram worthy! You open the app and try to upload it, but tragedy strikes. The upload has failed, we have no service! Much to your Intro You’re going for a hike. You wake up early, lace up your boots, and grab a banana as you walk out the door. You drive a ways out, into the thick of the woods, to find the trailhead. You hike for a few grueling hours, and finally, you get to the summit! You take the perfect picture; the sun is cresting over the ridgeline in just the perfect way. Now this is Instagram worthy! You open the app and try to upload it, but tragedy strikes. The upload has failed, we have no service! Much to your chagrin, you don’t have a network connection. Oh well, no worries. It’ll be a few more hours till your superfans (your mom + your other 64 followers) get to see your picture. Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. No matter, let’s hike back down the mountain. But now, the bad is getting worse. Our lack of networking means that we can’t access the updated Google Maps information. We’re officially lost! The plight of your hike is the same struggle that robots face every day. When robots are deployed to customer sites, such as factories, logistics centers, farms, and construction zones, networking is often poor. Sometimes, an enterprise customer will even firewall their facility, meaning that a robot can only communicate with a local server. Over-the-air (OTA) software updates are hard to get right in environments like this. And networking challenges are just one of the intricacies of OTA: You need visibility. What packages are running on what robot? What updates failed? What robots are out of sync? You need security. How can we securely (but also quickly) deliver and download software packages remotely? You need failsafes: Can we easily roll back an update if it causes a bug? Robots are nightmarishly complex systems. So much has to work perfectly for things to go right, and even still, there’s a long way the industry has to go to build dependable, robust robotic systems. Given this undertaking, most teams don’t have the time or bandwidth to stand up a solid OTA pipeline. What makes it more complicated is that there’s no clear standard for how it should be done. This means that they end up cobbling together a hacky solution that scales poorly, and usually bites them in the back as they grow their fleet! In this blog, I propose what that standard should be! We’ll walk through a setup using K3s and ArgoCD, using the principles of GitOps. We’ll briefly discuss its alternatives, its merits, and how the pipeline operates in production. Let’s go! The Common Ways Teams Deploy OTA Updates 1. SSH + SCP What is it? SSH ( Secure Shell ) lets you remotely access a robot’s terminal from your own workstation. You connect to the robot using its IP address and credentials. SCP ( Secure Copy ) lets you transfer files between your workstation and the robot over SSH. You specify the file path and the robot’s IP, and it securely copies the file to the destination. Manually SSH into each robot in the fleet. Then, use SCP (Secure Copy) to transfer your binaries or Docker containers. Finally, systemctl restart to restart your robot’s application and apply the update. Why do teams use it? This is a dead-simple way to get started with remote updates. It’s easy for any engineer to perform, and it’s instrumental in the early prototyping/testing phase. Plus, once you’re shelled in, it’s easy to debug. Why does it break down? Unscalable This is an inherently manual and unscalable model. If you have a small fleet of robots, it’s not that annoying. Once you get past 10, it becomes time-consuming. Error prone The manual nature of this process also means it can be error-prone. By the time you update robot #15, you might forget to copy over the object-detection module. There’s no declarative way to ensure that each package gets updated. Visibility It’s impossible to see which packages are on each robot in our fleet. Are they all running correctly? Is each robot on the right version? Rollbacks If something goes wrong, what then? We can’t easily roll back to a previous version. Security Every inbound port that you open via SSH is another attack vector for malicious actors (like a man-in-the-middle attack) You’ll have to monitor and audit this access, including who accessed which device, who made what change, and why. SSH keys need to be distributed securely and rotated often Once someone is shelled into the robot, they can run arbitrary code, including malicious updates, with little to no visibility or alerting. 2. Ansible (or other automation tools) What is it? Ansible is an open-source configuration management tool that enables you to automate the setup and updates of your infrastructure. You write YAML “playbooks” that describe the desired state of the robot, such as which packages should be installed, which files should exist, which environment variables should be set, and which services should be running. When you run a playbook (using ansible-playbook ), Ansible SSHs into each robot and executes your instructions step by step. It doesn’t require an agent on your device. Why do teams use it? Ansible’s power is that it’s declarative. Rather than having to manually copy each package in the terminal, you can specify the state you want your application to be in, and Ansible will handle the rest. It also helps keep things DRY . You can reuse playbooks across robots or environments with minimal changes. Templating, variables, conditionals, and roles let you write flexible, modular playbooks Why does it break down? It’s still SSH-based and, for that reason, faces most of the practicality and security challenges as SSH/SCP. Atomcity Updates are not atomic (all or nothing). If the network connection drops midway through an update, your device could be left in a bad state. Retries It’s a push-based system. The central server initiates updates, unaware of the current state of the device. If a robot is offline when the playbook runs, it will miss the update. There’s no way for the device to request or pull updates on its own. Ansible was originally designed to update thousands of server racks in a data center, not edge devices with inconsistent networking. That’s why it can be brittle for robotics! 3. Docker Compose What is it? Docker Compose is a tool for defining and running multi-container applications. You write a docker-compose.yml file that declares each container (aka service) your application needs. Each service can specify its image, volumes, environment variables, and dependencies. To deploy, you copy the Compose file onto the robot and run docker-compose up -d . This boots up all the containers with a single command. Why do teams use it? Compose is the easiest way to do container orchestration. It’s simple to learn, well-documented, and extremely easy to set up. The Compose file is practically a manifest. It becomes your source of truth for how the app should run: what images to use, what ports to expose, and how containers should interact. Why does it break down? No state enforcement Compose doesn’t monitor your system to ensure it stays in the desired state. If something crashes or a container exits unexpectedly, Compose doesn’t notice. No self-healing If a container dies, Compose won’t restart it automatically. You’ll need to log in and restart it manually. Kubernetes handles this automatically! Weak rollouts/rollbacks Compose has no canary rollouts and doesn’t natively support rolling back to a previous version in the case of issues. This won’t fly with a large fleet of production robots. Compose is great at first, but you’ll eventually want more and more functionality. Health checks and automatic restarts Resource limits per container Declarative state enforcement Centralized observability and metrics Rolling updates with version control You’ll start to duct-tape this together as you scale, until you realize that you should be using Kubernetes! Wrapping Up And before you beat me to the punch, you definitely don’t want to be building this yourself! The general inclinations of most engineers are. “Oh, this is easy! We can spin this up quickly ourselves.” Of course, it’s never that easy. You’ll be able to get a hacky v1, but the real problems will come with scaling and hardening the solution. But that’s nothing new; the same pattern happens in data and cloud software applications. The unfortunate part about scaling software for a robotic fleet is that things are much stickier for physical devices. A fleet of robots has uptime requirements. Switching out processes can require physical interventions: imagine that your robot is deployed to a firewalled site. To update an agent for a deployment, a technician must physically install it. Ripping and replacing an OTA system is really, really hard! So we need to get this right from day 1. So, we’ve looked at three of the most common ways teams approach OTA: SSH/SCP, Ansible, and Docker Compose. And we’ve seen where each of them breaks down. Now, let’s talk about what a better system looks like. Why to Use K3s, ArgoCD, and GitOps There’s a better way! Let’s talk through (in my opinion) the optimal architecture for your OTA stack. This setup uses K3s and ArgoCD, powered with the principles of GitOps to deliver a system that is: Declarative: We describe what the robot should be running, not how to make it run in that way. Resilient: Can handle network drops and power cycles. Tractable: Every change is tracked in Git. Secure: No inbound ports, no unfettered SSH access. Here’s how it works: Each robot runs its own K3s cluster. Within that cluster is an ArgoCD agent that monitors a remote Git repository. This Git repo will describe the desired state: what containers to run, config values, and startup instructions. When the Git repo changes, for example, if an image has a new tag, the agent detects the difference and pulls the update down. The local K3 cluster will then perform the necessary updates and synchronize itself with Git. No need to SSH, SCP, or execute any manual commands. Our cloud backend has a CI pipeline that builds images, pushes them to our container registry, and commits changes to Git. Our cloud will also host our ArgoCD control plane, which monitors the fleet status of our K3 clusters, providing us with fleet-wide observability on software versions and health status. TLDR: Commit a change in Git. Agent watches Git server. Update gets pulled down to the robot. K3 cluster applies update. Let’s dive deeper into each component. K3s K3s is a lightweight version of Kubernetes , designed to run on resource-constrained devices like robots or edge devices. It’s fast and compact, bundled into a single binary. For clarity, Kubernetes is a container orchestration tool that helps deploy, scale, and manage containers. You can think of it as a beefed-up Docker Compose. Kubernetes monitors your containers. If a container crashes or hangs, Kubernetes automatically restarts it. It uses: Liveness probes to check if a container is still functioning Readiness probes to check if a container is ready to serve traffic Everything is defined in the manifest (YAML file). This declarative approach means you don’t need to write hacky scripts. Just describe what you want, and Kubernetes does the rest. K3s gives you the power of Kubernetes in a single binary (<100 MB) and uses SQLite by default instead of etcd . This allows it to run on embedded devices like Jetson or Raspberry Pi. ArgoCD and GitOps ArgoCD: ArgoCD is a GitOps controller. It automates Kubernetes deployments by syncing your cluster to a Git repo. In a typical setup, ArgoCD runs as a centralized service in the cloud (the “control plane”) and pushes updates to your clusters when Git changes. The ArgoCD agent flips that model. It’s a lightweight version of ArgoCD that runs inside each robot’s local K3s cluster. Instead of a push system, it pulls updates from the Git repo and applies them locally. In this setup, the remote ArgoCD control plane doesn’t push code or directly interact with the cluster. Instead, it’s just used for observability. GitOps: As the name suggests, GitOps is the philosophy that Git is the source of truth for your application’s state. When using Ansible, Compose, or even Kuberenetes by itself, you have to execute imperative commands to deploy software ( kubectl apply -f deployment.yaml , docker-compose up -d , ansible-playbook deploy.yml ). With GitOps deploys kick off automatically when you push. By using GitOps, application versioning is more tractable. There’s a single truth, and it’s easy to change: just push to prod. ArgoCD detects the difference and automatically reconciles your local cluster to match. Note: You could run a traditional Kubernetes (K8s) cluster across your entire fleet, treating each robot as a node in that cluster. In that setup, the ArgoCD control plane would push updates directly to the nodes. This doesn’t work for two glaring reasons: It faces all the shortcomings of other push-based systems. If a robot disconnects from the network, Kubernetes might try to reschedule its pods onto a different node, which could be on a different robot! In summary, we use K3s, ArgoCD, and GitOps because the stack is declarative, version-controlled, resilient (self-healing), and secure (no inbound ports!). Now we’ll talk more tactically about how to structure this stack. Walking Through the Setup G it Structure We use a single shared Helm chart to define our robot’s application stack. This includes deployments, services, volumes, etc. Per-robot configuration (sensor calibrations or location-specific overrides) is stored in external ConfigMaps . You can manage these overrides with: Templating systems like Helm values.yaml Manual overrides via Kustomize patches Or, more scalably, with a dynamic config management with Miru These ConfigMaps are injected into the application at runtime. Here’s an example file tree: /robot-deployment/ /charts/ robot-app/ # Shared Helm chart /robot-values/ robot-001.yaml # Config for robot-001 robot-002.yaml # Config for robot-002 Sample robot-app apiVersion: apps/v1 kind: Deployment metadata: name: robot-app spec: replicas: 1 selector: matchLabels: app: robot-app template: metadata: labels: app: robot-app spec: containers: - name: robot-app image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" env: - name: ROBOT_ID value: "{{ .Values.env.ROBOT_ID }}" - name: WIFI_SSID valueFrom: configMapKeyRef: name: robot-config key: wifi_ssid - name: CAMERA_OFFSET valueFrom: configMapKeyRef: name: robot-config key: camera_offset readinessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 periodSeconds: 10 livenessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 10 periodSeconds: 20 Sample configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: robot-config data: wifi_ssid: "{{ .Values.config.wifi_ssid }}" camera_offset: "{{ .Values.config.camera_offset }}" Sample values.yaml image: repository: ghcr.io/my-org/robot-app tag: v1.5.3 env: ROBOT_ID: robot-001 config: wifi_ssid: "Factory_A" camera_offset: "0.12" CI/CD Pipeline Our pipeline uses Git as the source of the truth and the trigger for deployments. Here’s what happens when we release a new software version: An engineer pushes a new commit to Git (let’s say it bumps the image tag in values.yaml ) The CI pipeline (GitHub Actions, GitLab, Jenkins) Runs the test suite Builds a new Docker image ( robot-app:v1.5.4 ) Pushes it to a container registry (Docker Hub, GHCR, ECR) Commits the updated manifest to Git ArgoCD Agent Each robot runs an ArgoCD agent inside its K3s cluster. The agent intermittently polls Git for changes. It’s configured to watch a specific branch. If it sees a diff: A new commit on main A bumped Helm chart tag (e.g. tag: v1.5.3 ) It will pull down the new manifest and supply it to the K3 cluster. K3 Cluster Once the K3 cluster has the new manifest, it applies the update declaratively. Kubernetes handles this with its built-in controllers. Here’s what happens: If it sees a new Deployment that doesn’t exist, it creates it. If the Deployment already exists but something changed (like the container image tag), it rolls out a new ReplicaSet. If a Service, ConfigMap, or Volume changed, it updates those too, and may restart Pods if needed. If a referenced image isn’t cached locally, it pulls the new image from your container registry. We mentioned that it may restart a Pod if needed. It will do this when the Pod ‘spec’ changes. It needs to destroy and recreate the Pod to match the desired change. This could happen because the ports change, environment variables change, or a container image tag changes (among other reasons). Benefits of This Approach The K3s + ArgoCD + GitOps stack gives us a repeatable, reliable, and scalable way to deploy software to our fleet. Versioning, tractability, and rollbacks Every production change goes through Git. This means that we have a central source of truth for our deployments. We can be sure that we have a clear audit trail of what changed, when, and why. And, if things go wrong, we have an easy way to roll back just using Git. We don’t have to build any other fancy infrastructure to support it. Declarative + Scalable No more running hacky scripts (or even worse, manually SSHing one by one into each robot). Now, we can use Git to declare our desired state and let ArgoCD handle the rest. Push a change to the manifest, and let it be pulled to every robot in your fleet. It doesn’t matter if you have five robots or 100. Resilient to Network Drops and Power Cycles Other systems (like Ansible) that are push-based and rely on SSH have a gaping problem when it comes to edge devices. We can’t assume that we have a steady connection to our device. It’s common for our device to lose power connection to our network. In the case of a push-based system, this could mean that some devices (which aren’t connected to the network at the time) won’t receive the update. Moreover, if the network drops while an update is being applied, our device could be left in a bad state. Security A mortal sin in the world of edge devices is opening inbound ports to our devices. With ArgoCD agents, all communication is outbound. The robot polls the GIt server. Plus, because everything runs locally on-device, no remote commands are being executed over SSH. No chance of a man-in-the-middle attack! Open source and well supported Kubernetes is used everywhere for container orchestration. ArgoCD is the dominant GitOps tool for Kubernetes. Both tools have been battle-tested for years, from startups to Fortune 500s. Downsides of This Approach Kubernetes is Heavy Our robots have only a limited amount of compute and storage. We’ve heard war stories from our customers, where the perception and controls team are fighting for every last byte of resources. Often, the computer is pushed to its absolute limit to wring out the best performance for our robot. So, even though K3s is a lighter distribution of K8s, you can still argue that with its container runtime, control plane, and system daemons, it’s heavy. K3s typically takes ~100–300 MB of RAM and ~100–200 MB of disk space depending on your app, logs, and images If you’re running a Jetson or Raspberry PI, you should be good to go. Steep Learning Curve Kubernetes has reputation for being hard to master (and for good). My co-founder once spent an entire summer wrangling K8s! Most robotics teams will need to learn these behaviors/skills: Kubernetes basics: pods, deployments, services, volumes, probes, etc. Declarative manifests: writing and understanding YAML configs GitOps workflows: stricter Git practices (branches, PRs, approvals) CI/CD discipline: everything goes through the pipeline Team alignment: everyone has to operate within the system Renowned Infra Focus To use this stack to the fullest extent, you’ll need to lay down the proper infrastructure around it. Depending on your resources, this may be prohibitive. TLS certificates (for secure comms between agents and registries) Private container registries (with authentication + access control) Secrets management (Wi-Fi creds, tokens, API keys → Kubernetes Secrets) Networking edge cases (NAT traversal, firewalls, proxies) Structured observability (logs, metrics, status sync from ArgoCD) Conclusion Building OTA for robots is hard. Doing it well, securely, scalably, and reliably. is even harder. Most teams start with bright eyes and hacky internal tools, but those don’t hold up once the fleet grows. We’ve walked through the most common approaches (SSH + SCP, Ansible, Docker Compose) and why they don’t scale. The pair of K3s and ArgoCD is the most functional way to deploy OTA software updates. So the next time you’re evaluating your OTA stack, remember this blog! Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. --- ## Part 2: Why ROS Switched From TCPROS to DDS - URL: /blog/part-2-why-ros-switched-from-tcpros - Published: 2025-07-21T17:31:56.000+00:00 - Author: Vedant Nair - Tags: Insights - Summary: This post deep-dive into DDS for ROS. Did Open Robotics make the right call? - Excerpt: A deep-dive into DDS for ROS. Did Open Robotics make the right call? Introduction: Enter DDS In Part 1, we explored how ROS 1’s transport layer, TCPROS , became a bottleneck for real-time, production robots. It lacked support for multicast, fine-grained Quality of Service, and introduced failure risks that worried commercial customers. Open Robotics knew they had to rebuild their transport layer, but this time, they didn’t want to build it themselves. Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. The new middleware needed to be: Built for real-time performance Efficient enough for embedded systems Scalable to distributed systems with many nodes Reliable and fault-tolerant Widely adopted and proven in production environments They adopted DDS, a commercially proven pub/sub middleware used in aviation, defense, and industrial automation. In this post, we’ll break down what DDS is, how it works, and why it was the right backbone for ROS 2 . What is DDS? DDS, short for Data Distribution Service , is a standard protocol for pub/sub communication. It was developed in the early 2000s by the Object Management Group (OMG) , the same organization that developed CORBA . DDS wasn’t initially designed for robotics, but its initial use cases shared many qualities with robots. From avionics to air traffic control to defense, the systems it served were time-sensitive and mission-critical. Unlike niche academic protocol, DDS was already battle-tested. Companies like RTI and Prismtech (now ADLINK) had been commercializing and scaling DDS implementations in production for a decade. If the world’s biggest governments and enterprises could trust DDS, so could the fledgling ROS! That maturity and track record made DDS an attractive choice for ROS 2. How DDS Works DDS is fully decentralized. There is no broker, no master, or any other centralized service. Every participant handles discovery, negotiation, and communication on their own. Let’s walk through what happens when a ROS 2 node wants to publish or subscribe to a topic using DDS. DomainParticipants When a node wants to publish or subscribe to a topic, it creates a DomainParticipant . This is the core object in DDS. It represents a node joining a specific domain, which can be thought of as a namespace. From this participant, the node can create: DataWriters, publishing messages to a topic DataReaders, subscribing to messages from the topic Each participant describes their communication preferences. This includes the topic name, QoS policy, and finally, the message types. These types are defined using an Interface Definition Language written in .msg files. If a DataWriter and DataReader match on these three fronts, they connect and begin transferring data. Discovery via Multicast When a participant spins up, it announces itself over the network using multicast. Multicast allows a message to be delivered to many listeners simultaneously without opening separate connections for each one. This mechanism powers Simple Discovery, which DDS vendors implement out of the box. What Transport Does DDS Use? Most DDS implementations (including ROS 2’s) default to using UDP as their transport layer, specifically with a protocol called RTPS (Real-Time Publish-Subscribe) . UDP (User Datagram Protocol) is faster than TCP because it has minimal overhead: No connection handshake No guaranteed delivery No in-order packet enforcement This is why it’s widely used in video streaming, where it’s acceptable to drop a frame here and there, as long as the stream remains smooth throughout the video. As we discussed, TCP suffers from HOL Blocking. In UDP, packets can arrive out of order, and that’s okay. It takes the tradeoff in favor of speed for perfect reliability. And in robotics, this is the tradeoff we prefer! The awesome part about DDS is that you can control the knob between speed and reliability. You can configure retries, ordering, and other settings via QoS Policies. How Do ROS 2 and DDS Interact? ROS 2 doesn’t talk to DDS directly. Instead, it uses an abstraction layer called the ROS Middleware Interface ( rmw ). This layer acts as a bridge between ROS 2’s client libraries and the underlying DDS implementation. It lets ROS developers write portable code without being locked into a certain DDS vendor. Several DDS implementations are available for ROS 2, including: eProsima Fast DDS (ROS 2 default) open source, widely used, good community support RTI Connext open source, low-latency, well-suited for embedded and real-time use cases Eclipse Cyclone DDS commercial, mature, great tools and support, often used in safety-critical systems All of these vendors implement the same OMG DDS spec, but they differ in performance, licensing, tooling, and support. At the top of the stack, ROS 2 client libraries, like rclpp for C++ and rclpy for Python, plug into rmw, which in turn delegates to the configured DDS backend. This gives us a clear separation of concerns. Comparing DDS and TCPROS So, how do DDS and TCPROS stack up? DDS Advantages: Real-time capable: No HOL blocking and low latency Tunable QoS: can prioritize critical messages, tune deliverability, or set liveliness expectations Multicast support: One message to many recipients without duplication No ROS Master: Discovery is decentralized and removes a single point of failure DDS Downsides: 3rd party vendor: Have to choose a vendor, potentially jumping through implementation hoops and paying for an extra commercial license Increased complexity: QoS profiles and discovery settings can have a high learning curve Harder to debug: Decentralized systems with UDP can be tricky to trace Still, this tradeoff was more than worth it. ROS 2 needed a middleware built for real-time, distributed robotics. Moreover, DDS was the right choice for commercial customers with custom, highly-performant setups. The Bet Paid Off The switch to DDS wasn’t a small one. It broke compatibility with ROS 1. It introduced new complexity. It restarted the bar for developers and integrators. But it was the right call. ROS 2 continues to be the default robotics framework in the industry (though it is fraught with Rust-based competitors)! From warehouse fleets to surgical robots to defense companies, ROS 2 has scaled well to the most stringent use cases. The Magic 8 Ball says that OSRF made the right decision back in 2014. And for that, they deserve their kudos! Series Index Part 1: Exploring ROS 1's communication model and TCPROS’ shortcomings Part 2: A deep-dive into DDS for ROS. Did Open Robotics make the right call? Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. --- ## Config Wars - Chapter 3: CUE - URL: /blog/config-wars-chapter-3-cue - Published: 2025-07-18T17:05:37.000+00:00 - Author: Vedant Nair - Tags: Insights - Summary: This post compares configuration schema approaches and how they fit a robotics stack. It highlights trade‑offs in validation, composition, tooling, and long‑term maintenance. - Excerpt: Welcome to “Choosing a Schema for Your Robotics Stack!” This series is designed to help scaling robotics teams understand why schemas are useful for their configuration management and navigate which schema is best suited for their stack. Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. In this blog, we’ll do a deep dive into CUE, a popular choice for schemas in robotics. We’ll discuss its origin, features, and some of its pros and cons. History / M Welcome to “Choosing a Schema for Your Robotics Stack!” This series is designed to help scaling robotics teams understand why schemas are useful for their configuration management and navigate which schema is best suited for their stack. Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. In this blog, we’ll do a deep dive into CUE, a popular choice for schemas in robotics. We’ll discuss its origin, features, and some of its pros and cons. History / Motivation CUE, which is short for “Configure, Unify, Execute” was developed at Google by Marcel van Lohuizen, one of the original creators of Go. The original motivation stemmed from Google's internal struggles in managing complex, large-scale configurations for infrastructure and applications. JSON and YAML were too weak for expressing constraints, logic, or composition, so CUE set out to solve that. It was meant to replace: Handwritten configs, which were repetitive and error-prone External scripts for validation Inability to reuse and override structured data Config and code being separate, creating mismatches when keeping them in sync Instead of building and managing external tools to validate, patch, or generate configs, CUE itself is a fully programmable config engine. This means CUE wholly encapsulates the configuration data, the schema, and the logic (how to derive those config values). Usage in Practice CUE is still niche, but it’s gaining adoption in environments where configs are large, layered, and dynamic. It’s used by: Teams at Google Kubernetes projects that need validation beyond YAML Bazel , Kustomize , etc., where templating falls short Broadly, teams where JSON Schema isn’t expressive enough, and logic-based merging is necessary Community / Ecosystem: CUE has a small, active core team of maintainers, including the original Go team led by van Lohuizen. GitHub Activity: Tooling: CUE ships with a CLI that helps you validate, format, and export your configs cue vet : Validate data against constraints cue eval : Merge constraints and output final data cue export : Emit valid JSON/YAML cue fmt : Formatter, like gofmt for CUE cue def : Convert JSON/YAML into a base schema Editor Support: VS Code extension: First-class support with syntax highlighting, formatting, hover docs, and linting JetBrains IDEs: Limited community support; no official plugin yet Code Generation: CUE is tightly integrated with Go, but struggles with other languages. If you’re not using Go (which most robotics stacks aren’t), your best bet is to go from CUE → JSON Schema → code generation. For static languages like Rust, this works well. For dynamic languages, you’ll need CI checks to make sure your code and schema don’t drift. Feature Evaluation Validation Model: CUE uses a constraint-based, structural validation model. Instead of separating types from values, CUE treats both as constraints, so you define them in the same place. In CUE, the schema is the set of constraints your data must satisfy. Here’s an example: RobotConfig: { id: string & ! armLength: float & >0.1 & <2.0 payloadKg: int & >= 0 & <= 25 mode: "idle" | "active" | "emergency" ip: string & =~ "^(?:[0-9]{1,3}\\.){3}[0-9]{1,3}$" } Each field has an inline constraint to prevent misconfigurations. The & operator lets you intersect multiple constraints. Here’s an instance that conforms to the schema: myRobot: RobotConfig & { id: "arm-001" armLength: 1.2 payloadKg: 18 mode: "active" ip: "192.168.1.42" } When you run cue vet , it checks that all constraints from RobotConfig are satisfied by myRobot . If we change payloadKg to 100, the instance is now invalid: myRobot.payloadKg: conflicting values 100 and int & >=0 & <=25: conflicting values 100 and <=25 Code Generation: CUE has poor codegen. There is no way to natively go from CUE to Python, Rust, or C++. Here’s how you could work around this by going from CUE JSON Schema. First, create a .cue file. Here’s our robot.cue example. This defines a Robot type with: A required robot_id (string) A required enabled (bool) A max_velocity that must be ≥ 0 A sensors list that defaults to an empty array Robot: { robot_id: string enabled: bool max_velocity: number & >=0 sensors: [...string] | *[] } Next, export it to JSON Schema using cue export . cue export robot.cue --out=jsonschema Here’s the JSON Schema output! { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": { "robot_id": { "type": "string" }, "enabled": { "type": "boolean" }, "max_velocity": { "type": "number", "minimum": 0 }, "sensors": { "type": "array", "items": { "type": "string" }, "default": [] } }, "required": ["robot_id", "enabled", "max_velocity", "sensors"] } From the JSON Schema, we can generate our Rust struct. cargo install schemafy_cli schemafy robot.schema.json > robot_config.rs Here’s the output: #[derive(Debug, Serialize, Deserialize)] pub struct Robot { pub robot_id: String, pub enabled: bool, pub max_velocity: f64, pub sensors: Vec<String>, } This works, but it’s not first class. You’re stitching together a few different tools, which can cause some lossy translation. For example, going from CUE to JSON Schema means that you lose some of the CUE constraint logic if JSON Schema does not support it. Composability / Overrides: CUE does very well here as it has built composability into its language semantics. CUE uses a logical conjunction ( & ) to merge constraints and values. That means you can define reusable schemas, partial defaults, and environment-specific overrides as independent pieces and merge them when needed. Here’s an example of a base + override: BaseConfig: { armLength: 1.2 payloadKg: <= 20 mode: "idle" } ProductionOverride: { mode: "active" payloadKg: 18 } robot01: BaseConfig & ProductionOverride Evaluating robot01 merges the two objects. All constraints are preserved and checked at merge time. { "armLength": 1.2, "payloadKg": 18, "mode": "active" } This construct allows you to build layered configs, either merging various components together or creating overrides on a per-robot basis. This makes CUE uniquely good for managing configs at scale! Templating / Logic / Computation: CUE supersedes the functionality of a templating engine. It’s a declarative language, so you can write rules, expressions, and constraints that safely compute values based on inputs. CUE supports: Arithmetic & expressions ( torque / (payload + 1) ) Conditional logic ( status: if emergency then "offline" else "ready" ) List comprehensions ( [for id in robotIDs { id: id, zone: "default" }] ) Constraint merging ( limit: >10 , limit: <=25 → limit: >10 & <=25 ) CUE is not Turing-complete. It can’t do: Recursion Unbounded loops General-purpose function definitions Import values at runtime And that’s by design. It’s expressive enough to compute what you need, but also bounded enough to stay predictable. Sometimes, templating engines like Jsonnet and Jinja can be too powerful. They can introduce infinite loops and runtime failures that deliver a bad config, and therefore crash your application. Self-Documentation / Readability: CUE is reasonably self-documented. It has a strong type system, default values, and field declarations. However, compared to JSON Schema, it lacks richer metadata that are nice to haves with a schema language. Field types are explicit: name: string payloadKg: int & >= 0 & <= 25 You can define enums and constraints inline, which makes intent very clear: mode: "idle" | "active" | "emergency" You can annotate with comments, and they persist through evaluation and formatting: // Max load the robot can carry in kg payloadKg: int & >= 0 & <= 25 You can also use cue def or cue export to generate a "flattened" version of the config—useful for introspection or documentation tooling. That being said, it borrows a lot from Go, so for engineers not familiar with the language, there can be some ramp-up needed. Summary CUE is a schema language purpose-built for configuration. It combines validation, logic, and composition into a single language, making it especially useful for teams managing complex, dynamic configs. CUE’s strengths come from its logic and computation. You can write conditionals, compute derived values, and merge overrides natively. In terms of logic and expressiveness, CUE offers more out of the box than any other schema language we’ve evaluated. The downside is that CUE has limited language support outside of Go. While the ecosystem is growing quickly, this will remain a problem in the short to medium term. Teams using CUE alongside other languages will need to translate CUE into JSON Schema to unlock code generation. If you’re looking for a config language that has logic, inheritance, and validation all in one place, CUE is an option worth exploring. In the next post, we’ll explore Protocol Buffers, a data serialization format that’s been popularized as a schema format in embedded systems. See you in Chapter 4! Config Wars Series Index Config Wars - Chapter 1: Intro to Schemas Config Wars - Chapter 2: JSON Schema Config Wars - Chapter 3: CUE Config Wars - Chapter 4: Protocol Buffers (Protobuf) [Coming Soon] --- ## Part 1: Why ROS Switched From TCPROS to DDS - URL: /blog/part-1-why-ros-switched-from-tcpros - Published: 2025-07-15T17:48:31.000+00:00 - Author: Vedant Nair - Tags: Insights - Summary: This post exploring ROS 1's communication model and why TCPROS didn't cut it. - Excerpt: Exploring ROS 1's communication model and why TCPROS didn't cut it. Introduction As the new year rolled into 2014, a group of roboticists gathered around a table in Mountain View, California, debating how to solve a hairy problem in front of them. They were still in the same city where this journey first began, seven years earlier, in 2007, at Willow Garage . One of Silicon Valley’s most iconic robotics labs, it served as a launchpad for its alumni to found and lead some of the most exciting companies today: Agility , Dusty , Fetch , Zipline , and Simbe , among others. Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. Led by what is now called the Open Robotics , the Willow team set out to reduce the barrier to getting started with robots and kick-start a standardized robotics framework. So, they built ROS (Robot Operating System) . Only a few years after its launch, it had exploded in popularity. ROS was modular, customizable, and crucially, open-source. Academics, hobbyists, and startups all fervently adopted it. Through 2014, the growth continued to compound. ROS was far and away the most dominant robotics framework. Its usage had graduated past its small-sized early adopters. Commercial robotics teams with large fleets and exacting performance requirements were now heavy users of ROS. But ROS wasn’t performant enough for their use case; the cracks were starting to show. Notably, teams were frustrated with ROS 1’s native wire protocol, TCPROS. It wasn’t optimal for stringent, real-time speed. It didn’t scale well to large systems. And in some cases, the failure risks created safety concerns. Open Robotics had commanding ambitions. They wanted ROS to scale to millions of robots across every conceivable vertical. To do that, they knew they needed to rebuild their foundation. They started over with ROS 2. This time, they focused on delivering the performance commercial teams needed. This blog is the story of that transformation. Why ROS 1’s transport layer wasn’t good enough, what makes DDS better, and why the move to DDS was a bet worth making. What is Pub/Sub? Robots are real-time, distributed systems. They’re a messy collection of modules, each doing its own job: one node might handle perception, another localization, a third planning, and a fourth controls. Depending on the system, these modules might run synchronously or independently. Sometimes they’ll share compute. Other times, they’ll each get their own microcontroller. The point is that robots are complicated, and whatever software framework we use needs to manage this complexity and facilitate clean, efficient communication between modules. This means that the communication process needs to be fast, reliable, and allow asynchrony. Fast because latency kills performance. For real-time operations, every millisecond of overhead counts. Reliable because modules can be independent (a robot’s control module may rely on data from its perception module). If a message is dropped between the two, the robot will fail to actuate. Asynchrony because nodes can operate on distinct cadences. Sensors will collect and post data on their own schedule. Modules will consume data based on their own application’s needs. To meet those needs, ROS adopted a publish/subscribe (pub/sub) model . It’s a messaging paradigm where data producers (publishers) and consumers (subscribers) communicate indirectly by sharing a topic. Publishers write to a topic; subscribers listen to it. This decoupling means that there are no blocking handshakes or synchronous calls like in request/response systems. The pub/sub model is well-suited for robotics: Modularity: As we mentioned, a robot consists of multiple nodes that work together. We want to separate the concerns between them. This makes our system easier to build, test, and debug, rather than a monolithic monstrosity. The ability for a publisher to publish to many subscribers (in a many-to-many relationship) makes this possible. Low-latency, high-throughput: If every subscriber connection is handled in a single thread, high message rates, like from a LiDAR publishing thousands of messages per second, can create bottlenecks. Without concurrency or shared transport, the publisher has to serialize and send every message separately, which increases latency and wastes CPU resources. Asynchrony: Sensors produce data at their own pace. If we tried to use a request/response model like gRPC or REST, we’d constantly hit timeouts and backpressure. In this regard, pub/sub is much more efficient. Its ‘fire and forget’ approach allows modules to consume data at their own pace. So, ROS 1 used pub/sub. It was the right choice. But having the right messaging pattern is just the start. WE still need to move bytes across the system network. That’s where TCPROS came in, handling message delivery under the hood. Here's how it worked, and why it didn’t hold up. How did TCPROS work? To move messages between nodes, ROS needed a way to serialize, send, and receive data across a network. This is where the wire protocol operates: it’s the layer responsible for turning these messages into bytes and sending them to their receiver. ROS 1 implemented a custom protocol, TCPROS . As the name suggested, it used Transmission Control Protocol (TCP) as its transport layer. Here’s how it works in practice: Let’s say camera_node is publishing images to the topic /camera/image_raw , and perception_node subscribes to that topic. Discovery through the ROS Master First, the subscriber ( perception_node ) reaches out to the ROS Master. This is a centralized registry that keeps track of all publishers and subscribers in the system. Every ROS node registers itself with the Master when it launches. Topic Lookup The subscriber asks the Master, “Who’s publishing the /camera/image_raw topic?” Address Exchange The Master replies with the publisher’s URI (IP address and port). Direct Connection Setup The subscriber then opens a TCP socket directly to the publisher. It sends a handshake that includes: The name of the topic The message type A checksum of the message definition to ensure compatibility If the publisher agrees, it accepts the connection. Note: This connection is only scoped for these two nodes. If any other nodes want to talk to these nodes, they will have to open a separate TCP socket. All TCP sockets are 1:1 channels. Message Streaming From that point on, the publisher serializes and streams messages to the subscriber over that TCP connection. After the connection is established, the two nodes are no longer communicating with the ROS Master. It just handles discovery. It is not a message broker The Implications of TCPROS TCPROS was simple and pragmatic. Unfortunately, that turned out to be its undoing. There were no brokers that slowed down message routing. No heavy external libraries that made ROS more of a resource hog than it needed to be. And it was ~easy to debug and reason about. For these reasons, it performed well for hobbyists and researchers. The troubles revealed themselves as the commercial community adopted ROS. They found difficulties with their production robots, which had strict uptime requirements and performance constraints. No Shared Transport or Multicast As we discussed, each pub/sub pair gets its own TCP socket. So if five different subscribers want the same message from a publisher, the node must serialize the message five times, send the message five times, and maintain five independent TCP connections. This creates performance bottlenecks on three fronts: Repeated serialization burns unnecessary CPU cycles. More sockets mean more system overhead, which becomes a problem on memory-constrained edge computers. Duplicating large messages (like point clouds or images) for each subscriber saturates the network. Real-Time Constraints TCP is designed for in-order, reliable delivery. This is why it’s used for rendering web pages and email, where every packet matters. It’d be quite unpleasant to read an email with every fourth sentence missing! If a packet gets dropped, TCP will delay every packet behind it. This is called head-of-line blocking (HOL blocking) . Until the dropped packet is retransmitted, all other packets are stuck waiting in the queue. For example, if a LiDAR scan arrives 200ms late due to HOL blocking, your robot might miss its planning window. In most robotic workloads, dropping a frame is better than stalling the pipeline. The large volume of messages we’re sending means we’re insensitive if an individual packet gets dropped. No QoS Controls TCPROS didn’t support Quality of Service (QoS) customization. You couldn’t prioritize messages. You couldn’t specify delivery deadlines. You couldn’t configure retries, durability, or resource constraints. Not all data is treated equally. We may want to prioritize delivering safety-critical messages (like emergency stop commands) over general debug logs. With TCPROS, they were treated the same. This made it difficult to create custom, high-performance systems as teams scaled to dozens of nodes on their system. Single Point of Failure: ROS Master If the ROS Master went down, or if any node lost its connection to the Master, new pub/sub connections couldn’t be made. This created a critical single point of failure, which was unacceptable to commercial customers. Conclusion Overall, ROS 1 wasn’t built with production in mind. As more serious robotic teams adopted it, TCPROS’ fragility became a problem. It couldn’t offer the speed, reliability, or configurability their systems required. Open Robotics knew that they needed to reimagine their transport layer. This time, they searched outward for an off-the-shelf solution. And that’s what led them to DDS. In the next blog, we’ll explore how they overhauled the transport layer in ROS 2, why they bet on DDS, and what makes it better suited for modern robotics. Stay tuned! Series Index Part 1: Exploring ROS 1's communication model and TCPROS’ shortcomings Part 2: A deep-dive into DDS for ROS. Did Open Robotics make the right call? --- ## Config Wars - Chapter 2: JSON Schema - URL: /blog/config-wars-chapter-2-json-schema - Published: 2025-07-14T17:09:25.000+00:00 - Author: Vedant Nair - Tags: Insights - Summary: This post compares configuration schema approaches and how they fit a robotics stack. It highlights trade‑offs in validation, composition, tooling, and long‑term maintenance. - Excerpt: Welcome to “Choosing a Schema for Your Robotics Stack!” This series is designed to help scaling robotics teams understand why schemas are useful for their configuration management and navigate which schema is best suited for their stack. In this blog, we’ll do a deep dive into JSON Schema, a popular choice for schemas in robotics. We’ll discuss its origin, features, and some of its pros and cons. Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. His Welcome to “Choosing a Schema for Your Robotics Stack!” This series is designed to help scaling robotics teams understand why schemas are useful for their configuration management and navigate which schema is best suited for their stack. In this blog, we’ll do a deep dive into JSON Schema, a popular choice for schemas in robotics. We’ll discuss its origin, features, and some of its pros and cons. Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. History / Motivation: JSON Schema is a standard for describing and validating the structure of JSON data. Think of it as a contract that defines what a valid JSON object looks like, specifying types, required fields, constraints, and structure. Origins: JSON Schema was introduced in 2010. At this time, JSON was rapidly becoming the de facto format for APIs. However, it had a growing problem. For all its benefits, there was no standard way to validate its structure. JSON Schema filled this gap and gave JSON a similar schema definition to XML ’s XSD . Initial Use Case: The first major use case was API request and response validation. JSON Schema lets developers validate incoming payloads. It was soon adopted in OpenAPI (at the time called Swagger ) and became the backbone for API schemas and documentation. Usage in Practice: JSON Schema wasn’t originally designed for configs, but since it’s become the default option for validating JSON, it has begun to pop up in config systems. Outside of that, here are some places we see JSON Schema in use. OpenAPI : Uses JSON Schema to describe request and response payloads. If you’ve written an OpenAPI spec, you’ve used JSON Schema Docker Compose : Uses embedded schemas to validate service definitions and flag misconfigured keys. AsyncAPI : Builds on JSON Schema to define the structure of events and messages. Kubernetes Custom Resource Definitions (CRDs) : Kubes uses JSON Schema to validate resource specs and reject invalid YAMLs before it applies the resource to the cluster. Community / Ecosystem: JSON Schema has one of the most active and mature ecosystems of any schema format. It’s maintained by the JSON Schema organization , with regular drafts and updates. 2020-12 is the current draft. GitHub Activity: Tooling: The community and tooling around JSON Schema is huge. You’ll find production-grade libraries for nearly every language: JavaScript : ajv Python : jsonschema Go : gojsonschema Rust : jsonschema , schemafy C++ : json-schema-validator It also supports code generation with tools like: quicktype – generates idiomatic types for many languages datamodel-code-generator – for Python, generated via Pydantic schemars – for Rust via Serde Finally, it has editor support for VS Code , IntelliJ , and other editors to power autocompletion, validation, and inline documentation. Something to be careful about: JSON Schema doesn’t have an official validator or CLI. This means you have to use a third-party library. Please ensure the library is active and well-maintained before committing to it in production! Another consideration is that YAML is considered a second-class citizen in JSON Schema. While you can apply schemas to YAML (we often do internally, as you see with the examples we share in our blogs), the experience is less robust than using JSON. Feature Evaluation: Validation: JSON Schema was initially built for validation, so naturally, it is strong here. It gives you a declarative way to define rules like: Required fields Type checks ( string , number , boolean , etc.) Enum constraints ( status must be one of ["idle", "active", "error"] ) Min/max for numbers, length limits for arrays Regex patterns for strings Nesting for objects-of-objects Validation happens at runtime via a separate library like ajv or json-schema-validator. This means that there are no compile-time guarantees. You’re responsible for calling the validator while loading the config into your app. It’s strong for config validation for predictable, bounded checks; however, it lacks the expressive power for computed defaults or logic. You won’t be able to do, “if motor_type is servo , then gear_ratio must be set.” Code Generation: JSON Schema makes it easy to generate typed models in your language of choice from your schema. You can generate: Python classes with datamodel-code-generator : A codegen tool that converts JSON Schema, OpenAPI, or raw JSON samples into Pydantic model classes. Rust structs with schemafy : Takes a JSON Schema and generates Rust types that implement Serde traits. Great when you want strongly typed config parsing from schema definitions. C++ classes with quicktype : generates idiomatic C++ classes with nlohmann::json integration Here’s an example of going from JSON Schema to a Python class using datamodel-code-generator : JSON Schema { "type": "object", "properties": { "robot_id": { "type": "string" }, "enabled": { "type": "boolean" }, "max_velocity": { "type": "number", "minimum": 0 }, "sensors": { "type": "array", "items": { "type": "string" } } }, "required": ["robot_id", "enabled"] } Here’s the generated Python class: from typing import List, Optional from pydantic import BaseModel, Field class RobotConfig(BaseModel): robot_id: str enabled: bool max_velocity: Optional[float] = Field(default=None, ge=0) sensors: List[str] = Field(default_factory=list) This model: Enforces types and constraints at runtime Sets the minimum on max_velocity Defaults sensors to an empty list if not provided Composability / Overrides: JSON Schema doesn’t handle this well, natively. It’s a validation spec, so it was never designed to support overrides or schema extensions. However, there are some workarounds: Modular schemas using $ref : lets you break large schemas into smaller parts and reuse definitions. Note that $ref is reference only, so you can’t override or patch fields. Compose constraints from multiple schemas with allOf , anyOf , or oneOf . Use Jsonnet or Jinja to merge and perform overrides at build time. To use something like Jsonnet, you’ll need a shared base config plus specific overrides for your robot. Which will give you a workflow like the following: # Step 1: Compose or override the config jsonnet robot_07.jsonnet > robot_07.json # Step 2: Validate the result jsonschema --schema base.schema.json robot_07.json Overall, composability and overrides are clunky with JSON Schema. Note: Check out Miru to see how we make overrides easy w/ JSON Schema 🙂 Templating / Logic / Computation Since JSON Schema is declarative and static, it cannot define conditionals, compute derived fields, or loop through a list to generate nested objects. Any logic must be handled outside of JSON Schema (using a templating engine like Jsonnet, Jinja, or even Cue). This limits its usefulness where config values are dependent on each other. Self-Documentation / Readability: For JSON and YAML enthusiasts, JSON Schema is a familiar format. You can embed metadata directly into the schema: description : explain what a field is and how it’s used examples : show valid values or common patterns default : suggest a fallback value (not enforced) title : optional short label for fields Here’s what it looks like: { "type": "object", "properties": { "kp": { "type": "number", "description": "Proportional gain for motor controller (unit: N*m/rad)", "default": 0.05, "examples": [0.05, 0.1] } } } This metadata makes it easy for folks to get up to speed when reading the config for the first time. But for readability, JSON Schema can sometimes be a mixed bag: For simple schemas: readable and easy to follow. For large schemas: becomes deeply nested, verbose, and hard to write by hand. Using allOf + $ref + if/then/else can quickly make the file hard to parse Summary JSON Schema is one of the most widely adopted schema languages today. Originally built for API validation, it has since found its way into configuration systems across web, cloud, and now, robotics infrastructure. It has a litany of strengths: a mature ecosystem, great third-party validation libraries, and broad tooling support across languages. It does lack some functionality with regard to overrides, templating, and logic. Teams usually pair it with an external templating engine like Jsonnet or Jinja to fill these gaps. If your team is already using JSON or YAML and wants strong validation and wide compatibility, JSON Schema is a solid choice. In the next post, we’ll explore Cue, a schema language purpose-built for configs, with strong validation, expressive logic, and native support for overrides. I’ll meet you in Chapter 3! Config Wars Series Index Config Wars - Chapter 1: Intro to Schemas Config Wars - Chapter 2: JSON Schema Config Wars - Chapter 3: CUE [Coming Soon] --- ## Config Wars - Chapter 1: Intro to Schemas - URL: /blog/config-wars-chapter-1-intro-to-schemas - Published: 2025-07-08T17:27:32.000+00:00 - Author: Vedant Nair - Tags: Insights - Summary: This post compares configuration schema approaches and how they fit a robotics stack. It highlights trade‑offs in validation, composition, tooling, and long‑term maintenance. - Excerpt: Welcome to Config Wars! This series is designed to help scaling robotics teams understand why schemas are useful for their configuration management and navigate which schema is best suited for their stack. Setting the Stage You’ve done it! You’ve officially gone from 0 to 1. You started with a crazy idea for a robot that could automate a task for a customer, and after countless iterations, you built a prototype. After more testing and tuning, it worked, not just in the lab, but in the field. Welcome to Config Wars ! This series is designed to help scaling robotics teams understand why schemas are useful for their configuration management and navigate which schema is best suited for their stack. Setting the Stage You’ve done it! You’ve officially gone from 0 to 1. You started with a crazy idea for a robot that could automate a task for a customer, and after countless iterations, you built a prototype. After more testing and tuning, it worked, not just in the lab, but in the field. You shipped it to your first customers, and they love it. Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. Now, you’ve set your ambitions higher. Instead of just a few robots in production, you want 10s, 50s, or 100s across customer sites around the world. It’s time to scale. To support the scale-up, the entire team will have their hands full. The hardware team will need to finalize the BOM and establish QA for a growing fleet. The ML team needs to generalize their models across new customers and environments. And the software infra team? They’ll be tackling today’s topic: configuration! At the scale of just a few robots, most teams are using vanilla JSON/YAML for their configs. They’re iterating very quickly, constantly experimenting and manually tuning parameters. The robot fleet and codebase are small enough such that it’s tractable for SWEs to remember which configs do what, why specific fields have been modified, and manage the fleet’s configuration. Usually, along with some light version control, the fleet’s configuration is in a manageable state. However, to support your scaling efforts, you’ll want to move to a configuration system defined by a schema language. As I’ve observed, for teams comprised of engineers from the embedded and robotics world, it’s challenging to select the correct schema language. Historically, schema languages have been used in cloud software, either for SaaS applications or large-scale backend infrastructure. This means our friends in robotics don’t have much first-hand experience. In this blog series, we’ll give you all the information you need to choose the right schema for your production fleet. We’ll walk through what a schema language is, why they’re useful, our main criteria when choosing a schema, and finally, a few options for popular schemas in robotics. Chapter 1 is all about laying down the basics: What is a config? Why do they matter in robotics? Why YAML/JSON don’t scale What is a schema language? Why is it helpful? Criteria for choosing a schema language By the end, you’ll have the knowledge and confidence to evaluate a schema for your own use case. Let’s dive in! What is a Config? In software systems, a configuration (or config) is a set of parameters that define how the system behaves. The goal is to decouple application logic from runtime values, allowing behavior to be modified without needing to change code. Robotics follows the same principle. Configs are parameters, typically injected at runtime, that define how a robot operates. They make the system adaptable: we can swap out our configs to adjust for different hardware versions, environments, and use cases. This enables modularity. Instead of having to rewrite code every time we swap out a sensor, we can change the parameter values for the new serial number and calibrations. Rather than having a completely different perception module for our picking and packing robots, we can toggle a boolean depending on our use case. Configs are particularly important in robotics because of the complexity and diversity of the systems we build. Subsystems like firmware, perception, motion control, and ML are developed independently but must interoperate reliably. Configs are the shared interface that lets them coordinate without hardcoding any assumptions. Customers we work with often have thousands of config parameters per robot (one notably has 10,000 on a single system). This makes managing these configs for your fleet one of the most important infrastructure mandates as your fleet scales. Failure Mode: Why YAML / JSON Don’t Scale One of the most common failure modes in robotics config management is going schema-less. At the scale of a few robots, using raw JSON or YAML is often the right call. Some particularly daring teams even use .txt files! At that point, we’re iterating very quickly, constantly experimenting, and manually tuning parameters. The robot fleet and codebase are small enough such that it’s tractable for SWEs to remember which configs do what, why specific fields have been modified, and manage the fleet’s configuration. Usually, along with some light version control, the fleet’s configuration is in a manageable state. But once the fleet starts scaling, the flaws of this approach become evident. An unstructured config means that any fatfingering or typo can crash your application. If someone sets max_speed: -1 or misspells controller_mode , you won’t know until it’s deployed and running on your robot. By then, it’ll be too late. I’ve met a team that burned $50k because they misconfigured one of their robot arms, and it crashed into a wall. These mistakes are expensive! Config drift is also an issue. Each robot in the fleet will have a different permutation of its configs. They will vary based on their hardware version, business logic (such as a customer’s safety settings), sensor calibrations, etc. To account for this, we’d like to be able to override configs by layering robot-specific config values over our base configuration. But JSON and YAML don’t support this kind of inheritance. This means that we’ll be stuck copy and pasting files over every time we need to update a config, which is slow and brittle. Finally, you won’t have the benefit of the rich tooling ecosystem that comes with schema languages. They help validate configs, automatically generate structs and classes based on configuration files, and provide other useful utilities for your CI/CD pipeline. So What Is a Schema Language? Why are They Helpful? Let’s talk about how schemas save you from those aforementioned growing pains. Schema languages define the structure, types, and rules your configuration data must follow. In practice, they describe things like: What fields are required What types each field should be What values are allowed (enums, ranges, patterns) Depending on the language, what defaults or logic apply navigation: { max_speed: float & >=0 & <=5 // meters per second min_speed: float & >=0 & < max_speed safety_margin: float | *0.2 // default to 0.2 meters mode: "indoor" | "outdoor" // enum localization: { method: "ekf" | "fastslam" enabled: bool | *true } } In this example of a navigation module config, we can see how a schema is useful. It constrains speed values, sets defaults, uses enums for mode, and nests a localization block with its own rules. This will be important as we scale and are changing these values for each robot. No fat fingering that makes it to production! The most useful schema languages also solve scaling problems by making configs easier to write, validate, version, and generate code from. They can: Catch errors early with static validation Auto-generate boilerplate code or documentation Compose and override configs across environments Enable runtime introspection What to Look For in a Schema Language Not all schema languages were built for the same purpose. Some came from web APIs, others from serialization, and others were designed with configs in mind. Each comes with tradeoffs. When picking one, these are the dimensions we see robotics teams care about most: Core Dimensions: Validation Model: When does the schema catch errors? Runtime? Compile time? Does it support required fields, type checks, value constraints, etc.? Code Generation: Can you generate code from your schema in languages that we (robotics teams) care about (structs or classes in C++, Python, or Rust)? Composability / Overrides Can you override or extend configs cleanly across environments? How easy are these overrides to create? What kind of logic do we write? How powerful are these overrides? What level of granularity can we unlock? Templating / Logic / Computation: Can you express conditional logic, like if statements Can you use variables, functions, or computed values in your config? Is the language Turing-complete? If so, what are the potential side effects? Self-Documentation / Readability: Does the schema support built-in descriptions and field-level metadata? Can a new engineer understand the config by reading it? Is it easy to edit and debug? Can it double as internal documentation? Will your team hate you for choosing it? Tooling Ecosystem: How production-grade and well-maintained are its tools? Is there an official validate library that works well? Can you integrate schema validation into your CI/CD pipeline? Alongside these core features, here are some nice-to-haves worth mentioning: Bonus Dimensions: Change Safety / Drift Detection: Can you detect when a config change happens and perform taint tracking? Can you diff between config versions? Declarative vs Imperative Semantics: Is the language declarative (describe what the config should be) or imperative (describe how to compute it)? How much complexity is acceptable for your config system? Schema vs Instance: Does the schema live in a separate file from the config instance? Or is it all blended into a single file (logic, constraints, and data)? Summary: At a small scale, YAML and JSON are acceptable solutions for robotics configs. But as the fleet grows, it becomes a problem that’s too painful to ignore. Schema languages give your configuration much needed structure. They define what’s required, what values are valid, and how configs can vary across the fleet. This allows for config validation, overrides, and codegen of typed config objects. Instead of relying on manual, brittle processes, schemas and their tooling helps teams scale their config management solution alongside their fleet. Now that we know what configs are, why unstructured formats don’t scale, and what schema languages can offer, the next step is choosing the right one. In the next post, we’ll dive into JSON Schema, why it’s widely adopted, what it gets right, and where it falls short. Config Wars Series Index Config Wars - Chapter 1: Intro to Schemas Config Wars - Chapter 2: Coming Soon --- ## Choosing a Schema Language for Your Robotics Stack - URL: /blog/choosing-a-schema-language-for-your - Published: 2025-07-02T19:19:56.000+00:00 - Author: Vedant Nair - Tags: Guides, Featured - Summary: This post compares configuration schema approaches and how they fit a robotics stack. It highlights trade‑offs in validation, composition, tooling, and long‑term maintenance. - Excerpt: Intro You’ve done it! You’ve officially gone from 0 to 1. You started with a crazy idea for a robot that could automate a task for a customer, and after countless iterations, you built a prototype. After more testing and tuning, it worked, not just in the lab, but in the field. You shipped it to your first customers, and they love it. Now, you’ve set your ambitions higher. Instead of just a few robots in production, you want 10s, 50s, or 100s across customer sites around the world. It’s time t Intro You’ve done it! You’ve officially gone from 0 to 1. You started with a crazy idea for a robot that could automate a task for a customer, and after countless iterations, you built a prototype. After more testing and tuning, it worked, not just in the lab, but in the field. You shipped it to your first customers, and they love it. Now, you’ve set your ambitions higher. Instead of just a few robots in production, you want 10s, 50s, or 100s across customer sites around the world. It’s time to scale. Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. To scale, the entire team will have their hands full. The hardware team will need to finalize the BOM and establish QA for a growing fleet. The ML team needs to generalize their models across new customers and environments. And the software infra team? They’ll be tackling today’s topic: configurations! At the scale of just a few robots, most teams are using vanilla JSON/YAML. They’re iterating very quickly, constantly experimenting and manually tuning parameters. The robot fleet and codebase are small enough such that it’s tractable for SWEs to remember which configs do what, why specific fields have been modified, and manage the fleet’s configuration. Usually, along with some light version control, the fleet’s configuration is in a manageable state. However, to support your scaling efforts, you’ll want to move to a configuration system defined by a schema language. A schema defines the structure of your configuration. It lets you validate configurations before they are deployed, catch mistakes early, and prevent multiple teammates from stepping on each other’s toes. It also sets you up for versioning, diffing, CI/CD, and code generation, workflows that you’ll need in production. As I’ve seen, for teams comprised of engineers from the world of embedded/robotics, it’s difficult to choose the correct schema language. Historically, schema languages have been used in cloud software, either for SaaS applications or large-scale backend infrastructure. This means our friends in robotics don’t have much first-hand experience. In this blog, we’ll give you all the information you need to choose the right schema for your production fleet. We’ll walk through what a schema language is, why they’re useful, our main criteria when choosing a schema, and finally, a few options for popular schemas in robotics. Let’s dive in! What Are Schema Languages, and Why Do They Matter? Schema languages define the structure, types, and rules your configuration data must follow. In practice, they describe things like: What fields are required What types each field should be What values are allowed (enums, ranges, patterns) Depending on the language, what defaults or logic apply Schema Example: navigation: { max_speed: float & >=0 & <=5 // meters per second min_speed: float & >=0 & < max_speed safety_margin: float | *0.2 // default to 0.2 meters mode: "indoor" | "outdoor" // enum localization: { method: "ekf" | "fastslam" enabled: bool | *true } } In this example of a navigation module config, we can see how a schema is useful. It constrains speed values, sets defaults, uses enums for mode, and nests a localization block with its own rules. This will be important as we scale and are changing these values for each robot. No fat fingering that makes it to production! The most useful schema languages also solve scaling problems by making configs easier to write, validate, version, and generate code from. They can: Catch errors early with static validation Auto-generate boilerplate code or documentation Compose and override configs across environments Enable runtime introspection Choosing a Schema Language Not all schema languages were built for the same purpose. Some came from web APIs, others from serialization, and others were designed with configs in mind. Each comes with tradeoffs. When picking one, these are the dimensions we see robotics teams care about most: Core Dimensions: Validation Model: When does the schema catch errors? Runtime? Compile time? Does it support required fields, type checks, value constraints, etc.? Code Generation: Can you generate code from your schema in languages that we (robotics teams) care about (structs or classes in C++, Python, or Rust)? Composability / Overrides: Can you override or extend configs cleanly across environments? How easy are these overrides to create? What kind of logic do we write? How powerful are these overrides? What level of granularity can we unlock? Templating / Logic / Computation: Can you express conditional logic, like if statements Can you use variables, functions, or computed values in your config? Is the language Turing-complete? If so, what are the potential side effects? Self-Documentation / Readability: Does the schema support built-in descriptions and field-level metadata? Can a new engineer understand the config by reading it? Is it easy to edit and debug? Can it double as internal documentation? Will your team hate you for choosing it? Tooling Ecosystem: How production-grade and well-maintained are its tools? Is there an official validate library that works well? Can you integrate schema validation into your CI/CD pipeline? These are some nice-to-haves worth mentioning: Bonus Dimensions: Change Safety / Drift Detection: Can you detect when a config change happens and perform taint tracking? Can you diff between config versions? Declarative vs Imperative Semantics: Is the language declarative (describe what the config should be) or imperative (describe how to compute it)? How much complexity is acceptable for your config system? Schema vs Instance: Does the schema live in a separate file from the config instance? Or is it all blended into a single file (logic, constraints, and data)? Now that we’ve highlighted our criteria, let’s deep dive into a few schema languages! Deep Dive JSON Schema History / Motivation: JSON Schema is a standard for describing and validating the structure of JSON data. It’s like a contract that defines what a valid JSON object looks like, specifying types, required fields, constraints, and structure. Origins: JSON Schema was introduced in 2010. At this time, JSON was rapidly becoming the standard format for APIs. However, it had a growing problem. For all its benefits, there was no standard way to validate its structure. JSON Schema filled this gap and gave JSON a similar schema definition to XML ’s XSD . Initial Use Case: The first major use case was API request and response validation. JSON Schema lets developers validate incoming payloads. It was soon adopted in OpenAPI (at the time called Swagger ) and became the backbone for API schemas and documentation. Usage in Practice: JSON Schema wasn’t originally designed for configs, but since it’s become the default option for validating JSON, it has begun to pop up in config systems. Outside of that, here are some places we see JSON Schema in use. OpenAPI : Uses JSON Schema to describe request and response payloads. If you’ve written an OpenAPI spec, you’ve used JSON Schema Docker Compose : Uses embedded schemas to validate service definitions and flag misconfigured keys. AsyncAPI : Builds on JSON Schema to define the structure of events and messages. Kubernetes Custom Resource Definitions (CRDs) : Kubes uses JSON Schema to validate resource specs and reject invalid YAMLs before it applies the resource to the cluster. Community & Ecosystem: JSON Schema has one of the most active and mature ecosystems of any schema format. It’s maintained by the JSON Schema organization , with regular drafts and updates. 2020-12 is the current draft. Github Activity: Tooling: The community and tooling around JSON Schema is huge. You’ll find production-grade libraries for nearly every language: JavaScript : ajv Python : jsonschema Go : gojsonschema Rust : jsonschema , schemafy C++ : json-schema-validator It also supports code generation with tools like: quicktype – generates idiomatic types for many languages datamodel-code-generator – for Python, generated via Pydantic schemars – for Rust via Serde Finally, it has editor support for VS Code , IntelliJ , etc. to power autocompletion, validation, and inline documentation. Something to be careful about: JSON Schema doesn’t have an official validator or CLI. This means you have to use a third-party library. Please make sure the library is active and well-maintained before committing to it in production! Another consideration, YAML is considered a second-class citizen in JSON Schema. While you can apply schemas to YAML (we often do internally, as you see with the examples we share), the experience is less robust than using JSON. Feature Evaluation Validation: JSON Schema was initially built for validation, so it’s natural that it is strong here. It gives you a declarative way to define rules like: Required fields Type checks ( string , number , boolean , etc.) Enum constraints ( status must be one of ["idle", "active", "error"] ) Min/max for numbers, length limits for arrays Regex patterns for strings Nesting for objects-of-objects Validation happens at runtime via a separate library like ajv or json-schema-validator . This means that there are no compile-time guarantees. You’ll have to validate before loading the config in your app. It’s strong for config validation for predictable, bounded checks; however, it lacks the expressive power for computed defaults or logic. You won’t be able to do, “if motor_type is servo , then gear_ratio must be set.” Code Generation: JSON Schema makes it easy to generate typed models in your language of choice from your schema. You can generate: Python classes with datamodel-code-generator : A codegen tool that converts JSON Schema, OpenAPI, or raw JSON samples into Pydantic model classes. Rust structs with schemafy : Takes a JSON Schema and generates Rust types that implement Serde traits. Great when you want strongly typed config parsing from schema definitions. C++ classes with quicktype : generates idiomatic C++ classes with nlohmann::json integration Here’s an example of going from JSON Schema to a Python class using datamodel-code-generator : JSON Schema: { "type": "object", "properties": { "robot_id": { "type": "string" }, "enabled": { "type": "boolean" }, "max_velocity": { "type": "number", "minimum": 0 }, "sensors": { "type": "array", "items": { "type": "string" } } }, "required": ["robot_id", "enabled"] } Here’s the generated Python class: from typing import List, Optional from pydantic import BaseModel, Field class RobotConfig(BaseModel): robot_id: str enabled: bool max_velocity: Optional[float] = Field(default=None, ge=0) sensors: List[str] = Field(default_factory=list) This model: Enforces types and constraints at runtime Sets the minimum on max_velocity Defaults sensors to an empty list if not provided Composability / Overrides: JSON schema doesn’t handle this well, natively. It’s a validation spec, so it was never designed to support overrides or schema extensions. However, there are some workarounds: Modular schemas using $ref : lets you break large schemas into smaller parts and reuse definitions. Note that $ref is reference only, so you can’t override or patch fields. Compose constraints from multiple schemas with allOf , anyOf , or oneOf . Use Jsonnet or Jinja to merge and perform overrides at build time. To use something like Jsonnet, you’ll need a shared base config plus specific overrides for your robot. Which will give you a workflow like the following: # Step 1: Compose or override the config jsonnet robot_07.jsonnet > robot_07.json # Step 2: Validate the result jsonschema --schema base.schema.json robot_07.json Overall, composability and overrides are clunky with JSON Schema. Note: Check out Miru to see how we make overrides easy w/ JSON Schema 🙂 Templating / Logic / Computation: Since JSON Schema is declarative and static, it cannot define conditionals, compute derived fields, or loop through a list to generate nested objects. Any logic must be handled outside of JSON Schema (using a templating engine like Jsonnet, Jinja, or even Cue). This limits its usefulness where config values are dependent on each other. Self-Documentation / Readability: For JSON and YAML enthusiasts, JSON Schema is a familiar format. You can embed metadata directly into the schema: description : explain what a field is and how it’s used examples : show valid values or common patterns default : suggest a fallback value (not enforced) title : optional short label for fields Here’s what it looks like: { "type": "object", "properties": { "kp": { "type": "number", "description": "Proportional gain for motor controller (unit: N*m/rad)", "default": 0.05, "examples": [0.05, 0.1] } } } This metadata makes it easy for folks to get up to speed when reading the config for the first time. But for readability, JSON Schema can sometimes be a mixed bag: For simple schemas: readable and easy to follow. For large schemas: becomes deeply nested, verbose, and hard to write by hand. Using allOf + $ref + if/then/else can quickly make the file hard to parse CUE History / Motivation: CUE, which is short for “Configure, Unify, Execute” was developed at Google by Marcel van Lohuizen, one of the original creators of Go. The original motivation came from Google's internal struggles managing complex, large-scale configuration for infrastructure and applications. JSON/YAML were too weak for expressing constraints, logic, or composition, so CUE set out to solve that. CUE was open-sourced in 2019. It was meant to replace: Handwritten configs, which were repetitive and error-prone External scripts for validation Inability to reuse and override structured data Config and code being separate, creating mismatches when keeping them in sync Instead of building and managing external tools to validate, patch, or generate configs, CUE itself is a fully programmable config engine. This means CUE wholly encapsulates the configuration data, the schema, and the logic (how to derive those config values). Usage in Practice: CUE is still niche, but it’s gaining adoption in environments where configs are large, layered, and dynamic. It’s used by: Teams at Google Kubernetes projects that need validation beyond YAML Bazel , Kustomize , etc., where templating falls short Broadly, teams where JSON Schema isn’t expressive enough, and logic-based merging is necessary Community and Ecosystem: CUE has a small, active core team of maintainers, including the original Go team led by van Lohuizen. GitHub Activity: Tooling: CUE ships with a CLI that helps you validate, format, and export your configs cue vet : Validate data against constraints cue eval : Merge constraints and output final data cue export : Emit valid JSON/YAML cue fmt : Formatter, like gofmt for CUE cue def : Convert JSON/YAML into a base schema Editor Support: VS Code extension: First-class support with syntax highlighting, formatting, hover docs, and linting JetBrains IDEs: Limited community support; no official plugin yet Code Generation: CUE is tightly integrated with Go, but struggles with other languages. If you’re not using Go (which most robotics stacks aren’t) your best bet is to go from CUE → JSON Schema → code generation. For static languages like Rust, this works well. For dynamic languages, you’ll need CI checks to make sure your code and schema don’t drift. Feature Evaluation Validation Model: CUE uses a constraint-based, structural validation model. Instead of separating types from values, CUE treats both as constraints, so you define them in the same place. In CUE, the schema is the set of constraints your data must satisfy. Here’s an example: RobotConfig: { id: string & ! armLength: float & >0.1 & <2.0 payloadKg: int & >= 0 & <= 25 mode: "idle" | "active" | "emergency" ip: string & =~ "^(?:[0-9]{1,3}\\.){3}[0-9]{1,3}$" } Each field has an inline constraint to prevent misconfigurations. The & operator lets you intersect multiple constraints. Here’s an instance that conforms to the schema: myRobot: RobotConfig & { id: "arm-001" armLength: 1.2 payloadKg: 18 mode: "active" ip: "192.168.1.42" } When you run cue vet , it checks that all constraints from RobotConfig are satisfied by myRobot . If we change payloadKg to 100, the instance is now invalid: myRobot.payloadKg: conflicting values 100 and int & >=0 & <=25: conflicting values 100 and <=25 Code Generation: CUE has poor codegen. There is no way to natively go from CUE to Python, Rust, or C++. Here’s how you could work around this by going from CUE JSON Schema. First, create a .cue file. Here’s our robot.cue example. This defines a Robot type with: A required robot_id (string) A required enabled (bool) A max_velocity that must be ≥ 0 A sensors list that defaults to an empty array Robot: { robot_id: string enabled: bool max_velocity: number & >=0 sensors: [...string] | *[] } Next, export it to JSON Schema using cue export . cue export robot.cue --out=jsonschema Here’s the JSON Schema output! { "$schema": "http://json-schema.org/draft-07/schema#", "type": "object", "properties": { "robot_id": { "type": "string" }, "enabled": { "type": "boolean" }, "max_velocity": { "type": "number", "minimum": 0 }, "sensors": { "type": "array", "items": { "type": "string" }, "default": [] } }, "required": ["robot_id", "enabled", "max_velocity", "sensors"] } From the JSON Schema, we can generate our Rust struct. cargo install schemafy_cli schemafy robot.schema.json > robot_config.rs Here’s the output: #[derive(Debug, Serialize, Deserialize)] pub struct Robot { pub robot_id: String, pub enabled: bool, pub max_velocity: f64, pub sensors: Vec<String>, } This works, but it’s not first class. You’re stitching together a few different tools, which can cause some lossy translation. For example, going from CUE to JSON Schema means that you lose some of the CUE constraint logic if JSON Schema does not support it. Composability / Overrides: CUE does very well here as it has built composability into its language semantics. CUE uses a logical conjunction ( & ) to merge constraints and values. That means you can define reusable schemas, partial defaults, and environment-specific overrides as independent pieces and merge them when needed. Here’s an example of a base + override: BaseConfig: { armLength: 1.2 payloadKg: <= 20 mode: "idle" } ProductionOverride: { mode: "active" payloadKg: 18 } robot01: BaseConfig & ProductionOverride Evaluating robot01 merges the two objects. All constraints are preserved and checked at merge time. { "armLength": 1.2, "payloadKg": 18, "mode": "active" } This construct allows you to build layered configs, either merging various components together or creating overrides on a per-robot basis. This makes CUE uniquely good for managing configs at scale! Templating / Logic / Computation: CUE supersedes the functionality of a templating engine. It’s a declarative language, so you can write rules, expressions, and constraints that safely compute values based on inputs. CUE supports: Arithmetic & expressions ( torque / (payload + 1) ) Conditional logic ( status: if emergency then "offline" else "ready" ) List comprehensions ( [for id in robotIDs { id: id, zone: "default" }] ) Constraint merging ( limit: >10 , limit: <=25 → limit: >10 & <=25 ) CUE is not Turing-complete. It can’t do: Recursion Unbounded loops General-purpose function definitions Import values at runtime And that’s by design. It’s expressive enough to compute what you need, but also bounded enough to stay predictable. Sometimes, templating engines like Jsonnet and Jinja can be too powerful. They can introduce infinite loops and runtime failures that deliver a bad config, and therefore crash your application. Self-Documentation / Readability: CUE is reasonably self-documented. It has a strong type system, default values, and field declarations. However, compared to JSON Schema, it lacks richer metadata that are nice to haves with a schema language. Field types are explicit: name: string payloadKg: int & >= 0 & <= 25 You can define enums and constraints inline, which makes intent very clear: mode: "idle" | "active" | "emergency" You can annotate with comments, and they persist through evaluation and formatting: // Max load the robot can carry in kg payloadKg: int & >= 0 & <= 25 You can also use cue def or cue export to generate a "flattened" version of the config—useful for introspection or documentation tooling. That being said, it borrows a lot from Go, so for engineers not familiar with the language, there can be some ramp-up needed. Protocol Buffers (Protobuf) History / Motivation: Protobuf was created by Google in the early 2000s. The team needed a compact, fast, and language-agnostic way to serialize structured data. The performance offered by XML and JSON wasn’t cutting it, both in terms of size and speed. To keep up with their rocket ship growth, Google needed to efficiently pass structured messages across thousands of internal services written in various languages. Protobuf was created to solve these problems: Efficient cross-language serialization. They needed to encode and decode structured data across systems and languages (C++, Java, Python, Go, etc.) Smaller payload sizes. Protobuf produced a binary format that is smaller and more efficient than XML/JSON Schema guarantees. Define data schemas with . proto files, allowing for type-safe communication between services Forward/backward compatibility. With services constantly changing, schemas need to safely evolve over time. With Protobuf, you can add, remove, or modify fields with breaking old binaries Protobuf wasn’t designed for configuration, but over the years, it has been co-opted for this use case in environments where binary formats and strong typing matter. It’s become popular in the world of embedded systems and robotics. That’s great news for us! Usage in Practice: Protobuf operates in the infrastructure layer of many distributed systems: APIs & microservices: Especially with gRPC, which uses Protobuf as its IDL (Interface Definition Language) and serialization format. Internal service communication: For large-scale systems, Protobuf is often used to reduce payload size and parsing overhead Embedded systems: Robotics, IoT, etc., anywhere where bandwidth is constrained Protobuf wasn’t built for configuration, but many teams use .proto files as config schemas, given that they want: Typed configs and are okay with compiling the schema before runtime To control the full pipeline (generate configs at build time and deploy as binaries) Community & Ecosystem: Protobuf is still led by Google with a mature and production-grade ecosystem. It’s been around for ~15 years, so you’ll find wrappers, plugins, and tooling across every major language. Github Activity: Tooling: IDE Support VS Code: Extensions for syntax highlighting, autocomplete, and linting. IntelliJ, GoLand, etc: Built-in .proto support for major languages. CLI Tools protoc : the compiler that turns .proto files into code Protobuf’s CLI, buf , allows for: Linting Breaking change detections Schema Versioning Dependency management protovalidate : Adds runtime validation rules. Useful since native Protobuf doesn’t have strong validation. Feature Evaluation Validation Model: Protobuf’s native validation model is minimal. It enforces types but doesn’t support constraints like: Min/max values String length Regex patterns Cross-field logic For a real validation, most teams use protovalidate , a buf plugin that lets you define rich constraints directly in your .proto files. Example with protovalidate import "buf/validate/validate.proto"; message JointLimits { string joint_name = 1 [(buf.validate.field).string.min_len = 1]; float min_position = 2; float max_position = 3 [(buf.validate.field).float.gt = 0]; // Ensure max_position > min_position in application logic } This snippet ensures: joint_name can’t be an empty string. max_position must be > 0. You can enforce max_position > min_position at runtime via generated validation code. protovalidate allows you to validate at runtime and unlock most of the validation capability from both JSON Schema and CUE. Code Generation: Protobuf has excellent code generation. You define your schema once in a .proto file, and you can easily generate typed, structured code in your language of choice. Let’s say you have a schema for camera parameters on a robot: message CameraConfig { float focal_length = 1; int32 resolution_width = 2; int32 resolution_height = 3; bool auto_exposure = 4; } You can generate a C++ class using the protoc compiler: protoc --cpp_out=./generated camera.proto This generates a camera.pb.h file and a camera.pb.cc file. Here’s a snippet from the former: class CameraConfig : public ::google::protobuf::Message { public: CameraConfig(); virtual ~CameraConfig(); // Getters float focal_length() const; int resolution_width() const; int resolution_height() const; bool auto_exposure() const; // Setters void set_focal_length(float value); void set_resolution_width(int value); void set_resolution_height(int value); void set_auto_exposure(bool value); // Serialization bool SerializeToString(std::string* output) const; bool ParseFromString(const std::string& data); // ... }; This gives you: A single source of truth for shared config structures Type-safe config access across firmware, host software, and cloud services Taint tracking. If a field is added or removed, you get a compile-time error in every place it’s used Protobuf’s codegen is the best in the business. Composability / Overrides: Protobuf is not naturally composable in the way that Cue is. It’s more of a strict schema definition than a configuration language. That said, you can build some modularity using nested messages: message Resolution { int32 width = 1; int32 height = 2; } message CameraConfig { Resolution resolution = 1; float focal_length = 2; } This lets you define reusable components and assemble them into larger schemas. On the override side, oneof can give you some basic functionality. However, you can’t do hierarchical overrides (base config → fleet config → individual robot config) in the way that you might want to do once you have a fleet of production robots. You also won’t be able to implement any conditional logic. To get around this, teams either build their override logic into their application code, or use Protobuf strictly for static configs, and use another schema language for more dynamic, runtime configs. Templating / Logic / Computation: Protobuf has no support for: logic, conditionals, or computed fields. templating, variables, or macros. expressing constraints like “field A must equal field B + 1”. Which means that it’s very poor at templating! The schema is static and must be compiled ahead of time. If you want something more dynamic, you’ll need to couple it with a templating engine like Jinja to generate configs from .proto files. Self-Documentation / Human Readability: Protobuf schemas are readable if you know the syntax, but they aren’t designed to double as documentation. Comments are supported, but there’s no built-in support for markdown, structured metadata, or annotations. // Configuration for the main robot arm camera message CameraConfig { // Effective focal length in millimeters float focal_length = 1; // Image width in pixels int32 resolution_width = 2; // Image height in pixels int32 resolution_height = 3; // Whether to enable automatic exposure control bool auto_exposure = 4; } You can describe each field, but the schema can’t explain itself beyond these comments. Of course, once you serialize it into a binary blob, it is certainly not human readable. Honorable Mentions These tools/schema languages didn’t make the lineup, but they’re worth knowing. We’ll explore them briefly. Pkl: Pkl (short for “Pickle”) is a ~new config language developed by Apple. It combines JSON-like syntax with a real type system, a modular system, and even object-oriented features like inheritance and visibility modifiers. It stands our for its strong support for reusability and testing. You can write unit tests for your config and create base classes for components. The ergonomics feel like configs as code. It has positioned itself as a ‘configuration DSL’, with a promising direction. The problem is that it’s still young. The tooling ecosystem is limited, especially for robotics-adjacent languages. Dhall: Dhall is a functional programming language specifically designed for configs. It’s strongly typed and deterministic; you can write configs like code with imports, functions, types, and guarantees. However, it isn’t Turing-complete. One of Dhall’s core promises is safe refactoring. You can define types once, import them across files, and be sure that nothing breaks when you change things upstream. It also supports normalization and diffing, which is great for reviewing config changes in CI. Unfortunately, Dhall has a steep learning curve. Its syntax is unique: lambdas look like λ(x : Text) → x ++ "!" , and even simple expressions can feel complex. For most teams, the tradeoff to learn a foreign syntax isn’t worth it. It's a nail in the coffin that, like Pkl, it lacks codegen support for both Rust and C++. Jsonnet + Jinja: We’ve referenced these two a few times throughout the blog. They’re powerful, but are only an honorable mention because they aren’t a schema language. They’re templating engines. They’re widely used across robotics teams to generate structured configs. Jsonnet is a data templating language (surprise, surprise, created by Google!) that lets you write JSON with variables, conditionals, and functions. As we showed previously, you can write your config in Jsonnet and then validate it against JSON schema. Jinja is a string-based templating engine for Python. It’s less structured than Jsonnet in that you’re just generating text, but it’s familiar and easy to use in Python. You can loop over variables and render JSON/YAML. These tools are great for creating overrides or conditional logic, but by themselves, aren’t robust enough. You still want validation, type checking, and codegen offered by schema languages. Use them together! How should you decide? Here’s how JSON Schema, CUE, and Protobuf stack up for robotics configs: CUE If you want built-in validation and native override logic, CUE is your best choice. Because CUE is self-encapsulating, you won’t have to use any third-party libraries. However, its codegen is weak for robotics-specific languages. This means you’ll be on your own to build this tooling. Protobuf If you’re already using Protobuf for telemetry or message-parsing, it makes sense to use it for your config system as well. It has robust schema enforcement, versioning, and codegen. JSON Schema If you’re already familiar with YAML/JSON, JSON Schema is a good option. It’s perfectly workable for validation and tooling integration, and has large community support. Conclusion In this blog, we walked through configuration schemas, why they are important for scaling robotics teams, and introduced you to a few schema languages (along with some decision criteria). If you’re curious, for Miru ’s config management tool, we started by supporting JSON Schema. Here’s why: It was the easiest for teams to adopt. If they were using schema-less JSON or YAML the migration wasn’t too painful It has a large, mature community It supports codegen for all the languages we care about (C++, Rust, and Python) It has a litany of third-party tools from validators to templating engines and more For those who’d prefer to use a different schema language, good news! We’ll be supporting Protobuf and Cue soon. Thanks again for reading, and feel free to reach out with any questions. Happy building! --- ## Tutorial: Deploying a Config Instance with Miru - URL: /blog/tutorial-deploying-a-config-instance - Published: 2025-06-23T16:22:25.000+00:00 - Author: Vedant Nair - Tags: Tutorials, Product Updates & Features - Summary: This post first of multi-part series exploring how to use Miru, and diving deep into how the system works behind the scenes. - Excerpt: The first of multi-part series exploring how to use Miru, and diving deep into how the system works behind the scenes. Intro Your robot needs a new config. Maybe you're adjusting inference thresholds. Maybe you're tuning telemetry intervals for a customer fleet. Whatever the change, you want it to be versioned, deployed quickly, and safely. In this tutorial, we’ll show you how to do just that with Miru. We’ll walk through the full lifecycle: Loading a config in local development Deploying configs in production using the Miru agent And how Miru ensures the right config gets to the right robot, every time By the end, you’ll know how to: Use the Miru agent to pull configs in production Read parameters using the Miru C++ SDK Understand how config deployment works under the hood Scenario: Object Detection We're running a robot with an object detection module. It's using a YOLO model on a camera feed. We want to configure things like: Which YOLO version to run What FPS to use Thresholds like confidence and NMS Output settings like bounding boxes or filtered labels To define these parameters, we will use a configuration schema. The schema defines the shape of the config: what fields exist, their types, valid ranges, and default values. Currently, Miru supports JSON Schema , which is widely used and has code generation for many languages. However, support for other configuration languages is coming soon. Here are some popular ones. CUE Language Pkl Type Schema JSON Type Definition Dhall Throughout the lifecycle of our robot, our schema will change. We may add/delete parameters or update the constraints for our values. To track this evolution, we introduce a config type called object-detection . This config type groups every version of the schema and every instance we’ve ever deployed. It becomes the source of truth for how our object detection module should behave across our fleet. Quick Definitions Before we continue through our example, let’s define a few key terms: Config Type A named group of parameters, like object-detection , mobility , or motion-planner . It tracks every version of the schema and every instance that has ever been deployed. Defined automatically from the $miru_config_type_slug in your schema. Schema Defines the structure of the config, including valid fields, their types, default values, and allowed ranges. This is the contract your config instance must follow. Config Instance The actual values used by your robot at runtime. Must conform to the schema. Tags Labels attached to a device, like compute_device: Jetson Orin or region: EMEA . Tags carry metadata used to generate config instances. OverridesOverrides are a ruleset that overrides the default schema with the specific configuration values your robot needs. The tags supply the values used in the override.In development, you write the schema and config instance by hand (usually as YAML files). No cloud or agent required.In production, you push the schema to Miru. Miru renders a config instance for each device based on its tags and your overrides. File Structure Here’s the file tree for this example. You can follow along or check out the complete code here . . ├── cpp/ │ ├── main.cpp # Entry point — just calls run_object_detection() │ ├── object_detection.cpp # Core logic — loads and reads config │ └── object_detection.h # Header file for the module ├── schemas/ │ └── object-detection.schema.yaml # Defines the shape and defaults for the config ├── config-instances/ │ └── object-detection.config.yaml # Local-only config instance (used in dev) ├── CMakeLists.txt # Builds the project using the Miru C++ SDK This is a minimal setup: one config type, one schema, one module. How the Schema and Config Work Together Here’s the object detection schema: $miru_config_type_slug: "object-detection" title: Object Detection type: object properties: model_name: type: string default: "yolov9" inference_fps: type: integer minimum: 1 maximum: 60 default: 15 confidence_threshold: type: number minimum: 0.0 maximum: 1.0 default: 0.7 nms_threshold: type: number minimum: 0.0 maximum: 1.0 default: 0.4 image_resolution: type: object properties: width: type: integer default: 1280 height: type: integer default: 720 draw_bounding_boxes: type: boolean default: true label_filter: type: array items: type: string Miru uses the schema to extract the config type slug ( object-detection ) and to generate a hash of the schema, which acts as the version ID. Every time you change the schema, the hash changes. Miru uses it to track the schema versions over time and across the fleets. Here’s the matching config instance for the schema: model_name: yolov9 inference_fps: 15 confidence_threshold: 0.8 nms_threshold: 0.4 image_resolution: width: 1920 height: 1080 draw_bounding_boxes: true label_filter: - "person" - "car" This is the actual set of values a device will use at runtime. Miru guarantees that every config instance matches its schema, and if they don’t, the error is thrown at deploy time , not runtime , saving you from crashing your application. Local Development Now that we’ve set up our code environment, let’s start using Miru. In development, you load both the schema and the config instance from disk. The main draw of using Miru in development is speed. When you’re experimenting/tuning a parameter, you don’t want to version a config, push it to the cloud, write an override, and wait for your agent to pull the update. You just want to tweak a value and see it reflected immediately. Miru still enforces that the config instance matches the schema. You get a typed config object, and the SDK handles loading, validation, and parsing for you. Note the use of from_file() . This tells the SDK to load both the schema and config instance from disk: auto config_instance = miru::config::ConfigInstance::from_file( "schemas/object-detection.schema.yaml", "config-instances/object-detection.config.yaml" ); You can then query parameters like this: auto fps = miru::query::get_param(config_instance, "object-detection.inference_fps").as_int(); You can use get_param() to access any parameter in your config by path. The path matches the structure defined in your schema. If the parameter doesn't exist or has the wrong type, the SDK throws a descriptive error. Here’s the (dummy) output of my object-detection module: $ ./build/object_detection Using config instance: ======================= model_name: yolov9 inference_fps: 15 confidence_threshold: 0.7 nms_threshold: 0.4 image_resolution: width: 1280 height: 720 draw_bounding_boxes: true Starting object detection with the following parameters: - Model: yolov9 - FPS: 15 - Confidence Threshold: 0.7 - NMS Threshold: 0.4 - Resolution: 1280x720 - Draw Bounding Boxes: yes Production Deployment Once you’re ready to ship, swap from_file() with from_agent(). auto config_instance = miru::config::ConfigInstance::from_agent( "schemas/object-detection.schema.yaml" ); Now, instead of reading a config from disk, your app reaches out to the Miru agent running on the device. The agent fetches a config instance from the cloud—tailored to that device—based on its tags and the current schema. Here’s how it works under the hood: A: App requests a config from the SDK Your application calls from_agent(), passing in the schema from the local file system. B: SDK hashes the config schema The SDK reads your config schema file, hashes it (to get the digest), and extracts the slug. The hash and config slug uniquely identify the version of the schema that you are using. C: SDK requests the latest instance from the agent Next, the SDK sends that hash and slug to the Miru agent over a UNIX socket. The agent also grabs the device ID. We don’t cover auth in this tutorial, but it’s simple: run the install script, paste in the activation token from the Miru UI, and the agent ties itself to that device. D: Agent requests the latest instance from the cloud The agent sends a request to the cloud. At this point, the cloud knows: What schema version you're using (via digest + slug) Which device is asking (via device ID) Because network disconnections are common in robotics, Miru also keeps a local cache of the last known config instance. In the case the network is unavailable, it will pull the config instance from the cache. E: Cloud renders the config schema In the cloud, the rendering engine renders the config instance. It takes the information provided by the agent, along with the tags, metadata, and overrides defined in the cloud, to generate the config instance. F: Cloud sends the instance back to the agent Once rendered, the cloud sends the config instance back to the agent. The agent caches it on disk, so it can keep serving it even if the device goes offline. G: Agent returns the instance to the SDK The agent returns the config instance to the SDK. H: SDK returns the instance to the applicatio n Your app receives a fully typed config object, ready to use. Conclusion You’ve now seen how to use Miru to load configs in development and how to deploy them in production using the Miru agent. You saw how the SDK hashes your schema, how the agent identifies the device, and how the cloud returns the correct config instance. Miru handles versioning, delivery, and caching to simplify config deployment. Head over to the docs to try it yourself, and don’t hesitate to reach out if you have any questions! --- ## Robotics Config Management Is a Mess. Fix it with Miru. - URL: /blog/robotics-config-management-is-a-mess - Published: 2025-06-03T18:00:42.000+00:00 - Author: Vedant Nair - Tags: Company, Insights, Featured - Summary: This post exploring the challenges robotics teams frequently face with configs, and how we've designed Miru to alleviate these pains. - Excerpt: Exploring the challenges robotics teams frequently face with configs, and how we've designed Miru to alleviate these pains. Introduction What makes a robot a robot? Is it its hardware? Its software? Its sensors? Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. It can’t be just hardware. We don’t have much use for a hunk of tin that can neither think nor move. It’s not just the software either. Our goal is not to build super-intelligent software systems that can’t interact with the physical world. We can keep going on and on: beautiful LiDAR visualizations with no purpose, a ROS2 system with no subscribers to listen to it. A robot is the interplay between ALL of these parts. It’s a combination of its hardware, sensors, software, models, firmware, etc., working together at a given time. Or, put another way, a robot is its configuration. The configuration is a bulky, finicky amalgamation of text files that defines the robot’s behavior. And robots are complex systems. It’s a dense web of many thousands of parameters that control how that code runs: which models to load, how to tune each sensor, what safety limits to enforce. A single misconfiguration of a single parameter can bring the entire system to a halt. In this blog, we’ll discuss why configurations (also known as configs) are so hard and how Miru is making config management easy for robotics teams. Let’s jump in! What are Configs in Robotics? In practice, configs are runtime parameters that determine how a robot behaves. They aren't hardcoded, which makes the system easily adaptable to different tasks, environments, and hardware. It also avoids modifying application code for every change, no matter how trivial (such as swapping out a camera). For the scope of Miru, we consider these as ‘application configs’. They’re distinct from ‘system configs’, which control things like the OS, networking, or device permissions. System configs are usually baked into the system image, are more static, and should be carefully changed, as the failure paths include bricking the device. Where do Configs live? In robotics, configs are usually stored in text files — YAML , JSON , XML , or robot-specific formats like ROS params and Protocol Buffers (Protobufs ). Each format has its quirks, but we won’t get into that here. (If you're curious, stay tuned, we’ll cover it in a future post.) What are Configs used for? Almost all of the robot’s application behavior is encapsulated in its config, so the answer to this question could be ‘practically anything’. However, in the spirit of being precise, I’ve broken it down into five groups. Hardware Version Different robots in the fleet may use different compute boards or end effectors. These variations affect what software runs, what models are supported, and how the robot behaves in production. Calibration Every sensor, camera, LiDAR, and IMU needs calibration. These include intrinsics, extrinsics, and offsets. These values get stored in the config and loaded at runtime. Models and Algos Models are swapped in and out based on the task, fine-tuned on customer data, or throttled based on hardware limitations. Business Logic Whether it’s a meal-prepping robot or one delivering packages, configs define the business logic it needs to know, from ingredients to barcodes. Feature Flagging Teams want to toggle features based on their customer, development environment (dev, beta, prod). Challenges of Config Management Configs become even gnarlier once you move from a single robot to a production fleet. Now you have uptime requirements, legacy hardware versions, multiple environments, and human operators all interacting with configs in different ways. Let’s ground this with a simple example. Suppose we’re deploying intelligent robot arms to two customer sites, Customer X and Customer Y. Some of our robots have the latest GPU, running VLA models. Others are older, CPU-only, and rely on classical computer vision. Complex Config Groupings Across a fleet, robots are ‘snowflakes’. Their configs vary based on their attributes, such as hardware version, end effector, sensor calibration, and customer. These groups often overlap in complex ways: a single customer may have multiple hardware variants, some customers may choose to beta-test early releases, and others may want to stick with the latest stable release for as long as possible. Due to these overlapping relationships, it can be challenging to define a robot’s configuration and safely update it across the fleet. In practice, this could mean manually scouring thousands of values, editing fields, and taint tracking. And that’s for EACH robot config you want to edit! Let’s go back to our example with Customer X and Y’s robot arms. Even for this small matrix of customer/compute types, we face several challenges. If we want to change a feature for a given customer, we’re forced to edit the config manually. We can’t reuse sections of configs for a given compute type. The best we can do is copy and paste a file over, but this is a brittle and error-prone process. We can’t change any configs globally. And remember, this is just the cross-section for customer/compute type. In reality, we would have many such interdependent cross-sections for our various hardware components, models, and application logic. The groupings are complex and require a thoughtful data model to manage and apply programmatically. Most teams don’t have the luxury of building this themselves and instead make their edits and groupings manually. Volume A single device can have thousands of config parameters! As your fleet grows, it becomes prohibitively difficult to manage this manually. From conversations with customers, once you move past ~10 production robots, this becomes a burning problem. User-Facing Configs No self-respecting B2B robotics company allows itself to operate without a cavalry of Deployment/Field Engineers ;) These folks are usually semi-technical (ranging anywhere mechEs and tradespeople, but generally not software savvy) who operate at customer sites, provisioning robots, troubleshooting, and fixing errors. As part of their duties, they frequently interact with configs: Swap out a sensor? Change a config. Motor behaving weirdly? Change a config. Uptime shaky? Enable safety mode (by changing, you guessed it, a config). Onsite at Customer X and Y, our field engineers SSH into robots and edit YAML files in Vim. They’re supposed to commit these changes back to the cloud, keep version control sane, and not break anything. This almost never happens cleanly. The result? Misconfigurations, downtime, and a growing burden on infra engineers to trace who changed what, when, and why. (And yes, we’ve heard firsthand of companies with hundreds of robots deployed across F500 customers whose deployment engineers are using Vim daily to edit configs!!) Bidirectional Syncing Version control can be a real pain! If our software engineers want to push a new version of a config, they need to ensure it doesn’t cause any breaking changes for our production robots. Conversely, when deployment engineers want to edit configs locally, we want to have visibility into these changes and version these edits as well. This leads to questions like: What’s the source of truth? How can we ensure sanity with our configs when the cloud and local versions differ? How can we maintain versioning when we make local edits? These problems don’t just affect a few of our team members. They have knock-on effects throughout the entire org. Impacted Stakeholders Infra Engineers They want to skip schemas safely but are currently flying blind. There’s no guarantee a change won’t break a robot in the field. To combat this, they version their robot configs separately in their own git repo! For a fleet of 100 robots, that’s 100 separate repos that need tending to. They know it’s an imperfect solution, but if the fleet is scaling fast, they often have no other choice. QA Engineers QA depends on configs to run integration tests and system validation. Without types and constraints, they’re stuck chasing bugs caused by typos and invalid values. Robotics / Autonomy Engineers They want to experiment: swap in a new model, toggle a safety parameter, test a feature on just a few robots. But to do that, they have to SSH into each robot and manually change the configs. It's tedious and error-prone. Field / Deployment Engineers Whether it’s a meal-prepping robot or one delivering packages, configs define the business logic it needs to know, from ingredients to barcodes. End Users And at the end of the chain, it’s the customer who suffers. Misconfigurations and slow release cycles result in downtime, missed service-level agreements (SLAs), and a suboptimal customer experience. Back to our example, this means that Customer X and Y churn halfway through their contract, leaving our team stuck holding the bag! How Miru Solves Config Management Now that we’ve explored the challenges of config management, we’ll present a high-level overview of how Miru helps engineering teams manage, version, and deploy configs to their robots. We’ll start with a diagram of the user flow and then zoom in to different parts of the Miru application. If you’d rather watch a demo, feel free to check out this video. User Flow We’ll start by walking through the user flow at a high level: Local Development You begin by iterating on your config schema locally. Once you're happy with the changes, you push the new schema version to Miru using the CLI. This is typically done as part of your continuous integration (CI) pipeline. Cloud / Web App In the Miru dashboard, you define config values and set up logic for overrides (more on this later). Miru then renders these into a final config instance. Robot On the robot, the Miru Agent pulls the appropriate config instance from the cloud. The SDK (C++, Rust, Python) exposes the config instance to your application. Zoom-In: Config Instance Rendering In the old world, teams manually edited config values on a per-robot basis using a text editor. With Miru, that’s no longer necessary. You define config values in the cloud, set up override logic, and let Miru apply those values at scale across your fleet. This process is called rendering . It combines a schema with config values to produce a final, device-specific output: a config instance. Version and Validate Schemas Miru requires you to define a schema for every config. This schema enforces types, value constraints, and structure. We don’t want a misformatted, vanilla YAML to break our robot! When you're ready to publish a new version of your schema, run miru schema push . It uploads the schema to Miru and extracts Git metadata, such as the commit hash and branch, to track the version history. Most teams run this in CI. Critically, Miru does not have access to your application code / Git repository. Application code is sacred, and we want to give users full confidence that their IP is protected. Once the schema has been uploaded to the cloud, it’s time to define our config values! Define Config Values Miru utilizes a tagging system to define configuration data for our devices. We attach various tags to our devices, and these tags have metadata (as shown below) that define our configuration values. This allows us to define configuration data across our fleet of devices in a scalable manner. For example, the ‘Amazon’ tag has this metadata associated with it that defines its location , priority_level , robot_speed_limit , and safety_mode_enabled . In the rendering step, this metadata is used to create our configuration instance. Tags are just instances of a tag type. Each tag type has its own schema, which defines the metadata each tag can contain. For example, a Customer tag type might include Amazon , FedEx , and GXO . A Compute tag type might include Jetson Orin Nano or Raspberry Pi 4 . You can also tag by environment ( beta , prod ), end-effector, location, or anything else you care about. These tags are a flexible construct and allow your team to shape them based on the nuances of your fleet. Render Config Instance with Overrides Once you've uploaded your config schema and defined tag-based values, it's time to render the config instance . Rendering combines: The schema (which defines the structure and defaults) The tags assigned to a robot The override layers defined in Miru Each override layer is a named policy that pulls values from a specific tag type. For example, you might create override layers named Customer , Hardware Variant , or Safety Mode , each of which references a field in one of your tag types. Overrides are applied in order—later layers take precedence over earlier ones. So if Safety Mode overrides max_linear_speed_mps , it will replace the value set by the Customer or Hardware Variant layers. The result is a config instance : a concrete, validated config tailored to that robot’s tags and safe to deploy. Deploy Config Instance to Robot Once a config instance is rendered, it’s ready to be pulled down to the robot. On the device, the Miru Agent runs as a lightweight systemd service. It uses a pull-based model to fetch the correct config instance from the cloud, identified by the device’s ID, the config schema hash, and the config type slug. The agent is designed to be resilient. It handles flaky networks and unexpected reboots by caching the most recent config locally, so your robot can keep running even if it’s offline. The Miru SDK connects to the agent via a Unix socket and exposes the config instance to your application as a typed object. You can fetch values directly using your preferred language. Currently, we support C++, with Rust and Python SDKs coming soon. Conclusion In this blog, we walked through the challenges of configs in robotics and how Miru makes this easy for production fleets. If your team is struggling with this problem, please reach out! We’d love to help. If you’re interested in learning more about our architectural decisions or have any general questions, we’d love to hear from you as well! And lastly, if you’re feeling adventurous, why not try it yourself? ( Docs | App ) Thanks for reading Miru's Blog! Subscribe for free to receive new posts and support my work. --- ## Where Cloud Feature Flagging Fails for Robotics Configuration Management - URL: /blog/where-cloud-feature-flagging-fails - Published: 2025-05-23T21:48:29.000+00:00 - Author: Vedant Nair - Tags: Insights - Summary: This post compares configuration schema approaches and how they fit a robotics stack. It highlights trade‑offs in validation, composition, tooling, and long‑term maintenance. - Excerpt: Introduction As their fleet of robots grows, robotics teams look for scalable ways to manage, version, and deploy their configurations. Those with a cloud software background tend to first turn to cloud feature-flagging tools like LaunchDarkly, StatSig, or ConfigCat. Unfortunately, these tools were not built with the complexities of robotics in mind. As a result, they fall short of solving configuration management in robotics. This blog explores these different shortcomings. Bidirectional S Introduction As their fleet of robots grows, robotics teams look for scalable ways to manage, version, and deploy their configurations. Those with a cloud software background tend to first turn to cloud feature-flagging tools like LaunchDarkly , StatSig , or ConfigCat . Unfortunately, these tools were not built with the complexities of robotics in mind. As a result, they fall short of solving configuration management in robotics. This blog explores these different shortcomings. Bidirectional Sync and Local Caching Managing configuration in robotics is more complex than cloud apps because robots operate in dynamic, network-constrained environments. Their relationship with the server is different from that of cloud apps. Their propensity to lose network and power means they must be resilient to outages. To handle this, robots need to be able to sync configurations bidirectionally, store them locally, and recover from failures autonomously. Bidirectional Sync When a robot’s sensors are recalibrated, those new values should sync back to the cloud to be versioned with the rest of the configuration. If a developer pushes new config values from the cloud, the system should merge those changes without erasing local updates. Local changes should take precedence when necessary, and conflicts should be resolved automatically. Local Caching and Offline Recovery Robots operate in unstable network conditions, so they must store configs locally and handle failures without human intervention. To be resilient to network drops, the configuration should be cached, and polling should be retried once the connection is restored. If the robot reboots or loses power, it should revert to the last known working config and instantiate without relying on cloud access. Why Push-Based Models Fail Feature flagging tools are designed for cloud applications, not robotics. They rely on a push-based model where config updates are streamed from the cloud through a persistent connection (e.g., via server-sent events or WebSockets). This model assumes: A stable connection is always available. The cloud holds the single source of truth for config values. Clients don’t need to store config values locally. This setup works fine for cloud apps but not for robots. If the connection fails mid-update, the robot will be left in an undefined state. Robots must store configs locally and recover autonomously, which the push-based model isn’t designed to handle. State-Based Updates Robotic systems are stateful. Updating a robot mid-task—like during heavy lifting or precise movement—can lead to mechanical failures or safety hazards. Unlike cloud apps, robots must carefully time configuration updates to avoid interrupting critical operations. Cloud-based feature flagging tools rely heavily on Server-Sent Events (SSE) . SSE streams data continuously, which works fine for stateless environments. But in robotics, state drives behavior. Applying updates as soon as they arrive can be dangerous. To support this paradigm, robotics teams must write custom logic to monitor and manage state when applying updates — such as detecting whether the robot is active or at rest and deferring updates during critical tasks. This custom software is an additional burden on their infrastructure team and, if improperly maintained, can cause critical path errors. Hierarchical Overrides and Rollbacks Feature flagging tools are built for user-based segmentation — but robots need complex, hierarchical overrides. Cloud-based tools assume you’re targeting users, not machines. That mismatch makes overrides and grouping harder than they should be. Granular Targeting Robotics configurations aren’t flat; they’re hierarchical. You might want to update motor tuning for all industrial AMRs —except the fleet in Factory A , where the customer’s safety requirements mandate different settings. Even within that fleet, robots aren’t identical. One robot might have a misaligned sensor due to manufacturing tolerances—you’d want to apply a correction only to that specific unit without affecting the rest. Cloud feature flag tools struggle with this level of precision. They’re designed for simple user groups—like “all users in North America” or “beta testers”—not for multi-layered segmentation or per-device adjustments. Fleet-Wide Rollbacks Say you push an experimental update to 100 robots. Five robots experience issues—maybe due to mechanical wear or environmental differences—but the rest are fine. You’d want to roll back the update for those five without affecting the others. Most feature flag tools rely on simple user-based groupings. The rollback options will be limited if the grouping model isn’t granular enough to distinguish between different robot states or environments. Conclusion Robotics teams face unique challenges in configuration management that cloud-based feature flag tools weren’t built to handle: Bidirectional sync – Robots need to merge local changes with cloud updates without conflicts, even when offline. State-aware updates – Config changes must account for real-time state to avoid destabilizing active tasks. Granular targeting and rollbacks – Robotics configurations are hierarchical and require precise control at the fleet, model, and unit level. Flagging tools assume a stable connection and a stateless environment—neither of which apply to robots. To robotics teams looking for a way to solve this problem, you’ll have to keep looking for another solution. Thanks for reading! Fortunately, you won’t have to keep looking much longer :) At Miru , we’re building configuration management for robotics. We help robotics teams easily manage, version, and deploy their application configurations. If this is a pain you’re facing, please reach out! --- ## The Challenges of Application Configurations in Robotics - URL: /blog/the-challenges-of-application-configurations - Published: 2025-05-23T21:47:25.000+00:00 - Author: Vedant Nair - Tags: Company, Insights - Summary: This post compares configuration schema approaches and how they fit a robotics stack. It highlights trade‑offs in validation, composition, tooling, and long‑term maintenance. - Excerpt: Introduction A surprisingly common complaint I hear from robotics companies is their struggle to manage, version, and update application configurations (configs). At first glance, this seemed counterintuitive. Why should configs be harder here than in other fields? But robotics systems are uniquely complex: a patchwork of hardware, firmware, sensors, software, and ML. Configs are the glue that holds them together. In robotics, these configs define how a robot’s software interacts with its en Introduction A surprisingly common complaint I hear from robotics companies is their struggle to manage, version, and update application configurations (configs). At first glance, this seemed counterintuitive. Why should configs be harder here than in other fields? But robotics systems are uniquely complex: a patchwork of hardware, firmware, sensors, software, and ML. Configs are the glue that holds them together. In robotics, these configs define how a robot’s software interacts with its environment. They sit between code and execution, controlling everything from sensor calibration to safety protocols—all without requiring recompilation. In this blog, we’ll dive into what these configs are used for and why they’re so challenging for robotics teams to get right. What Are Application Configs Used For? Feature Flags: A/B test new features or gradually roll them out to mitigate risk Example: Enable a beta gripper control algorithm only for robots in your lab in your YAML features: autonomous_navigation: true # All robots have this enabled beta_gripper_control: true # Only enabled for beta robots in lab Environment Settings: Toggle system behavior in different environments from development to production. Example: .env file for switching between simulation and development. ENV=dev # dev vs. sim LIDAR_FIRMWARE=v2.3.1 # Specific to physical sensor LOG_LEVEL=info # trace -> info logs for dev Integration Settings: Connect to external systems: databases, auth servers, or APIs Example: A JSON config for uploading sensor data: { "storage": { "endpoint": "cloud-storage.example.com", "auth_token": "xyz", "sensors": ["lidar", "imu", "gps"] } } Hardware Definitions: Define how software interacts with motors, LiDAR, GPUs, etc. Example: YAML for a motor’s safety settings: motor_left: max_torque: 5.2Nm max_rpm: 120 emergency_stop: lidar_obstacle The Challenges of Application Configuration in Robotics As things scale—more robots, team members, and customers—configs can become difficult to manage, version, and update. Here’s why: Loosely Structured Data YAML and JSON are loosely structured —there’s no built-in way to enforce data types or validate values. This flexibility has its liabilities: A mismatch like lidar_scan_rate: "20" (string) instead of 20 (integer) can crash a strictly typed perception pipeline. Code/Configuration Versioning Robotics teams ship software updates frequently—but if the config schema doesn’t evolve in lockstep, things break. Struct Mismatches : The new code expects a field like motor_timeout_ms , but old configs omit it → robots ignore safety settings. Zombie Fields : You remove legacy_api_endpoint from the code, but leftover configs still reference it. When a subsystem or library tries to parse that field, you end up with cryptic errors or partial failures. Complexities of Scale : Imagine pushing v2.2.1 code to 100 robots only to realize that 30 still have v1.5 configs. Now you have to manually fix them! P.S. → If you’re a robotics team that needs help deploying over the air (OTA) software updates, check out Miru ;) Complex Configuration Groupings Robotics configs aren’t flat files—they’re hierarchical, conditional, and combinatorial. Hierarchical Configurations Customer X’s robots need a custom edge server IP. Without fleet-level configurations, you’re editing 50 YAMLs by hand. Conditional Logic: “Enable beta_feature_x only if the robot is in ‘ group_beta ’ and firmware > v2.1 .” Now, do that for 20 more features! Either find yourself an intern or get ready to be bored for the next 4 hours of your life. Combinatorial Complexity As shown below, an individual robot can have many configuration combinations, making managing and versioning challenging. Complexity Grows Exponentially with Fleet This pain multiplies as you deploy more robots. Differing Needs Between Stakeholders Multiple internal users rely on these configs, each with unique goals and skill levels: Software Engineers : Use configs in development, deployment, and debugging; highly technical. Deployment/Field Engineers : On-site installation and troubleshooting; less comfortable with raw data. Compliance : Must audit configs for regulatory reasons (e.g., speed-limit logging in autonomous vehicles). Deployment Challenges Even if you’ve versioned and grouped configs correctly, getting them onto the target device is tricky. Networking Constraints: Many robots are subject to poor networking. What happens if the network drops in the middle of an update? What happens if we are bandwidth-constrained and updates are slow? Restart & State Maintenance Some configuration changes require restarting an application. But what happens if the robot is in operation? How do we know whether to restart the application or do a hot reload? Rollbacks + Delta Updates We don’t want an update to download halfway, leaving the robot without some of the necessary parameters. How can we ensure that updates are atomic and roll back when something fails? In the same vein, most updates will be minor, so how can we be efficient by caching most of the updates and only pulling down the delta? User-Facing Configurations Some robotics products come with tablets or control planes that allow the end-user to configure the robot and change its settings, such as speed limits, task priorities, and safety zones. These user settings must map cleanly to underlying configs. If the UI writes arm_speed=9001 to a JSON file expecting a value from 0.1–1.0 , your robot is in trouble. Conclusion Configurations are one of the core software components that make robots operate smoothly. But as we’ve discussed, they have present gnarly problems in production: Invalid Data: Configurations lack strict typing or logic, which can lead to runtime errors due to unexpected values Config/Code Versioning: Both configurations and code need to be versioned in lockstep to ensure compatibility Fleet-Wide Complexity: As the fleet becomes more diverse, everything from configuration groupings to customer-specific parameters becomes more complex. Multiple User Groups: Software Engineers, Field Engineers, Compliance, and even end-users interact with configurations, making creating a centralized management platform difficult. Efficient Updates: Fast, reliable updates with minimal data transfer and rollback capability are needed. The kicker? None of this gets easier as you scale. Either pour engineering resources into a homegrown solution or keep doing it manually—risking the day a single YAML typo bricks a robot mid-mission. At Miru, we’re building tools to eliminate this pain for robotics teams and save them from standing up this infra themselves. If you’re having trouble managing, versioning, and deploying your application configs, let’s talk ! --- ## Tutorial: Building a Yocto Image for NVIDIA Jetson - URL: /blog/tutorial-building-a-yocto-image-for - Published: 2025-05-23T21:44:51.000+00:00 - Author: Vedant Nair - Tags: Tutorials, Featured - Summary: This post explains the core idea and why it matters in production. It ends with steps you can use without re‑architecting your stack. - Excerpt: Introduction Yocto is a framework that provides tools and processes for creating custom Linux images for embedded devices. It empowers developers with the flexibility to build Linux systems tailored to their needs while also offering production-grade quality, enhanced security, and long-term support. For these reasons, Yocto has become a popular choice among robotics engineers using NVIDIA Jetson devices to create custom images for their applications. However, Yocto's flexibility comes at a c Introduction Yocto is a framework that provides tools and processes for creating custom Linux images for embedded devices. It empowers developers with the flexibility to build Linux systems tailored to their needs while also offering production-grade quality, enhanced security, and long-term support. For these reasons, Yocto has become a popular choice among robotics engineers using NVIDIA Jetson devices to create custom images for their applications. However, Yocto's flexibility comes at a cost—it has a steep learning curve. Setting up an image can take hours (not to mention the lengthy build times), which means you might spend several days preparing your image before even having the opportunity to test it on your robot. I encountered this challenge myself, so I created this tutorial to help other engineers build with Yocto faster. In this guide, we’ll create a base Yocto image for the Jetson that includes ROS 2 and OpenCV, useful tools for whatever robotics applications you build. This tutorial is designed for beginner to intermediate Yocto users who want to go from 0 →1, starting from scratch, to successfully flashing a system image onto their Jetson device. By the end of this tutorial, you’ll understand how Yocto works, how to build a Yocto image, and how to customize it further to suit your specific application needs. Let's dive in! Yocto Basics This tutorial assumes you have a basic understanding of working with a Jetson and other embedded systems. If you’re new to Yocto, don’t worry—we’ll provide a quick rundown to help you get started. Check out this Miru blog for a deeper dive into Yocto’s core concepts and benefits. Here's some basic information to get you started: BitBake The build engine of Yocto. It processes recipes (instructions) to generate packages, kernels, and complete system images. Recipes (.bb Files) These are the core instructions that tell Yocto how to build a specific piece of software. Recipes include details like dependencies, source code locations, and build configurations. Layers Layers are modular collections of recipes, configurations, and other metadata. You can think of them as "building blocks" that can be stacked to customize your image. For example, the meta-tegra layer adds support for NVIDIA Jetson devices. Machine Configuration Yocto supports many hardware platforms, and you specify your target machine using a configuration file. This ensures the system image is optimized for your device (in our case, Jetson). Poky Yocto includes a reference distribution called Poky, which is often used as a starting point. Poky bundles essential tools and metadata for building a basic Linux system. Build Directory (Build Environment) This is where Yocto does its work. It stores temporary files, build artifacts, and configuration details. You’ll spend most of your time here when using Yocto. If any of these pieces are still confusing, don’t worry. As we go through the tutorial, things will make more sense. Requirements Hardware: x86 or ARM64 machine running Ubuntu 22.04 (recommended) or 20.04. You’ll need at least 50 GB of free space (but more is highly recommended, as Yocto builds are substantial). NVIDIA Jetson device SD card for flashing the Jetson Dependencies: Before starting, ensure you’ve installed the required dependencies. Run the following command on your host: sudo apt update sudo apt install -y gawk wget git diffstat unzip texinfo build-essential chrpath \ socat cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping \ python3-git python3-jinja2 libegl1-mesa libsdl1.2-dev pylint xterm curl lz4 \ qemu-user-static Preparing Jetson Files First, we’ll need to download the Jetson BSP (Board Support Package) and the Root File System (RFS) for Yocto to use in the build. What Are BSP and RFS, and Why Do We Need Them? BSP (Board Support Package): A BSP is a collection of drivers, config files, and tools that ensure Yocto can produce a system image optimized for your specific Jetson device. RFS (Root File System): The RFS is the base directory structure of the Linux OS. It contains libraries, binaries, and utilities that allow the OS to boot and function properly. I’m using NVIDIA Jetson Linux 36.4 , compatible with Jetpack 6.2 . You can download the BSP and RFS directly from this link . Once downloaded, extract the tarballs into your project directory: cd ~/Documents/yocto-jetson-ros-tutorial mkdir bsp cd bsp tar -xvpf ~/Downloads/jetson_BSP.tbz2 -C ~/Documents/yocto-jetson-ros-tutorial/bsp/ sudo tar -xvpf ~/Downloads/jetson_sample_root_filesystem.tbz2 -C ~/Documents/yocto-jetson-ros-tutorial/bsp/Linux_for_Tegra/rootfs/ You can also choose to download additional tools like CUDA or OTA Tools . While this tutorial skips these steps, make sure to extract any additional components into your /bsp directory using the same method. Once the BSP and RFS are extracted into your Yocto project, we must apply the binaries . sudo ~/Documents/yocto-jetson-ros-tutorial/bsp/Linux_for_Tegra/apply_binaries.sh Yocto Build Preparation We’re ready to download the Yocto-specific files needed to build our image. These files include the layers that BitBake will use in its recipes to build the final system image. First, navigate into your project directory: cd ~/Documents/yocto-jetson-ros-tutorial Next, we need to clone the Yocto Poky repository . Poky is the base layer for all Yocto projects and includes BitBake, the Yocto build engine, and other core layers. I’ll be using the Yocto Scarthgap release, but feel free to use any version you prefer: git clone --single-branch -b scarthgap git://git.yoctoproject.org/poky.git Navigate to the root of your project directory and create a /sources directory. This is where we’ll clone all the additional Yocto layers needed for our build. Layers to clone: meta-tegra is the Yocto layer for NVIDIA Jetson meta-ros is the Yocto layer for ROS and ROS2 meta-openembedded is a layer that provides a range of tools and libraries like OpenCV, Python, and OpenSSL/SSH Remember, layers are modular collections of recipes and config files. They combine to create the final Yocto image. As in our case, layers often focus on specific functionality or hardware. cd ~/Documents/yocto-jetson-ros-tutorial mkdir sources cd sources git clone --single-branch -b scarthgap https://github.com/OE4T/meta-tegra.git git clone --single-branch -b scarthgap https://github.com/ros/meta-ros.git git clone --single-branch -b scarthgap https://github.com/openembedded/meta-openembedded.git With all the required layers cloned, we’re ready to initialize the build environment and customize our image. cd ~/Documents/yocto-jetson-ros-tutorial/poky source oe-init-build-env Editing Configurations for Build: Since we’ve added custom layers to the Yocto build, we need to update the build configuration files so that Yocto can use these layers during the build process. First, let’s edit bblayers.conf . This file tells Yocto which layers to include in the build. nano ~/Documents/yocto-jetson-ros-tutorial/poky/build/conf/bblayers.conf Your original bblayers.conf file should look something like this, without the references to the meta-tegra , meta-ros , or meta-openembedded layers. POKY_BBLAYERS_CONF_VERSION = "2" BBPATH = "${TOPDIR}" BBFILES ?= "" BBLAYERS ?= " \ /home/miru/Documents/yocto-jetson-ros-tutorial/poky/meta \ /home/miru/Documents/yocto-jetson-ros-tutorial/poky/meta-poky \ /home/miru/Documents/yocto-jetson-ros-tutorial/poky/meta-yocto-bsp \ " Add the new layers to the file. POKY_BBLAYERS_CONF_VERSION = "2" BBPATH = "${TOPDIR}" BBFILES ?= "" BBLAYERS ?= " \ /home/miru/Documents/yocto-jetson-ros-tutorial/poky/meta \ /home/miru/Documents/yocto-jetson-ros-tutorial/poky/meta-poky \ /home/miru/Documents/yocto-jetson-ros-tutorial/poky/meta-yocto-bsp \ /home/miru/Documents/yocto-jetson-ros-tutorial/sources/meta-tegra \ /home/miru/Documents/yocto-jetson-ros-tutorial/sources/meta-ros/meta-ros-common \ /home/miru/Documents/yocto-jetson-ros-tutorial/sources/meta-ros/meta-ros2 \ /home/miru/Documents/yocto-jetson-ros-tutorial/sources/meta-ros/meta-ros2-humble \ /home/miru/Documents/yocto-jetson-ros-tutorial/sources/meta-openembedded/meta-oe \ /home/miru/Documents/yocto-jetson-ros-tutorial/sources/meta-openembedded/meta-python \ /home/miru/Documents/yocto-jetson-ros-tutorial/sources/meta-openembedded/meta-networking \ " The local.conf file customizes the build environment by specifying details like the target machine, included packages, and additional features. If you need to figure out the name of your target machine, the following command will help. My target machine is a jetson-orin-nano-devkit . ls ~/Documents/yocto-jetson-ros-tutorial/sources/meta-tegra/conf/machine Now open the local.conf file for editing. Feel free to follow the Yocto documentation to customize your build environment. I’m keeping it simple by adding ROS2 Humble and OpenCV . I’ve also changed my PACKAGE_CLASSES from .rpm to .deb. I've also included IMAGE_FSTYPES , which allows us to flash our target device with an SD card. nano ~/Documents/yocto-jetson-ros-tutorial/poky/build/conf/local.conf Update the file with these changes: Specify image file types for flashing: Adding the following lines lets you flash an SD card for Jetson: IMAGE_CLASSES += "image_types_tegra" IMAGE_FSTYPES = "tegraflash tar.gz" Add ROS 2 Humble and OpenCV to the build: IMAGE_INSTALL_append = " ros-humble opencv" Switch the package manager to .deb (optional): PACKAGE_CLASSES = "package_deb" Set the target machine: MACHINE = "jetson-orin-nano-devkit" Potential Fixes A few errors caused the build to fail on my setup. I’m sharing them with you in case you encounter any similar issues. You can either fix them proactively or as problems arise in your build. If doing the latter, remember to clean and rebuild the individual packages each time you make a fix. Feel free to skip these if none apply to you! CMake ( meta-ros layer): If you’re using ROS2 Humble, there’s a problem with the build process of ament-cmake-core-native . This happened because the PYTHONPATH environment variable was not correctly set, meaning that Python could not locate the ament_package module. We’ll need to fix the issue manually. You can also see this GitHub issue to reference the problem and the fix. Navigate to this directory and edit the ros_opt_prefix.bbclass file. cd ~/Documents/yocto-jetson-ros-tutorial/sources/meta-ros/meta-ros-common/classes nano ros_opt_prefix.bbclass We will need to change this line. export PYTHONPATH:prepend = "${STAGING_DIR_NATIVE}${PYTHON_SITEPACKAGES_DIR}:” This is the fixed version. export PYTHONPATH = "${STAGING_DIR_NATIVE}${PYTHON_SITEPACKAGES_DIR}:” Patch Resolution ( meta-ros layer) During the build, the patch 0001-CMakeLists.txt-prevent-building-zstd-with-ExternalPr.patch failed to be applied to the do_patch task. The patch was designed to modify the CMakeLists.txt file for the zstd-vendor package, but it’s likely outdated. The patch was referenced in the .bbappend file for the zstd-vendor package. ~/Documents/yocto-jetson-ros-tutorial//sources/meta-ros/meta-ros2-humble/recipes-bbappends/rosbag2/zstd-vendor_0.15.12-1.bbappend Find the line where the patch was referenced. SRC_URI += "file://0001-CMakeLists.txt-prevent-building-zstd-with-ExternalPr.patch" And disable it by commenting it out. # SRC_URI += "file://0001-CMakeLists.txt-prevent-building-zstd-with-ExternalPr.patch" hwloc ( meta-openembedded layer) The hwloc package failed the do_package_qa task because the libhwloc subpackage depended on libcudart.so.12 , which was not listed as a runtime dependency. So, we edited the recipe. ~/Documents/yocto-jetson-ros-tutorial/sources/meta-ros/meta-ros2-humble/recipes-bbappends/rosbag2/zstd-vendor_0.15.12-1.bbappend And we added this line to the recipe, adding the missing dependency. RDEPENDS:libhwloc += "cuda-cudart" Start Yocto Build! Now that we’ve set everything up, we have one more step before we start the build. We want to see all the layers in our image for a last check. This is also helpful for determining whether your build environment is active. cd ~/Documents/yocto-jetson-ros-tutorial/poky/build bitbake-layers show-layers If everything looks good, the only thing left to do is start the build. Yocto offers a range of options for the type of image you want to build . Choose the one that suits your needs. bitbake -v core-image-base Beware, Yocto builds take a LONG time. Now’s a great time to do something else—like walk your dog or wash your car. You won’t be missing out on much! Flashing your device Once your Yocto image is successfully built, we can flash it onto our target device. meta-tegra does not support flashing our target Jetson device with an SD card image, so we can’t use flashing software like Balena Etcher . Instead, we’ll flash it ourselves using the terminal. We need to locate the ext4 file to ensure it was successfully generated. For me, the file name was core-image-base-jetson-orin-nano-devkit.rootfs.tegraflash.tar.gz . ls -lh ~/Documents/yocto-jetson-ros-tutorial/build/tmp/deploy/images/<MACHINE> Extract the .tegraflash.tar.gz file into a temporary directory. mkdir ~/yocto-image cd ~/yocto-image tar -xvf core-image-base-<MACHINE>.tegraflash.tar.gz Now, prepare the SD card. Insert the SD card into your computer and identify the device. lsblk Once you identify your device (mine is /dev/sdb ), flash the SD card with the script provided by the tarball. sudo ./dosdcard.sh /dev/sdb -y Before we boot, we need to configure boot settings. Mount the SD card and edit the boot config file. sudo mount /dev/sdb1 /mnt sudo nano /mnt/boot/extlinux/extlinux.conf Update the APPEND line to point to the right root device. APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait ${bootargs} Now unmount the SD card. sudo umount /mnt Now, insert your SD card into your Jetson device and boot up. You should see the NVIDIA logo, followed by kernel boot messages. If this is your first time booting with Yocto (or you encounter boot issues), you may need to update your SPI flash. Go here for more instructions on how to do that. Depending on your device, you may need to follow Jetson documentation to put your device into recovery mode. Congrats—you’ve successfully used Yocto to flash your Jetson device with a custom Linux image. Happy building! --- ## Using Docker vs. Yocto in Robotics - URL: /blog/using-docker-vs-yocto-in-robotics - Published: 2025-05-23T21:19:37.000+00:00 - Author: Vedant Nair - Tags: Insights, Guides - Summary: This post explains the core idea and why it matters in production. It ends with steps you can use without re‑architecting your stack. - Excerpt: Introduction The debate over packaging robotics software is as old as robots themselves! As the industry has evolved from 'dumb' robots, like Kawasaki robot arms, to intelligent, autonomous systems, like Waymo’s robotaxis, this debate has intensified. Unlike their predecessors, intelligent robots require an iterative software development life cycle (SDLC) driven by rapid advancements in embodied AI, constantly pushing the state-of-the-art (SOTA) in robotics. Apple and Tesla have popularized Introduction The debate over packaging robotics software is as old as robots themselves! As the industry has evolved from 'dumb' robots, like Kawasaki robot arms , to intelligent, autonomous systems, like Waymo’s robotaxis , this debate has intensified. Unlike their predecessors, intelligent robots require an iterative software development life cycle (SDLC) driven by rapid advancements in embodied AI, constantly pushing the state-of-the-art (SOTA) in robotics. Apple and Tesla have popularized over-the-air (OTA) software updates for hardware. The standard is clear — to be defined as a modern hardware company, customers expect frequent software updates. The requirement to quickly ship software updates means robotics teams are reconsidering how to package their software. The ideal deployment flow must: Simplify software packaging, Integrate seamlessly with CI/CD pipelines, Enable rapid iteration, All without sacrificing performance! In this blog, we’ll explore two popular software packaging methods for robotics: Docker and Yocto . We’ll explain them, their benefits, and how to choose the right approach for your next robotics project. Buckle in! Docker The Docker platform uses containerization to package applications with their dependencies, libraries, and configurations, ensuring consistency across operational contexts. In this section, we’ll cover the core concepts and features essential to using Docker. Core Concepts: Container A lightweight, isolated runtime environment bundling an application and its dependencies. Image A pre-built, immutable template used to create containers. A container is an instance of an image. Dockerfile A text file with instructions for building a Docker image, specifying the base image, dependencies, configurations, and setup commands. Docker Compose A tool that manages and orchestrates multi-container systems using a .yml file. Simplifies running interconnected services. An easier but less powerful alternative to Kubernetes and Docker Swarm. Container Registry Stores and distributes Docker images, either cloud-hosted (e.g., Docker Hub, AWS ECR, Google Artifact Registry) or on-prem. Volume Persists data outside a container’s lifecycle, enabling data sharing between containers and ensuring it survives restarts. Layer Each instruction in a Dockerfile creates a cached layer, optimizing builds and storage. Features: Containerization What it is: Containerization isolates services into separate, lightweight units, ensuring portability and repeatability. Why it’s useful: A robotics development workflow has a variety of operational contexts— from developer workstations, simulations, hardware-in-the-loop (HIL) rigs, test robots, and production robots (whew, that’s a lot 😅). Containers reduce integration issues between these contexts. Isolation What it is: Docker containers run independently of the host operating system (OS) and other containers. Why it’s useful: Robotics systems run multiple services in conjunction, — such as path-planning, perception, and object detection. Isolation encapsulates dependencies and runtime environments, preventing conflicts. Ease of Deployment What it is: Once built, Docker images are reusable and, when stored in a container registry, can be quickly deployed to target devices. Why it’s useful: Robotics engineers test iteratively on real machines. Pre-built images eliminate the need to rebuild large applications (often 10+ GB), streamlining the workflow. Yocto Yocto , primarily used in embedded systems, allows engineers to create custom Linux-based system images. It provides the tools, templates, and workflows to build images from source, offering control, flexibility, and optimized performance for specific hardware. Here’s what you need to know about Yocto: Core Concepts: Image The final output of a Yocto build. Combines the kernel, bootloader, and software packages into a deployable format. Recipe Like a recipe for baking a cake! Defines how to fetch, configure, and build software components to create images. Uses .bb (BitBake) file format to specify source locations, dependencies, and build instructions. Layers Modular collections of recipes and configurations. Promotes separation of concerns between OS and application layer. Examples: Board Support Package (BSP) Layer meta-tegra for Jetson and meta-robotapp for ROS2. BitBake The task scheduler and engine at the core of Yocto. Executes instructions in recipes to build software components, packages, and images. Tasks Build steps handled by BitBake. Defined in recipes and executed in order, based on specified dependencies. Examples: fetching source code, compiling, and packaging software. Package Management Outputs build artifacts in formats like .deb ( Debian ), . rpm ( RedHat ), and .tar ( Tarball ), Enables software installation, updates, and removal on target devices. Features: Customizable Linux Distribution What it is: Yocto is like à la carte dining, while Ubuntu is a buffet. Ubuntu offers ALL components and libraries, even those you may not need. Yocto lets you select only the components that perfectly match your use case. Why it’s useful: Every robot has a unique configuration of hardware, firmware, software, and ML. Yocto allows you to build the Goldilocks image for that configuration and eliminate everything else. Lightweight Images What it is: Yocto generates minimal, efficient system images tailored to the hardware. Why it’s useful: Edge devices have limited CPU power, RAM, and disk space. By removing unnecessary components, Yocto images reduce overhead, allocating more resources to critical tasks like perception or control. This can lower latency, improve response times, and optimize power consumption. Real-Time Support What it is: Yocto supports kernel modifications like PREEMPT-RT that support real-time operations. Why it’s useful: Robots with complex actuation systems—like Tesla Bot , Swiss Mile quadrupeds, or Aescape ’s massage robots—rely on precise timing for motor control and sensor fusion. These tasks require real-time capabilities that Ubuntu and other off-the-shelf Linux-based OSs do not natively support. Just use a kernel modification with Yocto! Why Use Docker? Now that we understand Docker and Yocto’s role in robotics, let’s discuss why you may want to use one or the other in your workflow. Let’s start with Docker. Portability A robotics development workflow involves a variety of simulation, testing, and production environments. Because Docker isolates runtime into containers, porting an application across these contexts is easy. This allows for consistent performance, reducing the confounding variables of running on bare metal. Faster Dev Cycles Speed is everything in robotics development. Imagine you’re testing a change to your robot’s perception module. Waiting 10 minutes for your system image to rebuild for every tweak would be maddening. Docker solves this by using prebuilt, cached images. Instead of a full rebuild, you can pull a mostly cached container in say, 30 seconds. Over the life of a project, those seconds add up! Flexibility with Hardware Upgrades Over a robot's lifetime, its hardware spec will evolve—whether upgrading to the latest Jetson board or switching architectures from ARM64 to x86 . Yocto requires a custom build for each hardware configuration, which can be a painstaking process whenever you change hardware. I’ve met teams who stayed on a board for generations longer than they wanted to because a new Yocto build would require too much development time. Docker abstracts these dependencies, reducing both versioning and maintenance overhead. Robust CI The highest-fidelity testing you can get is running your software on a real robot. But let’s be honest: that’s slow and unscalable. A better CI/CD approach is to frontload as much testing as possible in the cloud or simulation. Containers shine here—they let you automate testing, scale simulations, and run everything in parallel. By the time you’re ready to deploy, most issues are ironed out. Then, you can focus on high-value validation testing on the robot itself, where it matters. Simplified Debugging Diagnosing and reproducing bugs is much easier with Docker. Containers are fully reproducible and can be shared and run on any machine. This allows teams to collaborate on troubleshooting even when working across different workstations or environments. The Docker Ecosystem! Docker’s mature ecosystem offers an expansive library of open-source containers for common applications like databases, monitoring, and more. These pre-built images save you the effort of re-creating infrastructure, letting you focus on what matters: building differentiated software. Why Use Yocto? Let’s discuss some of the benefits that make Yocto an attractive solution. Hardware-Specific Builds From the device drivers to the computer, your robot has a unique configuration. Thanks to Yocto’s custom builds, you can design your Linux distribution to fit. Optimized for Edge Devices Robots operate with limited RAM, CPU power, and disk space while collecting massive amounts of data that can only be offloaded intermittently. Every byte counts. Yocto strips away unnecessary components, reducing memory and CPU usage while improving boot times. Real-Time Capability Robotic applications with strict timing requirements need deterministic execution, which Ubuntu does not offer. Integrate a real-time kernel modification with Yocto to unlock this capability! Long-Term Control Robots are deployed in the field for months or even years, making complete control over your operating system essential. A custom Yocto Linux means you’re no longer subject to any 3rd party software release/update cycle. If Ubuntu ships a version release (e.g., 22.04 to 24.04), this update can break dependencies, introduce device driver incompatibility, or change kernel behavior. A Yocto build is more likely to be stable for the life of your project. Enhanced Security Device security is paramount, especially for robots operating in safety-critical environments. Embedded systems have numerous attack vectors, making this task difficult. Ubuntu provides security features like signed binaries and encrypted root filesystems, but its unnecessary components widen the attack surface. A custom Yocto build eliminates these components while allowing you to implement these mechanisms. Board Support Packages (BSPs) Though much of the Yocto process is do-it-yourself, BSPs reduce the development effort needed. BSPs are a pre-configured set of files, drivers, kernel, and middleware designed for a specific ‘board’ (Raspberry Pi, Jetson, among others). BSPs serve as the “base layer” for the image. The Yocto ecosystem provides a variety of open-source BSPs. Over-the-Air OS Updates Docker primarily operates at the application layer, so you must build separate infrastructure to update your OS or other lower-level software over the air (OTA). Fortunately, Yocto supports OTA OS updates. You can choose between approaches like image updates ( swupdate ), partition updates ( mender and RAUC ), or OSTree . When to use Docker vs. Yocto? Let’s summarize when to use Docker vs. Yocto. Of course, every design decision has its tradeoffs, so we’ll also discuss those. Where Docker Wins Prototyping: In the early stages of a robotics project, you can’t afford to sink time into a custom build system. You need to validate your application and move on quickly. Containers are perfect for an iterative cycle between your workstation and robot. Tradeoff: While skipping any packaging work and building directly from source can speed up prototyping, it introduces dependency management headaches later in the workflow. Collaborative Workflow: Robotics teams often have engineers working across diverse contexts, from simulation and hardware-in-the-loop (HIL) setups to actual robots. Docker’s portability allows seamless handoffs between teammates and contexts. Frequent Updates: Following an agile SDLC requires the infrastructure for bi-weekly updates. Docker makes application updates simple. Tradeoff: Docker can’t perform updates for the OS or other low-level dependencies. Where Yocto Wins Production-Grade: Yocto is great for building production-grade system images. Without reliance on third-party vendors, you maintain full control, security, and long-term stability. Tradeoff: Yocto has a STEEP learning curve. Expect to invest significant time learning Yocto before your system image runs successfully. Resource-Constrained Devices: For edge devices with limited CPU, RAM, and storage, Yocto eliminates unnecessary software components, minimizing resource usage and improving system performance Tradeoff: These builds are hardware-specific, making reusing them across different devices difficult. Real-Time Needs: If your device requires real-time or deterministic operations, Yocto is the best way to modify your kernel to support this. Docker and Yocto are NOT Mutually Exclusive! Throughout this blog, we’ve compared Docker and Yocto, and given you criteria to decide between them. This is a false dichotomy! Though most teams use them independently, Docker and Yocto are not mutually exclusive, and a hybrid approach can work splendidly! In a hybrid approach, Yocto is used to package the OS and lower-level dependencies. This delivers a custom, highly performant system image with the ability to implement over-the-air (OTA) updates for your OS. Docker complements this by handling your application software. Since application updates are far more frequent than OS updates, Docker’s ease of use becomes invaluable. Configuring a Yocto build alongside Docker images requires a significant upfront investment. However, this hybrid system has a high ROI in production. Thanks for reading! I hope this blog helped you decide how to package software for your next robotics project. Best of luck, and I can’t wait to see what you build. --- ## Why Server Automation Tools Fail at Robotics Configuration Management - URL: /blog/why-server-automation-tools-fail - Published: 2025-05-23T21:17:19.000+00:00 - Author: Vedant Nair - Tags: Insights - Summary: This post compares configuration schema approaches and how they fit a robotics stack. It highlights trade‑offs in validation, composition, tooling, and long‑term maintenance. - Excerpt: Intro When infrastructure engineers at robotics teams first learn about Server Configuration Management (CM)/Automation tools like Ansible, Salt, Chef, and Puppet, they jump for joy. They think they’ve found their panacea: “Robots are just edge servers, right? Let’s use CM tools for everything: provisioning, deployment, configs, and inventory management. Done.” Time to go home, kick up their feet, and let a CM tool handle their work for them! That optimism isn’t entirely misplaced. These tool Intro When infrastructure engineers at robotics teams first learn about Server Configuration Management (CM)/Automation tools like Ansible , Salt , Chef , and Puppet , they jump for joy. They think they’ve found their panacea: “Robots are just edge servers, right? Let’s use CM tools for everything: provisioning, deployment, configs, and inventory management. Done.” Time to go home, kick up their feet, and let a CM tool handle their work for them! That optimism isn’t entirely misplaced. These tools are solid. Robotics teams commonly use them in different parts of their CI/CD pipelines. However, things start to break down when using CM for robotics application configuration; this is where the mental model of “robots as servers” falls apart. Robots need dynamic, device-specific, and runtime-dependent configurations, which differ from the static config templates CM tools were built for. In this blog, we’ll discuss why CM tools fall short for robotics configs—and what we need instead. Incorrect Networking Assumptions CM tools were built to configure servers sitting comfortably in data centers, like the AWS us-east-1 region in Virginia. Every square inch of these facilities is supported by world-class networking infrastructure and teams of experts on standby to fix whatever goes wrong. At the time of writing, the last major outage in us-east-1 was in June 2023. It lasted two hours. In robotics, a mere two-hour network drop is a walk in the park. Depending on your use case, it’s common for your robot to drop off the network for hours, even days. CM tools, built with data center networking assumptions, aren’t designed for this level of flakiness. Push-Based vs. Pull-Based Model Some CM tools (Ansible or Salt’s default) rely on a push-based model . This means the server (cloud) pushes configuration updates to the client (robot). In robotics, this is risky. If the network’s shaky, you might push a new configuration while part of your fleet is offline. Now you’ve got different robots running different versions of your software. Worse, if a robot goes offline mid-update, it might end up in a broken state that needs manual recovery. A pull-based model is better. In this approach, the client polls the server for new configurations and pulls them down whenever an update is ready. Puppet and Chef use this approach. But whether they push or pull, these tools still assume a reliable network — and that’s where things fail. Queuing Let’s zoom in on a pull-based system. It defaults to pulling configurations from the server at some regular interval (Puppet’s is every 30 minutes). However, in the event of networking or power drops, we’d prefer that the client poll the server for a new update immediately upon restarting or re-establishing the network connection. But we don’t usually want the robot to apply the new configuration immediately once it has been downloaded. Robots are stateful systems. They might be mid-mission, interacting with a human, or doing something sensitive in the physical world. Applying a configuration change at the wrong moment can have dangerous consequences. Servers don’t face this problem as sharply. They run in predictable environments, with workloads that can be drained or paused with less disruption. So, for robots, instead of applying updates immediately, we want a local queue . The robot should pull down the config and store it locally. Then, when it’s safe, the robot can apply it. CM tools don’t give you this out of the box. No retry logic. No local job queue. If you care about when and how updates get applied (you should!), you’re left writing that logic yourself. Safe Updates and On-Device Versioning Atomic Updates It’s best practice to have atomic updates and rollback functionality. We don’t want to incrementally update configs and be left in a bad state if the update completes partially. We also want to easily roll back to a previous configuration version if our application isn’t behaving as expected. An atomic update is all or nothing. If any part of the update fails, the whole system is rolled back, leaving it in its original, working state. Salt, Puppet, and Chef apply updates sequentially, like scripts. If something fails, they move on. This assumption makes sense in a server farm, where you treat your machines like cattle. However, robots are more like pets, so we should treat our configuration updates with care. We can’t simply move on if the update only completes partially. Device Versioning CM tools rely on a central manifest that defines the server’s preferred device state. Because it is a central command/control system, it doesn’t care much about the device’s current state. This means that versioned, per-device history is not a first-class citizen. So if a robot misses config v3 and the latest version is v4, it will naively try to jump ahead—but that’s not always safe. You’d prefer to give the device the data for both v3 and v4 to upgrade sequentially, ensuring a smooth update. This lack of versioned per-device history also makes rollbacks difficult. Rollbacks require versioning and snapshots. We want to know the device’s version history, its last good state, and establish a safe way to reapply it. With traditional CM, you have to create version tracking, define migration paths, and handle rollbacks yourself. That’s critical path infrastructure we’d prefer not to build ourselves! Drift Detection Unlike servers, robots have configs that are edited locally. Whether an operator or end user modifies a setting in the business logic layer (like a CAD input in construction) or a robot recalibrates itself, these changes are not in lockstep with the cloud. When local configs and cloud versions are out of sync, we call this a configuration drift . The drift itself isn’t problematic, but what’s important is being able to resolve it when it happens. Teams have varying policies for resolving drift. Some want the robot to be the source of truth. Others want the cloud to win. Some even prefer one-off, snowflake configs. But all of them need visibility. CM tools don’t give you that. They only track the cloud’s desired state. They have no idea what the device is running right now. That means you can’t handle drift—you don’t even know it exists. And if you don’t know what’s running on the device, you can’t be confident that your next update will work. Analytics Weak device versioning also negatively affects our ability to run analytics. A robot’s application configs are its analytics primitives. When debugging a field issue, measuring uptime, or running an A/B test, you correlate performance with the config in place at the time. That requires device-level versioning. Again, CM tools miss this. Without it, we can't trace metrics back to the exact config that was running. We’re forced to hack around this with brittle logging or tagging systems that don’t scale. The result: we lose precision in our data, and any insight we draw about fleet performance is less trustworthy. Schema Management CM tools are process runners. They don’t know or care what’s in the config—they’ll happily deploy garbage as long as the YAML parses. But our deployment process can’t be that naive. A valid file != valid config. We must ensure that our configuration schemas are correct and safe to roll out, without piling more friction onto the software engineer deploying them. CM tools don’t natively integrate with configuration schemas like JSON Schema , Protobufs , or CUE that enforce types, valid values, or structural constraints. That means you can accidentally set max_torque: 1389 , launching a robot arm through a wall! CM tools don’t have a built-in way to check if a new config is compatible with the previous config and the application code. If there’s an incompatibility, errors get caught at runtime instead of deploy time, where the stakes are much higher. This puts the onus on the software engineers, who must perform this checking. But engineers don’t want to add this infrastructure to their laundry lists of tasks. They don’t want to build config validation infrastructure. They also don’t want to cross their fingers every time they deploy. They want something that just works, and control over the schema to make sure it does. Overrides Overrides should be simple. You should be able to specify: “These five robots at Customer A get a camera config X” or “Any robot running version 2.1 should increase its timeout to 25s.” You shouldn’t need to write a new file for every robot or fork your entire structure to change one value. But that’s exactly what traditional tools make you do. Salt lets you override through Pillars, but you end up writing conditionals in YAML or Jinja, hardcoding robot IDs into top.sls files. There's no layering, no merging — just string-matching and crossed fingers. Puppet uses Hiera, which sounds fancy until you realize it means a new file per override. You want to tweak something for ten robots? You’re now ten files deep in a folder maze. There's no way to say “robots with camera=flir” or “battery < 50%.” Chef does support layered overrides — default , normal , override , and force_override which sounds like a good idea. But it’s hard to trace where a value came from in practice. Overrides can come from cookbooks, roles, environments, or node-specific JSON. There’s no unified view or merge semantics you can reason about. So when something goes wrong, debugging turns into an expedition through scattered files and Chef's run-time state. Ansible is even simpler — and not in a good way. Host vars and group vars overwrite each other. There’s no merging, no targeting, and no live visibility. They all assume that configuration is something you set once, like a shipping label. But robotics doesn’t work like that. They need dynamic, layered, and functional overrides. Conclusion CM tools are still useful in robotics. They’re powerful for orchestrating software updates, doing OS-level configurations, and installing dependencies. But they weren’t built to safely manage configs in robotics for fleets of stateful, network-constrained robots. They lack the needed robustness, composability, and targeting. Robotics teams need something better. That’s why we’re building Miru : configuration management purpose-built for robotics. We help robotics teams manage, version, and deploy their application configs. If this has been challenging for you, reach out! We’d love to help. --- Edit with × Generated by: lapis trylapis.com