Robots built on the Robot Operating System are a web of small programs exchanging messages, and fuzzing them by throwing random messages at one node finds few bugs because most random inputs are wasted and code coverage of one node says nothing about the robot as a whole — ROFER mutates each field of a message according to how much it changes behaviour and measures coverage by the message patterns that flow between nodes, finding 88 real bugs in 13 ROS 2 programs, 46 confirmed by developers.
The argument in one glance
-
Problem
A robot is many programs passing messages, and bugs hide in the combinations
The Robot Operating System (ROS) is the most common framework for building robot software. A robot is a set of nodes, small programs, that publish and subscribe to typed messages on named topics. A velocity command, a laser scan or a map update is a message with many fields. A bug may only appear when one field of one message pushes the robot into a state that a second node then mishandles. Finding such bugs by hand does not scale.
-
What people did
Fuzz the nodes: send them random messages and watch for crashes
Fuzzing, generating many random or mutated inputs and checking whether the program misbehaves, has found thousands of bugs in ordinary software. Several recent tools adapt it to ROS by mutating messages, replaying them to nodes, and using code coverage or hand-written correctness rules as feedback.
-
Why it fell short
Wasted inputs, blind feedback, and one-off setups
The abstract names three limitations of existing ROS fuzzers. Test-case generation is inefficient: mutating a whole message at random mostly produces inputs that do nothing new. Feedback is ineffective: code coverage of a single node cannot see a state change that only shows up in another node. Generality and automation are weak: each target needs manual harnessing or manually written oracles.
-
This paper's idea
Mutate by dimension, and measure coverage by the messages between nodes
ROFER introduces two techniques. Dimension-level mutation treats each field of a message as a dimension and estimates how much mutating that dimension contributes to new coverage, so the fuzzer spends its effort on the fields that matter. Message-guided fuzzing adds a new coverage metric, the message feature, which captures the possible state transitions of the robot as seen through the messages exchanged by multiple nodes, so progress is measured at the level of the whole robot rather than one process. On 13 common ROS 2 programs, ROFER found 88 real bugs, 46 of them confirmed by ROS developers, and it found more bugs with higher coverage than four state-of-the-art ROS fuzzers.
Why this problem is worth solving
A delivery robot in a warehouse runs dozens of cooperating programs. One reads the laser scanner, one keeps the map, one plans a path, one converts the plan into wheel speeds. In the Robot Operating System
An open-source framework for robot software in which a robot is built as a set of communicating programs called nodes. node
One ROS program, running as its own process, that publishes messages on some topics and subscribes to others. messages
A typed data structure, such as a velocity command or a laser scan, sent on a named topic from one node to any node that subscribes.
This design makes robots modular, and it makes bugs hard to find. A message such as a velocity command has several numeric fields: linear speed on three axes and angular speed on three axes. A wrong or extreme value in one field may be handled correctly by the node that receives it and still push a downstream node into a state its author never considered. The failure surfaces two nodes away from the input that caused it. A crash of the navigation stack while the robot moves is a safety problem, not only a software one.
The paper’s evaluation gives a sense of how many such bugs are waiting: 88 real bugs across 13 common ROS 2 programs, of which 46 were confirmed by the ROS developers.
What people did before, and where it broke
- Coverage-guided fuzzing of ordinary programs. Tools such as AFL mutate an input file, run the program, and keep inputs that reach new code. Where it broke. A ROS node has no input file. Its inputs are streams of messages arriving on topics, and its behaviour depends on the timing and combination of messages from other nodes.
- ROS-specific fuzzers. Recent tools, including RoboFuzz, adapt fuzzing to ROS by mutating messages and replaying them to a node or a simulated robot, with feedback from code coverage or from hand-written rules that state what correct robot behaviour is. Where it broke. The abstract identifies three problems. Mutating whole messages at random wastes most test cases, because many fields do not affect behaviour at all. Code coverage inside one node misses transitions that only appear in the messages another node produces. And hand-written oracles or per-target harnesses make each new program a manual project.
- The gap. No tool combined an efficient way to decide what to mutate with a feedback signal that reflects the state of the robot as a whole, in a form that works across many ROS programs without manual setup.
Where ROFER sits
One fuzzing round in ROFER
-
Seed messages
Real messages recorded from the target ROS 2 program.
-
Dimension-level mutation new
Choose which message fields to mutate, weighted by how much each field has contributed to coverage so far.
-
Execute the program
Publish the mutated messages to the nodes under test and let them run.
-
Message-feature feedback new
Observe the messages exchanged between nodes and extract a coverage signal that reflects robot state transitions.
-
Keep or discard
Inputs that reveal new message features are kept as seeds; crashes and other bugs are reported.
The idea
This section explains the two techniques named in the abstract and the standard fuzzing ideas behind them. The paper’s precise algorithms, formulas and implementation choices were not available, and nothing below beyond the abstract’s statements should be read as the paper’s specific design.
Dimension-level mutation. A ROS message is a structure of fields, and each field is one dimension
One field of a message, treated as an independent axis along which the fuzzer can mutate the input.
Message feature as coverage. Standard fuzzers measure progress by code coverage: which branches of the program ran. For a robot that is the wrong unit, because interesting behaviour is a change of the robot’s state, and that state is spread across nodes. ROFER defines a new metric, the message feature
A coverage signal derived from the messages exchanged between ROS nodes, intended to reflect the possible state transitions of the robot.
Why these two fit together. The mutation strategy needs a feedback signal to learn which dimensions matter, and the message feature is exactly such a signal at the level of the whole robot. Dimensions that change the message patterns between nodes are the ones that change robot behaviour, so the two techniques reinforce each other.
Tracing one fuzzing round on a navigation node
How a single mutated field reaches a bug two nodes away
An illustrative scenario built from the abstract's description and standard ROS behaviour. Node and field names are typical of ROS 2 navigation; the paper's actual benchmark programs and bugs were not available.
Step 1 of 5
Start from a real seed
ROFER records normal traffic: the planner publishes a velocity command with linear x = 0.5 metres per second and all other fields near zero, the controller consumes it, and the odometry node reports the resulting motion.
seed: cmd_vel(linear x 0.5, y 0, z 0; angular x 0, y 0, z 0)
Why it works
- Not all fields are equal. Because many message fields do not affect behaviour, spending mutation budget in proportion to observed contribution puts most test cases where they can reach new states. Therefore fewer executions are wasted.
- Messages are where robot state becomes visible. Nodes influence each other only through messages, so the pattern of messages is a faithful trace of the robot’s state transitions. Feedback built on it sees progress that single-node code coverage cannot.
- Feedback and mutation form a loop. The message feature tells the mutator which dimensions changed robot behaviour, and the mutator uses that to find more new features. The loop converges on the inputs that drive the robot into rare states, where bugs live.
- Messages are generic. Every ROS program uses typed messages on topics, so a fuzzer that works at the message level applies to many programs without writing a new harness or oracle for each, which addresses the generality and automation limitation the abstract names.
What it costs
- A message-level view can miss bugs inside a node. If a bug changes nothing about what a node publishes, the message feature will not flag it. How ROFER combines message features with conventional code coverage was not available.
- Contribution estimates need warm-up. The mutator can only weight dimensions by contribution after it has observed some rounds, so early fuzzing is closer to uniform.
- Robots must be run to be fuzzed. Each test case executes real ROS 2 nodes, often with a simulator, so throughput is far lower than for a fuzzer of a single library function.
- Confirmed does not mean fixed or severe. Of the 88 bugs, 46 were confirmed by developers at the time of writing. The severity breakdown and how many were crashes versus logic errors were not available.
- Comparisons depend on the baselines chosen. The abstract states that ROFER beats four state-of-the-art ROS fuzzers on bugs found and coverage, but does not name them or give the margins.
The evaluation
ROFER against prior ROS fuzzers, as stated in the abstract
| Property | Prior ROS fuzzers | ROFER |
|---|---|---|
| Test-case generation | ||
| Feedback signal | ||
| Generality and automation | ||
| Bugs found |
Check yourself
Check your understanding
1. Why is code coverage of a single ROS node a weak feedback signal for fuzzing a robot?
2. What does dimension-level mutation change about how test cases are generated?
3. Predict what happens if a target program has a bug that corrupts an internal variable but never changes any message the node publishes.
4. What does the figure of 46 out of 88 bugs represent?
Sources
The paper is “Multi-Dimensional and Message-Guided Fuzzing for Robotic Programs in Robot Operating System” by Jia-Ju Bai, Hao-Xuan Song and Shi-Min Hu, Tsinghua University, published at ASPLOS 2024, Volume 2, pages 763 to 778, DOI 10.1145/3620665.3640425.
This explainer is marked as a draft because the full text was not accessible. The ACM Digital Library returned an anti-bot page, no arXiv preprint exists, Unpaywall and Semantic Scholar list the paper as closed access, the first author’s homepage lists the paper without a PDF, no artifact repository could be found, and the open-access survey that discusses the tool blocked automated retrieval on every route tried. Every claim attributed to the paper comes from its abstract: the three limitations of prior work, the two techniques, the 13 ROS 2 programs, the 88 bugs with 46 confirmed, and the comparison against four state-of-the-art approaches. The description of ROS, coverage-guided fuzzing and RoboFuzz is standard background, and the traced example is illustrative rather than taken from the paper.