ReCA: Integrated Acceleration for Real-Time and Efficient Cooperative Embodied Autonomous Agents

Teams of AI robots that plan and cooperate through large language models can do impressive long-horizon chores, but they are painfully slow — 18 to 23 minutes per task — because every step waits on several cloud LLM calls, and they scale badly as agents pile up; this work from Georgia Tech and partners profiles why, then co-designs a fix across algorithm, system, and hardware — running a small local model instead of a cloud one, letting one high-level plan drive many low-level moves, splitting agents into clusters that cooperate centrally inside and loosely across, keeping a compact dual memory, and offloading path-planning to a custom A-star chip — for a 10.2-times speedup at a slightly higher success rate.

Why this problem is worth solving

The systems being accelerated here are a fast-moving and genuinely useful class of AI.

Once the heavy planning is optimized, an unglamorous classical algorithm turns out to be the next thing in the way.

What people did before, and where it broke

• Fully decentralized cooperation (CoELA, COMBO). Every agent communicates peer-to-peer, negotiating actions through rounds of LLM-generated dialogue. Where it broke. As agents are added the chatter explodes — they repeat each other’s suggestions, dilute the shared context, and latency grows sharply, so the approach does not scale past a handful of agents.
• Fully centralized cooperation (MindAgent). A single central LLM plans for the whole team, avoiding all the peer-to-peer negotiation. Where it broke. One planner cannot handle the combinatorial explosion of coordinating many agents, so as the team grows its task success rate collapses.
• Route planning through a strong cloud LLM (GPT-4, Claude). Get the best decisions by calling the most capable available model over an API. Where it broke. API latency is unpredictable and it is the biggest time sink, the model cost limits scale, sending data off-device raises privacy concerns, and the network round-trip alone is 27 percent of total latency.
• Plan step-by-step on generic hardware. Ask the LLM for a fresh high-level plan before each low-level action, and run the resulting navigation on a stock CPU or GPU. Where it broke. It fires several LLM calls per step — and pre-generates messages that are almost never used, under 20 percent — while A-star navigation starves on hardware never built for it.
• The gap. Each fix addressed one layer and left the others untouched: an organization that scaled but still called a slow cloud model every step, or a faster model still trapped in a step-by-step pipeline on ill-suited hardware. Nobody had profiled the whole workflow end to end and then co-designed across algorithm, system, and hardware together, which is what it takes to make these agents actually run in real time.

How ReCA works

The idea

Fix the whole stack together, because no single layer is the whole problem. The distinctive move is not any one optimization but the decision to profile the entire workflow and then co-design across all of it. ReCA’s own measurements show why: the LLM dominates runtime, but memory, cooperation structure, execution scheduling, and low-level navigation each independently cap performance, and improving only one exposes the next. So ReCA changes the algorithm (a small local model), the system (dual memory, clustered cooperation, multi-step execution), and the hardware (a custom navigation accelerator) as one integrated design, each change clearing the way for the others. Treating performance as something won or lost across the full stack rather than in any single component is the same conviction the Trainium accelerator work covered elsewhere on this site is built on, which argues that fast large-model computation is a full-stack co-design problem spanning silicon, interconnect, and software rather than a matter of tuning one layer.

Keep a small, well-chosen working set of memory close, not the whole diluted history. A central inefficiency was memory: as a task drags on, cramming the entire dialogue and history into each prompt makes the model lose the thread and hallucinate, yet throwing memory away costs nearly double the steps. ReCA’s answer is a dual structure — a long-term graph holding the stable environment, which rarely changes, and a compact short-term buffer of recent state, refreshed by discarding the least recently used entries. The planner is handed exactly the facts that matter now, keeping context tight without forgetting the map. Managing a two-tier memory so that a small, high-value working set stays immediately available while the bulk is held in a cheaper, more permanent form is the same discipline the out-of-core random-walk work covered elsewhere on this site relies on, reserving a succinct set of what the computation needs next in scarce fast memory while the full data sits further away.

When the big cost falls, build specialized machinery for the bottleneck that surfaces. Optimizing the LLM does not end the story; it merely promotes A-star from a 5-to-15-percent cost to a 20-to-40-percent one, and A-star is precisely the kind of irregular, pointer-chasing, low-parallelism workload that a GPU handles badly. ReCA responds with a purpose-built A-star processing unit — hardware for the priority-queue heap, parallel units for the cost arithmetic, and a hash-indexed scratchpad that fetches neighboring cells by spatial locality — running the algorithm the way its structure actually wants. Recognizing that an irregular, memory-bound operation is starved on a general-purpose GPU and deserves tailored hardware is the same recognition the GPU graph-learning work covered elsewhere on this site acts on, building a framework fitted to an irregular, sampling-heavy workload that stock parallel hardware serves poorly.

Following the reasoning through ReCA

CoELA alone runs three separate LLM calls per step — message generation, planning, and action selection — with the cloud round-trip adding a quarter of total latency.

Why it works

• Running a small model locally removes both the biggest cost and the network delay at once. The cloud LLM planner dominated runtime and added unpredictable API and network latency → but a small open model fine-tuned and quantized to run on the agent itself does the planning in place → so the single largest time sink shrinks and the network round-trip, a quarter of total latency, disappears entirely.
• One high-level plan guiding many low-level actions amortizes the expensive planner. Calling the LLM before every single move put a costly inference on the critical path at each step → but letting one plan direct several consecutive actions, with a self-reflection check to catch drift, cuts the number of inferences sharply → so the per-action cost falls even though a few extra steps are taken.
• A compact dual memory feeds the planner what matters instead of the whole diluted history. Stuffing the entire growing dialogue into the prompt made the model lose track and hallucinate → but splitting memory into a long-term environment graph and a short-term recent-state buffer hands the planner just the relevant facts → so plans improve, and removing this memory was shown to cost 1.84 times more steps.
• Clustering agents captures centralized quality inside and decentralized scale across. A single planner for everyone collapses under many agents, while everyone talking to everyone explodes in chatter → but grouping nearby agents into clusters that plan centrally within and coordinate loosely between takes the strength of each → so twelve agents finish in under twenty minutes at ninety percent success where baselines took an hour and failed a third of the time.
• A custom accelerator relieves the path-planning that a GPU handles badly. A-star is sequential, heap-based, and branch-heavy, so a GPU starves on it, and after the LLM is sped up it grows to as much as forty percent of the time → but a dedicated unit with hardware heap management and parallel cost computation runs it the way its structure wants → so path planning speeds up 4.6 times at 281 times the energy efficiency, clearing the new bottleneck.

What it costs (trade-offs and scope)

• It is not evaluated on real robots. All results come from a desktop GPU plus an FPGA prototype running simulated embodied benchmarks — TDW, CuisineWorld, Minecraft, C-WAH — not physical robots in the world. Real sensing and actuation noise, which these simulations do not fully capture, would add challenges the paper does not test.
• The win is speed, not smarter plans. The 10.2-times speedup is the headline; task success rises only 4.3 percent, and ReCA actually takes 3.2 percent more steps. The plans are marginally worse per step — the gain is entirely in how fast each step runs.
• Most of the speedup comes from the LLM change, not the chip. The ablation shows the custom hardware trims runtime by 25.3 percent, while the larger levers are the local model and multi-step execution that cut the number and cost of LLM calls. The striking 4.6-times and 281-times A-star figures are for a slice that only became dominant after the LLM was already sped up.
• A small local model against cloud GPT-4 is not apples-to-apples. ReCA compares a carefully engineered Llama-3.1-8b pipeline — fine-tuned, chain-of-thought-prompted, quantized — against off-the-shelf cloud GPT-4 systems. Some of the advantage is engineering effort, and a smaller model carries an inherent ceiling on reasoning quality.
• Several knobs need per-workload tuning. The best number of multi-step actions and the best cluster size are found by pre-deployment sweeps rather than derived automatically, and a poorly chosen setting hurts both latency and success rate.
• The “communication barely matters” finding is model-dependent. Dropping most messages works largely because today’s LLMs communicate poorly and under 20 percent of generated messages were used. A future model that negotiates well could change that calculus and make the removed communication valuable again.
• The accelerator targets path planning only. A-star covers grid navigation; the rest of the low-level control stack, the perception and actuation pipeline, and any planning beyond shortest-path are not accelerated, and A-star’s optimality guarantee holds only under its usual assumptions.

The evaluation

Check yourself

Sources

The paper is “ReCA: Integrated Acceleration for Real-Time and Efficient Cooperative Embodied Autonomous Agents” by Zishen Wan, Yuhang Du, Mohamed Ibrahim, Jiayi Qian, Jason Jabbour, Yang (Katie) Zhao, Tushar Krishna, Arijit Raychowdhury, and Vijay Janapa Reddi, published at ASPLOS 2025, pages 981 to 996, DOI 10.1145/3676641.3716016. The authors are at the Georgia Institute of Technology, the University of Minnesota, Twin Cities, and Harvard University.

This explainer is written from the full 16-page text. Everything above comes from the paper’s own sections: the introduction and system characterization profiling CoELA, COMBO, and MindAgent to find per-step latencies of 19.9 to 21.4 seconds, LLM modules at 62.5 to 93.7 percent of runtime, network overhead at 27 percent, the scalability breakdown of decentralized versus centralized architectures, and the ablation showing communication is nearly dispensable, memory removal costs 1.84 times more steps, and execution removal causes failures; the system optimizations of efficient local model processing (LoRA fine-tuning, chain-of-thought and hierarchical prompting, MLC-LLM compilation, AWQ quantization, model-size sweeps), the dual long-term-graph and short-term-LRU-buffer memory, hierarchical intra-cluster-centralized and inter-cluster-decentralized cooperative planning with dynamic task decomposition, and planning-then-communication with planning-guided multi-step execution and a self-reflection check; the hardware architecture of an A-star subsystem and an LLM GPU subsystem interfaced through shared DRAM, with the A-star processing unit comprising a cost compute unit, a pipelined heap unit, and a hash-indexed dual-port scratchpad, and parent agents carrying both subsystems while child agents carry only the A-star one; and the evaluation on three systems and six benchmarks using Llama-3.1-8b, an A6000 GPU, an i7 CPU, and a ZC706 FPGA, reporting a 4.3 percent success-rate gain and 10.2-times speedup, 3.2 percent more steps, a user-study runtime from 3.3 down to 1.2 minutes, twelve-agent scaling to under 20 minutes at 90 percent success, the A-star unit’s 4.6-times speedup and 281-times energy-efficiency gain over a GPU with modest FPGA resource use, and the cumulative ablation attributing 25.3 percent runtime reduction to the hardware and further reductions to multi-step execution, hierarchical planning, and dual memory.