What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents

Why the form of memory matters

A coding agent that forgets a fact can inspect the codebase again or rewind the codebase to an earlier state; the lapse in memory costs some computation but is largely invisible to the user. On the other hand, a robot that forgets where it saw an object may have to navigate between rooms, open containers or move objects to find it again, and this information may no longer be retrievable if the environment has changed. Therefore, for an embodied agent, memory is crucial, the form in which it is stored is equally important. A textual record might note several mugs on a counter but omit the colors needed to retrieve a particular one. A stored camera frame is lossless but can take up space to store, whereas generating descriptions of a camera frame requires choosing what details to retain before the task is known, and hence could miss the task-relevant facts.

What we find

MeMoREval compares different memory modalities for a realistic embodied agent under a controlled protocol, measuring task success together with the inference cost of writing memory during experience and reading it to answer later requests. Through experiments with MeMoREval, we found that:

1. The agent solves 70% of tasks when camera frames are stored, against 60% with generated descriptions, and frames cost less tokens than descriptions not only at write-time but also at read-time (frames vs. descriptions). Keeping less information actually leads to higher read-time cost with more reasoning tokens when missing task-relevant information (read-time effort).
2. Storing frames with scene graphs gives the highest task success for every model we tested (across models).
3. Design choices matter more under deployment-oriented scenarios: different memory backends behave differently under longer histories, and designing memory writers that can choose which frames to keep is non-trivial (deployment variants).

The benchmark

We define each task as two phases. In the experience phase, the robot carries out a scripted sequence of actions, such as a tour of the environment or moving an object, and may hear a user mention a preference, while a memory module records the robot’s observations without knowing what will be asked. In the request phase, the agent receives two types of requests in a fresh context: information requests (“Which cabinet contains the mug?”) and action requests (“Fetch the mug”), and the agent must respond from memory alone. Action requests are responded to with a plan, which is a complete sequence of skill calls to be executed without returning new observations, so the agent cannot make up for missing memory by searching the scene again.

We separate the memory modality, the form in which evidence is retained, from the memory backend, which organizes that evidence and makes it available. We design six memory conditions with different modalities. The text condition keeps the textual observations alone. In desc@64, desc@256 and desc@1024, a describer generates a description of each camera frame at that token budget and stores that description. The frames condition stores the annotated camera frames directly, and frames+graph also stores scene-graph snapshots. Orthogonally, we test five different memory backends. A rolling transcript directly logs the observations verbatim. In a wiki of named file-system pages and in our adaptations of A-MEM, Mem0 and ObsMem, a memory writer turns those observations into organized memory records.

MeMoREval instantiates the protocol in 53 tasks across four simulated indoor environments and seven categories of remembered information, with a simulated Dexmate Vega 1 Pro bimanual mobile manipulator acting through navigation, opening and closing, and pick-and-place skills. See task categories, example tasks, and more below.

Results

Frames are more effective and more efficient than descriptions

Generated descriptions achieves task success rates of 56%, 59% and 60% as the description budget grows from 64 to 1024 tokens. Storing the frames themselves reaches 70%, and adding scene graphs 75%. Descriptions and frames are archived and retrieved the same way, so the gap only reflects what each modality stores and how well the agent uses it, not what the writer records. The limited gain from a larger description budget suggests that length alone does not solve the problem of keeping details before their relevance is known. Generating a description commits to an interpretation while memory is being built, whereas a stored frame leaves its visual details to be interpreted after the request arrives.

Cost-wise, storing camera frames directly also has an advantage. At write time, a description costs an extra model call at every experience step, while a frame is archived directly, so frames require less model inference ($9.30 per thousand steps against $11.08–$12.09 for descriptions). At read time, frames and desc@1024 cost about the same per attempted request, but frames succeed more often, so they need 14% fewer weighted tokens per successful information request and 7% fewer per successful action request.

From this cost analysis, we could gain another useful insight, that less retained memory actually leads to higher reasoning at read time. Per request, agents with only text records make the fewest tool calls (1.0; a tool call is a retrieval, an answer or a plan) but spend the most reasoning tokens (4.2k). Longer descriptions lower both tool calls and reasoning tokens, and frames use fewer tool calls and reasoning tokens than any description condition. This progression from text to descriptions to frames suggests that incomplete evidence leads to extra deliberation without recovering the missing information, and preserving visual evidence retains the most information and reduces the deliberation effort. Adding scene graphs raises tool calls from 2.1 to 2.5 while reasoning stays at 3.1k tokens, consistent with the extra read cost going to retrieving and processing the additional modality.

Where frames and scene graphs help

Frames improve mean success over desc@1024 in six of the seven categories, with the 95% intervals excluding zero for appearance binding (+18.0 points) and object location (+12.7). The advantage on appearance binding fits the account above: identifying a particular object can depend on visual attributes that a description leaves out. Adding scene graphs raises overall success by 5.2 points, with the 95% intervals excluding zero for spatial configuration (+9.7) and user preference (+8.0). Scene graphs make object positions and relations explicit, which can spare the agent from reconstructing spatial layout from camera views. Neither addition shows a clear gain on set aggregation or occluded state, so the benefit of an additional modality depends on the kind of information a task needs.

Across models, frames with scene graphs do best

We repeated the six conditions with GPT 5.6 Sol and GLM 5.3 Flash on one seed and three backends (transcript, wiki and A-MEM). Storing frames with scene graphs gives the highest success for all three models. Whether frames alone beat desc@1024 depends on the model: they do for Gemini 3.7 Flash (70% vs. 62%) and GPT 5.6 Sol (55% vs. 39%), but not for GLM 5.3 Flash (41% vs. 45%). Text, frames and scene graphs can encode the same fact in different forms, and the differing rankings suggest that models vary in how well they use each one. Combining them offers alternative routes to shared facts as well as complementary information, which may explain why the combined condition does best for every model. The models also differ in how many tokens they spend retrieving and using this evidence, which gives each its own trade-off between success and read cost.

Deployment choices change how memory configurations compare

The controlled comparison above uses short single tasks, a writer that receives only textual observations, and automatic storage of every frame. A deployed robot can introduce practical considerations beyond the controlled environment, and our benchmark tests such variants too. For longer histories, the experience phases of 10–11 tasks from one environment are stacked into one history, followed by all of their requests. The transcript degrades substantially under stacking: its rolling history exceeds the shared memory-context budget, so earlier observations and their retrieval cues are truncated from the agent’s context. The other backends respond in different ways, suggesting that memory organization and retrieval affect how accessible useful evidence remains as histories grow.

Archiving every frame makes storage grow with experience, so a deployed writer may have to decide which frames to save strategically. The retention rate ranges from 8–55% among the tested backends, and mean task success is lower than with every frame archived, suggesting that how to perform retention is still an open question.

Another reasonable extension is giving the writer access to the camera images, since writers might be able to synthesize data more effectively with multimodal data at hand. Our experiments show that this is beneficial only in some cases. In terms of task success, it does not always improve performance, and in some cases, it can even lead to a decrease in performance. In terms of the cost of writing memories, it increases the cost for some backends and decreases it for others. Overall, our results indicate that there are benefits to be gained from providing writers with access to camera images, but the extent to which these benefits are realized depends on the specific context and backend implementation.

On the real robot

We also deployed our evaluation protocol on a physical Dexmate Vega 1 Pro. A SLAM map with manually annotated objects and relations, including support, provides the scene-graph records, and the head camera provides visual observations, segmented with SAM 3. We ran two demonstrations of memory-dependent tasks, both with the frames+graph condition on the wiki backend. In the first, shown in the walkthrough above, the robot throws away a soda can and later fetches the fruit that had been beside it. In the second, below, it moves a lemon to a blue plate, later recalls the color of the lemon’s original plate, and fetches the fruit that had been to the lemon’s right. The robot completed both deliveries without human assistance.

Evaluate your own memory system

MeMoREval lets you try your own embodied agent memory system, or even your own agent altogether! In addition to the controlled environment, we also provide several deployment-oriented settings as explained above. We believe that this platform will be useful for much more than what we demonstrated, and we'd love to see your memory system and use case!

@misc{wang2026memoreval,
  title  = {What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents},
  author = {Wang, Junxiang and Liu, Yuchen and Wu, Yi and Chen, Guofei and Pechuk, Nir and Chen, Tao and Erickson, Zackory and Qin, Yuzhe and Li, Tianyu},
  year   = {2026},
  eprint = {2610.12561},
  archivePrefix = {arXiv},
  note   = {Dexmate technical report},
  url    = {https://arxiv.org/abs/2610.12561}
}

Correspondence: junxiang@cmu.edu, ty.li@dexmate.ai