Can Agents Manipulate?

Robots

• Simulation: 7-DoF xArm7 in MuJoCo [1] with a wrist-mounted RGB-D camera.
• Hardware: 6-DoF Piper for the head-to-head comparison; 7-DoF xArm7 for skill transfer and teleoperation (Secs. 06 and 07).

Agent and interfaces

• Agent: pi coding agent, limited to 150 steps per run.
• Models: GPT-6 Astra, Sol, Fable 5.1, Opus 5.5 and Argon.
• Interfaces: dimOS Python API or raw robot interface; only the interface differs between setups (Fig. 2). Without dimOS, the agent may install any library it needs.
• Hardware baseline: GPT-6 Astra operating end-to-end, estimating object poses directly from images.

Figure 2: What each agent receives (in) and what it can call (out). The harness, models and robot are shared, and both agents have the same harness tools (inside the agent box): read, bash, edit and write, so each can grep, run Python and write its own code through bash. Without dimOS, the agent reads wrist RGB and depth images, camera pose and arm and gripper state as raw topics, and acts only through end-effector (TCP) velocity and gripper commands, building its own perception and motion. With dimOS, everything goes through the dimOS Python API described in DIMOS.md. It reads the same camera and arm data, and adds dimOS skills such as perception (localize, grasps) and planned arm moves that report when they fail.

Tasks and grading

Table 1: Tasks. The prompt as given to the agent, the time limit and the complexity from Eq. (1); each task's score is defined in Sec. 04.

Hardware

Figure 3: First success on hardware, one run per setup on a 6-DoF Piper; the baseline is GPT-6 Astra operating end-to-end. Videos are shown at 60×.

• dimOS reduces time and cost. With dimOS, agents score better on every task and need less cost, time and turns per point of score; the dimOS factor stays above 1 on every task, including the most complex (Fig. 4).
• Failures occur at the last step. Agents miss steps that require interpreting the scene, such as a turned letter or a cap left on the bottle; dimOS narrows this gap (71% vs. 58% of Spell DIMOS runs pass; Fig. 8).
• Fewer turns is the main lever. Model latency dominates run time (Fig. 6), and dimOS cuts turns most on the longer tasks (42.5 vs. 60 on open bottle).
• The gap widens on hardware. On the Piper, dimOS reached first success 1.7–4.2× faster with 1.5–5.4× fewer tokens (Fig. 3), an early signal from one run per setup.
• The benefit varies by agent. It peaks where an agent struggles most without dimOS, as Opus 5.5 does on open bottle, and stays near 1 for Argon on Spell DIMOS (1.12×). Argon uses dimOS differently: with it, Argon takes more turns on Spell DIMOS (42 vs. 36.5) at about the same time and cost, so its effort barely drops.

Limitations and future work

• A physical harness. We plan to replace pi with a Dimensional harness built for physical tasks, one that helps agents perceive the scene, adapt to it and act on any robot.
• Time to full skill distillation. Rather than first success, we will track time and effort until a task is fully distilled into a reusable skill, where we expect the difference between interfaces to be more pronounced.
• Long-horizon tasks. Beyond tabletop tasks, we will extend the study to long-horizon manipulation, which the authors' prior work [13] identified as a weakness of generalist policies, and investigate how autoresearch combined with this harness can address it.

Dan B.W. Choe, Ruthwik Dasyam and Sundhar Vinodh Sangeetha. Can Agents Manipulate? Dimensional Research technical report, 2026.

BibTeX@misc{dimensionalcanagentsmanipulate,
  author       = {Choe, Dan B.W. and Dasyam, Ruthwik and Sangeetha, Sundhar Vinodh},
  title        = {{Can Agents Manipulate?}},
  howpublished = {Online research report, Dimensional Research},
  note         = {Technical report},
  year         = {2026}
}

References

1. Todorov, E., Erez, T., Tassa, Y. “MuJoCo: A physics engine for model-based control.” IROS 2012.
2. Liang, J. et al. “Code as Policies: Language Model Programs for Embodied Control.” ICRA 2023. arxiv.org/abs/2209.07753
3. Dimensional Research. “Can Jev Nav?” 2026. research.dimensional.org/system-one-navigation
4. dimOS source code. github.com/dimensionalOS/dimos · Documentation. docs.dimensional.org
5. Fitts, P. M. “The information capacity of the human motor system in controlling the amplitude of movement.” Journal of Experimental Psychology 47(6), 1954.
6. Zhao, T. Z., Kumar, V., Levine, S., Finn, C. “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware.” RSS 2023. arxiv.org/abs/2304.13705
7. Cadene, R. et al. “LeRobot: State-of-the-art Machine Learning for Real-World Robotics in PyTorch.” 2024. github.com/huggingface/lerobot
8. Hu, J., Martín-Martín, R., Stone, P., Duan, R., Jiang, Z., Shi, G. “SimEX: Simulation-Integrated Robotics AutoResearch.” 2026. arxiv.org/abs/2609.38982
9. Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., Martín-Martín, R. “What Matters in Learning from Offline Human Demonstrations for Robot Manipulation.” CoRL 2021. arxiv.org/abs/2108.03298
10. Anderson, P., Chang, A., Chaplot, D. S., Dosovitskiy, A., Gupta, S., Koltun, V., Kosecka, J., Malik, J., Mottaghi, R., Savva, M., Zamir, A. R. “On Evaluation of Embodied Navigation Agents.” 2018. arxiv.org/abs/1807.06757
11. Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., Press, O. “SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering.” NeurIPS 2024. arxiv.org/abs/2405.15793
12. Fleming, P. J., Wallace, J. J. “How not to lie with statistics: the correct way to summarize benchmark results.” Communications of the ACM 29(3), 1986. doi.org/10.1145/5666.5673
13. Choe, D. B. W., Sangeetha, S. V., Coogan, S., Kousik, S. “Make Your VLA More Robust Without More Data By Interleaving Motion Planning.” 2026. arxiv.org/abs/2606.00985