publications
2026
- What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied AgentsJunxiang Wang, Yuchen Liu, Yi Wu, Guofei Chen, Nir Pechuk, Tao Chen, Zackory Erickson, Yuzhe Qin, and Tianyu LiarXiv e-prints, 2026
Memory allows large language model (LLM) agents to use past experience, often stored as textual records, when responding to requests. For embodied agents that control robots, visual and spatial observations introduce a broader choice of memory modalities, affecting both the information available for subsequent tasks and the model inference costs of constructing and using memory. We introduce MeMoREval, a benchmark for studying this problem through a controlled experience-request protocol. Agents first experience a sequence of events without knowing the subsequent tasks, and subsequently they use memory to respond to information and action requests without receiving new observations. The benchmark comprises 53 tasks across four simulated indoor environments and measures task success alongside the inference costs of memory construction and use. Our reference evaluations reveal that storing frames outperforms storing generated descriptions at lower write-time inference cost for the primary model, while storing frames with scene graphs achieves the highest aggregate task success for every tested model. Extensions to accumulated experience, selective frame retention, and visual input to memory writers demonstrate how the benchmark accommodates further memory-design investigations. Two physical-robot demonstrations establish the feasibility of applying its protocol beyond simulation.
@article{wang2026memoreval, title = {What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents}, author = {Wang, Junxiang and Liu, Yuchen and Wu, Yi and Chen, Guofei and Pechuk, Nir and Chen, Tao and Erickson, Zackory and Qin, Yuzhe and Li, Tianyu}, journal = {arXiv e-prints}, year = {2026}, eprint = {2610.12561}, archiveprefix = {arXiv}, primaryclass = {cs.RO}, url = {https://arxiv.org/abs/2610.12561}, } - GenPHRI: Agentic Generative Simulation for Physical Human-Robot InteractionJunxiang Wang*, Nir Pechuk*, Xinwen Xu*, Nayon Kim, Tiancheng Wu, Julian Millan, and Zackory EricksonarXiv e-prints, 2026
Real-world physical human-robot interaction (pHRI) development is costly and slow, while simulation requires substantial task-specific scene and motion authoring. We present GenPHRI, a generative simulation framework for producing deployment-ready pHRI scenarios entirely from natural-language task descriptions using vision-language model (VLM) agents. Generated scenarios contain a room with furniture, a task-appropriate human pose and placement, and a robot executing the requested interaction. Generator and critic agents iterate on each component, and an orchestrator agent controls cross-stage decisions. With GenPHRI, we generate 50 different assistive scenarios in simulation with no manual intervention and deploy two tasks in a 12-participant user study. Vision policies trained on these scenarios achieve zero-shot task completion rates of 80% in real-world trials against 90% in simulation, while producing behaviors that participants rate as consistent with textual task prompts. Further, we demonstrate direct execution of generated motion as a complementary, training-free deployment pathway that enables rapid real-world inspection of generated trajectories at the cost of contact coverage. We release GenPHRI as a platform for generating, training, and deploying pHRI tasks from textual prompts.
@article{wang2026genphri, title = {Gen{PHRI}: Agentic Generative Simulation for Physical Human-Robot Interaction}, author = {Wang, Junxiang and Pechuk, Nir and Xu, Xinwen and Kim, Nayon and Wu, Tiancheng and Millan, Julian and Erickson, Zackory}, journal = {arXiv e-prints}, year = {2026}, eprint = {2604.08664}, archiveprefix = {arXiv}, primaryclass = {cs.RO}, url = {https://arxiv.org/abs/2604.08664}, } - Bidirectional Human-Robot Communication for Physical Human-Robot InteractionJunxiang Wang, Cindy Wang, Rana Soltani Zarrin, and Zackory EricksonIn Proceedings of the 21st ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2026
1 of 3 papers nominated for the Best Paper Award in the Systems track
Effective physical human-robot interaction requires systems that are not only adaptable to user preferences but also transparent about their actions. This paper introduces BRIDGE, a system for bidirectional human-robot communication in physical assistance. Our method allows users to modify a robot’s planned trajectory—position, velocity, and force—in real time using natural language. We utilize a large language model (LLM) to interpret any trajectory modifications implied by user commands in the context of the planned motion and conversation history. Importantly, our system provides verbal feedback in response to the user, either assuring any resulting changes or posing a clarifying question. We evaluated our method in a user study with 18 older adults across three assistive tasks, comparing BRIDGE to an ablation without verbal feedback and a baseline. Results show that participants successfully used the system to modify trajectories in real time. Moreover, the bidirectional feedback led to significantly higher ratings of interactivity and transparency, demonstrating that the robot’s verbal response is critical for a more intuitive user experience.
@inproceedings{wang2026bidirectional, title = {Bidirectional Human-Robot Communication for Physical Human-Robot Interaction}, author = {Wang, Junxiang and Wang, Cindy and Zarrin, Rana Soltani and Erickson, Zackory}, booktitle = {Proceedings of the 21st ACM/IEEE International Conference on Human-Robot Interaction (HRI)}, pages = {572-580}, numpages = {9}, year = {2026}, publisher = {Association for Computing Machinery}, doi = {10.1145/3757279.3785634}, } - Situated Understanding of Older Adults’ Interactions with Voice Assistants: A Month-long In-home StudyAmama Mahmood, Junxiang Wang, and Chien-Ming HuangACM Transactions on Accessible Computing, 2026
Our work addresses the challenges older adults face with commercial Voice Assistants (VAs), notably in conversation breakdowns and error handling. Traditional methods of collecting user experiences-usage logs and post-hoc interviews-do not fully capture the intricacies of older adults’ interactions with VAs, particularly regarding their reactions to errors. To bridge this gap, we equipped 15 older adults’ homes with smart speakers integrated with custom audio recorders to collect "in-the-wild" audio interaction data for detailed error analysis. Recognizing the conversational limitations of current VAs, our study also explored the capabilities of Large Language Models (LLMs) to handle natural and imperfect text for improving VAs. Midway through our study, we deployed ChatGPT-powered VA to investigate its efficacy for older adults. Our research suggests leveraging vocal and verbal responses combined with LLMs’ contextual capabilities for enhanced error prevention and management in VAs, while proposing design considerations to align VA capabilities with older adults’ expectations.
@article{mahmood2026situated, title = {Situated Understanding of Older Adults' Interactions with Voice Assistants: A Month-long In-home Study}, author = {Mahmood, Amama and Wang, Junxiang and Huang, Chien-Ming}, journal = {ACM Transactions on Accessible Computing}, year = {2026}, publisher = {Association for Computing Machinery}, doi = {10.1145/3796236}, }
2025
- CoRI: Communication of Robot Intent for Physical Human-Robot InteractionJunxiang Wang, Emek Barış Küçüktabak, Rana Soltani Zarrin, and Zackory EricksonIn 9th Annual Conference on Robot Learning, 2025
Clear communication of robot intent fosters transparency and interpretability in physical human-robot interaction (pHRI), particularly during assistive tasks involving direct human-robot contact. We introduce CoRI, a pipeline that automatically generates natural language communication of a robot’s upcoming actions directly from its motion plan and visual perception. Our pipeline first processes the robot’s image view to identify human poses and key environmental features. It then encodes the planned 3D spatial trajectory (including velocity and force) onto this view, visually grounding the path and its dynamics. CoRI queries a vision-language model with this visual representation to interpret the planned action within the visual context before generating concise, user-directed statements, without relying on task-specific information. Results from a user study involving robot-assisted feeding, bathing, and shaving tasks across two different robots indicate that CoRI leads to statistically significant difference in communication clarity compared to a baseline communication strategy. Specifically, CoRI effectively conveys not only the robot’s high-level intentions but also crucial details about its motion and any collaborative user action needed.
@inproceedings{wang2025cori, title = {Co{RI}: Communication of Robot Intent for Physical Human-Robot Interaction}, author = {Wang, Junxiang and K{\"u}{\c{c}}{\"u}ktabak, Emek Bar{\i}{\c{s}} and Zarrin, Rana Soltani and Erickson, Zackory}, booktitle = {9th Annual Conference on Robot Learning}, year = {2025}, url = {https://openreview.net/forum?id=dBaSaa7qi4}, } - A Digital Twin for Telesurgery under Intermittent CommunicationJunxiang Wang*, Juan Antonio Barragan*, Hisashi Ishida*, Jingkai Guo, Yu-Chun Ku, and Peter KazanzidesIn 2025 International Symposium on Medical Robotics (ISMR), 2025
Telesurgery is an effective way to deliver service from expert surgeons to areas without immediate access to specialized resources. However, many of these areas, such as rural districts or battlefields, might be subject to different problems in communication, especially latency and intermittent periods of communication outage. This challenge motivates the use of a digital twin for the surgical system, where a simulation would mirror the robot hardware and surgical environment in the real world. The surgeon would then be able to interact with the digital twin during communication outage, followed by a recovery strategy on the real robot upon reestablishing communication. This paper builds the digital twin for the da Vinci surgical robot, with a buffering and replay strategy that reduces the mean task completion time by 23 % when compared to the baseline, for a peg transfer task subject to intermittent communication outage.
@inproceedings{wang2025digital, title = {A Digital Twin for Telesurgery under Intermittent Communication}, author = {Wang, Junxiang and Barragan, Juan Antonio and Ishida, Hisashi and Guo, Jingkai and Ku, Yu-Chun and Kazanzides, Peter}, booktitle = {2025 International Symposium on Medical Robotics (ISMR)}, pages = {218--224}, year = {2025}, organization = {IEEE}, doi = {10.1109/ISMR67322.2025.11025988}, } - User Interaction Patterns and Breakdowns in Conversing with LLM-Powered Voice AssistantsInternational Journal of Human-Computer Studies, 2025
Conventional Voice Assistants (VAs) rely on traditional language models to discern user intent and respond to their queries, leading to interactions that often lack a broader contextual understanding, an area in which Large Language Models (LLMs) excel. However, current LLMs are largely designed for text-based interactions, thus making it unclear how user interactions will evolve if their modality is changed to voice. In this work, we investigate whether LLMs can enrich VA interactions via an exploratory study with participants (N=20) using a ChatGPT-powered VA for three scenarios (medical self-diagnosis, creative planning, and discussion) with varied constraints, stakes, and objectivity. We observe that LLM-powered VA elicits richer interaction patterns that vary across tasks, showing its versatility. Notably, LLMs absorb the majority of VA intent recognition failures. We additionally discuss the potential of harnessing LLMs for more resilient and fluid user-VA interactions and provide design guidelines for tailoring LLMs for voice assistance.
@article{mahmood2025user, title = {User Interaction Patterns and Breakdowns in Conversing with {LLM}-Powered Voice Assistants}, author = {Mahmood, Amama and Wang, Junxiang and Yao, Bingsheng and Wang, Dakuo and Huang, Chien-Ming}, journal = {International Journal of Human-Computer Studies}, volume = {195}, pages = {103406}, year = {2025}, publisher = {Elsevier}, doi = {10.1016/j.ijhcs.2024.103406}, }
2023
- Method for Robotic Motion Compensation during PET Imaging of Mobile SubjectsJunxiang Wang, Iulian Iordachita, and Peter KazanzidesIn 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023
Studies of the human brain during natural activities, such as locomotion, would benefit from the ability to image deep brain structures during these activities. While Positron Emission Tomography (PET) can image these structures, the bulk and weight of current scanners are not compatible with the desire for a wearable device. This has motivated the design of a robotic system to support a PET imaging system around the subject’s head and to move the system to accommodate natural motion. We report here the design and experimental evaluation of a prototype robotic system that senses motion of a subject’s head, using parallel string encoders connected between the robot-supported imaging ring and a helmet worn by the subject. This measurement is used to robotically move the imaging ring (coarse motion correction) and to compensate for residual motion during image reconstruction (fine motion correction). Minimization of latency and measurement error are the key design goals, respectively, for coarse and fine motion correction. The system is evaluated using recorded human head motions during locomotion, with a mock imaging system consisting of lasers and cameras, and is shown to provide an overall system latency of about 80 ms, which is sufficient for coarse motion correction and collision avoidance, as well as a measurement accuracy of about 0.5mm for fine motion correction.
@inproceedings{wang2023method, title = {Method for Robotic Motion Compensation during {PET} Imaging of Mobile Subjects}, author = {Wang, Junxiang and Iordachita, Iulian and Kazanzides, Peter}, booktitle = {2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, pages = {4648--4654}, year = {2023}, organization = {IEEE}, doi = {10.1109/IROS55552.2023.10341444}, } - Calibration and Evaluation of a Motion Measurement System for PET Imaging StudiesJunxiang Wang, Ti Wu, Iulian Iordachita, and Peter KazanzidesJournal of Medical Robotics Research, 2023
Positron Emission Tomography (PET) enables functional imaging of deep brain structures, but the bulk and weight of current systems preclude their use during many natural human activities, such as locomotion. The proposed long-term solution is to construct a robotic system that can support an imaging system surrounding the subject’s head, and then move the system to accommodate natural motion. This requires a system to measure the motion of the head with respect to the imaging ring, for use by both the robotic system and the image reconstruction software. We report here the design, calibration, and experimental evaluation of a parallel string encoder mechanism for sensing this motion. Our results indicate that with kinematic calibration, the measurement system can achieve accuracy within 0.5mm, especially for small motions.
@article{wang2023calibration, title = {Calibration and Evaluation of a Motion Measurement System for {PET} Imaging Studies}, author = {Wang, Junxiang and Wu, Ti and Iordachita, Iulian and Kazanzides, Peter}, journal = {Journal of Medical Robotics Research}, volume = {8}, number = {01n02}, pages = {2340003}, year = {2023}, publisher = {World Scientific}, doi = {10.1142/S2424905X23400032}, }
2022
- Evaluation of a Motion Measurement System for PET Imaging StudiesJunxiang Wang, Ti Wu, Iulian Iordachita, and Peter KazanzidesIn 2022 International Symposium on Medical Robotics (ISMR), 2022
Awarded to the best paper with a student first-author (4%)
Positron Emission Tomography (PET) enables functional imaging of deep brain structures, but the bulk and weight of current systems preclude their use during many natural human activities, such as locomotion. The proposed long-term solution is to construct a robotic system that can support an imaging system surrounding the subject’s head, and then move the system to accommodate natural motion. This requires a system to measure the motion of the head with respect to the imaging ring, for use by both the robotic system and the image reconstruction software. We report here the design and experimental evaluation of a parallel string encoder mechanism for sensing this motion. Our preliminary results indicate that the measurement system may achieve accuracy within 0.5mm, especially for small motions, with improved accuracy possible through kinematic calibration.
@inproceedings{wang2022evaluation, title = {Evaluation of a Motion Measurement System for {PET} Imaging Studies}, author = {Wang, Junxiang and Wu, Ti and Iordachita, Iulian and Kazanzides, Peter}, booktitle = {2022 International Symposium on Medical Robotics (ISMR)}, pages = {1--6}, year = {2022}, organization = {IEEE}, doi = {10.1109/ISMR48347.2022.9807554}, }