See every robot move before it happens.
Direct robots by speaking and pointing. MIRAI shows the AI's plan as a hologram in the real workspace first. Nothing moves until a person approves it.
Interactive simulation · not footage · pick a scenario or let them play
AI can read intent. Trust is the missing piece.
A plausible plan is not automatically a safe or correct one.
Programming industrial robots takes experts and setup time. Language models now let people simply say what they want, but the robot still has to get it right.
MIRAI pairs AI reasoning with a spatial interface, so a person can see, judge and stop what the robot intends to do, right where it happens.
From instruction to action, with a person in the loop
Four steps. The human decides at the third.
Instruct
Speak, point or type. “Put it there” actually means something.
Preview
The plan appears as a hologram: trajectory, target and how the request was understood.
Decide
Approve it, or reject it with a correction and get a revised plan.
Execute
Only approved plans reach the robot. You can still intervene at any point.
Anywhere a person should keep the final say
Wherever a robot acts on AI-generated plans, the same approval layer applies.
Pick, pack and sort
Direct picks by voice, or approve a whole sorting plan at once.
Working next to people
Stops that wait for an explicit resume, handovers that release only on a confirmed grip.
Rehearse in the twin
Dry-run a plan on a digital twin of the cell before it reaches the hardware.
HRI studies and teaching
A working platform for XR-based human–robot interaction research.
MIRAI is a research platform, not a certified safety system. Machine safety remains the responsibility of the integrator.
Three layers, one loop
Interaction, interpretation and execution, with the operator in the loop throughout.
- XR User InterfaceSpeech, gesture and text in, on a headset, an Android device or in the browser; live robot and object state shown back.
- MIRAI ServerInterprets instructions against the scene, checks them and schedules robot actions.
- Local Edge ComputerPerception, local safety analysis and execution, right next to the robot.
- Unity
- ROS
- LLMs
- Computer vision
- HoloLens 2
- Android
- WebGL
- Edge computing
A working proof of concept, end to end
Under active development. This is how it got here, in real recordings.
-
01 · Prototype
First prototype in Unity
AI-planned actions become a holographic preview instead of running straight away. Shown in the Unity editor.
-
02 · Deployment
Running on HoloLens 2
The full system runs end to end on a HoloLens 2, and now also on Android and as a simplified build in the browser. The operator inspects the plan, then approves or rejects it with a correction.
-
03 · Orchestration
Server and supervision interface
One interface for tasks, scene context, AI provider, execution state and approvals, from spoken instruction to robot program.
Published at APMS 2026
The architecture is published in the APMS 2026 proceedings (IFIP, Springer).
"MIRAI: Introducing a Human-in-Command Architecture for Multimodal AI-driven Human-Robot-Interaction"
doi.org/10.1007/978-3-032-38618-2_2 ↗Related research
Beyond the single arm
Simulator-ready Exploring
Isaac Sim, MuJoCo, Gazebo or Unity over MQTT, to rehearse before real hardware.
Autonomous and humanoid robots Exploring
The same approval layer for agentic robots and humanoids.
What people usually ask
Ask me directly about MIRAI. You'll usually hear back within two working days.
Ask a questionDo I need a HoloLens 2 to use MIRAI?
No. MIRAI runs on a HoloLens 2, on Android devices, and as a simplified build in the browser.
Which robots and AI models does it work with?
Robots connect through ROS, and the AI provider is configurable on the server.
What stops the AI from doing something unsafe?
A person approves every plan, the operator can intervene at any time, and local safety analysis runs next to the robot.
Is MIRAI a finished product?
No, it is a working proof of concept. Pilots are scoped around a concrete task and robot, or a simulation of one.
Want a human-in-command pilot on your robot?
Describe the task. You'll get a straight answer on whether MIRAI fits.