OPENTECHNICAL PUBLICATION
OT-2026-000001865

Controlling robots using multi-modal language models

Published: 2026-09-15T19:09:09.091488Z
Permanent URL: https://opentechnical.org/p/OT-2026-000001865
Categories: Robotics & Automation > Robot Architectures

------------------------------------------------------------------------

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a
computer storage medium, for controlling an agent interacting with an
environment. In one aspect, a method comprises: receiving one or more
observations of an environment; receiving an input text sequence that
describes a task to be performed by a robot in the environment; generating an
encoded representation of the input text sequence in an embedding space;
generating a corresponding encoded representation of each of the one or more
observations in the embedding space; generating a sequence of input tokens
that comprises the encoded representation of the input text sequence and the
corresponding encoded representation of each observation; processing the
sequence of input tokens using a language model neural network to generate an
output text sequence that comprises high-level natural language instructions;
and determining, from the high-level natural language instructions, one or
more actions to be performed by the robot.

------------------------------------------------------------------------

OpenTechnical publication OT-2026-000001865
Published 2026-09-15T19:09:09.091488Z
https://opentechnical.org/p/OT-2026-000001865