---
publication_id: "OT-2026-000001865"
title: "Controlling robots using multi-modal language models"
published_at: "2026-09-15T19:09:09.091488Z"
canonical_url: "https://opentechnical.org/p/OT-2026-000001865"
categories:
  - "robotics-automation/robot-architectures"
keywords:
  - "b25j9/1658"
  - "b25j9/1658"
  - "g06n3/045"
  - "g06n3/045"
  - "b25j9/16"
  - "b25j9/161"
---

# Controlling robots using multi-modal language models

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling an agent interacting with an environment. In one aspect, a method comprises: receiving one or more observations of an environment; receiving an input text sequence that describes a task to be performed by a robot in the environment; generating an encoded representation of the input text sequence in an embedding space; generating a corresponding encoded representation of each of the one or more observations in the embedding space; generating a sequence of input tokens that comprises the encoded representation of the input text sequence and the corresponding encoded representation of each observation; processing the sequence of input tokens using a language model neural network to generate an output text sequence that comprises high-level natural language instructions; and determining, from the high-level natural language instructions, one or more actions to be performed by the robot.