OpenTechnical
This is a record of a document published elsewhere, first published by the United States Patent and Trademark Office, as US-12576523-B2, on 17 March 2026. OpenTechnical did not first disclose it. The identifier, hashes and timestamps below record what this library served and when, not when the material became public. View the original.

OT-2026-000001865

Controlling robots using multi-modal language models

Published
Permanent URL
Classification
Robotics & Automation › Robot Architectures
Inventors
DRIESS DANNY MICHAEL, FLORENCE PETER RAYMOND, GREFF KLAUS, MORDATCH IGOR, SAJJADI SEYED MOHAMMAD MEHDI, ZENG ANDY
Assignee
GOOGLE LLC
Original source
US-12576523-B2, the United States Patent and Trademark Office, 17 March 2026
Document SHA-256
f744581fa844757eabd0e8876dd8350cc9bc81194e4052010434c16a459cbe54

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling an agent interacting with an environment. In one aspect, a method comprises: receiving one or more observations of an environment; receiving an input text sequence that describes a task to be performed by a robot in the environment; generating an encoded representation of the input text sequence in an embedding space; generating a corresponding encoded representation of each of the one or more observations in the embedding space; generating a sequence of input tokens that comprises the encoded representation of the input text sequence and the corresponding encoded representation of each observation; processing the sequence of input tokens using a language model neural network to generate an output text sequence that comprises high-level natural language instructions; and determining, from the high-level natural language instructions, one or more actions to be performed by the robot.

Publication information

Plain textMarkdownManifestIntegrity record

Citation

OpenTechnical Publication OT-2026-000001865, “Controlling robots using multi-modal language models,” published September 15, 2026 at 19:09:09 UTC, https://opentechnical.org/p/OT-2026-000001865.

To cite specific content, add the paragraph or figure numbers.

Terms

This record reproduces a document published elsewhere and is provided for reference and study. Rights in the original remain with its owner. See the publication policy.