Abstract
Implementations for robot control are provided. A method involves, based on vision data depicting an environment of a robot and a natural language instruction for the robot, determining an affordance plan for performing a task. The affordance plan comprises a sequence of intermediate representations of the robot in visual space, such as end effector poses. An action input prompt is assembled with data indicative of the vision data, the natural language instruction, and the affordance plan. The action input prompt is processed using one or more generative models to generate action output indicative of one or more actions to be performed by the robot. Subsequently, a robot control signal is generated based on the one or more actions. This provides a spatially precise and dimensionally concise form of guidance for robot manipulation tasks, which can improve performance and generalization.