Abstract
Generating output that is based on a sequence of actions performed by an entity and related to object(s) in an environment—and generating the output using both vision data that visually captures the entity performing the sequence of actions and event data that captures non-visually detected events that occurred in the environment during performance of the sequence of actions. Implementations utilize chain-of-modality techniques to process multiple modalities of data, each capturing corresponding aspects of the entity performing the sequence of actions, to generate output that reflects the sequence of actions. As opposed to incorporating all of multiple modalities of data in a single prompt, implementations of the chain-of-modality techniques generate and process multiple prompts in sequence, where each prompt includes only a subset of the multiple modalities of data and, when preceded by prior processing of a prior prompt, at least some of the output from the prior processing.