Abstract
Various aspects relate to a method of generating a dynamics model for predicting a surrounding of a robotic device resulting from an interaction of the robotic device, the method including: receiving a pre-trained diffusion model configured to generate an output image responsive to inputting an input image and a text prompt that indicates how to modify the input image; receiving a plurality of training tuples, each including a control action, a first image representing a surrounding of the robotic device prior to the robotic device executing the control action, and a second image representing the surrounding of the robotic device after execution of the control action; generating the dynamics model by training the pre-trained diffusion model for each training tuple to generate the second image of the training tuple as output image responsive to inputting the first image as input image and the control action as text prompt.