Abstract
One embodiment of a method for processing data includes receiving an image; segmenting, using a first trained machine learning model, the image to generate a segmentation mask; generating one or more descriptions of one or more objects using a second machine learning model and based on the segmentation mask; generating program code using a third trained machine learning model and based on the one or more descriptions, the image, and a task; and causing a robot to move based on the program code.