Abstract
A method includes receiving a mapping video of a scene; generating a map of the scene based on the mapping video; receiving a plurality of demonstration videos of a hand-held manipulation device performing one or more tasks in the scene; for each video among the plurality of demonstration videos, determining whether the hand-held manipulation device can be localized in the scene based on the video and the map; and for each video among the plurality of demonstration videos for which the hand-held manipulation device can be localized in the scene, localizing the hand-held manipulation device in the scene based on the video and the map, and storing the video as training data.