Abstract
A method and system for task planning for visual room rearrangement under partial observability is disclosed. The system or the robotic agent utilizes a visual input to efficiently plan a sequence of actions for simultaneous object search and rearrangement in an untidy room, to achieve a desired tidy state. Unlike search networks in the art that follow ad hoc approach, the method discloses a search network utilizing commonsense knowledge from large language models to find unseen objects. A Deep RL network used for task planning is trained with proxy reward, along with unique graph-based state representation to produce a scalable and effective planner that interleaves object search and rearrangement to minimize the number of steps taken and overall traversal of the agent, and to resolve blocked goal and swap cases. Sample efficient cluster-biased sampling is utilized for simultaneous training of the proxy reward network along with the Deep RL network.