Instructing Robots through gestures, text and speech

We are exploring the use of gestures and natural language (text to begin with and eventually speech), to instruct robots to do complex tasks. Not only do we want to tell robots what to do, we would like to also tell it how to do it as well as sometimes say what it should not do. More details of our current work is available in the two references below.

References:

Spatial reasoning from Natural Language Instructions for Robot Manipulation, Sagar et. al., ICRA 2021

Multi-Instance Aware Localization for End-to-End Imitation Learning, Sagar G.V., Raviteja Upadrashta, Shishir Kolathaya, Bharadwaj Amrutur, IROS 2020.