Conversational natural language is subject to noise, incompletions and grammatically ambiguous phrasing. To increase the robustness of communication, human conversation partners typically build on conversational repair (CR) to iteratively and interactively resolve misunderstandings. In the context of human-robot interaction, CR provides the possibility to interrupt and to repair a misunderstood instruction that is already being executed. However, current approaches do not consider the conversational repair of misunderstandings in human-robot dialog, even though this would significantly increase the robustness of human-robot interaction. The goal of this project is to fill this gap by addressing two core problems that have hindered existing approaches to successfully address conversational action repair for human-robot interaction. The first problem is the realization of an adaptive context-specific state model that integrates language with action. Most dialog systems consider only verbal communication, and they ignore that human communication is an embodied multi-modal process that is grounded in physical interaction. So how can we realize a scalable model that considers situated conceptual state representations for mixed verbal-physical interaction? To address this first problem, this project builds on a neuro-symbolic approach that integrates our previous work on embodied semantic parsing with our expertise in deep reinforcement learning. Herein, we will research a hybrid data- and knowledge-driven model for compositional interaction states that link the physical world state with semantics in language and dialog.The second problem pertains to the noise, disfluency, and polysemy of spoken natural language. Existing learning-based parsers are robust enough to parse noisy spoken language but they require large amounts of training data. So how can we realize a robust semantic parser that is data efficient while considering the mixed verbal-physical interaction? To address this second problem, this project complements our previous semantic parsing methods with a neural machine-translation approach. To this end, we will exploit the reward signal of the reinforcement learning as an additional data source to improve the data efficiency of the neural parser. The data required for this project will be generated using crowdsourcing, and the evaluation will be conducted on a humanoid robot. We expect the project to generate impact as a new approach for human-robot interaction, and to contribute novel methods for representation learning to the scientific communities in the fields of natural language understanding, machine learning, and intelligent robotics.

In this project MoReSpace, we will investigate the extent to which the transfer of learning is responsible for the development of a "self", and hypothesize that a conflict-driven attention model plays a major role. In the first part of our project, we investigate the transfer of previously learned action-effect associations to new unexpected environmental dynamics. Here, we put a strong focus on cognitive plausibility and motivate our model with psychological phenomena such as "haptic neglect". The phenomenon occurs, for example, when the computer mouse is inverted and the mouse pointer is directed in the opposite direction in each case. In such scenarios, psychologists have found reduced perception of the haptic and proprioceptive senses. Our hypothesis is that this is due to a conflict-driven attention mechanism that improves the ability to deal with such new dynamics. We will evaluate our model on a physical robot, and we will theoretically substantiate it with our collaboration partners from psychology. In the second part of the project, we will focus on imitation learning. Our hypothesis is that the attention model captures some psychological properties that are important for the human ability to change perspective and to imitate. We hypothesize that this will lead to novel methods of imitation learning for robots. We expect these methods to lead to significant improvements in the learning performance. We will evaluate this empirically and reproducibly.

In this project, we will investigate how far the transfer of learning is responsible for the development of a “self”. We will, therefore, present a computational ideomotor approach, and hypothesize that the transfer is possible due to a hierarchical structure of action-effect associations, such that training a specific narrow low-level task indirectly trains higher cognitive skills that are involved in other low-level tasks. For example, manipulating objects and balancing are two low-level tasks that both involve the cognitive skill of mental rotation. Consequently, according to our hypothesis, the mental rotation skill will benefit from balance training, which in turn also triggers an improvement of the learning of the grasping task. We will address our hypothesis by implementing a computational and neurocognitively plausible neural network architecture evaluated on a physical humanoid robot. Our expected contribution is a functional neurocognitively plausible deep reinforcement neural network model for ideomotor transfer learning that is verifiable on a reproducible robotic platform.