Grounding Statistical Machine Translation in Perception and Action

• Applicant: Professor Dr. Stefan Riezler
• Subject Area: Image and Language Processing, Computer Graphics and Visualisation, Human Computer Interaction, Ubiquitous and Wearable Computing
• Term: from 2014 to 2019
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 259623987

We propose a research program for grounding statistical machine translation (SMT) in perception and action. The crucial concept of preservation of meaning in machine translation will be defined and evaluated in the situational context of a world state. The criteria for optimization and evaluation are no longer translational adequacy of isolated sentences, ignoring the context of the discourse and ignoring the extrinsic task in which the translation is communicated. Instead, grounded SMT will focus on adequate meaning transfer with respect to human perception and action in a concrete task. For example, a successful translation of a natural language instruction enables an action that follows the instruction. In the context of instructions on game rules, this means that a description of a game rule is translated successfully, if correct game moves can be performed based only on the translation. In case of translating a description of a visual scene, the meaning to be transferred can be directly tied to the image: A description of a visual scene is translated successfully, if a monolingual speaker can identify the corresponding image among many similar images, based only on the translation. Grounded SMT introduces the concept of a task-specific evaluation of translation quality. It also offers the opportunity to deploy task-specific feedback on translations as data for learning SMT systems. This can be done online, e.g., by machine learning from feedback, or offline, e.g., by storing user-created and user-corrected translations as parallel data. The main challenges of our project will be as follows:+ Provide and popularize new mechanisms for translation data creation. We propose a gamification framework that potentially enables the creation of very large amounts of post-edited translations for training and evaluating SMT systems via simulated and human gameplay.+ Devise a machine learning framework that enables automatic learning in grounded scenarios. We will focus on response-based learning in which the only supervision signal available to the learner is the response from acting in the world. + Conduct experiments on grounding SMT in various concrete tasks, in which feedback for learning and evaluation will be provided. We will focus on game scenarios in which feedback is based on actions executed by a computer or a human in response to machine translations.

• DFG Programme: Research Grants