Multi-modal Analysis of Human-Object Interactions by a Socially Aware Agent:
Detecting and Fulfilling Needs

2023 - Multi-Layer Scene Graph Generation for Service Robotics

Project information

  • Category: PhD Thesis
  • Project start: October 2021
  • Expected completion: October 2024

Summary

Domestic Service Robots can help impair or elderly people in daily-life tasks such as cooking, cleaning, and taking care of the house in the same time as providing a basic social link. The robot must be able to interact with the human naturally, and adapt to their needs and goals to engage in natural collaboration tasks. In this paradigm, the understanding of implicit needs (i.e. needs that have not being explicitly expressed by the human) is not well covered by the current literature. This proposal presents a novel approach to the problem of modelling human needs from interactions with context as well as a novel algorithm to plan assistive actions in response. We based our approach on generating a symbolic representation of the scene and then infusing prior commonsense knowledge into it. Given this extended representation, we perform a complex reasoning process to infer the human needs and plan assistive actions. We especially use two type of external commonsense knowledge: knowledge about object affordances and temporal dependencies of events. We believe that this prior knowledge can model a correct perception of the user goals and associated relevant needs.