HR-SAT: Human-Robot Shared Assembly Taxonomy

Authors: Regina Lee*, Hao Zheng*, Yuqian Lu

* they contributed equally

Human-Robot Knowledge Transfer

Future manufacturing will witness a shift in human-robot relationships towards collaboration, compassion and coevolution. This will require seamless human-robot knowledge transfer. Differences in language and knowledge representation hinder the transfer of knowledge among humans and robots. Thus, a unified knowledge representation system that can be shared by humans and robots is essential.

Driven by this need in a product assembly scenario, we propose the Human-Robot Shared Assembly Taxonomy (HR-SAT). With HR-SAT, any comprehensive assembly task can be represented as a knowledge graph that both humans and robots can understand. HR-SAT enables utilisation in various applications as a hierarchical, consistent and information-rich representation.

HR-SAT

HR-SAT has a hierarchical structure which decomposes a task into primitive tasks and primitive tasks into atomic actions. Definitions of task, primitive task and atomic action are based on the functional state change of the manipulated object and the interaction state change between the subject and manipulated object.

Hover over image for more information

For better utilisation of HR-SAT, we also define nine atomic actions common in assembly processes.

More details in the Supplementary

Task description framework modeling

To flexibly describe the assembly task, we proposed the task description framework and modeled it as a knowledge graph.

Use the interactive window below to see the modeled knowledge graph

The RDF/XML file of the modeled knowledge graph can be downloaded below.

Case Studies

We present two application case studies to demonstrate the utilisation and significance of HR-SAT in diverse applications.

CS 1: Assistive Assembly and Quality Assurance

This study presents an assistive assembly and quality assurance framework for a hydraulic valve assembly. In this framework, HR-SAT can convert the disparate process information in the assembly instructions into a unified process representation in the form of a knowledge graph. From the knowledge graph, the primitive task and key atomic action nodes are extracted to obtain a unified instruction.

The generated unified instruction can then be used in to obtain personalised instructions, which is used to assist the worker as well as to cross-check with the assembly process monitoring for quality assurance.

The valve assembly in Case Study 1 is based on the 304 Check Valve from Oasis Engineering Ltd.

The knowledge graph for the unified process information can be downloaded below as an .xml file.
The instructions used by the HR-SAT and the BoM of the valve can also be downloaded below.

Case Study 2: Human-Robot Collaborative Assembly

This case study presents the human-robot collaborative assembly framework for the Generic Assembly Box (GAB). In this framework, HR-SAT converts the disparate assembly process information (such as video demonstrations) into a unified Subject-agnostic Task Graph, which provides a universal process description for any agent. From the Subject-agnostic Task Graph, the primitive tasks and even atomic actions are allocated to agents to obtain the HRC Task Graph. The allocation is based on the agent's capability and availability, guided by optimization objectives such as overall efficiency and human wellbeing.

From the obtained HRC Task Graph, personalized instructions can be generated for human workers and executable policies can be generated through motion planning for robots. During the collaborative task, the robot's policy is updated dynamically by monitoring and predicting the human to ensure worker safety and collaborative efficiency.

Two task graphs are created in Case Study 2: the Subject-agnostic Task Graph (STG) and the HRC Task Graph (HTG).
The knowledge graph for the task graphs can be downloaded below as an .xml file.
The BoM of the Generic Assembly Box (GAB) can also be downloaded below.

The process of a worker assembling the Generic Assembly Box (GAB) with instructions can be seen in the video below

Video of worker performing assembly with instructions

Acknowledgements

We would like to thank Oasis Engineering Ltd for giving us permission to use their 304 Check Valve in Case Study 2.

We would like to thank Vincent Link, Steffen Lohmann, Eduard Marbach, Stefan Negru, Vitalis Wiens for WebVOWL, which is used to visualise the modeled knowledge graph of the task description framework.

This project was funded by The University of Auckland FRDF New Staff Research Fund (No. 3720540)