A Novel Framework for Mixed Reality–Based Control of Collaborative Robot: Development Study

Authors

Md. Tanzil Shahria, University of Wisconsin - Milwaukee
Md. Samiul Haque Sunny, University of Wisconsin - Milwaukee
Md. Ishrak Islam Zarif, Marquette University
Md. Mahafuzur Rahaman Khan, University of Wisconsin - Milwaukee
Preet Parag Modi, University of Wisconsin - Milwaukee
Sheikh Iqbal Ahamed, Marquette UniversityFollow
Mohammad H. Rahman, University of Wisconsin - Milwaukee

Document Type

Article

Publication Date

2022

Publisher

JMIR Publications

Source Publication

JMIR Biomedical Engineering

Source ISSN

2561-3278

Original Item ID

DOI: 10.2196/36734

Abstract

Background:

Applications of robotics in daily life are becoming essential by creating new possibilities in different fields, especially in the collaborative environment. The potentials of collaborative robots are tremendous as they can work in the same workspace as humans. A framework employing a top-notch technology for collaborative robots will surely be worthwhile for further research.

Objective:

This study aims to present the development of a novel framework for the collaborative robot using mixed reality.

Methods:

The framework uses Unity and Unity Hub as a cross-platform gaming engine and project management tool to design the mixed reality interface and digital twin. It also uses the Windows Mixed Reality platform to show digital materials on holographic display and the Azure mixed reality services to capture and expose digital information. Eventually, it uses a holographic device (HoloLens 2) to execute the mixed reality–based collaborative system.

Results:

A thorough experiment was conducted to validate the novel framework for mixed reality–based control of a collaborative robot. This framework was successfully applied to implement a collaborative system using a 5–degree of freedom robot (xArm-5) in a mixed reality environment. The framework was stable and worked smoothly throughout the collaborative session. Due to the distributed nature of cloud applications, there is a negligible latency between giving a command and the execution of the physical collaborative robot.

Conclusions:

Opportunities for collaborative robots in telerehabilitation and teleoperation are vital as in any other field. The proposed framework was successfully applied in a collaborative session, and it can also be applied in other similar potential applications for robust and more promising performance.

Comments

Accepted version. JMIR Biomedical Engineering, Vol. 7, No. 1 (2022). DOI. © 2022 JMIR Publications. Used with permission.

Recommended Citation

Shahria, Md. Tanzil; Sunny, Md. Samiul Haque; Zarif, Md. Ishrak Islam; Khan, Md. Mahafuzur Rahaman; Modi, Preet Parag; Ahamed, Sheikh Iqbal; and Rahman, Mohammad H., "A Novel Framework for Mixed Reality–Based Control of Collaborative Robot: Development Study" (2022). Computer Science Faculty Research and Publications. 81.
https://epublications.marquette.edu/comp_fac/81