Microsoft Wants to Use ChatGPT to Control Robots through Language

Published on feb 27, 2023

Microsoft Autonomous Systems and Robotics Group researchers have demonstrated how OpenAI’s ChatGPT can be used for robotics applications, including how to design prompts and how to instruct ChatGPT to use specific robotic libraries.

Microsoft’s engineers explain that current robotics is based on a feedback loop between the robot and an engineer who is responsible for coding, observing, and correcting the robot’s behavior.

Microsoft vision can be used to translate a human-language description of the task to be accomplished into code for the robot by using ChatGPT. By doing so, the engineer (in the loop) would be replaced by a non-technical user (on the loop), whose only responsibility would be to provide the original task description in human language, observe the robot, and provide any feedback about its behavior, which ChatGPT would also translate into code for improving it.

Using their experimental approach, Microsoft researchers created a number of case studies that included zero-shot task planning to instruct a drone to inspect the content of a shelf; manipulating objects with robotic arms; searching for a specific object in an environment using object detection and object distance APIs; and others.

All of those cases, ChatGPT was able to generate the code to control the robot as well as to request clarifications when it found user input to be ambiguous, according to Microsoft.

In its efforts to make ChatGPT suitable for robotic applications, Microsoft investigated three main areas: how prompts can be designed to guide ChatGPT, how to utilize APIs and develop new high-level APIs, and how to provide human feedback via text. As a result, these three areas represent the cornerstones of a methodology for utilizing ChatGPT for robotic applications.

By using a simulation or inspecting the code, the user evaluates ChatGPT’s code and provides feedback for ChatGPT to improve.

Using the generated code, a robot can be programmed when the outcome is satisfactory to the user.

As part of the launch, Microsoft is launching a collaborative open-source platform that allows users to share prompting strategies for a variety of robot categories, which currently includes all the prompts and conversations the Microsoft team used during their research. Additionally, they plan to add robotic simulators and interfaces for testing ChatGPT-generated algorithms.