Let robots learn "common sense" by themselves? They can correct themselves and complete complex household chores without humans!

Robots are excellent human imitators, but most of them cannot do without human help, such as some preset programming. Especially for household robots, when faced with complex housework, if there is not enough common sense in their "brains", it will be difficult for them to meet the daily needs of human families.

Today, driven by large language models (LLMs), household robots can already acquire “common sense knowledge” and can correct themselves without human help and perform complex household chores excellently .

The related research paper, titled "Grounding Language Plans in Demonstrations Through Counterfactual Perturbations", has been published as a conference paper at the top artificial intelligence (AI) conference ICLR 2024.

Yanwei Wang, a Ph.D. student in the Department of Electrical Engineering and Computer Science at the Massachusetts Institute of Technology (MIT), is the corresponding author of the research paper.

He said that imitation learning is the mainstream method for realizing household robots. However, if the robot blindly imitates the movement trajectory of humans, small errors will continue to accumulate, eventually leading to other errors in the execution process. " With our method, the robot can self-correct execution errors and improve the overall task success rate. "

Let robots master some household chores

From wiping up spills to serving food, robots are learning how to perform increasingly complex household tasks. In fact, many household robots learn by imitating human behavior, and they are programmed to replicate the movements that humans instruct them to complete. However, because robots in the past did not have common sense, unless human engineers programmed them to adapt to every possible bump and nudge, they did not necessarily know how to handle these situations and would have to perform tasks from scratch .

Perhaps, by incorporating some "common sense knowledge," robots can be prepared when faced with situations that push them off their training tracks. According to the paper, Yanwei et al. verified the effectiveness of their proposed method through a simple everyday task. The task seems very simple, which is to scoop marbles from one bowl and pour them into another bowl.

However, in previous methods , in order to enable robots to complete this task, engineers often let the robot complete the "scoop" and "pour" actions on a fluid trajectory, and may repeat it many times, allowing the robot to imitate some human demonstration actions.

The problem is that while a human might be able to demonstrate a task in one go, that task depends on a series of subtasks, or trajectories. For example, the robot must reach into the bowl before it can scoop water, and it must scoop up marbles before moving to the empty bowl. If the robot gets jostled or makes a mistake during any of these subtasks, its only recourse is to stop and start over.

Unless human engineers explicitly label each subtask and program the robot or collect new demonstrations, it can recover from such failures and correct itself in a split second. “This level of planning is very cumbersome,” Yanwei said.

So, in this study, Yanwei and his team linked the robot's motion data with the "common sense knowledge" of a large language model.

They enabled the robot to logically parse many given household tasks into subtasks and adjust for interruptions in the subtasks. Based on this, the robot can continue to move forward without having to go back and start the task from the beginning. And importantly, human engineers don’t have to write detailed fixes for every possible failure.

According to reports, these deep learning models can process large text libraries and use them to establish connections between words, sentences, and paragraphs. Through these connections, large language models can generate new sentences based on the types of words that may appear after the previous word it has learned.

In addition to sentences and paragraphs, large language models can generate a logical list of subtasks involved in a specific task based on a prompt. For example, if asked to list the actions of scooping marbles from one bowl to another, the model might generate a list of verbs such as "reach", "scoop", "carry", and "pour".

“A large language model can tell the robot how to complete each step of a task using natural language. The human’s continuous demonstration is the embodiment of these steps in physical space,” Yanwei said. “We want to connect the two so that the robot can automatically know which stage of the task it is in and can replan and recover on its own. ”

Yanwei says their algorithm can now translate data collected by a teleoperation system into robust robotic behaviors, allowing the robot to complete complex tasks despite external interference.

Shortcomings and Prospects

Although this approach enables robots to self-correct without human assistance to complete complex household tasks, it also has certain limitations . For example, while their method does not require a large number of human demonstrations, it requires a large number of trial-and-error and reset-capable environments in order to collect task success labels for trajectories.

However, the research team said that this data inefficiency problem can be solved by active learning. In addition, it also requires some skills to prompt large language models to find suitable state representations for learning classifiers. In future work, they hope to combine pattern classifiers to learn state representations in an end-to-end manner.

Reference Links:

https://sites.google.com/view/grounding-plans

https://news.mit.edu/2024/engineering-household-robots-have-little-common-sense-0325

https://openreview.net/forum?id=qoHeuRAcSl

https://techcrunch.com/2024/03/25/large-language-models-can-help-home-robots-recover-from-errors-without-human-help/

https://www.youtube.com/watch?v=uerMlHmgPfI

https://yanweiw.github.io/