This APP can automatically convert the operation video into an AR tutorial, teaching you how to repair furniture step by step

[[203491]]

There are often such plots in novels or TV dramas: a girl asks a boy to fix her computer, a neighbor asks the guy next door to fix a light bulb, and the two become inseparable. But in real life, you don’t always know how to fix something when it breaks. At this time, you may need to search around the house and sweat profusely for a piece of manual.

Well now, an APP developed by German startup IOXP can help us solve this problem.

This application can first record a video of a person performing a task correctly, and then automatically convert it into an AR tutorial with the help of artificial intelligence technology. As long as our device supports AR, in the future we will be able to download the tutorial and perform actual repairs while receiving expert guidance.

[[203492]]

AR tutorials allow you to perform practical repairs while receiving expert guidance

Photo credit: Chesnot/Getty Images

To create an AR tutorial with the IOXP system, you first use a standard camera to record how the person who is completing the task performs the task. These operations may be changing the oil in a car, installing a boiler, or doing cross-stitch. Then a large number of computer vision algorithms are used to break down the video into different understandable modules: detecting the position of the human hand, the operation movements, and identifying different objects. The system will then generate an electronic manual that details how the task is completed step by step. Finally, it is automatically converted into an AR version of the tutorial.

Most importantly, the AR instruction manual will automatically start once it recognizes the task in front of the user. For example, if an engineer is performing some basic maintenance on a machine, the camera on his AR headset will automatically recognize the machine in front of him.

"Then you can see the hand of an expert appear in front of you and tell you what to do," said Didier Stricker, an expert at the German Research Center for Artificial Intelligence and one of the co-founders of IOXP. If the user touches the wrong dial or part, the hand will be projected in red and the system will replay the operation the user is about to perform, highlighting the human hand, the relevant knobs, dials, parts, etc., and the accuracy can be maintained within half a centimeter.

In Germany, Bosch Group, a Fortune 500 company, has already tried out the system with some engineers and is currently working on promoting it externally.

In fact, many AR device manufacturers promise that AR technology will turn all unskilled novices into experts in maintenance. Whether you want to change the oil in your car or learn to cut an onion without crying, the future AR headset will project an operating instruction (tutorial) in the real environment, and then guide us like an experienced professional, patiently correcting the mistakes made during the operation.

The idea sounds great, but a big obstacle is that it is not only difficult but also expensive to make these tutorials with operating instructions. This is why AR tutorials are currently usually used by wealthy companies, such as those that repair fighter jets. But this German startup wants to make AR more accessible to ordinary people.

“Our goal with this app is to allow anyone who fixes things at home to download a tutorial into their AR device so that the device can show them how to fix it and teach them how to do it in a very short time,” said Stricker, co-founder of IOXP.

Why would we need to watch AR tutorials when there is already a series of instructional videos on YouTube (“how-to” videos on YouTube)? Stricker explained that “YouTube does cover a lot of the repair techniques we want to learn, but those videos don’t automatically tell us when we make a mistake.”

Rab Scott from the Advanced Manufacturing Research Center at the University of Sheffield in the UK commented: "The application developed by IOXP can automatically extract operational information from ordinary videos to generate AR tutorials. The workload involved is enormous." This means that the information extraction technology of the IOXP system is already relatively mature.

Of course, for some people, it’s only a matter of time before flat-pack furniture with AR instructions appears. “Ikea has a lot of how-to videos on YouTube. It will be easier to turn that video content into an augmented reality app in the future using IOXP’s technology,” Scott said.

In addition, IOXP's technology is not necessarily limited to head-mounted devices, but also supports mobile phones or tablets. Imagine a scene like this: one day you want to replace the parts on the boiler, and then open this APP on your phone, and then the correct part position will appear in front of you, and it will also show you how to disassemble the old parts. After completion, it will either prompt you to continue to the next step, or tell you that the previous operation is wrong and needs to be readjusted.

Scott believes that such technology will become increasingly important in the coming years, especially for countries like the UK. "Sixty per cent of engineers in the UK are over 50 years old. Unless we can retain the knowledge of these engineers and pass it on effectively to the next generation of engineers, the UK will 'fall off the knowledge cliff' within 15 years."