Terence Tao: Beginners should not use AI tools to perform expert-level tasks, and GPT is not very helpful for experts

In recent months, the famous mathematician Terence Tao has been trying to use large-model AI tools such as ChatGPT to assist in solving mathematical problems, and share the results and exchange experiences with everyone, such as using ChatGPT to write code, prove mathematical theorems, and generate LaTeX expression program code.

How useful are AI tools like GPT for his own research and for people? In the past two days, Tao seems to have come to his conclusion.

Tao found that while AI tools weren’t that useful for the core math part of his research (or perhaps just because he didn’t want to try them out for tasks he thought he was good at), they were very useful for tasks like generating code and creating a rough draft of the flowcharts in his paper (he used a LaTeX package (tikz) that he had rarely used in the past).

In general, he found that GPT allowed him to abstract a specific language for a computational task, such as Python, SAGE, regular expressions, LaTeX, etc. He could express his request to GPT in natural language, and GPT would provide the appropriate code in the appropriate language. Although without full integration, he still had to copy and paste the GPT output into a document that could be compiled.

Tao admits that GPT has begun to change his workflow, and that he used to avoid using code-intensive solutions to solve tasks, but now that situation is disappearing. He finds himself more willing to make coding a part of his daily work.

Take GitHub Copilot for example. Tao was pleasantly surprised to find that after setting it up with VSCode+LaTeXworkshop, it could automatically complete standard letters. Given only the opening paragraph and the following sentences, the rest of the paragraphs provided by Copilot were very close to what he actually wanted to write. And by making minor changes to the content of Copilotshuchu, he could save more than half the time to complete a standard letter.

Tao shared a screenshot of himself using the autocomplete feature in his paper. While trying to autocomplete the proof outline, Copilot does cite relevant results from the rest of the paper, but then gives some random analysis of number theory nonsense.

So far, aside from some very short auto-completion jobs or writing repetitive text patterns, he has found Copilot more interesting than useful when writing math papers, but the future may have effects that surprise him.

It turns out that the regular expression provided by GPT doesn’t handle nested delimiters the way Tao wanted, but it’s close enough. He can adjust it to what he wants after reading the regular expression documentation himself. In his opinion, it takes some regular expression skills to accomplish a similar task, but less skills than without the assistance of GPT.

In general, GPT seems to allow people to improve their skills in text-based tasks, but with diminishing returns as skill levels increase. Someone completely new to a task might become an advanced beginner using GPT, a beginner might reach an intermediate skill level, and an intermediate might become an expert, at least in some areas. But for experts, using GPT provides relatively modest gains going forward.

Therefore, Tao does not recommend that beginners use AI tools to complete expert-level tasks because the output cannot be properly checked. However, he also believes that people can use AI tools to complete tasks with a skill level higher than the normal level, so that they have enough ability to check the output results and decide whether to accept or make additional adjustments.

Someone asked, what if people are not clearly aware of this gap in skill levels? Tao said he believes that as AI tools become more familiar, most people will have a more accurate mental model of what these tools can and cannot do, and calibrate accordingly.

Of course, Tao also agrees that public AI education and independent output verification will become increasingly important skills in the modern world.