AI decides your university? Calculate your 7 personal qualities in minutes

Currently, many universities adhere to the concept of comprehensive review.

In a recent survey by the National Association for College Admission Counseling (NACAC), 70% of admissions officers said they consider personal qualities to be an important factor when selecting applicants.

However, in reality, reviews do not ensure fairness, and admissions are limited by resources.

To address the above challenges, a research team led by Benjamin Lira of the University of Pennsylvania trained a language model called "Robustly Optimized BERT Pretraining Approach (RoBERTa)" to review applicants' extracurricular or work papers and look for seven personal qualities contained therein.

The results show that AI models trained on human ratings can generate millions of personal quality scores in minutes, accurately replicating human ratings while being explainable and fair.

The related research paper, titled "Using artificial intelligence to assess personal qualities in college admissions", has been published in Science Advances, a subsidiary of Science.

However, the research team also stressed: "No algorithm can determine the goals of the university admissions process, or which personal qualities are most important, and universities need to clarify their admissions goals before automating the review process."

AI successfully identifies personal qualities

At many universities, comprehensive reviews have become a widely adopted admissions assessment method that considers personal qualities as the most important consideration. The method is believed to promote fairness because it allows applicants to demonstrate non-cognitive skills and character that cannot be reflected in standardized tests.

However, historical cases have shown that comprehensive review systems can also lead to unfairness, especially when the evaluation criteria are unclear, lacking explanation or disclosure. The current comprehensive review lacks transparency, and admissions officers rely mainly on personal statements to assess the qualities of applicants, but these details remain unclear to applicants and the public.

Ways to improve comprehensive reviews include adopting more transparent and systematic assessment methods, using structured rubrics, and conducting multiple independent assessments. These recommendations reflect the application of psychometric principles and can improve the reliability, validity, and interpretability of assessments, thereby reducing potential bias.

However, there are still problems in reality, mainly including the increasing number of applications, time and resource constraints faced by university admissions officers. If resources are unlimited, then the evaluation can be better optimized and greater fairness can be achieved.

In this study, the team used AI technology to develop a method to better assess personal qualities. First, they selected a sample of 309,594 college applications that were de-identified, each of which included a 150-word essay describing the applicant's chosen extracurricular activities or work.

Next, researchers and admissions officers selected 3,131 of these essays to train a language model called RoBERTa. They looked for seven qualities in applicants’ essays: learning ability, perseverance, goal pursuit, teamwork, intrinsic motivation, leadership, and altruistic goals, which can help determine which applicants are most likely to succeed in college.

Finally, the researchers used these fine-tuned models to review an additional 306,463 articles and assign a score to each.

The researchers and admissions officers found evidence of seven personal qualities in every essay. Some personal qualities were more common than others. For example, they identified “leadership” in 42% and 44% of the essays, respectively; by contrast, they identified “perseverance” in only 19% and 21%.

AI assists judgment, not replaces it

However, this study also has certain limitations.

First, the study did not include the personal statement required by the Common Application, which limits the breadth of the study. Future studies should consider including this element, especially considering that recent applicants submit personal statements as PDF attachments.

Second, the high school GPA in the dataset is based on only a small number of applicants, which may affect the representativeness of the study. Future research needs to pay more attention to how to obtain more available data.

Third, personal qualities have a moderately high predictive effect on college graduation, but this is only relative to standardized test scores. The prediction of long-term life outcomes is complex and requires consideration of multiple factors. Future research could explore other unconsidered factors, such as tuition payment, academic preparation and support, etc.

Fourth, the study only focused on college graduation as an outcome, without considering other aspects of success, such as GPA, extracurricular activities, and community contributions. This requires a broader consensus on optimizing the goals and implementation of college admission decisions. Future research can consider more aspects of success and their impact on personal qualities.

Therefore, this study brings a critical perspective to the current state of comprehensive review and selective admissions. Future research and practice should focus on clarifying the goals of comprehensive review before automating the process.

In addition, the study found that Campbell’s Law suggests that the greater the weight given to evaluation in high-stakes decisions (as opposed to low-stakes research), the more distorted incentives are likely to emerge. For example, applicants may try to shape their essays, perhaps using AI tools such as ChatGPT, to cater to the requirements of admissions officers and trained algorithms.

It’s worth noting that algorithms can make mistakes, especially when it comes to finding patterns. For example, the RoBERTa model fine-tuned by the study assigned a very high social purpose score to the sentence “I donated heroin to the children’s shelter.”

Therefore, the study recommends using AI to supplement rather than replace human judgment. No algorithm can decide the goals of the college admissions process, or which personal qualities are most important. ** Viewing algorithms as a supplement to human judgment rather than a replacement can also address the problem of algorithmic avoidance, where people are more likely to trust human decision makers than algorithms in the presence of contradictory evidence.

Paper link:

www.science.org/doi/10.1126/sciadv.adg9405

Author: Yan Yimi

Editor: Academic