How the GMAT™ Exam Uses Artificial Intelligence

Since 1953, the GMAT™ exam has been the only admissions test designed specifically to be used for admissions to graduate business programs. It is also among the first worldwide testing programs that utilized the computerized adaptive testing design and the artificial intelligence (AI)-based essay scoring method.

As the Senior Director of Psychometrics, Chris Han has been instrumental in building on the foundation of the assessment that employs pioneering data science in its nearly seventh decade of development.

In a conversation with the mba.com team, he explains how the GMAT exam is able to stand the test of time as the leader in standardized assessment, and his view on the future use of AI technology.

What are the main ways artificial intelligence is used in GMAT testing?

Chris Han: The GMAT exam uses AI in a few different ways. First, it adopts an automated item calibration process to accurately analyze the test item (i.e., test question) characteristics. It also uses Computerized Adaptive Testing (CAT) algorithm to maximize the measurement efficiency so that the test scores are the most accurate and reliable in assessing the test takers with the least amount of time required for candidate to sit through the exam. In other words, CAT helps produce a much shorter exam time while retaining high test score reliability. Lastly, Natural Language Processing (NLP) is incorporated in machine scoring algorithm to score the written essays of the Analytical Writing Assessment (AWA) section.

Some argue that standardized testing is contributing to discrimination against underrepresented minorities in school admissions. In your belief and practice, how does AI help remove bias and capture talents specifically suitable for business school training?

CH: The field of data science specialized in test data is called psychometrics. My world-class psychometrician colleagues at GMAC and I work to design and develop the assessment tools that measure the most timely and relevant skills required for success in the ever-changing business world. GMAC psychometricians apply various data science techniques to refine the measurement construct for the exam, eliminating any measurement biases across different subpopulations. This is because we at GMAC take seriously its mission to ensure the quality and fairness of the GMAT exam for all test takers across the globe. It does so by subjecting every potential test question to a rigorous seven-step development and review process before it can be used in operational exams. I talked more about how this process works in my blog post for gmac.com.

GMAC is also the leader in the educational test industry, continuously bringing innovations and cutting-edge technologies to the field. For example, the latest generation of GMAT and Executive Assessment – another exam GMAC administers – is designed and operated based on two different U.S. patent pending technologies to maximize the measurement efficiency in the adaptive testing algorithm and to optimize the item utilization. In addition, we recently invented a ground-breaking differential item functioning (DIF) detection method that greatly improves our process of ensuring the fairness of test across various subpopulations. We just started introducing the new technology in our assessment products this past summer.

Some are concerned about AI replacing standardized testing in the future of student talent assessment. What is your view on that?

CH: We are witnessing the growing trend of AI-based approach in many learning applications in the past few years. Most of them focus on gathering information about student’s learning activities from the process data. For example, login information, use time, geotagging, traces of input devices (mouse, keyboard, and etc.). We’ve seen some success stories about getting meaningful insights and prediction about the students’ learning outcomes by having the AI process the big data. Such AI-based technologies can indeed have some potential to enhance the overall learning process with diagnostic information that is timely available near real-time.

The questions remain: will AI eventually replace standardized tests for college admission process, for example? Will (or should) the machine determines who should be admitted or rejected for higher education? More than anything, there will be fundamental and philosophical debates about the idea. With that aside, if we just focus on the technology itself, some recent AI techniques like the neural network (a.k.a., deep learning) have indeed shown their unique advantages. For example, the deep learning AI does not require humans to build any models for connecting the data points and their predictions, but the machines make their own inferences based on the training they went through, and it essentially makes the machine the ultimate learning machine (literally), taking in all the structured and unstructured data which human usually cannot even comprehend. As a result, in some specialized applications, the deep-learning-based AI often outperforms human experts.

There are, however, three critical issues with the deep-learning AI approaches, especially when we consider the technology for education evaluation and specifically for admissions decisions. First, the availability of big data about each individual drastically differs across students. While some students with enriched IT infrastructure may have an abundant amount of big data generated for their activities via their laptop, tablet, smartphone, fitness trackers, in-door cams (oh, yes, of course, there are privacy issues, too), other without access to such IT infrastructure and tools will not be properly recognized by AI, which will cause a new kind of fairness issue in educational evaluation. Secondly, when the deep learning-based AI is used for any decision-making process, it does not offer any rationale or explanation about their internal decision making process that humans can understand. As a result, humans will have to blindly take the decision from the machine. For some, it may be acceptable, but others will not agree with this practice. Lastly, the machines make their decisions based on the training data they received from the past, and it is unknown how they would react to the situation they have not been experienced. In other words, machines are always thinking retrospectively as opposed to forward-thinking.

Recognizing all those limitations of deep learning AI, I do not full-heartedly believe AI will be replacing standardized testing. However, we do see its advantages and are actively utilizing the various AI techniques to perfect the GMAT design so that it can eventually complete the human intelligence to make the best informed, holistic decision for GME admissions.