What is a Computer Adaptive Test? - GMAT PDF Download

What is a Computer Adaptive Test?

Computerized Adaptive Testing (CAT) is a computer-based assessment method that adjusts to the test taker's skill level, also known as "tailored testing." This means that the subsequent items or sets of items given in the test depend on the accuracy of the test taker's answers to previously administered items.

Computer-adaptive tests, considered cutting-edge in assessment technology, aim to accurately assess individual students' abilities by adjusting the difficulty of questions based on their answers. If a student answers incorrectly, the computer presents an easier question, and if the student answers correctly, the next question becomes more challenging. This approach seeks to eliminate the limitations of standardized tests which tend to be a "one-size-fits-all" method of assessment.

What is it like for a candidate to take a Computer Adaptive Test?

A Computer Adaptive Test (CAT) presents questions from a large pool of carefully calibrated questions to determine the level of difficulty. 

The test starts by presenting the candidate with a medium-difficulty question that is appropriate for their level and then adjusts the difficulty based on their answers. If the candidate answers correctly, the next question will be slightly harder, and if they answer incorrectly, it will be slightly easier. The system continually calculates the candidate's estimated ability and presents a personalized set of questions until a pre-defined level of confidence is reached or the maximum number of questions have been presented. Each candidate's test is unique.

Unlike other non-CAT tests, which may only provide useful results for average-ability learners, personalized assessments are designed to challenge all candidates. The number of easy questions for high-ability candidates and hard questions for low-ability candidates is reduced making the test more balanced to fit students of different calibres.

How does a Computer Adaptive Test work?

A CAT selects questions with the goal of maximizing the accuracy of the testing based on the examinee's previous answers. From the examinee's point of view, the test adapts to their ability level, with more difficult questions presented if they perform well on intermediate-difficulty items, and easier questions if they perform poorly. Unlike traditional multiple-choice tests that have a fixed set of questions for all examinees, computer-adaptive tests can achieve equivalent accuracy with fewer questions.

The steps in a basic computer-adaptive testing method are:

• Search the pool of available items for the optimal item based on the current estimate of the examinee's ability.
• Present the chosen item to the examinee and record their answer.
• Update the ability estimate based on all prior answers.
• Repeat steps 1-3 until a termination criterion is met.

At the start of the test, the algorithm usually selects a medium or medium-easy difficulty item as the first question, as nothing is known about the examinee prior to the test.

Due to the adaptive administration, each examinee receives a unique test. Despite the difference in the test content, their ability scores are still comparable to each other.

Some examples of Computer Adaptive Tests

Computer adaptive tests(CAT) have existed since the 1970s, and there are now many assessments that utilize it, some of them are

• Graduate Management Admission Test (GMAT)
• MAP test from NWEA
• SAT (will become multistage-adaptive in 2023)
• National Council Licensure Examination
• Armed Services Vocational Aptitude Battery

Also read: GMAT Handbook: Registration, Pattern, Syllabus, Preparation and More

Advantages of Computer Adaptive Tests

• Computer Adaptive Tests provide accurate scores for most test-takers with uniform precision, unlike standard fixed tests that are best suited for test-takers with average abilities and provide poor precision for those with extreme test scores. 
• The time for Computer Adaptive Tests can usually be shortened by 50% while maintaining a higher level of precision, resulting in time savings for both test-takers and the testing organization. 
• In recent years, CAT testing has grown greatly in these fields and is now accepted in medical facilities and laboratories as a tool for diagnostics. 
• Many Adaptive tests also provide immediate results after testing, like any other computer-based test.

Disadvantages of Computer Adaptive Tests

• The first challenge in computer-adaptive testing is calibrating the item pool. This requires pre-administering all the test items to a significant sample and analyzing the results. New items must be mixed into the operational exam items through a process called "pilot testing", or "pre-testing". This process poses logistical, ethical, and security problems.
• One security concern with CAT is that items can become frequent on tests for individuals with similar abilities, leading to groups having similar functional levels by sharing items. A randomized exam would be a more secure approach, but it's also less efficient.
• Since the test adjusts the difficulty according to the answer of the previous questions, it is impossible or candidates to go back to a previous question to review or redo the answer.
• It is challenging for a candidate in a time-limited CAT to allocate time effectively for each question and monitor their progress. They may face penalties for spending too much time on a challenging question at the beginning and failing to answer enough questions to accurately assess their proficiency in areas that are left untested due to running out of time. While untimed CATs are useful for formative assessments that inform future instruction, timed CATs are not suitable for high-stakes summative assessments used to evaluate aptitude for jobs and educational programs.

Components of a Computer Adaptive Test

Generally, Computer Adaptive tests have five technical components

• Calibrated item pool
• Starting point or entry level
• Item selection algorithm
• Scoring procedure
• Termination criterion

Calibrated item pool

• In a computer-adaptive test, a pool of test items must be available for the system to select from. These items can be created manually or through automatic generation. The pool must be calibrated using a psychometric model, with item response theory being a commonly used model. This model is popular because it allows for the placement of both persons and items on the same metric, making it useful for item selection.

Starting point or entry level

• In Computer-Adaptive Testing (CAT), items are chosen based on the examinee's performance. However, without any prior administration of items, it is impossible for the CAT to determine the examinee's ability. In such cases, the CAT uses a prior estimate of the examinee's ability based on previous information if available. If not, the CAT assumes that the examinee has an average level of ability, which is why the first item is usually of medium difficulty.

Item selection algorithm

• In a CAT, the ability of the examinee is placed on the same metric as the items using item response theory. This allows the CAT to choose an item that is best suited for the examinee's current ability estimate. The choice of item is made by selecting the one with the highest information content, which is based on factors such as the item's discrimination parameter, conditional variance, and pseudoguessing parameter.

Scoring procedure

• Once an item has been given, the CAT will adjust its evaluation of the examinee's ability according to whether they got the item right or wrong. This is done with item response theory to create a chance of the examinee's ability. Two approaches are called maximum likelihood estimation and Bayesian estimation. The Bayesian estimation assumes an initial ability level for the examinee and has two estimators - expectation a posteriori and maximum a posteriori. Maximum likelihood is similar to a maximum a posteriori Bayesian estimate when a uniform (f(x)=1) prior is used. Maximum likelihood is typically unbiased, but it is unable to give a theta evaluation for a response vector that is either all correct or all wrong, in which case a Bayesian method needs to be employed.

Termination criterion

• The CAT algorithm is designed to continually provide items and adjust the estimation of the examinee's skill. This will carry on until all the items have been given unless a condition for ending the test has been established. Generally, the test is halted when the examinee's standard error of measurement drops below an indicated value, thus the statement that an advantage is that the examinee's scores will be uniformly precise or "equiprecise". There are also other conditions for concluding the test, such as if the test is only intended to distinguish if the examinee should "Pass" or "Fail" the test, instead of getting an exact assessment of their ability.

Frequently Asked Questions (FAQs) about Computer Adaptive Tests

What is meant by a computer adaptive test?

A computer adaptive test (CAT) is an assessment that is tailored to the individual’s ability level. As the test taker answers questions, the computer algorithm adjusts the difficulty of the next question accordingly. This type of test is used to quickly and accurately assess a person’s knowledge, aptitude, or skills in a given area.

What are the examples of computer adaptive tests?

Many tests are moving to a computer adaptive format as of 2023, some of the famous CAT are Graduate Management Admission Test (GMAT), GRE, MAP, National Council Licensure Examination, Armed Services Vocational Aptitude Battery.

How does CAT determine the difficulty level of questions?

A CAT uses algorithms to select questions based on the examinee's previous answers. The level of difficulty increases or decreases based on whether the examinee answered correctly or incorrectly.

What are the advantages of CAT?

The advantages of CAT include a more efficient and accurate evaluation of the examinee's abilities, a reduced amount of time needed to complete the test, and the ability to administer the test in a standardized and secure environment.

What are the disadvantages of CAT?

The disadvantages of CAT include the need for a reliable and secure computer system, the lack of human interaction, and the possible lack of familiarity of the examinee with computer-based testing. The fact that they require a large question bank to select questions from requires a lot of effort and storage.