One of the most critical parts of the examination development process is setting an appropriate and defensible standard (also referred to as a passing point or cut score) on a credentialing exam.

NCCA defines standard setting as a “systematic method for determining the passing score on an examination based on the characteristics of the examination, particularly its level of difficulty. The result of the process is a pass/fail cut score that represents the lowest level of acceptable performance in the content area being assessed by an examination.” Though there are many methodologies that could be used for standard setting, most involve a subject matter expert judgement-based approach that is based on the concept of a “just qualified/minimally competent/borderline candidate.”

Since the goal of a credentialing exam is to categorize candidates into two groups: those that pass the exam (and are awarded the credential) and those who fail the exam (and are not awarded the credential), the passing score is expected to be achieved by a candidate whose knowledge and skills are right on the borderline.

Identifying the distinction in the level of knowledge and expertise that bumps a “borderline” candidate into the passing group compared to a “borderline” candidate that falls just below that level is a tedious process. Careful consideration should be given to the identification of subject matter experts to be selected for the panel to ensure that they are qualified to make these judgements.
Since the panel of judges must be well-versed in the examination content and must be familiar with the level of professional knowledge necessary for safe and effective practice measured by the exam, it is recommended that the panel is comprised of certified individuals. Judges who supervise or train certificants on the job should also be targeted. Inclusion of a few judges that are recently certified (within the last 3-5 years) is also recommended because they have fairly recent experience with the certification process, as compared to those who have been certified for a longer period of time, and they will have a more realistic and accurate perception of the “borderline” candidate.

Deciding the number of judges to include in the panel is also very important. With too few judges, the greater the results of the standard setting study will be impacted by one or two judges with either too high or too low standards. However, too many judges may be a wasteful use of resources, too expensive, more difficult to achieve a healthy dynamic during group discussions, and increases the exposure to the confidential examination items.

Though research has identified anywhere from as few as 5 to as much as 20+ judges, there are several factors that should be taken into consideration that can affect the number of judges necessary. For example, the panel of judges should be representative of the key demographics of the certified population (i.e., geographic location, practice area, workplace setting, gender, age, race, and other key demographics or interest groups that define the population of certificants).

The key is to ensure the composition of the panel is sufficiently balanced and neither underrepresented nor overrepresented by any demographic or special interest group to reduce the potential for unwanted bias and undue influence. Most organizations find that they can find a fair balance with about 10 participants.

It is important for organizations to have confidence that the panel selected is the best for the task at hand because the results of the study becomes a recommendation to the certification board and weighs in heavily into their decision for selecting the passing point for the examination. If the passing point is set in the wrong place, there are potential consequences that can put the integrity and value of the credential at jeopardy. For example, if the passing point is set too low, unqualified candidates will pass the examination and be awarded the credential. If those certificants go on to make serious job performance errors in practice, it would demean the value of the credential and would put the public (who the certification program seeks to protect) at risk. Conversely, if the passing point is set too high, qualified candidates would be denied the credential calling into question the fairness and validity of the credential. It would also discourage other qualified practitioners from seeking the credential.

Careful consideration in selecting the right subject matter experts to serve as judges is a tedious process, but it is worth the effort since the panel chosen sets the foundation for a successful, credible, and defensible standard.

Have a question? PTC is here to help. Contact us for more information.