Test Development, Scoring and Reporting Processes

ABPN examinations include initial certification examinations, continuing certification examinations, and short examinations tied to published articles as part of the Article-Based Continuing Certification (ABCC) pathway. This webpage was designed to describe the general processes required to create the examinations. As you can see, the process is rigorous and intensive in terms of both time and effort. A typical test development cycle is one year. Test scoring for our traditional proctored computerized examinations typically takes 8-12 weeks, depending on the number of examinations administered in the same time frame.

Each ABPN examination is developed and approved by a dedicated committee. The committee members are board-certified physicians in the field, all participating in Continuing Certification, who have been recognized by their colleagues as having the requisite expertise and skills. Committee members volunteer their time, for which the ABPN and our profession are enormously grateful. ABPN does cover travel-related expenses for committee members, as much of the work requires in-person meetings of the committees. All committee members receive training from the ABPN in the principles of item writing. Each committee is chaired by one of its veteran members; many committee chairs also serve on the ABPN Board of Directors. The committee members are supported in their work by ABPN’s dedicated staff, including trained personnel devoted to test development, editing, and psychometrics and scoring.

Board-certified physicians in our fields who are interested in being considered for committee membership may apply on an annual cycle, as described here. Candidates for committee membership must be certified in the applicable field by the ABPN and actively participate in ABPN Continuing Certification (CC), with timely documentation of CC requirements in the ABPN Physician Portal. They must also have at least three years of clinical experience post-training in their specialty and, if applicable, in their subspecialty. The ABPN emphasizes multiple elements of diversity for its test development committees, including but not limited to factors such as subspecialty or other specific expertise, practice type and work setting, geographic distribution, and sociodemographic characteristics and other experiences that foster the work of the ABPN.

Below you will find a Flow Chart showing the overall process of ABPN test development, scoring, and reporting processes. More detailed information about each step in the Flow Chart is provided below in drop-down text sections; please click on the downward-facing arrow next to each topic title to view the corresponding details.

Flow Chart of the ABPN Test Development, Scoring and Reporting Processes

Content Outline Development

Certification and Article-Based Continuing Certification (ABCC) examinations

Committee develop a content outline for each exam that specifies both the topics to be covered and the number (percent) of items devoted to each topic.
All content outlines are approved by the ABPN Board of Directors.
The outline is posted on the ABPN website for examinees to review.
The outline is periodically reviewed and updated.
Committee ensure that test items are coded properly when outlines are updated.
Examinations are constructed to follow the content specifications defined by the content outlines.

Article Selection

Article-Based Continuing Certification (ABCC) examinations

Committee members are given annual assignments to identify a specific number of articles within specific topic categories.
Articles are uploaded to our data storage software along with information pertaining to the article.
ABPN staff review the submissions, input the data into our database, and organize the materials for easy access for the committee members.
Committee members review the articles of their assigned partner in each topic to provide an alternate opinion on the article’s utility in our program.
Committee meet to discuss and select the articles that will become the focus of question writing.
ABPN staff takes notes during the meeting and records which are articles that have been chosen for usage, which are to be reviewed again at a future meeting, and which will not be kept in contention for use.

Item Writing and Review

Certification examinations

ABPN test development staff perform question item bank analysis and identify areas where more items are needed.
Committee members receive annual assignments and submit a specified number of items.
ABPN editors edit items for formatting and stylistic consistency and return them to authors for revision if needed.
Committee members are each assigned to review a section of the draft items ahead of the meeting.
Committee meet to review items, further modify them as needed, and decide for each item whether to accept if for use, send for further revisions, or reject.
Staff facilitate the meeting and track the committee’s revisions and decisions.
After the meeting, staff make all item revisions. Final versions of accepted items are saved in the item bank for potential use on subsequent examinations.
Staff compile feedback on any items that were flagged for revision or deletion. The feedback is provided to the original writer.

Article-Based Continuing Certification (ABCC) examinations

Committee members receive annual assignments and submit a specified number of question items.
ABPN editors edit for formatting and stylistic consistency and return them to authors for revision if needed.
Committee members are each assigned to review all submitted items in a particular topic. They read the articles and review the questions blindly (i.e. without answer keys) to simulate the experience of a test taker.
Committee meet to review items, make modifications as needed, and select the five questions per article to be used in the online examination.
Staff facilitate the meeting and track the committee’s revisions and decisions.
After the meeting, staff make all item revisions. Final versions of accepted items are saved in the item bank and are used in the online examinations.
Staff compile feedback on any items that were flagged for revision or deletion.

Patient Characteristics in Test Items

The items on ABPN’s article-based continuing certification (ABCC) examinations are carefully designed to measure meaningful and plausible testing points (e.g., diagnosis, management, etc.), without the influence of assumptions, bias, or stereotypes. When examinees select the correct (keyed) response, they are given credit because they are demonstrating what the examination item is designed to measure. ABPN examination committees encourage thoughtful consideration of patient characteristics, while at the same time striving to promote diversity and present patients who reflect the populations served by the examinees.

Many test items in this examination contain descriptions of patients. Characteristics of a patient such as age, sex, gender identity, race, ethnicity, sexual orientation, disability, socioeconomic status, native language, country of origin, and/or occupation are sometimes mentioned within case vignettes in test items. Some patient characteristics may be important inputs into the diagnostic reasoning process. Inclusion of some characteristics may increase the clinical verisimilitude of the patient cases. Their inclusion, however, as in actual clinical practice, may lead to incorrect conclusions and misdiagnoses. Among the latter are characteristics that could potentially be associated with harmful patient stereotypes.

In the context of item creation, race is considered a social construct not linked to biology or susceptibility to disease. This is similarly true of ethnicity and culture, heritage, or even country of origin. Ancestry, if known, may be biologically important, and thus may be relevant to factors relating to health and disease. In addition, when and if these characteristics are included in items, they should be considered based on patient self-report, not the assumption of the physician.

Based on the thinking above, some of the question stems can be brief and if the reference to a patient is general, patient characteristics are likely omitted unless directly relevant to the question at hand. On the other hand, if the question stem includes a portrayal of a specific patient (including vignettes for linked item sets), it is reasonable to at least include the patient’s age and sex.

Additional patient characteristics may be included for any of several reasons, including if they:

are clinically relevant or could aid in distractor quality.
are necessary for the examinee to better understand the context in which the patient is being seen (i.e., the item would be unreasonably difficult if excluded).
add to the overall exam-level representativeness of the referenced patient population.
increase the probability of detection, diagnosis, or recognition of an otherwise rare condition.
do not contain negative stereotypes.

Multimedia Development

Certification examinations

Staff perform resource bank analysis and identify needs for multimedia resources (i.e. standardized patient or other audio or video content).
Committee members receive annual script writing assignments that are used to create videos for the exam.
Committee review, revise, and finalize scripts.
Staff work with a video production vendor to create the videos. The recording is overseen for content accuracy by the ABPN committee chair.
The ABPN media editor processes all new media for item writing and potential exam inclusion.
The media product is reviewed and approved by the committee prior to creating item-writing assignments based on the media product.

Test Assembly

Certification examinations

Staff prepare a draft exam in addition to preparing pretest items and alternate items.
Staff compile an e-book containing the draft exam, pretest items, and alternate items to send to the committee ahead of the meeting.
Committee members are each assigned to review a section of the draft exam ahead of the committee meeting.
Committee review the draft and select final items. If any problematic items are identified, they are revised or sent back for further revisions before they are used on any exam.
Staff publish and proof final exam.
Committee chair reviews and approves final exam.

Test Administration

Certification examinations

All certification and continuing certification exams are administered under secure, proctored, standardized test conditions.
Exam data are delivered electronically to the ABPN.

Article-Based Continuing Certification (ABCC) examinations

Committee chairs review all finalized questions to determine if there are any changes.
ABPN staff upload questions to our online exam server. Editing staff proof all items a final time before setting them to live status.
Questions are activated as live early in any given year.

Test Scoring

Certification examinations

Psychometric staff conduct preliminary scoring to ensure that all data were transmitted accurately.
Psychometric staff review the scoring results to verify that each question on the examination is psychometrically acceptable.
Test development staff review all examinee comments to identify problematic items.
Based on these analyses, items that do not meet certain statistical criteria and/or have generated notable examinee comments are referred to the committee chairs for review.
Committee chairs review problematic items and come to an agreement on how to move forward.
Items that are deemed to be flawed by the committee chairs are removed from scoring, and the exam is rescored.
This phase typically takes 4-6 weeks, depending on the number of exam items and the number of exams being scored at the same time.

Article-Based Continuing Certification (ABCC) examinations

ABPN staff review statistics and all diplomate feedback to track item performance and determine if any questions may have flaws.
If a question is deemed to have possible flaws, the committee chairs will review the item and determine if the question should be taken offline for editing.
If the item is deemed flawed, ABPN staff rescore the item, meaning the item will not count toward the diplomate’s score on the exam.

Standard Setting

Certification examinations

The ABPN utilizes a criterion-referenced approach to standard setting. That is, there are no preset pass/fail rates for any group of examinees (referred to as norm-referenced approaches). It is always possible for all examinees to pass an examination.
Periodically a standard setting study is conducted by the committee chairs and other board-certified physicians in the field who are not current test development committee members. During a 2-day meeting, they review all of the items on the examination and use well-established, evidence-based procedures to set the passing standard for the examination.
After the standard has been set, statistical procedures are used to equate the pass/fail standard across different versions of the same examination.
This phase typically takes 2-4 weeks.

Score Reporting

Certification examinations

Score reports are generated, and several quality control procedures are performed to verify the accuracy of the scores.
Score reports are posted electronically in individual physicians' Physician Portal (specialty certification examinations) or mailed to the candidate.
This phase typically takes approximately 2 weeks.