Construction, administration and quality assurance of student assessment in HPE - Part II

Kathmandu, Nepal

In the last post, I have described the rationale and steps followed for creating SISAQ or PBQ. This post deals with the cut-off score (pass mark) determination of the SISAQ. Item analysis of SISAQ will be discussed in the next post.

SISAQ mostly test the Higher Order Thinking Skills (HOTS) and it will be usually difficult for the minimally competent "borderline" candidates. Since it does not allow the test-takers to see the options as in MCQ, it eliminates the chance of "guessing". Further, SISAQ questions are NOT direct so the test-takers should extract the answers based on the clinical/lab/public health based scenario and vignettes, which is again a challenging task for the borderline students.

Unlike OSPE/OSCE where students are observed for 5 - 8 minutes while doing the task and then awarding a holistic "global" rating at the end to facilitate the calculation of the pass-mark, SISAQ cannot be observed for such as short span of time for each students. Thus, SISAQ pass mark is determined based on the compromised method using standard setting done a priori of the exam by the concerned faculty and performance of students (marks obtained) in the same SISAQ.

SISAQ standard setting of faculty:

 

Once the SISAQ construction is done as per the rules laid down in the previous blog post in this thread, group of faculty responsible for the particular SISAQ decides on the range of the expected marks for five mutually exclusive groups using required competencies.

 

The five groups are as follows:

Very Poor:

Poor:

Borderline:

Good:

Very Good:   

The most important group is the borderline group as it is used to determine the pass mark. Thus, it is very important to assign the range of marks for this group as it is based on difficulty level of scenario, vignettes and questions from each disciplines. Faculty trained in criterion-referenced student assessment method also know that they have to think of the "borderline" group of students as those who could either pass or fail the SISAQ while referring to the students in this range.

Once the SISAQ is administered and marked by the faculty then a compromised method known as "borderline regression method" is used to calculate the pass mark. Actually, it is a simple linear regression analysis where marks obtained by the students in the SISAQ are taken as the dependent variable (y) and five mutually exclusive groups for the SISAQ are taken as independent variable (x). As the five standard setting groups are ordinal in nature, they are coded from 1 to 5 for data analysis. Code of the borderline group (i.e. 3) is then used to estimate the pass mark using the regression equation obtained.

 

Illustration:

 

SISAQ # 2

 

Full marks: 15 marks

Total time: 25 minutes

Criterion-referenced standard setting:

1. Very Poor:    0 - 3

2. Poor:             3 - 5

3. Borderline:   5 - 7

4. Good:           7 - 11

5. Very Good:  11 - 15

I found Google docs very easy for running the simple linear regression using the in-built "forecast" function to determine the pass-mark of SISAQ. You can see the details following this link:

https://docs.google.com/spreadsheet/ccc?key=0AoJkXbvD07ECdHh0NmUyUkFjNnA4aXozWnlJajM4QWc&usp=sharing

The borderline-regression (BLR) method reveals the pass-mark of this hypothetical SISAQ as 6.242 (41.61%) and based on this pass-mark there are 4 students (20%) fail and 16 (80%) pass it. The pass-mark (6.242) was obtained from using performance of the students in the exam and it lies between 5 and 7 which are the minimum and maximum criteria set for passing the SISAQ by faculty.

 

The Google Spreadsheet link above also shows the outcome for this SISAQ if the conventional 50% pass mark is used. It reveals that only 60% of the students would pass and 40% of them will fail it, which is different than the one obtained from using BLR method above. This happened because the "fixed" criteria of 50% does not take account of the "difficulty" level of the SISAQ in reference to the "borderline" group of students and thus is "biased". We all know that every SISAQ cannot have same level of difficulty. Thus, it is "injustice" to have a "fixed" pass-mark for each and every item and tests. So, I strongly suggest to have fluctuating pass-mark based on its difficulty in terms of content and construct for SISAQs.

 

In sum, theory paper should have both MCQ and SISAQ to test the knowledge from LOTS to HOTS. They must be based on the Angoff or Hofstee or Beuk cut-off score for MCQ and BLR cut-off score for SISAQ in order to get "holistic" and "criterion-referenced" assessment of the knowledge for any competency based curriculum. This method is defensible as they are "always" based on the decision of a group of faculty rather than a single faculty and "mostly" they also used students exam scores.

 

Most importantly, these criterion-referenced methods provide defensible and justifiable grounds for scientific and evidence based student assessment method for the undergraduate as well as post-graduate health professions education program.

 

Rest later, happy SISAQing ...