Kathmandu, Nepal
Once you run the full OSCEs then you can proceed with more complex statistical analysis to establish the "construct validity", "Predictive validity" & "Criterion validity" later ...
Developing and establishing Reliability and Validity of OSCE stations on PPH Management skills:
1. Write a checklist /series of checklists that can be assessed and feasible in 5 minutes (remember OSCE is usually of 5 minutes duration). If you want to test many things then break them down to multiple checklists to assess the "holistic" assessment of skills related to the topic. This will now become "modified OSCE".
2. You can take advantage of the currently available tools from articles, books, websites etc. to supplement/complement you own checklists. Make the skills and checklists "mutually exclusive" i.e. don't assess same thing on different checklists.
3. Once you develop the tools then discuss it in a team (experts in PPH management) to ensure the "content validity". Remember this will be iterative process and you will need to document the discussion in each step.
4. Give the tool/tools to review by the other content and non-content experts (GPs, Emergency Medicine, Public Health including statistician, Allied Health etc.) to ensure "face validity".
5. Think wisely if you want to use Standardized patients or real patients. You need to train them beforehand to get the "standardized" outcomes. Real patients are difficult to manage and handle (we recently faced a serious problem with our Obs/Gyn OSCE during formative exam as only 2 out of 8 volunteer patients were willing and available at the end of the OSCE circuit!).
6. Train the interviews rigorously to ensure inter-rater reliability.
3. Once you develop the tools then discuss it in a team (experts in PPH management) to ensure the "content validity". Remember this will be iterative process and you will need to document the discussion in each step.
4. Give the tool/tools to review by the other content and non-content experts (GPs, Emergency Medicine, Public Health including statistician, Allied Health etc.) to ensure "face validity".
5. Think wisely if you want to use Standardized patients or real patients. You need to train them beforehand to get the "standardized" outcomes. Real patients are difficult to manage and handle (we recently faced a serious problem with our Obs/Gyn OSCE during formative exam as only 2 out of 8 volunteer patients were willing and available at the end of the OSCE circuit!).
6. Train the interviews rigorously to ensure inter-rater reliability.
7. Standard set each OSCE station using holistic/subjective approach i.e. use global rating along with the objective checklist ratings. These scores in combined will give you the "competency" required to pass each OSCE station using Borderline, Contrast group or Borderline Regression Method. One can also use Angoffing but it will be tedious process.
8. Once it is done, then pilot test the OSCE stations with volunteer participants (students, interns etc.) to know about the feasibility, problem in checklist and reliabilities. Use at least 10-15 volunteers in this process. Pilot test will also give you the internal construct reliability commonly known as Cronbach's alpha.
8. Once it is done, then pilot test the OSCE stations with volunteer participants (students, interns etc.) to know about the feasibility, problem in checklist and reliabilities. Use at least 10-15 volunteers in this process. Pilot test will also give you the internal construct reliability commonly known as Cronbach's alpha.
9. Use at least 2 raters in each station to calculate inter-rater reliability.
10. If you use more than 1 station then you can calculate the Generalizability (G) Coefficient which shows how much reliable the whole OSCE circuit. You need special software for this but it is available free from PERD, McMaster University, Canada.
11. If you also want to assess the test-retest reliability of your PPH OSCEs then run the pilot test two times using same interviewers and same candidates immediately after a break of, say, 30 to 60 minutes in the same day. If possible make sure not to contaminate your test-re-test reliability estimate i.e. don't let students refer the OSCE related resources from any references and interact on the OSCEs).
12. Modify the checklists after the pilot test if required and document them properly. Re-run the pilot test if the reliability estimates are not in the acceptable range. If they are acceptable then you are ready to run the "full fledge" OSCE on PPH management.
12. Modify the checklists after the pilot test if required and document them properly. Re-run the pilot test if the reliability estimates are not in the acceptable range. If they are acceptable then you are ready to run the "full fledge" OSCE on PPH management.
13. Calculate the relevant statistics, interpret the results, document them and publish it on MedEdPortal so that others can assess it when required.
This will complete all the reliability assessment for your PPH OSCEs:
I. Inter-rater reliability
II. Internal construct reliability
III. Test Re-test reliability
And remember that "Rome was not built in a day" so go slowly and document all the process. Your tool development and/or pilot testing phase can be your curriculum innovation project.
I. Inter-rater reliability
II. Internal construct reliability
III. Test Re-test reliability
And remember that "Rome was not built in a day" so go slowly and document all the process. Your tool development and/or pilot testing phase can be your curriculum innovation project.
Once you run the full OSCEs then you can proceed with more complex statistical analysis to establish the "construct validity", "Predictive validity" & "Criterion validity" later ...
N.B. -
a. Same process holds true for pre-validate and/or validate the OSPE/OSCE stations properly.
b. If the tool needs to be used in the local language (say Nepali) then it should be translated from English to Nepali first by a professional translator/expert bilingual followed by back-translation into English using translated Nepali tool by another professional. Then original English tool and back-translated English tool should be compared by the study team and modifications should be reflected in the Nepali tool as well. This process will also be iterative one and must be documented properly. It can be resource demanding as well.
No comments:
Post a Comment