Yes. HEARTSMAP’s inter-user reliability was evaluated among pediatric ED clinicians locally at BC Children’s Hospital in 2014, and among ED clinicians in general EDs in 2016.
At BC Children’s Hospital, data from 104 ED mental health presentations was extracted to construct clinical narratives. A panel of ED clinician evaluators reviewed each of these narratives. The panel included: 2 pediatric emergency medicine physicians, 1 nurse practitioner, and 1 bedside nurse. The results below summarize the agreement among the clinician reviewers with regard to consulting psychiatry in the ED:
Comparisons |
Kappa Value |
Interpretation¹ |
Pediatric ED physician vs. Pediatric ED physician |
0.7 |
Substantial Agreement |
Pediatric ED physician vs. Nurse practitioner |
0.6 |
Moderate Agreement |
Pediatric ED physician vs. Bedside nurse |
0.5 |
Moderate Agreement |
In addition, eleven ED clinicians from 5 health authorities – Provincial Health Services Authority, Providence Health Care, Vancouver Coastal Health, Fraser Health Authority, and Interior Health Authority validated the HEARTSMAP tool in 2016 (2). These ED clinicians, including physicians, social workers, and psychiatric liaison nurses, were asked to use the HEARTSMAP tool to evaluate 50 fictional narratives describing a variety of pediatric mental health cases.
We report moderate to near excellent agreement, overall among clinicians for all 10 of the tool’s psychosocial sections (κ=0.43 to 0.93) and domain scores (κ=0.75 to 0.90), with acceptable agreement across all tool-triggered service recommendations (κ=0.36 to 0.65). Our findings show that HEARTSMAP may be reliably used by ED clinicians in assessing MH issues among youth. Results from this study will assist in informing the wider clinical implementation of HEARTSMAP as a standard assessment tool, in diverse emergency care settings.
- Viera AJ, Garrett JM. Understanding Interobserver Agreement¿: The Kappa Statistic. Fam Med. 2005;37(5):360-363.
- Virk, P., Stenstrom, R., & Doan, Q. (2018). Reliability testing of the HEARTSMAP psychosocial assessment tool for multidisciplinary use and in diverse emergency settings. Paediatrics & Child Health, 1–6. https://doi.org/10.1093/pch/pxy017