Validation of a Tool

When referring to measurement, validity is the degree to which a measurement measures what it purports to measure.[1] Validation is the process of establishing that a method is sound.

In validation studies, different measurements are made of the same variable (this might involve assessing a clinical condition, knowledge, risk, etc.), and the level of agreement between these measurements is determined.[2] It is important to note that one of the measurements is believed to be a “gold standard”, or a method of measurement that reveals the true value or outcome. The other measurement (the “non-gold standard measure) usually has some desirable aspect, such as being less invasive, cheaper, easier to administer or more readily available, and the aim is to determine whether this other measure can determine the truth. In other words, the key question is: does the simpler/cheaper/easier to administer test actually measure what we want to measure?

 

Examples of publications that have documented the validation of an assessment tool are presented below:

Olsen JR, Gallacher J, Piguet V, Francis NA. Development and validation of the Molluscum Contagiosum Diagnostic Tool for Parents: diagnostic accuracy study in primary care. Br J Gen Pract 2014; 64(625): e471-6.

Matlow AG, Cronin CMG, Flintoft V, Nijssen-Jordan C, Fleming M, Brady-Fryer B, et al. Description of the development and validation of the Canadian Paediatric Trigger Tool. BMJ Qual Saf 2011; 20(5): 416–423.

 

Method and Written Report Guidelines

(based on instructions for authors, Canadian Family Physician journal, July 12, 2011)

 Validation of a Tool

Manuscripts must be prepared in accordance with the “Uniform Requirements for Manuscripts Submitted to Biomedical Journals” available on the International Committee of Medical Journal Editors (ICMJE) website.

Abstract: Include Introduction (Tool being validated and objective of program), Description, Discussion and Conclusion and do not exceed 300 words. Up to 4 key words (MeSH headings) should be included.

Validation of a Tool should not exceed 2500 words, excluding abstract, tables and references.

Introduction: State the background in which the tool has been or might be implanted and why it is important to validate the tool.  Specific objectives of the program should be described and appropriate literature should be cited concisely.

Description: Include description of the tool, including content, indications for use, potential benefits and potential challenges.

Method: Discuss the gold standard to which the tool will be compared, and how this comparison will be made. Indicate the data collection process and plans for calculating agreement between the gold standard and the current tool being validated.

Discussion: Compare gold standard with the tool for validation. Include a presentation of agreement and narrative account of where the agreement is stronger or weaker. Indicate the implication and significance of the agreement.

Conclusion: Include recommendations regarding tool use, with specification of any modifications that are suggested based on the results of the validation.

Limitations: Planned improvements should be presented. Conclusion should summarize the main components of the program, relate to the problem addressed, and be justified by the information presented.

References: Must be current and complete. Check references for accuracy, completeness, and proper format (according to the Uniform Requirements for Manuscripts Submitted to Biomedical Journals; http://www.icmje.org/). References should be numbered in the order they appear in the text and should be limited to works cited in the article. List all authors when there are 6 or fewer; when there are 7 or more, list the first 6, followed by et al.

Key Points: Include a short, point-form “key points” section including 1 to 4 points in 50 to 100 words. Key points should not duplicate the abstract or summarize the article; they should highlight what is new, different, unexpected or surprising in the article.

Tables and figures: Attached separately from the main manuscript. Tables and figures should clarify and supplement, but not duplicate, the text. Tables must be self-explanatory and concise. Prepare each table or figure on a separate page. Give titles to tables and captions to figures and other illustrations. Ensure that all tables, figures, and illustrations are cited at appropriate places in the text. Prepare tables in Word; no spreadsheets, please. Use table structure, not spaces and tabs to format tables.

Images and any other visual material: Attached as separate electronic files (do not paste them into the manuscript). These may include photographs, digital illustrations or extra photographs for use if space is available. Images should be sent in a JPG file format.

Authorship: Include a description in written report of what each resident/author contributed to their Resident Scholar Project. In addition, if several residents are involved in one Resident Scholar Project, include a statement to describe the processes they underwent to decide the order of the authors on the Resident Scholar Project written report.

Acknowledgements: Name everyone (e.g. faculty, professionals, research assistants) who contributed to the work of the Resident Scholar Project who are not authors in the Acknowledgements section, describing what they did. Also, describe all financial support of the Resident Project in the acknowledgements.

 

Guidelines for Authorship and Acknowledgements

Please follow the latest authorship definition provided by the Uniform Requirements for Manuscripts Submitted to Biomedical Journals listed below from the website: www.icmje.org as follows:

“All persons designated as authors should qualify for authorship, and all those who qualify should be listed. Each author should have participated sufficiently in the work to take public responsibility for appropriate portions of the content. One or more authors should take responsibility for the integrity of the work as a whole, from inception to published article. Authorship credit should be based only on 1) substantial contributions to conception and design, or acquisition of data, or analysis and interpretation of data; 2) drafting the article or revising it critically for important intellectual content; and 3) final approval of the version to be published. Conditions 1, 2, and 3 must all be met. Acquisition of funding, the collection of data, or general supervision of the research group, by themselves, do not justify authorship.”

In addition, all residents and faculty should refer to the Department of Family Practice Authorship and Collaboration policy that was approved by the Post-Graduate Education Committee in 2003. This policy guides the order of authors for faculty and residents who work together on a Resident Scholar Project.

[1] Last JM, Porta M. A Dictionary of Epidemiology (5 ed.) Edited by Miquel Porta. Oxford University Press. 2008

[2] Statistics & Research Methodology. Reliability & Validity – Validation studies. Available from https://epilab.ich.ucl.ac.uk/coursematerial/statistics/reliability_validity/validation.html (accessed 18 February 2015).