Evaluation Strategy

After gathering a total of 318 documents, we undertook several methods to conduct a thorough evaluation of each. 

  • Literacy – We created an average grade level score for each material based on Lexiles scores and Gunning Fog, conducting further evaluation for the 94 materials that had an average grade level score < 10.0. 
  • Suitability Assessment of Materials (SAM) – We utilized the SAM to review the materials’ quality. After conducting a small pilot evaluation with a subset of the materials, we modified the measurement tool to be more applicable to this project, including reordering items, deleting items that were not relevant, and combining items that measured similar domains. The final evaluation tool consisted of 21 items that assessed materials’ purpose, scope, actionability, writing style, vocabulary, numerical literacy, graphics, and layout. Two reviewers independently completed the modified SAM for each material. 
  • Overall Rating – Reviewers assigned a score between 1 (poor) and 5 (excellent) to assess overall quality for each material, providing additional context to account for materials that had assets or flaws that weren’t sufficiently captured by the SAM. 
  • Favorites – The development team selected materials that we thought offered the best combination of readability, formatting, style, and medical accuracy. The designation of “Favorite” was based on objective measures (e.g., reading level), as well as the opinions of our internal experts in health literacy, communication, and CRC screening.