A Review of Guideline Dissemination in the United States
Guideline Length and Number of Recommendations
Indico Solutions undertook an analysis of more than 60 guideline development organization (GDO) dissemination efforts. The analysis consisted of a review of developers’ public website and mobile applications to assess the presence and relative strength of 38 guideline dissemination criteria.
Purpose
Effectiveness of guideline dissemination is difficult to assess in that it is both a function of implementation as well as the resonance of the difference between new recommendations and existing practice. Furthermore, dissemination has evolved beyond mere publication, indeed the underlying evidence base and/or systematic review that are the basis for recommendations are rarely published beyond electronic tables or summary articles – there is simply too much data. But with an electronic format, guidelines can be more easily re-purposed for physician, patient and machine level decision support. Initial observations of GDO disseminations efforts showed a transition from whole document to multiple formats, the degree of which this analysis attempts to quantify.
Introduction
The National Academy of Medicine’s 2011 “Clinical Practice Guidelines We Can Trust” [1] report highlighted the importance of several domains for effective dissemination and implementation of clinical guidelines. We analysed thirty-eight metrics across four dissemination domains: Web Site related, Mobile, Search and Guideline Collateral. Within each of these domains we tracked several factors that might illustrate the robustness of effective dissemination.
There are examples of excellent dissemination by several GDOs, however it appears on average, easier searching within the guidelines is a common area for improvement in guideline dissemination, this may be especially important for GDOs with larger libraries. Guideline integration into other platforms and applications is likely a good barometer of dissemination robustness and while not extensively reviewed here, mobile applications are a proxy of this robustness especially at the point of care.
Methodology
A number of steps were undertaken in this benchmarking analysis. First, a list of guideline developers in the United States was developed. This list began with a review of medical society’s websites to determine if they developed treatment guidelines or similar documents. Sixty-two medical societies were identified. Additionally, one allied health and two nursing societies were added to the list as they were active in guideline development.
Second, a list of metrics that GDOs would be measured against was developed. This was accomplished by reviewing the National Academy of Medicine’s 2011 report and relevant medical literature for suitable metrics which could be assessed remotely: a list of thirty-eight metrics was established. Examples of metrics include the presence of a mobile application, was the content fully available, was keyword searching possible on the guideline content.
The metrics were then assigned two weighting scores, one for dissemination to organizations (e.g. hospitals) and the other for dissemination to individuals. These weightings were assigned to each metric based on how important the metric was in relation to organizational dissemination and individual dissemination. Organizational dissemination ratings look at factors which assist organizations (e.g. hospitals) ability to disseminate guidelines to the relevant persons within that organization. Individual dissemination ratings look at factors that facilitate guideline access at the point of care. The full list of metrics and their relative weighting is outlined in Appendix 1.
For the purposes of analysis, the metrics were assigned into five domains (Table 1). The domains were created as a means of looking at the various types of guideline dissemination and the methods in which clinicians would attempt to access the guideline content.
Table 1: Guideline Dissemination Metric Domain Definitions
| Domain | Details |
| Website dissemination | Metrics that cover the dissemination of guideline content through a website. |
| Mobile dissemination | Metrics that look at the dissemination of guideline content in a mobile environment specifically. It does not include web enabled access via websites |
| Search features | Overview of various search metrics for the guideline content. |
| Guideline Collateral | Looks at guideline collateral has been developed to assist with guideline dissemination. Examples include slide sets, patient education material and summary documents. |
| IOM Metrics | Various metrics highlighted by the IOM report of 2011 that were thought to be important in guideline dissemination, that did not fit into one of the above domains. |
At the time of reviewing each society’s results, a random sample of 3-5 guidelines was taken to obtain an average number of pages and recommendations per guideline. This was done to ascertain if the length of guidelines, or the number of recommendations has any correlation with the dissemination metrics.
For each society, a review of their dissemination strategies was undertaken, and results were recorded for analysis. Based on a societies organizational and individual dissemination score we allocated them into one of four groups. Table 2 shows how each group was allocated.
Table 2: Dissemination Groupings
| Group | Score (organizational, individual and combined) |
| Basic | Below 25% of total score |
| Effective | Between 25% and 38% |
| Advanced | Between 38% and 55% |
| Exceptional | Above 55% |
Descriptive analysis was done on the results obtained and these are outlined in the results section below.
Results
These data demonstrate considerable variability in the organizational and individual dissemination robustness of guideline developer organizations throughout the United States. The full results of this analysis for each metric are shown in Appendix 2. The key findings are described below.
Figure 1 – Organizational Dissemination Score and Point of Care Score by Overall Rating
Figure 1 highlights the variability in guideline dissemination with respect to point of care and organization dissemination and a good correlation between the two (r=.76, p<001). Note that this analysis is independent of guideline quality, methodology or “trustworthiness”, it is a measure of the extent of dissemination options that factor into their implementation.
We found that 21 (31%) societies have Exceptional or Advanced overall dissemination implementations (Figure 2). Typically, these societies had guideline content available in mobile applications and had some form of dedicated guideline searching. Guideline collateral was a feature of the high performing GDOs as was full text available in formats other than PDF documents. This is important as PDF documents have limited utility in the clinical setting and this limit the use of guidelines at the point of care. The remaining 44 (69%) societies had PDF documents as the main dissemination option or had content that was restricted to members/paid subscriptions only.
Figure 2 – Percentage of Society Overall Dissemination Rating Group
Organizational Dissemination
Figure 3 shows the number societies who meet key metrics that significantly influence the ease of dissemination in the clinical setting. Several societies had their websites set up for specific searching of guideline content. Mobile applications have been developed by 21 (33%) of society’s most of which were freely available. Some of these mobile applications were the society journal publisher applications with freely available guideline content. However, most applications were dedicated to guideline content dissemination.
Figure 3 – Percentage of Societies with presence of key metrics
Another key finding of the analysis was that less than 10% of medical societies demonstrated evidence of guideline integration into third party software (E.G. EMR systems) through the use of technology solutions. The modern healthcare setting is increasingly demanding that clinical guidelines be integrated into EMR systems either as part of a clinical decision support system or as part of an Appropriate Use Assessment.
As it was not possible to review any such integration first hand this metric was assessed by looking for any indications on society websites of any such integration having occurred. Additionally, if a societies dissemination was limited to PDF documents it was inferred that third-party integration had not occurred. Only 1 society demonstrated evidence of guideline integration into third party systems.
Individual Dissemination
Analysis of individual dissemination metrics showed that 42 (65%) societies have effective or above systems to improve personal implementation of guidelines at the point of care. Central to this group of societies was the presence of a mobile application (60%), full content in formats other than PDF (52%) and guideline summaries (55%).
Dissemination Domains
Analysis of the metrics by domains the average score for each domain was less than 50%. The development of guideline collateral, including guideline summaries was the best domain with an average of 44%. For societies with Advanced or Exceptional overall dissemination scores all domain scores improved as seen in Figure 4.
Figure 4 – Average Scores Across Domains*
*White line shows average scores for societies with Advanced or Exceptional Overall Dissemination Score.
Guideline Length and Number of Recommendations
There was wide variability in the average length of guidelines and the number of recommendations contained within each guideline. Looking at the average number pages and recommendations per guideline it is interesting to match this with the presence of guideline summary documents and the presence of PDF only documents. Close to 50% (31) of medical societies had an average page length of more than 20 pages per guideline. Within this group 14 societies had no guideline summaries and PDF only content. This sub-group represents guideline developers for whom dissemination is less than optimal (Table 2).
Table 3: Guideline Summary and PDF content by Average Guideline Length
| Average length of guidelines | Societies with a Guideline Summary (%) | Societies with PDF Only Content (%) | ||
| Yes | No | Yes | No | |
| Less than 20 pages | 11 (32%) | 22 (65%) | 12 (35%) | 21 (62%) |
| 20 or more pages | 13 (41%) | 17 (55%) | 20 (65%) | 10 (33%) |
Discussion
This benchmarking analysis shows that many societies have advanced their dissemination strategies beyond the traditional method journal publication. Most societies provided more than one method of consuming the guideline content, which is the cornerstone of successful guideline dissemination in the modern healthcare setting. It is important to note that this analysis is independent of guideline quality, methodology or “trustworthiness”, it is a measure of the extent of dissemination options that factor into their implementation.
The results demonstrate the key criteria that affected the scores of all societies were
- the presence of a mobile solution for guideline content,
- guideline content specific searching,
- additional guideline collateral content and
- the presence/absence of non-PDF formats.
While identifying underlying technologies (vendor or in-house supported) that support these guideline dissemination efforts is beyond this analysis, clearly PDF linking is a minimally viable approach to dissemination. This seems to be especially true for organizations with bigger libraries of longer documents (and more recommendations).
Structured guideline representation is key to advanced dissemination practices whether on societies web pages via HTML encoding or in mobile applications that have many potential solutions. Unclear in dissemination are the roles of key personnel (project manager, guideline directors, panel chairs etc) in dissemination design and workflows. While recommendations and documents are generally finely nuanced consensus documents these distinctions may be lost in online representations where the audience has minimal searching and reading time. Accordingly, future consideration by GDO’s should be given to which dissemination methods will be translated into recommendation updates at the point of care. It was noted that several GDO’s craft and approve recommendations prior to document drafting, a practice that lends itself to structured documents and easier dissemination.
It can be argued that estimates of guideline integration into other IT solutions are speculative however since most GDO’s missions are to enhance patient outcomes claims of integration into the major EMR platform should be prominent. We found minimal evidence of such integration, however it may be that integration occurs via reference in third party content aggregators where the needed key word and clinical coding for machine level searching occur. There are two possible reasons that we found limited evidence of this type of integration. One, it may be that consideration has not been given to how such integration could be achieved. Alternatively, the process of structuring the data to allow integration is both time consuming and beyond a GDO’s budget.
Conclusion
The results of this benchmarking report found that 29 (44%) of GDO’s have dissemination is less than optimal. This group consists of GDO’s whose content is limited to PDF only, have no dedicated searching and no guideline summaries available for use at the point of care. Of this group only 8 (12%) GDO’s had a mobile application for their guideline content.
Conversely, 16 (25%) had full content in formats other than PDF documents with guideline summaries. Within this group 6 had dedicated search features and 8 had a mobile application. Two societies had all of these dissemination metrics giving rise to an Exceptional overall dissemination rating.
Guideline dissemination in the United States demonstrates many societies with multiple dissemination pathways. Further dissemination will be achieved with GDO’s making investment to improve the underlying structure of the guideline content which can facilitate dissemination by both organizations and individuals to the point of care.
Opportunity for Improvement
Using the Indico Solutions Platform Dissemination, most societies in the basic, effective and advanced groups can improve the reach and effectiveness of their guideline dissemination. The results of this analysis suggest that the following benefits of Indico Solutions Dissemination Platform will be of value to many guideline developers in the United States.
- Indico Mobile applications (Android and Apple) customized to your organization
- Dedicated guideline search page to host within your web environment or hosted by Indico
- Specialized medical synonym and keyword searching of guideline content
- Recommendation only searching
- API framework to import your updated guidelines into your existing mobile applications
- Specific full text searching of guideline content
- API framework for integration of recommendations into clinical decision support system
- Guideline delivery formats beyond PDF
Support for structured documents and discrete recommendation statements is key, allows proprietary keywording, standard code linking (ICD, CPTTM, HCPCS, NDC etc) that aids both manual and machine searching.
Appendix 1
| Metric | Description | Organizational Dissemination Metric Weighting | Individual Point of Care Metric Weighting |
| Website Domain | |||
| Society site includes guidelines | Guidelines were developed by the organization | 6 | 6 |
| Member Restrictions | Is content freely available or restricted to members | -1 | -1 |
| HTML Full Guideline Content | Was the full content available in non PDF formats | 1 | 3 |
| Guideline Summary | Were guideline summaries available | 2 | 4 |
| Was full content in PDF format only | -2 | -3 | |
| Journal Link | Was full content a link to Journal Publication | 1 | 2 |
| Journal Article Freely Available | Was Journal publication content freely available | 1 | 1 |
| Mobile Ready Site | Was society site easily displayed on mobile devices | 1 | 4 |
| API feed guideline content | Does an application programming interface exist for guideline content | 7 | 9 |
| AUC | Were Appropriate Use Criteria developed | 5 | 3 |
| Site Total | 21 | 28 | |
| Guideline Search Domain | |||
| Guideline Specific Search | Limiting searching to only guideline content as opposed to any document or page | 4 | 5 |
| Title Search | Did site searching include guideline titles | 1 | 1 |
| Keyword Search | Did site searching include keywords | 2 | 3 |
| Synonyms | Did site searching include synonyms | 2 | 3 |
| Text Search | Did site searching include guideline content | 3 | 2 |
| Recommendation specific search | Did site searching include limiting results to recommendations | 4 | 7 |
| Search Total | 16 | 21 | |
| Guideline Collateral Domain | |||
| Clinical Summary | Was a guideline summary available | 6 | 3 |
| Slides | Were slide sets summarizing guideline content developed | 1 | 0 |
| Pt Summaries | Were patient summaries of the guideline developed | 1 | 2 |
| Videos | Were videos outlining guideline content developed | 1 | 1 |
| Press Releases | Did guideline developer issue press releases for newly published guidelines | 1 | 0 |
| Quality Measures | Did guideline developer produce companion quality measures for any of the guidelines | 1 | 0 |
| Guidelines.gov Presence | Were any guidelines published in the National Guideline Clearinghouse | 1 | 2 |
| Choosing Wisely campaign | Did the society participate in the Choosing Wisely campaign | 2 | 2 |
| Collateral Total | 14 | 10 | |
| IOM Metric Domain | |||
| structure the format, vocabulary and content | Is the guideline developed in a manner to facilitate implementation (as per recommendations of IOM report) | 3 | 5 |
| active implementation | Have active implementation strategies been implemented | 1 | 0 |
| Algorithms | Have Algorithms been developed to assist clinicians ant point of care | 4 | 5 |
| Cross Guideline Searching | Do guideline recommendations link to other relevant guidelines | 1 | 3 |
| Link to CME | Is guideline linked to CME credits | 5 | 3 |
| Mass Media | Have mass media strategies been implemented | 1 | 0 |
| Tools for ‘Academic Detailing’ (education) | Have tools to facilitate “teaching” of guideline content been developed | 5 | 0 |
| IOM Metrics Total | 20 | 16 | |
| Mobile Application Domain | |||
| Has Mobile App | Does the society have a guidelines mobile application | 2 | 8 |
| Free or Paid or members | Is content freely available or restricted to members | 1 | 5 |
| Structured for mobile content | Has the guideline content been reformatted to improve dissemination at the point of care | 1 | 3 |
| PDFs in App | Is the application a collection of PDF’s within application | 0 | 0 |
| Link to Journal Article | Is the mobile application a link to journal articles | 1 | 0 |
| Modified Journal App | Is the mobile application a publishers application with guideline content freely available | 1 | 0 |
| Mobile Total | 6 | 16 | |
| Total | 77 | 91 | |
Appendix 2
| Metric | Number of Societies | Society (% total) |
| Member Restrictions | 10 | 15.4 |
| HTML Full Guideline Content | 14 | 21.5 |
| Guideline Summary | 24 | 36.9 |
| 41 | 63.1 | |
| Journal Link | 38 | 58.5 |
| Journal Article Freely Available | 40 | 61.5 |
| Mobile Ready Site | 28 | 43.1 |
| API feed guideline content | 1 | 1.5 |
| AUC | 11 | 16.9 |
| Guideline Content Specific Search | 16 | 24.6 |
| Title | 57 | 87.7 |
| Keyword | 15 | 23.1 |
| Synonyms | 1 | 1.5 |
| Text | 28 | 43.1 |
| Recommendation specific search | 1 | 1.5 |
| Clinical Summary | 24 | 36.9 |
| Slides | 10 | 15.4 |
| Pt Summaries | 19 | 29.2 |
| Videos | 9 | 13.8 |
| Press Releases | 47 | 72.3 |
| Quality Measures | 36 | 55.4 |
| Guidelines.gov Presence | 48 | 73.8 |
| Choosing Wisely campaign | 39 | 60.0 |
| structure the format, vocabulary, and content | 46 | 78.1 |
| active implementation | 2 | 3.1 |
| Algorithms | 18 | 27.7 |
| Cross Guideline Searching | 1 | 1.5 |
| Link to CME | 6 | 9.2 |
| Mass Media | 1 | 1.5 |
| Tools for ‘Academic Detailing’ (education) | 2 | 3.1 |
| Has Mobile App | 26 | 52.0 |
| Free or Paid or members | 0 | 23.6 |
| Structured for mobile content | 17 | 26.2 |
| Link to Journal Article | 7 | 10.8 |
| Modified Journal App | 2 | 3.1 |
[1] IOM (Institute of Medicine). 2011. Clinical Practice Guidelines We Can Trust. Washington, DC: The National Academies Press.