Patient Surveys Are Changing As Everyone Moves Online

The U.S. Department of Health and Human Services (HHS)’ Centers for Medicare and Medicaid Services (CMS) utilizes and administers the family of Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey protocols as the gold standard for measuring and reporting on patient experience (PX) in the U.S.

Download our new paper for insights into how to ensure accurate data using online surveys to supplement quality metrics.

---

INTRODUCTION

The U.S. Department of Health and Human Services (HHS)’ Centers for Medicare and Medicaid Services (CMS) utilizes and administers the family of Consumer Assessment of Healthcare Providers and Systems (CAHPS) survey protocols as the gold standard for measuring and reporting on patient experience (PX) in the U.S.

The Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS), one protocol in the CAHPS family, is a vital resource for promoting patient-centered care and improving overall care quality. Many stakeholders rely on the rigor and reliability of HCAHPS data, building its PX measures into performance-based reimbursement, board certification and licensing, and practice recognition programs.

HCAHPS is currently a paper-based survey, and its importance in improving overall healthcare quality emphasizes the need to continuously evaluate its methodological and analytical underpinnings. In parallel, technology has evolved to the point that survey research and healthcare fields can implement online and other technology-centered modes (text, etc.) to modernize and streamline data collection and analysis

To that end, CMS must carefully consider how to adopt online modes and other technological improvements into its protocols—a practice now common across survey research and healthcare fields.

Adopting online methodology into the HCAHPS protocol has become an important trend in the survey research field and the healthcare industry. There is a pattern of continuously declining response rates in traditional mail, telephone, and in-person studies (see Figure 1). Many large-scale national survey programs, including HCAHPS, have observed year-over-year response rate decreases, which suggests that this trend is due to a shift in people’s willingness of taking surveys or a shift in their preferences for how they want to respond, as opposed to program-level weaknesses in outreach efforts. 

Moreover, healthcare institutions have increasingly embraced technology to deliver care and better understand the needs of their populations. This is evident with the rise of electronic health records, health information exchanges, telehealth and data dashboards. Healthcare institutions are modernizing their operations and— in the case of telehealth—learning how to interact with patients to be as accessible as possible. These modernization efforts have also changed how healthcare institutions gather survey data, as many hospitals are using online surveys to supplement their HCAHPS data and quality metrics.

In short, online surveys will continue to grow in prominence and importance, so it is up to CMS to both adopt and thoughtfully consider the implications of incorporating this mode into the HCAHPS protocol.

In response to this need, Ipsos has highlighted four primary considerations as CMS works to understand the implications of adopting online modes of data collection into HCAHPS. Modernization efforts can disrupt normal operations and the quality of data and outputs. In the case of introducing an online mode to HCAHPS, this can potentially result in sharp changes to trend data due to mode rather than performance, leading to distrust in the insights that can be gleaned from them.

In the following sections, we highlight four primary considerations for implementing HCAHPS online so that CMS can maintain its status as having the most rigorous, reliable metrics for capturing patient experience in the U.S.

CONSIDERATION 1:

Ensure HCAHPS surveys are accessible by everyone

In thinking about the quality of HCAHPS survey results, consider employing the Total Survey Error concept, a framework where we can classify the different potential sources of error. As alternative research designs are evaluated, the cumulative sources of error can be minimized. Applying this paradigm, we can identify two types of accessibility-related errors as the HCAHPS programs consider how to adopt an online approach:

• Coverage error, if the sample frame excludes respondents without email addresses; and
• Nonresponse error, if the response patterns of individuals who receive email survey invitations are different from other modes of survey invitations.

Where possible, actions to mitigate these potential accessibility challenges should be undertaken to ensure the accuracy and representativeness of HCAHPS survey results.

Based on our considerations of these sources of error, which are presented in more detail below, Ipsos recommends that an online HCAHPS survey be executed in conjunction with a mail protocol.

To be clear, adding an online option should be embraced as a way to lower barriers for people to provide feedback on their care experiences. The reduction in barriers comes from the convenience of being able to receive and submit surveys anywhere there is internet coverage, as well as being able to forego the more involved processes of sealing and sending back responses via mail or engaging with an interviewer over the phone.

However, offering an online HCAHPS as the sole method of receiving feedback from patients risks excluding respondents who do not have reliable access to the internet. Moreover, online is subject to unique nonresponse factors that would need to be properly accounted for and explored. So, while the addition of online modes can further expand the reach of HCAHPS to incorporate as many patient assessments of care experiences as possible, it must be tested carefully to understand the implications of introducing this modality. We highlight a few factors for CMS to consider as they explore how to use online modes to ensure HCAHPS is as accessible and rigorous as possible.

Reducing Coverage Error

Offering HCAHPS online-only risks systematic under-coverage of underserved populations. The Federal Communications Commission (FCC) reports that nearly 25 million Americans lack stable access to broadband. A significant amount of the U.S. population remains underserved by broadband, and these pockets of inaccessibility include racial and ethnic minorities, people living on Tribal lands, older adults and those with lower levels of education and income. These individuals without internet access may experience care differently than their counterparts, who live in areas better served by internet access (given reduced access to the internet for postdischarge follow-up activities, reduced access to online health information and pre-hospitalization, etc.).

Additionally, their care experiences may differ in ways that weighting and case-mix adjustment may not fully capture. This concern is exacerbated when email is used, as a share of the population with broadband does not regularly use email or may not provide email addresses to healthcare institutions, thus increasing concerns of bias generated by under-coverage in online-only surveys. These issues point to a need to pair an online modality with something like mail, which is less prone to coverage error and has been sufficiently researched to mitigate other sources of error.

Reducing Nonresponse Error

There are unique nonresponse challenges with using email to deliver online surveys that warrant careful consideration. If not properly addressed, the online protocol might lose potential respondents that would have completed the survey otherwise.

First is the challenge of securing respondent trust in email survey invitations. Sections of the population view emails more cautiously as they attempt to discern whether a new message is spam or an attempt at a scam. A professionally presented and sponsored email message is critical to securing trust in its contents.

Testing for variations in these messages and measuring their impact on responses will be just as important as the testing of the actual programmed survey.

Beyond the challenge of securing respondents’ trust in email survey invitations, there can be a mismatch between electronic mail addresses on file at a health institution and those that survey respondents routinely use, resulting in a smaller share of the sample being aware that they were contacted.

For instance, a patient may have an obsolete email address on file with their provider or may rely more on a different one than they gave to their provider. Furthermore, even when the email address is accurate, the recipient may have “junk” email settings that deliver survey communications to the “junk” folder, or respondents may be comfortable working with emails but have lower-level skills when it comes to filling out web-based surveys.

Exploring the extent of these nonresponse issues across healthcare institutions will be important for informing what standards might need to be in place before a hospital can move forward with executing an online modality.

CONSIDERATION 2:

Make sure surveys are optimized and tested for mobile use

Introducing an online version of HCAHPS can help ensure the survey is as accessible as possible for patients. To that end, it is important for CMS to recognize that a large contingent of people complete online surveys on a smartphone or tablet. Currently, over 40% of people in our U.S. general population samples start Ipsos surveys on a smartphone. These Ipsos respondents tend to be younger, have less education, and are more likely to be Black or Hispanic.

As a result, it is critically important for CMS to consider how the HCAHPS survey may be optimized for smaller screens and study the impact that these optimizations might have on response rates and scores. This issue is related to some of the accessibility concerns highlighted in our previous section, but the extent to which people complete surveys on their mobile devices warrants this issue receiving a dedicated discussion.

Ipsos has years of experience conducting research-on-research to determine best practices in designing online surveys. The primary reason we recommended that surveys be tested and optimized for mobile devices is to ensure the online mode is as accessible as possible and that the experience is comparable to other forms of administration.

To that end, we have distilled learnings from over 100 internal experiments and feedback collected from thousands of respondents, and we present recommendations on how to best optimize and test for mobile use. The common themes around these recommendations are to minimize respondent burden, keep respondents engaged and ensure that representative results are collected.

We describe guiding principles to “mobile-first” survey design across three categories: limit cognitive effort, reduce manual effort and construct motivating surveys.

Limit Cognitive Effort

Participants expend cognitive effort in various ways depending on the number, complexity, repetitiveness, or redundancy of tasks that respondents are asked to perform. Using the fewest number of questions possible and having fewer and shorter response categories can significantly reduce cognitive effort. The majority of the current HCAHPS survey utilizes scales with four or five categories. This is favorable based on Ipsos research, which has shown that, in most cases, a scale with more than four categories does not improve the validity of the data. Further, four-category response scales are ideal for a smartphone design.

That said, the HCAHPS Overall Hospital Rating question, which relies on an 11-category, end-labeled scale, could be optimized for smaller screens. Ideally, Overall Hospital Rating would employ a four-category, fully-labeled scale. Not only will fewer response categories require less cognitive effort, but fully-labeled scales are superior to endlabeled scales and have been found to have the highest validity and greater discriminability for survey measurement based on Ipsos research. If the existing 11-category scale must be maintained for trending purposes and comparability, it will be important to optimize its design for presentation on smaller screens. Limiting the degree of vertical scrolling will be key.

Reduce Manual Effort

Survey designers create manual effort for respondents when they fail to minimize the number and difficulty of manual tasks required to complete the survey. To address this, our research-on-research revealed that surveys should not have horizontal scrolling, should present questions on as few screens as possible and should use clickable cells/radio buttons for response entry rather than use more taxing approaches (e.g., drop-down boxes). These factors can reduce occurrence of survey drop-offs and improve the overall quality of survey responses.

It is also important to allow respondents to complete a survey on their preferred device. Ipsos research has found that fewer than 10% of respondents will switch from their initial device of choice to another device when requested by researchers. Additionally, if people are asked to complete the survey on a non-preferred device they are less likely to start the survey in the first place. Asking respondents to take the survey on a non-preferred device risks reducing the representativeness of the sample and increasing bias.

Furthermore, it is vital that all questions and responses are presented consistently across all devices—on large or small screens. Respondents will want to use their devices with the screen orientation they prefer. Approximately 90% of those taking a survey on a smartphone will start the survey in portrait mode and less than 20% will take the time to switch to landscape when asked. Thus, the survey needs to be optimized for both orientations.

Construct Motivating Surveys

Generally, improved response rates and responses occur with shorter, well-designed surveys and interesting topics. This point is especially pertinent when considering guidance on any HCAHPS supplemental questions.

Some health agencies and institutions add custom survey questions at the end of HCAHPS surveys to gather additional feedback from their patients. These include large Federal agencies, such as the U.S. Department of Veterans Affairs (VA), Veterans Health Administration (VHA) and the U.S. Department of Defense (DoD), Defense Health Agency (DHA), who utilize the HCAHPS surveys and protocol and also administer custom survey questions.

While CMS does not provide guidance on this front, detailing some requirements might be helpful with the tenets outlined at the beginning of this section—reducing burden and increasing engagement to attain representative findings. The number of supplemental items should be kept to a minimum in order to reduce survey length and subsequently maintain motivation. These items should similarly follow the recommendations outlined for reducing cognitive effort and reducing manual effort.

CONSIDERATION 3:

Review and update current adjustment algorithms to account for any potential interactions between patient-mix factors and survey mode

Current HCAHPS protocols include mail-only, phone-only, interactive voice response (IVR)-only and combined mail-telephone approaches. People of different ages, education levels, language proficiency, cultural groups, gender and type of healthcare admission may answer questions in the survey very differently. Moreover, response rates for each of these modes may differ greatly by these demographic groupings. These reasons drive the need for adjustments to be applied to HCAHPS data in order to help prospective patients make informed judgments about their care.

CMS applies a mode and patient-mix adjustment (PMA) algorithm to PX data with the goal of creating hospital scores that are based on each hospital’s mix of patients and that account for differences in scores resulting from the mode of survey administration.

The current algorithm first adjusts a hospital’s score by their mix of patients using age, education, gender, language, race, response propensity level, self-reported health status, self-reported mental health status and the proportion of a sample receiving either medical, surgical, or obstetrics care. A hospital’s rates for each adjustment factor are compared to its national rates published by CMS. After scores are adjusted for their mix of patients, the approach adjusts scores for the mode of the hospital’s surveys.

Introducing a digital response option complicates how CMS and CAHPS vendors will need to equalize impact from the mode of survey administration and mix of patients. It will also make examining trends across time very difficult.

A real complication, which we have covered in the last section, is that people can access the internet using different technologies and devices. These include desktop computers, laptop computers, tablets or smartphones. Preference for the type of technology to use and comfort levels with using them may be related to age, education and other aspects of individuals. Ipsos recommends that CMS consider prospective respondents’ comfort with technology and device type when considering digital modes for HCAHPS. This would likely necessitate a question specific to the online modality to capture comfort with technology data.

An even greater consideration is that the current two-step adjustment process—to sequentially adjust for a hospital’s patient-mix and then adjust based on the mode of survey administration—may not be appropriate, particularly when assessing the impact of adding online modes. This approach implicitly assumes there is no interaction between the type of person; their preference for responding by mail, telephone or digitally; and their responses to the survey questions. An assumption of no interaction or no evidence of an interaction between the respondent, response propensity, responses and mode questions may have been acceptable for the seminal 2009 paper on mode and patient-mix adjustment, but this may no longer be valid today.

Ipsos foresees the need to test and develop more complicated models to account for the current adjustment factors—age, education, gender, health status, mental health status, language proficiency accounting for ethnic cultural differences, and service line—and their interactions with the mode of survey administration, with paper, telephone and digital modes. We expect intensive investigations will be necessary to examine the interactions for modes accounting for patient-mix and mode together.

The updated adjustment investigations should look into including the new models with the old for trending purposes. Many healthcare institutions track HCAHPS data trends to track the impact of newly implemented quality improvement programs and stay abreast of their patient experience performance. As we present in the next section, trends and comparisons to baseline performance are integral to how reimbursements are determined under current value-based programs.

As such, it is important to provide guidance on how institutions can maintain current trends and differentiate the impact from a move to online versus a true change in patient experience. Ipsos recommends conducting bridge studies between the existing method and the new method, completed concurrently, to understand the implications of any data outreach change and adjust accordingly. This can be done across hospitals (where a set of hospitals executes one mode and another set tests the alternative) or within hospitals (where patients at a single hospital are split into two groups that each receive the survey through a different mode).

In short, it will be important to set-up and execute adjustment investigations correctly as well as an HCAHPS digital pilot project to properly assess the impact an online mode might have on data trending.

CONSIDERATION 4:

Consider exempting the impact of the Person and Community Engagement domain on value-based reimbursement determinations while online HCAHPS is being assessed

HCAHPS is an integral component in calculating Medicare reimbursement for hospitals under the Value-Based Purchasing (VBP) Program. The VBP program plays a critical role in evolving the overall healthcare system to one that rewards providers for the quality of care they provide rather than the quantity of services they provide. Eight HCAHPS measures are used to calculate the Person and Community Engagement domain score, which represents 25% of a hospital’s Total Performance Score (TPS) that is used to ultimately determine how their reimbursement is adjusted based on quality. Introducing an online modality to the HCAHPS protocol, as we have presented in the previous sections, will likely impact HCAHPS scores in such a way that mode adjustments and trending adjustments would need to be fairly considered. These expected changes to HCAHPS scores will similarly require careful consideration in how VBP reimbursement calculations should be adjusted. There are two principal methodological concerns:

• How to address the VBP Improvement Points calculation to account for HCAHPS score deviations from the baseline performance of individual hospitals due to the change in mode; and
• How to address the current Performance Standards that are used to calculate Achievement Points to account for the standards being developed based on HCAHPS data that had not implemented an online modality.

In addition to these specific VBP methodological concerns, there is also the reality that hospitals nationwide are likely to differentially adopt online HCAHPS as they consider the infrastructure, technology and process-related investments needed to embrace that system. This might entail a multi-year process of planning, implementation and revisions on the part of the hospitals implementing this modality which would result in a potentially volatile set of Performance Standards if not properly controlled. In short, the VBP reimbursement formula’s reliance on baselines and scoring based on relative performance to those standards needs to be carefully considered with the addition of an online mode to HCAHPS. 

Our own research experience suggests that there might not be major differences in response patterns when comparing equivalent populations taking a survey online versus on paper. Rather, any differences that might emerge are more likely to be due to changes in the patient-mix of those returning surveys thanks to the greater level of accessibility offered to provide feedback. This, in theory, will be largely controlled for by current patient-mix adjustment formulas.

Nevertheless, as we presented in the previous section, it is important to re-assess this assumption and test more complicated models that account for interactions between the currently used patient mix adjustments and the mode of the survey.

One important point to keep in mind is that the HCAHPS scores that feed into the Person and Community Engagement domain constitute only 25% of the TPS of hospitals, meaning that we do not expect wild variation in TPS scores to occur from the historic performance of hospitals with this new mode, and we expect the impact on VBP to be minimal. Even with this expected outcome, however, we recommend that CMS offer some level of flexibility on implementing digital HCAHPS under VBP during this online transition period. There will be some level of apprehension from hospitals that might avoid online implementation if they believe it will negatively impact their performance (and thus, their reimbursement).

SUMMARY

Ipsos believes that the trend of decreasing response rates across mail, telephone and in-person modalities, as well as the healthcare industry’s continued incorporation of technology to engage with patients and gain efficiencies, point to the need to thoughtfully adopt an online HCAHPS mode into current protocols.

Moreover, adopting an online modality in conjunction with current modes will allow for a more accessible means for patients to provide feedback on their care experiences. This can ensure that as many patient voices as possible are incorporated into assessments of patient experience performance, which can strengthen the rigor of HCAHPS by improving its representativity. These positive results from embracing a digital mode for HCAHPS must be carefully balanced with prudent design, development and testing to mitigate the possible complications previously presented. We encourage CMS to consider the following points:

• Ensure HCAHPS surveys are accessible by everyone. This can be accomplished by offering the online HCAHPS in conjunction with the mail protocol, carefully assessing the extent that online-specific nonresponse factors exist and developing mitigating strategies to address them, as well as ensuring that online platforms support accessibility features and various technology platforms.
• Make sure surveys are optimized and tested for mobile use. As more individuals take online surveys on smartphones, it is important to ensure that mobile technology has a large role in any planned pilot testing. A mobile-first design philosophy should review how questions are asked to limit cognitive effort, review how questions are displayed to reduce manual effort, and ensure that appropriate guidance is provided to keep surveys short and engaging.
• Review and update current adjustment algorithms to account for any potential interactions between patient-mix factors and survey mode and to ensure trend data can be maintained. The current adjustment approach implicitly assumes there is no interaction between the type of person; their preference to responding by mail, telephone, or digitally; and their responses to survey questions. We foresee a need to test and develop models to account for how current patient-mix adjustment factors might interact with the survey mode of administration. Moreover, these models and any testing of the online mode will also need to consider how trending with past data can be maintained.
• Consider approaches to minimize the impact of the Person and Community Engagement domain on value-based reimbursement determinations while online HCAHPS is being assessed. Roll-out of an online mode will result in differential uptake by hospitals, which could lead to deviations from performance benchmarks that cannot be solely attributed to performance. We recommend that any online mode implementation carefully consider how to incorporate the Person and Community Engagement domain until baseline measurement periods are inclusive of data collected with this new mode.

In closing, an online mode of administration for HCAHPS has many benefits for improving the accessibility of patient experience surveys. However, the incorporation of an online modality must be thoroughly evaluated and tested in order to ensure the validity and tradability of the data.