PROMS+PREMS Pilot Project

Purpose | Benefits of Participating | Expectations | Domains and Measures | Assessment and Lessons Learned | Next Steps

Integrating the voices of individuals with lived experiences is a cornerstone in quality improvement. Thus, it is vital to routinely assess the quality of HIV outcomes and experiences with the healthcare system using the insights of the clients we serve. As emerging topics in the field of performance measurement and quality improvement, Patient-Reported Outcome Measures (PROMS) and Patient-Reported Experience Measures (PREMS) have value as additional measurement metrics to allow HIV providers to utilize these findings to continuously improve patients’ health outcomes and care/treatment experiences. 

The following definitions may help foster your understanding of PROMS and PREMS:

PROMS are defined as any measurement of patient’s well-being or function, which can only be determined by directly asking the patient. These are measured using standardized, often validated questions or instruments. Examples include depression, anxiety, pain, fatigue, etc.
PREMS are measures of the patient's personal experience of the healthcare they have received. These include respect, communication, privacy, engagement in shared decision-making, as well as the environment in which care is delivered.

To this end, the Center for Quality Improvement & Innovation (CQII) worked in close collaboration with the Institute for Healthcare Improvement (IHI) and its federal funder, the HRSA HIV/AIDS Bureau, to support an initial 10 sites (two sites were part of the same cooperative and were consolidated into one and one site dropped out due to competing priorities) participating in the PROMS+PREMS Pilot Project, starting in January 2022 and concluding in June 2022.

Purpose of the Pilot Project

Participating agencies in the PROMS+PREMS Pilot Project explored the value of these measures as vital data sources in HIV care and their feasibility for integrating the results in ongoing QI activities. The aims of the pilot project were to

Identify best practices to measure PROMS and/or PREMS in HIV ambulatory care setting as additional QI measures; and
Optimize agency-level strategies in the use of PROMS and/or PREMS data results to improve HIV care and complement ongoing QI efforts.

CQII’s expectations were that that the findings of the pilot project will assist national efforts to increase the awareness of PROMS and PREMS among RWHAP providers and to further empower HIV providers to utilize their findings to improve HIV health outcomes and patient experiences in HIV care.

CQII evaluated all aspects of the pilot project—using both quantitative and qualitative data.

Benefits of Participating in the Pilot

‘’We joined the pilot because we felt it would be good for our clients in the long run. Getting the patient perspective is important and we hope that it will allow us to provide even better care to our clients.’’ - PROMS+PREMS Pilot Project Participant

Participation in the PROMS+PREMS Pilot Project allowed for: Opportunities to jointly explore this emerging QI topic with CQII staff and consultants.
Access to national content experts from IHI and their coaching support and technical assistance.
Access to standardized PROMS and PREMS tools and measures for use beyond the pilot project including sample measures, topic-specific literature review, findings from a CQII/IHI Expert Meeting, and focus groups with RWHAP providers; and
Contributions of lessons learned documented in a PROMS and PREMS implementation guide.

Expectations

Starting in January 2022, each participating agency was asked to complete the following activities:

Participate in monthly one-hour webinars with presentations by national experts and updates by other PROMS+PREMS Pilot Project participants.
Participate in monthly coaching calls to provide one-on-one support to guide implementation. Implement at least one PROMS or PREMS in the agency.
Use the results in current QI activities.
Contribute to the development of the PROMS and PREMS implementation guide by identifying best practices.

PROMS and PREMS Domains and Measures

“Within our jurisdiction there were many providers that were not familiar with PROMS/PREMS but now there is a lot of interest. When you look at issues like quality of life, everyone is interested, especially frontline providers”. — Pilot Project Site

As CQII began exploring ways to support RWHAP providers in incorporating patient-reported outcomes and experiences, it became clear that many providers are already collecting patient-reported data and using them in their QI activities. In response, CQII sought to identify priority areas, also referred to as domains, that could be incorporated into ongoing QI activities. This approach was designed to allow RWHAP providers to build on existing efforts, utilize established data collection processes, and gradually build a more robust use of patient-reported outcome and experience measures.

The 2021 CQII/IHI Expert Meeting suggested domains participants could use for the Pilot Project. It is worth noting that these were not the only domains participants could use as the characteristics of the organization and their patient population would drive what issues to prioritize. Prior to selecting a PROM/PREM domain pilot participants were advised to consider whether they are:

Relevant – are the PROM/PREM domains relevant to the patients they serve? Do they generate the necessary momentum and interest among staff?
Measurable – to the extent possible, can existing data collection system be utilized?
Actionable – Does the organization have the capacity to address the issue that will be identified?
Sustainable – Can data be collected over time and integrated in ongoing QI efforts?
Tailorable – Do potential findings allow considerations of differences within the organization, patient population, geographic region, etc.

By February 15, 2022, pilot participants had selected their PROM/PREM domain. The table below list the sites and selected domains.

Pilot Site	Selected Domain	PROM/PREM
AIDS Ministries	Housing Stability	PROM
AIDS Project of the Ozarks	Communication	PREM
Centro Ararat	Quality of Life/Well-being	PROM
Cook County HIV Integrated Programs	Housing Stability	PROM
Eau Clair Cooperative Health	Housing Stability	PROM
Harris County Public Health	Quality of Life/Well-being	PROM
University of Pittsburgh Medical Center, Presbyterian Shadyside	Food Security	PROM
Tarrant County HIV Administrative Agency	Quality of Life/Well-being	PROM

In addition to the suggested domains, detailed domain-specific PROMS/PREMS survey questions were provided to each participating team to serve as an initial assessment tool. Participants were free to use all or some of the questions. Some questions do not constitute a validated measure, but they were drawn from existing measures.

The table below provides examples of PROMS domains and sample questions shared during the CQII/IHI Expert meeting.

Patient-reported Outcome Measures [PROMS] Domains
Priority Domain	Why is Important	Example of Validated Tool Measures/Questions
Quality of Life/Well-being (e.g., overall health [not related to related to HIV], quality of life, impact of side effects, issues such as loneliness)	Research suggests that physical symptoms, ART, psychological well-being, social support systems, coping strategies, spiritual support, and psychiatric comorbidities are important predictors of quality of life for people with HIV	100MLives Well-being Assessment (validated) Twelve (12) question survey plus demographics Sample Questions/Answers Imagine a ladder with steps numbered from zero at the bottom to ten at the top. The step of the ladder represents the best possible life for you and the bottom of the ladder represents the worst possible life for you. On which step of the ladder would you say you personally feel you stand at this time? On which step do you think you will stand about 5 Years from now? Now imagine the top of the ladder represents the best possible financial situation for you, and the bottom of the ladder represents the worst possible financial situation. In general, how would you rate your physical health? How would you rate your overall mental health? Note: This validated well-being assessment also has questions related to having a sense of purpose, loneliness, feeling a part of the local community, relationship with family and friends, and positive and negative emotions.
Housing Stability (whether the patient has stable housing, fears losing their housing)	In 2019, almost 29,000 RWHAP clients reported unstable housing. People experiencing homelessness or unstable housing are more likely to delay entry into HIV care, more likely to experience interruptions in healthcare, less likely to be prescribed ART, less likely to reach sustained viral suppression, and overall have poorer health outcomes.	Veterans Administration Homeless Screening Tool Five (5) question survey Sample Questions/Answers: In the past 2 months, have you been living in stable housing that you own, rent, or stay in as part of a household? Are you worried or concerned that in the next 2 months you may NOT have stable housing that you own, rent, or stay in as part of a household? Where have you lived for MOST of the past 2 months? • Apartment/ House/ Room - no government subsidy • Apartment/House/ Room - with government subsidy • With Friend/ Family • Motel/Hotel • Hospital, Rehab Center, Drug Treatment Center • Homeless: Shelter • Anywhere outside, (e.g., Street, Vehicle, Abandoned Building) • Other Would you like to be REFERRED to talk more about your housing situation?
Mental Health (depression, anxiety, substance use)	People with HIV are at increased risk of developing mood, anxiety, and cognitive disorders. Depression is one of the most common mental health conditions facing people with HIV. Some medications used to treat HIV, including ART, may have side effects that affect a person's mental health.	PHQ9 – Depression Severity Ten (10) question survey Sample Questions/Answers Over the last 2 weeks, how often have you been bothered by any of the following problems? • Little interest or pleasure in doing things. • Feeling down, depressed, or hopeless • Trouble falling or staying asleep or sleeping too much. • Feeling tired or having little energy. • Trouble concentrating on things, such as reading the newspaper or watching television. Answers: (Not at all, several days, more than half the days, nearly every day).
Perceived Discrimination (lifetime history of discrimination, daily microaggressions, trauma and re-traumatization)	Research indicates that the number of discrimination events experienced over life was positively associated with the number of HIV-related symptoms experienced.	Everyday Discrimination Scale (Short Version) Nine (9) questions plus a follow-up question on ‘perceived reasons Sample Questions/Answers In your day-to-day life how often have any of the following things happened to you? • You are treated with less courtesy or respect than other people • You receive poorer service than other people at restaurants or stores • People act as if they think you are not smart • People act as if they are afraid of you • You are threatened or harassed
Food Security (lack of food, fear of not having enough food, lack of access to nutritional food)	In 2019, 12.6 percent of RWHAP patients received food assistance (i.e., food bank, home-delivered meals) through the program. Studies indicate that food insecurity can lead to high levels of depression in people with HIV.12 Evidence also indicates lower rates of viral suppression, lower CD4 counts, and poorer health outcomes.	USDA Food Security Brief Survey Six (6) question survey Sample Questions/Answers The food that (I/we) bought just didn’t last, and (I/we) didn’t have money to get more. (Answers: Often true, Sometimes true, Never true, don’t know (DK) or Refused) (I/we) couldn’t afford to eat balanced meals. (Answers: Often true, Sometimes true, Never true, DK or Refused) In the last 12 months, did (you/you or other adults in your household) ever cut the size of your meals or skip meals because there wasn't enough money for food? (Answers: Yes, No, DK or Refused) [IF YES ABOVE, ASK] How often did this happen? (Answers: almost every month, some months but not every month, or in only 1 or 2 months) In the last 12 months, were you ever hungry but didn't eat because there wasn't enough money for food? (Answers: Yes, No, DK or Refused)

The table below provides examples of PREMS domains and sample questions shared during the CQII/IHI Expert meeting.

Patient-reported Outcome Measures [PREMS] Domains
Priority Domain	Why is Important	Example of Validated Tool Measures/Questions
Experience of Racism (has patient ever experienced racism while receiving care in the organization, does the patient feel unwelcome [e.g., none of the educational materials look like me])	Research by the Center for AIDS Research of the University of Alabama at Birmingham found that people with HIV who experienced racism at clinic visits were two times less likely to adhere to ART. The National HIV/AIDS Strategy recognizes racism as a serious public health threat that directly affects wellbeing and drives and affects HIV outcomes.	Krieger Experiences of Discrimination (EOD) scale Thirty (30) question survey (exact number depends on how many “Yes” answers) Sample Questions/Answers: Have you ever experienced discrimination, been prevented from doing something, or been hassled or made to feel inferior in getting medical care because of your race, ethnicity, or color? Answers Yes/No. For answers of “yes,” the follow-up question is: How many times did this happen? Once, two or three times, four or more times In the last year, how much did you worry about your experiencing unfair treatment because of your race, ethnicity, or color? Answers: Most of the time, some of the time, rarely or never How often do you feel that you, personally, have been discriminated against because of your race, ethnicity, or color? Choose the number that best represents how you feel. Answers: Never, rarely, sometimes, often
Respect/Dignity (are patients treated with kindness by all staff, even in a busy clinic, are they valued as a human being)	There are multiple variables related to respect and dignity (effective verbal communication, empathy, common courtesy, respect for privacy, modesty etc.). All these are important for patient-centered care.	CAHPS Home and Community Based Survey with over 100 questions based on experience of community-based care Sample Questions/Answers In the last 3 months, how often did {insert staff role} treat you with courtesy and respect? Would you say: never, sometimes, usually, always. Interviewee would also indicate don’t know, refused, unclear response
Privacy/Confidentiality (importance of privacy can be lost in a busy clinic, even more important in smaller/rural organizations)	Health Insurance Portability (HIPAA) and Accountability Act is not enough. People with HIV emphasize that maintaining their privacy in clinical settings is still an issue. CQII has heard from patients that they often feel that their privacy/confidentiality has been violated by provider staff.	Ontario Outpatient Experience Survey Survey has 63 questions Sample Questions/Answers Were you given enough privacy when discussing your condition or treatment? Definitely For the most part Somewhat Not at all (please tell us more in the open text box at the end of this survey)
Communication (do clinicians [and other staff] take the time to explain, do they allow patients to ask questions, do they recognize the value of questions and provide thoughtful responses [i.e., don’t dismiss questions])	Effective communication between clinician and patient has been shown to positively influence health outcomes, It increases patient satisfaction and supports greater patient understanding of health problems and possible treatments. This contributes to better adherence to treatment and provides support and reassurance to patients.	Ontario Outpatient Experience Survey Survey has 63 questions Sample Questions/Answers Did doctors and/or health professionals ask you what was important to you in managing your condition or illness? Definitely For the most part Somewhat Not at all This was not necessary If you had important questions to ask him or her, did you get answers that you could understand? Definitely For the most part Somewhat Not at all I did not need to ask I did not have an opportunity to ask
Shared Decision-Making (is the patient a partner in care, are their concerns acknowledged and addressed)	Shared decision-making is process in which patients, clinicians and caregivers make treatment and other health-related decisions together based on clinical evidence and reflecting the patient's personal preferences.	Ontario Outpatient Experience Survey Survey has 63 questions Sample Questions/Answers How often, during your most recent visit, were you involved as much as you wanted to be in decisions about your care and treatment? Always Usually Sometimes Never

PROMS + PREMS Pilot Project Assessment and Lessons Learned

Assessment of Pilot Study

Overall, the PROMS+PREMS Pilot Project met its pre-established goals for this national initiative. Several opportunities for improvement and further refinement were noted.

Value of PROMS/PREMS as vital data sources in HIV care

The PROMS/PREMS Pilot Project confirms the value of measuring PROMS and PREMS by HIV providers funded by the RWHAP. Participants saw the vital importance of capturing patient voices to better understand their self-reported outcome and experiences with their HIV programs. This finding is consistent with previously conducted focus groups and national surveys.
Established measurement frameworks should be widely shared to avoid duplication of efforts and expedite the measurement process.
The integration of PROMS/PREMS into existing data systems is critical. Best practices should be shared, and early adopters are critical in this process for national dissemination.
CQII has developed a PROMS/PREMS Guide as an important step to highlight PROMS/PREMS as vital data sources in HIV care. Best practices by pilot site participants are outlined in this resource.

Testing the feasibility of measuring PROMS/PREMS in busy clinic settings

The Pilot Project strongly affirms the feasibility of integrating PROMS/PREMS in busy RWHAP care settings and their integration as additional measurement concepts. Pilot sites welcomed the opportunity to look beyond viral suppression rates as their key performance measure.
Individual support and technical assistance by subject matter and experience experts are needed. Peer learning and exchange provided an important additional source of ideas to overcome measurement and implementation challenges.
Best practices need to be widely shared with other RWHAP sites to assist their efforts into measuring PROMS/PREMS in their HIV clinic environments.

Integrating the PROMS/PREMS results in ongoing quality improvement activities to advance HIV care and services

The PROMS/PREMS Pilot Project had an additional goal to utilize the PROMS/PREMS data results for advancing HIV care and services using QI methodologies and tools. While all pilot sites were able to set up effective measurement systems for PROMS/PREMS and receive results, several sites did not have the necessary time during the six-month pilot to fully integrate the PROMS/PREMS results in ongoing QI activities.
Using the PROMS/PREMS results in QI efforts take time. Implementers need to be reminded that measurement of patient-reported outcomes and experiences alone is not enough to make a difference in patient lives.
Real-world examples how to integrate PROMS/PREMS results in QI activities are needed to assist RWHAP recipients/subrecipients. Providers with successful past experiences should be made visible to other providers to learn from their experiences.

Identifying best practices to evaluate the measurement of PROMS and/or PREMS as additional quality improvement measures

The Pilot Project routinely identified and documented best practices from all pilot sites during each step of the measurement and QI integration process.
The Reflection Sessions - monthly 30-min calls facilitated by CQII to interview and document the experiences of all pilot participants - provided an important source to document best practices. This model was a novel process, which should be adopted in similar projects.
Notes from the Reflection Sessions were routinely shared with all faculty to keep everyone involved in the project well informed on each pilot site.
CQII developed a PROMS and PREMS Guide to increase awareness among RWHAP providers about the purpose of PROMS and PREMS as additional QI measures beyond viral suppression.

Lessons Learned

Pilot participants gained significant insights through the many lessons learned from this feasibility study, which will undoubtably position them to better manage any challenges that may arise as they consider making PROMS and PREMS a permanent part of their performance measurement and quality improvement efforts. Additionally, other HIV care providers will also benefit from the experiences of the PROMS and PROMS pilot participants in their own PROMS and PREMS implementation. Some of the lessons learned are listed below and are also featured in the PROMS+PREMS Guide.

PROMS/PREMS may seem challenging and complex at first, but they can be done. It can be done with basic quality improvement skills

Some participants made 'the perfect the enemy of good’ and made it too complicated than needed. These activities are like other quality improvement projects – think big and start small.

Patient involvement is critical in all steps

People with HIV should be involved in all aspects of planning and implementing PROMS/PREMS, especially in selecting the domain and the related measures.
Inform people with HIV about the reason questions are being asked (i.e., purpose of the survey) and how answering the questions will benefit them and other patients.
People with HIV wants continued feedback. Participating in the PROMS/PREMS survey process and providing their feedback, they are invested in it and want to know their input is acted upon.

It takes a team

Some pilot sites’ projects were initially driven by a single person. As they implemented the project, they quickly learned that it was not possible without the additional support from others in their organization. Especially important was the involvement of staff with direct experiences related to the domain (e.g., if the domain was housing, include the housing case manager).

It is important to communicate clearly and consistently to staff about goals, timeframe, and expected milestones and outcomes. Engagement of staff in this process generates ownership and buy-in.

Use data for PROMS/PREMS selection

Qualitative and quantitative data should be used to confirm the selection of the PROMS/PREMS before starting. Be aware that sometimes staff may perceive an issue that is not supported by data results.

Different domains require different actions

Housing stability requires more intervention than quality of life, which can vary according to site’s resources.
A domain that captures the interest of people with HIV should be selected. There should be a reason beyond the data---will the domain make a difference in their lives and their well-being?
Select a domain that staff are passionate about, since they will be involved in the implementation of the measures and more importantly, making subsequent improvements.
Involve staff representing various disciplines and role (e.g., front desk staff, peer navigators) in all aspects of planning and implementing PROMS/PREMS. Often, engagement by staff creates their ownership in the process.

Test early and often

Selected measures should be appropriate for people with HIV and feasible in existing work plan before incorporating them in any form of EMR.

Data should be actionable

Data collection should be worthwhile for both staff and people with HIV by channeling responses into action.

Acknowledge survey fatigue by both staff and patients

The survey should be kept short. The use of volunteers proved to be a good work around to staffing challenges. Incorporating survey into the people with HIV visit can be strategically significant.

Case Studies and Testimonies

Case Study – Eau Clair Cooperative Health

Eau Clair Cooperative Health in Columbia, South Carolina wanting to formalize and sustain the process of asking patients about their housing status, took an ambitious step by moving towards integrating measure into their EMR so that every patient will be asked about their housing situation. Given the short timeframe of the pilot this approach not only was a challenging undertaking, but it also slowed down the process of rolling out the measure and collecting data. Undeterred and committed to the pilot, the team at Eau Claire regrouped and began exploring where in the workflow they could best administer the survey as a temporary solution until it was fully incorporated into the EMR. While the delay was somewhat frustrating for the team, their work was recognized by senior leadership so much so that the medical director encouraged the collection of PROM data on multiple social determinants of health in the EMR so that the clinic can gain a more comprehensive view of patients’ life challenges.

Case Study - AIDS Ministries

AIDS Ministries in northern Indiana selected housing as their domain because it is a significant problem for their patients as they have difficulty finding permanent housing, even with support from their organization. The initial survey questions focused on the stability of housing, including where a patient was sleeping. Based on a short test cycle of the measures, AIDS Ministries found that the data they were collecting did not align with what was in their database about patients’ housing situations. In response, they revised the survey questions and focused on the financial issues that might result in patients becoming unhoused. They included a definition of housing stability in the survey, so patients understood what they meant when they asked about unstable housing. They also included questions on financial stability, such as inability to pay power bills and other utilities, and on situations that could lead to unstable housing. As a result of what they found AIDS Ministries is considering offering financial literacy training to their patients.

Testimonies by Pilot Participants

"There are two perspectives when it comes to quality improvement activities, the providers analysis and the patients’ response to their analysis. When you integrate the two, so that both parties have shared decision-making capacities, you illuminate a deeper understanding of the data. The data no longer translates as numbers or figures, but instead become the narrator of the untold stories for those with lived experience. Because the fact is, that the outcomes of your practice are directly related to the voices of your patients. Listen to your patients, even through the silence, they are speaking."

"When you think of quality improvement you tend to have a clinical focus. The project made us look at the bigger picture—what is going on in our clients’ lives. While our clinicians are good at checking in with clients adding these measures on a permanent basis will formalize our processes."

"Within our jurisdiction there were many providers that were not familiar with PROMS/PREMS but now there is a lot of interest. When you look at issues like quality of life, everyone is interested, especially frontline providers."

"Gathering the data gave me a clearer view of areas that we can focus on for improvement. Honing the questions help me better understand the issues."

"‘We’ve learned that responses can be complex. Patients indicate satisfaction in one area, such as having a purpose, or support from family and friends, but then say they feel lonely. We now need to craft a response to these needs."

"We joined the pilot because we felt it would be good for our clients in the long run. Getting the patient perspective is important and we hope that it will allow us to provide even better care to our clients."

"I think this is one of the most relevant and critical pilots that focuses on patient experiences and outcomes. As a person living with HIV and former consumer of Ryan White Service, I can honestly say that this pilot, and any future collaboratives around PROMS+PREMS will be one of the most important contributions to quality of care and services there has ever been – thank you for finally elevating the consumer experience."

Next Steps

With the assistance of the PROMS+PREMS Pilot Project faculty and participants, CQII brainstormed potential next steps to allow the exploration of the PROMS/PREMS measures as vital data sources in HIV care and their feasibility for integrating the results in ongoing QI activities. These next steps include:

Run a second pilot with an increased number of pilot sites (~25 sites) to further explore the opportunities for RWHAP recipients/subrecipients.
Run a national learning collaborative that focuses on implementing PROMS/PREMS and invite sites across the Unites States to join.
Integrate specific PROMS/PREMS in upcoming CQII learning collaboratives as additional performance measures.
Conduct a series of national TA Calls that addresses all major steps of implementing PROMS/PREMS, is open to all RWHAP providers, and include best practices from initial pilot sites.
Put together a series of Quality Academy Tutorials for asynchronous learning addressing all major steps of implementing PROMS/PREMS and using the data results in QI efforts.
Create a mentorship program where pilot sites mentor other RWHAP sites that are starting this journey.
Integrate PROMS/PREMS into existing Learning Labs. 
Publish a paper on the first pilot and submit abstracts to national conferences (e.g., IHI Forum).