MODULE 1: Desired outcomes
Overview:
The reason achieving desired outcomes is proving to be such a challenge for provider organizations is that producing many of the outcomes they are being asked to produce will require them to completely rework what they do and how they do it. What provider organizations do now, and how they do it, was often not designed with the intention of producing these specific outcomes that are now desired. Provider organizations will seldom be successful at producing a desired outcome by using an approach or practice that was designed to produce a very different desired outcomes in the past.
Oversight agencies who refer to provider organizations for services families and communities need are developing a relatively clear set of desired outcomes. Oversight agencies will also expect provider organizations to assure the services they are delivering to youth and families is through a lens that is culturally relevant and linguistically aligned with youth and families and the community.
Likewise, oversight agencies are expected to produce sustainable positive outcomes for the youth and families they engage. They are held accountable to these outcomes by various bodies such as state legislatures, county boards of supervisors, and federal oversight departments. The courts, advocacy organizations, general public, and press add additional levels of scrutiny, creating supplementary layers of accountability for oversight agencies s. It is critical for provider organizations to partner with their oversight agencies and have a voice in developing the desired outcomes for youth and families. This partnership should occur on multiple levels, including joint advocacy at the state, county, and federal levels. This type of partnership creates joint ownership of data collection and the development of desired outcomes, helping to shape and define the future direction of the field.
The clarity of knowing the desired outcomes expected can be very helpful to provider organizations since they will now know how the determinants of their success will be measured. Provider organizations will often have to produce outcome results that are being determined by someone else.
Provider organizations need to enlist their staff, youth, and families in helping to determine how to achieve these new desired outcomes while maintaining awareness of cultural and linguistic needs of those they serve. Successfully achieving these desired outcomes is one of the highest Mission-critical organizational goals.
Strategies:
- Collecting Data
- Reasons for Data Collection
- Long-Term Outcomes Data Collection
Resources:
- ACRC Position Paper #4
- ACRC Position Paper #9
- A Building Bridges Initiative Tip Sheet: Evaluating and Improving Outcomes for Youth who have Received Residential Services
- Book: “Residential Interventions for Children, Adolescents, and Families: A Best Practice Guide”, edited by Gary M. Blau, Beth Caldwell, Robert E Lieberman. Copyright 2014
- Book: “Transforming Residential Interventions: Practical Strategies and Future Directions“, edited by Beth Caldwell, Robert E. Lieberman, Janice LeBel, Gary M. Blau. Copyright 2020
- Add BBI Theory of Change – still to come when completed by Bob Lieberman
Introduction
There are several excellent resources offering significant insights into performance measures and outcomes that are referenced in the resource section of this Module. Various chapters in the two BBI books referenced offer great detail around tracking long-term outcomes and measuring the impact of residential interventions. BBI is also in the process of finalizing a Theory of Change (TOC) document discussed in its Transforming Residential Interventions book and the TOC will also become a resource in the toolkit once available. Additionally, two Association of Children’s Residential and Community Services (ACRC) Public Policy Position Papers are referenced that focus on performance indicators and outcomes and measuring functional outcomes. The information provided in this module will offer some broad insights in the area of outcome data collection and highlight a few key concepts from the above resources mentioned.
1: COLLECTING DATA
Organizations typically gather information on hundreds of different data points. Not all of this data is of equal significance for an organization and the data points collected may not be directly related to the desired outcomes being expected of the organization. Provider organizations gather data because of both internal and external requests.
A good approach for organizing a provider organization’s responses to outcome data is to first make a comprehensive list of all the outcome and indicator data they are presently gathering. Then prioritize that list by identifying those outcomes that are simply necessary, those that are operationally important and those that are strategically significant. The operationally important and strategically significant outcomes should then be put on a dashboard that is formally reviewed and responded to regularly.
It is advisable to develop a consistent pattern of reviewing and responding to results, those that are both positive and unfavorable. It is essential to carefully review the results achieved, adjust approaches whenever we experience unfavorable outcome results, and fully understand the reasons why others are working.
- SIMPLY NECESSARY – Organizations are asked to track scores of checklist kinds of outcomes and indicators that seem to be “busy work.” Few of them are true “busy work,” though they may seem to be. For example, the process of monthly checking the expiration dates on fire extinguishers, or the refrigerator temperature, are safety-assurance issues that shouldn’t be ignored. It’s simply necessary.
- OPERATIONALLY IMPORTANT – Organizations need to be able to track, identify, analyze, and respond to any contractual or regulatory outcome issues that they identify. They must be able to identify and address those issues on their own and not wait for others to identify them for the organization.
- STRATEGICALLY SIGNIFICANT – An organization’s Strategic Plans should identify those outcomes that are strategically significant. A strategically significant outcome is any outcome that the organization will have to be able to produce to be successful in the future. The successful installation of trauma-informed care would be a good example of a strategically significant outcome.
2: REASONS FOR DATA COLLECTION
Some of the reasons outcome data is collected is driven by contractual or regulatory requirements; some as a result of fiscal compliance and planning issues; some arise out of practice and process needs; some originate from local, state, and federal policies; and some come wanting to know about satisfaction with the organization. Some of this data directly helps provider organizations to determine if they are achieving their desired results and outcomes.
CONTRACTUAL – Performance-based contracting is clearly in the future for provider organizations. Oversight agencies who are making referrals want to purchase outcomes and not just services. When the expected outcomes are written into provider organization contracts, they must develop the ability to re-design what they do and how they do it to assure that they can get the outcomes that are specified.
REGULATORY – When the requirements that are necessary for licensing and certifications to operate programs and services are specific in what the provider organizations are expected to do or to not do, they must be able to assure that those specific requirements are being met on a consistent basis.
FISCAL – Some fiscal outcomes are internal, and some fiscal outcomes are external. Those internal outcomes that relate to the successful implementation of a business plan, or meeting audit requirements are particularly important. The external outcomes that relate to the procedures that must be followed in order to bill for services, assure contractual requirements are being met, etc. are likewise extremely important.
PRACTICE/PROCESS – The practice and process outcomes that we track are for both internal and external purposes. Organizations want to learn how to use both of these outcome results to improve their performance. Everyone involved in delivering services needs to be aware of our practice and process outcome results and committed to doing whatever needs to be done to improve them. These outcomes directly impact the quality of care and must be routinely shared with all staff.
POLICY – Outcomes that originate from policy are often a forecast of the practice results that service planners are expected to move towards. Reducing restraints and lengths of stay, increasing family involvement and success rates after discharge would be examples of policy outcomes that forecast the future direction for the service.
SATISFACTION – Understanding satisfaction is an important indicator for an organization. Satisfaction data collection should reflect satisfaction among consumers, both youth and family, staff job satisfaction, oversight agencies satisfaction with services provided, and the general public’s perception and satisfaction with the organization.
3: LONG TERM OUTCOMES DATA COLLECTION
The data needed to determine if an organization’s desired long-term outcomes post discharge is successfully being achieved is often the hardest and most costly data to collect. BBI has developed an Outcomes Tip Sheet that offers “a framework for measuring meaningful, long-term outcomes; promoting mutual accountability among residential and community programs, families, payers and regulators; and for using data to inform decision making to achieve better coordination of care for youth and families”.¹
The areas typically reviewed for those providing residential interventions that BBI has identified in its tip sheet include:
Home – a safe, stable, supportive living environment
Purpose – meaningful daily activities, such as a job, school, volunteerism, and the independence, income, and resources to participate in society
Community – relationships and social networks that provide support, friendship, love
Health – sustained basic physical and behavioral health, and overcoming or managing health challenges²
¹ A Building Bridges Initiative Tip Sheet: Evaluating and Improving Outcomes for Youth who have Received Residential Services
² Ibid
“For so long we just told stories to prove that what we were doing was important and was working. Stories aren’t enough anymore. Now we have to produce the results that people are expecting.”