Case study | Prioritizing the people’s perspective in evaluation

As an evaluator and public health practitioner, I've always enjoyed learning about others' journeys and lived experiences. This connection and curiosity have extended to my work in evaluation, which focuses on addressing systemic issues within government systems and answering social impact questions that touch people from many different walks of life who carry a range of lived experiences. 

Over the years, we have seen significant and necessary momentum to embed equitable practices in evaluation. There are initiatives such as the Equitable Evaluation Initiative (EEI) framework that helps practitioners embed concepts of equity, inclusivity, and rigor into everyday practice.  Increasing numbers of consultants and evaluators are using storytelling and creative participatory methods to capture the nuance and context of multiple perspectives to further inform the data. While as a field, we've been trending toward embedding an approach to more intentionally address equity and justice in evaluation, the current political landscape and concerted federal and state attacks on diversity, equity, and inclusion (DEI) have made it feel like a huge mountain has been placed in that path.  

In the midst of all the chaos, it's been helpful for me to think about what opportunities exist for me in my work to stay true to those values of equity, inclusion, and justice with the clients and projects that we work on and to continue to truly prioritize the voice of the people. As evaluators, we often seek methodological validity and rigor in our work, but it's also important that we hold room for participatory methods, community involvement, and community voice as the litmus test for rigor when measuring equitable change and progress. 

One example that comes to mind is a project we partnered on with the Department of Natural Resources (DNR) and the Michigan Department of Health and Human Services (MDHHS) as they embarked on developing the 2023-2027 Michigan Statewide Comprehensive Outdoor Recreation Strategic Plan (SCORP) used to guide programmatic decisions for state and local organizations around outdoor recreation, infrastructure, land acquisition,  physical activity spaces, and programming. The SCORP assessment and data collection process is designed to evaluate ongoing and emerging outdoor recreation trends, needs, and barriers to achieving outdoor recreation goals. 

As we reviewed the data, we saw a clear gap in who was represented in the survey responses. Because of how the survey data was collected, representation from specific communities was lacking. We took a step back and started to ask questions that could address the gaps in the data that we were seeing, such as:  

• What communities aren't represented in this data and why? 

• What would it take to connect with those communities? Who are the champions of those communities? 

• What do these communities value? What would it take for individuals from those communities to engage? 

Asking these questions allowed us to pivot and develop a data collection strategy to engage with participants whose perspectives had been historically excluded from previous data collection efforts. 

EC then worked with the SCORP project team to identify community leaders in predominantly BIPOC communities as focus group facilitators and as a gateway to connect with other community members who wouldn't ordinarily engage with the DNR or DNR affiliates. Community facilitators and participants were compensated with generous stipends for their time and expertise. The EC team developed and led a workshop for community facilitators to provide the training and tools to lead a community focus group to gather feedback about what outdoor recreation meant to individuals, including the barriers that stood in the way of those experiences. Community facilitators were key in finalizing the focus group protocol and determining the verbiage, framing, and questions that would resonate most with community members. 

We witnessed  overwhelming interest from community members who were willing to share their perspectives on their outdoor experiences and provide a wide range of perspectives on how BIPOC communities are defining outdoor recreation, the barriers that exist, and potential solutions to enhancing outdoor experiences in their local community. Community members felt seen, heard, and appreciated being included in the SCORP process. Participants left the focus groups eager to be involved in the next steps and, as a result, new relationships were formed that could potentially strengthen the feedback loop between government offices and the community.  

Cover page of current SCORP plan

What did we learn from this process? 

I learned that prioritizing the community in the evaluation process strengthened the data and provided necessary context and insight into a way forward that more fully benefits the community. Inclusion matters, and it often requires being willing to shift direction, invest time, and use unique methods to find the voices missing from the data. We also observed how it pushed the SCORP project team to reflect introspectively and answer questions from the community about internal processes that span wider than the SCORP process. At EC, we learned that allowing time to pivot in the work can have a ripple effect that opens new pathways to us and our partners,  answering questions and solving issues that weren't as obvious beforehand. We witnessed a shift that moved beyond just collecting feedback for the SCORP to asking questions about how to maintain these relationships and build ongoing communication between government departments and the community. This process may still be novel in government, but it’s a great step toward prioritizing people and community in evaluations conducted in the government sector. 

We'd love to hear if this resonates with you! Let us know how you've made a pivot in your data collection process to be more inclusive and what you're learning along the way.