2/26: Workforce Benchmarking Network’s Survey Q&A

Good morning, and welcome to our conversation with Kysha Frazier, Senior Policy Associate, and Megan Elyse Williams, Research and Evaluation Associate, both with the Corporation for a Skilled Workforce (CSW).  Kysha and Megan are joining us to discuss CSW's Workforce Benchmarking Network's National Survey.  For preliminary information about the network and the survey, please visit: https://skilledwork.org/what-we-do/workforce-benchmarking-network/wbn-national-survey/  

This survey is broader than adult education, but our field plays an important role in sharing how adult education helps to prepare learners for the workforce.   Kysha and Megan are interested in ABE and ESL programs that incorporate work readiness education, and Integrated Education and Training (IET) programs that prepare learners for entry into skilled positions.  

To begin, I want to ask Kysha and Megan to share some additional context for the survey.  Why was the survey developed and what are some of the benefits of participation? You use the term 'Apples to Apples' when referring to your comparative analysis of survey participants' responses.  What are examples of the kinds of service data and participant characteristics captured?  Would you walk us through what you mean by this term and why it is valuable to survey participants' report data? 

Mike Cruse

Career Pathways Moderator

michaelcruse74@gmail.com

Comments

Good morning everyone! Thanks so much for having us here today Michael. We welcome the engagement and conversation today and in the future! If you need to contact me or any member from the WBN team feel free to send a message to benchmarking@skilledwork.org. Now let's start our conversation!

The Benchmarking Project was launched in 2004 to begin to address questions like: What are the results of community-based workforce development efforts? What are “good” results for different types of programs?  How can CBOs—and the larger field—better use data to improve the effectiveness of   workforce programs?

From our research, we have found that the field knows relatively little about how CBOs are performing, and thus what improvements need to be made, so there are better job seeker and community results. The Benchmarking survey data is the largest collection of outcomes information to date for CBO programs (collecting aggregate data in the past from hundreds of programs on participant demographics, services received, job placement and retention rates, wages, and other outcomes). Our goal is to build the new dataset with updated outcomes data with your help! The survey offers an unprecedented opportunity to examine the outcomes of programs with varying characteristics.  This year, we are also collecting aggregate data to support understanding outcomes across race, ethnicity, and gender, allowing users to get a better sense of key disparities that may exist. While the project’s data cannot “prove” the effectiveness of any one approach, it can help funders and providers set more realistic expectations for performance and make better informed decisions about program design.

Improving performance matters more than ever in the workforce sector.  Employers struggle to keep positions filled, job seekers are addressing multiple challenges to career mobility, and funder resources are shrinking.  So, having and using data to understand what’s working – and for whom—is critical for local workforce systems and service providers. These data provide field-wide performance benchmarks and help illuminate program characteristics that matter for participant success. 

Our online reports are confidential- they are all password-protected.  Organizations submitting data can compare their results confidentially to those of programs with similar characteristics and use that information to create more effective programs.

In addition to the national survey, organizations can participate in regional peer learning workshops and receive technical assistance to help them make better use of the data they have by identifying system “leaks”, setting targeted goals and creating strategies for continuous improvement.

 

Thanks for that background, Kysha.  You noted that "past benchmarking data showed that programs providing ABE to more than 50% of their participants correlated in a statistically significant way with higher enrollee job placement and better job retention rates at 3-6-12 months than programs that didn't offer those services."

Would you talk more about the benefits to adult education programs participating in the survey to tell a more complete story about their organization's work and employment outcomes for learners?  Are there any examples of how a survey participant used their individualized report to engage funders and/or policy makers to develop or improve services?

Mike Cruse

 

Good Morning, Everyone! My name is Megan, and I will be answering questions about the Workforce Benchmarking Network Survey today along with my colleague, Kysha. 

What are the benefits to participating in a workforce development survey as a provider of Adult Basic Education or English as a Second Language services? Past Benchmarking data showed that workforce programs providing Adult Basic Education to more than 50% of their participants correlated in a statistically significant way with higher enrollee job placement and better job retention rates at 3-6-12 months than programs that didn't offer those services. While of course we can’t contribute these outcomes solely to ABE, it is clear that these services play an important role in driving better outcomes for workforce participants, and that data about these services are key for both the workforce field and all who are part of it. 

We believe it would be helpful and advantageous for programs that may be serving a high percentage of ABE students to see how their completion, employment and retention rates (and wages) compare to those of their peers. Our survey allows programs to benchmark their results to other organizations nationally, across multiple funding streams, that provide the same types and intensity of services. This unique capability has over the years helped organizations not only improve their services and outcomes but has opened doors to conversations with potential partners and funders by giving them access to this data that they can share.

For our updated survey that we launched end of last year, we spoke with Foundational Skills experts to update the survey to ensure we are asking about relevant services in the field today. On top of these expanded questions, we intend to do some deeper dive reporting on the new dataset once we have a critical mass of organizations that provide these services, so that we can share results relevant to these providers and the field at large. We will also be potentially collecting expanded data from a set of these providers so that we can better speak to not only "what" services are being offered, but also "how" they are being offered, which is information we know that our network members as well as the field would find of great value. This would include going deeper into areas like how long their services last, what tools they are using, other types of support that are provided, what reading levels are being served, as well as other data points of interest identified by our network members providing these services. We are hoping that you can help us reach that critical mass of Adult Basic Education and English as a Second Language providers by signing up to take our survey today.

What areas would you like to explore more deeply through research? What else would you like to know about your peer organizations and their work? Through virtual learning communities?

Thanks, Megan, for sharing about the survey's data collection, and it's relevance to ABE/ESL and IET practitioners.  You note that survey participants have the option to respond that they do not collect, or are not able to report individual measures, based on their circumstances.  Would you give us an example of how a participating organization that does not have data on earned credentials and learner placement outcomes could still benefit from participating in the survey, and accessing the resulting reports?

Mike Cruse

We ask survey participants about a number of characteristics that describes their participants services. By being thorough, we do hope that the survey participant can see themselves in multiple relevant places in the survey. We ask about individual program info, employer engagement activities, participant info, placement outcomes, employment retention, and financial outcomes. To get a more detailed view of some of the specifics under those categories you can download a summary handout describing the kinds of service data and participant characteristics captured here:

https://skilledwork.org/wp-content/uploads/2019/06/2019-WBN-National-Survey-summary-rev.-6.20.19.pdf . We are also happy to detail that out here if anyone want to see that. 

We do know that survey participants have varying needs in terms of what they wanting to get from the survey. For some program completion is a main goal, for others it’s employment or GED attainment.  We see a big opportunity for the field to do a deeper dive into possibly creating a customized survey focused on literacy providers’ questions to get to their specific outcome needs.

Some useful and what I would say are “cool” aspects of our survey tool is that users have the ability to see reports that compare their outcomes to other organizations using whatever filters are relevant. Users also have the ability to see results of organizations that provide data and outcomes related to services they might not provide or data they might not collect. We have heard over the years, how helpful it has been to see that many organizations are collecting data that maybe they have not thought of or how helpful it is to see that organizations that provide a different service see different or better results. A key benefit of participation beyond the reports, is the doors it opens to conversations at organizations about how they might do things differently based on the results other organizations are seeing. 

Kysha Frazier

 

 

Thanks for sharing more about the type of data the survey is looking to capture.  Part of your 2020-2021 Vision for the survey data is working with interested program sites to pilot use of participant level data.  Would you tell us more about the pilot, how sites will be chosen, and how you will work with these programs to use participant level data in meaningful ways?  

Mike Cruse

We would be happy to share more about our future vision for WBN! We do not have all of the details determined yet for piloting individual level data in terms of who we will work with or how. Some of this will be driven by funding or the communities most interested in participating.

However, we do know that there is only so far that we can go in understanding results using aggregate level data. We are very excited to be collecting disaggregated outcome data this year, but this is just a start. We are considering the respondents of our updated survey to essentially be helping us take this first step by providing this new and expanded data collection. And, as we begin reporting on this expanded dataset, we are interested to see how much more we learn, what gaps remain, and the questions partners and funders have about where to go next using individual level data.

At a high level, our planned next steps include:

  • Piloting use of participant-level data in the national survey to support in-depth analysis and reporting on disparities
  • Supporting additional deeper data collection and analysis around specific populations or strategy questions
  • Developing new reports, briefs, and related communication about survey data findings
  • Launching national data warehouse for sharing data in a variety of formats, including participant level information
  • Providing data infrastructure that can support local shared measures and other data integration efforts

If you are interested in learning more about the future phases of the Workforce Benchmarking Network, please contact us to discuss at benchmarking@skilledwork.org.

 

Thank you,

Megan 

Thank you both for your time explaining the the Workforce Benchmarking Network's Survey and its application to ABE/ESL and IET programs.  I hope the information you've shared with us sparks interest in exploring the potential benefits of participating in the survey.  Are you ready to register your program to participate?   Kysha and Megan can get you started today!

Mike Cruse

The WBN survey is open now to programs providing direct services that prepare individuals for employment (or better employment) and provide job placement assistance. There is a strong preference that organizations also offer job retention services, but that is not required. Organizations must be able to provide complete information about job placement results and median starting wage for participants enrolled in a specific program during a recently completed one-year period.

Register to indicate your interest in survey participation or other WBN Network activities by completing this brief questionnaire. You are also welcome to join us on March 12 from 2:00-3:00pm EST. Register Here for an informational webinar. You’ll learn more about what’s included in the survey and reports, the benefits of participation, and how to get started!

We do hope that you will consider participating and that you will tell others about the survey! Before we go, we want to summarize a few key pieces and share a few more benefits of participation.

Tool Capabilities and Reporting

Using our interactive reporting tools, participants are able to select from a number of characteristics/variables in which they can compare their participants and services against others. For example, they can select a set of outcomes, like rates for program completion, placements in full-time employment or retention for 6 months, and get a set of results. They can then “filter” or narrow down those results to see only programs that offer a similar service to theirs – like only those that offer training for an industry credential.  Or they could go even deeper by looking only at programs that share a focus on a similar population, e.g., those who are homeless.  The more specific, the more granular “apples to apples” comparison results the report will provide. Like going from a more common Red Delicious to a more unique Cortland apple. This flexibility is not often found in workforce development reporting and is the true bread and butter of the Workforce Benchmarking Network.

What Providers Say Are Key Benefits

We have heard many examples for how this data has been used over the years to support continuous program improvement or development efforts. One organization in Chicago that serves primarily homeless populations had experienced concerns from their board and their funders about their low job placement rate. But they were able to use information from the Benchmarking survey to show that compared to other organizations serving a similar population their results were actually quite good.  This helped re-frame funders’ expectations about what results were reasonable to expect. Funders in several cities have regularly used Benchmarking survey results to inform their grantmaking guidelines. 

Another organization in New York helped young adult obtain their high school equivalency and obtain employment.  They were able to use their survey results to identify that their program completion and job placement rates were comparable to those of their peers.  But their results in terms of six-month job retention were much lower than peers – which helped them identify where to focus their efforts.  They were able to use that data to support their proposal to funders to add additional staff capacity for longer-term participant follow-up.  

Some of the quotes we have collected include:

·       “It was helpful in being able to compare the apples to apples – those with similar eligibility requirements – we could break down into specific outcomes and characteristics – used it for proposals – reminding funders that you can’t throw everyone into the same bag.”

·       “Important to be compared to like organizations- talking to others that work with people with more barriers that can be frustrating for them that they have to go up against orgs that do not have those barriers. Where do we compare to like organizations- that is important.”

·       “It keeps us on our toes to keep challenging our data. If you aren’t talking about it, it is easy to do what you have always done. It just keeps challenging us to think about the data we are collecting, is it useful, do we need more, how do we get that? So staff sees we are using it and our performance is only as good as the data coming in.

·       “Just looking to see how are we doing and getting a sense of that- I did a lot of comparing of the characteristics to other orgs for us to get a sense to see where are we? Doing well? Middle of the pack?”

·       We have used benchmarking report data in grant proposals to show our comparisons of our impact to other organizations.”

·       “The ability to compare results with other providers was huge.”

·       “Great to access data and to communicate externally about our results and then also look internally ourselves. Before, we were giving ourselves an out, because when we compared ourselves to others, we assumed that because they weren’t serving the same harder to serve populations, it made sense, but then we could see other similar populations being served by other providers.”

·       Related to the new survey and providing outcomes across race, ethnicity, and gender: “This is data we have and could be looking at, at any time, but this was a good experience, because we realized through collecting the data for the survey, that even though our program has far and away more women, the men have higher retention rates, which we need to explore. Responding to the survey alone gave us time to look at data we wouldn’t have otherwise.”

Thanks for the opportunity to share with you today! Additional information can be found: https://skilledwork.org/what-we-do/workforce-benchmarking-network/.  Please continue the conversation with us at benchmarking@skilledwork.org!

Kysha Frazier