Independent Research on What Adult Fundamental (basic) Skills Programs Work

Colleagues,

As you may know, LINCS supports evidence-based practices. This is a new discussion thread whose focus is independent research evidence on what works in adult fundamental (basic) skills programs and adult schools, including ESL/ESOL programs and schools. When I learn about new independent research that is relevant to this topic I will post information about it in this discussion thread; I hope you will too.

David J. Rosen.Moderator

LINCS Community, Program Management group

Comments

Hello Program Management Colleagues,

In Stepping Up, researchers Anne Roder and Mark Elliott, from the Economic Mobility Corporation* document that JVS Boston’s English for Advancement (EfA) program generated large earnings gains for participants during the two years after they enrolled.

*The Economic Mobility Corporation (Mobility) describes itself this way It  "identifies, develops, and evaluates programs and policies that enable disadvantaged individuals to acquire the education, skills, and networks needed to succeed in the labor market so that they can support themselves and their families."

Read the short version: Stepping Up: Interim Findings on JVS Boston’s English for Advancement Show Large Earnings Gains

Read the Summary and Conclusions from the report (below): https://economicmobilitycorp.org/wp-content/uploads/2020/10/SteppingUp.pdf

Note: RCT stands for Randomized Control Study.

Summary

English for Advancement (EfA), a program offered by Jewish Vocational Service (JVS) in Boston, provides adult English-language learners with employment-focused language instruction, career coaching, and job placement assistance to help them obtain employment or advance to a better job. EfA is part of the Commonwealth of Massachusetts’ Pathways to Economic Advancement project, which seeks to help limited English speakers enter the workforce and progress up the economic ladder by providing English instruction and workforce development services. The project, managed by Social Finance, uses an innovative “Pay for Success” funding model in which private sector investors provide upfront capital to scale promising programs, and the government pays back the investors only if the programs achieve predetermined outcomes. As the project’s independent evaluator, Economic Mobility Corporation (Mobility) is conducting a study of EfA that uses a randomized controlled trial design to assess program effectiveness—that is, study participants were randomly assigned to a treatment group that could receive EfA services or to a control group that could not, then their outcomes were compared. In this interim report on the EfA program Mobility presents findings on employment and earnings impacts based on state administrative data for a cohort of nearly 800 study participants who enrolled between September 2016 and December 2017, and whose two-year follow-up period ended by December 2019.

In this report, we document that:

EfA had a substantial, statistically significant impact on annual earnings in the second year after random assignment—an average difference of $2,621 (Figure 1). Total earnings impacts over the two years after random assignment averaged $3,505.

EfA group members were significantly more likely than control group members to be employed starting in the third quarter after random assignment.

The EfA group’s quarterly earnings grew throughout the second year after random assignment, and at a faster rate than control group members’ earnings (15 percent versus 5 percent, respectively).

As the first RCT study of the earnings impacts of a workforce development program for English-language learners, the EfA study findings are important for the adult education and workforce development fields. EfA has proven to be an effective way to integrate language instruction with career services to improve the earnings of English-language learners. Although EfA was implemented during a strong labor market, its design—with rolling enrollment, individualized career coaching, and siting within partner community organizations—and its particularly large earnings effects for unemployed workers make it an important option for states and cities to consider as they develop policies in response to high unemployment.

Conclusions

With English for Advancement (EfA), JVS has developed a low-cost and effective way to integrate language instruction with career services to improve the labor market outcomes of English-language learners. EfA increased the proportion of job seekers who obtained employment and had positive earnings impacts that grew over the eight quarters after study enrollment. The model is worthy of wider utilization since many who enroll in ESOL classes are motivated by a desire to improve their job prospects. The Commonwealth of Massachusetts also benefited from increased employment and earnings that translated into higher tax revenue and larger contributions to the unemployment insurance system.

JVS demonstrated significant impacts in the tightest labor market in a decade, during which unemployment rates dropped to historic lows, particularly in the Boston metropolitan area. Key findings regarding EfA’s impacts for the early cohort of study participants include:

EfA significantly increased annual earnings in the second year after study enroll-ment by $2,621—or 15 percent—relative to the control group, and by $3,505 over two years.

EfA significantly increased employment rates starting in the third quarter after study enrollment and continuing through the eighth quarter.

While quarterly employment rates remained steady for both groups during the second year after study enrollment, the EfA group’s earnings continued to grow, and at a faster rate than control group members’ earnings.

As noted in the introduction, there are no other RCT studies of the earnings impacts of integrated ESOL and employment services for English-language learners. Occupational skills training programs that have succeeded in RCT studies have demonstrated earnings gains of 26 to 35 percent relative to the control group in the second year after study enrollment.26 These programs were more intensive than EfA—requiring full-time attendance over several weeks—and often helped participants earn certifications needed to obtain well-paying jobs. EfA is less intensive in terms of class hours, and many EfA participants obtained jobs in food preparation, cleaning, and transportation—not unlike the limited English proficient population generally. However, the findings demonstrate that EfA increased the percentage of those who were employed at all and the consistency of employment over the year. The results suggest that JVS’s decades of experience in working with immigrant communities, developing and delivering innovative curricula, and cultivating deep and extensive relationships with employers were important to EfA’s success.

Other key observations regarding the findings for the early cohort of EfA participants include:

EfA’s design—with rolling enrollment, individualized career coaching, and siting within partner community organizations—lends itself to rapid expansion and is especially useful in communities with large immigrant populations.

While EfA’s overall earnings impacts are impressive, they could be considerably larger if future programming focused on the groups who benefitted the most, including unemployed individuals with some prior U.S. work experience and individuals with more than the lowest level of English-language proficiency. The evidence also suggests that EfA is particularly effective in communities outside the city center, where ESOL services are limited.

The study provides evidence to support the Pay for Success funding model as an effective strategy to finance and scale innovative approaches to government-funded services.

We cannot know what EfA’s impact will be for the cohort of study participants whose two-year follow-up period will be affected by the economic slowdown caused by the COVID-19 pandemic. JVS is working to reengage participants in remote ESOL classes and career coaching, providing an opportunity for them to build their skills until the hiring environment improves. We will publish a report on EfA’s two-year impacts for the full sample of study participants in late 2022. That report will explore differences in the employment and earnings outcomes of the pre- and post-pandemic cohorts as well as whether and how EfA’s impacts change from the pre- to post-COVID periods.

David J. Rosen, Moderator

LINCS Community Program Management group

Hello Program Management colleagues,

Here's another article on the English for Advancement model, a short easily-read piece from today's Boston Globe. It's rare that that we have adult basic skills programs that have been studied using an experimental design model conducted by independent researchers. In this case, the results are impressive, two years after the program significant earnings gains were demonstrated, and the model is now supported by state funding and scaled up in Massachusetts, and perhaps eventually in other states. As a program manager, if you offer English language classes this might be worth your attention.

As always, your thoughts are welcome here!

David J. Rosen, Moderator

LINCS Community Program Management group

Hello Program Management Colleagues,

To register for this on-line Opportunity Forum about the English for Advancement report, hosted by Jewish Vocational Services (JVS) on Nov 18 at 12:30 pm EST, go to  https://jvs-boston.zoom.us/webinar/register/7316038966323/WN_VRjuC0eVSzq2dvy0Eu1OVQ

David J. Rosen, Moderator

LINCS Community Program Management group