Day Two: Impact of Basic Skills Program Participation-Research Design

Submitted by Cynthia Zafft on Tue, 10/20/2015 - 10:37
  • 2666 Views
  • 7 Comments

Greetings, Steve and Community Colleagues!

Welcome to Day Two of our discussion!  I want to begin with some of the questions that came up yesterday in the webinar and in the community.  They speak to the research design and I think Steve can help us here.

I think most of us read about studies in which researchers approach and partner with adult education programs to study teachers and adult learners but The Impact of ABS Program Participation Study has a different design (the 10,000 robo calls got my attention).  For those who were unable to attend the webinar, would you describe how people were located for the study?  And, why that method was used?

Then, here are two questions from David Rosen, Moderator of the Program Management group, that touch on how other states might replicate the study.

1.  Is it possible for other states to capture data to determine what the economic impact is of adults who have dropped out of school and who later enroll in ABS programs, and to compare those outcomes with school dropouts who do not participate in an ABS program?

2.  If so, how?

And, if possible, would you address questions by Dolores and Patricia:

3.  From Dolores Perin: Was it within the scope of the study to look at the relationship between any increase in reading scores, and increase in wages, in combination with, or separately from, program participation?

4.  From Patricia Meierdiercks: Did any of the program participants receive services through corrections?

Thanks,

Cynthia Zafft

Postsecondary Completion Moderator

Comments

Thanks, all, for the good questions.  I'll try to answer them here:

Cynthia's initial question: I think most of us read about studies in which researchers approach and partner with adult education programs to study teachers and adult learners but The Impact of ABS Program Participation Study has a different design (the 10,000 robo calls got my attention).  For those who were unable to attend the webinar, would you describe how people were located for the study?  And, why that method was used?

==>The survey phone lab that did the sampling actually wound up making close to 150,000 phone calls (random digit dialing).  The random-digit dialing phone lists were generated by commercial telephone survey companies.  That method was used to make sure that we identified both a random sample of participants and nonparticipants from our specified target population.  We could not rely on just program participants that local programs might identify because we needed a comparison group of similar individuals who might have gone to programs but did not.

Here are the additional numbered questions:

1.  Is it possible for other states to capture data to determine what the economic impact is of adults who have dropped out of school and who later enroll in ABS programs, and to compare those outcomes with school dropouts who do not participate in an ABS program?

2. If so, how?

Yes, it is possible for states to do this but not necessarily easy to bring the right data together from existing administrative databases.  Many states have received large federal grants to build State Longitudinal Data Systems (SLDS) that link individual data in state employment and other state data systems.  Which data sets they link and how they do it varies from state to state.  States have typically linked two or more of these systems: preschool, K-12, employment, postsecondary, adult education and training.  It's important to encourage your state to include adult education data in their SLDS so outcomes of our learners can be visible in other state data sets.  It's also very important, for the types of impact analysis I did, to include and identify individual records of those eligible for adult education who did not go through programs.  This can be the hardest group to identify within state data, typically requiring matching data from K-12 (i.e., individuals who did not complete high school who do not show up in adult ed programs).  There's much more to say about this, but let's leave it here for now.  I'm happy to follow up here or individually with folks from states wanting to look into possibilities further.

 

3.  From Dolores Perin: Was it within the scope of the study to look at the relationship between any increase in reading scores, and increase in wages, in combination with, or separately from, program participation?

It was possible within one of the methodologies I described, the fixed effects panel regression, to look at ongoing changes in assessed literacy scores and GED attainment at the same time as program participation.  GED attainment following program participation accounts for a small but significant portion of the program impact on earnings.  Like the effects of program impact, the GED impact on earnings takes years to "mature" within future earnings,, replicating findings of John Tyler's research on GED effects on earnings.  But program impact effects on earnings are independent of and much larger than those of GED attainment by itself.  So getting a GED was one of the routes to higher earnings for participants, but not the only one.  Further details about this are given in Brief #1 available on the LINCS site.  The story with respect to separating out the effects of literacy gains on future earnings is different.  After taking the effects of program participation and GED attainment on earnings into account, there is not a separate statistically significant impact of literacy proficiency evident in the panel regression models.  I am still in the process of trying to understand those results.  At the very least, I would say that the demonstrated impact of programs on assessed literacy proficiency is not easily separated from the broader impact of programs on earnings.

 

4.  From Patricia Meierdiercks: Did any of the program participants receive services through corrections?

This was a household survey so individuals who were incarcerated at the time the panel was originally sampled were not included.  But some individuals became incarcerated during the course of the study and they were interviewed and assessed in state or federal facilities.  Some of those did receive services but the subsamples are too small to report results separately for them.  

 

 

 

Hi Steve, David, Dolores, and Patricia:

Thank you for the questions and answers.  Are there other questions that community members have about the research design or the first brief on The Impact of ABS Program Participation on Long-Term Outcomes or any of the other briefs in the series?

I want to echo David's comment about the increased wages for those adult learners who participated in Adult Basic Skills (ABS) programs at the 100+ hour level because it is so astounding.  Those who enter ABS programs had an annual wage (before 1997) of approximately $6,000 (much lower than those who do not participate in ABS programs) but climbed to nearly $14,000 by the end of the study (after 2007; measured in 1997 dollars).  This is still not a family sustaining wage the the trajectory is in the right direction.

Cynthia Zafft

Postsecondary Completion Moderator

Cynthia makes a good point about the wage increases being in the right direction, but still not 'family sustaining'.  I'm wondering if there was any data collection regarding the types jobs that the ABS participants held, leading to these higher wages?  Were they predominantly new jobs, or advancements with existing employers?  This type of information would be helpful in better understanding how to support career pathways for participants in these programs.

Mike Cruse

Career Pathways Moderator

michaelcruse74@gmail.com

Thanks for your comments and questions, Cynthia and Mike. It's important to note that these growing income figures are *averages* of incomes that are much higher as well as ones that are much lower including "0"s for individuals not employed at all during a given year.  Many individuals did reach family sustaining wages but as Cynthia notes, many did not.

Mike asked about income growth being related to new jobs or advancements with existing employers. In general we saw indications of both patterns.  The income data had industry/occupation (NAICS) codes but did not identify specific employers so that makes it difficult to see all changes of individuals' employers.  Its also worth mentioning that many people had multiple jobs, some part-time, some full-time judging from the hours reported.  We also had information about work from our face-to-face interviews with respondents, but those data were not reported in the Briefs.

 

Dear Colleagues:

Here is the link to the study website:  LSAL The Longitudinal Study of Adult Learning at Portland State University.  Steve, do you have some suggestions for looking around the website.  For example, is the kind of information that Mike is asking about available in a report that is public, information not reported in the brief?  [Just an FYI to the community, the survey instruments are available on the site.]

Cynthia

Good question, thanks. The information is organized under the hyperlinked pages listed across the top of the home page. For example, most of the publications from the study are listed on the "Reports" page, many with links to pdf downloads available. The lengthy questionnaires used in the face-to-face interviews are available under "Instruments".  And so forth.  This is something of a legacy website by now, with things like broken links to external sites in some places.  We update these as we come across them.  Some of the content needs updating as well, for example, the 6th wave is not mentioned on all of the pages so a refreshing is in order.