Day 2 Discussion with Dr. John Strucker on Assessing the Four Components of Reading: The ARCS Study

Hi Everyone,

Many thanks to all those who contributed creative comments and posed perplexing questions during yesterday's discussion with Dr. John Strucker! If you missed the discussion about what silent reading tests do not tell us, you can still contribute to it.

I was privileged to be a STAR trainer for many years, and I can still hear John Strucker's voice echoing from a video where he extolled the benefits of doing specific diagnostic reading assessments in alphabetics, fluency, vocabulary, and comprehension. He said not giving these assessments is like "flying blind" in the classroom. He could say this because of the ground breaking study he and Rosalind Davidson conducted, the Adult Reading Components Study

John, thank you again so much for taking the time to add to our field's body of knowledge! I will lead off with today's first question:

You were the principal investigator on the Adult Reading Components Study (ARCS).  Please explain the findings of the study.

Steve Schmidt, Moderator

LINCS Reading and Writing CoP

 

Comments

Hello John, Thanks for outlining the reality that reading tests alone, such as TABE and CASAS, don't offer much guidance to instructors on the specific needs of learners. Teachers in my program have noticed that it is very hard for English learners who have tested out of ESL levels to show gains on TABE Reading. These learners are often placed into Levels E and M of TABE.  (We do not use the TABE CLAS-E.) Maybe we are the only program that has noticed this challenge, but I wonder if you might have any thoughts about this.

Thanks so much! Susan Finn Miller

Moderator, English Language Acquisition CoP

Hi Susan,

I'm sure other practitioners have noticed the challenges TABE and CASAS pose to ELLs who have tested out of ESOL classes and are now enrolled in ABE. I hope they'll weigh in on this, too.

Several questions come to mind. First, what are the native language literacy and school completion levels for these learners? Not surprisingly, the ARCS data for Spanish speakers showed that those who had the highest tested levels of Spanish literacy and background knowledge were those who also had completed middle school and above in their native countries scored the highest on our English literacy measures.  Interestingly, many of these HiSpanish/HiEnglish learners had been in the U.S. a shorter time and taken fewer years of ESOL than those who had lower Spanish and English literacy.

In the most extreme example, think of ESOL Literacy learners - the folks with little of no native language literacy and education. They are faced with acquiring English oral/aural skills, learning to read for the first time in a foreign language, and, because they had limited schooling in childhood, they have very little academic content knowledge. Even after they learn to decode English, it may take some time before they can comprehend texts above GLE 3.

The related area to look at is vocabulary. How do the ELLs perform on the DAR Word Meaning Test? It's possible that many learners are being transferred to ABE before they have mastered enough literate English vocabulary to support higher levels of comprehension.

Finally, speaking of literate vocabulary, I want to share a "quick-fix" or "hack" for improving ELLs' reading comprehension - teach English Signal Wordsespecially those that occur in text much more than in conversation. I'm talking about words like nevertheless, despite, moreover, contrary to, etc. Standardized texts are chock full of Signal Words, and i know from my own experience with Spanish and French, Signals Words in a foreign language are difficult to learn and difficult to remember. Like Tier 2 Words, they shape meaning and context. And, even worse, they defy definition except in the context of a sentence or clause. 

Thanks for raising this important issue, Susan!

Warmest wishes,

John

John, you and Susan brought up the challenge of addressing the needs of second-language speakers. You said, "They are faced with acquiring English oral/aural skills, learning to read for the first time in a foreign language..." Through many years of teaching English to adults who were illiterate in their own language, usually Spanish in my case, I often failed to direct students to learning to read in their own language first, before learning to perform in English. I knew, from experience and research, that they would learn to read much faster and under less stress, if they became literate before learning English. Unfortunately, those students consistently refused to cooperate. Their perception was that they would waste valuable time in the process. They wanted to learn English now! A little off topic, but I wonder if others have had that experience. Leecy

Hi Leecy,

Sorry I missed your post earlier. The Community Learning Center in Cambridge, MA, where I used to teach, did not offer Spanish literacy, but for a while we offered Kreyol literacy taught by a native speaker. The class only attracted Haitian grandmothers who, according to the teacher, were mostly interested in socializing. After a while, attendance dwindled and the class was discontinued. Younger Haitian immigrants were not at all interested in the class; they were totally focused on trying to acquire oral/aural English and English reading and writing.  There used to be a program in Cambridge (at Centro Presente, I think?) that offered a full range of Spanish literacy classes, as well as the Spanish GED. For Spanish speakers who just needed any high school certification, regardless of language, it seemed to offer a good alternative. 

In the case of ELL children, who have more time and aren't faced with finding work immediately, native language literacy-first is thought to be very effective. It used to be,and may still be the approach used in schools for Native Americans in Mexico. After a couple of years learning the alphabetic principle in their nat. lang, the kids were switched over to Spanish reading and writing. The idea is that it's easier to learn the alphabetic principle if the phonological targets are words that are in your native language.

Best wishes,

John  

Indeed, John. Your point is well taken: "if the phonological targets are words that are in your native language." In the case of Spanish, of course, phonological targets are far more predictable that those in English, which contributes to making it easier to become literate! Thanks for the note! Leecy

Welcome back everybody. Sorry I'm a little late responding to your question, Steve. I was busy getting caught up discussing Kristine's post from yesterday. She raised the important issue of what readability doesn't tell you about text selection. I urge anybody interested in this topic to take a look at her comment and my response in yesterday's discussion.  Now to Steve's question about the ARCS.

Along with Assistant Study Director, Rosalind Davidson, I led NCSALL’s Adult Reading Components Study. Our team administered a Background Questionnaire and an extensive battery of tests of the components of reading and related cognitive assessments such as phonemic awareness and rapid naming. The sample was drawn from seven states and included 676 adults placed in ABE classes and 229 placed in ESOL classes. Native Spanish speakers were also assessed Spanish reading skills, regardless of whether they were enrolled in ABE or ESOL classes,

 

The profiles of the 676 ABE enrollees were subjected to cluster analysis, a statistical procedure that groups people with similar characteristics. Experienced ABE practitioners helped us to arrive at 10 instructionally relevant clusters or profile groups, ranging from the most skilled ASE learners to those just beginning to read. There were three ASE clusters (34% of the sample), five intermediate clusters (56% of the sample), and two beginning level clusters (11 % of the sample). Beginners were probably underrepresented because we did not assess learners from volunteer programs or corrections facilities.

 

Rather than summarize the entire ARCS here, I suggest community members look at the three-page ARCS Research Brief brief_strucker2.pdf (ncsall.net). I will, however, list some of the findings that we found interesting, along with some other reflections on the ARCS, which is now nearly twenty years old.

 

  • We were surprised that so many learners had been kept back one or more grades when they were children, including 48% of those in the highest ASE cluster and up to 66% in some intermediate clusters. The cluster with the 66% kept back was made up of learners with “classic” dyslexia profiles – extreme trouble with phonemic and phonological processing, poor word reading, but stronger oral reading in context, and relatively strong vocabulary and comprehension.  These ARCS learners were in K-12 from the 1950s to the 1980s, when it was too often the case that dyslexic children were viewed as lazy or of limited intelligence.  Thankfully, this is no longer true in most US school systems.
  • There was an interesting distribution of learners by age: the youngest were found in the highest ASE clusters and the oldest among the beginning reader clusters. Perhaps this also reflects the improved reading instruction and help for children with mild reading disabilities that began to be widespread in the mid-1970s.
  • When the ARCS results were made public, some ESOL teachers were pleasantly surprised to learn that, based on Spanish reading assessments, the vast majority of native Spanish speakers were not uneducated or lacking in Spanish language literacy.
  • Speaking of being educated, compared to U.S.-born learners, a higher percentage of native Spanish speakers (most of whom were from Mexico) were able to correctly answer the question, “Who was president of the US during the American Civil War?” This speaks to the way in which K-12 reading problems can interfere with learning in the content areas like social studies. Content knowledge among US-born learners as assessed by the WAIS Information subtest was especially weak in the a areas of basic science and civics.
  • The ARCS data revealed something that we had noticed as teachers: Many learners who had difficulties with word reading and oral reading for fluency and accuracy, nevertheless demonstrated decent knowledge of basic phonics. Why did they consistently fail to use their phonics knowledge when they were reading words and texts?
  • Among US-born learners there was no relation between years of school completion and reading ability. But among native Spanish speakers, the more years of school they completed, the higher they scored in Spanish reading comprehension and vocabulary.
  • Our Background Questionnaire was a rich source of data. Several questions asked learners to describe their reading problems. Most learners’ self-reports about their reading tended to be confirmed by the assessment data.
  • Analysis of responses to our oral vocabulary tests highlighted special difficulty with academic or Tier 2 Words, words such as tend, abundant, or decline. Learners either didn’t know them at all, or gave highly colloquial definitions like, “Decline…that’s when your credit card comes back.”  Tier 2 Words are a major part of STAR vocabulary instruction because they are essential for comprehension of texts at higher levels.   

 

Specifically on this point you make, John:

"Our Background Questionnaire was a rich source of data. Several questions asked learners to describe their reading problems. Most learners’ self-reports about their reading tended to be confirmed by the assessment data."

There have been so many instances where I've reviewed DRA assessment data with students, and they've been excited (and relieved at times!) to have what they have always sensed or known about their reading difficulties validated. I think that for many of our students who believe they have a "reading issue," giving them some concrete assessment data about what they need to work on gives them confidence, hope and a solid plan for growth moving forward. 

And a big YES! on having a discussion about text selection, including visual as well as print texts. :-)

Your experience with sts who are relieved to have their reading assessed matches my experience,. I often add responses to students such as "I can see you have had to work HARDER than some others learning to read, which shows your persistence, dedication to your goals...". I also make sure to bring their reading assessment back full circle to their profile - if they have experienced many elementary school changes, I tell them that often school systems struggle to find and fill the gaps that can happen then, and that moving is tough on kids and that learning can be difficult at those time, so it was never the (then child) reader's fault, and that present difficulties "have nothing to do with intelligence or hard word, but simply exposure in early years."  I have experienced women simply start to cry at this point!

Hi Kristine,

Long before I worked on the ARCS, I noticed adult students'  ability to be thoughtful their own learning. It makes sense in the context of overall human development; one of the defining characteristics of being an adult is possessing metacognition, the ability to reflect on and analyze how we know things and how we learn.  

Having said that, I've found that we sometimes have encourage this reflection and analysis by asking learners specific questions like, "What makes reading difficult for you?" or, "What kind of practice works best for you?" Sad to say, many ABE learners are not used to having their thoughts listened to and respected. By its very nature, one-one-one diagnostic assessment helps to model that attention and respect.

Best wishes and thanks, 

John 

Not all ABE reading profiles are uneven. Beginning readers (GLE 0-1.9) (referred to as Beginning Adult Basic Education Literacy in OCTAE’s National Reporting System) tend to have flat profiles because they struggle in all aspects of reading. But even some beginners have oral vocabulary levels at GLE 6 and above because they watch TV nature shows or listen to informational podcasts and E-books. At the other end of the scale, the High ASE learners who are near to passing the HiSET have uniformly high levels of GLE 12 skills in all components.

 

It is the learners in the middle – GLE 4-8 – who are most likely to have uneven reading profiles like “Richard” and “Vanessa,” whom we talked about yesterday. The recognition of uneven levels among intermediate readers led OCTAE (then OVAE) to fund the development of STAR.  STAR training includes how to assess the components of reading, how to group learners by profiles, and how to teach vocabulary and fluency – two key aspects of reading development for adult intermediate readers. But, not even all intermediate adult readers have uneven profiles. NCSALL’s Adult Reading Components Study (ARCS) identified two subgroups of intermediate readers with even profiles around the GLE 5-6 level in all components. (One was distinguished from the other by having a much slower oral reading rate.)

 

To get back to Steve's question, why are there so many learners with uneven profiles in the ABE population? To gain some perspective, let’s consider the K-12 population, because our adult learners from the US were all once K-12 students. To my knowledge, nobody has ever given a battery of diagnostic reading assessments to a large sample of K-12 learners in the US, although the National Assessment of Educational Progress (NAEP) has looked at some of the components of reading from time to time.

 

Imagine a K-12 version of the ARCS. It would be likely to show some struggling readers with dyslexia or language-based reading difficulties – children who have trouble with print skills like phonics or fluency. It would also include children who have grade-level print skills, but severely limited vocabulary knowledge. The latter group might include children whose parents or caregivers are not well educated and/or are recent immigrants to the US. If children in of both these categories didn’t get enough reading support, especially in the middle grades, they might fail to graduate from high school, or they might graduate with limited literacy skills. Later in life they could end up in ABE programs – with uneven profiles similar to those they had in childhood.

 

The shorter (borderline tautological) answer as to why there are so many uneven profiles in ABE is this: almost by definition, many struggling readers fail to develop into good readers owing to key weaknesses in their profiles. You can’t be a good reader if your decoding is slow, uncertain, and inaccurate (Richard), nor can you be a good reader if you can decode the words, but don’t know what they mean (Vanessa).

Your last paragraph in your post on Uneven Profiles really spoke to me and my experience both as a classroom teacher who went through STAR training and as a current STAR trainer in Illinois. In order to help a struggling reader develop their skills, they need instruction in their areas of weakness, not continued practice in areas where they may have strengths. After I had gone through STAR training many years ago, I was able to assess and group my students for instruction on the reading components where they needed improvement. Others have said many of our students need vocabulary instruction on Tier 2 words and I found this to be the case. However, I had one older student who never finished 8th grade, but had an amazing vocabulary. He watched a lot of The Learning Channel and Discovery Channel on tv in the evenings when he came home from his warehouse job.  He placed out of vocabulary instruction of Tier 2 words and it would have been a waste of his time to be part of the Tier 2 explicit and direct instruction that I gave to the rest of the class. Instead, during that vocabulary time, he worked on the component he needed to improve which was comprehension strategy instruction. He was very grateful that he didn't have to stay with the rest of the class for something he already knew.

Hi Laura,

What an important point! Your mini-case study of the man who didn't need Tier 2 instruction was very informative. It illustrates so well why we want to avoid "one-size-fits-all". 

I'm sure you're well aware of this, but for the benefit of other participants who haven't had your training, it's not always the case that we don't provide instruction in areas of strength. For example, "Richard's" GLE 4 fluency was 2 GLEs higher than his than his alphabetics, but we didn't neglect it. Not only was his oral reading choppy with lots of self-corrections, but we know that word reading and phonics instruction will only become cemented if a learner has opportunities to apply what they are learning through the oral reading of meaningful texts - i.e., fluency practice. If Richard hadn't had been able to continue, we also would have worked to strengthen his GLE 6 vocabulary. even though it was a strength. Richard's goal was to get a high school credential, for which he would have needed a much stronger vocabulary

Thanks for sharing your experience. I continue to be impressed by all of you STAR practitioners.

Warmest wishes,

John

What a rich discussion this is! Thank you again everyone for your thought-provoking comments!

John, how should we apply the findings of the ARCS study to best assist our students in becoming proficient readers?

Thanks for sharing from your treasure trove of knowledge! 

Here are few of the findings from the ARCS that can inform teaching:

  • Always perform diagnostic reading assessments, including a background questionnaire, on every student. You can’t know what to teach if you don’t know what they need.
  • Fluency was a problem for many learners in the ARCS. If reading is too slow and effortful, comprehension suffers. Make sure there are opportunities to practice oral reading fluency for all levels up to ASE. Even some high ASE learners may need experience reading college-level texts fluently to get ready for post-secondary education.
  • Teach academic or Tier 2 Words. You can’t teach every one of the thousands of Tier 2 Words in English; however, working on a few each week in depth (the STAR approach) encourages learners to pay attention to them and realize that they are critical to comprehension.
  • Most adults are good at analyzing their reading. Take advantage of this in class. Ask learners why they find something challenging, or what they could do to master it.
  • The ARCS administered the Information subtest of the WAIS to all learners. Their background knowledge was surprisingly (and appallingly) limited – especially in science and social studies. Research shows that knowledge is essential to proficient reading. (See this article by Daniel Willingham for a nice summary How Knowledge Helps | American Federation of Teachers (aft.org).) The College and Career Readiness Standards for Adult Education (CCRSAE) urge us to use more informational texts to help build knowledge. We know can’t cover all the knowledge our learners missed out on, but sometimes targeted approaches can work for some people. For example, if we know a learner plans to become a medical technician, we can make sure she gets to do a lot of reading in the life sciences at an appropriate level.

  

Thanks so much for another day of rich discussion! One thing that was very common among my adult learners is that they'd readily say they "couldn't do math" but never said they couldn't read well. I'm not sure if that is because they assume that all adults should/could read well or if they just were not aware that there is a difference between physically reading words on a page and the ability to synthesize, digest, and apply their reading. Maybe I am unique in this experience. It seems to me that an across-the-board reading diagnostic would be really beneficial. Do you think that is practical, or is teaching the STAR method effort enough with diagnostic assessment only for those we intuit are in need of it?

Hi Susan,

Good questions. I'm just guessing, but maybe problems with math may be more "in your face" than reading problems. This might be the case if you read at a GLE 6 level, but neither your home life nor work require more advanced reading. You can accomplish most reading tasks you choose to take on. But math tasks are often required at jobs that do not otherwise have very high literacy demands. For example, you may need to keep your timesheet, or add up inventory, or understand percentages printed on labels. Also, according to the OECD tests, math ability is more determinative of income than literacy.

During the ARCS and at the Harvard Reading Lab, learners would often tell us that spelling was their biggest problem with reading. In the days before spell-checkers were everywhere, people's spelling errors were very obvious and at times embarrassing to them - such as when filling out job applications, writing phone messages, or filling out occasional brief reports..

Anybody else have any ideas on this question?

Thanks, Susan.

Best wishes,

John

Susan,

Your comment reminded me of a common answer to a STAR comprehension interview question: How do you know when you are reading well? So many students -- in fact, most -- answered either: "When I can read the words. When I know the words." This made me realize that their perception of being a reader was mostly about words, not meaning. So if adult readers are able to read the words they need to in everyday life, they may not realize they have difficulties with making meaning. They may also not realize they lack the skills to read beyond familiar materials (and perhaps avoid doing that?). Although DRA might be discouraging for some of these readers, it might reinforce the need to improve reading fluency, vocabulary knowledge, or use of comprehension strategies to become a more skilled, proficient reader. 

The reading process is so complex and fascinating...and there will always be something to learn.

Marn Frank

Such a great question, Susan. Not sure I have an answer, but would like to engage it, nonetheless.

I think Dr Strucker said this, but because in math instruction there is often (less these days, thankfully) an emphasis on 1 correct answer.  It is more difficult to engage if your chance at success is only ever 50%, whereas in reading instruction one can choose various levels of engagement if "called on". And how do teachers' responses differ to incorrect MATH vs READING responses from students? "Right or Wrong" or "Let's find the strength in that response."

An observation of mine based on your thoughts:  It can often be heard that 9th grade is a common point of departure from K12 because of Algebra. I never hear that 9th grade is a common point of departure because the reading level at that grade has not been achieved. Even educators use MATH as milestones of grades, because it's Algebra at 9th, Geometry at 10th, Algebra 2 at 11th, etc? It is simply easier to make assumptions of math skills encountered if a student says "I enjoyed math until 9th grade, than I checked out, and I was on track with the standard curriculum at that point." 

Hi John,

I love the quote from Maya Angelou which says, "Do the best you can until you know better. Then when you know better, do better.” 

You were an ABE instructor at the beginning of your career in adult education.  If you knew then what you know now, how would your classroom instruction be different?

Thanks! 

 

Great question, Steve!  But before getting into my response to it, I'd like to take this opportunity thank you again for setting up this discussion and helping me organize it. And, thanks to all who've logged in to follow it the last two days. I want to extend special thanks to those who've taken the time to post comments and questions. I've really enjoyed reading and responding to them. Your comments have made our discussion not only richer, but far more informative.

Now for Steve's last question, You were an ABE instructor at the beginning of your career in adult education.  If you knew then what you know now, how would your classroom instruction be different?

The biggest change in my teaching would be in the area of vocabulary. The STAR approach to Tier 2 Words was just being introduced to adult education by Mary Beth Curtis in the mid-1990s, based on her work at Boys and Girls Town. The key, as Curtis explains it, is you don’t teach how to learn new words from context, you encourage the learners to create a context for the new words they’re learning. This involves lots of focused discussion that learners really enjoy.

Related to this, the CCRSAE call for teaching academic or Tier 2 vocabulary to 0-3.9 learners. Readers at this level do not encounter many Tier 2 Words in the simple texts they are able to read. The solution is to for teachers to use these words and encourage learners to them in class discussion. Becoming familiar with some Tier 2 words could also make it easier for beginners to understand information-rich pod casts or videos. (See below.)

As mentioned in the previous question, the CCRSAE urge us to use a higher proportion of informational texts – at all levels, including beginning reading. Beginners can’t decode much in the way of informational texts, but they can listen to pod casts and watch videos. If I were teaching beginners again, I would make use of these kinds of resources.  Back in the old days, when we taught intermediate reading, most of us paid attention to only two things - the readability level of the material and whether the students would find it engaging. Nothing wrong with that as far as it goes, but there was too little continuity to my reading choices; it was too scattershot. If I were back in the classroom today, I would still use fiction or poetry. But, I’d also try to design units based around a coherent, sequenced approach over several weeks to a topic in science or social studies. Of course, HiSET (formerly GED) teachers have always been content-driven. However, they may face a different set of problems – how to find time for Tier 2 vocabulary or helping students gain fluency with complex texts. 

Finally, when I left classroom teaching in 1996, technology meant time in the computer lab where the students basically learned to turn the computers on and off, type, and write emails. We also had a few pieces of crude educational software – on floppy disks! If I were back in the classroom today, I imagine I’d be using technology in three ways:

  • For teaching – I might use brief YouTube videos to set up content reading, and I'd take notes of classroom discussions on the Smart Board.  We’d also work on how to use the Internet for research, with an emphasis on evaluating sources and evidence (now a CCRS principle).
  • But I’d also try to help the learners develop their own technological literacy – email, of course, but also more creative endeavors like using cell phone apps to make videos and presentations.
  • I’d also make sure the both beginner and intermediate learners were aware of how to get the most from assistive technology – speech-to-text and text-to-speech, translation, etc.  
  • I’d also like to experiment with using technology to help learners keep with their class when they have to miss school for a period of time. We know from research that most adult don’t miss class for frivolous reasons – maybe their childcare breaks down or their boss has changed their work schedule. All of you who have been teaching remotely for the last year may have some good ideas on how to deliver instruction remotely once we're back in face-to-face classes.  

 

Hello John and everyone, One of the Focus on Basics articles you wrote in 2006, John, "More Curriculum Structure: A Response to 'Turbulence" has stuck with me over the years and definitely shaped my practice. There is no question that having more structure in our teaching can make a difference for learner persistence. I think a lot of members would appreciate reading this article as well as your piece on the limitations inherent in silent reading tests, which you've outlined so well during our discussion here.

Thank so much for sharing your wisdom with us this week!

Take care, Susan 
 

Hi Susan,

I've officially "checked out" of the discussion, but I did want to sneak back in to highlight the importance of curriculum structure. Even in the pre-technology "old days", I wish I had planned better organized units with materials prepared several weeks in advance. Better organization on my part would have meant that when a learner had to miss classes for a week or two, I could mail them a packet of work. This would have helped them to keep up and stay engaged. And, who knows? Maybe it would have motivated them to get back to class sooner. Back then this would have been harder to do with ABE beginners, because so much of what they need requires aura/oral print-to-speech/speech-to-print work. But today, technology might help to present some alphabetics and word reading practice remotely .

Thanks for your generous participation, Susan.

Warmest wishes,

John  

Hi Everyone,

First, let me thank Dr. John Strucker for sharing his expertise with us over the past two days. He graciously gave us many hours of his time to carefully answer each question and consider each comment. Thank you so much John!

Please take some time and review the two-day discussion on Assessing the Four Components of Reading. Consider your practice and reflect on the following questions:

What is something in the discussion that:

1. Reinforced something good that I already am doing?

2. Reminded me of something I used to do and will try again?

3. Gave me a new idea that I plan to try?

This discussion remains open, so please continue to share your thoughts!

 

Thanks again to everyone who participated!

Steve Schmidt, Moderator

LINCS Reading and Writing CoP