7/1/2013
Avoid Misleading Claims About Adult Literacy Based on PIACC Results
Tom Sticht
International Consultant in Adult Education
On June 28, 2013 the Centre for Literacy in Montreal, QC completed a Summer Institute on “LEARNING FROM IALS, PREPARING FOR PIAAC” (Programme for the International Assessment of Adult Competencies). To inform people about the Summer Institute, the Centre produced a research scan for June 2013 with some 28 reports for background reading. Whereas most of these reports discuss the international assessments without questioning their validity, three reports critique these assessments and raise questions about their validity for portraying the literacy competence of a nation’s adults.
It is important that the field of adult education understand the conceptual and technical limitations of the PIACC whose results will be first released in in the in October 2013 with government reports and some non-profit organizations touting, likely uncritically, the results of the survey for the field. The following are ABRIGED versions of the three critical summaries provided in the Centre for Literacy research scan [available online at www.centreforliteracy.qc.ca].
1. Hauser, R.M., Edley C.F., Koenig, J.A. and Elliott, S.W. (2005). Measuring Literacy: Performance Levels for Adults. Washington, DC: National Academies Press. Retrieved May 14, 2013, from http://www.nap.edu/catalog/11267.html.
The U.S. Department of Education asked the National Research Council (NRC) to recommend literacy performance levels for the 2003 National Assessment of Adult Literacy (NAAL) that would permit comparisons with the 1992 National Adult Literacy Survey (NALS). …The Committee recommended … that a response probability value of 67% be used as the cut-off score for competency levels, as opposed to the more stringent 80% used in NALS, IALS and its successors. … The Committee warned against drawing sweeping conclusions about people’s literacy skills based on literacy survey results, since the test was designed to assess “information processing” employed in a range of tasks associated with the concept of “functional literacy”, and excluded the other uses of literacy, such as personal enjoyment, religious observance, and independent learning.
2. Sticht, T. G. (2005). The New International Adult Literacy Survey (IALS): Does it meet the Challenges of Validity to the Old IALS? Available at http://www.nald.ca/library/research/sticht/ials/ials.pdf.
Comparing the results of international assessments such as IALS with self-assessments of adult literacy, we find that most adults who are found to have inadequate literacy in these tests do not think they have a “literacy problem”…. IALS is ambiguous in its use of “real world tasks”: because of the complexity of the cognitive processes involved, it is unclear what specific knowledge or skills are actually being measured. Furthermore, these tasks may favour some groups over others – for example, older adults may be at a disadvantage due to the nature of the tasks. … In IALS, one must have an 80% probability of correctly fulfilling a task at a given literacy level to be considered proficient at that level. …this is too stringent [see Hauser report above]. … Sticht also asks on what basis IALS reports claim that Level Three represents the level needed to function in today’s society.”
3. St. Clair, R. (2012). The limits of levels: Understanding the International Adult Literacy Surveys (IALS). International Review of Education. Volume 58, Issue 6, pp 759-776
This article discusses the background and design of the “IALS family of surveys”. The author argues that while these surveys have value because they measure a particular type of literacy abilities strongly linked to economic outcomes such as income and are good at showing the links between this type of literacy and factors such as age, gender, education and occupation, they also have limitations and should not be used to make sweeping claims about the distribution of human capital across a society, or to make comparisons between countries over time. …The author …also notes that the literacy ability measured by the IALS surveys … a limited range of “functional” texts …and other studies suggest that the circumstances in which the test is taken influences results. St. Clair argues that while there is potential to build on the strengths of the IALS surveys, it has gone unrealized because of the persistent use of results to make misleading claims.
