Multifactor Placement for Adults

Dear Colleagues:

Have you found a "reasonably elegant solution" for assessing adult students entering college, other than the high-stakes placement test?

That's the question posed by Matt Reed, author of the Inside HigherEd blog, Confessions of a Community College Dean (see article: Multifactor Placement for Adults).  He is looking for specific practical, fair, and effective practices.  What do you suggest?

Cynthia

[Spoiler alert:  Reed thinks one- or two-week "boot camps" are okay but basically fine-tuning the single high stakes test.]

Comments

Hello All!

Our admissions office told us today that COMPASS will be going away in the next year and they wanted to know what assessments are available for college placement?  Accuplacer is used in other areas but is there another?

Brooke

(Got lucky and hit a working link to reply!) 

We are looking really hard at Accuplacer because we need to make a decision *fast* but our college has also been meeting and talking about solutions to good assessments.   The consensus is that just one assessment is just a bad idea.   (I also don't know whether you can, as we do w/ COMPASS, have some questions of your own that you add.   We have a supplemental set of basic arithmetic questions so that if a person does poorly on the 'pre-algebra' test we get to see if they know basic arithmetic. If they do, they get into Pre-Algebra no matter how badly they did with those questions.) 

Anything that happens one time is going to have the "snapshot effect" -- bad day, anxiety, etc. issues.   We are going to be trying out some other tools that assess other things that include time management skills, just how crowded your life is already, and ... I don't remember what else (I just hear updates at meetings about it ...)   

When it comes to math, I generally smell a hidden agenda:   the desire for, somehow, students to *be* better prepared for the math demands of college than they are -- and/or to have the math demands of college be more manageable & realistic. A week's "boot camp" can, yes, refresh the procedure for a 30-60-90 triangle, but applying that is another story.   There's also the sneaky habit the assessments have of presenting problems that are just slightly twisted from the "standard one you remember" -- giving extra information, making sure you're looking for "b" and not "C" in that right triangle so if you just plug in the formula, you're toast... 

One thing to be aware of is only certain approved government tests can be used for the ability to benefit option for federal financial aid purposes. Ability to benefit  was taken away a couple of years ago, but parts of it are going to be re-instated. This is important for students who don't have high school degrees or like many ESL students don't have access to those degrees or if they do, don't have the resources to pay for an official evaluation. The only test on the approved list besides COMPASS that has an ESL test is ACCUPLACER.

Hi Terry:

Excellent point regarding recent updates on the Ability to Benefit.  Here is a letter to colleagues on ability to benefit, mentioned in the June 1 OCTAE Connection.  Toward the second half of the letter is the current list the tests.

I am wondering, too, about the broader assessment processes in adult education and higher education that help inform adult learners (perhaps students preparing for their state's high school equivalency exam) about their readiness for college.  What are other ways that we and our students know they are ready?

Cynthia

Why is everyone blindly accepting that the expensive commercial tests are the only way to assess? Is there some requirement? 

It seems logical that giving incoming students "real" tasks = realia, and grading their skills or achievements would provide a fair and equitable assessment. 

If Ss can read a newspaper article, or short story, or several (a few) pages of material from a course, why would it not demonstrate competence, skill level, or ability to perform in reading?  Apply the same process to all competencies, skills, abilities that need assessing, and the results would be sufficient to actually reflect that student's likely-hood of readiness.

Why are we allowing the commercial test producers to con us into buying their products? Because they lobby harder? Or because it's easier?

Someone can create the first version - to prevent ad nauseum committee discussions, post it, get feedback, revise, repeat. The technology and content exist, why not bypass the costly, and not nearly as good commercial tests?

Arthur,

I agree wholeheartedly with you.  I am all for creating "real" tasks but for my college it is hard to find people who are willing to create these types of assessments and they want to take the easiest route (which is why they purchased COMPASS).  I agree that these assessments aren't good for adult learners and they do not accurately place students either.  The questions that I hear from the college are: who is going to monitor these tests? who is going to create it? who will grade it?  etc.  The college would rather spend money for something already made that is fully automated.  It makes me just shake my head at them.

Brooke

Dear Arthur, Brooke, and Others in the Same Boat:

First of all, I want to say "thank you" to both Arthur and Brooke.  I think you have described the overarching tension when helping students locate the best place to start in college.  Although college readiness and placement seems to be an issue for all students, both traditional high school graduates and nontraditional students, we might focus on nontraditional adult students that are often left out of the discussion or excluded from strategies because they lack a recent, traditional high school transcript and diploma (See Where to Begin). 

Has your institution and/or state rethought the way they work with students to assess where to start in college?  What is happening?  Are adult students able to access new entry strategies?  Do assessment options seem effective?  Do they have efficiencies in the short- and long-term?    And, how are career pathway programs for lower skilled adults addressing this issue?

Cynthia

   How long does it take to have a student read a few pages of material from a course and then "demonstrate competence"?   How many competencies and skills and abilities would need to be assessed?   It's not a short, straightforward process. Answering those questions is the ad nauseam issue ... 

   Then there's the whole technology thing.   Our "post-traditional" students didn't come through school using the tools the younger folks have been immersed in (tho' may not really know how to use them effectively for academic purposes). 

   We're always analyzing our assessments and our "cut scores."   The reading is considered reasonably effective and hopefully Accuplacer will be too.   Students read passages and answer questions... which is what they do in their courses.   

   I'm really curious about the "non-academic" assessments, because I think they just might be a worthwhile tool.   This way the student who may struggle with academic reading -- perhaps because they haven't done anything remotely resembling it for 10 years -- who has, basically, the self-discipline and habits to acquire a skill that will take sustained time and effort that another student might not know how to give -- can get a higher placement because they need less of our "how to learn in college" parts of the courses. 

   

I have been disgusted with the for-profit industries milking of educational resources at all level for many years. In mentoring and coaching teachers for over a decade, I have seen great strategies, great engagement and great people stuck in vortexes (time, bureaucracy, ignorant social pressures ...) that draw all those positive energies out there into feeding the negative cycles around education. After a few local discussion groups, after connecting with a few "early adopter" types, I am becoming more and more energized that we might be getting to the point in which teacher collaboration may be the most powerful option to getting what we want to see at all levels of education.  The texts and tests and other supporting materials offered for profit out there are so quick and easy to plug and play into our daily needs so we can turn our energies to combating all those other distractions that are part of the work we do. I propose the following idea for others to chew up, digest and report on how this tastes:

Massive amounts of money are going into resources in systems at many levels. During the summer months, when many educators have a little breathing room, divert a very small percentage of those resource costs into supporting people's time to get together and collaborate. From a financial standpoint, collaboration is relatively cheap if you look at the end product (once people are trained well in the art of collaboration of course). Collaboration's biggest expense is compensation for time. Looking at rates adult education pays teachers here in the state of Maine, I think it is safe to say we could pay teachers double their rate for a couple hours and still have used up what it costs for one cheap book. Using that equation of 1 book = 2 hours of human time does seem to favor the book in the short term. Remember that this is not an all or nothing suggestion though. I am suggesting we siphon a small percentage from our purchases, to get the ball rolling to create the collaboration system. What many do not factor in is the ability for home grown resources to be shared with each other. I am still looking for that optimal system of sharing and there are tons of efforts out there to build repositories, but the efforts thus far have been quite individually centered on specific "local" goals and national collaboration of resources are quite weak at the moment. As we improve our national sharing system (Imaging if we treated lessons or activities as free apps and they were shared that way?), we open ourselves to exponential returns on our investment in human capitol.  If we had 100 small collaboration teams meet around the country in teams for just a few hours to create one really nice activity with assessment each (over achievers may even include extensions), that would equate to at least 100 new activities we can add to our collective resources. That assumes just face to face efforts. As people get more comfortable with what collaboration really can feel like, we can migrate some efforts over to asynchronous experiences which can easily double the annual number of activities produced. Combined, I think it is realistic to think we could have at least 300, well collaborated and designed activities available to the educational communities each year. This again is only with a small percentage of text/test funding being diverted. As our pilot group finds success, which is assured because there are so many awesome educators out there just looking for the right forum, larger and larger percentages of that test/text funding can be diverted to the generation of collaborative products. Effectively, we can start using the money we already are giving to others to create our own production systems with our own human capitol. We have more human capitol than any for-profit agency can muster, we are close to that tipping point in which our rich human capitol is finally becoming more valuable than the financial capitol that seems to be more of a chain holding back progress rather than fuel to keep things progressing. How close do you think we are to really working on systems and methods of maximizing our human potential within our field instead of diverting resources to outside agencies claiming to have our next holy grail available, at an incredibly low price of course? Is the time right to propose Educational Collaboration Camps? Do we have technological tools in place to construct Education Collaboration Sandboxes? I don't personally think we are quite there yet on all counts, but I think I can see parts, and I think we can make this happen sooner rather than later based on the frustrations I hear daily online and in the field. Perhaps THE ANSWER we are looking for in terms of resources, diagnostics, assessments is dependent on our effectiveness of our collaboration infrastructure?

Hi Ed and all, Personally, I think this idea has real merit. Although it would be quite a challenge due to limited funding, I do believe it would be possible. You suggest that this could be mostly about the way and the timing for how monies are allocated (i.e., fund time for teachers to develop curriculum versus funding the purchase of published resources), but of course, this is a complex issue. We certainly seem to be facing a dearth of quality published materials that are well aligned to the standards. I agree, too, that we have a lot of talented teachers who would be eager to develop the materials we need. If we shared them through OER, this could very well save money in the long term. If this effort was effective, it would not only provide the field with much needed resources, it would be fantastic professional development for practitioners. Thanks for sharing your creative and big picture problem solving ideas, Ed!

What would it actually take?!

Cheers, Susan Finn Miller

Moderator, College and Career Standards CoP

It is frustrating how few resources available now are even partially aligned to the College and Career Readiness Standards (or even the HSE tests they're designed to prepare students for...), and I really don't forsee publishers making changes at anywhere near the pace needed. In addition, there are aspects of the CCRS that I think are difficult to address completely in a traditional textbook format. I think that a collection of materials developed and tested by teachers would be amazing, and I would love to be part of a program like this. I'm not sure how funding would work out (that is definitely not my expertise!), but between materials and professional development budgets, many programs might be able to swing participation for at least one teacher--if the administration was able to see the worth of such a project.

I've been hearing a lot recently about the Japanese practice of lesson studies, and I think the thing that is missing in the plans that I've heard is the final step: when a lesson has been taught and improved by a group of teachers, it is then published for other teachers in the country to use. This is how the teachers can spend so much time perfecting one lesson: their other lessons come out of a repository of lessons developed in the same way by others. I think creating such a repository for adult educators would quickly reveal the amount of creativity and ingenuity that is out there.

Teaching may usually be an individual sport, but that doesn't mean teachers need to be isolated.

 

 

Rachel you are spot on with the mention of lesson studies and the CCRS-SIA Phase 2 system that is rolling out this year in 12 states has a final component completely centered around lesson studies. I know Maine will be starting up training on lesson studies this spring and we have high expectations that this practice will be quite popular with many of our early adopters. 

I would like to offer an opinion about "what is needed" for real collaboration to get going nation wide. I have heard from many people, that money and funding are major issues in so many projects I have engaged in. I do not mean to disrespect that money does have value in any effort, but I would like to counter that I believe there are plenty of ways to get this going that do not involve a single extra cent of money in the system. Now that some are spitting out coffee and rolling up their sleeves in defense, allow me the flight of fancy below: 

The largest barrier to collaboration over distance is in the tools and systems currently out there right now. Every individual tool I have studied and used can do some wonderful things and has some severe limitations in terms of collaboration. I know we have the OER commons and we have well over 20 other wonderful repositories for materials nationally, but these are tools that are good for warehousing items, not connecting people. Many of the online tools that are good at connecting people (skype, google hangouts, other paid video conferencing ...) are deficient in storing and retrieving materials from the community. Then we have the creation element in which very few tools allow good collaboration on creating the lessons and activities over distance and time (Google Drive is awesome of course for this). What we need is a system of working with all phases of collaboration that does not involve complex technological shifts that will close some people out of the conversations. One tool to rule them all would be great, but I have not found such a tool and I would propose that Google has come the closest to fitting most of the needs, but that still is not at a level I believe we need to make collaboration really work. Please let me know what you think of these needs:

  1. Storage: Once good work is created, we need to be able to store and retrieve this work in efficient ways. It is ironic that so many repositories today that claim to be standards warehouses are all organized for retrieval based on a grade system (K, 1, 2, ...) instead of any indexing to the standards themselves. This is a HUGE flaw in most of the warehouses out there right now and must be fixed for our collaboration to really be of any use.
  2. Asynchronous Communications: LINCS and many other forums exist to help people share ideas around our needs, solutions and good practice. So many of these tools were designed by very talented technicians that may not have been concentrating on the end user's needs as a focal point. As such, clunky interfaces, difficult to navigate organization of content, and even occasional glitches offer way too many barriers for an easy to access collaboration effort in terms of asynchronous discussions. 
  3. Synchronous Communications: I may be quite ignorant here, but I am not finding any one resource for video or audio connections that the field is gravitating to. Many have experiences with many tools, but we lack a universally acceptable system. Each tool like Skype, Google Hangouts, GoTo Meeting, Adobe Connect and others each have a bevy of talented individuals out there with experience with the tools, but having personally used most of the tools out there extensively, the biggest challenge here is in finding a tool that everyone would feel comfortable with. This is very much like picking one type of ice cream we would all eat if we sat together on a hot day. Everyone has their favorites and although they are all flavors of ice cream, the opinions, experiences, and comfort levels of any one flavor will alienate many. 
  4. Creation Collaboration: I am bias here, but I feel this is one area we do have a free and wonderful system of tools for. Between Google Documents, Google Forms, Google Hangouts (synchronous shared white space feature specifically) I would challenge anyone to come up with a need for digital collaborative creation we can't engineer in this system of tools. There is a dark side of course. Not everyone trusts in the cloud or in any one company that offers storage in the cloud. Although companies like Google and others invest more in security than the combined IT funding in all of our states combined, there are still insecurities that the supporting company still has access and wrests some level of control.
  5. Integration: I think that if we were all locked into a room for a weekend and charged to come up with a system to crank out, I believe we could find great means of doing each of the 4 things above. What would be a huge challenge is in creating that integration of systems in such a way that the workflow makes sense and is easy to use in the field. If the end product(s) are not efficient and easy to use, we are wasting people's time and they will resort back to easy to access, expensive incomplete tools because that is the norm behavior already established. 

Note that no mention of money shows up in my critical list of 5 challenges above? Let's talk about money now. There are three main needs for money in this effort. Creation of the system, Creation of the lessons / activities, and maintenance and improvements over time. 

  • Creation of the system: In my experience creating systems there are two models of financially supporting innovation. First is the pitch and sell model in which ideas are flashed in front of the dollar holders in the hopes that they buy the pitch and throw money at something. The other model is the build it and the money will come model. This last model is much less used in many business worlds today, but if any of you have been to YouTube in the last 5 years, you can see how individuals are building excellence and going viral and the money is there for them. I have been working under build it and the money will come model and it obviously has not made me any money significantly, but it has created many systemic changes that have been significant to the point that stake holders are looking for money to support that work that started pro bono. We are entering an age in which people want to see the product working before the money flows as so many of the "buying the pitch" fails in execution. So to fund the creation of the system, I would propose that if a team of us is really passionate about building the right system, our donation of time will be well compensated once the product is accepted and adopted. For some this may seem a leap of faith and I can certainly respect that most will be uncomfortable with this model, but it is a working model today outside of education. I believe that with a system at least prototyped, grant funding would easily flow to support the creation efforts of such a powerful education movement. Education has some wonderful grant funding sources that would love to see effective collaboration in action. 
  • Creation of the lessons / activities: This one is actually a bit easier to fund. Every adult education program in the country has a professional development line in their funding I think. Initially, a program would spend some of those funds in year one to get their staff trained and comfortable with face to face collaboration efforts (see CCRS-SIA Phase 2 materials and system for a nice model). Once that training is effective, then we only need one 3-4 hour staff meeting a year dedicated to digital collaboration to really have an incredible number of people involved in a national collaborative effort. Some of those efforts may be synchronous, but I suspect most will be asynchronous. Meanwhile, there are ALWAYS early adopters that will be donating time to get samples, prototypes, and guidelines established. Something in the genetic make up of these early adopter "puppies" attracts them like a moth to flame to this start up work. Tossing these puppies a bone in terms of compensation would be a respectful gesture, but I personally know that early adopters do the work primarily for the passion and love of what they do. Getting paid is always nice though
  • Revision and Maintenance: Every vehicle needs fixing or tweaking to remain useful. We have people supporting these forums for instance. I don't see the financial cost formulas or sources needing to be much different BUT I do have an idea for you to chew on. Let us assume we have the vehicle built and are moving along nicely. Don't you think it is possible that an effective, home grown system would be attractive to for profit agencies? The model is already there! On YouTube, there is an individual by the name of Phillip DeFranko who offers 10 minute daily news recaps in a very unconventional manner. He is rude, crude and often unacceptable to introduce in an educational setting (may want to click the link at home), but his innovation and style placed him with more followers than ALL of the major news sites COMBINED on YouTube. He consistently gets job offers from the big news companies which he politely refuses with, "Why should I go to your medium when the people are clearly coming to mine. I would offer you a spot in my show if you would like." ( https://www.youtube.com/user/sxephil ) This model of build it and they will come applies to for-profits that wish to attach themselves like barnacles on the side of a massive whale or ship that is moving fast. Imagine major industries wanting to fund education like never before because they finally realize the power of our professional human potential to affect change in the lives of others. No more of this silliness of us paying for hardware or software that we effectively advertise to our students for 12 years worth of advertising. (sorry more on that in another post some time). 

I am sorry for the long post, but as you might have guessed, there has been much thought and discussion out there already about trying to make this system happen. I continue to study as many angles and issues as I can and I am hopeful the community here can offer perspectives that differ or support the above so we may all learn to build something better. Do you still see money as the primary hurdle? If so, perhaps locking a few of our talented LINCS people in a funded room for a week might produce the best results for the money invested? It would be great to hear from others to see what the field believes are the biggest barriers and greatest possible solutions to any challenges shared. 

Edward has a great idea. Hope a lot of you get on board, and make it real.

There is a huge amount of Teacher created materials on-line. Just google it.

Why not use one or more Crowd Funding sites to get started? There are lots of them.

And use Open Source software. There are tons of it available at zero cost. 

Would Lincs, or its parent org provide a central repository?

We, Elaine KIRN-Rubin & Arthur Rubin at AUTHORS & EDITORS at 2Learn-English.com, would donate some of our content for teachers to correlate and post to the central repository - NO Charge. All of Elaine's creations are Adult Education for English Acquisition.

Wish we could do more, but we don't get enough sales to break even, so we continue to fund our company out of our own pocket. 

BTW: We are signing in the Xprize Adult Literacy Competition. Anyone ready, willing, and able to help us load our content onto our chosen (once we actually choose) learning platform? It is a huge undertaking that will make 16 books of print available as interactive, online materials for download and use. 

 

I could chip in some time tho' I'm not sure what you mean by "loading our content."   

It occurred to me that along with Canvas and D2L... it's  just *possible* that lynda.com might be approachable.   LinkedIn recently acquired them and seems to be expanding its content to more basic "skills you need to get hired."   Their videos & courses have pretty high standards.  

I know one of our teachers spent hours and hours and hours recreating a document about objectives and CCRS, with the full understanding that it would then go on a shelf and sit somewhere and that yes, those hours could have been spent ... making materials for students to use or even ... working with students or teachers... 

   The OER movement seems to hold revision and refinement in high esteem and I think there's a lot of potential there. http://phet.colorado.edu/  is one example... 

   D2L (Desire to Learn -- tho' I *think* they're trying to go by "BrightSpace") and Canvas are really *trying* to make this kind of thing happen.   Each has options for "open" courses and our school will be workign with one of their instructional designers. IT seems to be a pretty good infrastructure. 

Last academic year my "when I'm not inundated with students in need of tutoring" time was spent with the OER-STEM project.   I'd like this year to be spent *making* OER and Android apps.   There are "design patterns" for all kinds of business apps, so I'm going to see what's out there for educational ones and see about designing them for our visual-kinesthetic learners.   

... I would also entertain getting a collaborative group together to work through some of the lynda.com tutorials (our school now gives all employees & students membership !)  and apply them to educational applications.   Lynda.com just has the videos -- not the rest of the stuff of a course (practicing, collaborating, etc)... 

Hi everyone.

Since I had one foot in adult ed and one in community college math for 5 years, I have a comment about the Accuplacer.

Colorado community colleges used if for years, and it did not produce accurate enough results in math. I had students the first day in my basic math class (whole numbers, decimals, fractions) who had passed pre-calculus in high school. I always asked: If you think you are in the wrong math course, see me after class.

Finally, Colorado worked with Accuplacer to develop a Colorado-specific test that the people in this state thought was more reliable. That was going on in 2013. I don't know whether that version is available in other states. Since I ended my community college math teaching in December 2013, I don't know the current status.

I agree, one test is not the answer, particularly when it is multiple choice for answers. Even when the student has to enter digits, the only result is the final answer. I want to see HOW the students arrive at the answer.

Dorothea Steinke

Lafayette, CO

 

Hi Dorothea:

The English teachers at the community college I worked for a long time ago did a similar readjustment during the first few classes and, hopefully, before the end of the Add/Drop period.  It seemed to be pretty effective but it didn't necessarily make a more informed system or student.  That reminded me of a few other possible solutions:

(1) Extended orientation by the college with students spending an intensive week or more learning about the institution, academic system (e.g., what is developmental education?), and student services, and interacting with a panel of adult learners who discuss their experience.  Activities do include brushing up for the placement test.  This helps a group of students that are fairly well prepared (and not overwhelmed by computer-based tests).

(2) A longer-term transition between adult education and college where students take a college course and workshops before leaving adult education to better understand their academic and personal readiness (e.g., the college's student success course).

(3) Bridge and career pathway programs where the placement test is given farther along (or, in a few cases, not at all).

I wonder if others could comment on methods that they use that go beyond the test.  How have they worked?  Is it possible to move beyond the single placement test?

Cynthia Zafft

Postsecondary Completion Moderator

 

 

 

I have read the comments and agree that high stakes and placement testing do not always indicate academic preparedness linked to success, but my questions are:  

  1. What are the true skills that we need to assess? Some discuss reading a passage /short article to demonstrate reading comprehension. Is this an accurate assessment of the various types of reading a student needs to demonstrate to be successful in multiple academic areas?
  2. How do you demonstrate writing abilities - thinking that writing is also about gathering information for research papers and demonstrating these skills accurately?
  3. What about Math? What are the real skills that are needed?  

Discussing alternative assessment ideas without having a very clear idea of what academic skills are needed is a very complex issue. Creating valid and reliable assessments is a very costly endeavor. Does an instittuion really save money when attempting to develop and implement independenly created assessments? I am not advocating for expensive tests, but rather - tossing additional ideas for consideration. 

Kathy 

 

 

Dear Kathy and Colleagues working with the CCRS for Adult Education:

Would using the College and Career Readiness Standards in Adult Education let a student and their program/institution know if the student is ready? Could it form a basis for a readiness assessment? Many states are working on this right now.  What have you found?

For example, take a look at the Reading Anchor Standard 1 for Reading at the E level (highest level).

Anchor 1: Read closely to determine what the text says explicitly and to make logical inferences from it; cite specific textual evidence when writing or speaking to support conclusions drawn from the text. Level E:  Cite strong and thorough textual evidence to support analysis of what the text says explicitly as well as inferences drawn from the text. (RI/RL.9-10.1) Application: cite specific textual evidence to support analysis of primary and secondary sources, attending to such features as the date and origin of the information. (RH.9-10.1) Application: cite specific textual evidence to support analysis of science and technical texts, attending to the precise details of explanations or descriptions. (RST.9-10.1)   How do see this standard being assessed?   Cynthia Zafft Postsecondary Completion Moderator

There is no such thing as "specific practical, fair, and effective practices" for assessing. At least not that I have ever heard of. Like so many issues in education, and Adult Ed. in particular, assessment of humans by its very nature  cannot be practical, fair, and effective, or elegant, and also be accurate. 

Until we accept this, we will continue to chase after the holy grail, wasting out time and money, and failing our students. When teaching becomes elegant and canned, and no longer needs a human to facilitate it, maybe we'll be able to assess better without human involvement. For now, we should save our resources and use them to provide the educators needed to properly assess students. 

Many, if not most students need individualized facilitation. Yet we do not have the resources to deliver this. Assessment is more complex. 

You want reasonably elegant? Then Keep It Simple and Straightforward! Performance on a real world task is as close as we can get. 

I'm sorry to burst this bubble, but we are being sold a bill of goods, and have been for decades. And the Feds and States have wasted billions chasing this magic bullet, promoted by big publishers and academic researchers. 

Face it: There is no "single high stakes test' that can truly do an accurate assessment.

 

 

Hi Arthur:

I think you're getting at the point of the article, that it is not a one-time, one-test event but a matter of more holistic assessment, placement, and support. And, while I think academic skill is usually the biggest driver, I don't think that giving someone a single reading, writing, or math assignment will do the trick, either.  It usually takes more than that for many adult learners to succeed when they go on to college and both adult education and postsecondary education have roles to play in the process.

Cynthia

Good day all. I wanted to share a thought and see what reactions come of it. Teaching is often referred to as an art and many experienced practitioners can share stories of how individualization or contextual application or maybe some other focus helped with a specific individual. It is often an organic process involving multiple touches over time. With so many feeling this way in education, it is quite disturbing to see how important the one-off standardized testing frenzy has developed over the years. Would the following project help the field at least have assessment discussions that could possibly produce universal products that might be useful?

First, let us establish that the College and Career Readiness Standards (CCRS) are a suggested framework of learning that highlights some very important goals, offers many flexible ways of creating experiences, and probably most importantly the developing of assessments that highlight learner successes in each standard would be useful. If that is true, it seems to make sense that we start with the standards and have some discussions around developing a good rubric for each individual standard. This collaborative rubric becomes a lens any teacher can use to look at any activity to gauge what level of success a learner might be demonstrating. In this way, any learning experience can now be valued differently simply by which lens (rubric set) the instructor chooses to focus on. These lenses, if well designed and agreed upon, become an invaluable tool that offers a common language when we say a student has some proficiency with anything. Please realize that there may be many interpretations of what a "good rubric" is and I believe we have experts with some research based opinions out there that should be in on design discussions. 

One work flow that comes to mind: Some collaborative space has all the standards listed individually with nice big boxes to house the different levels of student mastery worth assessing for that individual standard. For example, maybe there are three boxes that describe the student actions indicating three levels of success (Beginning to Explore, Demonstrated Some Success, Demonstrated Success). 

Additionally our collaboration space has a discussion space (forum) that is dedicated to just that one standard discussion. The whole focus is to develop descriptors that can be universally understood, accurate and focused on the students ability to demonstrate the goal of each standard. In each box is a succinct description of what student behavior might look like at each of those levels of accomplishment. This is much more than simply <50% there, > 50% there, 100% there. It should be a description of what observers would see in the student work at each of those levels of success. 

Armed with such a set of assessment lenses and a means of tracking student progress (I have a system already in development so there are ideas in motion around the country on this already), doesn't our jobs as educational coaches become more free to be that organic process many exclaim is the heart of real learning? 

Sadly, past experience and current policies/laws force many into a mindset that is quite divergent from what a standards based educational system is. Perhaps the system needs to be constructed first, or at least a prototype before those rigid confines (time based, grade based...)  can allow for exploration of alternatives? This alternative would need to include a good collection of standards-based learning experiences, effective and efficient tracking of goals and progress that includes student evidence, and reliable and consistent means of assessing standards. I feel very close to the tipping point of having all three of these aspects at least become part of discussions. If we build it, the rest will come. 

What are your thoughts on tackling each individual standard to aim for a good rubric? After we ensure everyone is on the same page, our community could easily flesh out one standard in ELA and Mathematics a week. That is 52 standards a year. I know that does not sounds like a huge total, but that is 104 assessment tools that could be an asset to our education community and it would not cost any of us a dime or a tremendous amount of time. Additionally, I feel that with the system established after a few standards, we have early adopters that would go nuts accelerating frameworks that can expedite discussions. As a bonus, our field would become so much more aware of what each standard is trying to suggest. Even after individually diving into CCRS for the last couple years, I feel I need so much more time looking at what each individual standard really intends and how it fits together with other standards in interesting ways. Getting a great set of tools and having a free forum of CCRS PD sounds like a win win to me. How about you?

Hi Ed, The ideas you've been sharing about the potential for collaboration related to standards-based lesson development and  now assessment seem quite doable and worthwhile to me. You wrote, "This alternative would need to include a good collection of standards-based learning experiences, effective and efficient tracking of goals and progress that includes student evidence, and reliable and consistent means of assessing standards." All adult education programs would benefit from having these tools, processes, and structures available.

With regard to assessments, in particular, having assessment criteria spelled out for each standard at each level, would surely be helpful. I tend to think in very practical terms; thus, I'm thinking about the practical application of the assessments during instruction. When teaching a lesson, I rarely focus on only a single standard at a time. For instance, an ELA lesson that reflects the key shifts, will likely touch on many standards at the same time. It would be unwieldy to apply a rubric to each standard involved in the lesson, so I guess I would just have to figure out which of the standards is the more important focus of the lesson.

Keep the ideas coming! And let's hear from others who've been reading and thinking about Ed's creative thinking.

Cheers, Susan Finn Miller

Moderator CCS CoP

Good day Susan and all. Susan, I think you have voiced a thought that many of us have. "...will likely touch on many standards at the same time." When many teachers think about standards, it is from a teacher-centric focus. "Am I covering the standards?", "Did I include something that hit this standards?" and many other questions are all about what the teacher is doing. In a standards-based environment, there is a huge shift of focus to, "What is the student able to show me after engaging in these activities?" This is a very big change for most of us in the educational world, but it is a vital shift for adoption of a truly standards based system. This is also why so many of our textbook resources will continue to fail us, but that is another discussion. 

Let us look beyond that need for a minute. In any activity, there will be so many things that CAN be assessed. For the sake of discussion let us imagine an activity that has over 12 standards that could be assessed. In a classroom full of individuals, the teacher has the power to pick and choose which of those 12 standards apply to each individual in the class. Maybe student #1 needs standard x, y, z, but student #2 needs only standards j, m, q. If we have those individual rubrics (along with the tracking tools I mentioned before) a teacher is well armed to offer the same activity while assessing to individual needs. EX: Susan needs 4 standards that are part of the 12 I determined were in my activity during planning. When Susan comes in, I hand her those 4 rubrics and with each student that comes in I hand them the appropriate rubrics each person needs. The class is now armed with individual lenses they each person is focusing on today. After a quick reminder that all of us will be engaged in the same activity today, but all of us have different focus, we begin. During the activity/lesson, I can now circulate and touch base with each student to see how they are engaging in their personal goals. After a while, we stop class. Pairs of students share what they have done and how the individual thinks they are progressing towards each rubric they have. Their partner offers his or her perspective with suggestions on how to help get to the next level if ideas come to the student (remember the descriptions of the thinking and actions are clearly laid out in each rubric box). Then we return to work. Now we have peers all learning the same content but through different lenses and able to celebrate their individual accomplishments that can differ so much from others doing the same work. What a wonderful opportunity for peers to really start taking pride in the fact that we are all experts and we all have something to offer those around us. 

Can you imagine some call me an idealistic dreamer?  In the LINCS forum, we all have the freedom to share ideas, so please offer your thoughts on what you read. Different perspectives make every conversation much more powerful. Do you see challenges in the ideas someone shares? Do you see ways an idea might be extended or maybe an alternative implementation? No one of us is THE EXPERT and all of us have experiences and skills to bring to bear. Please feel free to offer your thoughts. None of us have to agree on anything, nor do our posts have to be long dissertations (sorry, guilty) but we do need to hear from as many people as possible to enrich discussions, opportunities and resources that center on the work we all love to do. When you read something and have a thought, take a moment and share that with the community. We will all be richer for that moment of your time and your contribution!

Hi Ed,

The complexity of the adult basic skills formative assessment system you have described may be daunting to some teachers who read your description. How, they may wonder, could a teacher possibly manage that complexity, individualize assessment and the instruction that follows, for each student? Could you connect your description to the kind of online formative assessment system you have proposed that we need to create? Specifically, what would be the features of an online assessment system that a teacher would need to focus in the way you have suggested, and at each step, how would the teacher use these features?


David J. Rosen

djrosen123@gmail.com

What follows is a sample just whipped up this morning, so please realize it may be quite imperfect, but I am hopeful it helps illustrate the human and digital work flow. 

Teacher would like to present an activity that includes pairs of two digit numbers (EX: 43 and 26, 15 and 15, 39 and 74...) She has identified that there are two CCRS standards that are needed within her class and would like to use the same basic activity to address both of those standards. She is thinking of her class as three main groups, A, B, and C with group A needing to demonstrate 1.NBT.5, group B needing 1.NBT.3, and group C needing both. She goes to our online rubric repository and pulls out the two rubrics and shares these with the class. She ensures that students understand what each rubric is asking for and what each box is describing. Here is what she may handout to the students for discussion. (Note: she could print this off or actually use the digital tool at this point). She shares that there will be exploration time for students to work alone or in pairs and then there will be a reporting time. This will be a written description (with or without pictures) that demonstrates what they have discovered during exploration.

Now the lesson starts up. She may have grouped students to allow for collaboration. As students work, she is encouraging learners to self identify what progress they are making in terms of which box. "Which box to you think you are on right now and why?" can often be heard as well as "What do you think we are missing to get to that next level?". Manipulatives are on the tables as well as paper and pencils for students to use to process and possibly to articulate their work with peers. As she observes progress, she shares that it is time to start recording their findings. The teacher can now turn to the digital copy of each rubric for each student (this individualized organization can be demonstrated in another post if desired). If you like, you can try this as well right now by clicking on the link in the previous paragraph. For each rubric, simple click in one of the boxes below a descriptor and place an X or some other value in there. If you wish to delete the color, click in that same box and delete and it will return to white. Note: the third column has two boxes below it to allow for specific indication of what the student was doing. Now the student and teacher can have discussions, compare notes, discuss remediation needed and work together to upload the student finished work into the appropriate digital portfolio collection the student has going (again can show this in another post). 

The initial discussion in this thread centered on assessing learners for college ready. What I propose above is obviously a learning process and is not a one-off test. Instead, this system allows the learner and instructors a set of student works (in the digital portfolio collection) that demonstrate standards as they are met. When the student is ready to apply to college, the student can quickly create a specific college portfolio by pulling samples of their work that the student feels best represents his or her ability to learn and be successful in college. Perhaps the student pulls a series of standards that shows massive improvements from a struggling start. Perhaps the concentration is on specific skills the student knows (from college and career explorations) are vital to success in a given field of study or work. College admissions people are now getting a quick snapshot of what a student can do, how they do it and how the student presents themselves in the way that these evidences are assembled for presentation. This offers so much more information than some random number a testing company comes up with. It also costs the education community $0.00 in terms of testing while providing flexibility for ALL expressions of learning, rather than multiple choice / essay strengths, to be used to determine a level of college readiness. 

In short, we may not need a better College Ready Assessment tool, we may need a system to help learners better demonstrate the ways they are college ready in a quick snapshot that really helps admissions officers understand this student's chances of success. Having a standards based formative assessment system allows for a powerful and efficient end product the learner can tailor to any school or job he or she wishes to pursue. 

Please let me know if this helps clarify a bit  These are simply thoughts. Suggestions, additions, criticisms by all are appreciated. 

Hello Ed,

This helps me a lot to understand the process, and also what you mean by a rubric. In my view it could have some of the characteristics of a competency-based instruction and assessment system. For example:

  • The standard/competency(ies) and measures are detailed, observable and known to both the teacher and learner from the outset. The learner knows what kind of knowledge and skills s/he is expected to learn and what will be assessed.
  • The learner has options for (and control over) how to attain the knowledge and skills (e.g. work alone, work in pairs, work in a small group, use math manipulatives, get additional help from a teacher or tutor, learn from a video, etc.
  • There is no need for a one-shot, summative assessment, as mastery performance can be reliably measured (and subsequently re-measured to verify that it has been retained.) Students who demonstrate the mastery can be given a certificate of mastery of the competencies. Note: often in a competency-based system a learner performance standard is set for the standard/competency/ies, for example the standard might be something like: over a period of x number of weeks, assess the same competency/ies at least three times and the learner will master it/them all three times. Many psychometricians who design assessments, incidentally, find this kind of assessment system more reliable than a one-shot test.

So, if I understand correctly, what you propose is a blended/hybrid (face-to-face and online) competency-based/performance-based instructional-and- (formative and summative) assessment system as an alternative to a one-shot high-stakes multiple-choice placement test. What you propose, while an alternative to a one-shot placement test, is also much larger; it's an integrated instruction and assessment system built collaboratively by adult education teachers from CCR standards. Do I understand correctly what you are proposing? If not, please clarify. 

Thanks.


David J. Rosen

djrosen123@gmail.com

 

David, thank you for consolidating the example into a very accurate summary of what I propose. In particular, your underlined items really hit at the meat and potatoes what is being proposed and a few key elements. 

The resulting system and educational framework we could create would revolutionize how learners can experience and express success and how we can all do the work we know needs to be done. 

One issue is language:   many of my students who are perfectly capable of doing what the rubric describes and even explaining it in their language would be pretty confounded by those descriptions.  A way to address the issue would be to show an example of what they're talking about... this is an example of where collaboration would help.   If I made an adorable graphic of adding ten to a number (and I'll probably be doing that this week), we could share it... with simplified langauge ("can you get the answer?" "Can you hint at how/why?"   "Could you explain it all the way?")   ... 

I also found this site today when one of my tweeple said something "sounded like a good MARS fal."   Oy??? GOogle got me http://map.mathshell.org/   and "Math Assessment Resource Service" (MARS) "formative assessment lessons" (fal).   

(I'm still not sure it just wouldn't be too complex a thing to do but it's *definitely* worth exploring since our folks have such diverse needs...)

Cicadas are pretty ferocious here, too.  Tomorrow's our "Prep Week" beginning and I am hoping that our "Making Android Apps" course gets enough students to "make"  :-) 

I agree completely that having a graphics library that help describe the basic verbs being used in the assessment rubrics would be a great asset. As a visual learner I know I would end up just looking at the pretty pictures to determine what I needed to do. Of course, I have been known to miss some very key instructions that way. Still, I think are learners at Level A, B, C would especially benefit from graphical assistance. I know in my case this would have to be a collaboration unless everyone likes smiley faces and stick figures

Susan, Thanks so much for sharing the link to the Math Assessment Project.

To all members, I just spent a few minutes reviewing the detailed teacher resource guide complete with student handouts and assessments for one lesson. Along the lines of what Ed is suggesting, this site gives us an idea of what is currently available-- a standards-based system for middle school and high school math, complete with detailed lesson plans, assessments and student handouts. Each lesson is aligned to Math Practices and specific standards and can be searched accordingly.

I invite those who have the time and the inclination to take a look and weigh in on the design of this site and the resources that are available. What else would you like to see?

There are likely other sites that can offer ideas on how to organize instructional and assessment resources in viable ways for our purposes. What is already out there for lower level math? ELA?

Cheers, Susan Finn Miller

Moderator, College and Career Standards CoP

 

Susan and colleagues, this is a great resource.  Here is one of the lessons that I've used with teachers preparing students for their state's high school equivalency exam (teachers then use it with their students).  It has the type of questions that students often find on the test (mixing reading and quantitative skills).  The first link will take you right to the lesson in all its very interesting parts (including a table on page 4 with common issues students have and list of questions the teacher might ask students help develop their understanding).  The second link will take you to the Lesson Overview which includes lesson goals and standards, which is perhaps more central to our discussion.

Lesson Plan:  Interpreting Data:  Muddying the Waters

Lesson Overview

The Mathematics Assessment Project (see about MAP) lessons are aimed at bringing the Common Core State Standards to life in down-to-earth, explicit performance terms.  There are several collaborators and a key funder.  What we're looking at is the formative and summative assessment activities.  I'm wondering if someone who is also in the Numeracy group might weigh in on how the lessons might work from the individualized Learner-Standard connection that Ed is talking about.  The sixth grade lesson, Optimizing Coverage: Security Cameras, might have more general application.

Cynthia

Postsecondary Completion Moderator

 

Colleagues,

Although no one has asked for it, I think this engaging discussion that began on July 29th with a question from Cynthia Zafft, is important enough to summarize for the several CoPs involved, that I am going to give it a try. I hope my summary may be useful to those who have been participating in or just reading the discussion, and may also engage some other LINCS members to join in. You will see that the discussion has moved from the original question about an alternative to existing college placement tests to a particular proposed solution to the problem posed, and some of the details of that solution.  If I have misunderstood or misrepresented any of the posts in my summary, please let me/us know and I can correct it.

The discussion began when Cynthia Zafft, asked if there is a “reasonably elegant solution” for assessing adult students entering college, other than a high-stakes placement test such as the Accuplacer or Compass.

Arthur Rubin replied, “There is no such thing as ‘specific practical, fair, and effective practices’ for assessing,” that most students need “individualized facilitation,” not a single high stakes test.

Cynthia agreed, and said that was the point of the article, that a good solution is not a one-time, one-test event,  and she added that it requires “holistic assessment, placement, and support” in which more than academic skills are assessed.

Brooke Istas and Susan Jones wrote that their colleges are interested in having alternative ways to meet the placement need. Susan added that the consensus at her college is that a process that has multiple assessments is needed, not just one “snapshot” test.

Dorothea Steinke wrote that in Colorado Accuplacer modified its math test to create a Colorado-specific version that some believe is more reliable. She agreed that one test, particularly one multiple-choice test is not the best solution to assessment for placement in college courses.

Cynthia was reminded by Dorothea’s post of some other solutions, especially for students coming out of adult basic skills programs: 1) an Extended orientation for them by the college, 2) a longer term transition between adult education and college, where students in adult education take a college course and workshop, and 3) bridge and career pathway programs  that are offered before the placement test is given.

Kathy Tracey agreed that high stakes and placement testing do not always indicate academic preparedness and asked three questions about what should be assessed in reading comprehension, writing, and math, and how these should be assessed.

Cynthia replied to Kathy’s questions asking, “Would using the College and Career Readiness Standards in Adult Education let a student and their program/institution know if the student is ready? Could it form a basis for a readiness assessment?” and she gave a CCR Anchor example.

Terry Pruett-Said cautioned that only certain approved government tests can be used for the ability to benefit option for federal financial aid purposes.

Cynthia agreed with Terry’s point, and Brooke wrote that she is glad to know about this.

Arthur suggested that we need to give students “real” tasks, i,e, realia to assess their reading skills, and that we shouldn’t rely on commercial test producers.

Brooke agreed with Arthur, but pointed out that for her college it is hard to find people willing to create these types of assessments, and to grade and monitor them. The college wants something that is “fully automated”.

Cynthia appreciated the thinking of Arthur and Brooke. She suggested that, “Although college readiness and placement seems to be an issue for all students, both traditional high school graduates and nontraditional students, we might focus on nontraditional adult students that are often left out of the discussion or excluded from strategies because they lack a recent, traditional high school transcript and diploma (See Where to Begin).“ She also asked, “Has your institution and/or state rethought the way they work with students to assess where to start in college?  What is happening?  Are adult students able to access new entry strategies?  Do assessment options seem effective?  Do they have efficiencies in the short- and long-term?    And, how are career pathway programs for lower skilled adults addressing this issue?”

Susan Jones suggested that developing an assessment like this may not be easy, that it’s “not a straightforward process”. She added that she is also interested in assessments of  “non-academic” skills or habits, for example those needed “to acquire a skill that will take sustained time and effort.”

At this point Ed Latham proposed a specific solution and much of the conversation then focused on this solution.

Ed wrote that he wants to see adult education teacher collaboration in developing an assessment system. He suggested that resources be allocated for teachers to collaborate face-to-face – in the summer months – in 100 small collaboration teams, each to create one good learning activity and assessment, and that these learning activities and assessments could be added to “our collective resources.” This face-to-face collaboration could then be expanded to include asynchronous online collaboration to produce more activities and assessments. He thinks some – but not all – of the tools are there for this online collaboration.

Susan Finn-Miller wrote that  she thought Ed’s idea had merit, and although it would be challenging to implement because of limited funding, that she thought it would be possible. She pointed out that few of the existing published adult education materials are high quality and aligned with CCR standards. She suggested that activities developed should meet OER standards and be shared “through OER” (OER Commons?) which would save money in the long run. She added that this would also be “fantastic professional development for practitioners” in developing the assessments.

Rachel Baron agreed that few learning resources are aligned to CCR standards and wrote that she doesn’t think publishers will be meeting this need soon. She wrote that she would like to be part of a project such as Ed described, and thinks teachers and programs would respond well to the idea. She wrote that teachers need to share their lessons with others in an adult educator repository. Rachel also mention Japanese Lesson Studies as a promising practice for lessons.

Ed wrote about the CCRS-SIA Phase 2 system rolling out this year in 12 states that focuses on developing Lesson Studies resources.

He added that a collaborative assessment effort such as he described could get started without additional money.

He also wrote that the biggest challenge is not money but finding the right tools and systems for a good assessment project design. From his experience, he wrote, there is no single, perfect, ready-made set of online collaboration tools, but many of the online tools could be combined. That includes, he wrote, OER Commons and “over 20 other wonderful repositories for materials nationally” that are good for warehousing items but not for connecting people. The tools that are good for connecting people, skype, google hangouts, other paid video conferencing, he wrote, are not good for storing and retrieving materials. He wrote that there are other problems, as well, and suggested that a system of tools is needed that addresses the following needs, each of which he described in some detail: Storage, Asynchronous and synchronous Communication, Online Collaboration in Creation of Materials, and Integration of these needs so that the end products are easy for teachers to use.

Ed added that there are three areas for which money is needed and he described each in some detail: Creation of the System, Creation of the Lessons/activities, and Revision and Maintenance of the system.

Arthur liked Ed’s idea and suggested some ways to fund it. He offered to donate 2Learn-English content to the repository.

Susan Jones offered to help, and suggested that lynda.com might be approachable (to host this system?)  Susan would like to see a collaborative group “working through”  lynda.com tutorials and “applying them to educational applications.“

Susan Jones added that “The OER movement seems to hold revision and refinement in high esteem and I think there's a lot of potential there. http://phet.colorado.edu/  is one example... “ She also added that D2L (Desire to Learn), now called “BrightSpace,”  and Canvas may be doing what Ed has suggested.

Ed proposed, and described in detail, an adult education  “collaboration space” as an antidote to “the one-off standardized testing frenzy” based on:

·      Using the CCRS as a useful framework for creating assessments that highlight learner successes, and developing a good rubric for each individual standard, a “lens any teacher can use to look at any activity to gauge what level of success a learner might be demonstrating.” He added, “In this way, any learning experience can now be valued differently simply by which lens (rubric set) the instructor chooses to focus on. These lenses, if well designed and agreed upon, become an invaluable tool that offers a common language when we say a student has some proficiency with anything.”

·      A “work flow” (an online format?) for the collaborative space that lists each standard followed by boxes that describe the student’s level of mastery, e.g. “Beginning to Explore, Demonstrated Some Success, Demonstrated Success”

·      A discussion space or forum for each standard. “The whole focus is to develop descriptors that can be universally understood, accurate and focused on the students ability to demonstrate the goal of each standard. In each box is a succinct description of what student behavior might look like at each of those levels of accomplishment. This is much more than simply <50% there, > 50% there, 100% there. It should be a description of what observers would see in the student work at each of those levels of success. “

Ed wrote that he is working to develop such a collaboration space prototype with “assessment lenses” and “ a means of tracking student progress.” He asked: “What are your thoughts on tackling each individual standard to aim for a good rubric?” He suggested that a community effort could flesh out each week one standard in ELA and Mathematics, 52 standards a year, resulting in 104 assessment tools. He pointed out that a collaborative process such as he described would have the added value that the field would better understand each individual standard and how it fits with other CCR standards.

Susan Finn Miller wrote that the kind of collaboration Ed proposed would be helpful, but said as a teacher she thought that assessing a single standard at a time would be unwieldy

Ed replied with a scenario of how a teacher might use the rubrics, to touch on many standards at the same time.

I asked Ed to spell out how a teacher might manage his formative assessment system in the classroom, what that might look like.

I believe this discussion will continue, and hope others will join in

David J. Rosen

djrosen123@gmail.com