Checklists and Rubrics: More Discussion, Resources, & Strategies

Hello, all.

Yesterday, June 9, Jayme Adelson-Goldstein presented a webinar on using checklists and rubrics to differentiate instruction for adult English language learners. The webinar was attended by 88 participants who chatted, indicated their opinions on polls, and, in general engaged in a very interactive webinar. The webinar was hosted  by Dorjan Chaney at Kratos; Susan Finn Miller, subject matter expert (SME) of he Assessment CoP and I  want to invite you now to engage in further discussion with Jayme on using checklists and rubrics with your adult English language learners both as an instructional and assessment tool. We will continue the conversation started yesterday throughout this week.

Please feel free to post any questions or comments for Jayme on this discussion thread. I know we will hear from her soon today.

I have one to start the conversation: I guess I am not 100% sure of the differences between checlists and rubrics, and wonder if Jayme could explain that a bit more.
Thanks.

Miriam

SME, adult ELL CoP

 

 

 

 

Comments

Hello colleagues, A huge thank you to Jayme Adelson-Goldstein for her wonderful webinar yesterday on using rubrics and checklists in a multilevel ESL classroom. Everyone who attended took a way a great deal. Now we get the opportunity to extend and deepen our learning on this topic by engaging in discussion with Jayme within our two communities.

Miriam posed a good question that many folks may have. Here's another issue for Jayme to address: You mentioned briefly that adult learners could be engaged in helping to create checklists and rubrics. What would you say are the benefits of engaging learners? Under what circumstances would you advise involving learners in the process?

Thanks again, Jayme! Looking forward to a lively discussion!

Cheers,

Susan

Moderator, Assessment CoP

Great question, Susan--which we'll address first thing tomorrow! In the meantime, if any of you are anxious to get a head start thinking about the why and how of engaging learners in rubric and checklist creation, take a look at these resources:

Litz, D. (2007, November). Student-directed Assessment in ESL/EFL: Designing Scoring Rubrics with Students. Internet TESL Journal. Retrieved April 30, 2014 from http://iteslj.org/Lessons/Litz-StudentDirectedAssessment.html 

Henning, M. (nd) "Involving Students in Creating Rubrics" in Rubrics to the Rescue. Retrieved June 9, 2014 from http://www.teachersfirst.com/lessons/rubrics/involving-students.cfm

Teacher Vision Student-Generated Rubrics: Part Five in a Five-Part Series

(Note that while the second and third links are on K-12 sites, the concepts expressed on the subject have strong androgogical underpinnings and are relevant to our discussion.)

Onward!

Jayme

Wow!  I love that new word.  (I had to Goolge it.)  You'd think, after being in adult education for so long, that I would have known that word.  Well, I do now.  :)

Jayme, those websites are great resources. Already I feel like I am becoming a rubric junkie--learning so much and eager to try them out in classes with students and in PDs with colleagues. I look forward to our discussion of how to involve ESL students, many of whom have limited language, in the process of using that limited language to create rubrics.  I know we will discuss visuals and simple evaluative criteria.

I believe as educators we are challenged by the task of creating rubrics.  They force us to have very specific and differentiated objectives, often in a number of areas.  I think being exposed to many examples is key to becoming more comfortable with the process, and we must be comfortable and confident with them prior to asking our students to create them.  I trust, however, that if we and our students are interested and even excited about a learning task or project, we will be more able, eager, and quicker to come up with the rubric.  And practice and experience will help, too, of course.

Dave

 

 

Hi Dave!

I think you make a critical point re: instructors feeling confident about rubrics prior to getting learners involved in their creation. But I'd like to tweak that thought a bit  -- what if we were to say that instructors needed to feel confident about the purpose and value of rubrics, but could then go  on the journey with their learners to construct a basic analytic rubric on a single task with which they and the learners are very familiar. Say leaving a voicemail message,  or making a purchase at the food truck,  or completing a registration form.  The whole class or learners working in small groups could break down the important skills or elements within each task and then decide what would constitute the most successful way to demonstrate that skill. (This could also be a good way to contextualize practice with precise academic language as well as comparatives and superlatives. ) Here's an example.

When we want to start engaging learners in creating the assessment for a class task or project, it really means that we need to involve them in the creation of that task or project. This fits right in with adult learning principles that Knowles initially aid out in the 1970's and which are still referenced today. Self-directed, highly relevant, solution-oriented, and practical learning tasks are motivating for the adult learner. If the task or project is motivating, as you point out, creating the assessment will make it even more so.  Let's say that an instructor and his learners have agreed that there's a need to showcase the successes of adult ESL learners (in order to advocate for increased support of AE in the community). Having first brainstormed and selected the form this showcase will take (e.g. a print or online newsletter, YouTube/Vimeo/ video, blog posts on local news and community blogs, a PSA or radio spot on a public TV or radio station, a school fair, debate, etc.). The next step is for learners to work with the instructor to brainstorm the stages of the project and from there, the elements within each stage. (This might be a good time to start a checklist :-) ) With a complex project (newsletter, school fair) there could be a rubric for one or two aspects of the project ( a writing rubric for newsletter articles, a design rubric for the layout of the newsletter) and checklists could be used for the other elements. For a less-involved project: (writing blog posts) a single writing rubric would probably suffice. In each case though, you and the learners could determine the dimensions that should be evaluated.

Supplying a framework for the rubric and completing one row, is a good way to get things started.

What do you think?  j/me

Thanks Dave and Jayme, I agree that involving learners in the process of rubric creation can be very powerful. In my  experience as a student, it was very often unclear what the expectations were for assignments. I was often guessing until I received feedback from the teacher or professor on my work. By using a rubric, we have the opportunity to make expectations much clearer. I think Loraine Valdes-Pierce calls this "visible criteria" for evaluation. When students are involved in the creation of a rubric, they have a deeper understanding of what matters. Of course, this depends on the level of English learners have as well as various other factors, including how much time is available for the process.

Here's a rubric focused on job interviewing developed by a group of learners. What do you think this rubric illustrates about what the students have learned about job interviewing? About evaluation criteria, in general?  What feedback might you offer to this group of learners about their rubric?

Job Interview Rubric The interviewee ... 3 2 1 Eye Contact Maintains eye contact 100% of the time Occasionally maintains eye contact Never maintains eye contact Professionalism Maintains professionalism 100% of the time Is occasionally professional Is rarely professional Posture Maintains good posture all the time Occasionally maintains good posture Rarely maintains good posture Confidence Is confident 100% of the time Is occasionally confident Is not confident Speaking Answers all the questions clearly Occasionally answers questions clearly Does not answer questions clearly Attitude Generally maintains a good attitude Occasionally maintains a good attitude Does not maintain a good attitude

 

Cheers, Susan

Moderator, Assessment Community of Practice

Hi everyone! 

Thanks so much to all of you who sat in front of your computers on a lovely Monday afternoon in order to "chat" about the different and meaningful roles rubrics and checklists play in planning, differentiating, and assessing instruction in multilevel adult ESL classes. On this first day of our discussion, and in anticipation of some of the questions that might come in from those who did (and didn’t) attend the Webinar, I thought it might be helpful to recap the differences between rubrics and checklists as well as the rationale for using these valuable tools. Thanks for your question, MIriam!  Read on to find out more ;-)

First, a definition: “At its most basic, a rubric is a scoring tool that lays out the specific expectations for an assignment.” from Introduction to Rubrics. 2nd ed.  (Stevens and Levi, 2013) In Levi and Stevens book they call out these four parts of the rubric: (1) the assignment (task, project); (2) the scale (levels of achievement which can be labels, points or grades; (3)  the dimensions of the assignment (the key elements learners will focus on: e.g., accuracy, a “soft skill”; inclusion of specific content?, etc.) ; and (4) the descriptors for each level of performance on the assignment.

The analytic rubric separates out the dimensions and descriptors so that the instructor can give feedback on different aspects of the performance more easily.  The holistic rubric includes the scale, but provides a description of each level on the scale without breaking out the dimensions of the performance within that level.  For examples of holistic and analytic rubrics see the Webinar handout or check out these examples at http://teaching.berkeley.edu/rubrics-0

A rubric can play a very supportive role in helping us plan differentiated instruction. We can use the process of developing the rubric to help identify the different outcomes we expect for the beginners, intermediate and advanced learners in our class.  Think about the basic task of writing an email to request a day off. What would a level-appropriate email look like at those three general levels (B,I, A)? We can think about quantitative differences between levels of task performance,  (e.g. a one-line email, an email of three-five lines, an email of more than five lines) and qualitative aspects (a basic request, a polite request, a polite request and a concise rationale) . And we can also consider which aspects of effective writing we want to address, such as the format, organization, mechanics, and/or grammatical accuracy.

Identifying outcomes or lesson objectives is an essential part of backward lesson design, as is developing the assessments that will be used to determine learners’ success achieving those objectives.  Developing a rubric covers attends to both of these steps which lends a certain elegance to the process. As Stevens and Levi point out though, instructors new to rubrics may find that constructing their first rubrics takes more time than it saves. Happily, with practice (and the development of some generic rubrics that need only slight modifications for different assignments) rubric construction becomes easier.

While rubrics may feel a bit like rocket science, many of us have been using checklists in the classroom for years. They are used in all types of settings. (I bet you can name three without blinking).  And many of us have a definite fondness for the form. Kathleen Rowlands, in her 2007 English Journal article “Check It Out! Using Checklists to Support Student Learning,” points out the differences between checklist and rubrics

“Both provide ways of making task requirements explicit […]however rubrics are scaled and descriptive, unpacking levels of performance for the purpose of assigning grades or scores; rubrics align descriptive performance criteria with grades or scores. Checklists, on the other hand, […]make no attempt to identify descriptive criteria. They list the steps students should take as they are learning a process or highlight the features required for a complete assignment.”

For the email  assignment above, learners could work with a checklist to help them plan and organize their writing (being sure they include key information); and/or use a checklist as a guide for peer feedback and revision.  As a scaffolding strategy, the checklist supports learners throughout the task. It's also an effective differentiation tool. For example, the email task checklist for higher level learners can use more sophisticated language and have one or two additional steps. Beginning-level checklists may use visuals or examples to support learners’ understanding of the steps.

And there you have a recap of the differences between rubrics and checklists. And now a quick review of the rationale for making use of these tools in adult ESL multilevel instruction (or ABE, ASE or GED instruction, for that matter.)

Both checklists and rubrics….

  • allow for smoother management of the ML environment because they serve as guides for autonomous, self-directed group tasks.
  • use language and formats that can be found outside the classroom as well.
  • can serve as non-narrative reading lesson for beginning literacy learners while the more complex versions provide a meaningful challenge for advanced learners;
  • break down the complexity of a task by focusing on its dimensions (rubrics) and steps (checklists).
  • can call out the important soft skills and collaborative language that groups need to employ to collaborate successfully.
  • put a spot light on soft skills, team-building language, and participation which lends gravitas to the group work in a class. 
  • have formative assessment underpinnings—the task informs the rubric, informs the checklist which informs learners’ performance which informs the task and on and on.

I promise--that's the longest post I'll write for the week! Onward!

Jayme

 

Feel free to respond to any of these prompts or take the discussion in a direction that is meaningful for you.

  • What comments or questions do you have about the use of rubrics to help plan differentiated instruction?
  • Tell us about one lesson where you made use of a rubric: Was it analytic or holistic? What kind of scale did you use? Did learners see the rubric ahead of time? Why or why not?  How did you use the rubric to give feedback to learners?  How did you adjust instruction based on the rubric? What were the learners’ reactions?  
  • If you have used a checklist that was particularly successful, what were the elements that made it so? Were you able to re-use the checklist with more than one assignment? more than one class? Why or why not?
  • How would you decide when a task or activity merits a rubric? A checklist?  If you’d like to read more on this question, take a look at this page on the Learner-Centered Initiatives’ Wiki*: http://qualityrubrics.pbworks.com/w/page/11996588/Checklist_v_Rubric

If you have a rubric or checklist to share, you can post it on the Google Doc I started at http://tinyurl.com/pho6qlr  Just scroll down to a blank page and paste it in. (Then let us know it’s there!) It's not possible to post a downloadable doc to this community, but we can access each other's doc through the Google Drive link)

Correction:  The Wiki above was mistakenly attributed to Kathy Schrock during the Webinar and in the downloaded handout. It was part of her Assessment and Rubrics digest, but not her own work. Instead it should be attributed to Learner-Centered Initiatives out of Garden City, NY.

 

 

I like that both rubrics and checklists are beneficial when differentiating instruction.  I beleive rubrics provide focus, emphasis, and attention to particular details and allows for lesson to be evaluated according to the student’s ability. Checklists provide a way to keep students organized and on task and can be altered to fit students' ability levels by selection of wording and even adding pictures.

I have successfully used a rubric when assessing writing. I was able to reuse the rubric often. I think one great point made in the webinar was to use highlighters. They are useful .

Thanks so much for your contribution to the discussion, Melanie. I'm glad you liked the use of highlighters. I use them in my online classes and just highlight the word doc. It's a great time saver AND if it happens that one aspect of a description was not met...it's easy to leave the highlight off of that section and make a comment at the bottom of the rubric.

Would you be willing to post your writing rubric on the Google Doc? (Just scroll down to the first blank page and paste it in.)  Or if this is a rubric that you found online, I know we would love to be able to reference it ;-)

j/me
 

I really enjoyed the webinar and the continuing discussion.  I have always been a fan of rubrics, and I have used checklists extensively.  What I didn't put together until now was using them in combination.  Somehow I always tried to decide if a checklist worked better for a task OR a rubric.  However, it makes such sense to think of them in combination.  Both the checklist and the rubric help a teacher clearly define the intended results.  The checklist helps the learner prepare, and the rubric provides more specific details about how the project, speaking, reading, writing assignment will be evaluated.  Very helpful!

Over the past couple of days, I've fielded a few questions from colleagues (off the list) about the development of rubrics. The one question that keeps coming up is how to determine what criteria to include.  Here's a short list of guiding questions I found on Brown University's Sheridan Center for Teaching and Learning site:

As DeAnna has pointed out in our discussion, there is no need to reinvent the wheel in order to use rubrics.  We can go to template resources like RubiStar or iRubric and we can also pull models from all over the web to use as the basis for developing our own criteria.  

For a sample of writing criteria- you could look at the CASAS formative writing assessment rubric or REEP's or this adaptation from World Ed.: http://firstliteracy.org/uploads/File/NEW%20WRITING%20RUBRIC%20%28Oct.%202005%29.pdf  (Note that in 2005, (almost 10 years ago!!)  LINCS hosted a great discussion on the REEP writing assessment and the inclusion of voice as a dimension.  You can find the archive here: https://lincs.ed.gov/lincs/discussions/assessment/Voice-in-Writing.html

Here's a sample of an ESL speaking interview rubric developed on iRubric: http://www.rcampus.com/rubricshowc.cfm?sp=yes&code=KXX6W9&

And here's a cooperative group work sample rubric--designed for younger learners. What would you adjust (and how) to make this appropriate for your learners? http://www.readwritethink.org/files/resources/lesson_images/lesson95/coop_rubric.pdf

And, since we were talking about learner engagement yesterday--how about a rubric on teacher presentation/lecture skills? Can you imagine small groups of learners coming up with the criteria for an ESL teacher's presentation of new information? .http://www.ucdenver.edu/faculty_staff/faculty/center-for-faculty-development/Documents/Tutorials/Rubrics/documents/ex_faculty_lecture_evaluation.pdf  (This would take the "Design a job ad for the perfect teacher" task to a whole other level, don't you think?)

If you're new to rubrics, choose a task type or skill area and then search "Rubric for [...]"  to see what you find. Post a link you liked to this discussion, and your colleagues will be extremely grateful.

If you've been using rubrics for a while now, share a link to a sample rubric that has helped you develop the criteria for your own rubrics.  

Onward!

Jayme

 

 

 

 

Thank you, Jayme, for the wealth of information you are sharing with us, and thanks to the others joining in the discussion as well.

It is worth noting, I think, that the BEST Plus, the computer adaptive test of oral proficiency that is on the NRS approved list along with CASAS, also uses a rubric to assess the speakers listening comprehension, language complexity, and communication. Those three components are explored at http://www.cal.org/adultspeak/assessment/index.html

It is interesting that rubrics are currently being used in both formative and summative tests.

Miriam

 

 

 

 

 

So glad that you brought up BEST Plus Miriam!  And as to your point about the growing use of rubrics in summative and formative assessment in AE- I say "hooray!"  Performance-Based assessments (PBATS) are taking hold!

jme

Hello Miriam,

I wanted to provide the group with links to all the TABE and TASC related rubrics.  TABE CLAS-E is our assessment for Adult English Language Learners with NRS approval through 2017 in the content areas of Reading, Writing, Listening and Speaking. The TASC test is the newest High School Equivalency assessment available to states.

TABE Advance Level Writing Test Rubric

https://drive.google.com/file/d/0B-6nXdfwTqDzTjZsbndGcF9NNFE/edit?usp=sharing

TABE CLAS-E Speaking Test Rubric

https://drive.google.com/file/d/0B-6nXdfwTqDzbVZRcnZidHhRdVE/edit?usp=sharing

TABE CLAS-E Writing Test Rubric

https://drive.google.com/file/d/0B-6nXdfwTqDzY3dmQzhXOXVyNU0/edit?usp=sharing

TASC Writing Test Rubric

http://www.ctbassessments.com/pdfs/TASC_ScoringRubrics.pdf   

 

Regards,

Mike Johnson | National Adult Education Manager, CTB

McGraw-Hill Education | 1333 Burr Ridge Pkwy. Burr Ridge, IL 60527

P: 630-789-4586 | C: 630-995-6712  | mike.johnson@ctb.com

Miriam, to respond to the differences from a practical standpoint, in the classroom, I always find it easier to create a rubric first to help me identify exactly what it is I want students to produce and what I'm expecting from them; from that, I can provide them with a checklist of elements to include, which I think is more comprehensible when students review their assignments for completion before submitting.  This isn't super theoretical, but helps me differentiate between the two in my mind.

On another note, I don't find reinventing the wheel to be a productive use of my time, and that's why I love Rubistar.  It defines in the categories for me for different types of assignments, but lets me delete or edit the ones I don't want to include.  It also often lights a spark for me to consider elements I hadn't thought of, and of course, saves tons of time!  Since it's free, I can save my rubrics for future courses, go back and edit them, and create new ones if it's course I haven't taught before.  A true gem!
 

Hi DeAnna,

I think the way you visualize the differences will be very helpful for others. Thanks so much.  I agree that the development of the rubric is a very important planning tool for the instructor. I do like having learners use rubrics when they're working  in a group because it allows them to reach consensus on which aspects of a task they want to prioritize, but that's not something I do nearly as often as giving the group a checklist.

I, too, am a Rubistar user! Do you have any links to your rubrics that you'd be willing to share? You can put the link in a post, or use the Google Doc to paste the rubric directly on a page.

j/me

We've spent the last three days talking about the creation of rubrics and checklists--how they are different, how we can engage learners' in their creation, and how we can select appropriate criteria. I thought it would be useful to end the week with a reminder that rubrics can be used by learners as formative assessment tools at the end of a lesson.  We can ask learners to rate themselves in terms of their focus, their collaboration, their grasp of content, and their use of target language. Here's an example I knocked out quickly (probably too quickly.) If you have time, why not add your edits and comments right on the page?

An important consideration, when thinking about formative assessment, is that it's usually  a low-stakes, ‘grade-less’ method of helping learners and instructors monitor learning.  This would mean that assigning points to this type of rubric could be less effective than using labels such as Expert, Competent, Developing. for the scale.  Ideally, learners would want to look at the rubric dimension by dimension rather than looking at a total score or label.  By identifying learners’ strengths and weakness within the different dimensions of a task or assignment, rubrics inform instruction, helping the instructor address learners’ comprehension and production issues more effectively.

I hope this discussion continues on, because the topic certainly merits a deeper dive!

Why not put "lurk and comment on  LINCS Rubrics and Checklist discussion" on your summer "to-do" list.?

Onward!

Jayme

Thank you so much, Jayme, for the great, interactive webinar and the week of discussion with us on using rubrics and checklists. I just clicked on your example above and found my computer doesn't support all the tech on it, sadly.  I got this message: "You are using a version of Internet Explorer which is unsupported. Some features may not work correctly."  It seems I need to use google chrome as my default, but I can't do that without administrator approval.  Still, I was able to write a comment in the rubric itself, and I put the same comment  below Jayme's comment on how to make comments. Sorry to be confusing. I want to say I encourage you all to look at the rubrics she's added.  Add a rubric of our own, as she suggests, or comments as I did, in my low tech way, or the higher tech way if your computer allows that.

Thanks, again, Jayme.

Miriam

SME, Adult ELL CoP

 


 

Hi Miriam--

I put your comment in the comments section and edited the rubric accordingly. Thanks so much for starting the editing process!

And just FYI - we can also access Google Drive on our phones. I use my 2.0 reading glasses for that experience to be successful. ;-)  Here's a link to the Google Drive App info:  https://tools.google.com/dlpage/drive/?hl=en

Onward!  j/me

Hi all, I love Jayme's comment about students using rubrics to assess their own progress. With beginners, I want to be sure they are able to spell their names so that others can understand, i.e., they need to be able to pronounce the letters and spell fast enough for others to understand. Here's a rubric I developed which helps beginning level English learners to work on this skill and assess their own progress.

What do you think? Has anyone else used a tool similar to this? How has it worked for you?

Spelling My Name Yes, I can

No,

I need more practice

I can say the letters clearly.

People understand me.

    I can spell my name fast.    

 

Cheers, Susan

Moderator, Assessment Community of Practice

Jayme, I have been out of town and missed your webinar.   However, I was in your TESOL Certification online class and you are exceptional.   I am so sorry that I missed this one and will certainly be on the lookout for anything that you are doing.   You are a model for the rest of us and a leader in the field.

<p>I've read your comments and feel like I've learned more about Rubrics than I have in all my 34 1/2 years of teaching math!! &nbsp;And the idea of teaching with checklists is fantastic!! (I know: sentence fragment, but I had to express my enthusiasm.)</p>