Of programming projects, specifications grading, students dropping my course, and Charles Dickens

With apologies to Charles Dickens… It was the worst of days. It was the best of days.

My day started by learning that a student had dropped my CS 120 course. Yes, I’m usually a bit saddened when this happens, but I understand there are lots of reasons students drop a course for which they’ve enrolled, and that most of those reasons have nothing or little to do with me. But this time it was different; the initial reason she gave to me for her dropping the course was how I was grading her work. I was particularly saddened and discouraged that I appeared to be the major reason for her dropping the course. Further, since there are few females who enroll in computer science courses, I really like to keep the few who do enroll. To think I might have chased one away was unacceptable to me.

I am utilizing specifications grading for the first time in our CS 120 course this semester. (Read my previous discussion of specifications grading in a different course.) This course serves as our first introduction to programming for our students. I am evaluating most assessable items in the course as complete/incomplete—either they fully met the detailed specifications I provided to the students, or they did not. I am evaluating the outside-of-class, work-by-yourself projects based on an A-F scale, where students must complete certain specifications to earn a given grade. The better the grade, the more items that have to be successfully completed. However, the highest incomplete specification determines your grade.

In the instance of this student, she completed nearly all specifications correctly, and would have received an A, except for two of the lower-level specifications which were required for a grade of C. As a result, I recorded a grade of D for her, as I described would happen in the project specifications document. The items she did not complete should have been relatively easy to complete; that’s why they were at the C grade level. She took exception to this, saying her program generated the correct answer, so therefore she deserved an A, and felt it was totally unfair to receive a D. Further, she stated that she was thinking about dropping the course. I do have some empathy for her stance; it does seem a bit unfair. On the other hand, had she paid a bit more attention to details, she could have easily earned an A. Further, I’ve provided for this sort of situation by providing every student four Oops Bits at the start of the semester. These Oops Bit can be used to resubmit a variety of assessable items which are not completed successfully on the first submission, or can be used to gain an extra day to complete an assignment.

In an attempt to try to help her understand my rationale for establishing the grading for the project in the way I did, I asked her to consider this analogy.

What if you were an excellent employee, and there were few people that could do your job better, but you had a hard time showing up to work on time, maybe even missing some days altogether. Most employers would not be willing to keep you as an employee—even though your work was excellent when you showed up—because they couldn’t depend on you showing up, a fundamental quality of a good employee. They would be more likely to retain the employee they could depend on, even if their work quality was a bit lower.

Up to this point, our conversation about her grade was all via e-mail. I encouraged her to meet with me in person to discuss the situation further, and she agreed. We met yesterday and had a relatively pleasant, healthy conversation for nearly an hour about specifications grading, and how I had structured the specifications for this particular project. One thing I asked her to do was to arrange the items in the project specifications into what she considered to be a better set of requirements for each grade. My intent in asking her to do so was twofold: I was truly curious how a student would arrange the list, and I wanted her to critically think about what qualities of the project justified an A versus a B, etc. She obliged and came up with the following list. The first two columns in this table indicate what is required for a given grade. All items below a particular grade are required as well. If any of the items marked as a grade of D are missing, a grade of F would be recorded. The first column is how I organized the list, and the second column is her ordering. (I’ve arranged the list based on my specifications for the project.)

My grade category	Student grade category	Assessable item description
A	A	Program accurately creates all 31 possible versions of the shifted alphabet
A	A	Program accurately lists all 31 possible translations, based on character shifts of 0 through 30 positions to the right
A	A	At the end of your reflection paper, tell me to what this encoded message translates.
B	B	Program lists all 31 possible translations, based on character shifts of 0 through 30 positions to the right (correct or not)
B	C	Program uses descriptive function and variable names
B	D	Program capitalizes the input parameter value
C	C	Program is formatted/indented similar to the text book authors’ examples
C	D	ALL files for the project are placed into a folder named project0. Compress this folder. Submit this compressed folder.
D	B	Reflection paper completed and submitted according to specifications provided in Canvas
D	C	Program will run in JES
D	C	Program uses the character set provided
D	D	Program will load in JES
C		Program accepts one input parameter

I’ve highlighted the items where there was a difference in our ordering. The items highlighted in yellow are one grade different, sometimes higher, and sometimes lower. The darker highlighting are items with a difference of two grades, again in both directions. I find it interesting that most of the ordering is relatively similar. I also find it interesting to see what this student valued compared to other items.

During our conversation, she shared some about other courses she is taking this semester, and how none of them are required for her major, including CS 120. She is transferring to another university next semester, and is simply taking core courses and electives which will transfer. By the end of our conversation, I felt I’d made some progress, and asked her to give the course three more weeks before making a decision about dropping the course. This would provide her more time to experience the course, and sit for the first exam. By then, she’d have a good idea how the course was going for her. As we parted ways, I believed I’d see her in class today. Unfortunately, she did not attend, and I received word after class that she had dropped. 

Initially, I was rather discouraged to have a student drop my course because of how I am conducting the course, and specifically because of how I am grading. Upon reflection, and remembering what she shared with me during our conversation yesterday, I have come to realize her reason for dropping the course was much more than how I graded her project, and perhaps that was not even the major reason. She was taking an eighteen-credit hour load, had multiple demanding courses which are requiring significant amounts of time, and the courses are not directly required for her major. Dropping CS 120 likely bought her a lot of time to devote to other courses, and is not likely to have a significant impact on her college career. I likely would have made the same decision myself.

So, by now, I’m feeling a bit better than I was earlier in the day.

And then the student for my 5:15 PM appointment arrived.

This student is one who had met with me a few days ago because he felt he was struggling to understand the course material. When I sense a student is capable, but maybe just needs a bit of a push and assurance that they are capable—a shot of self efficacy—I will offer to set up a standing appointment to meet at the same time every week when they can come in and ask what ever questions they might have. Students are always welcome to meet with me anytime our schedules permit, but for some students, having a pre-scheduled time seems to make a significant difference in their progress. He was one of those students, and was arriving for his first appointment.

I expected that he might have a question about one of the recent activities we’d worked on in class. This was not the case. Rather, he came very prepared with a mental list of items and concepts for which he had questions, or felt like he needed clarification. We very productively moved from one topic to another, and I could tell, based on the type of follow-up questions he was asking, that he was gaining a much better understanding of the material. We worked through course material for nearly an hour, that very quickly passed—at least for me. The more I worked with him, the more he impressed me with how truly committed he is to learn and understand the course material. He was a total joy to work with.

So, was it the worst of days? Or was it the best of days?

I’m going to go with the best.

What we learned in CS 222, Largent's Fall 2017 edition

My colleague, Paul Gestwicki designed our CS 222: Advanced Programming course. He first taught it during the fall 2010 semester. Most semesters since then, he has blogged about an exercise performed as a final exam to identify what the students felt they learned during the course. You may review his fall 2017 blog entry, if you wish.  I've had the pleasure of teaching the course since fall 2015. Having just started to blog this fall, this is my first opportunity to create a blog post similar to Paul's. So here goes...

The final exam process for CS 222 consists of three parts. First, the students are asked to list everything they can think of that they learned--directly or indirectly--because of their participation in CS 222 this semester. Once the consolidated list is compiled (and cleaned of duplicates), each student has a limited number of votes to cast to identify the items they feel are the most significant. And lastly, each student then selects one of the items which received the most votes and writes a reflective essay about how they learned that item.

The consolidated list contained 103 items (after removing duplicates). Each student had seven votes to cast for the items they felt were the most significant for them. Because of a tie for seventh place, we included eight items in our "Top N List," which ended up including these items:

• Clean code techniques (15)
• TDD (15)
• Stack overflow (14)
• It’s easier to take the time to write cleanly than to write dirty code and fix it later. (8)
• Code is like art. Anyone can look at code and call it beautiful, but it’s rare to find someone who can tell you why. (7)
• Time-management is a key factor in a lot of things (7)
• Consider what the needs of the end user are when developing functions, rather than developing functions based on satisfying your own goals for your application. (5)
• Navigation of GitHub (5)

The students in Paul's fall 2017 section of the course developed this list:

• TDD (13)
• Good names (12)
• Refactor vs. Redesign (7)
• Self-reflection (7)
• git (6)
• Objects vs. Data Structures (6)
• Time management (6)

Comparing the two lists, I see many similarities. Both lists include references to clean code, TDD, Git, and time management. As Paul mentions in his blog post, clean code and TDD are major themes of the course, both of which are completely new concepts to the students. As such, they struggle with them. Similarly the use of Git and GitHub is new to virtually all students, and causes its own challenges, until they take the time to figure it out. Time management is a common issue students (and many of us!) have; seldom are they able to accurately estimate how much time it will take to develop their projects. Because of the nature of the course (with some limited scaffolding we "throw them in the deep end of the pool and ask them to swim"), it is not surprising that students would feel the use of Stack Overflow is significant.

I found the "Code is like art." item a bit interesting. One of the course's essential questions is "What is software craftsmanship?" Perhaps the discussions we held about that throughout the semester struck a chord with some of the students more than I realized. User needs versus your own project goals was an item I've not had show up before, but it may reflect a few of the groups' experiences when they performed user acceptance testing.

There were a few items that I found interesting but didn't receive enough votes to make the "Top N List."

• How much I’ve learned since CS 120
• Reflecting on yourself can prove to be useful
• Must make an effort to learn from past mistakes the first time I make them
• Asking for help and needing help are two different animals

These items all focus on learning, and our human tendencies; what has been, is being, and will be learned, and our inclination to avoid doing what we need to do to learn. The fact that a few students observed this in themselves, and a few even felt it was significant (as evidenced by a few votes) is a good thing.

Comparing the course grades of this semester's students to those of the spring semester, the grades appeared to drop, though not what I'd consider significantly. There were fewer As, more Bs and Cs, and fewer Ds.

	Spring 2017	Fall 2017
A	52%	42%
B	41%	48%
C	3.5%	10%
D	3.5%	0%
F	0%	0%

I did change how grades were calculated this semester, having switched to utilizing specifications grading, but the assessable items did not change--only how they were evaluated. I reflected on my experiences with this transition in my previous blog entry. Some of the over all grade change might be attributed to this, and may be reflecting a slightly more accurate representation of learning and effort than the previous point system, which tended to "average" my assessment of their learning. The students did show evidence of learning and growth in their final exam reflective essays.

According to specs…

During the fall 2016 semester, I participated in a multi-session faculty learning community that explored what specifications grading is, how it might be implemented, and what it might do for our students and us. In her book Specifications grading: Restoring rigor, motivating students, and saving faculty time, Linda Nilson claims that specifications grading can make grading faster, clearer, and more precise, allowing the instructor to focus more on promoting improvement in students, rather than worrying about justifying a grade.

The basic tenets of specifications grading are to provide a very clear, detailed description of what is expected of the student for a given assessment (the specification), and then evaluate their submitted work as complete or incomplete against that specification—they either met the specification, or they did not. Simply providing clear expectations can make a significant difference in a student’s ability to submit what you are wanting. It almost seems too good to be true.

Historically, I have assigned points to everything students do for a course, and then determined their course grade based on the total of their earned points compared to what was possible to earn. This means that a student can do poorly on some assignments (or not do it at all), and great on other assignments, and then depend on the “extra” points from the great assignment to offset the lack of points on the poor assignment. This effectively generates an “average grade”, but does not necessarily reflect what the student accomplished. Students were also inclined to beg for partial credit.

I decided if even one of the claims Nilson made in her book title was true, it would be worthwhile implementing specs grading. My first opportunity was for a discussion-based course (CS 200: Computers and Society) during the spring 2017 semester. I converted my evaluation of all assessments to complete/incomplete—they met the detailed specifications I provided, or they did not. One exception I made to grading complete/incomplete was for the exams, although, even here, I still altered my approach. Previously I often awarded partial points for a response that contained some of the answer for which I was looking, but not all of it. For this course, I evaluated each question on a complete/incomplete basis—full points or zero. Since most of the assessments were an all or nothing situation, I did offer the students limited opportunities to resubmit incomplete assignments. In the absence of points to add up, I determined the final course grade based on how many of each assessment category the student completed. For example, if a student successfully completed at least five of six quizzes, that gave them an A, provided they had an A in all other assessment categories. If they only passed four, they dropped to a B, etcetera.

I found it critical to clearly (and repeatedly) explain to the students why I was grading this way, and what the benefits were for them and their learning. I made minor adjustments, based on my experiences and student feedback, and implemented specs grading for the same course during the summer 2017 semester as well. I have just completed the fall 2017 semester during which I implemented specs grading on two more courses. One of these courses (CS 239: Social and Professional Issues) I implemented the same way I did the CS 200 course during the spring 2017 semester. For the second course (CS 222: Advanced Programming), I used specs grading on all assessments, excepting the programming projects. For the projects, I continued to assign points based on how close to a correct solution they were. The projects were a category of assessment items, and had a different minimum points the student had to accumulate for each letter grade.

Reflecting on my experiences teaching courses utilizing specifications grading, I observed a variety of positive outcomes, including the following.

• Students did not beg for missed points. Although there were a few (very few!) times when a student asked if I would consider passing a given assignment, rather than the failing evaluation I had provided.
• As a whole, final course grades stayed substantially the same in the specifications-graded courses, compared to before I implemented specifications grading. I believe however, that the grades more accurately reflect the level of course content mastery. That is, if a student received a course grade of “B”, they had mastered most all of the course material at a “B” level or better, as opposed to some material at an “A” level, and other material at a “C” level.

Not all was perfect, however. My experiences and student feedback did highlight some areas that needed adjustment, including the following.

• I needed to provide a better explanation and justification for using specifications grading at the start of the semester.
• I did not originally provide enough detail in a few of the specifications. Others contained minor ambiguities.
• Specification grading has not significantly reduced my grading time. In large part, this is because I want to provide the student with as much feedback as possible as soon as possible, rather than stopping my evaluation as soon as I found a specification they did not meet.
• Some students became discouraged and gave up when they failed too many assignments.

Thus far, students tend to either like, or dislike the use of specifications grading; there is not much middle ground. Some student quotes from the end-of-semester course evaluation may serve to illustrate this. Students in the CS 222 course offered these comments about specification grading (I list the positives, and then the negatives):

“The grading system was very clear. Everyone knew what needed to be done for a specific grade.”

“Unique grading system, which might seem daunting at first, but is overall a worthwhile system.”

“Grading system is a complete mess, I know it's newish, but it really doesn't have a place in an advanced programming class.”

“Specification grading encouraged me to not complete more achievements because I knew that I could not get a grader higher than a C because of my project scores. In that regard, I think standard weighted grading systems are better as every assignment counts. In this system, students will always have incentives to work hard on every remaining assignment because they all impact the student's grade.”

The student comments in the CS 239 course were all on the negative end of the scale.

“The fact that you could have one hundred percent on everything throughout the semester, but then get a B on the final and it would bring your final grade down to a B was kind of ridiculous, especially for a one credit hour class”

“This course has a horrible grading system. Pass/Fail for sections would have been alright, if they were then averaged.”

“The grading system needs revision.”

“I hate this grading scale. People are naturally going to be better in certain areas and it seems unfair for their skill in those areas not matter if they don't do well in another area. Because of the specification grading I did only the bare minimum on all of the assignments because I knew there was less weight placed on my actual skill on that assignment.”

This course has a clear grading system.

Interestingly, if we look at the student responses to Likert scale questions that are part of the course evaluation, we get a much more positive response for both courses. The response rate for CS 222 was 20 of 39 students or 64.5%, and for CS 239 was 22 of 30 or 73.3%. The questions related to specifications grading were the following:

The use of specification grading in this course encouraged me to complete more of my work well, rather than depending on a good grade on one item to cover lesser performance on another item.

The use of specification grading in this course encouraged me to attend class regularly, since I knew it would have a direct impact on my final course grade.

I felt like I could determine my final course grade (or my path towards it) more accurately at any time during the semester because of the use of specification grading.

Once I got used to it, I wish more instructors used specification grading for their courses.

All five questions had a combined strongly agree and agree that was beyond 65%, with many of them being 80% or better. Apparently those that disliked specifications grading must have been more inclined to provide comments.