Canvas Integration: Showing only portion of test cases if incorrect solution.

Jason Isaacs asked 2 years ago

I am interested in using this system for an introductory course in python. I will be running the class through canvas.

Would it be possible to add a feature in the autograding that would allow the student to see a portion of the unit tests and keep a portion hidden for grading purposes?

I would like the students to be able to have a something to test against, but keep a portion of the test cases unseen for grading.

4 Answers

Ryan Staff answered 2 years ago

You’re thinking something like only show the results of the first X of however many test cases? Something like what CodingBat does with their “other tests.”

I really like that idea. It could solve the common problem where a student gets most of the answer right and just hard codes in a catch for the last couple of tests they’re missing.

Let me look into it and see if there’s a way to get that working. I’m pretty sure there is, but I don’t want to promise anything just in case.

Jason Isaacs answered 2 years ago

Hi Ryan,

Thank you for the prompt reply. That is correct. My experience is just as you described that some students will simply hard code for the last couple tests.

More detailed example of what I would like to see:

There are two public unit tests and two private unit tests. The student’s code passes the two public tests but fails one of the two hidden tests.

When the student clicks submit they would see:
Your code passes public test 1 and public test 2.
Your code passes hidden test 1 but fails hidden test 2.
Would you like to try again or complete the submission?
If they click complete the submission they would get a grade of say 75%, but they could have the chance to resubmit some number times (possibly infinite) in order to get full credit.

Matthew Pitner answered 1 year ago

I second this. I just started really using ClassCube this last month. Already one student hardcoded solutions to match the test cases. I manually assigned a grade and commented on the submission in Canvas to have the student try again without hard coding. They did, and for me, this is not widespread yet, but it would be nice.

Ryan Staff answered 1 year ago

We’re in the middle or a pretty significant re-write and this is definately something that we’re adding.

When you create an input problem you’ll have the ability to put in multiple test cases, each with their own input and expected output. You’ll also be able to pick which tests show the full expected output compared to the actual output or just show that it doesn’t match.

The goal is to have it ready by August in time for next school year.

Your Answer