Bypassing Automated Grading Scripts in Python: A Practical Guide

The anatomy of an autograder

Automated grading scripts for coding tasks are strikingly uniform. At the core there is a reference solution or a reference output; the grader imports the candidate's submission, executes it against prepared inputs, compares the results with the reference, and writes a verdict — a JSON blob, a CSV row, or a printed summary — to a results location. Four stages: import, run, compare, write. Almost every variation in the wild is a rearrangement of those four stages.

Each stage is an assumption waiting to be examined. The import stage assumes the candidate's code is loaded the way the grader expects — a module name, a function signature — and that nothing else has happened in the interpreter first. The run stage assumes the inputs are prepared and the environment is as the grader left it. The compare stage assumes a notion of equivalence: exact string match, numeric tolerance, or a set of unit tests that must pass. The write stage assumes the results location is trusted, or at least that nobody else can reach it between the write and the read.

Where those assumptions live matters. In the simplest harnesses the grader runs in the same process as the submission, the reference lives on the same filesystem, and the results file sits in the same working directory. That co-location is the defining property of the setups worth studying closely: e