Monday, October 25, 2010

Measuring the Effectiveness of Error Messages Designed for Novice Programmers

My latest paper was accepted at SIGSCE. I will be speaking in Dallas in March.

If you have any interest on teaching programming to beginners, I invite you to read the pdf. It's an exploration of one factor that is largely ignored when people choose a language to teach with: how well do the error messages support learning? Of course, the answer is "not very well." But the surprise is that, even in a project that has spent considerable amount of time thinking and refining its error messages, we found considerable opportunity for improvement, simply by looking at students' behavior closely.

Here's the abstract, and a link to the paper.

Good error messages are critical for novice programmers. Many projects attempt to rewrite expert-level error messages in terms suitable for novices. DrScheme's language levels provide a powerful alternative through which error messages are customized to pedagogically-inspired language subsets. Despite this, many novices still struggle to work effectively with DrScheme's error messages. To better understand why, we study the effectiveness of DrScheme's error messages. Unlike existing work in this area, we study messages at a fine-grained level by analyzing the edits students make in response to various classes of errors. We present our rubric, apply it to a course-worth of student lab work, and describe what we have learned about using the rubric effectively.

[pdf]