Iterative Student Program Planning using Transformer-Driven Feedback

Problem planning is a fundamental programming skill, and aids students in decomposing tasks into manageable subtasks. While feedback on plans is beneficial for beginners, providing this in a scalable and timely way is an enormous challenge in large courses. Recent advances in LLMs raise the prospect of helping here. We utilize LLMs to generate code based on students' plans, and evaluate the code against expert-defined test suites. Students receive feedback on their plans and can refine them. In this report, we share our experience with the design and implementation of this workflow. This tool was used by 544 students in a CS1 course at an Austrian university. We developed a codebook to evaluate their plans and manually applied it to a sample. We show that LLMs can play a valuable role here. However, we also highlight numerous cautionary aspects of using LLMs in this context, many of which will not be addressed merely by having more powerful models (and indeed may be exacerbated by it).