Evaluating Automated Program Repair Techniques using Introductory Programming Course Datasets

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Debugging erroneous programs requires a great deal of human effort. To reduce human effort, automating debugging processes has been actively studied so far. One of such automation is automated program repair techniques for syntactic errors in programs. Researchers intend to support novice programmers such as students with these techniques because fixing syntactic errors is a difficult task for novice programmers. However, there exist few datasets that consist of programs written by novice programmers in universities and can be used to evaluate these techniques. Also, it is difficult to prepare such datasets from scratch. Indeed, prior studies usually utilized the Indian Institute of Technology Kanpur (IITK) dataset only. This limitation restricts the findings and implications in prior studies as a case study in the university. In this study, we intend to clarify which findings and implications in prior studies remain the same and which ones change in another university by a case study. We prepare three datasets that consist of over 21k programs collected from an introductory programming course in different divisions at our university. We compare the state-of-the-art automated program repair techniques, DeepFix, RLAssist, and Dr Repair, in these datasets. We found that (1) the best technique remains the same in all the datasets, (2) these techniques fix 8.3 % to 54.5 % syntactic errors in our datasets, which are 6.7 % to 32.4 % lower than those in the IITK dataset, (3) the error types that are fixed by them change in different datasets. Hence, the main finding in the IITK dataset remains the same; however, each technique fixes different errors.

Original languageEnglish
Title of host publicationProceedings - 2022 IEEE 46th Annual Computers, Software, and Applications Conference, COMPSAC 2022
EditorsHong Va Leong, Sahra Sedigh Sarvestani, Yuuichi Teranishi, Alfredo Cuzzocrea, Hiroki Kashiwazaki, Dave Towey, Ji-Jiang Yang, Hossain Shahriar
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages569-574
Number of pages6
ISBN (Electronic)9781665488105
DOIs
Publication statusPublished - 2022
Event46th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2022 - Virtual, Online, United States
Duration: Jun 27 2022Jul 1 2022

Publication series

NameProceedings - 2022 IEEE 46th Annual Computers, Software, and Applications Conference, COMPSAC 2022

Conference

Conference46th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2022
Country/TerritoryUnited States
CityVirtual, Online
Period6/27/227/1/22

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Hardware and Architecture
  • Software
  • Media Technology
  • Education

Fingerprint

Dive into the research topics of 'Evaluating Automated Program Repair Techniques using Introductory Programming Course Datasets'. Together they form a unique fingerprint.

Cite this