Publications
Publications
Toward Reproducible and Standardized Computer Architecture Simulation with gem5
conference
Kunal Pai, Harshil Patel, Erin Le, Noah Krim, Mahyar Samani, Bobby R. Bruce, Jason Lowe-Power
IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) 2026
To address the inconsistencies in simulation-based research, this work enhances the gem5 ecosystem by standardizing disk-image creation, introducing a flexible class-based exit event system for better guest-host communication, and implementing native tools like Suites and MultiSim to streamline and stabilize complex multi-workload workflows.
Computer Architecturegem5SimulationReproducibility
CoDocBench: A Dataset for Code-Documentation Alignment in Software Maintenance
conference
Kunal Pai, Premkumar Devanbu, Toufique Ahmed
International Conference on Mining Software Repositories (MSR) 2025: Data and Tool Showcase Track
Understanding and implementing code changes is a key aspect of software maintenance. To support this, we introduce a new dataset of coupled changes to code and documentation mined from high-quality GitHub projects, where each sample represents a single commit with simultaneous updates to code and docstrings. This dataset enables training and evaluation on realistic, change-related tasks, which remain challenging for current models like Llama 3.1 405B and Mixtral 8×22B.
Software EngineeringGitHub MiningLarge Language Models (LLMs)
Calibration and Correctness of Language Models for Code
conference
Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Amin Alipour, Sushmit Jha, Premkumar Devanbu, Toufique Ahmed
International Conference on Software Engineering (ICSE) 2025
Machine learning models often produce incorrect outputs, making reliable confidence measures essential for determining the trustworthiness of these outputs. This paper introduces a framework to evaluate and improve the calibration of code-generating models, finding that these models are generally poorly calibrated initially but can be improved using methods like Platt scaling, thereby enhancing decision-making in software engineering.
Software EngineeringMachine LearningNaturalness of Software
Automatic semantic augmentation of language model prompts (for code summarization)
conference
Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, Earl T. Barr
International Conference on Software Engineering (ICSE) 2024
Adding explicit semantic facts as prompts to Large Language Models improves their performance in code summarization tasks, with notable improvements exceeding 2 BLEU and, in some cases, even surpassing 30 BLEU, demonstrating the effectiveness of this approach in enhancing code analysis and extraction of essential information.
Software EngineeringMachine LearningNaturalness of Software