Understanding and Detecting Flaky Builds in GitHub Actions

Project Overview

This research project investigates flaky builds in continuous integration (CI) systems, providing comprehensive failure analysis data and predictive models. Our work aims to help developers identify, understand, and mitigate build failures through automated tools and machine learning approaches.

All data, models, and tools are openly available to support reproducible research and practical application in software development workflows.

Failure Analysis Data

We analyzed thousands of CI build failures and categorized them by root cause and mitigation patterns. The table below presents our comprehensive failure taxonomy with detailed case studies.

Key Findings

• Network-related issues constitute to largest category of flaky builds (15.8%)
• Simple rerun strategy effectively resolves 80% of network failures
• Dependency resolution issues often mask underlying network problems
• Artifact caching significantly reduces missing dependency failures

Hyperparameter Tuning Results

We performed extensive hyperparameter optimization across 10 repositories for baseline and proposed models. The optimal configurations are presented below.

Note: * Project shortened by omitting owner.

Optimization Process

Hyperparameter search was conducted over 500 iterations using 5-fold cross-validation across diverse repositories. Each model was tuned independently per project to account for project-specific characteristics.

Ablation Study

We conducted systematic ablation experiments to understand the contribution of each feature group to model performance. Results demonstrate the importance of temporal and code change features.

Insights

• Code change features have the highest impact (-5.8% when removed)
• Temporal features provide significant predictive power (-3.6%)
• Historical failure patterns contribute moderately (-2.1%)
• CI configuration features show least but still meaningful impact (-1.3%)

Performance Comparison

Comparative analysis of our predictive models against baseline and state-of-the-art approaches. Line charts below illustrate performance trends across different metrics.

Performance Summary

Our XGBoost-based model achieves state-of-the-art performance with 89.2% F1 score, representing a 12.4% improvement over the previous best approach. The model demonstrates robust generalization across different project types and CI platforms.

Code & Tools

All source code and tools are openly available. Our repository includes three main components: CI reproduction tool, feature extraction tool, and predictive models.

Dataset Access

Citation

If you use our data, tools, or models in your research, please cite our work:

@inproceedings{flakybuild2025,
  title={Understanding and Detecting Flaky Builds in GitHub Actions},
  author={[Your Name et al.]},
  booktitle={Proceedings of [Conference Name]},
  year={2025}
}