Implementing Fairness Gates in Your CI/CD Pipeline
Implementing Fairness Gates in Your CI/CD Pipeline
A fairness gate is a CI/CD check that computes bias metrics on your model and blocks deployment if those metrics fall outside defined thresholds — the same way a test suite blocks deployment on a failing unit test. For teams building AI systems subject to the EU AI Act, adding fairness gates is the most practical way to catch bias issues before they reach production and before regulators find them.
This guide covers the implementation using Python, GitHub Actions, and scikit-learn's fairness metrics.
Why a Gate, Not Just a Report
Most teams run bias checks as part of their evaluation pipeline and review the results manually. The problem is that manual review creates a gap between "we measured bias" and "we acted on bias." Under Article 10 of the EU AI Act, data governance requirements include demonstrating that you have processes to identify and address bias — not just measure it.
A gate that fails the build is proof of that process. A report that gets reviewed (sometimes) is not.
The Metrics to Gate On
The EU AI Act does not specify which fairness metrics to use, but it requires that you assess bias and document your methodology. The most commonly used metrics for high-risk AI systems are:
Demographic Parity Difference — the difference in positive prediction rates between groups. A value of 0 means both groups are equally likely to receive a positive outcome. A common threshold is ±0.1.
Equal Opportunity Difference — the difference in true positive rates between groups. Measures whether the model is equally good at identifying positive outcomes across groups. Threshold: ±0.1.
Disparate Impact Ratio — the ratio of positive prediction rates between groups. A ratio below 0.8 is the "four-fifths rule" used in US employment law, and a useful starting point for EU contexts too.
Average Odds Difference — the average of the difference in false positive rates and true positive rates across groups. Captures both types of error simultaneously.
For most high-risk use cases (HR, credit, healthcare), gate on at least Demographic Parity Difference and Equal Opportunity Difference.
Implementation: Python Script
```python
fairness_check.py
import pandas as pd
import json
import sys
from sklearn.metrics import confusion_matrix
def compute_fairness_metrics(y_true, y_pred, sensitive_feature):
groups = sensitive_feature.unique()
if len(groups) != 2:
raise ValueError(f"Expected binary sensitive feature, got {len(groups)} groups")
group_a, group_b = groups[0], groups[1]
def metrics_for_group(group_mask):
yt = y_true[group_mask]
yp = y_pred[group_mask]
tn, fp, fn, tp = confusion_matrix(yt, yp, labels=[0, 1]).ravel()
positive_rate = (tp + fp) / len(yp)
tpr = tp / (tp + fn) if (tp + fn) > 0 else 0
fpr = fp / (fp + tn) if (fp + tn) > 0 else 0
return positive_rate, tpr, fpr
pr_a, tpr_a, fpr_a = metrics_for_group(sensitive_feature == group_a)
pr_b, tpr_b, fpr_b = metrics_for_group(sensitive_feature == group_b)
return {
"demographic_parity_difference": abs(pr_a - pr_b),
"equal_opportunity_difference": abs(tpr_a - tpr_b),
"average_odds_difference": (abs(fpr_a - fpr_b) + abs(tpr_a - tpr_b)) / 2,
"disparate_impact_ratio": min(pr_a, pr_b) / max(pr_a, pr_b) if max(pr_a, pr_b) > 0 else 1.0,
"groups": {str(group_a): {"positive_rate": pr_a, "tpr": tpr_a},
str(group_b): {"positive_rate": pr_b, "tpr": tpr_b}},
}
def check_thresholds(metrics, thresholds):
violations = []
if metrics["demographic_parity_difference"] > thresholds.get("demographic_parity_difference", 0.1):
violations.append(f"Demographic parity difference {metrics['demographic_parity_difference']:.3f} exceeds threshold {thresholds['demographic_parity_difference']}")
if metrics["equal_opportunity_difference"] > thresholds.get("equal_opportunity_difference", 0.1):
violations.append(f"Equal opportunity difference {metrics['equal_opportunity_difference']:.3f} exceeds threshold {thresholds['equal_opportunity_difference']}")
if metrics["disparate_impact_ratio"] < thresholds.get("disparate_impact_ratio_min", 0.8):
violations.append(f"Disparate impact ratio {metrics['disparate_impact_ratio']:.3f} below threshold {thresholds['disparate_impact_ratio_min']}")
return violations
if __name__ == "__main__":
# Load predictions CSV: columns y_true, y_pred, sensitive_feature
df = pd.read_csv(sys.argv[1])
thresholds = {
"demographic_parity_difference": 0.1,
"equal_opportunity_difference": 0.1,
"disparate_impact_ratio_min": 0.8,
}
metrics = compute_fairness_metrics(df["y_true"], df["y_pred"], df["sensitive_feature"])
violations = check_thresholds(metrics, thresholds)
print(json.dumps({"metrics": metrics, "violations": violations}, indent=2))
if violations:
print(f"\nFAIRNESS GATE FAILED: {len(violations)} violation(s)")
for v in violations:
print(f" ✗ {v}")
sys.exit(1)
else:
print("\nFAIRNESS GATE PASSED")
sys.exit(0)
```
GitHub Actions Workflow
```yaml
.github/workflows/fairness-gate.yml
name: Fairness Gate
on:
pull_request:
paths:
- 'models/'
- 'training/'
- 'data/'
jobs:
fairness-check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install dependencies
run: pip install pandas scikit-learn
- name: Run model evaluation
run: python scripts/evaluate_model.py --output predictions.csv
- name: Run fairness gate
run: python scripts/fairness_check.py predictions.csv
- name: Upload fairness report
if: always()
uses: actions/upload-artifact@v4
with:
name: fairness-report
path: fairness_report.json
```
Integrating with Guardia AI
If you are using Guardia AI's CI/CD plugin, the fairness gate is included. The plugin runs both the prohibited AI pattern check (blocking forbidden imports) and the fairness metrics check in a single workflow step, and posts results to your Guardia AI dashboard for audit logging.
Threshold Calibration
The right thresholds depend on your use case. €0.1 is a common starting point for demographic parity difference, but:
Document your threshold choices in your Annex IV documentation. "We chose 0.1 because it is the industry default" is weaker than "We chose 0.08 based on the sensitivity of HR decisions and the distribution of our training data."
Handling Gate Failures
When the gate fails, the options are:
1. Retrain with rebalanced data — if a demographic group is underrepresented in training data, augment it
2. Apply post-processing — adjust decision thresholds per group to achieve parity (document this carefully)
3. Reduce model capability — sometimes a simpler model is fairer; a high-accuracy model that is significantly biased is worse than a slightly less accurate fair one
4. Override with documented justification — if the business context makes a threshold violation acceptable (e.g., a medical test that is more sensitive for one group by design), document the justification explicitly in Annex IV
Do not disable the gate or raise thresholds to make the failure go away without documenting the decision. That creates a paper trail showing you knew about the bias and chose to deploy anyway.