Packages Installation

First, install the holisticai package if you haven’t already:

!pip install holisticai[all]

Then, import the necessary libraries.

import warnings

import pandas as pd
from holisticai.bias.metrics import multiclass_bias_metrics
from holisticai.datasets import load_dataset
from holisticai.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

warnings.filterwarnings("ignore")

Data Loading

dataset = load_dataset('us_crime_multiclass', protected_attribute="race")
train_test = dataset.train_test_split(test_size=0.2, random_state=42)

train = train_test['train']
test = train_test['test']

1. Fair Scoring Classifier

Traditional implementation

from holisticai.bias.mitigation import FairScoreClassifier

# fit the mitigator
mitigator = FairScoreClassifier(objectives="ba", constraints={}, time_limit=200)
scaler = StandardScaler()
X_train = scaler.fit_transform(train['X'])
mitigator.fit(X_train, train['y'], train['group_a'], train['group_b'])

# predict the test set
X_test = scaler.transform(test['X'])
y_pred = mitigator.predict(X_test, test['group_a'], test['group_b'])

# compute bias metrics
metrics = multiclass_bias_metrics(test['group_a'], y_pred, test['y'], metric_type='both')
metrics

	Value	Reference
Metric
Max Multiclass Statistical Parity	0.857143	0
Mean Multiclass Statistical Parity	0.857143	0
Max Multiclass Equality of Opportunity	0.631313	0
Max Multiclass Average Odds	0.580808	0
Max Multiclass True Positive Difference	0.353535	0
Mean Multiclass Equality of Opportunity	0.631313	0
Mean Multiclass Average Odds	0.580808	0
Mean Multiclass True Positive Difference	0.353535	0

Pipeline implementation

mitigator = FairScoreClassifier(objectives="ba", constraints={}, time_limit=200)

# set pipeline
pipeline = Pipeline(steps=[('scalar', StandardScaler()),("bm_inprocessing", mitigator),])
pipeline.fit(train['X'], train['y'], bm__group_a=train['group_a'], bm__group_b=train['group_b'])

# predict
y_pred = pipeline.predict(test['X'], bm__group_a=test['group_a'], bm__group_b=test['group_b'])

# compute bias metrics
metrics_pipeline = multiclass_bias_metrics(test['group_a'], y_pred, test['y'], metric_type='both')
metrics_pipeline

	Value	Reference
Metric
Max Multiclass Statistical Parity	0.857143	0
Mean Multiclass Statistical Parity	0.857143	0
Max Multiclass Equality of Opportunity	0.631313	0
Max Multiclass Average Odds	0.580808	0
Max Multiclass True Positive Difference	0.353535	0
Mean Multiclass Equality of Opportunity	0.631313	0
Mean Multiclass Average Odds	0.580808	0
Mean Multiclass True Positive Difference	0.353535	0

Comparison

pd.concat([metrics['Value'], metrics_pipeline], axis=1, keys=['Traditional', 'Pipeline'])

	Traditional	Pipeline
	Value	Value	Reference
Metric
Max Multiclass Statistical Parity	0.857143	0.857143	0
Mean Multiclass Statistical Parity	0.857143	0.857143	0
Max Multiclass Equality of Opportunity	0.631313	0.631313	0
Max Multiclass Average Odds	0.580808	0.580808	0
Max Multiclass True Positive Difference	0.353535	0.353535	0
Mean Multiclass Equality of Opportunity	0.631313	0.631313	0
Mean Multiclass Average Odds	0.580808	0.580808	0
Mean Multiclass True Positive Difference	0.353535	0.353535	0