Measuring Bias in clustering

This notebook is a tutorial on auditing bias within a clustering task. We will use the holisticai library throughout, introducing some of the functions we have created to help study algorithmic bias.

The sections are organised as follows :

1. Load the data : we load the adult dataset as a pandas DataFrame

2. Data Exploration : some preliminary analysis of the data

3. Pre-Processing and Train a Model : we train a kmeans model (sklearn)

4. Measure Bias : we compute a few bias metrics, and comment on their meaning

Load the data

First of all, we need to import the required packages to perform our bias analysis and mitigation. You will need to have the holisticai package installed on your system, remember that you can install it by running:

!pip install holisticai[all]

# Imports
import numpy as np
import pandas as pd
import warnings
warnings.filterwarnings("ignore")

We will start by importing the adult dataset, which we host on our library. The adult dataset contains a set of informations extract from US 1994 Census database. It includes personal information about the individuals, specifically sex, race, and education. In this tutorial we will perform unsupervised learning to cluster the data, then measure whether this clustering contains gender or race information (clustering bias).

from holisticai.datasets import load_dataset

dataset = load_dataset('clinical_records', protected_attribute='sex')
dataset

[-] [Dataset]

Instances: 299

Features: X , y , p_attrs , group_a , group_b

Metadata: sex: {'group_a': '0', 'group_b': '1'}

Data Exploration

from holisticai.bias.plots import group_pie_plot, frequency_plot
import matplotlib.pyplot as plt

y = dataset['y']

fig,axs = plt.subplots(1, 2, figsize=(15,5))
group_pie_plot(dataset['group_a'], ax=axs[0], title='Group Pie Plot')
frequency_plot(dataset['group_a'], y, ax=axs[1], title='Frequency Plot')

<Axes: title={'center': 'Frequency Plot'}, xlabel='Group', ylabel='Frequency'>

Preprocess data and Train a model

from sklearn.cluster import KMeans

ks = range(1, 8)
inertias = []

X = dataset['X']

for k in ks:
    # create a KMeans instance with k clusters: model
    model = KMeans(n_clusters = k)

    # fit model to samples
    model.fit(X)

    # append the inertia to the list of inertias
    inertias.append(model.inertia_)

# Plot ks vs inertias
import matplotlib.pyplot as plt
plt.plot(ks, inertias, '-o')
plt.xlabel('Number of Clusters, k')
plt.ylabel('Inertia')
plt.xticks(ks)
plt.show()

# we choose to use 4 clusters
model = KMeans(n_clusters = 4)
model.fit(X)

# predict
y_pred = model.predict(X)

#centroids
centroids = model.cluster_centers_

Measure bias

The clustering is quite balanced in terms of gender representations.

from holisticai.bias.metrics import cluster_balance

cluster_balance(dataset['group_a'], dataset['group_b'], y_pred)

np.float64(0.5137457044673539)

The above metric reinforces the clustering is balanced in the sense of each group having similar representation in clusters to overall. The ideal value for this metric is 1.

Batch computation

from holisticai.bias.metrics import clustering_bias_metrics

centroids = model.cluster_centers_
data = X

metrics = clustering_bias_metrics(dataset['group_a'], dataset['group_b'], y_pred, data = data, centroids = centroids, metric_type = 'equal_outcome')
metrics

	Value	Reference
Metric
Cluster Balance	0.513746	1
Minimum Cluster Ratio	0.346154	1
Cluster Distribution Total Variation	0.096613	0
Cluster Distribution KL Div	0.035930	0
Social Fairness Ratio	1.139593	1
Silhouette Difference	-0.014285	0

Bias metrics report for clustering bias metrics.

from holisticai.bias.plots import bias_metrics_report

bias_metrics_report('clustering', metrics)