Bias mitigation with debiasing exposure, disparate impact remover and Fair top K

This demo will show how to implement the post-processing methods “debiasing exposure”, “disparate impact remover” and “Fair top K” to enhance the fairness of a recommender system’s output.

• Debiasing Exposure

• Disparate Impact Remover

• Fair Top K

First, install the holisticai package if you haven’t already:

!pip install holisticai[all]

Then, import the necessary libraries.

# Base Imports
import pandas as pd
import numpy as np
from holisticai.datasets.synthetic.recruitment import generate_rankings
from holisticai.bias.mitigation.postprocessing.debiasing_exposure.algorithm_utils import exposure_metric
from holisticai.bias.mitigation.postprocessing import DebiasingExposure

np.random.seed(0)
import warnings
warnings.filterwarnings("ignore")

The dataset that we will use is a synthetic ranking dataset generated following the procedure described by Yang and Stoyanovich in their research. This algorithm creates a ranked output of protected and unprotected candidates with a certain probability.

# Synthetic data
M = 1000
top_n = 20
p = 0.25
rankings = generate_rankings(M, top_n, p, return_p_attr=False)

baseline = exposure_metric(rankings, group_col='protected', query_col='X', score_col='score')
baseline

	Value
exposure_ratio	37027.714945
exposure difference	0.047272

Bias mitigation

Method: Debiasing exposure

Apply the debiasing exposure mitigator algorithm.

# create the DebiasingExposure class
dtr = DebiasingExposure(group_col="protected",
                        query_col = 'X',
                        doc_col = 'Y',
                        feature_cols = ['score', 'protected'],
                        score_col = 'score',
                        gamma=2,
                        number_of_iterations=100,
                        standardize=True,
                        verbose=1)

# train the model
dtr.fit(rankings)

<holisticai.bias.mitigation.postprocessing.debiasing_exposure.transformer.DebiasingExposure at 0x7f559cea5c10>

re_rankings = dtr.transform(rankings)

Observe the fairness metrics before and after applying the algorithm (lower is better).

df_deb_exp = exposure_metric(re_rankings, group_col='protected', query_col='X', score_col='score')
df_deb_exp

	Value
exposure_ratio	0.755373
exposure difference	0.002431

result = pd.concat([baseline, df_deb_exp], axis=1).iloc[:, [0,1]]
result.columns = ['Baseline','Mitigator']
result

	Baseline	Mitigator
exposure_ratio	37027.714945	0.755373
exposure difference	0.047272	0.002431

Method: Disparate impact remover

from holisticai.bias.mitigation import DisparateImpactRemoverRS

dir = DisparateImpactRemoverRS(query_col='X', group_col='protected', score_col='score', repair_level=1)
re_rankings = dir.transform(rankings)

df_dis_imp = exposure_metric(re_rankings, group_col='protected', query_col='X', score_col='score')
df_dis_imp

	Value
exposure_ratio	1.003761
exposure difference	0.001925

result = pd.concat([baseline, df_dis_imp], axis=1).iloc[:, [0,1]]
result.columns = ['Baseline','Mitigator']
result

	Baseline	Mitigator
exposure_ratio	37027.714945	1.003761
exposure difference	0.047272	0.001925

Method: Fair Top-K

Now, we will implement the Fair Top-K algorithm, this method works differently from the previous ones. Given a list of items, it will reorganize the list to ensure that the top-K items are fairer.

Let’s create a unfair list to apply the Fair Top-K algorithm.

def create_unfair_example(ranking, n):
    """
    Setting an unfair ranking where protected group examples are only the last n results.
    """
    ranking = ranking.copy()
    ranking['protected']=False
    ranking['protected'].iloc[-n:]=True
    return ranking

M = 1
k = 20
p = 0.1
ranking = generate_rankings(M, k, p)

unfair_ranking = create_unfair_example(ranking, 6)

from holisticai.bias.mitigation.postprocessing.fair_topk.transformer import FairTopK

# Bias Mitigation Post-processing
top_n = 20
p = 0.9
alpha = 0.15
fs = FairTopK(top_n=top_n,
              p=p,
              alpha=alpha,
              query_col='X',
              doc_col='Y',
              score_col='score',
              group_col='protected')

re_ranking = fs.transform(unfair_ranking)

Let’s observe how the original and unfair ranking was modified:

def compare_results(old , new):
    old = old.copy()
    new = new.copy()
    old.columns = pd.MultiIndex.from_tuples([['Old Rank',col] for col in old.columns])
    new.columns = pd.MultiIndex.from_tuples([['New Rank',col] for col in new.columns])
    return pd.concat([old.reset_index(drop=True),new.reset_index(drop=True)], axis=1)

compare_results(unfair_ranking , re_ranking)

	Old Rank				New Rank
	X	Y	score	protected	X	Y	score	protected
0	0	20	20	False	0	20	20	False
1	0	19	19	False	0	6	6	True
2	0	18	18	False	0	5	5	True
3	0	17	17	False	0	4	4	True
4	0	16	16	False	0	3	3	True
5	0	15	15	False	0	19	19	False
6	0	14	14	False	0	2	2	True
7	0	13	13	False	0	1	1	True
8	0	12	12	False	0	18	18	False
9	0	11	11	False	0	17	17	False
10	0	10	10	False	0	16	16	False
11	0	9	9	False	0	15	15	False
12	0	8	8	False	0	14	14	False
13	0	7	7	False	0	13	13	False
14	0	6	6	True	0	12	12	False
15	0	5	5	True	0	11	11	False
16	0	4	4	True	0	10	10	False
17	0	3	3	True	0	9	9	False
18	0	2	2	True	0	8	8	False
19	0	1	1	True	0	7	7	False