Metrics

holisticai.bias.metrics is a python module measuring bias in algorithms. Metrics are included for classification, regression, clustering, recommender and multiclass tasks.

Binary Classification

holisticai.bias.metrics.abroca(group_a, ...)	ABROCA (area between roc curves).
holisticai.bias.metrics.accuracy_diff(...)	Accuracy Difference.
holisticai.bias.metrics.average_odds_diff(...)	Average Odds Difference.
holisticai.bias.metrics.classification_bias_metrics(...)	Classification bias metrics batch computation.
holisticai.bias.metrics.cohen_d(group_a, ...)	Cohen D.
holisticai.bias.metrics.disparate_impact(...)	Disparate Impact.
holisticai.bias.metrics.equal_opportunity_diff(...)	Equality of opportunity difference.
holisticai.bias.metrics.false_negative_rate_diff(...)	False negative Rate difference.
holisticai.bias.metrics.false_positive_rate_diff(...)	False positive rate difference.
holisticai.bias.metrics.four_fifths(group_a, ...)	Four Fifths.
holisticai.bias.metrics.statistical_parity(...)	Statistical parity.
holisticai.bias.metrics.true_negative_rate_diff(...)	True negative Rate difference.
holisticai.bias.metrics.z_test_diff(group_a, ...)	Z Test (Difference).
holisticai.bias.metrics.z_test_ratio(...)	Z Test (Ratio).

Multiclass Classification

holisticai.bias.metrics.accuracy_matrix(...)	Multiclass Accuracy Matrix.
holisticai.bias.metrics.confusion_matrix(...)	Confusion Matrix.
holisticai.bias.metrics.confusion_tensor(...)	Confusion Tensor.
holisticai.bias.metrics.frequency_matrix(...)	Frequency Matrix.
holisticai.bias.metrics.multiclass_average_odds(...)	Multiclass Average Odds.
holisticai.bias.metrics.multiclass_bias_metrics(...)	Multiclass bias metrics batch computation.
holisticai.bias.metrics.multiclass_equality_of_opp(...)	Multiclass Equality of Opportunity.
holisticai.bias.metrics.multiclass_statistical_parity(...)	Multiclass statistical parity.
holisticai.bias.metrics.multiclass_true_rates(...)	Multiclass True Rates.
holisticai.bias.metrics.precision_matrix(...)	Multiclass Precision Matrix.
holisticai.bias.metrics.recall_matrix(...[, ...])	Multiclass Recall Matrix.

Regression

holisticai.bias.metrics.avg_score_diff(...)	Average Score Difference.
holisticai.bias.metrics.correlation_diff(...)	Correlation difference.
holisticai.bias.metrics.disparate_impact_regression(...)	Disparate Impact quantile (Regression version).
holisticai.bias.metrics.mae_ratio(group_a, ...)	MAE ratio
holisticai.bias.metrics.max_statistical_parity(...)	Max absolute statistical parity.
holisticai.bias.metrics.no_disparate_impact_level(...)	No disparate impact level.
holisticai.bias.metrics.regression_bias_metrics(...)	Regression bias metrics batch computation
holisticai.bias.metrics.rmse_ratio(group_a, ...)	RMSE ratio.
holisticai.bias.metrics.statistical_parity_auc(...)	Statistical parity (AUC).
holisticai.bias.metrics.statistical_parity_regression(...)	Statistical Parity quantile (Regression version).
holisticai.bias.metrics.zscore_diff(group_a, ...)	ZScore Difference.

Clustering

holisticai.bias.metrics.cluster_balance(...)	Cluster Balance.
holisticai.bias.metrics.cluster_dist_entropy(...)	Minority Cluster Distribution Entropy.
holisticai.bias.metrics.cluster_dist_kl(...)	Cluster Distribution KL.
holisticai.bias.metrics.cluster_dist_l1(...)	Cluster Distribution Total Variation.
holisticai.bias.metrics.clustering_bias_metrics(...)	Clustering bias metrics batch computation.
holisticai.bias.metrics.min_cluster_ratio(...)	Minimum Cluster Ratio.
holisticai.bias.metrics.silhouette_diff(...)	Silhouette Difference.
holisticai.bias.metrics.social_fairness_ratio(...)	Social Fairness Ratio.

Recommender

holisticai.bias.metrics.aggregate_diversity(...)	Aggregate Diversity.
holisticai.bias.metrics.avg_f1_ratio(...[, ...])	Average f1 ratio
holisticai.bias.metrics.avg_precision_ratio(...)	Average precision ratio
holisticai.bias.metrics.avg_recall_ratio(...)	Average recall ratio
holisticai.bias.metrics.avg_recommendation_popularity(...)	Average Recommendation Popularity.
holisticai.bias.metrics.exposure_entropy(...)	Exposure Entropy.
holisticai.bias.metrics.exposure_kl(group_a, ...)	Exposure KL Divergence.
holisticai.bias.metrics.exposure_l1(group_a, ...)	Exposure Total Variation.
holisticai.bias.metrics.gini_index(mat_pred)	GINI index.
holisticai.bias.metrics.mad_score(group_a, ...)	Mean Absolute Deviation.
holisticai.bias.metrics.recommender_bias_metrics([...])	Recommender bias metrics batch computation.
holisticai.bias.metrics.recommender_mae_ratio(...)	Recommender MAE ratio.
holisticai.bias.metrics.recommender_rmse_ratio(...)	Recommender RMSE ratio.