Privacy risk score analysis for classification models

In this notebook, we will analyze the privacy risks of a classification model. We will use the Adult Income dataset from the UCI Machine Learning Repository.

This is useful for understanding some privacy concepts related to membership inference attacks in machine learning models. To do this, we will use the shadow training technique commonly used in membership inference attacks and then we will evaluate the privacy risk score of the model concerning the dataset.

Let’s start by importing the necessary libraries and loading the dataset.

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from holisticai.datasets import load_dataset
import numpy as np
from sklearn import metrics

from holisticai.security.metrics import privacy_risk_score

To understand the privacy risk score, we will use the shadow training technique. This technique consists of training a shadow model to mimic the target model.

For our purposes, we will assume that the adversary has access to part of the training dataset and full access to the testing dataset. The adversary will use this dataset to train the shadow model, which is a model that tries to mimic the target model.

To do this, once we have our training and testing sets, we will split the training dataset into two parts: the target dataset, which will be used to calculate the probabilities from the target model; and the shadow dataset, which will be used to train the target model. We will use the following proportions: 60% for the target dataset and 40% for the shadow dataset.

dataset = load_dataset('adult', protected_attribute='sex', preprocessed=True)
train_test = dataset.train_test_split(test_size=0.2, random_state=42)
train = train_test['train']
target_shadow = train.train_test_split(test_size=0.4, random_state=42)

X_target_train = target_shadow['train']['X']
y_target_train = target_shadow['train']['y']
X_shadow_train = target_shadow['test']['X']
y_shadow_train = target_shadow['test']['y']

X_test = train_test['test']['X']
y_test = train_test['test']['y']

X_train = train['X']
y_train = train['y']

print("Training set size:", X_train.shape[0])
print("Test set size:", X_test.shape[0])

print("Target Model Training set size:", X_target_train.shape[0])
print("Shadow Model Training set size:", X_shadow_train.shape[0])

Training set size: 36177
Test set size: 9045
Target Model Training set size: 21706
Shadow Model Training set size: 14471

Models training

Now, we can train the target model. In this case, we will use a Random Forest classifier.

# Train the Target Model
target_model = RandomForestClassifier(random_state=42)
target_model.fit(X_train, y_train)
y_target_pred = target_model.predict(X_test)
print("Target Model Performance:")
print("Accuracy:", accuracy_score(y_test, y_target_pred))

Target Model Performance:
Accuracy: 0.851077943615257

We will do the same for the shadow model. Just remember that the shadow model will be trained with the shadow dataset.

# Train the Shadow Model
shadow_model = RandomForestClassifier(random_state=42)
shadow_model.fit(X_shadow_train, y_shadow_train)
y_shadow_pred = shadow_model.predict(X_test)
print("\nShadow Model Performance:")
print("Accuracy:", accuracy_score(y_test, y_shadow_pred))

Shadow Model Performance:
Accuracy: 0.8486456605859591

Privacy risk score calculation

Once we have the target and shadow models, we can calculate the privacy risk score for each sample in the dataset. With this metric, we can estimate the probability of a sample belonging to the training dataset of the target model. In this way we can identify which samples present a higher risk of privacy leakage. To read more about the privacy risk score, you can check the original paper.

To measure the privacy risk score, we first need to calculate the probability of the training and testing samples belonging to the target and shadow models.

shadow_train_probs = shadow_model.predict_proba(X_shadow_train)
shadow_test_probs = shadow_model.predict_proba(X_test)
target_train_probs = target_model.predict_proba(X_target_train)
target_test_probs = target_model.predict_proba(X_test)

Having the probabilities of the input samples belonging to the target and shadow model, we can then calculate the probability of the input samples belonging to the training set by using the privacy_risk_score function. This function receives as input tuples with the probabilities of the training and testing samples for the target and shadow models.

We will first calculate the privacy risk score for the training dataset and then for the testing dataset.

risk_score_train = privacy_risk_score((shadow_train_probs, y_shadow_train), (shadow_test_probs, y_test), (target_train_probs, y_target_train))
print("Mean Privacy Risk Score for train: ", risk_score_train.mean())

risk_score_test = privacy_risk_score((shadow_train_probs, y_shadow_train), (shadow_test_probs, y_test), (target_test_probs, y_test))
print("Mean Privacy Risk Score for test: ", risk_score_test.mean(), risk_score_test.std())

Mean Privacy Risk Score for train:  0.5550201103971106
Mean Privacy Risk Score for test:  0.4502210917990319 0.19709890203635058

Now that we have calculated the privacy risk score of the training and testing set, we can then analyze the privacy risk of the target model by varing the threshold of the privacy risk score.

To do this we will calculate the precision and recall of the privacy risk score for different thresholds. We will use the ROC curve function to calculate the false positive rate and true positive rate for different thresholds and then, we will use the calculated values to analyze the privacy risk of the target model at different thresholds.

To create the ROC curve, we will use the roc_curve function from the sklearn.metrics module. This function receives as input the true labels and the predicted probabilities of the positive class. As labels, we know previously if the samples belong to the training or testing set, so we can use this information to create the labels for the ROC curve. On the other hand, the scores are the calculated privacy risk scores.

labels = np.concatenate((np.ones(len(risk_score_train)), np.zeros(len(risk_score_test))))
scores = np.concatenate((risk_score_train, risk_score_test))

# Compute ROC curve
fpr, tpr, thresholds = metrics.roc_curve(labels, scores, drop_intermediate=False)

# Define threshold list and find meaningful thresholds
threshold_list = [1., 0.9, 0.8, 0.7, 0.6, 0.5, 0.4]
max_prob = max(risk_score_train.max(), risk_score_test.max())
meaningful_thresholds = [threshold for threshold in threshold_list if threshold <= max_prob]

# Find indices for meaningful thresholds
indices = [np.argwhere(thresholds >= threshold)[-1][0] for threshold in meaningful_thresholds]

# Calculate precision and recall
precision_list = tpr[indices] / (tpr[indices] + fpr[indices])
recall_list = tpr[indices]

Finally, the previous calculations allow us to analyze the privacy risk of the target model at different thresholds. We can do this by analyzing the precision and recall of the privacy risk score to extract insights about the privacy risk of the model.

for idx, threshold in enumerate(meaningful_thresholds):
    print(f"Threshold: {threshold}, Precision: {precision_list[idx]}, Recall: {recall_list[idx]}")

Threshold: 0.7, Precision: 0.6794304779738761, Recall: 0.05436284898184834
Threshold: 0.6, Precision: 0.6408083273937311, Recall: 0.2475352437114162
Threshold: 0.5, Precision: 0.5715656917862854, Recall: 0.9259651709204828
Threshold: 0.4, Precision: 0.5715656917862854, Recall: 0.9259651709204828

With these results, we can say the following:

• If the privacy risk score is set as 0.7, there are ~5% of the training set members that can be inferred correctly by the adversary with a ~68% of precision.

• If we decrease the privacy risk score to 0.6, there are ~25% of the training set members that can be inferred correctly by the adversary with a ~64% of precision.

• Finally, if the privacy risk score is set as 0.5, there are ~92% of the training set members that can be inferred correctly by the adversary with a ~57% of precision. The same is maintained if we decrease the privacy risk score to 0.4.

So we can say that the privacy risk of the model is not so high, since the precision is not so high for the different thresholds. However, we can see that the recall value is considerable at least when a threshold of 0.6 is set, which means that the adversary can infer a considerable percentage (almost 25%) of the training set members. This is a privacy risk that should be taken into account when deploying the model.

Summary

In this notebook, we have analyzed the privacy risk of a classification model using the shadow training technique. We have calculated the privacy risk score for the training and testing datasets and then we have analyzed the privacy risk of the model at different thresholds. We have seen that the model presents a considerable privacy risk, since the adversary can infer a considerable percentage (24%) of the training set members with a considerable precision (67.8%). This is a privacy risk that should be taken into account when deploying the model and it is important to analyze the privacy risk of the model to avoid privacy leakage.