Analyzing privacy risk score for classification models

In this notebook, we will analyze the privacy risks of a classification model. We will use the bank marketing dataset from the UCI Machine Learning Repository. This dataset is related to direct marketing campaigns of a Portuguese banking institution. The classification goal is to predict if the client will subscribe a term deposit.

This is useful for understanding some privacy concepts related to membership inference attacks in machine learning models. To do this, we will use the shadow training technique commonly used in membership inference attacks and then we will evaluate the privacy risk score of a target model concerning the dataset. Then, we will analyze the privacy risk score by changing the model architecture to observe the impact of the model choice on the privacy risk score.

Let’s start by importing the necessary libraries and loading the dataset.

from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from holisticai.datasets import load_dataset
import numpy as np
from sklearn import metrics

from holisticai.security.metrics import privacy_risk_score

To understand the privacy risk score, we will use the shadow training technique. This technique consists of training a shadow model to mimic the target model.

For our purposes, we will assume that the adversary has access to part of the training dataset and full access to the testing dataset. The adversary will use this dataset to train the shadow model, which is a model that tries to mimic the target model.

To do this, once we have our training and testing sets, we will split the training dataset into two parts: the target dataset, which will be used to calculate the probabilities from the target model; and the shadow dataset, which will be used to train the target model. We will use the following proportions: 60% for the target dataset and 40% for the shadow dataset.

dataset = load_dataset('bank_marketing', protected_attribute='marital', preprocessed=True)
train_test = dataset.train_test_split(test_size=0.2, random_state=42)
train = train_test['train']
target_shadow = train.train_test_split(test_size=0.4, random_state=42)

X_target_train = target_shadow['train']['X']
y_target_train = target_shadow['train']['y']
X_shadow_train = target_shadow['test']['X']
y_shadow_train = target_shadow['test']['y']

X_test = train_test['test']['X']
y_test = train_test['test']['y']

X_train = train['X']
y_train = train['y']

print("Training set size:", X_train.shape[0])
print("Test set size:", X_test.shape[0])

print("Target Model Training set size:", X_target_train.shape[0])
print("Shadow Model Training set size:", X_shadow_train.shape[0])

Training set size: 36168
Test set size: 9043
Target Model Training set size: 21700
Shadow Model Training set size: 14468

Models training

Now, we can train the target model. In this case, we will use a Random Forest classifier.

# Train the Target Model
target_model = RandomForestClassifier(random_state=42)
target_model.fit(X_train, y_train)
y_target_pred = target_model.predict(X_test)
print("Target Model Performance:")
print("Accuracy:", accuracy_score(y_test, y_target_pred))

Target Model Performance:
Accuracy: 0.9027977441114674

We will do the same for the shadow model. Just remember that the shadow model will be trained with the shadow dataset.

# Train the Shadow Model
shadow_model = RandomForestClassifier(random_state=42)
shadow_model.fit(X_shadow_train, y_shadow_train)
y_shadow_pred = shadow_model.predict(X_test)
print("\nShadow Model Performance:")
print("Accuracy:", accuracy_score(y_test, y_shadow_pred))

Shadow Model Performance:
Accuracy: 0.8974897710936636

Privacy risk score calculation

Once we have the target and shadow models, we can calculate the privacy risk score for each sample in the dataset. With this metric, we can estimate the probability of a sample belonging to the training dataset of the target model. In this way we can identify which samples present a higher risk of privacy leakage. To read more about the privacy risk score, you can check the original paper.

To measure the privacy risk score, we first need to calculate the probability of the training and testing samples belonging to the target and shadow models.

shadow_train_probs = shadow_model.predict_proba(X_shadow_train)
shadow_test_probs = shadow_model.predict_proba(X_test)
target_train_probs = target_model.predict_proba(X_target_train)
target_test_probs = target_model.predict_proba(X_test)

Having the probabilities of the input samples belonging to the target and shadow model, we can then calculate the probability of the input samples belonging to the training set by using the privacy_risk_score function. This function receives as input tuples with the probabilities of the training and testing samples for the target and shadow models.

We will first calculate the privacy risk score for the training dataset and then for the testing dataset.

risk_score_train = privacy_risk_score((shadow_train_probs, y_shadow_train), (shadow_test_probs, y_test), (target_train_probs, y_target_train))
print("Mean Privacy Risk Score for train: ", risk_score_train.mean())

risk_score_test = privacy_risk_score((shadow_train_probs, y_shadow_train), (shadow_test_probs, y_test), (target_test_probs, y_test))
print("Mean Privacy Risk Score for test: ", risk_score_test.mean())

Mean Privacy Risk Score for train:  0.5572013708112826
Mean Privacy Risk Score for test:  0.4623020668629128

Now that we have calculated the privacy risk score of the training and testing set, we can then analyze the privacy risk of the target model by varing the threshold of the privacy risk score.

To do this we will calculate the precision and recall of the privacy risk score for different thresholds. We will use the ROC curve function to calculate the false positive rate and true positive rate for different thresholds and then, we will use the calculated values to analyze the privacy risk of the target model at different thresholds.

To create the ROC curve, we will use the roc_curve function from the sklearn.metrics module. This function receives as input the true labels and the predicted probabilities of the positive class. As labels, we know previously if the samples belong to the training or testing set, so we can use this information to create the labels for the ROC curve. On the other hand, the scores are the calculated privacy risk scores.

labels = np.concatenate((np.ones(len(risk_score_train)), np.zeros(len(risk_score_test))))
scores = np.concatenate((risk_score_train, risk_score_test))

# Compute ROC curve
fpr, tpr, thresholds = metrics.roc_curve(labels, scores, drop_intermediate=False)

# Define threshold list and find meaningful thresholds
threshold_list = [1., 0.9, 0.8, 0.7, 0.6, 0.5, 0.4]
max_prob = max(risk_score_train.max(), risk_score_test.max())
meaningful_thresholds = [threshold for threshold in threshold_list if threshold <= max_prob]

# Find indices for meaningful thresholds
indices = [np.argwhere(thresholds >= threshold)[-1][0] for threshold in meaningful_thresholds]

# Calculate precision and recall
precision_list = tpr[indices] / (tpr[indices] + fpr[indices])
recall_list = tpr[indices]

Finally, the previous calculations allow us to analyze the privacy risk of the target model at different thresholds. We can do this by analyzing the precision and recall of the privacy risk score to extract insights about the privacy risk of the model.

for idx, threshold in enumerate(meaningful_thresholds):
    print(f"Threshold: {threshold}, Precision: {precision_list[idx]}, Recall: {recall_list[idx]}")

Threshold: 1.0, Precision: 1.0, Recall: 0.0014285714285714286
Threshold: 0.9, Precision: 0.9192934519839431, Recall: 0.03400921658986175
Threshold: 0.8, Precision: 0.9192934519839431, Recall: 0.03400921658986175
Threshold: 0.7, Precision: 0.9192934519839431, Recall: 0.03400921658986175
Threshold: 0.6, Precision: 0.6012458562151735, Recall: 0.5895852534562211
Threshold: 0.5, Precision: 0.5775898398535104, Recall: 0.7221658986175116
Threshold: 0.4, Precision: 0.5480712814344799, Recall: 0.9387557603686636

With these results, we can say the following:

• If the privacy risk score is set above of 0.7, there are ~3% of the training set members that can be inferred correctly by the adversary with a ~92% of precision.

• If we decrease the privacy risk score to 0.6, there are ~58% of the training set members that can be inferred correctly by the adversary with a ~60% of precision.

• Surpringly, there is a very small percentage (0.1%) of the training set members that can be inferred correctly by the adversary with a ~100% of precision if the privacy risk score is set as 1. This is because the privacy risk score is calculated as the probability of the sample belonging to the training set, so if the privacy risk score is 1, the sample belongs to the training set with a probability of 1.

• Finally, if the privacy risk score is set as 0.5, there are ~72% of the training set members that can be inferred correctly by the adversary with a ~57% of precision.

We can see that for this particular model, we could have serious privacy risks since the adversary can infer a considerable percentage of the training set members with a considerable precision. This is a privacy risk that should be taken into account when deploying the model.

Changing the model architecture

Now that we have analyzed the privacy risk of the model with a Random Forest classifier, we can analyze the privacy risk of the model with a different architecture. In this case, we will change the target model to a Logistic Regression classifier maintaining the same shadow model. We will repeat the same steps as before to analyze the privacy risk of the model with a different architecture.

We will this analysis to observe the impact of the model choice on the privacy risk score.

Let’s train the target model.

# Train the Target Model
target_model = LogisticRegression(random_state=42)
target_model.fit(X_train, y_train)
y_target_pred = target_model.predict(X_test)
print("Target Model Performance:")
print("Accuracy:", accuracy_score(y_test, y_target_pred))

Target Model Performance:
Accuracy: 0.8906336392790003

Although the accuracy of the model decreases, this change is not significant.

Now, let’s analyze the privacy risk of the target model built with a Logistic Regression classifier.

shadow_train_probs = shadow_model.predict_proba(X_shadow_train)
shadow_test_probs = shadow_model.predict_proba(X_test)
target_train_probs = target_model.predict_proba(X_target_train)
target_test_probs = target_model.predict_proba(X_test)

risk_score_train = privacy_risk_score((shadow_train_probs, y_shadow_train), (shadow_test_probs, y_test), (target_train_probs, y_target_train))
print("Mean Privacy Risk Score for train: ", risk_score_train.mean())

risk_score_test = privacy_risk_score((shadow_train_probs, y_shadow_train), (shadow_test_probs, y_test), (target_test_probs, y_test))
print("Mean Privacy Risk Score for test: ", risk_score_test.mean(), risk_score_test.std())

labels = np.concatenate((np.ones(len(risk_score_train)), np.zeros(len(risk_score_test))))
scores = np.concatenate((risk_score_train, risk_score_test))

# Compute ROC curve
fpr, tpr, thresholds = metrics.roc_curve(labels, scores, drop_intermediate=False)

# Define threshold list and find meaningful thresholds
threshold_list = [1., 0.9, 0.8, 0.7, 0.6, 0.5, 0.4]
max_prob = max(risk_score_train.max(), risk_score_test.max())
meaningful_thresholds = [threshold for threshold in threshold_list if threshold <= max_prob]

# Find indices for meaningful thresholds
indices = [np.argwhere(thresholds >= threshold)[-1][0] for threshold in meaningful_thresholds]

# Calculate precision and recall
precision_list = tpr[indices] / (tpr[indices] + fpr[indices])
recall_list = tpr[indices]

for idx, threshold in enumerate(meaningful_thresholds):
    print(f"Threshold: {threshold}, Precision: {precision_list[idx]}, Recall: {recall_list[idx]}")

Mean Privacy Risk Score for train:  0.39248846087198713
Mean Privacy Risk Score for test:  0.390332725228539 0.1548584345529434
Threshold: 1.0, Precision: 0.5151883448597119, Recall: 0.002350230414746544
Threshold: 0.9, Precision: 0.5036339840160396, Recall: 0.007741935483870968
Threshold: 0.8, Precision: 0.5036339840160396, Recall: 0.007741935483870968
Threshold: 0.7, Precision: 0.5036339840160396, Recall: 0.007741935483870968
Threshold: 0.6, Precision: 0.5124566467179488, Recall: 0.02870967741935484
Threshold: 0.5, Precision: 0.4995475163227362, Recall: 0.12230414746543779
Threshold: 0.4, Precision: 0.5014759264484899, Recall: 0.7450691244239631

From these new results, we can observe and extract the following insights:

• First, we could observe that although the target model has changed, the accuracy of the model is still high. But the real change is in the privacy risk score. For this case, we have a mean privacy risk score of 0.4, while for the Random Forest classifier, the mean privacy risk score was 0.55.

• Next, we can see that the privacy risk score is lower for the Logistic Regression classifier than for the Random Forest classifier. This means that the adversary can infer fewer training set members with a lower precision for the Logistic Regression classifier than for the Random Forest classifier.

• Finally, we can observe the same behavior for different thresholds. With an almost 2% of the training set members that can be inferred correctly by the adversary with a ~51% of precision for a privacy risk score of 0.6, while for the Random Forest classifier, this percentage was 58% with a ~60% of precision.

From these results, we can see that the model choice has an impact on the privacy risk score. In this case, the Logistic Regression classifier has a lower privacy risk score than the Random Forest classifier. This is an important insight to consider when deploying a model, as the privacy risk score can vary depending on the model architecture.

In addition, although the privacy risk score can be a useful metric to analyze the privacy risks of a model, it is important to consider other metrics and techniques to ensure the privacy of the data and the model. For example, the implementation of defenses against membership inference attacks, such as differential privacy or adversarial training, can help to mitigate the privacy risks of a model.

Summary

In this notebook, we have analyzed the privacy risk of a classification model using the shadow training technique. We have calculated the privacy risk score for the training and testing datasets and then we have analyzed the privacy risk of the model at different thresholds. We have seen that the model presents a considerable privacy risk, since the adversary can infer a considerable percentage of the training set members with a considerable precision if we choose the target model as a Random Forest classifier. We have also observed that the model choice has an impact on the privacy risk score, as the Logistic Regression classifier has a lower privacy risk score than the Random Forest classifier.

This analysis can help to understand the privacy risks of a model and to take actions to mitigate these risks, such as implementing defenses against membership inference attacks.