Local Privacy Protection for Sensitive Areas in Multiface Images.

The privacy protection for face images aims to prevent attackers from accurately identifying target persons through face recognition. Inspired by goal-driven reasoning (reverse reasoning), this paper designs a goal-driven algorithm of local privacy protection for sensitive areas in multiface images (face areas) under the interactive framework of face recognition algorithm, regional growth, and differential privacy. The designed algorithm, named privacy protection for sensitive areas (PPSA), is realized in the following manner: Firstly, the multitask cascaded convolutional network (MTCNN) was adopted to recognize the region and landmark of each face. If the landmark overlaps a subgraph divided from the original image, the subgraph will be taken as the seed for regional growth in the face area, following the growth criterion of the fusion similarity measurement mechanism (FSMM). Different from single-face privacy protection, multiface privacy protection needs to deal with an unknown number of faces. Thus, the allocation of the privacy budget ε directly affects the operation effect of the PPSA algorithm. In our scheme, the total privacy budget ε is divided into two parts: ε_1 and ε_2. The former is evenly allocated to each seed, according to the estimated number of faces ρ contained in the image, while the latter is allocated to the other areas that may consume the privacy budget through dichotomization. Unlike the Laplacian (LAP) algorithm, the noise error of the PPSA algorithm will not change with the image size, for the privacy protection is limited to the face area. The results show that the PPSA algorithm meets the requirements ε-Differential privacy, and image classification is realized by using different image privacy protection algorithms in different human face databases. The verification results show that the accuracy of the PPSA algorithm is improved by at least 16.1%, the recall rate is improved by at least 2.3%, and F1-score is improved by at least 15.2%.

1. Introduction

In the information age, the importance of data becomes more and more prominent. The value of data is highlighted in various fields in our society. Data-based services are widely applied in different industries. In particular, the application of image data has been developing rapidly. Thanks to the fast development of information technology and multimedia technology, it is now easier to acquire and share digital face images. The users can publish their own photos on social network platforms and other channels. Statistics show that more than 3.2 billion face images are shared by users of major social network platforms around the world. These digital images usually contain a wealth of personally sensitive information. If the information is collected and analyzed by attackers, unmeasurable losses will occur, in addition to the leak of personal privacy.

The privacy protection of image data often relies on techniques like k-anonymity, access control, and privacy encryption. Fung et al. [1] and Xiao and Tao [2] proposed the k-same method based on the anonymization mechanism. The method anonymizes each published gray image, reducing the probability of attackers deriving user identity from the published image to less than 1/k. Li et al. [3] applied access control to restrict user access to social network images. The transfer and visitor volumes of these images are reduced to protect privacy. This approach, which protects images via access settings, is not a fundamental privacy protection method for images. If an attacker has a certain background knowledge, he/she may bypass the access control and acquire the user images and privacy information. To prevent the communication process from being eavesdropped, Terrovitis et al. [4] implemented homomorphic encryption on gray images. However, every data encryption technique makes some assumptions and designs the corresponding encryption algorithm based on these assumptions. However new it is, the encryption algorithm will soon be breached by attackers. Focusing on anonymized images on Facebook, Sweeney, and Anonymity [5] revealed that attackers can deduce the social security number (SSN) of the person in each anonymized image, based on the unique Friendster feature of Facebook, and identify the sensitive information (e.g., disease and address) from the SSN.

In 2006, Dwork [6] proposed differential probability, which disturbs sensitive data by adding noise to the output. Differential privacy can hide the impact of a single record: whether a record is in the dataset or not, the output probability of the same result will not change significantly. In this way, attackers are hampered from further reasoning. Compared with other privacy protection techniques, the differential probability is superior in that it makes no assumption about the background knowledge of attackers. Dwork further investigated differential privacy in a series of theses [7-11] and proposed its implementation mechanism [12, 13]. McSherry [14] pointed out that the differential privacy algorithm for complex privacy problems needs to satisfy two composition properties: sequence combination and parallel combination. In recent years, differential privacy has been mainly applied to the field of data publication. The differential privacy protection for data publication mainly attempts to ensure the accuracy of published data or query results while satisfying differential privacy. The relevant research focuses on adjusting the publication mechanism and algorithm complexity. The primary research method is a quantitative analysis based on calculation and learning theories.

By the realization environment, the differential privacy protection for data publication can be divided into interactive data publication and noninteractive data publication [12]. On interactive data publication, Roth and Roughgarden [15] presented a median algorithm that can respond to more queries. Hardt and Rothblum [16] developed a pulse wave modulation (PMW) mechanism capable of increasing the number of queries. Gupta et al. [17] proposed a universal iterative dataset generation mechanism. Fan and Xiong [18] designed a novel approach with filtering and adaptive sampling for releasing time series under differential privacy (FAST). Kellaris et al. [19] put forward a flow data publication algorithm with no limit on the number of publications. On noninteractive data publication, Xiao et al. [20] came up with the Privelet algorithm. Xu et al. [21] created the noise first and structure first algorithms. Li et al. [22] proposed the matrix mechanism. Li et al. [23] put forward a data- and workload-aware algorithm (DAWA) algorithm. Owing to the complexity of image data, the current research is still in the exploratory stage concerning the differential privacy protection for sensitive information in images.

Images are often represented as matrices in the real number field. Any pixel in an image can be mapped to a value at the corresponding location in a two-dimensional (2D) matrix. To satisfy ε-differential privacy, the most direct approach is to add a Laplace noise to all the values in the matrix. However, the disturbed image may be over distorted and become useless. Fourier transform and wavelet transform are common ways to compress images. Zhang et al. [24] proposed an image compression method based on discrete Fourier transform (DFT), which adds a unique Laplace noise to each target image. Despite suppressing the noise error, their approach introduces the reconstruction error to image compression. Considering the uncorrelation between the values in the image matrix, Liu et al. [25] converted the image gray matrix into a one-dimensional (1D) ordered data flow and modeled the data flow with the sliding window model. To protect the privacy of images, the privacy budget was allocated dynamically based on the similarity between the data of adjacent sliding windows. This highly available strategy is restricted to the 1D space. Liu et al. [26] utilized regional growth to expand the comparison between adjacent subgraphs to the 2D space and further optimized the privacy protection for face images.

The noises of the above methods cover the entire image. For face images, the sensitive information of the face only concentrates in the face area and even in some specific areas. The exposure of other nonsensitive areas will not lead to the leak of privacy. Therefore, this paper proposes a local privacy protection method for the sensitive areas in multiface images.

2. Background

2.1. Differential Privacy

Definition 1 .

Adjacent datasets of the face image

For a given image X, the gray matrix X can be obtained by normalizing the image. Then, there exists , where x in matrix X represents the gray of the corresponding element. If there exists an X′ with only one element difference from X, |X − X′|=x(1 ≤ i ≤ m, 1 ≤ j ≤ n), then X and X′ are adjacent datasets.

Definition 2 .

Differential privacy

For a given random algorithm M of image data publication, with the output range of Range(M), the algorithm can provide ε-differential privacy, if its arbitrary outputs on two adjacent gray images X and X′ satisfy the following:where ε is typically a small positive number that balances privacy with accuracy. If ε is small, the privacy is high and accuracy is low. The inverse is also true. Normally, ε is selected by the user by executing a privacy policy. When the adjacent datasets vary by only one record, the algorithm satisfies ε-differential privacy. When the adjacent datasets vary by k records, the algorithm satisfies kε-differential privacy.

Definition 3 .

Global sensitivity

Let Q be a random query function meeting Q : D⟶R. Then, the global sensitivity of Q can be expressed as follows:

Theorem 1 .

Laplace mechanism

Let Q be a query series of the length d. The random algorithm M receives database D and outputs the following vector that satisfies ε-differential privacy:

As the most common noise addition mechanism, the Laplace mechanism disturbs the real output by adding the noise generated by Laplace distribution, thereby achieving differential privacy. The probability density function (PDF) of its noise distribution satisfies f(x|μ, b)=1/2be|.

Property 1 .

Differential privacy-serial combination property

For a given dataset X and a set of differential privacy algorithms M1(X), M2(X),…, M(X) related to X, if the algorithm M(D) satisfies ε- differential privacy, and the random processes of any two algorithms are independent of each other, then the algorithm combined from these algorithms satisfies ∑ε- differential privacy.

2.2. Multitask Cascaded Convolutional Network (MTCNN)

With the development of CNN [27], Sun et al. [28] suggested that the CNN can be applied to localization face landmarks by virtue of its strong feature extraction ability. A three-layer deep CNN (DCNN) was designed to solve a thorny problem in landmark extraction: the inability to obtain the global optimal solution, due to the improper setting of initial values. Zhang et al. introduced multitask learning to face landmark localization and proposed a task-driven DCNN for face landmark localization (TCDCN). The TCDCN, a multitask learning model of four subtasks, has a smaller time complexity than traditional CNN [29]. Two years later, Zhang et al. developed the MTCNN, which effectively integrates face area detection with face landmark localization. The three cascading neural networks in MTCNN are responsible for face classification, bounding box regression, and key point localization, respectively [30]. Wu et al. [31] presented a tweaked CNN (TCNN), which relies on a mixed Gaussian model to cluster the features on different layers, and concluded that a deeper network layer can more accurately mirror face landmarks. In addition, many other methods have been adopted to locate face landmarks, namely, principal component analysis (PCA) [32], support vector machine [33], Bayesian probabilistic network (BPN) [34], dynamic link architecture (DLA) [35], and Gabor wavelet network (GWN) [36].

The MTCNN is a multitask parallel face recognition method based on deep learning. This method has been widely recognized in the industry because it operates rapidly, and its accuracy meets the general requirements of face detection. The multitask parallel capacity of the MTCNN algorithm manifests as the simultaneous detection of multiple faces (detecting whether an image contains faces and finding the face locations in the image, as shown in Figure 1) and the localization of face landmarks (locating the five landmarks, namely, two eyes, nose tip, and mouth corners, as shown in Figure 2).

Figure 1

Face detection.

Figure 2

Landmark localization.

The MTCNN algorithm can be realized as a three-stage cascaded CNN. The first stage is called a proposal network (P-Net), which mainly obtains the candidate windows for the face area and the regression vector of the bounding box, carries out regression with the bounding box, calibrates the candidate windows, and merges highly overlapped candidate boxes through nonmaximum suppression (NMS). The structure of the P-Net is shown in Figure 3.

Figure 3

Structure of P-net.

The second stage is called a refine network (R-Net), which eliminates the false positive areas through bounding box regression and NMS. That is, lots of nonface windows are denied by a more complex CNN in order to refine the face window. The structure of the R-Net is shown in Figure 4.

Figure 4

Structure of R-net.

The third stage is called an output network (O-Net). With one more convolutional layer than the R-Net, the O-Net outputs more refined results. The functions of the O-Net are the same as those of the R-Net. But the O-Net carries out more supervision of the face area and output five landmarks. The structure of the O-Net is shown in Figure 5.

Figure 5

Structure of O-net.

Each stage of the MTCNN is a multitask network, responsible for handling tasks like face/nonface judgement, face box regression, and landmark localization. The face/nonface judgement adopts the cross-entropy loss function:where p is the probability for the image to contain faces; ydet ∈ {0,1} is the true background label.

The regression loss of face box can be calculated by Euclidean distance:where is the background coordinates predicted by the network; y is the true background coordinates; ybox ∈ ℝ4(ℝ4 is a 4-tuple composed of x of the upper left corner, y of the upper left corner, length, and width).

Landmark localization is calculated similarly as face box regression. The Euclidean distance between the predicted landmark location and the true landmark location is calculated and minimized:where ylandmark ∈ ℝ10 is a 10-tuple, for five landmarks are being localized.

The simplest function of the training process can be expressed as follows:where N is the number of training samples; α is the importance of a task; β ∈ {0,1} is a sample label. In the P-Net and R-Net, αdet = 1, αbox = 0.5, and αlandmark = 0.5. In the O-Net, αdet = 1, αbox = 0.5, and αlandmark = 1.

3. Methodology

3.1. Laplacian (LAP) Algorithm

This paper proposes the LAP algorithm based on the Laplace mechanism. In the LAP algorithm, every element x in the gray matrix X is regarded as an independent individual. This division method is the basis for applying the interactive mechanism to privacy protection. Each x (1 ≤ i ≤ m, 1 ≤ j ≤ n) consumes a privacy budget of ε/(m × n). The LAP algorithm can be realized in the steps in Algorithm 1.

Algorithm 1

LAP.

Compared with the original image X, the privacy protected image X′ contains an additive noise of 2m × n × (ΔQ × m × n/ε)2. Admittedly, the LAP algorithm satisfies ε-differential privacy. However, the noise results show that the LAP algorithm will have a huge error if it is applied to protect the privacy of an excessively large image. If so, the noisy image will be of low availability.

3.2. Fusion Similarity Measurement Mechanism (FSMM)

This paper divides a face image X into multiple nonintersecting subgraphs T(1 ≤ i ≤ I, 1 ≤ j ≤ J), each of which contains multiple pixels:

Through the division, the nonintersecting subgraphs T carry as much information of the original image as possible.

The key to regional growth is to determine the criterion. Due to the natural complexity of face images, the criterion should be designed by judging the various features of the original image. The traditional criterion for regional growth only focuses on the difference between gray values. The gray value of a single pixel cannot provide rich information about the image. By contrast, subgraphs as the basic units of regional growth can retain the luminance, contrast, structure, color, texture, and spatial distribution features of the image. This paper presents a new regional growth criterion called FSMM, which further improves the accuracy of area merging with subgraphs as the basic units. The FSMM can be realized in the steps in Algorithm 2.

Algorithm 2

FSMM.

As its name suggests, the new regional growth criterion relies on the FSMM, a brand-new similarity measure between images. In rws 3-4 of the algorithm, parameter σ, a very small positive number, appears to ensure the denominator is nonzero. In rows 7–9, u and u are, respectively, the mean values of images X and Y, reflecting the luminance features of the images; σ and σ are, respectively, the variances of images X and Y, reflecting the contrast features; C1, C2 and C3 are very small positive numbers that ensure the denominator is nonzero. In Column 10, α, β,  and γ are parameters adjusting the proportion of different eigenvalues: if α=β=γ=1,  then C1=(K1L)2, C2=(K2L)2,  and C3=C2/2,  with  K1 ≪ 1,  K2 ≪ 1,  and L being the dynamic range of the image. The final calculation method for FSMM(X, Y) can be expressed as follows:

3.3. Privacy Protection for Sensitive Areas (PPSA) Algorithm

To achieve privacy protection of sensitive information (face area) in multiface images and reduce noise impact on the privacy protected images, this paper proposes a local privacy protection method for face images, which combines face recognition, regional growth, and differential privacy. The proposed algorithm is called the PPSA.

Inspired by goal-driven reasoning (reverse reasoning), the PPSA was designed with goal as the starting point. The privacy protection for face images aims to prevent attackers from recognizing the target persons through face detection. Therefore, the first step of PPSA design is to understand how attackers identify faces. Next, a privacy protection method should be provided for face images, targeting the face identification technique of attackers. In this way, the designed privacy protection algorithm can effectively curb privacy leak and mitigate the effects of noise on the availability of the privacy protected image.

The design of the PPSA needs to solve three key problems: (1) locate the sensitive area in the multiface image and recognize the landmarks in the area; (2) formulate the criterion for regional growth; (3) allocate the privacy budget reasonably to areas requiring different levels of protection.

The PPSA algorithm is designed to protect the sensitive information (face information) in multiface images in batches. Different from single-face privacy protection, multiface privacy protection needs to deal with an unknown number of faces. Thus, the allocation of the privacy budget is crucial to PPSA design. The privacy protection should be tailored to areas with different protection requirements. In our scheme, the total privacy budget ε is divided into two parts: ε1 and ε2. The ε1 is evenly allocated to each seed Seed (any subgraph containing a landmark is a seed), according to the estimated number of faces ρ contained in the image. During regional growth, Seed′ (noisy seed) replaces the T( (adjacent subgraph) meeting the growth criterion, thereby protecting the privacy of T(. Note that the privacy protection of T( does not consume any privacy budget. Meanwhile, the ε2 is allocated through dichotomization. This part of the privacy budget is consumed by the Seed outside the estimated number of faces ρ, the T( failing to meet the growth criterion, and the belonging to the sensitive area but not involved in regional growth. The PPSA algorithm can be realized in the steps in Algorithm 3.

Algorithm 3

PPSA.

The image preprocessing, privacy budget allocation, and regional growth are described in rows 1–6, rows 7–9, and rows 10–29, respectively. In row 30, is the addition of Laplace noise to an unprotected subgraph in S′. In row 18, the PPSA limits the range of regional growth: if a T( meeting the growth criterion but lying beyond the coverage of the current S′ or the growth range of the current Seed, it will not be able to complete this area merging; the growth range of Seed will be limited to half of the Euclidean distance to the nearest adjacent Seed. If ω ≤ ρ, only rows 10–27 will be executed for regional growth. If ω > ρ, all the rows from 10–29 will be executed for regional growth.

Theorem 2 .

The PPSA consumes a privacy budget smaller than ε and satisfies ε-differential privacy.

Proof

In the PPSA, the total privacy budget ε is divided into two parts. Among them, ε1 is used to add noise to seeds. If ω < ρ, there exists ε1left > 0 after the end of the algorithm; if ω ≥ ρ, there exists ε1left=0 after the end of the algorithm. Meanwhile, ε2 is allocated by dichotomy:

Thus, there exists the following:

Therefore, the PPSA consumes a privacy budget smaller than ε. According to Property 1, the PPSA satisfies ε-differential privacy.

Q.E.D.

Theorem 3 .

The error of the PPSA is no greater than that of the LAP, i.e.,

In the PPSA, the privacy protected image contains four kinds of subgraphs, namely, noise-free subgraph P(, seed Seedn, subgraph merged in regional growth T(, and subgraph belonging to a sensitive area but not involved in regional growth .

For P(:

Subtracting the two formulas above, we have the following: Error(PPSA(P()) < Error(LAP(P()).

For :

If 2 ≤ I × J, there exists Error(PPSA(P()) < Error(LAP(P()).

If 2 > I × J, there exists Error(PPSA(P()) ≈ Error(LAP(P()).

The validity of Error(PPSA(P()) ≈ Error(LAP(P()) depends on the uniqueness of the gray image. In the gray image matrix, the gray value falls between 0 and 255. Under the Laplace mechanism, the smaller the ε, the greater the noise. For any other type of data, the noise effect will be huge, if ε is sufficiently small. For gray value, however, the fluctuation induced by noise is limited. Take for example. When lap(ΔQ × 2/ε2)⟶+∞ 时, value in the gray matrix will become 255. When lap(ΔQ × 2/ε2)⟶−∞, that value will change to 0. Thus, Error(PPSA(P()) ≈ Error(LAP(P()) holds, as long as I × J is sufficiently great.

For Seed:

There are two possible scenarios: ω ≤ ρ and ω > ρ.

When ω ≤ ρ, Seed′ only consumes ε1. Then, there exists the following:

If 5 × ρ ≪ I × J, the numerical gap between ε1 and ε is so small as to be negligible. Subtracting the above two formulas, we have the following: Error(PPSA(Seed)) < Error(LAP(Seed)).

When ω > ρ, the first ρ face areas consume ε1 until no budget in this part is left. In this case, there exists Error(PPSA(Seed)) < Error(LAP(Seed)). For the lack of space, the proof is omitted.

In the following ω − ρ face images, Seed′ needs to consume ε2. According to the proof of , there exists Error(PPSA(Seed)) ≈ Error(LAP(Seed)) during the consumption of ε2. Thus, it can be deduced that, in any scenario, there exists the following:

For T(:

There are also two possible scenarios: ω ≤ ρ and ω > ρ. When ω ≤ ρ, the proof is as follows:

Subtracting the above two formulas, we have the following:

For a gray image, when the privacy budget is sufficiently small under the Laplace mechanism, Error(T(′) is equivalent to Error(Seed′). Thus, we have the following:

Thus, Error(PPSA(T()) < Error(LAP(T()).

Similarly, when ω > ρ, there exists Error(PPSA(T()) ≈ Error(LAP(T()). For the lack of space, the proof is omitted.

Q.E.D.

4. Experiments and Results Analysis

4.1. Experiments

To demonstrate its feasibility, the PPSA was tested on a 750 ∗ 1020 image containing 8 faces from the WIDER FACE dataset. The original image was divided into 7,650 subgraphs of the size 10 ∗ 10. During the implementation of the PPSA, the original image (Figure 6) was first transformed into a gray image (Figure 7). Then, the MTCNN algorithm was called to recognize the face area S and landmark K (Figure 8). To meet the requirements of the PPSA on regional growth, the face image was divided into multiple equal size T( (Figure 9), using the preset parameters. The subgraph area (Figure 10) overlapping the landmark location was taken as the Seed (Figure 11) of regional growth. However, the S detected by the MTCNN is not compatible with the size fT( (Figure 12). To solve the problem, S was resized to S′ (Figure 13), according to the size of T(.

Figure 6

Original image (RGB).

Figure 7

Gray image.

Figure 8

S and K.

Figure 9

Rendering of T(i, j).

Figure 10

Rendering of K and T(i, j).

Figure 11

Rendering of Seed.

Figure 12

Rendering of Seed.

Figure 13

Rendering of Seed and S′.

Figure 14 provides the results of the LPA. Figure 15 presents the PPSA results under the same privacy budget. Observations show that the PPSA operates within the face area while the LAP acts across the image. Besides, within the scope of the regional growth algorithm, the PPSA retained the recognizability of landmarks. Although not excluding the possibility of privacy leaks, this strikes a balance between the privacy and availability of the privacy protected multiface image. In addition, the relationship between the number of faces ρ estimated by the PPSA and the true number of faces ω determines how much privacy budget can be obtained by a seed. The allocation of the privacy budget will directly affect the error of the protected image. Figures 16 and 17 present the local results at different relationships between ρ and ω.

Figure 14

LAP results.

Figure 15

PPSA results.

Figure 16

Local results at ω ≤ ρ.

Figure 17

Local results at ω > ρ.

4.2. Results Analysis

To verify the feasibility of our algorithm, multiple face images were collected from WIDER FACE dataset, i-bug face dataset, and AFW dataset and subjected to experiments using TensorFlow + AlexNet CNN. The experiments were carried out in the environment of Intel® Core i9-9900K CPU @ 3.60 GHz, 32G memory, GTX 21080TI GPU, and Windows 10. The privacy budget was set to 1, 2, 3, 4, and 5, in turn. The face recognition performance was measured by precision, recall, and F1-score. The experimental results are displayed in Figures 18–26.

Figure 18

WIDER and precision.

Figure 19

WIDER and recall.

Figure 20

IDER and F1-score.

Figure 21

I-bug and precision.

Figure 22

I-bug and recall.

Figure 23

I-bug and F1-score.

Figure 24

AFW and precision.

Figure 25

AFW and recall.

Figure 26

AFW and F1-score.

The experimental results show that the PPSA achieved different results on images from different datasets. The difference arises from the varied image sizes of different datasets: 760 ∗ 1000 in WIDER, 500 ∗ 800 in i-bug, and 1000 ∗ 1500 in AFW. In addition, the different background complexities between the face datasets also influence the operation effects of LAP, sliding window publication (SWP) algorithm, and region growing publication (RGP). By contrast, the operation of the PPSA is not affected by image size and background complexity.

From the experimental results, it can be learned that the operation results of the PPSA mainly depend on the number of estimated faces ρ and S′. In the images from i-bug, there are relatively few faces in each image, but the face area takes up a large portion of the entire image. In the images from WIDER, there are many faces in each image, and the face area takes up a large portion of the entire image. In the images from the AFW, there are relatively few faces in each image, but the face area takes up a small portion of the entire image, for the images tend to be large. Therefore, it is expected that the PPSA's privacy protection effect is positively correlated with ω, and negatively with S′. The prediction is consistent with the experimental results.

5. Conclusions

To protect the privacy of multiface images, this paper combines face detection, regional growth, with the Laplace mechanism of differential privacy to add noises to local sensitive areas in face images and realizes the privacy protection for the local sensitive areas in multiface images under an interactive framework. Compared with the LAP, SWP, and RGP, the proposed PPSA can effectively suppress the noise impact on the protected image and improve the availability of the privacy protected image. Moreover, the PPSA applies to images of various sizes, and its error does not increase with the image size.

Although the PPSA can effectively protect the face information in multiface images, the attackers may choose to attack the hair, clothes, and body of persons in the images. Besides, privacy leak may arise from the relationship between persons, and the correlation between each person and the background. The future work will further improve image privacy protection, trying to solve these potential risks.