Logistic regression over encrypted data from fully homomorphic encryption.

BACKGROUND: One of the tasks in the 2017 iDASH secure genome analysis competition was to enable training of logistic regression models over encrypted genomic data. More precisely, given a list of approximately 1500 patient records, each with 18 binary features containing information on specific mutations, the idea was for the data holder to encrypt the records using homomorphic encryption, and send them to an untrusted cloud for storage. The cloud could then homomorphically apply a training algorithm on the encrypted data to obtain an encrypted logistic regression model, which can be sent to the data holder for decryption. In this way, the data holder could successfully outsource the training process without revealing either her sensitive data, or the trained model, to the cloud.
METHODS: Our solution to this problem has several novelties: we use a multi-bit plaintext space in fully homomorphic encryption together with fixed point number encoding; we combine bootstrapping in fully homomorphic encryption with a scaling operation in fixed point arithmetic; we use a minimax polynomial approximation to the sigmoid function and the 1-bit gradient descent method to reduce the plaintext growth in the training process.
RESULTS: Our algorithm for training over encrypted data takes 0.4-3.2 hours per iteration of gradient descent.
CONCLUSIONS: We demonstrate the feasibility but high computational cost of training over encrypted data. On the other hand, our method can guarantee the highest level of data privacy in critical applications.

Background

Since 2014, iDASH (integrating Data for Analysis, Anonymization, and Sharing) has hosted yearly international contests around the theme of genomic and biomedical privacy. Teams from around the world participate to test the limits of secure computation on genomic and biomedical tasks, and benchmark solutions on real data sets. Such contests serve to bring together experts in security, cryptography, and bioinformatics to quickly make progress on interdisciplinary challenges. The task for outsourced storage and computation this year was to implement a method for private outsourced training of a logistic regression model.

Motivation

Machine Learning (ML) over encrypted data has important applications for cloud security and privacy. It allows sensitive data, such as genomic and health data, to be stored in the cloud in encrypted form without losing the utility of the data. For the third task in the 2017 iDASH Secure Genome Analysis Competition, participants were challenged to train a machine learning model on encrypted genomic data that will predict disease based on a patient’s genome. In a non-interactive (with outsourced storage) setting, training ML models on encrypted data had up until now only been done for very simple models, such as Linear Means Classifiers and Fisher’s Linear Discriminant Analysis [1]. Interactive settings, where multiple parties hold shares of the data and communicate throughout the training process, have been developed for several more complicated models, but they require high communication costs and a non-colluding assumption between several clouds [2].

The 2017 iDASH competition task was to train a logistic regression model, and although in theory it can be done using Fully Homomorphic Encryption (FHE) [3, 4], until now the feasibility and efficiency of this approach had not been studied.

Summary of results

In this work, we show that training a logistic regression model over binary data is possible using FHE. In particular, we use gradient descent and stochastic gradient descent algorithms with mini-batches, and demonstrate that it takes several minutes to one hour to run each gradient descent step. Our solution can run for an arbitrary number of steps, as opposed to the now commonly used practical homomorphic encryption (PHE) approach [5], where the size of the computation is determined beforehand, and parameters chosen once and for all to support a computation of that size. This is possible using Craig Gentry’s bootstrapping operation [4], which we have implemented for the first time for the Fan-Vercauteren scheme [6] using the publicly available homomorphic encryption library SEAL (http://sealcrypto.org; accessed on 9 April, 2018).

More precisely, in fully homomorphic encryption each ciphertext contains a component called the noise, which grows in all homomorphic operations, and eventually reaches a maximum value. Once this maximum is reached, the ciphertext cannot be decrypted correctly anymore. Bootstrapping is the process of “refreshing” FHE ciphertexts to reduce the noise levels during deep computations to ensure correct decryption at the end of the computation.

Another challenge in the approach we take is the plaintext data type supported by the homomorphic encryption scheme. Namely, it is only possible to encrypt fairly small integers with SEAL, and indeed with many homomorphic encryption schemes. In machine learning the model weights are typically rational numbers, which need to be scaled to integers. Unfortunately, this quickly causes an overflow to occur in our rather small integer data type, unless the integers can be scaled down. We describe a modified bootstrapping operation which merges bootstrapping and such a scaling into one step, significantly reducing the complexity of our algorithm.

Besides noise growth and message expansion, another challenge in implementing Logistic Regression with FHE is applying the sigmoid function. We present two methods to approximate this function with a polynomial, and compare them both in terms of the accuracy of the trained model and in terms of computation time.

Related work

At the time of writing this, very little directly comparable prior work exists. The closest to our approach is [7], where the authors achieve remarkably good performance in training small logistic regression models; in their solution it is necessary that the number of features is very small (logarithmic in the number of training records).

A slightly different approach is taken in [8], where the authors use the homomorphic encryption library HEAAN, that natively supports scaling down of plaintext numbers [9, 10]. The authors report good performance numbers, but unlike us and [7] they only allow a very small number of iterations. Extending to more iterations will be computationally very costly, and require bootstrapping.

Methods

Fan-Vercauteren scheme

Fully Homomorphic Encryption (FHE) refers to a type of encryption scheme, envisioned already a few decades ago [3], that allows arbitrary computations to be performed directly on encrypted data. A blueprint for a solution was first proposed by Gentry [4] in 2009, and since then numerous schemes have been proposed. In this work we use the Fan-Vercauteren scheme (FV) [6], and its implementation in the SEAL library [11].

Parameters and notation.

We start by defining the parameters of the FV scheme. Let q≫t be positive integers and n a power of 2; often t is a prime such that 2n∣(t−1). Denote Δ=⌊q/t⌋. We define , , and . Here, is the set of polynomials with integer coefficient and is the set of polynomials with integer coefficient in range [0,q−1). Therefore, R is the set of polynomials of degree at most n−1, with coefficients integers modulo q. Multiplications of polynomials in R is similar to usual polynomial multiplication, except that x should in every step be replaced by − 1. In the FV scheme plaintext elements are polynomials in R, and ciphertext elements are pairs of polynomials in R×R. Let χ denote a narrow (centered) discrete Gaussian error distribution. In practice, most implementations of homomorphic encryption use σ[χ]≈3.2. Finally, let U denote the uniform distribution on .

Key generation

The first step in using the FV scheme is generating a public-secret key pair (pk,sk). To do this, sample , , and e←χ; here s, a, and e are all considered as elements of R, where the n coefficients are sampled independently from the given distributions. To form the keys, we let where [·] denotes the (coefficient-wise) reduction modulo q. In reality there are other types of keys involved, in particular so-called evaluation keys and Galois keys, but for the sake of simplicity we will omit discussing them here, and refer the reader to [6, 11].

Encryption.

Let m∈R be a plaintext message. To encrypt m with the public key , sample , , and e1,e2←χ. Consider u, a, and e as elements of R as in key generation, and create the ciphertext

Decryption.

To decrypt a ciphertext ct=(c0,c1) given a secret key sk=s, write where c0+c1s is computed as an integer coefficient polynomial, and scaled by the rational number t/q, b is an integer coefficient polynomial, the underlying message, and v the leftover fractional part.

It is easy to see that when q is sufficiently larger than t, then , and ∥v∥≪1/2. This means that the original message can be recovered by computing where ⌊·⌉ denotes rounding to the nearest integer. For details, see [6, 11].

Homomorphic computations

A final fundamental piece in the puzzle is how to enable additions and multiplications of two ciphertexts. For addition, this is easy; we define an operation ⊕ between two ciphertexts ct1=(c0,c1) and ct2=(d0,d1) as follows: We denote this homomorphic sum by , and note that if then as long as ∥v1+v2∥<1/2. Thus, ⊕ passes through the encryption to the underlying plaintexts, and results in an encryption of the sum [m1+m2] as long as ∥v1+v2∥<1/2.

It is similarly possible to define an operation ⊗ between two ciphertexts, that results in a ciphertext decrypting to [m1m2], as long as ∥v1∥ and ∥v2∥ are small enough. Since ⊗ is much more difficult to describe than ⊕, we refer the reader to [6, 11] for details.

Noise

In the decryption formulas presented above the rational coefficient polynomials v are assumed to have small enough infinity-norm, namely less than 1/2. This is clearly necessary, as otherwise the ciphertext will result in the incorrect plaintext being recovered. Given a ciphertext ct=(c0,c1) encrypting a plaintext m, let such that The polynomial v is called the noise polynomial, ∥v∥ is called the noise, and the ciphertext decrypts correctly as long as the noise is less than 1/2 [11].

When operations such as addition and multiplication are applied to encrypted data, the noise in the result may be larger than the noise in the inputs; this is referred to as noise growth. This noise growth is very small in homomorphic additions, but substantially larger in homomorphic multiplications. Thus, given a specific set of encryption parameters (n,q,t,χ), one can only evaluate computations of a bounded size (in practice, of bounded multiplicative depth), until the noise grows too large making the ciphertext impossible to decrypt even with the correct secret key.

To mitigate the problem of high noise growth rates Craig Gentry [4] described a clever approach which is commonly known as bootstrapping. In this process, an encrypted version of the secret key is used to decrypt the message using homomorphic operations. Therefore, the result of this process is similar to a freshly encrypted message and hence it has only a small amount of noise. This bootstrapping process is considered to be a very costly operation in most schemes [10, 12, 13], but not in all [14, 15].

Batching

The FV scheme (and many other homomorphic encryption schemes) inherently support SIMD operations. This capability is commonly called “batching” in literature, and is explained in detail e.g. in [11] in the context of the SEAL library that we use.

The idea is that by choosing the plaintext modulus t appropriately, the plaintext space R is isomorphic as a ring to the k-fold product , for some k∣n. In other words, operations in R translate automatically into k concurrent operations in the extension field , for example allowing us to perform k-fold SIMD operations on integers up to t by using only the subfield .

Using batching efficiently can be non-trivial, and typically requires one to carefully design the computation to maximize the benefit.

Logistic regression

Logistic Regression is a common tool used in machine learning to build a model that can discriminate between samples from two or more classes. It arises from the need to model the posterior probabilities of K classes via linear functions of input . In this work we consider two-class classification, so K=2. To simplify the notation, we assume the input vector x always has 1 as the first element, which accounts for the bias term in the linear function. Then the logistic regression model has the form where Y denotes the class, and is the weight vector that we need to learn in model training. The above model is specified in terms of log-odds ratio, reflecting the constraint that the probabilities sum to one. An alternative and more common form is to represent it as the following posterior probability for class 0: where σ(t)=1/(1+e−) is known as the sigmoid function. Next we present two algorithms for learning w.

Training algorithms

Our goal is to evaluate a training algorithm for a logistic regression model on homomorphically encrypted data. In this section we present the two training algorithms that we evaluated for this purpose.

Gradient descent

The standard method for training logistic regression is gradient descent. To fix notation, let D be the number of (binary) features, and N the number of training records of the form (X,y), where , . In this case the weight vector w is in .

Gradient descent proceeds in iterations, where in each iteration the weight vector w is updated as where σ is the sigmoid function, and α>0 a learning rate parameter. We formalize the gradient descent algorithm below.

1-bit gradient descent

A direct application of Algorithm 1 suffers from the problem of quickly growing plaintext size—a problem which was briefly mentioned in “Summary of results”. Namely, the parameter t in the homomorphic encryption scheme is typically quite small, causing integer plaintext data to quickly become reduced modulo t. This is similar to the problem using a too small data type in normal programming, except that in this case it is difficult to switch to a larger one. For this reason, we need to be able to control the growth of our encrypted numbers either by scaling them down, and/or by designing our computation in a way that minimizes the increase in the size of the numbers.

For the first approach, we need a homomorphic floor function, which we discuss in “Fixed point arithmetic over plaintext data”. For the second approach, we note that multiplying by just a sign never increases the size of a number, so replacing one multiplicand by its sign allows the plaintext size to remain much smaller. Unfortunately, homomorphic sign extraction is very difficult, but turns out to be still faster than the homomorphic floor function. For this reason, we opt to use sign information instead of evaluating floor function to make our homomorphic training faster. By using the 1-Bit Gradient Descent (1-Bit GD) algorithm, which was invented to compress the gradient in order to reduce communication during training [16], our homomorphic training becomes much faster.

In the 1-Bit GD method, in each iteration we update each weight by a learning rate multiplied by the sign of the corresponding coordinate of the current gradient, plus a residue term. The unused part of the gradient is then added back into the residue. We also introduce a new parameter β, which reduces the magnitude of the accumulated residues in the past. Our modified 1-Bit GD is presented formally in Algorithm 2.

The 1-Bit GD approach can be done easily also in the stochastic setting, where either individual records or mini-batches are processed at a time. In this work, for the sake of simplicity, we will only focus on full gradient descent.

Fixed point arithmetic

Fixed point arithmetic over plaintext data

Logistic Regression is naturally performed over floating point numbers. However, in the FV scheme there is no easy way to encrypt numbers of this type directly, so they need to be first scaled to integers of some fixed precision.

In fixed point number representation we choose an integer base p (in this work we will fix p to be an odd prime), the number of integral digits l, and the number of fractional digits f. Then a fixed point number is a rational number x of the form with . That is, every fixed point number has l integral digits and f fractional digits in base p. We need f extra digits to hold an intermediate result from multiplication, hence we let r=l+2f and set the modulus to be p (see also [17]). To encode a number, we multiply by p and round to an integer, i.e. the representation of x is . See

To add/subtract two fixed numbers, we simply add/subtract their representations modulo p. To multiply two fixed point numbers x and y, we compute

Note that although standard fixed point arithmetic requires us to perform scaling after every multiplication, it is not strictly needed. For example, if we are going to compute , then it is possible to not scale after each product, but only scale after the sum. This may not save a lot of work over plaintext, since scaling is fast; however, since scaling is expensive over encrypted data, this technique is useful in our setting.

Bootstrapping

Even for relatively small examples, Algorithm 1 and Algorithm 2 result in (multiplicatively) high-depth arithmetic circuits; the depth is equal to the number of iterations times the depth of a single iterative step. Recalling the noise growth problem discussed above in “Noise”, a straightforward implementation will have to use bootstrapping regularly to maintain the correctness of the final result. Since bootstrapping is a costly operation, we introduce below in “Combining bootstrapping with scaling” a modification to this step that does both the noise cleaning and also scaling, which is used to prevent plaintext size expansion.

We modified the bootstrapping algorithm from [13], where the crucial part of the bootstrapping procedure is a homomorphic digit removal process. Namely, suppose the plaintext modulus of our homomorphic encryption scheme is a prime power t=p, and the plaintext is (for simplicity) just an integer . Then as an intermediate result in bootstrapping we have an encryption of M=pm+v, where e>r, p is an intermediate plaintext modulus, and |v|