Characterizing large-scale quantum computers via cycle benchmarking.

Quantum computers promise to solve certain problems more efficiently than their digital counterparts. A major challenge towards practically useful quantum computing is characterizing and reducing the various errors that accumulate during an algorithm running on large-scale processors. Current characterization techniques are unable to adequately account for the exponentially large set of potential errors, including cross-talk and other correlated noise sources. Here we develop cycle benchmarking, a rigorous and practically scalable protocol for characterizing local and global errors across multi-qubit quantum processors. We experimentally demonstrate its practicality by quantifying such errors in non-entangling and entangling operations on an ion-trap quantum computer with up to 10 qubits, and total process fidelities for multi-qubit entangling gates ranging from [Formula: see text] for 2 qubits to [Formula: see text] for 10 qubits. Furthermore, cycle benchmarking data validates that the error rate per single-qubit gate and per two-qubit coupling does not increase with increasing system size.

Introduction

Practical methods to characterize quantum processes acting on large-scale quantum systems are required to assess current devices and steer the development of future, more powerful devices. In principle, quantum processes can be fully characterized using, for example, quantum process tomography[1] or gate set tomography[2-4]. However, any protocol for fully characterizing a quantum process requires a number of experiments and digital post-processing resources that grows exponentially with the number of qubits, even with improvements such as compressed sensing[5,6]. As a result, the largest quantum processes that have been fully characterized to date acted only on three qubits[7].

The exponential resources required for a full characterization can be circumvented by extracting partial information about quantum processes. A partial characterization typically yields some figure of merit comparing the noisy implementation of a quantum process to the desired operation. We will consider the process fidelity (also known as the entanglement fidelity), which is equivalent to the average gate fidelity up to a dimensional factor that is approximately 1[8,9].

The process fidelity can be efficiently estimated by randomized benchmarking[10-12] or direct fidelity estimation[13-15]. Direct fidelity estimation can be efficient and hence has been implemented for up to 7 qubits[16] but conflates state preparation and measurement (SPAM) errors with the process fidelity, limiting its value for realistic systems. SPAM errors increase with the system size and so robustness to SPAM is increasingly important for many qubits. Randomized benchmarking decouples the SPAM errors from gate operation errors by applying multiple random elements of the -qubit Clifford group[11,12]. However, implementing each Clifford operation requires primitive two-qubit operations[17], so that randomized benchmarking provides very coarse information about the primitive operations. Furthermore, for error rates as low as per two-qubit operation, a single 10-qubit Clifford operation will have a cumulative error rate on the order of , which substantially increases the number of measurements required to accurately estimate the process fidelity.

Owing to these practical limitations, randomized benchmarking has only been applied on operations involving three or less qubits[18]. While randomized benchmarking can be performed on small subsets of the qubit register[19], such experiments do not explore the full Hilbert space and therefore will not detect important performance-limiting error mechanisms such as cross-talk. Moreover, errors in operations must be characterized in the context in which they are used because control sequences for a specific gate are often distorted by other gates performed in parallel. One method to achieve this is to only perform gates in fixed modes of parallel operation. We refer to a parallel set of gates as a cycle, in analogy with a digital clock cycle. In typical architectures, there are two types of cycles, namely, cycles of single-qubit gates and cycles of multi-qubit gates. Undetected calibration and cross-talk errors will typically lead to coherent and spatially correlated errors that can lead to substantially larger algorithmic errors and can require higher overheads in fault-tolerant quantum error correction schemes[20]. Such errors can be converted to stochastic Pauli errors by randomizing the cycles of single-qubit gates in such a way that the overall ideal circuit remains unchanged, a technique known as randomized compiling (RC)[21]. The error rate due to the resulting stochastic Pauli errors can then be accurately quantified by the process fidelity.

In this paper, we introduce cycle benchmarking (CB), a protocol for estimating the process fidelity of a global noise process affecting a quantum device that occur when a cycle of operations is applied to a quantum register. Under the assumption of Markovian noise such that the noise on each cycle of independent single-qubit gates is independent of the specific gates being implemented (see Supplementary Note 1), we prove that CB is robust to SPAM errors and that the number of measurements required to estimate the process fidelity to a fixed precision is approximately independent of the number of qubits. We demonstrate the practicality of CB for many-qubit systems by using it to experimentally estimate the process fidelity of both non-entangling Pauli operations and the multi-qubit entangling Mølmer–Sørensen (MS) gate[22,23] acting on up to ten qubits. We also confirm that the protocol and analysis methods, derived under theoretical assumptions, produce consistent results in our experimental system.

Results

The CB protocol

We now outline how the CB protocol can quantify the effect of global and local error mechanisms affecting different primitive cycle operations of interest.

Mathematically, the ideal operation of interest is described by the corresponding unitary matrix . Its action is expressed by a map that acts on the state of the quantum register, described by the density matrix . We denote the map of an ideal operation by capital calligraphic letters, such as , and their noisy experimental implementations will be indicated by an overset tilde, such as . We denote the composition of gates by the natural matrix operations for the map representation, so, e.g., means first apply then apply , and means apply a total of times. A particularly important class of processes are Pauli cycles , where the unitary matrix of the process is the -qubit Pauli matrix .

We evaluate the quality of a noisy process by its process fidelity to the ideal target , which can be written as[13]where

Each quantity can be experimentally estimated by preparing an eigenstate of , applying the noisy gate , and then measuring the expectation value of the ideal outcome . The process fidelity may be estimated by averaging over a set of Pauli matrices. However, a sampling protocol (as in direct fidelity estimation[13,14]) for estimating these individual terms is not robust to SPAM errors. Robustness to SPAM is particularly important because SPAM errors can dominate the gate errors.

Inspired by randomized benchmarking[10], SPAM errors can be decoupled from the process fidelity by applying the noisy operation of interest a total of times and extracting the process fidelity from the decay of as a function of the sequence length . Extracting a meaningful error per application of the gate of interest is nontrivial for generic noise channels[24]. However, decay rates can be extracted straightforwardly for Pauli noise channels, that is, classical mixtures of Pauli operations that are applied to the register randomly with given probability. Mathematically, a Pauli noise channel is a mapfor some probability distribution . Such channels cannot exactly describe, for example, small over-rotation errors or amplitude damping channels.

Since the noise in our system is generic, we want to engineer the noise such that it can be described well by a Pauli noise channel. It has been shown that this can be accomplished by introducing a random Pauli cycle at each time step between each application of the cycle of interest [25,26]. This additional random Pauli cycle comes with an additional overhead that will increase the number required gates to implement a given algorithm. RC has been developed to eliminate this overhead[21]. The resulting noise channel when using RC is then associated with the composition of with a random Pauli cycle , called a dressed cycle , which is an important characterization primitive for any algorithm implemented via RC[21]. Therefore CB estimates the average of the process fidelities of the dressed cycle In addition to the dressed cycle fidelity, the process fidelity of the noisy gate alone is of interest. The process fidelity of a specific gate may be estimated by taking the ratio of the estimates obtained for and the identity process , in analogy to interleaved benchmarking[27]. It should be noted that this method of estimating the fidelity of the noise on alone is generally subject to a large systematic uncertainty[28], so the CB method is most precise in the important context of characterizing errors on dressed cycles[21].

CB can be used to efficiently characterize non-Clifford gates by selecting random gates and correction operators using RC[21]. However, the general protocol for non-Clifford gates is more complex, so a simplified version for characterizing the errors occurring under a fixed cycle of Clifford gates composed with a random Pauli cycle is as follows (the protocol is illustrated in Fig. 1, where we explain the motivation for each step further below):

Fig. 1

Schematic circuit implementation of the experimental cycle benchmarking protocol. The protocol can be subdivided into three parts, depicted by the different colors. The green gates describe basis changing operations for the state preparation and the measurement (SPAM) procedure. The red gates are the noisy implementations of some gate of interest (in this work, the global Mølmer–Sørensen gate acting on all qubits). The blue gates are random Pauli cycles that are introduced to create an effective Pauli channel per application of the gate of interest, where denotes the tensor factor of the gate. Creating an effective Pauli channel per application enables errors to be systematically amplified under -fold iterations for more precise and SPAM-free estimation of the errors in the interleaved red gates . The blue and the red gates together form the random circuit . The sequence of local operations before the first and last rounds of random Pauli cycles are identified as conceptually distinct but were compiled into the initial and final round of local gates in the experiment. The experimental parameters , and of this work and the exact definitions of and are given in Supplementary Note 7.

Select a set of -qubit Pauli matrices with elements.

Select two lengths and such that the multiple application of composes to the identity .

Perform the following sequence for each Pauli matrix , length , and , where describes the number of random sequences per Pauli.

Select random -qubit Pauli cycles , and define the randomized circuitas illustrated in Fig. 1.

Calculate the expected outcome of the sequence assuming ideal gate implementations.

Main experiment: Implement and estimate the overlapbetween the expected outcome and the noisy implementation for some initial state that is a -eigenstate of . State preparation and measurement are realized by applying the operations and that are described in Supplementary  Note 2.

Estimate the composite process fidelity via

Step 1 ensures that the action of the -qubit process is accurately estimated. In Supplementary Note 5, we prove that the uncertainty of the fidelity estimate is independent of the number of qubits , and the number of Pauli matrices that need to be sampled depends only on the desired precision. This highlights the scalability of the protocol for large quantum processors.

Step 2 ensures that the measurement procedures for circuits in Eq. (8) with two different values of are the same. Having the same measurement procedures for the two values of is crucial to decouple the SPAM errors from the decay in the process fidelity via the ratio in Eq. (7). In our experiment, we always choose and to be an integer multiple of 4, as, for the considered gates, applying the operation four times subsequently yields the identity process .

In step 3a, we choose random Pauli cycles to engineer an effective Pauli noise process across the randomizations. This enables us to extract a process fidelity from the decay of with the sequence length . Note that unlike typical randomized benchmarking protocols, the above protocol does not have an inversion gate. Formally, the final random Pauli can be regarded as a correction gate for the random Pauli gates in the rest of the circuit composed with another random Pauli that we use to isolate exponential decays as in character benchmarking[29].

In step 3b, for any Clifford cycle , Pauli matrix , and Pauli cycles , the expected outcome of the ideal implementation is a Pauli matrix that can be efficiently calculated. Note that only the sign of depends on the random Pauli cycles. This sign is accounted for when estimating the expectation value with the procedure outlined in Supplementary Note 2. Incorporating the sign engineers a measurement of the expectation value of that is robust to SPAM errors, as otherwise the expectation values result from a multi-exponential decay[24,29].

In step 3c, we experimentally prepare an eigenstate of a Pauli matrix , apply a circuit with interleaved random Pauli cycles, and measure the expectation value of . The explicit procedures we use for preparing the eigenstate and measuring the expectation value are described in Supplementary Note 2. As discussed in Supplementary Note 5, the number of measurements required to estimate the expectation value to a fixed additive precision is independent of the number of qubits.

As we prove in Supplementary Note 4, the expected value of in Eq. (7) for two values of and as in step 2 is equal to the composite process fidelity in Eq. (4) up to and always provides a lower bound.

Experimental results

We demonstrate the practicality of CB for multi-qubit systems by using it to experimentally estimate the process fidelity of cycles acting globally on quantum registers containing 2, 4, 6, 8, and 10 qubits. The specific cycles we consider consist of simultaneous local Pauli gates and multi-qubit entangling MS gates[22,23] combined with simultaneous local Pauli gates. We confine ions in a linear Paul-trap and encode a single qubit in the electronic states of each atomic ion. The encoding utilizes the ground-state and the metastable excited state. Our quantum computing toolbox comprises independent arbitrary single-qubit operations and fully entangling -qubit MS gates, acting on all qubits in the register simultaneously (see Supplementary Note 7). An experimental run consists of: (i) Doppler cooling, (ii) sideband-cooling of the two motional modes with lowest frequencies, (iii) optical pumping to the initial state , (iv) coherent manipulation, and (v) readout of the ions. Each sequence is repeated 100 times to gather statistics (for experimental details, see Supplementary Note 7 and ref. [30]).

Under Markovian noise, the estimate of the process fidelity from Eq. (7) is independent of the sequence lengths and used to estimate it (see Supplementary Note 3). We tested whether our experimental apparatus satisfied this assumption by performing measurements at three values of (4, 8, and 12) on a register containing 6 qubits and comparing the results obtained from pairs of sequence lengths against each other. The data are tabulated in Supplementary Table 2, where the variation of the estimated fidelities is within 0.1%, which is smaller than the corresponding uncertainties of 0.4%. This suggests that the errors are Markovian and the estimated process fidelity is independent of the chosen sequence lengths for our system and henceforth we only use two sequence lengths to estimate the process fidelity.

The CB protocol is practical to implement on large processors because the fidelity can be accurately estimated using a number of Pauli matrices that is independent of the number of qubits (see Supplementary Note 5). To illustrate the rapid convergence under finite sample size, we performed CB of local Pauli operations on a 4-qubit register by exhaustively estimating all possible decay rates. We estimate the average fidelities via Eq. (7) for multiple subsets of the set of all Pauli matrices. For each , we evaluate the fidelity for 30 randomly chosen subsets containing Pauli matrices. The mean and standard deviation of the estimated fidelities as functions of the subset size are shown in Fig. 2. In Fig. 2b, we introduce two boundaries between which the observed standard deviation should lie if we are choosing appropriate sequence lengths and sample sufficiently many random circuits per sequence length. For the lower bound, we assume quantum projection noise to be the only noise source. We evaluate the shot noise for the measured data and perform error propagation to calculate the lower bound . This lower limit could be reached if the noise in the system is completely isotropic (e.g., global depolarizing). Biased noise or drift (see Supplementary Note 10) will lead to uncertainties bigger than those originating from quantum projection noise. We furthermore test that the fluctuations between different Pauli channels is bounded by an error model that assumes worst-case fluctuations between channels. This bound does not depend on the register size but only on the fidelity and can be estimated via (see Supplementary Note 5).

Fig. 2

Experimental evidence demonstrating rapid convergence under finite sample size with favorable constant factors. a Mean fidelity estimates from 30 randomly sampled subsets of Pauli matrices as a function of the size of the subset. The error bars illustrate the standard deviation of the 30 samples, that is, the standard error of the mean. The green line describes the mean fidelity % calculated from the complete data set. b The standard deviation of the fidelity from plot a against including a bound due to finite sampling of Pauli channels in orange, a fit of the standard deviation in green, and a fit of the expected projection noise in red (see Supplementary Note 5).

The observed standard error of the mean is larger than the lower bound given by quantum projection noise but smaller than the worst-case bound from sampling finite Pauli channels. The data demonstrate that we can estimate the process fidelity to an uncertainty smaller than independent of the register size with other experimental parameters held fixed (the parameters are listed in Supplementary Table 1).

We performed CB on local operations and with an interleaved MS gate on registers containing 2, 4, 6, 8, and 10 qubits. The process fidelity as a function of the number of qubits in the register is shown in Fig. 3 and Table 1. While it is expected that the fidelity over the full register decreases with increasing register size, an important question is whether the effective error rate per qubit increases or significant cross-talk effects appear, with increasing numbers of qubits.

Fig. 3

Experimental estimates of how rapidly error rates increase as the processor size increases. a Process fidelities obtained under cycle benchmarking for local gates (blue circles) and for sequences containing dressed Mølmer–Sørensen (MS) gates (red diamonds), that is, MS gates composed with a random Pauli cycle, plotted against the number of qubits in the register. The local operations are consistent with independent errors fitted according to Eq. (8). b Estimate of the process fidelity of an MS gate obtained by taking the ratio of dressed MS and local process fidelities. The data are fitted to Eq. (9) and is consistent with a constant error per two-qubit coupling.

Table 1

Process fidelities (%) estimated via CB. Measured fidelities for local gates, dressed MS gates, and the inferred MS gate fidelity as depicted in Fig. 3.

Qubits	Local gates	Dressed MS gate	MS gate
2	99.37 (7)	98.92 (8)	99.6 (1)
4	97.25 (8)	94.3 (1)	97.0 (2)
6	96.9 (2)	91.2 (3)	94.1 (4)
8	92.8 (8)	85 (1)	91 (2)
10	90.9 (6)	78 (1)	86 (2)

We observe that the fidelity for local CB (blue circles in Fig. 3a) decays linearly with register size , aswith . The linear decay of the fidelity indicates that our single-qubit Pauli operations do not show increasing error rates per qubit or a significant onset of cross-talk errors as the register size increases. Each single-qubit Pauli operation requires native gates, where on average , independent of the system size. Therefore, the effective process fidelity of a native single-qubit gate is .

The CB measurements with interleaved MS gates give the process fidelity of the MS gate composed with a round of local randomizing gates as in Eq. (4) (a dressed MS gate, see red diamonds in Fig. 3a). This determines the error rate when a circuit is implemented by RC[21]. The process fidelity of the interleaved gate can be estimated by the ratio of the dressed MS and local fidelities as in interleaved randomized benchmarking[27]. The resulting estimates are plotted in Fig. 3b. We note that these estimates may have a large systematic error that is on the same order as the overall error rate[28]. This systematic uncertainty primarily arises due to coherent over- and under-rotations with similar rotation axes. The MS gate performs rotations around the non-local axes , which are substantially different from the single-qubit rotation axes. Therefore, it is unlikely that any coherent errors on the MS gate accumulate with the errors on the single-qubit rotations, and so we neglect this systematic error. We conjecture that the process fidelity of the MS gate should decay quadratically due to an error in each of the couplings between pairs of qubits introduced by the MS gate. If we assume an average error rate per two-qubit coupling, we can describe the MS gate fidelity asFitting this model to the results in Fig. 3b gives an estimated error per two-qubit coupling of . However, we cannot harness these two-qubit couplings individually in the experiment and thus they cannot be compared to individually available gates. The deviations of the fidelity estimates from the model defined in Eq. (9) are within the expected statistical uncertainty and we believe that these deviations arise mainly from day-to-day fluctuations in the experiment.

Discussion

In summary, we have developed CB and demonstrated its practicality by implementing it on quantum registers containing , 4, 6, 8, and 10 qubits. In comparison, a single random Clifford gate for 8 and 10 qubits would require >50 MS gates and so randomized benchmarking for 8 and 10 qubits would require a large number of measurements to achieve a useful statistical precision. CB is practical in regimes where randomized benchmarking is impractical because it uses local randomizing gates. A similar approach was independently considered in refs. [29,31] to characterize a two-qubit Clifford gate. However, the approach implemented here and proposed previously in ref. [26] can be applied in a scalable manner to processors with arbitrary numbers of qubits.

The total experimental time and post-processing resources required for our implementation were approximately independent of the number of qubits (see Supplementary Table 1), after accounting for the additional tests performed on specific numbers of qubits. This is achieved because, as we provide proof in Supplementary Note 5, the uncertainty of the fidelity estimate is independent of the number of qubits , and the number of Pauli matrices that need to be sampled depends only on the desired precision. In addition, we demonstrated experimentally that the estimate of the fidelity and its error converges rapidly under finite sample size (Fig. 2) and that the estimated fidelities are approximately independent of the sequence lengths used. The data from CB also gives estimates of the diagonal of the Pauli–Liouville representation of the effective noise. A natural open question is to use this procedure to reconstruct the underlying noise model, which we leave for future work.

CB can be readily implemented on general quantum computing architectures to estimate the fidelity of multi-qubit processes. The fidelity corresponds to the effective error rate under RC[32]. It should be noted that the performance of the same operation in a circuit without RC can differ significantly from the estimated fidelity of its constituents due to the addition or cancellation of coherent errors[33]. This is a general issue with performance metrics for quantum operations[34] and we want to emphasize that RC has been designed to eliminate these coherent errors. The protocol also provides insight into how noise scales within a fixed architecture. In our ion trap, the fidelity of local gates across the whole register decreased linearly with , demonstrating that our native single-qubit gates have an average fidelity of 99.2(1)% and do not deteriorate with the register size. Thus we have demonstrated a scalable method to validate a major requirement for fault-tolerant quantum computation. In addition, we performed interleaved CB protocols to estimate the performance of the multi-qubit entangling MS gate. From the ratio between the dressed MS and the local CB fidelities, we infer entangling gate fidelities ranging from 99.6(1)% to  86(2)% for 2–10 qubits. While this inference is in principle subject to a large systematic uncertainty[24], we have argued that the systematic uncertainty should be small for our set of operations. We leave the problem of quantifying or reducing this systematic uncertainty open, but note that a natural approach would be to quantify how coherent the errors are using generalizations of either purity benchmarking[35] or iterated interleaved benchmarking[36].