Characterizing the Reproducibility of Noisy Quantum Circuits.

The ability of a quantum computer to reproduce or replicate the results of a quantum circuit is a key concern for verifying and validating applications of quantum computing. Statistical variations in circuit outcomes that arise from ill-characterized fluctuations in device noise may lead to computational errors and irreproducible results. While device characterization offers a direct assessment of noise, an outstanding concern is how such metrics bound the reproducibility of a given quantum circuit. Here, we first directly assess the reproducibility of a noisy quantum circuit, in terms of the Hellinger distance between the computational results, and then we show that device characterization offers an analytic bound on the observed variability. We validate the method using an ensemble of single qubit test circuits, executed on a superconducting transmon processor with well-characterized readout and gate error rates. The resulting description for circuit reproducibility, in terms of a composite device parameter, is confirmed to define an upper bound on the observed Hellinger distance, across the variable test circuits. This predictive correlation between circuit outcomes and device characterization offers an efficient method for assessing the reproducibility of noisy quantum circuits.

1. Introduction

Quantum computing leverages phenomena such as superposition and entanglement to offer novel capabilities relative to conventional computing, such as super-polynomial reductions, in time to solution and energy consumption, as well as memory storage [1]. Quantum advantages may be found for solving a variety of problems, such as simulating quantum many-body systems [2], solving unstructured optimization problems [3], achieving complex linear algebra computations [4], efficiently sampling high-dimensional probability distributions [5], and enhancing the security of communication networks [6]. These assessments are based on an idealized quantum computing model [7,8,9], comprising of an n qubit register that encodes a -dimensional Hilbert space . After initializing the register to the state , a sequence of logical unitary transformations , known as quantum gates, transform the state as . Ultimately, a measurement operation reads out an n-bit string s, where with has the probability to observe the outcome.

In practice, on-going efforts to realize a quantum computer introduces many additional physical processes that complicate the operational description above. For example, a crude approximation to an actual device can be conceptualized as a stack of interacting layers [10,11,12] that includes physical qubits, a physical control layer, a hardware-aware compiler, a logical control layer (including fault tolerant quantum error correction protocols), and a logical compiler and circuit optimizer, as well as the quantum algorithms and applications. Each of these intermediate processes introduce the possibility for noise and errors that make modeling the system more complicated. Subsequent certification that an actual physical system is performing as expected represents a leading concern for validating experimental results [13]. This is made more difficult by the inherent randomness that often manifests in the computed results, inability to pin-point exactly where in a circuit an error has occurred, curse of dimensionality, and the inability to step through program execution [14,15,16].

While quantum computers using fault-tolerant operations may eventually alleviate some of these concerns [17], current noisy intermediate-scale quantum (NISQ [18]) computers are influenced heavily by noise and errors in physical operations [19]. Computational errors arise due to noise in the register, imperfect gate implementations, and faulty thermodynamic control systems [20,21,22,23,24]. For example, noise in the register may arise from spontaneous decay, decoherence, coupling to the environment, cross-talk, and leakage. Errors in gate operations also arise when one of the control operations goes awry, due to lack of precision in the applied pulses, e.g., pulse attenuation and distortion, pulse jitter, and frequency drift. Thermodynamic control systems that maintain the stability of the operating environment may also be a source of noise, due to limits on the cooling power of the dilution refrigerators, contaminants in the vacuum chamber, or fluctuations in the electromagnetic shielding and vibration suppression systems.

The varying environment and fluctuating controls present in current NISQ computing platforms lead to transient sources of noise that impact the ability to reproduce the results of a quantum circuit [25,26]. While errors arising from fixed sources of device noise can frequently be mitigated using tailored methods [27,28,29,30,31], ill-characterized and transient noise impedes the reproduction of NISQ circuit results and prevents the replication of quantum computed solutions. It is, therefore, important to assess the conditions under which a noisy quantum circuit may be expected to be reproducible, as well as the correlation with the corresponding noise in the quantum device. Bounding statistical variations, expected from a noisy quantum circuit, in terms of device noise itself, is potentially an efficient method for assessing reproducibility on NISQ platforms.

Here, we show how device characterization may be used to bound the reproducibility of a noisy quantum circuit. First, we formalize the problem of quantifying the reproducibility of results from a quantum circuit (), executed on a noisy quantum device (D). We express this statistical variation, in terms of the Hellinger distance between the observed noisy output and expected ideal distribution, and we examine the conditions under which this distance deviates. Then, we derive an upper bound on the Hellinger distance by setting a threshold on these deviations that may be expressed in terms of the device noise itself. This yields a test as to whether the quantum circuit () is reproducible when executed on the given device (D). We show that this test is efficient when cast in terms of a single composite parameters computed by estimating the device noise. We then validate this method using an ensemble of single qubit test circuits executed on a well-characterized, but noisy, superconducting transmon processor.

The remainder of the presentation is as follows: in Section 2, we show how the composite parameter (), representing a quantum circuit (), executed on a device (D), is composed from available device characterization, and that, when stays below the threshold , then the circuit is reproducible. In Section 3, we present results from applying these methods to single qubit circuits, executed on a superconducting transmon processor, to validate agreement between the theory and experiment. Finally, in Section 4, we discuss the success of this approach, as well as its potential limitations.

This work is a significant extension of an earlier publication [32], in which the methods and results have been revised and refined to validate experimentally-observed bounds.

2. Method

Consider to denote the initial density matrix representing the state of the n qubit quantum register. In the ideal setting, the quantum state of the register evolves under a unitary transformation that describes the circuit as:

Let represent the orthonormal computational basis, with , and let be the corresponding set of orthonormal projectors. The corresponding probability distribution for the results generated by an ideal quantum device () is then given as: where is the probability to observe the i-th outcome. In general, it is not efficient to construct the set using conventional methods, i.e., classical computing, and the resources for such calculations scale exponentially in the size of n. However, such demanding calculations are feasible for values of , while calculations of highly structured circuits, such as the quantum Fourier transform, quantum search, or arithmetic circuits, may be simulated efficiently. We limit our current approach to the consideration of for such instances.

In the presence of noise, the evolution of the quantum register is no longer modeled by a unitary operator, and the final state is generally a mixed-state. Let be the super-operator representing a noisy quantum channel, in terms of Kraus operators () that represent the execution of the circuit . The corresponding state of the register is then: where the Kraus operators are a function of the device reliability parameters. Thus, the corresponding probability distribution from executing on the noisy quantum device is given by: where , and is the measurement operator for a noisy readout channel [33].

We quantify the variation between the ideal and actual distributions for the results from executing the circuit using the Hellinger distance. We denote the Hellinger distance between and as: with the Bhattacharyya coefficient defined as:

Note that the Hellinger distance vanishes for identical distributions and approaches unity for those distributions with no overlap. Now, let be a parameter that defines a threshold for the Hellinger distance, i.e., when: then, the results from a quantum circuit () on a noisy, quantum device (D) are reproducible within a distance () of the idealized outcomes. The equation for characterizing the reproducibility of a noisy quantum circuit introduces a test for deciding if the observed value of the Hellinger distance lies below the elected threshold.

In general, the above test for reproducibility requires estimating the noisy probability distribution () by repeatedly sampling the circuit outcomes. The number of samples required to estimate an arbitrary distribution over outcomes with precision () scales exponentially with the Hilbert space dimension (n) and as an inverse square of the required precision () (see Appendix A). Such a direct method for estimating the Hellinger distance is inefficient and requires repetition across devices. We next consider how to correlate the bound placed on the Hellinger distance in Equation (7), with the noise assumptions for the given device. We show, by example, that a composite parameter (), characterizing the noisy quantum circuit, can be similarly bound from above as:

Example

Consider an n qubit state prepared as a uniform superposition across the computational basis states as:

The corresponding quantum circuit shown in Figure 1 corresponds to: for which the ideal distribution is:

Figure 1

Circuit used for our experiment. In this figure, H represents the Hadamard gate. The meter symbol denotes measurement gate.

We next assume that the quantum register is reliably initialized as and that inter-qubit cross-talk is negligible during noisy circuit execution. Rather, gate error and readout fidelity capture the principal sources of noise in this circuit, each of which is modeled as a noisy process.

Let and denote the identity matrix, Pauli-X matrix, Pauli-Y matrix, and the Pauli-Z matrix, respectively, in the computational basis as:

Additionally, let denote the rotation by an angle () about the Y-axis on the Bloch sphere:

An ideal Hadamard gate is then given by:

We model a noisy Hadamard gate () by the unitary: where is the implementation error that is assumed to be small, i.e., .

The operator representation for a unitary control error has only one term which can be seen as follows. Write a noisy unitary as where is the ideal unitary. Thus, where E is the operator representing the unitary control error that arises due to imperfections in the control system. For our example, which is the 2D rotation matrix (no error will correspond to an identity channel with ).

Thus,

In the absence of readout noise, when we initialize a qubit in the ground state, subject it to a noisy Hadamard gate, and measure in the -basis, we get the probabilities for observing the output as:

We next consider what happens when the Hadamard gate is followed by a noisy measurement. The readout channel is characterized as a quantum channel [29,34,35] as shown in Figure 2, using two parameters for each qubit.

Figure 2

The quantum channel maps and to their expected outcomes, with probabilities and , respectively.

The first parameter () defines the probability of observing 0 post readout when the channel input state is , and the second () defines the probability of observing 1 post readout when the channel input state is .

A classical representation for the single qubit readout channel is: where is the readout error matrix with elements = probability of observing when the input to channel is :

Equivalently, the quantum channel representation for a single qubit noisy measurement has two terms, i.e., and , and is specified by a super-operator (), whose action on the quantum state is as follows [33]: which is equivalent to Equation (24), when you consider the action of the measurement operators (), given by . Thus, the representation equivalently shows how a noisy measurement of a quantum state yields a particular readout i.

Let Pr be the probability of observing 0 when we prepare a qubit in the ground state, subject it to a Hadamard gate, and then measure it. Let Pr be the corresponding probability of observing 1 for the same experiment (i.e., we prepare a qubit in the ground state, subject it to a Hadamard gate, and then measure it). Additionally, let f be the average readout fidelity and be the readout fidelity asymmetry:

Thus, in the presence of readout noise, the probability of observing 0 and 1 for each qubit in Figure 1 is given by [33]: where

Let represent the n-bit string with , and let be the decimal integer equivalent. In the absence of cross-talk between gates, where: and refers to the i-th register element. Thus, and the sum is over all bit strings with decimal integer equivalent . If is the user-defined threshold on the acceptable Hellinger distance, then it follows that: or

Without loss of generality, we consider that all register elements have identical readout and gate errors, which allows us to replace with for each of the n register elements. Thence:

As shown in Appendix B, when assuming that is small, this yields: with and

Here, is a composite parameter, estimated using characterization of the individual gate operations, e.g., during periodic device calibration. For our specific circuit and device noise model, the parameter is a function of the gate angle error and measurement fidelity and serves as a combined measure of the hardware noise and output variation; the exact relationship is a function of the noisy circuit model used. Thus, we have shown that if exceeds the bound , then the threshold on the Hellinger distance is also exceeded. Notably, the former statement does not require experimental execution of the circuit itself or even estimation of the Hellinger distance, but rather depends solely on our model for the noisy device and accuracy of the parameter estimation.

As we have assumed, is small, and Equation (35) may be reduced as: which tells us that the tolerance specification for an experimentally-observed output distribution when using a noisy device (D) that is lower bounded. Attempts to reproduce the experiment must use an error bar larger than this minimum, otherwise it will likely fail the accuracy test, and the ability of the device to reproduce the output distribution will be questioned.

3. Results

We validated our method for testing the reproducibility of the previously described test circuits on the superconducting transmon hardware, called ibmq_toronto, from IBM, whose schematic is shown in Figure A1. The test circuit shown in Figure 1 was programmed using the IBM qiskit toolkit [36] and compiled and executed remotely on the ibmq_toronto device on 8 April 2021.

Figure A1

Schematic of the ibmq_toronto device, produced by IBM. Circles represent register elements, while edges denote the connectivity for performing two qubit operations.

To estimate the device parameters, we repeated our experiments to estimate the readout fidelity and gate angle error L times (each). Let l denote the index of the l-th experiment. For any instance of circuit execution, the ibmq_toronto device prepared an ensemble of S identical circuits, where S denotes the number of shots and s denotes the s-th shot in a particular experiment. In the tests reported below, was the number of repetitions successfully executed during a 30-min reservation-window on ibmq_toronto, and was the number of shots, the maximum allowed by the device. We separately analyzed the results for the case in Equation (36) using each of the 27 register elements available in ibmq_toronto.

Next, we characterized the device parameters required to estimate and . Here, we introduce the convention that a random variable is denoted by bold font, and a caret sign denotes a particular realization of that random variable. We first characterized readout fidelity, in which SPAM(0) denotes an experiment with a register element, prepared as and measured. Similarly, SPAM(1) denotes an experiment in with a register element, prepared as measured. Then, let denote the binary outcome of measuring in the computational basis, when collecting the s-th shot of the l-th experiment of the SPAM(0) circuit on the q-th register element. Additionally, let denote the initialization fidelity observed in the l-th SPAM(1) experiment for the q-th register element. Similarly, let denote the initialization fidelity observed in the l-th SPAM(0) experiment for the q-th register element. Thus: and

Let denote the realized fidelity asymmetry of the q-th register element in the l-th experiment. Thus:

Let denote the mean of the fidelity asymmetry for the q-th register element over the L experiments, and let be the corresponding observed value. Thus: and

To quantify the error on these measurements, we define as the standard deviation of population mean , such that:

The average readout fidelity for each qubit q is then calculated using Equation (26).

Figure 3 shows the experimental results for the fidelity distributions of and for the first nine register elements of ibmq_toronto, collected on 8 April 2021, between 8:00–10:00 p.m. (UTC-05:00). The initialization fidelities of the computational states are not the same and, for some register elements, they are very distinct. The asymmetric nature of the single qubit noise channel is brought out starkly by the negligible overlap between the distributions of and for qubit 5. Additionally, observe the significant spread in values (indicating channel instability) in qubit 3, relative to the others. These results show that the naive approach of assuming a single value for readout error for a qubit is dangerous. Not only do we have to characterize and separately, our work must also take into account the significant dispersion around the mean. The remaining register elements are shown in the Appendix B in Figure A2 and Figure A3. The register variation of the readout asymmetry is shown in Figure A5.

Figure 3

Fidelity distributions for computational basis states of ibmq_toronto for qubits 0–8. Raw data for ibmq_toronto, collected on 8 April 2021, between 8:00–10:00 p.m. (UTC-05:00). The remaining register elements are shown in the Appendix B in Figure A2 and Figure A3.

Figure A2

Fidelity distributions for computational basis states of ibmq_toronto for qubits 9–17. Raw data for ibmq_toronto, collected on 8 April 2021, between 8:00–10:00 p.m. (UTC-05:00).

Figure A3

Fidelity distributions for computational basis states of ibmq_toronto for qubits 18–26. Raw data for ibmq_toronto, collected on 8 April 2021, between 8:00–10:00 p.m. (UTC-05:00).

Figure A5

Each of the 27 register elements was used to verify Equation (36) for . The dotted red line denotes the register mean for readout asymmetry (averaged over all qubits). Qubit 0 has the best performance for this parameter, while qubit 24 was the worst. The error bars show the standard deviation of the population mean across L = 203 experiments. A consistent register color scheme has been used for all the figures.

The probability (Pr) for each qubit, as defined by Equation (28), was estimated by executing the circuit in Figure 1 and counting the faction of zeros in the 8192-bit long binary string, returned by the remote server. Let denote the random measurement outcome (a classical bit) when we conduct a experiment and measure the q-th register element in the l-th experiment’s s-th shot (measurement done in the computational Z-basis). Let denote the corresponding observed value. Similarly, let denote the probability of observing 1 as the outcome in the l-th experiment for the q-th register element. Similarly, let denote the probability of observing 0 as the outcome in the l-th experiment for the q-th register element. Thus: and

Let denote the Hellinger distance between the noisy and ideal outcomes in the l-th experiment for the q-th register element. Let be the corresponding observed value (or realization). Let denote the mean (a random variable) of the Hellinger distance for the q-th register element over L experiments. Let be the corresponding observed value (or realization). Thus:

To quantify the error on these measurements, define as the standard deviation of population mean . Thus:

The register variation of the experimentally-obtained Hellinger distance is shown in Figure A4. The Hadamard gate angle error was subsequently estimated using Equation (30). The register variation of the the Hadamard gate angle error (in degrees) is shown in Figure 4. The dotted red line denotes the register mean for the gate error, averaged over all qubits. The gate implementation on qubit 21 is the closest to ideal, while qubit 24 is the farthest.

Figure A4

Register variation of the experimentally-obtained Hellinger distance. Each of the 27 register elements was used to verify Equation (36) for . The dotted red line denotes the register mean for Hellinger distance (averaged over all qubits). Qubit 19 is the closest to ideal, while qubit 24 is the farthest. The error bars show the standard deviation of the population mean across experiments. A consistent register color scheme has been used for all the figures.

Figure 4

Register variation of the experimentally-obtained Hadamard gate error (in degrees). Each of the 27 register elements was used to verify Equation (36) for . The dotted red line denotes the register mean for the gate error (averaged over all qubits). Qubit 21 is the closest to ideal, while qubit 24 is the farthest. A consistent register color scheme has been used for all the figures.

Figure 5 shows the values for and for ibmq_toronto on 8 April 2021, when is set to be the observed Hellinger distance. The blue dots are experimentally-observed data for each register element (see Table A1 for the full list), using the characterization data versus the actual observed Hellinger distance. It validates our noise model as Equation (8) holds. The dashed line in Figure 5 provides the decision boundary to test circuit reproducibility, using characterization data as a proxy. Given a user-defined threshold on the statistical distance between the observed distribution and reference to be reproduced, the plot provides an upper bound for the device parameter (the register variation of is shown in Figure A6). The latter must lie below this boundary for reproducibility by the device. We conjecture that the magnitude of serves as a reliability indicator, i.e., higher value provides greater cushion against temporal fluctuations (see [25]). Thus, Table A1 can serve as a basis for register selection tailored to a specific unitary when channel characterization data is available.

Figure 5

Characterizing circuit reproducibility on ibmq_toronto. Plot of (dashed line) and (blue dots) for ibmq_toronto on 8 April 2021, when is set to be the observed Hellinger distance . Equation (8) should hold for all and hence it must hold for the corner case when is set to be the experimentally observed Hellinger distance. The blue dots are experimentally-observed data plotted using the characterization data versus the actual observed Hellinger distance for each register element. Only a subset of qubits are shown. See Table A1 for all 27 qubits.

Table A1

Register values for and .

Register #	γ_max	γ_D(τ)	Register #	γ_max	γ_D(τ)
0	1.4590	1.3040	14	11.9104	11.9038
1	1.1365	0.6755	15	2.0713	2.0228
2	2.7284	2.7118	16	1.3392	0.2359
3	6.9946	6.9931	17	4.8557	4.8553
4	4.3229	4.3226	18	1.5986	1.4980
5	5.8171	5.8157	19	1.0322	0.4378
6	4.5425	4.5325	20	9.0893	9.0886
7	2.6946	2.6649	21	1.2259	1.0620
8	5.5066	5.4724	22	10.9146	10.9136
9	8.9672	8.9666	23	3.0018	3.0017
10	2.7272	2.7231	24	14.1254	14.1241
11	11.5502	11.5486	25	1.4325	1.2624
12	3.2212	3.0797	26	1.3103	0.9567
13	1.7818	0.6460

Figure A6

Register variation of the derived parameter . A high adversely impacts the computational reproducibility. However, low does not necessarily mean that the device is close to perfect because the terms in Equation (30) can cancel each other out and lead to an improvement in the output distance. For an unstable device, will vary with time and must be re-estimated. The dotted red line denotes the register mean averaged over all qubits. Qubit 16 has the best performance for this composite parameter, while qubit 24 has the worst. A consistent register color scheme has been used for all the figures.

4. Conclusions

Reproducibility is important for validating the performance of applications in quantum computing and as a measure of consistency in computation. Current NISQ devices are strongly affected by intrinsic noise that lead to a variety of computational error mechanisms. Here, we have characterized the ability to reproduce the output generated by noisy quantum circuit, using single qubit rotation gates and asymmetric noisy readout. We showed how to derive a composite parameter that can be easily computed from the available device characterization data (e.g., readout fidelity and gate error), such that, if the derived parameter stays below a defined threshold, then the circuit is assured of reproducible output. Our model led to an analytic bound, in terms of the Hellinger distance between computational outputs, and we validated the resulting test using experiments on a publicly available transmon processor.

Despite the successful validation, we note that this approach has its limitations. A preliminary version of these results used only readout asymmetry and ignored the Hadamard gate error for describing circuit noise [32]. For example, on 21 August 2021, the typical mean error rates were for single-qubit Pauli-X error and for readout assignment error on the 27-qubit IBM quantum device ibmq_toronto. See Figure A7 for the results from that first-order model that failed the test of Lemma A1 for 5 of the 27 qubits, due to incorrect assumptions about the device model. An analytically-derived composite parameter, using a wrong noise model, cannot serve as a suitable proxy for circuit reproducibility. Similarly, if the device parameters themselves cannot be estimated from the available device characterization data, then the scheme is impractical.

Figure A7

Plot of readout asymmetry vs. observed Hellinger distance for experiments. Each register element is shaded by a different color.

While our method is generalizable in principle, a closed form expression for may not be obtainable for complex circuits. As the number of parameters needed to characterize the channel increases with circuit complexity (i.e., number of qubits and circuit depth); one will need to either utilize numerical methods to deduce or make simplifying assumptions. A circuit-specific modular or layered characterization for the device may reduce the parameter estimation overhead, but this remains to be investigated. There is still an open question regarding how to systematically extend this method to any model and what conditions must the model meet.

As quantum computations enter the realm of advantage over classical, we expect that the task of practically characterizing circuit reproducibility will become an important area. To our knowledge, this is a first attempt to frame the problem of linking circuit reproducibility to device reliability. Such reproducibility characterization serves four purposes. Firstly, it shows how to make use of the device characterization data in a targeted way for inferring circuit reproducibility. Second, it provides a basis for selecting a device for a circuit prior to execution. Third, it provides insight on how the quality of final digital output is related to the characteristics of the intermediate quantum channels and, hence, understand the device improvement pathways for a specific circuit. Lastly, it helps to estimate a lower bound for the error that can be expected from the circuit execution.