Sub-projection-noise sensitivity in broadband atomic magnetometry.

We demonstrate sub-projection-noise sensitivity of a broadband atomic magnetometer using quantum nondemolition spin measurements. A cold, dipole-trapped sample of rubidium atoms provides a long-lived spin system in a nonmagnetic environment, and is probed nondestructively by paramagnetic Faraday rotation. The calibration procedure employs as known reference state, the maximum-entropy or "thermal" spin state, and quantitative imaging-based atom counting to identify electronic, quantum, and technical noise in both the probe and spin system. The measurement achieves a sensitivity 1.6 dB (2.8 dB) better than projection-noise (thermal state quantum noise) and will enable squeezing-enhanced broadband magnetometry.