Nonlinear cavity optomechanics with nanomechanical thermal fluctuations

Although the interaction between light and motion in cavity optomechanical systems is inherently nonlinear, experimental demonstrations to date have allowed a linearized description in all except highly driven cases. Here, we demonstrate a nanoscale optomechanical system in which the interaction between light and motion is so large (single-photon cooperativity C-0 approximate to 10(3)) that thermal motion induces optical frequency fluctuations larger than the intrinsic optical linewidth. The system thereby operates in a fully nonlinear regime, which pronouncedly impacts the optical response, displacement measurement and radiation pressure backaction. Specifically, we measure an apparent optical linewidth that is dominated by thermo-mechanically induced frequency fluctuations over a wide temperature range, and show that in this regime thermal displacement measurements cannot be described by conventional analytical models. We perform a proof-of-concept demonstration of exploiting the nonlinearity to conduct sensitive quadratic readout of nanomechanical displacement. Finally, we explore how backaction in this regime affects the mechanical fluctuation spectra.