Nanoparticle-on-mirror plasmonic nanocavities, capable of extreme optical confinement and enhancement, have triggered state-of-The-Art progress in nanophotonics and development of applications in enhanced spectroscopies. However, the optical quality factor and thus performance of these nanoconstructs are undermined by the granular polycrystalline metal films (especially when they are optically thin) used as a mirror. Here, we report an atomically smooth single-crystalline platform for low-loss nanocavities using chemically synthesized gold microflakes as a mirror. Nanocavities constructed using gold nanorods on such microflakes exhibit a rich structure of plasmonic modes, which are highly sensitive to the thickness of optically thin (down to 15 nm) microflakes. The microflakes endow nanocavities with significantly improved quality factor (2 times) and scattering intensity (3 times) compared with their counterparts based on deposited films. The developed low-loss nanocavities further allow for the integration with a mature platform of fiber optics, opening opportunities for realizing nanocavity-based miniaturized photonic devices for practical applications.