Just as exactly marginal operators allow one to deform a conformal field theory along the space of theories known as the conformal manifold, appropriate operators on conformal defects allow for deformations of the defects. When a defect breaks a global symmetry, there is a contact term in the conservation equation with an exactly marginal defect operator. The resulting defect conformal manifold is the symmetry breaking coset, and its Zamolodchikov metric is expressed as the two-point function of the exactly marginal operator. As the Riemann tensor on the conformal manifold can be expressed as an integrated four-point function of the marginal operators, we find an exact relation to the curvature of the coset space. We confirm this relation against previously obtained four-point functions for insertions into the 1/2 BPS Wilson loop in N=4 SYM and 3D N=6 theory and the 1/2 BPS surface operator of the 6D N=(2,0) theory.