Optical spin pumping of color centers promises to revolutionize the practice of nuclear magnetic resonance via efficient dynamic nuclear polarization (DNP) protocols. Significant progress has been attained with systems such as the nitrogen-vacancy (NV) center in diamond, but extreme spectral broadening and nuclear spin-lattice relaxation from co-existing paramagnetic defects are hurdles still difficult to circumvent. The technique we present in this manuscript addresses both issues through the implementation of magnetic-field-sweep-induced NV cross-relaxation with ancilla point defects, here acting as proxies to mediate the transfer of polarization to the nuclear spin ensemble. The governing spin dynamics is robust to field misalignment and ideally suited for shallow NVs, indirectly coupled to outside nuclei through paramagnetic defects at the diamond surface.