We measure the cross-correlation between weak lensing of galaxy images and of the cosmic microwave background (CMB). The effects of gravitational lensing on different sources will be correlated if the lensing is caused by the same mass fluctuations. We use galaxy shape measurements from 139 deg2 of the Dark Energy Survey (DES) Science Verification data and overlapping CMB lensing from the South Pole Telescope (SPT) and Planck. The DES source galaxies have a median redshift of zmed ∼ 0.7, while the CMB lensing kernel is broad and peaks at z ∼ 2. The resulting cross-correlation is maximally sensitive to mass fluctuations at z ∼ 0.44. Assuming the Planck 2015 best-fitting cosmology, the amplitude of the DES×SPT cross-power is found to be ASPT = 0.88 ± 0.30 and that from DES×Planck to be APlanck = 0.86 ± 0.39, where A = 1 corresponds to the theoretical prediction. These are consistent with the expected signal and correspond to significances of 2.9σ and 2.2σ, respectively. We demonstrate that our results are robust to a number of important systematic effects including the shear measurement method, estimator choice, photo-z uncertainty and CMB lensing systematics. We calculate a value of A = 1.08 ± 0.36 for DES×SPT when we correct the observations with a simple intrinsic alignment model. With three measurements of this cross-correlation now existing in the literature, there is not yet reliable evidence for any deviation from the expected LCDM level of cross-correlation. We provide forecasts for the expected signal-to-noise ratio of the combination of the five-year DES survey and SPT-3G.