Doppler optical coherence tomography (DOCT) allows simultaneous micrometer-scale resolution cross-sectional imaging of tissue structure and blood flow. Here we demonstrate a fiber-optic, polarization diversity-based differential phase contrast DOCT system as a method to perform self-referenced velocimetry in highly scattering media. Using this strategy we reduced common-mode interferometer noise to < l Hz and improved Doppler estimates in a scattering flow phantom by a factor of 5.