Many approaches have been developed to characterize cell elasticity. Among these, atomic force microscopy (AFM) combined with modeling has been widely used to characterize cellular compliance. However, such approaches are often limited by the difficulties associated with using a specific instrument and by the complexity of analyzing the measured data. More recently, quantitative phase imaging (QPI) has been applied to characterize cellular stiffness by using an effective spring constant. This metric was further correlated to mass distribution (disorder strength) within the cell. However, these measurements are difficult to compare to AFM-derived measurements of Young's modulus. Here, we describe, to our knowledge, a new way of analyzing QPI data to directly retrieve the shear modulus. Our approach enables label-free measurement of cellular mechanical properties that can be directly compared to values obtained from other rheological methods. To demonstrate the technique, we measured shear modulus and phase disorder strength using QPI, as well as Young's modulus using AFM, across two breast cancer cell-line populations dosed with three different concentrations of cytochalasin D, an actin-depolymerizing toxin. Comparison of QPI-derived and AFM moduli shows good agreement between the two measures and further agrees with theory. Our results suggest that QPI is a powerful tool for cellular biophysics because it allows for optical quantitative measurements of cell mechanical properties.