Estrogen signaling, through estrogen receptor (ER)α, has been shown to cause hypertrophy in the prostate. Our recent report has shown that epithelial ERα knockout (KO) will not affect the normal prostate development or homeostasis. However, it remains unclear whether ERα in different types of stromal cells has distinct roles in prostate development. This study proposed to elucidate how KO of ERα in the stromal smooth muscle or fibroblast cells may interrupt cross talk between prostate stromal and epithelial cells. Smooth muscle ERαKO (smERαKO) mice showed decreased glandular infolding with the proximal area exhibiting a significant decrease. Fibroblast ERαKO mouse prostates did not exhibit this phenotype but showed a decrease in the number of ductal tips. Additionally, the amount of collagen observed in the basement membrane was reduced in smERαKO prostates. Interestingly, these phenotypes were found to be mutually exclusive among smERαKO or fibroblast ERαKO mice. Compound KO of ERα in both fibroblast and smooth muscle showed combined phenotypes from each of the single KO. Further mechanistic studies showed that IGF-I and epidermal growth factor were down-regulated in prostate smooth muscle PS-1 cells lacking ERα. Together, our results indicate the distinct functions of fibroblast vs. smERα in prostate development.