BACKGROUND: Androgen deprivation therapy (ADT) for prostate cancer (PC) has detrimental effects on physical function and quality of life (QoL), but the addition of androgen receptor signalling inhibitors (ARSI) on these outcomes is unclear. PURPOSE: To compare body composition, physical function, and QoL across progressive stages of PC and non-cancer controls (CON). METHODS: In men with hormone sensitive PC (HSPC, n = 43) or metastatic castration-resistant PC (mCRPC, n = 22) or CON (n = 37), relative and absolute lean and fat mass, physical function (6 m walk, chair stands, timed up and go [TUG], stair climb), and QoL were determined. RESULTS: Relative body composition differed amongst all groups, along with ~39% greater absolute fat mass in mCRPC vs. CON. TUG and chair stands were ~71% and ~33% slower in mCRPC compared to both CON and HSPC, whereas stair climb was ~29% and 6 m walk was ~18% slower in mCRPC vs. CON. Relative body composition was correlated with physical function (r = 0.259-0.385). Clinically relevant differences for mCRPC were observed for overall QoL and several subscales vs. CON, although body composition and physical function did not influence QoL. CONCLUSIONS: PC progression is associated with deteriorations in body composition and physical function. As ADT length was similar between groups, ARSI use for mCRPC likely contributed in part to these changes. Given the difficulties of improving lean mass during ADT, interventions that reduce adiposity may lessen the side effects of hormone therapy.