PURPOSE: Setup errors and prostate intrafraction motion are main sources of localization uncertainty in prostate cancer radiation therapy. This study evaluates four different imaging modalities 3D ultrasound (US), kV planar images, cone-beam computed tomography (CBCT), and implanted electromagnetic transponders (Calypso/Varian) to assess inter- and intrafraction localization errors during intensity-modulated radiation therapy based treatment of prostate cancer. METHODS: Twenty-seven prostate cancer patients were enrolled in a prospective IRB-approved study and treated to a total dose of 75.6 Gy (1.8 Gy/fraction). Overall, 1100 fractions were evaluated. For each fraction, treatment targets were localized using US, kV planar images, and CBCT in a sequence defined to determine setup offsets relative to the patient skin tattoos, intermodality differences, and residual errors for each patient and patient cohort. Planning margins, following van Herk's formalism, were estimated based on error distributions. Calypso-based localization was not available for the first eight patients, therefore centroid positions of implanted gold-seed markers imaged prior to and immediately following treatment were used as a motion surrogate during treatment. For the remaining 19 patients, Calypso transponders were used to assess prostate intrafraction motion. RESULTS: The means (μ), and standard deviations (SD) of the systematic (Σ) and random errors (σ) of interfraction prostate shifts (relative to initial skin tattoo positioning), as evaluated using CBCT, kV, and US, averaged over all patients and fractions, were: [μ CBCT = (-1.2, 0.2, 1.1) mm, Σ CBCT = (3.0, 1.4, 2.4) mm, σ CBCT = (3.2, 2.2, 2.5) mm], [μkV = (-2.9, -0.4, 0.5) mm, Σ kV = (3.4, 3.1, 2.6) mm, σ kV = (2.9, 2.0, 2.4) mm], and [μ US = (-3.6, -1.4, 0.0) mm, Σ US = (3.3, 3.5, 2.8) mm, σ US = (4.1, 3.8, 3.6) mm], in the anterior-posterior (A/P), superior-inferior (S/I), and the left-right (L/R) directions, respectively. In the treatment protocol, adjustment of couch was guided by US images. Residual setup errors as assessed by kV images were found to be: μ residual = (-0.4, 0.2, 0.2) mm, Σ residual = (1.0, 1.0,0.7) mm, and σ residual = (2.5, 2.3, 1.8) mm. Intrafraction prostate motion, evaluated using electromagnetic transponders, was: μ intrafxn = (0.0, 0.0, 0.0) mm, Σ intrafxn = (1.3, 1.5, 0.6) mm, and σ intrafxn = (2.6, 2.4, 1.4) mm. Shifts between pre- and post-treatment kV images were: μ kV(post-pre) = (-0.3, 0.8, -0.2), Σ kV(post-pre) = (2.4, 2.7, 2.1) mm, and σ kV(post-pre) = (2.7, 3.2, 3.1) mm. Relative to skin tattoos, planning margins for setup error were within 10-11 mm for all image-based modalities. The use of image guidance was shown to reduce these margins to less than 5 mm. Margins to compensate for both residual setup (interfraction) errors as well as intrafraction motion were 6.6, 6.8, and 3.9 mm in the A/P, S/I, and L/R directions, respectively. CONCLUSIONS: Analysis of interfraction setup errors, performed with US, CBCT, planar kV images, and electromagnetic transponders, from a large dataset revealed intermodality shifts were comparable (within 3-4 mm). Interfraction planning margins, relative to setup based on skin marks, were generally within the 10 mm prostate-to-planning target volume margin used in our clinic. With image guidance, interfraction residual planning margins were reduced to approximately less than 4 mm. These findings are potentially important for dose escalation studies using smaller margins to better protect normal tissues.