The analytic energy gradients of the optimized effective potential (OEP) method in density-functional theory are developed. Their implementation in the direct optimization approach of Yang and Wu [Phys. Rev. Lett. 89, 143002 (2002)] and Wu and Yang [J. Theor. Comput. Chem. 2, 627 (2003)] are carried out and the validity is confirmed by comparison with corresponding gradients calculated via numerical finite difference. These gradients are then used to perform geometry optimizations on a test set of molecules. It is found that exchange-only OEP (EXX) molecular geometries are very close to the Hartree-Fock results and that the difference between the B3LYP and OEP-B3LYP results is negligible. When the energy is expressed in terms of a functional of Kohn-Sham orbitals, or in terms of a Kohn-Sham potential, the OEP becomes the only way to perform density-functional calculations and the present development in the OEP method should play an important role in the applications of orbital or potential functionals.