Intraoperative 3D imaging has great potential for enhancing surgical visualization. This is especially so in ophthalmic surgery where live volumetric imaging from optical coherence tomography systems recently incorporated into surgical microscopes has freed surgeons from the otherwise universal top-down viewpoint. New viewpoints, however, disorient surgeons when directions of their hand motions and viewed tool motions do not align. We propose introducing a robotic surgery system to decouple surgeons' hands from their tools and ensure that viewed tool motions align in arbitrary viewpoints. We present a framework entitled Arbitrary Viewpoint Robotic Manipulation (AVRM) which governs how hand and tool motions should interact to minimize disorientation and thereby enable operations from desirable but previously untenable viewpoints. A crossover study in which 20 subjects completed mock surgical scenarios with an AVRM testbed system demonstrated that arbitrary viewpoints do not improve task performance unless automatic hand-tool misalignment correction is provided. When provided together with arbitrary viewpoints, automatic hand-tool misalignment correction reduces task completion time by 50% on average compared to a fixed top-down viewpoint.