We construct a qubit regularization of the O(3) nonlinear sigma model in two and three spatial dimensions using a quantum Hamiltonian with two qubits per lattice site. Using a worldline formulation and worm algorithms, we show that in two spatial dimensions our model has a quantum critical point where the well-known scale-invariant physics of the three-dimensional Wilson-Fisher fixed point is reproduced. In three spatial dimensions, we recover mean-field critical exponents at a similar quantum critical point. These results show that our qubit Hamiltonian is in the same universality class as the traditional classical lattice model close to the critical points. Simple modifications to our model also allow us to study the physics of traditional lattice models with O(2) and Z2 symmetries close to the corresponding critical points.