Accurate force fields are crucial for molecular dynamics investigation of complex biological systems. Building accurate protein force fields from quantum mechanical (QM) calculations is challenging due to the complexity of proteins and high computational costs of QM methods. In order to overcome these two difficulties, here we developed the residue-based systematic molecular fragmentation method to partition general proteins into only 20 types of amino acid dipeptides and one type of peptide bond at level 1. The total energy of proteins is the combination of the energies of these fragments. Each type of the fragments is then parametrized using neural network (NN) representation of the QM reference. Adopting NN representation can circumvent the limitation of the analytic form of classical molecular mechanics (MM) force fields. Using MM force fields as the baseline, our method adds NN representation of QM corrections at the length scale of amino acid dipeptides. We tested our force fields for both homogeneous and heterogeneous polypeptides. Energy and forces predicted by our force fields compare favorably with full QM calculations from tripeptides to decapeptides. Our development provides an efficient and accurate method of building protein force fields fully from ab initio QM calculations.