A series of linear peptides (6a-6o) were designed based on the known non-covalent 20S proteasome inhibitors TMC-95A and compound 5 via a fragment-based approach. These compounds were synthesized and evaluated against the chymotrypsin-like activity of the human 20S proteasome. Three of them (6d, 6e and 6k) were potent inhibitors with IC50 values at the submicromolar level. These three compounds were selective to the β5-subunit and showed no obvious inhibition against trypsin-like and caspase-like activities of the human 20S proteasome. Docking study of the most potent compound 6e revealed its key interactions with the β5-subunit of the 20S proteasome. These findings have provided a new chemical template for non-covalent proteasome inhibitors, which is ready for further structural optimization to improve both potency and subunit selectivity.