Gastrointestinal (GI) conditions are widespread and significantly impact the quality of life and healthcare. Stool appearance is a valuable GI diagnostic indicator, and with the advent of the Internet of Things (IoT), daily monitoring of excreta from a toilet is emerging as a promising digital health tool. This article describes a stool image analysis approach that classifies two physiologically relevant stool characteristics: 1) form and 2) color for an IoT-based smart toilet. We constructed a stool image data set with 3275 images, spanning all seven types of the Bristol stool-form scale (BSFS), a widely used metric for stool consistency and a variety of colors. We used ground-truth data obtained through the annotation of our data set by two gastroenterologists and developed a stool-color card to standardize the labeling of stool colors. We addressed two limitations associated with the application of computer-vision techniques to a smart-toilet system: 1) uneven separability between different stool-form categories and 2) class imbalance in the data set. We present results on hierarchical convolutional neural network (CNN) architectures for training a stool-form classifier and on perceptual color quantization coupled with machine-learning techniques to optimize the color-feature space for the classification of stool color. We utilized an edge-cloud approach to pursue an optimal balance between accuracy and latency and for the classification of stool form, we achieved a balanced accuracy of 84.4% and 84.2% reduction in latency compared to a cloud-model only. For color classification, the logistic-regression (LR) classifier provided 80.8% balanced accuracy and 47.3% reduction in communication latency.