A simple adjustment to the Pearson chi-square test has been proposed for comparing proportions estimated from clustered binary observations. However, the assumptions needed to assure the validity of this test have not yet been thoroughly addressed. These assumptions will hold for experimental comparisons, but could be violated for some observational comparisons. In this paper we investigate the conditions under which the adjusted chi-square statistic is valid and examine its performance when these assumptions are violated. We also introduce some alternative test statistics that do not require these assumptions. The test statistics considered are then compared through simulation and an example presented based on real data. The simulation study shows that the adjusted chi-square statistic generally produces empirical type I errors close to nominal under the assumption of a common intracluster correlation coefficient. Even if the intracluster correlations are different, the adjusted chi-square statistic performs well when the groups have equal numbers of clusters.