A four group cross-over design for measuring irreversible treatments on web search tasks
When trying to measure the effect of irreversible treatments such as training interventions, the choice of the experimental design can be difficult. A two group cross-over experimental design cannot be used due to longitudinal effects during the course of the experimental run, which can be especially large in dynamic web search environments. A standard case/control two group design also can be problematic because it is negatively impacted by variability among participants. Our solution uses a four group cross-over design that combines features from standard cross-over and case/control designs. We illustrate the effectiveness of this design through a case study in which participants are shown a video on how to use "control-F" to search for text within a web page. We quantify the improvement of our four group cross-over design compared to the standard case/control two group design with respect to measuring the effect of this video on participants. Finally, we compare the magnitude of our estimated "control-F" effect to similar studies on web ranking and user-interface changes, revealing that teaching this skill produces a potentially large improvement in web searchers' ability to search rapidly.
