Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study
Publication
, Conference
Kim, T; Bock, K; Luo, C; Liswood, A; Poroslay, C; Wenger, E
Published in: Proceedings of the 2025 ACM Internet Measurement Conference
October 28, 2025
Duke Scholars
Published In
Proceedings of the 2025 ACM Internet Measurement Conference
DOI
Publication Date
October 28, 2025
Start / End Page
541 / 557
Publisher
ACM
Conference Name
IMC '25:ACM Internet Measurement Conference
Citation
APA
Chicago
ICMJE
MLA
NLM
Kim, T., Bock, K., Luo, C., Liswood, A., Poroslay, C., & Wenger, E. (2025). Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study. In Proceedings of the 2025 ACM Internet Measurement Conference (pp. 541–557). ACM. https://doi.org/10.1145/3730567.3764471
Kim, Taein, Karstan Bock, Claire Luo, Amanda Liswood, Chloe Poroslay, and Emily Wenger. “Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study.” In Proceedings of the 2025 ACM Internet Measurement Conference, 541–57. ACM, 2025. https://doi.org/10.1145/3730567.3764471.
Kim T, Bock K, Luo C, Liswood A, Poroslay C, Wenger E. Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study. In: Proceedings of the 2025 ACM Internet Measurement Conference. ACM; 2025. p. 541–57.
Kim, Taein, et al. “Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study.” Proceedings of the 2025 ACM Internet Measurement Conference, ACM, 2025, pp. 541–57. Crossref, doi:10.1145/3730567.3764471.
Kim T, Bock K, Luo C, Liswood A, Poroslay C, Wenger E. Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study. Proceedings of the 2025 ACM Internet Measurement Conference. ACM; 2025. p. 541–557.
Published In
Proceedings of the 2025 ACM Internet Measurement Conference
DOI
Publication Date
October 28, 2025
Start / End Page
541 / 557
Publisher
ACM
Conference Name
IMC '25:ACM Internet Measurement Conference