Skip to main content

RIOT: I/O efficient numerical computing without SQL

Publication ,  Journal Article
Zhang, Y; Herodotou, H; Yang, J
Published in: CIDR 2009 - 4th Biennal Conference on Innovative Data Systems Research
December 1, 2009

R is a numerical computing environment that is widely popular for statistical data analysis. Like many such environments, R performs poorly for large datasets whose sizes exceed that of physical memory. We present our vision of RIOT (R with I/O Transparency), a system that makes R programs I/O-efficient in a way transparent to the users. We describe our experience with RIOT-DB, an initial prototype that uses a relational database system as a backend. Despite the overhead and inadequacy of generic database systems in handling array data and numerical computation, RIOT-DB significantly outperforms R in many large-data scenarios, thanks to a suite of high-level, inter-operation optimizations that integrate seamlessly into R. While many techniques in RIOT are inspired by databases (and, for RIOT-DB, realized by a database system), RIOT users are insulated from anything database related. Compared with previous approaches that require users to learn new languages and rewrite their programs to interface with a database, RIOT will, we believe, be easier to adopt by the majority of the R users.

Duke Scholars

Published In

CIDR 2009 - 4th Biennal Conference on Innovative Data Systems Research

Publication Date

December 1, 2009
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Zhang, Y., Herodotou, H., & Yang, J. (2009). RIOT: I/O efficient numerical computing without SQL. CIDR 2009 - 4th Biennal Conference on Innovative Data Systems Research.
Zhang, Y., H. Herodotou, and J. Yang. “RIOT: I/O efficient numerical computing without SQL.” CIDR 2009 - 4th Biennal Conference on Innovative Data Systems Research, December 1, 2009.
Zhang Y, Herodotou H, Yang J. RIOT: I/O efficient numerical computing without SQL. CIDR 2009 - 4th Biennal Conference on Innovative Data Systems Research. 2009 Dec 1;
Zhang, Y., et al. “RIOT: I/O efficient numerical computing without SQL.” CIDR 2009 - 4th Biennal Conference on Innovative Data Systems Research, Dec. 2009.
Zhang Y, Herodotou H, Yang J. RIOT: I/O efficient numerical computing without SQL. CIDR 2009 - 4th Biennal Conference on Innovative Data Systems Research. 2009 Dec 1;

Published In

CIDR 2009 - 4th Biennal Conference on Innovative Data Systems Research

Publication Date

December 1, 2009