Skip to main content

ON TRAJECTORY AUGMENTATIONS FOR OFF-POLICY EVALUATION

Publication ,  Conference
Gao, G; Gao, Q; Yang, X; Ju, S; Pajic, M; Chi, M
Published in: 12th International Conference on Learning Representations, ICLR 2024
January 1, 2024

In the realm of reinforcement learning (RL), off-policy evaluation (OPE) holds a pivotal position, especially in high-stake human-centric scenarios such as e-learning and healthcare. Applying OPE to these domains is often challenging with scarce and underrepresentative offline training trajectories. Data augmentation has been a successful technique to enrich training data. However, directly employing existing data augmentation methods to OPE may not be feasible, due to the Markovian nature within the offline trajectories and the desire for generalizability across diverse target policies. In this work, we propose an offline trajectory augmentation approach, named OAT, to specifically facilitate OPE in human-involved scenarios. We propose sub-trajectory mining to extract potentially valuable sub-trajectories from offline data, and diversify the behaviors within those sub-trajectories by varying coverage of the state-action space. Our work was empirically evaluated in a wide array of environments, encompassing both simulated scenarios and real-world domains like robotic control, healthcare, and e-learning, where the training trajectories include varying levels of coverage of the state-action space. By enhancing the performance of a variety of OPE methods, our work offers a promising path forward for tackling OPE challenges in situations where human-centric data may be limited or underrepresentative.

Duke Scholars

Published In

12th International Conference on Learning Representations, ICLR 2024

Publication Date

January 1, 2024
 

Citation

APA
Chicago
ICMJE
MLA
NLM
Gao, G., Gao, Q., Yang, X., Ju, S., Pajic, M., & Chi, M. (2024). ON TRAJECTORY AUGMENTATIONS FOR OFF-POLICY EVALUATION. In 12th International Conference on Learning Representations, ICLR 2024.
Gao, G., Q. Gao, X. Yang, S. Ju, M. Pajic, and M. Chi. “ON TRAJECTORY AUGMENTATIONS FOR OFF-POLICY EVALUATION.” In 12th International Conference on Learning Representations, ICLR 2024, 2024.
Gao G, Gao Q, Yang X, Ju S, Pajic M, Chi M. ON TRAJECTORY AUGMENTATIONS FOR OFF-POLICY EVALUATION. In: 12th International Conference on Learning Representations, ICLR 2024. 2024.
Gao, G., et al. “ON TRAJECTORY AUGMENTATIONS FOR OFF-POLICY EVALUATION.” 12th International Conference on Learning Representations, ICLR 2024, 2024.
Gao G, Gao Q, Yang X, Ju S, Pajic M, Chi M. ON TRAJECTORY AUGMENTATIONS FOR OFF-POLICY EVALUATION. 12th International Conference on Learning Representations, ICLR 2024. 2024.

Published In

12th International Conference on Learning Representations, ICLR 2024

Publication Date

January 1, 2024