Doubly robust nearest neighbors in factor models [2211.14297]