In multi-group learning, a single predictor must perform nearly as well as the best hypothesis in a class on every group in a rich, possibly infinite and overlapping, collection of subpopulations. The known algorithms with the optimal sample complexity keep track of every pair of a group and a hypothesis, which is impossible for the large classes of interest, and the known oracle-based algorithms have a worse rate. We ask whether the optimal rate can be attained by a learner that accesses the classes only through an empirical risk minimization (ERM) oracle. We give an algorithm that uses
O~((dG+dH)/ε2) samples and polynomially many calls to a weighted ERM oracle over pairs of a group and a hypothesis, where
dG and
dH are the VC dimensions of the group and hypothesis classes and
ε bounds the excess risk on every group, weighted by its probability. We also show a matching lower bound of
Ω((dG+dH)/ε2) samples for every learner even with unlimited computation, so oracle efficiency costs no additional samples. We also consider a natural special case in which every input belongs to at most
k groups and these groups can be listed. In this case, an ERM oracle for the hypothesis class alone suffices, with
O~((dH+logk)/ε2) samples in the agnostic setting and
O~((dH+logk)/ε) samples when every group has a hypothesis that is correct on it.