We study the facial structure of the set ${cal E_{ntimes n$ of correlation matrices (i.e., the positive semidefinite matrices with diagonal entries equal to 1). In particular, we determine the possible dimensions for a face, as well as for a polyhedral face of ${cal E_{ntimes n$. It turns out that the spectrum of face dimensions is lacunary and that ${cal E_{ntimes n$ has polyhedral faces of dimension up to $approx sqrt {2n$. As an application, we describe in detail the faces of ${cal E_{4times 4$. We also discuss results related to optimization over ${cal E_{ntimes n$.