This study explores 12-month-olds' understanding of face-to-face conversation, a key contextual structure associated with engagement in a social interaction. Using a violation-of-expectations paradigm, we habituated infants to a “face-to-face” conversation, and in a test phase compared their looking times between “back-to-back” (conceptually novel) and “face-to-face” (conceptually familiar) conversations, while simultaneously manipulating perceptual familiarity in a 2 × 2 factorial design. We also analyzed dynamic changes in pupil dilation, which are considered a reliable measure of cognitive load that may index processing of social interactions. Infants looked relatively longer at perceptual changes (new speaker positions) but not at conceptual change (back-to-back conversation), suggesting that face-to-face conversation may not elicit particular expectations, and so may not carry any particular conceptual significance. Moreover, on the first test trial, larger pupil dilation was observed for familiar conditions, suggesting that familiarity with perceptual features could enhance processing of conversations. Thus, this study undermines assertions regarding infants' conceptual understanding of the social signals underlying engagement. Infants may rather recognize such signals through their perceptual familiarity and associated positive feelings. This may then increase their engagement when observing and participating in others' collaborative activities, in turn allowing for the development of knowledge regarding others' intentions.