Maternal pregnancy nutrition influences fetal growth. Evidence is limited, however, on the relationship of maternal diet during pregnancy and lactation on infant postnatal growth and adiposity. Our purpose was to examine associations between maternal diet quality during pregnancy and lactation with offspring growth and body composition from birth to six months. Maternal diet quality was serially assessed in pregnancy and at one and three months postpartum, using the Healthy Eating Index(-)2015 in a cohort of 354 fully breastfeeding mother(-)infant dyads. Infant length-for-age (LAZ), weight-for-age (WAZ), and weight-for-length (WLZ) Z-scores were assessed at birth, one, three, and six months. Infant body fat percent (BF%), fat mass (FM), and fat-free mass (FFM) were measured at six months using dual-energy X-ray absorptiometry. Higher maternal diet quality from pregnancy through three months postpartum was associated with lower infant WLZ from birth to six months (p = 0.02) and BF% at six months (p = 0.05). Higher maternal diet quality at one and three months postpartum was also associated with lower infant FM at six months (p < 0.01). In summary, maternal diet quality during pregnancy and lactation was inversely associated with infant relative weight and adiposity in early postnatal life. Additional research is needed to explore whether associations persist across the life course.