Segmentation of neonatal and fetal brain MR images is a challenging task due to vast differences in shape and appearance across age and across subjects. Expert priors for atlas-based segmentation are often only available for a subset of the population, leading to a reduction in accuracy for images dissimilar from the atlas set. To alleviate the effects of limited prior information on atlas-based segmentation, we present a novel semi-supervised learning framework where labels are propagated among both atlas and test images while modelling the confidence of propagated information. The method relies on a voxel-wise graph interconnecting similar regions in all images based on a patch similarity measure. By iteratively allowing information flow from voxels with high confidence to voxels with lower confidence, segmentations in test images with low similarity to the atlas set can be improved. The method was evaluated on 70 fetal brain MR images of subjects at 22–38 weeks gestational age. Particularly for test populations dissimilar from the atlas population, the proposed method outperformed state-of-the-art patchbased segmentation.