Studies based on single cells have revealed vast cellular heterogeneity in stem cell and progenitor compartments, suggesting continuous differentiation trajectories with intermixing of cells at various states of lineage commitment and notable degrees of plasticity during organogenesis1–5. The hepato-pancreato-biliary organ system relies on a small endoderm progenitor compartment that gives rise to a variety of different adult tissues, including the liver, pancreas, gall bladder and extra-hepatic bile ducts6,7. Experimental manipulation of various developmental signals in the mouse embryo has underscored important cellular plasticity in this embryonic territory6. This is reflected in the existence of human genetic syndromes as well as congenital malformations featuring multi-organ phenotypes in liver, pancreas and gall bladder6. Nevertheless, the precise lineage hierarchy and succession of events leading to the segregation of an endoderm progenitor compartment into hepatic, biliary and pancreatic structures have not yet been established. Here we combine computational modelling approaches with genetic lineage tracing to accurately reconstruct the hepato-pancreato-biliary lineage tree. We show that a multipotent progenitor subpopulation persists in the pancreato-biliary organ rudiment, contributing cells not only to the pancreas and gall bladder but also to the liver. Moreover, using single-cell RNA sequencing and functional experiments we define a specialized niche that supports this subpopulation in a multipotent state for an extended time during development. Together these findings indicate sustained plasticity underlying hepato-pancreato-biliary development that might also explain the rapid expansion of the liver while attenuating pancreato-biliary growth.