Without further immigration, population growth this decade is expected to be the lowest in U.S. history.