Some Brexit supporters want the U.K. to seek economic opportunity in its former colonies to replace trade with the European Union, which has twice the economic activity of the ex-colonies.