Social fabric is more frayed in places that voted to leave the EU