Twenty-five houses have been torn down in five years -- a big number for a town of just a couple thousand residents.