Where did the narrator live growing up? At what point did his family move?

The narrator lived in two houses growing up. The first he lived in for seven years, from when he was five until he was twelve. That house was knocked down, and a second house was built at the bottom of the garden.