What does the speaker realize while spending time in Peter's grandmother's home in the poem, Nothing?

While shoveling, the speaker reflects on the passage of time, noting how he and Peter are standing in the same place that Peter's grandmother arrived after fleeing Nazi Germany.