Who are the "ghosts" in the poem, What Lips My Lips Have Kissed?

The "ghosts" to which the speaker refers in Line 4 symbolize the past, specifically the speaker's sensual past that she spent with her various lovers. The speaker says that the ghosts exist within the rain that strikes her window, cultivating a sense of death and haunting around her now-lonely bedroom that used to be full of vitality.