Ray's first wife was Ethel Fleming (1922–1961). He fell in love with her before he was eighteen and got a job to convince his father he could marry her. They had a daughter together and were seemingly happy until Ray got too obsessed with work and spent less and less time at home. They divorced when Ray fell out of love with her.