Why are the Aborigines afraid of white men in the novel, Rabbit-proof Fence?

The first white men who came did not treat the Aborigines well. They tricked the men by offering to take them to Green Island in their ships. The men wanted to go there to collect eggs, so happily agreed. The white men left the men on the island, then sailed back and stole the undefended women. They used the Aborigine women as sex slaves, then killed them. The Aborigines had thought that the men were spirits, and assume this about all the white people they see. However, they are quick to trust Major Lockyear and his men when they see that they are not hurting their people.