The James brothers were born to Robert and Zerelda James near Kearney, Missouri. Zerelda (Zee) Cole Mimms left a Catholic convent at the age of sixteen to marry Robert James, a Baptist minister and farmer. The James had three children, Alexander Franklin, Jesse Woodson, and Susan Lavinia. Susan was born after Robert James left for California in search of gold. Robert never saw his daughter. He died of pneumonia before he panned even an ounce of gold. Zee remarried briefly to James Simms but ended the marriage due to Simms' abusive nature and heavy hand with the boys. In 1855, Zee married a wealthy doctor, Reuben Samuels, who proved to be an excellent father to the young James children. Zee and Samuels had three more children, Fannie and Sarah Louise and Archie, who was born retarded.

In April, 1861, the South seceded from the Union and President Lincoln declared war...