The regiment remained at the race course for several months, for drill and instruction.  In February, 1863, they were moved to the west end of James’ Island, near Secessionville, for guard and picket duty.  After this, they were transferred to Sullivan’s Island, and quartered in the old Moultrie House and cottages adjacent.  Four companies were ordered to Battery Marshall, on the east side of the Island, to assist in the management of the siege guns at that point.

On the 7th of May the Federal gunboats crossed the bar and made an attack upon Forts Sumter, Moultrie, and the batteries on Morris’ Island.  Here the regiment was subjected to a heavy cannonading from the three hundred pounders from the Federal ironclads.  Colonel Dansler, however, moved the regiment to the east, in the sandhills, thus avoiding the direct fire of the enemy.  One of the ironclads was sunk and others badly crippled, drawing off after dark.  In December eight companies were moved over to Mt.  Pleasant and two to Kinloch’s Landing.

During the memorable siege of Morris’ Island, the Twentieth did its turn at picketing on that island, going over after dark in a steamer and returning before day.

On the night of the 30th July, 1863, while the regiment was returning from Morris’ Island, the tide being low, the steamer Sumter, on which the regiment was being transported, was forced to take the main ocean channel.  It was the duty of those on garrison duty at Fort Sumter to signal Moultrie and the shore batteries of the movements of the transport steamer.  For some cause or other Sumter failed to give the signals, and Moultrie being aware that there was a steamer in the harbor and no signals up, opened upon the ill-fated steamer with all her guns, thinking it one of the enemy’s ironclads.  This was a signal for the shore batteries to open their guns, and in a few moments shells came crashing through the decks and cabins of the crowded steamer from all sides.  This created a panic among the troops, and had it not been for the self-possession and coolness of the captain of the steamer, the loss of life would have been appalling.  The captain turned his boat and beached it as soon as possible, not, however, before the men began leaping over the sides of the vessel in one grand pell-mell.  The dark waves of unknown death were below them, while the shells shrieked and burst through the steamer.  There was but little choice for the panic stricken men.  Fortunately the waters here were shallow enough for the men to touch bottom and wade out, some to Fort Johnson, some to Fort Sumter, while others remained in the shallows until relieved by small boats from shore.  The regiment lost sixteen men, either killed or drowned.

On the 16th or 18th of May, 1864, the regiment was ordered to Virginia, and reached Richmond about the twenty-second, and was ordered to join Kershaw’s Brigade, reaching it about the 28th of May, near South Anna River.