Lord Altham’s creditors, as has been stated, were very clamorous, and his brother Richard was practically a beggar: they were both sadly in want of money, and only one way remained to procure it. If the boy were out of the way, considerable sums might be raised by his lordship by the sale of reversions, in conjunction with the remainder-man in tail, who would in that case have been Lord Altham’s needy brother Richard. Consequently the real heir was removed to the house of one Kavanagh, where he was kept for several months closely confined, and in the meantime it was industriously given out that he was dead. The boy, however, found means to escape from his confinement, and, prowling up and down the streets, made the acquaintance of all the idle boys in Dublin. Any odd work which came in his way he readily performed; and although he was a butt for the gamins and an object of pity to the town’s-people, few thought of denying his identity or disputing his legitimacy. Far from being unknown, he became a conspicuous character in Dublin; and although, from his roaming proclivities, it was impossible to do much to help him, the citizens in the neighbourhood of the college were kindly disposed towards him, supplied him with food and a little money, and vented their abuse in unmeasured terms against his father.
In 1727 Lord Altham died in such poverty that it is recorded that he was buried at the public expense. After his death, his brother Richard seized all his papers and usurped the title. The real heir then seems to have been stirred out of his slavish life, and declaimed loudly against this usurpation of his rights, but his complaints were unavailing, and, although they provoked a certain clamour, did little to restore him to his honours. However, they reached his uncle, who resolved to put him out of the way. The first attempt to seize him proved a failure, although personally superintended by the uncle himself; but young Annesley was so frightened by it that he concealed himself from public observation, and thus gave grounds for a rumour—which was industriously circulated—that he was dead. Notwithstanding his caution, however, he was seized in March 1727, and conveyed on board a ship bound for Newcastle in America, and on his arrival there was sold as a slave to a planter named Drummond.