Invisible Man Chapter 18
The narrator receives an anonymous note warning him to slow down or the very people who supported him would be the ones to cut him down. The note unnerves him, but he defies it. A few weeks later the narrator goes to a meeting where one of the black Brothers accuses him of using the Brotherhood to make himself important and attempting to become a tyrant by controlling the Harlem district.
The narrator is shocked and disappointed when the Brotherhood decides to investigate the ludicrous allegations. They ask him to work on issues of women's rights in a different part of the city or just step down from his position all together. The narrator chooses to leave Harlem until the investigation clears him of the accusations made against him.