Who is the antagonist in the novel, My Brother Michael?

The novel's antagonist is not a specific individual, but rather, the group of thieves and smugglers, who are somehow tied to the death of Simon's brother.