Why does the speaker urge his mistress to set aside her gems in the poem, To His Mistress Going to Bed?

Gems are external ornaments, used, as the speaker insists, to distract from the pure glories of a woman's beauty. The speaker urges his mistress to set these ornaments aside, focusing instead on the glory of the exposed body.