What is the difference of Toulon living "in China," and Gallimard living "with the Chinese?"

Toulon seems to have no personal connection with the native people. He doesn't appear to want that connection either, although he sees Gallimard as a man who gets along with and understands the people of China.