If all that is true, and I'm in no position to say that it isn't, then what's wrong with some empathy? What's wrong with acknowledging phrases like "those people" and "you people" are coded specifically to make others feel less than? The purpose is derogatory. How would such a phrase make one of your immigrant friends feel if it were directed at them?
As Eskimo said, this isn't about poppies and it's not about "white[s]...speaking freely". It's about giving a shit about other people and how words (yours, mine, and Don Cherry's) have an effect.