Most on the left - and some on the right - play the race card in their discussions about immigration issues in an attempt to cow their opponents and avoid a real discussion about immigration matters. However, in the vast majority of cases those card players are engaging in the logical fallacy of Guilt by Association: if Matt likes cabbage, and Matt is a bad person, that doesn't mean that cabbage is itself bad.
In a similar fashion, just because some people who oppose various kinds of immigration are racially motivated doesn't imply that opposition to those kinds of immigration is per se racist. In fact, all you need to do is find one person who opposes that kind of immigration, and who doesn't do so out of racist grounds, for their entire argument to fall to pieces. Thus, you can ask the person making this argument whether they believe that every single person opposed to illegal immigration is a racist. When they say "no", you can inform them that they've been engaging in a logical fallacy.
It's a different argument for someone to point out that a particular person is racially motivated, but that's not how almost all who play this card operate. They simply broadly state that they think that opposition to illegal immigration or massive immigration is racist and then think that that should end debate. In many cases it does, because many people are cowed by such an argument. However, as more and more people learn everything involved in this issue, and as more and more people tire of corrupt elites making the same fallacious argument over and over, this argument has become less and less effective.