What Color? Poem

Observing the poem "What Color?" I find that the author contradicts himself at times or sometimes makes a few points that I find slightly confusing. The author talks about a pastor who has a "white soul". However, they then say that "His skin so black, they say, his skin so black in color." After that they again talk about how (he was) "on the inside snow." I find this point a little confusing though. I understand how white stands for peace (dove) and innocence (generally a blank slate), but I don't understand why the author is making it seem like the pastor on the inside is "white" and that there "wasn't one stain on his impeccable interior." I find the points that he is white on the inside and black on the outside very contradicting. Does the author mean to say that white always stands for good while as black does not? It seems to me that the author stresses the point that this pastor is technically black, but he really isn't since he is white on the inside. I'm not too sure but this poem to me sounds a little racist. Why would the author describe the pastor's soul as (not having) "one stain on his impeccable soul."? Does that mean that if black (or any color not white) stains get on the interior of this pastor, then he is no longer pure or noble? Does this mean that the pastor stands for all things good and evil at the same time? The part that sounds most contradicting to me is the part about "His skin was black, but with the purest soul, white as the snow..." since it should not matter what color the pastor's skin or soul is.