Why do people think racism has to do with hating your skin color?

Your correct, I view a black guy totally different then an African guy. Even tho they can be the exact same complexion.