It is important to take what he says in context. I read what he said and I agree for the most part. He says over and over again he is against separation of races and isn't racist. I see it as he is searching for true constitution approved laws. And he states why it isn't a legal act. I agree to a point. I think it did more good than bad.
Read for yourself.
http://www.lewrockwell.com/paul/paul188.html
Is there something there I am missing that says because he is against it he is racist?
Just because I agree that KKK have the right to have hate parties in their own home doesn't mean I am racist. I don't think this makes him racist either, he could be, but I don't think his position on the issue does.