It is reasonable to say that it is not surprising that he was racist toward white people because of the discrimination that existed at the time. It is a totally different argument to say that he "wasn't very racist" because he associated with whites or that at the same time supported and voiced racist ideology.

By that reckoning one could argue that someone is not very racist to dislike black people because in their town more street crime is committed by black people than is proportionate to the population.

You can have one way of thinking or the other but not both.
You can perhaps argue that racism is understandable in certain circumstances but not acceptable, unless it is acceptable for all.
Fighting oppression, suppression or segregation does not have to involve racism. Non whites do all three to each other all over the world. The fact that most ruling class Americans happened to be white is insignificant in the grand scheme of things. Ali's slave ancestors were probably sold to white slave traders by black slave traders.
If we are going to uphold ideologies then you have to be be true to them no matter what or they lose credibility.

Racism is never acceptable, but it can be understandable and even inevitable. MLK should be celebrated for rising above that, but in doing so, he got a lot of people, white and black, killed, arrested and injured.

Also, if you experience crime at say the hands of a black person, I think its understandable to be affected by the racial aspect. Its not right, and ideally something that should be redressed, but its understandable. Same as women being 'cautious' around men, we as men should understand, that their experience of sexual politics can be very radically different, and the experience of harassment, sexual violence etc is very different.

We can only really try to overcome fear and prejudices by understanding why and how they manifest within us, and why other people may or may not fear us - collectively - and in doing so, resolve to personally try to set aside those misconceptions and deal with the problems that generate these stereotypes.

But its probably more significant that the 'dominant' social power group makes the most significant 'sacrifices' first and listen to the issues that create that prejudice against us; and try to resolve those issues. In doing so, its generally the case that prejudice on both sides is reduced.