I get what you're saying, and I agree that there’s definitely a weird disconnect between how fans perceive Negan and what he actually did, especially when it comes to the Sanctuary and the whole "wives" situation. You're absolutely right to point out that coercion is coercion, and calling it anything else doesn't erase the moral problem. But I’d argue that instead of glossing over it, we can look at it as part of what makes Negan such an interesting and flawed character, precisely because of his hypocrisy.
Yes, he says "rape is bad", but he also created a system where consent was murky at best. That contradiction doesn’t excuse his behavior, but it does show that he THINKS he’s doing something moral, even when he's not. That’s not a free pass, it’s character depth. It’s like watching someone try to impose order on chaos, justify their power through rules, and still completely miss the moral core of what they’re doing. That’s fascinating in a storytelling sense. It doesn’t make him a good person, but it makes him a great character.
And when it comes to redemption, the show doesn’t just hand it to him. He’s forced to reckon with his past over and over. He’s isolated. Hated. He has to face Maggie, who literally represents one of the worst things he’s done. His journey isn’t about "he’s hot now and we forgive him". It’s about whether someone who did unforgivable things can change. And even then, the show leaves a lot of that up to interpretation, some characters forgive him, others never do, and the audience is to this day put in a place where we’re supposed to wrestle with it.
So no, I don’t think fans should ignore what he did or rewrite it just because he has charisma. But I do think the writers intended for him to be morally complex, even uncomfortable. He’s a former villain trying (and often failing) to live with what he’s done, and that, to me, is far more compelling than if he had been a black-and-white bad guy.