Idk what trans man needs to hear this but you're NOT evil or disgusting for being a man. You do NOT have to suffer for the sins of the patriarchy committed by cis dudes. Being a man doesn't invalidate the misogyny you experienced growing up or experience now. Being a man doesn't mean you deserve to be isolated. Being a man doesn't mean you're inherently predatory or scary. You didn't "choose" this, and finding your true self is NOT "betraying the community" because you happen to be a man and/or masculine rather than a woman and/or feminine. You ARE allowed to be upset when people "affirm" your gender by malgendering you.
You DO deserve a community that uplifts you. You DO deserve to experience trans joy. You DO deserve to have your voices heard and your struggles recognized. Wanting the bare minimum of solidarity is NOT "making everything about trans men".