I've had this problem too - not wanting to abandon boys because I could see their goodness, and they didn't deserve to be abandoned!

But now I've realized, if the relationship isn't good for me, if it's stunting my growth, it can't be good for him either. So abandon them I did, and it was always a relief.