The vast, vast majority of people don’t quit their job or their employer, but their boss and coworkers.
Don’t underestimate how much healthy relationships at work matter when you spend so much of your time there. Yes, in tech jobs as well. So stick with IT if you like it, but don’t stick around in a bad environment. Especially if you plan to have a family in X years, because then it gets a lot harder and riskier to jump ship and change your situation.
That’s like saying “black people can use the N-word (as an in-group term), so I can too (as an insult)”.
You really don’t see the difference?