Here are some common reasons people in UX leave their current role for new opportunities:
This usually boils down to not having access to users to test with. For whatever bureaucratic reason, all the requirements come from business or engineers. The designer has no support to contact users to validate assumptions with testing exercises. This leads to a rather "defeatist" mindset of "we tried our best with what we had," but that often feels empty because understanding users is a core tenet of UX.
A designer who has never interacted with their user is a sad designer.
Additionally, lack of support can translate into lack of investment in tools, time to build and iterate on processes and space to learn and grow.
You might find yourself in a situation where all your coworkers think you are a graphic designer and see you as a "mockup machine." Maybe you are not being involved in discussions about requirements because "this isn't a visual matter." They don't understand your timelines or your requests for more information because they see some of the work in the UX process as "irrelevant to design."
Often, projects just get tossed over with a week deadline which leaves no room for any insightful research. If you are just churning out artifacts and deliverables in a vacuum you might start questioning what are you really there for? It is important for designers to feel like their work is meaningful and serves a purpose, not just to check a box.
In any career there are divas. If you have the bad luck of working with someone with a huge ego, then that just sucks. Pretty similar to other professions here... whether it is belittling your ideas, micromanaging, purposely excluding your from things, spreading personal drama, etc., it's pretty hard to deal with this in the workplace.
This could be at a level below you, a peer or even many levels above you. Negative influences from any co-worker can make your day-to-day worse.