I worked as a freelance data scientist for 2 years, without pursuing a deep specialization in either NLP or CV. Now I'm thinking about shifting to software engineering. It is more interesting to me.BTW, I am a physician. Do you think this is a good idea?

Yes, I think it can be a good idea.

If you were already working as a data scientist for 2 years, then software engineering should not be some impossible jump. It is a shift, but not a reset. You already have technical problem-solving, coding exposure, and experience shipping work for real clients. The bigger question is not “am I allowed to switch?” but “which kind of software engineering do I actually want?” because SWE is very broad.

Also, being a physician is not a disadvantage here. If anything, it gives you a useful edge if you ever work in healthtech, medical products, clinical workflows, or healthcare data systems. I would not treat your lack of deep NLP/CV specialization as a problem either, especially if software engineering is genuinely more interesting to you. Interest matters, because this move will require some focused effort on engineering fundamentals: data structures, system design basics, testing, debugging, APIs, databases, and building production-style projects. Backend is probably the most natural entry point, but that depends on what kind of work you enjoy.

So yes, reasonable idea. I would just avoid doing it as a vague escape. Pick a target, build 2–3 solid projects that look like real software rather than notebooks, and see if you actually enjoy the engineering workflow enough to commit.