Network Effect is the biggest hurdle for sure. I think it it true for so many other services too. I think we can agree there is no real technical problem to solve, we only look at the technical problems because trying to “fix” the social and political issues is a lot harder. Digital Markets Act is supposed to address this but time will tell if it has any lasting impact (in the EU).
Few weeks late to pitch in now but I can +1 docker-mailserver.
It has almost everything included and the configuration files are quite straightforward and flexible enough that you can drop little edits into the individual services if you need to tweak something.
My setup is very close to what you want: I use fetchmail to pull in from my old gmail and yahoo inboxes, I also have my own domain so I configured the MX records so that emails go straight to my server, with a fallback to my email provider (any mail that doesn’t make it directly to my mailserver will still get pulled into my inbox with fetchmail when it comes back online).
Docker-mailserver allows you to set the SMTP of your instance to use your provider. This is important because it means that they do all the reputation stuff so that your emails work properly (and both my home ISP and my VPS provider don’t do sending over pot 25 anyway).
So when I need to connect a new client (like Thunderbird) to my email I don’t need to manually config anything as docker-mailserver has all the auto config messages so its really seamless. At the same time my risk is low because even if my sever is off my provider will still receive anything on my behalf. I can only send using the username I have paid for from my provider and switching between gmail and yahoo is not possible without rewriting configs and restarting services but its not something I want anyway. On the receiving side I can have any number of aliased usernames that will all be received by my server (but only when its on so i use them rarely and for disposable addresses).
Big downsides are:
Most of my complaints stem from the fact that I’m not very good at this but in the end it has been very satisfying to drop the occasional: “I host my own email BTW”
Good luck! Let us know how you get along!