How is it bloated if you decide what to put on? Gentoo isn’t a traditional distribution, its what you make it to. Still you have to answer for yourself why you even want to switch to Gentoo. And if all the extra work for compilation is worth it. Do you even want to compile everything and customize the compilation process? If not, maybe Gentoo is not for you. That’s the thing. Only you can answer that.