Are there any risks or disadvantages to building software from source, compared to installing a package? Can it mess with my system in any way?
I usually avoid it because I’ve found it to be a faff and often doesn’t work anyway but in a couple of cases it has been necessary.
I thought the optimizations didn’t actually do they much.
They don’t in general, but things that do heavily detailed graphics work (like your compositor or browser) or lots of cryptography work on the CPU can get a bit more out of those newer instructions than many other programs.
Very approximately, things that Gentoo offers prebuilt versions of because compiling them is so resource intensive are often the things that can get the best benefit out of your architecture variant. (Not singling out Gentoo here as an example of “doing it badly” - they do the sensible thing by providing these prebuilt binaries, but in some ways it defeats the purpose of optimised source distributions.)
It’s a Hard Problem™ to solve.