Are there any risks or disadvantages to building software from source, compared to installing a package? Can it mess with my system in any way?
I usually avoid it because I’ve found it to be a faff and often doesn’t work anyway but in a couple of cases it has been necessary.
Well, compiling from source is the “installing dodgy freeware .exe” of the Linux world. You have to trust whoever is distributing that particular version of the source code, and ideally vet it yourself. When installing a binary package from your distro’s repositories, presumably someone else did the vetting for you already. Another slight risk is that technically you are running some extra build scripts before you can even run the application, which is a slight security risk.
Yeah, unless you take precautions and compile in a container or at least a sandbox, the build scripts have complete unadulterated access to your user account, which is pretty much game over if they turn out to be malicious (see: https://xkcd.com/1200). Hopefully most FOSS software is not malicious, but it’s still a risk.
If you “install” the software on your system, it also becomes difficult to uninstall or update, because those files are no longer managed from any centralized location.
I recommend using a source-based package manager, and package your software with it (typically won’t be any more difficult than just building from source) to mitigate all of those (as typically source-based PMs will use sandboxing and keep track of the installed files for you).