All you gotta do is put workbenches down. They dont spawn inside the build area they create. I make a big ass fence around my base and cover the whole thing in workbenches. Benches get in the way so I hide them by digging a small hole and putting floor on top. Make sure they are inside the perimeter, greydwarves usually seek the workbenches out first. Eventually you can layer on the fence defense by adding spikes around the outside of the fence and/or digging a trench around the outside so they get stuck and you dont have to keep repairing. I just make little hanging outcroppings near the gates so its easier for me to jump across or complete a bridge temporarily to get animals across if I need to.

This also helps with some raids. The raid enemies will spawn outside the base area. Beware that advanced raids involve flying enemies, otherwise they usually shoot so if you idle too much your fence will probably get destroyed anyway. Still, it is nice to have, but tedious to set up. I usually make this huge and close to water so I can have a nearby dock for hauling metals and a portal hub tower in the main base for everything else. Workbenches have a limited height so if you make towers hide another on the upper floors too.