I host my mail server on a VPS.

I suggest making sure you get DMARC / DKIM / SPF working, and having an anti-spam strategy (greylisting helps, but there are a few ASNs that just exist to send spam). Also make sure your IP is not on any public spam list.

The next problem you might face is that Microsoft and especially Google like to make it hard for anyone not using their services. With Microsoft, you fill in a form and jump through some hoops and they’ll start accepting your email enough to land it in spam. Unless you are regularly sending to Microsoft, it is hard to keep them accepting mail, but just sending to a free Hotmail address (owned and occasionally marked as read and deleted by you!) on cron is enough to keep occasional mail deliverable as long as none of your mail ever gets marked as spam. Google can be more of a pain to small email servers in terms of not landing in spam, but I think occasional reports of not spam will help you.

In terms of keeping down spam:

• postgrey or similar for greylisting keeps out the least serious spammers.
• The notorious spammers / bulletproof hosting is best blocked by ASN since they regularly shift IP addresses. Try a script like this on daily cron (assuming you jump to the custom BAD_AS table from your INPUT iptables rule) - please don’t run it too often since routeviews is a free public service and you should be respectful of them:

#!/bin/bash -e

TEMPDIR=$(mktemp -d)
trap 'rm -r "$TEMPDIR"' EXIT

curl https://archive.routeviews.org/oix-route-views/oix-full-snapshot-latest.dat.bz2 -Lo "$TEMPDIR/snapshot.bz2"
bzgrep -e " (15828|213035|400377|399471|210654|46573|211252|62904|135542|132372|36352|209641|7552|36352|12876|53667|138608|150393|60781|138607) i" $TEMPDIR/snapshot.bz2 | cut -d" " -f 3 | sort | uniq > $TEMPDIR/badranges

iptables -N BAD_AS || true
iptables -D INPUT -j BAD_AS || true
iptables -A INPUT -j BAD_AS
iptables -F BAD_AS

for ROUTE in $(cat "$TEMPDIR/badranges"); do
    iptables -A BAD_AS -s $ROUTE -j DROP;
done

• Despite Google being so hostile to very infrequent emails from IPs that have years of never sending spam, just because they are small, Gmail and Firebase are one of the most significant spam sources. I find client-side filtering works best for things like that which get through your other defences.
• Another spam source is Docusign. These types of companies tend to shut down individual scammer / spammer accounts, but then allow them back in for the same scam with another account.

Note that of the spam that gets through if you have the basic defences, it’s probably a similar level to big corporate hosted mail, so don’t let this deter you (I just hate spammers).