https://en.wikipedia.org/wiki/Private_Use_Areas
I came across a Python library that passed the ASCII range into one of these non printable character ranges and then into a database. If someone was doing that manually with a hex table, how is that detected and mitigated?
Also, the answer to your actual question is no. There’s definitely no way to block people from using any particular characters at the kernel level.
What you seem to be asking for is a way to absolutely forbid all software from writing certain characters to files, and/or from reading those characters. Aside from requiring that the kernel inspect all data in detail before letting other software have it, which would slow everything way down, it would prevent anyone from reading or writing binary data which happens to contain those sequences of bytes by coincidence. Binary data includes things like the programs which make the system work, so blocking those characters would be terminal
Also, (a) userspace could have some higher-level encoding or encryption or compression that happens without the kernel seeing the non-encoded data, and (b) whatever particular Unicode encoding OP is probably thinking of isn’t the only Unicode encoding out there.
That doesn’t, strictly-speaking, mean that it’s impossible to have kernel-level blocking — you could create some kind of emulated system that inspects everything, but it does mean that you couldn’t just inspect data at points where one normally enters the kernel.
The answer that is probably most useful to OP is that if it’s a problem for his application, he should validate it in userspace.