https://en.wikipedia.org/wiki/Private_Use_Areas
I came across a Python library that passed the ASCII range into one of these non printable character ranges and then into a database. If someone was doing that manually with a hex table, how is that detected and mitigated?
Whatever you’re trying to achieve: You’re decidedly approaching it from the wrong angle.
It’s not detected or mitigated, because it’s nothing bad. People can use whatever data formats or file formats they want. This is just another weird data format.
Globally we’ve agreed that the ASCII code for a space is 32, 65 for the letter A.
Unicode characters are also globally defined, so when someone uses an agreed upon code, everyone sees the same thing, like this grimace smiley 😁
A private area is a place that we’ve all agreed is for “private use”. If a trademark owner wants to use their special character in their documentation, they can define one area to represent their character, but the only people who will see it in the same way, are people who installed their particular font.
Anyone without that font would see whatever the font on their own machine displayed.
Putting random stuff in such a place is no more than putting gobbledygook in a text and it might even be used as a way to fingerprint text.
I’m not sure what you want to “detect” or “mitigate”.
I can’t work out what you’re asking.
You use “mitigated” like this is some kind of exploit but it’s just unicode text still.
What is the problem with private use areas of unicode?
It is non printing. It cannot be seen or scanned or highlighted. It looks like nothing, except the file size is large with more hex than should be in the binary.
I’m still not seeing why that is a problem. The information remains even if it has no glyphs.
It ought to look like a bunch of □, which is the glyph generally used to indicate that the font has nothing to represent the character.
Specifically you’d expect
U+25A1□ WHITE SQUAREAlso, the answer to your actual question is no. There’s definitely no way to block people from using any particular characters at the kernel level.
What you seem to be asking for is a way to absolutely forbid all software from writing certain characters to files, and/or from reading those characters. Aside from requiring that the kernel inspect all data in detail before letting other software have it, which would slow everything way down, it would prevent anyone from reading or writing binary data which happens to contain those sequences of bytes by coincidence. Binary data includes things like the programs which make the system work, so blocking those characters would be terminal
Also, (a) userspace could have some higher-level encoding or encryption or compression that happens without the kernel seeing the non-encoded data, and (b) whatever particular Unicode encoding OP is probably thinking of isn’t the only Unicode encoding out there.
That doesn’t, strictly-speaking, mean that it’s impossible to have kernel-level blocking — you could create some kind of emulated system that inspects everything, but it does mean that you couldn’t just inspect data at points where one normally enters the kernel.
The answer that is probably most useful to OP is that if it’s a problem for his application, he should validate it in userspace.
