Task
I’m working on my final project for school, we are supposed to make a web app of our choosing and there has to be specific features in it. One of it is all data must be encrypted, and the other is that we have to have a search functionality. My app (A customer support framework) has a ticket functionality where customers can submit help request tickets, the contents of these tickets need to be encrypted at rest, at the same time admins need to be able to search contents of tickets.
Current Plan
My current plan is to store an AES-256 encrypted copy of the message message.content to meet the encrypted requirement, and also store a tokenized and hashed version of the message message.hashed to meet the searchability requirement.
The tokenization/hashing method will be:
- strip the message to alphanumeric + whitespace ([a-zA-Z0-9 ])
- tokenize by splitting the message by whitespace,
- SHA-256 each token,
- rejoin all the hashed tokens into a space seperated string and stored in the
message.hashedfield.
Thus this is a test string becomes <hash of this> <hash of is> <hash of a> <hash of test> <hash of string>
When the user searches their search string goes through all of the steps in the tokenization/hashing method, then we query the message table for message.hashed LIKE %%<hashed string>%% and if my thinking is right, we should be able to find it.
Concerns
- Statistical analysis of hashed tokens
- I really don’t see a way around this, to make the string searchable the hashing needs to be predictable.
message.hashedfield could potentially be huge, if each word is getting a SHA256 hash, a large message could result in a very large hash string- maybe we just store the last 4 of the hash?
- This would increase collisions, but the likelihood of multiple last 4’s colliding in a given search string should be pretty dang small, and any collisions would likely not be valid language.
- Would this help with the statistical analysis concern? Increasing collisions would decrease the effectiveness of statistical analysis. It would be a performance hit, but after returning all matches against the hashes I could decrypt the
message.contentdata and search the raw search query against the unencrypted text and remove any incorrect returns caused by collisions.
- maybe we just store the last 4 of the hash?
I’m interested in hearing everyone’s thoughts, am I being logical in my reasoning?
What you describe seems sound, but I’d say you need a final “decrypt and confirm” pass on each matched result, to work around the hash collisions, especially if you only store partial hashes, like you describe (which also seems sound).
Also, depending on your database implementation (whether it has good text-search-indexing support), it might make more sense to not recombine the hashed tokens, and instead store them in a 1-to-many mapping table.