I’m working on a proposal to rip out the existing PostgreSQL ILIKE/fuzzy_search based search in Lemmy (the federated Reddit clone) and replace it with Tantivy, a Rust full-text search library. The current implementation is basically WHERE title ILIKE '%query%' which obviously doesn’t scale and has zero relevance ranking. The proposal adds BM25 scoring, phrase queries, boolean operators, field-specific searches (author:name), tag filtering, and date ranges - all while keeping the existing API endpoint unchanged. I’d need to maintain a separate Tantivy index on disk, hook into all CRUD operations to keep it in sync, and build a query parser that translates user input into Tantivy’s query language. The alternative is just sticking with PostgreSQL’s built-in full-text search (tsvector/tsquery) which would be way less work to implement but wouldn’t give us the same level of advanced querying. Is rolling our own search index with Tantivy worth the complexity, or should we just use what Postgres already provides and call it a day?
way less work
Okay so do postgres tsvector then
Still an improvement
You probably still have other things you want to do, too
Also, the more advanced the feature is, reality is that the fewer users can make use of its full capability
But forget all that, if you feel like doing it anyway, then do it, this is FOSS
Make a PR or ask the devs. The search could be improved for sure.
Why is this post labeled AI? Who put that label there? OP or something automated or a mod?
your comment is 42 minutes old as i write this buy that tag doesnt show up for me
That’s a Piefed thing.
Because it was proofread using ai
So you added the label yourself. I think it’s way exaggerated; it would’ve been enough to put it in a PS. Maybe you’re getting too little exposure because of this. E.g. Piefed has an option to completely hide AI-generated content. Which doesn’t even apply to your post.
