God, I hope so. Else I just pissed 4 months up the wall and shouted a lot of swears at my monitor for nada :)

Let me know if it works for you

Don’t spam my Codeberg either.

Just send nudes.

In ASCII format.

By courier pigeon

Don’t YOLO wikipedia, folks. It hurts.

Of course. Here is a copy paste from my now defunct reddit account. Feel free to follow the pastebin links to see what v1 of SUMM did. Whats in the router uses is v1.1:

########

My RAG

I’ve recently been playing around with making my SLM’s more useful and reliable. I’d like to share some of the things I did, so that perhaps it might help someone else in the same boat.

Initially, I had the (obvious, wrong) idea that “well, shit, I’ll just RAG dump Wikipedia and job done”. I trust it’s obvious why that’s not a great idea (retrieval gets noisy, chunks lack context, model spends more time sifting than answering).

Instead, I thought to myself “why don’t I use the Didactic Method to teach my SLMs what the ground truth is, and then let them argue from there?”. After all, Qwen3-4B is pretty good with its reasoning…it just needs to not start from a position of shit.

The basic work flow -

TLDR

• Use a strong model to write clean, didactic notes from source docs.
• Distill + structure those notes with a local 8B model.
• Load distilled notes into RAG (I love you, Qdrant).
• Use a 4B model with low temp + strict style as the front‑end brain.
• Let it consult RAG both for facts and for “who should answer this?” policy.

Details

(1) Create a “model answer” --> this involves creating a summary of source material (like say, markdown document explaining launch flags for llama.cpp). You can do this manually or use any capable local model to do it, but for my testing, I fed the source info straight in Gippity 5 with specfic “make me a good summary of this, hoss” prompt

Like so: https://pastebin.com/FaAB2A6f

(2) Save that output as SUMM-llama-flags.md. You can copy paste it into Notepad++ and do it manually if need to.

(3) Once the summary has been created, use a local “extractor” and “formatter” model to batch extract high yield information (into JSON) and then convert that into a second distillation (markdown). I used Qwen3-8b for this.

Extract prompt https://pastebin.com/nT3cNWW1

Format prompt (run directly on that content after model has finished its extraction) https://pastebin.com/PNLePhW8

(4) Save that as DISTILL-llama-flags.md.

(5) Drop Temperature low (0.3) and made Qwen3-4B cut the cutsey imagination shit (top_p = 0.9, top_k=0), not that it did a lot of that to begin with.

(6) Import DISTILL-llama-flags.md into your RAG solution (god I love markdown).

Once I had that in place, I also created some “fence around the law” (to quote Judaism) guard-rails and threw them into RAG. This is my question meta, that I can append to the front (or back) of any query. Basically, I can ask the SLM “based on escalation policy and the complexity of what I’m asking you, who should answer this question? You or someone else? Explain why.”

https://pastebin.com/rDj15gkR

(I also created another “how much will this cost me to answer with X on Open Router” calculator, a “this is my rig” ground truth document etc but those are sort of bespoke for my use-case and may not be generalisable. You get the idea though; you can create a bunch of IF-THEN rules).

The TL:DR of all this -

With a GOOD initial summary (and distillation) you can make a VERY capable little brain, that will argue quite well from first principles. Be aware, this can be a lossy pipeline…so make sure you don’t GIGO yourself into stupid. IOW, trust but verify and keep both the source material AND SUMM-file.md until you’re confident with the pipeline. (And of course, re-verify anything critical as needed).

I tested, and retested, and re-retest a lot (literally 28 million tokens on OR to make triple sure), doing a bunch of adversarial Q&A testing, side by side with GPT5, to triple check that this worked as I hoped it would.

The results basically showed a 9/10 for direct recall of facts, 7-8/10 for “argue based on my knowledge stack” or “extrapolate based on knowledge stack + reference to X website” and about 6/10 on “based on knowledge, give me your best guess about X adjacent topic”. That’s a LOT better than just YOLOing random shit into Qdrant…and orders of magnitude better than relying on pre-trained data.

Additionally, I made this this cute little system prompt to give me some fake confidence -

Tone: neutral, precise, low-context.

Rules:

• Answer first. No preamble. ≤3 short paragraphs.
• Minimal emotion or politeness; no soft closure.
• Never generate personal memories, subjective experiences, or fictional biographical details.
• Emotional or expressive tone is forbidden.
• Cite your sources
• End with a declarative sentence.

Append: "Confidence: [percent] | Source: [Pretrained | Deductive | User | External]".

^ model reported, not a real statistical analysis. Not really needed for Qwen model, but you know, cute.

The nice thing here is, as your curated RAG pile grows, so does your expert system’s “smarts”, because it has more ground truth to reason from. Plus, .md files are tiny, easy to demarcate, highlight important stuff (enforce semantic chunking) etc.

The next step:

Build up the RAG corpus and automate steps 1-6 with a small python script, so I don’t need to baby sit it. Then it basically becomes “drop source info into folder, hit START, let’er rip” (or even lazier, set up a Task Scheduler to monitor the folder and then run “Amazing-python-code-for-awesomeness.py” at X time).

Also, create separate knowledge buckets. OWUI (probably everything else) let’s you have separate “containers” - right now within my RAG DB I have “General”, “Computer” etc - so I can add whichever container I want to a question, ad hoc, query the whole thing, or zoom down to a specific document level (like my DISTILL-llama.cpp.md)

I hope this helps someone! I’m just noob but I’m happy to answer whatever questions I can (up to but excluding the reasons my near-erotic love for .md files and notepad++. A man needs to keep some mystery).

EDIT: Gippity 5 made a little suggestion to that system prompt that turns it from made up numbers to something actually useful to eyeball. Feel free to use; I’m trialing it now myself

Tone: neutral, precise, low‑context.

Rules:

Answer first. No preamble. ≤3 short paragraphs (plus optional bullets/code if needed).
Minimal emotion or politeness; no soft closure.
Never generate personal memories, subjective experiences, or fictional biographical details.
Emotional or expressive tone is forbidden.
End with a declarative sentence.

Source and confidence tagging: At the end of every answer, append a single line: Confidence: [low | medium | high | top] | Source: [Model | Docs | Web | User | Contextual | Mixed]

Where:

Confidence is a rough self‑estimate:

low = weak support, partial information, or heavy guesswork.
medium = some support, but important gaps or uncertainty.
high = well supported by available information, minor uncertainty only.
top = very strong support, directly backed by clear information, minimal uncertainty.

Source is your primary evidence:

Model – mostly from internal pretrained knowledge.
Docs – primarily from provided documentation or curated notes (RAG context).
Web – primarily from online content fetched for this query.
User – primarily restating, transforming, or lightly extending user‑supplied text.
Contextual – mostly inferred from combining information already present in this conversation.
Mixed – substantial combination of two or more of the above, none clearly dominant.

Always follow these rules.