I’m still sanguine that 1.58 BITNET models take off. Those could plausibly run at good clip on existing CPUs, no GPU needed.
Super basic medium article for those not in know
Necessity spite is usually a good driver…though given BITNET is Microsoft IP…ehh…I won’t hold my breath for too long. Still waiting for their 70B model to drop… maybe this year…
I get where you’re coming from (and why) but am of the “rip the bandaid off clean in one go” school of thought.
A smaller start (like using that RPI to self host Jellyfin server for your home) puts you on the road to sovereignty straight away. A Pi 4 costs what…$60 (plus $30 for power supply and SD card)? Hell, use an old laptop.
Once you have one thing running, your on the right path for the next and the next.
Doing it on the cloud I think is paradoxically harder and ultimately self defeating.
Don’t get me wrong - if you need the cloud (say, you need to rent a H100 for a few hours to fine tune your LLM), I’m all for it. But if sovereignty is to goal - and the gateway drug is a SBC and a few days / weeks of self learning…you may as well start eating the elephant. IMHO and YMMV of course.
The cloud? You mean someone else’s computer? 🤣
Yes, very possible. Cheaper even…but will require some elbow grease.
There’s a lot there. Feel free to skip to the security cameras section (linked above), look at the intro, watch videos etc. Rossman is GOATed for putting all this together. It’s written for the layman willing to roll their sleeves up.
Yep! I will mirror it here -
(Its empty rn / place holder only).
I had a bunch of prelim write-ups on r/LocalLLM and r/LocalLlama and r/homelab but they’re in the shadowrealm now due to reddit ban (fuck reddit)
I will also post it on @homelabs and @privacy here; I think my MoA design is worthwhile enough to maybe even merit a post on HackerNews…but I want to cross all T’s and dot all I’s before I get into that bar fight lol.
I’m exactly doing this atm. I’m running a homelab on a $200 USD lenovo p330 tiny with a Tesla P4 GPU, via Proxmox, CasaOS and various containers. I’m about 80% finished with what I want it to do.
Uses 40W at the wall (peak around 100W). IOW about the cost of a light bulb. Here’s what I run -
LXC 1: Media stack
Radarr, Sonarr, Sabnzdb, Jellyfin. Bye bye Netflix, D+ etc
LXC 2: Gaming stack
Emulation and PC gaming I like. Lots of fun indie titles, older games (GameCube, Wii, PS2). Stream from homelab to any TV in house via Sunshine / Moonlight. Bye bye Gforce now.
LXC 3: AI stack
Llama.cpp + llama-swap (AI back ends)
Qdrant server (document server)
Openwebui (front end)
Bespoke MoA system I designed (which I affectionately call my Mixture of Assholes, not agents) using python router and some clever tricks to make a self hosted AI that doesn’t scrape my shit and is fully auditble and non hallucinatory…which would otherwise be impossible with typical cloud “black box” approaches. I don’t want black box; I want glass box.
Bye bye ChatGPT.
LXC 4: Telecom stack
Vocechat (self hosted family chat replacement for WhatsApp / messenger),
Lemmy node (TBC).
Bye bye WhatsApp and Reddit
LXC 5: Security stack
Wireguard (own VPN). NPM (reverse proxy). Fail2Ban. PiHole (block ads).
LXC 6: Document stack
Immich (Google photos replacement), Joplin (Google keep), Snapdrop (Airdrop), Filedrop (Dropbox), SearXNG (Search engine).
Once I have everything tuned perfectly, I’m going to share everything on Github / Codeberg. I think the LLM stack alone is interesting enough to merit attention. Everyone makes big claims but I’ve got the data and method to prove it. I welcome others poking it.
Ultimately, people need to know how to do this, and I’m doing my best to document what I did so that someone could replicate and improve it. Make it easier for the next person. That’s the only way forward - together. Faster alone, further together and all that.
PS: It’s funny how far spite will take someone. I got into media servers after YouTube premium, Netflix etc jacked their prices up and baked in ads.
I got into lowendgaming when some PCMR midwit said “you can’t play that on your p.o.s. rig”. Wrong - I can and I did. It just needed know how, not “throw money at problem till it goes away”.
I got into self hosting LLM when ChatGPT kept being…ChatGPT. Wasting my time and money with its confident, smooth lies. No, unacceptable.
The final straw was when Reddit locked my account and shadow banned me for using different IP addresses while travelling / staying at different AirBNBs during holiday “for my safety”.
I had all the pieces there…but that was the final “fine…I’ll do it myself” Thanos moment.
They’re holding value remarkably well, sadly lol. By all rights they should be $50 or $60…but I decent ones in the $100+ range…which is about where they start to cross over with Switch OG and Switch lites.
At least the DS lite and DS XL are still reasonably priced. DS lite is an excellent form factor, I think…if only the screen was larger.
I hear you. Oddly enough, I seem to have gone the other way - I have more fun ekeing out the last erg of performance than actually playing sometimes.
Its a sickness, I admit :).
The tinkering mentality (roused from the dead thanks to emulation and retro gaming) has now led me into adjacent areas, like self host local LLM, media servers, coding etc. I credit the revival entirely to the delightful Wii (and the gentle introduction to modding provided by all the guides).
It really is a great system.
Second hand market. I saw a bunch yesterday in Akihabara for the $120-150 USD range…but for the life of me couldn’t find a decent 3DS LL
I had no issue making Wii Sports Resort with motion plus. Do you use the wii mote with it inbuilt or the dongle?
Also (and fwiw) some of the older versions of Dolphin seem to handle motion plus a bit more gracefully. I can’t recall if I played Sport resort with Ishii Dolphin or an older v5 version of mainline but yes, it for sure worked for me.
Just FYI; you can completely erase the pain point of this by using Playnite as front end (in windows) and configuring your emulators (Dolphin, PCSX2 etc) to start full screen WITH controller support started at run time, along with enabling hot key macros for shutting down game and returning to dashboard etc. Playnite has native scripting control.
Its glorious and once set, you’ll never need to use your keyboard for anything (unless you want to).
Arguably better than OG hardware due to shaders, upscaling etc. Dolphin especially is excellent for this (I found Ishiiruka branch a bit better for motion controls but YMMV)
I agree! Upgraded from a Wii to a Lenovo M93p tiny ($50 USD) so that I could emulate all my old games and play select PC games I loved from back in the day (Just Cause 2, GTA IV, Fallout 3 etc).
Then I went to m710q…which I broke by being a little to hasty swapping CPU :(
(Turned out to be a blessing in disguise… as the next tiny I got - a P330 - got me onto self hosting etc as well as gaming).
TL;DR: yeah, those “mini-micro-tiny” machines are pretty sweet and cheap.
The OG Wii
Reasons -
And all that for $20.
NPUs yes, TPUs no (or not yet). Rumour has it that Hailo is meant to be releasing a plug in NPU “soon” that accelerates LLM.