I run a 5 bit quantized Qwen 3.8 27b at home on a gaming PC with an RX 7900 XTX and RAG memory using Gemma MoE (i forget what one)running on an old 1080 ti on my Valheim server.
It takes 3-15 minutes to run depending on how many tool calls the model makes and that’s mostly memory or web scrapes since that runs on the slower hardware. I mostly just use it as a discord bot to look stuff up for me and to fetch the news.
“I wrote it down and glued it to the screen and its not fixed. Do you even know how to do your job?”
Making up demon names is fun and whimsical. Neth’sin the Ethereal Tormentor likes chin scratches, will rip strangers to shreds, and sometimes defiles laundry.
Because Samsung likes knowing when the wash is done from across the world.
Too tight of a schedule to cast that counter spell.
Another citizen, a pastor named Rick Bick, rushed to the podium. “Let’s just reverse that curse that was just put on our city,” he said. “That ending with hail Satan and hail dark god. That’s not welcome here.”
“Mayor, that should be done at the end,” city councilor Cara Galloway said.
“It needs to be done now, OK.” Rick said. “I appreciate what Ms. Galloway is saying but that was a curse put on our city when he said ‘Hail Satan’ and the dark god and most of you know that. And we need to just stop and reverse it.”
“So what I’m gonna do now…I’m just gonna invite us into a 30 second time of prayer — time of silence.”
“No,” Galloway said. “We are at the consent agenda. I’m sorry, but it’s time to move forward. The invocation happened.”
“As mayor of this city…I’m going to ask right now that we do a moment of silence to reflect on whatever it is we need to reflect on at this time. And then we will move to the consent agenda,” mayor Benjamin Foote said.
I should start giving incantations, that reaction is gold.
What if they try to go past NPC speak and I have to talk about my cringe ass hobbies, then they find out I’m a weird loser?
You should put in some Disney lyrics so they issue a takedown request if it leaks.
Probably not, or if it runs it may take hours or days to get a full output. VRAM on the GPU really decides what will run reasonably. Try an 8 billion parameter phone sized model and see if its faster than sending a letter, then keep moving up until your setup can’t handle it.
8B models are a bit too dumb to use any tools reliably but should handle a basic conversation.
llama cpp will allow you to manually load a model and tweak settings.
unsloth will give you a basic setup with some tools, manage model downloads, and settings.