I am always interested in the potential of LLMs, and specifically free, private LLMs. That's why, in this blog post, I will share my experience trying the local LLM Qwen 3.8 with LM Studio on 

• GPU: NVIDIA GeForce RTX 4060 with 8GB Dedicated GPU Memory

• Processor: 13th Gen Intel(R) Core(TM) i7-13700H

• Installed RAM: 16 GB (15.7 GB usable)

My first intuition was that I would use the quantized version that would allow the whole model to load and fit on the GPU memory, so that I avoid reading back from or transferring data back and forth from CPU to GPU and vice versa, since that will kill the tokens/s. 

source: unsloth/Qwen3.8-27B-GGUF · Hugging Face

I decided to go initially with UD-IQ1_M (6.73 GB), and with the rest of the 8GB, I would increase the context until it fully utilized the whole 8GB VRAM. This accounted for an 8192 context window. And with that, I could reach a speed of 19 tokens/s.

LM Studio chat window
 

LM Studio Settings with GPU offload full

Now, Testing time

Trial 1 (Thinking not enabled)

Trivial mistake in a popular problem-solving question with lots of online judges: Product Except Self

The mistake was very trivial: we should start looping from 0, not 1, in the first loop only.

He didn't figure that out and hallucinated, along with providing the wrong answer, but at least the answer was instant. XD

Trial 2 (Thinking  enabled)

We will try the same question as in Trial 1, but with thinking enabled. 

His Thinking Consumed My Small Context Quickly

However, what about his thinking? 

Let's quickly analyze a small part of his thinking, ignoring the hallucination, since it's still just thinking, so we can't judge it on that aspect.

From what I noticed, he was close to figuring out the mistake (see the highlighted yellow text below)

Beginning of his thinking (I truncated the middle, and some of the beginning and ending):

Let me look at code more carefully:

• resinitialized to[1]*(len(nums)+1)— length len+1, filled with 1s..

• prefix loop: for i in range(1, len(nums)-1): so i from 1 to len-2 (since range(1, len-1) means i values 1..len-2). Waitrange(1,len(nums)-1)gives start=1, stop=len(nums)-1, so i takes values 1,2,...,len-2. So iterates over indices 1 to len-2 (inclusive? range stops before len-1, so last value is len-2). For each i: prefix *= nums[i]; res[i+1] = prefix..

So for i=1: prefix = nums[0]*nums[1]? Wait initial prefix=1, then prefix = nums[1] → prefix = nums[1]. res[2] = nums[1]. Hmm wait no: prefix starts at 1, first iteration i=1: prefix = nums[1] → prefix = nums[1]; res[i+1]=res[2] = nums[1]. Second iteration i=2: prefix = nums[2] → prefix = nums[1]nums[2]; res[3] = product of nums[1],nums[2]. So res[k] for k>=2 holds cumulative product starting from index 1? Actually res[i+1] after first iter is res[2]=nums[1], second res[3]=nums[1]*nums[2], etc. So res[j] = product of nums[1..j-1]? Let's see: at iteration i, res[i+1] = prefix where prefix = product of nums[1..i]. So res[k] for k>=2 equals product of nums[1..k-1]. Hmm interesting.

Then postfix loop: for i in range(len(nums)-1,0,-1) → i from len-1 down to 1 (step -1). For each i: postfix = nums[i]; then res[i-1] = postfix. So this multiplies into res positions... Let's see: first iteration i=len-1: postfix = nums[len-1]; res[len-2] = that → res[len-2] = 1 nums[len-1]. Next i=len-2: postfix = nums[len-1]*nums[len-2]; res[i-1]=res[len-3] *= product. So res[k] for k<=len-2 holds suffix products? Hmm.

This is getting complicated; maybe the user just wants a general review of what's wrong with their code. The question: "can you tell me whats wrong in here" — they want me to identify bugs/bugs. I should point out:

1. Syntax error: missing closing parenthesis on return line: return res[:len(res)-1 → should be return res[:len(res)-1]? Wait, slicing syntax: res[:len(res)-1] — but that's not valid; they wrote return res[:len(res)-1 without the closing bracket. Actually in code it's return res[:len(res)-1 — missing ). So syntax error..

2. Logic errors:.

◦ The loops are off-by-one / wrong ranges. For prefix loop, they want to compute something but range(1, len(nums)-1) misses the last element? Let's see: if we want res of length n+1 where res[0]=1 and res[i] = product of nums[0..i-1]? Hmm..

.

At the end of his thinking, he tends to keep repeating, as if he got stuck in an infinite loop the EOS token never gets generated.

Thinking Repetition 

So the conclusion is that he didn't reach the answer because the context window was limited, but rather because there was another issue, either caused by poor accuracy due to quantization or inference settings. Let's change to a smaller math problem that doesn't need as much accuracy, unlike programming with variables and complexity.

Trial 3 (Thinking  enabled)

Simple Math Problem

Even though it reached the answer very quickly, his thinking kept repeating until all the context was consumed again. So now it's probably not about the accuracy or the quantization, since it appears that he is capable of answering short, simple questions, which can be utilized in many ways, such as classification. So even though it appears he's not useful for agentic coding, he could be used for other small, specified purposes.

Trial 4 (Thinking  enabled)

Let's try to limit thinking tokens to 2000.  

As is obvious, that was not a good idea at al. his repetition started to become more frequent and incomplete.

Also, sometimes specifying the prompt more helps him finish, like below, but in the programming question that did not work. 

Now, with Recommended Unsloth Settings 

Note that Repetition penalty default value was 1.1 so it was higher than the one that Unsloth recommend however the temperature in thinking mode was 0.7 not 1 like what Unsloth recommend so let's try this

With the Unsloth recommended settings, we faced the same issues documented above: repetition in thinking, despite giving it the largest context window of 70k tokens along with trimming the middle if the context exceeded it.

Trying 4-bit quantization 

We tried running 4 bit quantization and offloading the rest of the model size into RAM and computation onto CPU cores, and as expected, the speed dropped to 2 tokens/s. That was very slow, but the accuracy was noticeably better than 1-bit.

Conclusion

1. The reasoning was repetitive and redundant for small to mid problems.

2. That repetitive thinking quickly consumed my entire context window, which was already limited.

3. The generation speed on an 8 GB VRAM RTX 4060 laptop with 16 GB of system RAM was consistently around 20 tokens/s.

4. Although GPU offloading was set to 64/64 (meaning all layers were kept on the GPU to maximize token throughput), the cache and local server overhead still occupied several gigabytes of system RAM. This caused other applications like VS Code to be very slow.

It appears that, for now, limited free tier cloud models easily outperform extreme low bit local models on average laptops for coding problems or generic usage; however, the local LLM could be used to classify small words automatically and quickly. What I mean is that it depends on developers' creativity in how they utilize these small models. Also there might be hope in 4 bit quantization with bigger Uniform memory or VRAM.