I use MSTY Studio which is a GUI front-end to an embedded Ollama server. Check it out here: https://msty.ai/ It's basically an alternative to things like LMStudio and AnythingLLM, but has its own version of OpenClaw and can also support LLaMa.cpp and has an Agent Mode. Supports local and online models.
Yes, it's a cross-platform GUI with an embedded Ollama server which it uses as its own local server accessible by any tool that uses Ollama (but on a different server endpoint: http://localhost:11964).
You can also use Huggingface GGUF models as well as Ollama models in addition to any online model from any provider. I use Kimi K2.7 from Nvidia's NIM Developer program for free.
And it has an "experimental" LLaMa server setup.
They also have a "Vibe CLI Proxy" they describe as:
"Connect command-line model providers to use their AI models
Integrate models from supported CLI applications directly into Msty Studio. This integration supports multiple CLI providers and makes their models available for conversations.
Supported CLI Providers
Claude Code
Codex
Google Gemini
iFlow
Github Copilot
Antigravity"
I haven't used that yet.
They also have Agent Mode:
"Agent Mode
Run coding agents in Msty Studio Desktop
Agent Mode lets you run coding agents inside Msty Studio Desktop. You can chat with an agent, review plans, inspect diffs, and approve actions in one workspace-aware flow."
It's main features are a comprehensive set of chat functions such as chatting with multiple LLMs at once, splitting chats, etc. They have four types of chat conversations:
"Normal Conversations (Conversations): Best for everyday chatting, drafting, Q&A, and iterative work with one model.
Persona Conversations (Persona Conversations): Best when you want a conversation guided by a specific role, voice, or expertise.
Crew Mode Conversations (Crew Conversations): Best when you want multiple personas to respond to the same prompt so you can compare perspectives.
Agent Mode Conversations (Agent Mode): Best for multi-step tasks where an agent plans, uses tools, and executes with less manual steering."
They also have RAG callled "Knowledge Stacks":
Knowledge Stacks are Msty Studio's take on Retrieval-Augmented Generation (RAG), designed to help you create, manage, and use collections of knowledge in a structured, flexible way. They let you organize your data for easy access in conversations, proving models with deep, domain-specific understanding.
They also have "Tolboxes" which is their version of MCP:
"The Toolbox allows for seamless communication with tools installed on your local machine or a remote one. These tools can interact with various data sources, perform file operations, and connect to repositories like GitHub, further empowering Msty Studio. This is Msty Studio's implementation of Model Context Protocol (MCP)."
Msty Studio supports local MCP server that run on your machine as well as remote MCP servers that can be connect to over Streamable HTTP.
Plus they have MSTY Claw which is their version of OpenClaw:
Nice article, thank you very much! One thing i'am missing is a greater view on security. I personally would put the coding instance and project files in a virtual machine and remote connect to it. So agents are clearly disconnected from private data and vulnerabilities are sandboxed to some temporary/check pointed system. However keeping your coding projects / packages up to date requires agent web access features that can bring the risk of prompt injection. Do you have experience with safety guards for this, like building a extraction pipeline that omits such injections?
Thanks. And yes, I mentioned this sandboxing throughout the article. I my case I run them on a separate machine. You may also turn off web access but that limits capabilities severely. I think the sweet spot is doing an audit of the harness code base (as I mentioned in this article) and start using it in a separate (or at least virtual) machine with no auto-permissions. I.e., without auto-permissions, it will ask you each time which command to run before it can execute it.
You are totally right, my comment reads like you didn't discussed it at all. Sorry!
From my understanding, command execution and file read / write are separated from each other in general. So one can use agents to automatically implement a bunch of features without requiring permission for each single file edit, but not use yolo mode, where commands also are executed automatically.
When putting the code base and coding agent into a VM and enable the sandbox mode (as you described) with no automatic command execution, this is considered as "safe".
But are there best practices to handle prompt injections hitting the agent at all? Like allowed / disallowed domains for the agent, some kind of converting pipelines (html to markdown) etc.?
We use Pi with Deepseek running on DS4, and Geodesia.ai as safety layer.
Our developers started tweaking a local stack after the repeated outages and slowdown of Claude and Codex. They are afraid to be cut off one day. They showed me the Stanford survey showing how open weight models are really close in capabilities to the big AI labs, so we assigned time, budget and focus on building it.
Thanks for sharing. Yeah, DS4 is great. It’s tad more expensive to run, but GLM 5.2 is even better. Apparently better than GPT 5.5 and Claude Mythos level.
@Sebastian Raschka, PhD Local coding agents are becoming the default workflow. Better privacy, lower latency, and full control over the development environment—great overview.
Great article, thanks for this and all the LLM architecture material you produce Sebastian 🙏🏼
Prompted me to switch to Qwen Code (I've been using OpenCode) with Qwen3.6 35b MoE as the model, and I have to say, so far it seems faster than OC. Specifically, it seems to generate less and more structured thinking output, so spends less time thinking.
Thank you, Sebastian, for this exceptional article.
Beyond the technical breakdown, this approach to local AI offers a wonderful bridge between modern agentic engineering and sysadmin culture, with that invaluable 80s open-world spirit running in the background: remaining master of your own machine and sovereign over your data.
A truly delightful and rigorous read, thanks for sharing!
Really nice article! Thank you for sharing. I really like the benchmarking example repo you published. It is often really hard to come out with any kind of evals or at least with useful structure for evals.
I was wondering, since Ollama quantizes everything heavily, does it degrade quality by a lot? I am curious how you think about trade off or quality vs. quantization.
Thanks, and good question. Regarding the benchmarks, there are also some personal ones I have based on concrete projects that I can’t share in their current form, but in general, I suggest occasionally add or collect some of the tasks you come across so that you have them in ready when comparing models.
Regarding the quantization, yeah, there’s unfortunately a bit of a performance degradation. But looking at the default quantized model vs the original unqantized bf16 variant (qwen3.6:35b-a3b-bf16) of the 35B MoE (https://ollama.com/library/qwen3.6/tags), the latter uses 3x more storage and RAM when loaded and it could make a difference between being able to run and not run the model at all.
But yeah you might get like a 0% to 5% quality hit, which I think is totally acceptable in practice.
Thanks a lot for publishing this setup and the agent problem pack. I really appreciate the practical angle of the article.
I tested the five tasks with a slightly different local setup: a used HP Z8 G4 workstation acting as my local AI server, equipped with two NVIDIA RTX A6000 GPUs. LM Studio was running as the local OpenAI-compatible backend on that machine, with the model split evenly across both GPUs. On the client side, I used OpenCode Desktop on Windows as the coding-agent harness.
My setup was:
Agent / harness: OpenCode Desktop on Windows
Backend: LM Studio Server, OpenAI-compatible endpoint
Result: OpenCode Desktop + LM Studio + Qwen3 Coder Next Q4 solved all five tasks successfully in my run. I verified the generated changes afterwards by running the pytest suites manually in each isolated workspace. All five passed.
From the LM Studio server log, the five benchmark tasks produced about 49 chat-completion requests, around 48.8k prompt/input tokens and 4.8k generated tokens. Aggregated over the benchmark run, I saw roughly 1.1k prompt tokens/s and about 72 generated tokens/s. No context truncation was reported in the log.
I found this especially interesting because it confirms your point that the agent harness matters. This was not just a model test; it was specifically OpenCode Desktop as the coding harness, using LM Studio as the local backend. For your small problem pack, that combination worked surprisingly well.
I have not yet isolated how much the dual-GPU setup contributed to throughput versus mainly providing VRAM headroom for the model and long context.
Nochmals Vielen Dank für Deinen Artikel und das bereitgestellte Setup! :-)
Hey Sebastian, thank you very much for that nice article. I run currently very similar setup with Qwen3.6 35b on my local DGX Spark with OpenCode and I currently happy.
For most coding task this setup works quite good, but sometimes it would be great to habe more intelligent capabilities. Currently I looking a little bit in the direction of Apple and there approach to use edge models for the most tasks and the ability to escalate to a much more capable cloud version. Did you have then something like this stack within the open weight space currently?
I am not affiliated with them, but you could also switch to a cloud model in Ollama (like GLM 5.2). Their's is a monthly subscription. Alternatively, you could use e.g., OpenRouter (which also exposes an OpenAI-compatible API for the coding agents) with usage-based per-token costs.
The local angle gets underrated as a control story, not just a cost or privacy one. With a coding agent that mostly drafts, running open-weight locally is a nice-to-have, because you can always fall back to an API. The calculus changes the moment the agent starts acting on its own: running commands, touching files, hitting your systems. Then owning the weights is what lets you pin the behavior, guarantee where the data goes, and keep the thing working when the API has an outage or shifts under you. The benchmark gap is closing. The part that compounds is that local means you control the path the agent actually runs on.
Hello, could you provide a little more information about the hardware and electrical cost ranges for such an implementation? How might it compare the proprietary agent subscriptions?
The prices for Mac and DGX fluctuate because of the RAM constraints etc. It's something I use for other work projects anyways, so repurposing them for local agent work is no extra cost in my case but yeah is a significant investment.
The DGX Spark running under full load for like 6 hours a day (which would be a lot) would be about 10 cents per day where I live, so approximately $3/month
"Transparent, inspectable, and free to run" — those three properties are underrated as an architectural requirement, not just a cost decision. When an agent can read and manipulate files directly, knowing exactly what model version is running and what data it touches isn't a nice-to-have. It's the audit trail that makes local AI a categorically different product than an API call.
Thank you for another great article, Sebastian. Very much enjoy reading your content. I’ve been testing mtp + dflash to speed up inference on my GB10 machine with Gemma + Qwen models. Selecting the ideal coding harness for the models was next on my list. This is going to save me a lot of time!
You mentioned LLMs are often optimized for a specific harness. In more open-ended domains (beyond SWE), how do we effectively co-optimize the model and the scaffolding? I'm curious to know your thoughts on how fine-tuning specifically for the harness reshapes things like memory compaction and context management.
This is great. thanks for sharing. Have you tried running two harnesses in the same machine? I have the impression that a harness like Hermes or paperclip are more suited for productivity than coding, so I am wondering if one can run both and used them for different purposes (given that you can host both corresponding LLMs somewhere). Loved the post!
Terrific article!
I use MSTY Studio which is a GUI front-end to an embedded Ollama server. Check it out here: https://msty.ai/ It's basically an alternative to things like LMStudio and AnythingLLM, but has its own version of OpenClaw and can also support LLaMa.cpp and has an Agent Mode. Supports local and online models.
Interesting, if I understand correctly, it's the Codex GUI analog (as opposed to Codex CLI analog)?
Yes, it's a cross-platform GUI with an embedded Ollama server which it uses as its own local server accessible by any tool that uses Ollama (but on a different server endpoint: http://localhost:11964).
You can also use Huggingface GGUF models as well as Ollama models in addition to any online model from any provider. I use Kimi K2.7 from Nvidia's NIM Developer program for free.
And it has an "experimental" LLaMa server setup.
They also have a "Vibe CLI Proxy" they describe as:
"Connect command-line model providers to use their AI models
Integrate models from supported CLI applications directly into Msty Studio. This integration supports multiple CLI providers and makes their models available for conversations.
Supported CLI Providers
Claude Code
Codex
Google Gemini
iFlow
Github Copilot
Antigravity"
I haven't used that yet.
They also have Agent Mode:
"Agent Mode
Run coding agents in Msty Studio Desktop
Agent Mode lets you run coding agents inside Msty Studio Desktop. You can chat with an agent, review plans, inspect diffs, and approve actions in one workspace-aware flow."
It's main features are a comprehensive set of chat functions such as chatting with multiple LLMs at once, splitting chats, etc. They have four types of chat conversations:
"Normal Conversations (Conversations): Best for everyday chatting, drafting, Q&A, and iterative work with one model.
Persona Conversations (Persona Conversations): Best when you want a conversation guided by a specific role, voice, or expertise.
Crew Mode Conversations (Crew Conversations): Best when you want multiple personas to respond to the same prompt so you can compare perspectives.
Agent Mode Conversations (Agent Mode): Best for multi-step tasks where an agent plans, uses tools, and executes with less manual steering."
They also have RAG callled "Knowledge Stacks":
Knowledge Stacks are Msty Studio's take on Retrieval-Augmented Generation (RAG), designed to help you create, manage, and use collections of knowledge in a structured, flexible way. They let you organize your data for easy access in conversations, proving models with deep, domain-specific understanding.
They also have "Tolboxes" which is their version of MCP:
"The Toolbox allows for seamless communication with tools installed on your local machine or a remote one. These tools can interact with various data sources, perform file operations, and connect to repositories like GitHub, further empowering Msty Studio. This is Msty Studio's implementation of Model Context Protocol (MCP)."
Msty Studio supports local MCP server that run on your machine as well as remote MCP servers that can be connect to over Streamable HTTP.
Plus they have MSTY Claw which is their version of OpenClaw:
"Agents: execution identity (workspace, runtime, routing, permissions)
Briefing and chat flow: Pulse, Side Chat, chat organization, and response controls
Mission Control: run room for subagents, approvals, and handoffs
Memory: live state (Working Brief) plus durable memory packs
Playbooks, Tasks, and Rules: reusable, scheduled, and event-based automation
Providers and Tools: model routing, MCP capabilities, and search/tooling
Security and Settings: sender policy, folder/network controls, and app defaults"
It's under consistent development and has an active Discord channel for support and the developers are pretty responsive.
It's got so many features I haven't even attempted to use them all yet.
Oh, and most of the features are free. There are some enhanced features at $149/year or $349 lifetime plus a team plan at $300/user.
Nice article, thank you very much! One thing i'am missing is a greater view on security. I personally would put the coding instance and project files in a virtual machine and remote connect to it. So agents are clearly disconnected from private data and vulnerabilities are sandboxed to some temporary/check pointed system. However keeping your coding projects / packages up to date requires agent web access features that can bring the risk of prompt injection. Do you have experience with safety guards for this, like building a extraction pipeline that omits such injections?
Thanks. And yes, I mentioned this sandboxing throughout the article. I my case I run them on a separate machine. You may also turn off web access but that limits capabilities severely. I think the sweet spot is doing an audit of the harness code base (as I mentioned in this article) and start using it in a separate (or at least virtual) machine with no auto-permissions. I.e., without auto-permissions, it will ask you each time which command to run before it can execute it.
..or filtering out urls that could be hard coded somewhere in the project.
You are totally right, my comment reads like you didn't discussed it at all. Sorry!
From my understanding, command execution and file read / write are separated from each other in general. So one can use agents to automatically implement a bunch of features without requiring permission for each single file edit, but not use yolo mode, where commands also are executed automatically.
When putting the code base and coding agent into a VM and enable the sandbox mode (as you described) with no automatic command execution, this is considered as "safe".
But are there best practices to handle prompt injections hitting the agent at all? Like allowed / disallowed domains for the agent, some kind of converting pipelines (html to markdown) etc.?
We use Pi with Deepseek running on DS4, and Geodesia.ai as safety layer.
Our developers started tweaking a local stack after the repeated outages and slowdown of Claude and Codex. They are afraid to be cut off one day. They showed me the Stanford survey showing how open weight models are really close in capabilities to the big AI labs, so we assigned time, budget and focus on building it.
Thanks for sharing. Yeah, DS4 is great. It’s tad more expensive to run, but GLM 5.2 is even better. Apparently better than GPT 5.5 and Claude Mythos level.
You got it. I heard through the vines they are tinkering with GLM 5.2. Exciting times?
@Sebastian Raschka, PhD Local coding agents are becoming the default workflow. Better privacy, lower latency, and full control over the development environment—great overview.
Thanks! And yes, there are several advantages (plus getting into it is also kind of future-proofing oneself)
Great article, thanks for this and all the LLM architecture material you produce Sebastian 🙏🏼
Prompted me to switch to Qwen Code (I've been using OpenCode) with Qwen3.6 35b MoE as the model, and I have to say, so far it seems faster than OC. Specifically, it seems to generate less and more structured thinking output, so spends less time thinking.
Oh nice, thanks for sharing.
Thank you, Sebastian, for this exceptional article.
Beyond the technical breakdown, this approach to local AI offers a wonderful bridge between modern agentic engineering and sysadmin culture, with that invaluable 80s open-world spirit running in the background: remaining master of your own machine and sovereign over your data.
A truly delightful and rigorous read, thanks for sharing!
Wow thanks for this very kind comment!
Really nice article! Thank you for sharing. I really like the benchmarking example repo you published. It is often really hard to come out with any kind of evals or at least with useful structure for evals.
I was wondering, since Ollama quantizes everything heavily, does it degrade quality by a lot? I am curious how you think about trade off or quality vs. quantization.
Thanks, and good question. Regarding the benchmarks, there are also some personal ones I have based on concrete projects that I can’t share in their current form, but in general, I suggest occasionally add or collect some of the tasks you come across so that you have them in ready when comparing models.
Regarding the quantization, yeah, there’s unfortunately a bit of a performance degradation. But looking at the default quantized model vs the original unqantized bf16 variant (qwen3.6:35b-a3b-bf16) of the 35B MoE (https://ollama.com/library/qwen3.6/tags), the latter uses 3x more storage and RAM when loaded and it could make a difference between being able to run and not run the model at all.
But yeah you might get like a 0% to 5% quality hit, which I think is totally acceptable in practice.
Thanks a lot for publishing this setup and the agent problem pack. I really appreciate the practical angle of the article.
I tested the five tasks with a slightly different local setup: a used HP Z8 G4 workstation acting as my local AI server, equipped with two NVIDIA RTX A6000 GPUs. LM Studio was running as the local OpenAI-compatible backend on that machine, with the model split evenly across both GPUs. On the client side, I used OpenCode Desktop on Windows as the coding-agent harness.
My setup was:
Agent / harness: OpenCode Desktop on Windows
Backend: LM Studio Server, OpenAI-compatible endpoint
Model: Qwen3 Coder Next GGUF
Quantization: Q4_K_M
Context: 64k (`n_ctx=65536`)
GPU setup: 2× NVIDIA RTX A6000, LM Studio split strategy: split evenly
Result: OpenCode Desktop + LM Studio + Qwen3 Coder Next Q4 solved all five tasks successfully in my run. I verified the generated changes afterwards by running the pytest suites manually in each isolated workspace. All five passed.
From the LM Studio server log, the five benchmark tasks produced about 49 chat-completion requests, around 48.8k prompt/input tokens and 4.8k generated tokens. Aggregated over the benchmark run, I saw roughly 1.1k prompt tokens/s and about 72 generated tokens/s. No context truncation was reported in the log.
I found this especially interesting because it confirms your point that the agent harness matters. This was not just a model test; it was specifically OpenCode Desktop as the coding harness, using LM Studio as the local backend. For your small problem pack, that combination worked surprisingly well.
I have not yet isolated how much the dual-GPU setup contributed to throughput versus mainly providing VRAM headroom for the model and long context.
Nochmals Vielen Dank für Deinen Artikel und das bereitgestellte Setup! :-)
Hey Sebastian, thank you very much for that nice article. I run currently very similar setup with Qwen3.6 35b on my local DGX Spark with OpenCode and I currently happy.
For most coding task this setup works quite good, but sometimes it would be great to habe more intelligent capabilities. Currently I looking a little bit in the direction of Apple and there approach to use edge models for the most tasks and the ability to escalate to a much more capable cloud version. Did you have then something like this stack within the open weight space currently?
I am not affiliated with them, but you could also switch to a cloud model in Ollama (like GLM 5.2). Their's is a monthly subscription. Alternatively, you could use e.g., OpenRouter (which also exposes an OpenAI-compatible API for the coding agents) with usage-based per-token costs.
Thank you Sebastian. Dankesehr ! This was a wonderful explanation and exercise of something i had been trying to learn for several weeks.
Nice, glad to hear!!
The local angle gets underrated as a control story, not just a cost or privacy one. With a coding agent that mostly drafts, running open-weight locally is a nice-to-have, because you can always fall back to an API. The calculus changes the moment the agent starts acting on its own: running commands, touching files, hitting your systems. Then owning the weights is what lets you pin the behavior, guarantee where the data goes, and keep the thing working when the API has an outage or shifts under you. The benchmark gap is closing. The part that compounds is that local means you control the path the agent actually runs on.
Hello, could you provide a little more information about the hardware and electrical cost ranges for such an implementation? How might it compare the proprietary agent subscriptions?
The prices for Mac and DGX fluctuate because of the RAM constraints etc. It's something I use for other work projects anyways, so repurposing them for local agent work is no extra cost in my case but yeah is a significant investment.
The DGX Spark running under full load for like 6 hours a day (which would be a lot) would be about 10 cents per day where I live, so approximately $3/month
"Transparent, inspectable, and free to run" — those three properties are underrated as an architectural requirement, not just a cost decision. When an agent can read and manipulate files directly, knowing exactly what model version is running and what data it touches isn't a nice-to-have. It's the audit trail that makes local AI a categorically different product than an API call.
Thank you for another great article, Sebastian. Very much enjoy reading your content. I’ve been testing mtp + dflash to speed up inference on my GB10 machine with Gemma + Qwen models. Selecting the ideal coding harness for the models was next on my list. This is going to save me a lot of time!
Thanks for sharing your write-up, excellent work.
You mentioned LLMs are often optimized for a specific harness. In more open-ended domains (beyond SWE), how do we effectively co-optimize the model and the scaffolding? I'm curious to know your thoughts on how fine-tuning specifically for the harness reshapes things like memory compaction and context management.
This is great. thanks for sharing. Have you tried running two harnesses in the same machine? I have the impression that a harness like Hermes or paperclip are more suited for productivity than coding, so I am wondering if one can run both and used them for different purposes (given that you can host both corresponding LLMs somewhere). Loved the post!