Learning AI engineering and improving my skills while being efficient requires a smart toolset that combines cloud LLM services and local models to deal with performance, resource constraints, and offline usability for code assistance.
This led me to design a hybrid local/cloud LLM architecture for coding assistance, focusing on balancing:
- Hardware capacity vs model size: My MacBook Pro’s Apple M4 chip has 24G unified RAM, which limits the size of models I can run locally without glitching.
- Cloud for complex generation vs local for simple editing: I want Claude Code for complex code generation and architecture/design needs, while using local LLMs for quick edits and production tweaks.
- Offline support: Local models provide offline coding assistance when disconnected.
- Maintainability and adaptability: The stack should be modular and allow adding newer models or tools, such as exploring new open-weight models or frontier models in the future.
I chose a layered architecture where my primary IDE, Zed Editor, connects both to cloud-based Claude Code and a chain of local LLMs managed through OpenCode and LM Studio, with Qwen 2.5 as the primary local model.
Prioritizing local compute constraints and model selection
My MacBook Pro’s M4 chip delivers 24GB unified RAM is good, but still limited for running very large LLMs locally.
I decided to use a carefully sized model: Qwen 2.5 with 14 billion parameters. This model fits into memory without resorting to disk swap, providing reasonable latency and response times for typical coding completions.
Trying to push for larger models locally causes unacceptable latency and memory pressure whereas very small models fall short of code comprehension and generation quality.
Balancing model size and hardware capacity:
- Selecting Qwen 2.5 as the current local default.
- Using MLX for ARM-optimized runtime performance.
This ensures my local stack can operate smoothly with my available hardware.
Delegating complexity: Claude Code for heavy-duty tasks
While the local stack handles simpler edits and incremental code generation, cloud-based Claude Code provides high-end reasoning, architecture design assistance, and large-scale context windows.
This separation leverages Claudes’s power when connected, without burdening my local resources.
OpenCode and LM Studio as local orchestration platforms
OpenCode manages the LLM invocation cycle locally by handling prompt construction, token management, and caching. LM Studio loads the model and exposes an OpenAI compatible server.
Their modular architecture allows swapping Qwen with other local models or cloud-based ones for offloading complex LLM calls.
Responsibilities include:
- Context management
- Translating IDE input into well-formed prompts (including code-focused system prompts)
- Delivering results back to the IDE with minimal latency.
Embedding LLMs and agents into Zed
Zed is my central code editor, chosen for its extensibility and performance.
I configured it using only native Agent Client Protocol (ACP) that connect Zed to both the local LLM stack (OpenCode/LM Studio/Qwen) and the cloud Claude Code API.
This lets me:
- Invoke local LLM completions inline
- Switch to cloud completions only on demand
- Seamlessly view responses without context loss
Planning for future extensibility
The stack is designed as a living architecture:
- I am thinking about integrating tools such as Caveman a tool that reduces input token use to save Claude credits
- Alternative LLM models with compatible runtimes can be swapped in without major rework
- Additional plugins could manage context window expansion and local vector databases for local RAG
This approach maintains momentum while supporting experimentation.
Reflecting on hybrid local/cloud LLM design
Building this stack taught me many things about AI and LLM architectures:
- Implementing local inference servers
- Exploring open-weight LLMs and the Hugging Face ecosystem
- Using OpenCode and integrating it within local tooling
- Finding the sweet spot between local and cloud use
While this is a pure developer experience enhancement, I see a wider and actual long-term approach to gaining “local AI” skills: sovereignty.
In a world where data sovereignty comes more and more into play, being able to implement local AI is a critical step to help mission-critical applications (government, banking, healthcare, …) secure their data and operations. I will cover this topic in more detail in the future.