Blog / model configuration
Run Hermes Agent on Local Models (Ollama and More)
Hermes Agent can use local models when you point it at an OpenAI-compatible endpoint such as Ollama, llama.cpp, or vLLM instead of a hosted provider. Local models remove per-token cost and keep data on your own hardware, but they need enough memory, a realistic context window, and a model that can call tools.
- 01
Check memory and storage first
Confirm the machine can hold the model weights plus the context you need. Memory, not raw compute, is usually the limiting factor.
- 02
Serve the model locally
Install Ollama, llama.cpp, or vLLM, pull a model, and start the server on a port the agent runtime can reach.
- 03
Expose an OpenAI-compatible endpoint
Confirm the server exposes the OpenAI-compatible chat completions route, and set an API key value if the server expects one.
- 04
Point Hermes Agent at the local endpoint
Add a custom provider with the base URL and model name. Keep the hosted provider configured as a fallback.
- 05
Verify tool calling and long context
Send a request that requires a tool and one that needs a long context. Local models often pass the first test and fail the second.
- 06
Decide what work stays local
Keep private or high-volume work on the local model and send the difficult reasoning tasks to a hosted provider when quality matters more.
Four-step path
How do you run Hermes Agent on a local model?
Five checkpoints: size the hardware, serve the model, expose an OpenAI-compatible endpoint, point Hermes Agent at it, then verify tool use.
- Step 01
Check memory and storage first
Confirm the machine can hold the model weights plus the context you need. Memory, not raw compute, is usually the limiting factor.
- Step 02
Serve the model locally
Install Ollama, llama.cpp, or vLLM, pull a model, and start the server on a port the agent runtime can reach.
- Step 03
Expose an OpenAI-compatible endpoint
Confirm the server exposes the OpenAI-compatible chat completions route, and set an API key value if the server expects one.
- Step 04
Point Hermes Agent at the local endpoint
Add a custom provider with the base URL and model name. Keep the hosted provider configured as a fallback.
- Step 05
Verify tool calling and long context
Send a request that requires a tool and one that needs a long context. Local models often pass the first test and fail the second.
- Step 06
Decide what work stays local
Keep private or high-volume work on the local model and send the difficult reasoning tasks to a hosted provider when quality matters more.
What hardware do you need for a local model?
You need enough memory to hold the quantized weights plus the key-value cache for your context length. A model that barely fits will fail as soon as the conversation grows.
- Quantized weights reduce memory at a small quality cost.
- Longer context consumes more memory for the same model.
- A GPU speeds generation dramatically but is not always required.
- Leave headroom for the runtime, the OS, and other services.
Which local server should you use?
Ollama is the quickest way to get a model running. llama.cpp gives finer control over quantization, and vLLM offers higher throughput for multi-user workloads.
What matters for Hermes Agent is the OpenAI-compatible API surface, not the server brand. Choose the one you can operate reliably.
Do local models support tool calling?
Some do, and the capability varies by model and by server. Verify tool calls explicitly, because a model that chats well may still fail to emit a correct structured call.
- Test a request that must invoke a tool, not just answer a question.
- Check the server version; tool support is added over time.
- Prefer models with explicit tool or function-calling training.
- Keep a hosted fallback for workflows that depend on tools.
Is a local model actually more private?
Yes for inference, because prompts and completions never leave your hardware. The benefit disappears if the agent still sends the same content to a hosted provider for part of the workflow.
Map where each request goes before claiming privacy gains. A hybrid setup that silently routes some traffic to a cloud provider is not a private deployment.
Should you run everything locally?
Rarely. A hybrid setup is usually better: local models for high-volume or sensitive work, hosted models for difficult reasoning and tool-heavy tasks.
Routing between a local and a hosted provider is the same configuration problem as any multi-provider setup, and it gives you a fallback at the same time.
Which primary sources support this page?
Product behavior is checked against the current deployment flow. These external links provide the upstream project or channel documentation used for setup details.
Direct answers
Frequently asked questions
Is a local model free?
There is no per-token charge, but you pay for the hardware, electricity, and the time spent operating it.
Can Hermes Agent use Ollama directly?
Yes, through the OpenAI-compatible endpoint that Ollama exposes. Add it as a custom provider with the model name you pulled.
Why does the local model fail on long conversations?
Long contexts need more memory. If the machine is at its limit, requests fail or degrade once the conversation grows past the available capacity.
Can I mix local and hosted models?
Yes, and it is usually the best configuration. Route sensitive or high-volume work locally and keep a hosted provider for harder tasks.
Key takeaways
- Hermes Agent can use local models when you point it at an OpenAI-compatible endpoint such as Ollama, llama.cpp, or vLLM instead of a hosted provider. Local models remove per-token cost and keep data on your own hardware, but they need enough memory, a realistic context window, and a model that can call tools.
- Start with one model, one channel, and one controlled test conversation.
- Use managed hosting when deployment speed matters more than operating the server yourself.
Continue the setup
Related Hermes Agent guides
Blog guide
How to Choose a Model for Hermes Agent
Choose the right Hermes Agent model for support, briefings, operations, and channel workflows across GPT, Claude, Gemini, DeepSeek, Kimi, and GLM.
Read nextBlog guide
Hermes Agent Provider Routing Across Multiple Models
Route Hermes Agent across multiple model providers by task, cost, and latency. Covers rule design, verification steps, and the failure modes to avoid.
Read nextBlog guide
Hermes Agent Fallback Providers: Survive an Outage
Configure Hermes Agent fallback providers so one outage, rate limit, or bad key does not stop the assistant. Includes trigger rules and failure testing.
Read nextBlog guide
Best VPS for Hermes Agent: 24/7 Hosting Guide
Choose a VPS for Hermes Agent by workload, memory, persistence, gateway uptime, backups, and security—or decide when managed hosting is the better fit.
Read next