The problem
LLMs will happily make up a router ID, and giving one a shell on your network is a bad idea. I wanted an agent that could diagnose a live lab without being able to change anything or invent anything.
Network automation
Finished August 2026Homelab NetOps Agent
7.1 → 7.9 / 12tasks passed before and after fine-tuning on runs of the same tasks, mean of 10 runs
A small local model (Qwen2.5-3B) that answers questions about my OSPF lab by running read-only commands on the real routers and quoting what comes back. No agent framework, just Python and SSH.
PythonQwen2.5-3BOllamaNetmikoVyOSEVE-NG
THE SHORT VERSION
What I tested, built, and learned.
3VyOS routers
59verified training runs
7.9 ± 0.32fine-tuned score out of 12
What I built
The model picks from a short list of read-only show commands. The allowlist is enforced in code, not in the prompt. It runs them over SSH and has to answer from the output. The model is served from my MacBook and never talks to anything outside my network.
How I checked it
The benchmark is frozen: 12 tasks, fixed graders, 10 runs each so variance shows up. Early on it scored 8/12, but the device inventory file was leaking answers the model was supposed to discover. With that stripped out, the real baseline was 7.1. Fine-tuning on 59 verified runs brought it to 7.9 ± 0.32. Those runs came from the same 12 tasks, so this measures consistency on tasks it had seen, not new ability.
What didn't work
Four tasks still score zero. On the DR question, the model reads "Full/DR" in R1's neighbor output as describing R1 itself instead of the neighbor. The other three need multi-step reasoning a 3B model doesn't manage. Fine-tuning made the agent consistent on tasks it could already sometimes pass. It didn't teach it anything new.
The lab
Cloud0 is just the management connection into EVE-NG and isn't part of OSPF. The three routers below are the actual lab.
Where most questions start
The agent reads R1's neighbors, interfaces, DR election, and routes more than anything else.
- Links
- R2 · 10.0.12.0/30
R3 · 10.0.13.0/30
The destination
Its loopback is what the reachability questions aim at. I changed its router ID from 2.2.2.2 to 10.50.0.2 so the model couldn't guess the pattern.
- Links
- R1 · 10.0.12.0/30
R3 · 10.0.23.0/30
The backup path
Closes the triangle. If the R1-R2 link dies, traffic to R2 should go through R3, and one question tests exactly that.
- Links
- R1 · 10.0.13.0/30
R2 · 10.0.23.0/30
7.9Mean score/ 12 tasks
0.32Std devσ across runs
66%Mean pass rate10 runs
OSPFProtocolArea 0 · Triangle
Select a task to trace what part of the topology it tests.
98%Direct lookup
75%Interpretation
25%Multi-step reasoning
Every run, 120 attempts
Click an attempt to see the commands it ran and what the routers said back.
Loading the verified evidence export…
How a run gets graded
A right answer isn't enough. The run also has to include the router output that backs it up, or it fails.
def evaluate(task, result):
answer = result.get("answer")
if is_abstention(answer):
return False, "abstained"
if not check(answer, task["check"], task["expect"]):
return False, "wrong_answer"
if not check_evidence(result.get("messages", []), task.get("evidence")):
return False, "missing_or_invalid_evidence"
return True, NoneDetails
- Command allowlist enforced in code
- Frozen graders and repeated runs
- LoRA fine-tune exported to Q4_K_M