A framework-free Qwen2.5-3B agent that selects allowlisted VyOS commands, queries a live three-router OSPF lab, and answers from device evidence.
PythonQwen2.5-3BOllamaNetmikoVyOSEVE-NG
LAB TOPOLOGY / NODE ROLES
Three peers, three jobs in the evaluation.
Cloud0 carries private management access into EVE-NG. It is deliberately outside OSPF; the routed data plane is the R1–R2–R3 triangle below.
R1 · 1.1.1.1
Primary observation point
Most diagnostic questions start here. The agent reads R1's neighbor table, interface state, DR election, and routing table to prove what this router can see and forward.
Links
R2 · 10.0.12.0/30 R3 · 10.0.13.0/30
Why it matters
Turns raw CLI state into grounded network answers.
R2 · 10.50.0.2
Reachability target
R2's loopback is the destination used to test route learning and next-hop selection. If the direct R1–R2 link fails, the same prefix tests whether the alternate path through R3 remains available.
Links
R1 · 10.0.12.0/30 R3 · 10.0.23.0/30
Why it matters
Makes forwarding and failover measurable.
R3 · 3.3.3.3
Alternate-path peer
R3 closes the OSPF triangle. It supplies the second adjacency at R1 and the two-hop path toward R2, letting the benchmark distinguish direct lookup from multi-step topology reasoning.
Links
R1 · 10.0.13.0/30 R2 · 10.0.23.0/30
Why it matters
Exposes whether the agent understands path redundancy.
7.9Mean score/ 12 tasks
0.32Std devσ across runs
66%Mean pass rate10 runs
OSPFProtocolArea 0 · Triangle
EVE-NG TOPOLOGYAREA 0
Select a task to trace what part of the topology it tests.
TASK STABILITY MATRIX stable mixed fail
98%Direct lookup
75%Interpretation
25%Multi-step reasoning
RUN EXPLORER / 120 ATTEMPTS
Follow the evidence.
Loading the verified evidence export…
EVALUATOR CODE / FROZEN LOGIC
The answer alone is not enough.
A run passes only when the response is correct and the recorded live-device evidence satisfies the task-specific rule.
def evaluate(task, result):
answer = result.get("answer")
if is_abstention(answer):
return False, "abstained"
if not check(answer, task["check"], task["expect"]):
return False, "wrong_answer"
if not check_evidence(result.get("messages", []), task.get("evidence")):
return False, "missing_or_invalid_evidence"
return True, None
3VyOS routers
59sanitized trajectories
7.9 ± 0.32fine-tuned score / 12
01 / PROBLEM
What had to change.
Network assistants become risky when they invent topology facts or can execute arbitrary commands. The project needed useful live diagnosis without giving the model broad control of the lab.
02 / BUILD
How the system works.
The agent chooses from a code-enforced read-only command allowlist, connects over SSH, and composes answers only from returned device evidence. Model traffic remains private-network-only.
03 / PROOF
Why the result holds up.
A frozen evaluation harness separates answer correctness from safe abstention, repeats runs to expose variance, and removes topology metadata that previously leaked benchmark answers.