LAB-NOTES

Network Troubleshooting Evidence Ledger and Restart Point

Animated Unix Magick terminal intro showing a network troubleshooting evidence ledger, known-good baseline, and restart-point checks.
Preserve what is known, separate proof from hypothesis, and resume from the narrowest remaining fault domain.

This is Part 3 of the network troubleshooting series.

Part 1 kept the chronological investigation: the 100 Mbps injector, the marginal coupler panel, the one-way TCP collapse, the bad VLAN test, and the local U6 result that finally narrowed the fault domain.

Part 2 turned the commands into a reusable OSI-layer runbook.

This final post is the evidence ledger and restart point. Its job is different: preserve what was actually observed, distinguish proved faults from reduced suspects and open hypotheses, and make sure the next troubleshooting session starts where the previous one ended instead of reopening already-answered questions.

For publication, private addressing, VLAN numbers, endpoint names, and location-specific labels have been replaced with documentation networks and generic names. The measured results and device models that matter to the diagnosis are retained.

Topology under test

The sanitized topology was:

remote wired / Wi-Fi clients
          │
         U6 ─────┐
          │      │
       Juniper EX3300
          │
   gigabit passive-PoE injector
          │
   remote PowerBeam
          )) 5 GHz RF bridge ((
   core PowerBeam
          │
        router
          │
   core test server

For the examples below, documentation addresses stand in for the real private networks:

Documentation address Role during testing
198.51.100.10 core iperf3 server
198.51.100.13 remote wired client on TEST_VLAN
192.0.2.20 remote Wi-Fi client used through the bridge
192.0.2.19 Wi-Fi 5 test laptop on USER_VLAN
192.0.2.11 wired test laptop on USER_VLAN for corrected AP isolation
192.0.2.8 Wi-Fi 6 test endpoint on the U6

The exact addresses are not the point. What mattered was whether two endpoints were actually local to the same Layer-2 segment or whether the test silently required routing through the core.

Chronological evidence ledger

Stage Test or path Observed result What the evidence supported
Initial symptom Internet speed test from remote site ~80 Mbps A problem existed, but not where
Injector inspection Physical component check One injector was 100 Mbps Confirmed a hard Layer-1 ceiling
After gigabit injector Wired across bridge ~400–500 Mbps initially Path could exceed 100 Mbps
Link instability PowerBeam-facing Ethernet link 1 Gbps later fell to 100 Mbps A copper/autonegotiation problem remained
Coupler panel inserted Wired path after time as bad as ~13/400 Mbps, then 100 Mbps link Strong causal evidence against the inline coupler panel
Coupler panel bypassed Wired bridge roughly ~400/400 Mbps Direct path restored 1 Gbps negotiation and useful symmetric throughput
Early directional iperf3 remote wired → core ~430 Mbps One physical direction could perform normally
Early reverse iperf3 core → remote wired 18.1 Mbps, 3,753 retransmissions Severe directional impairment, not a simple clean rate cap
Early reverse, four streams core → remote wired ~22.5 Mbps, thousands of retransmissions Parallel streams could not overcome the fault
Core → remote Wi-Fi Same bridge, U6 endpoint 209 Mbps, 27 retransmissions The earlier 18 Mbps failure was not inherent to the entire bridge path
Wired path after state change core → remote wired 418 Mbps, 111 retransmissions The wired-client path recovered; some temporary endpoint/port/path state had changed
Long ping remote ↔ core 0.365% loss, 4.056 ms average, 113.599 ms maximum Normally low latency with transient spikes/loss
Wi-Fi 5 live state iw at 80 MHz RX 390–433 Mbps NSS1; TX 650–780 Mbps NSS2; ~-47 dBm Strong signal; client TX radio was not simply capped near 200 Mbps
Two Wi-Fi clients on one U6 same local VLAN ~241/193 Mbps single stream; ~231 Mbps one four-stream run The test consumed shared Wi-Fi airtime twice and was not an AP maximum-throughput test
Wrong-VLAN “local” test 198.51.100.13 ↔ 192.0.2.8 fell from ~45 toward ~10–30 Mbps Traffic routed through the core and crossed the bridge twice
Correct same-VLAN local AP test wired 192.0.2.11 ↔ Wi-Fi 192.0.2.8 ~672 Mbps Wi-Fi→wired; ~481 Mbps wired→Wi-Fi single stream U6 path was far above 200 Mbps; one single stream was affected by a transient loss event
Correct local, four streams same isolated path 721 Mbps / 722 Mbps U6, its Gigabit uplink, EX3300 local switching, and Wi-Fi 6 endpoint were cleared for this tested path and load

The last wording matters. A successful test does not prove that a component can never fail. It shows that the component was capable of the measured result in the path and conditions that were actually tested.

Decisive raw excerpts

Catastrophic early reverse direction

[  5] 0.00-30.00 sec 64.9 MBytes 18.1 Mbits/sec 3753 sender retransmissions

Interpretation: TCP repeatedly experienced loss or disruption and reduced its congestion window. This did not look like a clean 100 Mbps ceiling.

Recovered wired bridge path

[  5] 0.00-30.00 sec 1.46 GBytes 418 Mbits/sec 111 sender retransmissions

Interpretation: the same general bridge path later exceeded 400 Mbps, rejecting the idea that the PowerBeam path had a permanent ceiling near 200 Mbps.

Final isolated U6 path

[SUM] 0.00-20.00 sec 1.68 GBytes 720 Mbits/sec 186 sender
[SUM] 0.00-20.02 sec 1.68 GBytes 721 Mbits/sec receiver

Reverse test:

[SUM] 0.00-20.02 sec 1.68 GBytes 722 Mbits/sec sender
[SUM] 0.00-20.00 sec 1.68 GBytes 722 Mbits/sec receiver

Interpretation: the isolated local wired-to-Wi-Fi path was roughly symmetric at 720 Mbps aggregate TCP.

What the evidence actually eliminated

Confirmed faults

100 Mbps passive-PoE injector

The original injector physically imposed a Fast Ethernet ceiling. Replacing it allowed the path to move into the several-hundred-megabit range.

Inline female-to-female coupler panel

Repeated insertion recreated degradation, severe asymmetry, and eventual 100 Mbps renegotiation. Repeated bypass restored 1 Gbps negotiation and roughly 400/400 Mbps observed wired bridge throughput.

The repeatability of that A/B test is why the panel was treated as a confirmed fault.

Cleared for the tested path and load

The corrected local four-stream test substantially cleared:

  • the U6’s local throughput capability;
  • the U6 Gigabit Ethernet uplink;
  • the EX3300 local Layer-2 switching path;
  • the Wi-Fi 6 test endpoint.

The same test also rejected the idea that a displayed 1.2 Gbps Wi-Fi PHY rate should equal 1.2 Gbps of TCP payload.

Rejected interpretations

“The bridge is an 80 Mbps bridge because the first speed test was 80 Mbps.”

Rejected. Different controlled paths produced results ranging from tens of megabits to more than 700 Mbps.

“The U6 is capped around 200 Mbps.”

Rejected for the tested local path. The isolated wired-to-Wi-Fi test exceeded 700 Mbps.

“Two clients beside the same AP guarantee a local one-hop Wi-Fi test.”

Rejected. Two wireless endpoints on the same AP consume shared wireless airtime in both directions of the relay.

“If the devices are physically in the same room, the traffic must stay local.”

Rejected. Different VLANs/subnets caused the supposedly local test to route through the core and cross the PowerBeam twice.

What remains open

The evidence did not fully eliminate:

  • PowerBeam RF tuning or capacity as a factor in the roughly 400/400 Mbps wired result;
  • possible 5 GHz coexistence effects between the U6 and the PowerBeam;
  • queueing or flow-control behavior at the PowerBeam Ethernet/radio boundary;
  • long-term stability of the direct remote-side copper channel;
  • a second marginal physical component if the 1 Gbps → 100 Mbps fallback ever returns.

The 5 GHz coexistence point remains a hypothesis. Nothing in the completed testing proved that the U6 and PowerBeam were interfering with one another.

Current working model

By the end of the session, there were at least two confirmed Layer-1 faults and one temporary condition that was observed but not fully explained:

Confirmed fault 1:
  100 Mbps passive-PoE injector

Confirmed fault 2:
  defective or marginal inline coupler panel

Observed temporary condition:
  wired client / NIC / cable / switch-port state
  associated with the one-way 18–22 Mbps behavior

After the confirmed faults were removed and the endpoint/path state changed:

observed wired PowerBeam path: ~400/400 Mbps
tested local U6 path:          ~720/720 Mbps

That leaves a much narrower question:

Why was the combined Wi-Fi-through-PowerBeam path
often closer to 200–300 Mbps when the isolated
local Wi-Fi path and wired bridge path both tested faster?

That question is worth measuring. Reopening the already-confirmed injector or coupler-panel findings without contradictory evidence is not.

Preserve the known-good baseline

Before doing anything else in the next session:

  • keep the inline coupler panel out of the PowerBeam path;
  • keep the gigabit-capable passive-PoE injector installed;
  • do not force 1 Gbps speed as a substitute for repairing a marginal copper channel;
  • record the exact EX3300 ports used by the PowerBeam and U6.

The baseline itself is evidence. Do not casually alter it before collecting the next round of measurements.

Capture before the next test

EX3300 PowerBeam-facing port

show interfaces ge-0/0/X extensive | no-more
show interfaces ge-0/0/X media | no-more

EX3300 U6-facing port

show interfaces ge-0/0/Y extensive | no-more
show interfaces ge-0/0/Y media | no-more

Wi-Fi client

iw dev WIFI_IFACE link
iw dev WIFI_IFACE station dump

Route selection

ip route get CORE_SERVER

PowerBeam data to record at both ends

Capture:

  • exact operating frequency;
  • channel width;
  • signal per chain;
  • noise floor;
  • airMAX quality/capacity;
  • current modulation/MCS;
  • TX/RX rate;
  • retries/errors;
  • Ethernet link speed;
  • uptime and recent link events.

Take these measurements before changing RF settings.

Run the comparison matrix

A. Wired remote client → core server

iperf3 -c CORE_SERVER -t 30 -O 3
iperf3 -c CORE_SERVER -R -t 30 -O 3
iperf3 -c CORE_SERVER -P 4 -t 30 -O 3
iperf3 -c CORE_SERVER -P 4 -R -t 30 -O 3

B. Wi-Fi 6 endpoint → core server

Run the same four tests from the Wi-Fi 6 endpoint, or arrange the endpoint roles so that the same physical directions are measured.

C. Observe the path while the tests run

During every run, watch:

  • EX3300 errors, carrier transitions, and PAUSE counters;
  • U6 airtime and client PHY rates;
  • PowerBeam capacity, modulation/MCS, and retries;
  • rapid-ping latency and loss.

Change only one variable between tests.

Decision tree for the next session

Does the PowerBeam-facing Ethernet link remain at 1 Gbps?
├─ No
│  └─ Return to Layer 1:
│     copper / injector / termination / PowerBeam PHY
│
└─ Yes
   │
   ├─ Is the wired remote↔core path still ~400/400?
   │  ├─ No
   │  │  └─ Investigate PowerBeam RF or
   │  │     Ethernet/radio-boundary behavior
   │  │
   │  └─ Yes
   │
   └─ Is local Wi-Fi still ~700 Mbps while
      Wi-Fi↔core remains ~200–300 Mbps?
      ├─ Yes
      │  └─ Measure combined RF coexistence,
      │     airtime, queueing, and directionality
      │
      └─ No
         └─ Re-test the local AP path and client
            association before changing bridge settings

That is the exact restart point.

The next troubleshooting session should begin by confirming that this baseline still exists, capturing the state of the switch, bridge, Wi-Fi client, and route selection, and then running the comparison matrix without changing multiple variables at once.

The purpose of an evidence ledger is not to make the investigation look tidy after the fact. It is to keep the next session from forgetting what the previous one already earned.