Network Troubleshooting Command Reference by OSI Layer

This is Part 2 of the network troubleshooting series. Part 1 kept the full incident chronology: the 100 Mbps injector, the marginal coupler panel, the one-way TCP collapse, the bad VLAN test, and the local U6 result that finally narrowed the fault domain.

This post strips that story down into a field runbook.

The commands below are either commands used during the incident or checks prepared for the next recurrence. Private addressing, VLAN numbers, interface names, connection-profile names, endpoint names, and location-specific labels have been replaced with documentation networks and generic placeholders.

One local convention matters: this Gentoo system uses doas for privileged commands. If your system uses a different privilege mechanism, adapt those lines accordingly. Read-only commands that work without elevation on your system do not need doas.

The order is intentional:

Layer 1: prove the physical link and radio state
Layer 2: prove local switching, VLAN placement, and airtime assumptions
Layer 3: prove the route and source address
Layer 4: measure throughput in both directions
Layer 7: confirm the user-visible result last

The point is not to worship the OSI model. The point is to stop higher-layer symptoms from distracting you from a lower-layer fault.

Layer 1 — Physical link, negotiated speed, cabling, and radio state

Juniper EX3300: capture the full interface before touching anything

show interfaces ge-0/0/X extensive | no-more

Purpose: Capture physical/link state, speed, duplex, carrier transitions, traffic counters, errors, drops, and flow-control counters.

Why it mattered: The PowerBeam-facing switch port intermittently changed from 1 Gbps to 100 Mbps. The extensive output is the evidence capture to take before unplugging, reseating, or swapping a cable.

High-value fields include:

• Speed
• Link mode / duplex
• Carrier transitions
• CRC, alignment, or framing errors
• input/output drops
• input/output PAUSE packets
• last-flapped time

Juniper defines Carrier transitions as transitions from down to up. Rapidly increasing transitions are a clue to the cable, far-end device, or interface hardware.¹

For a faster media-focused view:

show interfaces ge-0/0/X media | no-more

To reduce a long capture to the fields most useful during an unstable-link event:

show interfaces ge-0/0/X extensive | match "flap|error|carrier|pause|CRC|drop"

Important: Run the capture before changing the physical path. Reseating a cable can restore service and destroy the evidence at the same time.

Linux wired NIC: negotiated state

doas ethtool IFACE

Purpose: Show current speed, duplex, autonegotiation, link detection, advertised modes, and link-partner modes.

For a copper path expected to be Gigabit Ethernet:

Speed: 1000Mb/s
Duplex: Full
Auto-negotiation: on

is the expected general state.

A path that previously negotiated at 1000 Mb/s but now shows:

Speed: 100Mb/s

moves the investigation toward the copper channel, connector, injector, PHY, or another component participating in negotiation. ethtool is the standard Linux utility for querying and controlling network-driver and hardware settings.²

Linux wired NIC: Energy Efficient Ethernet

doas ethtool --show-eee IFACE

Purpose: Show whether Energy Efficient Ethernet is supported, enabled, and active.

During this incident, EEE was one possible explanation for a temporary directional wired-client condition. It was a diagnostic variable, not a proven root cause.

Linux wired NIC: driver-specific counters

doas ethtool -S IFACE | grep -Ei 'crc|error|drop|miss|pause|timeout|reset|carrier'

Purpose: Search NIC/driver-specific statistics for physical errors, missed packets, PAUSE activity, resets, and carrier events.

Counter names are driver-dependent, so the filter is deliberately broad. Linux documents ethtool -S as a source of NIC- and driver-specific interface statistics.³

Linux wired NIC: kernel interface counters

ip -s link show dev IFACE

Purpose: Show generic RX/TX bytes, packets, errors, drops, overruns, and carrier counts.

For more detailed standard error counters, Linux also supports specifying -s twice:

ip -s -s link show dev IFACE

The kernel documentation specifically calls out ip -s -s link show dev ... when the detailed members of the standard link statistics are needed.³

Linux Wi-Fi: current association

iw dev WIFI_IFACE link

Purpose: Show the current AP association, frequency, signal, current RX/TX PHY rates, channel width, MCS, and spatial-stream information when exposed by the driver.

Representative output from the incident looked like:

signal: -48 dBm
rx bitrate: 433.3 MBit/s VHT-MCS 9 80MHz VHT-NSS 1
tx bitrate: 780.0 MBit/s VHT-MCS 8 80MHz VHT-NSS 2

Those are PHY rates, not application throughput.

Linux Wi-Fi: station counters

iw dev WIFI_IFACE station dump

Purpose: Inspect signal averages, retries/failures, beacon behavior, current rates, MCS, channel width, and NSS when supported.

One observed sample was:

tx failed: 0
signal avg: -44 dBm
tx bitrate: 650.0 MBit/s ... VHT-NSS 2
rx bitrate: 390.0 MBit/s ... VHT-NSS 1

Watch Wi-Fi rates during a throughput test

watch -n 0.5 'iw dev WIFI_IFACE link | grep -E "signal:|rx bitrate:|tx bitrate:"'

Purpose: PHY rates are dynamic. Watching them while iperf3 is active helps show whether a rate collapses only under load.

Check Wi-Fi power-save state

iw dev WIFI_IFACE get power_save

Temporarily disable it for a controlled test:

doas iw dev WIFI_IFACE set power_save off

Purpose: Remove client power management as a variable during a repeatable test. Linux Wireless documents both querying and setting power-save state with iw.⁴

This was prepared as a diagnostic. It was not the final explanation after the isolated U6 path reached roughly 720 Mbps in both directions.

Reactivate a NetworkManager profile

doas nmcli con up WIFI_PROFILE

Purpose: Reactivate the intended saved connection profile.

NetworkManager documents nmcli connection up as the command for activating a stored connection profile.⁵

Why this deserves caution: During the incident, reconnecting the intended Wi-Fi profile also changed the active source interface and subnet. That changed the test path. A reconnect can alter the experiment even when it appears to be merely a client-side repair.

After changing interfaces or connection profiles, re-check:

ip addr
ip route

and then prove the path again.

Layer 2 — VLANs, local switching, MAC forwarding, PAUSE, and Wi-Fi airtime

Prove endpoint locality before calling a test “local”

For the isolated AP test, both endpoints must be on the same local VLAN/subnet:

wired endpoint: USER_VLAN, 192.0.2.x
Wi-Fi endpoint: USER_VLAN, 192.0.2.x

With both endpoints local, the EX3300 can switch frames directly based on destination MAC after ARP resolution. No Layer-3 routing is required for the test flow.

A misleading test used endpoints equivalent to:

wired endpoint: TEST_VLAN, 198.51.100.13
Wi-Fi endpoint: USER_VLAN, 192.0.2.8

That forced routing through the core and crossed the PowerBeam path twice.

Physical proximity is not proof of Layer-2 locality.

Incorrect AP-isolation topology: wireless to wireless on one AP

client A )) U6 )) client B

Each payload must use the same wireless channel twice: once to reach the AP and once for the AP to transmit it again.

That is useful for testing client-to-client behavior, but it is not a clean measurement of one client’s maximum single-hop AP throughput.

Correct AP-isolation topology: one wired, one wireless

wired endpoint
  → EX3300
  → U6 Ethernet
  → U6 5 GHz radio
  → Wi-Fi endpoint

Only one wireless transmission is required for each payload direction.

The corrected topology was the test that eventually reached approximately 721 Mbps and 722 Mbps aggregate TCP throughput.

Layer 3 — IP path, subnet selection, routing, and latency

Verify the selected route and source address

ip route get 192.0.2.8

Purpose: Confirm which interface, gateway, and source address Linux plans to use for the destination.

In the failed “local” test, the equivalent public-safe example was:

source:      198.51.100.13
destination: 192.0.2.8

Those addresses represent different VLANs/subnets. The test therefore required routing instead of local switching.

For a truly local test, the route should select the intended local interface and a source from the same documentation subnet, for example:

source 192.0.2.11
destination 192.0.2.8

Do this check before starting iperf3, not after a suspicious result.

Rapid ping while throughput testing

ping -i 0.1 "$SERVER"

Options:

• -i 0.1 — one ICMP echo request every 100 ms.
• "$SERVER" — destination stored in a shell variable.

Purpose: Record baseline latency, transient spikes, and packet loss while throughput tests are running.

One long run during the incident produced:

0.364697% packet loss
0.804 ms minimum
4.056 ms average
113.599 ms maximum

That was useful context, but ping did not identify the failed component by itself.

Treat it as supporting evidence.

Layer 4 — TCP throughput, direction, retransmissions, and congestion behavior

Start the iperf3 server

iperf3 -s

-s / --server starts server mode. By default, iperf3 listens on TCP port 5201.⁶

Standard single-stream client test

iperf3 -c "$SERVER" -t 30 -O 3

Options:

• -c "$SERVER" — connect to the test server.
• -t 30 — run the measured transfer for 30 seconds.
• -O 3 — omit the first three seconds from reported measurements to get past startup behavior such as TCP slow start.⁶

Direction:

client → server

Reverse the test direction

iperf3 -c "$SERVER" -R -t 30 -O 3

-R / --reverse makes the server send the test data to the client.⁶

This option was essential in the original incident because one physical direction carried roughly 430 Mbps, while the reverse direction collapsed to roughly 18 Mbps.

If you only test one direction, you can miss the fault.

Four parallel TCP streams

iperf3 -c "$SERVER" -P 4 -t 30 -O 3
iperf3 -c "$SERVER" -P 4 -R -t 30 -O 3

-P 4 / --parallel 4 runs four concurrent test streams.⁶

Why it is useful:

• it can show aggregate path capacity when one TCP flow is temporarily constrained;
• it helps distinguish a single-flow limitation from a path-wide ceiling;
• it does not make a damaged link healthy—major loss will still appear as retransmissions and poor aggregate throughput.

Shorter final AP validation

The final isolated U6 check used:

iperf3 -c "$SERVER" -P 4 -t 20 -O 3
iperf3 -c "$SERVER" -P 4 -R -t 20 -O 3

Observed aggregate results:

wired → Wi-Fi: 721 Mbps
Wi-Fi → wired: 722 Mbps

That did not prove every possible workload or radio condition. It cleared the tested local path well enough to reject the earlier idea that the U6 itself had a hard throughput ceiling around 200 Mbps.

Read the output fields that explain the bitrate

Bitrate

Application-level average data rate for the reporting interval.

Retr

TCP retransmissions reported by the sender.

A small number can occur during real-world testing. Thousands of retransmissions combined with collapsed throughput are evidence of a severely impaired TCP path, not merely a lower nominal link rate.

Cwnd

The TCP congestion window.

During the early bad direction, thousands of retransmissions drove the congestion window down to only a few kilobytes. That explained why the transfer stayed near 18–22 Mbps despite much faster nominal links elsewhere in the path.

sender and receiver

The end-of-test summaries from each endpoint’s perspective.

Use completed tests and compare both lines. Interrupted tests, asymmetric loss, or version differences can make partial summaries misleading.

Layer 7 — User-experience confirmation

Internet speed test

Use a browser-based internet speed test after the internal path has been measured.

It is useful for answering:

Does the user-visible internet experience now look right?

It is poor at answering:

Which internal segment is broken?

An internet speed test includes the client, AP, switch, PowerBeam bridge, router, ISP, and remote test server. It should be the final confirmation after local and bridge-only iperf3 tests.

Physical A/B tests that need no command

Substitute the injector

100 Mbps passive-PoE injector
                ↓
gigabit-capable passive-PoE injector

Observed result: throughput increased into the several-hundred-megabit range.

That was a confirmed physical ceiling, not a tuning problem.

Bypass the inline coupler panel

Compare:

EX3300 → coupler panel → injector

against:

EX3300 → injector directly

Observed behavior:

panel present:
  degradation
  severe asymmetry
  eventual 100 Mbps renegotiation

panel bypassed:
  1 Gbps negotiation
  roughly 400/400 Mbps observed wired bridge throughput

The repeated A/B result convicted the coupler panel more strongly than a single clean counter snapshot could have cleared it.

Swap order for a future 100 Mbps recurrence

Change only one variable at a time:

1. Replace the short EX3300-to-injector patch cable.
2. Move to another EX3300 port.
3. Substitute a known-good gigabit-capable passive-PoE injector.
4. Inspect or reterminate the outdoor cable and RJ45.
5. Suspect the remote PowerBeam Ethernet PHY last.

Before each physical change, capture the negotiated state and counters. After each change, record what changed.

Compact field sequence

When the problem returns, this is the short version:

1. Capture switch/NIC link state before touching anything.
2. Confirm 1 Gbps/full duplex where expected.
3. Check counters, carrier transitions, EEE, and radio state.
4. Prove the two test endpoints are on the intended interfaces/VLANs.
5. Use ip route get to prove the actual Layer-3 path.
6. Run iperf3 forward.
7. Run iperf3 reverse.
8. Compare one stream with four streams.
9. Watch retransmissions, congestion window, signal, and PHY rate.
10. Shorten the path until the fault disappears.
11. A/B one physical component at a time.
12. Run the internet speed test last.

The most important habit is still the one from Part 1:

Capture state first. Shorten the path. Test both directions. Change one variable at a time.

Part 3 will turn the same incident into an evidence ledger: what each test proved, which suspects were actually eliminated, what remained open, and the exact restart point for the next troubleshooting session.

References