Cisco MDS troubleshooting: missing LUNs, isolated VSANs and slow drain

01Establish impact and preserve the evidence

1. Record the affected host/application, first/last failure time, persistent LUN identity, missing fabric/path and most recent change. Determine whether I/O is stopped or continuing with reduced resilience.
2. Ask the host/application owner to verify the surviving service path. Pause additional work on the other fabric.
3. Save relevant interface, zoning, login and event outputs with timestamps. Avoid clearing counters, resetting ports or restarting a switch before collecting the failing state.

show version
show module
show environment
show interface brief
show logging logfile
show zone status vsan 10

02Scenario: the host FC port is down

1. Read the interface reason and administrative state. Compare the port with the approved worksheet before considering no shutdown.
2. Verify the host HBA is enabled, the peer and local optics are supported, negotiated/forced speed is compatible and fiber reaches the correct endpoint.
3. For media investigation, inspect/clean or replace one approved component at a time after confirming service resilience. Record the old/new component and retest the same link.
4. Verify stable online state, the expected WWPN login and no increasing physical-error delta under representative traffic.

show interface fc1/1
show interface fc1/1 transceiver details
show interface fc1/1 counters detailed
show running-config interface fc1/1
show vsan 10 membership

03Scenario: port online but the expected initiator is absent

An online link without the expected pWWN points to an earlier stage than target LUN presentation. Check the host adapter identity/state, correct physical port/VSAN, NPIV requirements and relevant login or security policy. Do not add a guessed WWPN to zoning just because a zone name looks familiar.

show flogi database interface fc1/1
show flogi database vsan 10
show fcns database vsan 10
show feature
show logging logfile

04Scenario: local devices work but remote targets disappeared

1. Determine whether the entire ISL/channel is down or only one VSAN is isolated. Compare physical state, operational channel members and allowed/active/up VSAN sets.
2. Check that the intended VSAN exists and is active on both peers. An allowed-list replacement can accidentally omit the production VSAN.
3. Read isolation/merge events and compare domain identities, zone-mode/policy/database conflicts and relevant security settings. Resolve the stated mismatch through an approved scoped change.
4. Verify remote FCNS registrations, target sessions and host paths after correction. Do not delete the zoning database or change a domain ID simply to make the link join.

show topology vsan 10
show fcdomain domain-list vsan 10
show interface trunk vsan 10
show interface fc1/47
show trunk protocol
show port-channel summary
show fcns database vsan 10

05Scenario: both WWPNs exist but the host sees no target

1. Compare the exact initiator and target pWWNs in the active policy. Similar aliases and node WWNs are not sufficient.
2. Check whether the new zone exists only in the full database, or whether enhanced activation is still pending an owned commit.
3. If aliases changed, compare their current mappings and actual consumer mode. Coordinate the correct activation/commit rather than force-clearing a lock.
4. Validate the target-side login and client service after the approved correction. Retain the new active policy and persistence record.

show zone status vsan 10
show zoneset active vsan 10
show zoneset vsan 10
show zone pending-diff vsan 10
show device-alias database

06Scenario: target reachable but one LUN is missing

When the initiator-target session works and other LUNs are available through it, inspect the array’s host identity, igroup membership, LUN mapping and reporting/path restrictions. Confirm the actual LUN identity and correct OS procedure for discovery. Formatting or initializing an unexpected device is not a diagnostic step. A switch does not need a separate ordinary WWPN zone for each SCSI LUN.

# ONTAP admin:
vserver fcp initiator show -vserver svm_lab
lun igroup show -vserver svm_lab
lun mapping show -vserver svm_lab
# Linux host:
multipath -ll
lsblk -o NAME,SIZE,TYPE,MOUNTPOINTS
# Windows host:
mpclaim -s -d
Get-Disk

07Scenario: CRC or invalid-word counts increase

1. Collect both link ends and the relevant traffic interval. Locate where receive errors are observed; correlate signal/sync loss, resets and diagnostic alarms.
2. Review the cable route, connector condition, supported optics and peer transmitter. Optical power alone does not prove an error-free link.
3. Replace or clean one approved suspect component, then collect a comparable interval. Retain evidence rather than declaring success from reset-to-zero counters.

show interface fc1/1 counters detailed
show interface fc1/1 transceiver details
show logging onboard error-stats

Synthetic example:
12:00 receive CRC count = 120
12:05 receive CRC count = 135
Delta = 15 in 5 minutes; this is an active increase.

08Scenario: application latency with TxWait and few CRC errors

Credit waiting is a flow-control observation. Correlate transmit waiting, traffic rate and timeout/discard behavior with downstream paths, receiving endpoints and application timing. A slow receiver can back up shared upstream resources. High utilization, optical corruption and credit pressure can overlap, but they are different hypotheses.

1. Map the direction of the affected traffic and identify shared ISLs and the suspected receiver. Compare unaffected ports and the same time interval.
2. Check endpoint queues, target latency and host behavior. A switch counter cannot by itself assign fault to storage or server software.
3. Review OBFL TxWait records with their logging threshold: no stored entry does not prove no credit waiting occurred.
4. Apply isolation, portguard, timeout or congestion-feature changes only through the platform-specific reviewed design. First test the identified endpoint/path cause and measure the result.

show interface fc1/1 counters detailed
show interface fc1/47 counters detailed
show logging onboard txwait
show logging onboard error-stats

09Scenario: a port-channel member will not join

1. Verify both members actually connect the same intended switch pair. Compare speed/capabilities, port VSAN, mode, trunk settings and allowed VSANs.
2. Read the stated incompatible/down reason on both ends. Confirm the channel ID and peer membership in the actual design.
3. Correct only the documented mismatch in the window, acknowledging any link flap. Do not use force to overwrite a live interface blindly.
4. Recheck member/channel health, intended VSAN carriage, traffic and host/application path counts.

show port-channel summary
show port-channel database
show interface port-channel 10
show interface fc1/47
show interface fc1/48
show running-config interface fc1/47
show running-config interface fc1/48

10Close with service evidence and useful escalation data

1. Recheck modules, links, expected registrations and effective zoning. Compare required objects, not just the absence of an alert.
2. Have the owner accept the intended application transaction and restored supported path resilience. Observe a relevant load/backup cycle where the incident was intermittent.
3. Retain the accepted configuration and authorized support evidence. Diagnostic bundles can contain sensitive configuration; store/share them through the approved support process.