ONTAP workload triage · AFF/FAS example · host and application correlation required
Illustrative incident: a database team reports write response time rising from 3 ms to 40 ms at 14:10. Another application on the same cluster is healthy. The task is to find the layer responsible for the observed delay, not to tune every component at once.
About the command and output examples
Names, addresses and results are illustrative. Commands have not been executed against a live storage system or SAN switch for this publication. Confirm syntax and compatibility for your release, then rehearse in an approved lab. Existing-data recovery and destructive changes need an environment-specific plan. Read the editorial standards.
01Define the operation and comparison group
| Metric | Record together | Reason |
|---|---|---|
| Latency | Read/write/other and percentile or average | Different operations and averages can hide tail delay. |
| IOPS | Operation type and interval | A small number of slow requests can dominate an average. |
| Throughput | MiB/s and average request size | High bandwidth is not automatically high IOPS. |
| Errors | Host/fabric counters and reset times | A counter total without a time window is ambiguous. |
| Placement | Current node/LIF and recent change | Remote-node paths can alter the route. |
- Ask whether the delay is application commit time, host block I/O latency, a file open, a metadata listing or bulk transfer. Record the measured unit, sampling interval and source of the number.
- Map the affected application to its host, protocol, SVM, volume/LUN and owning node. Note failovers, volume moves, backups, security product changes and imports in the same window.
- Choose a comparison: the same operation before 14:10, a healthy volume on the same node, or another client accessing the same export. Synchronize timestamps; a five-minute application average cannot be directly compared to a one-second storage peak.
02Gather a bounded ONTAP sample
- Use System Manager or Unified Manager to locate the affected workload. Capture a short sample during the symptom and the same view during a healthy interval.
- Where supported, collect volume performance and latency breakdown. The following 9.14-style commands run a finite number of iterations; field names and subsystem labels differ by ONTAP release.
- Also capture health, capacity and relevant EMS events. Keep the session read-only while establishing the cause.
qos statistics volume performance show -vserver svm_prod -volume vol_db -iterations 10
qos statistics volume latency show -vserver svm_prod -volume vol_db -iterations 10
system health alert show
volume show-space -vserver svm_prod -volume vol_db
event log show -severity ERRORThe sample covers the actual affected workload and the time of the application symptom.
03Interpret an illustrative latency breakdown
A component correlation gives you a candidate test, not automatic proof. For instance, a QoS Max delay can justify reviewing a ceiling against the approved service objective. It does not authorize removing a ceiling that protects other tenants. Compare the policy, observed request rate and shared workload impact.
| Illustrative observation | Hypothesis | What could disprove it |
|---|---|---|
| Application commit 40 ms; ONTAP volume 3 ms | Delay may be before/after the storage operation or in application dependencies. | An aligned storage sample showing a matching write spike. |
| QoS Max component rises after policy change | The workload may be waiting at its ceiling. | No matching policy or no observed ceiling delay. |
| Disk component rises across workloads sharing a pool | Storage resource pressure may be shared. | Other affected workloads do not use that resource/time window. |
| Only a remote-node workload has Cluster delay | Placement or interconnect is worth examining. | The measured route does not cross that node boundary. |
| SMB opens slow; sustained copying normal | Metadata, identity, Vscan or FPolicy may be involved. | The same delay occurs in an unrelated block workload. |
04Include the host and network in the diagnosis
- For SAN, examine multipath states, device queueing, HBA/NIC errors, transport retries and fabric congestion in the same period. A degraded path can increase application delay even when the LUN remains online.
- For NAS, separate DNS/session establishment from metadata and data I/O. Review retransmits and interface errors, then identity services and scanning servers when the slow operation is consistent with those dependencies.
- Compare block or file request sizes. As a sanity check, 20,000 requests/second at 8 KiB is about 156.25 MiB/s before protocol overhead; a 600 MiB/s backup with large sequential requests is a different workload.
The candidate layer explains both the affected workload and the healthy comparison group.
05Test one reversible hypothesis and keep a counterexample
Worked exercise: an hourly report job begins at 14:10; the shared workload hits its configured throughput ceiling, and QoS Max delay rises. The application team pauses only that report in an approved test. Delay falls while the production write rate remains comparable. This supports a scheduling or service-allocation change. If production traffic also fell, the result would be ambiguous and needs a better comparison.
- Define the predicted result before changing anything. Capture a baseline, authorize the narrow change and set an end time.
- Apply one controlled change such as rescheduling a confirmed batch job. Capture application and storage metrics again. Avoid simultaneous LIF, QoS and host tuning.
- Restore the prior setting if the prediction fails or another service regresses. Retain the counterexample; it reduces the search space for the next test.
The intervention improves the original operation under a comparable workload and does not cause an unacceptable shared-service regression.
06Close with evidence and a measurable service objective
- Record the symptom, aligned metrics, confirmed cause, change and before/after result. Preserve relevant logs and any support case reference.
- Set workload-specific thresholds based on a known healthy baseline and service objective. One universal latency threshold is inadequate for metadata browsing, database writes and backup transfer.
- Review again during the next peak period. A quiet-period test is useful but cannot prove peak capacity.
Use the section links to revisit a step, or follow a related guide below.
Official references and further reading
Choose the matching release in the vendor documentation before using a command or design. References open on the publisher’s website; linked material remains with its publisher.