Checking Disk Health with smartctl and Diagnosing I/O Failures (iotop, dmesg)

smartctl, from the smartmontools package, queries a drive’s SMART (Self-Monitoring, Analysis and Reporting Technology) data to reveal disk health: smartctl -a /dev/sda dumps the health verdict, attribute table, and self-test log in one shot. The critical skill is interpretation — the overall status can read PASSED right up until a drive dies, so you diagnose by trending attributes like Reallocated_Sector_Ct over time and corroborating with kernel-level evidence from dmesg.

What SMART tells you (and what it doesn’t)

SMART is firmware-level self-monitoring built into virtually every modern HDD, SATA SSD, and NVMe drive. The drive continuously tracks internal counters — remapped sectors, seek errors, temperature, power-on hours — and exposes them to the host. The overall health assessment is a single pass/fail verdict computed by the drive vendor’s own thresholds, and those thresholds are deliberately conservative: the verdict flips to FAILED only when an attribute crosses a limit that usually means the drive is already in serious trouble.

That design has a practical consequence: PASSED means “no threshold crossed yet,” not “healthy.” A drive can be actively deteriorating — remapping sectors week after week — while still reporting PASSED. Administrators who only check the verdict get blindsided; administrators who trend the raw attribute values see failures coming.

Essential smartctl commands

smartctl -i /dev/sda        # identity: model, serial, firmware, SMART support
smartctl -H /dev/sda        # overall health verdict only
smartctl -a /dev/sda        # everything: health, attributes, error and self-test logs
smartctl -x /dev/sda        # extended everything, incl. vendor-specific pages
smartctl -t short /dev/sda  # run a short self-test (runs in-drive, ~1-2 min)
smartctl -t long /dev/sda   # run an extended surface self-test
smartctl -l selftest /dev/sda   # read results of completed self-tests
smartctl -l error /dev/sda      # drive's internal error log

All of these need root, since they talk to the device directly. NVMe drives use the same tool (smartctl -a /dev/nvme0) but report a different, simpler attribute set — media errors, percentage used, available spare.

Self-test	Short	Extended (long)
Duration	Roughly 1–2 minutes	Minutes to many hours (full surface)
Coverage	Electrical/mechanical checks plus a small read sample	Reads every sector on the drive
Impact	Negligible; safe during production	Runs in-drive at low priority, but plan for the duration
Use case	Quick triage, scheduled daily	Suspected media problems, pre-deployment burn-in

Self-tests run inside the drive itself — the host just requests them and reads results later from the self-test log.

The attributes that predict failure

In the ATA attribute table, the RAW_VALUE column is where the diagnosis lives. Four attributes correlate strongly with impending death:

• Reallocated_Sector_Ct (ID 5) — sectors the drive found bad and remapped to spare area. Any nonzero value is a flag; a rising trend is the classic signature of progressive media failure.
• Current_Pending_Sector (ID 197) — sectors the drive couldn’t read and is waiting to remap. These represent data at risk right now.
• Offline_Uncorrectable (ID 198) — sectors that failed even offline correction.
• UDMA_CRC_Error_Count (ID 199) — interface CRC (cyclic redundancy check) errors; a climbing count usually implicates the SATA cable or backplane, not the platters — a genuinely different repair.

The trend matters more than the snapshot. A drive whose reallocated count climbed from 12 to 187 in a month is consuming its spare-sector pool at speed. The correct conclusion is that the drive is failing progressively and should be replaced proactively — backed up and swapped on your schedule, not the drive’s — even though the overall SMART verdict still says PASSED. Waiting for the verdict to flip means waiting until spares are nearly exhausted, at which point the next bad sector becomes unrecoverable data loss.

Confirming an active failure: dmesg and the kernel log

SMART shows the drive’s self-assessment; the kernel log shows what’s happening on the bus right now. When a disk is actively failing, the kernel ring buffer fills with lines like ata1.00: exception Emask, I/O error, dev sda, sector ..., blk_update_request: critical medium error. Read it with dmesg (or dmesg -T for human timestamps), or query the persistent journal for kernel messages with journalctl -k, covered in depth in journalctl commands.

This pairing is the standard proof pattern for a failing disk: kernel-logged I/O errors on the device plus SMART data showing media degradation. Together they distinguish hardware failure from the look-alikes — a corrupted filesystem (fsck findings, no underlying I/O errors), a flaky cable (CRC errors but clean media attributes), or a kernel bug (panic with no disk errors preceding it). In a boot-time panic where I/O errors on the root device appear immediately before the panic, those two sources — the kernel log and the drive’s SMART/self-test data — are what confirm the disk as the root cause. A failing drive you’re pulling from service should then be sanitized properly; see secure data destruction on Linux.

Watching live I/O: iotop and friends

iotop displays per-process disk read/write bandwidth in a top-like view — the tool for “which process is hammering the disk?” It works by reading the kernel’s taskstats accounting interface, and that interface is restricted: iotop requires root privileges. An unprivileged user running it gets a permission error even though the binary itself is world-executable — the restriction is kernel-side (per-process I/O accounting exposes information about other users’ processes), not a filesystem permissions problem on the binary. Run it with sudo iotop, or sudo iotop -oPa to show only processes actually doing I/O, per-process, with accumulated totals.

When you can’t use root, iostat -x 2 (from sysstat) reports per-device utilization, await times, and throughput without special privileges — you lose the per-process attribution but keep the workload picture. High %util with high await on one device corroborates a saturated or struggling disk.

How the XK0-006 exam tests this

• Trend-interpretation scenarios: a weekly smartctl -a habit reveals a climbing Reallocated_Sector_Ct while health still shows PASSED — the exam wants “drive is failing, replace proactively,” not “PASSED means fine” and not “reinstall/fsck.”
• Choose-two evidence questions: a crash or panic follows disk I/O errors, and you must pick the two sources that confirm hardware failure — kernel log (dmesg/journalctl -k) and SMART/self-test data — over distractors like application logs or package verification.
• Permission-error diagnosis: a non-root user gets a permission error from iotop despite an executable binary; the tested fact is that iotop needs root because of the kernel taskstats interface.
• Attribute discrimination: distinguishing media-failure attributes (5, 197, 198) from cabling symptoms (CRC error count) to pick the right fix.

Disk-health diagnosis anchors the Troubleshooting domain — the full XK0-006 study guide covers how that domain is tested. Reading SMART output cold is a skill; Linux+ practice questions are the fastest way to build it.

Quick reference

• smartctl -a /dev/sda = full report; -H = verdict only; -t short|long = run in-drive self-tests; -l selftest = read their results. Root required.
• PASSED is not a clean bill of health — it only means no vendor threshold has been crossed yet.
• Rising Reallocated_Sector_Ct (ID 5) = progressive media failure; back up and replace the drive proactively.
• Current_Pending_Sector (197) = unreadable sectors awaiting remap — data at risk now.
• UDMA_CRC_Error_Count (199) climbing points at the cable/backplane, not the platters.
• Confirm active failures with dmesg / journalctl -k: look for I/O error, dev ... and ATA exception lines.
• iotop shows per-process I/O but requires root (kernel taskstats); iostat -x gives unprivileged per-device stats.
• Failing-disk proof = kernel I/O errors + SMART degradation, together ruling out filesystem and kernel-bug explanations.