A CentOS server at 97% disk is not a maintenance item — it is minutes or hours from taking your services down in ways that look nothing like a disk problem. MySQL starts refusing writes, caches fail silently, logs stop recording the very errors you need to debug. When it happened on one of my own AWS boxes, the fix took under fifteen minutes with zero downtime, because every layer in the chain — EBS, the partition table, the XFS filesystem — was built to grow live. This is the exact sequence I ran, plus the triage step that decides whether you even need it.
First, prove it's a capacity problem
SSH in and look before you buy anything:
ssh -i "your-key.pem" centos@your-server-ip
df -h
The output that triggered this incident:
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p1 50G 48G 1.2G 97% /
Anything over 85–90% on a root volume is the danger zone. But do not resize yet — first find out what is eating the space:
sudo du -sh /* 2>/dev/null | sort -hr | head -n 10
sudo du -sh /var/* | sort -hr | head -n 10
This step matters because a bigger disk cannot fix a disk that is full of garbage. If the top entry is a runaway log file or a cache directory nobody rotates, deleting is the fix and you will be back at 97% next month if you expand instead. That was exactly the shape of a different incident I wrote up in how I saved my WordPress server from repeated 100% disk crashes — cleanup, not capacity.
One caution from lived ops: if /var/lib/mysql dominates the list, that is your database, not detritus. Do not "clean" it. Growing data in the database is legitimate growth, and it is the strongest signal that expanding the volume is the right call.
In my case the usage was real — application data and the database had simply outgrown 50 GiB. On to the expansion.
Snapshot, then resize the volume in the console
Take a snapshot of the volume first. The modify operation is safe in practice; a sixty-second snapshot converts "safe in practice" into "safe, period," and gives you a forensic copy if anything downstream surprises you.
Then, in the EC2 console:
- Volumes under Elastic Block Store
- Select the volume attached to your instance
- Actions → Modify Volume
- Change the size — I went from 50 GiB to 70 GiB
- Modify and confirm
No reboot, no detach, no downtime. The instance keeps serving traffic throughout. Wait for the volume state to leave modifying before the next step.
Grow the partition, then the filesystem — live
Back in the SSH session, confirm the kernel sees the new capacity:
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 70G 0 disk
└─nvme0n1p1 259:1 0 50G 0 part /
The disk is 70G; the partition is still 50G. Two commands close the gap, both safe on a mounted, running filesystem:
sudo growpart /dev/nvme0n1 1
sudo xfs_growfs -d /
Order matters and is easy to forget a year from now: partition first, filesystem second. growpart rewrites the partition table to claim the new blocks; xfs_growfs tells XFS to use them. (On ext4, the second command is sudo resize2fs /dev/nvme0n1p1 instead — everything else is identical.)
Verify:
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p1 70G 48G 22G 69% /
From 97% to 69%, services untouched. On my box, the symptoms that had started creeping in — failed cache writes, a service restart loop — stopped immediately.
The device-name confusion that stalls people here
One nuance trips up more people at this step than anything else: the AWS console may list the volume's device as /dev/xvda or /dev/sda1, while the instance itself shows /dev/nvme0n1. Both names refer to the same volume. On Nitro-based instances — which is most modern instance types — EBS volumes attach through the NVMe driver, so the OS names them nvme0n1, nvme1n1, and so on regardless of what the console says. Trust lsblk on the box, not the console label, when constructing the growpart command.
Two related syntax details worth pinning down, because getting them wrong produces confusing errors rather than helpful ones. First, growpart takes the disk and the partition number as two separate arguments — growpart /dev/nvme0n1 1, with a space, not growpart /dev/nvme0n1p1. Second, xfs_growfs takes the mount point, not the device — xfs_growfs -d /. Mixing those conventions up is the most common reason the "instant" fix turns into twenty minutes of man-page reading at the worst possible moment.
If you run several attached volumes and are not certain which NVMe device maps to which EBS volume, sudo nvme id-ctrl /dev/nvme0n1 -v | grep -i vol (or simply matching sizes in lsblk against the console) settles it before you grow anything. On a single-volume web server this is overkill; on a box with a separate data volume for the database, checking first is the difference between growing the right disk and merely growing a disk.
Why this works with zero downtime
Understanding the chain tells you what this procedure can and cannot do. EBS volumes resize under load. growpart updates the partition table while the partition is mounted. XFS and ext4 both grow online — the kernel simply sees more blocks and the filesystem claims them. Nothing unmounts, nothing restarts.
The same chain defines the limits: it grows, never shrinks, and it cannot rescue a disk being actively filled by a misbehaving process. Hence the du triage first.
Two post-expansion checks worth thirty seconds: confirm your application is writing successfully (not just that df looks right), and note that the volume state in the console moves to optimizing — on larger volumes that phase can run for hours with slightly reduced performance. It is normal; do not misread it as a new incident. Keep the pre-change snapshot until your first successful backup after the expansion.
Stop doing this reactively
Every expansion I have done under pressure was avoidable with two pieces of boring hygiene:
A CloudWatch alarm at 80% disk. The metric requires the CloudWatch agent on the instance, and it turns a 2 a.m. emergency into a calendar item three weeks out. The best version of this entire procedure is the one you schedule calmly for a Tuesday morning.
Log rotation that actually runs. Growth should be data, not detritus. If logs are a meaningful slice of your du output, fix rotation before you buy gigabytes.
The arithmetic favors headroom, too: the difference between 50 and 70 GiB of gp3 is a couple of dollars a month, against an outage that costs you an evening plus whatever your downtime is worth. Storage is the cheapest insurance in the AWS catalog. Size for a year and a half of growth and spend the worry budget elsewhere — the same reasoning behind the rest of my baseline server setup, from Supervisor-managed queues on Amazon Linux to automated SSL on the web tier: make the failure modes boring before they page you.
The one line to bookmark
You will need this in a year and forget it exists:
sudo growpart /dev/nvme0n1 1 && sudo xfs_growfs -d /
Partition first, filesystem second, done live. Sixty seconds from console resize to usable space — snapshot, resize, grow, verify, and back to work.
If your servers are the kind that only get attention when they scream, that is fixable: I set up disk alarms, log rotation, and capacity planning as part of the infrastructure work I do for clients — reach out and the next disk-full event on your stack can be an email instead of an outage.
