CONCERTO: Extend the size of root partition
By Paul Christian Ella, Solutions Architect, Versa Networks 25/06/2024Introduction
In many production deployments, concerto runs 3-node cluster made of 1 primary, 1 secondary and 1 arbiter having each 16 cores CPU, 32 GB RAM and 512 GB DISK, preferably SSD.
In some corner cases, it may happen that the root partition is configured with a size less than the VM disk space and this can lead to concerto cluster instability when root partition is full.The purpose of this document is to provide a procedure to resize the system root partition to the VM disk size.
The procedure must apply at each node of the cluster without service restart. Even though this activity is not service impacting, we highly recommend performing it during maintenance window.
DisclaimerWe are not responsible of any issue which may occur while applying this procedure. We highly recommend you to get in touch with Versa representative or Versa TAC to assess and validate you setup before proceeding with any change.
Procedure
Before the change:
- Perform Concerto backup of save a copy to an external drive (procedure out of scope).
- Run the following commands and save the session: lsblk, df -kh
Activities to perform during the change:
- Identify the partition to resize and extend
admin@concerto-primary:~$ sudo lvdisplay
--- Logical volume ---
LV Path /dev/system/root
LV Name root
VG Name system
LV UUID MN1U2c-WfV9-Gy9C-d3Jo-SXcD-NAoi-8XciMC
LV Write Access read/write
LV Creation host, time versa-ecp, 2023-09-25 09:47:01 -0700
LV Status available
# open 1
LV Size <149.04 GiB
Current LE 38154
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0
…
admin@concerto-primary:~$ sudo pvdisplay
--- Physical volume ---
PV Name /dev/sda1
VG Name system
PV Size <150.00 GiB / not usable 2.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 38399
Free PE 0
Allocated PE 38399
PV UUID 0cNbJm-WZfm-aqBO-4QRI-V8Pb-IDrJ-0aeEIO
Resize the root partition:
- sudo growpart <PV Name>
- Resize the lvm partition:
- – sudo pvresize <PV Name>
- – sudo lvm lvextend -l +100%FREE <LV Path>
- Resize the filesystem: sudo resize2fs -p /dev/mapper/system-root
After the change:
- Perform Concerto backup and save a copy to an external drive.
- Run the following commands and save the session: lsblk, df -kh
Example
Concerto Virtual Disk is set to 198.8G but root partition has 149G.admin@versa-ecp:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk sda 8:0 0 198.8G 0 disk `-sda1 8:1 0 150G 0 part |-system-root 253:0 0 149G 0 lvm / `-system-swap_1 253:1 0 980M 0 lvm [SWAP] sr0 11:0 1 1024M 0 rom admin@versa-ecp:~$1. Resize the root partition using grow part command
sudo growpart /dev/sda 1 (1 means the 1st partition on /dev/sda or /dev/sda1) admin@versa-ecp:~$ sudo growpart /dev/sda 1 [sudo] password for admin: CHANGED: partition=1 start=2048 old: size=314568704 end=314570752 new: size=416970719,end=416972767As we can see below, the volume of /dev/sda1 is already resize to 198.8G
admin@versa-ecp:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk sda 8:0 0 198.8G 0 disk `-sda1 8:1 0 198.8G 0 part |-system-root 253:0 0 149G 0 lvm / `-system-swap_1 253:1 0 980M 0 lvm [SWAP] sr0 11:0 1 1024M 0 rom2. Resize the lvm / (root) to 198.8G
admin@versa-ecp:~$ sudo pvresize /dev/sda1 Physical volume "/dev/sda1" changed 1 physical volume(s) resized / 0 physical volume(s) not resized admin@versa-ecp:~$ sudo lvm lvextend -l +100%FREE /dev/system/root Size of logical volume system/root changed from <149.04 GiB (38154 extents) to <197.87 GiB (50654 extents). Logical volume system/root successfully resized. admin@versa-ecp:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT fd0 2:0 1 4K 0 disk sda 8:0 0 198.8G 0 disk `-sda1 8:1 0 198.8G 0 part |-system-root 253:0 0 197.9G 0 lvm / `-system-swap_1 253:1 0 980M 0 lvm [SWAP] sr0 11:0 1 1024M 0 rom3. The lvm disk is successfully extended. But if we run the df -h command, it still show 146G. Filesystems resize is required.
admin@versa-ecp:~$ df -kh Filesystem Size Used Avail Use% Mounted on udev 7.8G 0 7.8G 0% /dev tmpfs 1.6G 7.3M 1.6G 1% /run /dev/mapper/system-root 146G 17G 122G 12% / tmpfs 7.9G 12K 7.9G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup tmpfs 1.6G 0 1.6G 0% /run/user/1000 admin@versa-ecp:~$ admin@versa-ecp:~$ sudo resize2fs -p /dev/mapper/system-root resize2fs 1.44.1 (24-Mar-2018) Filesystem at /dev/mapper/system-root is mounted on /; on-line resizing required old_desc_blocks = 19, new_desc_blocks = 25 The filesystem on /dev/mapper/system-root is now 51869696 (4k) blocks long.Now,the lvm partition is resized.
admin@versa-ecp:~$ df -kh Filesystem Size Used Avail Use% Mounted on udev 7.8G 0 7.8G 0% /dev tmpfs 1.6G 7.3M 1.6G 1% /run /dev/mapper/system-root 194G 17G 168G 9% / tmpfs 7.9G 12K 7.9G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup tmpfs 1.6G 0 1.6G 0% /run/user/1000This concludes our article.
