Concerto: Extend the size of root partition

What are your feelings
Updated on June 25, 2024

CONCERTO: Extend the size of root partition

By Paul Christian Ella, Solutions Architect, Versa Networks 25/06/2024

Introduction

In many production deployments, concerto runs 3-node cluster made of 1 primary, 1 secondary and 1 arbiter having each 16 cores CPU, 32 GB RAM and 512 GB DISK, preferably SSD.

In some corner cases, it may happen that the root partition is configured with a size less than the VM disk space and this can lead to concerto cluster instability when root partition is full.

The purpose of this document is to provide a procedure to resize the system root partition to the VM disk size.

The procedure must apply at each node of the cluster without service restart. Even though this activity is not service impacting, we highly recommend performing it during maintenance window.

Disclaimer

We are not responsible of any issue which may occur while applying this procedure. We highly recommend you to get in touch with Versa representative or Versa TAC to assess and validate you setup before proceeding with any change.

Procedure

Before the change:

  • Perform Concerto backup of save a copy to an external drive (procedure out of scope).
  • Run the following commands and save the session: lsblk, df -kh

Activities to perform during the change:

  • Identify the partition to resize and extend
Get details about LV Path, LV Name and VG Name
  admin@concerto-primary:~$ sudo lvdisplay
  --- Logical volume ---
  LV Path                /dev/system/root
  LV Name                root
  VG Name                system
  LV UUID                MN1U2c-WfV9-Gy9C-d3Jo-SXcD-NAoi-8XciMC
  LV Write Access        read/write
  LV Creation host, time versa-ecp, 2023-09-25 09:47:01 -0700
  LV Status              available
  # open                 1
  LV Size                <149.04 GiB
  Current LE             38154
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0
  …

  admin@concerto-primary:~$ sudo pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda1
  VG Name               system
  PV Size               <150.00 GiB / not usable 2.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              38399
  Free PE               0
  Allocated PE          38399
  PV UUID               0cNbJm-WZfm-aqBO-4QRI-V8Pb-IDrJ-0aeEIO
Resize the root partition:
  • sudo growpart <PV Name>
  • Resize the lvm partition:
    • – sudo pvresize <PV Name>
    • – sudo lvm lvextend -l +100%FREE <LV Path>
  • Resize the filesystem: sudo resize2fs -p /dev/mapper/system-root

After the change:

  • Perform Concerto backup and save a copy to an external drive.
  • Run the following commands and save the session: lsblk, df -kh
 

Example

Concerto Virtual Disk is set to 198.8G but root partition has 149G.
admin@versa-ecp:~$ lsblk
NAME              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
fd0                 2:0    1     4K  0 disk 
sda                 8:0    0 198.8G  0 disk 
`-sda1              8:1    0   150G  0 part 
  |-system-root   253:0    0   149G  0 lvm  /
  `-system-swap_1 253:1    0   980M  0 lvm  [SWAP]
sr0                11:0    1  1024M  0 rom  
admin@versa-ecp:~$
1. Resize the root partition using grow part command
sudo growpart /dev/sda 1 (1 means the 1st partition on /dev/sda or /dev/sda1)
admin@versa-ecp:~$ sudo growpart /dev/sda 1
[sudo] password for admin:
CHANGED: partition=1 start=2048 old: size=314568704 end=314570752 new: size=416970719,end=416972767
As we can see below, the volume of /dev/sda1 is already resize to 198.8G
admin@versa-ecp:~$ lsblk
NAME              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
fd0                 2:0    1     4K  0 disk 
sda                 8:0    0 198.8G  0 disk 
`-sda1              8:1    0 198.8G  0 part 
  |-system-root   253:0    0   149G  0 lvm  /
  `-system-swap_1 253:1    0   980M  0 lvm  [SWAP]
sr0                11:0    1  1024M  0 rom  
2. Resize the lvm / (root) to 198.8G
admin@versa-ecp:~$ sudo pvresize /dev/sda1
  Physical volume "/dev/sda1" changed
  1 physical volume(s) resized / 0 physical volume(s) not    resized

admin@versa-ecp:~$ sudo lvm lvextend -l +100%FREE /dev/system/root
  Size of logical volume system/root changed from <149.04 GiB (38154 extents) to <197.87 GiB (50654 extents).
  Logical volume system/root successfully resized.

admin@versa-ecp:~$ lsblk
NAME              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
fd0                 2:0    1     4K  0 disk 
sda                 8:0    0 198.8G  0 disk 
`-sda1              8:1    0 198.8G  0 part 
  |-system-root   253:0    0 197.9G  0 lvm  /
  `-system-swap_1 253:1    0   980M  0 lvm  [SWAP]
sr0                11:0    1  1024M  0 rom  

3. The lvm disk is successfully extended. But if we run the df -h command, it still show 146G. Filesystems resize is required.
admin@versa-ecp:~$ df -kh
Filesystem               Size  Used Avail Use% Mounted on
udev                     7.8G     0  7.8G   0% /dev
tmpfs                    1.6G  7.3M  1.6G   1% /run
/dev/mapper/system-root  146G   17G  122G  12% /
tmpfs                    7.9G   12K  7.9G   1% /dev/shm
tmpfs                    5.0M     0  5.0M   0% /run/lock
tmpfs                    7.9G     0  7.9G   0% /sys/fs/cgroup
tmpfs                    1.6G     0  1.6G   0% /run/user/1000
admin@versa-ecp:~$

admin@versa-ecp:~$ sudo resize2fs -p /dev/mapper/system-root
resize2fs 1.44.1 (24-Mar-2018)
Filesystem at /dev/mapper/system-root is mounted on /; on-line resizing required
old_desc_blocks = 19, new_desc_blocks = 25
The filesystem on /dev/mapper/system-root is now 51869696 (4k) blocks long.
Now,the lvm partition is resized.
admin@versa-ecp:~$ df -kh
Filesystem               Size  Used Avail Use% Mounted on
udev                     7.8G     0  7.8G   0% /dev
tmpfs                    1.6G  7.3M  1.6G   1% /run
/dev/mapper/system-root  194G   17G  168G   9% /
tmpfs                    7.9G   12K  7.9G   1% /dev/shm
tmpfs                    5.0M     0  5.0M   0% /run/lock
tmpfs                    7.9G     0  7.9G   0% /sys/fs/cgroup
tmpfs                    1.6G     0  1.6G   0% /run/user/1000
This concludes our article.