By Radu Pavaloiu
In this article We’ll discuss extensively the upgrade process of a Highly redundant HeadEnd Controller complex and VOS devices from 16.1R2 release to 20.2.3 release.
The HeadEnd Controller contains the following components:
- Versa Director
- Versa Analytics cluster/standalone node
- Versa Controller
Our lab topology looks like the bellow one:
Director-1 (Active Director), Analytics-1, Controller-1 are part of one DC (Datacenter-1).
Director-2(Standby Director), Analytics-2, Controller-2 are part of a second DC (Datacenter-2). Routing between the two Datacenters is provided by two layer 3 devices: IOU1 and IOU2. HA relationship between the Versa Directors is established via this routed connectivity.
Download the required software
For Versa Director, download the following packages:
1. versa-director-patch-20.2.3.bin
2. versa-director-20201002-061050-2d43eaa-20.2.3.bin
3. versa-director-osspack-20210428.bin
4. fusion_migration.v2.3.tar (or whichever is latest).
For Analytics, download the following packages:
1. versa-analytics-20201002-063050-969d704-20.2.3.bin
2. versa-analytics-osspack-20210428.bin
For Versa FlexVNF (Controller), download the following packages:
1. versa-flexvnf-20201002-063050-969d704-20.2.3.bin
2. versa-flexvnf-20201002-063550-969d704-20.2.3-wsm.bin (in case if CPU is Intel Westmere architecture)
3. versa-flexvnf-osspack-20210428.bin
Upgrade the controller complex
Versa Director HA upgrade from 16.1R2 to 20.2.3
Regarding compatibility Versa Director Release 20.2.3 is compatible with VOS devices running the following software versions:
- Release 20.2.3
- Release 20.2.2
- Release 16.1R2S11
- Release 16.1R2S10.1
- Release 16.1R2S9
Upload the software to the Versa Director nodes
Using WINSCP and/or SCP, upload the image files to the respective Versa Director nodes. Use the target directories as in the bellow examples.
Director-1
versa@Radu_Server:~/tmp$ scp versa-director-20201002-061050-2d43eaa-20.2.3.bin admin@Director-1:/var/versa/packages/vnms/ Warning: the ED25519 host key for 'director-1' differs from the key for the IP address '10.40.213.1' Offending key for IP in /home/versa/.ssh/known_hosts:5 Matching host key in /home/versa/.ssh/known_hosts:27 Are you sure you want to continue connecting (yes/no)? yes admin@director-1's password: versa-director-20201002-061050-2d43eaa-20.2.3.bin 100% 787MB 72.6MB/s 00:10 versa@Radu_Server:~/tmp$ scp versa-director-osspack-20210428.bin admin@Director-1:/var/versa/packages/vnms/ Warning: the ED25519 host key for 'director-1' differs from the key for the IP address '10.40.213.1' Offending key for IP in /home/versa/.ssh/known_hosts:5 Matching host key in /home/versa/.ssh/known_hosts:27 Are you sure you want to continue connecting (yes/no)? yes admin@director-1's password: versa-director-osspack-20210428.bin 100% 511MB 82.3MB/s 00:06 versa@Radu_Server:~/tmp$ scp versa-director-patch-20.2.3.bin admin@Director-1:/var/versa/packages/vnms/ Warning: the ED25519 host key for 'director-1' differs from the key for the IP address '10.40.213.1' Offending key for IP in /home/versa/.ssh/known_hosts:5 Matching host key in /home/versa/.ssh/known_hosts:27 Are you sure you want to continue connecting (yes/no)? yes admin@director-1's password: versa-director-patch-20.2.3.bin 100% 3532KB 51.6MB/s 00:00
Following the upload verify the MD5 checksum of the packages from Versa Director cli:
Administrator@Director-1> request system package list
packages {
name *.deb
}
packages {
name versa-director-20191010-151235-84ad5cf-16.1R2S10.bin
}
packages {
name versa-director-20201002-061050-2d43eaa-20.2.3.bin
}
packages {
name versa-director-osspack-20210428.bin
}
packages {
name versa-director-patch-20.2.3.bin
}
[ok][2021-07-15 09:12:00]
Administrator@Director-1> request system package check-integrity ?
Description: Bin file to be verify checksum
Possible completions:
Administrator@Director-1> request system package check-integrity versa-director-patch-20.2.3.bin
status Ok
[ok][2021-07-15 09:12:18]
Administrator@Director-1> request system package check-integrity versa-director-osspack-20210428.bin
status Ok
[ok][2021-07-15 09:12:46]
Administrator@Director-1> request system package check-integrity versa-director-20201002-061050-2d43eaa-20.2.3.bin
status Ok
[ok][2021-07-15 09:13:25]
Director-2
versa@Radu_Server:~/tmp$ scp versa-director-20201002-061050-2d43eaa-20.2.3.bin admin@Director-2:/var/versa/packages/vnms/ The authenticity of host 'director-2 (10.40.213.2)' can't be established. ED25519 key fingerprint is SHA256:G4CamORrzfZL21PuCGNPhVEZgUCDx5bj0N2SzExtR0E. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'director-2,10.40.213.2' (ED25519) to the list of known hosts. admin@director-2's password: versa-director-20201002-061050-2d43eaa-20.2.3.bin 100% 787MB 81.4MB/s 00:09 versa@Radu_Server:~/tmp$ scp versa-director-osspack-20210428.bin admin@Director-2:/var/versa/packages/vnms/ admin@director-2's password: versa-director-osspack-20210428.bin 100% 511MB 81.8MB/s 00:06 versa@Radu_Server:~/tmp$ scp versa-director-patch-20.2.3.bin admin@Director-2:/var/versa/packages/vnms/ admin@director-2's password: versa-director-patch-20.2.3.bin 100% 3532KB 58.2MB/s 00:00
Following the upload verify the MD5 checksum of the packages from Versa Director cli:
Administrator@Director-2> request system package list
packages {
name *.deb
}
packages {
name versa-director-20191010-151235-84ad5cf-16.1R2S10.bin
}
packages {
name versa-director-20201002-061050-2d43eaa-20.2.3.bin
}
packages {
name versa-director-osspack-20210428.bin
}
packages {
name versa-director-patch-20.2.3.bin
}
[ok][2021-07-15 10:30:24]
Administrator@Director-2> request system package check-integrity versa-director-20201002-061050-2d43eaa-20.2.3.bin
status Ok
[ok][2021-07-15 10:31:21]
Administrator@Director-2> request system package check-integrity versa-director-osspack-20210428.bin
status Ok
[ok][2021-07-15 10:32:07]
Administrator@Director-2> request system package check-integrity versa-director-patch-20.2.3.bin
status Ok
[ok][2021-07-15 10:32:23]
– Pre-upgrade steps and checks
Verify HA status and sync between the Versa Director nodes.
Director-1:
Administrator@Director-1> request vnmsha actions status status MASTER [ok][2021-07-15 23:44:35] Administrator@Director-1> request vnmsha actions check-sync-status postgres-status IN_SYNC ncs-status IN_SYNC [ok][2021-07-15 23:44:50]
Director-2:
Administrator@Director-2> request vnmsha actions status
status SLAVE [ok][2021-07-15 23:46:15] Administrator@Director-2> request vnmsha actions check-sync-status postgres-status IN_SYNC ncs-status IN_SYNC [ok][2021-07-15 23:46:20]
Create backup for the Master Director and Standby Director.
The generated backup is located at /var/versa/backups directory.
Note: Starting from 16.1R2S11 the backup is also copied across to the Slave Director. So, it is not required to generate an additional backup of the Slave Director in case the Versa Director is running 16.1R2S11, 20.2.x or 21.x.
As the current release in our case is 16.1R2S10 Versa Directors will be back-ed up.
Director-1
Administrator@Director-1> request system recovery backup
status Versa director backup initiated ..
[ok][2021-07-16 00:05:08]
Administrator@Director-1> request system recovery list
files {
name vnms@2021-07-08T06:25:15.backup
}
files {
name vnms@2021-07-12T06:25:04.backup
}
files {
name vnms@2021-07-15T06:25:05.backup
}
files {
name vnms@2021-07-16T00:05:08.backup
}
Director-2
Administrator@Director-2> request system recovery backup
status Versa director backup initiated ..
[ok][2021-07-16 00:15:55]
Administrator@Director-2> request system recovery list
files {
name vnms@2021-07-08T06:25:15.backup
}
files {
name vnms@2021-07-12T06:25:05.backup
}
files {
name vnms@2021-07-15T06:25:04.backup
}
files {
name vnms@2021-07-16T00:15:55.backup
}
[ok][2021-07-16 00:16:06]
Use SCP utility to copy the backup files out of the Versa Director nodes.
Director-1
versa@Radu_Server:~$ scp admin@Director-1:/var/versa/backups/vnms@2021-07-16T00:05:08.backup tmp/Director-1.bak Warning: the ED25519 host key for 'director-1' differs from the key for the IP address '10.40.213.1' Offending key for IP in /home/versa/.ssh/known_hosts:5 Matching host key in /home/versa/.ssh/known_hosts:27 Are you sure you want to continue connecting (yes/no)? yes admin@director-1's password: vnms@2021-07-16T00:05:08.backup 100% 5918KB 71.7MB/s 00:00
Director-2
versa@Radu_Server:~$ scp admin@Director-2:/var/versa/backups/vnms@2021-07-16T00:15:55.backup tmp/Director-2.bak admin@director-2's password: vnms@2021-07-16T00:15:55.backup 100% 5919KB 64.2MB/s 00:00
– OSSPACK update on Versa Director nodes
Director-1
cd /var/versa/packages/vnms/ sudo chmod a+x versa-director-osspack-20210428.bin sudo ./versa-director-osspack-20210428.bin
osspack update process takes several minutes. At the end verify if the osspack was installed successfully:
Administrator@Director-1> show security osspack info Version 20210428 Update Type full [ok][2021-07-16 00:53:32]
Director-2
cd /var/versa/packages/vnms/ sudo chmod a+x versa-director-osspack-20210428.bin sudo ./versa-director-osspack-20210428.bin Administrator@Director-2> show security osspack info Version 20210428 Update Type full [ok][2021-07-16 01:05:20]
– Install the patch file and run the validation script
Director-1 (Active Director)
cd /var/versa/packages/vnms/ sudo chmod a+x versa-director-patch-20.2.3.bin admin@Director-1:/var/versa/packages/vnms$ sudo ./versa-director-patch-20.2.3.bin Verifying archive integrity... All good. Uncompressing Versa Director Patch 100% => Patching Versa Director.. => Copying validation scripts.. => Fixing file permissions.. => Done.
To validate the upgrade procedure from release 16.1R2 to 20.2.3, use the following validation script on Primary Director
admin@Director-1:/var/versa/packages/vnms$ sudo /opt/versa/vnms/upgrade/scripts/validate.py -f 16.1R2S10 -t 20.2.3 INFO - = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ INFO - Executing validation script: ha-pair-config-validation.py ... Executing validation script: ha-pair-config-validation.py ... INFO - Successfully executed ha-pair-config-validation.py Successfully executed ha-pair-config-validation.py INFO - = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ INFO - Executing validation script: stale-bind-data-remover ... Executing validation script: stale-bind-data-remover ... INFO - Executing stale bind data remover script for 16.1R2 Executing stale bind data remover script for 16.1R2 INFO - Successfully executed stale-bind-data-remover Successfully executed stale-bind-data-remover INFO - = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ INFO - Executing validation script: auth-connector-validation.lua ... Executing validation script: auth-connector-validation.lua ... INFO - Successfully executed auth-connector-validation.lua Successfully executed auth-connector-validation.lua INFO - = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ INFO - Executing validation script: org-validation.py ... Executing validation script: org-validation.py ... INFO - Successfully executed org-validation.py Successfully executed org-validation.py INFO - = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~
Director-2 (STNDBY Director)
Remove /opt/versa/vnms/upgrade/validate/scripts/stale-bind-data-remove directory:
admin@Director-2:~$ sudo rm -rf /opt/versa/vnms/upgrade/validate/scripts/stale-bind-data-remover/
Run the validation script:
admin@Director-2:~$ sudo /opt/versa/vnms/upgrade/scripts/validate.py -f 16.1R2S10 -t 20.2.3 INFO - = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ INFO - Executing validation script: ha-pair-config-validation.py ... Executing validation script: ha-pair-config-validation.py ... INFO - Successfully executed ha-pair-config-validation.py Successfully executed ha-pair-config-validation.py INFO - = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ INFO - Executing validation script: auth-connector-validation.lua ... Executing validation script: auth-connector-validation.lua ... INFO - Successfully executed auth-connector-validation.lua Successfully executed auth-connector-validation.lua INFO - = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ INFO - Executing validation script: org-validation.py ... Executing validation script: org-validation.py ... INFO - Successfully executed org-validation.py Successfully executed org-validation.py INFO - = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~ = ~
If any of the scripts fail, take corrective actions to update the auth-connectors, ha-pair configs and org configurations. If you are unable to resolve the errors, engage Versa TAC for additional assistance.
– Upgrade software on Standby Versa Director
Start the upgrade process on the Standby Versa Director.
A second ssh session could be used to verify the upgrade progress and catch the eventual errors, by using the bellow command:
tail -f /var/log/vnms/upgrade.log
In the first ssh session start the upgrade process. Answer “yes” for restart versa director.
Administrator@Director-2> request system upgrade package versa-director-20201002-061050-2d43eaa-20.2.3.bin Will restart versa director (all processes). Are you sure? [no,yes] yes status Installing.. [ok][2021-07-19 07:09:46] Administrator@Director-2> Starting upgrade.. Stopping vnms service.. admin@Director-2:~$ Taking postgres DB backup.. Taking vnms backup.. Removing existing vnms package.. Installing vnms package.. Run migration scripts.. From 16.1R2 to 20.2.. Starting NCS in phase0.. Running phase0 scripts.. Starting NCS in phase12.. Starting postgres.. Merge xml files.. Running phase12 scripts.. Executing run-always scripts.. Migrating postgres database... Stopping NCS service Stopping postgres.. Migration completed. Starting vnms service.. admin@Director-2:~$ Status of vnms service.. Running Post Migration scripts after normal start. /opt/versa/vnms/upgrade/scripts/post_migrate.py --from-version 16.1R2 --to-version 20.2 Run post normal migration scripts.. From 16.1R2 to 20.2.. Running Post Normal scripts.. Post Normal Migration completed. Restarting vnms service.. admin@Director-2:~$ Cleaning up /tmp directory.. VNMS started successfully.. Upgrade to versa-director-20201002-061050-2d43eaa-20.2.3.bin successful!
Please be patient as the process could take more than 30 minutes.
Check if the new release running on the standby Versa Director is reported correctly:
Administrator@Director-2> show system package-info Package Versa Director Software Release 20.2.3 Release Type GA Release date 20201002 Package id 2d43eaa UI Package id 8e5261a Package name versa-director-20201002-061050-2d43eaa-20.2.3 Branch 20.2 Spack Version 1744
Stop services on the standby Director node
Run vsh stop on the standby Versa Director and verify the services stopped correctly.
admin@Director-2:~$ vsh stop Stopping VNMS service ------------------------------------ Stopping VNMS:TOMCAT.............[Stopped] Stopping VNMS:REDIS..............[Stopped] Stopping VNMS:POSTGRE............[Stopped] Stopping VNMS:SPRING-BOOT........[Stopped] Stopping VNMS:SPACKMGR...........[Stopped] Stopping VNMS:NCS................[Stopped] * Stopping daemon monitor monit [ OK ] admin@Director-2:~$ vsh status VNMS:NCS[4.7.3.3] is NOT RUNNING VNMS:SPACKMGR is NOT RUNNING VNMS:POSTGRE[9.5.17] is NOT RUNNING VNMS:SPRING-BOOT is NOT RUNNING VNMS:REDIS[5.0.5] is NOT RUNNING VNMS:TOMCAT[Apache Tomcat/9.0.37] is NOT RUNNING
– Upgrade software on Active Versa Director
Using the same procedure the software is upgraded on the Active Versa Director.
Administrator@Director-1> request system package upgrade versa-director-20201002-061050-2d43eaa-20.2.3.bin Will restart Versa Director (all processes). Are you sure? [no,yes] yes Verify package checksum.. Upgrading Versa Director from versa-director-20191010-151235-84ad5cf-16.1R2S10.bin to versa-director-20201002-061050-2d43eaa-20.2.3.bin status Installing.. [ok][2021-07-19 08:14:52] Administrator@Director-1> Starting upgrade.. Stopping vnms service.. admin@Director-1:~$ admin@Director-1:~$ Taking postgres DB backup.. Taking vnms backup.. Removing existing vnms package.. admin@Director-1:~$ Installing vnms package.. admin@Director-1:~$ Run migration scripts.. From 16.1R2 to 20.2.. Starting NCS in phase0.. Running phase0 scripts.. Starting NCS in phase12.. Starting postgres.. Merge xml files.. Running phase12 scripts.. Executing run-always scripts.. Migrating postgres database... Stopping NCS service Stopping postgres.. Migration completed. Starting vnms service.. Status of vnms service.. Running Post Migration scripts after normal start. /opt/versa/vnms/upgrade/scripts/post_migrate.py --from-version 16.1R2 --to-version 20.2 Run post normal migration scripts.. From 16.1R2 to 20.2.. Running Post Normal scripts.. Post Normal Migration completed. Restarting vnms service.. Cleaning up /tmp directory.. VNMS started successfully.. Upgrade to versa-director-20201002-061 050-2d43eaa-20.2.3.bin successful!
Check if the new release running on the active Versa Director is reported correctly:
Administrator@Director-1> show system package-info Package Versa Director Software Release 20.2.3 Release Type GA Release date 20201002 Package id 2d43eaa UI Package id 8e5261a Package name versa-director-20201002-061050-2d43eaa-20.2.3 Branch 20.2 Spack Version 1744
Verify that all services are running on the active Director node:
admin@Director-1:~$ vsh status [sudo] password for admin: VNMS:NCS[4.7.3.3] is RUNNING VNMS:SPACKMGR is RUNNING VNMS:POSTGRE[9.5.17] is RUNNING VNMS:SPRING-BOOT is RUNNING VNMS:REDIS[5.0.5] is RUNNING VNMS:TOMCAT[Apache Tomcat/9.0.37] is RUNNING
Start services on the standby Director node.
Start the services on the Standby Versa Director node and verify if they started successfully:
admin@Director-2:~$ vsh status [sudo] password for admin: VNMS:NCS[4.7.3.3] is NOT RUNNING VNMS:SPACKMGR is NOT RUNNING VNMS:POSTGRE[9.5.17] is NOT RUNNING VNMS:SPRING-BOOT is NOT RUNNING VNMS:REDIS[5.0.5] is NOT RUNNING VNMS:TOMCAT[Apache Tomcat/9.0.37] is NOT RUNNING admin@Director-2:~$ vsh start .---., ( ``. _ \ ) __ ________ _____ _____ ( `. \ / \ \ / / ____| __ \ / ____| /\ \ `. ) / \ \ / /| |__ | |__) | (___ / \ \ | / \ \/ / | __| | _ / \___ \ / /\ \ \ | / \ / | |____| | \ \ ____) / ____ \ \ | / \/ |______|_| \_\_____/_/ \_\ \ | / \_|/ ___ ___ ___ ___ ___ _____ ___ ___ | \_ _| _ \ __/ __|_ _/ _ \| _ \ | |) | || / _| (__ | || (_) | / |___/___|_|_\___\___| |_| \___/|_|_\ Starting VNMS service ------------------------------------ Starting VNMS:NCS................[Started] Starting VNMS:POSTGRE............[Started] Starting VNMS:SPRING-BOOT........[Started] Starting VNMS:REDIS..............[Started] Starting VNMS:TOMCAT.............[Started]
– Verify the HA status between Versa Director nodes.
Verify the Versa Director nodes role and the sync between them.
Director-1 (Active Versa Director)
Administrator@Director-1> request vnmsha actions status status MASTER [ok][2021-07-19 09:54:57] Administrator@Director-1> request vnmsha actions check-sync-status postgres-status IN_SYNC ncs-status IN_SYNC [ok][2021-07-19 09:55:06]
Director-2 (Standby Versa Director)
Administrator@Director-2> request vnmsha actions status status SLAVE [ok][2021-07-19 09:22:09] Administrator@Director-2> request vnmsha actions check-sync-status postgres-status IN_SYNC ncs-status IN_SYNC [ok][2021-07-19 09:22:19]
This concludes Versa Director HA upgrade process.
Versa Analytics upgrade from 16.1R2 to 20.2.3
For this upgrade we’ll use only one analytics/search node in each Datacenter: Analytics-1 and Analytics-2.
Upload the software to the Versa Analytics nodes
Using WINSCP and/or SCP, upload the image files to the respective Versa Analytics nodes. Use the target directories as in the bellow examples.
Analytics-1
versa@Radu_Server:~/tmp$ sudo scp versa-analytics-20201002-063050-969d704-20.2.3.bin versa@Analytics-1:/tmp versa@analytics-1's password: versa-analytics-20201002-063050-969d704-20.2.3.bin 100% 384MB 75.7MB/s 00:05 versa@Radu_Server:~/tmp$ sudo scp versa-analytics-osspack-20210428.bin versa@Analytics-1:/tmp versa@analytics-1's password: versa-analytics-osspack-20210428.bin 100% 177MB 77.3MB/s 00:02
Move the .bin files to /home/versa/packages:
[versa@Analytics-1: ~] $ sudo mv /tmp/versa-analytics-20201002-063050-969d704-20.2.3.bin /home/versa/packages/ [versa@Analytics-1: ~] $ sudo mv /tmp/versa-analytics-osspack-20210428.bin /home/versa/packages/
Following the upload verify the MD5 checksum of the packages from the Analytics cli:
[versa@Analytics-1: packages] $ md5sum versa-analytics-20201002-063050-969d704-20.2.3.bin 00fc14691cb610d5ba36c2697fc37ccd versa-analytics-20201002-063050-969d704-20.2.3.bin [versa@Analytics-1: packages] $ md5sum versa-analytics-osspack-20210428.bin 6c381aed1ad1be88b52f47345024e3b6 versa-analytics-osspack-20210428.bin
Analytics-2
versa@Radu_Server:~/tmp$ sudo scp versa-analytics-20201002-063050-969d704-20.2.3.bin versa@Analytics-2:/tmp The authenticity of host 'analytics-2 (10.40.213.4)' can't be established. ED25519 key fingerprint is SHA256:0wDqko/NwXIc19K4G2mwMmsTaFpFoyJlDaxkKeLmOm8. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'analytics-2,10.40.213.4' (ED25519) to the list of known hosts. versa@analytics-2's password: versa-analytics-20201002-063050-969d704-20.2.3.bin 100% 384MB 78.2MB/s 00:04 versa@Radu_Server:~/tmp$ sudo scp versa-analytics-osspack-20210428.bin versa@Analytics-2:/tmp versa@analytics-2's password: versa-analytics-osspack-20210428.bin
Move the .bin files to /home/versa/packages:
[versa@Analytics-2: ~] $ sudo mv /tmp/versa-analytics-20201002-063050-969d704-20.2.3.bin /home/versa/packages/ [sudo] password for versa: [versa@Analytics-2: ~] $ sudo mv /tmp/versa-analytics-osspack-20210428.bin /home/versa/packages/
Following the upload verify the MD5 checksum of the packages from Analytics cli:
[versa@Analytics-2: packages] $ md5sum versa-analytics-20201002-063050-969d704-20.2.3.bin 00fc14691cb610d5ba36c2697fc37ccd versa-analytics-20201002-063050-969d704-20.2.3.bin [versa@Analytics-2: packages] $ md5sum versa-analytics-osspack-20210428.bin 6c381aed1ad1be88b52f47345024e3b6 versa-analytics-osspack-20210428.bin
Pre-upgrade steps and checks
Save the existing Analytics nodes config and copy the files to an existing server:
Analytics-1
versa@Analytics-1> configure Entering configuration mode private [ok][2021-07-20 06:40:42] [edit] versa@Analytics-1% save ? Possible completions: versa@Analytics-1% save pre_upgrade_Analytics-1.cfg [ok][2021-07-20 06:41:07] [edit] versa@Analytics-1% exit [ok][2021-07-20 06:41:11] versa@Analytics-1>
Analytics-2
versa@Analytics-2> configure Entering configuration mode private [ok][2021-07-20 06:42:06] [edit] versa@Analytics-2% save pre_upgrade_Analytics-2.cfg [ok][2021-07-20 06:42:22] [edit] versa@Analytics-2% exit [ok][2021-07-20 06:42:26] versa@Analytics-2> versa@Radu_Server:~$ scp versa@Analytics-1:/home/versa/pre_upgrade_Analytics-1.cfg tmp/ versa@analytics-1's password: pre_upgrade_Analytics-1.cfg 100% 2629 2.9MB/s 00:00
Copy the cfg files to a separate server:
versa@Radu_Server:~$ scp versa@Analytics-1:/home/versa/pre_upgrade_Analytics-1.cfg tmp/ versa@analytics-1's password: pre_upgrade_Analytics-1.cfg 100% 2629 2.9MB/s 00:00 versa@Radu_Server:~$ scp versa@Analytics-2:/home/versa/pre_upgrade_Analytics-2.cfg tmp/ The authenticity of host 'analytics-2 (10.40.213.4)' can't be established. ED25519 key fingerprint is SHA256:0wDqko/NwXIc19K4G2mwMmsTaFpFoyJlDaxkKeLmOm8. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'analytics-2' (ED25519) to the list of known hosts. Warning: the ED25519 host key for 'analytics-2' differs from the key for the IP address '10.40.213.4' Offending key for IP in /home/versa/.ssh/known_hosts:7 Are you sure you want to continue connecting (yes/no)? yes versa@analytics-2's password: pre_upgrade_Analytics-2.cfg 100% 2328 2.5MB/s 00:00
Verify if services are in running state on the Analytics nodes prior the upgrade:
Analytics-1
[versa@Analytics-1: ~] $ vsh status versa-confd is Running versa-lced is Running versa-analytics-driver is Running versa-analytics-app is Running versa-monit is Running
Analytics-2
[versa@Analytics-2: ~] $ vsh status [sudo] password for versa: versa-confd is Running versa-lced is Running versa-analytics-driver is Running versa-analytics-app is Running versa-monit is Running
Verify Cassandra DB status using nodetool cli utility on the Analytics nodes. UN means Status = UP, State = Normal
Analytics-1
[versa@Analytics-1: ~] $ nodetool status Datacenter: Search-Analytics ============================ Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Owns Host ID Token Rack UN 127.0.0.1 1.35 MB ? d0fc1546-d9b2-422c-9664-a8fb7b4c00c1 4072798800309564077 RAC1 Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
Analytics-2
[versa@Analytics-2: ~] $ nodetool status Datacenter: Search-Analytics ============================ Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Owns Host ID Token Rack UN 127.0.0.1 306 KB ? 1027040c-695e-4670-ae7f-d066e23202d1 -6992659615691841717 RAC1 Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
OSSPACK update on Versa Analytics nodes
Before you upgrade the Analytics software to Releases 20.2 or later, upgrade the OS SPack on all Analytics nodes to the latest release. If you do not upgrade the OS SPack, the Analytics upgrade may fail.
Analytics-1
[versa@Analytics-1: packages] $ pwd /home/versa/packages [versa@Analytics-1: packages] $ chmod +x versa-analytics-osspack-20210428.bin [versa@Analytics-1: packages] $ sudo ./versa-analytics-osspack-20210428.bin [sudo] password for versa: Verifying archive integrity... All good. Uncompressing OSSPACK 100% ... Processing triggers for ca-certificates (20190110~14.04.1~esm1) ... Updating certificates in /etc/ssl/certs... 10 added, 31 removed; done. Running hooks in /etc/ca-certificates/update.d....done.
Analytics-2
[versa@Analytics-2: packages] $ pwd /home/versa/packages [versa@Analytics-2: packages] $ ls -l total 899588 -rw-r--r-- 1 versa versa 332665708 Oct 10 2019 versa-analytics-20191010-144348-c1a1363-16.1R2S10.bin -rw-r----- 1 versa versa 403044383 Jul 20 03:52 versa-analytics-20201002-063050-969d704-20.2.3.bin -rw-r----- 1 versa versa 185454639 Jul 20 03:52 versa-analytics-osspack-20210428.bin [versa@Analytics-2: packages] $ chmod +x versa-analytics-osspack-20210428.bin [versa@Analytics-2: packages] $ sudo ./versa-analytics-osspack-20210428.bin [sudo] password for versa: Verifying archive integrity... All good. Uncompressing OSSPACK 100% ... Processing triggers for ca-certificates (20190110~14.04.1~esm1) ... Updating certificates in /etc/ssl/certs... 10 added, 31 removed; done. Running hooks in /etc/ca-certificates/update.d....done.
Upgrade Analytics nodes software
In the table below you can find the supported upgrade paths for Versa Analytics software.
Old New Supported
16.1R2.Sx 20.2.2 Yes
16.1R2Sx 20.2.3 Yes
16.1R2Sx 21.1.1 Yes (if DSE is 4.8 or Fusion)
16.1R2Sx 21.1.2 Yes (if DSE is 4.8 or Fusion)
16.1R2Sx 21.2.1 Yes (if DSE is 4.8 or Fusion)
20.2.2 20.2.3 Yes
20.2.2 21.1.1 Yes (if DSE is 4.8 or Fusion)
20.2.2 21.1.2 Yes (if DSE is 4.8 or Fusion)
20.2.3 21.2.1 Yes (if DSE is 4.8 or Fusion)
21.1.1 21.1.2 Yes
21.1.2 21.2.1 Yes
In case of an Analytics cluster upgrade, it is recommended that you upgrade nodes with the analytics personality first, followed by nodes with the search personality. If there are external log forwarder nodes, they can be upgraded after the analytics and search nodes.
To follow the upgrade process the bellow command can be used:
tail -f /var/log/versa/upgrade.log
In case on any issue the above file should give information about what is wrong with the upgrade process.
Analytics-1
versa@Analytics-1> request system package upgrade versa-analytics-20201002-063050-969d704-20.2.3.bin
Will restart the appliance (all processes). Are you sure? [no,yes]yes
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
2021-07-21 02:00:49.258 INFO Exec permission missing on bin. Adding it
2021-07-21 02:00:49.262 INFO Validating checksum
2021-07-21 02:01:04.481 INFO Initiating upgrade from versa-analytics-20191010-144348-c1a1363-16.1R2S10.bin => versa-analytics-20201002-063050-969d704-20.2.3.bin
2021-07-21 02:01:04.484 INFO Installing versa-analytics-20201002-063050-969d704-20.2.3.bin
2021-07-21 02:01:04.486 INFO Loading default configuration
2021-07-21 02:01:04.690 INFO Ret: 0
[versa@Analytics-1: ~] $ 2021-07-21 02:05:09.381 INFO Upgrade succeeded. Starting services
2021-07-21 02:05:11.944 INFO Upgrade task ended.
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Analytics-2
versa@Analytics-2> request system package upgrade versa-analytics-20201002-063050-969d704-20.2.3.bin
Will restart the appliance (all processes). Are you sure? [no,yes]yes
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
2021-07-21 02:09:25.700 INFO Exec permission missing on bin. Adding it
2021-07-21 02:09:25.703 INFO Validating checksum
2021-07-21 02:09:40.574 INFO Initiating upgrade from versa-analytics-20191010-144348-c1a1363-16.1R2S10.bin => versa-analytics-20201002-063050-969d704-20.2.3.bin
2021-07-21 02:09:40.577 INFO Installing versa-analytics-20201002-063050-969d704-20.2.3.bin
2021-07-21 02:09:40.580 INFO Loading default configuration
2021-07-21 02:09:40.772 INFO Ret: 0
[versa@Analytics-2: ~] $ 2021-07-21 02:13:36.372 INFO Upgrade succeeded. Starting services
2021-07-21 02:13:38.883 INFO Upgrade task ended.
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Verify upgrade Analytics nodes software outcome
Verify the running software release on the Analytics nodes.
Analytics-1
versa@Analytics-1> show system package-info
Package Versa Analytics Software
Release 20.2.3
Release Type GA
Release date 20201002
Package id 969d704
Package name versa-analytics-20201002-063050-969d704-20.2.3
Branch 20.2
Creator jenkins
[ok][2021-07-21 02:24:38]
Analytics-2
versa@Analytics-2> show system package-info
Package Versa Analytics Software
Release 20.2.3
Release Type GA
Release date 20201002
Package id 969d704
Package name versa-analytics-20201002-063050-969d704-20.2.3
Branch 20.2
Creator jenkins
[ok][2021-07-21 02:26:02]
Verify that Cassandra DB is running correctly.
Analytics-1
[versa@Analytics-1: ~] $ nodetool status
Datacenter: Search-Analytics
============================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns Host ID Token Rack
UN 127.0.0.1 1.59 MB ? d0fc1546-d9b2-422c-9664-a8fb7b4c00c1 4072798800309564077 RAC1
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
Analytics-2
[versa@Analytics-2: ~] $ nodetool status
Datacenter: Search-Analytics
============================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns Host ID Token Rack
UN 127.0.0.1 331.68 KB ? 1027040c-695e-4670-ae7f-d066e23202d1 -6992659615691841717 RAC1
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
Verify System Configuration from Analytics UI.
After the upgrade is finalized, ensure that search node IP addresses are listed under Search Hosts, Analytics node IP addresses are listed under Analytics Hosts, and all log collector or forwarder IP addresses are listed under Driver Hosts. In Releases 16.1R2S2.2 and later, the search and Analytics hosts are listed separately.
In the Active Versa Director node go to Analytics – – – > Administration – – – > Configuration – – – > Settings
Upgrade the Analytics Database
In Releases 20.2 and later, Versa Analytics nodes support a new database platform called Fusion, which is based on open-source technology. The Fusion database improves scaling and performance and fixes many security vulnerabilities. The Fusion database replaces the DSA database that was used in Release 16.1R2. The DSA database is now at end of life. In Release 20.2, the features of the new Fusion database are compatible with the features of the DSE database. All Release 16.1R2 features and reports are available in Release 20.2. To take advantage of the Fusion database features, including the many new features, reports, and vulnerabilities, you must upgrade to the Fusion database.
Upgrade Analytics Database Documentation could be found at the following location:
https://versanetworks.app.box.com/s/8pdi9ppyjzfq8cx53s10l3zbwt6k2kbw
To upgrade from the DSE to the Fusion database, the Analytics nodes must be running DSE Version 4.8. Upgrade DSE from a prior version to Version 4.8 documentation could be found at the following location:
The script for upgrading to the Fusion database to Fusion is a cluster upgrade script that uninstalls the DSE packages and installs the Fusion packages. The upgrade process occurs one node at a time. During the upgrade, historical data is preserved. However, real-time search data is truncated. The upgrade process takes approximately 1 to 2 hours, depending on the size of the cluster. During this time, database operations are interrupted. However, during the upgrade process, no data or logs are lost, and streaming to third-party collectors is not interrupted.To reduce the downtime, you can bring up a new cluster that is running Release 20.2.x. Then, configure the Controller node to use the server IP addresses of the new cluster so that logs start flowing to the new cluster.
Verify the running Analytics Database.
Analytics-1
[versa@Analytics-1: ~] $ service dse status
* dse is running
[versa@Analytics-1: ~] $ dse -v
4.8.16
Analytics-2
[versa@Analytics-2: ~] $ service dse status
* dse is running
[versa@Analytics-2: ~] $ dse -v
4.8.16
Copy the Fusion migration script to the Active Versa Director node.
versa@Radu_Server:~/tmp$ scp fusion_migration.v2.3.tar admin@Director-1:
The authenticity of host 'director-1 (10.40.213.1)' can't be established.
ED25519 key fingerprint is SHA256:Y+JbgBjGxwFy59PoJxMLaOAfwlRNEiQgf9qLOl0EUjw.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'director-1' (ED25519) to the list of known hosts.
Warning: the ED25519 host key for 'director-1' differs from the key for the IP address '10.40.213.1'
Offending key for IP in /home/versa/.ssh/known_hosts:5
Are you sure you want to continue connecting (yes/no)? yes
admin@director-1's password:
fusion_migration.v2.3.tar 100% 427MB 69.9MB/s 00:06
Extract the contents of the package
admin@Director-1:~$ pwd
/home/admin
admin@Director-1:~$ ls -l
total 436844
-rw-r----- 1 admin versa 447324160 Jul 21 10:19 fusion_migration.v2.3.tar
admin@Director-1:~$ tar -xvf fusion_migration.v2.3.tar
fusion_migration/
fusion_migration/.pip/
fusion_migration/.pip/urllib3-1.25.9-py2.py3-none-any.whl
fusion_migration/.pip/requests-2.23.0-py2.py3-none-any.whl
fusion_migration/.pip/chardet-3.0.4-py2.py3-none-any.whl
fusion_migration/.pip/idna-2.9-py2.py3-none-any.whl
fusion_migration/.pip/certifi-2020.4.5.1-py2.py3-none-any.whl
fusion_migration/.pip/ptyprocess-0.6.0-py2.py3-none-any.whl
fusion_migration/.pip/pexpect-4.8.0-py2.py3-none-any.whl
fusion_migration/session_manager.py
fusion_migration/logger.py
fusion_migration/defaults/
fusion_migration/defaults/vansetup.conf
fusion_migration/defaults/vandriver.conf
fusion_migration/fusion_migration.py
fusion_migration/packages/
fusion_migration/packages/zookeeper-3.6.0.tar.gz
fusion_migration/packages/solr-7.7.2.tgz
fusion_migration/packages/zookeeper-3.4.10.tar.gz
fusion_migration/packages/dependencies/
fusion_migration/packages/dependencies/libpcsclite1_1.8.10-1ubuntu1.1_amd64.deb
fusion_migration/packages/dependencies/libnspr4_2%3a4.13.1-0ubuntu0.14.04.1_amd64.deb
fusion_migration/packages/dependencies/libnss3-nssdb_2%3a3.28.4-0ubuntu0.14.04.5+esm4_all.deb
fusion_migration/packages/dependencies/libnss3_2%3a3.28.4-0ubuntu0.14.04.5+esm4_amd64.deb
fusion_migration/packages/dependencies/liblcms2-2_2.5-0ubuntu4.2_amd64.deb
fusion_migration/packages/dependencies/libavahi-common-data_0.6.31-4ubuntu1.3_amd64.deb
fusion_migration/packages/dependencies/libavahi-common3_0.6.31-4ubuntu1.3_amd64.deb
fusion_migration/packages/dependencies/ca-certificates-java_20130815ubuntu1_all.deb
fusion_migration/packages/dependencies/libsctp1_1.0.15+dfsg-1_amd64.deb
fusion_migration/packages/dependencies/libpulse0_1%3a4.0-0ubuntu11.1_amd64.deb
fusion_migration/packages/dependencies/tzdata-java_2019c-0ubuntu0.14.04_all.deb
fusion_migration/packages/dependencies/libasyncns0_0.8-4ubuntu2_amd64.deb
fusion_migration/packages/dependencies/libavahi-client3_0.6.31-4ubuntu1.3_amd64.deb
fusion_migration/packages/dependencies/dpkg_1.17.5ubuntu5.8_amd64.deb
fusion_migration/packages/dependencies/libflac8_1.3.0-2ubuntu0.14.04.1_amd64.deb
fusion_migration/packages/dependencies/java-common_0.51_all.deb
fusion_migration/packages/dependencies/libcups2_1.7.2-0ubuntu1.11_amd64.deb
fusion_migration/packages/dependencies/libvorbisenc2_1.3.2-1.3ubuntu1.2_amd64.deb
fusion_migration/packages/dependencies/libvorbis0a_1.3.2-1.3ubuntu1.2_amd64.deb
fusion_migration/packages/dependencies/libogg0_1.3.1-1ubuntu1_amd64.deb
fusion_migration/packages/dependencies/openjdk-7-jre-headless_7u211-2.6.17-0ubuntu0.1_amd64.deb
fusion_migration/packages/dependencies/libsndfile1_1.0.25-7ubuntu2.2_amd64.deb
fusion_migration/packages/dependencies/lksctp-tools_1.0.15+dfsg-1_amd64.deb
fusion_migration/packages/dependencies/tzdata_2019c-0ubuntu0.14.04_all.deb
fusion_migration/packages/cassandra-2.2.21/
fusion_migration/packages/cassandra-2.2.21/cassandra_2.1.21_all.deb
fusion_migration/packages/cassandra-2.2.21/cassandra-tools_2.1.21_all.deb
fusion_migration/packages/solr-6.6.1.tgz
fusion_migration/packages/cassandra-3.0.9/
fusion_migration/packages/cassandra-3.0.9/cassandra_3.0.9_all.deb
fusion_migration/packages/cassandra-3.0.9/cassandra-tools_3.0.9_all.deb
fusion_migration/migration_prechecks.py
fusion_migration/README
fusion_migration/logging.properties
fusion_migration/fusion_migration.conf
fusion_migration/device.py
Edit fusion_migration.conf file accordingly with the existing Analytics Config and run the python script.
admin@Director-1:~/fusion_migration$ python fusion_migration.py
.---.,
( ``.
_ \ ) __ ________ _____ _____
( `. \ / \ \ / / ____| __ \ / ____| /\
\ `. ) / \ \ / /| |__ | |__) | (___ / \
\ | / \ \/ / | __| | _ / \___ \ / /\ \
\ | / \ / | |____| | \ \ ____) / ____ \
\ | / \/ |______|_| \_\_____/_/ \_\
\ | /
\_|/ _ _ _ _ _ __ _______ ___ ___ ___
/_\ | \| | /_\ | |\ \ / /_ _|_ _/ __/ __|
/ _ \| .` |/ _ \| |_\ V / | | | | (__\__ \
/_/ \_\_|\_/_/ \_\____|_| |_| |___\___|___/
VERSA ANALYTICS - FUSION MIGRATION
v2.3
/home/admin/.ssh/known_hosts updated.
Original contents retained as /home/admin/.ssh/known_hosts.old
2021-07-21 10:44:36,407 - SessionManager - INFO - Session established with 'STANDALONE' on 10.40.213.3
2021-07-21 10:44:36,408 - FusionMigration - INFO - Configuration for Cluster read from fusion_migration.conf
2021-07-21 10:44:43,820 - FusionMigration - INFO - Backing up DSE configs on 'STANDALONE'
2021-07-21 10:44:44,771 - FusionMigration - INFO - Copying packages to 'STANDALONE'
2021-07-21 10:44:50,925 - FusionMigration - INFO - Verifying copied packages on 'STANDALONE'
2021-07-21 10:45:07,841 - FusionMigration - INFO - Copying vansetup.conf to 'STANDALONE'
===> Proceed with installation? [y/N]: y
===> Waiting for 30 second(s) [ 30 |]
2021-07-21 10:45:46,391 - FusionMigration - INFO - Dropping DSE & Search Keyspaces on 'STANDALONE'
===> Waiting for 30 second(s) [ 30 |]
===> Waiting for 60 second(s) [ 60 -]
2021-07-21 10:47:44,481 - FusionMigration - INFO - Stopping monit & DSE on 'STANDALONE'
2021-07-21 10:47:57,261 - FusionMigration - INFO - Purging DSE + dependencies on 'STANDALONE'
2021-07-21 10:48:08,908 - FusionMigration - INFO - Cleaning up obsolete DSE files on 'STANDALONE'
2021-07-21 10:48:10,565 - FusionMigration - INFO - Setting hostname on 'STANDALONE'
2021-07-21 10:48:14,022 - FusionMigration - INFO - Installing dependencies on 'STANDALONE'
2021-07-21 10:48:45,609 - FusionMigration - INFO - Preparing Zookeeper on 'STANDALONE'
2021-07-21 10:48:46,602 - FusionMigration - INFO - Preparing Solr on 'STANDALONE'
2021-07-21 10:48:47,577 - FusionMigration - INFO - Installing Cassandra 2.2.21 on 'STANDALONE'
===> Waiting for 20 second(s) [ 20 -]
2021-07-21 10:49:13,590 - FusionMigration - INFO - Draining Analytics node 'STANDALONE'
2021-07-21 10:49:17,587 - FusionMigration - INFO - Installing Cassandra 3.0.9 on 'STANDALONE'
2021-07-21 10:49:25,788 - FusionMigration - INFO - Restarting Cassandra on 'STANDALONE'
===> Waiting for 60 second(s) [ 60 -]
2021-07-21 10:50:26,713 - FusionMigration - INFO - Performing vansetup on 'STANDALONE'
sudo /opt/versa/scripts/van-scripts/vansetup.py
VAN Setup configuration start
cluster_name : 'VAN Standalone'
mode : standalone
personality :
rpc_address : localhost
listen_address : localhost
seeds : "127.0.0.1"
zookeeper_node : false
zookeeper_id : 1
zookeeper_server.1 : 127.0.0.1
search_hosts : localhost
analytics_hosts : localhost
search_shards : 1
search_replication_factor : 2
analytics_replication_factor : 2
authentication : false
geo_support : true
archive_dir : /var/tmp/archive
archive_interval : hourly
VAN Setup done for the first time
Stopping Versa Analytics if running
* Stopping versa-lced
* -n ... waiting for versa-lced to exit
* Stopping versa-confd
* Stopping versa-analytics-app
* Stopping daemon monitor monit
...done.
* Versa Analytics Stopped
...done.
Stopping database service if running
start-stop-daemon: warning: failed to kill 9937: No such process
Update config files
Delete the database ? (y/N) n
Skipping deleting database
solr: unrecognized service
zookeeper: unrecognized service
Starting Zookeeper
update-rc.d: warning: start runlevel arguments (none) do not match zookeeper Default-Start values (2 3 4 5)
update-rc.d: warning: stop runlevel arguments (none) do not match zookeeper Default-Stop values (0 1 6)
ZooKeeper JMX enabled by default
Using config: /opt/versa_van/apps/zookeeper-3.4.10/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
Zookeeper is up and running on this node
Starting Cassandra...
Waiting for host 127.0.0.1 to come up
0
UN 127.0.0.1 1.35 MB 100.0% d0fc1546-d9b2-422c-9664-a8fb7b4c00c1 4072798800309564077 RAC1
Creating database schema...
Connected to cluster: VAN Standalone
Datacenter: Analytics, Host: 127.0.0.1, Rack: RAC1
Database version is up to date (4.2.0)
Installing search engine
id: solr: no such user
Extracting Solr...
Creating new user: solr
Adding system user `solr' (UID 109) ...
Adding new group `solr' (GID 113) ...
Adding new user `solr' (UID 109) with group `solr' ...
Creating home directory `/var/lib/solr/data' ...
Extracting /opt/versa_van/apps/solr-6.6.1.tgz to /etc
Installing symlink /etc/solr -> /etc/solr-6.6.1 ...
Installing /etc/init.d/solr script ...
Installing /etc/default/solr.in.sh ...
Adding system startup for /etc/init.d/solr ...
/etc/rc0.d/K20solr -> ../init.d/solr
/etc/rc1.d/K20solr -> ../init.d/solr
/etc/rc6.d/K20solr -> ../init.d/solr
/etc/rc2.d/S20solr -> ../init.d/solr
/etc/rc3.d/S20solr -> ../init.d/solr
/etc/rc4.d/S20solr -> ../init.d/solr
/etc/rc5.d/S20solr -> ../init.d/solr
Service solr installed.
Customize Solr startup configuration in /etc/default/solr.in.sh
Not starting Solr service (option -n given). Start manually with 'service solr start'
Authentication disabled
Done
Starting search engine
ZooKeeper JMX enabled by default
Using config: /opt/versa_van/apps/zookeeper-3.4.10/bin/../conf/zoo.cfg
Starting Zookeeper...
Starting zookeeper ... already running as process 10211.
Waiting up to 180 seconds to see Solr running on port 8983 [\]
Started Solr server on port 8983 (pid=10812). Happy searching!
Starting Versa Analytics package
...done.
* Restarting daemon monitor monit [ OK ]
* Starting versa-analytics-app Optimizing Search Engine ...
Sending stop command to Solr running on port 8983 ... waiting up to 180 seconds to allow Jetty process 10812 to stop gracefully.
Waiting up to 180 seconds to see Solr running on port 8983 [\]
Started Solr server on port 8983 (pid=11390). Happy searching!
* Versa Analytics Started
.---.,
( ``.
_ \ ) __ ________ _____ _____
( `. \ / \ \ / / ____| __ \ / ____| /\
\ `. ) / \ \ / /| |__ | |__) | (___ / \
\ | / \ \/ / | __| | _ / \___ \ / /\ \
\ | / \ / | |____| | \ \ ____) / ____ \
\ | / \/ |______|_| \_\_____/_/ \_\
\ | /
\_|/ _ _ _ _ _ __ _______ ___ ___ ___
/_\ | \| | /_\ | |\ \ / /_ _|_ _/ __/ __|
/ _ \| .` |/ _ \| |_\ V / | | | | (__\__ \
/_/ \_\_|\_/_/ \_\____|_| |_| |___\___|___/
update-rc.d: warning: start runlevel arguments (none) do not match cassandra Default-Start values (2 3 4 5)
update-rc.d: warning: stop runlevel arguments (none) do not match cassandra Default-Stop values (0 1 6)
Setup log archiving
Source Directory /var/tmp/log
Destination Directory /var/tmp/archive
Interval hourly
Job-name log-archive
Archive backup
Skipping rebooting the node
VAN Setup Complete
[versa@Analytics-1: ~] $ 2021-07-21 10:52:40,385 - FusionMigration - INFO - Adding Fusion signature to Analytics Cluster
2021-07-21 10:52:41,748 - FusionMigration - INFO - Restarting Versa services on 'STANDALONE'
2021-07-21 10:52:56,833 - FusionMigration - INFO - Fusion migration completed on Analytics, Search & Forwarder nodes
Analytics-2
Config file has to be modified accordingly with the new cluster or new standalone node.
admin@Director-1:~/fusion_migration$ python fusion_migration.py
.---.,
( ``.
_ \ ) __ ________ _____ _____
( `. \ / \ \ / / ____| __ \ / ____| /\
\ `. ) / \ \ / /| |__ | |__) | (___ / \
\ | / \ \/ / | __| | _ / \___ \ / /\ \
\ | / \ / | |____| | \ \ ____) / ____ \
\ | / \/ |______|_| \_\_____/_/ \_\
\ | /
\_|/ _ _ _ _ _ __ _______ ___ ___ ___
/_\ | \| | /_\ | |\ \ / /_ _|_ _/ __/ __|
/ _ \| .` |/ _ \| |_\ V / | | | | (__\__ \
/_/ \_\_|\_/_/ \_\____|_| |_| |___\___|___/
VERSA ANALYTICS - FUSION MIGRATION
v2.3
/home/admin/.ssh/known_hosts updated.
Original contents retained as /home/admin/.ssh/known_hosts.old
2021-07-21 10:59:22,487 - SessionManager - INFO - Session established with 'STANDALONE' on 10.40.213.4
2021-07-21 10:59:22,487 - FusionMigration - INFO - Configuration for Cluster read from fusion_migration.conf
2021-07-21 10:59:29,927 - FusionMigration - INFO - Backing up DSE configs on 'STANDALONE'
2021-07-21 10:59:30,747 - FusionMigration - INFO - Copying packages to 'STANDALONE'
2021-07-21 10:59:36,996 - FusionMigration - INFO - Verifying copied packages on 'STANDALONE'
2021-07-21 10:59:52,989 - FusionMigration - INFO - Copying vansetup.conf to 'STANDALONE'
===> Proceed with installation? [y/N]: y
===> Waiting for 30 second(s) [ 30 |]
2021-07-21 11:03:11,579 - FusionMigration - INFO - Dropping DSE & Search Keyspaces on 'STANDALONE'
===> Waiting for 30 second(s) [ 30 |]
===> Waiting for 60 second(s) [ 60 -]
2021-07-21 11:04:59,533 - FusionMigration - INFO - Stopping monit & DSE on 'STANDALONE'
2021-07-21 11:05:12,407 - FusionMigration - INFO - Purging DSE + dependencies on 'STANDALONE'
2021-07-21 11:05:22,936 - FusionMigration - INFO - Cleaning up obsolete DSE files on 'STANDALONE'
2021-07-21 11:05:24,597 - FusionMigration - INFO - Setting hostname on 'STANDALONE'
2021-07-21 11:05:27,993 - FusionMigration - INFO - Installing dependencies on 'STANDALONE'
2021-07-21 11:06:00,113 - FusionMigration - INFO - Preparing Zookeeper on 'STANDALONE'
2021-07-21 11:06:01,098 - FusionMigration - INFO - Preparing Solr on 'STANDALONE'
2021-07-21 11:06:02,071 - FusionMigration - INFO - Installing Cassandra 2.2.21 on 'STANDALONE'
===> Waiting for 20 second(s) [ 20 -]
2021-07-21 11:06:27,501 - FusionMigration - INFO - Draining Analytics node 'STANDALONE'
2021-07-21 11:06:31,484 - FusionMigration - INFO - Installing Cassandra 3.0.9 on 'STANDALONE'
2021-07-21 11:06:39,320 - FusionMigration - INFO - Restarting Cassandra on 'STANDALONE'
===> Waiting for 60 second(s) [ 60 -]
2021-07-21 11:07:40,248 - FusionMigration - INFO - Performing vansetup on 'STANDALONE'
sudo /opt/versa/scripts/van-scripts/vansetup.py
VAN Setup configuration start
cluster_name : 'VAN Standalone'
mode : standalone
personality :
rpc_address : localhost
listen_address : localhost
seeds : "127.0.0.1"
zookeeper_node : false
zookeeper_id : 1
zookeeper_server.1 : 127.0.0.1
search_hosts : localhost
analytics_hosts : localhost
search_shards : 1
search_replication_factor : 2
analytics_replication_factor : 2
authentication : false
geo_support : true
archive_dir : /var/tmp/archive
archive_interval : hourly
VAN Setup done for the first time
Stopping Versa Analytics if running
* Stopping versa-lced
* -n ... waiting for versa-lced to exit
* Stopping versa-confd
* Stopping versa-analytics-app
* Stopping daemon monitor monit
...done.
* Versa Analytics Stopped
...done.
Stopping database service if running
start-stop-daemon: warning: failed to kill 8219: No such process
Update config files
Delete the database ? (y/N) n
Skipping deleting database
solr: unrecognized service
zookeeper: unrecognized service
Starting Zookeeper
update-rc.d: warning: start runlevel arguments (none) do not match zookeeper Default-Start values (2 3 4 5)
update-rc.d: warning: stop runlevel arguments (none) do not match zookeeper Default-Stop values (0 1 6)
ZooKeeper JMX enabled by default
Using config: /opt/versa_van/apps/zookeeper-3.4.10/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
Zookeeper is up and running on this node
Starting Cassandra...
Waiting for host 127.0.0.1 to come up
0
UN 127.0.0.1 226.19 KB 100.0% 1027040c-695e-4670-ae7f-d066e23202d1 -6992659615691841717 RAC1
Creating database schema...
Connected to cluster: VAN Standalone
Datacenter: Analytics, Host: 127.0.0.1, Rack: RAC1
Database version is up to date (4.2.0)
Installing search engine
id: solr: no such user
Extracting Solr...
Creating new user: solr
Adding system user `solr' (UID 109) ...
Adding new group `solr' (GID 113) ...
Adding new user `solr' (UID 109) with group `solr' ...
Creating home directory `/var/lib/solr/data' ...
Extracting /opt/versa_van/apps/solr-6.6.1.tgz to /etc
Installing symlink /etc/solr -> /etc/solr-6.6.1 ...
Installing /etc/init.d/solr script ...
Installing /etc/default/solr.in.sh ...
Adding system startup for /etc/init.d/solr ...
/etc/rc0.d/K20solr -> ../init.d/solr
/etc/rc1.d/K20solr -> ../init.d/solr
/etc/rc6.d/K20solr -> ../init.d/solr
/etc/rc2.d/S20solr -> ../init.d/solr
/etc/rc3.d/S20solr -> ../init.d/solr
/etc/rc4.d/S20solr -> ../init.d/solr
/etc/rc5.d/S20solr -> ../init.d/solr
Service solr installed.
Customize Solr startup configuration in /etc/default/solr.in.sh
Not starting Solr service (option -n given). Start manually with 'service solr start'
Authentication disabled
Done
Starting search engine
ZooKeeper JMX enabled by default
Using config: /opt/versa_van/apps/zookeeper-3.4.10/bin/../conf/zoo.cfg
Starting Zookeeper...
Starting zookeeper ... already running as process 8485.
Waiting up to 180 seconds to see Solr running on port 8983 [\]
Started Solr server on port 8983 (pid=9050). Happy searching!
Starting Versa Analytics package
...done.
* Restarting daemon monitor monit [ OK ]
* Starting versa-analytics-app Optimizing Search Engine ...
Sending stop command to Solr running on port 8983 ... waiting up to 180 seconds to allow Jetty process 9050 to stop gracefully.
Waiting up to 180 seconds to see Solr running on port 8983 [\]
Started Solr server on port 8983 (pid=9656). Happy searching!
* Versa Analytics Started .---., ( ``. _ \ ) __ ________ _____ _____ ( `. \ / \ \ / / ____| __ \ / ____| /\ \ `. ) / \ \ / /| |__ | |__) | (___ / \ \ | / \ \/ / | __| | _ / \___ \ / /\ \ \ | / \ / | |____| | \ \ ____) / ____ \ \ | / \/ |______|_| \_\_____/_/ \_\ \ | / \_|/ _ _ _ _ _ __ _______ ___ ___ ___ /_\ | \| | /_\ | |\ \ / /_ _|_ _/ __/ __| / _ \| .` |/ _ \| |_\ V / | | | | (__\__ \ /_/ \_\_|\_/_/ \_\____|_| |_| |___\___|___/ update-rc.d: warning: start runlevel arguments (none) do not match cassandra Default-Start values (2 3 4 5)
update-rc.d: warning: stop runlevel arguments (none) do not match cassandra Default-Stop values (0 1 6)
Setup log archiving
Source Directory /var/tmp/log
Destination Directory /var/tmp/archive
Interval hourly
Job-name log-archive
Archive backup
Skipping rebooting the node
VAN Setup Complete
[versa@Analytics-2: ~] $ 2021-07-21 11:09:55,867 - FusionMigration - INFO - Adding Fusion signature to Analytics Cluster
2021-07-21 11:09:57,235 - FusionMigration - INFO - Restarting Versa services on 'STANDALONE'
2021-07-21 11:10:12,448 - FusionMigration - INFO - Fusion migration completed on Analytics, Search & Forwarder nodes
Note: Please contact Versa Support if you are upgrading a large database or if you encounter any issues while running the upgrade scripts.
Verify Analytics (Cluster) Database status
Versa Analytics Fusion introduces a new architecture using various open-source components. These components include:
Cassandra: A distributed database that is used to store summary information & metrics.
Solr: A distributed Search Engine used to store alarms, NGFW logs & other events.
Zookeeper: A distributed co-ordination service used to manage Solr configurations, node discovery & leader election.
Verify that dse database is not present on the Analytics nodes
Analytics-1
[versa@Analytics-1: ~] $ dse -v
No command 'dse' found, did you mean:
Command 'dsc' from package 'dsc-statistics-collector' (universe)
Command 'dsh' from package 'dsh' (universe)
Command 'nse' from package 'ns2' (universe)
Command 'ds9' from package 'saods9' (universe)
Command 'ase' from package 'python-ase' (universe)
dse: command not found
Analytics-2
[versa@Analytics-2: ~] $ dse -v
No command 'dse' found, did you mean:
Command 'nse' from package 'ns2' (universe)
Command 'ds9' from package 'saods9' (universe)
Command 'dsc' from package 'dsc-statistics-collector' (universe)
Command 'dsh' from package 'dsh' (universe)
Command 'ase' from package 'python-ase' (universe)
dse: command not found
Verify Cassandra, Solr and Zookeeper status.
Analytics-1
[versa@Analytics-1: ~] $ vsh dbstatus
[sudo] password for versa:
Datacenter: Analytics
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns (effective) Host ID Token Rack
UN 127.0.0.1 1.35 MB 100.0% d0fc1546-d9b2-422c-9664-a8fb7b4c00c1 4072798800309564077 RAC1
Zookeeper Status
=====================
ZooKeeper JMX enabled by default
Using config: /opt/versa_van/apps/zookeeper-3.4.10/bin/../conf/zoo.cfg
Mode: standalone
=====================
Zookeeper version: 3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
Clients:
/0:0:0:0:0:0:0:1:56504[0](queued=0,recved=1,sent=0)
/127.0.0.1:51860[1](queued=0,recved=1348,sent=1370)
Latency min/avg/max: 0/1/788
Received: 1364
Sent: 1387
Connections: 2
Outstanding: 0
Zxid: 0x92
Mode: standalone
Node count: 38
Datacenter: Search
=====================
Found 1 Solr nodes:
Solr process 11390 running on port 8983
solr_home: /var/lib/solr/data/data,
version: 6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 22:00:32,
startTime: 2021-07-21T10:52:33.166Z,
uptime: 0 days, 0 hours, 38 minutes, 22 seconds,
memory: 205.8 MB (%10.5) of 1.9 GB,
cloud:
ZooKeeper: 127.0.0.1:2181,
liveNodes: 1,
collections: 1
Analytics-2
[versa@Analytics-2: ~] $ vsh dbstatus
[sudo] password for versa:
Datacenter: Analytics
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns (effective) Host ID Token Rack
UN 127.0.0.1 226.19 KB 100.0% 1027040c-695e-4670-ae7f-d066e23202d1 -6992659615691841717 RAC1
Zookeeper Status
=====================
ZooKeeper JMX enabled by default
Using config: /opt/versa_van/apps/zookeeper-3.4.10/bin/../conf/zoo.cfg
Mode: standalone
=====================
Zookeeper version: 3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
Clients:
/127.0.0.1:53760[1](queued=0,recved=841,sent=855)
/0:0:0:0:0:0:0:1:38756[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/1/99
Received: 857
Sent: 873
Connections: 2
Outstanding: 0
Zxid: 0x8d
Mode: standalone
Node count: 38
Datacenter: Search
=====================
Found 1 Solr nodes:
Solr process 9656 running on port 8983
solr_home: /var/lib/solr/data/data,
version: 6.6.1 9aa465a89b64ff2dabe7b4d50c472de32c298683 - varunthacker - 2017-08-29 22:00:32,
startTime: 2021-07-21T11:09:50.124Z,
uptime: 0 days, 0 hours, 22 minutes, 23 seconds,
memory: 128.9 MB (%6.6) of 1.9 GB,
cloud:
ZooKeeper: 127.0.0.1:2181,
liveNodes: 1,
collections: 1
This concludes Analytics (Cluster) upgrade process.
Versa Controllers upgrade from 16.1R2 to 20.2.3
You can upgrade VOS devices to Release 20.2 from Releases 16.1R2 (Release16.1R2S8) and later. If you are using an earlier software release, upgrade first to the latest 16.1R2 service release and then upgrade to Release 20.2.
Upload the software to the Versa Controller nodes
Using WINSCP and/or SCP, upload the image files to the respective Versa Controller nodes. Use the target directories as in the bellow examples. Pay attention what is the processor architecture for the appliance. In case it is Westmere architecture the wsm binary has to be used. (e.g versa-flexvnf-20201002-063550-969d704-20.2.3-wsm.bin)
Controller-1
versa@Radu_Server:~/tmp$ scp versa-flexvnf-20201002-063050-969d704-20.2.3.bin admin@Controller-1:/home/versa/packages/
The authenticity of host 'controller-1 (10.40.213.7)' can't be established.
ED25519 key fingerprint is SHA256:bPGwdTqLHQBMi8nGpOwUguG/ykMCpyIegOal9aj9+Oc.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'controller-1,10.40.213.7' (ED25519) to the list of known hosts.
admin@controller-1's password:
versa-flexvnf-20201002-063050-969d704-20.2.3.bin 100% 620MB 41.8MB/s 00:14
versa@Radu_Server:~/tmp$ scp versa-flexvnf-osspack-20210428.bin admin@Controller-1:/home/versa/packages/
admin@controller-1's password:
versa-flexvnf-osspack-20210428.bin 100% 173MB 47.4MB/s 00:03
Controller-2
versa@Radu_Server:~/tmp$ scp versa-flexvnf-20201002-063050-969d704-20.2.3.bin admin@Controller-2:/home/versa/packages/
The authenticity of host 'controller-2 (10.40.213.8)' can't be established.
ED25519 key fingerprint is SHA256:bPGwdTqLHQBMi8nGpOwUguG/ykMCpyIegOal9aj9+Oc.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'controller-2,10.40.213.8' (ED25519) to the list of known hosts.
admin@controller-2's password:
versa-flexvnf-20201002-063050-969d704-20.2.3.bin 100% 620MB 50.1MB/s 00:12
versa@Radu_Server:~/tmp$ scp versa-flexvnf-osspack-20210428.bin admin@Controller-2:/home/versa/packages/
admin@controller-2's password:
versa-flexvnf-osspack-20210428.bin 100% 173MB 55.4MB/s 00:03
Following the upload verify the MD5 checksum of the packages from VOS cli:
Controller-1
admin@Controller-1-cli> request system package list
packages {
name versa-flexvnf-20191010-145033-c1a1363-16.1R2S10.bin
}
packages {
name versa-flexvnf-20201002-063050-969d704-20.2.3.bin
}
packages {
name versa-flexvnf-osspack-20210428.bin
}
[ok][2021-07-22 01:19:03]
admin@Controller-1-cli> request system package check-integrity versa-flexvnf-20201002-063050-969d704-20.2.3.bin
status Ok
[ok][2021-07-22 01:19:25]
admin@Controller-1-cli> request system package check-integrity versa-flexvnf-osspack-20210428.bin
status Ok
[ok][2021-07-22 01:19:37]
Controller-2
admin@Controller-2-cli> request system package list
packages {
name versa-flexvnf-20191010-145033-c1a1363-16.1R2S10.bin
}
packages {
name versa-flexvnf-20201002-063050-969d704-20.2.3.bin
}
packages {
name versa-flexvnf-osspack-20210428.bin
}
[ok][2021-07-22 01:21:36]
admin@Controller-2-cli> request system package check-integrity versa-flexvnf-20201002-063050-969d704-20.2.3.bin
status Ok
[ok][2021-07-22 01:21:50]
admin@Controller-2-cli> request system package check-integrity versa-flexvnf-osspack-20210428.bin
status Ok
[ok][2021-07-22 01:22:02]
Pre-upgrade steps and checks
Save the current Controllers configuration.
Controller-1
admin@Controller-1-cli> configure
Entering configuration mode private
[ok][2021-07-22 01:58:01]
[edit]
admin@Controller-1-cli(config)% save ?
Possible completions:
admin@Controller-1-cli(config)% save pre_upgrade_Controller-1.cfg
[ok][2021-07-22 01:58:30]
Controller-2
admin@Controller-2-cli> configure
Entering configuration mode private
[ok][2021-07-22 01:59:26]
[edit]
admin@Controller-2-cli(config)% save pre_upgrade_Controller-2.cfg
[ok][2021-07-22 02:00:32]
Copy the Controllers config file to an external server.
versa@Radu_Server:~$ scp admin@Controller-1:/home/admin/pre_upgrade_Controller-1.cfg tmp/
admin@controller-1's password:
pre_upgrade_Controller-1.cfg 100% 65KB 16.0MB/s 00:00
versa@Radu_Server:~$ scp admin@Controller-2:/home/admin/pre_upgrade_Controller-2.cfg tmp/
admin@controller-2's password:
pre_upgrade_Controller-2.cfg 100% 70KB 17.6MB/s 00:00
Verify that the Versa services are in Running state
Controller-1
[admin@Controller-1: ~] $ vsh status
[sudo] password for admin:
versa-service is Running, [*] process 1624
versa-infmgr is Running, [-] process 1583
versa-rfd is Running, [-] process 1850
versa-vmod is Running, [-] process 1847
versa-ip2user is Running, [-] process 1853
versa-imgr is Running, [-] process 1862
versa-acctmgrd is Running, [-] process 1842
versa-fltrmgr is Running, [-] process 1593
versa-vstated is Running, [-] process 1585
versa-addrmgrd is Running, [-] process 1855
versa-rt-cli-xfm is Running, [-] process 1791
versa-rtd is Running, [-] process 1828
versa-dhcpd is Running, [-] process 1586
versa-eventd is Running, [-] process 1845
versa-vrrpd is Running, [-] process 1590
versa-dnsd is Running, [-] process 1594
versa-ppmd is Running, [-] process 1783
versa-snmp-xform is Running, [-] process 1797
versa-certd is Running, [-] process 1857
versa-ntpd is Running, [*] process 1569
versa-dhclient6 is Running, [-] process 1799
versa-redis is Running, [-] process 2787
versa-av-redis is Running, [-] process 1395
versa-spackmgr is Running, [-] process 1835
versa-monit is Running, [*] process 1986
versa-confd is Running, [*] process 1343
versa-fail2ban is Running, [*] process 2108
versa-auditd is Running, [*] process 1087
versa-nodejs is Running, [-] process 1679
Controller-2
[admin@Controller-2: ~] $ vsh status
[sudo] password for admin:
versa-service is Running, [*] process 1635
versa-infmgr is Running, [-] process 1580
versa-rfd is Running, [-] process 1811
versa-vmod is Running, [-] process 1806
versa-ip2user is Running, [-] process 1821
versa-imgr is Running, [-] process 1839
versa-acctmgrd is Running, [-] process 1795
versa-fltrmgr is Running, [-] process 1600
versa-vstated is Running, [-] process 1579
versa-addrmgrd is Running, [-] process 1829
versa-rt-cli-xfm is Running, [-] process 1751
versa-rtd is Running, [-] process 1788
versa-dhcpd is Running, [-] process 1583
versa-eventd is Running, [-] process 1797
versa-vrrpd is Running, [-] process 1592
versa-dnsd is Running, [-] process 1594
versa-ppmd is Running, [-] process 1744
versa-snmp-xform is Running, [-] process 1759
versa-certd is Running, [-] process 1832
versa-ntpd is Running, [*] process 1565
versa-dhclient6 is Running, [-] process 1763
versa-redis is Running, [-] process 2777
versa-av-redis is Running, [-] process 1388
versa-spackmgr is Running, [-] process 1790
versa-monit is Running, [*] process 1986
versa-confd is Running, [*] process 1339
versa-fail2ban is Running, [*] process 2106
versa-auditd is Running, [*] process 1083
versa-nodejs is Running, [-] process 1737
Check the BGP sessions state.
Controller-1
admin@Controller-1-cli> show bgp neighbor brief
routing-instance: Provider-Control-VR
Neighbor V MsgRcvd MsgSent Uptime State/PfxRcd PfxSent AS
11.0.192.2 4 246 250 01:37:18 25 33 64512
11.0.192.101 4 234 250 01:37:27 4 33 64512
11.0.192.102 4 234 252 01:37:31 4 33 64512
192.168.3.1 4 223 244 01:37:41 2 22 65001
routing-instance: Customer1-Control-VR
Neighbor V MsgRcvd MsgSent Uptime State/PfxRcd PfxSent AS
11.1.64.2 4 237 242 01:37:28 10 19 64512
11.1.64.101 4 233 242 01:36:47 4 19 64512
11.1.64.102 4 234 247 01:37:31 4 19 64512
[ok][2021-07-22 02:21:04]
Controller-2
admin@Controller-2-cli> show bgp neighbor brief
routing-instance: Provider-Control-VR
Neighbor V MsgRcvd MsgSent Uptime State/PfxRcd PfxSent AS
11.0.192.101 4 241 266 01:40:36 4 32 64512
11.0.192.102 4 241 256 01:39:57 4 32 64512
11.0.192.1 4 255 251 01:39:30 23 27 64512
192.168.3.5 4 230 252 01:40:41 2 21 65002
routing-instance: Customer1-Control-VR
Neighbor V MsgRcvd MsgSent Uptime State/PfxRcd PfxSent AS
11.1.64.101 4 240 251 01:40:36 4 19 64512
11.1.64.102 4 239 249 01:39:57 4 19 64512
11.1.64.1 4 247 243 01:39:40 10 10 64512
[ok][2021-07-22 02:23:17]
Check the Netconf session between the active Versa Director and the Controller.
Controller-1
admin@Controller-1-cli> show netconf-state sessions
IN OUT
SESSION IN BAD RPC OUT
ID TRANSPORT USERNAME SOURCE HOST LOGIN TIME RPCS RPCS ERRORS NOTIFICATIONS
------------------------------------------------------------------------------------------------------------
15 netconf-ssh admin 10.40.213.1 2021-07-22T00:44:41-07:00 1 0 0 0
Controller-2
admin@Controller-2-cli> show netconf-state sessions
IN OUT
SESSION IN BAD RPC OUT
ID TRANSPORT USERNAME SOURCE HOST LOGIN TIME RPCS RPCS ERRORS NOTIFICATIONS
------------------------------------------------------------------------------------------------------------
15 netconf-ssh admin 10.40.213.1 2021-07-22T00:44:42-07:00 1 0 0 0
Check the session state with the Log Collector
Controller-1
admin@Controller-1-cli> show orgs org-services Provider lef collectors LEF-Collector-Analytics-1-1 status
VSN SOURCE DESTINATION DESTINATION ROUTING PENDING
ID SOURCE IP PORT IP PORT INSTANCE STATUS MSGS
--------------------------------------------------------------------------------------
2 169.254.0.3 1026 11.0.192.1 1234 Analytics-VR Established 0
Controller-2
admin@Controller-2-cli> show orgs org-services Provider lef collectors LEF-Collector-Analytics-2-1 status
VSN SOURCE DESTINATION DESTINATION ROUTING PENDING
ID SOURCE IP PORT IP PORT INSTANCE STATUS MSGS
----------------------------------------------------------------------------------------
2 169.254.0.3 1026 11.0.192.2 1234 Analytics-VR Established 0
OSSPACK update on Controller nodes
Before you upgrade the VOS to Releases 20.2 or later, upgrade the OS SPack on all Controller nodes to the latest release.
Controller-1
[admin@Controller-1: ~] $ cd /home/versa/packages/
[admin@Controller-1: packages] $ pwd
/home/versa/packages
[admin@Controller-1: packages] $ ls -l
total 1354520
-rw-r-----+ 2 versa versa 555352760 Oct 11 2019 versa-flexvnf-20191010-145033-c1a1363-16.1R2S10.bin
-rwxrwx--x+ 1 admin versa 650566707 Jul 22 01:03 versa-flexvnf-20201002-063050-969d704-20.2.3.bin
-rwxrwx--x+ 1 admin versa 181092701 Jul 22 01:07 versa-flexvnf-osspack-20210428.bin
[admin@Controller-1: packages] $ sudo ./versa-flexvnf-osspack-20210428.bin
[sudo] password for admin:
Verifying archive integrity... All good.
Uncompressing OSSPACK 100%
...
Updating certificates in /etc/ssl/certs... 10 added, 31 removed; done.
Running hooks in /etc/ca-certificates/update.d....done.
Controller-2
[admin@Controller-2: ~] $ cd /home/versa/packages/
[admin@Controller-2: packages] $ pwd
/home/versa/packages
[admin@Controller-2: packages] $ ls -l
total 1354520
-rw-r-----+ 2 versa versa 555352760 Oct 11 2019 versa-flexvnf-20191010-145033-c1a1363-16.1R2S10.bin
-rwxrwx--x+ 1 admin versa 650566707 Jul 22 01:06 versa-flexvnf-20201002-063050-969d704-20.2.3.bin
-rwxrwx--x+ 1 admin versa 181092701 Jul 22 01:10 versa-flexvnf-osspack-20210428.bin
[admin@Controller-2: packages] $ sudo ./versa-flexvnf-osspack-20210428.bin
[sudo] password for admin:
Verifying archive integrity... All good.
Uncompressing OSSPACK 100%
...
Updating certificates in /etc/ssl/certs... 10 added, 31 removed; done.
Running hooks in /etc/ca-certificates/update.d....done.
Verify the OS Spack was installed successfully.
Controller-1
admin@Controller-1-cli> show security osspack info
Version 20210428
Update Type full
[ok][2021-07-22 03:17:45]
Controller-2
admin@Controller-2-cli> show security osspack info
Version 20210428
Update Type full
[ok][2021-07-22 02:51:22]
Upgrade Controller nodes software
A second ssh session could be used to verify the upgrade progress and catch the eventual errors, by using the bellow command:
tail -f /var/log/versa/upgrade.log
The Controllers upgrade should be performed node by node, so always there will be a running Controller(BGP Route Reflector). When a Controller node upgrade is finished, please redo the checks in 2.1.5.2 before to start the upgrade process on the next controller. Do not forget to perform the checks also on the last upgraded Controller.
Controller-1
admin@Controller-1-cli> request system package upgrade versa-flexvnf-20201002-063050-969d704-20.2.3.bin
Will restart the appliance (all processes). Are you sure? [no,yes]yes
[admin@Controller-1: packages] $ 2021-07-22 03:22:34.696 INFO Installing ... (events will be logged to console)
2021-07-22 03:22:34.704 INFO Taking a backup of CDB files to /var/lib/vs/tmp.Y1iTeRNcSW.upgrade (in case we need to rollback)
2021-07-22 03:22:37.797 INFO No Osspack installation in progress, proceeding to upgrade
2021-07-22 03:22:37.841 INFO Upgrade checkpoint #1: recovery handler installed
2021-07-22 03:22:37.900 INFO Stopping and removing current package 16.1R2 (VCSN)
2021-07-22 03:23:02.725 INFO Upgrade checkpoint #2: Previous package removed
2021-07-22 03:25:36.003 INFO Deleting /run/vinterfaces
2021-07-22 03:25:50.623 INFO Package Status: install ok installed
2021-07-22 03:25:50.628 INFO Upgrade checkpoint #3: New package installed
2021-07-22 03:25:50.641 INFO Restoring vsboot.conf
2021-07-22 03:25:50.646 INFO isolcpu - was not enabled
2021-07-22 03:25:50.672 INFO UPGRADE from 16.1R2 to 20.2.3
2021-07-22 03:25:50.676 INFO Starting ConfD in Phase 0
2021-07-22 03:25:59.448 INFO Upgrade transaction created - run upgrade scripts...
2021-07-22 03:25:59.454 INFO Invoking /opt/versa/upgrade/scripts/aaarm.lua
2021-07-22 03:25:59.839 INFO Invoking /opt/versa/upgrade/scripts/addr_objid.lua
2021-07-22 03:26:00.185 INFO Invoking /opt/versa/upgrade/scripts/av.lua
2021-07-22 03:26:01.473 INFO Invoking /opt/versa/upgrade/scripts/banner_test.lua
2021-07-22 03:26:01.883 INFO Invoking /opt/versa/upgrade/scripts/bgp2lmgr_rtds2i3_join_upgrade.elf
2021-07-22 03:26:02.304 INFO Invoking /opt/versa/upgrade/scripts/bgp_gr_upgrade.elf
2021-07-22 03:26:02.629 INFO Invoking /opt/versa/upgrade/scripts/bgp_local_address_upgrade.elf
2021-07-22 03:26:02.931 INFO Invoking /opt/versa/upgrade/scripts/bgp_local_id_upgrade.elf
2021-07-22 03:26:03.249 INFO Invoking /opt/versa/upgrade/scripts/cgnat_port.lua
2021-07-22 03:26:03.624 INFO Invoking /opt/versa/upgrade/scripts/cgnat_sourceport.lua
2021-07-22 03:26:03.954 INFO Ignore /opt/versa/upgrade/scripts/confd_dyn_delta.xml
2021-07-22 03:26:03.957 INFO Invoking /opt/versa/upgrade/scripts/ddslam.lua
2021-07-22 03:26:04.322 INFO Invoking /opt/versa/upgrade/scripts/decrypt_profile.lua
2021-07-22 03:26:04.636 INFO Invoking /opt/versa/upgrade/scripts/dhcpv6.lua
2021-07-22 03:26:04.947 INFO Invoking /opt/versa/upgrade/scripts/dns-proxy-config.lua
2021-07-22 03:26:05.250 INFO Invoking /opt/versa/upgrade/scripts/dns-settings.lua
2021-07-22 03:26:05.553 INFO Invoking /opt/versa/upgrade/scripts/fec-dup.lua
2021-07-22 03:26:05.863 INFO Invoking /opt/versa/upgrade/scripts/global_label_space_upgrade.elf
2021-07-22 03:26:06.223 INFO Invoking /opt/versa/upgrade/scripts/instance_import_upgrade.elf
2021-07-22 03:26:06.616 INFO Invoking /opt/versa/upgrade/scripts/inter_chassis_ha_params.lua
2021-07-22 03:26:06.982 INFO Invoking /opt/versa/upgrade/scripts/interface_entry_upgrade.elf
2021-07-22 03:26:07.353 INFO Invoking /opt/versa/upgrade/scripts/ipsec_controller_auth_domain.lua
2021-07-22 03:26:07.681 INFO Invoking /opt/versa/upgrade/scripts/ipsec_controller_tvi.lua
2021-07-22 03:26:07.984 INFO Invoking /opt/versa/upgrade/scripts/ipsec_peer.lua
2021-07-22 03:26:08.299 INFO Invoking /opt/versa/upgrade/scripts/mpls_vpn_mgr_use_bgp_upgrade.elf
2021-07-22 03:26:08.662 INFO Invoking /opt/versa/upgrade/scripts/nacm-edit.lua
2021-07-22 03:26:09.046 INFO Invoking /opt/versa/upgrade/scripts/nameserver.lua
2021-07-22 03:26:09.419 INFO Invoking /opt/versa/upgrade/scripts/next_rti_key_upgrade.elf
2021-07-22 03:26:09.782 INFO Invoking /opt/versa/upgrade/scripts/path_policy.lua
2021-07-22 03:26:10.101 INFO Invoking /opt/versa/upgrade/scripts/pim_local_id_upgrade.elf
2021-07-22 03:26:10.404 INFO Invoking /opt/versa/upgrade/scripts/pppoe_base_intf.lua
2021-07-22 03:26:10.780 INFO Invoking /opt/versa/upgrade/scripts/qos_upgrade.lua
2021-07-22 03:26:11.471 INFO Invoking /opt/versa/upgrade/scripts/redist_monitor_upgrade.elf
2021-07-22 03:26:11.840 INFO Invoking /opt/versa/upgrade/scripts/redist_set_list_upgrade.elf
2021-07-22 03:26:12.175 INFO Invoking /opt/versa/upgrade/scripts/redist_src_dest_proto.elf
2021-07-22 03:26:12.482 INFO Invoking /opt/versa/upgrade/scripts/rest_https.lua
2021-07-22 03:26:12.844 INFO Invoking /opt/versa/upgrade/scripts/routing_R16_upgrade.elf
2021-07-22 03:26:13.474 INFO Invoking /opt/versa/upgrade/scripts/rti_alarm_upgrade.lua
2021-07-22 03:26:13.775 INFO Invoking /opt/versa/upgrade/scripts/rti_alarms.elf
2021-07-22 03:26:14.080 INFO Invoking /opt/versa/upgrade/scripts/rti_icmp_rt_upgrade.elf
2021-07-22 03:26:14.449 INFO Invoking /opt/versa/upgrade/scripts/rti_icmp_upgrade.elf
2021-07-22 03:26:14.808 INFO Invoking /opt/versa/upgrade/scripts/rti_name_upgrade.elf
2021-07-22 03:26:15.181 INFO Invoking /opt/versa/upgrade/scripts/rti_owned_rti_upgrade.lua
2021-07-22 03:26:15.507 INFO Invoking /opt/versa/upgrade/scripts/rti_tenant_name_upgrade.elf
2021-07-22 03:26:15.908 INFO Invoking /opt/versa/upgrade/scripts/sdwan_site_upgrade.elf
2021-07-22 03:26:16.270 INFO Invoking /opt/versa/upgrade/scripts/sdwan_sla_loss_pct.lua
2021-07-22 03:26:16.606 INFO Invoking /opt/versa/upgrade/scripts/skulimits.lua
2021-07-22 03:26:17.329 INFO Invoking /opt/versa/upgrade/scripts/sla_interval.lua
2021-07-22 03:26:17.709 INFO Invoking /opt/versa/upgrade/scripts/subj_noip.lua
2021-07-22 03:26:18.017 INFO Invoking /opt/versa/upgrade/scripts/upgrade_bgp_auto_group_type.lua
2021-07-22 03:26:18.316 INFO Invoking /opt/versa/upgrade/scripts/upgrade_bgp_nm.elf
2021-07-22 03:26:18.677 INFO Invoking /opt/versa/upgrade/scripts/upgrade_bgp_peer_group.elf
2021-07-22 03:26:19.016 INFO Invoking /opt/versa/upgrade/scripts/upgrade_bgp_rm_tenant_name.lua
2021-07-22 03:26:19.321 INFO Invoking /opt/versa/upgrade/scripts/upgrade_branch.elf
2021-07-22 03:26:19.614 INFO Invoking /opt/versa/upgrade/scripts/upgrade_controller.elf
2021-07-22 03:26:19.911 INFO Invoking /opt/versa/upgrade/scripts/upgrade_mrib_routing.elf
2021-07-22 03:26:20.278 INFO Invoking /opt/versa/upgrade/scripts/urlf.lua
2021-07-22 03:26:20.650 INFO Invoking /opt/versa/upgrade/scripts/vfp_rule.lua
2021-07-22 03:26:20.951 INFO Invoking /opt/versa/upgrade/scripts/vfp_vrf.lua
2021-07-22 03:26:21.253 INFO Invoking /opt/versa/upgrade/scripts/vwire_retain.lua
2021-07-22 03:26:21.601 INFO Upgrade checkpoint #4: recovery handler removed
2021-07-22 03:26:21.606 INFO Fully starting ConfD only...
2021-07-22 03:26:26.095 INFO Waiting for ConfD to start...
2021-07-22 03:26:27.108 INFO ConfD fully started
2021-07-22 03:26:27.117 INFO Reboot required, checks deferred
2021-07-22 03:26:27.121 INFO Reboot required for upgrade to take effect (USB modem not installed)
2021-07-22 03:26:27.125 INFO Rebooting system in 5 seconds...
Broadcast message from root@Controller-1
(unknown) at 3:26 ...
The system is going down for reboot NOW!
Controller-2
admin@Controller-2-cli> request system package upgrade versa-flexvnf-20201002-063050-969d704-20.2.3.bin
Will restart the appliance (all processes). Are you sure? [no,yes]yes
[admin@Controller-2: packages] $ 2021-07-22 02:55:02.616 INFO Installing ... (events will be logged to console)
2021-07-22 02:55:02.626 INFO Taking a backup of CDB files to /var/lib/vs/tmp.bSkPCWG1uH.upgrade (in case we need to rollback)
2021-07-22 02:55:04.221 INFO No Osspack installation in progress, proceeding to upgrade
2021-07-22 02:55:04.265 INFO Upgrade checkpoint #1: recovery handler installed
2021-07-22 02:55:04.305 INFO Stopping and removing current package 16.1R2 (VCSN)
2021-07-22 02:55:28.405 INFO Upgrade checkpoint #2: Previous package removed
2021-07-22 02:58:09.028 INFO Deleting /run/vinterfaces
2021-07-22 02:58:24.127 INFO Package Status: install ok installed
2021-07-22 02:58:24.132 INFO Upgrade checkpoint #3: New package installed
2021-07-22 02:58:24.147 INFO Restoring vsboot.conf
2021-07-22 02:58:24.155 INFO isolcpu - was not enabled
2021-07-22 02:58:24.175 INFO UPGRADE from 16.1R2 to 20.2.3
2021-07-22 02:58:24.179 INFO Starting ConfD in Phase 0
2021-07-22 02:58:31.529 INFO Upgrade transaction created - run upgrade scripts...
2021-07-22 02:58:31.535 INFO Invoking /opt/versa/upgrade/scripts/aaarm.lua
2021-07-22 02:58:31.938 INFO Invoking /opt/versa/upgrade/scripts/addr_objid.lua
2021-07-22 02:58:32.283 INFO Invoking /opt/versa/upgrade/scripts/av.lua
2021-07-22 02:58:32.622 INFO Invoking /opt/versa/upgrade/scripts/banner_test.lua
2021-07-22 02:58:32.992 INFO Invoking /opt/versa/upgrade/scripts/bgp2lmgr_rtds2i3_join_upgrade.elf
2021-07-22 02:58:33.399 INFO Invoking /opt/versa/upgrade/scripts/bgp_gr_upgrade.elf
2021-07-22 02:58:33.695 INFO Invoking /opt/versa/upgrade/scripts/bgp_local_address_upgrade.elf
2021-07-22 02:58:33.992 INFO Invoking /opt/versa/upgrade/scripts/bgp_local_id_upgrade.elf
2021-07-22 02:58:34.294 INFO Invoking /opt/versa/upgrade/scripts/cgnat_port.lua
2021-07-22 02:58:34.601 INFO Invoking /opt/versa/upgrade/scripts/cgnat_sourceport.lua
2021-07-22 02:58:34.903 INFO Ignore /opt/versa/upgrade/scripts/confd_dyn_delta.xml
2021-07-22 02:58:34.908 INFO Invoking /opt/versa/upgrade/scripts/ddslam.lua
2021-07-22 02:58:35.277 INFO Invoking /opt/versa/upgrade/scripts/decrypt_profile.lua
2021-07-22 02:58:35.622 INFO Invoking /opt/versa/upgrade/scripts/dhcpv6.lua
2021-07-22 02:58:35.993 INFO Invoking /opt/versa/upgrade/scripts/dns-proxy-config.lua
2021-07-22 02:58:36.376 INFO Invoking /opt/versa/upgrade/scripts/dns-settings.lua
2021-07-22 02:58:36.749 INFO Invoking /opt/versa/upgrade/scripts/fec-dup.lua
2021-07-22 02:58:37.105 INFO Invoking /opt/versa/upgrade/scripts/global_label_space_upgrade.elf
2021-07-22 02:58:37.466 INFO Invoking /opt/versa/upgrade/scripts/instance_import_upgrade.elf
2021-07-22 02:58:37.825 INFO Invoking /opt/versa/upgrade/scripts/inter_chassis_ha_params.lua
2021-07-22 02:58:38.127 INFO Invoking /opt/versa/upgrade/scripts/interface_entry_upgrade.elf
2021-07-22 02:58:38.494 INFO Invoking /opt/versa/upgrade/scripts/ipsec_controller_auth_domain.lua
2021-07-22 02:58:38.863 INFO Invoking /opt/versa/upgrade/scripts/ipsec_controller_tvi.lua
2021-07-22 02:58:39.190 INFO Invoking /opt/versa/upgrade/scripts/ipsec_peer.lua
2021-07-22 02:58:39.509 INFO Invoking /opt/versa/upgrade/scripts/mpls_vpn_mgr_use_bgp_upgrade.elf
2021-07-22 02:58:39.874 INFO Invoking /opt/versa/upgrade/scripts/nacm-edit.lua
2021-07-22 02:58:40.253 INFO Invoking /opt/versa/upgrade/scripts/nameserver.lua
2021-07-22 02:58:40.630 INFO Invoking /opt/versa/upgrade/scripts/next_rti_key_upgrade.elf
2021-07-22 02:58:41.022 INFO Invoking /opt/versa/upgrade/scripts/path_policy.lua
2021-07-22 02:58:41.389 INFO Invoking /opt/versa/upgrade/scripts/pim_local_id_upgrade.elf
2021-07-22 02:58:41.691 INFO Invoking /opt/versa/upgrade/scripts/pppoe_base_intf.lua
2021-07-22 02:58:41.999 INFO Invoking /opt/versa/upgrade/scripts/qos_upgrade.lua
2021-07-22 02:58:42.709 INFO Invoking /opt/versa/upgrade/scripts/redist_monitor_upgrade.elf
2021-07-22 02:58:43.016 INFO Invoking /opt/versa/upgrade/scripts/redist_set_list_upgrade.elf
2021-07-22 02:58:43.384 INFO Invoking /opt/versa/upgrade/scripts/redist_src_dest_proto.elf
2021-07-22 02:58:43.747 INFO Invoking /opt/versa/upgrade/scripts/rest_https.lua
2021-07-22 02:58:44.121 INFO Invoking /opt/versa/upgrade/scripts/routing_R16_upgrade.elf
2021-07-22 02:58:44.874 INFO Invoking /opt/versa/upgrade/scripts/rti_alarm_upgrade.lua
2021-07-22 02:58:45.243 INFO Invoking /opt/versa/upgrade/scripts/rti_alarms.elf
2021-07-22 02:58:45.587 INFO Invoking /opt/versa/upgrade/scripts/rti_icmp_rt_upgrade.elf
2021-07-22 02:58:45.951 INFO Invoking /opt/versa/upgrade/scripts/rti_icmp_upgrade.elf
2021-07-22 02:58:46.288 INFO Invoking /opt/versa/upgrade/scripts/rti_name_upgrade.elf
2021-07-22 02:58:46.592 INFO Invoking /opt/versa/upgrade/scripts/rti_owned_rti_upgrade.lua
2021-07-22 02:58:46.897 INFO Invoking /opt/versa/upgrade/scripts/rti_tenant_name_upgrade.elf
2021-07-22 02:58:47.236 INFO Invoking /opt/versa/upgrade/scripts/sdwan_site_upgrade.elf
2021-07-22 02:58:47.596 INFO Invoking /opt/versa/upgrade/scripts/sdwan_sla_loss_pct.lua
2021-07-22 02:58:47.941 INFO Invoking /opt/versa/upgrade/scripts/skulimits.lua
2021-07-22 02:58:48.680 INFO Invoking /opt/versa/upgrade/scripts/sla_interval.lua
2021-07-22 02:58:49.077 INFO Invoking /opt/versa/upgrade/scripts/subj_noip.lua
2021-07-22 02:58:49.445 INFO Invoking /opt/versa/upgrade/scripts/upgrade_bgp_auto_group_type.lua
2021-07-22 02:58:49.822 INFO Invoking /opt/versa/upgrade/scripts/upgrade_bgp_nm.elf
2021-07-22 02:58:50.159 INFO Invoking /opt/versa/upgrade/scripts/upgrade_bgp_peer_group.elf
2021-07-22 02:58:50.465 INFO Invoking /opt/versa/upgrade/scripts/upgrade_bgp_rm_tenant_name.lua
2021-07-22 02:58:50.780 INFO Invoking /opt/versa/upgrade/scripts/upgrade_branch.elf
2021-07-22 02:58:51.140 INFO Invoking /opt/versa/upgrade/scripts/upgrade_controller.elf
2021-07-22 02:58:51.496 INFO Invoking /opt/versa/upgrade/scripts/upgrade_mrib_routing.elf
2021-07-22 02:58:51.934 INFO Invoking /opt/versa/upgrade/scripts/urlf.lua
2021-07-22 02:58:52.323 INFO Invoking /opt/versa/upgrade/scripts/vfp_rule.lua
2021-07-22 02:58:52.655 INFO Invoking /opt/versa/upgrade/scripts/vfp_vrf.lua
2021-07-22 02:58:52.957 INFO Invoking /opt/versa/upgrade/scripts/vwire_retain.lua
2021-07-22 02:58:53.337 INFO Upgrade checkpoint #4: recovery handler removed
2021-07-22 02:58:53.341 INFO Fully starting ConfD only...
2021-07-22 02:58:57.006 INFO Waiting for ConfD to start...
2021-07-22 02:58:58.019 INFO ConfD fully started
2021-07-22 02:58:58.030 INFO Reboot required, checks deferred
2021-07-22 02:58:58.060 INFO Reboot required for upgrade to take effect (USB modem not installed)
2021-07-22 02:58:58.067 INFO Rebooting system in 5 seconds...
Broadcast message from root@Controller-2
(unknown) at 2:59 ...
The system is going down for reboot NOW!
Verify the new installed VOS
Controller-1
admin@Controller-1-cli> show system package-info
Package Versa FlexVNF software
Release 20.2.3
Spack api version 11
Spack lib version 5
Release Type GA
Release date 20201002
Package id 969d704
Package name versa-flexvnf-20201002-063050-969d704-20.2.3
Branch 20.2
Creator jenkins
[ok][2021-07-22 03:35:26]
Controller-2
admin@Controller-2-cli> show system package-info
Package Versa FlexVNF software
Release 20.2.3
Spack api version 11
Spack lib version 5
Release Type GA
Release date 20201002
Package id 969d704
Package name versa-flexvnf-20201002-063050-969d704-20.2.3
Branch 20.2
Creator jenkins
[ok][2021-07-22 03:36:29]
This concludes the Controller nodes VOS upgrade process.
Summary
In this article, we saw how a redundant controller complex could be upgraded from a 16.1R2 release to 20.2.3 release. For further information please consult Versa Networks documentation.
