Stop Vertica Gracefully

[Praveen Venkatesan]

Hi All,

 How can i stop the vertica db on a single node gracefully during automatic patch update for a single node.

[David DuPre]

Data Repository (DR) Host
-----------------------------
The DR host only requires the DB be stopped and shut down prior to system restarts.
The Vertica processes that run the DB can be left as is during a restart. The key detail is
ensuring the DR DB is stopped.

Also, as noted above in the DA section, the dadaemon service for the DA MUST but shut down
before stopping the DR DB. If this is not done an error will be seen. When trying to stop
the DB via the adminTools utility, if the DA remains running a message will be seen that states:

"Error: NOTICE 2519: Cannot shut down while users are connected"

If that message is observed in a popup message in adminTools when trying to step the DB,
check to ensure the dadaemon service is shut down and not running.
To shut down the DR DB (can be done from any active node in a multi-node cluster):
Log into the DR DB host as the dradmin or equivalent DB admin user created during installation.
Go to the /opt/vertica/bin directory and run:
./adminTools
If the environment is configured properly you may alternatively run this from any location:
/opt/vertica/bin/adminTools
Choose option 4 "Stop the Database".
Choose the DB name to stop. Standard environments should only have one entry.
Enter the password and wait for the DB to stop.

When complete in the main menu for adminTools, select option 1 "View Database Cluster State"
to ensure all nodes show as down before they are rebooted.

To start the DB again after the host is rebooted, if not automatically restarted, choose option 3
"Start the Database", choose the DB to restart, enter the correct password and wait for the
restart to complete.

Once more to check status, choose option 1 "View Database Cluster State" to ensure all nodes
show as up post reboot.

Note that while the DB will not allow shut down while open connections from the dadaemon exist,
once that is shut down, we have seen instances where the DB didn't fully complete shut down
despite the status in adminTools showing as down. This is known as a 'dirty' DB shut down whereby
the DB appears to be down to the user and the system but isn't fully and truly down. Thus when
the restart is performed the DB won't restart without restoration using the last known good epoch.

While there is no **** way to check for the DB state outside of the adminTools UI,

the **** act of patience will often help avoid this problem.
--------------------------------------------------------------
When stopping the DB allow if possible an extra 20-40 minutes before restarting the DR host.

------------------------------
[JobTitle]
------------------------------

[David DuPre]

If this is a 3 or more node cluster, you can use the Vertica Management Console - a downloadable tool from Vertica to stop a single node and restart same.  (Install this on a separate server and connect it to the VerticaDB to see status and control start,stop, restart of nodes.

David

------------------------------
[JobTitle]
------------------------------

[Praveen Venkatesan]

Hi David ,

 I want to do this without any manual intervention .

[Jeffrey Pinard]

Vertica services/systemd script to shutdown DB on reboot is eh at best, it seems.  So to be the safest...
I suggest making a script that does:

#!/bin/bash
su - <dradminaccount> -c "/opt/vertica/bin/admintools -t stop_db -p <PASSWORD> -d <DBNAME> -i​​"

And call it by the tool (via ssh or whatever) that handles installing patches and rebooting.
Call it first before applying patches.

Note: shutting down the DB will tell DA to stop automatically, when DA can't reach DR.

[Praveen Venkatesan]

In case it is a 3 node cluster  and we are stopping only 1 node,it should not tell the DA to stop . Its right ?

[Jeffrey Pinard]

So with a 3 node cluster, as long as the DB remains up with 2 nodes, the DA will remain up.

But if you are doing a cluster, you want to use a diff command to stop vertica on host.

#!/bin/bash
su - <dradminaccount> -c "/opt/vertica/bin/admintools -t stop_node -s <hostname>"

But once the node comes back up, you will need to start it manually, because it was stopped manually.

#!/bin/bash
su - <dradminaccount> -c "/opt/vertica/bin/admintools -t restart_node -s <hostname> -d <DBNAME> -p <DBPASSWORD> -i"

[Jeffrey Pinard]

To add to my last post...

The commands I gave allow you to do 1 node patch update at a time.   So stop the node's vertica, patch, reboot, start the node's vertica.
Then move onto next node.  This way the DB stays up.

You can run this command to see the status of the cluster.   Might be good to run after you think the node is up, to confirm it's up before moving onto next node to patch.  Otherwise a 2nd node going down will cause the DB to shutdown.

[dradmin@dr ~]$ /opt/vertica/bin/admintools -t view_cluster
DB | Host | State
--------+------+-------
drdata | ALL | UP