Backup of CommServe DR server fails after Disaster Recovery Test

Recently I performed a Disaster Recovery test of the CommVault Simpana 10 software. The Disaster Recovery test has been executed by using the SQL Mirroring functionality as described in this procedure on the CommVault website.

My environment is configured as follows:

  • An active CommServe installed on server1 as instance1;
  • An standby CommServe installed (services are stopped) on server 2 as instance1;
  • A filesystem agent has been deployed on the StandBy CommServe as instance2. The agent is used to protect data locally by using the software installed on server1.

During my Disaster Recovery test, I executed the following steps:

  1. Took a Disaster Recovery backup;
  2. Enabled activity control on CommCell level;
  3. Stop the CommVault Simpana services on server1;
  4. Logon onto the active CommVault Simpana SQL instance on server1 and failover the database to server2;
  5. Stop the CommVault Simpana instance (instance2) on server 2;
  6. Perform a name change of the CommServe by using the CommServeDisasterRecoveryGui.exe application which can be found in the base folder;
  7. Start the services (instance1) on server2;
  8. From within the CommVault Simpana console migrate some clients from server1 to server2;
  9. Disable activity control on CommCell level;
  10. Do a test backup & test restore to validate if a client can be protected and if data recovery is possible;
  11. Fallback the CommVault application to server1 by executing the same steps 1 to 9;
  12. Start the CommVault services (instance2) on server2.

Once done, all data can perfectly protected (and recovered) in exception of data stored on server2. New backups executed are put in queue with the following error message: “Error Code: [19:1327] Description: Attempt start error: [Connection to remote machine [server2.domain.local] refused. Please check that the services are running on the remote machine.] Source: server1, Process: JobManager“.

First of all, it’s required to determine if the client is reachable over the network:

  • Perform a ping from server1 to server2 to validate if the hostnames can be perfectly resolved.
  • Perform a ping from server1 to server2 to validate if the hostnames can be perfectly resolved.
  • Perform a cvping (application can be found in the base folder) from server1 to server2 to on the CommVault ports. If you don’t know which ports are being used, open the CommVault console and consult the advanced properties of the client. The ports can be found on the general tab.
  • Do the same cvping test from server2 to server1 to validate if the connectivity works in both ways.

If all above mentioned tests are succesful, check the properties of the client. The client should be published towards server1 by using it’s hostname server2. In my case the server was published as server1 towards server1. Needless to say this caused a communication conflict. When I tried to alter the hostname back to server2, I received an error message stating a duplicate hostname is already available. However it was not visible within the console.

After some digging in the database I found a SQL view called “CommCellClientConfig” in the database “CommServ”. In this table I found some duplicate entries.

In case the table contains too many records, this SQL query can be used to efficiently lookup the different records: “use commserv;┬áselect * from CommCellClientConfig where client like '%SERVER2%'“, where SERVER2 is the name of your CommServ Disaster Recovery server.

So now that we found what is causing the issue, it’s required to get it resolved. As we are unable to change the properties of the client within the console and deleting records in the CommServ database is not advised, we used the following workaround. Please note, the execution of this workaround will erase the backupsets containing backupdata of this server. In case you don’t want to erase the media, do not execute this procedure!

  1. Release the license of server2 within the CommVault Simpana console;
  2. Delete the client server2 in the CommVault Simpana console, when prompted fill in “erase and reuse media“;
  3. Cycle the services on the active CommServe;
  4. The client will be visible again in the client;
  5. Reconfigure the client by assigning a storage policy and so on;
  6. Execute a backup of the client to check if Data Protection operations are successfully completed.

Hope this helps!

Leave a Reply