Recovering a corrupted HP Data Protector software StoreOnce store

In one of the remote sites, the customer is running an HP Data Protector 9.02 environment in combination with a software-based HP Data Protector StoreOnce deduplication store based on a Windows Server 2012.

Unfortunately during the initial installation, the consultant did not implement any anti-virus exclusions causing the software deduplication store to get corrupted last week. Files which got deleted were from the type: “$StoreRoot\**\containerData_*\*.js“.

Magically we were still able to bring the store back online in read-only mode and restore data from it! Thumbs up!

In the logfiles we found the following errors:

  • Failure in (src\corededupe\store\CStoreHK.cpp:428 dedupe::store::CStoreHouseKeeper::threadRoutine))
  • Exception [HousekeepingException]: thrown in (src\dedupe\core\CONISTON\src\corededupe\housekeeping/ctasks/Reclaimer.h:72 dedupe::housekeeping::Reclaimer::processContainers) Error (14) Bad address
  • Exception [HousekeepingException]: thrown in (src\dedupe\core\CONISTON\src\corededupe\housekeeping/ctasks/Reclaimer.h:72 dedupe::housekeeping::Reclaimer::processContainers) Error (14) Bad address boost::filesystem::file_size: The system cannot find the file specified: "D:\B2D\StoreOncelibrary\dvol_00000014\committed\storemgr\store.d\store_01\ containerData.d\containerDatagroup_0039.d\containerData_00399c8c"

We used the procedure below to get everything environment back online as advised by HP. Before giving it a try, we advise to open a case @HP to ensure you are using the correct procedure.

Step 1: stop the StoreOnce service by executing the following command “sc stop StoreOnceSoftware” or by stopping the service called “Data Protector StoreOnceSoftware“. We changed the startup type to “Disabled” to ensure the monitoring system would not try to start the service again.

Step 2: move the “s.bad_integrity” file which is on the following location: “<StoreRoot>\StoreOnceLibrary\dvol_00000001\commited\storemgr\store.d\store_01\” to a staging location. You don’t need the file anymore, but the copy is more a safety measure.

Step 3: move the files in the following directory to another location: “<StoreRoot>\StoreOnceLibrary\dvol_00000001\commited storemgr\store.d\store_01\retired“.

Step 4: configure the service “Data Protector StoreOnceSoftware” with an “automatic” startup type. Once done, start the service.

Step 5: verify if the store came online and is accessible for the software. This can be easily performed by executing the following command:

C:\Users\backupadmin>storeoncesoftware --list_stores
Store Name: HPDP001_B2D
Store Id: 1
Store Description: Data protector store
Store Status: started
Store Autostartable: ON
Store Readonly: OFF
User Data Stored: 97373 GB
Store Data Size: 4068 GB
Deduplication Ratio: 23.9 : 1
Listing of the stores succeeded.

Step 6: if the store goes offline after some time (give it at least 30 minutes), it can be a possible solution to configure the store in read-only mode. The read-only mode can be changed by executing the following command: “C:\Users\backupadmin>StoreOnceSoftware --set_readonly=ON --name=HPDP001_B2D“. When executing the previous command, you will see the store will be set in readonly by the value “Store Readonly: ON“.

Step 7: bring your backup environment back operational. In both cases (store remains online & store had to be put in read-only), it’s adviseable to create a new store. Take the following things in account when executing this operation.

  • When creating a new store, the store will be merged in the folder structure in the root of the configuration (execute the following command to retrieve this information:  “C:\Users\backupadmin>storeoncesoftware --get_server_properties“).
  • A new baseline needs to be created. Ensure the disk hosting the Store Root has sufficient disk space to be allocated for the new baseline.

Step 8: once the store is created, you can go forward with modifying the Data Protector selection lists with the freshly created destination.

That’s about it! I hope you will never need it… But anyhow here it is :).

Thanks for reading!

Leave a Reply