In one of the remote sites, the customer is running an HP Data Protector 9.02 environment in combination with a software-based HP Data Protector StoreOnce deduplication store based on a Windows Server 2012.
Unfortunately during the initial installation, the consultant did not implement any anti-virus exclusions causing the software deduplication store to get corrupted last week. Files which got deleted were from the type: “
Magically we were still able to bring the store back online in read-only mode and restore data from it! Thumbs up!
In the logfiles we found the following errors:
Failure in (src\corededupe\store\CStoreHK.cpp:428 dedupe::store::CStoreHouseKeeper::threadRoutine))
Exception [HousekeepingException]: thrown in (src\dedupe\core\CONISTON\src\corededupe\housekeeping/ctasks/Reclaimer.h:72 dedupe::housekeeping::Reclaimer::processContainers) Error (14) Bad address
Exception [HousekeepingException]: thrown in (src\dedupe\core\CONISTON\src\corededupe\housekeeping/ctasks/Reclaimer.h:72 dedupe::housekeeping::Reclaimer::processContainers) Error (14) Bad address boost::filesystem::file_size: The system cannot find the file specified: "D:\B2D\StoreOncelibrary\dvol_00000014\committed\storemgr\store.d\store_01\ containerData.d\containerDatagroup_0039.d\containerData_00399c8c"
We used the procedure below to get everything environment back online as advised by HP. Before giving it a try, we advise to open a case @HP to ensure you are using the correct procedure.
Step 1: stop the StoreOnce service by executing the following command “
sc stop StoreOnceSoftware” or by stopping the service called “
Data Protector StoreOnceSoftware“. We changed the startup type to “
Disabled” to ensure the monitoring system would not try to start the service again.
Step 2: move the “
s.bad_integrity” file which is on the following location: “
<StoreRoot>\StoreOnceLibrary\dvol_00000001\commited\storemgr\store.d\store_01\” to a staging location. You don’t need the file anymore, but the copy is more a safety measure.
Step 3: move the files in the following directory to another location: “
Step 4: configure the service “
Data Protector StoreOnceSoftware” with an “
automatic” startup type. Once done, start the service.
Step 5: verify if the store came online and is accessible for the software. This can be easily performed by executing the following command:
C:\Users\backupadmin>storeoncesoftware --list_storesStore Status: started
Store Name: HPDP001_B2D
Store Id: 1
Store Description: Data protector store
Store Readonly: OFF
Store Autostartable: ON
Store Data Size: 4068 GB
User Data Stored: 97373 GB
Deduplication Ratio: 23.9 : 1
Listing of the stores succeeded.
Step 6: if the store goes offline after some time (give it at least 30 minutes), it can be a possible solution to configure the store in read-only mode. The read-only mode can be changed by executing the following command: “
C:\Users\backupadmin>StoreOnceSoftware --set_readonly=ON --name=HPDP001_B2D“. When executing the previous command, you will see the store will be set in readonly by the value “
Store Readonly: ON“.
Step 7: bring your backup environment back operational. In both cases (store remains online & store had to be put in read-only), it’s adviseable to create a new store. Take the following things in account when executing this operation.
- When creating a new store, the store will be merged in the folder structure in the root of the configuration (execute the following command to retrieve this information: “
- A new baseline needs to be created. Ensure the disk hosting the Store Root has sufficient disk space to be allocated for the new baseline.
Step 8: once the store is created, you can go forward with modifying the Data Protector selection lists with the freshly created destination.
That’s about it! I hope you will never need it… But anyhow here it is :).
Thanks for reading!