Out-of-the-blue we encountered that two of our Oracle databases started failing with the error code “62:342″ and description “Encountered an I/O error while performing the operation”. The odd thing is that we have more than 15 databases on the server and only two of them were having the issue.
Important to note is the fact that the server itself is a MediaAgent, which means the server has access to a shared tape device (in our case a VTL). The VTL itself is shared with approximately 90 MediaAgents and the storage policy used for database backup is associated with more than 250 databases. And… only 2 were failing…
After some digging we noticed the following entry in the logfile (CVD.log) of the MediaAgent: “03/31 11:21:40 [MEDIAFS ] 523157-483733 The operation failed because Unable to communicate with the device <name_of_the_VTL>. Please check whether the correct version of drivers have been installed for this device. Please check whether the device have been configured properly.”
Eventually we noticed in the Media Agent Properties the option “Use Native device driver for data transfer for tape media” under the tab “control”. After checking the checkbox and retrying the backupjob the issue was resolved.
The issue occurred on a CommVault Simpana 9.0 Service Pack 14 environment. Both the CommServe and the MediaAgent/Client were installed with the same binaries. The MediaAgent is RedHat Enterprise Linux 5 (x64) installed with an Oracle 10.2 database. The size of both databases were larger than 65GB.
Our hunch is related to a timeout which occurs during mount/unmount of the tape device as only these two databases – on this server – were larger than 50GB, which is the size of the virtual tapes. We are awaiting confirmation from CommVault for this.
Hope this helps.