Hi All
We are migrating from tape to Azure backup, progressively adding more volumes and data until everything is initially seeded. Unfortunately I have been constantly up against this issue which is making me seriously doubt our continued use of Azure backup.
Our setup:
DPM Server: 2012 R2
DPM: 2012 R2 UR10
Azure backup agent: 2.0.9040.0
Volume being backed up is general user file shares on a 2012 R2 Server. Deduplication is enabled on this volume. Data size is 1.9TB and our typical data churn each night is approximately 10-20GB.
Our online recovery points are failing at a very high rate due to errors encounted with individual files.
DPM console:
An error was encountered while performing an operation for C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\Replica\File System\vol_db2a28f4-1813-4284-872a-8ac5b73d6478\78f1d767-8092-4d0e-953e-90d8700e2d9f\Full\<file path\file name>. (ID 100091)
Event Viewer:
Creation of online recovery points for H:\ on <server> have failed. The last online recovery point creation failed for the following reason: (ID: 3188)
An error was encountered while performing an operation for C:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\Volumes\Replica\File System\vol_db2a28f4-1813-4284-872a-8ac5b73d6478\78f1d767-8092-4d0e-953e-90d8700e2d9f\Full\<file path\file name>. (ID:
100091)
CBEngine.errlogs
14D4 27B4 09/15 08:00:03.778 18 mtffile.cpp(684) [00000000207CF300] B352E3B1-CCF7-4F32-BD30-D63730AE9E36 WARNING Failed: Hr: = [0x00000000] Corruption detected, m_pFileWriter->Cancel() returned
14D4 27B4 09/15 08:00:03.778 70 continueonfailurehandler.cpp(401) [000000001C2536B0] B352E3B1-CCF7-4F32-BD30-D63730AE9E36 WARNING CContinueOnFailureHandler :: SetNextFailedFile entered ( hrReason:0x80990aa5 ,
dsmStatus:3, Logical File File Path:\\?\Volume{ab95b202-46d1-11e6-80fc-b8ca3a61122d}\78f1d767-8092-4d0e-953e-90d8700e2d9f\Full\<file path\file name>, Physical file Path: \\?\Volume{718213cc-83d1-4723-938f-97b0124da768}\78f1d767-8092-4d0e-953e-90d8700e2d9f\Full\<file
path\file name>, File Info Type:0, FileId: 0)
14D4 27B4 09/15 08:00:03.778 70 continueonfailurehandler.cpp(424) [000000001C2536B0] B352E3B1-CCF7-4F32-BD30-D63730AE9E36 WARNING Can't continue for failure in backup in data phase.
14D4 27B4 09/15 08:00:03.778 70 continueonfailurehandler.cpp(465) [000000001C2536B0] B352E3B1-CCF7-4F32-BD30-D63730AE9E36 WARNING Failed: Hr: = [0x80990aa5] CContinueOnFailureHandler:SetNextFailedFile : Adding
Record Failed,Returning Error.
14D4 27B4 09/15 08:00:03.778 18 mtffile.cpp(696) [00000000207CF300] B352E3B1-CCF7-4F32-BD30-D63730AE9E36 WARNING Failed: Hr: = [0x80990aa5] Callback returned
I have tried:
-chkdsk on the source volume and the DPM replica volume
-consistency checks
-full deduplication integrity scrubbing job (source volume only) followed by consistency checks, sync and new recovery point.
-DPM server reboot
-ensuring the file is not open on the network when a sync/recovery point is created.
-there is no corruption or issues accessing this file on the network.
-there are no issues with the DPM local syncs and recovery point creations, only the online recovery point creation fails.
This single file has been causing this issue over the consecutive past 4 nights despite the troubleshooting steps. This is only the current situation however we have been having this issue occur many times over a short period trying to migrate to Azure with many different files and file types on the volume we are trying to backup however one-way or the other I seem to be able to get it working again - this time not so much.
Thank you for any assistance in advance.