I am trying to restore files from Azure Online Backup. However, the job keeps failing, saying that there is an internal error with the Azure service.
In the log files, I keep seeing the following block logged:
044C 03AC 03/23 08:12:09.130 79 WcfClient.cs(739) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING FMBlock: Executing retry policy | Params: {CurrentAttempt = 11}{MaxAttempts = 18000}{BackOffIntervalInSec = 18}{OperationCode
= WcfProxyGenericAsyncOperationPollingCall}{Retry policy id = 7}
044C 03AC 03/23 08:12:21.754 79 WcfClient.cs(871) A5067E21-49D4-4196-AFF0-84BD23980C1B NORMAL Executing web service call | Params: {ServiceInterface = Microsoft.Internal.CloudBackup.Common.Recovery.Interface.IRecoveryServiceChannel}{Description
= GetStatus}{TargetEndpoint =
https://pod1-rec1.ne.onlinebackup.microsoft.com/RecoveryService.svc}
044C 03AC 03/23 08:12:22.598 79 msoidcrlutil.cpp(385) A5067E21-49D4-4196-AFF0-84BD23980C1B NORMAL User [goweb@gemspace.onmicrosoft.com] successfully authenticated
044C 03AC 03/23 08:12:22.598 69 OrgLiveUserIdentity.cs(156) A5067E21-49D4-4196-AFF0-84BD23980C1B NORMAL Returning auth header | Params: {Name =goweb@gemspace.onmicrosoft.com}
044C 03AC 03/23 08:12:22.598 69 CBEngineWcfClientHelper.cs(298) A5067E21-49D4-4196-AFF0-84BD23980C1B NORMAL Setting RequestId header for outgoing request | Params: {RequestId = 30818bd3-a50a-4a02-bb73-43314e012270}{WorkitemId
= 1b0d4177-be74-40d3-8e86-66fb5a701c48}{TaskId = a5067e21-49d4-4196-aff0-84bd23980c1b}
044C 03AC 03/23 08:12:22.692 79 WcfClient.cs(892) A5067E21-49D4-4196-AFF0-84BD23980C1B NORMAL Finished web service call | Params: {ServiceInterface = Microsoft.Internal.CloudBackup.Common.Recovery.Interface.IRecoveryServiceChannel}{Description
= GetStatus}{TargetEndpoint =
https://pod1-rec1.ne.onlinebackup.microsoft.com/RecoveryService.svc}
I also see the following logged:
044C 03AC 03/23 08:25:24.010 79 msoidcrlutil.cpp(352) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING Failed: Hr: = [0x80048051] : LogonIdentityEx failed for User=[goweb@gemspace.onmicrosoft.com]
044C 03AC 03/23 08:25:24.010 79 msoidcrlutil.cpp(600) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING Failed: Hr: = [0x80048051] : Encountered Failure: : lVal : LogonUserInternal(hId, pszUser, pszPwd)
044C 03AC 03/23 08:26:15.370 79 WcfClient.cs(892) A5067E21-49D4-4196-AFF0-84BD23980C1B NORMAL Finished web service call | Params: {ServiceInterface = Microsoft.Internal.CloudBackup.Common.Recovery.Interface.IRecoveryServiceChannel}{Description
= GetStatus}{TargetEndpoint =
https://pod1-rec1.ne.onlinebackup.microsoft.com/RecoveryService.svc}
044C 03AC 03/23 08:26:15.370 79 WcfClient.cs(739) A5067E21-49D4-4196-AFF0-84BD23980C1B NORMAL FMBlock: Retry succeeded! | Params: {OperationCode = WcfProxyGenericAsyncOperationPollingCall}{CurrentAttempt = 62}
044C 03AC 03/23 08:26:15.370 79 WcfClient.cs(648) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING FMBlock: FM exception caught | Params: {OperationCode = WcfProxyAsyncCall}{OperationStartTime = 23/03/2013 08:09:52}{FMExceptionDetail
= FMException: [ErrorCode:CloudInternalError, DetailedCode:0, Source:None, Message:
044C 03AC 03/23 08:26:15.370 79 WcfClient.cs(648) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING RequestId: 75c8da8a-5743-47b4-a212-4fd7c3fa43eb
044C 03AC 03/23 08:26:15.370 79 WcfClient.cs(648) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING TaskId: a5067e21-49d4-4196-aff0-84bd23980c1b
044C 03AC 03/23 08:26:15.370 79 WcfClient.cs(648) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING CustomerId: 337769972052787285
044C 03AC 03/23 08:26:15.370 79 WcfClient.cs(648) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING IdentityType: User
044C 03AC 03/23 08:26:15.370 79 WcfClient.cs(648) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING Id: 94]
044C 03AC 03/23 08:26:15.370 79 WcfClient.cs(648) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING {WorkItemId = 345ebd54-8dc9-4199-9dd8-46d01e4c3b0e}
044C 03AC 03/23 08:26:15.370 79 WcfClient.cs(648) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING }
044C 03AC 03/23 08:26:15.370 69 PrepProxy.cs(397) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING FMBlock: FM exception caught | Params: {OperationCode = PrepareRecoveryMetadata}{OperationStartTime = 23/03/2013 08:09:49}{FMExceptionDetail
= FMException: [ErrorCode:CloudInternalError, DetailedCode:0, Source:None, Message:
044C 03AC 03/23 08:26:15.370 69 PrepProxy.cs(397) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING RequestId: 75c8da8a-5743-47b4-a212-4fd7c3fa43eb
044C 03AC 03/23 08:26:15.370 69 PrepProxy.cs(397) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING TaskId: a5067e21-49d4-4196-aff0-84bd23980c1b
044C 03AC 03/23 08:26:15.370 69 PrepProxy.cs(397) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING CustomerId: 337769972052787285
044C 03AC 03/23 08:26:15.370 69 PrepProxy.cs(397) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING IdentityType: User
044C 03AC 03/23 08:26:15.370 69 PrepProxy.cs(397) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING Id: 94]
044C 03AC 03/23 08:26:15.370 69 PrepProxy.cs(397) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING {WorkItemId = 345ebd54-8dc9-4199-9dd8-46d01e4c3b0e}
044C 03AC 03/23 08:26:15.370 69 PrepProxy.cs(397) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING }
044C 03AC 03/23 08:26:15.448 71 prepunmanagedutils.h(119) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING Error occured, DlsErrorCode - 1073871825, Exception - FMException: [ErrorCode:CloudInternalError, DetailedCode:0,
Source:None, Message:
044C 03AC 03/23 08:26:15.448 71 prepunmanagedutils.h(119) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING RequestId: 75c8da8a-5743-47b4-a212-4fd7c3fa43eb
044C 03AC 03/23 08:26:15.448 71 prepunmanagedutils.h(119) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING TaskId: a5067e21-49d4-4196-aff0-84bd23980c1b
044C 03AC 03/23 08:26:15.448 71 prepunmanagedutils.h(119) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING CustomerId: 337769972052787285
044C 03AC 03/23 08:26:15.448 71 prepunmanagedutils.h(119) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING IdentityType: User
044C 03AC 03/23 08:26:15.448 71 prepunmanagedutils.h(119) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING Id: 94]
044C 03AC 03/23 08:26:15.448 71 prepunmanagedutils.h(119) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING {WorkItemId = 345ebd54-8dc9-4199-9dd8-46d01e4c3b0e}
044C 03AC 03/23 08:26:15.448 71 prepunmanagedutils.h(119) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING Stack trace: at Microsoft.Internal.CloudBackup.Common.WcfClient.WcfClient`1.WaitForCompletion(Guid
workitemId)
044C 03AC 03/23 08:26:15.448 71 prepunmanagedutils.h(119) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING at Microsoft.Internal.CloudBackup.Common.WcfClient.WcfClient`1.<>c__DisplayClassd.<CallWcfService>b__c()
044C 03AC 03/23 08:26:15.448 71 prepunmanagedutils.h(119) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING at Microsoft.Internal.Common.FailureModeling.FMComponent`3._FMBlock(String fileName, Int32 lineNumber,
FMBlockArgs args, Action fmPrecheckBlock, Action fmUserBlock)
044C 03AC 03/23 08:26:15.448 71 prepunmanagedutils.h(119) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING at Microsoft.Internal.CloudBackup.Common.WcfClient.WcfClient`1.CallWcfService(Action webServiceCall,
String webServiceDescription)
044C 03AC 03/23 08:26:15.448 71 prepunmanagedutils.h(119) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING at Microsoft.Internal.CloudBackup.Common.WcfClient.WcfClient`1.CallWcfService[Tin,Tout](Func`2
webServiceCall, Tin argsIn, Tout& responseOut, String webServiceDescription)
044C 03AC 03/23 08:26:15.448 71 prepunmanagedutils.h(119) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING at Microsoft.Internal.CloudBackup.Client.Proxy.PREP.PrepProxy.<>c__DisplayClass2e.<PrepareRecoveryMetadataBlobs>b__2d()
044C 03AC 03/23 08:26:15.448 71 prepunmanagedutils.h(119) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING at Microsoft.Internal.Common.FailureModeling.FMComponent`3._FMBlock(String fileName, Int32 lineNumber,
FMBlockArgs args, Action fmPrecheckBlock, Action fmUserBlock)
044C 03AC 03/23 08:26:15.448 71 prepunmanagedutils.h(119) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING at Microsoft.Internal.CloudBackup.Client.Proxy.PREP.PrepProxy.PrepareRecoveryMetadataBlobs(PrepareRecoveryMetadataBlobsArgs
recoveryInformation)
044C 03AC 03/23 08:26:15.448 71 prepunmanagedutils.h(119) A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING at PrepareRecoveryMetadataBlobs(HResult* , _GUID guidIDContextStoreKey, PREP_RECOVERY_METADATA_INFORMATION_V2011_09*
metadataInfo, PREP_RECOVERY_METADATA_RESPONSE_V2011_09* pMetadataResponse_V2011_09, CErrorInfo* pErrorInfo)
044C 03AC 03/23 08:26:15.448 71 prepunmanagedutils.h(124) A5067E21-49D4-4196-AFF0-84BD23980C1B NORMAL ErrorMapper: Set Dls error [DLS - 1073871825]
044C 03AC 03/23 08:26:15.464 70 senddatasetsubtask.cpp(1197) [000000001A5CE4E0] A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING Failed: Hr: = [0x80131500] IFileProvider::PrepareStream failed
044C 03AC 03/23 08:26:15.479 18 dsmsubtaskbase.cpp(86) [000000001A652B70] A5067E21-49D4-4196-AFF0-84BD23980C1B NORMAL CDsmSubTaskBase: destructor [000000001A652B70]
044C 03AC 03/23 08:26:15.479 70 senddatasetsubtask.cpp(513) [000000001A5CE4E0] A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING Failed: Hr: = [0x80131500] CSendDatasetSubTask::Initialize failed
044C 03AC 03/23 08:26:15.479 71 dscontext.cpp(151) [000000001A5BE8F0] A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING Ds Id 16 failed. DLS: 1073871825 HRESULT: 0x80131500
044C 03AC 03/23 08:26:15.479 71 replicator.cpp(315) [000000001A5BD960] A5067E21-49D4-4196-AFF0-84BD23980C1B WARNING Failed: Hr: = [0x80131500] SetupReplicationForReplica unsuccessful for replica id ({AB75F3AB-DF08-496B-AD0A-3C696075254B})
Volume (\\?\Volume{bd67db32-c2d7-11dd-b046-806e6f6e6963}\))