Slow MARS Agent Backup Peformance

We have been using Azure Backup (MARS Agent) to backup an Azure IaaS VM 2012R2 file server (Files Folders) to an Azure Backup Vault in this clients Azure subscription.

About 10 data ago a 'large' DSF/R data set was seeded to the server and synced up using DFS replication. The data set size is ~ 1.5TB in ~13.5K Directories and ~132K files. The MARS agent is set to create 2 recovery points / day (06:00 & 18:00). The rate of change on the data set is minimal.

Based on prompting in the MARS Agent and information in KB  3109750 / 3090067, we upgraded the MARS agent about 5 days ago to version 2.0.9028.0 (which I believe is the most current).

Since the addition of the DFS/R data the creation of a new recovery point has taken a backup has taken a extremely long time to run to completion - i.e. a recovery point started on 2016/2/07 @ 18:00 UTC completed as at 2016/2/14 @ ~05:00 UTC [it moved ~ 2.3 GB of changed data].  7+ days to create a recovery point  is not acceptable [during this time any additional recovery point creation operations fail of course] !!

It appears as if the CBEngine process is reading every byte in every file for both the backup and validation phases of the backup.

I initiated a Backup Once recovery point @ 2016-02-15 ~19:07 UTC - it again appears that the CBEngine process is reading every byte in every file and given the rate at which it is doing it will take a number of days to complete ….  

The CBEngine process does not tax the server resources (CPU, Disk, Network) to any great extent. A proxy is not configured and the MARS agent throttling is not set. Observation of resource utilization in Resource Monitor on the server show very low CPU usage (< about 1%), essentially no network usage, and the disk utilization is low  ~ <= 10 MB/sec for CBEngine.exe (as a point of reference copy / WSB operation on this VM can move data at 100+ MB/sec on a sustained consistent basis)

These warning are continuously repeated in the CBEngine.errlog file:

ProcessId ThreadId Date Time ComponentCode FileName(FileLine) This TaskId Level TraceMessage
0934 0404 02/15 13:48:01.192 71 dscontext.cpp(157) [000000001F804A30] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70368933128041) is 6
0934 0404 02/15 13:48:04.387 71 dscontext.cpp(157) [000000001F803DB0] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70369239887403) is 5
0934 0404 02/15 13:48:06.410 71 dscontext.cpp(157) [000000001F804A30] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70368933128041) is 6
0934 0404 02/15 13:48:06.410 71 dscontext.cpp(157) [000000001F803DB0] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70369239887403) is 5
0934 0CDC 02/15 13:48:07.423 75 OMUtils.cs(70)  03D87099-E25A-40BC-BF5A-A6B6DC00EEE6 NORMAL GetFQDNForServer = wrl-neu-fs-01.wrl.local
0934 0404 02/15 13:48:08.418 71 dscontext.cpp(157) [000000001F804A30] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70368933128041) is 6
0934 0404 02/15 13:48:08.418 71 dscontext.cpp(157) [000000001F803DB0] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70369239887403) is 5
0934 137C 02/15 13:48:10.445 75 OMUtils.cs(70)  1A224C87-EB7B-4296-BE62-87C2FEE1637E NORMAL GetFQDNForServer = wrl-neu-fs-01.wrl.local
0934 0404 02/15 13:48:10.445 71 dscontext.cpp(157) [000000001F804A30] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70368933128041) is 6
0934 0404 02/15 13:48:10.445 71 dscontext.cpp(157) [000000001F803DB0] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70369239887403) is 5
0934 0404 02/15 13:48:12.463 71 dscontext.cpp(157) [000000001F804A30] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70368933128041) is 6
0934 0404 02/15 13:48:12.463 71 dscontext.cpp(157) [000000001F803DB0] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70369239887403) is 5
0934 137C 02/15 13:48:13.493 75 OMUtils.cs(70)  068D3620-9F44-48C7-B056-9D4929B0C871 NORMAL GetFQDNForServer = wrl-neu-fs-01.wrl.local
0934 0404 02/15 13:48:14.465 71 dscontext.cpp(157) [000000001F804A30] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70368933128041) is 6
0934 0404 02/15 13:48:14.465 71 dscontext.cpp(157) [000000001F803DB0] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70369239887403) is 5
0934 0404 02/15 13:48:16.474 71 dscontext.cpp(157) [000000001F804A30] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70368933128041) is 6
0934 0404 02/15 13:48:16.474 71 dscontext.cpp(157) [000000001F803DB0] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70369239887403) is 5
0934 06A0 02/15 13:48:16.521 75 OMUtils.cs(70)  ADDD7EDE-9ABF-4758-BDE2-D0CB19E33682 NORMAL GetFQDNForServer = wrl-neu-fs-01.wrl.local
0934 0404 02/15 13:48:18.484 71 dscontext.cpp(157) [000000001F804A30] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70368933128041) is 6
0934 0404 02/15 13:48:18.484 71 dscontext.cpp(157) [000000001F803DB0] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70369239887403) is 5
0934 06A0 02/15 13:48:19.590 75 OMUtils.cs(70)  651BE907-5210-4FEF-949C-5C2361337333 NORMAL GetFQDNForServer = wrl-neu-fs-01.wrl.local
0934 0404 02/15 13:48:20.487 71 dscontext.cpp(157) [000000001F804A30] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70368933128041) is 6
0934 0404 02/15 13:48:20.487 71 dscontext.cpp(157) [000000001F803DB0] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70369239887403) is 5
0934 0404 02/15 13:48:22.490 71 dscontext.cpp(157) [000000001F804A30] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70368933128041) is 6
0934 0404 02/15 13:48:22.490 71 dscontext.cpp(157) [000000001F803DB0] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70369239887403) is 5
0934 13C4 02/15 13:48:22.630 75 OMUtils.cs(70)  4D664649-0433-4142-8090-3EC5C24F1390 NORMAL GetFQDNForServer = wrl-neu-fs-01.wrl.local
0934 0404 02/15 13:48:24.497 71 dscontext.cpp(157) [000000001F804A30] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70368933128041) is 6
0934 0404 02/15 13:48:24.497 71 dscontext.cpp(157) [000000001F803DB0] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70369239887403) is 5
0934 13C4 02/15 13:48:25.666 75 OMUtils.cs(70)  1EC49670-5AFE-447E-8326-89CC24B0BE94 NORMAL GetFQDNForServer = wrl-neu-fs-01.wrl.local
0934 0404 02/15 13:48:26.505 71 dscontext.cpp(157) [000000001F804A30] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70368933128041) is 6
0934 0404 02/15 13:48:26.505 71 dscontext.cpp(157) [000000001F803DB0] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70369239887403) is 5
0934 0404 02/15 13:48:28.519 71 dscontext.cpp(157) [000000001F804A30] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70368933128041) is 6
0934 0404 02/15 13:48:28.519 71 dscontext.cpp(157) [000000001F803DB0] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70369239887403) is 5
0934 13C4 02/15 13:48:28.712 75 OMUtils.cs(70)  992DFE73-BBC2-493E-A2FF-F5FEADF467E3 NORMAL GetFQDNForServer = wrl-neu-fs-01.wrl.local
0934 0404 02/15 13:48:30.527 71 dscontext.cpp(157) [000000001F804A30] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70368933128041) is 6
0934 0404 02/15 13:48:30.527 71 dscontext.cpp(157) [000000001F803DB0] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70369239887403) is 5
0934 13C4 02/15 13:48:31.758 75 OMUtils.cs(70)  ACD9FF25-9E6E-4EFB-A828-6DE435D33C33 NORMAL GetFQDNForServer = wrl-neu-fs-01.wrl.local
0934 0404 02/15 13:48:32.529 71 dscontext.cpp(157) [000000001F804A30] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70368933128041) is 6
0934 0404 02/15 13:48:32.529 71 dscontext.cpp(157) [000000001F803DB0] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70369239887403) is 5
0934 0404 02/15 13:48:34.541 71 dscontext.cpp(157) [000000001F804A30] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70368933128041) is 6
0934 0404 02/15 13:48:34.541 71 dscontext.cpp(157) [000000001F803DB0] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70369239887403) is 5
0934 137C 02/15 13:48:34.793 75 OMUtils.cs(70)  243A471F-10E2-45F4-BEC6-80EAA57E5DA9 NORMAL GetFQDNForServer = wrl-neu-fs-01.wrl.local
0934 0404 02/15 13:48:36.549 71 dscontext.cpp(157) [000000001F804A30] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70368933128041) is 6
0934 0404 02/15 13:48:36.549 71 dscontext.cpp(157) [000000001F803DB0] 2CA76FB7-DDD8-491B-8653-2DEA735F7857 WARNING Last completed state for Ds Id (70369239887403) is 5
0934 0674 02/15 13:48:37.828 75 OMUtils.cs(70)  7E65A1FF-3FDC-4F00-B5A6-8B66EACAFC9A NORMAL GetFQDNForServer = wrl-neu-fs-01.wrl.local

A check of Azure backup form on MSDN has a couple of threads describing similar behaviour (since Dec), at least from the perspective of slow MARS agent backup performance - but provide non-answers / non-solutions to the issue(s).

Please advise on what might be causing this situation and how it can be resolved.

Regards,

Introduced in MAB Agent update KB 3090067  This new value helps optimize / increase performance for incremental Azure backups performed on file servers that contain a large number of files. This is applicable for Windows server Azure backup, DPM 2012 R2 online backup and Microsoft Azure Server backup.