DPMRA not starting on the DPM Server

The following issue I encountered in our own DPM environment.

When checking the DPM console we found that all jobs were stuck. During investigation we found that the the DPMRA service on the DPM server keeps failing to start.The event log shows countless number of the following errors.

Log Name:      System Source:        Service Control Manager Event ID:      7009 Task Category: None Level:         Error Keywords:      Classic Description: A timeout was reached (30000 milliseconds) while waiting for the DPMRA service to connect.

Log Name: System Source: Application Popup Event ID: 26 Task Category: None Level: Informatio Keywords: Classic Description: Application popup: DPMRA.exe - Application Error : The exception unknown software exception (0x8007000e) occurred in the application at location 0xfd74bccd

There are no other event in the eventlog that indicate why the DPMRA service keeps failing. The DPMRA1.errlog logs gives us a better indication of the root cause.

2430 28D4 02/04 09:31:11.131 03 agentcfg.cpp(475) [0000000000A5CC20] NORMAL Cound not find configuration for DPMRA 2430 28D4 02/04 09:31:11.131 29 dpmra.cpp(125) [0000000000A594D0] NORMAL CDPMRA: destructor [0000000000A594D0]

With help of this Technet forum post we found that this error indicates that one of the registry keys for DPM is missing.

did a procmon trace, and found that the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Data Protection Manager\Agent\2.0\Configuration is missing ! Which would co-incide with the agent saying "Element not found"

We checked the registry and the key is indeed missing. As shown in the image below.

 

This is a complex key with a lot of unique information, therefor the best way to fix this issue is by restoring the key. In our case we have a System State back-up available of the server (created at the time the server was working fine).

The steps below show you how to restore the registry key from a System state back-up (VHD). This procedure uses Windows 2008 R2 server. Other OS versions involve different steps

Windows 2012 Hyper-V VMs enters a paused state or a CSV volume goes offline during back-up

A serious issue when backing up CSV 2.0 volumes. Host-based backups could bring down most of the VMs on the cluster. This can be a serious show stopper if you plan to migrate to Windows 2012 Hyper-V

When creating a back-up of a Windows 2012 Hyper-V CSV  volume you might get the following error in your cluster log.

Event id: 5120 Cluster Shared Volume xxxx is no longer available on this node because of 'STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR(c0130021)'. All I/O will temporarily be queued until a path to the volume is reestablished.

• You enable the Cluster Shared Volumes (CSV) feature on a Windows Server 2012-based failover cluster.
• You create a virtual machine on a CSV volume on a cluster node.
• You start the virtual machine.
• You try to create a backup of the virtual machine on the CSV volume by using Microsoft System Center Data Protection Manager (DPM).

In this scenario, one of the following issues occurs:

• The backup is created, and the virtual machine enters a paused state.
• The CSV volume goes offline. Therefore, the virtual machine goes offline, and the backup is not created.
• VMs stop responding while the backups runs

There is a hotfix available from Microsoft unfortunately it looks like this doesn’t solve the issue. In addition after applying this hotfix there seemed to be a severe memory leak on the node storage owner.

More info and a workaround available on the Technet Forum thread.

More information also available the Microsoft support article: Issues that occur after you deploy System Center 2012 DPM SP1.

The DPM 2012 database grows and the console crashes after installing Update Rollup 3

 

After installing Update Rollup 3 for System Center 2012 Data Protection Manager you notice that the DPM database grows and eventually the DPM console will crash.

On one of the days when this didn't happen, the dpm system becomes unresponsive around 2-3am and all the jobs just pile up. By restarting the server or killing the DPM service will correct the blockage and as above the Sharepoint catalogue fails to complete. If I run the powershell to create the Catalogue nanually, the server again becomes unresponsive. This has all happen after Rollup 3 was installed. It is very clear that when DPM runs the Sharepoint Catalogue job, something starts to go wrong.

This is fixed in SP1 but there’s an alternate workaround here:

KB2809773 - The Data Protection Manager database grows and the console crashes after installing Update Rollup 3 (http://support.microsoft.com/kb/2809773)

forum link

Update Rollup 1 for System Center 2012 (DPM) Service Pack 1

 

Here is what is fixed in this roll-up release

Issue 1
Client backups fail when there is a case difference between the client computer name on the computer and the client computer name that is stored in Active Directory.
Issue 2
The Administrator Console may stop working when a user starts the Create Protection Group Wizard on a non-English or a non-Russian version of Data Protection Manager.
Issue 3
The Agent Management Wizard may stop working on the Spanish version of Data Protection Manager.

Data Protection Manager (KB2802095)

System Center Data Protection Manager CSV Serialization Tool

This tool is mostly interesting for people trying to back-up Windows 2008 R2 CSV.

The back-up of Windows Server 2008 R2 CSV van be tricky. It is recommended to use a hardware snapshot provider. If there is no hardware provider available then you need to switch to serial CSV back-ups.

This process creates an XML file that is read by DPM.  This works well when you have only a couple of Hyper-V servers to manage.  But what if you had lots of clusters backed up by a single DPM server?  With the current approach, if there are multiple clusters managed by DPM, the user needs to run DSConfig.ps1 on one node of each cluster and then XMLs of each cluster needs to be merged and replace the DataSources.xml

Microsoft has released the System Center Data Protection Manager CSV Serialization Tool to help you in this scenario.  This tool is intended to be used when backing up Windows Server 2008 R2 Hyper-V clusters with one or more CSVs using DPM 2010 with QFE 3 and above or DPM 2012.

Read more at Aidan Finn, IT Pro

System Center 2012 SP1 DPM documentation

The following documentation is available for the new release of Data Protection Manager.

Title	Size	Link
SC2012_DPM_Cmdlets.doc	3.6 MB	DOWNLOAD
SC2012_DPM_Cmdlets.pdf	3.6 MB	DOWNLOAD
SC2012_DPM_CompleteDocumentation.docx	1.2 MB	DOWNLOAD
SC2012_DPM_CompleteDocumentation.pdf	5.7 MB	DOWNLOAD
SC2012_DPM_Deployment.docx	155 KB	DOWNLOAD
SC2012_DPM_Deployment.pdf	778 KB	DOWNLOAD
SC2012_DPM_GettingStarted.docx	134 KB	DOWNLOAD
SC2012_DPM_GettingStarted.pdf	407 KB	DOWNLOAD
SC2012_DPM_Operations.docx	929 KB	DOWNLOAD
SC2012_DPM_Operations.pdf	4.4 MB	DOWNLOAD
SC2012_DPM_Planning.docx	181 KB	DOWNLOAD
SC2012_DPM_Planning.pdf	810 KB	DOWNLOAD
SC2012_DPM_Troubleshoot.docx	475 KB	DOWNLOAD
SC2012_DPM_Troubleshoot.pdf	5.2 MB	DOWNLOAD

Plenty to read for some hours…

HOTFIX: Hotfix rollup package 7 for System Center Data Protection Manager 2010

This article describes the issues in Microsoft System Center Data Protection Manager (DPM) 2010 that are fixed in hotfix rollup package 7 for System Center DPM 2010.
This hotfix rollup resolves the following issues:

• When a Microsoft SharePoint site collection name contains a space, SharePoint item-level recovery fails in System Center Data Protection Manager 2010.
  For more information about this issue, click the following article number to view the article in the Microsoft Knowledge Base:

  2755678 SharePoint item-level recovery fails when the site collection name contains a space in System Center Data Protection Manager 2010

• After you rename a SharePoint site in System Center Data Protection Manager 2010, you cannot restore the site.
  For more information about this issue, click the following article number to view the article in the Microsoft Knowledge Base:

  2763004 System Center Data Protection Manager 2010 is unable to restore SharePoint after renaming a site

• In System Center Data Protection Manager 2010, the SharePoint Recovery Point Status report displays incorrect data.
  For more information about this issue, click the following article number to view the article in the Microsoft Knowledge Base:

  2763010 System Center Data Protection Manager 2010 SharePoint Recovery Point Status Report shows stale data

Source and Download link