DPM 2012 SP1 replica inconsistent - datasource is owned by a different DPM server

Recently we took the leap of faith to DPM 2012 SP1 + Update Rollup 1. SP1 offers proper compatibility with SQL 2012 and Server 2012, products we have begun using within our organization  The initial install and management went without a hitch, in fact it went eerily too well.

The follow month however wasn't such smooth sailing, within a few days a number of SQL data sources belonging to two different protection groups began failing with "DPM could not run the backup/recovery job for the data source because it is owned by a different DPM server.". The error description went on to say the "Owner DPM Server: ." claiming "." owned the DPM job.

This was an unusual error to receive as there has only ever been a single DPM server within the organization  so the possibility of another DPM server owning the job was highly unlikely.

The problem in detail

The 5 data sources that were failing were all SharePoint data sources. We are using a Sharepoint 2010 Farm protection group (PG) and backing up any SQL resources that arn't covered in this PG with a simple SQL PG. "Sharepoint_Config" was one of the failing resources, as well as 4 SQL jobs "Application_Registry_Service", "Bdc_Service_DB", "Managed Metadata Service" and "PerformancePoint Service Application".

DPM complained that the "Replica is inconsistent" and attached the following detailed error description:

"The replica of SQL Server 2008 database SERVER\Application_Registry_Service on server.contoso.internal is inconsistent with the protected data source. All protection activies for data source will fail until the replica is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent.
For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106)
DPM could not run the backup/recovery job for the data source because it is owned by a different DPM server.
Data Source: SERVER\Application_Registry_Service
Owner DPM Server: . (ID 3184 Details: The file or directory is corrupted and unreadable (0x80070570))"

DPM suggests to take ownership, I attempted this, re-ran the consistency check and within 5 minutes received the errors messages again.

Our logs were also complaining of communication problems, which we initially put down to network issues, but this theory was quickly debunked as other data sources on the same server were successfully backing up.

FsmBlock.cs(178)        2DE6593E-B086-4002-9205-0A57B65BDC8E    WARNING    Backup.DeltaDataTransferLoop.CommonLoop : RAReadDatasetDelta, StatusReason = Error (StatusCode = -2147023671, ErrorCode = CommunicationProblem, workitem = 70aeaa93-b090-4a3c-bea0-c6fd1a1b4625)

01    TaskExecutor.cs(843)        2DE6593E-B086-4002-9205-0A57B65BDC8E    FATAL    Task stopped (state=Failed, error=RmAgentCommunicationError; -2147023671; WindowsHResult),

We also tried removing the Protect Group and re-adding it, checking SQL permissions, repairing the DPM agent, un-installing and re-installing the Agent, all of which failed.



The Solution

This was a very difficult one to troubleshoot, as SP1 was so new, no one had published details of experiencing similar problems.

Eventually we tracked down the issue to be ActiveOwner problems on the SQL server. The ActiveOwner files are located in "c:\program files\Microsoft Data Protection Manage\dpm\activeowner" on the server hosting the databases (SQL server being backed up). These ActiveOwner are used to manage ownership of databases, important for ensuring multiple DPM servers aren't attempting to backup/restore resources contemporaneously.

After opening the directory and locating the ActiveOwner files for the failing databases, we noticed they were all 0 KB, while healthy ActiveOwner files were 1 KB and contained the name of the owner DPM server.

1. Open  "c:\program files\Microsoft Data Protection Manage\dpm\activeowner" on the database server. 

2. Rename any 0 KB files to <name>.old 

3. Run SetDpmServer from ""c:\program files\Microsoft Data Protection Manage\dpm\bin"Syntax: SetDpmServer.exe -DpmServerName <SERVER>
Replace server with the computer name of your DPM server. 

4. Re-run your synchronization

This fix literally takes 3 minutes, yet it took us an entire week of investigation to come to this conclusion.

Sharepoint error "The security token username and password could not be validated" during claims based authentication configuration

There are many great tutorials on configuring Sharepoint 2010 (SP) to use claims based authentication such as this one by SharepointChick. Claims based authentication is a handy way to pair SP with a lightweight LDAP directory.

The configuration is reasonably complex and determining where an error lies is a balancing act between windows event viewer, LDAP logs and ULS logs.

One such error I received during my SP and OpenLDAP configuration was an event viewer application log.

Source: SharePoint Foundation
Event ID: 8306
Task: Claims Authentication
Error: An exception occured when trying to issue security token: The security token username and password could not be validated..

This error is accompanied with the inability to login to your SP site. A quick google search turned up a number of users in the same boat and plenty of suggestions but no definitive answers.

Clumsy Configuration

When you are working from configuration samples and writing large portions of text based configuration there is always the possibility of data entry errors.

Any of the tutorials you follow, such as this one, will ask you to create a new web application, select claims based authentication and then do some manual configuration in some of the web.config files on your SP site.

This is where the doubt creeps in, during the configuration of a membership provider, depending on the tutorial you are reading, you may be asked to configure a "ASP.net membership provider" with  "useDNAttribute" or "userDNAttribute" or both.

As soon as I removed "userDNAttribute" and set "useDNAttribute" to false, everything worked for me and the security token validation error message disappeared. I am not sure if this is because I am using OpenLDAP or if that makes any difference at all.

The below configuration is an example of my working membership provider syntax.

<membership>
<providers>
      <add name="LdapMember"
         type="Microsoft.Office.Server.Security.LdapMembershipProvider,
 Microsoft.Office.Server, Version=14.0.0.0, Culture=neutral,
 PublicKeyToken=71e9bce111e9429c"
         server="192.168.1.1
         port="389"
         useSSL="false"
         useDNAttribute="false"
         userNameAttribute="cn"
         userContainer="ou=users,dc=contoso,dc=local"
         userObjectClass="inetOrgPerson"
         userFilter="(ObjectClass=inetOrgPerson)"
         scope="Subtree"
         connectionUsername="Cn=sharepointuser,dc=contoso,dc=local"
         connectionPassword="password"
         otherRequiredUserAttributes="title,sn,cn,mail,description"/>
   </providers>
</membership>

Looking back retrospectively it seems like an obvious and easy to fix problem, but when there are so many variables at play it isn't always so easy.

Querying the Sharepoint User Profile Service with Javascript and a content editor web part

So you've gone to the trouble of setting up the Sharepoint User Profile Service (UPS), mapping the attributes you want to use in Sharepoint and possibly troubleshooting some of UPS problems, what's next?

More than likely you want to start using some of these attributes and one of the easiest and most flexible ways to do this is with some Javascript and a content editor web part.



Some things you need to know

The content editor web part is a funny one, and acts differently depending on how you use it. Firstly you can paste Javascript directly into the content editor, but it is much easier to host your script in a .txt file and use the content link functionality to reference it. This gives you much more flexibility in editing, you can use your editor of your choice, instead of the flaky, awkward Sharepoint editor.

While waiting for the UPS XML query to complete there is a slight delay, therefore a simply Javascript document.write operation won't work for the response of the query. Instead we need to use Javascript ID tags in conjunction with syndication to insert the data dynamically into predefined areas of your script.

The actual request is a SOAP XML request, to which I need to credit this thread for the base code.



Enough rambling, how do I do it?

I am assuming you already have the User Profile Service configured and any attributes you want to import ready to go, configuration of UPS is outside the scope of this tutorial.

The below code gives an example of how I generated a Sharepoint entry page. My requirements were to pull a Student ID number from active directory that I could then use to generate URL's and pull data from external databases.

Click HERE for example code

In this simple example I am querying the UPS for a Student ID (stored in the Active Directory department attribute) and the users "Title" to create a welcome page for my Parent Portal.

Example dynamically generated content

Example dynamically generated link with "Student ID"

Most of the code is self explainatory but there are a few changes you will need to make to get started.

1. Please replace the sharepoint.contoso.local in the below line with the address of your Sharepoint UPS.

xmlHttpReq.open('POST', 'http://sharepoint.contoso.local/_vti_bin/UserProfileService.asmx', true);

2. I am pulling the Student ID, stored in the AD "department" field. If you change department in the below line, you can pull any attribute you want, providing you have configured the UPS to poll that attribute.

GetUserPropertyByAccountName(loginName, "department");

The returned value of this attribute will be set as "responsestuid" to which you can manipulate any way you like.


3.  At the top of the file I have multiple Javascript ID's, welcomeheader, linksheader, links. You can create as many IDs as you need and apply HTML to them such as font size and color. You can also place them in the other in which you want them to appear on the page.

For example because I want my "linksheader" above my "links", I position them as follows.

<font size = "4">
<script type="text/javascript" id="linksheader" src="syndication.js">
</script>
</font>

<font size = "2">
<script type="text/javascript" id="links" src="syndication.js">
</script>
</font>

4. Once you have made all the required changes to the example code, save it as "javascript.txt" and either upload it to a Sharepoint Library or any web server.

a. Create a new content editor web part.
b. Edit the web part.
c. Put the link to your "javascript.txt" in the "Content Link" box and press save.

With any luck you will being pulling attributes from UPS and creating dynamic pages based on those attributes!

If you are having troubles getting it working use an alert('error'), and move it further and further down the javascript.txt until it now longer appears when you load your page. This will give you a strong indication of where the error (normally a syntax error) might be in the code.

Troubleshooting the Sharepoint 2010 User Profile Service Application

The Sharepoint 2010 User Profile Service (UPS) application allows the Sharepoint administrator great flexibility and is a "must have" feature if you are taking your Sharepoint to the next level.

Unfortunately UPS has a dark side that too many administrators have to face at one time or another, more often than not it is related to the "Forefront Identity Management Service" and the "Forefront Identity Manager Synchronization Service" service.



Some Tips

• Upgrade to Sharepoint 2010 SP1 and the August 2011 CU at before attempting to resolve problems. Both of these updates resolve a number of issues that might impact the User Profile Service application.
• Don't ever try to change the "Forefront Identity Manager Service" or "Forefront Identity Manager Synchronization Service" settings manually from the Services MMC snap-in. It simply doesn't work as SP needs to do a great deal of configuration.
• Sharepoint can be super slow, sometimes you need to wait 10-15 minutes for things to happen, so when you click start and nothing happens, wait 15 minutes then check again.
• If you are re-creating the User Profile Service application, it is a good idea to use different names for the databases, the service application itself and the application pool. This will ensure there are no conflicts with any old settings that may be floating around in your Sharepoint configuration or registry.
• Be IISRESET "happy". It is a good idea to perform IISRESET's after major parts of the setup process. I normally follow a pattern such as: Start/Create the service, wait 10 minutes, IISRESET, next step. This will ensure all of Sharepoint is "on the same page" before moving forward to the next step of the process.

Possible Issues and Resolutions

Problem:
After starting the "User Profile Synchronization Service" from Central Administration only the "Forefront Identity Manager Synchronization Service" starts, or both services fail to start.

Resolution:
If you don't have too much already setup in the UPS or it is your first time setting it up, it can be much easier to delete the UPS, stop the Synchronization services and then recreate it, than mess around. See my rebuild process below.


Problem:
"Forefront Identity Manager" source logs an Event ID 3 in event logs.

.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: HostId is not registered

Resolution:
Most of the time simply restarting the "Forefront Identity Manage Service" and "Forefront Identity Manager Synchronization Service" from the services MMC snap-in will resolve this issue. If either of them is in a Disabled state or the problem persists after a service restart, then I recommend rebuilding the UPS from scratch as per my instructions below.


Problem:
"ILM Web Service Configuration" source logs an Event ID 234 in event logs.

ILM Certificate could not be created: Cert step 2 could not be created: C:\Program Files\Microsoft Office Servers\14.0\Tools\MakeCert.exe -pe -sr LocalMachine -ss My -a sha1 -n CN="ForefrontIdentityManager" -sky exchange -pe -in "ForefrontIdentityManager" -ir localmachine -is root

Resolution:
If you have tried to provision the "User Profile Synchronization Service" a number of times you might see this error. It occurs because there are multiple "ForefrontIdentityManager" certificates stored in the Certificate store.

Firstly you need to stop the "User Profile Synchronization Service" under Sharepoint Central Administration > System Settings > Manage Services on Server.

Then open an MMC console, add a Certificates snap-in, select Computer Account. Check the Personal, Trusted Root Certification Authorities and Trusted People stores for duplicate "ForefrontIdentityManager" certificates and delete ALL the FIM certificates.

Next under Sharepoint Central Administration > System Settings > Manage Services on Server, Start the "User Profile Synchronization Service" again, you will be prompted for the password of the Sharepoint service account it is using. It should successfully restart and create a new certificate without conflicts.


Problem:
"Microsoft Resource Management Service" source logs a Event ID 0 in event logs.

Service cannot be started. System.InvalidOperationException: Cannot find the X.509 certificate using the following search criteria: StoreName 'My', StoreLocation 'LocalMachine'

Resolution:
The "User Profile Synchronization Service" can't find a certificate when it is starting. You need to stop and then start this service. Firstly you need to stop the "User Profile Synchronization Service" under Sharepoint Central Administration > System Settings > Manage Services on Server. After waiting 5 minutes, perform an IISRESET and then press Start to restart the service. You will be prompted for the password of the Sharepoint service account it using.


Problem:
"Microsoft.ResourceManagement.ServiceHealthSource" source logs an Event ID 2 in event logs.

The Forefront Identity Manager Service could not bind to its endpoints.  This failure prevents clients from communicating with the Web services.

Resolution:
You can try restarting the "Forefront Identity Manage Service" and "Forefront Identity Manager Synchronization Service" from the services MMC snap-in. If this does not work then I recommend rebuilding the UPS from scratch as per my instructions below.


Problem:
"Forefront Identity Manager" source logs an Event ID 3 in event logs.

.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: Cannot open database "DBNAME" requested by the login. The login failed.
Login failed for user 'DOMAIN\service-spsql'

Resolution:
This issue occurs if you have recreated the User Profile Service application and the database settings have not updated in the registry. It is normally a smart idea to stop the "User Profile Synchronization Service" under Sharepoint Central Administration > System Settings > Manage Services on Server, wait 5 minutes, then restart it.

If it is simply a database name wrong you can edit it in the registry under the following paths.

HKLM\system\currentcontrolset\services\FIMService

HKLM\system\currentcontrolset001\services\FIMService

HKLM\system\currentcontrolset002\services\FIMService

HKLM\system\currentcontrolset\services\FIMSynchronizationService

HKLM\system\currentcontrolset001\services\FIMSynchronizationService

HKLM\system\currentcontrolset002\services\FIMSynchronizationService

The main two values you will want to look at are "DatabaseName" and "DatabaseServer", ensure those two are correct then restart the "Forefront Identity Manage Service" and "Forefront Identity Manager Synchronization Service" from the services MMC snap-in.

If this doesn't work, then stopping the "User Profile Synchronization Service"and then restarting it from Central Administrator is your best solution.


Problem:
 "User Profile Service" source logs an Event ID 1511 in event logs. The "event user" will be one of your Sharepoint service accounts.

Windows cannot find the local profile and is logging you on with a temporary profile. Changes you make to this profile will be lost when you log off.

Resolution:
This is a fairly common problem and one of the easier ones to fix. Firstly go into your IIS management console > Application Pools and search for any application pools that have the same username as the event log user. Stop all of those application pools and then run an IISRESET.

Then from the command line run the commands. The first command adds the problem user account to the local administrator group (to aide the profile creation) and the second command creates the user profile.

net localgroup administrators DOMAIN\AppPoolAccount /add
runas /u:DOMAIN\AppPoolAccount /profile cmd

When this process is complete remove the user from the local administrators group.

net localgroup administrators DOMAIN\AppPoolAccount /delete

Then you will need to restart all the IIS Application Pools you previously stopped.


Problem:
After adding a "Synchronization Connector" to the UPS you can no longer get into "Manage User Profiles". When clicking "Manage User Profiles" the web browser simply times out with no errors in the event log or ULS logs.

Resolution:
Unfortunately I am still struggling with this one and have no resolution. On the other hand it seems to make no difference, unless you want to map custom properties, which you can do manually through the Forefront Identity Manager console from the Sharepoint server desktop. While it is annoying to not be able to access, it doesn't seem to have any functional restrictions, in my environment at least.


Recreating the User Profile Service application

1. First we need to get rid of the broken instance of UPS.

a. Under Central Administration > System Settings > Manage services on server, stop the "User Profile Service" and "User Profile Synchronization Service"

b.  Under Central Administration > Application Management > Manage service application, delete the "User Profile Service application"

c.  On the Sharepoint server itself, open an MMC console, add a Certificates snap-in, select Computer Account. Check the Personal, Trusted Root Certification Authorities and Trusted People stores for "ForefrontIdentityManager" certificates and delete ALL the FIM certificates.

d.  Open a Sharepoint 2010 Management Shell, issue the command get-spserviceapplicationpool. Remove any service application pools that are associated with previous UPS applications with the command:

remove-spserviceapplicationpool "PoolName"

e.  Delete any pending timer jobs related to the UPS synchronization service provisioning. Under Central Administration >Monitoring > Check job status, check the Running section for any related jobs and delete them.

f.  Wait 15 minutes and then issue an IISRESET before proceeding to the next step.

2. Under Central Administration > System Settings > Manage services on server, start the "User Profile Service.

3.  Under Central Administration > Application Management > Manage service application, create a new UPS. Use a different name than you did for the previous UPS instance, different database names and a different application pool name. Wait 15 minutes then issue an IISRESET.

4. Under Central Administration > System Settings > Manage services on server, start the User Profile Synchronization service. Wait 15 minutes and if both FIM services are running from an MMC services snap-in as below, issue an IISRESET.

If you get this far and have no problems opening your UPS application from Central Administration > Application Management > Manage service application then congratulations, more than likely you have resolved your problems.

You can now proceed to adding Synchronization connectors and bringing those attributes in from Active Directory.

If you are still having issues getting your UPS to work after following all these steps, then the Sharepoint MSDN forums are a great place to get help from Sharepoint gurus with much more experience than myself.

Enumerating user active directory attributes with Infopath and Sharepoint 2010 SP1

Sharepoint 2010 and Infopath are an extremely powerful team, from IT support request forms, staff leave request forms and the ability to enumerate values from SQL databases, the options are endless.

Recently we began working on a "Parent Portal" which will allow parents of our students to login and view school dates, student marks and timetables. Most of the data we want to pull is stored on our SQL 2008 R2 SP1 servers, so Sharepoint and Infopath are potentially a great match.



The Problem

We quickly realized that all of these external SQL databases didn't reference our users by their Active Directory samAccountName but instead by the students ID number. The solution we came up with was to populate an unused Active Directory attribute with the student ID.

So then when a user logs in, Sharepoint sees the samAccountName then can enumerate the AD attribute (we are using the Department attribute) and use that Student ID to do lookups in SQL.

So how do we enumerate AD attributes with Infopath? Easy!




The Solution


Prerequisites

1. Sharepoint 2010 (preferably with SP1 installed as it fixes a number of User Profile problems).
2. The User Profile Service application needs to be running and a Active Directory synchronization connector created. This AD synchronization connector needs to have completed a full synchronization before you can enumerate any attributes.
3. An AD attribute needs to be pre-populated.


The Process

1. Open Infopath Designer 2010 and create a new Blank form.

2. Click on the "Data" tab and select "Data Connections"

3. Select "Create a new connection to" and select "Recieve Data" then click next.

4. Select "SOAP Web Service" and click next.

5. Enter the URL of the User Profile Service. In my environment it is http://sharepoint.domain.internal/_vti_bin/userprofileservice.asmx

6. Select "GetUserPropertyByAccountName", press next.

7. Leave "tns:AccountName" as is, set the value of "tns:propertyName" to the name of the Active Directory attribute you want to enumerate.

It is best to use something standard like "Department" or else you need to mess around in Forefront Identity Manager and the Sharepoint User Profile Service and believe me its something you don't want to do. For this example I am using "Department" Active Directory attribute which I have pre populated, then click next.

8. Click next on the following screen.


9. You can give this connection a name that means something to you, I will call mine, "sharepointEnumerateDepartment", then click Finish.


10. Now for demonstration we will simply display this "Department" attribute on the screen as a proof of concept. Click on the Fields dialog on the far right hand side of the screen, select "myFields" then click "Add Field".

11. Name it "Test" and click OK.


12. Drag the "test" field onto the form and drop it. This will insert the test field into the form.

13. Right click the test box and select "Text Box Properties" then select the "Data" tab, and the "fX" symbol under the Default Value heading.

14. Select "Insert Field or Group".


15. Click the "Fields" drop down and select "sharepointEnumerateDepartment" or whatever you named this DataConnection in step 9.


16. Please select the "Value" item as in the image below.

myFields > dataFields > tns:GetUserPropertyByAccountNameResponse > GetUserPropertyByAccountNameResult > Values > ValueData > Value

Then click OK, OK, OK.

17. Now it is time to preview the form. Click on the "Home" tab and select "Preview".

If all has worked correctly, you will see the Active Directory attribute you want to enumerate displayed in the Test box above. My attribute is Department and I have set it to my example student ID of 20010.


Completion

Now that we have pulled this value from Active Directory we have so many more options. I can use this ID to pull student marks from our Accelerus database or student timetables from the timetable SQL database. You might want to pull a totally different attribute from AD and display custom SQL content to the end user, this same process can be modified and used to achieve that.






Things to check if its not working

• The Sharepoint User Profile Service application. Ensure it is running and both the "Forefront Identity Manager Service" and "Forefront Identity Manager Synchronization Service" are running on the Sharepoint 2010 server. 

• Ensure the Sharepoint User Profile Service application has done a full synchronization. You can check this in Sharepoint Central Administration > Manage Service Applications > User Profile Service. On the far right side of the page, there should be a number relating to your AD users in the "Number of User Profiles" field and a number relating to the number of Active Directory properties you are synchronizing in the "Number of User Properties" field.

If these are blank. you need to create a new Active Directory Synchronization connector and then start a full profile synchronization.

• Of course ensure the "Department" attribute is populated in Active Directory.

 

Sharepoint User Profile Service application can be extremely difficult to configure correctly, if you have problems I have found the folks on the Sharepoint MSDN forums are extremely helpful.

Sharepoint Config SQL logs growing out of control

Today I woke up, bags packed, ready to go to Taiwan for Computex, but I also found a notification from SCOM telling me the SQL box had nearly filled up its harddrive.

Upon investigating I found the Sharepoint Configuration Log has grown to over 6GB in only a few short months, HUH? Why was this happening? I only have a couple of Sharepoint sites and they arn't even heavily used and I do daily SQL backups via Microsoft DPM 2010, shouldn't the logs be truncated at this point?

Actually no, DPM 2010 won't truncated the logs because the Sharepoint databases are set to full recovery mode by default, which is critical for log shipping and mirroring funtionality that you may need with a large Sharepoint farm deployment. I only have a very small Sharepoint deployment but I still don't want to disable the full recovery functionality at this stage.

Instead I have scripted a weekly log truncation via the following SQL Query script.


SHAREPOINT CONFIGURATION LOG TRUNCATION

USE Sharepoint_Config

GO

            DBCC SHRINKFILE(Sharepoint_Config_log, 1)

    BACKUP LOG Sharepoint_Config WITH TRUNCATE_ONLY

            DBCC SHRINKFILE(Sharepoint_Config_log, 1)

This took a LONG time for my 6GB log, but the result was beautiful, huge reduction on my logs and SCOM notification dismissing themselves. Of course you may want or be required to keep your logs in your organiziation. If you have this requirement you can simply export the log before you truncate it.


SHAREPOINT CONFIGURATION LOG BACKUP

BACKUP LOG [Sharepoint_Config] TO DISK=’D:configLogBackup.bak’



Hopefully one is all resolved now!