We were doing some routine checks of a newly provisioned Lync 2013 front end fail-over pool when we discovered a CMS error on the backup pool in the event logs. Our deployment is configured with two standard edition front end pools with resiliency.
The error is as follows:
Event ID 4082
LS Backup Service
Microsoft Lync Server 2013, Backup Service central management backup module failed to complete import operation.
Configurations:
Backup Module Identity:CentralMgmt.CMSMaster
Working Directory path:\\lync002.contoso.info\lyncstore\1-BackupService-20\BackupStore\Temp
Local File Store Unc path:\\lync002.contoso.info\lyncstore\1-BackupService-20\BackupStore
Remote File Store Unc path:\\lync002.contoso.info\lyncstore\1-BackupService-21\BackupStore
Additional Message:
Exception: Microsoft.Rtc.BackupService.ModuleUnavailableException: Backup module is temporarily unavailable at this point. Reason: Instance (local)\rtc points to a CMS store that is being moved to a different pool.
at Microsoft.Rtc.BackupService.BackupModules.CentralMgmtBackupModule.CheckModuleAvailability(Nullable`1 primaryPool)
at Microsoft.Rtc.BackupService.BackupModules.CentralMgmtBackupModule.GetBackupCookie()
at Microsoft.Rtc.BackupService.BackupModuleHandler.ReceiveBackupDataTask.GetBackupCookie(Boolean& isModuleInitialized)
at Microsoft.Rtc.BackupService.BackupModuleHandler.ReceiveBackupDataTask.InternalExecute()
at Microsoft.Rtc.Common.TaskManager`1.ExecuteTask(Object state)
Cause: Either network or permission issues. Please look through the exception details for more information.
Resolution:
Resolution
This error message was repeated every 15 minutes as the backup jobs were run. A separate error was created for the import and export process.
During our testing phase we did some fail-over testing which included moving the CMS database to the fail-over node. We believe this may have caused this error to occur. While it was not causing any known issues in the environment, it may come to a head should we ever need to fail over to our resiliency node.
The Resolution
The resolution was fairly simple but did require a few attempts before we got the process order correct.
1. Open the Lync Topology Builder and remove the associated backup pool from the resiliency tab of the primary front end pool.
2. Publish the topology
3. Use the Get-CsManagementStoreReplicationStatus powershell applet to verify the topology has replicated to the associated backup pool.
4. Open a command prompt to "%ProgramFiles
5. Remove the CMS database from the backup pool using the following powershell command.
Uninstall-CsDatabase -CentralManagementDatabase -SqlServerFqdn lync002.contoso.info
6. Go back to the Topology Builder and re-add the associated backup pool to the primary front end pool.
7. Publish the topology
8. Wait for the topology to replicate
9. Re-run the bootstrapper on the backup pool node
This 5 minute process fixed the error messages for us and hopefully saves us in the unfortunate even of a fail-over.
I wanted to add something to this as my removal kept failing. Had to add -SqlInstanceName RTC to the end of your command.
ReplyDeleteCorrect, if you are using a named instance you will need to specify the instance name, however if you are using the default then you do not need to specify it.
DeleteI had this issue in a SfB Server 2015 environment, and tried the steps above, but got a 1603 fatal error on Step 9. MgmtServer.msi and BackupService.msi were not installing because of a version mismatch. I found that bootstrapper.exe was trying to run a Lync Server 2013 version of the above files from "C:\ProgramData\Microsoft\Skype for Business Server\Deployment\cache\6.0.9319.0\setup". I should state here that the server was previously in-place upgraded from Lync Server 2013 to SfB Server 2015. I tried copying the SfB Server 2015 versions of the files from the original source media to the directory above, but it still failed, and the files in that directory were rolled back to the Lync Server 2013 version. Only when I updated the files in the original Lync Server 2013 installation media that was on the server did it succeed! It clearly remembers the original Lync Server 2013 installation media location. Also after implementing the entire fix in the article above, you will end up with redundant (previous) BackupService and CentralMgmt folders in the Skype file shares. You can find out what the current ones are by running "Get-CsService -BackupServer" and "Get-CsService -CentralManagement"
ReplyDelete