Last week I endured the day from hell! Heed this warning otherwise it will happen to you.
Picture the scene. You’ve successfully migrated 16,000 users across the globe from Skype for Business to Microsoft Teams after months of painstaking voice infrastructure deployments, number porting, and a truck load of user adoption and change management. The losing service provider now wants to decommission Skype for Business.
You have a meeting and I called out that removing the Skype for Business schema is probably a bad idea since all user’s phone numbers are managed by the msRTCSIP-Line attribute as they are migrated accounts.
I even cited the article I wrote a few weeks back to them: https://online.commsverse.com/before-you-uninstall-skype-for-business/
After a lot of persuasion due to lack of clarification from Microsoft I convinced them to see caution and not remove the Schema until such time as we have successfully moved people’s phone number control over to Teams.
Script to Migrate LineURIs to Teams
I had been working on a script that would collect people’s LineURIs from the msRTCSIP-Line attribute, clear the offending attribute and reapply the lineuri directly into Teams releasing the dependance to this attribute and thus allowing for the schema to be removed.
Meetings with the customer I highlighted that this process would lead to an outage period for the affected users because clearing, waiting for AzureAD sync etc. would mean they temporarily lose dial capabilities to the PSTN. With all good due diligence I highlighted the importance of a test before arranging change windows with each site.
Testing the Script
The day of the test came and I had 10 test accounts to do my worst. Due to lack of pre-production environment, this would be a test on production infrastructure. Procedures were placed and script double checked to make sure that nothing untoward should happen.
12pm came along and testing began. As expected the script worked and performed properly.
Then Came the dreaded phone call…
3:15pm a call from the customer. “Mark. A site has reported they have lost calling, I’ve checked and their LineURIs are missing”
You can imagine my dread. I think I aged 25 years in 25 seconds. How was this possible. My script outputted every line, only 10 accounts where affected. I asked if there was any work that had been done that could cause this as well. I was told that the losing service provider had removed Skype servers from the topology around 2-3pm that day.
An export of AD showed that EVERYONE had their msRTC* attributes wiped, not only the LineURI, but primary user address, Home server etc. All which we would expect to remain even for hybrid accounts. The last changed time on the objects showed all this happened between 14:10 and 14:40, so I breathed a heavy sigh of relief that it wasn’t me.
Root Cause
In order to remove a front end pool from the Skype for Business topology all active user accounts must either be moved to another pool or disabled. We knew there was one account left on-prem which needed to be removed and they hypothesis was that the engineer tasked to remove the topology ran Get-CsUser | Disable-CsUser to remove that account.
This command is massively destructive in that it will collected every user in AD with msRTCSIP-UserEnabled attribute set to TRUE and pipe them into the Disable-CsUser command.
Migrated accounts even in Teams Only will still have this attribute set on AD.
Disable-CsUser will clear all msRTCSIP* attributes for all enabled users…
Whilst Teams doesn’t pay attention to the enabled attribute on-prem, it does for phone numbers applied to Direct Routing users as that is where the -OnPremLineURI is inherited from.
This single action took out every user, meeting room, common area phone, call queue and auto attendant across the globe for the customer.
What should have been done is a Disable-CsUser that was targeted specifically to the account on the pool preventing removal using the -Identity parameter.
Fixing it…
At this stage the only option left it to reapply the Line URIs to all user accounts. It was by sheer good planning that we had regular outputs from Get-CsUser throughout our migration so that we could quickly use those as a reference point for all Line URIs assigned.
Still, even though we managed to reapply lineuris directly into Teams thousands of users within 2 hours, it still took over 10 hours for Teams BVD backend to re-provision call routing for them.
Learning from it
- Always have a backup of your user phone number assignments and policies
- Make sure that before you remove the topology you understand the risk and the impact of making mistakes
- Always have a backup of your user phone number assignments and policies
Had we not had a backup then the customer would have been in serious trouble as there would have been no way to recover assigned phone numbers. This would have been catastrophic and a very extended outage whilst people run around trying to find numbers of people and services manually.
What does Microsoft say on this?
Fortunately, (but unfortunately for my case as it came too late) Microsoft have now published guidance on disabling hybrid post migration.
https://docs.microsoft.com/en-us/skypeforbusiness/hybrid/cloud-consolidation-disabling-hybrid
Note the big BLUE Important Box!
I can confirm that this does cause an outage 🙂 if you remove the schema or clear the attributes.
If you continue with the schema you need to remember that any phone number changes will need to be done On-Prem using AD Attribute Editor because your on-prem SfB Powershell will no longer work.
If you decide to move the management to Teams so that you can run Set-CsUser -Identity [email protected] -OnPremLineURI commandlet without reliance on on-prem, then you can, but it is not recommended by Microsoft.
If you do, be prepared for an outage for users. Our analysis showed that the outage in small, controlled changes where between 1 and 2 hours. So never do this in a big bang like we were forced to because that can be 10+ hours.
Where is the Script?
Due to the potential outage and impact of this and that it is directly in contradiction with recommended Microsoft guidelines, I will not be releasing it for public use. I do not want the lawsuits.
Heed this warning.
