In the past month, I have been helping out our customer look into some troubles their user account provisioning jobs have been encountering. One particular job is to take employee data files generated from their SAP system and determine the changes and provisioning the Active Directory domain accordingly. (i.e. update particulars, move OUs, set expiry dates, etc). This job of late has been terminated abruptly every morning as it crosses its five-hour execution limit, meaning it has not successfully completed its run. Catastrophic for the account administrators, as they have to provision user accounts manually upon user complaints.
This account provisioning module is a custom .NET program developed back in 2004 (In other words, .NET Framework 1.1). And naturally, makes use of the System.DirectoryServices namespace. From the error logs, we observed this slow processing was due to repeated "Directory Service is busy" errors that blow up when attempting to update a user's properties.
“The Directory service is Busy”
When I was originally just given this error message alone by my colleage, I searched around the Internet and was rather surprised at just how little information there is regarding this situation. I was equally dismayed when nobody could offer any accurate insight on AD or Directory Services communities. Fearful that we won't be able to quickly determine the actual cause of the errors and be entrenched in the problem for a long time, I did a fundamental thing - asked for a copy of the error log file. With the full details at hand, I began to observe vital information:
- The "Directory service is busy" error always happens at the same line of code. (CommitChanges())
- The block of logic before CommitChanges() deals with unassigning the Manager property of a user, if it is found that a user has no supervisor.
- Matching up with the SAP employee data file, users who are throwing those errors are users with no supervisors.
- Each "Directory service is busy" error occurred around half a minute apart from each other, which suggested why the whole job took so long to run.
The code written to "unassign" a user's manager goes like
updateUser.Properties["manager"].Value = " ";
When I tested such a pattern on my home domain, it threw up the exact "Directory service is busy" error, after a similar half-minute delay. Interesting. On further study of this particular property, it turns out that an AD user's Manager has to be defined in the format of a Distinguished Name (DN). It has to point to another valid user object in the AD store. Giving a single " " space value probably sends the domain controller into a confused state looking for "void" before it gives up thinking it is simply too busy to find the real location. You know, like using a mirror to look into another mirror ad infinitum.
Why doesn't the domain controller return an immediate "Invalid DN for Manager" error message is beyond me.
By adjusting the code to be more respectful of AD conventions, I managed to eliminate the error.
But then everybody would probably be asking, did the previous developers even tested the code before rolling out to production? How could they blindly assign a mere " " space? Indeed we all asked. To cut long story short, it turns out that back in those Windows 2003 RTM days, a domain controller would happily accept a nonsenscial " " space for a user's Manager property. Since the original developers did not encounter any errors with that type of logic, they probably assumed all the world was at peace with each other.
After patching it to at least Service Pack 1, the "Directory service is busy" error comes into play. The latter problem is actually better, though. Because storing an invalid DN value in the Manager property would later induce unsavoury error and behaviour when attempting to edit the user's properties under the Active Directory Users and Computers administrative console. It was essentially corrupting the AD user data.
So if you are witnessing this error message in your ADSI code, take care no funky values are being offered to the domain controller. Do not feed your goldfish junk.