At a present project we are delivering for a customer, the infrastructure includes a fairly simple set of stand-by servers to be situated at a disaster-recovery site. It is a scaled-down mirror of production environment, with the application database being backed up from the production database server over to the DR database server via means of SQL Server 2005's Log Shipping. Naturally, part of the UAT plans was to carry out DR fail over procedures to prove the viability and readiness of the DR setup.
The entire system we developed is pretty elaborate:
- A main web application for users. The main setup is load-balanced between two web servers.
- A couple of web service applications as well as Windows services spread across the servers. They process their own information and carry out inter-server communication as part of the system's workflow.
- Several SQL Server Agent batch jobs for additional data processing. This includes fetching data from external database servers.
Because of the wide distribution of so many system components on different server types, DR scenarios and fail-over plans can become a complex sequence of actions. As the application developers, we understand the need to document information and steps to help administrators (who are from the IS department of the customer) carry out proper fail-over and recovery of all the application components. Ensuring users have minimum disruption with their use of the system is utmost. Again, let me emphasize that helping the administrators understand the design and nature of the application system is our job. So that they can do their job of supporting the system when disaster strikes. When we have long disengaged.
Now comes the real point of this post. In the documentation of the DR failover guide, I listed down steps of what needs to be done on each server. Like services to stop, SQL Server agent jobs to disable, files to copy or edit. There are even description notes explaning why those need to be done. Some of the administrators commented the guide is not detailed enough. They wanted it down to instructions on how to open up SQL Server Management Studio or Windows Services MMC, drill down to which graphical nodes, click on which menu item or button. To paraphrase them,
A good DR document will list explicit steps A to Z so that any person can carry out the fail-over procedure without knowing what is happening. Just follow the steps and see the expected results. No need for any explanation.
| For starters, I absolutely know what their concern is and can understand that. It is important to jot down as much information as possible so that a possibly unskilled staff who may be called in to carry out the job can still have some chance of deflecting disastrous consequences to a future date. I don't know about you, but my first impression was that I may enter an operation theatre to be serviced by a "surgeon" reading an instruction guidebook step by step while cutting into my body and arteries, removing my organs, and patching it up. All without knowing what he was really doing. Now that is a great surgical guidebook. |
 |
The thing that put me into discomfort is that they can actually consider getting an individual unqualified at administration duties to handle a dire situation of such magnitude. Of course, it will be unfair to label the whole group as clueless, pretentious liars who got their jobs by smoking an equally clueless boss. That is not certainly true for their case; in fact most of them exhibit good competency and sound technical knowledge. However, the key point I am highlighting here is that I am, unfortunately, totally familiar with the mentality of individuals who possess the concept of professional jobs equating to that of reading and executing step by step. as per documentation. That were not prepared by themselves.
They have neither the motivation to know and understand what they are doing, nor why they are doing it. They just want a guide book.
I have seen this type of attitude with past colleagues; I have seen it with many customer engagements. Supposed professional personnel engaged to take care and manage a sufficiently significant environment. Yet having absolutely no interest in learning about the items under their charge.
It is my opinion that hiring such people is a highly dangerous affair. You cannot afford to entrust critical matters like disaster recovery to individuals who have zero awareness of the systems they need to touch. One has to have a damn good grasp of human anatomy and physiology before attempting surgery, so jolly well engage one who has a good understanding of the Windows operating system and Active Directory. System disasters happen out of a great variety of reasons; if you think you can get away without adequate system knowledge and diagnostic skills and referring to just a DR guide, you couldn't be further from the truth. If you want a job where all the documentation and step-by-step guides are pre-written for you to follow blindly, then you are seeking a position of User, not System Administrator or Engineer.
I have witnessed this kind of "risk ventures" get their due deserves before. And undoubtedly will see more in the future. Why? Because they still get hired anyway. Who would have known better? Since the people with the hiring authority rarely has the knowledge and experience to truly grade the competency of their subordinates. This is why the IT industry is so messed up. Folks who refuse to realize their jobs require a whole lot more dedication and responsibility, ultimately sow the seeds of destruction that cause others ultra hurt. Guess who gets called in to clear up the mess.