By Eric Secrist — Senior Technical Consultant
From a systems administrator perspective, change management can be challenging. You are ready to resolve an issue, patch a system, or otherwise improve the state of a server, but you must first submit a change record for approval. In many cases, the change record must meet particular requirements or it’s rejected. Multiple parties may need to approve the change, delaying the implementation. In the meantime, you are watching your server languish, and likely receiving grief from the application and service owners. You know that with a few quick keystrokes, the server could be performing better; it could be configured more robustly, that its redundancy could be improved. You are at the mercy of others, others that may not fully understand the change (or at least not as well as you).
Change Management should not stifle proactivity. Consider a new operating system version that has recently been implemented in a production environment. The new version has already been through the change management process. The SA(s) involved in the implementation painstakingly completed all of the necessary change artifacts. A major change such as implementing a new OS version certainly is not a standard change, so it would have been approved by the CAB. After the new OS version is implemented a flaw is found. Although the SA team agrees that the flaw needs to be corrected, there is a general feeling of reluctance. No one wants to complete all of the necessary change documentation associated with fixing the flaw. For this reason, the correction of the flaw is prioritized a little lower. Other more pressing issues move ahead of this worthy proactive effort. In this example, change management is not reducing business risk as it should be; it’s, in fact, increasing risk, because the flaw remains longer than necessary or is not removed from the production environment at all.
The two situations illustrated above seem inefficient. Most people don’t like red-tape, and change management certainly feels like red-tape bureaucracy.
However, change management is here to stay. It’s likely that change management processes will become even more work-intensive in the future as the need for IT controls increases. The challenge for the SA in today’s change management world is to learn patience, acceptance, and maybe even a little humility. The old days are over when the SA could perhaps take down a system with little or no notice, and make what were deemed necessary changes. The power to make those types of decisions has been removed. Financial scandals such as Enron in 2001 led to the Sarbanes-Oxley act of 2002, which resulted in the widespread adoption of ITIL and its associated change management best practices.
Since change management isn’t going anywhere, businesses have plenty of opportunities (and motivation) to improve the process. A proper change record has associated artifacts that must exist and be reviewed before implementation. Artifacts are items that explain different aspects of the change, such as a detailed statement of the change, a rollback plan, and various testing results. When the quest for “artifact perfection” occurs, the change management process can degrade, resulting in possible rogue changes.
Consider the following statement:
“During the review of artifacts, a well-intentioned drive to perfection may arise. Impractically high standards weaken the change control system. During the implementation of a change management system, it is easy to list requirements and label them all as mandatory. As an analogy, generals easily move pins on a map, but the troops on the ground must go through the formality of “making it happen.” The artifacts required should be as minimal as possible, consistent with the organization’s appetite for risk. More depth is certainly welcome, but if requirements are excessive, the system will break under the day-to-day pressures of production. Reasonable standards persist.” (Yarberry, p. 12, emphasis added)
In short, a change management system should be reasonable, minimal, and practical. Sometimes change management processes are overly detailed, slow, and nit-picky. The auditors must be appeased and the systems must be compliant. Change Management is a step towards that compliance. However, impractical change management processes can create an adversarial environment, making change management burdensome for change requestors and implementers alike.
One method to minimize change management processes is to empower staff who are “closer to the change” with change record approval authority. For example, a system administrator submits a change request to update the kernel on a Linux server. Since kernel updates are somewhat routine, the change can be submitted as a standard change. Standard changes require less documentation because they are less likely to have a significant impact. The criteria for a standard change should be defined in an SOP or another document. Approval for a standard change can be placed with a systems administration lead, which actually understands the technical aspects of the change and has a working relationship with the change requestor. The lead can make an informed decision based on the implementation plan and easily contact the requestor if there are any questions. Instead of subjecting a routine change to a CAB (Change Approval Board) the change can be rubber-stamped and implemented quickly.
Avoiding the CAB whenever possible is a reasonable approach for a systems administrator. Such avoidance should not involve rogue activities. A thorough understanding of how changes are classified can help SA successfully navigate the world of change management. If your client has an SOP or another document where standard, normal, and emergency changes are defined, study the documentation carefully and know it well. If your client does not have such documentation, ask your lead and/or other managers for more information about change management processes. Implementing a change without proper approval, not providing all of the necessary information, or otherwise pushing unapproved changes into the production environment is unethical, and in some cases can lead to the removal of access to the change management system. If you cannot submit change records because your access to the change system has been removed, your value to the client is significantly diminished. You are putting your client at risk when you do not participate fully in your client’s change management processes. Comply with change management processes. When you have questions, ask. Don’t put your client or your personal career at risk.