"The purpose of the change management process is to control the lifecycle of all changes, enabling beneficial changes to be made with minimum disruption to IT services.
The objectives of change management are to:
- Respond to the customer’s changing business requirements while maximizing value and reducing incidents, disruption and re-work.
- Respond to the business and IT requests for change that will align the services with the business needs.
- Ensure that changes are recorded and evaluated, and that authorized changes are prioritized, planned, tested, implemented, documented and reviewed in a controlled manner.
- Ensure that all changes to configuration items are recorded in the configuration management system.
- Optimize overall business risk – it is often correct to minimize business risk, but sometimes it is appropriate to knowingly accept a risk because of the potential benefit."
- Quote from ITIL
Due to the fact that ITIL has not defined, in Version 2011, a definitive change process flow, but instead listed examples of it, we like to recommend a general flow based on our operational experience.
General Change Approval remark:
ITIL is unspecific about the planning state of a change when it comes to approvals. Our experience shows that there are on one side changes of minor nature which may or may not need oversight by a change management team. On the other side there are important, risky and complex changes which absolutely require the full attention of an operational change management team. The change management team must be aware of the planning of such changes and so it is our recommendation that no CAB (Change Advisory Board) approval is granted for changes without a specific plan, including dates, times and teams. Essentially the change must be fully planned prior to approval and release of the change. After approval only minor changes may be applied without revoking the approval.
General Vendor inclusion remark:
As technology implementations get more and more complex we recommend the involvement of key relevant vendors in change validations prior to the approval of a change. There are generally two types of involvements possible:
- Vendor reviews a planned change and provides feedback for improvements to the plan
- Vendors create the change plan for a service provider/customer.
General Run book remark:
For Major, Significant or Special Management Focus Changes we recommend writing a so called Run Book for the change. This is a document which outlines the activities of the change with a time-line overview. Additionally, it outlines at which points decisions are taken like Start, Point of no return, Back out, etc. It also outlines which teams or team members (if these are a unique critical resources) perform which task.
Usually Service Management Tools provide the capability to input all of this data, but it can become very difficult to read and review by upper management, which is required to provide approval and accept any documented risks.
General remark on build and test environments
ITIL defines that regular Change Management also reviews changes for build and test environments. Our recommendation is:
Where there is a specific requirement in build & test environments, organizations should perform regular change management i.e. legal requirements, certification requirements, etc.
Where there is no specific requirement, our recommended best practice is to perform full change management for productive or production related systems/services only. In this case the development and test organization must perform change oversight in these environments themselves.
For production related changes the Change Manager and CAB generally need to request documentation and/or declarations that the change has been successfully tested in build or test environments. Where changes are required on production systems, to enable development to continue their work, a production change is always required.
Example: A test environment requires a new port to be opened on the production firewall. The firewall is in production and therefore opening the port requires a change, although the test environment is non-productive.
Change Management Process Overview
Central Change Advisory Boards (CCAB)
The Central Change Advisory Board is the central authority regarding all changes within the Service Providers organization. It approves all major and critical changes and supervises their implementation. It sets global and company-wide standards for successful changes and their implementation and ensures that those standards are being observed. In an international organization all local business units and subsidiaries need to act with the same standards to ensure a similar quality level globally.
Change Advisory Board (CAB)
The Central Change Advisory Board is supported by local Change Advisory Boards in the local business units which manage standard and less critical changes as part of their daily operations. It operates under the same standards and procedures as the Central Change Advisory Board. The knowledge transfer between CCAB and CAB is ensured by a virtual community of all local CABs and the CCAB. The community also works on the continuous improvement of the agreed change management process and discusses and agrees on necessary adaptations.
ITIL outlines 5 levels of change authorities. We suggest that Level 1 and 2 are combined and that both are owned and organized by the outlined CCAB organization:
- Local Authorization – Level 5
- Change Manager – Level 4
- CAB – Level 3
- CCAB – Level 1 + 2
- Check if RFC is reasonable, feasible and compliant
- Identify correct entry point, which operations unit is responsible.
- Ask for prepared change models.
- Provide information about cause/target of RfC.
- Consider preparation time of the change.
- Identify responsible change manager.
Points to consider:
- Operational teams need to be involved.
- Configuration items /service chain to be altered.
- Customers affected by the change request.
- Avoid requests on short notice.
- A single request might raise several changes.
- Use change models (pre-approved, pre-released).
- Be aware of the risks related to the change; identify and plan counter measures.
- Assign activities to necessary teams (tasking), consider using a 4-eyes-principle.
- Identify change type and risk type.
- Consider mandatory approver and lead times.
- Identify configuration items.
Important change parameter:
- Criticality of the changed configuration item (CI) and service impact will determine the change type.
- Change type and complexity will determine the risk type.
- Major changes might require a hyper-care phase after the change to detect side effects, or previously undetected instabilities, early. During hyper-care, systems are watched with particular focus to detect potential reoccurrence immediately.
Suggested lead times
- Preapproved Changes: No lead time
- CAB: minor changes (depends on organisation, usually between 1- 4 days).
- CCAB: major &significant changes 10 days
40 day announcement in advance for high changes.
Make sure that all important change information is available before entering the next phase
The Service Provider and its vendors/suppliers agree on a joint run book that is mandatory for complex or high risk changes. The accountability lies with the service provider, even though the partners can be engaged to the required level to create or modify the run book,
Verification by partners (strongly recommended for complex or high risk changes):
- Complexity of the changes determines the required lead time to prepare the run book, but in general expect 5 -10 working days (verify your specific contract status with relevant suppliers/partners).
Run book creation by partners (some partners offer this service)
- Agree with your supplier’s preparation timelines (incl. procurement & delivery) and reflect agreed lead times within change planning.
- Accountability cannot be outsourced! The requester of the runbook has full accountability and therefore should verify the created runbook for validity/errors/fit for purpose.
During the planning phase all possible outcomes and effects of the change need to be considered. Risks are to be taken into consideration and actions for risk mitigation should be defined. The enhanced planning phase with risk management and optimized timing is one of the main aspects of Change Management according to the Zero Outage Industry Standard.
Plan the change execution in detail prior Authorisation which means:
- The exact time and resource plan must exist prior request for change approval.
- No updates of planning after the approval except minor updates
- Responsible for the change planning should be the change coordinator and not the change management
- The service provider and his vendors/suppliers should agree on a joint run book that is at least mandatory for complex or high risk changes.
- An adequate backout plan must be defined for at least Significant, Major and Special Focus changes. This Plan must include Go/No Go decision points prior execution
- Point of no return must clearly be defined and no action shall be performed prio a formal Go decision prior the point of no return. Also the responsible Manager need to be defined prior which will take the Go / No Go decision.
- 4-Eyes Principle must be included and visible from the change tasks
Provide a change calendar for all organisational units with a 6 month forecast, where frozen zones, blocked maintenance windows and changes with their criticality are listed. Additionally, there might be a central forecast list of special focus changes.
Additionally a Maintenance Window forecast for the next year should exist at least during Q3 of the current year.
Improved Change Forecast
Implementing those tools and guidelines will ensure resource availability and long term resource planning. It will reduce conflicts within different changes in the same environment and create enhanced management attention for critical and important changes. Additionally, the service provider can enable prioritization of incident solution and security enhancement.
- Change coordinator :
- ensures mandatory approvals.
- Present the changes to mandatory change advisory boards.
- Get Service Delivery Manager/customer approval. If necessary, consider both general commitment and formal approval.
Points to consider:
- Major and significant changes should be reviewed and approved by the CCAB
- Minor changes have to be approved by (regular) the CAB.
- Standard changes do not need any approval.
- Emergency changes and short lead time changes should be approved by Manager on Duty or similar responsibility.
- MoD approval is the default escalation procedure for exceptional requirements (short term, outside business hours) while Change Management should handle everything inside business hours.
- Special focus changes (usually on management request) have to be safeguarded and approved at top management level.
We recommend that the following change types and the corresponding review and approval rules are used for larger organizations:
Special Focus changes are special in every way. There is no formal definition. Selected management is allowed to name a change as “Special Focus” change to ensure the highest attention level for an organization.
The reasons can vary from extremely high risk changes, never been done before, to touching a critical system at an inconvenient time which may result in severe damages or issues if it is going wrong. It may also be simply at the heart of top management through political reasons that nothing shall go wrong with this change or any other reason.
A Special Focus change will be reviewed and safeguarded with highest organizational attention to:
- Focussing on known hot spot areas or areas core to the customer or overall company.
- Risk-oriented safeguarding of changes.
- Extended supplier support in planning & implementation.
- Extended management support (top management).
Authorization for Major Changes with is provided by the Central Change Advisory Board (CCAB), together with the business owner There is always a Safeguarding for Major changes.
Authorization for Significant Changes is provided by the Central Change Advisory Board (CCAB) after the corresponding local CAB has approved the change. There might be a Safeguarding for Significant changes.
Authorization for Minor Changes is provided by the Change Advisory Board (CAB). In large organisations there are usually multiple “local” CAB which approve minor changes within their responsibility.
Standard Changes are pre-authorized and therefore need no CAB approval. We suggest two types of Standard Changes.
- Pre-approved Standard Change
- These Changes are pre-approved, but not pre-released. A local CAB still need to review and release the standard change prior execution. Therefore these changes can not be performed immediately
- Pre-released Standard change
- These changes can be performed immediately, no futher activity is required by a CAB
Change Safeguarding is a detailed review for high risk changes or special focus changes. An organisation must define who is entitled to request special focus changes to be handled as such. After definition, all further calls and organisational items to review the change and gain defined management approvals is done by the assigned Change Manager. The review of such changes is performed by a CCAB with special participants due to additional mandatory top management participation. The entitled requester of the Special Focus change must at least attend the change review.
Initial Safeguarding Call
- First call to review the prepared safeguarding documentation
- Is the run book ok, are all relevant milestones listed, is there a story told in the milestones?
- Are all (management) responsible known and named in the document?
Follow-up Safeguarding Calls
- Optional call to check the settlement of the conditions from the previous call
- Review of the changes made in the change and run book
- Check of the participants invited to Final Safeguarding Call
Final Safeguarding Call
- Final presentation of the change to top management and CCAB
- Final decision for the change (denial/approval) by top management and CCAB
Our recommendation is that a change coordinator should ensure the change implementation.
The coordinator is a person close to the technical/business topic that is being changed
- Pre- and post-health checks
- Ensure that all involved people are available (supplier, customer, ...) prior to start
- Document test results and customer sign-off if needed
- React on issues of test result (e.g. back-out, ...)
- Do not exceed point of no return without MoD involvement
- Secure performance of defined 4-Eyes Principle for critical tasks
- Secure CMDB update
Points to consider
- Only start in a healthy environment
- Stick to the (CAB approved) plan
- In case of critical deviation react and escalate appropriately and early
- Consider a hyper-care phase after change execution when special occurrences have happened
- Perform adequate tests after a performed change back-out to ensure the environment(s) work properly again.
4EP is a core component of error free services and must be implemented within the planning of changes as well as in the Change reviews and execution. This means that important change activities are only performed if a second pair of eyes have reviewed it to secure human errors do not happen. Think about it, if a key DNS change is performed at 03:00 am morning, which, if completed incorrectly, may impact all your communication… wouldn´t it be smarter to ensure the command submitted is absolutely correct being verified by a second independent person prior submission ?
- Review of run book in a 4-eyes-principle (4EP)
- Within the change team
- Together with a supplier
- 4EP tasks need to be planned within the planned change tasks
- Confirmation of the 4EP task have to be given by another colleague than the main implementer
- The additional colleague shall not sit next to the implementer. The person shall only verify the command, activity result, prior submission or after execution to be fully correct and complete. Experience shows that if colleagues sit together in front of the screen, they often make the same mistake
- Mandatory for all major and significant changes
- Critical steps within change implementation need be verified by a second person with similar skills
- The success of each critical step has to be verified with the 4EP implementer
- This is valid for the back-out as well
- Update CI in CMDB
- Closure comments have to provide information about problems, deviations, reduced scope and potentially more
- Transfer results back to project plan
- Define follow-up activities, lessons learned, best practice
- Provisioning of implementation feedback to change requester that his desired functionality is now in operation
Points to consider
- In case of deviation to change planning perform a Post Implementation Review (PIR)
- RfC - Closing must not be done before completion of enhanced monitoring phase