I work in IT for a public university that prides itself on delivering an exceptional value for our students and making education accessible to people in our region. It’s one of the key reasons I believe in the university I work for. Obviously, with all else being equal, to offer an exceptional value you need to have lower costs than the competition. One of the ways we do this is by being leaner staffed than comparable universities. This is true across the board - but this being a tech blog, I’m mostly interested in IT which is no different. We staff IT very lean; in many cases we have half of the IT staff to student ratio than comparable universities or less.
I petitioned to get this sign put over the conference room we use, but it got shot down pretty hard
We are proud that we run lean-and-mean. We stay focused and hit goals that keep us ahead of the curve in many areas, and we do it with far lower costs than you’d usually expect to see. However - like any major research institution - we found a few years ago that our portfolio of applications and supporting infrastructure continued to get more complex with each passing month, and the interdependencies between different stacks were only getting more likely to cause downstream impacts when changes were not coordinated properly. This clearly presented an opportunity to introduce some additional process and structure in the form of change management.
The first challenge we ran into was one that is likely familiar to anyone who works in a large enough IT shop to have been through a formal change management program – the immediate pushback from the folks “doing the work” that such a structure would slow them down, possibly so significantly as to be a detriment to the larger IT team’s overall goals. However, the bigger issue we realized very quickly that our “lean-and-mean” structure was ill-suited to traditional change management programs. After talking to several other institutions both in and outside of education, we heard horror stories of full day length Change Advisory [or Approval] Board (CAB) meetings and multiple stand-up reviews and presentations. The thought of pulling multiple managers and individual contributors off of their responsibilities for a full day wasn’t something any of us in IT leadership found tenable.
We had some outside help in the form of a very knowledgeable consultant who had done some work with educational ITSM and ITIL implementations, and he assured us that the extra structure would be beneficial in the long run. I think we honestly believed him, but we also felt like there should be a better way that reduced friction and achieved the overall goals of stability and coordination. We stepped back a bit and formed a Change Management “tiger team” that was small and focused, because I think we all realized that getting into multiple round table conversations with the entire IT staff would have been both fruitless and frankly, pretty annoying or boring for almost everyone there (depending on their role).
Our first action was to figure out what our actual objectives for a change management process were, and we came up with a few overarching criteria that we called “the imperatives” – a few key characteristics that our end result had to deliver on in order for the project to be considered successful. They were:
Coordination
We wanted everyone in IT to make sure they were aware of the upcoming forward schedule of changes, and be aware of changes to components their applications or services may be dependent upon.Efficient
We thought it was essential that whatever process we came up with didn’t need to consume anyone - let alone an entire group for more than an hour or two per week at most. We also didn’t want the process to inhibit simple, low-risk changes or changes to a configuration item that would be so isolated as to be out of scope.Understandable
We absolutely did not want something that would take anyone months to understand. It needed to be something you could explain in a 10 minute all-hands meeting and everyone would pretty much get it.Equitable
We wanted to make sure that no one felt left out of the decision-making process, and that every working group within IT had an opportunity to be heard and provide feedback.Flexible
We had heard a lot of horror stories about how processes were so rigid as to cause problems when something new came up that didn’t fit into one of the categories that the CAB group had defined. We wanted to make sure our process could be changed to accommodate future needs.
With those overarching goals, we started thinking through different models and we kept ending up at the same problem – that a CAB review seemed to be unavoidable. How on earth could we achieve all of the transparency and coordination goals without convening a large meeting on a regular basis and making sure everyone was on the same page. I’ll admit that this was something we chewed on for a while until I had a bit of a epiphany one day. I was looking through Twitter (where you can find me @steveathanas) and realized that there was some moderately nuanced conversation going on – but it was happening totally asynchronously. It dawned on me that such a thing could be possible with a CAB review; it didn’t have to be done all at the same time.
Once we got past that logjam, we quickly got to something that we feel is a really solid process and it’s been working very well for us for over three years now. I’ll only detail the high-level points here, but we’re very happy to open source this so if you have any interest just drop me an email or note and I’ll make sure to give you as many details as you want.
Each Request for Change (RFC) is entered into a (admittedly rudimentary) workflow system that manages the approval process. Each RFC has to include the full process, documentation, rollback plan, timing, and any other pertinent details. From there, it’s routed around IT for review and approval. We ultimately settled on having two distinct levels of review, one that is generally for technical fit and readiness and another for business or organizational fit – though either group can comment on either topic.
We don’t give out stickers for voting on an RFC… but maybe we should!
The first level of review for every RFC is done by peers within each working team in IT. For instance, if the Systems Engineering group submits an RFC, the workflow app emails the details for review to the Network Services group, the Applications team, the InfoSec group, and so forth. Each of these working groups have nominated two people to be their representatives to “speak” on behalf of the team – and they can (and do) bring up specific RFCs in their individual team meetings if there’s something they want to crowdsource feedback on. Each of these representatives can “vote” on whether they are supportive of the change right from the email they are sent via an included link. If they have no issues with the change, they simply need to notate that they are supportive. If they are not supportive they can say as much, but they must leave an explanation of why they are not - both as feedback to the requestor and as a notice of dissent to the CAB.
Once all of the groups have weighed in, the RFC is reviewed by the formal CAB, which I have had the responsibility to chair since it’s inception. (Note to self: work on a revolving chairperson approach.) There are only three CAB members, and each review the RFC and any associated feedback prior to meeting. I will say that my colleagues and I cheat and one of the more enterprising managers on my team consolidates the information for us a day before CAB. If we had a better workflow system, it would do it for us. Anyways, we review this information which includes the feedback and votes from the peer reviews. We then get together for an hour once per week and vote on the changes. We occasionally – though infrequently – have the requester come to CAB to explain certain aspects of the change, especially if there is dissent or notes of non-support from peer reviews. Generally, we find that those issues are resolved by the time they get in the room because the requester saw the concern and addressed it either in the RFC or with the team raising the concern before we meet and the issues have been worked out. Most of the time we just discuss the timing of the change to make sure we all agree it’s the right schedule to support the university’s mission and then vote as a group to formally approve it.
After CAB approves the RFC it becomes a scheduled change and it goes into the Forward Schedule of Changes (FSC). In order to get everyone to think about change management frequently, we actually got a few large screen TVs that we put around our office suite that continuously display the FSC for everyone to see. It’s a nice little webapp that we wrote in-house that rotates through each of the upcoming weeks along with some on-campus and world news and local weather, as we found that people pay more attention when you throw in some other non-IT information. (2020 side-note: In all fairness, we probably should have switched to email digests now that no one is in the office with any regularity – though last time I was in the TV was still scrolling the FSC and news.)
Now, since I stressed efficiency - I should be clear that this process has some hard-and-fast deadlines that need to be met. All RFCs must be submitted by noon on Monday. All peer reviews need to be completed by Tuesday at close of business. CAB meets on Thursday mornings so that the CAB members have a chance to review RFCs on Wednesday. After a change is completed, we occasionally do a debrief - especially if we think that it could have been done better, more efficiently, or could be a candidate to be pre-approved in the future. These debriefs take about ten minutes. In general, the entire process takes about 15 minutes from the requester, about 3 from a peer reviewer, and maybe 10 from a CAB member, and is transacted over the course of a single workweek start to finish. Of course, if something needs to be addressed more quickly than that, we do have lightweight and separate options for expedited and emergency changes.
I’m not saying an empty conference room is a productive conference room, but…
What I think makes this process work is that everyone feels like they have some ownership of the changes that are happening, and it doesn’t take up much of anyone’s precious calendar – because like I said at the beginning, pretty much everyone in IT is pulling double duty continuously. Since we implemented this, we have found that buy-in has become almost universal (there’s always one or two conscientious objectors) and that it has actually improved our speed of innovation and changes because we have a system to keep everyone coordinated.
It may sound counterintuitive, but you can – and should – balance the needs for operational efficiency with controls that keep things stable, secure, and auditable. I feel like we’ve struck a great balance, and I think you probably can, too.
Some questions for reflection:
Have you or anyone at your organization ever broken something because you forgot to notify a stakeholder that a change was coming?
Does your company have a formal change management process? How do you feel about it?
What’s more important to you or your company: efficiency or control? Have you balanced them?