1807 hours on Friday, 03/23/2018. By this time on Friday evening I’m usually finishing my first Martini as I cook dinner, and well into “The entire world can kiss my a**” mode, but I got a call at 1730 that the User Acceptance Testing (UAT) site was down, and a question from my technical team leader as to whether or not it mattered, as she “wasn’t sure anyone was planning on working in UAT this weekend”.
“Are you sure that none of the other project teams are working in that environment?” I asked.
“Well, I wouldn’t know…You wouldn’t either, would you? We’re only responsible for our project.”
You know what happened next, right? I dropped mentally back in time to the last command I held:
“Give me a second to see what activity was taking place right before the server went down,” I told her, hearing traffic noise on her end of the line. A quick look at SVN and the deployment logs showed that one of our developers had added some XML and CSS to a new SVN branch and created a new code build, then deployed it to UAT…Then apparently packed his laptop and headed home without testing. I’ll have a discussion with that individual on Monday – But in the meantime, I took the phone off mute and told my technical team leader: ” ________________ from OUR development team created a new build and pushed it to UAT 30 mikes from the time it went down. We own this. It’s Friday night, but I wrote up an SOP (Standard Operating Procedure) for this exact type of situation two weeks ago; SOP ________. We’re not going to have our collective a**es in the wringer come Monday morning, so I’m calling ________________ and having him go back to the office and roll that build back to the previous, working one, regression test it, then I’ll let him go. I want you on the P2 bridge I’m going to open so that you know exactly what’s being done – Once it’s fixed I’ll send a note out to management, again per SOP, describing the issue, the root cause analysis, and the solution. This is going to eat up a couple of hours of our Friday night, but the SOP is approved and signed off, and on file, so that’s how we’re going to do it.
Dismay in her voice she asked “Fickle Green, can’t you just Putty in and roll back the build yourself? It would only take ten minutes, I’m driving, I can’t do it right now.”
“_________________,” I said, “Affirmative. I COULD Putty in and fix it myself in ten minutes, but what would the team learn from that? Any time there’s a problem in off-hours call the Program Manager and he’ll fix it? That’s not the lesson I want to be the takeaway from this dog and pony show, the point is that there’s a reason I’m spending two hours a day writing SOP’s, and that reason is that I expect them to be followed. I hope you’ve got a phone charger in your car, this is going to be a one or two hour exercise, but it will be done RIGHT. I’m opening the P2 bridge and sending the page out now, talk to you on the bridge in five.”
So here I sit, phone on mute, copy of the SOP on the screen in front of me, listening to the team follow it and resolve the issue. I suppose I COULD be drinking a Martini as I listen, but if something else gets screwed up in the process, I want a clear head to deal with it.
So how is everyone else’s Friday evening going?