Battling the Perfect Storm
It’s a rarity in Ireland to have snow; let alone blizzards and meters high snow drifts. As we settled in to our working week, talk abounded of the “Beast from the East”; cold Siberian air currents arriving on our shores on Wednesday. Initially, we shrugged it off and dismissed it. We’ve grown accustomed to orange weather alerts issued from Met Eireann, the Irish Meteorological Service and even soaked in some sunshine Tuesday on the way to lunch.
Wednesday morning however was a much different story. With a large dump of snow overnight and icy roads around Dublin, the usually uneventful trek to work for Monsoon staff was just the opposite. Traction was non-existent with part of my morning drive spent horizontally across two lanes of a main road on a steep hill! Bus, train and tram operators were running limited services with some of our staff getting stuck in various parts of the county. The UK and Ireland, unfortunately is not prepared to handle such weather events as they are such a rarity.
Awaiting us in work was news from Magento of multiple security patches to resolve some critical vulnerabilities, requiring immediate action.
Ensuring the integrity and security of any digital platform is unfortunately now part of the cost of doing business with constantly evolving threats in the wild. The criticality of these applications cannot be underestimated with revenue, business reputation and customer data just some of what is at stake. At Monsoon Consulting, we support applications that collectively generate over €300m per year so this is serious business.
On Wednesday morning, we jumped straight in with a deep analysis of the patches and releases to understand what has changed, how critical the updates were and how we could validate once installed and deployed. Both our support lead and QA lead worked together to create test cases on how to validate functionality affected by the changes.
The weather was deteriorating, with “The Beast from the East” soon to meet “Storm Emma” rising from the south with a forecast of 100km per hour blizzard conditions for Thursday and Friday.
At the same time, our Sysops team were kicking off our business continuity plan and verifying remote working procedures for all staff, with the impending shut down of public transport on Wednesday evening for two full days.
After a team meeting to organise our plan around security patching and with the evening approaching, we broke early to ensure staff could get home and prepare for the impending storm.
It was a surprisingly calm Thursday morning with no one on the roads, and a light snow falling. Our traditional 10am morning stand-up was transformed into a virtual conference with everyone dialled in from the safety of their home. There was a quiet determination from each update with everyone knowing the task ahead. Version upgrades and patches of 20+ enterprise ecommerce applications in 48 hours is no easy task, with variations in implementations, infrastructure, environments all to be tackled and clients to be briefed.
As the day progressed, our update meetings tracked the progress of both security patches and the worsening weather with snow falls getting heavier and heavier across the country. JIRA tickets were gaining traction through the workflow with our QA team bolstered by the arrival of product owners to get releases out. Our general no-release Friday policy meant that we were up against it to get patches delivered and by 9pm, Thursday the majority of patches were installed, tested, verified by customers and deployed to production.
On Thursday evening the blizzard hit dumping vasts amounts of snow, delivering meters high drifts. Friday mornings virtual stand-up was to the backdrop of a blowing gale with everyone snowed in and some without power. Thankfully most of the work was done, with the last application patched just before 11am on Friday.
With only one minor escape defect in production, it was a highly successful security patching exercise under pretty difficult circumstances. The value of teamwork shone through with everyone united in the common goal, while also battling against the storm.
Forecast is clear for Sunday, so as the snow thaws it’ll be business as usual Monday morning.
The perfect storm will be remembered for a long time.
