xy - Fotolia

Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

How a cloud issue opened a can of IT shadow worms

When a rogue administrator went undetected at this enterprise, its entire cloud infrastructure was at risk. How can you protect yourself?

Normally I write in the mornings after the kids are off to school. I play hooky from the office and have coffee in hand and my knocked around Mac balanced on my knees.  But, after hearing a particularly disturbing shadow IT story from an attendee at IP Expo in London, I serialized this to disk as I flew back over the Atlantic.

A little background: I'm not one to panic, and I believe that any dedicated IT team with determination, logic and time can solve any issue—we own the wires after all.  But this is an example of a cloud issue gone terribly wrong and needs to be explored while the details are fresh.

We're all too familiar with shadow IT: cowboys in our organizations with just enough technical savvy to be dangerous as they execute outside the authority of IT and sometimes even corporate policy.  How often do you discover YADBA (Yet another Dropbox account), even though every employee signs an agreement not to ship documents offsite through prohibited means?  Fortunately, Dropbox is relatively easy to solve-- provide an acceptable alternative in the form of a secure file-sharing app running in your data center and then use packet inspection, application signature detection and/or NetFlow to the access control list, or shame it out of your organization.

IT death by a thousand credit cards

The story I heard at IP Expo, however, goes way beyond out-of-policy application use, or even sloppy infrastructure as a service (IaaS) instance management. And take heed, because without vigilance, it could happen in organizations of any size. 

In a nutshell, a part-time WebDev or content management system jockey decided he'd found a great way to allow customers to track fulfillment progress on customer orders. He got a purchase order approved and opened an IaaS account with a major provider (rhymes with Blamazon Jeb Dervishes).  He also built out a passable app that even included responsive layout for mobile.  Finally, IT had helpfully set up a VPN to his virtual private cloud (VPC) so he could access data service APIs back on the corporate network. Customers were happy and management was pleased.  The only problem was he was getting airline miles on his credit card. His personal credit card.

You probably just winced slightly reading that last sentence because you know what's coming.

Hit by a bus

The first sign of trouble was not at the admin's going-away happy hour a year later when, freshly promoted, he left to join another company. It was about two months after that, when emails started coming in to the default email queue at webhelp@thecompanyisscrewd.com. The heavily promoted, hyperlinked and globally used tracking site was offline.  At first, there was no panic; websites break and admins fix. Lather, rinse, repeat. But soon, the Web operations team realized the service wasn't coming from its colo, or even from its hybrid cloud instance. It was just an IP address on the Internet, now nothing more than a whiff of vapor from what was once an unknown production system. The whole account, including machine instances, storage, relational database and VPN endpoint had been copied to /null.

In hindsight, this should have been as easy to prevent as kicking Dropbox off the network.

What had happened was the "account owner" removed his personal billing details from the account and sent an email -- a single email -- to a junior IT admin reminding him to get a corporate card set up for the account. The admin did his youthful best, but ran up against the purchase order process and the ticket died in the help desk queue.  The cloud provider let the account run for 60 days, then nuked it from orbit. IT and ultimately legal begged and pleaded with the provider, but the answer was simple and final: The account was the property of an individual, and irrespective of what it was connected to, domain name server (DNS) resolution, or the logo and copyright splashed across Web interfaces, there would be no transfer of ownership.

Missed opportunities to illuminate IT shadows

In hindsight, this should have been as easy to prevent as kicking Dropbox off the network, maybe even easier. In addition to detecting the VPN traffic that clearly showed transactions with the provider, IT had provisioned the VPC VPN and added requested new subdomains to DNS. They failed not because they didn't care, but because they didn't know what they didn't know. They didn't have a service audit process in place to verify ownership of company-owned services in the cloud. They didn't track dev or admin ownership or enforce standard documentation. 

Fortunately, the company was back online in a couple of days because the guy who'd created the environment was (1) available, apologetic and not a jerk; and (2) had some backups running out to the approved storage account IT could service. The organization was incredibly lucky and made some substantial changes after this cloud issue. Managers implemented new audit processes, started scanning traffic to identify unknown service delivery sites and most of all, became much more suspicious of shadow IT. 

And as for me, I know what scans I'm going to run as soon as I get back to Austin.

About the author:
Patrick Hubbard is a head geek and senior technical product marketing manager at SolarWinds. With 20 years of technical expertise and IT customer perspective, his networking management experience includes work with campus, data center, storage networks, VoIP and virtualization, with a focus on application and service delivery in both Fortune 500 companies and startups in high tech, transportation, financial services and telecom industries. He can be reached at Patrick.Hubbard@solarwinds.com.

Next Steps

Controlling identity management

What to do when shadow IT moves to the cloud

Confessions of former IT shadow worker

This was last published in October 2014

Dig Deeper on IP Networking

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.