Pakhnyushchyy - Fotolia
In this edition of The Subnet, we catch up with Sidney Pendelberry, a senior systems administrator/engineer at the Rochester Institute of Technology (RIT) in upstate New York, where he also teaches courses in networking and computer science.
Being able to identify and prevent network threats is an uphill battle for any IT department. But for a university as large as RIT -- where BYOD is the norm and the number of active users runs between 16,000 and 21,000 -- striking a balance between network protection and academic freedom is especially challenging.
With so many users -- and so many tech-savvy users -- what's your biggest challenge these days?
The number of devices on the network has grown tremendously. This is really the first year that a child who is now a student in college has been connected their entire lives, whereas I grew up just as black and white TVs were turning to color.
We used to be able to dictate which computer equipment you had and the apps that are safe. Now in a university, BYOD is the heart and soul of the network. Everybody brings their own device. And we have a class B network, so we have 65,000 IP addresses out for the taking. There is no firewall between you and the world -- no ISP doing any kind of protection. If you are actually on the 129.21 class B, we do provide some filtering to try to prevent the stupid [behavior], but you have to be careful with that because you surely can't encroach on any academic freedom.
We have our share of problems [with network threats] though. We have freshman that are setting up their webpages for the first time, and great, they got Apache to run, but they never turned on their firewalls, so they get pwned pretty quick, and we do our best to shut them down. We've also got students using RIT's network for business. About 3% or 4% of the students make up about 80% of the bandwidth use from 2:00 to 6:00 in the morning, so we do some packet shaping to prevent that.
Additionally, we have the battles of trying to get XP expunged from the network. There's also always a lot of angst over authentication because you have so many different OSes, and they each have their own separate cipher suites. So trying to find the one that will work best usually takes us down into NTLMv2, which is scary. We do our best with it. We've got a whole bunch of old lab equipment that [someone] thought, back in the day, was a great idea to attach to the network -- some of the old projectors and those sorts of things -- and they're hard to defend. Printing is also really hard to defend. I don't know what happened, but big print manufacturers -- whether it's HP, Xerox or what have you -- definitely took a shortcut when it came to security. Network threats are always a struggle. But since we have no firewall, a lot of the devices are subject to endpoint protection. So what we focus on is user education and scanning the network to make sure that we keep the vulnerabilities down to a minimum.
Which do you find more effective of those two to combat network threats: user education or vulnerability scanning?
It depends on what class of user you're talking about. The students do surprisingly well on the security side, especially given the volume of students we have. We have had staff and faculty members that have been pwned two or three times, and that's just frustrating.
We did a napkin calculation a couple years back: What does it cost us when a staff or faculty member responds to a phish? Turns out it's about $14,000 in effort and lost productivity. When you respond to a phish, usually what happens is they'll go in and erase everything in your mailbox. They do that so that when they start spamming out of your mailbox, it doesn't fill up. We have to restore the mailbox and ensure that we don't have any personally identifiable information, because that's a reporting issue if we do. Then we have to beg forgiveness to all of the ISPs that have blocked us for being silly and allowing spamming to occur. Sometimes there's a security investigation that has to go along with that, beyond just the email, if we can't identify the attack vector. All of that can be rather expensive. Thank goodness these [network threats] have been decreasing over time, but we still get them.
So what's on your plate now in terms of networking projects?
The biggest one we're looking at now is energy, and there's a lot of ways that we can use reporting and control our power consumption. We're not interested in monitoring the individual. There are just too many problems with that. We don't have the data governance that would allow us to do that anyway, and most of the students would think it's creepy if we sent them a note saying, ‘We've been watching your activity and you're using too much power.' We try to protect the students, anyway. We don't keep our logs any longer than we have to, and we do the bare minimum amount of logging. If it's required, we do it; if it's not, we don't.
Sidney Pendelberrysenior systems administrator/engineer, RIT
Personally, I'm doing a lot of big data. RIT wants to tie together some of the environmental and building conditions we already monitor -- temperature, humidity and air flow, along with outside conditions, scheduling data and network utilization data -- to give us a more accurate demand model for RIT's cooling and heating needs.
Specifically, the wireless APs' cumulative session data will help us determine the usage in real time instead of the latent response from the temperature sensors you'll see well after people have walked into or left a building. For example, in the mornings, you heat up the buildings. But as people come in, they're little heat and humidity machines, so they're generating heat and expelling humidity. And by the end of the day, you end up having to cool the building. You get this yo-yo effect, so we'd like to be more efficient in our environmental building controls, and we feel the network data has a big play in that because people are connected to their devices, and mobile devices are a great indicator of building utilization.
New technologies like SDN have changed this field dramatically. As an instructor at RIT, how is that influencing your curriculum?
The curriculum is constantly in flux. That's what makes it so difficult -- what was current yesterday will change tomorrow, so the labs constantly have to be rewritten. I used to love to do Windows XP unpatched so I could show the evolution of password cracking, but now not so much. I don't think we even have those OSes available anymore.
The biggest thing we have to do now is configuration management. We spend a lot of time on that, whether it's SCCM, Puppet or Chef. Active Directory is huge, so there's LDAP. In terms of authentication methods, Kerberos is still the big standard; it's been around forever. But then we go into implementing things like Shibboleth and some of the other network-centric authentication out there that are constantly evolving.
The software-defined networking piece is critical. We use it in our own environments. We have a fairly substantial cloud environment as well. We do a lot of networking within the cloud, and it makes sense to do it there. We have a lot of edge appliances as well as switches and routers that are virtual. In some of the labs, the first thing the students are doing is setting up a virtual subnet, and whether they're using pfSense or Vyatta, they basically have their virtual network. But you don't see that until your third or fourth year. Some students will push ahead and pick up those classes in their sophomore years so that when they go on co-op they're much better prepared.
How did you get into IT and specifically networking?
It was an accident. I worked at Xerox as a systems engineer back around 1999 and 2000. Another guy and I decided we were going to start a business: UniteU Technologies. He had just gotten his MBA from Simon Business School at the University of Rochester, and I had just gotten my master's in systems engineering from RIT. So I took the tech side, he took the business side, and we put together UniteU Technologies. It's still in existence, and they probably have 30 or 40 people working for them. It's a point-of-sales integration company.
I had never seen a router before I had to buy one there and install it. We grew the business and I left Xerox, but it was so time-consuming. I was spending 16 hours a day [on it], and I gave up a significant amount of salary to do this. After a while my wife told me that was silly, so I had to go find another job and I fell into RIT.
Last question: What's your biggest non-tech hobby?
I'm an outdoors kind of guy. I do a lot of hiking and backpacking. My kids and I have been working on the Appalachian Trail. We've done a couple states and keep working at it on the weekends, and I like that the best because there are no phones. We leave them at home. When I'm out backpacking with the kids, the only things we have are ourselves and the dog.
Amid constant threat of network breaches, visibility is key
Network security challenges, by industry
How to use DNS monitoring to detect network breaches