To deliver server-based slot machines to its newest casino, Station Casinos had to build and manage an IP network 10 times as dense as any network it had built before.
Gaming vendors are starting to transition to server-based slot machine games, according to Van Baltz, CTO of Station Casinos. And with that transition come new demands on the network.
"What happens is a slot machine ends up being a PC, and if you want to change the theme or the game, it's all controlled centrally," Baltz said. To make changes to older slot games, game technicians have to visit each machine. "[Games] are constantly being shuffled, so it's quite labor intensive and disrupting to the customers. Going to this centralized deployment of server-based games changes that, so you can do a change in the middle of the night when the floor is slow. To do that, you obviously need to have a bigger pipe."
Server-based games are a new model for the casino industry. Most slot machines on the market today have a single game written into the physical machine. These games use their network connections simply to report on activity.
"It's a new design for us," said Bryan Miles, director of networking for Station Casinos. "We're going forward with server-based gaming, and the first leap into that was having games with IP addresses. [In other casinos] we probably would have a couple hundred ports because the games were connected on a serial. Now, each game has its own network connection, and we had to expand the number of ports."
Building a very large network
Miles said a typical Station casino has about 40 to 60 switches. The Aliante Station Casino & Hotel in Las Vegas, which opened in November, has 500 network switches on site, including 400 just for the casino floor and its 2,400 server-based slot machines. The casino runs gigabit Ethernet (GbE) to each game. Each bank of six games has its own switch, a Cisco 3560, and each 3560 has a fiber connection to two closet switches, Cisco 6500s. The closet switches connect to core Juniper routers and switches in the casino's data center. And the data center has dual connections to a 10 GbE private dark fiber ring, which connects the 18 casinos that Station owns in Las Vegas.
In addition to the dense casino network, Aliante is also Station's first casino to have a 100% IP telephony system for both employees and guests. The casino has a Cisco Call Manager system that supports 1,000 phone lines.
Miles also deployed a distributed antenna system (DAS) to boost the casino's internal radio communications, its wireless LAN (serving both employees and customers), and the signals of the four major wireless carriers. Wireless IP phones in particular had to be able to run on the WLAN, and signal coverage was critical.
"One of the install tests I did after the phone went online and on the IP network was to take a wireless phone and start a call in the data center on the east side of the property down in the basement," Miles said. "I walked all the way to the west side of the property, got on an elevator, went up to the ninth floor, got out and went to the end of that floor, came back down the elevator and out to the pool. Then I criss-crossed … the casino floor and [went] back down to the data center. We were able to do that on one call."
Network management challenges
Deploying a network this large presented a few major management challenges to Miles and Baltz. The sheer number of switches on one property was more than the company had ever dealt with before. In addition to this, Aliante was also Station's first property to use IP telephony, adding more management complexity. Even the casino's environmental controls were run over the IP network. But the goal for Station's IT organization, which is quite lean, was to have just one full-time person on site at the casino.
"Now your requirements for up-time are much higher than they would be in a normal casino," Miles said. "So just keeping [the network] up, trying to make sure all the operating systems are at the same level, keeping track of ports when they go up and down are challenges. The slot technicians are their own group, so we need to keep track of when they're moving things around, and we need to make things easier for them so they don't need to involve us all the time when they move stuff around."
Deploying networking management and monitoring technology
To manage and monitor all of these complicated network elements, Station has deployed two management products from CA Inc.: eHealth Network Performance Manager and Spectrum Network Fault Manager.
Station is using Spectrum for basic network management such as fault detection, Miles said. He is also doing some configuration management with it, but he plans to expand his deployment of the product to make it a complete configuration management tool.
He is using eHealth to do network trending, particularly capacity planning and trend analysis.
"As more and more features get used on the slot floor, it will need more bandwidth," Miles said. "I want to make sure we stay ahead of that curve, so we're making sure we're upgrading links as we need to."
Station also uses eHealth to detect network anomalies. "We run a daily report in there, but we've also taken that a little further so that it can tell us about anomalies every five minutes," he said. "We can see spikes that help us with capacity trending and analysis. But we can also see where spikes are that [might not require action]. For instance, say you grew by 20% today, and your calculation for how many days before you use up that pipe is out of whack if you rely on trending. But if you look at this other report [on anomalies], you can see it was just a spike for that day and it was just a promotion, or maybe someone was downloading a large amount of files. We can catch that as they come along."
Critical network failure strikes early
Although the new casino has been open for only a few months, its network management technologies were put to the test at the very beginning.
"Right before we came online, we were in the middle of bringing up the 10 gigabit fiber ring when we had an outage hit us," Miles said. "We were able to use fault isolation in Spectrum to figure out exactly where the problem was and get it fixed right away. Given that the gear was new, had we not had that tool in place, it would have taken us quite a bit longer to figure out what was broken."
The dual fiber ring was set up with reverse entry into the casino, so there were two entry points coming into both the east and west sides of the casino. Miles said his team took down one side of the ring to do some work on it. When they did this, the other side of the ring went down.
"The first thing you think when that happens with new gear is, 'What did I do?' " Miles said. "As we pulled up Spectrum and took a hard look at it, we saw that it was actually that a module on a switch was bad. We got it replaced and we were on our way. But in those first few minutes, everyone was looking at each other and saying, 'What did we do? Did we mess something up in our plan?' But trusting the tools, it was real fast for us to identify the problem and move forward."
Let us know what you think about the story; email: Shamus McGillicuddy, News Editor