Recently, we got a unique opportunity to check out Verizon Wireless? Regional Operations Control Center or ROCC. This facility is responsible for connecting all of the cellular towers together while simultaneously connecting to the rest of the Verizon network. This facility is given the purpose of covering all of San Diego and Imperial counties with some redundancy built in to support other regions as well. If you were to drive up to this building, you would have no idea that this facility served hundreds of thousands if not millions of connections a day. Since Verizon cannot provide us with actual usage figures, we can?t necessarily give you exact user figures, but with San Diego and Imperial counties combined, you?re looking at about 3.5 million people, with many of them using Verizon as their carrier of choice.
The reason why Verizon Wireless is so popular in this market is primarily because they have consistently provided better call and data coverage in the region for many years. This can be supported by Verizon Wireless? dominance in Rootmetrics? annual tests where Verizon Wireless has consistently been the winner against AT&T, T-Mobile, Sprint, and Cricket. This year, AT&T did improve significantly almost tying Verizon Wireless in a 4-way tie, however, Verizon Wireless still had superior call performance which has been one of the primary reasons why people have switched to them (my mom included).
Now, getting back to the facility itself you would have no idea that the facility is serving potentially a million or more connections considering its relatively small size and stature. But when you?re running a facility that is responsible for serving so many connections, you have to maintain a low profile. The parking lot itself was fairly empty and made me believe that I was actually in the wrong place, since I expected such a facility to be heavily staffed like the SBC/AT&T/PacBell ROCC I visited in San Diego when I was a child.
Upon entering this facility, I was made aware of the place?s high security and no-tolerance for unexpected visitors. I was directed to a meeting room where I met with my tour guides for the day that explained to me that their meeting room was mostly used for training Verizon employees and for emergency situations. After we chatted a bit, we made our way over to the first room in the NOCC, but not after crossing through a secure hallway which goes around the center of the entire facility. This provides an added level of security and monitoring to make sure that everything is always visible.
The first room was their power room, which had a gigantic DC power board. This power board serves many purposes including the protection of the circuits going to different parts of the building with double redundant fuses. It also gives the ability to give a direct and instantaneous power shutoff for certain parts of the building by removing both fuses, if an electrical fire were to break out. The grey cables coming out are power and the green cables coming out are ground. This room also has tons of rectifiers to help convert the DC power from the battery room to AC.
The second room we got to take a look at was the battery room. As you can see here, these are gigantic acid batteries, which are constantly charged and prepared for use if the power were to go out. Because these batteries are directly hooked into the building?s power, if there were ever a power outage the people inside of it would not notice any difference of operation. These backup batteries can run anyone from 4 to 8 hours until a generator needs to kick on and provide backup power. The batteries? longevity depends on the size of the hardware inside of the building running the network and it?s always expanding. Each battery is constantly inspected in order to maintain voltage and each string of batteries is tested to make sure all of the connections are in working order. There are around 700 batteries in this room, each weighing in around 700 lbs. As you can tell, Verizon does not mess around with their backup batteries.
The next room we got to take a look at was the power room where all of the external power from the grid comes into the facility. This facility is designed to facilitate more than a megawatt of power if necessary; however, this facility does not currently consume that much power. The building has multiple sources of power (continuing on the redundancy aspect of things) and has the ability to accept even more than two sources.
As we walked by the (green) ground cables for all of the power in the building, we walked outside the door and took a look at the external generators for the building. Each of these generators holds up to 10,000 gallons of diesel fuel and are periodically checked to make sure that they are about 80% full since they don?t want to overfill them. These two generators combined can deliver over 1 Megawatt of power, which is enough to keep the building powered for about 8 hours at which point they will need to get a delivery of more diesel fuel. The building also has hookups for 2 more mobile generators, if needed and Verizon Wireless already has contracts with partners to deliver backup generators for their backup generators (yes, a backup for a backup) at a moment?s notice.
The next room we went to was the main server and switch room; this is where most of the most of the actual work that this building was designed for gets done. We were shown some networking equipment with double redundancy as well as off-site redundancy, but since it is sensitive equipment we were not allowed to photograph it. Verizon?s own networking and server redundancy system does not only have on-site redundancy, but it also has off-site redundancy in other centers like it across the region. This enables one ROCC to assist another in the event if something were to go wrong at one location. It?s more of a pool-like off-site redundancy system, not necessarily 1 to 1.
The first thing you can see (above), though, is the only copper that exists within the whole building that is not networking cable for internal servers. This copper is in order to support land line connections and to bring in any kind of older telecommunications. As time goes by, this part of the facility will likely completely disappear as a whole.
The next row (above) is for all of the networking cables, mostly Cat5 cables to help manage all of the countless network cables that are needed to connect the different equipment in the building. This does not have as many routers and switches as you would think as it is only a way for Verizon Wireless to manage the cabling.
The following row (above) is where all of the internal and external fiber for the entire building is managed. This row of boxes allows Verizon Wireless employees to easily run new fiber connections while simultaneously being able to manage old ones as well. As you can tell, there are hundreds, if not thousands of fiber optic cables running through this facility. Each box can hold up to 24 separate connections, and the fiber connections you see are actually redundant, so somewhere in the middle each box is redundant to another on the other side of the row from the middle. Once again, showing how serious Verizon Wireless has to be about having backups, of everything.
The next row is where all of the 3G calls and data are handled by Verizon Wireless. As Verizon Wireless has expanded the use of their 4G LTE network, fewer users are using 3G data and as a general trend people are calling each other less anyways. All calls and 3G data are handled by the racks on the right, while the racks on the left are for cross connecting equipment on the floor to each other.
Next, we have the 4G LTE racks, which actually take up even less space than the 3G equipment. In the future, voice and data should be handled by the same equipment, so the amount of equipment used once 4G LTE starts to deliver VoLTE should not change. That would mean that those 3G racks will eventually disappear completely and be replaced by 4G LTE racks instead. A total of 4 racks are what we were shown to run the 4G LTE network, which is amusing since the 4G LTE network has so much more bandwidth and so many users now. Virtually every device Verizon Wireless sells nowadays is 4G LTE enabled in one way or another.
After that, we got to look at some proprietary equipment that allows Verizon to be able to continue to real world test their 3G and 4G networks to make sure that they are operational. We were not able to take a picture of this equipment or the setup, but it basically consisted of off-the-shelf devices that are connected to Verizon?s network while being connected to computers that monitor their call quality and data speeds.
Once we left the server room, we got to take a look at the actual control center. Once again, we could not take any pictures in here since there was a lot of proprietary and customer data being displayed, but Verizon Wireless does have some really interesting graphs constantly running that allow you to view the network load at any given moment and to see what the status of the power in the building is as well. There are also antennas inside of this room to allow for engineers in the control center to run any tests on the network without having to actually go outside of the thick concrete building to test something. I was really surprised to see that there were only two people running the entire ROCC from that room, granted there was an engineer or two in one of the two server rooms.
After we passed through the mostly empty second server room, we headed towards the outside rear of the building. Do note that the reason why the second server room is mostly empty is to allow for the facility to grow with network demand, not because Verizon Wireless overestimated on how much space they needed. A calculated decision was made to allow for room to grow, preventing the need for construction or even worse, a move.
Outside, we saw two things. First, we saw all of the cooling equipment (below) that Verizon uses to cool the entire facility, since all of that computer equipment needs to be cooled somehow. They use a huge water cooling system which they run outside to chillers which cool down the water. Currently, Verizon has 3 installed, but as always, they have left room for a fourth in the event that one becomes needed as the facility grows. There are also hookups for backup chillers in the event that some are needed for some emergency.
We also got to take a look (above) at some of Verizon?s field deployable equipment including dozens of generators, which only represent a fraction of the generators Verizon Wireless has deployed. The big truck is one of Verizon Wireless? Cells on Wheels or CoWs. There is also a cellular repeater and a cellular antenna to help catch signal in remote areas. It was definitely reassuring to see how much backup equipment Verizon Wireless has deployed.
After going outside, we went back inside and took a look at the pumps that run the cooling system followed by the air handlers which are a part of the HVAC system that the coolers and pumps are a part of as well.
Following this, we got to take a look at a separate server room, which is specifically placed on the outer ring of the building away from the rest of the server equipment. This room is where telecom companies like AT&T, T-Mobile, and Sprint would put their own hardware to enable their network to communicate with Verizon?s. This room has its own access and its own wiring and is specifically designed to keep non-Verizon Wireless employees out of the main server room where all of Verizon Wireless? and their customers? hardware is located.
Also, in almost every corner of every room where there is either electrical or computer hardware, Verizon has a fire evacuation system which is designed to suck the air out of the room rapidly to extinguish any fires. It dumps a bunch of retardant and air into the room that is affected to quickly and effectively put out any fires. You can see these white canisters in many of the photos that we took throughout the article.
The main purpose of this article was to get an idea of what kinds of hardware, manpower and work are needed in order to keep a network like Verizon?s online. The truth of the matter is that Verizon Wireless is not the cheapest network to be on, but I can personally attest to its reliability. Back when all of San Diego county had a blackout for about 12 hours, I was able to maintain an internet connection to Verizon?s LTE network completely uninterrupted. I was able to stream a football game online on my laptop without any hiccups, while I still did not have any power in the house.
The truth of the matter is that you don?t really realize how important all of the measures Verizon have gone to until you really need them. If the guys at Verizon Wireless? NOCC are doing their job right, you have no idea that they or the NOCC exist. You also have to take into account how much our mobile network operators have to spend on multiple redundancy equipment to enable a reliable network. There is a reason why Verizon is rated highest in reliability above all of their competitors.
We want to thank the kind people at Verizon Wireless for letting us into their NOCC and showing us around with complete all-access. It was a great experience and I hope that our readers gained some knowledge from it as well. Special thanks to Ken Muche for setting up this tour.