Edge Data Center Markets and the Redundancy Problem

By Raul K. Martynek | Chief Executive Officer

How DataBank and Zayo Networks are Teaming-Up to Deliver on the Promise of Edge Computing

In a previous post, I argued that despite all the buzz about micro-data centers at the base of cell towers, the first place “The Edge” will most likely materialize is inside traditional data centers located within secondary “NFL cities”. The incremental improvements in latency that come from deploying content and workloads locally in these large population centers (as opposed to distant cloud data centers), coupled with the ability of full-sized data centers to better scale and maintain infrastructure, actually make this approach far superior to alternatives like micro-data centers.

But while this thesis addresses the latency considerations and benefits of data center location, we all know that the effective delivery of bits depends on another critical element in the infrastructure chain: the network behind the data center. The connectivity available in a building, and the paths from which it enters and exits the property, are just as important as the redundancy and configuration of the mechanical, electrical and plumbing (MEP) systems in the facility. A weak link in this part of the chain can negate all the benefits of the data center’s proximity and redundancy to users in that metropolitan region.

Let me propose an analogy. Everyone knows that best practices dictate that a properly designed data center must have a minimum of two Points of Entry (POEs) for fiber, and a minimum of two Meet Me Room (MMRs) to terminate the fiber and distribute it to IT workloads. This ensures the proper degree of physical diversity both within and outside the data center.

And what is true for a data center needs to be true for a geographic market. If we imagine that interconnection data centers are effectively the “meet me rooms” of a metro area, and the long-haul fiber paths into and out of those data centers are the “Points of Entry” for that market, it stands to reason that on this macro scale we need the same level of diversity. Within a given metropolitan market, redundant interconnection locations and diverse long-haul fiber access must be available in order to provide the market level diversity our industry is accustomed to at the micro-level. As we will see, that’s not always the case, and it has the potential to slow the development of the edge for all providers and users.

In this article, we will explore the current problem with network interconnection and fiber redundancy in secondary markets. I will argue that multiple interconnection points and physically diverse long-haul fiber connectivity options must emerge for mission-critical deployments to flourish at the Edge.

Let’s start by looking at how most large Tier 1 markets are configured for data center interconnection and long-haul fiber. Figure 1 below depicts the Zayo fiber map in metro NYC. While New York is unique in many ways (and one that I have known for years) the broad configuration I will describe is the same in all Tier 1 markets. The three orange dots are the three main “carrier hotels” in the New York area – 60 Hudson St., 111 8th Ave., and 165 Halsey St. in Newark, NJ. In a market such as NYC, there are other significant interconnection points (32 Ave of Americas, 325 Hudson, 75 Broad, etc.) but most people would agree that these three locations form the “core” of peering and interconnection for the New York Metro market. The reason there are multiple locations is obvious: redundancy. With three locations, users have the ability to deploy in multiple locations to ensure uptime. A failure in one facility or portion of a facility will not impact traffic in another location. Similarly, each of these locations benefits from dozens of metro and long-haul fiber options that exit the New York area heading north, west, east, northwest and southwest from Manhattan. With multiple interconnection points and multiple routes coming into/out of the area, a provider can carefully construct a resilient architecture for connectivity and availability.

When you survey the landscape, all major Tier 1 markets also have 2-3 major interconnection points and diverse metro and long-haul fiber connectivity. Dallas (my new home) has three main locations – 1950 Stemmons, 2323 Bryan and 400 South Akard (DataBank’s DFW1 location). Long-haul fiber enters the market from multiple, physically separate routes. Chicago has 350 E. Cermak and 600 S. Federal. From each of these facilities or by tethering to a sub-tending datacenter in the same market, users can avail themselves of diverse data center locations and redundant metro fiber and long-haul fiber options to ensure that no one location can become a “single point of failure”.

Large cloud, content and technology providers understand this. DE-CIX is one of the world’s leading interconnection platforms. Until this year, their only location in Dallas was at the Infomart Building at 1950 Stemmons. As their traffic grew, they decided to create a secondary location that offers its customers an alternative peering point for diversity. In August, DE-CIX, deployed their fully redundant Apollon platform at our 400 South Akard location. In talking to the DE-CIX folks, the two main criteria for their selection were: (a) a robust data center location physically diverse from Stemmons with good fiber connectivity; (b) available long-haul fiber routes fully independent from Stemmons. South Akard checked both boxes. In addition to offering a diverse local data center facility, the 400 South Akard location provides access to fully diverse long-haul fiber routes to exit the Dallas region.

The situation is not the same in most secondary markets. In many of the markets that DataBank operates, multiple, diverse interconnection locations and physically diverse long-haul fiber routes have not yet developed. Figure 2 below shows the situation in Minneapolis. In that market, the Cologix facility located at 511 Main Street is the predominant carrier interconnection point. To exit the market, the main long-haul fiber routes go either south to Chicago or west to Seattle (we show the Zayo routes in this figure). A third route heads due south towards Omaha. Most of the peering in the Minneapolis market happens within the 511 building. Today, there does not exist another interconnection point within the market that has both direct access to all the diverse long-haul fiber routes and enough critical mass of local metro fiber providers to be a viable alternative location. For Minneapolis, the 511 building is the proverbial single point of failure and a bottleneck that must be addressed for the region to support fully diverse edge workloads.

Recently, DataBank’s lead investor, Digital Bridge Holdings, announced the acquisition of Zayo Networks in conjunction with EQT Equity Partners. Zayo owns and operates the largest fiber network in the United States with dense metro and long-haul fiber routes. At Digital Bridge, we believe that having complementary internet infrastructure assets in the same portfolio will foster the development of innovative solutions. DataBank and Zayo are doing just that. We have joined forces to bring alternative interconnection and long-haul fiber diversity to secondary edge markets, and we have picked Minneapolis as our initial market.

Figure 3 depicts the proposed new routes for long-haul dark fiber connectivity out of the Minneapolis market using Zayo fiber. Zayo will extend its Omaha long-haul route to terminate at DataBank’s MSP1 facility and its Chicago route to terminate at DataBank’s MSP2 location. Zayo metro dark fiber already connects MSP1 and MSP2. The key element of this route is that it bypasses the 511 Main location heading West, while also providing long-haul access to Chicago and Omaha via metro fiber splices. As an interconnection point, the DataBank MSP2 facility is the designated termination point. DataBank’s MSP2 facility is a certified Tier III data center location with over 15 onsite metro fiber providers. Recently, the Midwest Internet Cooperative Exchange (MICE) deployed an extension switch in MSP2. MSP2 is also located in Egan, a suburb in Minneapolis about 10 miles south of the downtown 511 Main Street location. MSP2 provides an alternative interconnection facility with diverse long-haul dark fiber options, giving Minneapolis the resilient internet infrastructure required to support the mission-critical workloads and make it a viable edge market. DataBank and Zayo intend to partner in other markets where there is a significant overlap in assets such as Kansas City, Indianapolis, and Salt Lake City. We will share those plans as we develop the solutions.

At DataBank, we believe that for Edge markets like Minneapolis to fully develop, redundant interconnection and long-haul fiber options must emerge, so that providers of scaled, mission-critical applications can build the redundant architectures they have come to rely on in Tier 1 markets. In our conversations with large cloud, content and technology companies, this topic is a reality. With the exponential growth in internet traffic, large content providers and ISPs are increasingly looking to “regionalize” their traffic flows and drain traffic from Tier 1 markets into adjacent metropolitan areas. However, in our conversations with such users, they are reticent to expand their footprint without the availability of diverse options for colocation, peering and dark fiber transit. For edge workloads to fully evolve and deliver the 100% uptime that will be expected from these new technologies, a robust physical internet infrastructure topology will be required.

Diversity of options also provides another non-technical benefit: competition. As the importance of internet infrastructure grows, so does the importance of truly neutral interconnection locations. Yet we continue to hear stories of aggressive incumbent behavior, incessant price increases and anti-competitive terms of supposedly “neutral” operators of interconnection facilities. We believe that owning/controlling the dominant interconnection point in a market is not an excuse to abuse your customers, but an opportunity to foster and build a better internet, which will ultimately benefit all users and consumers of internet infrastructure.