This decision by Google, while small in terms of units purchased, is enormous in terms of the disruptive impact it should have on 10GbE switching equipment providers and their component supply chains. It is as if a MACHO just arrived in the Enterprise networking business and the orbits of the existing satellites have begun to shift without observers knowing why – until now.

We were watching shipments of SFP+ components for 10GbE in the market but simply couldn’t account for their end destination – sort of an optical component dark matter problem. After a great deal of investigation we have reached the following opinion:

Through conversations with multiple carrier, equipment, and component industry sources we have confirmed that Google has designed, built, and deployed homebrewed 10GbE switches for providing server interconnect within their data centers. This is very similar to Google’s efforts to build its own server computers (excellent article here). Google realized that because its computing needs were very specific, it could design and build computers that were cheaper and lower power than off the shelf alternatives. The decision to do so had a profound impact on server architecture and influenced the market’s move to lower power density solutions that Sun (JAVA) , Intel (INTC) and AMD (AMD) now embrace.

It now appears that the process Google trail blazed in the server computing market will repeat itself in the enterprise switching market. Given the relative dearth of low-cost 10GbE switching solutions, it isn’t surprising to see Google revisit this approach.

We believe Google based their current switch design on Broadcom’s (BRCM) 20-port 10GE switch silicon (BCM56800) and SFP+ based interconnect. It is likely that Broadcom’s 10GbE PHY is also being employed. This would be a repeat of the same winner-take-all scenario that played out in 1GbE interconnect. Vendors of standalone 10GbE PHY silicon ( AMCC (AMCC), VTSS (VTSS.PK), Netlogic/Aeluros (NETL) ) should take close note. Broadcom’s role in hollowing out equipment is something we previously profiled in depth (see Cisco’s Fear of a Broadcom Planet).

What is interesting about Google’s approach is that it has eschewed traditional 10GBASE optical standards and instead adopted off-standard solutions that better suit its needs for time-to-market, power and port density, and cost. While Google makes use of the SFP+ cage format, it does not use the receive dispersion compensation (EDC) function typically associated with SFP+. Instead Google is looking to employ a combination of twinax cabling for short reach (<10m) intra-rack cabling and a motley 850nm SR-like standard. Off the shelf SR optical modules appear to work well up to 100m over without receive equalization. Ironically, Finisar (FNSR) proposed such a solution several years ago.

This non-standard and very low cost optical format should prove just as attractive to other datacenter customers. Given the delays in deploying production grade EDC solutions it is possible vendors will move forward with an SFP+ SR standard without EDC. This would be a boon to suppliers of SR based SFP+ modules such as Finisar and Avago as adoption of the SFP+ standard will accelerate faster once decoupled from the complexity and cost of EDC. (see Five Misconceptions About the 10G Optical Market)

It is difficult to determine the precise amount of components Google is purchasing. Google is believed to have in excess of 500,000 servers. Based on shipments of 10G SFP+ modules, our best guess puts Google’s current usage at approximately 5k ports of 10GbE a month. This would include both server based SFP interconnect as well as the switches themselves. While the number is low, it is Google’s implementation and motivation for building their own switches that will resonate through the equipment and component industries.

At this time, other purveyors of large data centers like Yahoo, Microsoft, and Equinix do not appear to be following the same aggressive path with SFP+ optics. This is likely to change as new low cost per port 10GbE equipment from Arastra, Woven, Force10, Cisco (CSCO), and Juniper (JNPR) come into production that make use of the new format.

To us, it is Arastra that is the most interesting company in the context of Google’s decision. Arastra is building a system that closely matches what Google appears to be doing in secret. A picture of Arastra’s 7148S system with 48x 10GbE ports is below.

Arastra presents the pseudo-IEEE standard 10GBASE-CR which appears to match the twinax approach Google is taking. Furthermore, Arastra was founded and funded by Andy Bechtolsheim, Chief Architect at Sun Microsystems and who is closely tied to Eric Schmidt, the CEO of Google and an ex-Sun executive. Andy Bechtolsheim was also one of the first investors in Google. With these connections, Arastra may be the commercialization of Google’s technology and the ultimate supplier to Google itself.

Through our investigative research, Nyquist Capital reached the conclusion  that 12 months ago Google took a look at the state of the art in 10GE switching equipment and decided that it could do better. The reasons behind this decision will have a large impact on how the small but rapidly growing 10GbE equipment and component market evolves.

Author holds positions in Broadcom, Vitesse, Finisar and AMCC.

In the past, optical transport has been hardware intensive and light on software content. The big volume metro boxes of the bubble such at the Nortel 5200 and the long reach systems in particular didn’t have differentiating software. The R&D was in the optics. It is for this reason I viewed this sector as ripe for commoditization, particularly by vendors such as Huawei (see “Ciena Sees Red at BT“). It is for this reason we avoided Ciena as an investment. This is now a flawed assumption.

The future of the core network increasingly appears to be Layer 2 based- whether it is Ethernet, T-MPLS, PBT is a secondary issue. Implementation of these new Metro Ethernet protocols is about as far away from a commodity  as you can get. Systems vendors, such as Ciena, are producing systems that merge Layer 2 Ethernet switching with traditional optical transport functionality. In the past carriers would avoid such ‘god boxes’ for a number of reasons both good and bad, preferring to keep the switching and transport elements separate.

The Chinese wall between these functions now appears to be breaking down. The fact is that without some compression of transport and switching equipment carriers will not drive their operational expenditures down. This is a primary concern that trumps the political, technical, and historical reasons that they were separate. The inclusion of Layer 2 functionality is deflecting transport equipment from a commodity future, at least temporarily. This is a major shift and is forcing a change to a core investment thesis.

Market Summary

The Ciena 4200 series attracts the most investor attention but it is a glorified transport multiplexer with limited L2 intelligence. It filled a market need but it not at the vanguard of innovation. The new Ciena 5060 platform evolved from their Wavesmith product, which bridged legacy ATM networks and allowed carriers to cap ATM switch expenditures. This new system converts a fistful of protocols into Ethernet and is a better example of a Ciena Layer-2 Ethernet product. If Ciena is successful in the carrier Ethernet market, it will be because of the 5060, not the 4200.

It is surprising to me that Cisco (CSCO) and Juniper (JNPR) are still so notably absent from this market. Both have the deep Layer 2 (and Layer 3, and Layer 4…) expertise to be a force in the Carrier Ethernet market, particularly with the strength of their network OS. The common refrain is that their Enterprise networking roots make them unsuitable for carrier consumption but the recent success of the GSR12k refutes this. How long will it be until Cisco repositions itself in the metro market with a junior version of the GSR12k?

There are other systems such as the Fujitsu Flashwave 9500 and Nortel (NT) OME 6500. These are heavy iron telecom systems with myriad functionality. As an engineer at a major dark fiber carrier explained to me “they do everything but do nothing well.” These vendors are developing systems with one foot in legacy TDM and another in Layer 2.

Market Scenarios

There are four Metro Ethernet scenarios that we see developing.

1. Simple L2 aggregation combined with transport. Limited support for legacy interfaces and no support for TDM. Limited traffic management and service awareness.
2. The God box. Support for TDM grooming, L2 switching and perhaps even MPLS routing.
3. Sophisticated L2 switching and L3 routing. Deep packet classification and traffic management. No support for TDM.
4. The stupid network. Buy dirt cheap commodity transport equipment and manage everything at the wavelength level. Backhaul everything to enormous Cisco and Juniper routers and sort it out there.

All of these scenarios will see deployment but it is too early to tell which will dominate. In theory, the new business plan of Soapstone (aka Avici (AVCI) ) makes sense since a platform agnostic software layer would allow carriers to mix and match platforms and avoid being held hostage to one equipment vendors native operating system. (see “The End of Telecom As We Know It And I Feel Fine“

Ciena has masterfully played the Ethernet PR card but the race for market share is only beginning. Ciena, at it’s core, is still a Layer 1 transport company, and is positioning itself as a Layer 2 Carrier Ethernet company. If successful, this would remove the threat of commoditization and is deserving of closer study.

Note I still haven’t changed my mind on NPU’s…

There is one exception to my stereotype. British Telecom’s (BT) 21st Century (21CN) initiative. No Victorian Bric-a-Brac here.

I’ve spent some time ripping apart BT21CN in order to understand the short and long term impact on the telecom equipment and component supply chain. So far, I really like what I see.

There is a ton of information on 21CN in the public domain. BT has stated transparency is an important characteristic of this initiative and it shows. The 21CN website is a good place to start.

MSAN – King of the Ring

Most impressive is BT’s commitment to take all voice and data services at the edge and homogenize them into IP and Ethernet. The star of the show is the MSAN (MultiService Access Node), a DSLAM-on-steroids. All protocol complexity is pushed to the extreme edge of the network and collapsed into the MSAN, which feeds an Ethernet and IP core.

Every voice line is converted to VoIP right where the copper pair is terminated. All 30 million of them. DSL services are provided from the same linecard. BT is using these systems to rollout ADSL2+ to 95% of households nationwide. There’s no separate DSLAM, POTS termination, SONET/SDH Add Drop Mux. MSAN collapses all.

Fractional TDM based Frame Relay and IP services (500k lines!) are packetized and bundled right at the POP. If it isn’t TDM leased line (E1 or bigger), it gets packetized and sent through the core using MPLS. BT has indicated that support for fractional rate leased lines after 2012 is questionable.

The Major Players

Fujitsu (FJTSY) and Huawei provide the MSAN boxes that sit at the edge of the network and act as the bridge between legacy services and an all-IP core, while providing robust support for transporting legacy TDM containers. This is a 40% share of project capex, or $2.8 Billion Dollars.

Speculation was rife several weeks ago that Huawei had lost it’s MSAN contract. I’ve subsequently learned that it is the optical transport contract that Huawei is struggling to keep. Ciena (CIEN) and Huawei are providing the optical transport equipment, though multiple people have informed me that Ciena is capturing a greater share due to problems with the Huawei solution. This is rumor, not fact, so don’t assume it is true.

It certainly was a shock to me, as I have long expected and written about how Huawei will meet initial success in western networks through it’s optical transport equipment. Regardless, I expect the optical transport portion of the contract to be the least important in both strategic and dollar terms. In the near term BT is bootstrapping all of their old transport equipment and not buying a great deal of new kit.

Cisco (CSCO), Lucent/Alcatel (ALA), Siemens (SI), Juniper (JNPR), and most recently Nortel ::ticker(“NT”) supply a motley assortment of switching and routing gear. The picture is murky but it appears to me that, as usual, Cisco wins and everyone else is invited to ensure a decent hand of poker.

Much attention is focused on all the vendors named in the 21CN contract, but the reality is one MSAN vendor is likely to capture at least 1/3 of total capex. It will either be Fujitsu or Huawei. If you have an opinion, please do share.

The Death of TDM Access

Going forward, TDM as an enterprise access technology is over in the UK. BT announced a new contract win with ADVA (ADVOF.PK) to deploy Ethernet demarcation boxes using a multitude of backhaul technologies, mostly fiber and mid-band Ethernet. This is part of their push to roll out Ethernet services.

This is a really big deal. Instead of forcing another copper TDM connection down the throats of their customers (you thought English food was bad) BT will be offering carrier grade Ethernet connectivity, fiber based in some cases.

Anyone betting on Ethernet over PDH or any native TDM as customer connectivity should be concerned. All legacy TDM based Frame Relay and PPP protocols are homogenized into IP right in the MSAN. New customers will be receiving Adva boxes using copper or fiber based Ethernet. It’s safe to assume those E1 lines used for voice will vaporize sometime shortly after.

TDM based optical transport (i.e. SONET/SDH or Ethernet over SONET/SDH) is still used throughout the network to transport both legacy TDM leased lines and IP services.

It is notable that even with a network as radically advanced as BT’s, SONET/SDH still plays a central role, Ethernet over SONET/SDH in particular. Take that, evil anti-SONET Sith Lords. I will fight you to the last.

VoIP 2.0

The other notable characteristic of BT’s architecture is they chose not to rely on TDM to VoIP conversion while using legacy copper TDM termination equipment. This choice, if a trend for other carriers, is death for companies like Sonus (SONS), who provide a solution that bootstraps existing TDM investments. The drawback to their solution is they end up adding more equipment and complexity rather than reducing it.

BT embeds the VoIP functionality right into the MSAN, and does the conversion as close to the customer as possible. This has the effect of reducing, not increasing network elements and is the only long term method for driving down operational costs.

As carriers aggressively deploy low-cost DSL and DSL penetration increases, the MSAN model makes more and more sense. There won’t be legacy TDM connections for Sonus to convert as the edge equipment itself will incorporate this function.

I believe companies like Acme Packet (APKT) are better representative of the long term future of VoIP and other connection based services yet to be invented. Once VoIP is a pervasive service, the growth will be in managing and securing the connections successfully, not converting legacy TDM connections.

Conclusion? Sonus = Carrier VoIP 1.0, Acme Packet = Carrier VoIP 2.0.

Summary

The Victorian age of Telecom is nearing an end, with it’s assortment of bric-a-brac equipment destined for retirement.

BT 21CN is a sleeker, flatter, radically modernistic alternative these Victorian networks of old. If successful, Tech historians will draw parallels between Victorian interior design of the 19th century and Telco Central Offices of the 20th century. More Gropius. Less Cruft.

I could write pages about how much I don’t like England…. somehow the country doesn’t match the fine characteristics of the people who live there.

The author is long Acme Packet.