Verizon does agree with Jeff Pulver -- but neither realizes it

Jeff Pulver of FreeWorldDialup and Pulver.com and the VON Coalition got into it a bit with Brian Whitton of Verizon Labs at IPTComm 07 last week. Pulver argues that VoIP systems should be open, and new ideas should be tried. Brian Whitton argued that "business realities" were what caused Verizon to do the things it does -- like deploy FTTX using Voice over ATM at first. Both of them are probably right, though there was definitely antagonism. Verizon comes across as the enemy of VoIP for their lawsuit against Vonage. Pulver really sounds like a naive 1997 dot-com bubblemaker, except that he's fully self aware that his ideas don't make economic sense up front.

It strikes me that Whitton has it tougher than Pulver. I've worked at some larger service providers helping them make VoIP work; I'm a system integrator. Telcos have it tough because they have tons of requirements:

• The system has to work right out of the gate
  
• They can never turn it off for maintenance
  
• They can't get bored and cancel it
  
• They have to account for (be able to bill) everything
  
• They want full redundancy (at least two of everything)
  
• All the pieces have to be fully tested so they know all of the features work
  
• They have to be able to stand behind all of the pieces used; e.g., their customers can't just "try" a new SIP phone
  
• It has to satisfy the tastes, styles, and preferences of their engineers
  
• It usually has to work over a significant/large market area; they usually just don't try something in part of one city
  
• It's supposed to be delivered on a predictable timeline
  
• It has to meet regulatory standards; 911 of course, but also any tariffs in place
  
• It's supposed to follow all the known rules. (As one friend I had who worked for BellSouth's labs said: people in the telephone companies advance not by how creative or innovative they are, but by how closely they follow the written rules.)
  
• Everybody has to be trained on everything they're ever supposed to touch; e.g., the sales guys all have to understand it, so it's best if it acts just like ordinary telephone service
  
• It has to support any features that might be imagined to be needed someday
  
• And anything else they dream up as "problems" along the way, such as security protections against attacks not shown to be actually feasible
  

But the two most damning features of the large telcos are these:

• They have plenty of money to do it.
  
• They have plenty of time to design it.
  

Jeff Pulver talked about "forget about ARPU and EBIDAS for a year or 18 months"; he was trying to say: not everything cool can be expected to be self-sustaining immediately. Some things actually require an extensive investment. And he was aiming the jab at Whitton and Verizon. But in reality, organizations like Verizon Labs spend far more than 12-18 months playing with their systems, and they know it'll take a long time to recoup their investment. The Telcos do ignore ARPU and EBIDAS while they're building and designing.

My company has customers just like Verizon; or, more precisely, exactly like Verizon. And I've seen enough to know that practically-unlimited-time and practically-unlimited-money can sink good and creative ideas. The ideas and good intentions get all bogged down in all of the Bellhead traditional practices, and they can afford to do it. They've got loads of money and time! They don't have to re-think the Bellhead ways; they don't have to say, "that's not perfectly redundant, but if it fails it won't actually affect service"; they don't have to tell the engineers, "look, you're right, we really should do it like that, but let's just get this thing to market so we can test it, then we can decide what's most important next".

If telephone companies would follow one piece of Jeff's advice and actually limit their product-rollout time to no more than 12 months, they'd have a much easier time of things. They'd be able to focus on the really important pieces.

Henning Schulzrinne: We need a PHP for VoIP

New York, NY: IPTComm 2007 at Columbia University was organized by Henning Schulzrinne, one of the co-inventors of SIP and an ongoing developer of VoIP applications. Schulzrinne commented that he thinks one of the big reasons people aren't developing new communication services is because there's nothing like PHP that can control calls.

He's referring to the PHP programming language, which was one of the first widespread ways that people wrote programs to develop web applications. Perl programming of CGI web applications came earlier, but required integration with Apache and mod_perl. PHP came out-of-the-box ready to do web pages. It was simple to do simple things, and possible to do harder things.

Jonathan Rosenberg: No new applications because all the ideas are old.

New York, NY: Jonathon Rosenberg, the principle inventor of SIP IPTComm 2007 at Columbia University, claims that we're not seeing new interesting VoIP-based services because people have been working on Voice services for a long time. All the obvious ideas have already been tried. It's not easy to have new useful ideas.

Brian Whitton of Verizon Labs: Integration is the Problem

New York, NY: At IPTComm 2007 at Columbia University, Brian Whitton spoke about the network. He disagreed with Jeff Pulver's assertion that innovation wasn't happening because people didn't have guts. Whitton showed slides of Verizon's Fiber-to-the-home network that uses Voice over ATM, and showed the new architecture where SIP FTTX NIDs (ATAs) that talk to BroadWorks. The new architecture also included the Nortel CS2K (likely for PSTN access from BroadWorks).

He said the innovation is stymied by the diversity of vendors; there's no one source for all of the devices, and they don't all interoperate very well. It's a lot of work to integrate all the pieces. He also pointed out that there are more independent pieces (servers, routers, SBCs, call control servers, etc.) than in the traditional PSTN, not fewer.

So the complexity of integration seems to be the largest difficulty.

Jeff Pulver: Why aren't there advanced services? Nobody has any guts.

New York, NY. According to Jeff Pulver at the IPTComm 07 conference in Columbia: The reason there aren't any advanced services is because everybody lost their courage. We took the closed network environment of traditional telcos and replicated it in VoIP. So now people can't easily build new services on top of VoIP services.

"People need to have guts. People need to take changes. We got real, and we forgot about the fun...We've got to forget about ARPU and EBIDA for 12-18 months...We're forgetting about the core values that make innovation happen every day."

This is the first time I've heard Jeff speak. Anbody talking about the need for courage gets my attention.

Cisco Catalyst 3750 and VoIP (CORRECTED)

The performance of the Cisco Catalyst 3750 is a funny thing. Software-based routing, called process switching, can only switch 2500-3000 packets per second[1]. But the box itself is rated at 6.5 million packets per second[2], using hardware-based switching. That's a huge disparity, larger than on Cisco's official "router" products. For example, the Cisco 2821 can process-switch 75,000 pps, and CEF-switch 170,000 pps.[5]

According to Cisco: nearly all packets are switched in hardware using Cisco Express Forwarding. But some packets are still switched in software. So, which packets take the slow boat?

As best I can tell: the first packet to a destination is process-switched, and thereafter, on average one packet every twenty minutes to that destination [3].

Also, any packet through a GRE tunnel is process-switched [4]. These are the "tunnel" interfaces.

So, what's the net-effect of this on VoIP? There are two.

According to [3], every minute, 1/20 of the CEF cache is invalidated. But we don't care about averages, since this 5% invalidation happens all at once. (I'm assuming.) [CORRECTION: This is referring to the Cisco fast-switching cache invalidation. IF you're not using fast switching (a distinct feature from CEF), then you're not subject to this. But I'll leave the rest of this article intact, just in case it's later useful.]

We can process-switch 2500 packets per second. So in any 20 ms period following the CEF table invalidation, it can process-switch packets for 50 different destinations. If it takes longer than 20 ms, then an RTP flow will miss a packet.

The router can keep up if these 50 packets represent 5% of destinations going through the box. 50 is 5% of 1000.

1000 different destinations might represent 1000 different RTP streams; at 20 ms packetization, that's 20000 pps. That's well within the 6.5 mpps [2].

So I'd expect the Catalyst 3750 to be able to keep up with routing VoIP traffic to 1000 different destinations, using the typical 20 ms RTP packetization.(*)

But what if you don't have lots of different destinations -- just lots of calls to only a few destinations? Then you're more likely to be bound by the 6.5 mpps or the backplane throughput capacity of 32 Gbps. Under those conditions, your maximum is 65,000 concurrent calls (based on mpps) or 168,000 concurrent calls (based on Gbps).

Sources:

[1] http://www.cisco.com/en/US/products/hw/switches/ps5023/products_qanda_item09186a00801b0971.shtml

[2] http://www.cisco.com/en/US/products/hw/switches/ps5023/prod_models_comparison.html

[3] http://www.cisco.com/warp/public/63/tuning.html

[4] http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a00807213f5.shtml

[5] http://www.cisco.com/warp/public/765/tools/quickreference/routerperformance.pdf

Footnote:

(*) What does 1000 different destinations look like in a hosted VoIP company? Well, we can estimate the revenue size of such a company. Suppose you're selling residential Bring-Your-Own-Broadband VoIP for $14-$25/month. Suppose you're using media steering because these are NAT'd devices, so all the media has to go to and from your SBC; thus, each call appears on your router twice. Suppose also that you're using Level(3), so the media for your calls is routed to numerous different trunking gateways. (To be honest, I don't really know how many Level(3) gateways Let's make a guess that you're talking to about half as many trunking gateway IP addresses are you have sessions.

1000 [destinations] = calls + 1/2 calls + 1 [sbc IP] 
calls ~= 667
subscribers = calls * 4 [oversubscription ratio] = 2,668
revenue = $19.50 [average monthly rate] * 2,668 [subscribers] * 12 [months/year] ~= $624,000/year

So this gives us a ballpark idea for how big a BYOB VoIP provider might grow using Level(3) and the Catalyst 3750 for routing. This is really just for fun, though.

Suppose this BYOB VoIP company switches to a carrier that routes all of their PSTN calls to just ten different trunking gateways; that increases the supported subscriber count to around 3,950 and revenues of $925,704/year.

Giant Telco, meet Demanding Customers (Or: AT&T and the iPhone)

I'm no expert on telephone company management, nor the iPhone itself, nor the internal AT&T wireless-side organization. But I have been around numerous telephone companies, and have some ideas about AT&T and the iPhone.

At the time of this post, lots of people are complaining of trouble activating the iPhone. My guesses about some sources of trouble:

• The iPhone doesn't have an external SIM slot (from what I've read), even though AT&T customer reps have been trained that all GSM phones have a SIM slot.
  
• The iTunes-based activation system is, in part, a front-end to AT&T wireless' provisioning platform. It's really hard to test provisioning software under load without doing it a lot; after all, who wants to load up their dozens of multi-million dollar telephone switches with a bunch of bogus subscribers? It's really hard to test any software under load without doing it a lot.
  
• It doesn't help that they started this on Friday evening. Most telcos avoid even disconnecting non-paying customers on Friday because they (the telco) doesn't intend to be around on the weekend to fix things. So while customer-service call centers were up and operational, I suspect a lot of the back-end provisioning and switch-management staff were hoping to actually get the Independence-Day weekend off.
  
• The iPhone itself might have some bugs.
  

Giant Telcos like to control the whole thing: the network, your device, etc. Why, up to a few years ago (in telco history terms), the American telco owned every thing connected to its network -- you couldn't even plug in your own telephone.