We responded today to an attack on a nationwide Service Provider. They reported up to 69 REGISTERs per second originating from an IP address in Anhui province, China. 69 REGISTERs per second is roughly the equivalent load of 5,000 users.

Unfortunately for the victims, the “friendly scanner”, SIPVicious runs very hot and fast, apparently blasting out lots of requests without even waiting for earlier attempts to fail. The SIPVicious tool is focused on cracking SIP PBXs, and will be only so slightly less effective on Carrier VoIP systems.

The main reports of problems due to SIP Registration scanning are server overloads. But if the registration scanner users are smart, they’ll slow down their rates so they don’t alarm the parties being probed.

How do you defend against SIP Registration storms?

1. For registering endpoints like SIP phones and IADs always use SIP authentication! use quality passwords.
2. If you have a competent Session Border Controller like the Acme Packet OS-C system, you can blacklist devices after they fail a few REGISTER attempts.
3. If you’re using non-registering SIP (such as SIP peerings for SIP Trunking), you should have a small number of SIP signaling IP addresses. Use firewall rules / or ACLs to block all SIP except for what comes from that small list.
4. Use heavy-hitter detectors to spot SIP devices that are sending more-than-normal traffic loads, and alarm your staff.

Many Hosted PBX providers based on VoIP are surprised by the network load caused by the SIP Features “Shared Call Appearance” (SCA), “Shared Line Appearance”, “Simultaneous Ring”, “Busy Lamp Field” (BLF), or “Line State Monitoring”. Both of the big Application Server players, Metaswitch and BroadSoft, offer these features. And carriers are starting to deploy them in spades. They’re often together, so I’ll call them collectively BLF/SCA/Simring.

All these features are fundamental departures in the underlying signaling model. I’ll explain why, and what a VoIP Service Provider needs to do.

1,000% Increase in Signaling

Normal Modern call control requires only a few signaling SIP messages to setup or end a telephone call: INVITE, 100, 180, PRACK, 200, 200, ACK, BYE, 200. You’ll get more messages when calls are put on hold, or switch to fax mode, and Metaswitch does session-audits with a re-INVITE every 30 seconds. But 10 or 20 messages are typical for a phone call.

Enter BLF/SCA. Now every phone in the group can get 6 or more NOTIFY-200 SIP messages for every call placed by other people in the group. Plus they get a call setup attempt for every single call for every person in their group.

The SIP signaling load per user grows enormously. A user who has 20 calls per day might only need 200 signaling messages for call control, but a user in a 5-person BLF/SCA group could have 1,100 messages per day. Raise that to ten-person group, and now it’s 2,180 messages per day.

The signaling load grows superlinearly — in this case, it grows as n² with the number of users in the group!

Surprise! Your system is full earlier than expected.

The real danger is not the messages per day — it’s the messages per second at your peak. If your customers are clustered into one geographic area, that peak probably happens around 10:00am or 3:00pm local time. But for typical users, peak load will correspond to the daily workload.

And the problem is typically not in the routers and switches; signaling load is just more IP traffic. Solving problems in the transport network would be easy. But signaling load affects the application plane — i.e., the devices that process the SIP. Solving application-layer problems is much more complex, because (obviously) the application has to track the state and progress of all of the user-oriented business logic.

So what is most likely to get overloaded during the peak?

1. The Session Border Controller.
   Even some folks at the SBC vendor have been caught off-guard in a few cases by this. In one case, a sales engineer told me that he had planned for a single SBC installation, but needed 3x SBC systems to handle all the BLF/SCA traffic. “The planning tool we had was all wrong,” he said.
2. The Application Server.
   When you’re using BLF/SCA, the AS or Call Feature Server (CFS) may have to process 10x as many SIP dialogs, and has to handle 10x as many SIP messages. “The most expensive thing an Application Server does is process a SIP packets,” a BroadSoft Systems Engineer once told me.

BLF/SCA and Simultaneous Ring are certainly very useful — but they come with a price.

Cool features — Big cost differences.

For example: suppose an SBC costs $67,500 for a non-redundant system and three years of support. Without BLF/SCA/Simultaneous Ring, you could expect to support 30,000 users on the system; i.e., that’s $0.75/user/year CapEx for the SBC. But with BLF/SCA/Simultaneous Ring, your efficiency could could drop significantly — reasonably down to 5,000 users — i.e., $4.50/user/year CapEx. That’s a 6x difference in cost per user — but still a great price for all the features an SBC provides.

The revenues from BLF/SCA/Simultaneous Ring are also great, but probably not proportional to the signaling load. Can you really charge 6x or 10x the price for a BLF/SCA/Simultaneous Ring customer? Of course not; but remember that the signaling load is not the only cost of the system. Technical talent, network transport, Customer Premise Equipment, sales and marketing etc., are all significant expenses that are largely unaffected by these features.

Not all SIP Phones are Created Equal

You’ve got to consider the Customer Premise Equipment selection. BLF/SCA are advanced new features, and do not enjoy the robust, mature, time-and-customer-tested support of ordinary call control. While Polycom has been supporting BLF/SCA for years and wide deployment, many other phones do not have the history. There’s nothing magical about Polycom — they just have a head start on software maturity and reliability. And they have the experience with SIP-over-TCP necessary to make BLF work well.

Finally, test scale before you deploy. Testing a three-member BLF/SCA group in the lab is not adequate preparation to sell a ten-member BLF/SCA group. Because of the non-linear growth in signaling, and the requirement to use SIP over TCP for reliable BLF/Line State updates, these features do not scale up in clean intuitive ways.

Further, you’ve got to test the transport. Very low levels of packet loss can seriously affect SIP over UDP deployments of Busy Lamp Field. Your Gigabit-Ethernet VoIP lab probably isn’t the best evaluate the reliability of your service, if that service is deployed using T1s to customers.

Before you deploy a big Simultaneous Ring / BLF / SCA group, you need to test and prove the reliability of that big group. You’ll be sorry if you don’t, and your customer will let you know just how sorry.

Proceed — with Caution.

Busy Lamp Field, Shared Call Appearance, Shared Line Appearance, and Simultaneous Ring are cool features, and well worth selling. But they constitute a genuine disruption to the ordinary Hosted IP PBX model that has been built with BroadWorks and Metaswitch for years. Network engineering, costs, and support all change in serious ways.

ECG has a new, high-speed training class on Acme Packet SIP Trunking and Registration. This is a high-speed class intended to cover just the basics you need to configure SIP trunking in the Acme Packet OS-C (4250, 4500, 3800, 9200), or configure SIP registration. ECG has been installing, configuring, and troubleshooting Acme Packet OS-C systems since 2004. We really know the box inside and out.

Let us show you the tactical details you need. Learn how to piece together a SIP Trunking or registration config — complete with realms, local-policy’s, sip-interface’s, steering-pool’s, session-group’s, session-agent’s, etc. Understand the key options, and learn which options to study later.

Read more on the Acme Packet Boot Camp: Building SIP Trunks & Registrationg page. You can email sales@e-c-group.com or call +1-229-244-2099.

ECG is rolling out new training courses. Another new class we have is on BroadWorks Call Processing and Troubleshooting.

This is a class for the people who really need to understand how BroadWorks does what it does. They need to know how to fix it when it breaks, and how to determine what’s broken.

This class covers all the arcane dark-magic BroadWorks tools, like healthmon, ttIsql, bwcli, and repctl, peercmd. We cover all the logs, like XSLogs, PSLogs, AuditLogs, and ms.syslog. We show you how the protocols interact.

This is a week-long class, taught at your location, or at ECG locations near Denver, Raleigh, or Valdosta, Georgia.

If you’re interested, read more on the BroadWorks Boot Camp: Call Processing & Troubleshooting page. You can email sales@e-c-group.com or call +1-229-244-2099.

We at ECG are pleased to announce a new training course: BroadWorks Installation and Upgrades.

This class is intended to give engineeers and technicians tons of experience installing, patching, and upgrading BroadWorks. There are a numerous small rules, and a few tricks. We at ECG have been doing Broadworks installations since R9 — back in 2002. We’ve done hundreds of installs for carriers around the world.

We have the scars. Let us save you some pain. In addition to being a crack team of special-forces consultants, ECG is a bona-fide BroadWorks service provider providing service in southern Georgia.

This is a week-long class, taught at your location, or at ECG locations near Denver, Raleigh, or Valdosta, Georgia.

If you’re interested, read more on the BroadWorks Boot Camp: Installation & Upgrades page. You can email sales@e-c-group.com or call +1-229-244-2099.

ECG is a leading Network Design and Operation consultancy for the Metaswitch Networks — Carrier Systems division. Since 2007, ECG has accelerated deployments by performing network planning, installation, training, and operational maintenance.

ECG’s brings in-depth knowledge of MetaSwitch Call Agent, Media Gateway, EAS, and N-Series Systems to solid IP Network Design practice. ECG’s Carrier IP and PSTN/SS7 Network Design experience extends to 1995.

The new CPE Management System is integrated with BroadSoft’s BroadWorks system, the industry-leading softswitch for Commercial VoIP Carriers. Features of the new system include:

• Zero-Touch Maintenance: Centennial’s Customers and Provisioning Team can enjoy new features, line moves, adds, and deletes that are automatically propagated to the CPE.
• BroadWorks Integration: The CPE Management system is fully integrated with BroadSoft BroadWorks, supporting all advanced hosted IP PBX features.
• SSL Verification: Centennial’s Cisco/Linksys SPA phones will refuse to communicate with unsigned CPE Management Servers. This provides defense for Centennial’s customers against man-in-the-middle and DNS hijacking attacks.
• Configuration Signing/Encryption: Centennial’s Cisco/Linksys SPA phones are pre-programmed with Centennial credentials. This prevents a customer or Man-In-The-Middle from recovering key system signaling and management details. Attackers cannot redirect Centennial’s phones to another system.
• Fault Tolerance: Centennial’s system maintains multiple copies of each configuration, on separate servers, to protect against server or network fault.
• Traffic Engineering: The new system supports traffic engineering, allowing Centennial to intelligently route customer SIP traffic to backup data centers.
• VLAN Management: Customer devices can be configured to use a Voice VLAN using 802.1q tagging. This can be managed through BroadWorks directly.
• Advanced Troubleshooting: Centennial’s customer devices report on network issues before they become problems. These data are transmitted to a central logging server, where they can be analyzed by Centennial operations staff.

Centennial PR Customer Technology Integration (CTI) Engineers Fernando Jorge and Alberto Leo were principle technicians on the project.

ECG, Inc. developed and implemented the system for Centennial PR. ECG Engineers Mark Lindsey and Jonathan Stanley worked with Centennial through the design and specification process, then implemented custom software and systems.

The tested design allows customers of each service provider to take full advantage of the breadth of CODECs and services the SIP feature servers and endpoints provide including HD voice and SIP-based video.

Centennial PR upgraded from BroadWorks Release 14sp3 to Release 14sp9. The coordination of the upgrade is complex with numerous interconnected components:

• The Nortel Communication Server 2000 provides SS7 PSTN access for BroadWorks.
• The Convergy’s Ceon platform is integrated with BroadWorks for provisioning.
• The Acme Packet SD provides security and session management features both for Hosted PBX VoIP and for IP Trunking to several carriers.
• Centennial’s CONUS-based Billing Department receives CDRs from six different BroadWorks application servers.
• Many tens-of-thousands Scientific Atlanta MGCP/NCS eMTAs are integrated with BroadWorks
• The mPathix voicemail system is integrated with BroadWorks.
• VoIP Customer Premise equipment from Cisco, Adtran, Linksys, Polycom, and Digium/Asterisk are used in numerous configurations.

“The BroadWorks software activation is the simple part,” according to project engineer Mark Lindsey. “The complexity comes in verifying the software will work with all of the integrated components.” The upgrade from BroadWorks R14sp3 to R14sp9 involves several hundred individual changes. ECG analyzed each change to ensure it would not create a problem in Centennial’s specific network, and designed and conducted regression tests to be used in the system upgrade.

ECG has extensive software integration experience with the BroadSoft BroadWorks platform, including integration with OCI-P for provisioning and feature management, as well as OCI-C for call control. The Attaché Call Control tool for Mac OS X is a working example of both of these interfaces.