Machine Design

Industrial Networks Get Synced Up

The advent of techniques for putting Ethernet nodes on a common time base may make it possible to synchronize even widely separated connections.

The advent of techniques for putting Ethernet nodes on a common time base may make it possible to synchronize even widely separated connections.

Barry Dropping
Sr. Director Product
Line Management
Symmetricom Inc.
San Jose, Calf.

Edited by Leland Teschler

The hardware setup for a system deploying IEEE 1588 generally incorporates a time stamping unit that sits between the physical layer (PHY) of Ethernet and the media access control (MAC) layer. The TSU issues a precise time stamp whenever it senses a 1588 timing packet.

The sequence of packets used to transfer time from the PTP master to a PTP slave can be visualized as shown here. Sync packets are stamped as they leave the master and arrive at the slave. The difference in the master and slave time-stamps represents the offset of the slave plus the message transmission delay. The slave clock adjusts itself to compensate for this difference. To correct for message-transmission delay, the slave uses a second set of sync and follow-up messages with its corrected clock to calculate the master-slave delay. The second set of messages accounts for variations in network delays. The slave time-stamps and sends a delay request message. The master time-stamps the arrival of this message and sends a delay response message with the delay request arrival time-stamp. The difference between the two time stamps is the slave-to-master delay. The slave averages the two directional delays and then adjusts its clock by the delay to synchronize the two clocks. The sequence repeats periodically to keep the two clocks synchronized.

A standard called IEEE 1588 Precision Time Protocol (or just PTP for short) has gotten considerable attention since its introduction in 2002. The reason is it forms the basis for defining Ethernet links that can transport synchronization signals with small and well-defined delays. The delays can be defined with enough accuracy, on the order of nanoseconds, to let Ethernet links synchronize signals and tasks over large physical distances.

A variety of silicon vendors are producing hardware that supports PTP. And an IEEE committee is defining the next version of the 1588 protocol, which is expected to have even better accuracy than today's standard. It will have better fault tolerance and higher update rates.

Nevertheless, today's standard is good enough to do useful work. For example, the initial applications for PTP in telecom will be to drive servocontrol loops for reference oscillators in remote network elements such as street cabinet access platforms and wireless base stations. Historically this equipment has been synchronized over T1 connections that are far more expensive than Ethernet. Industrial applications are expected to include the synchronization of distributed motion controllers and measurement systems.

Prior to the PTP standard, Ethernet was not well suited for applications involving precise synchronization. By nature it is nondeterministic, meaning there is no guarantee its data packets will ever arrive. And those that do arrive may experience different delays or latency that vary from packet to packet. This is true even if all the packets come from the same source. PTP overcomes the Ethernet latency and jitter issues through a technique called hardware time stamping, which takes place at the physical layer of the network. The result can be an unprecedented accuracy in the 100-nsec range. Consequently, several schemes for defining deterministic versions of Ethernet now incorporate IEEE 1588 techniques as a way to track packet arrival times.

There is more than one way to deliver synchronization over Ethernet networks. Equipment manufacturers currently are examining several means toward this end. Three main techniques besides PTP are under consideration:

Adaptive Clock Recovery (ACR): ACR algorithms attempt to reproduce the master network clock at far-away nodes. In telecom applications, many proprietary schemes transfer voice or video traffic across a network through circuit emulation services that implement ACR. These applications need ACR because voice and video, unlike data traffic, is sensitive to delay and delay variance. Though ACR-based techniques are seeing some interest, the proprietary equipment on which they are generally deployed makes manufacturers leery of using them.

Synchronous Ethernet: The International Telecommunications Union recently began defining a standard for Synchronous Ethernet as a way to synchronize frequencies over Ethernet networks. But deployment is years away. And Synchronous Ethernet will be suited only for new applications because all elements in the network will need to be upgraded to support the standard.

Network Time Protocol (NTP): NTP is the most widely used protocol for time synchronization over LANs and WANs. It is one of the oldest Internet protocols still in use. NTP is relatively inexpensive to implement, requiring little in the way of hardware. It can usually maintain time to within 10 msec over the public Internet and can hit accuracies of 200 sec or better in LANs under ideal conditions. However, the current version of NTP does not meet the higher precision requirements for modern synchronized networks. There are studies underway to improve it, but more precise versions are still in the future.

PTP, on the other hand, offers the economies of NTP by using existing Ethernet distribution networks. It exceeds the NTP accuracy through the use of hardware-based time stamping. Network hardware inserts a time stamp into each packet, which the receiver uses to synchronize networks precisely. Thus time stamps serve as synchronization signals so receivers can properly reconstruct the initial data.

PTP can coexist with normal network traffic on a standard Ethernet using high-speed switches. The addition of IEEE 1588 boundary clocks or transparent switches makes possible submicrosecond synchronization accuracy.

A boundary clock simply serves as a time-transfer standard between the subnets defined by routers or other network devices. The boundary clock has a network connection to each of the subnets. Ordinary clocks within each subnet synchronize with the boundary clock. The boundary clock resolves all of the times of the several subnets by establishing a parent-child hierarchy of clocks.

One point to note, however, is that use of cascading boundary clocks can cause nonlinear time offsets to accumulate in the servoloops that generate these clock signals, degrading their accuracy to an unacceptable degree.

The two primary problems that must be overcome in network timekeeping are oscillator drift and time-transfer latency. Regardless of the protocol used, oscillator drift can be mitigated by using high-quality oscillators and by deriving time from accurate sources. In telecom settings, for example, GPS might serve as a time source. Clocks based on temperature-controlled or oven-controlled crystal oscillators typically have stability measured in parts per billion.

The time-transfer latency problem is more difficult and is twofold in nature: First, there is latency associated with the processing of time packets by the operating system. Second, there is network latency created by routers, switches, cables and other hardware that resides between clocks on a network. It is in the area of reducing operating system latency and jitter that PTP is most successful.

PTP combines time stamping units (TSUs) with an innovative method for exchanging timestamp details between master and slave clocks. A TSU sits between the Ethernet Media Access Control (MAC) protocol and the Ethernet transceiver. It sniffs both inbound and outbound traffic and issues a time stamp when it sees the leading bits of an IEEE 1588 PTP packet, precisely marking the arrival or departure of PTP time packets.

(As a quick review, the MAC protocol provides addressing and channel access control that lets nodes in a network communicate with each other. It gives a physical address to each network adapter so data packets can get to specific destinations.)

To estimate and mitigate latency caused by the operating system, the master clock periodically sends a Sync message based on its local clock to a slave clock on the network. The TSU marks the exact time the Sync message is sent. A Follow_Up message with the exact time information then immediately goes to the slave clock. The slave clock time-stamps the arrival of the Sync message and compares the arrival time to the departure time provided in the Follow_Up. This information lets it identify the amount of latency in the operating system and adjust its clock accordingly.

Network-related latency is handled differently. It is reduced by measuring the round-trip delay between master and slave clock. The slave periodically sends a delay request message (Delay_Req) to the master clock, which issues a delay response message (Delay_Resp). Because both messages are precisely time-stamped, the slave clock can combine this information with the detail from the Sync and Follow_Up messages to gauge and adjust for network-induced latency.

Those familiar with Ethernet protocols will realize IEEE 1588 PTP is similar in concept to NTP. Both protocols distribute time packets in-band with the payload traffic. NTP is ubiquitous and operates at the upper layers. PTP is a specialized layer-two (data link) protocol. Three main factors determine performance: The resolution and accuracy of the time-stamping engines in the master and slave (the accuracy you begin with); latency/packet delay variation (PDV) through the network; and servoprocessing gain and oscillator implementation at the slave side (how effectively PDV uncertainty can be filtered out).

Given a high degree of starting precision, Packet Delay Variation (PDV) over networks rapidly becomes the dominant source of error for packet-based timing schemes. Layer-two switched networks (that is, those containing network switches and bridges) will perform best with regard to PDV. This aligns well with IEEE 1588 PTP as this protocol is optimized for layer-two switching environments.

Performance over layer-three routed networks (that is, those containing network switches and routers) will show little difference in the time-stamping stability between NTP and PTP. That is because there is a lot of PDV uncertainty through layer-three software-based routers. PDV will be the main source of time variations for layer-three software-routed networks. Oscillator stability and design at the slave side will be the main factors that assure network nodes remain synchronized.

In PTP, the desired accuracy of timing determines how often sync messages are broadcast and the kind of oscillator used. More frequent broadcasts result in more accurate sync. But the broadcasts also generate more network traffic, although the bandwidth used is extremely small. Higher-quality oscillators also result in more accurate sync.

It may be tempting to try to get accuracy economically through use of a lower quality oscillator broadcasting more frequently, but this is unadvisable. Low-quality oscillators generally lack the stability for the precision needed in networks, so shortening the broadcast interval offers diminishing returns.

For example, clocks based on crystal oscillators that are not temperature controlled generally exhibit stability measured in parts per million. They drift apart more rapidly than temperature-compensated clocks so they need more frequent corrections.

Accuracy is also a function of the IEEE 1588 master clock, called the grandmaster, which is the ultimate source of time on the network. Grandmasters are typically both stable and accurate. In telecom applications they are often referenced to GPS. With a GPS reference, accuracy to the Coordinated Universal Time (UTC) standard is typically 30-nsec rms or better. Starting with such an accurate clock with an absolute time reference makes PTP-enabled networks well synchronized. A quality grandmaster also provides other measurement features both to characterize the latency and jitter of network elements and to measure how accurate the slave is relative to the grandmaster.

PTP has been implemented on Ethernet networks where router-buffer delay and switch latencies undermine the accuracy of time transfer. Advanced wire-speed routers can switch fast enough to rival traditional layer-two switching in terms of latency and PDV. This makes them suitable for distributing PTP sync signals. On the other hand, the high latency and PDV associated with software-based routers can be a limitation as noted above.

The transparent clock is another potential hardware option for the PTP-based network. This is a PTP-enhanced switch that modifies the precise time stamps in the Delay_Resp and Follow_Up messages to account for receive and transmit delays within the switch itself. The result is better sync between slave and master clocks. However, the transparent clock can also create security issues when the original packet crypto checksum doesn't match the final packet arriving at the slave.

To learn more about IEEE 1588,
Symmetricom Inc.,

TAGS: Technologies
Hide comments


  • Allowed HTML tags: <em> <strong> <blockquote> <br> <p>

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.