Machine Design and its sister publication Electronic Design held a contest to collect stories from engineers that involved troubleshooting and solving an issue on their own. Last year, Dave Wright (see photo above) won the prize when he had to troubleshoot an important component of his measurement equipment. Here’s his story.
Many years back, I was working as a technician for a local shop that was specializing in creating custom industrial control systems. We cobbled them up from 8-bit micros and whatever glue logic and hardware that was needed for the application. As was typical in the day, a client decided to capture the company and ride the wave of microprocessor-controlled everything.
Choosing to not stay on, this left an opportunity for me to support the equipment we had built and deployed through the area. Only one problem—I had no test gear of my own other than a hand-me-down digital voltmeter. I had grown up with the Tektronix 465B 'scope; my hands knew where the controls were without looking, I could operate it as easily as a familiar car with manual shift. I needed one, I couldn’t afford one. A carefully crafted letter to Grandma yielded a loan of some two kilobucks, and I was able to buy a refurbished rental unit. I was in business for myself..!
Fast forward 35 years, and that old 465B is still on my bench. The calibration has surely drifted a bit, but I have plenty of new digital gear if I need to pin a number on something. When I need to probe into a mysterious malfunction, I reach for the old Tek probes first because I can move fast and see the waveforms with practiced motions.
These days, I prowl around on eBay looking for bargains in test gear. I was on the hunt for a precision frequency reference, and thought I had found a steal in a GPS-disciplined rubidium oscillator.
Winning the bid easily, I wondered if perhaps there was something I did not know. Turned out the unit required a down-converting type of remote antenna.
NOTE: A GPS-disciplined rubidium oscillator increases the accuracy and stability of time-related measurements. Multiple instruments can share the signal, allowing all to be synchronized to a single high-precision reference frequency.
Working through the email and telephone mazes eventually put me contact with a sympathetic tech who identified the particular antenna I needed to get the unit in operation. The search was on, and it quickly became apparent that the particular antenna was “unobtanium,” hence the bargain price for the oscillator.
Patience was rewarded, however, and a used antenna showed up on eBay some months later. It was snatched up at the buy-it-now price before some other desperate soul could get to it before me. Stringing a coax out to the yard on a frosty December day, I hooked the antenna to the receiver and watched the status display change from “looking for satellites” to “2D time.” Eureka! We were in business.
There followed weeks of learning the intricacies of configuration, measuring ionospheric propagation delays from WWV in Colorado (the call sign of the United States National Institute of Standards) to my location in Louisville, and watching the phase error change throughout the day. Seeing the phase-error number hovering around in the E-09 territory was a certain sort of joy only a “time-nut” can understand…
All went well for the winter. Glances at the status readout indicated that the oscillator was tracking well, the rubidium package was behaving, and satisfaction surrounded the whole affair. The antenna was relocated to the attic, coax fished through to the basement shop, and my attention turned to other things.
NOTE: Receiving the time-and-frequency-standard radio signal from WWV allowed Dave to measure the length of the path that the radio waves took from Colorado to Louisville. He was measuring the time interval between the PPS (pulse per second) “tick” out of the GPSDO, and the arrival of the radio “tick” signal from WWV. Because the two “ticks” represent the same instant in time, any observed delay will be due to the speed of the radio waves and the distance traveled.
Spring arrived with warmth and flowers, the basement lab lost its appeal for a time, and the GPS oscillator was taken for granted as a reliable piece of gear. It was not to be. One afternoon, the status displayed “looking for satellites” when all had been well for months. Early the next morning, all was well again, and then in the afternoon it lost the satellites. This went on for several days and I finally hit on the thought that the attic was heating up in the afternoon sun and raising the temperature of the antenna. I brought the antenna down to the kitchen, put it in the refrigerator for a while, and then back outside. Yup, it was thermal. As long as the antenna was below 70ºF it worked ok, above that and no-go.
A phone call to the helpful tech at the manufacturer of the obsolete unit revealed a telling comment, “Oh yeah, we had a lot of trouble with that particular antenna as I remember.” Oh great! My bargain GPS rubidium frequency reference is turning out to be not such a bargain. I had continued to watch eBay for additional antennas as a spare, and none turned up. As most techs will understand, when a system they have assembled stops working, a certain determination to put it right starts to take over. It was with this determination that I undertook an investigation of the antenna.
Some testing with a signal generator and spectrum analyzer revealed that the down-converter in the antenna developed spurious sidebands when the temperature went up, so most likely there was a phase-locked loop (PLL) inside somewhere that was misbehaving. I just needed to get to it. Being sealed in a plastic dome, filled with expanding foam insulation, and soldered into an RF-tight enclosure, just getting to a point where I could put a probe on a circuit node was an epic journey in itself.
Surgery on the lathe with a Dremel cutter:
Burrowing down through the insulating foam:
The guts exposed:
Once opened up on the bench, and peering through the magnifier at a few square inches of surface-mount components, I was getting my bearings. Reading device numbers off a few of the ICs and Googling for datasheets began to reveal what was what. Here’s the input RF amp, a regulator, another RF amp, and look at this one.
Just put the probe there…
It’s a GPS down-conversion function block with internal local oscillator (LO), PLL, and mixer. Ok, we’re getting somewhere. Looking at the application example in the datasheet, and comparing it with the circuitry on the PCB, we see little resemblance. Hmmm… we’re poking around with the probe on my faithful old 465B, looking at PLL error voltage, and watching the spectrum analyzer.
All of a sudden, the erroneous sidebands disappear. I lift the probe and they return; probe down and the signal clears up again! Sound familiar? A feedback circuit, operating on the hairy edge of stability, falls over the edge only to be rescued by a few picofarads (pF) of added capacitance at the right place. Looking up the spec on my 'scope probe reveals that 13 pF seems to make my PLL stable. Digging into the SMD capacitor assortment, I find a 12-pF unit and rather clumsily manage to get it soldered on the right spot.
Two winters and summers later, and we’re still good. I’m left to wonder what the circuit engineers were thinking when that particular design was developed. Never attribute to malice what stupidity can explain came to mind, but there is a lot I still don’t know about electronics. Perhaps it was cost, perhaps it yielded better temperature stability within some narrow conditions, or something I have yet to learn about. Anyway, my story is not unique. Many a strange problem has been fixed by nothing more than putting a ’scope probe on the right node.
If you want to share your engineering stories, send them to [email protected]. We might select your story to publish next.