Stop Patching the Noise and Start Finding the Root Cause
I spent about two hours this past week staring at a trend line, absolutely convinced I’d found a fundamental flaw in how computers do math.
I was doing some final testing on revisions for our upcoming SCADA class, running a robot simulation of an anodizing system (or a t-shirt factory, if you use a little imagination). Everything looked great until I noticed a funky jump in the A3 axis.
On the screen, it looked like classic signal noise—sharp, jagged spikes on a sine wave. But here was the catch: the robot wasn't physically jumping. It was smooth as silk. The "noise" only existed in the data.
The Search for the Source
My first instinct was to check the PLC. I went into Studio 5000, trended the A3 tag, and sure enough, the spikes were there too. Was it a bug in the Yamaha robot code? I checked the A2 axis, which was perfectly smooth. If the code could handle one axis, it could handle the other.
Then I questioned the math. Floating-point math can do weird things, but usually not until you get into very high numbers. I exported the trend to a spreadsheet and found a value of 30,900 that should have resulted in 30.9 after being divided by 1,000. Instead, it was showing 15. When I tested that specific calculation in a vacuum, it worked perfectly.
The Cross-Reference Lie
I did what any of us would do: I right-clicked the tag and selected Cross Reference. I looked down the "Destructive" column to see what was writing to that tag. There was only one "Y"—my own scale instruction.
At this point, I did what I always tell my students to do: I took my hand off the mouse and thought. I changed the A3 position to a temporary real number tag that wasn't associated with anything else. Immediately, the signal was smooth. That proved it: something else was writing to that memory address, even if the PLC cross-reference didn't see it.
The Hidden Culprit
Since the PLC wasn't doing it, it had to be coming from the network. The only other thing connected was the Inductive Automation Ignition system. To narrow it down, I switched the Ignition connection to Read Only. Instantly, the spikes vanished in the PLC, and Ignition started throwing errors.
I had accidentally checked the "bidirectional" box on an HMI component. It was trying to write back to the PLC while the PLC was trying to update the HMI, creating a data conflict that looked exactly like electrical noise.
Stop Patching, Start Solving
The most interesting part was the feedback I got when I posted a picture of the trend online. Over half the comments said, "It's just noise, just add a filter."
While a filter would have hidden the spikes, it wouldn't have fixed the problem. It would have just masked a logic error. Before you reach for a software filter or start blaming "bad math," ask yourself: Is something else writing to this? As technicians, we have to move beyond just patching symptoms. Whether it’s using the "View Diagnostics" tool in Ignition or understanding why Wireshark requires port mirroring to see point-to-point traffic, we need to find the root cause. Sometimes the "ghost" in your machine is just a checkbox you forgot to uncheck.