IT Horror Stories: Tales of Wireless Interference
1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...

IT Horror Stories: Tales of Wireless Interference

Our resident IT expert, Ross Thomas, recounts a couple of amusing Wi-Fi tales

When I took this job nearly a year ago, there were some lingering wireless issues within our office. Just having come from a position where I dealt quite a bit with wireless, I had a lot of experience with different wireless devices and had just the thing: Ubiquiti AP AC Pro. Shameless shoutout: this hardware is fire (good)! In fact, I insisted that my personal one be used to demonstrate just how excellent that piece of hardware is for a wireless solution and, given that the results were satisfactory, the company purchased one like a week later.

Of course, when I got it setup in an area that was central-ish to the office, there was great improvement in wireless performance. The access point moved around a few times, not too far from that central location, before I decided to replace the then “current” solution of a wireless router repurposed as an access point (despite the fact that routing was turned on and it was serving up IP addresses in its own little subnet :-/). That supplanting actually moved the device a little ways away from central-ish of the office.

Everything seemed to be fine.

Since then, there were a few reports of our Chromecast not working well in our conference room. I had seen it work fine while other times it would be slow, have some poor throughput and sometimes would drop from the wireless prompting to be reconnected. How would moving an access point 15 linear feet display such a degradation of performance? The coverage appeared to be fine anywhere else in the office. The truth is that I currently do not know, for sure. But, my suspicion is that the way the office is constructed, with some sheet metal-looking ceiling elements and various metal beams, a kitchen with microwave/toaster oven/Keurig (not to mention other rooms) in between the access point and the conference room and an entire shared office building of wireless access points flooding the spectrum (as shown in an app called Wifi Analyzer) could be attributing to that performance degradation.

I moved the access point back to the general and original position and it appears that everything is working great. I’ll let it go for a bit and listen out for reports of problems and then look to fixate the AP AC Pro around that location as its new final destination. If we need another access point to serve the area where the AP AC Pro was shifted away from, then so be it. It appears that everything is currently working fine and we should be back to happy wireless for all (within my office). Ill know for sure in like 2-3 weeks.

Wireless is a finicky thing despite its ubiquity and can lead to frustrations with the proper timing. Interference plays a big part in wireless because the air is so very flooded with signals that reside in the provided spectrum. Obviously, this is not the first time in my career that I have seen wireless interference cause so many disruptions. There are 2 noteworthy times (probably more, actually) in which interference with wireless has provided that “gotcha”, or, more appropriately from the receiving side, “I got got”, that I will share and that are similar in explanation of what happened recently with the Chromecast performance degradation.

New Orleans – Beautifully Historic – Historically Outdated Architecture

One of my earlier dips into the world of digital signage had me working as a field technician (amongst many other roles): installing and configuring servers at hotels, casinos, etc, troubleshooting technical issues and running on-site training for how the software is best used. New Orleans is no stranger to hotels, casinos, etc., so we had plenty of business in the most (in)famous part, the French Quarter. This takes us to the Royal Sonesta, a traditional looking and wonderfully located hotel (another shameless shout out) that sits right in the heart of Bourbon Street. You could imagine that any young, fun seeking individual (which I was and still am) would leap at the chance to visit and be put up at this location. There is nothing quite like finishing a hard day of work, stepping out the front door of the hotel onto Bourbon street to have a plate of blackened Redfish and some fine, locally brewed Abitas at a local restaurant.

When this project first started, my initial task was to survey the building and the areas as to where the digital signage would be located. It is not a huge hotel, but big enough and those conference rooms, where a lot of the digital signage would live, were spaced plenty far apart. I met the IT staff who immediately stated that it is difficult to run any kind of cable in the building down the walls. They can run it through the ceiling and they don’t like construction when they can help it and they would prefer to go wireless. No problem. Even though wireless was somewhat in its adolescent stage, it should work plenty fine.

“You’re gonna need more then you think,” said one of the IT guys in his cool New Orleans Creolish accent. “This building is about 40 years old. The French Quarter’s old,” he continued.

“Cool,” I said. “Let’s mark out where we think we’ll need it.”

We walked around the commons area.

“Access Point/Repeater here……here…….one here will cover those 3……etc, etc,” we agreed upon.

“Yeah, let’s start there und see how we do,” he said. “So you know, the reason we have so many problems with wireless is this building has metal in the walls. It’s like a cage. This is also why we have problem running cable through the walls.” Hmmmm.

I returned a few weeks later after we had some stuff configured and we looked to get everything in place.

“These two screens have a bad connection,” he informed me. “They catch but they are not always connected.”

The way the system worked, it wasn’t really that big of a deal. Since everything within that system was local, as in, the information fed to the screen originated from the server, and the screen would continue playing whatever it had last, it should be OK given that it catches updates to information, as needed.

“They work for the most part so we’re not too concerned now.”

A few weeks later, I get a call:

“Uh, those two screens haven’t contacted in a week or so. Reboots are fixing them immediately but they go off again.”

I flew back out, and we decided to put another access point closer based on our range of where the cable could be ran to the ceiling tile. That seemed to be holding it steady.

A week later, I get another call:

“This room is a killer. That one is still dropping but the other one is OK now,” he spoke with a chuckle.

“Can we have that newest AP moved a little closer?” I asked.

“Yeah, we already got the work order in for that. Ill let you know,” he said.

He called back a few days and told me that moving that newest AP location had killed the original one it fixed.

“So, what I’m hearing is that we need an AP for, like, each of the last 4 rooms of that hall? That seems a bit overkill but I understand if it is necessary. Are you sure we can’t run cable down those walls?” I inquired.

“That tile and décor has been there since the hotel opened. We are gonna PO the GM if we ask to tear it out,” he replied.

“Access point it is, then. That’s like 8 Access point for 9 computers. Wow.”

Just the way that building was constructed in the later 60s would affect the future technology. At my later digital signage position, there were certain parts of downtown Philadelphia, also a historical city, as you are aware, that had similar problems but were able to correct some of it by running cable through the walls. They would not have vintage looking tiles/décor that could not be messed up.

South Carolina Hospital – Major Medical Machinery or Mealtime/Munchie Menace?

My later dip into the industry of digital signage involved a network that did indeed traverse the public networking space. The reason for this is because that particular use case was driven by advertisements meaning that certain information had to be reported to get credit for honoring advertising contracts.

The system was designed to call into our servers every so often, 20-30 minutes with some basic play stats as well as general (end unit) health. So, if the end unit, computer or player, does not call in within that 20-30 timeframe, we could assume that something happened to the machine itself or there is a problem with the networking. Most often, it was a networking issue but the lifespan of these players would also lead to problems. We always gave a 3 day grace period before we would label a player as officially MIA (missing in action). The player would continue playing original content and if it was offline and the content expired, we had some default content so the failure would be considered “graceful.”

Now, our target market was medical facilities so, we were exclusively in hospitals and certain types of doctor’s offices. As you probably know, there is a lot of complex machinery in medical facilities so, we had to consider that when setting up the system. Some of it was no problem and some of it was chronically problematic.

One such problematic office was a hospital in South Cackalacky. There were 2 signs that would always drop for a few hours and not correctly display their consistent custom message updates. They had many signs and really, just these 2 would be problematic. They had recently put an X-Ray room near those signs. While those rooms are typically properly shielded, we couldn’t help but think that is what was going on. Problem was that the timing was not exact.

We actually had sent a technician on site to kind of monitor and it was a slow day for the X-Ray machine. As we were monitoring, all of sudden, the signs dropped connectivity. We asked questions to the technician: was the X-Ray firing? Was there any other machinery going off? Other Wifi APs that may have stomped out the signal?

The technician then said, “Listen! Do you smell something? I hear something: pop, pop, pop, beepbeepbeep”.  He then went into the kitchen and there were a lot of people, eating lunch and using the microwave. The positioning couldn’t have been more perfect. It was likely the culprit.

“Let’s wait until it dies down in the kitchen and then recheck,” I suggested.

2PM rolls around, the kitchen crowd dies down and everything works great. OK, so the microwave was the problem and not the X-Ray machine (thank goodness)? Somebody comes in late and uses the microwave and down the screens go. I then had my technician test it by turning the microwave on, pushing the screens to report in and no go. He then would wait until the microwave stop and it would connect. Boom! Problem identified. We hardwired the screens (at our expense) and now that problem was solved.

Wrapping this up

So, these 2 anecdotes were probably not very complex investigating but it just goes to show that working with wireless can be pretty unpredictable since we can’t see what is happening and the tools we have to measure, at a simple form, are as good as our cell phones. There is so much “noise” flooding the wireless spectrum that we can attribute some of these problems to that crowded space. Many wireless APs are left to default settings so they will stomp on each other all the time. There are things to help prevent that, including adding the 5GHz spectrum to wireless availability and autoscanning for the least crowded channel. But, there are countless environmental factors, predictable and unpredictable and probably not fully realized (as far as intensity of disruption) that can cause issues. This is why most IT people recommend hard-wiring when available (not to mention the overhead of error checking amongst other things).

Hope you liked the stories and can relate (especially just being in New Orleans, which I love). Scrutinize, and be happy doing it!