When working with networks, especially wireless networks, it is important to see what's going on. Communication links always have two sides, at least. When one side is a third party and is trusted to work, things are straightforward.
But oftentimes as product developers, we develop both sides of a communication link. For example, a mobile phone app and a wearable device. Both the the app software and the device firmware are custom. In this discussion, I am talking about many more than two devices.
The project I'm talking about here is a wireless water leak detection system for commercial buildings. It's a large scale mesh network over the IEEE 802.15.4 protocol. The network architecture was not strictly mesh. We actually called it a hybrid mesh. And by the way, my name is actually on a patent for this ;) .
The network architecture consists of a single gateway to the Internet, a proper mesh network of routers, and then each router is the hub of a star network of sensors. So the minimal complete end to end system has three devices, two communication links, and two messages to get across.
I created all of the application firmware on all of these devices. I did not create the underlying network stack, although I did need to be intimately familiar with its source code implementation as well as needing to tune parameters and compile the stack.
Once you have control of the stack and are able to send and receive point to point messages, it is very straightforward to develop the application on top of it. Most IoT devices, particularly sensors, are extremely simple. All they do is read the physical sensor and communicate the readings.
However, once you add real world constraints such as battery life, message throughput, range, and harsh field environment, things get complicated very fast. Even a perfectly functional application can be rendered useless with no recourse. Solving problems from these added constraints necessarily involves dealing with the lower levels of the stack, and when you are working down there, it becomes harder to see what's going on.
This is where the sniffer becomes a critical and indispensable tool.
We were working with Atmel (now Microchip) RF transceivers, and we were actually given a sniffer solution by Atmel to help us at the early stages of the project. Basically, you take an off the shelf transceiver module, flash it with a binary provided by the vendor, and then there is a Wireshark interface to read the data from the sniffer. This worked great for most use cases because not only is Wireshark very useful already, there also was plugin functionality for Wireshark to be able to unpack, decode and display header information about received packets ie. frames. Even less technical colleagues were able to make use of this Wireshark system, for example, collect data from different field sites and report characteristic information to me.
However, this solution reached its limits, for two reasons. One is that there was a limited supply of these hardware adapters. One colleague and myself had one piece each. As the scale of the network under test increased, I needed more than one.
Additionally, I needed remote access to the information. For example, I might want to place the sniffer in a remote location and be able to look at the readings from my main workstation.
There is also the obvious fact that the product we are developing is itself a wireless RF transceiver. A sniffer is basically a transceiver without an application. And over the course of my time working with this project, I have literally boxes full of these devices. So to me, the idea of purchasing more of these off the shelf modules and loading them with an unknown binary that only can communicate with Wireshark is unacceptable. So I set out to create a sniffer firmware which would run on any one of the devices in our product ecosystem.
Conceptually, a Sniffer is just a trivial application that instead of doing things when it receives a RF frame, it does nothing but send the exact received frame out to a serial port. However, in this case there is one thing required of a sniffer, which is called Promiscuous Mode.
The transceiver we were using handles more than just the physical layer. The chip takes care of stuff at the bottom of the stack, so there is less CPU overhead and less for the application programmer to worry about. This is common in modern, highly integrated transceiver modules, eg. Bluetooth modules and System on Chip's.
Promiscuous mode tells the transceiver to accept every frame, even frames that are not addressed to it. Fortunately, there is a register in the tranceiver where this mode can be enabled. This is basically all that needs to be done on the the transceiver side.
Because the sniffer itself does nothing with the frames it receives, that means any further functionality needs to be handled on the host side. For example, unpacking and decoding the frame and the layers of headers inside of the frame.
The PHY and MAC layers are standard defined by IEEE. The network layer is a Atmel specific implementation and then the application layer is of course specific to the product being developed. Thus, this sniffer can be used on any 802.15.4 transceiver, (in the 900MHz band in this case), even ones completely unrelated to the product I am developing. So there is the common and the specific but parsing out the fields and headers is the same process. In any case, it is just a matter of what is to be held proprietary and secret or not.