Bootladder Engineering

You back in 2015. I created the firmware for a wireless water leak detection product which launched in 2016 and it was acquired in 2020. Production had been steady and stable, but after being acquired there were new interests and direction. Being an IoT product acquiring is all about acquisition, is all about integration, ecosystem, value, chain, et cetera. Not only is there the product and market ecosystem, there is also the internals of the greater organization. For example, whole infrastructure, ops and as we will be talking about here, testing.

a

As the Solo creator of the firmware, I saw the project through the full cycle from the beginning to beta testing to production and launch, to the first part of the bathtub curve curve after launch, to the creation of new spinoff products. All of these stages bring new perspectives and challenges into the equation. One thing here made things very difficult, which is the large scale wireless networking. In fact, some of these challenges were so great that they were intractable to solve in a timely manner and so compromises were made for the good of the business. To me, this is always disappointing. But at scale, some problems truly are intractable. It as the Solo principal working with a small startup, less than ten people with limited resources, many options were off the table. For example, testing at live customer sites. It was absolutely critical during beta testing to have friendly customers willing to let us test on their site. There are so many real world parameters that simply cannot be simulated or isolated to a bench test. But even with such generous access, it was not enough. For example, we had restricted access to walk around and placing our equipment had to be either temporary or, if installed, gone through a process of notifying the client, perhaps having an electrician wire, an outlet or bracket into a drop ceiling. For example, many devices would be going in utility closets, which we needed to request for somebody's help on site. The point being, we had barely enough access and leeway to set up a beta test system. It but any past that was not going to work.

a

So scaling up manual testing on site in a real environment was not going to work. Especially when if we were trying to change things for example a new feature or tuning a parameter of the network stack or placing two devices farther away from each other or behind RF obstructions. For a while we, or I should say I was able to do testing outside. For example, to test point to point range which was theoretically and advertised to be 1000ft line of sight, I was able to find a parking lot with a 1000 foot clearance in the middle of Silicon Valley. This is harder than you might think. I drove around with hardware on my car looking at my laptop. At one point I was stopped by a police officer who ran my ID. But that is just a quick and easy test. Another test I could do outside was an end to end multi hop test. This could be performed by lowering the transmit power of the RF transceiver on a set of devices, let's say ten, and placing them in a line on the road. Even this can be tough in a densely populated commercial area because when you put a device down it is either on the road, on the sidewalk or on somebody else's property. Our office was way too small to do this inside. Generally people are okay with this, especially in Silicon Valley. But again, the theme of these stories is that there is a clear limit of scale and ultimately doing as I've described eventually will not be enough. It.

Now that I have described the situation at hand, now I will discuss the options it what can I do about it? I broke down the problem situation into three categories. There are three approaches which can be done despite all of the constraints simulation, automated testing, end to end and unit testing. Simulation is a classic topic of debate. My humble opinion here is that simulation will not reflect reality when we are dealing with such complicated and dynamic wireless networks and environments. This is of course debatable and there are nuances. But my feeling was that I would really not enjoy the moment after I've spent so much time in simulation to discover that it is completely inaccurate. Unit testing is sort of how do I say this? It the best thing you can do for very limited scope unit testing enough? Unit testing alone is not nearly enough. However, everything that can be captured with unit testing is extremely important to do so and valuable, especially when making critical changes at the late stage of a product lifecycle before production we are all human and mistakes will happen. Unit tests are very good at catching these off by one mistakes. So what remains is automated testing. To me. I would define automated testing to be a physical jig to fix the position and local environment around a device under test, a test harness which exposes communication interfaces and control interfaces to the device under test, and an orchestration. System which can prepare, trigger and execute routines on a device under test, as well as being able to support systems with multiple devices under tests under multiple test harnesses. When I speak of automated testing for an IoT product like this, there is so much more than software here's. All of the techniques and tools used to test web based software is important and useful as of course this product does have a web app. However, that alone is not enough. And because of that, as well as that, these ideas are new and foreign to the executives. Typical executives. What I describe is not something you can simply have like software is. The physical aspects of the system I describe involves acquiring resources, permission and project planning. It is very much not an obvious thing compared to say unit testing where unit testing is something you can just say I want this, you should do this and then it will happen.