How We Test Smart-Home Products
Last updated: 2026-07-13
A product that works for an afternoon is not necessarily a good smart-home product. We test for the ordinary frustrations that show up later: a missed automation, a router restart, an app change, or a household member who simply wants the lights to work.
Testing philosophy: real-world and long-term
We begin with normal use rather than a glossy demo. Devices stay installed through routine schedules, power and network interruptions, firmware updates, and changes to the surrounding system. We repeat the useful actions—switching lights, locking doors, checking sensors, and running automations—because reliability is earned over time, not claimed at setup.
Our test environment
Our working rig is a lived-in home network: a Wi-Fi router with separate main and guest networks, Ethernet where a hub permits it, and Home Assistant running on a Raspberry Pi 5. We pair devices directly and through vendor bridges, then compare Matter over Thread, Zigbee, Z-Wave, Wi-Fi, and Bluetooth paths where the product supports them. The bench also includes a Thread border router, a Zigbee coordinator, a Z-Wave 800-series USB controller, mixed Android and iPhone household use, and deliberate internet outages.
What we measure
| Criterion | What we look for |
|---|---|
| Reliability | Whether commands, schedules, and automations work repeatedly without unexplained dropouts or repair work. |
| Latency | How quickly an action reaches the device in everyday use, including when the cloud or network is under strain. |
| Local control | Whether essential controls and automations still work on the local network when the internet is unavailable. |
| Privacy | What account, permissions, telemetry, and third-party services are required—and whether those demands are proportionate. |
| Ten-year livability | Whether the product can remain useful, repairable, and interoperable as apps, subscriptions, and ecosystems change. |
How the scoring rubric works
We score the criteria above as a whole rather than letting one clever feature hide a weak foundation. Reliability and local control carry the most weight; latency, privacy, setup burden, and the explicitly named ten-year livability criterion decide whether a product remains recommendable. A simpler device that is dependable and easy to leave behind for a guest is often the better choice.
What disqualifies a product
Some shortcomings are not trade-offs we can recommend around. A product may be disqualified for being cloud-only for essential functions, requiring a subscription to keep basic hardware working, lacking a credible firmware-update path, collecting unnecessary personal data, or failing repeatedly during normal use. We also step back from products that cannot explain their protocol support or make a household dependent on a single abandoned app.
Who does the testing
Testing is led by Priya Nandakumar, Founder & Lead Reviewer. Her background in network and systems engineering informs the failure testing, but the conclusions are written for people who do not want to become network administrators to turn on a lamp. The editorial team reviews the notes, compatibility claims, and recommendation before publication.