Reading Time: 8 minutes

What Happens on the Device, Stays on the Device: Edge AI

A video doorbell has one job: notice a person and say so. For most of the last decade, it did that job with a detour. The camera compressed a stream, pushed it through your Wi-Fi, across your ISP, into a data center a few hundred miles away, where a GPU decided “yes, that’s a person,” and the verdict traveled all the way back. Round trip: a few hundred milliseconds on a good day. On a bad day, a spinner.

Around 2019, the doorbells quietly stopped doing that. Not all of them, and not all at once. But person detection moved into the doorbell itself, onto silicon that costs less than the retail packaging. The vacuum that maps your living room made the same move. So did hearing aids, wake words, and the camera watching a production line. The devices kept their features and stopped calling home for every little decision.

That migration has a name: edge AI. The reasons behind it are physics and accounting, in that order. Neither one cares what’s trending this quarter.

What Edge AI Actually Means

Strip the buzzwords and the answer to what is edge AI comes down to where the model lives. Take a trained neural network, shrink it until it fits, and run it on the hardware that produces the data: the camera’s SoC, the sensor’s microcontroller, the gateway one shelf above them. The weights sit in flash next to the firmware. Inference runs on local silicon. The network connection becomes a nice-to-have.

“Edge” is a stretchy word, and marketing stretches it daily. A rack in the factory closet gets called edge. So does a phone. So, occasionally, does a regional data center, which is edge only in the sense that Iceland is close to New York. In this article, edge means the device itself or something within arm’s reach of it, because that’s where the interesting constraints live.

One boring clarification that saves whole meetings: training stays in the cloud. You teach the model on big GPUs somewhere air-conditioned, then freeze it, shrink it, and ship it to the field. The device only runs it. Teams that separate those two ideas early argue less later.

The Physics Bill

Start with latency. Light in fiber covers about 200 kilometers per millisecond, which sounds generous until you add everything that isn’t light: the radio waking up, TLS handshakes, load balancers, and the queue in front of a shared GPU. (Fine, the queue is usually the whole story.) A realistic cloud round trip for one camera frame lands anywhere from 80 to 400 milliseconds. A walking adult covers about half a meter in that window. The doorbell shrugs. A robot arm next to a human hand can’t afford to.

Bandwidth is the second line on the invoice. One 1080p camera at a modest 2 Mbps produces about 21 gigabytes a day, close to 8 terabytes a year, and it produces that whether anything happens in the frame or not. Run the detector on the device and the same camera sends a few kilobytes per event. Our engineers watched this math decide an architecture on a municipal project: air-quality, noise, and weather nodes across a city, fed by a small solar panel, talking over LoRa. LoRa moves bytes like they’re billed by the gram. Those nodes summarize on the spot because streaming was never on the menu.

The battery makes the same argument, louder. On most battery-powered devices the radio is the most expensive item in the power budget, the SUV of the whole operation. Sending a kilobyte over a cellular link burns orders of magnitude more energy than running a small neural network that decides whether the kilobyte deserves to exist. We reworked the long-range link on a consumer smartwatch region by region and cut power draw by 60 percent, and the lesson travels well beyond watches: transmit less, decide more.

Then there’s the data that isn’t allowed to travel at all. A camera in a hospital corridor, a microphone in a car cabin, anything filming an EU citizen: the moment raw footage leaves the building, legal gets involved and stays involved. When frames are processed where they’re born and only “fall detected, room 12” goes upstream, the compliance review gets shorter and the sales team gets a slide it can actually show. What happens on the device stays on the device. Your DPO will buy you coffee.

Connectivity itself is the last force, and the most underrated. Mines, ships, tunnels, tractors in the far corner of a field: plenty of products spend half their life where “check the cloud” is a joke. A model that lives on the device keeps working through an outage, a roaming gap, or a backhaul contract dispute, and degrades to “sync later” instead of “sorry, try again.” Nobody writes a support ticket about the feature that kept working when the Wi-Fi didn’t. That silence is the deliverable.

Where the Cloud Bill Bites

The edge AI vs cloud AI argument reads differently at different fleet sizes, so run the numbers before picking a religion. Cloud inference is a subscription: per-device bandwidth, per-call compute, storage that only grows. Edge inference is mostly a purchase: a few extra dollars of silicon per unit, once, plus the engineering to make the model fit.

For a pilot of fifty devices, the subscription wins comfortably. Nobody sane quantizes a model to save forty dollars a month. Somewhere in the low thousands of devices the lines cross. Past ten thousand, the cloud bill becomes a standing agenda item with its own gravity, quietly bending product decisions around it. We’ve sat in those meetings, usually right after the invoice landed.

Two honest caveats. The crossover point moves with model size and data volume, so treat any chart, including ours, as a shape rather than a quote. And edge brings its own quiet costs: shipping model updates across a fleet, telemetry to prove accuracy hasn’t rotted, a bench of test devices that somebody has to dust. The subscription survives in a smaller body under a different name.

When You Don’t Need It

Time for the unpopular part. A fair share of the edge AI applications we get asked to scope don’t need a neural network at all. Vibration above a threshold, temperature drifting out of band, a door open longer than thirty seconds: an if-statement covers these, and the if-statement never hallucinates and never asks for a retraining budget. We’ve seen a team burn a quarter on an anomaly model that a moving average with two thresholds matched to within noise. Ship the boring version first. Upgrade when the data says you must.

The cloud keeps plenty of legitimate territory too. Mains power, relaxed latency, a small fleet, a model that changes every week: run that in the cloud and enjoy one-click deploys plus observability that embedded people only dream about. Our developers build that side as well. The same municipal nodes from earlier report their summaries to a cloud backend, because a fleet only becomes a picture where the dashboards are.

Big models are the third case, and the arithmetic is unsentimental. A 7-billion-parameter language model still wants about 3.5 gigabytes of memory after aggressive 4-bit quantization. That fits a phone or a Jetson. It does not fit a microcontroller with 512 kilobytes of SRAM, and no vendor slide changes that. If your feature needs a frontier model, your feature lives in the cloud, full stop.

The grown-up answer is often both, wired as a cascade. A small, fast model on the device handles the easy 99 percent and stays quiet about it. The rare ambiguous cases, plus a trickle of random samples for honesty, go upstream where a heavier model or a human takes a look. You get local latency and tiny bandwidth on the common path, cloud-grade accuracy on the hard path, and a steady supply of labeled edge cases for the next training round. Most of the systems we respect in the wild are built this way, whatever their marketing calls it.

Sounds like a lot of ways to say no? That’s the point. Edge AI earns its keep inside a specific box: hard latency, scarce power, expensive links, private data, stable tasks. Inside that box it’s unbeatable. Outside, it’s a hobby with a BOM.

What It Takes to Build

Here’s the part vendor decks skip. Getting a model onto small silicon has a discipline name, TinyML, and the workflow is the same at every scale: train big, then shrink until it fits, then find out what you broke.

Shrinking mostly means quantization, FP32 down to INT8: four times smaller, several times faster on the right hardware, and typically a point or two of accuracy left on the table. Quantization-aware training buys some of it back. The toolchain is LiteRT, which every standup on Earth still calls TFLite, with CMSIS-NN kernels underneath on Arm cores and a vendor compiler in front of whatever NPU you picked. It works. It also produces error messages you’ll want to frame.

Then memory. On a microcontroller the model shares SRAM with the TLS stack, the radio buffers, and whatever the RTOS keeps for itself, so a “500 KB model” is a negotiation, not a spec. Zephyr at least makes the negotiation honest, since the linker settles it at build time. On one bring-up our team spent two days shuffling tensor arenas so the BLE stack and the model would stop elbowing each other out of RAM. Nobody puts that week in the demo video, and yet that week is the job. Most of our computer vision work on devices runs through this exact pipeline, minus the glamour.

Debugging changes character in the field. You can’t printf a tensor over LoRa, so telemetry gets designed up front: confidence histograms, input statistics, a ring buffer of the hard cases. Model drift is real and it’s sneaky. The factory swaps a lightbulb, your vision model loses four points of accuracy, and the only witness is the telemetry you did or didn’t build six months earlier.

Updates deserve their own paragraph, because models age faster than firmware. The fleet needs an OTA path sized for weights, staged rollouts, and a rollback story that works when the device is bolted to a roof. We took apart that machinery in a separate piece on large-scale IoT firmware management; one fleet behind it grew to 75,000 devices while update failures fell from 5.8 to 1.7 percent, and model weights ride those same rails. Tooling decides whether any of this ships at all. We relearned that from an odd angle when our engineers rebuilt the interface of NeuroCAD, ORBAi’s design tool for spiking neural networks: a different corner of the field, same rule. When people fight the tools, the model never meets the product.

Add it up and the neural network is maybe a third of an edge AI project. The rest is disciplined embedded work: memory budgets, power profiling, OTA, telemetry. Which is good news for anyone who already runs device fleets, because the hard part is a skill they own.

Where This Is Going

The silicon is moving faster than the discourse. ST shipped the STM32N6 with a neural accelerator inside a general-purpose MCU line. Alif builds Cortex-M55 parts with Ethos-U55 NPUs baked in, and Arm’s Ethos-U85 stretches the same idea toward small transformers. An NPU is turning into a datasheet checkbox, somewhere between the second UART and the CAN controller. Once acceleration costs pennies, the question flips from “can we afford to run the model on the device” to “can we justify streaming raw data at all.”

Our bet, for whatever a vendor’s bet is worth: within a few product cycles the default architecture inverts. The cloud stays in charge of training and auditing, plus the fleet-wide view no single device can have. The device becomes the place where you infer. Most teams will arrive there gradually, one bandwidth invoice at a time.

If you’re weighing it for a product right now, three moves cost little and teach a lot. Run your feature through the five questions in the chart above. Prototype on a sixty-dollar dev kit before committing to custom silicon, and read our harder-nosed checklist on evaluating edge AI for smart devices before the platform shortlist exists. And budget telemetry from day one, because an edge model without telemetry is a rumor with a serial number. If you’d rather stress-test the whole plan against your actual product, that’s our day job: bring the napkin math to our AI and machine learning development team, or just start the conversation. We’ll bring the whiteboard.