How did this system actually track calories? Detecting that the user is consuming food seems like a fairly solvable problem; tracking what they're eating - which is going to have order-of-magnitude effects on caloric intake - seems like a much harder problem.
I can't seen any obvious ways to do it, other than by requiring significant user input, and that would rather negate any benefits that a passive, low effort tracker had.
Am I missing something here?
(Was it a "beep to remind you not to snack" device, rather than a calorie tracker?)
It didn't track calories. It tracked bites. The problem we were attacking was eating awareness and frequency. It might've been possible to track food types too but we didn't get that far.
(Was it a "beep to remind you not to snack" device, rather than a calorie tracker?)
Yes.
Even a tool like this was useful enough that people repeatedly and independently, on their own initiative, asked us to build it for them. They were all from the same demographic: women, often stay-at-home, between the ages of 30 and 50. Their problem, as they saw it, wasn't eating calorie-dense food. It was frequent unconscious snacking.
Thank you for sharing this story!
I think there's a small bit of illusion of transparency going on here. You mention "IMU" ("inertial measurement unit"?) twice but don't explain the acronym, so I think some readers didn't necessarily understand what you'd built, or what the value-add is.
Now, that also means I am not quite sure whether I've understood you correctly, but I think it's supposed to be:
Picture a tracker or smartwatch, e.g. a Fitbit. Or even some smartphones. These all include various sensors, like accelerometers and gyroscopes. The combination of the latter two allows tracking time-series data of body movements. If you wear a tracker around your wrist, that includes arm movement, among other things.
People wear these trackers for various reasons, e.g. to change their behavior. The features offered by popular trackers like the Fitbit depend partly on the included sensors, and partly on the algorithm used to interpret the sensor data. For instance, if you were able to identify the signal in the noisy data, you could interpret certain accelerometer & gyroscope data as "you're taking a bite of a snack".
As I understand it, there is not in fact a commercial bite tracker, so Lsusr is saying his startup found a way to reliably identify when a tracked user takes a bite of a snack, and then to alert the user of this behavior, in a way companies like Fitbit have not managed to. And if you have something like that, you have a prototype of a valuable product, e.g. because it could help people become conscious of their unconscious bad habit of snacking, which in turn would make it easier to stop.
Lsusr, I understand that you've personally decided to stop this project, because your time and resources are limited and better spent elsewhere. That said, if you still think this is an unusually good idea but that you can't execute on it, consider posting a more technical version of this story on the EA forum. Most likely nothing will come of it, but very occasionally there are people in search of projects. I haven't done the EV-calculation myself, but this project is the kind of thing that, if it actually worked, sounds like it could actually have a somewhat competitive EV, given the abysmal efficiency of other obesity interventions?
If you have patents or would (for the right price) be available to consult others who gave this project a try, mention that, too.
Your description of the invention is precisely correct. Cross posting on the EA forum is a good idea.
I'd enthusiastically do part-time contract work to help make a machine like this happen again, for the right price (which might be surprisingly high or low, depending on one's expectations). It'd be way faster and cheaper for a company to hire me than to do it themselves from scratch. It'd be less risky too. My mental model is that the bottleneck on a project like this is good founders/executives. It is a question of leadership and will.
So, you and your team spent six years of effort working full time for no pay (what did you even eat then?). You developed a product that worked just great, was in demand and could make a difference in fighting obesity by making a beep whenever the wearer eats. But even though the product was ready - "just put it on and good to go" - and you can easily reconstruct it, you and your whole team decided to abandon it and part ways. Because you simply aren't that into diabetes prevention, and also your time is limited and you have more important things to do. But you would enthusiastically do part-time contract work on this project again.
I feel like this story doesn't quite make sense. If the company was doing so well and you just didn't want to run it anymore, why didn't you sell it?
“Turn the continuous datastram into discrete symbols by feeding it into a random forest classifier.”
Would you be willing to say any more about this? It sounds really interesting.
The LCSS algorithm (from which WarpingLCSS is an optimization) operates on a sequence of discrete symbols such as letters or nucleotides. This is in contrast to a neutral network whose natural inputs come from a continuous vector space. IMU data is continuous. We needed to bucket it before feeding it into LCSS. The random forest buckets continuous data into discrete categories. We tried support vector machines too but random forest classification worked better.
Aha - so like how something like word2vec maps discrete symbols into a shared vector space, this does the other direction, mapping vectors into discrete symbols. Never thought about that direction; thanks!
Did you really throw away the software and not keep it in a VCS or on an extra storage device? I feel a sort of pain thinking about it - that if it really worked as well as described, it may not at all be easy to rebuild.
We kept the software. It's somewhere on the cloud in a VCS. Not sure about the data. This was an embedded system. The hardest, most frustrating part to maintain was hardware integration. The prototype was a specific wearable device attached to a specific laptop. I erased that laptop and gave it away to a relative.
In a perfect world, I'd redo this project as a contractor for an established wearables company. I'd do the machine learning, they'd do the hardware and we'd outsource the data annotation to Mechanical Turk. (Data collection is easy. The data bottleneck is annotation.) But that takes industry connections I don't have.
My other comment is that you probably didn't succeed as well as you thought you did. I am taking your story at face value - that your model was spookily accurate, that it worked way better than you could reasonably expect, etc. But scale matters. Many tech prototypes work perfectly at small scales but would fail if you had built a few thousand or million hardware instances and tried them with the user wearing them across the full range of human activity and cultures.
But say you did, and your model that's cheap enough to train on a laptop and run on an embedded CPU is perfectly accurate. How does this solve the actual problem? Obesity is caused by humans getting incredibly strong, insidious urges to eat (and maintain their weight once fat). The actual root cause is probably poisons in the food supply, explaining the delta between specific countries of genetically comparable people, but in any case, how does this gadget help?
In The Circle everyone wears cameras all the time and thus gets peer shamed into not eating, but this has issues other than the privacy ones, such as most people not being attractive enough/popular enough to have any peers to shame them.
I've on many occasions tried the myfitnesspal calorie counting. It does work but the longer you do it the stronger your urges to restore your missing mass. It's like compressing a spring. Significant weight loss is very difficult. It's not impossible, I've been -35 lbs for 4 years now, but I have not been able to get from being merely 'overweight' to 'normal' BMI.
If this technique you're talking about is actually as good as you say it is, and can run very performantly on regular PCs, I would be willing to pay you to help build a PoC for a particular problem class that this kind of pipeline would add a ton of value to. Although to be honest, I'm kind of curious why your cofounders decided to quit at this particular moment, if people liked your product so much and it worked so well.
I'm obese and struggle with weight loss, so this is a particularly sad story to hear. My experience makes me think a lot of my issues could be improved just by having someone, like, standing 24/7 by my side going "hey" when I go to buy ice cream, stress-eat etc.
Would you be willing to make the pieces you made available to someone who wanted to pick up where you ended? I'd probably not be that person, though (because don't have the spoons).
I don't recommend anyone use that actual literal hardware we used. Hardware advances fast and some of the components we used are no longer manufactured by Nordic Semiconductor. It would be better to start from scratch with new hardware. The hardware was not complicated. It was just an industry-standard IMU attached to an industry-standard microcontroller attached to a battery, a vibrating motor and a charging light.
If someone wants to take on this project the thing to steal from my experience would be the machine learning architecture. That's where all the hard technical challenges were. I think I have left behind enough hints in my story to save them 80% of the algorithmic work. Anyone competent enough to pull this project off could probably muddle through the remaining 20% on their own, but I recommend they hire me instead.
Not on an Apple Watch. At least, not on an Apple Watch the one time we tried porting our code to it. But it's theoretically possible to implement the algorithm on any wearable with an IMU which meets basic power efficiency requirements and which let's you run raw C close to the metal. Apple didn't let us run the app with the necessary privileges.
In the very beginning we wrote our software for Android smartwatches but that approach caused several problems.
The problem is that no wearable platform good enough was already owned by enough of the population. As counterintuitive as our approach might seem to a software developer, it made more sense for us to manufacture our own hardware.
You sound very confident your device would have worked really well. I'm curious, how much testing did you do?
I have a Garmin Vivosmart 3 and it tries to detect when I'm either running, biking, or going up stairs. It works amazingly well considering the tiny amount of hardware and battery power it has, but it also fails sometimes, like randomly thinking I've been running for a while when I've been doing some other high heart rate thing. Maddeningly, I can't figure out how to turn off some of the alerts, like when I've met my "stair goal" for the day.
Only eating with a fork. A full system would require more data than that. We tested on real people in real-world conditions who were not part of the training dataset. If someone ate in a different style we could add just a little bit of annotated training data for the eating style, run the toolchain overnight and the algorithm would be noticeably better for that person and everyone else. The reason why I'm so confident in our algorith was because ① it required very little data to do updates and ② I had lots of experience in the field which meant I knew exactly what quality level was and wasn't acceptable to customers.
To update the code in response to user feedback we would have to push the new code. Building an update system was theoretically straightforward. It was a (theoretically) solved problem with little technical risk. But it was not a problem that we had personally built a toolchain for and the whole firmware update system involved more technical maintenance than I wanted to commit myself to.
Obesity is one of the world's greatest health problems. It contributes to everything from diabetes to cancer. Obesity is hard to prevent because tracking calories requires effort. It's too easy to eat absent-mindedly if you're a middle-aged woman in charge of your household's food. Manually logging your food intake is hard. Food tracking should be automatic, like a FitBit. If we had a cheap machine that could effortlessly track food intake it would save many lives and improve the quality of life of even more people.
I talked to the lead engineer at company that is a leading manufacturer of wearable electronics. The company is worth tens of billions of dollars. His manager tasked his team to make a device that used IMU data to detect when a person is eating. He failed.
Me and my team succeeded. Our device worked so well our test subjects thought we were cheating. They thought we had a man behind the curtain. It was all software. My friends and family would try to fake eating by pantomiming it and the device wouldn't go off until they took a bite for real. Our algorithm was so good it often detected when people were eating before they even lifted their fork. We had to program a delay into the alarm system so the predictive signal didn't disrupt the user's actual behavior.
We trained our machine learning model on six hours of annotated data. The algorithm we deployed was so small and efficient you could run it all day on a Nordic SoC the size of a fingernail. I trained the whole thing on a laptop from 2016.
Our system was perfectly general. You could dump a different hand movement into it and the algorithm could detect the new behavior too. All you had to do was strap a sensor to your test subject, video record them and then tell the computer where in the video the new hand movements began and ended. Press one button and my toolkit would turn that training data into a binary executable ready to be deployed on your embedded system. It did not need to be customized for individual users (except left-handers). Just put it on and you're good to go. Individual units cost us $30 to produce. Large-scale manufacturing could drive the price much lower.
You might wonder "Where is this magical machine?" I threw it away.
I could reconstruct it. Strap an IMU onto peoples' wrists. Manually annotate where fork movements begin and end. Apply a complimentary filter and a moving average filter. Normalize everything. Turn the continuous datastram into discrete symbols by feeding it into a random forest classifier[1]. Dump the random forest classifier into the recently-invented WarpingLCSS algorithm[2]. Add some threshold cutoffs for trigger and reset. Use Facebook's Adaptive Experimental Platform to tell you what hyperparameters to use. Port the whole thing to C. Deploy it on a microcontroller.
We had experience selling consumer electronics. We had the hardware for hundreds of these wearable food trackers just sitting in boxes.
But startups are hard. We had been working full-time for (basically) no pay for six years. We shut the project down and parted amicably. We all had different reasons for leaving. Personally, I didn't want to spend the rest of my life running a diabetes prevention program. The reason our algorithm could be trained on my laptop instead of a supercomputer was because I had invented a machine learning trick so new it had been invented independently and published in an academic journal within the previous two years. I had more tricks where that came from. I wanted to find out what would happen if I pushed my machine learning skills as hard as I could.
I see obese people everywhere. People die every single day from a health problem I could have done something about. I was the only person on Planet Earth who could have built a passive eating tracker with the technology available in 2020. I didn't.
There's a tiny little trick we discovered here that makes a big difference. ↩︎
There's a little more secret sauce here. ↩︎