My sleep is unpredictable. Not in a technical sense, but a colloquial one. To be literal, I have no idea how to predict my sleep. I just as often sleep through the day as I do through the night. My sleep itself, as far as a sleep study can tell, is normal. I can vaguely say, 60% confidence, if I'm likely to fall asleep in a given 3-4 hour period, and occasionally I will be fairly sure, 80% confidence, 6-10 hours beforehand, of a 1-2 hour period. I can similarly predict the length of my sleep (which is relatively normal--generally distributed 7, 8, 9.5, 13 hours at .1, .4, .6, .9).
My sleep is seriously disturbed. Without understanding the process behind my sleep, without being able to predict it days beforehand and understand the variables behind it, I find it impossible to wake up at a consistent time every day (+/- 8 hours), despite years of trying, which makes it extremely hard to hold down a job, or do dozens of other normal things. There could be a profession that I could make my sleep work with, but I'm still searching for it.
So I ask you readers: Is there some sort of pattern detecting thing, whose name perhaps includes something like "markov" or "kolmogorov" or "bayesian", that could automatically take a time series data and predict the next values based on an unknown, complex model?
So, I could like enter the times I go to sleep and wake up, and when I have caffeine or I exercise, and maybe other things, and it would puzzle out how my sleep works and forecast my next few sleep cycles?
To have an accurate tool like that would transform my life.
"Hidden Markov models" comes to mind, but at first glance I don't see how a sleep model would count as a Markov process, given that you have to factor in sleep debt, time of day (because of sunlight), and perhaps other variables. But then I know nothing about HMMs.
Also, this is my first post. Is this the sort of thing that goes better in LessWrong or Less Wrong Discussion?
I'd start by playing around with the data, trying to find different ways to organize it, identifying and defining relevant variables, and making graphs to look for patterns. Then you can start to look at relatively simple tests of relationships, and once you have a better idea of how the data are structured and what sorts of relationships seem to be present then you can try fancier math (if necessary).
For instance, you linked to your data organized by sleep cycle, starting a new row of data each time you fell asleep. I made a spreadsheet here using that organization (but slightly different formatting which was easier for me to work with) and used it to make this graph, which shows how much time you slept in blue and then the following amount of time you spent awake in orange, and then starts the next sleep period on the row below it. Two things that jump out from the graph are that you have lots of short cycles (under 5 hours, even) and that most of the cycles are under 24 hours. If you tend to find yourself on a cycle of over 24 hours (going to bed later and later each day), that must be from sleeping multiple times in a single calendar day.
Another way to organize the data is by calendar day. You could have one row for each calendar day (possibly starting each day at a time other than midnight, like the time when you are most often awake or when you're most often asleep) and make a two-color graph of when you're asleep & awake, like the one I linked to (except this time it would be a rectangle). With the data organized that way, you could look at questions like: how often am I awake at each time of day? If I am awake at a certain moment, how likely am I to be awake 24 hours later? If I am asleep? Or x hours later - you could make a graph where the x-axis is the number of hours later, and the y-axis is the probability of being awake that long later (give that you're awake/asleep at t0). How does this change depending on time of day?
There are also various correlations that you could look at, or graphs that you can plot of 2 variables, like bedtime, length of sleep, wakeup time, length of awakenness, sleep debt (percent of time asleep during previous 72 hours? during previous 4 sleep cycles?). You might need to do something about the short cycles (leave them out? combine them with adjacent sleep/wake periods?). Here is a graph of amount of sleep vs. bedtime, which seems to show a pattern (you sleep longer when you fall asleep earlier) if we ignore the short sleeps.
Or, come up with some definition of a "normal" sleep cycle (e.g. one lasting close to 24 hours, which includes close to 8 hours of sleep) or a normal day (e.g. sleeping at least 6 hours from 10pm-10am and then being awake for at least 9 hours from 10am-10pm). How common are these "normal" cycles/days, are there any patterns to when they occur, do they come in streaks, and what seems to happen to precipitate the end of a streak of normality?
There are a bunch of other things like these that you could try. If you've already done some, you could share them here.