🍩 Doughnut Reader 🍩

Time series data are inherently context sensitive, unlike natural languages which follow predictable grammar patterns. The patterns in time series data vary based on context. For example, flight data often show seasonal trends, while electric signals depend on the type of sensor used. There's also data that appear random, like stock data, though firms like Rentech manage to consistently find unlerlying alphas. Training a multivariate time series data would be challenging, but I don't see why not for specific applications.

Why do you think language is so special?

There's an extensive body of literature across numerous domains that demonstrates the benefits of Multi-Task Learning (MTL). Actually I have a whole folder of research papers on this topic, here's one of the earliest references on hand that I feel captures the idea succinctly in the context of modern ML:

“MTL improves generalization by leveraging the domain-specific information contained in the training signals of related tasks" [Caruana, 1998]

I see repetition and structure everywhere in life. To me it's not far fetched that a model trained on daily or yearly trends could leverage that information in the context of e.g. biological signals which are influenced by circadian rhythm etc.

Disclaimer: my background is in ML & bio-signals, I work with time series too much.

The things that we are typically interested in have very clear patterns. In a way, if we find that there are no patterns, we don't even try to do any forecasting.

"The Unreasonable Effectiveness of Mathematics in the Natural Sciences" [1] hints that there might be some value here.

[1] https://en.m.wikipedia.org/wiki/The_Unreasonable_Effectivene...

Why not? There are plenty of time series that have underlying patterns which means you can do better than a total guess even without any knowledge of what you are predicting.

Think about something like traffic patterns. You probably won't predict higher traffic on game days, but predicting rush hour is going to be pretty trivial.

There is potential for integrating ML with time series data in industrial applications (things like smelters, reactors etc.), where you have continuous stream of time series measurements from things like gauges and thermocouples. If you can detect (and respond) to changing circumstances faster then a humans in control room reacting to trends or alarms then potential big efficiency gains...

Operator guidance is often based on heuristics - when metric A exceeds X value for Y seconds take action Z. Or rates of change if the signal is changing at a rate of more than x etc.

So in these areas there exists potential for ML solution, especially if it's capable of learning (i.e. last response overshot by X so trim next response appropriately).

Fundamentally, the pre-trained model would need to learn a "world model" to predict well in distinct domains. This should be possible not regarding compute requirements and the exact architecture.

After all, the physical world (down to the subatomic level) is governed by physical laws. Ilya Sutskever from OpenAI stated that next-token prediction might be enough to learn a world model (see [1]). That would imply that a model learns a "world model" indirectly, which is even more unrealistic than learning the world model directly through pre-training on time-series data.

[1] https://www.youtube.com/watch?v=YEUclZdj_Sc

> I'm curious why we seem convinced that this is a task that is possible or something worthy of investigation.

There's a huge industry around time series forecasting used for all kinds of things like engineering, finance, climate science, etc. and many of the modern ones incorporate some kind of machine learning because they deal with very high dimensional data. Given the very surprising success of LLMs in non-language fields, it seems reasonable that people would work on this.

+1 for “any underlying process”. It would be interesting what use case they had in mind.

Watch this talk from Albert Gu:

Efficiently Modeling Long Sequences with Structured State Spaces

https://www.youtube.com/watch?v=luCBXCErkCs

They made one of the best time series models and it later became one of the best language models too (Mamba).

well... if you look at a language in a certain way, it is just a way to put bits in a certain order. if you forget about the 'language' part, it kinda makes sense to try because why shouldn't it work?

I think there are some generalizable notions of multiscale periodicity that could get embedded into some kind of latent space.

as you say, without knowing anything about the underlying process, we can't predict generally. Some other comments point to contexts in which we do know something about the underlying. For instance, I don't think finance is something where you can apply this kind of stuff.

There was a paper written a while back that proved mathematically how you can correlate any time series with any other time series, thus vaporizing any perception of value gained by correlating time series (at least for those people that read the paper.) just wanted to share

Not really. It's true it would usually need more context than a single series dataset but you can predict broadly accurate-ish bandwidth usage trends just using simple statistical extrapolation, we've been doing that since the early 90s. If you give a model your subscriber numbers and usage data as time series it should be able to tell you quite accurately how much electricity|bandwidth|gas|road traffic levels| metro passenger levels at station Z... you'll be using at 4pm on January 4th 2026.

And like all deep learning forecasting models thus far, it makes for a nice paper but is not worth anyone using for a real problem. Much slower than the classical methods it fails to beat.

They also have Amazon Forecast with different algos - https://aws.amazon.com/forecast/

Something I've had issues with time series has been having to use relatively custom models.

It's difficult to use off the shelf tools when starting with math models.

Curious George here: why are you trying to predict where the mouse is going? :)

What is the latency?

Specifically, its referring to univariate, contiguous point forecasts. Honestly, I'm a little puzzled by the benchmarks.

Even if it was for multivariate time series, the model would first need to infer what machine are we talking about, then its working conditions, and only then make a reasonable forecast based on an hypothesis of its dynamics. I don’t know, seems pretty hard.

I have a few qualms with this app: 1. For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.

2. It doesn't actually replace a USB drive. Most people I know e-mail files to themselves or host them somewhere online to be able to perform presentations, but they still carry a USB drive in case there are connectivity problems. This does not solve the connectivity issue.

3. It does not seem very "viral" or income-generating. I know this is premature at this point, but without charging users for the service, is it reasonable to expect to make money off of this?

> Would you look on a time based graph and buy according to that? Or you rather look at financial data, see earnings, profits?

Things affecting financials happen through time.

LSTM is common for this.

also https://facebook.github.io/prophet/

What do you mean by “observability”?

The next index fund should use AI. What could possibly go wrong?

it doesn't apply. Checkout the Incerto by Nassim Nicholas Taleb.

if I knew it was good, why would I tell you that?

If you have a univariate series, just single values following each other -

[5, 3, 3, 2, 2, 2, 1, …]

What is the next number? Well let’s start with the search space - what is the possible range of the next number? Assuming unsigned 32bit integers (for explanation simplicity) it’s 0-(2^32-1)

So are all of those possible outputs equally likely? The next number could be 1, or it could be 345,654,543 … are those outputs equally likely?

Even though we know nothing about this sequence, most time series don’t make enormous random jumps, so no, they are not equally likely, 1 is the more likely of the two we discussed.

Ok, so some patterns are more likely than others, let’s analyse lots and lots of time series data and see if we can build a generalised model that can be fine tuned or used as a feature extractor.

Many time series datasets have repeating patterns, momentum, symmetries, all of these can be learned. Is it perfect? No, but what model is? And things don’t have to be perfect to be useful.

There you go - that’s a pre-trained time series model in a nutshell

Third paragraph of the introduction of the mentioned paper[1] in the first paragraph of the repo.

[1] https://arxiv.org/abs/2310.10688

My understating is that, while your eye can naturally spot a dependency over time in time series data, machines can’t. So as we did for imaging, where we pre-trained models to let machines easily identify objects in pictures, now we are doing the same to let machines “see” dependencies over time. Then, how these dependencies work, this is another story.

Map matching to a road network might be helpful here. For example, a Hidden Markov Model gives good results. See for instance this paper:

"Hidden Markov map matching through noise and sparseness" (2009)

https://www.microsoft.com/en-us/research/wp-content/uploads/...

Kalman filter

I imagine they're both worse than good old exponential smoothing or SARIMAX.

Prophet was pretty bad so yes, but it doesn't seem much better than ARIMA

how is the series of the first 100 prime numbers a time series ?

Some times series are more predictable than others. Being good at predicting the predictable ones is useful.

For example you can easily predict the weather with descent accuracy. Tomorrow is going to be about the same than today. From there you can work on better models.

Or predicting a failure in a factory because a vibration pattern on an industrial machine always ended up in a massive failure after a few days.

But I agree that if a model is good at predicting the stock market, it’s not going to be released.

and yet we have those huge llamas publicly available. these are computers that talk, dammit