When I was learning DSP, I was surprised by the fact that generating pink (1/f) noise, sample by sample, is not mathematically easy at all.
One practical approach is passing white noise (every sample independent random) through a "pinking filter", which is a sum of a bunch of lowpass filters at different frequencies, so that the sum of their cutoff "knees" approximates the frequency curve of pink noise. Another approach is generating in chunks using Fourier transform from a desired shape. Given the ubiquity of 1/f noise, you'd think there would be a simpler and more direct algorithm, but no.
1/f noise does not truly kill averaging, observe how for increasing f, the noise spectrum looks "white" locally, but with a decreasing noise power for ever higher frequencies.
the 1/f noise at significant power levels is restricted to the lower and lower frequencies, effectively a slowly varying reference ("0") voltage of the amplifier, measuring ADC ground every other sample effectively recalibrates the offset voltage of the amplifier, think of "correlated double sampling".
Effectively measuring in sequence 0V, signal, 0V, signal, ... moves the signal of interest to a higher frequency band, where the 1/f noise is more tame.
When a cliff blocks the way of a vehicle, we don't say "cliffs kill vehicle travel", insteas we just drive around it...
Yup, it has the fun property that you'll get the same error in your estimate of the mean regardless of averaging time (though since it's not stationary this isn't even really that well defined).
(though of course, a random walk means you'll get even worse as you measure for longer...)
This makes me think of the Cauchy distribution, a probability distribution whose average follows the distribution itself rather than converging (hence, the distribution has no "mean" despite being symmetric). Is there any connection here, or is that just a coincidental similarity?
Just as interesting and surprising to me is that this 0-100Hz line is unexplained. Given that signal processing is the foundation of pretty much all digital technology, as well as the analog technologies that came before, I've kind of assumed every segment of a frequency plot be well studied and understood, with half a dozen of names to choose for (doublesine quefrency this, Kowalski-Shannon that...) and a heap of decades old textbooks about it.
I confirm, the low frequency range looks weird on every DFT plot I ever saw, particularly the audio ones. I just assumed it has something to do with ADC and is probably explained on Wikipedia. It's literally one of the last place on Earth when I'd expect to find unsolved mysteries.
It's not completely unexplained. Roughly speaking you can get that power spectrum in the limit when you are adding up many different events where the magnitude of the event is inversely proportional to its likelihood (and in practice, there is a limit to the magnitude of the events that will cause it to level off at some point, but for some processes this is not measurable even over decades). The main mystery in most cases is what exactly is the physical process that is causing it. For some electronics it looks like it is due to trapped charges sometimes tunneling around, but it doesn't explain every case of it in electronics let alone everything else. Convective thermal effects can also be a good candidate in a lot of systems, since turbulence also has 1/f noise properties.
(Also, the low frequency range on a DFT can look weird for reasons other than noise: it'll also tend to rise up if there's any longer-term structure to the signal as well, so you need to be careful interpreting them blindly if you're trying to measure noise)
Low frequencies are studied quite extensively, especially the mHz-10Hz region for noise characterization of solid state materials. 1/f is quite well studied depending on your field. In semiconductor physics, for example, one possible explanation is electrons trapped in defects or on charged islands and then slowly trickling down. Of course this explanation cannot be used in other fields where 1/f noise shows up as well. The problem is that no model gives a satisfying answer as to why it occurs therefore its unexplained. Lack of model doesn't mean that you can't engineer your way around 1/f noise for example the chopper amp works so well because it shifts away towards frequencies above the 1/f cutoff.
Compounding the mystery is that it crops up everywhere, not just electronic noise plots. But outside of a few systems such as electronic noise in a laboratory setting, it's phenomenally hard to measure. For one thing, to get into the 1/f domain, you have to measure things for a long time. And the noise measurement itself is noisy. And the number of things that you need to control, such as environmental conditions, increases.
So its existence is often largely treated as an empirical rule of thumb rather than having a specific physical cause.
The other thing to note is that the noise plot in a dataset is probably a curve fit.
Wikipedia has an article with some other information on pink noise (a more common name), including a random generator: https://en.wikipedia.org/wiki/Pink_noise. Some music generation algorithms use pink noise, as it (supposedly) strikes a better balance between randomness and predictability.
Pink noise is also perceptually flat noise, because it contains the same energy in each octave (or decade). 'truly' white noise (equal energy for equal bandwidth) tends to sound quite tinny/hissy in comparison.
I went through a phase of using ~whitenoise while working to block out distractions.
Actual white noise indeed sounds really bad and grating. The best for me was a mix of pink and brown noise, pink for an ~equal baseline and brown to make it sound a little more mellow.
I suspect most/all generators meant to block out noise do something similar. It really sounds pretty bad without that, especially for anything more than a few seconds.
Yeah. pink noise is often confused with white noise in audio because it looks flat on a lot of equaliser displayers (because they show energy per octave instead of energy per Hz).
One practical approach is passing white noise (every sample independent random) through a "pinking filter", which is a sum of a bunch of lowpass filters at different frequencies, so that the sum of their cutoff "knees" approximates the frequency curve of pink noise. Another approach is generating in chunks using Fourier transform from a desired shape. Given the ubiquity of 1/f noise, you'd think there would be a simpler and more direct algorithm, but no.
the 1/f noise at significant power levels is restricted to the lower and lower frequencies, effectively a slowly varying reference ("0") voltage of the amplifier, measuring ADC ground every other sample effectively recalibrates the offset voltage of the amplifier, think of "correlated double sampling".
Effectively measuring in sequence 0V, signal, 0V, signal, ... moves the signal of interest to a higher frequency band, where the 1/f noise is more tame.
When a cliff blocks the way of a vehicle, we don't say "cliffs kill vehicle travel", insteas we just drive around it...
(though of course, a random walk means you'll get even worse as you measure for longer...)
I confirm, the low frequency range looks weird on every DFT plot I ever saw, particularly the audio ones. I just assumed it has something to do with ADC and is probably explained on Wikipedia. It's literally one of the last place on Earth when I'd expect to find unsolved mysteries.
(Also, the low frequency range on a DFT can look weird for reasons other than noise: it'll also tend to rise up if there's any longer-term structure to the signal as well, so you need to be careful interpreting them blindly if you're trying to measure noise)
So its existence is often largely treated as an empirical rule of thumb rather than having a specific physical cause.
The other thing to note is that the noise plot in a dataset is probably a curve fit.
Surely there is some connection to entropy.
(The frequency is 42.3338481 nanohertz.)
Turns out this property is not unusual. There are many such pairs - there’s a reasonably well-known journal paper with a construction technique.
If you want to be boring you could call it the uniform distribution for log frequency.
Actual white noise indeed sounds really bad and grating. The best for me was a mix of pink and brown noise, pink for an ~equal baseline and brown to make it sound a little more mellow.
I suspect most/all generators meant to block out noise do something similar. It really sounds pretty bad without that, especially for anything more than a few seconds.