Filter self-oscillation happens when the filter's resonance is increased to a point where it begins to generate a tone on its own, even without an input signal. This occurs because the filter's output is fed back into its input, creating a feedback loop. The frequency of this self-generated tone corresponds to the filter's cutoff frequency.