Screen Shot 2026-03-02 at 4.18.40 PM.png
- 444.29 KB
(2003x1640)
So I've been working on a project to re-implement the VOCALOID1 engine. I'm basing it on the description in Jordi Bonada's PhD thesis "Voice Processing and Synthesis by Performance Sampling and Spectral Models" and not the original papers as the former is more detailed, easier to follow, and also describes the VOCALOID2 engine. After a lot of trouble with getting TWM f0 estimation to work, I've finally gotten to implementing MFPA. And amazingly, it seems to have worked first try. Compare my results: https://i.ibb.co/dsvgv0fd/Screen-Shot-2026-03-02-at-3-54-48-PM.png To the results in the study: https://i.ibb.co/C3fjdWVd/Screen-Shot-2026-03-02-at-3-55-09-PM.png
I was thinking about why the modulation occurred in the case of the overlap-and-add method. Thinking about it, when the fundamental frequency is stationary and the MFPA onsets are perfect, the trapezoidal window function is equivalent to a weighted average between two adjacent voice pulses over the duration of twice the border interpolation size. However, when the MFPA onsets are inaccurate, or even just when the fundamental frequency is non-stationary, this is no longer true. Even worse, thinking about it from the weighted average point of view, the sum isn't necessarily one everywhere anymore, hence the modulation. I then devised a method that would not result in modulation. This method works by first synthesizing the 'inner' portion of each pulse (by 'inner', I mean starting at the end of the border interpolation at the start, and ending before the start of the next border interpolation towards the end of the pulse). Then, for the gaps in between each pulse, we calculate each sample value by a weighted average of two values. These are values are the values of each voice pulse at that time. Since the gap extends beyond the boundaries of each voice pulse, we use the periodic nature of the pulses to compute the effective position in the voice pulse by taking the position modulo the period of the fundamental frequency at that voice pulse. The fundamental frequencies of each of the voice pulses may differ, so we actually change step in time linearly. At each end of the gap, the step size for the voice pulse it is next to is one sample in time, while the step for the former voice pulse is the equivalent of one sample in the latter voice pulse relative to the former's fundamental frequency (e.g. if the second voice pulse has twice the fundamental frequency as the first; the step size for the first would be 2 and tep size for the second would be 1, at the end of the gap). For the start of the gap, it is the same except relative to the first pulse having a step of 1. In between, we the step size interpolate linearly. It is worth noting that in the ideal case where the onsets are exactly correct and the fundamental frequency is stationary, the result of this approach is the same as using the trapezoidal window. FREQUENCY WARP-CORRECTION - As noted in Bonada's thesis, WBVPM assumes that the fundamental frequency is stationary within each pulse, however this is not actually true, and that the artifacts from this are particularly apparent for low fundamental frequency voice signals, because each period of the signal is longer in time and thus the internal state of the system has more time to change. One of the changes that can happen over time is modulation of the fundamental frequency. This can be thought of actually as a time-domain remapping function that distorts each voice pulse according to a continuous fundamental frequency trajectory. I discovered a way of correcting this, largely by accident as I was thinking about solving the modulation issue I discussed in the previous section. I was thinking about how I proposed changing the step size linearly in the gaps between the 'inner' pulses. I was thinking, we have a discrete sequence of fundamental frequencies. So, what if instead of changing the step size linearly, we instead created a spline from the fundamental frequencies and instead changed the step size based on that? Then I realized that we could also use this for the whole voice pulses and just sample everything with a step size based on the fundamental frequency trajectory. I then realized that this would actually act like the distortion from changing parameters within each voice pulse, at least in the synthesis stage. Further more, since we are already computing splines for each voice pulse to downsample it, this comes at very little additional computational cost. (continued in the next post)
However, the voice pulses in analysis are already distorted. So then I already we can do the inverse resampling in the upsampling stage of WBVPM analysis to correct for non-stationary frequency, then it is redistorted according to the transformed fundamental frequency trajectory in the synthesis stage. This makes this method effectively invariant to modulations in fundamental frequency, so long as the modulation is less than the fundamental frequency and it is modeled well by the spline, which should be the case for modulation period is at least several voice pulses. PITCHED/UNPITCHED DECOMPOSITION - As mentioned in Bonada's thesis, WBVPM only models sinusoids, and thus residual in the input signal is encoded as flucuations between the spectra of voice pulses. I devised a post-processing technique to separate the voice pulses into sinusoidal and residual components. I have not actually implemented and test this yet, because as it is post-processing step, it will not improve the residual level, and in fact, will probably make it worse. The benefit of it is in the transformation stage, which is much harder to quantify and which I have not finished implementing. However, I believe this approach should work. The technique works as follows: a) First, for each voice pulse, and then for each harmonic of its spectra, we compute a spline based on the values of the amplitude of that harmonic in the voice pulse as well as a fixed number of surrounding voice pulses. b) Since the time delta between voice pulses can vary, we then resample each local harmonic spline with fixed steps in time. c) We compute the fourier transform of these resampled local harmonic trajectories d) We apply a low-pass and high-pass filter to separate it into low-frequency and high-frequency components. e) We then apply the inverse fourier transform to each of these. We can then sample the low pass trajectory at the time of the voice pulse to get the amplitude value of the denoised harmonic for that voice pulse. The same can be done for the high pass trajectory to obtain a pseudo-pulse representing the residual. These residual voice pulses can then be synthesized using the WBVPM synthesis method to obtain a time-domain residual signal which can be processed separately from the main harmonic signal. A significant source of error in this process presumably would come from the resampling step. This can be decreased by using a smaller time step, at an increased computational cost. However, the error could probably be greatly reduced by first calculating the difference between the original amplitudes and the amplitudes at the same times in a spline computed from the resampled harmonic spline before applying the band filters, this difference can later be added back to the low-pass amplitude trajectory. The denoised harmonic phase can also be computed via the same method, using Bonada's method for unwrapping phase across both frequency and time. The residual phase can be calculated by taking difference of the original phase from the denoised phase and dividing it by the residual amplitude. RESULTS: I have tested these improvements and obtained the following results for the aforementioned audio sample: Original WBVPM: -36.355dB Warp-correction improvement only: -36.74595dB Warp-correction & Resilient border interpolation in synthesis: -37.41177dB More research is needed to properly evaluate these improvements across more samples with more variety, and to see if these techniques still result in improvements with more accurate pitch and MFPA estimation and with proper handling of unvoiced/voiced frames.
Bump for visibility, merged with original thread.
I feel like I've been going insane over the last day. I've scoured every paper mention Excitation plus Resonance, but at least as far as I can tell, none of them actually mentioned the procedure for actually estimating the parameters of the EpR resonances. Even the expired patent for EpR doesn't seem to mention it. After thinking about it for a while, I attempted to come up with something. No idea if it will actually work though. The procedure I came up with is as follows: a) An amplitude spectrum representing N discrete sampled amplitudes of sinusoids in a frame or voice pulse is calculated such that for every index i in [0, N), Amp[i] is estimation of the value of the amplitude of a sinusoid having frequency i*F at a time or over a time period, where F is a constant frequency step. In the case of WBVPM, F is the fundamental frequency. b) A logarithmic amplitude spectrum comprising an array of N elements is created such that LogAmp[i] = log(Amp[i]) for i in [0, N), where N is the number of discrete values in the amplitude spectrum and Amp is the amplitude spectrum. c) A cubic natural spline LogInt comprising N - 1 segments is created from the values of the logarithmic amplitude spectrum and positions represent the frequency values of the sinusoids estimated in the amplitude spectrum. d) An array capable of holding up to 2N-2 objects representing the resonance parameters (frequency, amplitude, and bandwidth) is created and it's initial length is set to zero. e) For each segment S of the spline LogInt having index i and parameters offset o; span h; and coefficients a, b, c, and d (representing the cubic, quadratic, linear, and constant values, respectively); the following is performed: e) 1) The derivative of S is computed. e) 2) The roots of said derivative is computed, and for each of them, the following steps are performed: e) 2) a) It is checked whether the root, when the offset o and span h is accounted for, lies in the range [o, o+h] of segment S. If it does not, it is discarded and the further computation is skipped. e) 2) b) The value of the second derivative of the segment S is computed at the root. If it is not negative, the root is discarded and the further computation is skipped. e) 2) c) Then a resonance parameters object is appended to the end of the resonance parameters array, and that array's length is increased by one. The frequency value is set to the root. The amplitude value is set to the difference of the interpolated logarithmic amplitude spline value at the root and the value of the source curve at the root. The bandwidth value is set to the reciprocal of the negation of the second derivative of the segment at the root. f) An iterative refinement process of the resonances is performed for M iterations. For each iteration, the following is performed: f) 1) For each resonance O with index i in the resonance array, the following is performed: f) 1) a) A sum s is set to 0. f) 1) b) Then, for each resonance O2 (including resonance O) with index j in the resonance array, the sum is increased by the value of the resonance O2 at the frequency of the resonance O. f) 1) c) The difference between the sum s and the interpolated value of the logarithmic amplitude spline LogInt at the frequency of O is computed. It is multiplied by a fixed step size S and then the amplitude of O is subtracted by this value.
