Screen Shot 2026-03-02 at 4.18.40 PM.png
- 444.29 KB
(2003x1640)
So I've been working on a project to create a singing voice synthesizer I'm basing it on the description in Jordi Bonada's PhD thesis "Voice Processing and Synthesis by Performance Sampling and Spectral Models". After a lot of trouble with getting TWM f0 estimation to work, I've finally gotten to implementing MFPA (Maximally Flat Phase Alignment". And amazingly, it seems to have worked first try. Compare my results: https://i.ibb.co/dsvgv0fd/Screen-Shot-2026-03-02-at-3-54-48-PM.png To the results in the study: https://i.ibb.co/C3fjdWVd/Screen-Shot-2026-03-02-at-3-55-09-PM.png
>>12957 also you can just store and serve them, you dont have to handle them at all
The main goal of this project is to create a realistic traditional singing voice synthesizer (although integrating machine learning ideas into it could also be interesting), but side goal is also to recreate VOCALOID2 (actually this was the original goal, but the objective changed early on). Part of this task is to implement the expression system. I had long realized that there were two separate expression systems, but initially there had been some confusion between what belonged to which. Initially, I worked based on the expression system described in Jordi Bonada's 2008 PhD thesis, in Chapter 3, because this was by far the most complete description. For a long time, I had thought the expression briefly mentioned in the 2003 paper "Sample-Based Singing Voice Synthesizer Using Spectral Models and Source-Filter Decomposition", in part due to the reused figures. Because of this, I referred to this system as the "Bonada 2003" expression system, because that's where I thought it had been described, although only briefly. Later, I read parts of Jaume Ortola's 2001 Master's Degree. In there, some of the "2001" (or "Ortola 2001") expression should be described. Good detail is provided on the dynamics curves, which use Manfred Clynes' Predictive Amplitude Shaping algorithm. On the other hand, the pitch model is not really described, stating only: "The pitch contour of the singing voice has to be carefully generated in order to obtain a faithful synthesis. So we have designed a mathematical model for reproducing the smooth pitch transitions between notes. This model allows us to control the transition duration and the tuning deviations at the end and the beginning of the notes in accordance with the musical context.". Nothing about what happens in between the note transitions was stated at all, so this was a total mystery at the time. With what I know now, I am still not fully sure. The basic model is probably either flat lines or linear interpolation between the start and end. On the other hand, looking at the figures, besides the vibrato present in some of them, there is clearly something else as well. In one expired patent I read, the mean was subtracted from timbre variations and those were added on top of interpolated timbre, so perhaps something could be happening like that for pitch, although it is unclear then what would happen for the areas that don't correspond to stationary PhUs. Another possibility is some kind of random noise that is added. Actually, the figure looks suspiciously like something that has been linearly interpolated at the edges. You can clearly seen in the transitions and expressive parts, full pixel-level resolution; on the other hand, in these areas, it looks like straight line interpolation. So perhaps it is random noise that is interpolated. Anyway, this "mathematical model" in reference in many other places, but not described in any of them. For example, in "Sample-Based Singing Voice Synthesizer Using Spectral Models and Source-Filter Decomposition": "In the case of note transitions, the process is the same but whenever no template is specified, a pitch model is applied that overwrites the absolute pitch track of the score, like shown in Fig. 3, so to avoid pitch discontinuities. This pitch model has to be carefully generated to obtain a natural sounding pitch curve in the output synthesis. A mathematical model has been designed to produce smooth pitch transitions between notes and allow the control of some parameters like duration, shape and synchronization to phonetics and musical rhythm."
So recently I have been reading the expired VOCALOID patents, and made an effort to catalog and then read them all. I read https://patents.google.com/patent/JP2006330615A which describes the expression system described in Bonada's 2008 PhD thesis. Because of this, I am now calling this expression system the Bonada 2005 expression system. Interestingly, this finally described that mathematical model. The mathematical model for the legato transitions is not mentioned at all in Bonada's thesis, so I think it was replaced by the performance-sampling method for producing legato transitions that he also described in that thesis was used instead. Another interesting thing that this patent mentions is the addition of supplemental points before attacks and after releases. This was not mentioned at all in Bonada's thesis. Interestingly however, the thesis did provide the parameters for these points. Second, another very interesting thing is that in the patent, all of the parameters have fixed values, with the option to randomly scale them. On the other hand, in the thesis, the values were generated according to gaussian distributions. This must have been entirely different methods for generation and not just a writing quirk because, for example, the patents mentions that the points A and C are above the nominal pitch, and B below, while in the thesis, all points' gaussian distributions have a mean of zero. For this, along with other differences, such as the description of the supplemental point pitches being inconsistent with the given parameters in the thesis, I believe these two descriptions actually referred to two different iterations of this expression system, which I am now referring to as Bonada 2005 V1 and Bonada 2005 V2. This clears some things. On the other hand, there are also things where there is now more ambiguity, so more things to test. Other things are still unclear, such as how the point based model is used to generate the dynamics.
In the last post, I talked about some potential experiments I would have to do on the pitch/dynamics model for my VOCALOID2 recreation project. I had already programmed the base code for Bonada 2005 expression system, actually many months ago now. This was all in anticipation of the day I finally compare my system to samples that were actually generated by VOCALOID2. Well today I decided that day is now... and I was totally wrong about everything. Well not everything,- I was wrong about the expression system. It was actually the Ortola 2001 expression system that was still used in VOCALOID2 seemingly, except using a different dynamics model from the Predictive Amplitude Shaping algorithm that was used in VOCALOID1. Anyway, at first this made me even more concerned, because there was much less information about the Ortola 2001 expression system. One of the big mysteries was how the pitch was handled over the note duration. Well, as you can see in the attached image, it is very simple. In fact it is actually the simplest thing possible - it is a straight line. Another thing that concerned me was the dynamics model, since there was no description of this at all anywhere. But luckly, it proved to be incredibly simple. Simple enough that I was able to recreate it just by fiddling around for a couple hours in Jupyter notebook. In general, I feel like I have enough from the creators of VOCALOID that I have in a way gain an intuition for their thought process and what mathematical structures they favored. Below, I have presented in a demonstration of a partial recreation of the dynamics model and template-less portamento pitch transition model (my [orange] vs V2 [blue] portamento pictured). For an example of this thought process understanding, the first thing I tried for note dynamics decay was a formula of the form y*(e^-x - 1), which was inspired from the formula used for computing the source curve in the Excitation plus Resonance model, and it worked. Another example is the way the exponent is used in the portamento transition function, which was inspired in a way from the formula used for that same transition mentioned in the Bonada 2005 expression system patent (https://patents.google.com/patent/JP2006330615A).
import numpy as np
def synthesize_note_dynamics(note_on, duration, amplitude, sr=22050):
v = np.zeros(round((duration + note_on + 1.0) * sr))
for itr in range(len(v)):
t = itr / sr
if t >= note_on and t < note_on + 1.25:
amp_attack = np.cos(((t - note_on) / 1.25) * (np.pi / 2.0)) * 0.4 + 0.6
else:
amp_attack = 0.0
if t > note_on + 1.25:
amp_decay = 0.6 + (np.exp(-(t - (note_on + 1.25)) / (duration - 1.25)) - 1.0) * 0.175
else:
amp_decay = 0.0
if t >= note_on - 0.015 and t < note_on + 0.005:
amp_spike1 = (1.0 - abs((t - (note_on - 0.005)) / 0.01)) / 4.0
else:
amp_spike1 = 0.0
# Note: There is also a fade of about 33% that occurs near the end that lasts about 100ms. The rest of the fade is handled by an articulation from phoneme to silence. The fade is not modeled here since its time depends on the length of that articulation.
v[itr] = (amp_spike1 + amp_attack + amp_decay) * amplitude
return v
# In cents
def synthesize_portamento(start_time, duration, start_pitch, end_pitch, sr=22050):
v = np.zeros(round((duration + start_time + 1.0) * sr))
for itr in range(len(v)):
t = itr / sr
if t >= start_time and t < start_time + duration:
if end_pitch >= start_pitch:
v[itr] = start_pitch + (end_pitch - start_pitch) * (((t - start_time) / duration) ** 0.8)
else:
v[itr] = end_pitch + (start_pitch - end_pitch) * ((1.0 - (t - start_time) / duration) ** 1.5)
elif t < start_time:
v[itr] = start_pitch
else:
v[itr] = end_pitch
return v
Update on my proposed improvement to the Wide-Band Harmonic Sinusoidal Modeling algorithm: https://listserv.cuit.columbia.edu/scripts/wa.exe?A2=MUSIC-DSP;8bd93c70.2608C&S=
)
Feel free to use this thread to share cool images 
stupid 
smart 
gamble 
yeah 
