The main goal of this project is to create a realistic traditional singing voice synthesizer (although integrating machine learning ideas into it could also be interesting), but side goal is also to recreate VOCALOID2 (actually this was the original goal, but the objective changed early on).
Part of this task is to implement the expression system. I had long realized that there were two separate expression systems, but initially there had been some confusion between what belonged to which. Initially, I worked based on the expression system described in Jordi Bonada's 2008 PhD thesis, in Chapter 3, because this was by far the most complete description. For a long time, I had thought the expression briefly mentioned in the 2003 paper "Sample-Based Singing Voice Synthesizer Using Spectral Models and Source-Filter Decomposition", in part due to the reused figures. Because of this, I referred to this system as the "Bonada 2003" expression system, because that's where I thought it had been described, although only briefly.
Later, I read parts of Jaume Ortola's 2001 Master's Degree. In there, some of the "2001" (or "Ortola 2001") expression should be described. Good detail is provided on the dynamics curves, which use Manfred Clynes' Predictive Amplitude Shaping algorithm. On the other hand, the pitch model is not really described, stating only: "The pitch contour of the singing voice has to be carefully generated in order to obtain a faithful synthesis. So we have designed a mathematical model for reproducing the smooth pitch transitions between notes. This model allows us to control the transition duration and the tuning deviations at the end and the beginning of the notes in accordance with the musical context.".
Nothing about what happens in between the note transitions was stated at all, so this was a total mystery at the time. With what I know now, I am still not fully sure. The basic model is probably either flat lines or linear interpolation between the start and end. On the other hand, looking at the figures, besides the vibrato present in some of them, there is clearly something else as well. In one expired patent I read, the mean was subtracted from timbre variations and those were added on top of interpolated timbre, so perhaps something could be happening like that for pitch, although it is unclear then what would happen for the areas that don't correspond to stationary PhUs. Another possibility is some kind of random noise that is added. Actually, the figure looks suspiciously like something that has been linearly interpolated at the edges. You can clearly seen in the transitions and expressive parts, full pixel-level resolution; on the other hand, in these areas, it looks like straight line interpolation. So perhaps it is random noise that is interpolated.
Anyway, this "mathematical model" in reference in many other places, but not described in any of them. For example, in "Sample-Based Singing Voice Synthesizer Using Spectral Models and Source-Filter Decomposition": "In the case of note transitions, the process is the same but whenever no template is specified, a pitch model is applied that overwrites the absolute pitch track of the score, like shown in Fig. 3, so to avoid pitch discontinuities. This pitch model has to be carefully generated to obtain a natural sounding pitch curve in the output synthesis. A mathematical model has been designed to produce smooth pitch transitions between notes and allow the control of some parameters like duration, shape and synchronization to phonetics and musical rhythm."
. Fresh air and sunlight is nice. I hope things go better for you.
)
