3.1 Training Data

The pre-processed ABC dataset of Irish traditional music called folk-rnn data_v_3, used for previous research, was used for model training ^[11,12]. The ABC data was crawled from The Session, a community-driven website featuring transcriptions of Irish traditional music.

The pre-processed dataset transposed each tune to C and C# key. Among them, only the tunes in the C key were used. Any slurs or ornamentation were omitted during pre-processing. Although the pre-processed dataset lacked rhythmic genre information, such as ‘reel’ or ‘jig’, information was added by matching the tunes with those in The Session.

Among the dataset, we filtered out tunes with duration errors such as overfilled measure duration (e.g., nine 8th notes in 4/4 measure) or pickup measures without following corresponding pairs. Tunes with multiple time signatures or key signatures were excluded. Of the 22,925 tunes in the dataset, 18,612 tunes were retained after filtering. Furthermore, a few errors in repetition symbols, such as a missing seconda volta, were manually corrected.

3.2 Tokenizing

Based on the pyabc library, the ABC parser was implemented in parse repetition, and the ABC notation was tokenized.

When a token represents a note or a rest, the token has two features: pitch and duration. On the other hand, when a token did not represent a note or a rest, such as a barline, its duration was set to <pad>, and its pitch was set to the string of symbols. This is one of the key differences between the previous models ^[11,12], which employ duration text as independent tokens. Using duration text as independent tokens results in a note being encoded with either one or two tokens because the ABC notation omits the duration character if the duration is equal to the unit length, with a duration value of 1. For example, ``C2DE/E/'' represents C with a duration of 2 and D with a duration of 1, and two E with a duration of 0.5. In our encoding scheme, each note is always encoded as a single token.

Other characters, such as barline ``|'', repetition mark ``:|'', and triplet mark ``(3'' are also grouped as a single token. Metadata, such as key information ``K: C Major'' were all handled as a single word in a vocabulary, without splitting the characters.

3.3 Encoding

Previous studies on Irish music using ABC notation, such as those by Sturm et al. ^[11,12], used similar encoding methods to natural language processing, where the input sequence $\boldsymbol{X}$ is represented as $X\in \mathrm{\mathbb{W}}^{T}$, with $\mathrm{T}$ being the number of tokens. Each token is represented solely by its index in a unified vocabulary, encompassing all the distinct musical symbols.

The generated results of previous work ^[11,12] showed that deep neural networks could learn the meaning and semantic relationship between tokens in these previous encoding schemes. However, the encoding of features and the structure of the musical information in each token can have a significant impact on the performance of the model, as highlighted by Peracha ^[13] and Zeng et al. ^[14]. Considering the limited training set, this study aimed to encode tokens in a musically structured format with rich features.

We employ musically-structured encoding schemes as similar to previous research on handling symbolic music with neural network, such as expressive performance modeling ^[15] or symbolic music generation ^[14], to enrich the tokens with a more comprehensive set of features. This approach provides each note token with the pitch, duration, and several other features, including bar position, metrical position, time signature, or key signature.

Using this encoding method, the input sequence $\boldsymbol{X}$ can be expressed as $\boldsymbol{X}\in \mathrm{\mathbb{W}}^{T\times F}$, where $F$ is the number of feature categories. Table 1 provides a list of features.

A straightforward approach to encoding musical information is to assign a separate category to every unique numeric value. For example, if there were 25 different pitches from MIDI pitch 60 (C4) to 84 (C6), each pitch can be considered an independent "word" that is unrelated to any other pitch. Nevertheless, this approach disregards musical knowledge, such as the fact that pitches with the octave relation share similar characteristics, such as 60 (C4), 72 (C5), and 84 (C6). By encoding pitch as a combination of the pitch class (e.g., C, G) and octave (e.g., 4, 5, and 6), these notes can be grouped in the pitch class feature and represented by the same embedding. Hence, the MIDI pitch was also included as a distinct feature so the model could learn individual embeddings for each pitch and concatenate them with other feature embeddings.

In addition, the measure position of the current token was included as a feature. This encoding captures the token offset from the start of the measure and the measure’s offset from the start of the repeat. The measure offset from the repeat start provides valuable information to the model for learning typical 4 or 8-bar phrase patterns because repetition often occurs at the 4$^{\mathrm{th}}$ or 8$^{\mathrm{th}}$ bar in Irish traditional music. A modulo 4 value of the measure offset was included as an independent feature for each token to enhance the model understanding of a typical 4-bar phrase. This additional feature enables the token to convey information about its position within a 4-bar phrase.

3.4 Hierarchical GRU Language Model

Our model VirtuosoTune is trained in the form of a language model, which predicts the probability distribution of the next token $x_{t}$ given the preceding tokens $x_{0},\ldots ,x_{t-1}$. The model is trained to minimize the negative log-likelihood over a sequence of tokens $x_{1\colon T}$, defined as $-\sum _{t=1}^{T}\log p\left(x_{t}|x_{0\colon t-1}\right)$, where $T$ is the length of the sequence. The negative log-likelihood measures how well the model predicts the ground-truth sequence of tokens.

One of the key aspects of Irish traditional music is its clear measure structure. In addition, ABC notation always includes a bar (measure) boundary as a barline character ``|'', so the bar boundary is provided. To exploit this information, instead of using a single recurrent neural network (RNN), we employed a hierarchical RNN, which was also used for VirtuosoNet, a system for modeling expressive piano performance from a given music score ^[9,16]. VirtuosoNet structure was modified into a melody language model, VirtuosoTune, with the gated recurrent unit (GRU) ^[17] as an RNN cell, because GRU showed better performance than vanilla RNN or long short-term memory in these experiments. Three GRUs, a note-level GRU, a measure-level GRU, and a final GRU, were used to compose hierarchical structures, as shown in Fig. 1.

First, the input sequence $X\in \mathrm{\mathbb{W}}^{T\times F}$ was transformed into a sequence of the dense vector by concatenating the embedding vector of each feature in Table 1. For each feature $f$ there is a corresponding embedding layer $E_{f}\in \mathrm{\mathbb{R}}^{{v_{f}}\times {d_{f}}}$, where $v_{f}$ and $d_{f}$ represent the vocabulary size and embedding size of feature $f$, respectively.

The token-level embeddings were fed into note-level GRU. The output of note-level GRU was summarized in each measure using context attention ^[18]. For a given sequence of note-level hidden states of single measure $\boldsymbol{h}=\left[\mathrm{h}_{0},\,\,\mathrm{h}_{1},\ldots ,\,\,\mathrm{h}_{\mathrm{t}}\right]\,,$ hierarchical context attention summarized it as a $\boldsymbol{y}=\sum _{t}^{T}\alpha _{t}h_{t}\,,$ where $\alpha _{\mathrm{t}}=$ $\text{Softmax}\left(\tanh \left(\boldsymbol{W}\mathrm{h}_{\mathrm{t}}+\mathrm{b}\right)^{\top }\mathrm{c}\right)\,,$ and $\boldsymbol{W},$ $b,$ and $c$ represent the weight, bias, and context vector of the attention module, respectively, which are trainable parameters.

The measure-level summarizations were fed into the measure-level GRU. The measure-level output was then broadcasted into the note level so that each note or token’s hidden output was concatenated with the measure-level output of the previous measure. The final GRU took the concatenated vectors as its input and returned the hidden states, which were then projected to the logit for the entire vocabulary of pitch and duration. The softmax was applied separately for pitch and duration.

Fig. 1. Diagram of the proposed hierarchical GRU.}

5.1 Generating Tunes

The model generated a tune using various random seeds and fixed metadata conditions, including a meter of 4/4, C major, and reel rhythm. Sampling strategies, such as beam search or nucleus sampling, were not used, unlike previous work ^[12]. Following the generation process, basic rules were applied to ensure the generated tune adhered to valid measure durations, repeats, structure, rhythm, and cadence. The generation of tunes was continued until the model had 1000 tunes.

A rule-based post-processing technique was applied to transpose the generated tune. The transposed key was determined by the pitch range of the generated tune while imitating the key distribution among the 350 reels in ``The Dance Music of Ireland: O’Neill’s 1001''.

5.2 Result

The evaluation result was presented on the challenge site ^[19], where the submission was named Clare. Among the ten tunes of the submission selected for evaluation, three judges flagged one as plagiarized and another as having a rhythm that did not meet the Irish reel standards. These results indicate room for improvement in the model performance.

On the other hand, the tune titled ``Reel 507'', presented in Fig. 2 received perfect scores (five out of five) from all four judges and was nominated as a potential award winner. Ultimately, this tune was selected as the first-prize winner.

The judges' remarks for ``Reel 507'' were as follows <sup>[19]</sup>: ``Easy to remember, easy to follow and catch; uplifting, bright and fun to dance to"; "Excellent! Everything right! That's a 5+. Not plagiarised as far as I can see (I suspected it might be since it was so good)"; "Very simple reel, but adheres to all the variables regarding reel structure. Great Rhythm. Makes good use of repetition. Very simple but consistent and phrases are easy to remember. Excellent simple reel that sits well with the tradition.''

As the evaluation comments showed, the generated tune proved that the AI model could produce a novel tune that was both novel and well-suited to the characteristics of traditional music. Among the Ai Music Generation Challenge from 2020 to 2022, this tune was the first to obtain perfect scores from all the human judges.

Fig. 2. Score of Reel 507, which won first prize.