|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Table of Contents | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
About ZZT Sound Plus | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
ZZT Sound Plus was created by Christopher Allen for the purpose of modeling sampled waveforms using relatively simple sound generation capabilities. Even though waveform audio may seem at first to be impossibly complex to take apart, this tool can break it down into its fundamental building blocks (sine waves). The main advantage of ZZT Sound Plus is that any type of recording can be modeled as sheet music. The program can tell what notes are being played, and how loud they are relative to other notes played at the same time. One does not need preexisting metadata such as that found in MIDI or tracker formats--this tool is capable of capturing anything. You can whistle into a microphone and the tool will capture the whistled notes! ZZT Sound Plus supports a variety of output formats, including MIDI, ZZT #PLAY syntax, and bitmap renditions of sheet music in musical staff notation.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The Science Behind It All | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
ZZT Sound Plus works by utilizing a system of mathematical formulas called Fast Fourier Transforms. You might be like, "fast furry WHAAAAT?!" Fourier Transforms are named after the inventor Joseph Fourier (pronounced "for ee ay"), who discovered that any repeating-interval function can be modeled as the sum of sine waves at specific amplitudes, frequencies, and phases. What is truly amazing about Fourier Transforms: it is possible to model any interval this way, including functions that are completely discontinuous. As long as the modeled function covers a repeating interval, the interval can be treated as a sum of sines.
From each of the resulting waves used to model the interval, one picks out coefficients representing specific amplitudes and phases. We aren't concerned with the phases in our results; we are only looking for the amplitudes. What ZZT Sound Plus does with the results of the Fast Fourier Transform, or FFT, is sort the coefficients by amplitude, such that the "loudest" ones are captured, with the "quietest" ones thrown away. We can't just model an entire waveform as a continuous interval with a giant window, because we must limit our computations somehow. Therefore, ZZT Sound Plus splits the source waveform into smaller subsections, whereupon it models them on an individual basis. By letting the user configure the shortest measured sound interval and total number of voices modeled, ZZT Sound Plus can structure the FFT parameters appropriately. Frequencies in ZZT #PLAY statements are limited to only a finite set of F-numbers, or predefined frequencies representing a note on a scale. These F-numbers are measured along standard 2(1/12) half-steps. Any frequency modeled by the FFT that does not fall exactly on one such F-number will be rounded to the nearest F-number. While it seems that Fourier Transforms were tailor-made for automated computations, it is quite remarkable that they were invented long before automation with electronic computers was possible. Originally meant to solve heat equations, the transforms are excellent at taking apart anything harmonic. Just be glad you don't need to do these calculations by hand! | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Fundamentals of Note Capturing | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The sampling rate of a waveform is determined at the time the waveform audio is recorded digitally. Modern sampling rates are fairly high, along the lines of 44,100 Hz (CD quality). This means that every 1/44,100 of a second, there exists a sample point. A detailed, digitized waveform requires an appropriate amount of storage space, which is why .WAV files tend to be rather large, even for only a few seconds of audio. This also requires ZZT Sound Plus to do a commensurate amount of work in order to process such a large volume of data. The longer the audio, and the higher the sampling rate, the longer ZZT Sound Plus must spend analyzing the data. ZZT Sound Plus subdivides the audio into discrete-time sections based upon the shortest playback interval. Each of these sections is then subject to a window function and run through a Fast-Fourier Transform. The FFT translates an analyzed section from the time domain into the frequency domain.
In the frequency domain, vertical distance from the center of the
waveform is broken down not by position in the waveform timeline, but
rather, the frequency bin of the waveform peak. When summed, the
frequencies contribute to the composite waveform over the entirety of the
interval. One can think of any waveform as a sum of a potentially infinite
number of sine waves, each with their own amplitudes, frequencies, and phases.
ZZT Sound Plus locates peaks in the frequency distribution and assigns all captured frequencies an amplitude. Depending on which of these peaks represent frequencies that the user wants to capture, and the number of voices the user wants to allocate for the capture, the results are in the form of frequency-amplitude pairings, totaling between zero and the maximum number of voices for any one discrete-time section. A typical pairing is represented in #PLAY string format as:
(Shortest Playback Interval) V(Volume Number) (Note) For example, TV39D# represents a thirty-second note (T), played at volume level 39 (V39), at the frequency 311.127 Hz (D#). Octave is accounted for on a running basis, starting at octave 4 and moving up or down when "+" or "-" characters appear. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Using ZZT Sound Plus | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
There is also the option to open a file of .MID format, or MIDI. MIDI files already contain note metadata and will not require special evaluation using FFTs in order to display the note data. For more information on loading MIDI files, see the Note Consolidation Interface.
After the sample data is loaded, a summary of the content is shown in the
text box below the buttons. The number of channels, sampling rate, size
per sample, and other information are displayed in the box.
Configuring Note Capture OperationsThe options below the box configure how the note capture operation will work.
Keep in mind that captures will still work even if the tempo is incorrect.
If the song is played back at a different tempo, a high resolution for the
shortest playback interval will still model the song fairly well (unless
the song has a large number of extremely short notes).
The Note Capture Ranges control which frequencies are picked from the
FFT for long-term representation in the output. In effect, a single range
acts as a band-pass filter. The default range covers the entire
capturable range of notes for this tool (C octave 2 through B octave 7).
If desired, one can set as many as three separate ranges.
Volume levels are organized logarithmically, such that 50 represents the maximum level that is digitally characterizable, with each 5 units below cutting the amplitude in half. Thus 45 is half the amplitude of 50, 40 is one quarter the amplitude of 50, and 35 is one eighth the amplitude of 50. To convert between decibel levels and volume levels, use the following formulas.
dB = log10(exp((baseVol - vol) / -5 * ln(2))) * 20 vol = baseVol + int(ln(10 ^ (dB / 20)) / ln(2) * 5) The following are useful conversions between decibel levels and volume levels when V45 is set as the maximum.
The FFT generates many sine waves in order to characterize a discrete time
interval. A person can limit the number of simultaneous sine waves (notes to
capture per interval) with the Total Voices Captured setting.
ZZT Sound Plus sorts all notes in an interval by volume, such that
only the loudest (i.e. highest amplitude) waveforms are captured. It is
recommended that this number be something relatively high (like 12). It
can go as low as 1 and as high as 15.
Running the Note Capture Operation
Important: MIDI files do not require running FFT after load. One should proceed directly to the note consolidation interface after loading a MIDI file. When the capture is finished, the status box below the button indicates completion. One can proceed to the Note Consolidation Interface or make immediate use of the output via the Rough Cut Notes.
It is possible to use this "rough cut" in a variety of ways, but the most immediate use is a test in The Hall of Music. When one copies the contents of the text box and pastes it into the Hall of Music, it is possible to play back the notes from the capture and listen to the song in a non-PCM playback device.
While The Hall of Music is designed for testing ZZT-related game
chiptunes, as with the game engine
ZZT Ultra,
it can play back any tune
the user desires. One should realize, however, that the notes taken from
the "rough cut" are just that--rough. The tune will appear to have
significant arpeggio and noise due to the fact that nothing
has been consolidated yet.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Note Consolidation Interface | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The timeline of the song interpreted from the waveform audio is shown as a horizontally scrolling image in the center of the window. For those familiar with musical staff notation, little is left to the imagination. Time signature, key signature, clefs, measure boundaries, notes, and rests all appear as they would in sheet music. Because of the highly subjective nature of note capture output, the more nuanced notation usually written in sheet music does not appear in this interface. All notes appear as singles, all notes at or above middle C appear in the treble clef, and all notes below middle C appear in the bass clef. Aesthetic decisions about how to orient note stems in close proximity are not configurable; the tool does its best job to guess what looks like a reasonable layout on the page.
ZZT Sound Plus also supports its own unique notation. Measure
numbers and section numbers are enumerated at the base of the image.
Section boundaries are shown within measures using dotted lines. There
are also several postfixes that the user can choose to show next to each
note: volume numbers, decibel numbers, and voice numbers.
If one wants to generate quick consolidated output, one can select the
Auto-Consolidate button, which merges adjacent notes at similar
volumes and assigns preliminary voice numbers. The note output after
auto-consolidation yields much better-quality results in
The Hall of Music
or other destinations than the "rough cut" output would yield.
Data Sources for NotesIf the data for the notes displayed in this interface originated from a .WAV file, most of the tempo, signature, and spacing idiosyncrasies will not be known at first. The user will need to experiment with the settings within the interface in order to get a feel for which settings work best with the song characterized. If the data for the notes originated from a .MID file, most of the song information will be known, and ZZT Sound Plus will fill in this information automatically if the MIDI file contains it. MIDI note metadata is characterizable in most circumstances, although not all nuances present in the original MIDI file are used here. Note on/off events will be used, but program changes, control changes, aftertouches, and pitch bends will not be reflected in this interface.
ZZT Sound Plus remembers interface settings between sessions. For each
file loaded in the interface, the application saves the last settings chosen
for that song. This is to make it easier to model songs multiple times, so
that one does not need to store metadata in a separate location. Saved
settings can be removed by deleting entries from the ZZT_SP.INI file
in the user's roaming profile.
Pointer Mode
One can select multiple notes by holding down the Control key and clicking several notes in succession. One can also select entire sections at once (which includes all notes per section) by holding the Shift key and clicking a different section from the last section clicked. A related mode is box selection mode, accessible by pressing F1 a second time. This mode differs from section selection mode in that one can drag a "box" with a vertical component to select notes at specific X/Y locations in the staff, as opposed to whole sections at once.
Move note mode changes the behavior to allow the user to drag a note from one spot on the staff to another. This mode is chosen by pressing F2 twice. A note's pitch can be modified by dragging it up or down the staff, and a note's starting location can be modified by dragging it to a different section.
Set measure start mode lets the user pick a specific section to act as
the start of a measure boundary. This mode is chosen by pressing F3
twice. Since measures are periodic, the section chosen will be MOD'ed by the
number of sections per measure, as needed. This mode can be useful when
aligning measure intervals to audio that might not have started on a
"complete" measure boundary.
Show/Hide Options
The staff only shows a limited subset of the total notes captured. The notes shown are filtered by the F-Num Show Range and the Volume Show Minimum. Only those notes within the frequency range are displayed on the staff, and only those notes greater than or equal to the volume level minimum are displayed on the staff. One can fine-adjust the volume show minimum with the keys Ctrl+Plus and Ctrl+Minus. These filter options are useful when distinguishing notes by relevance. The fact that most waveforms are not perfect sines means that artifacts, in the form of shorter, quieter notes than the ones desired, will appear at various locations in the staff. While the option exists to display all of the notes from the capture operation, the user generally only wants to view the notes most relevant for editing. The act of filtering the displayed notes limits what is selectable on the staff. An excluded note cannot be selected with the mouse (or even the Select All feature), and cannot be copied to the clipboard. Most operations will skip over excluded notes in favor of those visible.
It is not normally necessary to show voice numbers because ZZT Sound Plus color-codes the notes by their assigned voice numbers. See the Voices buttons for the color key. For those without good color vision, though, it can be helpful to enumerate voice numbers.
Global Settings
The Tempo setting, in quarter notes per minute, defaults to the setting used in the original note capture operation. However, it can be changed so that output formats assume this new tempo specification. In copied clipboard data, this is a "Unnn" code. In MIDI, this is a tempo meta-command. It is possible to "guess" what the song tempo should be with the Infer Tempo button. This button analyzes a user-determined selection to come up with a more accurate assessment of the tempo than what had been selected in the note capture operation. While not perfect, it can be helpful to use these suggestions for a second note capture operation later. The way ZZT Sound Plus infers tempo is by interpreting the user's selection of multiple sections as a "complete" measure. The tool posts several suggestions for what the tempo would be based on a re-interpreted timeline and the original note capture tempo setting. The suggestions, of course, are only as accurate as the user's selection would indicate. The Time Signature sets the measure boundaries via the numerator and denominator numbers of the signature. After changing the time signature, the staff is updated. A time signature is also set as a MIDI meta-command when MIDI output is generated. The Key Signature sets the key signature on the staff, and also sets the corresponding MIDI meta-command when MIDI output is generated. The key signature modifies how notes appear on the staff, manipulating both the vertical positions for some notes and the appearance of accidental sharps, flats, and normals for others. The Section Spacing slider adjusts the normal section width. Sections have a default length based on the number of postfixes that need to be displayed next to notes within a section. This slider can grow or shrink the standard size. Measures are not necessarily spaced evenly in ZZT Sound Plus. If there is no need to display a section for lack of any visible notes present, it is skipped in favor of a rest of appropriate duration. This means that less complicated music (more quarter, half, and whole notes) has smaller measure widths than more complicated music (more sixteenth, thirty-second, and sixty-fourth notes). The user must decide the best way to trade off readability with functionality.
ZZT Sound Plus remembers global settings between individual
sessions, and which settings one had chosen for specific files. MIDI files
populate many of the global settings automatically, but the user will need
to pick his or her own ideal settings for other types of files.
The Chord and Keyboard GaugesAs notes are selected, they are highlighted using pink arrows and reticles. Whole sections, when selected, are also tinged pink, indicating selection. There are two additional selection indicators that one can use to analyze the selected notes: the Chord Gauge and the Keyboard Gauge.
The notes on the keyboard gauge can be toggled on and off with the mouse.
Clicking on a key will select or deselect all notes that match the chosen
frequency. This is a useful way to identify notes within a specific range
of a broader selection.
[C level] = [2-C dur]*[2-C vol] + [3-C dur]*[3-C vol] + [4-C dur]*[4-C vol] + [5-C dur]*[5-C vol] + ... [C# level] = [2-C# dur]*[2-C# vol] + [3-C# dur]*[3-C# vol] + [4-C# dur]*[4-C# vol] + [5-C# dur]*[5-C# vol] + ... [D level] = [2-D dur]*[2-D vol] + [3-D dur]*[3-D vol] + [4-D dur]*[4-D vol] + [5-D dur]*[5-D vol] + ... [D# level] = [2-D# dur]*[2-D# vol] + [3-D# dur]*[3-D# vol] + [4-D# dur]*[4-D# vol] + [5-D# dur]*[5-D# vol] + ... [E level] = [2-E dur]*[2-E vol] + [3-E dur]*[3-E vol] + [4-E dur]*[4-E vol] + [5-E dur]*[5-E vol] + ... [F level] = [2-F dur]*[2-F vol] + [3-F dur]*[3-F vol] + [4-F dur]*[4-F vol] + [5-F dur]*[5-F vol] + ... [F# level] = [2-F# dur]*[2-F# vol] + [3-F# dur]*[3-F# vol] + [4-F# dur]*[4-F# vol] + [5-F# dur]*[5-F# vol] + ... [G level] = [2-G dur]*[2-G vol] + [3-G dur]*[3-G vol] + [4-G dur]*[4-G vol] + [5-G dur]*[5-G vol] + ... [G# level] = [2-G# dur]*[2-G# vol] + [3-G# dur]*[3-G# vol] + [4-G# dur]*[4-G# vol] + [5-G# dur]*[5-G# vol] + ... [A level] = [2-A dur]*[2-A vol] + [3-A dur]*[3-A vol] + [4-A dur]*[4-A vol] + [5-A dur]*[5-A vol] + ... [A# level] = [2-A# dur]*[2-A# vol] + [3-A# dur]*[3-A# vol] + [4-A# dur]*[4-A# vol] + [5-A# dur]*[5-A# vol] + ... [B level] = [2-B dur]*[2-B vol] + [3-B dur]*[3-B vol] + [4-B dur]*[4-B vol] + [5-B dur]*[5-B vol] + ... By selecting a portion of the timeline, one can determine what the "chord" is based on the frequencies that stand out. The above example shows that the most prominent notes are D# (222), G (267), and A (430). This gives a good idea of the underlying chord at that portion of the song. The chord gauge offers a handy mechanism for removing artifacts that do not match the chord:
This has the effect of leaving the selected sections with only those frequencies
that are part of the chord.
Selection Editing
The clipboard format for notes copied from the staff is the same format used in the ZZT Ultra #PLAY string syntax. When copied, the notes can be pasted into The Hall of Music or a text editor. Similarly, these note codes can be copied from external locations and pasted into the staff at any selected section.
While the goals of editing the notes on the staff are entirely up to the
user to decide, perhaps the most important goal should be noise removal.
Out-of-place notes of short duration (usually in the extreme high or low
octaves) are likely artifacts from the note capture operation, and their
removal (using Delete, Crop to Visible, etc.) will help round
out the results of the captured notes significantly.
Changing Selected Note Attributes
The Set Volume button, or V key, sets all selected notes to the volume level shown in the Active Update Info. This is handy for smoothing out fluctuating volume from multiple notes captured. One can make fine adjustments to this volume level with the +/- keys. The Set Duration button, or D key, sets all selected notes to the duration shown in the Active Update Info. This is handy for lengthening, shortening, or otherwise making uniform one or more selected notes. Merging of multiple notes at the same frequency will occur if a note is lengthened to the point where it would "bleed into" notes in adjacent sections. One can quickly set the active duration with the keys W, H, Q, I, E, S, T, J, and period. The Change Voice button, or C key, sets the voice number for all selected notes to the voice shown in the Active Update Info. Setting the voice number for a note will recolor the note to reflect the color key shown in the voice buttons. Voice numbers aren't altogether pertinent to musical staff notation, because it is usually to the musician's discretion how the note should be played. The reason for selecting voice numbers is output metadata generation (clipboard format or MIDI file).
Until voice numbers are assigned to individual notes, notes will have no
default voice number. Copying the notes without voice numbers to the clipboard
or creating a MIDI file will assign voice numbers in a subjective fashion.
Selection Modification
If merged notes have different volumes, the highest of the volumes is picked.
When auto-consolidating notes, adjacent notes are only merged if the notes are (A) directly next to each other, section-wise, and (B) within a volume tolerance of +/- 3 levels. When merged, the composite note will have the maximum volume of all merged notes.
Important: Auto-consolidation cannot be undone.
The maximum number of voices selected in this fashion is set by the Limit combo box to the right. For example, if this is set to 10, it means that only 10 voices will have unique numbers assigned; all the rest will be assigned voice zero. The idea behind this operation is that frequencies with the highest energy spectral density (see notes above on the Chord Gauge) will have the lower voices, starting at 1, and those frequencies with lower density will have higher voices, ticking upwards until the maximum limit is reached. Those frequencies with the least density are then assigned voice zero, which can be easily removed if desired.
Important: Spectral voice reassignment cannot be undone.
In other words, eighth note rounds to eighth-note duration anything
eighth-note-dot or shorter, eighth note dot rounds to eight-note-dot duration
anything quarter note or shorter, and quarter note rounds to quarter-note
duration anything quarter-note-dot or shorter.
Voice-by-voice Selection Toggle
Context Menu Actions
Many of the actions are also available as buttons and/or keyboard shortcuts, including Merge, Break Apart, Copy, Cut, Paste, Delete, Delete Section, and Crop to Visible. There are also some actions that are specific to workflow and are accessible only via the context menu. Select Entire Measure expands the selected portion or last clicked location in the staff to cover one or more measures. This is a quick way to perform operations that apply to only one measure in the song. Select Active Duration Interval expands the selected portion or last clicked location in the staff to cover the length of the interval set by the active update info. For example, if the active update duration is set to quarter note, the interval selected will be the length of a quarter note, and rounded to the start of the measure using quarter-note multiples (4/4 time signature will have exactly four different possible selection starting points per measure). Delete Notes in Measure Here and Above clears the measure of all notes at or above the selected note's frequency. This can be useful when removing artifacts. Delete Notes in Measure Here and Below clears the measure of all notes at or below the selected note's frequency. This can be useful when removing artifacts. Move Notes 1 Section Left/Right performs a quick cut-and-paste operation with the selected notes, moving them one section left or right. It is handy to use this feature when cleaning up a song with notes that are slightly "off" their preferred starting time. Shift Octave +/-1 performs a quick octave shift of the selected notes. This is generally faster than using the Shift Key feature when a note only needs to be modified by a single octave. Consolidate towards Middle Octave performs a special "chord consolidation" within the selected sections. Selected notes provide the basis for the frequencies that will be consolidated towards middle C. If a note exists closer to middle C (lower than self for treble clef; higher than self for bass clef) whose frequency is only different by octave multples, such notes are "absorbed" into the notes closer to middle C. If the removed notes are longer and/or louder than the retained notes, the retained notes assume these longer and/or louder characteristics. One would consolidate notes up or down an octave if it is known that high or low harmonics will appear as a result of instrument aliasing, and such harmonics are not considered necessary for playback.
Subtract Clipboard from Selection performs removal of notes in the
selected sections that match equivalent note content found in the clipboard.
This somewhat unusual operation can be useful for getting rid of notes that
are seen as repeated across multiple measures, and one wants to purge the
"regular" notes to get a better view of the "solo" notes.
Other Actions
Playback is of limited quality because sine functions are used to synthesize the audio, resulting in a relatively cheap waveform profile. The audio will appear to be subjected to a low-pass filter of about 4000 Hz. Note that The Hall of Music delivers somewhat better output, albeit with little control over instrumentation. Playback can be limited to only selected sections, or it can continue until the end of the timeline. If only a single section is selected, you will be prompted to confirm playback of the remainder of the song.
PNG output translates the notes to a series of sheet music image files.
You can number the individual pages of the file by placing "[]" brackets in
the filename; the brackets are replaced by a number 1, 2, 3, 4...etc. as more
pages are generated. Sheet music has a lot of nuance to it; an additional
dialog box opens for configuring the output before it is saved.
The resolution of the generated images is determined by the width, height, and margin settings. The greater the number of pixels, the finer quality the notes will be when the image is printed out.
Because sheet music can be quite subjective when displayed, it may be
worthwhile to edit the images after they are generated. ZZT Sound Plus
attempts to display notes in a very basic fashion; it cannot possibly
know the "right" way to organize them in every context.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Key Commands | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The following key commands (hotkeys) are implemented in the note consolidation interface in ZZT Sound Plus.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Note Capture Issues | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
When notes are captured in ZZT Sound Plus, results may vary in quality based on a variety of factors. Most of these factors are due to the nature of the audio the user is trying to capture. Others are based upon the parameters used in the capture operation. A user can expect to deal with the following issues.
Upper voice limitAs many as 15 simultaneous voices are capturable. Depending on what the user is looking for, setting the number of voices higher or lower will change the results of the capture operation. ZZT Sound Plus sorts all captured frequencies within a section by amplitude. Frequencies with the highest amplitudes are assigned voices; frequencies with lower amplitudes are discarded. This may have consequences if there are significant differentials in the amplitudes of the waveforms in the audio. A reduced number of voices may inadvertantly hedge out frequencies the user wants, yet are quieter than others that may or may not be wanted. Two ways exist to deal with this problem: increase the number of voices used in capture, and limit the frequency capture ranges to only those desired frequencies. Often, a capture will only make sense if the expected results are limited to only high-range, low-range, or mid-range. By manipulating the capture ranges, the issue of undesired (but high-volume) bands can be mitigated. Inadvertant noise captureThis problem depends on how many voices are used in the capture, as well as what sort of noise exists in the audio itself. Inadvertant noise capture is a problem actually made worse by having a higher upper limit for voices. While more frequencies are captured, this is not always a good thing. Artifacts from percussive effects and notes adorned with awkward frequency modulation profiles will increasingly show up as short, scattered notes. Limiting frequency capture ranges and reducing the number of voices can help to mitigate inadvertant noise. The note consolidation interface also helps to identify and remove irregular notes. Percussion and bass tracks tend to be rather loud. It might be useful to capture a song twice with two different capture ranges: once with the lower limit set to something like 3-C, and again with the lower limit set at the very bottom of the capturable scale. Too-quiet thresholdZZT Sound Plus uses a Hamming window function to mitigate spectral leakage. This will result in noise artifacts appearing below -40 dB. If the volume differentials within audio are so significant that the minimum captured waveform is under -40 dB from the maximum captured waveform, the quieter waveforms cannot be captured accurately. This is usually not a significant problem, but it can happen. The overall volume of the original sample data is not a problem during capture--all note captures are relative to the maximum volume number specified. Noise artifacts from the window function only become a problem when there are significant volume differentials of waves over the course of the audio. If differentials as a result of changing volume (e.g. fade in or fade out) make this prospect unacceptable, one could break up the audio into multiple files and model them separately. This gives each capture result a unique maximum (some loud, some quieter) to use as a point of reference. Scalloping lossThe window function used in the capture operation will often "slice" some of the amplitude off the true maximum for frequencies modeled outside of their captured bins. The closer the frequency to the bin representing the frequency, the less loss occurs. Maximum loss happens when a bin is exactly halfway between two half-steps on the scale. This so-called scalloping loss will reduce amplitude for some frequencies, but the loss is usually limited to a few dB. The problem only becomes serious when a variety of frequencies must be represented, and scalloping loss drives many of the quieter tones below a capturable threshold. Instrument aliasingHuman hearing interprets frequencies based upon perception of the most dominant, sine-like shape of a wave at a specific frequency. But most waveforms that are "played" at a frequency are not perfect sines; they naturally contain modulation characteristics that could be interpreted as other, quieter frequencies. Some instruments also naturally generate frequencies at multiple points in the band per single note played. This can result in notes being captured one octave above and/or one octave below the frequency expected. Depending on how loud the instrument is, a single note could potentially occupy two or more voices, not just one. The note consolidation interface can help deal with this problem. Aliased notes one octave higher and/or lower can be identified, shifted, and possibly removed. Out-of-capturable-range frequenciesZZT Sound Plus limits the capturable band of frequencies to 2-C in the lowest octave and 7-B in the highest octave. This band reflects the original ZZT engine playback limitations. Notes above and below this band are not capturable. This is usually not a problem, as the band encompasses nearly all notes likely to be played on most instruments. Interstitial frequency representationZZT Sound Plus analyzes the peak shapes of the frequency distribution as a way to find the true position of the frequency relative to the bins used in the FFT. While this makes sense most of the time, it does create a problem if the user for some reason really does want to capture a frequency between the expected half-steps of the musical scale. If a tone actually would naturally fall outside of a capturable half-step, ZZT Sound Plus will round the captured frequency to the nearest half-step. This has the effect of stabilizing some tones, but ironically destabilizing others. Human voices and some instruments will not necessarily maintain perfect pitch over the entirety of a note's envelope. Vibrato effects will also transcend the target pitch over the sustained portion of a note's envelope. ZZT Sound Plus addresses this lack of consistency rather mercilessly, in the form of rounded-to-half-step frequencies over the entire course of the note. One can think of this as a "malevolent" form of auto-tune. If one only wishes to capture a consistent frequency at a natural half-step, the note consolidation interface is the best mechanism for cleaning up potentially out-of-tune notes. Unfortunately, there is no recourse offered for capturing a frequency outside of natural half-steps. Small differences in instrument tuning might affect where half-steps are expected. ZZT Sound Plus allows the user to configure tuning by setting the frequency for middle A. This defaults to 440 Hz, but it can be set to any frequency. Inability to represent percussive effectsWaveforms are captured in an FFT based upon the apparent consistency of the waveform's period. Percussive effects tend to be far less consistent over time, and are not capturable in any straightforward way. Percussion affects the frequency distribution in a variety of ways. The tendency of percussion to be loud often means that it will intrude within the frequency distribution in a random and inconsistent fashion. The "frequencies" reported from such an effect might not be very useful other than to indicate locations within the timeline of such effects. Section edge casesThe shortest playback interval determines the minimum resolution used for representing a single note. The shorter the interval, the less likely a single section will fall on an edge case. An edge case is when the section selected for FFT analysis is between the start of one combination of tones and the end of another combination of tones. The result tends to be noise or frequencies that drift outside of expectations. This is not a serious problem if the shortest playback interval is set to something very resolute, such as 32nd note. One should be aware of the problem if a resolution is picked that would increase the likelihood of edge cases. Low-octave minimum intervalThe lower the frequency on the scale, the more difficult it becomes to capture frequency bins accurately from an FFT. ZZT Sound Plus handles the lower octaves by running multiple passes on each section. The first pass is always the shortest playback interval. Subsequent passes double the interval time, which also doubles the resolution of the frequency bins. This allows for more accurate captures of lower octaves. The downside of this strategy is that lower frequencies cannot be accurately characterized unless they are sustained for longer periods of time. Edge cases will end up being worse at the lower octaves; little can be done to mitigate this. Misrepresented tempo specificationThe tempo of an audio recording is likely not known from the outset. ZZT Sound Plus interprets section size for a typical note as a function of the shortest playback interval and the specified tempo. If the shortest playback interval has high resolution (e.g. 32nd note), the fact that the tempo could be wrong tends not to be a problem, because captured sections can be consolidated with minimal edge cases for notes longer than the playback interval. But incorrect tempo will become significant when notes must be represented for durations exactly matching the shortest playback interval. For these short-duration notes, edge cases will mutilate the captures and make note consolidation a tricky endeavor. The note consolidation interface has a special Infer Tempo button that analyzes the selected sections after consolidation, and then arriving at tempo suggestions of its own. While not necessarily accurate, the suggestions can be used to re-run the FFT at a new tempo, resulting in better capture output. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Q & A | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Q: Is ZZT Sound Plus Free? A: Yes. You can use the results of the note capture output how you wish as long as it is in a non-commercial capacity. If you have composed music using this tool, please mention my name (Christopher Allen) in credits as the creator of ZZT Sound Plus. Q: Where did you get the idea for this tool? A: It had bounced around in my mind ever since I learned about Fourier Transforms (nearly 20 years ago). I had dismissed it because I thought it was too ambitious to take seriously. Obviously, ambition is relative. Completion of other ambitious projects eventually stored enough courage in the bank to facilitate taking a stab at this one. As for the overall reason for creating the tool, I thought it would be cool to be able to generate note metadata and sheet music on the cheap, and to "pull the apple right off the tree" if you have an album containing tracks you want to model. The main advantage I was looking for in making this tool was capturing the subtle nuances of songs, the solos, and all the finer, quieter elements of the music that often get ignored by people not listening for it. A lot of sheet music I've seen does not go into the level of detail I'd like. Q: Are there copyright considerations with notes captured from recordings to which I do not own the rights? A: Yes. The original copyright owner of the song you are modeling also owns the rights to the note capture data (at least in raw form). If you are the composer, it means you own the rights. Of course, ZZT Sound Plus operates in a significant gray area of the legal system when it comes to transformative works. By editing the notes in the Note Consolidation Interface (even if you are only removing notes you consider noise), you are transforming the song. These so-called transformative works occupy a special section of copyright law that is too involved for me to talk about here. The key to whether the note capture data is "safe" for usage depends on how it is deployed and distributed. At the present time, it is rather difficult to know the full scope of how such works will be used. When in doubt, you should talk to the copyright holder of the song you are modeling. Q: I made a cover tune based on note captures from this tool. Who owns the rights to the cover tune? A: The cover tune is still owned by the original copyright holder. If you are thinking about showcasing the tune or performing it, this constitutes a reproduction that the copyright holder will have to decide is acceptable or not. There is no way to gauge what reaction an artist or label will have to a cover tune, but it is typically not in their best interest to send Cease and Desist letters to everyone attempting such covers. Circulation of the tune, on the other hand, nearly always benefits them. Once again, talking with the copyright holder is the best bet. Q: I captured notes for an original tune I recorded. Who owns the rights to the captured notes? A: You own the rights. I don't claim anything. But it would be really nice if you credit me somewhere. :-) Q: Can this tool be used to generate karaoke tracks? A: Yes. Files from the MIDI output option can be used as a general framework for constructing a karaoke song. While formats for karaoke sound files can vary, it is possible to add lyrics (shown in a karaoke display as text overlays) to the notes. ZZT Sound Plus does not configure MIDI instruments, MIDI effects, or any other special meta-events that you might want to add to the MIDI output. Once the MIDI output is generated, you're on your own. Q: There is this one song I am trying to model, but it is way too noisy, no matter which settings I pick. Help! A: Read the Note Capture Issues section. I have found that some songs are so filled up with relatively loud notes and percussive effects that it is very hard to model quieter melodies buried under the heap. Dynamic range compression makes this problem either worse or better depending on whether or not the amped-up instrument of the song is something you want to capture versus something you don't want to capture. Probably the best way to model these songs is to run multiple FFTs on them, each time setting the note capture range to a specific band. I have found it worthwhile to independently capture treble clef and bass clef, because relatively loud treble and/or bass tracks do not interfere with each other in the captured output. The results can be merged later using copy and paste. Q: Your staff notation drawing decisions are off in a very specific way that is driving me crazy. A: Well excuuuuuuuuse me! I can read music, but I wouldn't consider myself an expert. If there is something I could easily fix with the notation that I am not aware of, you can suggest a comment at chris@chriskallen.com. Q: Can this tool figure out which instruments play which frequencies? A: Unfortunately, no. To do this, you would need an instrument database. Having said that, it would be interesting to design a next-generation version of the tool that makes use of such a database, which would contain frequency distribution info that allows individual instrument modeling. Right now, that sort of feature is above my pay grade. Q: Human singers are almost intelligible in the captured output. That's creepy. A: I wasn't expecting human voices to be modeled all that well during the concept stages of the tool. It turns out, though, that sound is really just a sum of a whole lot of waves. As you add more of them together, you get closer and closer to the waveform in real life. Fourier Transforms seem at times like they are working miracles. I suppose it's a bit of a mixed blessing that they don't lie about what they represent. For better or for worse, they deliver the sound waves they observe. A creep factor might be inevitable. Q: Why do human singers sound out-of-tune in the captured output? A: In the Note Capture Issues section, I talked about how perfect pitch is difficult to maintain for some instruments over the entirety of a note's envelope. It is actually very punishing to try and get a human voice to be so on-key that the frequency is nailed to the wall. ZZT Sound Plus, using its rounding-to-half-step strategy, highlights this unpleasant fact of life in a way that might even be seen as cruel. If nothing else, you might gain a little respect for the use of auto-tune. Sure, it can be abused, and a lot of the music produced with it risks sounding extremely unnatural. But if used subtly, it can pave over the hurdles that lead singers face. Sorry to disappoint those who think this tool should implement a better form of auto-tune. You need context to know which key is "right" when using auto-tune. A polyphonic note capture tool isn't going to know this on its own, unless it utilizes some kind of expert system. Having this feature would make the tool a whole tier higher than it is now. Q: Is it possible to model non-music with this tool? A: Yes. While the usefulness of the output might be questionable, it can be interesting to model sound effects as melodies. Whatever perverse ideas you have in mind, though, please keep them to yourself. Q: Are there any other tools like ZZT Sound Plus? A: Sort of. The closest type of modeling tool I am aware of is something called a phase vocoder. Using Fourier Transforms to model sound waves is nothing new; software applications have been doing this for some time now. I don't know of any other tools capable of capturing notes directly into staff notation in a polyphonic manner. There are quite a few types of operations one can perform on sound waves transformed into the frequency domain that don't involve capturing the notes directly. Audacity is an example of a sound editing tool that makes heavy use of FFTs to perform creative effects on sample data. One operation of particular interest is changing the speed of audio without changing the pitch. While FFTs are deployed, the results are used in a manner completely different from how ZZT Sound Plus handles its own results. At some point, I became aware of an application known as ScoreCloud. While the implementation of the tool seems to be a stripped-down version of what ZZT Sound Plus offers, the overall concept appears to be quite similar. I was taken aback at how poor a reception ScoreCloud had gotten from reviewers. It is possible that the makers of such an application wanted to create an overly simple and elegant consumer product to handle a scientific phenomenon that is neither. From the Note Capture Issues section, you would know just how many pitfalls await someone trying to do what a polyphonic note capture tool does! Q: This tool can't do "X" and I want this tool to do "X". A: Send your suggestion to chris@chriskallen.com. Q: You keep talking about "The Hall of Music." What exactly is this? A: I can't do the The Hall of Music justice here; this link sends you to the website where it is hosted. Below the musical note interface, there is a much longer explanation about The Hall of Music and what it can do. It also has links to pages that discuss the #PLAY string syntax in detail. Q: What are ZZT and ZZT Ultra? How do these tie into ZZT Sound Plus? A: ZZT was created by Tim Sweeney, and was the first breakout game engine released by Epic Megagames. The versatility of this old editor caused it to develop a large fan following, which is why so many ZZT enthusiasts (like myself) keep supporting it. ZZT Ultra is a suped-up version of ZZT designed to play in a web browser. I created ZZT Ultra back in 2015, and have been updating it ever since. Q: Why do you go into so much detail about musical staff notation? I'm not interested in any of that. A: Going into so much detail on musical staff notation benefits people who care about it. Musicians, both professional and amateur, stand to benefit from an interface that uses the same "gold standard" for written compositions everywhere in the world. It's true that modern "tracker" interfaces don't resemble staff notation much anymore. Also, some people just want to play and listen to music; they can't read music all that well. You can still perform basic operations with the tool if you don't know how to read music. How you use the tool is up to you. Q: Why did you cover so much math and science at the start of the documentation? I'm not interested in any of that. A: Tying together multiple disciplines carries a risk of losing people. But this implies that math and music are not related--they are quite related. If nothing else, I don't like the prospect of people just considering that technology they don't understand is "magic." I can and will try to educate people if they want to be educated. Q: If Fourier Transforms are so great, why haven't I heard about them in any math class? A: They are high up there. You won't hear about them until you've already mastered calculus, possibly multivariable calculus. It doesn't mean they are out of reach. Thanks to the Internet, nothing is. Q: I have a friend who says higher math isn't needed after graduation. What do you have to say about that? A: I say to your friend, shut up. This tool was developed long after I was in school. And it won't be the last time I will use higher math in my ultimate plan to take over the world. I probably shouldn't have mentioned that. Christopher Allen, signing off. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
© Copyright 2018-2020 Christopher Allen. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ZZT Sound Plus is a polyphonic note capture tool.
This tool will break down any composite sound into notes as represented
in sheet music.
The term Fast Fourier Transform refers to a mechanism that produces
an efficient combination of these sine waves with easily-generated
coefficients for waveforms at various levels. The idea is that one can
define a "window" of sampled points at specific locations along the waveform,
and from this window, one can obtain a sum of sine waves, starting with the
lowest frequency and adding more and more multiples of the frequency
until the sum is satisfied. These individual "frequency bins" form the basis
for the "simultaneously-playing notes" captured within a time interval.
When sound is represented as a waveform, it is manifested in a time domain.
This is to say, at any one point in time during playback, one can observe the
extent of vibration away from the center of the waveform. By packing a
sequence of these sample points together, it is possible to represent any
digitized waveform as a series of numbers.
A typical representation of a waveform in the frequency domain is
amplitude on the Y axis and frequency bin on the X axis.
Often, these two axes are represented logarithmically given the
accelerating nature of frequency and amplitude scales as perceived by human
hearing. The result of this graph, or frequency distribution,
is a series of bins at specific frequencies spanning the extent of the FFT
parameters.
The first action to take after opening ZZT Sound Plus is to open a
data file. The typical input is a file of .WAV format. One chooses
the Open File button or the Paste from Clipboard button. Audio can
only be read from the clipboard if it is encoded in an uncompressed PCM sample
data format.
The note data is captured and modeled at the resolution set by the Tempo
and Shortest Playback Interval. The number
of samples grouped together in a single FFT calculation is determined by
these two quantities and the sampling rate of the source audio.
The A Standard defaults to 440 Hz. One would change this if the
instruments in the song are tuned to some other frequency; the FFT will be
more accurate.
The Maximum Volume Level controls how "loud" the note metadata is
captured in the output. Volume is treated as an abstract concept by
ZZT Sound Plus, with the highest peak in the capture output assuming
the maximum volume level (default 45) and all quieter peaks assuming a level
at or below this level. No matter how loud the original sample was, the
capture results are always pegged to the level selected here.
The Run FFT button executes the note capture operation. The amount of
time this takes will depend on the size of the .WAV file loaded and the
resolution selected for the capture operation. If notes in the lower octaves
(i.e. octaves 2 and 3) are captured, ZZT Sound Plus must perform
multiple passes per interval, which also lengthens the amount of time required.
The Rough Cut Notes represent the raw output from the note capture
operation. Each note is identified by a duration code, a volume code, and a
note code. Individual voice codes (representing the number of voices
chosen for the capture) assign each of these code combinations to a specific,
independent voice.
The Note Consolidation Interface has the ability to transform the raw
output into something that sounds and functions much better. To enter this
interface, click the button Consolidate Note Sequence.
The Note Consolidation Interface is a sophisticated suite for editing
note capture data with an interactive musical staff. The main
difference between this type of interface and other music-editing software
is that the origin here is waveform audio, while other software can typically
generate content only from other metadata or direct user input.
It's easy to be overwhelmed by the interface at first glance. With practice,
though, the Note Consolidation Interface is a very powerful tool for
a sound engineer to have at his or her disposal.
All notes in the timeline are selectable with the mouse. The default pointer
mode is section selection mode, accessible by pressing F1.
A single click on a note will select it and show information about it in the
area just below the mode selection buttons.
Add note mode changes the behavior to add new notes to the staff at
the click location. This mode is chosen by pressing F2.
The Active Update Info located to the right of the
information text indicates the volume, duration, and
voice number chosen for new notes added to the staff.
Insert sections mode changes the behavior to allow the user to insert
sections into the timeline. This mode is chosen by pressing F3.
A section's duration is the shortest playback interval chosen for the note
capture operation. One can "bump forward" the timeline with empty sections
with this mode.
At the lower-left portion of the window, there are several options for
configuring what is shown in the staff display.
The checkboxes below the filter options control the postfixes and other
notations shown on the staff. One can show or hide the volume number,
decibel number, and voice number next to each note. The
section numbers and measure numbers, drawn at the base of the image, can
also be shown or hidden.
The duration extenders are optional dashed lines that can be turned
on or off. These lines extend from each note as a way to highlight how many
trailing sections a note "occupies." Duration extenders are useful in
identifying how note durations characterize the entire affected portion
of the timeline, instead of just where the note begins.
Global settings control high-level options that are not strictly
related to note capture data or the consolidation process, but can be
useful in characterizing the output.
The Keyboard Gauge is a straightforward representation of how a
piano keyboard would look if the selected notes were depressed. White keys
are shown with a pink highlight, while black keys are shown with a red
highlight. This gives a good indication of where a person's fingering would
need to be as the notes are played. If the configuration would appear
impossible, some editing of the notes might be warranted.
The Chord Gauge provides a rough representation of the energy
spectral density for all twelve of the half-steps in the music scale. As
notes are selected, the number for each note in the scale increases based
on the sum of products of duration and volume level at all octaves:
A person can select, copy, cut, paste, and delete notes on the staff. The
typical hotkeys for doing so are implemented (Ctrl+C, Ctrl+X,
Ctrl+V, Ctrl+A, Delete).
The Ctrl+Delete operation works differently from Delete. This
operation removes selected sections from the timeline altogether, moving later
sections into the deleted-section gap.
The Ctrl+R operation crops the selected sections to the notes visible
with the existing show/hide filters. Notes removed in this manner cannot be
selected or edited afterwards, even if the filters are changed to allow for
notes within the range.
The Ctrl+Z "undo" operation reverses the last selection editing or
modification event. Any operation can be undone except for
auto-consolidate and Reassign Voices by Spectral Density.
The Active Update Info is used in many of the modification operations.
Direct adjustment of the volume, duration, and voice number is possible with
the mouse. There also exist several 
The user can perform more involved modification to the selected notes or
sections. The buttons in the Modify Selection box in the upper right
portion of the window operate on the selected notes or sections. If no notes
are selected, the operation applies to all notes in the capture output.
The Merge button merges selected adjacent notes unconditionally, and does
not assign voices. For notes to be merged, they must have the same frequency,
and the end of the duration of the earlier note must exactly touch the start of
the later note.
The Break Apart button separates notes into shorter durations based on
the following criteria.
The Auto-Consolidate button merges adjacent notes at the same frequency
and assigns voice numbers based on a frequency-affinity system. Each voice
number has a "preference" for a specific frequency, which means that notes
on a specific part of the staff (treble clef or base clef) will tend to select
voices closest to that affinity if they are available. The resulting
coloration also helps to distinguish overall pitch patterns.
The Reassign Voices by Spectral Density button reassigns note voice
numbers based on the need to set a single voice per frequency for each measure.
After this change has taken place, one can guarantee that a note frequency will
have a voice "to itself" if there are sufficient voices available.
The Round to Duration button "rounds" the selected notes to a duration
pattern resembling the duration from the Active Update Info. Selected
notes will be set only if the selected note has a duration less than or equal
to one denomination above the duration specified in the active update info.
The Round to Start Time button rounds note starting sections to "whole"
locations based on the existing note duration. The time signature is used
to help judge where these positions should be. For example, for 4:4, a whole
note is rounded to the first section of a measure, a half note is rounded to
the first section or halfway-point of a measure, and quarter notes are rounded
to sections offset from the start of a measure by +0, +1/4, +1/2, and +3/4.
The Shift Key button shifts the selected notes up or down by the
specified number of half-steps. Shifting frequencies in this manner has the
effect of changing the key in which the song is played back. Keep in mind
that this action does not modify the key signature.
The colored Voice Buttons toggle specific voices in the selection on or
off. This is a useful way of isolating the scope of multiple selected notes to
just one or more voices out of many, so that an entire set of voices can be
subjected to common changes, while ignoring other voices.
Right-click the staff to open a context menu. The actions in the context menu
apply to the current selection.
The Play button plays to the speakers the selected sections using
perfect sines as the waveform for each note. There are brief attack and
release portions of the envelope for the notes played, but for most of the
duration, a note is just played as a sine wave sustained at the volume level
specified.
MIDI output translates the notes to MIDI format and saves the file to
the disk. A MIDI song generated from ZZT Sound Plus only contains
note F-numbers, note durations, and note volume levels (implemented as
velocity) plus the appropriate signatures and other general metadata. Voice
numbers set for notes translate to channel numbers. The tool does not save
instrument names, program or control changes, aftertouches, or other types
of metadata.
The Sheet Music Generation Settings are in an early prototype stage.
There are settings to control the standard, maximum, and minimum percentages
of page width that the user wishes to allocate to measures, as well as the
limits on measures per row and rows per page.