Development: OpenMPT Format Extensions

From OpenMPT Wiki
Jump to: navigation, search

ModPlug Tracker and OpenMPT have extended the IT and XM formats in various ways. In general, these hacks are frowned upon, but here is some documentation on those hacks in case you want to support them in your own player. I am really sorry about all this mess, but all of this has grown historically way before I joined OpenMPT development.

Any numeric values are stored in little-endian format, unless noted otherwise.

Data types used in this document:

  • uint8, uint16, uint32: Unsigned integers with the given bit width.
  • int8, int16, int32: Signed integers with the given bit width.
  • char: A single character (i.e. a byte)
  • float32: Single precision IEEE float
  • VarInt: A MIDI-like variable-length unsigned integer (big-endian value where the highest bit of each byte indicates if another byte follows).
  • Square brackets [] denote an array of values. [] is a variable-length array (length is deduced from some other attribute), [42] denotes an array with 42 entries.

ModPlug Song Extensions[edit]

The following extensions exist since the (closed-source) ModPlug Tracker days. These extensions are found in IFF-like chunks, but without any padding bytes. The chunks are placed right after the header data in the IT format (i.e. after the edit history / MIDI macro block). In the XM format these chunks are placed right at the end of the file (i.e. after the sample data). At the time of writing, these chunks are always written out in the order described here, but if possible you should probably try to read them without expecting a certain order. All chunks are optional.

Chunk Header Layout[edit]

Offset Type    Content
0      char[4] Magic bytes (FOURCC)
4      uint32  Size of this chunk, excluding the header

Chunks[edit]

text (XM only)[edit]

Contains the song message (CR line endings), its length is determined by the chunk size.

MIDI (XM only)[edit]

Contains the MIDI macro configuration, in the same format as in the IT format.

PNAM[edit]

Contains the pattern names. Each pattern name is 32 bytes long and not necessarily null-terminated. The encoding is unspecified (Windows code page). The pattern names are stored continuously, i.e. there are (chunk size / 32) pattern names in the chunk, for pattern 0, pattern 1, ... pattern (chunk size / 32 - 1).

CNAM[edit]

Contains the channel names. Each channel name is 20 bytes long and not necessarily null-terminated. The encoding is unspecified (Windows code page). The channel names are stored continuously, i.e. there are (chunk size / 20) channel names in the chunk, for channel 1, channel 2, ... channel (chunk size / 20).

CHFX[edit]

Contains the plugin assignment for each channel. For every channel, there is a 32-bit integer plugin index. 0 means no plugin, 1 is the first plugin, etc...

FX00, ... FX99, F100, ... F255[edit]

Contains plugin information for each plugin slot. FX00 contains the information for the first plugin slot, FX99 for the 100th, F100 for the 101st, etc...

Offset Type     Content
0      char[4]  Plugin type ("PtsV" for VST, "OMXD" for DMO plugins)
4      char[4]  Plugin unique ID
8      uint8    Routing Flags
9      uint8    Mix Mode
10     uint8    Gain Factor * 10 (9 = 90%, 10 = 100%, 11 = 110%, etc.)
11     uint8    Reserved
12     uint32   Output Routing (0 = send to master 0x80 + x = send to plugin x)
16     char[16] Reserved
32     char[32] User-chosen plugin name (Windows code page)
64     char[64] Library name (Original DLL name / DMO identifier - UTF-8 starting from OpenMPT 1.22.07.01, Windows code page in older versions)
128    uint32   Length of plugin-specific data (parameters or opaque chunk)
132    char[]   Plugin-specific data

Routing Flags:

  • 0x01: Apply to master mix
  • 0x02: Bypass effect
  • 0x04: Wet Mix (dry added)
  • 0x08: Expand Mix [0%,100%] → [-200%,200%]
  • 0x10: Plugin will automatically suspend on silence

Mix Modes:

  • 0: normal processing
  • 1: MIX += DRY - WET * wetRatio
  • 2: MIX += WET - DRY * dryRatio
  • 3: MIX -= WET - DRY * wetRatio
  • 4: MIX -= middle - WET * wetRatio + middle - DRY
  • 5: MIX_L += wetRatio * (WET_L - DRY_L) + dryRatio * (DRY_R - WET_R)
    MIX_R += dryRatio * (WET_L - DRY_L) + wetRatio * (DRY_R - WET_R)

The first four bytes of the plugin-specific data determine the type of data. If they are all 0, then the plugin parameters follow as an array of float32 values. Otherwise, plugin type specific data follows which should be treated as an opaque chunk. For example, for VST plugins that support the effGetChunk / effSetChunk chunk, the first four bytes will be "fEvN", followed by the data to be sent to the plugin.

After the plugin information described above, more information may follow in OpenMPT modules. This information is again stored in chunks. However, there are two legacy chunks which do not have any size stored alongside the chunk identifier:

FOURCC Data Type Description
DWRT float32 Dry/Wet Ratio of the plugin. This chunk does not denote its size!
PROG uint32 Default plugin program (preset) to restore. This chunk does not denote its size!

Any chunks that will be added in the future will have proper IFF-like chunks with a 32-bit size field.

ModPlug Instrument Extensions (IT only)[edit]

To be able to address more than 256 samples in the IT format, ModPlug Tracker has an extension to store the high byte of sample indices for the instrument sample map.

By default, the last four bytes of an IT instrument (right after the pitch envelope) are unused. If they read MPTX or XTPM, 120 extra bytes follow, one for each note in the sample map. These bytes are the high byte of the sample index, i.e. they need to be multiplied by 256 and then added to the already read sample index.

OpenMPT Extensions (General Information)[edit]

In the XM format, OpenMPT instrument extensions may follow the ModPlug song extensions and may in return be followed by OpenMPT song extensions.

In the IT format, OpenMPT instrument extensions may follow the sample block and may in return be followed by OpenMPT song extensions. This is very ugly, because there might not be any samples, in which case the last thing before the extension block would be the last pattern. If there are no patterns, the last thing before the extension block would be a sample header (at least one sample header will be present even if there is no sample data – but you may even want to cover the case where there are no sample headers, for being compatible with possible future changes). So you will somehow have to keep track of the highest offset you have read into the file. If the last sample is IT-compressed, things become even more complicated: There is no way to know the compressed size of a compressed sample, so in case you want to skip sample loading, and the last sample happens to be compressed, you can do the following: Since we know that the extended instrument and song properties start right after the last sample, and since IT-compressed samples consist of chunks with a prepended 16-bit length field, you can simply read that 16-bit number, skip this amount of bytes, then check if you can read OpenMPT's XTPM or STPM magic bytes, and if not, read the next 16-bit length, skip the bytes, etc...

In pseudo code, finding the OpenMPT extensions in an IT / MPTM file could look somewhat like this:

offset = 0
lastSampleCompressed = false

// In case there are no patterns and no sample data (just empty sample slots):
if(number of samples > 0):
    offset = last sample header pointer + sizeof(ITSampleHeader)

for all samples:
    if sample is not compressed or if samples are decoded:
        lastSampleCompressed = false
        offset = max(offset, pointer to sample data + sample size)
    else if sample is compressed and samples are not decoded:
        lastSampleCompressed = true
        offset = max(offset, pointer to sample data)

// In case there is no sample data:
for all patterns:
    offset = max(offset, end of pattern data)

if lastSampleCompressed:
    while not eof:
        if next four bytes are XTPM or STPM:
            chunkID = next four bytes
            if chunkID only contains ASCII characters (all bytes are in 32…127)
                offset = current position - 4; break
            else
                skip back 8 bytes
        read uint16 value and skip as many bytes

Try reading XTPM and STPM extensions at offset

The provided data types are just for orientation, i.e. the minimum recommended field size for storing the values in memory. You must expect fields to have a different size, since sometimes they do so for historic reasons. Defensive programming is your friend. Most of the FOURCCs also just make sense when read backwards, again for historic reasons. Note that some of these extensions duplicate existing functionality of the IT/XM format. In this case, the extensions take precedence over the value previous found in the file.

OpenMPT Instrument Extensions[edit]

Instrument extensions start with the XTPM magic bytes, but there is no size indication of the total size of this block. So you have to read the following chunks and as soon as you read the STPM magic bytes for a chunk, you know you have read to far and can continue with reading OpenMPT Song Extensions instead.

Instrument extensions are stored in a peculiar way: There is one chunk per property, and it contains the values for all instruments at once. The layout is as follows:

Chunk Header Layout[edit]

Offset Type    Content
0      char[4] Magic bytes (FOURCC)
4      uint16  Size of this chunk's entry for one instrument (i.e. total chunk content size is this field × number of instruments)

Chunk Contents[edit]

The following instrument properties exist. Some of them are redundant depending on the file type and thus not present in all files. If they are redundant, they overwrite the values that were obtained from the format-specific structures.

FOURCC Data Type Formats Description
..OF uint32 IT, MPTM, XM Fade-out
...P uint32 IT, MPTM, XM Panning (0...256)
..EV uint32 MPTM Number of volume envelope nodes (if ≥ 25 nodes)
..EP uint32 MPTM Number of pan envelope nodes (if ≥ 25 nodes)
.EiP uint32 MPTM Number of pitch envelope nodes (if ≥ 25 nodes)
..BM uint16 IT, MPTM, XM MIDI Bank
..PM uint8 IT, MPTM, XM MIDI Program
..CM uint8 IT, MPTM, XM MIDI Channel
.[PV uint16[] MPTM Volume Envelope Ticks (if ≥ 25 nodes)
.[EV uint8[] MPTM Volume Envelope Values (if ≥ 25 nodes)
.[PP uint16[] MPTM Pan Envelope Ticks (if ≥ 25 nodes)
.[EP uint8[] MPTM Pan Envelope Values (if ≥ 25 nodes)
[PiP uint16[] MPTM Pitch Envelope Ticks (if ≥ 25 nodes)
[EiP uint8[] MPTM Pitch Envelope Values (if ≥ 25 nodes)
.PiM uint8 IT, MPTM, XM Plugin, 0 = no plugin, 1 = first plugin, etc.
..RV uint16 IT, MPTM, XM Ramping / Attack
...R uint32 IT, MPTM, XM Resampling Mode
  • 0: No Interpolation (1 tap)
  • 1: Linear (2 taps)
  • 2: Cubic Spline (4 taps)
  • 3: Polyphase (8 taps)
  • 4: XMMS-ModPlug (8 taps)
  • 5: Default
..SC uint8 IT, MPTM, XM Cutoff Swing
..SR uint8 IT, MPTM, XM Resonance Swing
..MF uint8 IT, MPTM, XM Filter Mode
HEVP uint8 IT, MPTM, XM Plugin Velocity Handling
HOVP uint8 IT, MPTM, XM Plugin Volume Handling
NREV uint8 IT, MPTM, XM Volume Envelope Release Node
NREA uint8 IT, MPTM, XM Pan Envelope Release Node
NREP uint8 IT, MPTM, XM Pitch Envelope Release Node
DWPM uint8 IT, MPTM Pitch Wheel Depth
LTTP uint16 IT, MPTM, XM Integer part of Pitch / Tempo Lock
PTTF uint16 MPTM Fractional part of Pitch / Tempo Lock (0...9999)

Legacy Extension[edit]

But wait, there is more! Some really old OpenMPT versions (1.17 RC1 and older, but not 1.17 RC2) do not use the instrument extensions described above. However, they also need to store the plugin reference for each instrument, which is done in another modular block following each instrument header (and possibly the MPTX / XTPM extension of that instrument header). If this legacy extension is present, the instrument header is followed by the magic bytes MSNI and an uint32 containing the modular data size.

Currently there is only one chunk in this modular data block. Its FOURCC is GULP, has no size information and contains an uint8 for the plugin index (0 = no plugin, 1 = first plugin, etc.).

OpenMPT Song Extensions[edit]

Song extensions start with the STPM magic bytes, but there is no size indication of the total size of this block. In an IT / XM file, these extensions are the last chunks in the file, so you can continue reading until you are out of data. In the case of MPTM files, some further MPTM-specific data follows. This data starts with the bytes "228" followed by ASCII charater 4 (not the digit), so you can keep reading the song extensions until you read this FOURCC.

Chunk Header Layout[edit]

Offset Type    Content
0      char[4] Magic bytes (FOURCC)
4      uint16  Size of this chunk, excluding the header

Chunk Contents[edit]

FOURCC Data Type Formats Description
..TD uint32 IT, MPTM, XM Integer part of the default tempo (required if it is larger than 255 in IT / MPTM, but also found in legacy XM files)
DTFR uint32 MPTM Fractional part of the default tempo (0...9999)
.BPR uint32 IT, MPTM, XM Rows Per Beat
.MPR uint32 IT, MPTM, XM Rows Per Measure
...C uint16 IT, MPTM Number of channels
SnhC - IT, MPTM If there are more than 64 channels in the IT / MTPM format, this chunk contains the default panning, volume and flags for channels 65+. They are encoded the same way as in the IT header, except that volume and panning are stored in an interleaved way (i.e. volume for channel 65, pan for channel 65, volume for channel 66, ...)
..MT uint32 IT, MPTM, XM Tempo Mode
  • 0: classic
  • 1: alternative
  • 2: modern
.MMP uint32 IT, MPTM, XM Mix Levels
  • 0: Original
  • 1: 1.17 RC1
  • 2: 1.17 RC2
  • 3: 1.17 RC3
  • 4: Compatible
  • 5: Compatible (FT2 Pan Law)
.VWC uint32 IT, MPTM, XM OpenMPT "Created With" version (e.g. OpenMPT 1.23.45.67 = 0x01234567)
VWSL uint32 IT, MPTM, XM OpenMPT "Last Saved With" version
.APS uint32 IT, MPTM, XM Sample Pre-Amp
VTSV uint32 IT, MPTM, XM VSTi Pre-Amp
.VGD uint32 XM Global Volume (0...256)
..PR uint16 IT, MPTM Restart Position
RSMP uint32 MPTM Resampling Mode
  • 0: No Interpolation (1 tap)
  • 1: Linear (2 taps)
  • 2: Cubic Spline (4 taps)
  • 3: Polyphase (8 taps)
  • 4: XMMS-ModPlug (8 taps)
  • 5: Default
CUES - MPTM Sample cue points for a single sample (only MPTM format). The first uint16 in the chunk indicates for which sample slot the cue points are meant. The rest of the chunk contains all cue points as uint32s.
SWNG - MPTM Contains tempo swing factors (only MPTM format). The first uint16 in the chunk indicates the number of swing rows. The rest of the chunk contains all swing factors as uint32s. A factor of 16777216 (224) is considered to be unity, i.e. does not modify the row duration. After loading, the number of swing factors should be resized to the actual number of rows per beat (in case of malformed file) and re-normalized.
.FSM - IT, MPTM, XM A bit field of generic compatibility flags. For modules made with OpenMPT 1.25 and older, the most important one is 0x01 (first bit set), which is IT-/XM-compatible playback mode. All other flags indicate which compatibility settings are toggled.
AUTH uint8[] IT, MPTM, XM Song artist, as UTF-8 string
AMIM - IT, MPTM, XM MIDI Mapping settings

MPTM Extensions[edit]

Detecting an MPTM file[edit]

There are two types of hacked IT files: In early versions of the MPTM format (used in OpenMPT 1.17.02.4x), the IMPM magic bytes are replaced by tpm., so they are not backwards compatible. Newer MPTM files use the original magic, but use a cwtv value between 0889h and 0FFFh (inclusive) in the header.

In both cases, the last four bytes of the file point to the start of the MPTM extensions. If the magic bytes "228" can be found at this start position, it is a valid MPTM file.

Order list (old)[edit]

If the cwtv is between 088Bh and 088Dh (inclusive), the order list at the start of the file is replaced by the following struct:

Offset Type     Content
0      uint16   Version of the order list. Only version 0 is defined. Reject any other values.
2      uint32   Number of items in the order list
6      uint32[] The order list, as a series of uint32 values. The number of values is determined by the previous field.

External Samples[edit]

MPTM files can reference external samples. If the cvt value in the sample header is 80h, then the sample is external. In this case, the sample pointer does not point to actual sample data but to a filename:

Offset Type   Content
0      VarInt Length of the filename
?      char[] Filename as UTF-8 string (not null-terminated)

Note that only the sample waveform should be loaded, but not its metadata: Frequency, loop points, volume, panning, auto-vibrato etc. should be read from the IT file. If the sample length according to the IT header is shorter than the actual sample, the sample data should be trimmed, too.

External samples are used exactly the same way in ITI files.

228 extensions[edit]

This extension block is not yet documented. Sorry.

RIFF WAVE[edit]

OpenMPT uses its own xtra chunk in RIFF WAVE files to store some sample properties which could otherwise not be represented in the format.

Its layout is as follows:

Offset Type   Content
0      uint32 Sample flags (0x20: Default panning is enabled)
4      uint16 Default panning (0...256)
6      uint16 Default volume (0...256)
8      uint16 Global volume (0...64)
10     uint16 Reserved (must be 0)
12     uint8  Auto-vibrato type
13     uint8  Auto-vibrato sweep
14     uint8  Auto-vibrato depth
15     uint8  Auto-vibrato rate

Optionally, when copying a sample to the system clipboard, the sample name (32 characters, null-padded) and filename (22 characters, null-padded) follow.

FLAC[edit]

Since the FLAC format has no native and standardized way to store loop information, OpenMPT follows Renoise′s way of storing loop information: FLAC supports application-defined metadata, so OpenMPT writes a metadata block with application ID riff. The block contains a smpl chunk (as defined in the RIFF specification). Similarly, extended sample properties are stored in another riff application block contaning the OpenMPT-specific xtra chunk, and sample cue points are stored in a cue  chunk.

OpenMPT also reads and writes the following non-standard vorbis comments:

  • SAMPLERATE: Contains the sample rate (as text) in case it would exceed the maximum sample rate supported by the FLAC format, 655350 Hz.
  • LOOPSTART: The start of the sample loop in frames. This tag is only read, not written.
  • LOOPLENGTH: The length of the sample loop in frames. This tag is only read, not written.