Development: Formats/IT

2024-06-14T14:10:30Z

Paper: schism's version.c is not dependent on CRT anymore

General information concerning the IT file format. Any variable names in this document refer to the original variable names from [https://github.com/schismtracker/schismtracker/wiki/ITTECH.TXT ITTECH.TXT]. Variable names and hexadecimal numbers (in little-endian formats) are indicated by a <tt>monospace font</tt>.

Note: Impulse tracker is now open source! You can grab the [https://github.com/sagamusix/impulsetracker source code] at GitHub.

== Tracker IDs ==

The IT header contains a field, <tt>Cwt/v</tt>, which is used to identify the application that was used to create the file.
The following custom tracker IDs (read as little-endian hexadecimal numbers) are known to be found in the <tt>Cwt/v</tt> field:

* <tt>0xyy</tt> - Impulse Tracker x.yy - Many other trackers, like ModPlug Tracker, disguise as various versions of Impulse Tracker, so this is <strong>not</strong> a reliable way to tell if a file was really made with Impulse Tracker. See [[#Detecting other trackers|below]] for detecting such trackers.
* <tt>1xyy</tt> - Schism Tracker x.yy up to Schism Tracker v0.50, later versions encode a timestamp in the version number - see Schism Tracker’s <tt>[https://github.com/schismtracker/schismtracker/blob/master/schism/version.c version.c]</tt> or OpenMPT’s <tt>[https://source.openmpt.org/browse/openmpt/trunk/OpenMPT/soundlib/Load_it.cpp Load_it.cpp]</tt>.
* <tt>4xyy</tt> - pyIT x.yy
* <tt>5xyy</tt> - OpenMPT x.yy - lower two bytes of version number are stored in the "reserved" header field starting from OpenMPT 1.29.10.00 in compatible mode (otherwise OpenMPT song extensions are used as before).
* <tt>6xyy</tt> - BeRoTracker x.yy
* <tt>7xyz</tt> - ITMCK x.y.z (N.B. the user can override the value of the <tt>Cwt/v</tt> header by the <tt>-w</tt> command-line switch or the <tt>#TRACKER-VERSION</tt> command in the input file)
* <tt>7FFF</tt>, <tt>Cmwt = 0215</tt>: munch.py
* <tt>8xyy</tt> - Tralala x.yy; prealpha versions, built from svn, before the first release identify as version 0.00 (i.e. Cwt/v=8000)
* <tt>Cxyy</tt> - ChickDune ChipTune Tracker x.yy (currently non-public alpha version)
* <tt>DAEB</tt> - spc2it (after commit d3d1b524032f668be8ee20d25339e1c635fb9bcf)

Are you a tracker developer and your tracker can save IT files? Please add your Tracker ID to this list!

Once we run out of tracker IDs, a more sophisticated approach will have to be found. For example, the ID <tt>0FFF</tt> could be reserved, and a pointer to extended tracker information could be placed in the <tt>reserved</tt> header field.

== Detecting other trackers ==

Some trackers disguise as Impulse Tracker 2, but there are ways to uncover this diguise:

=== BeRoTracker ===

Earlier versions of BeRoTracker, which did not use the tracker ID described above, can be identified by the four byte <tt>MODU</tt> magic which can be found after the Edit History / MIDI Macro / ModPlug extension block. <tt>Cwt/v</tt> is <tt>0217</tt> and <tt>Cmwt</tt> is <tt>0214</tt> or <tt>0200</tt> (overlaps with ModPlug Tracker). The edit history flag is set and there are always 0 entries in the edit history.

=== CheeseTracker ===

* <tt>Cwt/v</tt> and <tt>Cmwt</tt> are both <tt>0214</tt>
* Instrument mode flag is always set. The only other flags that can be optionally set are: Stereo Mode, Linear Slides, Old Effects, Compatible Gxx.
* The only <tt>special</tt> flag ever set is the song message flag (even though row highlights are always written).
* <tt>reserved</tt> is <tt>0</tt>
* All sample and instrument filenames read <tt>XXXXXXXX.YYY</tt> (obviously this should not be used as a stand-alone detection criterion, as files re-saved with other trackers will still use these filenames).

=== ChibiTracker ===

<tt>Cwt/v</tt> and <tt>Cmwt</tt> are both <tt>0214</tt>, <tt>reserved</tt> reads <tt>CHBI</tt>.

Further interesting criteria: Sample data is stored directly after header. All sample and instrument filenames read <tt>-DEPRECATED-</tt> (obviously this should not be used as a stand-alone detection criterion, as files re-saved with other trackers will still use these filenames). ChibiTracker also uses a line feed character (<tt>0A</tt> or <tt>\n</tt>) instead of a carriage return (<tt>0D</tt> or <tt>\r</tt>) in the song message.

=== ModPlug Tracker ===

* '''ModPlug Tracker 1.0 pre-alpha 4 (and possibly earlier) - 1.0 alpha 4''': <tt>Cwt/v</tt> is <tt>0202</tt>, <tt>Cmwt</tt> is <tt>0200</tt>, <tt>reserved</tt> is <tt>0</tt>, row highlights are both 0. These attributes alone are not enough to fingerprint these ModPlug Tracker versions, but luckily it also stores the remaining data in a very peculiar order: patterns first, then instrument headers, and then interleaves sample headers and their corresponding sample data. These versions also do not optimize empty 64-row patterns with a 0 parapointer, so they can be identified by checking if the first pattern parapointer is not 0 and less than the first sample parapointer.
* '''ModPlug Tracker 1.0 alpha 5''': <tt>Cwt/v</tt> is <tt>0214</tt>, <tt>Cmwt</tt> is <tt>0200</tt>, <tt>reserved</tt> is <tt>0</tt>. Pattern highlights and edit history flags are not set. Instruments are 560 bytes apart.
* '''ModPlug Tracker 1.0 alpha 6 / beta 1 / beta 2''': <tt>Cwt/v</tt> is <tt>0214</tt>, <tt>Cmwt</tt> is <tt>0200</tt>, <tt>reserved</tt> is <tt>0</tt>. Pattern highlights and edit history flags are set. Instruments are 560 bytes apart up to and including beta 1, in beta 2 they are 557 bytes apart.
* '''ModPlug Tracker 1.0 beta 3.2 - 1.07''': <tt>Cwt/v</tt> is <tt>0214</tt>, <tt>Cmwt</tt> is <tt>0202</tt>, <tt>reserved</tt> is <tt>0</tt>. Instruments are 557 bytes apart.
* Newer versions of '''ModPlug Tracker''', '''OpenMPT 1.17 in compatible mode''': <tt>Cwt/v</tt> is <tt>0217</tt>, <tt>Cmwt</tt> is <tt>0200</tt>, <tt>reserved</tt> is <tt>0</tt>. If there are any instruments in the file, the instrument <tt>TrkVers</tt> field is <tt>0211</tt> for ModPlug Tracker and <tt>0220</tt> for OpenMPT. ModPlug Tracker writes <tt>FF</tt> for unused channels in the panning map, OpenMPT never does this. When an IT file uses all 64 channels (so the panning map cannot be used for distinction), the presence of ModPlug extensions (channel names, plugin chunks, etc.) can help telling MPT 1.16 and OpenMPT 1.17 compatibility export apart. The latter also does not save a final "---" order list item, while MPT 1.16 does.

=== OpenMPT ===

* OpenMPT 1.17.02.20 to OpenMPT 1.17.02.25 wrote the value <tt>0300</tt> in both the <tt>Cwt/v</tt> and <tt>Cmwt</tt> fields.
* OpenMPT 1.17.02.26 to OpenMPT 1.18 wrote the value <tt>0888</tt> in both the <tt>Cwt/v</tt> and <tt>Cmwt</tt> fields.
* Later versions of OpenMPT use the tracker ID provided above. In that case, the <tt>reserved</tt> field reads <tt>OMPT</tt> if the file was saved in normal mode, and is <tt>0</tt> (or the lower two bytes of version information starting from OpenMPT 1.29.10.00) if it was saved in compatible mode.

=== OpenSPC conversions ===

OpenSPC is a tool to convert SNES music from the SPC format to the IT format. If all the following conditions are true, you can be fairly sure that a module was written with OpenSPC:
* <tt>Cwt/v</tt> is <tt>0214</tt>.
* <tt>Cmwt</tt> is <tt>0200</tt>.
* <tt>Flags</tt> is <tt>09</tt> (Linear Slides, Stereo Playback).
* <tt>Special</tt> is <tt>0</tt>.
* Both pattern highlight values are <tt>00</tt>.
* No instruments are used.
* <tt>PatNum</tt> + 1 == <tt>OrdNum</tt>.
* Global volume is 128 decimal.
* Master volume is 100 decimal.
* Initial Speed is 1.
* Pan Separation is 128 decimal.
* Pitch wheel depth is 0.
* There is no song message, i.e. message length and offset are 0.
* <tt>reserved</tt> is 0.

Earlier versions of spc2it (before commit d3d1b524032f668be8ee20d25339e1c635fb9bcf) will be mis-identified as OpenSPC, as the tool closely follows the way its predecessor writes out IT files.

=== Unknown XM to IT conversion tool ===

There is a currently unknown XM to IT conversion tool which can be identified easily by the fact that it writes unsigned samples (which was the default only until Impulse Tracker 2.01). However, only sample slots with actual sample data should be considered for this test, as re-saving such a file in Impulse Tracker will not reset the unsigned flag for empty slots. If the following conditions are true, you can be fairly sure that a module was written with the unknown converter:
* <tt>Cwt/v</tt> is <tt>0204</tt>.
* <tt>Cmwt</tt> is <tt>0200</tt>.
* <tt>Flags</tt>: Stereo Playback, Instrument Mode, Old Effects are always set, Linear Slides are optional.
* <tt>Special</tt> is <tt>0</tt>.
* Both pattern highlight values are <tt>0</tt>.
* Global volume is 128 decimal.
* Master volume is 48 decimal.
* Pan Separation is 128 decimal.
* Pitch wheel depth is 0.
* There is no song message, i.e. message length and offset are 0.
* <tt>reserved</tt> is 0.
* Song title is no longer than 20 characters.
* All channels are panned centre (32) or muted, and their volume is 64 decimal.

To be extra sure, you can check all of these properties, but just checking <tt>Cwt/v</tt>, <tt>Cmwt</tt> and unsigned sample data is probably enough.

=== UNMO3 ===

UNMO3 is a tool to convert an MO3-compressed module file back into its original format.

In IT files created with UNMO3, the following are always true:

* <tt>Cwt/v</tt> and <tt>Cmwt</tt> are <tt>0214</tt>
* <tt>reserved</tt> is <tt>0</tt>
* <tt>pwd</tt> is 0
* Both row highlights are 0
* Flags: Use MIDI Pitch Controller (bit 6) and Request embedded MIDI configuration (bit 7) are never set

This partially overlaps with Impulse Tracker 2.14 and CheeseTracker fingerprints, however:
* CheeseTracker should write non-zero row highlights, even when resaving an UNMO3-ed file.
* It is highly unlikely that Impulse Tracker 2.14 writes a <tt>reserved</tt> value of <tt>0</tt>. If it writes zero row highlights, the row highlight <tt>special</tt> flag is set.
* Impulse Tracker 2.14 never writes an empty edit history (see below).
* CheeseTracker always uses instrument mode.

Some older versions of UNMO3 write out IT files that are malformed in some way or another, but it is easy to detect and work around. Some UNMO3-ed IT files are floating around in the wild, so it may be worthwile to implement those work-arounds:

==== Special flags prior UNMO3 2.4.0.2 ====

* UNMO3 2.4 (released in 2008) and older sets the <tt>special</tt> flag bits 1 (edit history) and 2 (row highlights) whenever bit 3 (MIDI macros) is set as well.
* UNMO3 2.4.0.1 (January 2011) always sets the <tt>special</tt> flag bit 1 (edit history), but the behaviour for bit 2 (row highlights) remains as in previous versions.
* UNMO3 2.4.0.2 (July 2011) never sets the row highlights flag.

==== Sample mode prior to UNMO3 2.4.0.1 ====

UNMO3 2.4 and older reserves space for the instrument parapointers (as many as there are samples) even when the IT file is in sample mode. This can be detected by the fact that 4 * <tt>SmpNum</tt> zero bytes follow after the pattern parapointers. These bytes need to be skipped in order to be able to read MIDI macros and ModPlug plugin settings which may potentially follow.

UNMO3 2.4.0.1 fixes this problem.

==== Edit History prior to UNMO3 2.4.0.1 ====

As mentioned above, UNMO3 older than version 2.4.0.1 may or may not set the edit history flag, but they always write two edit history length bytes (zeroes) in the file anyway. Keep this in mind, as MIDI Macros and ModPlug plugin settings will not load properly if you don’t.

* '''Edit history flag is set''': Nothing needs to be done. We can still use this for identifying UNMO3-saved IT files, because Impulse Tracker never writes an edit history of length 0.
* '''Edit history flag is not set''': This is the troublesome case due to MIDI macros and plugin settings. However, a reliable way to detect such malformed IT files would be to try to read the edit history length if all of the above detection criteria for UNMO3 are fulfilled. For those old UNMO3 versions, you should read two zero bytes. If that is the case, it means that the first MIDI macro would be empty (if MIDI macros are embedded) or the first plugin chunk ID starts with two zero bytes (impossible). Since UNMO3 writes out default values for the nine global MIDI macros, the first MIDI macro would have to start with “FF” (MIDI Start) and not be empty, so it is clear that the two zero bytes belong to the (empty) edit history.

Later version of UNMO3 set the correct header flag for the edit history and write the two zero length bytes.

== Special Header Flags ==

The following "Special" flags are not documented:

* Bit 1 (<tt>02</tt>): If set, edit history information (see [[#Edit History|below]]) is present.
* Bit 2 (<tt>04</tt>): If set, the row highlights (at offset <tt>1E</tt> - <tt>1F</tt>) should be read. If not set, they should be ignored.

== Edit History ==

It is little known that the latest versions of Impulse Tracker keep track of when an IT file is opened and how long it is left open.
The edit history can be found directly after the IT header and before the possibly present MIDI macro configuration.

A 16-Bit integer <tt>num</tt> tells how many edit history blocks are present. Every block is 8 bytes long, so the 16-bit integer is followed by <tt>8 * num</tt> bytes of edit history information.

Each history block consists of a 16-Bit FAT date, a 16-Bit FAT time and a 32-Bit DOS timer, which contains the time how long the file was open in the editor, in 1/18.2th seconds.
Information on how to decode the FAT date and time stamps can be found on the [https://msdn.microsoft.com/en-us/library/windows/desktop/ms724247%28v=vs.85%29.aspx MSDN].

Note: Many programs don’t write an edit history, but they set the edit history flag in the header and write the two zero bytes for the history length. Many of them also ignore the edit history flag when loading files and always read the two <tt>num</tt> num bytes (ignoring the special flag), so if you want to write out an IT file that does not have an edit history and do not mind about the two extra bytes, the safest (most compatible) way is to set the edit history flag and write two zero bytes for <tt>num</tt>.

Also note that none of this is really relevant if you are sure that you never write out the MIDI macro configuration or ModPlug extensions, since these are the only two things that could normally follow the edit history. So if you have no MIDI macros and no ModPlug extensions, it is always safe to not set the special flag and not write the two extra bytes.

== Edit Timer ==

Before the addition of the full Edit History, a simple Edit Timer was stored in the <tt>reserved</tt> field of the file header. Starting from Impulse Tracker 2.07, this field contains the total edit time as a 32-Bit DOS timer (same resolution as in the Edit History). Starting from Impulse Tracker 2.08, the timer is encrypted. To decrypt:

editTime = editTime XOR 'ITRK'
editTime = editTime RoR 7
editTime = -editTime
editTime = editTime RoL 4
editTime = editTime XOR 'JTHL'

== Proposed way to store compressed stereo samples ==

Since Impulse Tracker itself does not support stereo samples, a common way of how to decode compressed stereo samples had to be found - After all, it is possible to just treat them as one continuous stream (not stopping the encoding / decoding process after the end of the left channel), or to treat them as two separate, consecutive streams. XMPlay was probably the only player that could handle compressed stereo samples even before there was a single implementation for saving compressed stereo samples. GreaseMonkey has decided to follow [https://www.un4seen.com/forum/?topic=12448.msg86479#msg86479 XMPlay's way] of decoding compressed stereo samples by treating them as two streams of the same length, so OpenMPT follows this example. If you intend to add IT sample compression support in your application, you should do the same to ensure compatibility across as many applications as possible.

== OpenMPT Hacks ==

For a list of all the file format hacks that ModPlug Tracker / OpenMPT added to the IT format over the years, see the [[Development: OpenMPT Format Extensions|OpenMPT Format Extensions]] article.

[[Category:Development|Formats/IT]]
[[Category:IT Format|Format Specification]]

OpenMPT Wiki - User contributions [en]

Development: Formats/IT