Clever Geek Handbook

How I recovered data in an unknown format from a magnetic tape

Background



As a lover of retro iron, I once bought a ZX Spectrum + from a UK seller. Complete with the computer itself, I got several audiocassettes with games (in the original packaging with instructions), as well as programs recorded on cassettes without any special designations. Surprisingly, the data from 40-year-old cassettes was readable and I was able to download almost all the games and programs from them.







However, on some cassettes I found recordings that were clearly not made by the ZX Spectrum computer. They sounded completely different and, unlike the recordings from the aforementioned computer, did not start with a short BASIC bootloader, which is usually present in the recordings of all programs and games.



For some time I was haunted by this - I really wanted to know what is hidden in them. If you could read the audio signal as a sequence of bytes, you could look in them for characters or something that indicates the origin of the signal. A kind of retro archeology.



Now that I've come all the way and look at the labels of the cassettes themselves, I smile because

the answer was right in front of my eyes all this time
— TRS-80, : «Manufactured by Radio Shack in USA»


(If you want to keep the intrigue to the end, do not go under the spoiler)



Comparison of audio signals



The first step is to digitize audio recordings. You can listen to how it sounds:





And as usual the recording from the ZX Spectrum computer sounds:





In both cases, at the beginning of the recording, there is a so-called pilot tone - a sound of one frequency (on the first recording it is very short <1 sec, but distinguishable). The pilot tone signals the computer to prepare to receive data. As a rule, each computer recognizes only "its" pilot tone by the waveform and its frequency.



I must say about the signal shape itself. For example, on the ZX Spectrum, its shape is rectangular:







When a pilot tone is detected, the ZX Spectrum displays alternating red and blue stripes on the border of the screen, indicating that the signal is recognized. The pilot tone ends with a synchro pulse, which signals the computer to start receiving data. It is characterized by a shorter (compared to the pilot tone and subsequent data) duration (see figure)



After the sync pulse is received, the computer records each rise / fall of the signal, measuring its duration. If the duration is less than a certain limit, bit 1 is written to memory, otherwise 0. Bits are collected into bytes and the process is repeated until N bytes are received. The number N is usually taken from the header of the downloaded file. The boot sequence is as follows:



  1. pilot tone
  2. header (fixed length), contains the size of the loaded data (N), name and file type
  3. pilot tone
  4. the data itself


To make sure that the data is loaded correctly, ZX Spectrum reads the last byte of the so-called byte parity (parity byte), which is calculated when you save the file operation XOR over all bytes of the recorded data. When reading the file, the computer calculates the parity byte from the received data and, if the result differs from the saved one, displays the error message "R Tape loading error". Strictly speaking, the computer can issue this message earlier if, when reading, it cannot recognize the impulse (it is missed or its duration does not correspond to certain boundaries)



So, now let's see what the unknown signal looks like:







This is a pilot tone. The waveform is significantly different, but you can see that the signal consists of repetitive short pulses of a certain frequency. At a sampling rate of 44100 Hz, the distance between the "peaks" is approximately 48 samples (which corresponds to a frequency of ~ 918 Hz). Let's remember this figure.



Now let's look at the fragment with the data:







If we measure the distance between individual pulses, it turns out that the distance between the “long” pulses is still ~ 48 samples, and between the short ones - ~ 24. Running a little ahead, I will say that in the end it turned out that "reference" pulses with a frequency of 918 Hz follow continuously, from the beginning to the end of the file. It can be assumed that during data transmission, if an additional pulse occurs between the reference pulses, we consider it as bit 1, otherwise 0.



What's with the synchro pulse? Let's look at the beginning of the data:







The pilot tone ends and the data begins immediately. A little later, after analyzing several different audio recordings, we found out that the first data byte is always the same (10100101b, A5h). The computer may start reading data after it receives it.



You can also pay attention to the shift of the first reference pulse immediately after the last 1 in the synchrobyte. It was discovered much later in the process of developing a program for data recognition, when the data at the beginning of the file could not be read stably.



Now let's try to describe an algorithm that will process an audio file and load data.



Loading data



First, let's look at a few assumptions so as not to complicate the algorithm:



  1. We will consider files only in WAV format;
  2. The audio file must start with a pilot tone and must not contain silence at the beginning
  3. The source file must have a sampling rate of 44100 Hz. In this case, the distance between the reference pulses of 48 samples has already been determined and we do not need to calculate it programmatically;
  4. The sample format can be any (8/16 bit / floating point) - since when reading, we can convert it to the desired one;
  5. We assume that the original file is normalized in amplitude, which should stabilize the result;


The reading algorithm will be as follows:



  1. We read the file into memory, at the same time we convert the sample format to 8 bits;
  2. Determine the position of the first pulse in the audio data. To do this, you need to calculate the number of the sample with the maximum amplitude. For simplicity, let's count it manually once. Let's save it to the prev_pos variable;
  3. Add 48 to the position of the last impulse (pos: = prev_pos + 48)
  4. 48 , ( , ), pos. (pos-8;pos+8) . , , pos. 8 = 48/6 — , , . , 48, , ;
  5. , . , , . , , . , . 2 : , . ;
  6. ( 0 1), (prev_pos;pos) middle_pos middle_pos := (prev_pos+pos)/2 middle_pos (middle_pos-8;middle_pos+8) . 10, 1 0. 10 — ;
  7. prev_pos (prev_pos := pos)
  8. 3, ;
  9. . - , 8, . - 8 . . A5h,


Ruby,
Ruby, .. . , . 



#  gem 'wavefile'
require 'wavefile'

reader = WaveFile::Reader.new('input.wav')
samples = []
format = WaveFile::Format.new(:mono, :pcm_8, 44100)

#  WAV ,    Mono, 8 bit 
#  samples       0-255
reader.each_buffer(10000) do |buffer|
  samples += buffer.convert(format).samples
end

#    ( 0)
prev_pos = 0
#   
distance = 48
#       
delta = (distance / 6).floor
#        "0"  "1"
bits = ""

loop do
  #    
  pos = prev_pos + distance
  
  #       
  break if pos + delta >= samples.size

  #   pos     [pos - delta;pos + delta]
  (pos - delta..pos + delta).each { |p| pos = p if samples[p] > samples[pos] }

  #    [prev_pos;pos]
  middle_pos = ((prev_pos + pos) / 2).floor

  #     
  sample = samples[middle_pos - delta..middle_pos + delta]

  #    "1"           10
  bit = sample.max - sample.min > 10
  bits += bit ? "1" : "0"
end

#  -       256   (  ) 
bits.gsub! /^[01]*?10100101/, ("0" * 256) + "10100101"

#   ,    
File.write "output.cas", [bits].pack("B*")






Having tried several variants of the algorithm and constants, I was lucky to get something extremely interesting:







So, judging by the character strings, we have a program for plotting graphs. However, there are no keywords in the program text. All keywords are coded as bytes (each value> 80h). Now we need to figure out which computer from the 80s could save programs in this format.



This is actually very similar to a BASIC program. In approximately the same format, the ZX Spectrum computer stores in memory and saves programs to tape. Just in case, I checked the keywords against the table . However, the result was obviously negative.



I also checked the BASIC keywords of the popular Atari computers, the Commodore 64 and several others, for which I managed to find documentation, but to no avail - my knowledge of the types of retro computers was not so wide.



Then I decided to go through the list , and then my eyes fell on the name of the manufacturer of Radio Shack and the TRS-80 computer. These names were written on the labels of the cassettes that lay on my table! After all, I did not know these names before and was not familiar with the TRS-80 computer, so it seemed to me that Radio Shack was an audio cassette manufacturer, such as BASF, Sony or TDK, and TRS-80 was the duration of playback. Why not?



Computer Tandy / Radio Shack TRS-80



It is very likely that the audio recording in question, which I gave as an example at the beginning of the article, was made on such a computer:







It turned out that this computer and its variants (Model I / Model III / Model IV, etc.) were very popular in their time (of course, not in Russia). It is noteworthy that the processor used in them is also Z80. A lot of information can be found on this computer on the Internet . In the 1980s, information about the computer was circulated in magazines . At the moment, there are several computer emulators for different platforms.



I downloaded the trs80gp emulatorand for the first time I was able to see how this computer worked. Of course, the computer did not support color output, the screen resolution was only 128x48 pixels, but there were many extensions and modifications that could increase the screen resolution. There were also many options for operating systems for this computer and options for implementing the BASIC language (which, unlike the ZX Spectrum, in some models was not even "flashed" into ROM and any option could boot from a floppy disk, as well as the OS itself)



. I found a utility for converting audio recordings into CAS format, which is supported by emulators, but for some reason I could not read the recordings from my cassettes with their help.



Having figured out the format of the CAS file (which turned out to be just a bitwise copy of the data from the tape, which I already had in my hands, with the exception of the header with the presence of a sync byte), I made several changes to my program and was able to get a working CAS file at the output, which it worked in the emulator (TRS-80 Model III):







The last version of the utility for converting with automatic detection of the first pulse and the distance between the reference pulses I designed as a GEM package, the source code is available on Github .



Conclusion



The traversed path turned out to be an exciting journey into the past, and I am glad that in the end I found a solution. Among other things, I:



  • I figured out the format for saving data in the ZX Spectrum and studied the built-in ROM routines for saving / reading data from audio tapes
  • I got acquainted with the TRS-80 computer and its varieties, studied the operating system, looked at examples of programs and even had the opportunity to debug in machine codes (after all, all the Z80 mnemonics are familiar to me)
  • I wrote a full-fledged utility for converting audio recordings to CAS format, which can read data that is not recognized by the "official" utility


More articles:


All Articles