I decided to write this post in the hope that the experience I gained and the solutions I found may be useful to those who face similar problems in projects where data exchange between several Teensy and a PC is supposed to be.
It started at the end of March. My friend (a pure electronics engineer and never a programmer) asked for help with programming the installation he wanted to start. Outwardly, everything looked quite simple: there are 2 high-speed ADCs ( ADS7886 ) plus some additional peripherals that need to be controlled. ADCs should be polled as often as possible, preferably with their maximum frequency (1 MHz), some part of the data should be saved unchanged, and the rest should be processed. And all data, after completion of measurements and processing, transfer to a PC. Considering the requirement "as often as possible" I suggested using 2 pieces of Teensy 4.0- they have excellent parameters (600 MHz clock frequency, a lot of memory, a good connection, and all this is in a microscopic form factor). Well, in addition - the ability to program in the advanced version of the Arduino IDE (something more complicated for such a task is already overkill).
The coding and debugging was done remotely - first using TeamViewer, and then when it started to interrupt the session after 60 seconds, we switched to Google Remote Desktop. Physically, I have never appeared near the installation and I do not know how it looks and generally works.
A couple of months before that, I made my project on Teensy - an 8-channel biopotential meter based on ADS1298, and was very pleased with this "crumb" (as the name of the controller sounds loosely translated). Therefore, I recommended my friend to use a pair of Teensy 4.0, with which he agreed and was very pleased.
In addition, one Tincy must be the master (COM3), and the other - the slave (Slave, COM5), i.e. the master is controlled directly from the PC, and the slave (at least partially) is controlled by the UART from the master. At the same time, it was initially assumed that both controllers would be connected to a PC and transfer the accumulated data to it via USB.
Initially, the vision of the problem looked something like this: we debug the functionality on the master Tinsey, then transfer the software to the slave, adjust it in accordance with its functionality, and make sure that everything works. Profit.
Initially, I assumed that 2-2.5 weeks would be enough for everything. One week will obviously not be enough, but as many as THREE weeks ... Well, what is there to do for so long ?? After all, all the pitfalls should be raised when debugging the software for the leader, and then - you just need to carefully examine the logic of the follower.
Ok, no sooner said than done.
In the beginning, everything went quite briskly. Host Tinsey collected data (with an interval of slightly less than 2 microseconds, it didn't work out faster) and sent it to the PC. Pasha (as I will conventionally call my comrade) was quite happy with it. The next stage is to carry out sequentially 100 measurement cycles, processing each cycle, storing the processing results for each cycle in Tinsi memory.
But when transferring the received data, the first problem arose. When using the same transmitting function, the first data array ("raw") was truncated in the most cruel and merciless way, while the second array (with the same data structure and approximately the same length) was passed without problems. Those. the transfer of the first array could be interrupted at the 1048th, 2036th or 4076th byte (with an array length of 28800 bytes), after which Tinsey considered his task to be completely completed.
Since in this project the main task is not the beauty and elegance of the code, but its maximum reliability and maximum forward speed, I decided to transfer the first array in packets. The packet size was chosen to be 512 bytes. Tincy sends the total size of the file to the PC, the PC calculates how many packets should be sent, and asks Tincy for the next packet, indicating its offset and length. The length should be indicated because the last packet is usually less than 512 bytes in size.
By the way, in the end, the second array was no longer transmitted entirely, and I also transferred it to transmission in packets.
Okay, we survived this trouble. Host Tinsey began to transfer all the necessary data to the PC without any problems. Now it was the follower's turn.
Naturally, it was also connected to a PC, and I registered in the PC program (it was written in WPF) to work with two COM ports: COM3 - Tinsi's master, COM5 - slave. Excellent. We turn on the installation, and ... I see a command in the COM5 received data handler that MUST be transferred to COM3! Naturally, since this command does NOT go to COM3, everything goes to hell. How is this, comrades ??
Following our main principle ("forward and fast"), it was decided to completely abandon direct communication between the slave Tincy (COM5) and the PC, and transfer all control of COM5 and transfer of data from it exclusively through the UART, using the master Tincy (COM3) in as a transfer station. As a result, the normal operation of COM3 was restored. Now fine.
However, after that I had to spend several days to carefully write down the logic of interaction between PC, COM3 and COM5. Considering that what is happening in the depths of Tincy remains a black box for the programmer (if you do not use purchased software for debugging for Arduino, for example Visual Micro ), the only way I can judge its more or less normal operation is the data that it issues to PC.
Therefore, all the work of the host Tincy was completely and completely subordinated to the commands of the PC: the PC issues a "preparatory" command, Tingxi confirms the receipt of the command, the PC issues the command for execution. A kind of handshake. Tincy's responses are transmitted in binary or text form (depending on the operating mode), for example: "6 28800", where 6 is the command code (I want to transmit the first data array), 28800 is the amount of data to be transferred, in bytes. A similar handshake was used for the interaction between both Tinsey.
This approach to interaction allows you to keep a log of what is happening, which can, theoretically, identify the place where the problem arises. If you need to go to the transfer of binary data (batch transmission of measurement results), then in this case the data is not written to the log.
Another advantage of the handshake is that it allows you to get rid of the "sticking" of commands: due to the fact that Tinsey transfers data to the PC at a high speed (data transfer via USB is at a speed of 12 Mbps), he has time to transmit the next command so that it goes to the end of the PC receive buffer, for example: "6 28800 5"), where 5 is the command for the next operation. Data transfer delays didn't help. Here you either need to parse the received string (considering all possible options), or use a handshake, which allows you to work without a headache.
We return to our rams. Connect COM3 and COM5, launch the program. COM3 works without problems. But COM5 ... Instead of the expected (conditionally) "6 28800" - we have "6 1" or "6 8400" or in general "6 0". That is, COM5 receives a control command from COM3 and reacts to it, but the data that it sends is in a deep spherical vacuum. So who's to blame?
The first thing that comes to mind is that there is a problem somewhere in the UART. There is simply no one else to sin against. Excellent. We make a test program: COM5 generates 100 numbers (from 0 to 99) and sends them to COM3, and that one directly sends the received data to the PC. Indeed, the data is transmitted with large gaps. Even if the baud rate is reduced from 500 kbaud to 9600 baud. WTF ???
I lost a couple more days trying to defeat this garbage by a scientific poke. Did not help.
But here, as they say, "there would be no happiness, but misfortune helped."
From the very beginning of working with two Tincy, we noticed that you can program them only one at a time: if, when connecting only COM3 (or only COM5), you can upload the program to them using the Arduino IDE buttons on the screen, then if both Tincy are connected to the PC, they can be more or less successfully programmed only by physically pressing the reset button on the Tinsey board. Those. compiled software for COM3 - pressed the button, wait until the cherished Reboot Ok appears. After that I compiled the software for COM5, pressed its button, waiting for Reboot Ok.
So, this mechanism also broke down at some point. Those. COM5, when trying to reprogram, immediately ceased to be detected as a Tinsey device. Regardless of how many times we disconnected it, waited for this device to disappear from the list of devices and connected, waiting for it to appear in the list. Moreover, the disease has spread to COM3 as well!
Then vague doubts began to torment me. Isn't the fact of simultaneous connection of COM3 and COM5 to the same PC a source of problems?
To test this hypothesis, I persuaded Pasha to power COM5 from a third-party PowerBank (old laptop). And - lo and behold! - UART began to transmit data without loss and at any speed (from 9600 to 500k). After that, Pasha powered COM5 from a + 5V source, which is available in the installation, placing a diode in series for protection, as recommended in the Tinsi manual.
The only but: initially in the tests COM5 did not want to transfer more than 64 bytes (1 command byte + 63 data bytes), after 63 bytes of data the counter of transmitted data was reset to 0. The problem was solved by expanding the transmit / receive buffers to 9000 bytes:
Serial1.addMemoryForRead (buffer, 9000);
Serial1.addMemoryForWrite (buffer, 9000);
So in operating mode, COM5 is powered from its own source, and during programming, it is connected to a PC via USB (with COM3 disconnected). Unfortunately, this did not solve the problem with the transfer of measurement results from COM5, but now I knew for sure that the UART had nothing to do with it, and I had to look in my own code. After that, the victory of reason over sansaparilla was perceived only as a matter of time.
It was the 28th day since the start of work on the project ... And then another month was lost because I fell ill with COVID-19 ... But in the end, work on the project resumed. And then there was another problem related to the UART.
Data transfer - both single-byte commands and 3-5 byte arrays (command + parameters) from COM5 to COM3 took place without problems - I saw the transmitted data on the PC, and there were no errors. But a similar array did not pass from COM3 to COM5 - only the first byte. Moreover, the function that receives data in both controllers was the same. Rearranging the controllers did not solve the problem. Since the array transferred from COM3 to COM5 contained information about the offset of the start of data reading and about the amount of data requested by the PC, in response I, naturally, did not receive anything. Adding delays to the UART buffer read routine did not solve the problem.
We already wanted to use I2C to transfer the accumulated data from the slave Tincy to the master, since there were enough free legs, but here there was a bummer - there was a high potential at the output, where the clock signal should have been, and there were no pulses. While browsing the Web in search of a solution to the problem, I came across on some forum a mention that in Arduino, simultaneous operation of UART and I2C is impossible. True, this applied to another controller, but, possibly, to us too.
I could not completely abandon the UART - I needed the start of measurements in both controllers to be strictly tied to the sync signal (the negative edge of the same sync pulse), coming with a frequency of several kHz, and I could not guarantee this using I2C ... UART, operating at a frequency of 500 kHz, coped with this problem quite well.
So I gave up on the idea of using I2C and went back to a "pure" UART. There was also an idea to use an additional SPI channel, but Pasha resisted here - he didn’t like the need to climb into the Tincy board with a soldering iron: in Tincy, only SPI0 is connected to the connectors, and the SPI1 and SPI2 pins are pads on the bottom side of the board.
The idea to put an intermediate buffer and use the same SPI0 bus to operate COM5 in Master mode when polling the ADC and in Slave mode when transferring data arrays to COM3 also met resolute resistance - Pasha was panicky that somewhere in the program there would be a failure, conclusions controllers will go to the wrong state, and both controllers will kill each other. Since it was still Pashin's project, I did not punch this solution much, although, given the clarity of work with SPI with peripherals, I assumed that with synchronous data transfer everything should work like clockwork.
Thus, I had no choice but to resist with all my might and push this UART cart until it gets where it needs to be.
After a couple of days trying to find the answer to the question "what the hell ?!" I wrote another micro-test, and suddenly - lo and behold! - saw a data packet transmitted without distortion! A cursory analysis showed that in the "operating" mode, the data from the UART receive buffer was read only once - at the beginning of the command processing function. She accessed the subroutine for reading from the buffer, and considered the first received byte as a command, and the subsequent ones as its parameters or data. At the same time, as the "reverse" output to the PC showed (an array of bytes received by the slave Tincy and transmitted back to the master, and from him to the PC), the slave Tincy "saw" only the first byte, as mentioned above.
Approximately the same trouble awaited when transferring a large test data array from COM5 to COM3 - during testing, the command was received and correctly processed, but instead of data I saw only zeros. However, in the new test, I "automatically" put another call to the read function from the UART input buffer inside the test. And the data appeared!
After inserting the extra read into the code for the slave Tinsey, the problem of "disappearing" data was gone. True, along the way it turned out that data is best transmitted only in small packets (up to 64 bytes, despite the "expansion" of the read / write buffers in both Tincy), well, the delays had to be inserted, but all this suited Pasha.
In the end, after additional marching of the logic in the Vlendish way (read - debugging and debugging), everything worked as expected.
The End.