Douglas Hurley and Bob Behnken in the Endeavor capsule



SpaceX is rapidly developing projects in several directions. The first stage of the Falcon 9 rocket, after launching the payload into space, returns to Earth for reuse, similar to testing for Starship prototypes. Crew Dragon is delivering the crew to the ISS, the second generation of Dragon trucks is being prepared. A swarm of Starlink communication satellites is producing over a hundred megabits per second for real users of the open beta test.



All of these projects require a high degree of automation, which is the responsibility of the software development team. SpaceX experts talk about it with pleasure: it is not only important for attracting applicants for open positions, but also interesting for everyone else.

Complexity of space

On December 14, 1966, in an unmanned test launch of the Soyuz, the 7K-OK No. 1 spacecraft landed on the launch pad: the lighter on one of the engines did not work. The automation stopped the launch sequence before the rocket could separate from the surface of the launch pad. The personnel approached the rocket to inspect it and assess the possibility of re-launching. Soyuz's emergency rescue system (SAS) suddenly went off. Its powder engines carefully carried the descent vehicle to an altitude of 700 meters and put it under the power of parachutes, but also ignited the spilled flammable liquid of the thermal control system of the instrument compartment, which remained on Earth. The top of the rocket lit up. As Chertok recalls , the memory of the Nedelin disasterforced people to leave the launch pad at a run. One person died.



The elucidation of the reasons for the activation of the SAS on a stationary rocket began even before extinguishing the launch pad. During flight, the rocket constantly compares the deviations of the inertial reference system from the calculated trajectory. If the difference is too large, then SAS is triggered. The rocket standing on the launch pad still moves: it rotates with the Earth, and the gyroscopes are "tied" to the stars. When designing emergency systems, the Earth was assumed to be stationary. In 27 minutes , about 8 degrees " ran ", and an ignition signal was received on 32 SAS pyrotechnic charges.



This "bug" was overlooked even without the complexities of modern computers. All critical systems of any spacecraft are now controlled by electronics. This is not even the " most expensive hyphen in the history " of the 1962 Mariner 1 probe: in modern systems, the space for errors amounts to tens, hundreds of thousands of lines of code. Any noticeable software failure degrades the quality of work or even leads to the loss of the device.



It seems that the maturity of the direction and the very culture of aerospace development have negated any "childish" mistakes. But this is not the case. Until now, rockets explode with variable conversion error , rovers hang with priority inversion , and orbital probesfall due to a value in the wrong measurement system . Even Starliner, a direct competitor to Crew Dragon, did not make it to the ISS due to the quality of the software.





SpaceX has posted this video with a love of its culture of trial and error



. SpaceX's example is curious against this backdrop. This relatively young company with notable accomplishments would not have happened without software that small teams quickly develop and test.

Falcon on x86

It's not just the reliability of the code that causes problems - space is hostile even to hardware. On Earth, we are protected from radiation by the magnetic field and the atmosphere. In space, a stream of high-energy particles can easily turn one into zero in a memory cell, or even disable components altogether. Space microelectronics at least requires special certification, and sometimes resorts to special microcircuits, for example, silicon-on-sapphire semiconductor technologies .



SpaceX is ignoring the established squeamish attitude towards space equipment. Since its inception , the company has stood out for its love of relatively cheap components. For example, in 2005, journalists reported that computers in a Falcon 1 rocket communicated over a regular Ethernet cable.



As SpaceX engineers told ( 1 , 2 ) at the GDC 2015 conference, on the Falcon 9 rocket, three dual-core computers of x86 architecture are installed at each stage. In each computer, on each of the two cores, an operating system with flight software runs independently of each other. The rocket also contains PowerPC microcontrollers. They control various actuators: motors, lattice rudders, and so on.



All this equipment is integrated into the actor-judge system.

• Almost everything is expressed in the form of a control loop, which is traditional for TAU: many times a second data is received from the sensors, a decision is made on them and the past states of the system, the computer issues signals to the devices.
• . . , .
• . , , . , .
• Falcon 9 .

This architecture both simplifies testing on Earth and provides the required level of resistance to radiation. There is no need for special expensive microcircuits, which, moreover, may have an architecture that is familiar only to a small number of developers in the labor market.



Control computers are tested on the so-called table rocket bench, "a rocket on a table." Falcon 9 "brains" are laid out on a flat surface and connected as they work in a real rocket. Then the specialists launch a full simulated flight, monitor the system's behavior, its performance and possible failures. During the simulation, one of the flight computers can be turned off in order to understand how the rocket will respond to this.



Also, the entire control system is able to virtualize one workstation, which makes it possible to mass automated testing and code verification for flight in just a day.



A similar triple-redundant system is installed in the Dragon spacecraft, it was said at GDC 2015. In 2020 responses, employees vaguely hinted that the Crew Dragon spacecraft is controlled by a separate quad-core processor, comparable in processing power to a smartphone five years ago.



In general, SpaceX does not report exact processor models. Possibly PowerPC controllers are radiation-resistant RAD750well-proven in rovers and probes, and x86 processors are industrial embedded solutions with low heat generation and modest performance.



But space is not needed much. The ISS itself is controlled by an Intel 80386SX-20 processor with an 80387 math coprocessor. Even at the time of the station's launch, it was a product of ten years ago.

Into space in a browser

On May 30, SpaceX launched humans into orbit for the first time. Finally, there was a second option for delivering the crew to the International Space Station. Since 2011, the only solution for this has been the Russian Soyuz.



The audience of the online broadcast noticed how much the Crew Dragon ship looked larger than the three-seat Soyuz capsule. With a similar internal volume, the SpaceX spacecraft is 4 meters in diameter versus the Union’s 2.2 meters. SpaceX initially advertised the spacecraft as a seven-seater, but NASA will launch four astronauts on the manned Dragons.



In addition to the physical differences, the most noticeable was the way people navigated the ship. "Soyuz" does not deviate from the traditions of aerospace engineering: the crew presses the toggle switches and keys, and signal boards display information. Cru Dragon does everything in its own way. For a futuristic SpaceX spaceship, the main element is the touchscreen.



Crew Dragon screens are powered by the Chromium browser engine, that is, the interfaces are built in HTML using web components, JavaScript and CSS. We wrote our own reactive library internally. The development of interfaces follows the Agile methodology with a high bar for coverage by unit tests.





Screenshot of the browser docking simulator.



Before the first manned launch, SpaceX publishedbrowser-based simulator of Crew Dragon docking to the ISS. The simulator started out as a hobby for two of the company's developers. Then they decided to finish it and published it for the general public.



The simulator remains a simulator: it has nothing to do with the real code. Although both products were written by the same team, they are two completely different projects.



All this does not mean that the spaceship flies in JavaScript. Chromium on the ship is only used as a rendering tool for UI elements. The flight software interface has all the necessary redundancy and sits outside the displays, SpaceX officials say. The backend is written in C / C ++.



A regular browser engine in a spaceship is a daring decision. In the responses, SpaceX employees assuredthat regardless of the technological stack, the development standards are the same, the principles of writing reliable and efficient software do not change, and to understand possible errors, testing goes under various conditions. In case of rejections, there are appropriate notifications and procedures. Finally, hundreds of hours of astronaut training on flight simulators on Earth are added to the testing.



Just in case, physical keys are located under the touch-screens. They are designed for critical situations such as a fire in the cockpit of a ship. There are also physical "start" and "abort" buttons for most of the operations that can be called from the displays. NASA has developed specific requirements for the screens themselves, and SpaceX's solution matches them .



SpaceX employees also explainedhow they came to such a seemingly unconventional solution for the aerospace industry. The project started out as a prototype to showcase a sample design for NASA. Then the prototype was successfully launched on a real flight hardware. The developers liked the modern features that were present in the browser engine, and there are enough developers for it on the market.





The diagram shows how JavaScript code is isolated from the main control systems of the James Webb telescope.



Aside from the SpaceX experience, there is nothing outrageous in choosing JavaScript for the space field. In the case of the James Webb Space Telescope, the JS code will run directly on the vehicle. It will not directly control, for example, motors, it will only call other programs.

Why is there no sound in space?

Because the rocket flies on Linux.



Falcon 9 software is regularly updated. Almost every rocket launch flies with a slightly updated code. Although updates are so frequent, the "base" for each specific launch is not specifically adapted. This is done by other SpaceX departments, which make their own adjustments to the flight configs: they set the wind variables on the launch day, change the fault tolerance limits, and so on.



Crew Dragon is controlled by the Linux operating system with the PREEMPT_RT patch for real-time operation. SpaceX doesn't usesome standard third-party distribution. The company has assembled its own kernel and related utilities. During the years of development, the kernel has hardly been modified. There were only minor changes and a few specialized drivers to work with the hardware.



Among the open source projects used are the Das U-Boot operating system loader , the Buildroot distribution build system , the C ++ standard library, and the Musl C library . But in general, SpaceX uses not so much software written outside the company and chooses open-source projects only with the highest possible quality.



At SpaceX, tests are written in Python, tested in LabVIEW, and they flyin C ++. When writing in C ++, they use the language's object-oriented techniques, although they prefer to keep everything as simple as possible.



SpaceX at the code level provides the possibility of normal operation with the loss of sustainer or shunting engines with compensation algorithm. State machines contain all kinds of emergencies. For example, in the state machine of the Dragon ship , an autonomous transition from approaching a station to leaving is incorporated if some failures are observed.



SpaceX says there is no "AI" in Dragon (probably means neural network algorithms), although some kind of machine vision is used for navigation. Developers explainedthat does not rule out the use of machine learning programs sometime in the future.

Starlink

The Starlink Satellite Internet Project is even more computers. In each launch, the Falcon 9 rocket launches 60 satellites into orbit, which contain more than 4,000 Linux computers. SpaceX has put tens of thousands of Linux nodes and more than a thousand microcontrollers into low-earth orbit.





The animation shows how the satellite's solar panel is being



deployed Six months ago, Starlink generated about 5 TB of telemetry per day, and the constellation has only become larger. The number of satellites is growing, and work is underway to reduce the amount of data sent. To reduce the amount of data that is stored on board and sent to Earth, some of the problems are diagnosed on the device itself.



Each of the Starlink satellites is powered by a Hall effect motor. Due to this, the satellite takes the desired position in the swarm and avoids collision with space debris. But these maneuvers still need to be performed correctly, and the command department of the largest constellation of satellites is tiny.



Therefore, the programmers were puzzledhow to avoid collisions and control the position of satellites automatically. In the developed system, the satellites are asked which window to enter - and they independently go there. Also, several times a day, satellites receive data from the Earth on approaches to other objects in orbit. Then the satellites independently calculate the necessary maneuvers and move away from space debris.



Duplication of internal systems in the case of Starlink satellite is only performed up to a certain limit. Due to the total number of companions, the squad will not notice the loss of a soldier. If one node fails, the user on Earth will connect to another satellite visible in the sky.



During development and testing, each of the Starlink flotilla satellites is not considered as a separate satellite, but as a server in the data center. Some of the tasks are critical: management, software updates, power and security. Separate tests are written for these applications. Many other features allow for a more flexible approach, similar to developing web services. Therefore, the team is deploying a test build to a small number of satellites and comparing how they behave in comparison with the rest of the constellation. Further, if problems arise, the software is improved or rolled back to the previous version.



This testing process is needed to quickly improve the system. SpaceX developers claim that many times it was possible to find and fix failures that were impossible to predict on Earth.



The Starlink satellite leaks user data through it, and a computer hack threatens to be tapped. SpaceX has provided for this and added end-to-end encryption. Also, each of the components - satellites, gateways and user terminals - executes only the signed code, so the attacker does not register in the system for a long time.

Development culture

Most of SpaceX's software engineers work in Seattle, Washington, and Hawthorne, California, and some from offices in Texas.





SpaceX Software Development Team, 2013 The



traditional aerospace industry is overwhelmed by both the speed of development and the size of SpaceX's divisions. As in 2019 ( podcast , mark 44:00), Chief Software Officer of the US Air Force, Nicholas Chillan, said that where government agencies would need at least 2.5 thousand programmers, SpaceX is hiring 50. At the same time, the development team writes software for nine different machines and checks the code in 24 hours.



SpaceX is trying to attract developers from the gaming industry. At GDC 2015, company representatives said that those with a computer science degree did not have memory management skills. Unexpectedly, the pace of work and optimization methods of game developers are suitable for space. As Elon Musk says , compared to an MMO, docking two ships is elementary simple.



As part of the AMA at Reddit in 2013, employees talked about the structure of the software development departments.

• 7 35 . : Falcon 9, Grasshopper Dragon. , , . .
• SpaceX.  — - . , , , . , SpaceX .  — - : C#, MVC4, EF, SQL; Javascript, Knockout, Handlebars, LESS; REST API, «super sexy».
• 2013 9 , . , . .
• . .  — .

The company constantly hires developers and engineers, and not every position requires special education. There are many different accents in SpaceX offices , including from the space of the former USSR. The company can hire not only the holder of an American passport, although there are restrictions on the export control regime for military technologies . To be hired, a foreigner will need a residence permit - a US green card. It will take several years to get it from scratch. If there is already a "green", then the question is only in the ability to show the level of qualifications at the interview.



SpaceX CEO and CTO Elon Musk is known for his dislike of 40-hour workweeks. He has repeatedly statedthat works 80-120 hours a week. What is the pop, so is the parish. A common complaint about Glassdoor about SpaceX  is the poor balance of life and work. In anonymous testimonials, employees and interns talk about frequent burnout and the now-common 12-hour shift .