But Fly is kind of like a black sheep among other similar platforms. Our hardware works in data centers scattered around the world. Our servers are connected to the Internet via the Anycast network, and they are connected to each other using the WireGuard network. We take Docker containers from users and turn them into Firecracker microvirtuals. And when we first got started, we did just that in order to give our customers the ability to run "edge applications". These applications are usually relatively small, self-contained pieces of code that are highly sensitive to network performance. As a result, these code snippets need to run on servers located as close to the users as possible. In such an environment, the ability to connect to the server via SSH is not that important.
But now not all of our clients use Fly in this way. Nowadays, in the Fly environment, you can easily execute all code related to an application. We have simplified the procedure for starting an ensemble of services in a clustered environment. Such services can, using secure communication channels, interact with each other, they can store data on a permanent basis, they can, via the WireGuard network, communicate with their operators. If I continue the story about our system in the same spirit, then I will have to provide links to all the materials that we have written over the past couple of months.
But, in any case, we did not have normal SSH support.
It is clear, of course, that you could simply build a container with an SSH service that you can connect to over SSH. Fly platform supports work with common TCP ports (and UDP ports too). If the client, using the file
fly.toml
, "tells" our Anycast network about his strange SSH port, the system will organize routing of his SSH connections, after which everything will work as it should.
But those who create containers usually do not do this, and we do not suggest that they do so. As a result, we equipped Fly with SSH support. What we have done is arranged in a rather unusual way. In this article, which consists of two parts, I will talk about this.
Part 1: 6PN and Hallpass
I wrote a lot about how private networks are arranged in Fly. To put it in a nutshell, it turns out that what we have can be compared with a simplified IPv6 version of the "virtual private clouds" GCP or AWS. We call this system 6PN. When an application instance (Firecracker microvirtual machine) is launched in Fly, we assign a special IPv6 prefix to this instance. Several identifiers are encoded in the prefix: the identifier of the application, the organization that owns the application, and the hardware resources on which the application is running. We use a bit of eBPF code to statically route such IPv6 packets on our internal WireGuard network and to ensure that clients cannot connect to the systems of organizations they are not involved with.
You can also use WireGuard to bridge the private IPv6 networks we create with other networks. Our API is able to create WireGuard configurations that can be used, for example, on EC2 hosts for RDS Postgres proxying . Or, if needed, you can use WireGuard clients (on Windows, Linux, or macOS) to connect the development computer to your own private network.
You probably already know what I'm getting at.
We wrote a very small and very simple SSH server in Go called Hallpass. It can be compared to "Hello, World!" Created using the Go library
x/crypto/ssh
... (If I did that again, I would probably just use the Glider Labs package for building SSH servers. Using this package, our server would literally be a "Hello, World!" Initialization of all instances of Firecracker microvirtual machines is performed and Hallpass is launched with binding to their 6PN addresses.
If you are capable of running your organization's 6PN network (say, via a WireGuard connection), that means you can log into your microvirtual instance using Hallpass.
There is only one interesting detail about how Hallpass works. It is about authentication. Infrastructure elements in our production network usually do not have direct access to our APIs or to their underlying databases. And the instances of Firecracker themselves, of course, do not have this access either. This leads to some difficulties associated with changing the communication settings. How, for example, can you answer the question of what kind of keys you need to have to connect to certain instances of microvirtual machines?
We found a workaround for this problem by resorting to SSH client certificates. Instead of having to deal with handing over the keys every time a user wants to log in from a new host, we create a root certificate to organize that user. The public key for this root certificate is hosted in our private DNS system, and Hallpass contacts the DNS to obtain this certificate every time a login is attempted. Our API signs new certificates for users, these certificates can be used to log in to the system.
You may have questions about this solution. Therefore, I will reveal some more details about him.
First, let's talk about certificates. Decades of X.509 Madness”May have caused the word“ certificate ”to give you an unpleasant aftertaste. And I don’t blame you for that. But certificates should be used when organizing SSH connections, since such certificates in this case are a good solution. However, SSH certificates are not X.509 certificates. It uses its own OpenSSH format , and, in general, nothing special can be said about these certificates. They, like all other certificates, have an "expiration date", which allows you to create short-lived keys (and this is, almost always, exactly what you need). And, of course, they allow you to assign a single public key to an entire group of servers that can authorize an arbitrary number of private keys. There is no need to constantly update the corresponding servers.
Next is our API and certificate signing. Well! We are very careful, but these certificates are generally as secure as Fly access tokens. At the moment, certificates cannot be better protected than tokens, since the token allows the deployment of new versions of application containers. Working with Web PKI X.509 CA involves a lot of formalities. We do without them.
And finally, our DNS. She, I agree, looks like complete nonsense. But it really isn't that bad. Each host running Firecracker microvirtual instances runs a local version of our private DNS server (a small program written in Rust). The eBPF code ensures that Firecracker machines can only interact with this DNS server, referring to it from the 6PN address of their server. (From a technical point of view, a user can only make queries to the private DNS API of this server, and all other users' queries will be processed recursively.) A DNS server can (I know it looks unusual) reliably identify an organization by analyzing the source IP addresses requests. In general, this is how we work.
All this happens in the depths of our system, users cannot see all of this. Users saw only a command
flyctl ssh issue -a
that requested a new certificate from our API, and then passed it to the local SSH agent, after which SSH connections, in general, turned out to be operational. All this was arranged neatly enough. But any business can always be done more accurately than before.
Part 2: working on a WireGuard network from user mode using TCP / IP
There is one problem with the above scheme of using SSH, which is that not everyone has WireGuard installed. The corresponding program, however, should be installed by everyone. WireGuard is a great technology that helps a lot to manage applications running on the Fly platform. But, be that as it may, some of our users do not have WireGuard.
True, such users also need to work with their systems via SSH.
At first glance, the fact that someone doesn't have WireGuard installed can seem like an insurmountable hindrance. How does WireGuard work? A new network interface is created on the user's computer. This is either a kernel-level WireGuard interface (in Linux), or a tunnel with a user-mode WireGuard service attached to it (in all other operating systems). Without this network interface, you cannot work with the WireGuard network.
But if you look at WireGuard from the right angle, you can see that, from a technical point of view, this is not the case. Namely, operating system-level privileges are required to configure a new network interface. But to send packets to
51820/udp
no privileges are needed. Anything needed to make the WireGuard protocol work can be started as an unprivileged process running in user mode. This is how the wireguard-go package works .
This will only allow you to go through the WireGuard handshake procedure. But at the same time, we are not talking about the exchange of information with the nodes of the WireGuard network, since you cannot simply take and send some arbitrary data to another system connected to this network. Such a system listens for packets that would normally be transmitted over TCP / IP networks. Standard system tools that support UDP sockets are of no help in establishing a TCP connection using such sockets.
Would it be difficult to write a small piece of code that enables TCP in user mode, solely designed to support communication over the WireGuard network, again in user mode? Such a code would allow Fly users to connect to their systems via SSH without having to install the software that powers WireGuard.
I was reckless in discussing all of this on the Slack channel that Jason Donenfeld was on. Namely, after thinking aloud, I went to bed. When I woke up, Jason had already implemented all this using gVisor, and included in the WireGuard library.
The most interesting thing here is gVisor. We already wrote about it ... If anyone doesn't know, gVisor is essentially a user-space Linux OS, Linux implemented in Golang, used as a replacement
runc
for running containers. This is actually a completely insane project. And if you use it, then I suppose you can proudly tell others about it, because it's just a gorgeous thing. In its depths there is a complete TCP / IP implementation, written in Go, which operates on input and output data represented as ordinary buffers
[]byte
.
Then a few tweets were tweeted, and then a couple of hours later I got a very nice email from Ben Barkert... Ben was already involved in various tasks related to the gVisor networking subsystem, he was interested in what we were working on, he wanted to know if we would like to cooperate with him. We liked his idea of working together on this project. And now, without going into details, we have a certificate-based SSH implementation that runs through the user-mode gVisor TCP / IP implementation. This all interacts with the WireGuard network through a custom mode package
wireguard-go
. And finally, this thing is built into the
flyctl
.
In order to use SSH using
flyctl
- just enter a command like this:
flyctl ssh shell personal dogmatic-potato-342.internal
And now, so that you can realize the incredibleness of what is happening, I will tell you a little about this command. So -
dogmatic-potato-342.internal
is an internal DNS name that is only resolved by a private DNS server on the 6PN network. All this is efficient due to the fact that in the mode the
ssh shell
utility
flyctl
uses the TCP / IP stack gVisor user mode. But there is no code in gVisor to do a DNS lookup. This is just a standard Go library that we fooled by slipping our special TCP / IP interface into it.
Flyctl
, by the way, this is an open source project(It should be so, since clients need to use it on their own computers on which they are engaged in development). Therefore, if you're interested, you can just read its code. Ben wrote some nice code in the pkg folder . And the rest of the code, horrible, I wrote. In Go, providing IP communications on the WireGuard network is surprisingly simple. If you've ever done low-level TCP / IP programming, then you might find this simplicity incredible. Objects from the gVisor TCP stack connect directly to the network code of the standard library.
Take a look at this code:
tunDev, gNet, err := netstack.CreateNetTUN(localIPs, []net.IP{dnsIP}, mtu) if err != nil { return nil, err } // ... wgDev := device.NewDevice(tunDev, device.NewLogger(cfg.LogLevel, "(fly-ssh) "))
CreateNetTUN
Is a part
wireguard-go
. This is where gVisor's capabilities are used. First of all, we have at our disposal a synthetic tunnel device that can be used to read and write ordinary packets that provide WireGuard operation. Secondly, we have the net.Dialer function , a wrapper for gVisor, which can be used in Go code and through it interact with the corresponding WireGuard network.
It's all? In general, yes. For example, here's how we use these mechanisms to work with DNS:
resolv: &net.Resolver{ PreferGo: true, Dial: func(ctx context.Context, network, address string) (net.Conn, error) { return gNet.DialContext(ctx, network, net.JoinHostPort(dnsIP.String(), "53")) }, },
This is normal networking code written in Go. In general, it turned out well.
Obviously everyone should do this.
Thanks to a couple of hundred lines of code (this is - apart from the Linux user-mode implementation code that we get from gVisor; but what to do - there is no escape from the dependencies), you can get a new network with cryptographic authentication at your disposal. A network that is accessible at any time and from almost any program.
It is clear that such a network is significantly slower than the one based on the core TCP / IP implementation. But how often does it really matter? And, in particular, does it often have any meaning in solving periodically arising problems, for the solution of which they usually build strange, unknown from what, assembled TLS tunnels? When speed matters, you can simply switch to the regular WireGuard implementation.
In any case, what I said solved our huge problem. After all, this system is suitable not only for organizing the work of SSH. We also host Postgres databases. It is very convenient when it is possible, by executing a simple command, to literally open a shell from anywhere
psql
, regardless of whether it is possible, at the right time, to install WireGuard for macOS.
Are you using WireGuard?
