Lightweight, (Almost) Crypto-Free Remote System Operation

By Ray Ingles

"There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies, and the other is to make it so complicated that there are no obvious deficiencies." - C.A.R. Hoare

"Sure I'm paranoid, but am I paranoid ENOUGH?" - Unknown

Introduction

System administrators frequently want to be able to work on the machines they run even when they are far away from them. There are secure tools that allow full remote shell access, like ssh and lsh, but due to their complexity they have suffered critical exploits from time to time. In addition, their overhead can be excessive for some purposes. Fortunately, other options are available that can be used alone or can be combined with remote shells to create a more secure overall system.

Overview

Maybe the pager has just gone off when you're home in bed, and the boss wants you to fix the broken database now. Or perhaps you're out for lunch and someone calls to tell you the mailserver has been cracked and is currently spamming the world, and you need to bring it down fast. Possibly you've checked and your Web server has wedged itself and needs to be restarted. Or suppose you're just on vacation and find you want to update your home Web site with some new photos. In all these cases, you'd like to do something to the machine over the Internet without having to actually sit in front of it - things you don't want just anybody to be able to do.

The Problem

Tools like ssh and lsh are great for allowing secure remote access to your system. They offer essentially full, flexible remote control of a machine, in an encrypted and authenticated manner. But they are complex pieces of software; there's no way to do what they do without being complex. And with complexity comes bugs. SSH and lsh, and related tools like Webmin, have all had serious flaws that would allow an attacker to get full control over your system. Leaving them available all the time is a risk - sometimes it's necessary, but it's still a risk. And in some cases, you'd like to be able to tell the machine to do something, but it's not even attached to the network on a regular basis.

Some Solutions

It would be nice to enable remote shell access only when necessary. And perhaps (for something like shutting down a mail server) you don't even need a full shell, just a way to fire off a script remotely. Of course, the problem then becomes, how do you know that the alternative software is any more secure than ssh itself? Various people have worked on this problem in the past, and several potential solutions are available, ranging from the simple and venerable to the new and exotic.

Xringd uses a modem to control a machine remotely. Mail filters can be used to trigger actions based on special messages. Some solutions (like 'port knocking' and 'Net::Pcap') use the network, but without requiring even a single open port. Lando runs commands over a network, using username and password. Most recently, a program specifically for secure remote execution called Ostiary has been developed.

The Options

Old School Remote Operation - Xringd

The eXtended Ring Daemon, or "Xringd", uses a modem to monitor rings on a phone line. It counts the number of rings, and the time between them. If a 'sequence' matches one of the ones that it has been set up to detect, Xringd will run an associated command.

This is very nice from a security perspective. Since it uses no network connection at all, it's entirely immune to network attacks like buffer overflows. It can be used even when a network connection is unavailable (it's often used to cause a computer to initiate a dialup connection). The only 'client' you need is a phone. If you use it to start up ssh on demand, then the attacker needs to know the right phone number and the right ring pattern - it's quite hard to sniff that kind of thing remotely. It's also highly resistant to a man in the middle attack. (If you have to worry about someone rerouting your phone calls, you're in more trouble than Xringd can save you from.)

There are some practical issues that may make this unattractive in some circumstances. You need a modem and a telephone line to the server. (Fortunately, you don't need a fast modem at all; even a 1200 baud one will do nicely, but some servers are not placed close to a telephone jack.) Also, things like answering machines or voicemail (or even other people answering the telephone) can interfere with Xringd. If you give the server a dedicated line, you can avoid these problems, but that can be costly.

Finally, note that the rings you hear when making a call are not necessarily synchronized with the ring signals actually sent to the telephone. In most circumstances, they are close enough, but reliability can be an issue at times.

Procmail And Other Mail Filters

Most of the mail filtering programs have a way to invoke scripts when mail matching a pattern is received (in the simplest case, mail to a particular address). Assuming the server is running an SMTP daemon, this can be a nice way of triggering actions remotely. Technically, one could even send a shell script to be run, and have it e-mail the results back to you, giving you the equivalent of a very slow remote shell. The only client needed is an e-mail program, or even a webmail account.

The first problem is that if the box you want to talk to doesn't accept e-mail, this obviously won't work. (Adding an entire mail server, with the attendant risks of bugs, spam load, etc., just for remote execution doesn't make a lot of sense.) Some machines only periodically collect e-mail from a primary server, so there can be a substantial delay between when a command is sent and when it is acted upon.

Furthermore, if you don't encrypt the traffic in some way (or at least sign it with PGP), then anyone sniffing traffic between you and your server may be able to take advantage of the same channel to do mischief, or perform a man-in-the-middle-attack. (E-mail traffic is notoriously easy to falsify; hence the avalanche of spam these days.)

CVTSA, or "ClairVoyanT SysAdmin", is a system designed specifically for running commands through e-mail. It has some support for using passwords, but does not (currently) encrypt them in transit, so a sniffer could capture them and use them again.

Of course, if the only things you want to do with this type of system are emergency shutdowns and other such (hopefully rare) crisis management, then even an unencrypted channel might work. However, you'll need to change the 'magic trigger pattern' each time after you use it, or you take the risk that an attacker might capture it and 'replay' it at an inconvenient time.

Port Knocking

With port knocking, a daemon monitors firewall logs, looking for particular sequences of connection attempts to particular (closed) ports. When it sees a sequence it recognizes, it runs the associated command. This isn't terribly bandwidth efficient, but it has some nice properties. First, it's hard to tell if a server is listening for port knocks. Second (and most important), it's awfully hard to crack a closed port. (Linksys routers have had a simple version of this for a while, BTW, that they call port triggering.)

However, a clever attacker with a sniffer could notice this traffic, and duplicate it for their own use. More complicated encodings could express something like a PGP signature (indeed, in theory one could create an entire network protocol based on port knocks), but things rapidly become difficult to work with. As with 'mail filtering' solutions, one can either use it sparingly in emergencies, or move to real cryptography.

It's also important to realize that this system is critically dependent on the probe packets actually being delivered, and delivered in the order that they were sent. This is not guaranteed on the Internet. What's more, depending on where you're at (e.g., an Internet cafe or behind a business firewall), you might not be allowed to connect out to arbitrary ports. The more complex you make the 'knocks', the less reliable the system will be.

Also, notice that at least one entire IP packet (28 bytes or so minimum) is used to transmit roughly one bit of information. In terms of network efficiency, it's almost hideous. For a simple 'open up ssh' message, it's not a consideration, but actually adding cryptographic security to this system could use up a decent chunk of the available bandwidth.

Finally, this increases the CPU load for each entry in the firewall log. Depending on how detailed the logs are, and how fast and busy the network is, this can be a significant drain on resources.

'Sniffer' Triggers - libpcap and friends

Another interesting approach is to use Net::Pcap or other network capturing software to look for specific packets on the network (e.g., DNS requests) and examine them for particular data (e.g., a particular address). If found, it can enable ssh temporarily, or perform other actions.

One potential benefit of this approach is that a computer doesn't have to have an address on a network in order to monitor traffic on that network. You can set the card to 'promiscuous mode' and examine all the traffic on the wire. (It's very hard to hack a machine you don't even know is there.) Once the 'trigger' is spotted, the sniffer can use other means (a separate network, a serial link, even Xringd) to open up SSH on a target machine. Of course, you can also simply run the sniffer directly on the target.

Again, a clever attacker with their own sniffer may be able to detect the unusual activity and correlate it. To make this system truly secure, you would need more complex encoding/encryption of the 'trigger' traffic.

Additionally, the CPU load for this solution can be even worse than for 'port knocking' systems. A 'port knocking' daemon monitors firewall logs, which can have variable levels of detail. By necessity, a 'sniffer' solution must examine every packet on the network segment, which can be a substantial task for a busy gigabit line.

Lando - Simple & Flexible

Lando allows a user to run a preconfigured set of commands remotely, using passwords, and even allowing the user to supply arguments to them. While it currently has only a Windows client, and passwords are sent in the clear (making it suitable only for use on a trusted local network, or perhaps on a VPN), it can be very useful for, e.g. operating a local firewall box without going to the trouble of logging in.

Okay, A Little Crypto - Ostiary

All of the above solutions have their advantages, but each has some practical issues that can make them unsuitable for particular applications. Ostiary was designed to be a secure alternative that uses minimal resources. It tackles this problem with what might be termed "aggressive simplicity". It does require an active connection to the network (unlike Xringd and sniffing), but allows for much better default security with very low CPU, RAM, disk, and network bandwidth requirements.

An Ostiary server has one open port that it listens on. When someone connects, the server sends a random fixed length 'salt' message 16 bytes in size - the size of an MD5 hash. It then waits (with a timeout) for a reply from the client. It reads (at most) 16 bytes of reply, and closes the connection.

Ostiary has a list of commands to run, with associated passwords. It runs through the list, and hashes these passwords with the 'salt' it sent to the client. If one of these hashes matches the reply from the client, the associated command is run. (One final touch is that a record is kept of connections, and clients with too many failed attempts are 'locked out', and all subsequent communication from them is ignored.)

A detailed security analysis is available, but a few things about this system should be clear. With a protocol this simple, the chances for dangerous bugs are drastically reduced. Using fixed-length messages essentially eliminates the chances of a buffer overflow or other memory error. (Indeed, Ostiary does no dynamic memory allocation of any kind - everything is stored in static, fixed-size data structures.) Replay and man-in-the-middle attacks are also effectively useless. Ostiary limits how fast it accepts connections, enforcing low CPU and network usage. (The first production Ostiary server was a 16MHz 68030 machine.) Client requirements are even lower: Clients are available for Palm Pilots and even Windows.

Unlike a procmail-based solution, where you can put arbitrary commands (with arguments) in the message, Ostiary can only run the fixed set of commands you have preconfigured. The only argument it supplies to the commands is the IP address of the client that initiated the command. It requires an active network connection (unlike Xringd) and an open port (unlike port knocking or sniffing), which may entail configuring a firewall to open a new port. (Although one could run Ostiary on, say, port 22, and upon receipt of the correct command, it could terminate itself and spawn sshd...)

Since Ostiary uses TCP, it is as reliable as the network it uses to communicate. Problems with miscounted phone rings (a la Xringd) or randomly dropped packets (a la port knocking) are not a concern.

Summary

The following table summarizes the pros and cons of the various systems outlined above. "Replay" and "Man-in-the-middle" indicate if the default system is vulnerable to the corresponding attacks. "Command arguments" indicates if the system can run arbitrary commands with arguments. "CPU load" indicates that CPU time can be a significant consideration. "Special client" indicates that a specific client program is needed to work with that system.

System	Xringd	Mail filter	Port knocking	Sniffers	Lando	Ostiary
Network Required?		Yes	Yes	Yes	Yes	Yes
Port Required?		Yes			Yes	Yes
Modem Required?	Yes
Replay?		Yes	Yes	Yes	Yes
Man-in-the-middle?		Yes	Yes	Yes	Yes
Command arguments?		Yes			Yes
CPU load?		Sometimes	Yes	Yes
Special client?			Sometimes	Sometimes	Yes	Yes

None of these approaches is right for everyone. But all of them can be used to make attacks at least more inconvenient, and in many cases far more difficult. Remember, though, to analyze their pros and cons relative to your specific situation. Also remember that true security is a process, not a goal - you can never just install some software and be done thinking about it.

 

Ray Ingles has been involved with Linux since 1995. In addition to being an active member of the Metro Detroit Linux User's Group, he has made minor contributions to the UPS HOWTO and the Linux Joystick Driver.