Now that I’ve explained some about SSH and the concept of a jump server (a secondary machine that takes requests and proxy forwards them to other machines on the LAN that are not exposed to the Internet) I’ll continue by explaining how to conceptually configure a jump server. I’ll start by reiterating a little about how one works and why we’d use it.
A jump server provides extra layers of protection from the bad guys and script kiddies on the Internet. What you may not know, understand, believe is that someone is attacking your computers (including your phones — whatever has an OS they are probing and trying to exploit) at your location all the time from all over the world. There’s a whole computer discipline (field of study) about cyber security. The purpose being that students go to college and learn how to protect computer systems from break in attempts by securing them.
Effective cyber security practices are best described as done in layers. By layers we mean using tools that compound security, such as having a firewall at the router, a firewall on the workstation, tool such as fail2ban that monitors for failed or fraudulent login attempts, restricting access to specific IPs, placing services in sandboxed containers, placing servers on VLANs that don’t route internally, using ad and tracking blockers, using blacklists to block DNS queries to known fraud, ransomware, and other malicious sites, limiting emails to exclude all foreign countries, and blocking all IPs outside of your country or geographical region, locking SSH logins down to only permit RSA keys, etc. These layers are not 100% effective, yet they will deter even some of the staunchest threat actors — as they’ll move on to softer targets.
You might not consider yourself in need of cyber security practices. We all do need it and we all should want it, but if you don’t you don’t. You risk your privacy and security by neglecting or ignoring it. By virtue of a lack of privacy and security you risk the privacy and security of those around you.
There are those business and individuals that do a lot on the internet and want to learn a lot about how to protect themselves, and over time they’ve devised these programs and practices. They’ve done this as a matter of practice, because they need remote access to their computers for one reason or another. Sometimes it is about having their websites available. Other times it is about having a local email server, other times it is about just wanting to grab a file from a remote computer. It could be a lot of things, such as being able to access a database on a remote server to update client records, etc. Often it is because the network management personnel want to be able to remotely administer their systems. There’s nothing like planning a day with the family on a weekend only to have the office call saying that you need to come in to fix this or that problem. Remote administration is a huge help to all sorts of people, such as sales force, managers at all levels, network administrators, etc. It’s not realistic to just tell people to close off the access and deal with getting their job done in some other way.
One of the tools that they use is SSH. It is so powerful a tool. Network administrators and other skills workers use it all the time It is so well known and such a capable tool that even the bad guys know it intimately. Therefore they use it to try to break in. Sometimes these bad guys are just a script kiddies (some young kid not really experienced whom are following a written script executing commands with the hopes that the network administrator is inept or just inattentive to the whole security thing). SSH is so powerful, even though it has mostly good purposes/uses, it is good enough that it is used to try to break into most systems. SQL injection exploits are a big use case too.
I use SSH whenever possible because SSH allows the use of encryption keys. The most common is the RSA encryption key. That key can be extensive meaning it can have a large bit set for encryption. Now, our federal government will have you believe that 512 bits is enough, but that’s actually quite easy for them and the bad guys to break. I’ve seen episodes on TV that tell you to limit it to 512 bits because you need really no more. That’s a scam. However, the more bits of encryption you use the harder it is to break and the more data that must be transferred with each packet of data. What I mean is that 1024 bits is better than 512 bits and 2048 is better than 1024 and 4096 is better than all the others examples that I gave, however the larger the number of bits means the more overhead information is sent along with the packets when transferred.
Everything you send over the Internet, be it a movie from Netflix or a music file you download from iTunes, or a web page that you load into your browser is chopped up into packets and overhead is added to tell it where/who (as well as things like the sequential order of the packets) the packet is destined. When you use RSA keys (you encrypt your transfer) so more data is added to each packet. That means that it slows things down slightly. Packets are generally a certain size, so the more overhead you add the less actual data is included in each packet; which means the more packets. If you cut two equally size chicken breasts up one with big chunks and one with little chunks you’ll end up with more smaller chunks for the chicken breast cut into smaller pieces. That means you’ll also need more batter to coat the chicken before cooking.
This might imply that it is a performance penalty to encrypt your data to send over SSH. That is factually true. That performance is relative though. With modern Internet Service Providers the bandwidth is pretty good so you do not really notice. With today’s modern computers they can decrypt the stream much faster than computers of 5-10 years ago. So, using a high level of encryption (4096 bits) isn’t impacting your performance as much as you’d think. What it will do is protect you much better than the impotent 512 bit recommended by the government.
When you set up SSH keys on your computer you are asked to provide it with how many bits of encryption that you want to use and the encryption type. 4096 bits with RSA is a good choice. You should use it or at least 4096bit or higher. When these keys are created there are two resulting files. The “id_rsa” and “id_rsa.pub“. The one with the .pub extension is the public key and the other is the private key. You never give out your private key. You do transfer at least in part your public key to the remote computer so the remote computer can compare it to the private key that you try to connect with. That means the remote server has your public key, and when, at some time down the road, you try to access the remote server it will check it against your locally stored private key and let you in if it checks and if the server is so configured.
This is pretty good security if you require RSA keys to check for valid access and you don’t permit password login (unless you made a password part of your RSA key creation). We’ll not get into that as it isn’t necessary for purposes of setting up the SSH jump server.
The way a SSH jump server works is this. When you want to make a computer accessible via the Internet you go to your router (yeah, that little box you have that sometimes gives you WiFi access also). There are settings that can be configured to “port forward” to a specified machine (that “port” forwarded to only one machine — as well as specifying the protocol to further limit access). In that configuration you say that you want any data coming in on port # to go to a specific machine on the LAN. Yes, you have a LAN if you have one or more devices connecting to the router.
What happens is that whatever data is coming in on that port that data will be forwarded to the machine that you specified in your port forward settings in the router. Ports are designated by numbers, from 1 to 65535. Think of a port as like phone number where you talk to that number you know you are talking to a specific program on the specified computer. I used to describe it like a port hole. Remember the old old movies of ships that didn’t have electronic communication and the captains when they wanted to talk to engineering they’d grab the funnel connected to a long tube that lead down to the engineering. The captains would have a different funnel for the cook. So, think of a port like that. You’d never use the port assigned to the cook to give orders to the engineers.
That means that most programs that operate that work over the Internet have a specific port assigned (or a range of ports) specifically to them. In the case of the world wide web (WWW) when you use your browser to load a web page, your computer is talking on port 80 to the web server at the other end which also talks to you on port 80. There are default ports for lots of functions. The default port for SSH is 22. Remember, the bad guys and script kiddies know SSH operates by default on port 22 and will try to break in on port 22.
When you port forward at your router as described above you are telling any data that comes on that port to forward it to a machine which happens to be listening on that port. You can configure most Internet software to use a different (non-standard) port other than the default, but few people do that. You can configure the SSH service on the computer listening to listen on port 2222 instead of port 22. However, the bad guys and script kiddies know that too. You can tell SSH to listen on any port that’s not actively being used and can be forwarded to that machine — meaning it isn’t already forwarded to some other computer.
Further complicating this is the fact that most businesses have more than one computer that they want to talk to over SSH, and if we did this the traditional way (without a jump server) we’d modify the default port on all those other computers to listen each on a different port and then create a port forward with the router to forward to each of those on those non-standard ports. But that brings with it a bunch of tediousness and complexity. For instance, you have to know which port goes to which machine and you have to manage each machine’s port by editing config files, etc. If that machine changes or needs to be reinstalled you have to reconfigure it all over. If you need to reconfigure the router you likely will have to reconfigure each machine too. And of course the biggy, the real biggy — you have to have multiple port forwardings in your router — which becomes a “larger vector” for the baddies to attack. They can scan your public IP to find your open ports and notice that you have a bunch of ports open. It is harder to secure many publicly exposed computers than it is to secure just one. So, doing that isn’t always good.
What some do is they port forward to one machine and then jump from there to another machine by issuing the SSH command again once there to connect to another machine on the LAN. This is known as island hopping. If you think that’s secure you are likely going to have an eye opening experience some time in your career as you find that your computers were violated because you configured SSH in a way that permitted island hoping.
A way to secure this is with a jump server. “One that you can’t SSH out of” and thus neither can the baddies if they exploit and break into it. This means that if configured properly you’ll have only one port forwarding in your router for SSH because you’ll have a small computer set aside that operates as the jump server that, from within it, you can’t jump around into machines on your LAN.
If you use RSA keys internally every machine should be configured to only permit the use of RSA keys. Each computer will have their set generated and registered with the other computers “as is necessary”. But…but the small SSH jump server has no private RSA key itself so no one can jump into it and then jump around the different islands (computers) from there.
So, technically, an SSH jump server should only allow SSH connection through it without the need to log into it. By virtue of that configuration this means that whomever is jumping through it they have to have their keys registered on the target machine and the jump server, and they must know the internal network layout of the company in order to know the names and IP addresses of each other machine. If you are a bad guy you won’t have this information unless you broke into some other system that has the information. An example might be a computer that was stolen (and to take advantage of it it would need to be in the possession of someone capable of understanding the configuration). The good thing is that if the person that lost the computer is responsible they’ll alert the network support staff quickly so they can revoke that person’s keys on the jump server. This will result in the thief being immediately locked out. Quick and simple.
Remember nothing is absolutely secure. If a computer is stolen or someone breaks into it in some other way, your network could be left wide open. This is why we have network administration staff.
The next post on the SSH jump server will cover how to configure the RSA key setup and the config files used to tell the jump server who to actually connect to when someone attempts to connect.