Chroot SSH jump server in a Proxmox Container.

26Aug 2021 by Jim Blaich

The purpose of this is to

  • use chroot to contain specific users to specific folders and to limit the commands that they can use while in chroot. The chroot has no root control.
  • convert all the Proxmox containers that are operating as SSH jump servers into containers that if broken into the intruder will have an extremely limited environment in which to operate.
  • determine the security concerns and to work out how to quickly set up the service environment for users. Not all users need a full container with their own processors, memory and storage. Many users can operate under a single container only to be restricted to chroot and the commands available therein.

Proxmox is just one hypervisor out of many. It is the one I use. This even can be done on bare metal. My goal is not to teach you how to use Promox nor how to create containers nor how to do anything within the container as far as diagnosis goes unless that goes directly to the creation of the jump server.

There is technically no need for shell access in the jump server environments, but at some point having the shell access part “worked out” will aid in the speed of setting it up and in working out security concerns. There is technically no need for multiple user accounts either. Having them allows for future subscription offerings, e.g., a drop box of sorts for backups.

A privileged container is necessary. Being this is the only access (via the chroot) the fact that the container is privileged should not impact security. The hacker would have to escape the chroot and then escape the container. Without a compiler to compile custom code in the chroot and root access the hacker can not escape the chroot.

It also appears that the scp program is required in the chroot — meaning you have to copy the program to the chroot. A second problem that I found with using scp to copy to the chroot is that scp would not copy files into the chroot if it can’t determine user names. Once the nsswitch, etc are properly installed and configured scp will copy files. Otherwise it will fail stating that it can’t determine the user name. Without “scp” installed in the chroot you’ll get the following message.

bash: scp: command not found
lost connection

The vast majority of these steps are done in the host container. Very little is done within the chroot once ssh’d in.

The steps are as follows …

1. Create a privileged container in Proxmox.

Use Ubuntu 18.04 or 20.04 LTS template. Ensure that it is created as a privileged container and set the option for quotas.

Start the container and log in to it from the Proxmox host using “lxc-attach”

sudo lxc-attach ###

Where ### is the container ID

Once inside the work of setting up the chroot begins.

As you work through the steps you’ll encounter issues. Most of those have been “noted”. Once done then create the folder structure to be used in the jail as outlined in step 2.

2. After everything has been decided then create the necessary structure under the jail. This includes the dependent folders that are necessary to operate various commands and a bash shell.
mkdir /jails
cd /jails
mkdir -p {dev,bin,home,etc,lib}
chown -R root:root /jails

The device nodes need to be created too. Under the jail folder should be a dev folder and under that should be the device nodes such as null, random, urandom, zero, etc. Some guides only mention null. These device nodes are made with the command mknod.

  • Note: This program needs to be installed as it doesn’t come in the base OS install.

Install mknod into the host environment. Some examples instruct you to install coreutils, however not all distributions have mknod. In Ubuntu 18.04 it is not present so you have to install it as described below.

sudo apt update
sudo apt install libunix-mknod-perl

Set up the device nodes. Keep in mind that if these are not present when the user attempts to log in they will immediately be disconnected.

cd /jails/dev
mknod -m 666 null c 1 3
mknod -m 666 tty c 5 0
mknod -m 666 zero c 1 5
mknod -m 666 random c 1 8
mknod -m 666 urandom c 1 9

Along with the shell comes the need to view, rename, delete, and make and remove files and folders. This is done by copying the applicable programs and their dependencies to the chroot jail.

Some have suggested that you should do the following or something similar. I recommend keeping the environment minimal and therefore there is no need for the mounts described in the below block. It needlessly exposes elements of the OS to the user. Providing that information/access is likely outside the chroot’s raison d’etre.

  • Note: The following block is not to be done as it is just an example of what others have suggested.
mount -t proc proc /jails/proc/
mount -t sysfs sys /jails/sys/
mount -o bind /tmp /jails/tmp/
mount -o bind /dev /jails/dev/
3. Locate and copy the files from the host environment to the structure under the jail that the users will have access to while in the chroot. Both config files and program binaries.

Copy specific config files from the host’s /etc into the jail folder’s equivalent. Files such as resolved.conf, passwd, shadow, etc need to be copied and have root ownership, and have group and everyone else be granted “read” access. If all users have no read rights to these files the environment may act in unusual ways.

For example, if you end up with a bash prompt that says “I have no name!” it means you have not copied and/or set permissions on either the nsswitch files (the libnss included), or the libn and/or passwd and/or the shadow file(s) are missing.

The following files and folders go into the jail’s etc folder (/jails/etc).

passwd
hosts
resolv.conf
nsswitch.conf
shadow
ld.so.cache
ld.so.conf
terminfo (the complete folder)
  • Note: Cut down to size the passwd.conf so any potential hacker can’t be educated as to the host environment.

The resolv.conf gives access to DNS lookup so you may wish to skip it. You can still ping an ip address if you include ping in the programs made available. If you don’t copy ping into the jail it is no big deal as you will not need it being there are no LAN/WAN related programs installed in the chroot.

Terminfo is also necessary so as to allow for proper terminal emulation. You can just copy the whole folder into the jail’s etc folder. These files are in the /usr/share/terminfo folder. Copy all sub folders.

Copy the libnss libraries to their respective folders under /jails.

cp -p --parent /lib/libns{l,s_} /jails/

Copy the programs and their dependencies into the jail’s respective program folders. Copy only what is necessary to allow enough functionality to perform the tasks that the chroot was created for.

/jails
/jails/bin
/jails/usr
/jails/usr/bin
/jails/usr/lib
/jails/usr/lib/x86_64-linux-gnu
/jails/usr/share
/jails/usr/share/terminfo
/jails/lib
/jails/lib/x86_64-linux-gnu
/jails/lib/terminfo
/jails/lib64
/jails/etc/terminfo

Use “ldd” to determine the dependencies of each program and copy those into their respective folders under the jail folder. Use the command ldd in the host’s program folder against the program to determine the program’s dependencies.

cd /bin
ldd bash

This will display the dependencies along with the path to where they are located. The program you are checking should be copied (as well as the dependencies) to the jails respective folders.

linux-vdso.so.1 (0x00007ffdb63cf000)
libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007fc3f6663000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc3f665d000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc3f646b000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc3f67c6000)
  • Note: the first entry (linux-vdso.so.1) is virtual so there is no need to try to copy it.
4. Set the ownership and rights ensuring that everything has root only ownership and rights. Do not set root ownership on the home folder nor on the user’s folders and files under it.

At any time while in the system folders such as /jails/bin you can chmod 644 to set the permissions to what they should be.

The user folders should be owned by them alone. The system binaries (programs) should be 644. The config files in the /jails/etc need to 644 also. Double check to ensure everyone has at least read access.

5. Set up users and group.

Create a user in the host environment. Then again create the user skel structure in the user structure under the jail folder. Technically you can simply add the user telling the add user program to create the home structure under the jail itself.

Create the chroot users group and add the users to the group.

https://www.thegeekstuff.com/2010/07/disk-quota/

Useradd, groupadd, usermod to include users into the chroots group.

6. Modify the hosts /etc/ssh/ssh_config file.

Set options to limit root login, to ensure only keys are used, and to create a match section that either matches the users and or the group of chroot users. I won’t explain how to create a user or a group.

Configure SSH via the ssh_config file to set up an environment where keys are required and root can’t log in and all users must use keys.

PermitRootLogin no
PasswordAuthentication no
PermitEmptyPasswords no

Using the match command limit the login to the chroot environment to users, or users in a specific group. Ensure that the proper authorizedkeysfile option is set. Failing to do this will cause SSH to look to the wrong folder for the authorized_keys file.

Match Group chroots
ChrootDirectory /jails
X11Forwarding no
AllowTcpForwarding yes
AuthorizedKeysFile /jails/home/%u/.ssh/authorized_keys
  • I place this at the bottom of the file so as to not cause issues with other options in this file. Here is where you put the authorizedkeyfile path for the users and maybe some other limiting options for the chroot users.
  • AllowTcpForwarding yes is requires to use this as a jump server. Without it set to yes there will be an administrative error message presented as the ssh fails.

Copy or create the actual authorized_keys file to the user’s folder under the jail folder. All files and folders in user’s home folder must be only owned by the user. Limit rights to rw. No other user or group should have even read rights.

7. Set quotas at file system and for users. Hopefully find a way to set a quota via sshd_config file in chroot environment. (Optional)

This is a difficult procedure and it requires that you understand where Proxmox begins and where it ends. Remember the Proxmox people do as little as possible in hopes of separating their stuff from the regular parts of Linux. Quotas are separate from Proxmox. However, there is a bug in the current quota software that does not permit it to operate inside of a container. You can step down to an older version than the one shipped with Proxmox but that quickly becomes a dependency hell scenario.

  • In Proxmox for the container under resources and then root disk there is an option to enable quotas. If you set the option after creating the container, you must restart the container before it will take effect. More research is required as to how to set quotas by user automatically such as setting it in the sshd_config file.
  • If this is being done in a Proxmox container it must be done in a privileged container otherwise mknod will not work. If disk quotas are to be enforced then this also requires a privileged container. It also requires legacy cgroups. It further requires the ext4 file system.
8. Along with quotas comes the need to allow users to copy files to and from the chroot. In the case of using dolphin one would need to have the sftp-server and its dependencies copied into the /jails/usr/lib/openssh/ folder.
linux-vdso.so.1 (0x00007ffd9e7de000)
libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007fa48c6e9000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa48c4f7000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa48c4f1000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa48c4ce000)
/lib64/ld-linux-x86-64.so.2 (0x00007fa48c9e7000)
  • Note: Most of the dependencies are already met by other programs. Just make sure they are all in the proper folder.
9. Implement this in the jump servers. This can be done in a privileged container or in bare metal and can be done without interfering with their current configuration.

Most of the above steps are to this end. Once you can ssh into the chroot you can use it as a jump server. The note about allowing tcp forwarding applies.

This has been tested and the target container inherits the nameserver from the Proxmox host. This allows for updating the target container using apt.