Proxmox is set up in an exceptional way, except one. That is obviousness for certain features. It tries but fails. Here’s what I mean.
When set up proxmox you tell it to create storages. In my case I have the SSD that I installed promox onto. It sets up a small portion for the OS and the remainder is left to allow for you to use as you see fit. In my case I always set up a second drive or array of drives. This is where all my containers go and is typically much larger than the boot drive that promox was installed on.
Here’s the issue that I dealt with yesterday and today. I set up an email container based on Ubuntu 18.04 LTS. In here I installed the good stuff. It can handle multiple domains and lots of email accounts. It uses dovecot, postfix, antivirus, etc. It is very complete.
In order to make this work I gave it 1tb of the 16tb of space. I chose to make a copy of the container after doing this. I thought I’d created the container on the ZFS storage volume but apparenly because it isn’t obvious I created it on the small boot drive. I wonder why? Because of the inobvious aspect when setting this up I didn’t really notice and because proxmox doesn’t assign all the space at the time of creation everything looked right. When I sent to make a clone of the container that’s when things went wrong. The container finished copying too fast. I thought that was odd. I deleted the clone and had issues restarting the email container, as you have to shut it down to clone it. I couldn’t figure it out so I decided try again. Same issue. Only this time I had to reboot the whole proxmox server to recover. But when I did this it took 1/2 hour or more to come back up. That’s a systemd issue. It has shutdown timers and those are poorly designed and implemented meaning the people that decided the timer mechanism were ill guided and the specifics of the timers can cause you long delays or even permanent ones.
Anyway, when it finally did come back up I saw that the email container didn’t start automatically. I checked the options and it said yes to start at boot. I was able to get the email container to start so I chose to then test to see if email and everything still worked. It did.
I decided that I’d check to see if the backup had been done overnight. It had. At this point I chose to see if I could update this to Ubuntu 20.04LTS. I did the updates and at the end noticed all the errors having to do with mysql 8.0 (typical of mysql and that’s why I think poorly of it). So, because of those errors I actually blew away the container and I restored it from last nights backup.
This went OK but it wouldn’t start.
So, I started looking at journalctl and looking up the error messages online. Of the hits that matched my search almost none of them addressed my issue, one went so far as to talk about the date/time of the computer — really? A container doesn’t start and you worry about the date and time and that’s not even what the error message you received was about. I read through many hits and no one addressed the error head on. This is what is wrong with the web today.
I finally decided to look at free space on the drive, figuring that maybe, somehow, I had exceeded the 16tb of space. Nope. However I discovered that the container had been stored in the wrong place. When I manually looked in the CLI I saw the container ID wasn’t listed.
To speed up my progress I just flat out destroyed the email container and when to restore again from last night’s backup. I ensured that it created the container on the proper volume. Since it had failed to start after rebooting the proxmox host I decided to test this. I can’t have a container that won’t start at boot especially one that is critical to the mission — email. After checking I noticed that it had completed the full proxmox host reboot and I looked and found that the email container did start. I doubly verified that the container was in the proper location on the zfs pool on the 16tb volume.
Lesson learned was to ensure that the containers you create, big or small, go to the volume you intended and don’t rely on proxmox to store it in the same location as source, etc — such as when cloning.