Nextcloud and error 503 Service Unavailable (seemingly suddenly seemingly randomly)

Some of these things just annoy you to death.  Suddenly I find that I am getting this error whenever I access my nextcloud instance, which I happen to be using more and more and more every day.

I had been tightening down my security on my containers on the proxmox server and had discovered that I needed to put some exclusion IPs in fail2ban.  I did that for several containers and must have been interrupted by a customer or two and I know I never got back to it.

I also put in fail2ban on several containers that were missing it and I ensured that certificates were valid and being updated properly by letsencrypt.  So, I was tightening the screws and sealing the doors so to speak, but interruptions are events that can limit your focus.  Anyway, I found the cause of the above error.  I looked at a bunch of online posts but some were so nonsensical such as “access this file through the web interface”.  What made it nonsensical is that you can’t test by accessing that file because the service is down.

In any event, after seeing a bunch of suggestions and trying a few nothing worked.  That’s a pet peeve of mine.  So many people tend to offer suggestions when they should not be because it costs us time to sift through them.

I looked in my log file and found this error.  I almost missed it.

client denied by server configuration: /var/www/html/nextcloud/data/.ocdata

I looked into that. The first notable suggestion to fix it was nonsensical.  How can I access that file if the service is down.  This error comes from the /var/log/apache2/error.log file.  That should have given the person that suggested it their first clue that their suggestion was nonsensical.

Someone suggested that this was a .htaccess file error.  So I renamed it and restarted the instance.  Nope.  No luck.  I renamed it back.

Then it hit me.  Stop the fail2ban service and try accessing nextcloud again.  Bingo.  I was in.

I then opened the /etc/fail2ban folder and edited the jail.local file.  In there I have an ignoreIP entry for each jail.  I added to the end of it the local IPs for the lan and vlans.  I then restarted fail2ban and everything worked properly.

Funny how one error leads you down a path to a solution that doesn’t seem to be associated with the original error.