published: 23rd of June 2019
After a power failure of the instance hosting my gitlab server I began getting errors when trying to access existing projects (404) or create new projects (503) from the web interface. I could see the list of projects but if I clicked on the link to any of them I would get a 404 errors. Likewise, if I tried to create a new project I would end up with a 503 error.
My Gitlab instance is running on CentOS 7 using the following software versions.
Looking at the /var/log/gitlab/gitlab-rails/production.log I could see errors related to the 404 and 503 errors I was getting in the web interface. Initially I thought the repos may have either been wiped or the database was corrupted.
Looking in the /var/opt/gitlab/git-data/repositories/ directory I could see all my repos still in tact phew!
Ok time to check the status of the Gitlab services. Running the gitlab-rake gitlab:check command revealed that there was an error with the Gitaly service.
sudo gitlab-rake gitlab:check # output Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 9.1.0 ? ... OK (9.1.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... FAIL: 14:Connect Failed Checking Gitaly ... Finished Checking Sidekiq ... . . . <snip>
Cool. WTF is Gitaly? I went on a duck hunt and to my surprise found no references to burly Italian men and beautiful Italian women doing a git merge :/
I did find the documentation page which states:
Gitaly is the service that provides high-level RPC access to Git repositories. Without it, no other components can read or write Git data. GitLab components that access Git repositories (gitlab-rails, gitlab-shell, gitlab-workhorse) act as clients to Gitaly. End users do not have direct access to Gitaly.
Sure enough, looking at the /var/log/gitlab/gitaly/current directory revealed the same errors as the issue linked above.
sudo tail /var/log/gitlab/gitaly/current # output 2019-06-22_09:48:35.16733 time="2019-06-22T09:48:35Z" level=fatal msg="find gitaly" error="strconv.Atoi: parsing \"\": invalid syntax" wrapper=16691 2019-06-22_09:48:36.17871 time="2019-06-22T09:48:36Z" level=fatal msg="find gitaly" error="strconv.Atoi: parsing \"\": invalid syntax" wrapper=16706 2019-06-22_09:48:37.19001 time="2019-06-22T09:48:37Z" level=fatal msg="find gitaly" error="strconv.Atoi: parsing \"\": invalid syntax" wrapper=16721
After checking the /var/opt/gitlab/gitaly/gitaly.pid I found that it was empty so I rm the file and restarted the Gitlab services with the sudo gitlab-ctl restart command. After that the gitlab-rake gitlab:check came back all green.
Success! I am now able to access existing projects and create new projects without getting any errors. Problem solved.