Gitlab Gitaly Error
23rd June 2019
After a power failure of the instance hosting my gitlab server I began getting errors when trying to access existing projects (404) or create new projects (503) from the web interface. I could see the list of projects but if I clicked on the link to any of them I would get a 404 errors. Likewise, if I tried to create a new project I would end up with a 503 error.
My Gitlab instance is running on CentOS 7 using the following software versions.
- GitLab 11.11.3
- GitLab Shell 9.1.0
- GitLab Workhorse v8.7.0
- GitLab API v4
- Ruby 2.5.3p105
- Rails 5.1.7
- PostgreSQL 9.6.11
Looking at the /var/log/gitlab/gitlab-rails/production.log I could see errors related to the 404 and 503 errors I was getting in the web interface. Initially I thought the repos may have either been wiped or the database was corrupted.
Looking in the /var/opt/gitlab/git-data/repositories/ directory I could see all my repos still in tact phew!
Ok time to check the status of the Gitlab services. Running the
command revealed that there was an error with the Gitaly
sudo gitlab-rake gitlab:check # output Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 9.1.0 ? ... OK (9.1.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... FAIL: 14:Connect Failed Checking Gitaly ... Finished Checking Sidekiq ... . . . <snip>
Cool. WTF is Gitaly? I went on a duck hunt and to my surprise found no references to burly Italian men and beautiful Italian women doing a git merge :/
I did find the documentation page which states:
Gitaly is the service that provides high-level RPC access to Git repositories. Without it, no other components can read or write Git data. GitLab components that access Git repositories (gitlab-rails, gitlab-shell, gitlab-workhorse) act as clients to Gitaly. End users do not have direct access to Gitaly.
Sure enough, looking at the /var/log/gitlab/gitaly/current directory revealed the same errors as the issue linked above.
sudo tail /var/log/gitlab/gitaly/current # output 2019-06-22_09:48:35.16733 time="2019-06-22T09:48:35Z" level=fatal msg="find gitaly" error="strconv.Atoi: parsing \"\": invalid syntax" wrapper=16691 2019-06-22_09:48:36.17871 time="2019-06-22T09:48:36Z" level=fatal msg="find gitaly" error="strconv.Atoi: parsing \"\": invalid syntax" wrapper=16706 2019-06-22_09:48:37.19001 time="2019-06-22T09:48:37Z" level=fatal msg="find gitaly" error="strconv.Atoi: parsing \"\": invalid syntax" wrapper=16721
After checking the /var/opt/gitlab/gitaly/gitaly.pid
I found that it was empty so I
rm the file and restarted the Gitlab
services with the
sudo gitlab-ctl restart command. After that the
gitlab-rake gitlab:check came back all green.
Success! I am now able to access existing projects and create new projects without getting any errors. Problem solved.