Gitlab Gitaly Error

published: 23rd of June 2019

Intro

After a power failure of the instance hosting my gitlab server I began getting errors when trying to access existing projects (404) or create new projects (503) from the web interface. I could see the list of projects but if I clicked on the link to any of them I would get a 404 errors. Likewise, if I tried to create a new project I would end up with a 503 error.

My Gitlab instance is running on CentOS 7 using the following software versions.

  • GitLab 11.11.3
  • GitLab Shell 9.1.0
  • GitLab Workhorse v8.7.0
  • GitLab API v4
  • Ruby 2.5.3p105
  • Rails 5.1.7
  • PostgreSQL 9.6.11

The Problem

Looking at the /var/log/gitlab/gitlab-rails/production.log I could see errors related to the 404 and 503 errors I was getting in the web interface. Initially I thought the repos may have either been wiped or the database was corrupted.

Looking in the /var/opt/gitlab/git-data/repositories/ directory I could see all my repos still in tact phew!

Ok time to check the status of the Gitlab services. Running the gitlab-rake gitlab:check command revealed that there was an error with the Gitaly service.

cmd 
sudo gitlab-rake gitlab:check

# output

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 9.1.0 ? ... OK (9.1.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK

Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... FAIL: 14:Connect Failed


Checking Gitaly ... Finished

Checking Sidekiq ...
.
.
.
<snip>

Cool. WTF is Gitaly? I went on a duck hunt and to my surprise found no references to burly Italian men and beautiful Italian women doing a git merge :/

I did find the documentation page which states:

Gitaly is the service that provides high-level RPC access to Git repositories. Without it, no other components can read or write Git data. GitLab components that access Git repositories (gitlab-rails, gitlab-shell, gitlab-workhorse) act as clients to Gitaly. End users do not have direct access to Gitaly.

- https://docs.gitlab.com/ee/administration/gitaly/

The Fix

More duck hunting for the error Gitaly: ... default ... FAIL: 14:Connect Failed lead me to this page that looked similar to the issue I was facing.

Sure enough, looking at the /var/log/gitlab/gitaly/current directory revealed the same errors as the issue linked above.

cmd 
sudo tail /var/log/gitlab/gitaly/current

# output

2019-06-22_09:48:35.16733 time="2019-06-22T09:48:35Z" level=fatal msg="find gitaly" error="strconv.Atoi: parsing \"\": invalid syntax" wrapper=16691
2019-06-22_09:48:36.17871 time="2019-06-22T09:48:36Z" level=fatal msg="find gitaly" error="strconv.Atoi: parsing \"\": invalid syntax" wrapper=16706
2019-06-22_09:48:37.19001 time="2019-06-22T09:48:37Z" level=fatal msg="find gitaly" error="strconv.Atoi: parsing \"\": invalid syntax" wrapper=16721

After checking the /var/opt/gitlab/gitaly/gitaly.pid I found that it was empty so I rm the file and restarted the Gitlab services with the sudo gitlab-ctl restart command. After that the gitlab-rake gitlab:check came back all green.

Outro

Success! I am now able to access existing projects and create new projects without getting any errors. Problem solved.

https://docs.gitlab.com/ee/administration/gitaly/

https://docs.gitlab.com/ee/administration/raketasks/maintenance.html

https://forum.gitlab.com/t/gitaly-fail-14-connect-failed/26137