Troubleshooting package installation
- Tier: Free, Premium, Ultimate
- Offering: GitLab Self-Managed
To help with troubleshooting, run the following commands.
sudo gitlab-ctl status
sudo gitlab-rake gitlab:check SANITIZE=true
For more information on:
- Using
gitlab-ctl
for maintenance, see Maintenance commands. - Using
gitlab-rake
for configuration checking, see Check GitLab configuration.
PG::UndefinedColumn: ERROR:..
message in logs
500 errors with After upgrading, if you start getting 500
errors in the logs that show messages similar to PG::UndefinedColumn: ERROR:...
,
these errors could be cause by either:
- Database migrations not being complete. Wait until migrations are completed.
- Database migrations being complete, but GitLab needing to load the new schema. To load the new schema, restart GitLab.
Error: Failed to connect to the internal GitLab API
If you receive the error Failed to connect to the internal GitLab API
on a separate GitLab Pages server,
see the GitLab Pages administration troubleshooting
An error occurred during the signature verification
If you receive this error when running apt-get update
:
An error occurred during the signature verification
Update the GPG key of the GitLab packages server with this command:
curl --silent "https://packages.gitlab.com/gpg.key" | apt-key add -
apt-get update
Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails] [..] Command timed out after 3600s
If database schema and data changes (database migrations) must take more than one hour to run,
upgrades fail with a timed out
error:
FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails] (gitlab::database_migrations line 51)
had an error: Mixlib::ShellOut::CommandTimeout: bash[migrate gitlab-rails database]
(/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16)
had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
To fix this error:
-
Run the remaining database migrations:
sudo gitlab-rake db:migrate
This command may take a very long time to complete. Use
screen
or some other mechanism to ensure the program is not interrupted if your SSH session drops. -
Complete the upgrade:
sudo gitlab-ctl reconfigure
-
Hot reload
puma
andsidekiq
services:sudo gitlab-ctl hup puma sudo gitlab-ctl restart sidekiq
Missing asset files
Following an upgrade, GitLab might not correctly serve up assets such as:
- Images
- JavaScript
- Style sheets
GitLab might generate 500 errors, or the web UI might fail to render properly.
In a scaled out GitLab environment, if one web server behind the load balancer is demonstrating this issue, the problem occurs intermittently.
The Rake task to recompile the
assets doesn't apply to a Linux package installation which serves
pre-compiled assets from /opt/gitlab/embedded/service/gitlab-rails/public/assets
.
The following sections outline possible causes and solutions.
Old processes
The most likely cause of old processes is that an old Puma process is running. And old Puma process can instruct clients to request asset files from a previous release of GitLab. Because the files no longer exist, HTTP 404 errors are returned.
A reboot is the best way to ensure these old Puma processes are no longer running. Alternatively, you can:
-
Stop Puma:
gitlab-ctl stop puma
-
Check for any remaining Puma processes, and kill them:
ps -ef | egrep 'puma[: ]' kill <processid>
-
Verify with
ps
that the Puma processes have stopped running. -
Start Puma
gitlab-ctl start puma
Duplicate sprockets files
The compiled asset files have unique filenames in each release. The sprockets files provide a mapping from the filenames in the application code to the unique filenames.
/opt/gitlab/embedded/service/gitlab-rails/public/assets/.sprockets-manifest*.json
Make sure there's only one sprockets file. Rails uses the first one.
A check for duplicate sprockets files runs during Linux package upgrades:
GitLab discovered stale file(s) from the previous install that need to be cleaned up.
The following files need to be removed:
/opt/gitlab/embedded/service/gitlab-rails/public/assets/.sprockets-manifest-e16fdb7dd73cfdd64ed9c2cc0e35718a.json
Options for resolving this include:
-
If you have the output from the package upgrade, remove the specified files. Then restart Puma:
gitlab-ctl restart puma
-
If you don't have the message, perform a reinstall to generate it again. For more information, see Incomplete installation.
-
Remove all the sprockets files and then follow the instructions for an incomplete installation.
Incomplete installation
An incomplete installation could be the cause of missing asset file problems.
Verify the package to determine if this is the problem:
-
For Debian distributions:
apt-get install debsums debsums -c gitlab-ee
-
For Red Hat/SUSE (RPM) distributions:
rpm -V gitlab-ee
To reinstall the package to fix an incomplete installation:
-
Check the installed version:
-
For Debian distributions:
apt --installed list gitlab-ee
-
For Red Hat/SUSE (RPM) distributions:
rpm -qa gitlab-ee
-
-
Reinstall the package, specifying the installed version. For example 14.4.0 Enterprise Edition:
-
For Debian distributions:
apt-get install --reinstall gitlab-ee=14.4.0-ee.0
-
For Red Hat/SUSE (RPM) distributions:
yum reinstall gitlab-ee-14.4.0
-
NGINX Gzip support disabled
Check whether nginx['gzip_enabled']
has been disabled:
grep gzip /etc/gitlab/gitlab.rb
This might prevent some assets from being served. Read more in one of the related issues.
ActiveRecord::LockWaitTimeout error, retrying after sleep
In rare cases, Sidekiq is busy and locks the table that migrations are trying to alter. To resolve this issue:
-
Put GitLab in read-only mode.
-
Stop Sidekiq:
gitlab-ctl stop sidekiq
GPG signature verification error: Bad GPG signature
You might receive the following error when running yum update
or dnf update
:
Error: Failed to download metadata for repo 'gitlab_gitlab-ee-source': repomd.xml GPG signature verification error: Bad GPG signature
To resolve this issue:
- Run
dnf clean all
. - Fetch the latest signing keys.
- Attempt to upgrade again.
If the error persists even after dnf clean all
, manually remove the affected repository cache directory. In this example:
- Remove
/var/cache/dnf/gitlab_gitlab-ee-source
. - Run
dnf makecache
.