Skip to content
Snippets Groups Projects

Troubleshooting package installation

  • Tier: Free, Premium, Ultimate
  • Offering: GitLab Self-Managed

To help with troubleshooting, run the following commands.

sudo gitlab-ctl status
sudo gitlab-rake gitlab:check SANITIZE=true

For more information on:

500 errors with PG::UndefinedColumn: ERROR:.. message in logs

After upgrading, if you start getting 500 errors in the logs that show messages similar to PG::UndefinedColumn: ERROR:..., these errors could be cause by either:

  • Database migrations not being complete. Wait until migrations are completed.
  • Database migrations being complete, but GitLab needing to load the new schema. To load the new schema, restart GitLab.

Error: Failed to connect to the internal GitLab API

If you receive the error Failed to connect to the internal GitLab API on a separate GitLab Pages server, see the GitLab Pages administration troubleshooting

An error occurred during the signature verification

If you receive this error when running apt-get update:

An error occurred during the signature verification

Update the GPG key of the GitLab packages server with this command:

curl --silent "https://packages.gitlab.com/gpg.key" | apt-key add -
apt-get update

Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails] [..] Command timed out after 3600s

If database schema and data changes (database migrations) must take more than one hour to run, upgrades fail with a timed out error:

FATAL: Mixlib::ShellOut::CommandTimeout: rails_migration[gitlab-rails] (gitlab::database_migrations line 51)
had an error: Mixlib::ShellOut::CommandTimeout: bash[migrate gitlab-rails database]
(/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16)
had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:

To fix this error:

  1. Run the remaining database migrations:

    sudo gitlab-rake db:migrate

    This command may take a very long time to complete. Use screen or some other mechanism to ensure the program is not interrupted if your SSH session drops.

  2. Complete the upgrade:

    sudo gitlab-ctl reconfigure
  3. Hot reload puma and sidekiq services:

    sudo gitlab-ctl hup puma
    sudo gitlab-ctl restart sidekiq

Missing asset files

Following an upgrade, GitLab might not correctly serve up assets such as:

  • Images
  • JavaScript
  • Style sheets

GitLab might generate 500 errors, or the web UI might fail to render properly.

In a scaled out GitLab environment, if one web server behind the load balancer is demonstrating this issue, the problem occurs intermittently.

The Rake task to recompile the assets doesn't apply to a Linux package installation which serves pre-compiled assets from /opt/gitlab/embedded/service/gitlab-rails/public/assets.

The following sections outline possible causes and solutions.

Old processes

The most likely cause of old processes is that an old Puma process is running. And old Puma process can instruct clients to request asset files from a previous release of GitLab. Because the files no longer exist, HTTP 404 errors are returned.

A reboot is the best way to ensure these old Puma processes are no longer running. Alternatively, you can:

  1. Stop Puma:

    gitlab-ctl stop puma
  2. Check for any remaining Puma processes, and kill them:

    ps -ef | egrep 'puma[: ]'
    kill <processid>
  3. Verify with ps that the Puma processes have stopped running.

  4. Start Puma

    gitlab-ctl start puma

Duplicate sprockets files

The compiled asset files have unique filenames in each release. The sprockets files provide a mapping from the filenames in the application code to the unique filenames.

/opt/gitlab/embedded/service/gitlab-rails/public/assets/.sprockets-manifest*.json

Make sure there's only one sprockets file. Rails uses the first one.

A check for duplicate sprockets files runs during Linux package upgrades:

GitLab discovered stale file(s) from the previous install that need to be cleaned up.
The following files need to be removed:

/opt/gitlab/embedded/service/gitlab-rails/public/assets/.sprockets-manifest-e16fdb7dd73cfdd64ed9c2cc0e35718a.json

Options for resolving this include:

  • If you have the output from the package upgrade, remove the specified files. Then restart Puma:

    gitlab-ctl restart puma
  • If you don't have the message, perform a reinstall to generate it again. For more information, see Incomplete installation.

  • Remove all the sprockets files and then follow the instructions for an incomplete installation.

Incomplete installation

An incomplete installation could be the cause of missing asset file problems.

Verify the package to determine if this is the problem:

  • For Debian distributions:

    apt-get install debsums
    debsums -c gitlab-ee
  • For Red Hat/SUSE (RPM) distributions:

    rpm -V gitlab-ee

To reinstall the package to fix an incomplete installation:

  1. Check the installed version:

    • For Debian distributions:

      apt --installed list gitlab-ee
    • For Red Hat/SUSE (RPM) distributions:

      rpm -qa gitlab-ee
  2. Reinstall the package, specifying the installed version. For example 14.4.0 Enterprise Edition:

    • For Debian distributions:

      apt-get install --reinstall gitlab-ee=14.4.0-ee.0
    • For Red Hat/SUSE (RPM) distributions:

      yum reinstall gitlab-ee-14.4.0

NGINX Gzip support disabled

Check whether nginx['gzip_enabled'] has been disabled:

grep gzip /etc/gitlab/gitlab.rb

This might prevent some assets from being served. Read more in one of the related issues.

ActiveRecord::LockWaitTimeout error, retrying after sleep

In rare cases, Sidekiq is busy and locks the table that migrations are trying to alter. To resolve this issue:

  1. Put GitLab in read-only mode.

  2. Stop Sidekiq:

    gitlab-ctl stop sidekiq

GPG signature verification error: Bad GPG signature

You might receive the following error when running yum update or dnf update:

Error: Failed to download metadata for repo 'gitlab_gitlab-ee-source': repomd.xml GPG signature verification error: Bad GPG signature

To resolve this issue:

  1. Run dnf clean all.
  2. Fetch the latest signing keys.
  3. Attempt to upgrade again.

If the error persists even after dnf clean all, manually remove the affected repository cache directory. In this example:

  1. Remove /var/cache/dnf/gitlab_gitlab-ee-source.
  2. Run dnf makecache.