Introduction

gitlab-monitor is a Prometheus Web exporter that does the following:

1. Collects GitLab production metrics via custom probes defined in a YAML configuration file.
2. Custom probes gather measurements in the form of key/value pairs.
3. For each probe, gitlab-monitor creates an HTTP endpoint /<probe_name> (by default on port 9168) that delivers these metrics to a Prometheus scraper.

A central Prometheus process is configured to poll exporters at a specified frequency.

Supported Probes

Below is a list of probes added by this exporter, and their corresponding metrics.

1. Database
   • Per-table tuple stats -- gitlab_database_stat_table_*
   • Row count queries -- gitlab_database_rows
   • CI builds -- ci_pending_builds, ci_created_builds, ci_stale_builds, ci_running_builds
   • Bloat -- gitlab_database_bloat_$type_$key with type btree (index bloat) or table (table bloat) and keys bloat_ratio bloat_size extra_size real_size (see below)
   • Remote mirrors -- project_remote_mirror_last_successful_update_time_seconds, project_remote_mirror_last_update_time_seconds
2. Git
   • git pull/push timings -- git_pull_time_milliseconds, git_push_time_milliseconds
   • git processes stats (see Process below)
3. Process
   • CPU time -- process_cpu_seconds_total
   • Start time -- process_start_time_seconds
   • Count -- process_count
   • Memory usage
     • Data from /proc//cmdline:
     • process_resident_memory_bytes
     • process_virtual_memory_bytes
     • Data from /proc//smaps -- probe_smaps (off by default):
     • process_smaps_size_bytes
     • process_smaps_rss_bytes
     • process_smaps_shared_clean_bytes
     • process_smaps_shared_dirty_bytes
     • process_smaps_private_clean_bytes
     • process_smaps_private_dirty_bytes
     • process_smaps_swap_bytes
     • process_smaps_pss_bytes
4. Sidekiq -- sidekiq_queue_size, sidekiq_queue_paused, sidekiq_queue_latency_seconds, sidekiq_enqueued_jobs, sidekiq_dead_jobs, sidekiq_running_jobs, sidekiq_to_be_retried_jobs

Setup with GitLab Development Kit

gitlab-monitor can be setup with the GitLab Development Kit for development. When using the gitlab-monitor CLI, you'll need to set the --db-conn flag to connect to the PostgreSQL instance in your GDK folder. For example:

bin/gitlab-mon row-counts --db-conn="dbname=gitlabhq_development host=/Users/<user>/gitlab-development-kit/postgresql"

Running gitlab-monitor as a Web exporter

When serving the pages on localhost, you'll need to edit the YAML configuration file. An example can be found under config/gitlab-monitor.yml.example. For each probe that has to connect to the database, set the connection_string to dbname=gitlabhq_development host=/Users/<user>/gitlab-development-kit/postgresql

Once you have this configured, you can then run:

bin/gitlab-mon web -c config/gitlab-monitor.yml

Once running, you can point your browser or curl to the following URLs:

• http://localhost:9168/database
• http://localhost:9168/git_process
• http://localhost:9168/process
• http://localhost:9168/sidekiq
• http://localhost:9168/metrics (to get all of the above combined)

Database Bloat Metrics

Database bloat is measured for indexes (btree) and/or tables (table). Returned metrics contain:

• bloat_ratio: estimated ratio of the real size used by bloat_size.
• bloat_size: estimated size of the bloat without the extra space kept for the fillfactor.
• extra_size: estimated extra size not used/needed by the index. This extra size is composed by the fillfactor, bloat and alignment padding spaces.
• real_size: real size of the index

Also see the original documentation.

Note that all metrics returned are estimates without an upper bound for the error.

Contributing

gitlab-monitor is an open source project and we are very happy to accept community contributions. Please refer to CONTRIBUTING.md for details.