Browse Source

graphite: fix xFilesFactor

When we migrated this to ansible I missed that we didn't bring across
the storage-aggregation.conf file.

This has had the unfortunate effect of regressing the xFilesFactor set
for every newly created graphite stat since the migration.  This
setting is a percentage (0-1 float) of how much of a "bucket" needs to
be non-null to keep the value when rolling up changes.  We want this
to be zero due to the sporadic nature of data (see the original change
I5f416e798e7abedfde776c9571b6fc8cea5f3a33).

This only affected newly created statistics, as graphite doesn't
modify this setting once it creates the whisper file.  This probably
helped us overlook this for so long, as longer-existing stats were
operating correctly, but newer were dropping data when zoomed out.

Restore this setting, and double-check it in testinfra for the future.
For simplicity and to get this back to the prior state I will manually
update the on-disk .wsp files to this when this change applies.

Change-Id: I57873403c4ca9783b1851ba83bfba038f4b90715
changes/76/847876/8
Ian Wienand 2 months ago
parent
commit
6cd7433086
  1. 19
      playbooks/roles/graphite/tasks/main.yaml
  2. 9
      playbooks/roles/graphite/templates/docker-compose.yaml.j2
  3. 29
      playbooks/roles/graphite/templates/storage-aggregation.conf.j2
  4. 15
      testinfra/test_graphite.py
  5. 1
      zuul.d/system-config-run.yaml

19
playbooks/roles/graphite/tasks/main.yaml

@ -1,7 +1,11 @@
- name: Ensure docker-compose directory exists
- name: Ensure docker-compose directories exists
file:
state: directory
path: /etc/graphite-docker
path: '{{ item }}'
loop:
- /etc/graphite-docker
- /etc/graphite-docker/graphite
- /etc/graphite-docker/graphite/conf
- name: Write settings file
template:
@ -11,7 +15,12 @@
- name: Write storage config file
template:
src: storage-schemas.conf.j2
dest: /etc/graphite-docker/storage-schemas.conf
dest: /etc/graphite-docker/graphite/conf/storage-schemas.conf
- name: Write storage aggregation config file
template:
src: storage-aggregation.conf.j2
dest: /etc/graphite-docker/graphite/conf/storage-aggregation.conf
- name: Write nginx override config
template:
@ -47,11 +56,13 @@
shell:
cmd: docker image prune -f
# This is handy to have on the host for checking stat ingestion
# net is handy to have on the host for checking stat ingestion
# whisper utils are handy for inspecting .wsp files for debugging
- name: Install netcat
package:
name:
- netcat
- python3-whisper
state: present
# Removes files not updated for ~9 months, and clears out empty directories

9
playbooks/roles/graphite/templates/docker-compose.yaml.j2

@ -8,9 +8,10 @@ services:
image: docker.io/graphiteapp/graphite-statsd
network_mode: host
volumes:
- /etc/graphite-docker/graphite-statsd.conf:/etc/nginx/sites-enabled/graphite-statsd.conf
- /etc/graphite-docker/statsd.js:/opt/statsd/config/udp.js
- /etc/graphite-docker/storage-schemas.conf:/opt/graphite/conf/storage-schemas.conf
- /etc/letsencrypt-certs:/etc/letsencrypt-certs
- /etc/graphite-docker/graphite-statsd.conf:/etc/nginx/sites-enabled/graphite-statsd.conf:ro
- /etc/graphite-docker/statsd.js:/opt/statsd/config/udp.js:ro
- /etc/graphite-docker/graphite/conf/storage-schemas.conf:/opt/graphite/conf/storage-schemas.conf:ro
- /etc/graphite-docker/graphite/conf/storage-aggregation.conf:/opt/graphite/conf/storage-aggregation.conf:ro
- /etc/letsencrypt-certs:/etc/letsencrypt-certs:ro
- /opt/graphite/storage:/opt/graphite/storage
- /var/log/graphite:/var/log/

29
playbooks/roles/graphite/templates/storage-aggregation.conf.j2

@ -0,0 +1,29 @@
[min]
pattern = \.lower$
xFilesFactor = 0
aggregationMethod = min
[max]
pattern = \.upper(_\d+)?$
xFilesFactor = 0
aggregationMethod = max
[sum]
pattern = \.sum$
xFilesFactor = 0
aggregationMethod = sum
[count]
pattern = \.count$
xFilesFactor = 0
aggregationMethod = sum
[count_legacy]
pattern = ^stats_counts.*
xFilesFactor = 0
aggregationMethod = sum
[default_average]
pattern = .*
xFilesFactor = 0
aggregationMethod = average

15
testinfra/test_graphite.py

@ -56,3 +56,18 @@ def test_graphite_data(host):
found_value = True
assert found_value
def test_graphite_wsp(host):
# seed some timer data
cmd = ('timeout 20 bash -c '
'\'while true; do echo -n "example:$((RANDOM % 1000))|ms" '
'| nc -6 -w 1 -u localhost 8125; done\'')
host.run(cmd)
wsp_file = '/opt/graphite/storage/whisper/stats/timers/example/mean.wsp'
wsp = host.file(wsp_file)
assert wsp.exists
cmd = host.run('whisper-info %s' % wsp_file)
assert 'xFilesFactor: 0.0' in cmd.stdout

1
zuul.d/system-config-run.yaml

@ -692,6 +692,7 @@
graphite02.opendev.org:
host_copy_output:
'/var/log/graphite': logs
'/etc/graphite-docker': logs
files:
- playbooks/bootstrap-bridge.yaml
- playbooks/letsencrypt.yaml

Loading…
Cancel
Save