Use dstat to record performance of system-config-run hosts

We have seen some poor performance from gitea which may be related to
manage project updates. Start a dstat service which logs to a csv file
on our system-config-run job hosts in order to collect performance info
from our services in pre merge testing. This will include gitea and
should help us evaluate service upgrades and other changes from a
performance perspective before they hit production.

Change-Id: I7bdaab0a0aeb9e1c00fcfcca3d114ae13a76ccc9
changes/51/775051/5
Clark Boylan 2 years ago
parent 7a5041140b
commit 1560b01f7e
  1. 6
      playbooks/roles/dstat-logger/README.rst
  2. 28
      playbooks/roles/dstat-logger/tasks/main.yaml
  3. 3
      playbooks/service-dstatlogger.yaml
  4. 2
      playbooks/zuul/run-base.yaml
  5. 1
      zuul.d/system-config-run.yaml

@ -0,0 +1,6 @@
Install, configure, and run a dstat logger
This is primarily useful for testing where we don't have instances hooked
up to cacti. You can use this to get a csv log file at
/var/log/dstat-csv.log in test jobs that records similar system performance
information.

@ -0,0 +1,28 @@
- name: Install dstat
become: yes
package:
name: dstat
state: present
- name: Install dstat unit
become: yes
copy:
dest: /etc/systemd/system/dstatlogger.service
owner: root
group: root
mode: 0644
content: |
[Unit]
Description=Dstat logging service
[Service]
User=root
ExecStart=/usr/bin/dstat -tcmndrylpg --tcp --swap --output /var/log/dstat-csv.log
KillMode=process
TimeoutStopSec=300
[Install]
WantedBy=multi-user.target
- name: Reload systemd
become: yes
command: systemctl daemon-reload
- name: Start dstat unit
become: yes
command: systemctl start dstatlogger

@ -0,0 +1,3 @@
- hosts: "*:!disabled"
roles:
- dstat-logger

@ -92,6 +92,8 @@
command: ansible-playbook -f 50 -v /home/zuul/src/opendev.org/opendev/system-config/playbooks/base.yaml
- name: Run bridge service playbook
command: ansible-playbook -v /home/zuul/src/opendev.org/opendev/system-config/playbooks/service-bridge.yaml
- name: Run dstat logger playbook
command: ansible-playbook -v /home/zuul/src/opendev.org/opendev/system-config/playbooks/service-dstatlogger.yaml
- name: Run playbook
when: run_playbooks is defined
loop: "{{ run_playbooks }}"

@ -17,6 +17,7 @@
'/var/log/messages': logs_txt
'/var/log/docker': logs
'/var/log/containers': logs
'/var/log/dstat-csv.log': logs
'/etc/iptables/rules.v4': logs_txt
'/etc/iptables/rules.v6': logs_txt
host-vars:

Loading…
Cancel
Save