This updates our user management system to use the userdel --force flag
when disabling and removing distro cloud image users like 'ubuntu',
'centos' and 'admin'. The reason for this is when we switch from using
the distro user to boot strap launchnode over to root the distro user
may still have running processes that prevent userdel from succeeding.
This should address that problem and delete the user anyway.
The last step in the launch node process is to reboot which should clear
out any stale processes.
We don't do this for normal users as they aren't removed at node launch
time and this may be too forceful for them. It would be better for us to
error in that case and clean up any stale processes.
Change-Id: I79caf2a996566ecaec4cb4a70941bb3f03a5fb73
In gitea 1.14.0 they dropped the macaron http router for go-chi. This
seems to have changed how the request context's RemoteAddr is parsed in
logging. Importantly instead of a valid source port we get :0 which
makes it difficult to trace a connection from apache to gitea.
The origin of this behavior seems to be handling of X-Forwarded-For
headers that apache is setting. To address this we drop those headers
in hopes that gitea will log raw details for the apacher -> gitea
connection in that case. Due to not using x-forwarded-for anymore we
need to log the source port that apache is using for the proxy pass
connection which is done by modifying the apache log format.
Change-Id: I1e69431bf703947dc5c223df2a9e1b55bd0d841c
Switch because our overridden format does not seem to do what we expect
anymore. It always shows the port as :0 which is problematic as it
doesn't include necessary info. The update was made to accomodate
macaron which gitea has apparently moved away from (in favor of go-chi)
it is possible that the default format does what we want now.
Change-Id: If9bcf5cdb11911a46a8fd728346f2f35ffa5e8ba
Apparently "30d" as a ttl value breaks badger, but "720h" is okay.
Also, run the process as the jaeger user.
Change-Id: I00ad6b529aa86b0051343f828e05df8212b34656
We have instructed zuul to connect to tracing.opendev.org, but
we are generating a certificate using opendev-ca with
S=tracing01.opendev.org. Update the certificate with the correct
subject.
This also corrects the opendev-ca role which assumed that the cert
filename would always be inventory_hostname.
Change-Id: I9b6b0534f058d386e01910bb7efc30312f3d72ad
The env variables we set for the TLS configuration were for the
jaeger native gRPC server rather than the otlp server.
This corrects the settings.
Change-Id: Id8b9157a104af243ee62a9d0aadbe56d09ea4ae5
Right now latest ceph release is Quincy. In order to test it in CI and
not fetch packages via external network, it's proposed to add relevant
mirror. At the moment OpenStack-Ansible and Loki does use these mirrors.
Change-Id: Icb836b9046571c5f824a3b57dafca05d37f94372
This generates TLS certs for Zuul using the jaeger CA and enables
tracing on all Zuul components, exporting to tracing.opendev.org.
Change-Id: I821e5ce4738ea0c93e116684033fa7b78e2da8c6
This renames zk-ca to opendev-ca and allows us to operate more than
one ca on bridge. This way we can keep the CAs for ZooKeeper and
Jaeger distinct (so that a compromise of the jaeger server could not
be used to access the ZooKeeper cluster).
This also starts a new jaeger-ca and uses it on the Jaeger server.
Change-Id: I4e5bc4e3ccd78284ce785c971f7e6ad6e721f887
In switching to all-HTTPS for Mailman sites, it was missed that only
the plain HTTP vhosts set a DocumentRoot of /var/www. This was only
used for publishing metadata so went unnoticed until now. Rather
than add a DocumentRoot to the new HTTPS vhosts, simply use Aliases
to map the specific files we want to expose, for improved clarity
and to make it less likely they'll be overlooked in configuration in
the future.
In order to make sure the archives.yaml file exists at server
creation, before its cronjob fires for the first time, add a direct
invocation of the script which builds it. Move all tasks related to
this after the tasks which create the mailing lists, so that the
generated file will include them. This also simplifies testing.
For the non-multihost configuration, only robots.txt is expected to
be present, so don't add an alias for archives.yaml there.
Also add regression tests to ensure we keep these working.
Change-Id: I6b54b0386f0ea9f888c1f23580ad8698314474b9
This fixes the JVB connection info to use IP addrs instead of names
since nginx can't seem to do name lookups. Additionally, we modify the
cert CN to match the IP address used.
Change-Id: I6bbca44b60559d9586741c6540cb390371e3c120
We are currently running an all in one jitsi meet service at
meetpad.opendev.org due to connectivity issues for colibri websockets to
the jvb servers. Before we open these up we need to configure the http
server for websockets on the jvbs to do tls as they are on different
hosts.
Note it isn't entirely clear yet if a randomly generated keystore is
sufficient for the needs of the jvb colibri websocket system. If not we
may need to convert an LE provisioned cert and key pair into a keystore.
Change-Id: Ifbca19f1c112e30ee45975112863fc808db39fc9
The previous code had attempted to handle the case where the container
isn't running and we exec a zuul graceful stop in the container.
Unfortunately I got the string to check for wrong. I think I must've
checked the `docker exec` output and not the `docker-compose exec`
output.
This change updates the string to match exactly what ansible complained
about:
TASK [zuul-merger : Gracefully stop Zuul Merger] *******************************
fatal: [zm05.opendev.org]: FAILED! => {
"changed": true,
"cmd": "docker-compose exec merger zuul-merger stop",
"delta": "0:00:00.573561",
"end": "2022-09-14 01:59:41.721044",
"failed_when_result": true,
"rc": 1,
"start": "2022-09-14 01:59:41.147483"
}
STDERR:
No container found for merger_1
MSG:
non-zero return code
Specifically we check for 'No container found' in stderr.
Change-Id: I737b9da14c210215926804816d1e032540d694dc
There are two issues in the zuul merger and executor shutdowns. The
first is that `docker-compose ps -q` will report exited containers
unlike `docker ps -q`. This means we may try to exec into a non running
container which is an error. Handle this by checking the error message
and proceeding if the 'is not running' string is present.
The second issue is a race between stopping a container and running an
exec in the container. If a container stops while an exec is running in
it that exec appears to be treated with some equivalent of kill -9. The
result is the exec returns 137. While theoretically possible for both
executor and merger graceful stop command we seem to only hit this with
the merger so we handle exit code 137 for the merger only. This way
we'll get info if the executors start running into this too.
Change-Id: Ia6dc2d7e397631d72968ffa89c4492b803c89c47
In the graceful shutdown for mergers and executors if we skip the docker
exec to stop the container we also need to skip the docker wait. The
reason for this is docker wait exits with an error code if not provided
with any arguments to wait for.
Change-Id: Id09666ee23e1a9599d477b63a89559e4ab1d21bf
Ansible syntax got me. When I updated the apt tasks to retry on apt/dpkg
locks I overindented the register, delay, retries, and until parameters.
These are to the task not the module.
Change-Id: I955d96b5467597503e0e5563e37ffa736ef2fcdc
The way the currently graceful stop tasks are written for zuul expects
there to always be a running zuul container to exec into. There are
situations where there may not be a running container in which case
there is nothing to stop. Avoid this being an error by checking if the
containers are running before execing into them. If no containers are
running then we'll noop the docker exec step allowing the rest of the
ansible tasks to continue.
Change-Id: I6c47147a589ae12cc33e37e40e49673396d120f7
The zuul_reboot playbook runs on each zuul server at what essentially
become random times based on how long the previous servers took to be
updated. We have seen this result in our apt tasks colliding with
unattended upgrades on the server.
Latest ansible would let us workaround this using the lock_timeout
parameter to the apt module, but the version we use on bridge does not
support this parameter. Instead we check the failure message for
'Failed to lock apt for exclusive operation' and if present we retry. We
wait 30 seconds between retries and will perform up to 40 attempts for a
total of 20 minutes of waiting. This method should also be forward
compatibile with new Ansible.
If the lock is held for longer than 20 minutes it likely implies
something has gone wrong and we will need to perform manual intervention
anyway.
Change-Id: I3171838a30e3ea496bb08f8b6ab1c95755b2ff3c
Normally this isn't an issue because we run logrotate more frequently
than our weekly cron to upgrade and reboot zuul. But if you need to
manually run the playbook and are referring to the crontab entry to
determine how to run the playbook then the resulting command could
truncate a recent run. Simply append to the file in all cases to avoid
this.
Change-Id: I393741317cccaf447912b1f1517e846c32ee7677
For some reason, the JVB servers now seem to tell clients to connect
to 8443/tcp on localhost rather than the actual server. It seems it
wants to build the URL based on the PUBLIC_URL envvar, but we
previously did not pass that through to the JVB containers. Add it
to their configuration so they'll have it available.
Change-Id: I10c761105490a72c4eb9ac0b08a304b7d5d1e18c
It appears upstream container init now copies this from defaults,
overwriting our modifications. Shadow the one in the container with
ours so it gets copied into the eventual destination.
Also switch back to the old muting variables we were using before,
since the new "with" bools seem not to work (still worth looking
into later).
Change-Id: I7e91e82e6f91b44c5c7eb1406ba0c64d30e6b8ff
Bring our 5 configs into line with current upstream versions
(jitsi-meet_7648 and stable-7648-4 tags from the jitsi-meet and
docker-jitsi-meet repositories respectively). Attempt to preserve
most of our earlier overrides:
* configure Etherpad integration
* disable background blurring
* disable watermarks
* open shared document on join
* start with audio and video muted
* redirect HTTP to HTTPS
* disable XMPP WebSockets
* disable P2P connections
* templated credentials
* templated unique JVB server identifiers
Drop any options we previously set which later became defaults (like
useRoomAsSharedDocumentName or UTC as the TZ). Identify the upstream
repo and tag on which each file is based. Stop claiming Firefox is
not recommended, now that the default configuration adds a pre-join
page which helps browsers realize they should not treat the audio
stream as unsolicited. Switch to newer vars for muting audio and
video as a boolean rather than at a participant threshold.
Update the docker-compose files to use the stable tag instead of
latest, since upstream seems to just stop refreshing the latest tag
far too often. Clean up extra envvars we were setting for JVB which
we didn't pass through to the containers.
Change-Id: I1e5a3836917f3d90ad7dd1c0771871740fda3cda
This reverts commit cc2dd16d3a.
Reason for revert: rax mirrors not synched for 15 hours and causing
issues, facebook mirror is up to date so let's switch to it.
Change-Id: Iaf94540f22e2b49c74ab0704ac94fd1554ce5bbc
Related-Bug: #1988397
We do this to prevent lookups to third parties for information in this
case avatar info. We should verify this doesn't break local avatar
storage usage which we manage directly.
Change-Id: I612bf1629bd211ed14203cc9e39f34ba0be041bf
Please carefully review the changelog:
https://github.com/go-gitea/gitea/blob/v1.17.1/CHANGELOG.md
and ensure that we've properly addressed the items listed there.
I have listed the breaking changes list here and any actions we've taken
or justification for why they don't affect us:
* Require go1.18 for Gitea 1.17 (#19918)
We were already using go 1.18.
* Make AppDataPath absolute against the AppWorkPath if it is not (#19815)
Path is already absolute:
playbooks/roles/gitea/templates/app.ini.j2:APP_DATA_PATH = /data/gitea
* Nuke the incorrect permission report on /api/v1/notifications (#19761)
This has to do with how that api endpoint returns permissions. We
don't use this anywhere as far as I can tell.
* Refactor git module, make Gitea use internal git config (#19732)
In the gitea container /data/git/.gitconfig is present but we don't
appear to manage this in system-config. I think that means this
change is a noop for us as gitea will move its managed .gitconfig
from /data/git/.gitconfig to /data/git/repositories/.gitconfig.
I expect the contents to be the same since gitea must be managing
the file old content today.
* Remove RequireHighlightJS field, update plantuml example. (#19615)
This was a flag that toggled syntax highlighting on and off as best
as I can tell. The default is to just have it turned on and we don't
check the flag in any of our templates.
* Increase minimal required git version to 2.0 (#19577)
Debian Bullseye ships with 2.30.2-1.
* Add a directory prefix gitea-src-VERSION to release-tar-file (#19396)
They were tarbombing people and their tarballs extracted into the
current dir. They now no longer do that. We build from git so this
doesn't affect us.
* Use "main" as default branch name (#19354)
We explicitly set the default branch name to master for both gitea and
gerrit. This should be a noop for us. Testing has been added to check
this.
https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/gitea-git-repos/library/gitea_create_repos.py#L129-L132https://opendev.org/opendev/jeepyb/src/branch/master/jeepyb/cmd/manage_projects.py#L488
* Make cron task no notice on success (#19221)
I'm not aware of us relying on any cron tasks or any cron task
notifications.
* Add pam account authorization check (#19040)
We don't integrate with pam so the change in behavior to check
authorization does not affect us.
* Show messages for users if the ROOT_URL is wrong, show JavaScript errors (#18971)
This message shows up in CI because ROOT_URL is https://opendev.org
but we access gitea in testing via localhost. I don't think this
is worth fixing. Its a good reminder that the instance is a test
instance.
* Refactor mirror code & fix StartToMirror (#18904)
We don't mirror repos with gitea. Should be a noop for us.
* Remove deprecated SSH ciphers from default (#18697)
hmac-sha1-96, diffie-hellman-group1-sha1, and arcfour{128,256} are
removed. The only ssh user is gerrit's replication. MINA should
be able to support more modern ciphers and be fine.
* Add the possibility to allow the user to have a favicon which differs from the main logo (#18542)
Previously, logo.svg was used as the favicon.svg and gitea only fell
back to favicon.png if the browser couldn't so the .svg. But now they
want to support users having different logo.svg and favicon.svg. This
necessitates explicitly adding a favicon.svg. Something we already do.
Details at https://github.com/go-gitea/gitea/pull/18542
* Update reserved usernames list (#18438)
This shouldn't be a problem for us as we don't have regular users and
gerrit is not a reserved name.
* Support custom ACME provider (#18340)
We run ACME with LE out of band. This doesn't affect us.
* Change initial TrustModel to committer (#18335)
This changes the signed commits trust model from collaborator
to committer. THis won't affect us as we aren't maintaining trusted
keys. But basically this now shows if the signed commit by the
committer matches the committer's key.
* Update HTTP status codes (#18063)
This changed redirect HTTP codes from 302 to 307. Shouldn't
affect us.
* Upgrade Alpine from 3.13 to 3.15 (#18050)
We build on Debian and not alpine. The alpine nodejs version did
change from 14 to 16 in this change and we've updated to match.
* Restrict email address validation (#17688)
If we had real users this may pose a problem as they are limiting
the set of emails gitea would accept to a smaller set than they
accepted before. Also fewer than actually allowed by email. But
we don't have real users so this should be fine.
* Refactor Router Logger (#17308)
This streamlines and improves the log format of some of the gitea
logs. We aren't automatically processing these logs today so this
shouldn't have a major impact on us.
Additionally this release adds a new git.HOME_PATH setting to set the
location for writing out git configs and potential gnupg configs. We
should be fine to let gitea write this content out to the default path,
but there is potential for this to impact our ssh daemon.
Changes made include:
* Minimal updates to web templates to match 1.17
* Updating nodejs to v16 as v14 failed to build gitea
* Disabling the new enabled by default "packages" feature
* New test to check repos have a master branch by default instead of
Gitea's new default of main.
Change-Id: I88105eccd118e3daca72f0b86a6b351c35e37413
We've seen CI systems consume all of our threads which causes the web UI
to become non responsive. To address this increase the number of httpd
threads from 100 to 150. Note that we do not modify sshd.threads beacuse
sshd.threads determines the max number of git requests across both ssh
and http.
In theory what this means is that httpd has an additional 50 threads to
process non git requests (for example web UI requests) which will
hopefully keep that responsive even if git requests are max'd out.
It is possible that we also need to increase the sshd.threads value to
handle those git requests, but we will start by modifying one config
value at a time. If we do bump sshd.threads we should increase
httpd.maxThreads to give it that additional headroom.
Finally, I believe this is likely to be safe as we doubled the size of
our Gerrit server when we moved it to vexxhost. The old server was
pretty well maxed out though so increase these values on the new server
slowly and monitor the results.
Details on the configuration can be found at:
https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#httpd
Change-Id: I57a1e248c3c01597bb29c7afc304688e834a64cc
Facebook mirror is out of sync for some days, so i'm proposing to use
rackspace one. This reverts [1] as it seems it is accepting rsync
connections properly.
[1] https://review.opendev.org/c/opendev/system-config/+/824829
Change-Id: Ic0076191157be8947f62ce18d5dd37f1f0ac3337
We still have some Ubuntu Xenial servers, so cap the max usable pip
and setuptools versions in their venvs like we already do for
Bionic, in order to avoid broken installations. Switch our
conditionals from release name comparisons to version numbers in
order to more cleanly support ranges. Also make sure the borg run
test is triggered by changes to the create-venv role.
Change-Id: I5dd064c37786c47099bf2da66b907facb517c92a
This is the latest 1.1.18 release, and from the changelog there
doesn't seem to be anything important we need to take into account
from 1.1.14.
Just as a note the 1.2 series is released, but this requires much more
thought when updating.
Change-Id: I949c40e9046008d4f442b322a267ce0c967a99dc
As noted inline, make a create-venv role that brings in appropriate
versions on Bionic.
This was noticed because pip is trying to install borgbackup with
setuptools_scm 7.0.5, which doesn't support Python 3.6. We use this
new role to create the venv correctly.
Change-Id: I81fd268a9354685496a75e33a6f038a32b686352
This is a follow-on to Ica63860f3221e99ca0a2aa2636d573fc134447bb to
make what's happening with the various exit points clearer.
Also sneak in an explaination of the weird arg input for clarity.
Change-Id: Ib059f1de465430d6e6f674b6649817105b7ef9a0
Currently we discard the exit code of the acme.sh call and swallow any
possible errors. Although they are logged, it means the Ansible calls
won't fail and you'll have to debug much later on why you didn't get a
certificate as expected.
Capture the failure of the call and log it better. Note that when
skipping renewal due to current valid certificates acme.sh returns
"2". After [1] acme.sh is returning "3" when it exits with a TXT
entry requiring validation; anything else is an error on the request
path. Valid issues should be "0" and anything else will be an error.
While we here, make sure we always output the end stamp by putting it
in a exit trap.
[1] 2d4ea720eb
Change-Id: Ica63860f3221e99ca0a2aa2636d573fc134447bb
Keeping the testing nodes at the other end of the namespace separates
them from production hosts. This one isn't really referencing itself
in testing like many others, but move it anyway.
Change-Id: I2130829a5f913f8c7ecd8b8dfd0a11da3ce245a9
Similar to Id98768e29a06cebaf645eb75b39e4dc5adb8830d, move the
certificate variables to the group definition file, so that we don't
have to duplicate handlers or definitions for the testing host.
Change-Id: I6650f5621a4969582f40700232a596d84e2b4a06
Currently we define the letsencrypt certs for each host in its
individual host variables.
With recent work we have a trusted CA and SAN names setup in
our testing environment; introducing the possibility that we could
accidentally reference the production host during testing (both have
valid certs, as far as the testing hosts are concerned).
To avoid this, we can use our naming scheme to move our testing hosts
to "99" and avoid collision with the production hosts. As a bonus,
this really makes you think more about your group/host split to get
things right and keep the environment as abstract as possible.
One example of this is that with letsencrypt certificates defined in
host vars, testing and production need to use the same hostname to get
the right certificates created. Really, this should be group-level
information so it applies equally to host01 and host99. To cover
"hostXX.opendev.org" as a SAN we can include the inventory_hostname in
the group variables.
This updates one of the more tricky hosts, static, as a proof of
concept. We rename the handlers to be generic, and update the testing
targets.
Change-Id: Id98768e29a06cebaf645eb75b39e4dc5adb8830d
Our infra prod jobs use a strftime format string to update log file
modication times. This format string had a stray '%' in it leading to:
"Error while obtaining timestamp for time 2022-08-04T18:02:59 using
format %Y-%m%-%dT%H:%M:%S: '-' is a bad directive in format
'%Y-%m%-%dT%H:%M:%S'"
Fix that by removing the extra '%'.
Change-Id: I934ecb4b24244fdd00fa16de6e4c4ae67542e2fe
Replace the optional src parameter with the required path parameter,
this seems to have been a simple typo.
Change-Id: Ib95a11b28eed138225b57062faeb7c344067c991
The etsencrypt_certs variable defined here in the "static" group file
is overwritten by the host variable. This is not doing anything (and
we don't have a logs.openstack.org any more as it is all in object
storage), remove it.
Change-Id: I6910d6652c558c94d71b1609d1194b654bc5b42d
Jammy nodes appear to lack the /etc/apt/sources.list.d dir by default.
Ensure it exists in the install-docker role before we attempt to
install a deb repo config to that directory for docker packages.
Change-Id: I859d31ed116607ffa3d8db5bfd0b805d72dd70c0
This is the first step in running our servers on jammy. This will help
us boot new servers on jammy and bionic replacements on jammy.
Change-Id: If2e8a683c32eca639c35768acecf4f72ce470d7d
The most recent version of the grafana-oss:latest container seems to be
a beta version with some issues, or maybe we need to adapt our
deployment. Until we do this, pin the container to the latest known
working version.
Change-Id: Id50bf3121f3009f36f0f9961cf5211053410a576
How we got here - I3e99b80e442db0cc87f8e8c9728b7697a5e4d1d3 split the
log collection into a post-run job so we always collect logs, even if
the main run times out. We then realised in
Ic18c89ecaf144a69e82cbe9eeed2641894af71fb that the log timestamp fact
doesn't persist across playbook runs and it's not totally clear how
getting it from hostvars interacts with dynamic inventory.
Thus take an approach that doesn't rely on passing variables; this
simply pulls the time from the stamp we put on the first line of the
log file. We then use that to rename the stored file, which should
correspond more closely with the time the Zuul job actually started.
To further remove confusion when looking at a lot of logs, reset the
timestamps to this time as well.
Change-Id: I7a115c75286e03b09ac3b8982ff0bd01037d34dd
Clark Boylan
yatin