[00:14:07] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 73186MiB (16% inode=99%);
[00:14:26] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 26783MiB (5% inode=99%);
[00:15:13] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 73271MiB (16% inode=99%);
[00:16:26] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 72197MiB (15% inode=99%);
[03:25:02] <icinga-miraheze>	 PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 19.66, 20.28, 10.29
[03:29:02] <icinga-miraheze>	 PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 4.97, 11.48, 8.91
[03:31:02] <icinga-miraheze>	 RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 5.64, 9.60, 8.54
[10:25:20] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/850e711cf2ac50dce171726bee5940a076d1b587
[10:25:20] <MacFanBot>	 02ssl/03main 07WikiTideBot 03850e711 Bot: Auto-update domain lists
[14:46:26] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49784MiB (10% inode=99%);
[15:15:22] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/3a71246cf2ffb4b4435e7a4f2069f8175355a9b9
[15:15:22] <MacFanBot>	 02ssl/03main 07WikiTideBot 033a71246 Bot: Auto-update domain lists
[15:19:13] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49870MiB (10% inode=99%);
[15:22:07] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49847MiB (10% inode=99%);
[17:21:13] <icinga-miraheze>	 PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 14979MiB (6% inode=94%);
[17:25:14] <icinga-miraheze>	 RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 55971MiB (25% inode=94%);
[17:28:22] <icinga-miraheze>	 RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:33:55] <MirahezeLSBot>	 !log [www-data@mwtask181] Began automatic backing up
[17:34:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:40:33] <MirahezeLSBot>	 !log [www-data@mwtask181] Began automatic backing up
[17:40:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:41:04] <RhinosF1>	 MacFan4000: should we stop that if it's broken then ^
[17:41:24] <MacFan4000>	 SRD already did
[17:41:43] <MacFan4000>	 IA said they fixed it
[17:41:49] <RhinosF1>	 Ah ok
[17:41:50] <RhinosF1>	 Cool
[17:41:59] <MacFan4000>	 I can now see on the web side that we can edit the collection now]
[17:51:52] <MirahezeLSBot>	 !log [www-data@mwtask181] Began automatic backing up
[17:51:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:58:44] <MacFan4000>	 actually i was wrong, the script was never stopped
[17:59:14] <MacFan4000>	 though it def needs to be stopped and restarted, as it seems to be consistently failing
[17:59:28] <MacFan4000>	 has to be done by somebody with root access though
[18:11:20] <RhinosF1>	 MacFan4000: what's the pid?
[18:12:09] <RhinosF1>	 Or service naw
[18:12:12] <RhinosF1>	 Name
[18:13:01] <MacFan4000>	 976794 is the pid
[18:13:05] <MacFan4000>	 mediawiki_job_backup-all-wikis-ia is the service
[18:18:28] <RhinosF1>	 MacFan4000: killed the pid
[18:19:01] <MacFan4000>	 the service needs a restart for it to run again
[18:19:44] <MirahezeLSBot>	 !log [www-data@mwtask181] Began automatic backing up
[18:19:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:20:01] <MacFan4000>	 ^ perfect
[18:20:12] <RhinosF1>	 MacFan4000: it's starting again
[18:20:26] <RhinosF1>	 sudo start is hanging
[18:21:54] <RhinosF1>	 Looks to be running
[18:22:19] <RhinosF1>	 MacFan4000: all look good?
[18:22:46] <MacFan4000>	 yup, looks like things are uploading without issue
[18:23:09] <RhinosF1>	 Yey!
[18:35:25] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/7a88b9a85543c5f3e601cd7885ae29e64a94e125
[18:35:25] <MacFanBot>	 02ssl/03main 07WikiTideBot 037a88b9a Bot: Auto-update domain lists
[20:45:29] <MirahezeRelay>	 <abaddriverlol, replying to MacFan4000> I had stopped the service using systemctl yesterday, but maybe puppet restarted it
[20:55:16] <MirahezeRelay>	 <abaddriverlol> !log "SET GLOBAL slow_query_log=1; SET GLOBAL slow_query_log_file = "/var/log/mysql/mysql-slow.log";" on db171
[20:55:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:08:38] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push
[21:16:38] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push
[21:47:45] <MirahezeRelay>	 <abaddriverlol> !log "SET GLOBAL slow_query_log=0;" on db171
[21:47:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log