[00:01:37] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 1734 MB (7% inode=93%); [00:02:33] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 52.69, 30.11, 18.59 [00:03:37] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2683 MB (11% inode=93%); [00:06:24] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.09, 23.14, 18.31 [00:10:24] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.79, 20.04, 18.13 [00:38:30] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2648 MB (10% inode=93%); [01:00:44] PROBLEM - secularknowledge.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for secularknowledge.com could not be found [01:04:36] PROBLEM - mon1 APT on mon1 is CRITICAL: APT CRITICAL: 8 packages available for upgrade (3 critical updates). [01:07:36] RECOVERY - secularknowledge.com - reverse DNS on sslhost is OK: rDNS OK - secularknowledge.com reverse DNS resolves to cp6.miraheze.org [01:41:37] PROBLEM - services1 APT on services1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [01:57:34] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [02:14:17] PROBLEM - db7 APT on db7 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (3 critical updates). [02:20:22] PROBLEM - cloud2 APT on cloud2 is CRITICAL: APT CRITICAL: 83 packages available for upgrade (3 critical updates). [02:31:20] PROBLEM - test2 APT on test2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [02:39:54] PROBLEM - mw7 APT on mw7 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [03:15:35] PROBLEM - misc1 APT on misc1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [04:14:31] PROBLEM - puppet2 Current Load on puppet2 is WARNING: WARNING - load average: 7.07, 5.12, 3.89 [04:16:26] RECOVERY - puppet2 Current Load on puppet2 is OK: OK - load average: 1.48, 3.74, 3.53 [05:00:05] PROBLEM - mw4 APT on mw4 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [05:32:10] PROBLEM - cp7 APT on cp7 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [05:47:26] PROBLEM - mw5 APT on mw5 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [06:00:12] PROBLEM - cloud1 APT on cloud1 is CRITICAL: APT CRITICAL: 83 packages available for upgrade (3 critical updates). [06:02:20] RECOVERY - cloud2 APT on cloud2 is OK: APT OK: 80 packages available for upgrade (0 critical updates). [06:03:25] RECOVERY - db11 APT on db11 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:03:32] RECOVERY - cp3 APT on cp3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:06:12] RECOVERY - cloud1 APT on cloud1 is OK: APT OK: 80 packages available for upgrade (0 critical updates). [06:06:52] RECOVERY - jobrunner2 APT on jobrunner2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:11:27] RECOVERY - cp9 APT on cp9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:14:44] PROBLEM - puppet2 Current Load on puppet2 is WARNING: WARNING - load average: 6.86, 5.35, 4.14 [06:16:38] RECOVERY - puppet2 Current Load on puppet2 is OK: OK - load average: 1.27, 3.79, 3.71 [06:17:51] RECOVERY - mw7 APT on mw7 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:18:03] RECOVERY - mw4 APT on mw4 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:19:50] RECOVERY - gluster2 APT on gluster2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:22:17] RECOVERY - cp7 APT on cp7 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:22:44] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.32, 19.79, 18.20 [06:24:39] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.41, 18.54, 17.92 [06:29:38] RECOVERY - mw6 APT on mw6 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:32:15] RECOVERY - db7 APT on db7 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [06:32:15] RECOVERY - jobrunner1 APT on jobrunner1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:33:24] RECOVERY - test2 APT on test2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:33:40] RECOVERY - services1 APT on services1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:34:17] PROBLEM - mail1 APT on mail1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [06:35:16] RECOVERY - services2 APT on services2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:44:25] PROBLEM - rdb1 APT on rdb1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [06:44:35] RECOVERY - mon1 APT on mon1 is OK: APT OK: 5 packages available for upgrade (0 critical updates). [06:45:35] RECOVERY - misc1 APT on misc1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:46:23] RECOVERY - rdb1 APT on rdb1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:51:46] RECOVERY - phab1 APT on phab1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:52:52] RECOVERY - bacula2 APT on bacula2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:53:26] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [06:55:25] RECOVERY - mw5 APT on mw5 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:57:12] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu] [07:05:12] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:24:54] PROBLEM - gluster1 APT on gluster1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [07:47:30] PROBLEM - puppet2 APT on puppet2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [07:50:37] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.17, 20.40, 17.96 [07:52:30] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.09, 19.34, 17.87 [08:06:24] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.39, 19.51, 17.89 [08:10:24] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.19, 17.97, 17.70 [08:11:50] RhinosF1 around? [08:12:20] I am [08:12:51] DMs on discord [08:35:41] PROBLEM - ldap1 APT on ldap1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [08:38:25] PROBLEM - cp6 APT on cp6 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [08:53:25] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.00, 20.48, 18.82 [08:55:19] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.53, 18.39, 18.26 [09:28:41] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.79, 20.51, 18.25 [09:30:35] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.91, 19.15, 18.00 [09:53:52] PROBLEM - db13 APT on db13 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (3 critical updates). [09:54:39] PROBLEM - puppet2 Current Load on puppet2 is WARNING: WARNING - load average: 6.83, 5.04, 3.94 [09:56:33] RECOVERY - puppet2 Current Load on puppet2 is OK: OK - load average: 1.35, 3.65, 3.57 [10:56:24] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.21, 20.62, 18.37 [11:00:23] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJdV5 [11:00:24] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.02, 22.95, 19.97 [11:00:25] [02miraheze/services] 07MirahezeSSLBot 03a7c526c - BOT: Updating services config for wikis [11:02:24] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.44, 20.96, 19.60 [11:04:24] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.19, 19.72, 19.29 [11:19:38] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [11:45:35] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.77, 18.18, 17.45 [11:45:36] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1444 MB (5% inode=93%); [11:47:29] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.77, 16.85, 17.04 [12:02:36] PROBLEM - wiki.iufs.jp - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.iufs.jp' expired on Mon 27 Jul 2020 19:57:14 GMT +0000. [12:03:06] RECOVERY - wiki.iufs.jp - reverse DNS on sslhost is OK: rDNS OK - wiki.iufs.jp reverse DNS resolves to cp6.miraheze.org [12:14:36] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 8.14, 5.76, 4.46 [12:16:34] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 4.41, 5.19, 4.41 [12:37:17] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [12:43:46] Reception|away, JohnLewis: neither of you can do ^ please [12:44:58] RhinosF1: looking [12:46:49] !log umount/mount static on gluster1 [12:46:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:55:16] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:13:22] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JJdX2 [13:13:24] [02miraheze/ssl] 07Reception123 03e5619c5 - readd infectowiki cert [13:25:07] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [13:25:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJd1t [13:25:14] [02miraheze/services] 07MirahezeSSLBot 03424af60 - BOT: Updating services config for wikis [13:51:39] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JJdMT [13:51:40] [02miraheze/ssl] 07Reception123 03cd2b9ff - add history.sdtef.org cert [13:57:40] PROBLEM - db12 APT on db12 is CRITICAL: APT CRITICAL: 31 packages available for upgrade (3 critical updates). [14:10:22] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJdMa [14:10:23] [02miraheze/services] 07MirahezeSSLBot 03a4e66c9 - BOT: Updating services config for wikis [14:14:41] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JJdMM [14:14:43] [02miraheze/ssl] 07Reception123 03ff949ac - add usagihimepedia.ddns.net cert [14:15:22] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.08, 18.63, 16.28 [14:19:16] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.82, 19.11, 17.03 [14:30:22] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJdDH [14:30:24] [02miraheze/services] 07MirahezeSSLBot 03b3a2d73 - BOT: Updating services config for wikis [15:35:33] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.15, 21.93, 19.40 [15:37:31] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.49, 20.29, 19.07 [15:43:19] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 23.22, 22.91, 20.56 [15:44:07] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 9.05, 6.52, 4.98 [15:46:02] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 4.34, 5.81, 4.91 [15:47:13] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.44, 18.24, 19.25 [15:48:17] Alert to Miraheze Staff: It looks like the MirahezeRC bot has stopped! Recent Changes are no longer available from IRC. [15:48:26] !log restarted rc bot [15:48:27] We know [15:48:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:01:50] PROBLEM - rdb2 APT on rdb2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:54:45] PROBLEM - puppet2 Current Load on puppet2 is WARNING: WARNING - load average: 6.85, 5.22, 4.13 [16:56:42] RECOVERY - puppet2 Current Load on puppet2 is OK: OK - load average: 1.18, 3.68, 3.70 [17:35:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJdbR [17:35:15] [02miraheze/services] 07MirahezeSSLBot 0353cd8ee - BOT: Updating services config for wikis [18:48:07] @SigmaBotMH, why'd you nip out? (You probably won't respond to this.) [18:50:09] dmehus: you'd be best asking MacFan [18:50:40] @RhinosF1, good point. [18:51:33] .tell @MacFan4000 why did @SigmaBotMH nip out @ 10:05 AM (UTC -07:00) on 16 August 2020? [18:51:33] dmehus: I'll pass that on when @MacFan4000 is around. [18:51:44] dmehus: he's in -bots [18:52:03] @RhinosF1, oh, I looked there; didn't see him. [18:52:51] dmehus: I looked into us hosting channel logs and the plugin isn't opt in per channel so asked the dev to make it [18:53:07] And then we can move it to sopel.bots.miraheze.wiki [18:53:56] @RhinosF1, what do you mean opt-in per channel? Was that my earlier request asking whether @MirahezeBot could be added to one of the -cvt channels? [18:54:32] dmehus: I mean that the current channel logging plugin only allows every channel to be logged or none at all [18:54:42] PROBLEM - puppet2 Current Load on puppet2 is WARNING: WARNING - load average: 7.20, 5.29, 4.08 [18:55:27] @RhinosF1, ah, okay, the channel logging question I wondered about. [18:55:30] Thanks. [18:55:48] dmehus: if you poke Zppix then they can add MirahezeBot to -cvt, would rather local op requests it [18:56:20] .help [18:56:21] RhinosF1: I've published a list of my commands at: https://sopel.bots.miraheze.wiki/help_prod.html [18:56:30] @RhinosF1, by local op, you mean a current steward/CVT? [18:56:38] Yes [18:56:39] RECOVERY - puppet2 Current Load on puppet2 is OK: OK - load average: 1.21, 3.69, 3.64 [20:46:35] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.98, 18.48, 16.05 [20:48:35] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.44, 18.58, 16.38 [22:32:55] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 24.91, 19.22, 16.02 [22:34:53] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.92, 18.14, 16.02 [23:04:29] PROBLEM - puppet2 Current Load on puppet2 is WARNING: WARNING - load average: 7.67, 5.09, 3.95 [23:06:26] RECOVERY - puppet2 Current Load on puppet2 is OK: OK - load average: 1.44, 3.69, 3.58 [23:35:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJFk7 [23:35:14] [02miraheze/services] 07MirahezeSSLBot 03adcbc09 - BOT: Updating services config for wikis