[03:10:21] <icinga-miraheze>	 RECOVERY - bacula1 Bacula Static on bacula1 is OK: OK: Full, 3066142 files, 232.8GB, 2019-02-24 03:08:00 (2.3 minutes ago)
[03:14:44] <icinga-miraheze>	 PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.74, 1.81, 0.86
[03:18:45] <icinga-miraheze>	 RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 1.43, 1.67, 1.03
[03:22:44] <icinga-miraheze>	 PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 2.00, 1.84, 1.24
[03:24:44] <icinga-miraheze>	 RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 1.18, 1.59, 1.21
[03:42:46] <icinga-miraheze>	 PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.72, 2.07, 1.61
[03:48:45] <icinga-miraheze>	 PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.43, 2.00, 1.77
[03:52:46] <icinga-miraheze>	 PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.94, 2.25, 1.89
[03:53:53] <icinga-miraheze>	 PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:54:45] <icinga-miraheze>	 PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.46, 1.98, 1.84
[03:55:47] <icinga-miraheze>	 RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Wed 15 May 2019 09:02:45 PM GMT +0000.
[04:00:46] <icinga-miraheze>	 PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.12, 1.87, 1.79
[04:02:46] <icinga-miraheze>	 RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 1.13, 1.59, 1.70
[04:11:58] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhbsX
[04:11:59] <Not-8d6a>	 [02miraheze/puppet] 07paladox 034449307 - Removed needed def from ordered_yaml
[05:48:32] <icinga-miraheze>	 PROBLEM - spiral.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[05:50:34] <icinga-miraheze>	 RECOVERY - spiral.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spiral.wiki' will expire on Wed 08 May 2019 09:46:50 PM GMT +0000.
[07:08:57] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 320
[07:10:57] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[08:45:14] <icinga-miraheze>	 PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:47:12] <icinga-miraheze>	 RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 07 May 2019 10:15:37 PM GMT +0000.
[12:16:57] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 352
[12:20:57] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[15:12:16] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhb4v
[15:12:18] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox 03a0c8a99 - Update AbuseFilter
[15:35:52] <MacFan4000>	 !log DELETE FROM mw_permissions WHERE perm_dbname='metawiki' AND perm_group='reviewer';
[15:35:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:39:14] <MacFan4000>	 !log DELETE FROM mw_permissions WHERE perm_dbname='metawiki' AND perm_group='oversight'; (was only adding one perm)
[15:39:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:40:14] <MacFan4000>	 !log DELETE FROM mw_permissions WHERE perm_dbname='metawiki' AND perm_group='forumadmin';
[15:40:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:43:11] <MacFan4000>	 !log DELETE FROM mw_permissions WHERE perm_dbname='metawiki' AND perm_group='chatmod';
[15:43:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:43:48] <MacFan4000>	 !log DELETE FROM mw_permissions WHERE perm_dbname='metawiki' AND perm_group='comentadmin';
[15:43:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:46:53] <MacFan4000>	 !log UPDATE mw_permissions SET perm_permissions='["editinterface","editusercss","edituserjson","edituserjs","editsitecss","editsitejson","editsitejs"]' WHERE perm_dbname='metawiki' AND perm_group='interface-admin';
[15:46:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:51:46] <MacFan4000>	 !log UPDATE mw_permissions SET perm_permissions='["rollback"]' WHERE perm_dbname='metawiki' AND perm_group='rollbacker';
[15:51:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:54:05] <MacFan4000>	 !log UPDATE mw_permissions SET perm_permissions='["move","createpage","translate","mwoauthproposeconsumer","mwoauthupdateownconsumer","editsemiprotected","autoconfirmed","skipcaptcha"]' WHERE perm_dbname='metawiki' AND perm_group='confirmed';
[15:54:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:05:17] <MacFan4000>	 !log UPDATE mw_permissions SET perm_addgroups='["interface-admin","bot","bureaucrat","sysop"]' WHERE perm_dbname='metawiki' AND perm_group='bureaucrat';
[16:05:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:06:10] <MacFan4000>	 !log UPDATE mw_permissions SET perm_removegroups='["interface-admin","bot","sysop"]' WHERE perm_dbname='metawiki' AND perm_group='bureaucrat';
[16:06:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:08:32] <paladox>	 !log depool mw2
[16:08:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:09:27] <paladox>	 !log upgrade php on mw2 to 7.3.2 (bug fixes) && also dist upgrade to 9.8
[16:09:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:11:19] <paladox>	 !log rebooted mw2
[16:11:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:13:37] <MacFan4000>	 !log UPDATE mw_permissions SET
[16:13:37] <MacFan4000>	 perm_permissions='["autopatrol","block","blockemail","browsearchive","createaccount","createpage","createtalk","delete","deletechangetags","deletedhistory","deletedtext","deletelogentry","deleterevision","edit","editcontentmodel","editinterface","editprotected","editsemiprotected","editusercss","edituserjson","edituserjs","import","importupload","minoredit","move","noratelimit","override-export-depth","pagelang","patrol","patrolmarks","protect"
[16:13:37] <MacFan4000>	 ,"purge","rollback","suppressredirect","unblockself","undelete","upload_by_url","abusefilter-modify","abusefilter-log-detail","abusefilter-view","abusefilter-log","abusefilter-modify-restricted","abusefilter-revert","abusefilter-view-private","abusefilter-log-private","override-antispoof","gadgets-edit","gadgets-definition-edit","globalblock-whitelist","nuke","oathauth-enable","oathauth-api-all","mwoauthproposeconsumer","mwoauthupdateownconsume
[16:13:37] <MacFan4000>	 r","tboverride","tboverride-account","titleblacklistlog","translate","translate-import","translate-manage","translate-messagereview","translate-groupreview","pagetranslation","massmessage"]' WHERE perm_dbname='metawiki' AND perm_group='sysop';
[16:13:40] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:13:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:13:53] <icinga-miraheze>	 PROBLEM - mw2 php-fpm on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:13:59] <icinga-miraheze>	 PROBLEM - mw2 HTTPS on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:14:00] <MacFan4000>	 !log
[16:14:00] <MacFan4000>	 perm_permissions='["autopatrol","block","blockemail","browsearchive","createaccount","createpage","createtalk","delete","deletechangetags","deletedhistory","deletedtext","deletelogentry","deleterevision","edit","editcontentmodel","editinterface","editprotected","editsemiprotected","editusercss","edituserjson","edituserjs","import","importupload","minoredit","move","noratelimit","override-export-depth","pagelang","patrol","patrolmarks","protect"
[16:14:16] <MacFan4000>	 !log
[16:14:16] <MacFan4000>	 ,"purge","rollback","suppressredirect","unblockself","undelete","upload_by_url","abusefilter-modify","abusefilter-log-detail","abusefilter-view","abusefilter-log","abusefilter-modify-restricted","abusefilter-revert","abusefilter-view-private","abusefilter-log-private","override-antispoof","gadgets-edit","gadgets-definition-edit","globalblock-whitelist","nuke","oathauth-enable","oathauth-api-all","mwoauthproposeconsumer","mwoauthupdateownconsume
[16:14:17] <icinga-miraheze>	 PROBLEM - mw2 SSH on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:14:22] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2
[16:14:24] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2
[16:14:36] <MacFan4000>	 !log r","tboverride","tboverride-account","titleblacklistlog","translate","translate-import","translate-manage","translate-messagereview","translate-groupreview","pagetranslation","massmessage"]' WHERE perm_dbname='metawiki' AND perm_group='sysop';
[16:14:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:14:42] <icinga-miraheze>	 PROBLEM - mw2 Current Load on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:14:57] <icinga-miraheze>	 PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw2
[16:15:11] <icinga-miraheze>	 PROBLEM - mw2 Disk Space on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:16:14] <icinga-miraheze>	 RECOVERY - mw2 SSH on mw2 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u5 (protocol 2.0)
[16:16:38] <icinga-miraheze>	 RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 0.12, 0.03, 0.01
[16:17:09] <icinga-miraheze>	 RECOVERY - mw2 Disk Space on mw2 is OK: DISK OK - free space: / 59065 MB (76% inode=99%);
[16:17:37] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[16:17:41] <icinga-miraheze>	 RECOVERY - mw2 php-fpm on mw2 is OK: PROCS OK: 3 processes with command name 'php-fpm7.3'
[16:17:46] <icinga-miraheze>	 RECOVERY - mw2 HTTPS on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 22921 bytes in 0.008 second response time
[16:18:10] <paladox>	 !log repool mw2
[16:18:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:18:22] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[16:18:24] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy
[16:18:51] <MacFan4000>	 !log UPDATE mw_permissions SET perm_addgroups='["flood","translator","translationadmin","autopatrolled","confirmed","rollbacker"]' WHERE perm_dbname='metawiki' AND perm_group='sysop';
[16:18:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:18:57] <icinga-miraheze>	 RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy
[16:19:15] <paladox>	 !log depool mw3
[16:19:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:19:25] <paladox>	 !log upgrade php on mw3 to 7.3.2 (bug fixes) && also dist upgrade to 9.8
[16:19:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:21:31] <paladox>	 !log reboot mw3
[16:21:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:22:12] <MacFan4000>	 !log UPDATE mw_permissions SET perm_permissions='["autoconfirmed","editsemiprotected","move","createpage","translate","mwoauthproposeconsumer","mwoauthupdateownconsumer","skipcaptcha","createaccount"]' WHERE perm_dbname='metawiki' AND perm_group='autoconfirmed';
[16:22:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:23:14] <icinga-miraheze>	 PROBLEM - mw3 JobQueue on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:24:02] <icinga-miraheze>	 PROBLEM - mw3 HTTPS on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:24:09] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:24:21] <icinga-miraheze>	 PROBLEM - mw3 Disk Space on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:24:22] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3
[16:24:24] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3
[16:24:33] <icinga-miraheze>	 PROBLEM - mw3 SSH on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:24:41] <icinga-miraheze>	 PROBLEM - mw3 JobChron Service on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:24:57] <icinga-miraheze>	 PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw3
[16:25:04] <icinga-miraheze>	 PROBLEM - mw3 JobRunner Service on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:25:22] <icinga-miraheze>	 PROBLEM - mw3 Current Load on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:25:48] <icinga-miraheze>	 PROBLEM - mw3 php-fpm on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:26:28] <icinga-miraheze>	 RECOVERY - mw3 SSH on mw3 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u5 (protocol 2.0)
[16:26:42] <MacFan4000>	 !log UPDATE mw_permissions SET perm_permissions='["translate-review","move","move-subpages","move-rootuserpages","move-categorypages","movefile","read","edit","createtalk","writeapi","upload","reupload","reupload-shared","minoredit","editmyusercss","editmyuserjson","editmyuserjs","purge","sendemail","applychangetags","changetags","editcontentmodel","requestwiki","user","translate-messagereview","mwoauthmanagemygrants"]' WHERE
[16:26:42] <MacFan4000>	 perm_dbname='metawiki' AND perm_group='user';
[16:26:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:27:04] <icinga-miraheze>	 RECOVERY - mw3 JobRunner Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[16:27:07] <paladox>	 !log repool mw3
[16:27:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:27:17] <icinga-miraheze>	 RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 0.66, 0.17, 0.05
[16:27:18] <icinga-miraheze>	 RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs
[16:27:46] <icinga-miraheze>	 RECOVERY - mw3 php-fpm on mw3 is OK: PROCS OK: 3 processes with command name 'php-fpm7.3'
[16:28:00] <icinga-miraheze>	 RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 22921 bytes in 0.022 second response time
[16:28:02] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures
[16:28:15] <icinga-miraheze>	 RECOVERY - mw3 Disk Space on mw3 is OK: DISK OK - free space: / 59065 MB (76% inode=99%);
[16:28:23] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[16:28:24] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy
[16:28:29] <icinga-miraheze>	 RECOVERY - mw3 JobChron Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobChronService'
[16:28:57] <icinga-miraheze>	 RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy
[16:29:06] <paladox>	 !log upgrade misc1 to debian 9.8
[16:29:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:33:36] <MacFan4000>	 !log UPDATE mw_permissions SET perm_permissions='["createaccount","read","edit","createpage","createtalk","writeapi","viewmywatchlist","editmywatchlist","viewmyprivateinfo","editmyprivateinfo","editmyoptions","abusefilter-log","abusefilter-log-detail","abusefilter-view","autocreateaccount","centralauth-merge","oathauth-enable""]' WHERE perm_dbname='metawiki' AND perm_group='*';
[16:33:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:37:55] <MacFan4000>	 !log deleting permission cache files for metawiki
[16:37:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:40:28] <paladox>	 !log depool mw1
[16:40:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:40:36] <paladox>	 !log upgrade php on mw1 to 7.3.2 (bug fixes) && also dist upgrade to 9.8
[16:40:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:45:17] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:46:10] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:46:13] <icinga-miraheze>	 PROBLEM - mw1 Disk Space on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:46:39] <Not-8d6a>	 [02miraheze/IncidentReporting] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhbBo
[16:46:40] <Not-8d6a>	 [02miraheze/IncidentReporting] 07MacFan4000 03ccd1561 - add missing i18n
[16:47:10] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1
[16:47:10] <icinga-miraheze>	 PROBLEM - mw1 php-fpm on mw1 is CRITICAL: connect to address 185.52.1.75 port 5666: Connection refusedconnect to host 185.52.1.75 port 5666: Connection refused
[16:47:13] <icinga-miraheze>	 PROBLEM - mw1 HTTPS on mw1 is CRITICAL: connect to address 185.52.1.75 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[16:47:15] <icinga-miraheze>	 PROBLEM - mw1 MirahezeRenewSsl on mw1 is CRITICAL: connect to address 185.52.1.75 and port 5000: Connection refused
[16:47:31] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1
[16:47:38] <icinga-miraheze>	 PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1
[16:47:40] <paladox>	 !log repool mw1
[16:47:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:48:08] <icinga-miraheze>	 RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 0.40, 0.11, 0.03
[16:48:12] <icinga-miraheze>	 RECOVERY - mw1 Disk Space on mw1 is OK: DISK OK - free space: / 53005 MB (69% inode=99%);
[16:49:04] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[16:49:05] <icinga-miraheze>	 RECOVERY - mw1 php-fpm on mw1 is OK: PROCS OK: 3 processes with command name 'php-fpm7.3'
[16:49:07] <icinga-miraheze>	 RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 22921 bytes in 0.010 second response time
[16:49:11] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:49:14] <icinga-miraheze>	 RECOVERY - mw1 MirahezeRenewSsl on mw1 is OK: TCP OK - 0.001 second response time on 185.52.1.75 port 5000
[16:49:29] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy
[16:49:38] <icinga-miraheze>	 RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy
[16:51:35] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhbBi
[16:51:36] <Not-8d6a>	 [02miraheze/puppet] 07paladox 0397186b4 - Update nginx.pp
[16:59:46] <Not-8d6a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhbBD
[16:59:47] <Not-8d6a>	 [02miraheze/puppet] 07paladox 0390260d3 - Upgrade php to 7.3 for matomo
[17:00:39] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb
[17:00:55] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1
[17:02:04] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb
[17:02:08] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[17:02:33] <icinga-miraheze>	 PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw2
[17:02:39] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[17:02:55] <icinga-miraheze>	 RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy
[17:03:05] <icinga-miraheze>	 PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures
[17:04:03] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[17:04:06] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy
[17:04:32] <icinga-miraheze>	 RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy
[17:05:05] <icinga-miraheze>	 RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[17:05:41] <icinga-miraheze>	 PROBLEM - misc2 HTTPS on misc2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 372 bytes in 0.008 second response time
[17:09:34] <icinga-miraheze>	 RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 45587 bytes in 0.340 second response time
[17:13:08] <JohnLewis>	 !log UPDATE mw_permissions SET perm_permissions = '["createaccount","read","edit","createpage","createtalk","writeapi","viewmywatchlist","editmywatchlist","viewmyprivateinfo","editmyprivateinfo","editmyoptions","abusefilter-log","abusefilter-log-detail","abusefilter-view","autocreateaccount","centralauth-merge","oathauth-enable"]' WHERE perm_dbname = "metawiki" and perm_group = '*';
[17:13:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:25:47] <icinga-miraheze>	 PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:27:43] <icinga-miraheze>	 RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 07 May 2019 10:15:37 PM GMT +0000.
[17:46:39] <icinga-miraheze>	 PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:48:35] <icinga-miraheze>	 RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Wed 15 May 2019 09:02:45 PM GMT +0000.
[19:00:34] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhb0T
[19:00:35] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox 032efa2a8 - Update VoteNY
[19:01:12] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhb0k
[19:01:13] <Not-8d6a>	 [02miraheze/mediawiki] 07paladox 03294b1b5 - Update Comments
[19:05:09] <Not-8d6a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhb0q
[19:05:11] <Not-8d6a>	 [02miraheze/mw-config] 07paladox 034aa0c99 - Blog Page depends on VoteNY and Comments too so add dependancy on those extensions
[23:03:28] <paladox>	 !log running lc on mw*
[23:03:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master