[00:05:21] <Not-5be1>	 [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhjNO
[00:05:23] <Not-5be1>	 [02miraheze/mw-config] 07Southparkfan 0363ddd0b - CirrusSearch migration for nonsensopediawiki (1/2)
[00:07:36] <Not-5be1>	 [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhjNs
[00:07:37] <Not-5be1>	 [02miraheze/mw-config] 07Southparkfan 034840671 - Make sitenotice less wiki specific
[00:10:07] <Not-5be1>	 [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhjNZ
[00:10:08] <Not-5be1>	 [02miraheze/mw-config] 07Southparkfan 039c38718 - Prepare nonsensopediawiki for CirrusSearch migration
[00:12:04] <Not-5be1>	 [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhjNl
[00:12:06] <Not-5be1>	 [02miraheze/mw-config] 07Southparkfan 034c08518 - Enable search updates again on nonsensopediawiki
[01:24:59] <Not-5be1>	 [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS
[01:25:01] <Not-5be1>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fhjA4
[01:25:02] <Not-5be1>	 [02miraheze/puppet] 07paladox 032a54049 - php: set pcre.backtrack_limit to 5000000 for cli
[01:25:04] <Not-5be1>	 [02puppet] 07paladox opened pull request 03#989: php: set pcre.backtrack_limit to 5000000 for cli - 13https://git.io/fhjAB
[01:26:56] <Not-5be1>	 [02puppet] 07Southparkfan closed pull request 03#989: php: set pcre.backtrack_limit to 5000000 for cli - 13https://git.io/fhjAB
[01:26:57] <Not-5be1>	 [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhjAR
[01:26:59] <Not-5be1>	 [02miraheze/puppet] 07paladox 03519d65c - php: set pcre.backtrack_limit to 5000000 for cli (#989)
[02:37:41] <icinga-miraheze>	 PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[02:38:24] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb
[02:39:26] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb
[02:42:46] <icinga-miraheze>	 PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 53%
[02:43:10] <icinga-miraheze>	 PROBLEM - cp5 HTTPS on cp5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[02:46:50] <icinga-miraheze>	 PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[02:48:47] <icinga-miraheze>	 PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 57%
[02:49:19] <icinga-miraheze>	 RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/1.1 200 OK - 23753 bytes in 1.599 second response time
[02:49:26] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[02:49:40] <icinga-miraheze>	 RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy
[02:50:24] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[02:50:44] <icinga-miraheze>	 RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 0%
[10:30:53] <Not-5be1>	 [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS
[10:30:54] <Not-5be1>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1
[12:30:15] <Not-5be1>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjeUL
[12:30:16] <Not-5be1>	 [02miraheze/services] 07MirahezeSSLBot 0314835fb - BOT: Updating services config for wikis
[13:11:46] <PuppyKun>	 !log changed the misc1 shell passwd for ndkilla (root@misc1:/home/ndkilla# passwd ndkilla)
[13:11:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:12:29] <Zppix>	 PuppyKun:  you forgot to add what it was changed too :P Jk
[13:13:21] <PuppyKun>	 :p funny
[15:02:46] <icinga-miraheze>	 PROBLEM - misc2 Current Load on misc2 is CRITICAL: CRITICAL - load average: 2.17, 1.71, 1.16
[15:04:47] <icinga-miraheze>	 RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 1.26, 1.55, 1.16
[15:11:40] <icinga-miraheze>	 PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 1.75, 1.77, 1.40
[15:13:37] <icinga-miraheze>	 RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 0.65, 1.38, 1.30
[15:30:38] <Not-5be1>	 [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjeL5
[15:30:40] <Not-5be1>	 [02miraheze/mw-config] 07Southparkfan 03ab66099 - Switch search on nonsensopediawiki to ES
[15:31:37] <Not-5be1>	 [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjeLF
[15:31:39] <Not-5be1>	 [02miraheze/mw-config] 07Southparkfan 032c3ac44 - Disable sitenotice
[15:35:22] <SPF|Cloud>	 !log drop searchindex table for nonsensopediawiki
[15:35:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:36:45] <icinga-miraheze>	 PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 48459 MB (13% inode=96%);
[16:38:06] <PuppyKun>	 MacFan4000: revi: I've disabled several global rules that (could) add subscribers to security tasks because global rules bypass policy controls. If you want to be subscribed to objects you can see, please create personal rules.
[16:43:31] <MacFan4000>	 I mean the one for my user project probably could have stayed, since I am staff anyways
[16:44:20] <PuppyKun>	 That could change in the future and I don't think "review herald rules for security" is part of off-boarding
[16:48:31] <SPF|Cloud>	 !log disabling puppet on mw1
[16:48:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:51:36] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 6 minutes ago with 0 failures
[16:52:31] <revi>	 PuppyKun: it doesn't let me see it though
[16:52:44] <revi>	 I actually don't care, just noting that
[16:53:02] <revi>	 oh if it is in S1 it would
[16:53:10] <revi>	 whatever I don't care
[16:53:18] <revi>	 then seeya
[16:53:33] <PuppyKun>	 My bad revi I re enabled the rule
[16:53:49] <PuppyKun>	 It apparently only adds subscribers to tasks visible *before the rule fires*
[17:06:26] <Not-5be1>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjemM
[17:06:27] <Not-5be1>	 [02miraheze/mw-config] 07paladox 032bbab85 - Enable CirrusSearch && disable search updates for allthetropeswiki
[17:08:10] <Not-5be1>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjemD
[17:08:11] <Not-5be1>	 [02miraheze/mw-config] 07paladox 03f716529 - Set sitenotice
[17:10:21] <Not-5be1>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjem7
[17:10:22] <Not-5be1>	 [02miraheze/mw-config] 07paladox 03e0fd33e - Fix wrong db for site notice Change-Id: I5da03d227682600a35e3b14a45c54790b85ef31d
[17:12:02] <paladox>	 !log generate elasticsearch index for allthetropeswiki
[17:12:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:13:42] <Not-5be1>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjemj
[17:13:44] <Not-5be1>	 [02miraheze/mw-config] 07paladox 0346a2298 - Re enable search updates for allthetropeswiki
[17:14:43] <paladox>	 !log indexing allthetropeswiki for elasticsearch
[17:14:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:26:43] <icinga-miraheze>	 PROBLEM - misc2 HTTPS on misc2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:27:26] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb
[17:28:40] <icinga-miraheze>	 RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 45587 bytes in 0.198 second response time
[17:29:26] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[17:30:04] <SPF|Cloud>	 !log disable puppet on cp4
[17:30:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[17:32:28] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 9 minutes ago with 0 failures
[17:38:24] <SPF|Cloud>	 !log depool mw1
[17:38:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:19:36] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[18:35:47] <SPF|Cloud>	 !log repool mw1 and enable puppet on mw1/cp4
[18:35:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:36:28] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[20:45:17] <Not-5be1>	 [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjes6
[20:45:19] <Not-5be1>	 [02miraheze/dns] 07paladox 030f5c7fa - Replace cp5 with cp3
[20:45:20] <Not-5be1>	 [02dns] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbQXl
[20:45:23] <Not-5be1>	 [02dns] 07paladox opened pull request 03#107: Replace cp5 with cp3 - 13https://git.io/fjesi
[21:33:05] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file
[22:07:15] <Not-5be1>	 [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjeZc
[22:07:17] <Not-5be1>	 [02miraheze/puppet] 07JohnFLewis 034b576de - add cp3 to puppet
[22:07:18] <Not-5be1>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjeZC
[22:07:20] <Not-5be1>	 [02miraheze/puppet] 07paladox 03d91f0bf - base::puppet: Fix apt::source key and server
[22:09:37] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[22:27:37] <Not-5be1>	 [02miraheze/puppet] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fjeZ5
[22:27:38] <Not-5be1>	 [02miraheze/puppet] 07JohnFLewis 03ad6dd5c - add ufw for cp3
[22:27:40] <Not-5be1>	 [02miraheze/puppet] 07JohnFLewis 035be3d40 - Merge branch 'master' of www.github.com:miraheze/puppet
[22:39:40] <Not-5be1>	 [02puppet] 07MacFan4000 opened pull request 03#990: Monitoring: +cp3 -cp5 - 13https://git.io/fjenY
[22:39:49] <Not-5be1>	 [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjenO
[22:39:50] <Not-5be1>	 [02miraheze/puppet] 07JohnFLewis 03dcb6b0a - cp3
[22:40:02] <Not-5be1>	 [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjen3
[22:40:04] <Not-5be1>	 [02miraheze/mw-config] 07JohnFLewis 03c6179df - cp3
[22:40:32] <Not-5be1>	 [02puppet] 07JohnFLewis closed pull request 03#990: Monitoring: +cp3 -cp5 - 13https://git.io/fjenY
[22:40:57] <Not-5be1>	 [02dns] 07JohnFLewis closed pull request 03#107: Replace cp5 with cp3 - 13https://git.io/fjesi
[22:41:17] <Not-5be1>	 [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbQXl
[22:41:18] <Not-5be1>	 [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1
[22:43:25] <Not-5be1>	 [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjenZ
[22:43:27] <Not-5be1>	 [02miraheze/puppet] 07JohnFLewis 034ad3e3b - add cp3 to mon groups
[22:43:56] <Not-5be1>	 [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjenW
[22:43:58] <Not-5be1>	 [02miraheze/dns] 07JohnFLewis 0361889f9 - cp5->cp3
[22:53:40] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::f001/cpweb
[22:55:21] <Not-5be1>	 [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjeni
[22:55:23] <Not-5be1>	 [02miraheze/dns] 07JohnFLewis 032904f76 - fix IPv6 for cp3
[23:03:40] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online