[01:55:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvLlM
[01:55:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03f26efe8 - BOT: Updating services config for wikis
[05:46:57] <icinga-miraheze>	 PROBLEM - db5 Disk Space on db5 is WARNING: DISK WARNING - free space: / 20070 MB (10% inode=99%);
[06:26:51] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2998 MB (12% inode=94%);
[06:52:07] <Reception123>	 !log reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki WageUglyDollandMarlaPlaymobil12 "Just Wage" --logwiki=metawiki
[06:52:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[06:52:26] <RhinosF1>	 Morning
[06:52:59] <RhinosF1>	 I thought we added a deleted.dblist to avoid that Reception123
[06:57:29] <Reception123>	 RhinosF1: hi. I was told it was fixed too but it seems to still not be
[06:58:40] <RhinosF1>	 paladox: when you wake up ^
[06:58:46] <RhinosF1>	 Reception123: so I saw
[07:01:11] <RhinosF1>	 Reception123: I’m going to busy a bit tonight but if you want keep updating the stateof wikis stats pages
[07:01:47] <RhinosF1>	 Especially top 10 wikis
[07:01:53] <Reception123>	 I'll also be busy tonight unfortunately
[07:04:14] <RhinosF1>	 Reception123: ok, I’ll do wiki counts but might have to wait to top10wikis
[08:45:41] <Not-5a0a>	 [02mw-config] 07HeartsDo-Dev opened pull request 03#2866: Make WikiForum disable for anon user on allthetropeswiki per T5140 - 13https://git.io/JvLuo
[08:46:00] <travis-ci>	 HeartsDo-Dev/mw-config/patch-2/69fe6ce - HeartsDo The build passed. https://travis-ci.com/HeartsDo-Dev/mw-config/builds/145504123
[11:13:49] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 3.31, 2.39, 1.47
[11:15:43] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.66, 1.71, 1.43
[11:17:37] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.28, 0.95, 1.18
[11:52:36] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%);
[12:32:38] <paladox>	 Well we added it to fix the undefined index errors, no gurrentee that would fix it. (That’s what we thought broke it)
[14:49:15] <icinga-miraheze>	 PROBLEM - wiki.mastodon.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mastodon.kr' expires in 15 day(s) (Fri 07 Feb 2020 02:45:20 PM GMT +0000).
[14:49:32] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvL6n
[14:49:33] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot 0354cb6a4 - Bot: Update SSL cert for wiki.mastodon.kr
[15:03:13] <icinga-miraheze>	 RECOVERY - wiki.mastodon.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mastodon.kr' will expire on Tue 21 Apr 2020 01:49:25 PM GMT +0000.
[16:22:33] <Not-5a0a>	 [02mw-config] 07HeartsDo-Dev closed pull request 03#2866: Make WikiForum disable for anon user on allthetropeswiki per T5140 - 13https://git.io/JvLuo
[16:27:33] <Not-5a0a>	 [02mw-config] 07paladox commented on pull request 03#2866: Make WikiForum disable for anon user on allthetropeswiki per T5140 - 13https://git.io/JvLXy
[16:29:50] <Not-5a0a>	 [02mw-config] 07paladox edited a comment on pull request 03#2866: Make WikiForum disable for anon user on allthetropeswiki per T5140 - 13https://git.io/JvLXy
[17:21:58] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[17:22:14] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[17:22:31] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[17:23:13] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[17:23:22] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw3
[17:24:09] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15296 bytes in 9.021 second response time
[17:24:14] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 2.479 second response time
[17:24:38] <icinga-miraheze>	 RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15288 bytes in 8.075 second response time
[17:25:13] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy
[17:25:22] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy
[18:30:56] <paladox>	 !log root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=spectrobeswiki --report=1 spectrobes_pages_full.xml.7z - per request through email
[18:31:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:31:29] <paladox>	 !log root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=spectrobeswiki --report=1 spectrobes_pages_full.xml - per request through email
[18:31:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:43:22] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.89, 1.75, 1.41
[18:45:22] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.49, 1.08, 1.21
[19:04:23] <RhinosF1>	 paladox: is electron still duff for private wikis?
[19:08:01] <paladox>	 yes
[19:12:22] <RhinosF1>	 paladox: thx
[19:20:53] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvL9Y
[19:20:55] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 0309fbca0 - Blacklist oauth-disable-for-user
[19:26:04] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvL90
[19:26:06] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 0303188e2 - fix typo
[19:29:25] <paladox_UK_IN_EU>	 !log remove oauthauth-disable-for-user from all wikis
[19:29:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:43:23] <icinga-miraheze>	 PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 0.72, 2.07, 1.66
[19:45:23] <icinga-miraheze>	 RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.30, 1.19, 1.40
[20:45:13] <icinga-miraheze>	 PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 47%
[20:47:14] <icinga-miraheze>	 RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 17%
[20:48:57] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 87%
[20:59:00] <icinga-miraheze>	 RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 3%
[21:01:51] <Zppix>	 paladox:  whats up with the alerts?
[21:02:15] <paladox>	 good question
[21:10:33] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvL5O
[21:10:35] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 03b203e60 - Blacklist oathauth-view-log
[21:11:24] <RhinosF1>	 paladox: what’s that one do?
[21:13:26] <paladox>	 https://github.com/wikimedia/mediawiki-extensions-OATHAuth/commit/808df6dc551ddc5e788f41dd546950f379142636
[21:13:29] <ZppixBot>	 [ Add private logging when user disables 2fa for someone else · wikimedia/mediawiki-extensions-OATHAuth@808df6d · GitHub ] - github.com
[21:14:26] <RhinosF1>	 paladox: ah
[22:38:08] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvLFx
[22:38:09] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 0367686f4 - Fix typo
[22:42:49] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[22:49:44] <hispano76>	 Hello.
[22:49:53] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvLb3
[22:49:54] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 03c7763ec - Blacklist stopforumspam
[22:51:32] <paladox>	 !log remove oathauth-disable-for-user from all wikis
[22:51:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:53:47] <icinga-miraheze>	 PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.66, 1.95, 0.90
[23:54:31] <paladox>	 !log apt-get upgrade cp3
[23:54:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:55:47] <icinga-miraheze>	 RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 0.93, 1.53, 0.88
[23:59:49] <Not-5a0a>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvLAL
[23:59:51] <Not-5a0a>	 [02miraheze/mw-config] 07paladox 03bd7d755 - Move translates sql stuff to ManageWikiExtensions We doin't need translate schema to be created on every wiki creation, only when the extension is created.
[23:59:52] <Not-5a0a>	 [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3
[23:59:54] <Not-5a0a>	 [02mw-config] 07paladox opened pull request 03#2867: Move translates sql stuff to ManageWikiExtensions - 13https://git.io/JvLAq