[00:06:07] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [00:06:53] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [00:07:32] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 1 files, 0B, 2019-02-10 00:05:00 (1.1 weeks ago) [00:08:08] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:12:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [00:16:48] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:18:08] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:02:50] Hello Void-bot2! If you have any questions feel free to ask and someone should answer soon. [03:32:15] sorry for flooding a bit with the joins and quits :) [03:37:29] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:39:25] RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 07 May 2019 10:15:37 PM GMT +0000. [04:04:56] for whoever's interested, Void-bot2 will be replacing Void-bot sometime tomorrow [05:05:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fh57b [05:05:11] [02miraheze/services] 07MirahezeSSLBot 03b403029 - BOT: Updating services config for wikis [05:15:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fh55J [05:15:13] [02miraheze/services] 07MirahezeSSLBot 03534000b - BOT: Updating services config for wikis [05:35:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fh55c [05:35:12] [02miraheze/services] 07MirahezeSSLBot 03e890686 - BOT: Updating services config for wikis [08:52:15] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 386 [08:54:15] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [09:00:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fh5bH [09:00:15] [02miraheze/services] 07MirahezeSSLBot 031e089c6 - BOT: Updating services config for wikis [09:10:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fh5Nk [09:10:12] [02miraheze/services] 07MirahezeSSLBot 03253cbd3 - BOT: Updating services config for wikis [13:51:03] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhdeD [13:51:05] [02miraheze/CreateWiki] 07paladox 0311c08ad - Fix undefined wgCreateWikiUseEchoNotifications variable [13:52:18] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhdeS [13:52:19] [02miraheze/mediawiki] 07paladox 039edf648 - Update CW [13:56:07] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhded [13:56:09] [02miraheze/mw-config] 07paladox 0356b15c1 - ManageWiki: Add missing sql fro ApprovedRevs [13:57:51] miraheze/mw-config/master/56b15c1 - paladox The build was broken. https://travis-ci.org/miraheze/mw-config/builds/494904082 [13:59:28] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhdvk [13:59:29] [02miraheze/mw-config] 07paladox 03e107d69 - Fix syntax [13:59:37] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 2 backends are down. mw1 mw2 [14:00:21] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [14:00:58] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [14:01:37] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [14:02:21] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [14:02:58] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [14:05:20] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhdvG [14:05:21] [02miraheze/mw-config] 07paladox 0300d698a - Wikibase: Set siteLinkGroups and specialSiteLinkGroups to a empty array [16:05:11] [02mw-config] 07JohnFLewis commented on commit 0300d698a15969d98b6bdd9c6aa5149d4f579d40d2 - 13https://git.io/fhdUA [16:06:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhdUx [16:06:17] [02miraheze/mw-config] 07paladox 0355d066a - Remove duplicate siteLinkGroups configs [16:06:37] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhdUp [16:06:38] [02miraheze/mw-config] 07paladox 03eb46b4d - Update Wikibase.php [16:06:59] [02mw-config] 07paladox commented on commit 0300d698a15969d98b6bdd9c6aa5149d4f579d40d2 - 13https://git.io/fhdTe [16:38:15] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 400 [16:40:15] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [17:00:15] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 742 [17:46:15] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [18:00:04] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 337 [18:02:00] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [19:25:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhdtC [19:25:12] [02miraheze/services] 07MirahezeSSLBot 03ee6d60e - BOT: Updating services config for wikis [19:34:58] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fhdtg [19:34:59] [02miraheze/mw-config] 07paladox 035f44c55 - Add WikibaseClient and enable it for ffgxvwiki [19:35:00] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [19:35:02] [02mw-config] 07paladox opened pull request 03#2632: Add WikibaseClient and enable it for ffgxvwiki - 13https://git.io/fhdt2 [19:35:27] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhdta [19:35:28] [02miraheze/mw-config] 07JohnFLewis 03787467d - add service providers as incident component [19:36:11] [02mw-config] 07JohnFLewis reviewed pull request 03#2632 commit - 13https://git.io/fhdtV [19:37:02] miraheze/mw-config/paladox-patch-2/5f44c55 - paladox The build was fixed. https://travis-ci.org/miraheze/mw-config/builds/495073669 [19:37:19] [02mw-config] 07paladox reviewed pull request 03#2632 commit - 13https://git.io/fhdtw [19:49:00] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fhdt7 [19:49:02] [02miraheze/mw-config] 07paladox 03017ee21 - Update LocalExtensions.php [19:49:03] [02mw-config] 07paladox synchronize pull request 03#2632: Add WikibaseClient and enable it for ffgxvwiki - 13https://git.io/fhdt2 [19:49:30] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fhdt5 [19:49:32] [02miraheze/mw-config] 07paladox 0350a3591 - Update LocalSettings.php [19:49:33] [02mw-config] 07paladox synchronize pull request 03#2632: Add WikibaseClient and enable it for ffgxvwiki - 13https://git.io/fhdt2 [19:51:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fhdtF [19:51:17] [02miraheze/mw-config] 07paladox 038dac519 - Update ManageWiki.php [19:51:19] [02mw-config] 07paladox synchronize pull request 03#2632: Add WikibaseClient and enable it for ffgxvwiki - 13https://git.io/fhdt2 [19:53:19] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fhdtA [19:53:21] [02miraheze/mw-config] 07paladox 033e65f34 - Update extension-list [19:53:22] [02mw-config] 07paladox synchronize pull request 03#2632: Add WikibaseClient and enable it for ffgxvwiki - 13https://git.io/fhdt2 [19:54:17] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [19:55:22] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [20:03:19] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fhdqY [20:03:20] [02miraheze/mw-config] 07paladox 031ec5ded - Update Wikibase.php [20:03:22] [02mw-config] 07paladox synchronize pull request 03#2632: Add WikibaseClient and enable it for ffgxvwiki - 13https://git.io/fhdt2 [20:03:37] [02mw-config] 07paladox closed pull request 03#2627: change list in siteLinkGroups to 'miraheze' - 13https://git.io/fhygx [20:03:40] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [20:03:41] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [20:05:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhdqO [20:05:13] [02miraheze/services] 07MirahezeSSLBot 03808d2c2 - BOT: Updating services config for wikis [20:05:32] [02mw-config] 07paladox closed pull request 03#2632: Add WikibaseClient and enable it for ffgxvwiki - 13https://git.io/fhdt2 [20:05:33] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/fhdq3 [20:05:35] [02miraheze/mw-config] 07paladox 0397cb3ad - Add WikibaseClient and enable it for ffgxvwiki (#2632) * Add WikibaseClient and enable it for ffgxvwiki * Update LocalExtensions.php * Update LocalSettings.php * Update ManageWiki.php * Update extension-list * Update Wikibase.php [20:07:47] !log running lc on mw* [20:07:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:10:40] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 41922 MB (11% inode=95%); [20:12:10] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [20:13:21] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:13:58] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [20:13:59] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [20:16:55] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhdq2 [20:16:56] [02miraheze/MirahezeMagic] 07paladox 03a1c9365 - Update populateWikibaseSitesTable.php [20:17:50] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhdqo [20:17:51] [02miraheze/mediawiki] 07paladox 0324e1ae0 - Update MM [20:20:15] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhdqP [20:20:17] [02miraheze/mw-config] 07paladox 03d3dc3ed - Set repoNamespaces [21:19:39] PROBLEM - mai.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mai.gyaanipedia.co.in' expires in 15 day(s) (Wed 06 Mar 2019 09:17:12 PM GMT +0000). [21:19:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhdmi [21:19:56] [02miraheze/ssl] 07MirahezeSSLBot 03b7890ce - Bot: Update SSL cert for mai.gyaanipedia.co.in [21:23:39] RECOVERY - mai.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'mai.gyaanipedia.co.in' will expire on Sun 19 May 2019 08:19:48 PM GMT +0000. [21:54:15] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 729 [21:58:15] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [22:04:15] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 353 [22:06:15] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [23:33:55] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:57:10] [02miraheze/IncidentReporting] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fhd3X [23:57:12] [02miraheze/IncidentReporting] 07JohnFLewis 035bf0ccd - reduce number of / 60 so we work in mins not hours [23:57:47] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fhd31 [23:57:49] [02miraheze/mediawiki] 07JohnFLewis 0339a2a95 - update IR