[00:11:16] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 5.20, 3.68, 2.32 [00:15:15] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.15, 1.13, 1.61 [00:19:13] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 0.91, 2.11, 1.92 [00:21:14] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.34, 1.08, 1.54 [02:19:25] SantaRhino: I’m currently mobile [02:21:30] [02puppet] 07paladox reviewed pull request 03#1167 commit - 13https://git.io/JebTm [03:04:04] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [03:12:03] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:29:49] PROBLEM - wiki.apap04.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.apap04.com' expires in 15 day(s) (Fri 10 Jan 2020 06:26:34 AM GMT +0000). [06:30:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JebLr [06:30:04] [02miraheze/ssl] 07MirahezeSSLBot 03c6ad668 - Bot: Update SSL cert for wiki.apap04.com [06:35:47] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 21979 MB (5% inode=95%); [06:43:38] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown): Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [06:43:48] RECOVERY - wiki.apap04.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.apap04.com' will expire on Tue 24 Mar 2020 05:29:56 AM GMT +0000. [06:51:39] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:14:20] PROBLEM - oecumene.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'oecumene.org' expires in 15 day(s) (Fri 10 Jan 2020 07:10:49 AM GMT +0000). [07:14:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jebtq [07:14:35] [02miraheze/ssl] 07MirahezeSSLBot 03143f41e - Bot: Update SSL cert for oecumene.org [07:22:20] RECOVERY - oecumene.org - LetsEncrypt on sslhost is OK: OK - Certificate 'oecumene.org' will expire on Tue 24 Mar 2020 06:14:27 AM GMT +0000. [08:03:03] [02puppet] 07RhinosF1 reviewed pull request 03#1167 commit - 13https://git.io/JebtR [08:35:37] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members],Exec[puppet-users_ensure_members] [08:43:37] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:38:34] [02puppet] 07RhinosF1 synchronize pull request 03#1167: Update with new role titles - 13https://git.io/JeFhg [09:39:08] [02puppet] 07RhinosF1 commented on pull request 03#1167: Update with new role titles - 13https://git.io/Jebqu [09:40:15] paladox: ^ changed to tech@ [11:10:25] PROBLEM - www.mh142.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mh142.com' expires in 15 day(s) (Fri 10 Jan 2020 11:06:58 AM GMT +0000). [11:10:26] PROBLEM - mh142.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mh142.com' expires in 15 day(s) (Fri 10 Jan 2020 11:06:58 AM GMT +0000). [11:10:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jebmu [11:10:41] [02miraheze/ssl] 07MirahezeSSLBot 036fde1cb - Bot: Update SSL cert for mh142.com [11:22:23] RECOVERY - www.mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Tue 24 Mar 2020 10:10:34 AM GMT +0000. [11:22:26] RECOVERY - mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Tue 24 Mar 2020 10:10:34 AM GMT +0000. [11:38:11] MERRY CHRISTMAS [11:57:29] Merry Christmas! :) [12:07:26] Reception123, Zppix: if you see a steward can you get an import that was sent to them and look into it. [12:07:37] Reception123: would it be on misc1 somewhere [12:08:40] Ok [12:15:19] SantaRhino: if its on misc1 there aint a damn thing i can do about it [12:30:41] Zppix: until ot’s forwarded [12:30:47] Poke a stew if you see one [13:02:24] PROBLEM - wiki.ciptamedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ciptamedia.org' expires in 15 day(s) (Fri 10 Jan 2020 01:00:21 PM GMT +0000). [13:02:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JebOm [13:02:39] [02miraheze/ssl] 07MirahezeSSLBot 031a2d514 - Bot: Update SSL cert for wiki.ciptamedia.org [13:05:20] PROBLEM - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.gesamtschule-nordkirchen.de' expires in 15 day(s) (Fri 10 Jan 2020 01:02:03 PM GMT +0000). [13:05:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JebOB [13:05:35] [02miraheze/ssl] 07MirahezeSSLBot 03994ad17 - Bot: Update SSL cert for wiki.gesamtschule-nordkirchen.de [13:09:47] PROBLEM - adadevelopersacademy.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'adadevelopersacademy.wiki' expires in 15 day(s) (Fri 10 Jan 2020 01:06:00 PM GMT +0000). [13:10:01] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JebOr [13:10:02] [02miraheze/ssl] 07MirahezeSSLBot 03c50c4b2 - Bot: Update SSL cert for adadevelopersacademy.wiki [13:12:25] RECOVERY - wiki.ciptamedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ciptamedia.org' will expire on Tue 24 Mar 2020 12:02:31 PM GMT +0000. [13:13:20] RECOVERY - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gesamtschule-nordkirchen.de' will expire on Tue 24 Mar 2020 12:05:28 PM GMT +0000. [13:21:47] paladox, Zppix: any reason why back button in gerrit doesn’t work? [13:22:31] If i come from WM Phab to Gerrit then press back, no matter how much i press it, I stay on same gerrit screen [13:23:44] RECOVERY - adadevelopersacademy.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'adadevelopersacademy.wiki' will expire on Tue 24 Mar 2020 12:09:54 PM GMT +0000. [13:23:48] because it uses url params on almost every click SantaRhino [13:29:15] GWTUI is just well... what zppix says, try PolyGerrit. [13:29:48] paladox: ok [13:30:53] paladox: yep, New UI works perfect [13:33:56] * SantaRhino is slowly learning to speak Gertiy [13:34:24] s/gertiy/gerrit [13:34:40] Zppix: ^ spelling module is dead [13:34:45] Merry Christmas Everyone [13:34:48] s/dead/not working [13:34:48] SantaRhino meant to say: Zppix: ^ spelling module is not working [13:34:53] ol [13:34:55] lol [13:34:55] Oh no just action msg [13:35:09] * SantaRhino adds it to the fix list [13:35:20] Merry Christmas TreesideMiners [13:37:15] Zppix: are you okay with the 2020 plans [13:38:43] Fix for what i just noticed is in next update [13:40:59] SantaRhino: lgtm [13:53:45] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 6.43, 2.45, 1.24 [13:55:43] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.87, 1.36, 1.06 [14:02:00] PROBLEM - cp4 Varnish Backends on cp4 is UNKNOWN: NRPE: Unable to read output [14:05:58] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [14:06:50] PROBLEM - cp4 Puppet on cp4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [14:07:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jeb3r [14:07:02] [02miraheze/puppet] 07paladox 0388f94e8 - Add support for varnish 6 [14:07:03] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS [14:07:05] [02puppet] 07paladox opened pull request 03#1168: Add support for varnish 6 - 13https://git.io/Jeb3o [14:07:35] [02puppet] 07paladox closed pull request 03#1168: Add support for varnish 6 - 13https://git.io/Jeb3o [14:07:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeb3K [14:07:38] [02miraheze/puppet] 07paladox 037cfd62c - Add support for varnish 6 (#1168) [14:07:59] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [14:08:51] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [14:14:32] paladox: on wmf phab how do i change a task visibility? [14:15:23] Zppix i doin't think you can change it to public, but i thinks there's a "Protect as a security issue" button. [14:15:35] Since you need the perms to make it public [14:37:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeb3A [14:37:48] [02miraheze/puppet] 07paladox 03fa95946 - redis: Add stop-writes-on-bgsave-error (setting it to no and also disabling it) [14:40:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeb3j [14:40:09] [02miraheze/services] 07MirahezeSSLBot 039a556dd - BOT: Updating services config for wikis [17:22:01] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3 [17:23:59] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [18:58:44] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:59:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [19:00:29] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [19:00:39] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18598 bytes in 0.274 second response time [19:01:30] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:02:29] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:51:40] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 42% [20:54:53] paladox ^ [20:57:38] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 21% [21:09:48] Zppix seems to have recovered. [21:33:57] Hello Guest61581! If you have any questions, feel free to ask and someone should answer soon. [21:39:04] Hello paladoxs! If you have any questions, feel free to ask and someone should answer soon. [21:54:22] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 44% [21:55:42] paladox_: ^ [21:56:38] [02puppet] 07JohnFLewis closed pull request 03#1159: Add owen to the donate alias - 13https://git.io/Jey8Z [21:56:50] ok [21:57:49] seems to be working for me [21:58:00] [02puppet] 07JohnFLewis closed pull request 03#1167: Update with new role titles - 13https://git.io/JeFhg [21:58:02] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeblY [21:58:03] [02miraheze/puppet] 07RhinosF1 036d33787 - Update with new role titles (#1167) * Update with new role titles T5016 Ensure full consensus and check any config for maniphest * sysadmins -> tech [21:58:13] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 30% [22:02:25] JohnLewis: thx [22:02:40] yw [22:05:09] JohnLewis: per task, will update any links on meta when i’m at a charger. Might need someone to use the run jobs script though a few times [22:21:39] hm? [22:24:09] JohnLewis: replace text uses the JobQueue as far as I know but i don’t know how large the job will be to update all refreneces to old emails in help guides. [22:24:35] Can you make sure the phabricator emails still work? [22:24:51] We haven't use phab emails in months [22:26:08] JohnLewis: config is still in aliases [22:26:29] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 40% [22:26:30] Although it goes to sre@ [22:26:36] which isn't an issue as we used them in the past, and one is a very important email [22:26:51] Okay [22:27:02] As long as it all still works [22:27:43] Or works as it should [22:28:24] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 35% [22:44:31] JohnLewis, paladox: can you run jobs on meta [22:46:39] !log running jobs for meta on mw1 [22:46:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:47:32] JohnLewis: ping me when it’s ran [22:47:41] (Watching api as well) [23:00:27] JohnLewis: could run jobs go much slower? —run-faster not an option yet? [23:00:46] there's a fatal [23:04:17] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [23:04:44] JohnLewis: what’s the fatal say? [23:05:40] paladox is resolving it, 1.34 merge issue [23:05:51] Okay [23:06:18] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:06:23] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jeb8G [23:06:25] [02miraheze/mediawiki] 07paladox 03499ea73 - Fix merge issue [23:11:29] JohnLewis: https://meta.miraheze.org/w/index.php?title=Community_noticeboard&diff=92670&oldid=92614 [23:11:47] [ Difference between revisions of "Community noticeboard" - Miraheze Meta ] - meta.miraheze.org [23:11:57] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jeb8Z [23:11:58] [02miraheze/mediawiki] 07paladox 0321ced43 - Fix tabbing and mark custom changes [23:12:00] [02mediawiki] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbL5b [23:12:02] [02mediawiki] 07paladox opened pull request 03#131: Fix tabbing and mark custom changes - 13https://git.io/Jeb8n [23:12:30] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:12:31] okay [23:13:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:13:52] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [23:14:28] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:14:39] [02mediawiki] 07paladox closed pull request 03#131: Fix tabbing and mark custom changes - 13https://git.io/Jeb8n [23:14:40] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_34 [+0/-0/±2] 13https://git.io/Jeb8C [23:14:42] [02miraheze/mediawiki] 07paladox 035189cc9 - Merge pull request #131 from miraheze/paladox-patch-1 Fix tabbing and mark custom changes [23:14:43] [02miraheze/mediawiki] 07paladox deleted branch 03paladox-patch-1 [23:14:45] [02mediawiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbL5b [23:15:15] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:18:33] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:23:28] JohnLewis: when the JQ clears can you remark everything for translation and clean JQ up again [23:23:37] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:23:38] * SantaRhino might be asleep soon [23:23:56] Will do [23:24:52] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:26:02] JohnLewis: thanks and happy christmas [23:26:13] Did you do anything nice? [23:26:25] Saw family, that was all [23:26:56] Same [23:27:16] Surprisngly not a drunk family this year [23:30:12] !log restart jobchron on lizardfs6 [23:30:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:34:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeb8w [23:34:11] [02miraheze/puppet] 07paladox 0328ba3a3 - Revert "Jobrunner: Use nutcracker" This reverts commit c932457727ce356920d95ef1706391f5c766d242. [23:40:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeb8X [23:40:13] [02miraheze/puppet] 07paladox 038e13755 - jobrunner: Do not run webVideoTranscodePrioritized jobs [23:50:13] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jeb8D [23:50:15] [02miraheze/mw-config] 07paladox 03e3f45f3 - Set wgFFmpegLocation to /usr/bin/ffmpeg [23:50:16] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [23:50:18] [02mw-config] 07paladox opened pull request 03#2840: Set wgFFmpegLocation to /usr/bin/ffmpeg - 13https://git.io/Jeb8y [23:50:57] [02mw-config] 07paladox closed pull request 03#2840: Set wgFFmpegLocation to /usr/bin/ffmpeg - 13https://git.io/Jeb8y [23:50:59] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeb8S [23:51:00] [02miraheze/mw-config] 07paladox 03a0555a1 - Set wgFFmpegLocation to /usr/bin/ffmpeg (#2840) [23:51:02] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [23:51:03] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [23:51:36] miraheze/mw-config/paladox-patch-1/e3f45f3 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/629502679 [23:51:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeb89 [23:51:38] [02miraheze/puppet] 07paladox 037f8c28d - jobrunner: Allow webVideoTranscode and webVideoTranscodePrioritized to be processed again [23:53:37] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22]