[00:00:33] PROBLEM - lizardfs6 GlusterFS port 49152 on lizardfs6 is CRITICAL: connect to address 54.36.165.161 and port 49152: Connection refused [00:05:31] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static-new] [00:06:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JewIa [00:06:58] [02miraheze/puppet] 07paladox 0374668a7 - Update secure-access [00:06:59] [02puppet] 07paladox synchronize pull request 03#1144: gluster: Add secure-access file - 13https://git.io/JewIC [00:08:27] RECOVERY - lizardfs6 GlusterFS port 49152 on lizardfs6 is OK: TCP OK - 0.013 second response time on 54.36.165.161 port 49152 [00:13:37] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:15:51] [02puppet] 07paladox synchronize pull request 03#1144: gluster: Add secure-access file - 13https://git.io/JewIC [00:15:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JewI1 [00:15:54] [02miraheze/puppet] 07paladox 03b7baccb - Update secure-access [00:20:23] PROBLEM - lizardfs6 GlusterFS port 49152 on lizardfs6 is CRITICAL: connect to address 54.36.165.161 and port 49152: Connection refused [00:22:54] PROBLEM - lizardfs6 Puppet on lizardfs6 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [00:24:54] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:25:45] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static-new] [00:26:23] RECOVERY - lizardfs6 GlusterFS port 49152 on lizardfs6 is OK: TCP OK - 0.012 second response time on 54.36.165.161 port 49152 [00:33:51] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:01:35] PROBLEM - public.revi.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'public.revi.wiki' expires in 15 day(s) (Wed 27 Nov 2019 12:59:13 AM GMT +0000). [01:01:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewLt [01:01:50] [02miraheze/ssl] 07MirahezeSSLBot 0391c695f - Bot: Update SSL cert for public.revi.wiki [01:03:41] PROBLEM - udc-japan.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'udc-japan.xyz' expires in 15 day(s) (Wed 27 Nov 2019 01:00:42 AM GMT +0000). [01:03:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewLs [01:03:55] [02miraheze/ssl] 07MirahezeSSLBot 033a875c3 - Bot: Update SSL cert for udc-japan.xyz [01:04:18] PROBLEM - zw.fontainebleau-avon.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'zw.fontainebleau-avon.fr' expires in 15 day(s) (Wed 27 Nov 2019 01:00:55 AM GMT +0000). [01:04:30] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewLG [01:04:32] [02miraheze/ssl] 07MirahezeSSLBot 03eba06ce - Bot: Update SSL cert for zw.fontainebleau-avon.fr [01:13:33] RECOVERY - public.revi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'public.revi.wiki' will expire on Sun 09 Feb 2020 12:01:42 AM GMT +0000. [01:13:43] RECOVERY - udc-japan.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'udc-japan.xyz' will expire on Sun 09 Feb 2020 12:03:48 AM GMT +0000. [01:14:18] RECOVERY - zw.fontainebleau-avon.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'zw.fontainebleau-avon.fr' will expire on Sun 09 Feb 2020 12:04:25 AM GMT +0000. [04:05:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewqG [04:05:09] [02miraheze/services] 07MirahezeSSLBot 03f8bca0b - BOT: Updating services config for wikis [04:48:13] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.mxlinuxusers.org_private] [04:54:16] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:00:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jewm3 [06:00:09] [02miraheze/services] 07MirahezeSSLBot 030bc7832 - BOT: Updating services config for wikis [06:26:25] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2896 MB (12% inode=94%); [07:14:05] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/Jewmp [07:14:06] [02miraheze/dns] 07Reception123 03543e65a - add www.mashedpotaytoes.com zone [07:24:16] !log renamed customwiki to mashedpotaytoeswiki [07:24:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [07:26:48] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JewYk [07:26:50] [02miraheze/ssl] 07Reception123 033fee387 - add www.mashedpotaytoes.com cert [07:51:17] !log recreated mashedpotaytoeswiki (empty wiki; login issues) [07:51:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [08:02:56] !log bash ./deletedbackups.sh /home/reception/temp.dblist [08:03:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [08:09:52] !log ran deletedwikis script on mw1 [08:09:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [08:25:19] !log DROP DATABASE from this list: https://phabricator.miraheze.org/P235\ [08:25:20] [ Login ] - phabricator.miraheze.org [08:25:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [11:55:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewG7 [11:55:09] [02miraheze/services] 07MirahezeSSLBot 0365c3b88 - BOT: Updating services config for wikis [12:15:47] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2644 MB (10% inode=94%); [12:34:16] PROBLEM - bacula1 Bacula Databases db4 on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:36:17] PROBLEM - bacula1 Bacula Databases db4 on bacula1 is WARNING: WARNING: Diff, 46119 files, 36.42GB, 2019-10-20 01:25:00 (3.2 weeks ago) [12:38:27] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [12:42:26] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [13:41:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewcJ [13:41:14] [02miraheze/puppet] 07paladox 0343d59d9 - Revert "Update mfsmaster.cfg.erb" This reverts commit ea5c2a4d5ef08c8f4994ad122a01843b2838d82e. [13:42:01] !log restart lizardfs-master on lizardfs6 [13:42:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:09:57] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 22.35, 14.26, 7.66 [14:13:51] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 7.27, 14.01, 9.30 [14:14:31] i must say lizardfs6 handled that load quite good [14:15:50] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 2.67, 10.03, 8.41 [14:36:01] apoop: The reason you are getting banned is because of your nickname is not on the access list [14:36:12] you will need to log in with nick serv [14:36:25] im apap [14:39:42] why can't you just stick to one username? wouldn't that be easier? [14:44:03] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 3.50, 2.55, 1.40 [14:44:04] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 10.61, 4.36, 2.10 [14:46:05] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 7.94, 4.27, 2.15 [14:49:23] im on my phone [15:00:06] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:00:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:01:50] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [15:01:51] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:02:00] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:02:28] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [15:02:51] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:03:45] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [15:03:52] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 19021 bytes in 2.391 second response time [15:04:01] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 19015 bytes in 0.945 second response time [15:04:25] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [15:04:27] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:04:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:04:47] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.685 second response time [15:12:30] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 0.21, 1.41, 3.66 [15:14:31] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 0.12, 0.97, 3.22 [15:14:55] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 0.22, 0.96, 3.60 [15:16:54] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 0.21, 0.70, 3.18 [16:02:24] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JewWP [16:02:26] [02miraheze/CreateWiki] 07translatewiki 03259c527 - Localisation updates from https://translatewiki.net. [16:02:27] [ Main page - translatewiki.net ] - translatewiki.net. [16:02:28] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JewWX [16:02:29] [02miraheze/ManageWiki] 07translatewiki 0380e58fe - Localisation updates from https://translatewiki.net. [16:02:30] [ Main page - translatewiki.net ] - translatewiki.net. [16:02:31] [02miraheze/IncidentReporting] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JewW1 [16:02:32] [02miraheze/IncidentReporting] 07translatewiki 03365e1a6 - Localisation updates from https://translatewiki.net. [16:02:33] [ Main page - translatewiki.net ] - translatewiki.net. [16:02:33] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+3/-0/±0] 13https://git.io/JewWM [16:02:35] [02miraheze/MirahezeMagic] 07translatewiki 034698b87 - Localisation updates from https://translatewiki.net. [16:02:36] [ Main page - translatewiki.net ] - translatewiki.net. [16:02:37] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewWD [16:02:38] [02miraheze/WikiDiscover] 07translatewiki 030d4d475 - Localisation updates from https://translatewiki.net. [16:02:39] [ Main page - translatewiki.net ] - translatewiki.net. [16:05:00] We might need to update them [16:05:05] In prod [16:05:58] yeah feel free to [16:06:37] Probably best [16:07:05] My only concern is ManageWiki given JohnLewis pushed some fixes [16:07:16] JohnLewis: are they 1.33 compatible [16:07:46] 1.34 is Nov 29 deadline according to phab [16:08:11] my changes should have been to a branch, not master [16:08:13] JohnLewis: 2019-11-09 - 13:06:48UTC tell JohnLewis Check your inbox [16:09:06] JohnLewis: ok [16:09:09] Good [16:09:15] That was from revi [16:10:03] He did it nonetheless [16:10:22] and that reminds me of something to do... ouch [16:10:51] Good [16:11:49] ouch [16:12:08] I originally thought COCC election was due today and just realized it is tomorrow [16:12:10] >_> [16:12:12] timezone sucks [16:12:35] I support flst earth theory so we have no timezone issue [16:31:22] [02miraheze/mediawiki] 07RhinosF1 pushed 035 commits to 03REL1_33 [+0/-0/±5] 13https://git.io/Jewla [16:31:24] [02miraheze/mediawiki] 07RhinosF1 0333c690f - Updating CW [16:31:26] [02miraheze/mediawiki] 07RhinosF1 03fb5a971 - Updating ManageWiki [16:31:27] [02miraheze/mediawiki] 07RhinosF1 035f1cdc7 - Updating IR [16:31:28] [02miraheze/mediawiki] ... and 2 more commits. [16:36:05] .in 5 minutes puppet [16:36:06] RhinosF1: Okay, I will set the reminder for: 2019-11-11 - 16:41:05GMT [16:39:06] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki and sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw[123] and test1 [16:39:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:39:23] paladox: do we need do anything with LFS6 running as a MW server [16:40:02] oh! right [16:40:10] well that needs sign off for mw-admin to access [16:40:17] cc JohnLewis Reception123 PuppyKun ^ [16:40:23] (and SPF|Cloud ^) [16:40:44] paladox: can you run the i18n scripts then on it please [16:40:49] yup [16:41:05] RhinosF1: puppet [16:41:18] paladox: you never answered whether to replicate the updates on test1 [16:41:26] 1.34 branch even [16:41:32] !log running lc on lizardfs6 [16:41:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:41:45] RhinosF1 huh? [16:41:49] i doin't understand? [16:42:06] paladox: do you want me to push the changes i just made to the rel1_34 branch as well [16:42:14] you can [16:42:15] yeh [16:44:58] [02mediawiki] 07RhinosF1 opened pull request 03#128: Update extensions - 13https://git.io/JewlQ [16:45:20] RhinosF1 it's going to be temporarily anyways. [16:52:13] [02mediawiki] 07RhinosF1 closed pull request 03#128: Update extensions - 13https://git.io/JewlQ [17:00:42] [02miraheze/mediawiki] 07RhinosF1 pushed 032 commits to 03REL1_34 [+0/-0/±6] 13https://git.io/Jew8O [17:00:43] [02miraheze/mediawiki] 07RhinosF1 03187a75e - Update extensions [17:00:45] [02miraheze/mediawiki] 07RhinosF1 03e7dcf5b - Update MW [17:10:43] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:12:08] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jew8S [17:12:09] [02miraheze/mediawiki] 07paladox 031d7787d - Update MW [17:16:21] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03DBConnRef [+0/-0/±1] 13https://git.io/Jew8d [17:16:23] [02miraheze/ManageWiki] 07paladox 0387051f2 - DB_SLAVE -> DB_REPLICA [17:16:41] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [17:17:05] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jew8N [17:17:06] [02miraheze/mediawiki] 07paladox 03c74843a - Update MW [17:17:08] RECOVERY - db4 Disk Space on db4 is OK: DISK OK - free space: / 40793 MB (11% inode=96%); [17:50:36] !log add valkyrienskieswiki discord webhook (re-add) [17:50:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:32:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:34:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:58:40] PROBLEM - cp2 Current Load on cp2 is WARNING: WARNING - load average: 1.85, 1.59, 1.13 [19:00:38] RECOVERY - cp2 Current Load on cp2 is OK: OK - load average: 1.45, 1.49, 1.14 [19:35:18] after re-opening a closed wiki, do I need to do something to make the "This wiki has been closed because..." header go away for everyone? [19:46:04] the notice should disappear automatically, if it still appears try clearing the cache [19:46:14] sparr, ^ [20:03:09] thanks, took a minute [20:04:26] much more subjective inquiry... I'm struggling to decide on a structure for my wiki. If anyone can spare an eye, I would appreciate feedback on what parts of this wiki idea to implement as categories vs [dynamic] lists vs namespaced pages, what to simplify via templates, etc. https://www.modesofdiscourse.com/ [20:04:29] [ Modes of Discourse ] - www.modesofdiscourse.com [20:08:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JewEJ [20:08:07] [02miraheze/puppet] 07paladox 03fc459d4 - Update site.pp [20:25:35] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 40291 MB (10% inode=96%); [21:01:39] I put Spam around here... https://phabricator.miraheze.org/T4352 [21:01:40] [ ⚓ T4352 Create the Miraheze-feed-es channel ] - phabricator.miraheze.org [21:03:58] paladox: * [21:04:00] ^ [21:04:41] i wouldn't know how to do that :( [21:04:53] i guess we need to duplicate things? [21:05:05] paladox: how do we do the current irc feed [21:05:31] https://github.com/miraheze/puppet/blob/master/modules/irc/templates/ircrcbot.py [21:05:32] [ puppet/ircrcbot.py at master · miraheze/puppet · GitHub ] - github.com [21:06:46] https://github.com/miraheze/mw-config/blob/71481b5fde0c50f43d1affebbc8c7e5ee6bf511d/LocalWiki.php#L55 [21:06:47] [ mw-config/LocalWiki.php at 71481b5fde0c50f43d1affebbc8c7e5ee6bf511d · miraheze/mw-config · GitHub ] - github.com [21:07:30] my guess is we need to duplicate both that [21:07:35] *them [21:07:50] then use some config that says if lang == es then send [21:09:07] I suppose yeah we’d duplicate that but pick the Lang parameters from the dB as well [21:12:48] I’m mean we filter private wikis out anyway now [21:51:05] oh [22:23:56] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jewgk [22:23:57] [02miraheze/mediawiki] 07paladox 0357aa386 - Update PF [22:28:21] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/Jewgt [22:28:24] [02miraheze/mediawiki] 07paladox 038030db1 - Update PF