[00:03:35] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.60, 1.77, 1.28 [00:05:36] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.36, 1.24, 1.14 [00:16:12] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JfSAi [00:16:13] [02miraheze/MirahezeMagic] 07paladox 037d4a6fb - sendBulkEmails does not need to depend on WikimediaMaintenance [00:16:15] [02MirahezeMagic] 07paladox created branch 03paladox-patch-2 - 13https://git.io/fQRGX [00:16:16] [02MirahezeMagic] 07paladox opened pull request 03#127: sendBulkEmails does not need to depend on WikimediaMaintenance - 13https://git.io/JfSAP [00:17:21] miraheze/MirahezeMagic/paladox-patch-2/7d4a6fb - paladox The build passed. https://travis-ci.com/miraheze/MirahezeMagic/builds/170786692 [00:18:19] [02MirahezeMagic] 07paladox synchronize pull request 03#127: sendBulkEmails does not need to depend on WikimediaMaintenance - 13https://git.io/JfSAP [00:18:20] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JfSAS [00:18:22] [02miraheze/MirahezeMagic] 07paladox 0321cd7f9 - Update sendBulkEmails.php [00:18:42] [02MirahezeMagic] 07paladox edited pull request 03#127: sendBulkEmails does not need to depend on WikimediaMaintenance - 13https://git.io/JfSAP [00:21:56] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 8.52, 4.45, 2.34 [00:22:02] [02MirahezeMagic] 07paladox closed pull request 03#127: sendBulkEmails does not need to depend on WikimediaMaintenance - 13https://git.io/JfSAP [00:22:04] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfSAb [00:22:05] [02miraheze/MirahezeMagic] 07paladox 03ec0fdbb - sendBulkEmails does not need to depend on WikimediaMaintenance (#127) * sendBulkEmails does not need to depend on WikimediaMaintenance * Update sendBulkEmails.php [00:22:07] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/fQRGX [00:22:08] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-2 [00:23:00] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JfSAA [00:23:02] [02miraheze/mediawiki] 07paladox 0305c0346 - Update MirahezeMagic [00:24:32] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfSAj [00:24:34] [02miraheze/MirahezeMagic] 07paladox 037b67eb7 - Fix path to Maintenance.php [00:24:53] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JfSxe [00:24:55] [02miraheze/mediawiki] 07paladox 03867908b - Update MirahezeMagic [00:39:59] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [00:41:29] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [00:41:42] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [00:42:15] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [00:42:53] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.82, 1.28, 1.97 [00:43:33] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [00:43:46] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:44:18] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:44:27] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:44:54] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JfSxi [00:44:55] [02miraheze/mw-config] 07paladox 03578d34b - Update sitenotice [00:44:57] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [00:44:58] [02mw-config] 07paladox opened pull request 03#3106: Update sitenotice - 13https://git.io/JfSxP [00:46:55] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.66, 1.00, 1.70 [00:55:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfSpk [00:55:13] [02miraheze/services] 07MirahezeSSLBot 0367acb84 - BOT: Updating services config for wikis [01:29:40] [02mw-config] 07paladox synchronize pull request 03#3106: Update sitenotice - 13https://git.io/JfSxP [01:29:41] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JfSpN [01:29:43] [02miraheze/mw-config] 07paladox 03b385cbb - Update Sitenotice.php [01:29:54] [02mw-config] 07paladox closed pull request 03#3106: Update sitenotice - 13https://git.io/JfSxP [01:29:56] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfSpA [01:29:57] [02miraheze/mw-config] 07paladox 033387dc7 - Update sitenotice (#3106) [01:29:59] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [01:30:00] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [01:31:16] miraheze/mw-config/paladox-patch-2/b385cbb - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/697067255 [01:31:50] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfShT [01:31:51] [02miraheze/mw-config] 07paladox 038662336 - test against php 7.4 [02:55:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf9eU [02:55:14] [02miraheze/services] 07MirahezeSSLBot 03119ae43 - BOT: Updating services config for wikis [03:23:01] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.86, 1.74, 1.19 [03:27:03] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.64, 1.54, 1.29 [06:02:57] RECOVERY - services1 APT on services1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:10:05] RECOVERY - rdb1 APT on rdb1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:11:15] RECOVERY - cp8 APT on cp8 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:15:08] RECOVERY - rdb2 APT on rdb2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:22:12] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf9kV [06:22:13] [02miraheze/mw-config] 07Reception123 03fea5914 - c/e [06:31:04] RECOVERY - cp7 APT on cp7 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:31:32] RECOVERY - mw6 APT on mw6 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:31:56] RECOVERY - puppet2 APT on puppet2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:44:47] RECOVERY - gluster2 APT on gluster2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:48:50] RECOVERY - phab1 APT on phab1 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [06:55:39] RECOVERY - ldap1 APT on ldap1 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [07:07:34] [02mw-config] 07Reception123 commented on pull request 03#3102: simcitywiki configuration - 13https://git.io/Jf9La [07:08:46] [02mw-config] 07Reception123 edited a comment on pull request 03#3102: simcitywiki configuration - 13https://git.io/Jf9La [08:10:19] RhinosF1: deal with scratch via reception [08:14:33] Reception123: ^ i messaged you [08:16:12] Ah yes [08:28:49] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 4.04, 3.78, 2.58 [08:30:53] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.25, 3.16, 2.50 [09:15:09] PROBLEM - wiki.velaan.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.velaan.org' expires in 15 day(s) (Sat 27 Jun 2020 09:11:22 GMT +0000). [09:21:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf9sT [09:21:34] [02miraheze/ssl] 07MirahezeSSLBot 03cf7e2b0 - Bot: Update SSL cert for wiki.velaan.org [09:33:14] RECOVERY - wiki.velaan.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.velaan.org' will expire on Wed 09 Sep 2020 08:21:26 GMT +0000. [09:35:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf9sE [09:35:14] [02miraheze/services] 07MirahezeSSLBot 03f2ac0d5 - BOT: Updating services config for wikis [11:06:14] Good morning [11:52:36] PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 4.64, 4.13, 2.52 [11:54:38] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.34, 2.99, 2.29 [13:00:12] Good morning [13:11:04] hi AmandaCath [13:12:47] * hispano76 greetings [13:14:43] [02mw-config] 07Amanda-Catherine commented on pull request 03#3102: simcitywiki configuration - 13https://git.io/Jf985 [13:20:39] [02puppet] 07Amanda-Catherine opened pull request 03#1402: add wildcards for googleusercontent and imgbox T5614 - 13https://git.io/Jf94J [13:23:24] AmandaCath: hi. for the PR if JohnLewis has allowed it previously I will let him make the decision when he's available. [13:23:40] I'm not really opposed it's just that it's a bit unusual [13:24:55] RhinosF1: any idea - https://phabricator.miraheze.org/T5734 ? [13:24:56] [ ⚓ T5734 How to alter site line spacing, e.g. between paragraphs or bullet list items? Common.css not having effect ] - phabricator.miraheze.org [13:25:20] no [13:25:24] void will [13:26:29] I was able to reproduce T5732, which seems to be a strange/interesting MediaWiki bug [13:28:13] hmm yeah that might need to be upstreamed [13:33:18] * AmandaCath clears the RequestWiki backlog [13:34:47] We really need to recruit more wiki creators that speak Asian/Far Eastern languages [13:35:17] Such requests seem to be the ones that take the longest to be reviewed, and Google Translate is unreliable for characters [14:10:22] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf9R2 [14:10:24] [02miraheze/services] 07MirahezeSSLBot 037bbfdf7 - BOT: Updating services config for wikis [14:55:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf9EZ [14:55:15] [02miraheze/services] 07MirahezeSSLBot 03c924665 - BOT: Updating services config for wikis [15:32:28] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jf9ud [15:32:30] [02miraheze/mw-config] 07paladox 03313b216 - Update sitenotice [15:32:31] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [15:32:33] [02mw-config] 07paladox opened pull request 03#3107: Update sitenotice - 13https://git.io/Jf9uF [15:34:26] [02mw-config] 07paladox closed pull request 03#3107: Update sitenotice - 13https://git.io/Jf9uF [15:34:28] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf9ze [15:34:29] [02miraheze/mw-config] 07paladox 030a8f3fa - Update sitenotice (#3107) [15:34:31] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [15:34:31] miraheze/mw-config/paladox-patch-2/313b216 - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/697275400 [15:34:32] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [15:37:32] 08:34:47 We really need to recruit more wiki creators that speak Asian/Far Eastern languages [15:37:32] ^ We don't really recruit Wiki Creators. They come to us. That might be part of our problem though... [15:38:02] * Texas thinks that Miraheze does an absolutely awful job of giving new users things to help with [15:59:08] So are the wikis Mediawikis? What's the selection of installable extensions? And once again thanks to the Miraheze people that wrote the Matomo plugin. Love ya! [16:01:39] And sorry about the spamming yesterday, it's just that I care about the humankind [16:02:15] jukebohi: you can view a list of currently installed extensions by going to Special:ManageWiki/extensions on any wiki [16:02:17] jukebohi: No problem thank you for your apology. We do run MediaWiki wikis. If you would like to request one I would be glad to make it for you ASAP [16:03:05] I've been thinking about Automatic Basic Income needing a wiki. Automation necessitates, automation enables. [16:03:34] jukebohi: Hold on. I can find you a list of extensions. [16:04:04] jukebohi: Please see https://github.com/miraheze/mw-config/blob/master/extension-list [16:04:05] [ mw-config/extension-list at master · miraheze/mw-config · GitHub ] - github.com [16:05:10] Thank you oy Lonestar State :D [16:06:04] ...and my new wiki is up and running! [16:07:27] Impressive list of extensions. So is there self-service automation for enabling extensions onwiki? [16:08:27] jukebohi: yes, if you have bureaucrat rights (by default) you can use Special:ManageWiki/extensions [16:08:52] Thanks AmandaCath. Very cool. [16:08:57] jukebohi: and if there is any extension you need that we do not have, just put in a request on phabricator [16:09:04] * Texas loves handling extension requests [16:09:08] when he has time [16:09:22] so are the wikis like on a nameofwiki.miraheze.org or where? [16:09:36] jukebohi: Yes or you can request a custom domain [16:09:51] https://meta.miraheze.org/wiki/Custom_domains [16:09:52] [ Custom domains - Miraheze Meta ] - meta.miraheze.org [16:14:16] I'm using wildcard certs for all my domains on my servers, but I think LetsEncrypt would be ok with issuing a specific subdomain cert via Miraheze automation [16:15:32] jukebohi: that sounds like it would be fine [16:16:18] coz two places issuing same cert will likely set off some low-level alarm at LE [16:16:36] * I mean for the same exact domain [16:22:38] uh oh [16:23:24] Wow, and just like that it starts pouring out of nowhere [16:23:49] One minute it is cloudy and the next it is raining cats and dogs [16:24:17] Heyyy! Your cookie consent banner is an unblocking and unobtrusive one, took me minutes to notice it is there. Which one is it? [16:26:06] jukebohi: Hm? [16:26:11] Coz I should probably comply with the GDPR [16:26:41] jukebohi: Miraheze is in compliance with the GDPR [16:27:07] Texas: yeahyeah, I was admiring how nice your cookie consent extension is. I would like the same [16:28:27] jukebohi: If you host a wiki on Miraheze, it will have that by default [16:28:39] annd Sygnal is messing up again [16:28:42] !reset [16:28:48] ugh [16:28:49] !ping [16:28:52] Yeah, but could you please tell me the name of the extension that does that banner? [16:29:13] CookieWarning [16:29:39] jukebohi: See https://www.mediawiki.org/wiki/Extension:CookieWarning [16:29:57] !ping [16:29:57] Pong! [16:30:00] !reset [16:31:19] Thanks paladox and Texas [16:31:27] np :) [16:32:40] I c https://adsense.googleblog.com/2015/07/introducing-new-user-consent-policy.html ... if I did not know this, my Google Ads project would not have gone through. So I need to enable [[mw:Extension:CookieWarning]] before Google will let me advertise [16:32:41] [ Inside AdSense: Introducing a new user consent policy ] - adsense.googleblog.com [16:32:41] https://meta.miraheze.org/wiki/mw:Extension:CookieWarning [16:35:15] jukebohi: You may get better help by asking in #mediawiki. This channel is mainly for support with Miraheze wikis. [16:36:58] Yeah, sure. I'm reading the documentation now. [16:47:17] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jf921 [16:47:18] [02miraheze/mw-config] 07paladox 03cfd6be9 - Blacklist "investigate" perm Not currently used but is added a newer CU update and also is behind a flag. [16:47:20] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [16:47:21] [02mw-config] 07paladox opened pull request 03#3108: Blacklist "investigate" perm - 13https://git.io/Jf92M [16:47:29] hey [16:47:39] can i get a backup of my wiki? mathorg.miraheze.org [16:47:50] [02mw-config] 07paladox edited pull request 03#3108: Blacklist "investigate" perm - 13https://git.io/Jf92M [16:48:21] rain1 hi, you can do that using Special:DataDump [16:48:56] nice! [16:48:57] thank you [16:49:06] i fgot an error [16:49:10] Although this PHP script (/wiki/Special:DataDump/download/mathorgwiki_xml_fa42ad508f4db1028871.xml.gz) exists, the file requested for output (mwstore://dumps-backend/dumps-backup/mathorgwiki_xml_fa42ad508f4db1028871.xml.gz) does not. [16:49:13] [02mw-config] 07paladox closed pull request 03#3108: Blacklist "investigate" perm - 13https://git.io/Jf92M [16:49:15] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf929 [16:49:16] [02miraheze/mw-config] 07paladox 03e4f5dfb - Blacklist "investigate" perm (#3108) Not currently used but is added a newer CU update and also is behind a flag. [16:49:18] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [16:49:19] oh its queued [16:49:19] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [16:53:29] ty paladox [17:03:39] So, we're continually looking at extensions now for new rights - and not at MediaWiki upgrade..? [17:03:41] JohnLewis: 2020-06-10 - 16:16:03UTC tell JohnLewis If I am eligible I would like a Miraheze cloak, please [17:04:22] JohnLewis: I'm looking for any now on all extensions. Code is on master. [17:04:46] You said we are fine to add rights before they come in use before [17:06:07] I'm aware I said its fine [17:06:12] But I'm asking about the process [17:06:31] Is this the process now? Or is this a one-off reactionary response? [17:07:57] JohnLewis: one-off [17:08:16] This is me manually checking & flagging issues [17:08:55] Okay [17:28:15] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf9aS [17:28:16] [02miraheze/MirahezeMagic] 07translatewiki 03e188f87 - Localisation updates from https://translatewiki.net. [17:28:17] [ Main page - translatewiki.net ] - translatewiki.net [17:28:18] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jf9a9 [17:28:19] [02miraheze/CreateWiki] 07translatewiki 03bbfb106 - Localisation updates from https://translatewiki.net. [17:28:20] [ Main page - translatewiki.net ] - translatewiki.net [17:47:49] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 2.59, 3.43, 2.50 [17:49:50] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.01, 2.64, 2.33 [18:02:53] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jf9w8 [18:02:55] [02miraheze/mediawiki] 07paladox 032f34c44 - Update Echo [18:05:41] !log root@jobrunner1:/srv/mediawiki/w# sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/Echo/db_patches/patch-increase-varchar-echo_unread_wikis-euw_wiki.sql [18:05:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:14:28] hello! is there a page where I as admin can see all members? [18:14:45] ja_: [[Special:UserList]] I believe [18:14:45] https://meta.miraheze.org/wiki/Special:UserList [18:14:58] ^ except on your own wiki [18:15:31] Texas , cheers! [18:15:40] np always glad to help :) [18:28:30] PROBLEM - dbt1 Current Load on dbt1 is CRITICAL: connect to address 51.77.109.151 port 5666: Connection refusedconnect to host 51.77.109.151 port 5666: Connection refused [18:28:35] PROBLEM - dbt1 APT on dbt1 is CRITICAL: connect to address 51.77.109.151 port 5666: Connection refusedconnect to host 51.77.109.151 port 5666: Connection refused [18:28:41] PROBLEM - dbt1 Disk Space on dbt1 is CRITICAL: connect to address 51.77.109.151 port 5666: Connection refusedconnect to host 51.77.109.151 port 5666: Connection refused [18:29:36] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf9rw [18:29:38] [02miraheze/dns] 07paladox 031f2eede - dbt1 -> db9 [18:29:54] PROBLEM - dbt1 Puppet on dbt1 is CRITICAL: connect to address 51.77.109.151 port 5666: Connection refusedconnect to host 51.77.109.151 port 5666: Connection refused [18:29:56] PROBLEM - bacula2 Bacula Databases dbt1 on bacula2 is CRITICAL: CRITICAL: Timeout or unknown client: dbt1-fd [18:32:57] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 26.60, 21.07, 17.59 [18:35:00] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.63, 17.79, 16.79 [18:37:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf9rD [18:37:02] [02miraheze/puppet] 07paladox 03115fe61 - dbt1 -> db9 [18:37:03] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS [18:37:05] [02puppet] 07paladox opened pull request 03#1403: dbt1 -> db9 - 13https://git.io/Jf9ry [18:37:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+1/-1/±0] 13https://git.io/Jf9rS [18:37:19] [02miraheze/puppet] 07paladox 03d673f87 - Rename dbt1.yaml to db9.yaml [18:37:21] [02puppet] 07paladox synchronize pull request 03#1403: dbt1 -> db9 - 13https://git.io/Jf9ry [18:37:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf9rQ [18:37:42] [02miraheze/puppet] 07paladox 03c67616e - Update site.pp [18:37:43] [02puppet] 07paladox synchronize pull request 03#1403: dbt1 -> db9 - 13https://git.io/Jf9ry [18:37:59] what happened to db8? [18:38:51] PROBLEM - dbt1 Puppet on dbt1 is UNKNOWN: NRPE: Unable to read output [18:39:04] oh, I think I see [18:39:17] dbt1 -> db8 [18:39:27] The old db8 -> db9 [18:39:30] RECOVERY - dbt1 Disk Space on dbt1 is OK: DISK OK - free space: / 94778 MB (98% inode=98%); [18:39:42] RECOVERY - dbt1 Current Load on dbt1 is OK: OK - load average: 1.12, 0.66, 0.26 [18:39:42] PROBLEM - dbt1 APT on dbt1 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [18:42:31] RECOVERY - dbt1 Puppet on dbt1 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:43:45] RECOVERY - dbt1 APT on dbt1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:49:11] paladox: will you watch load please as wikimedia have had an incident on InstantCommons [18:50:14] https://phabricator.wikimedia.org/T255179 [18:50:15] [ ⚓ T255179 Session failures preventing edits, login, logout, etc ] - phabricator.wikimedia.org [18:50:20] ok [18:50:37] well that's for sessions, hasn't actually taken the wikis down? [18:50:53] paladox: not afaics [18:50:58] Just keep an eye [18:51:25] See -operations [18:51:39] yup, i've just seen. [18:52:18] * RhinosF1 back in 29 [18:52:20] 20 [18:54:55] [02puppet] 07paladox closed pull request 03#1403: dbt1 -> db9 - 13https://git.io/Jf9ry [18:54:56] [02miraheze/puppet] 07paladox pushed 034 commits to 03master [+2/-2/±3] 13https://git.io/Jf9oP [18:54:58] [02miraheze/puppet] 07paladox 0398fd863 - Merge pull request #1403 from miraheze/paladox-patch-8 dbt1 -> db9 [18:54:59] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [18:55:01] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS [19:02:29] paladox: train held and traffic at codfw [19:03:37] ok [19:03:45] i'm watching :) [19:06:01] PROBLEM - db9 MySQL on db9 is CRITICAL: Can't connect to MySQL server on '51.77.109.151' (115) [19:06:15] PROBLEM - db9 Disk Space on db9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:06:17] PROBLEM - db9 APT on db9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:06:48] PROBLEM - ping4 on db9 is CRITICAL: PING CRITICAL - Packet loss = 100% [19:07:21] PROBLEM - db9 Current Load on db9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:07:57] PROBLEM - ping6 on db9 is CRITICAL: PING CRITICAL - Packet loss = 100% [19:08:29] PROBLEM - db9 SSH on db9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:09:00] known ^ [19:09:02] PROBLEM - db9 Puppet on db9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:09:38] PROBLEM - Host db9 is DOWN: PING CRITICAL - Packet loss = 100% [19:15:18] RECOVERY - Host db9 is UP: PING OK - Packet loss = 0%, RTA = 0.77 ms [19:15:20] RECOVERY - db9 Disk Space on db9 is OK: DISK OK - free space: / 94514 MB (97% inode=98%); [19:15:20] RECOVERY - db9 APT on db9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:15:20] RECOVERY - ping4 on db9 is OK: PING OK - Packet loss = 0%, RTA = 0.55 ms [19:15:20] RECOVERY - db9 Puppet on db9 is OK: OK: Puppet is currently enabled, last run 12 minutes ago with 0 failures [19:15:23] RECOVERY - db9 Current Load on db9 is OK: OK - load average: 0.06, 0.02, 0.00 [19:15:51] RECOVERY - ping6 on db9 is OK: PING OK - Packet loss = 0%, RTA = 0.68 ms [19:16:25] RECOVERY - db9 SSH on db9 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [19:18:11] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jf9KX [19:18:12] [02miraheze/mw-config] 07paladox 032347b30 - db8 -> db9 [19:18:14] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [19:18:15] [02mw-config] 07paladox opened pull request 03#3109: db8 -> db9 - 13https://git.io/Jf9K1 [19:18:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf9KD [19:18:58] [02miraheze/puppet] 07paladox 03a80c358 - dbt1 -> db9 [19:18:59] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS [19:19:01] [02puppet] 07paladox opened pull request 03#1404: dbt1 -> db9 - 13https://git.io/Jf9Ky [19:19:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf9KH [19:19:36] [02miraheze/puppet] 07paladox 0301feb54 - Update nrpe.cfg.erb [19:19:37] [02puppet] 07paladox synchronize pull request 03#1404: dbt1 -> db9 - 13https://git.io/Jf9Ky [19:19:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf9KQ [19:19:53] [02miraheze/puppet] 07paladox 036642f9c - Update director.pp [19:19:54] [02puppet] 07paladox synchronize pull request 03#1404: dbt1 -> db9 - 13https://git.io/Jf9Ky [19:20:52] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.50, 1.89, 1.30 [19:20:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf9K5 [19:20:54] [02miraheze/puppet] 07paladox 03b92f7b8 - Update db.pp [19:20:55] [02puppet] 07paladox synchronize pull request 03#1404: dbt1 -> db9 - 13https://git.io/Jf9Ky [19:23:03] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.74, 1.42, 1.20 [20:01:18] paladox, Voidwalker: Could we knock on a quick +i ? [21:35:39] [02mw-config] 07paladox closed pull request 03#3100: Fix T5708 - 13https://git.io/JfyzZ [21:35:41] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf9M4 [21:35:42] [02miraheze/mw-config] 07Amanda-Catherine 03877df9e - Fix T5708 (#3100) [21:38:19] [02mw-config] 07JohnFLewis commented on pull request 03#3100: Fix T5708 - 13https://git.io/Jf9Mg [21:38:25] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Jf9M2 [21:38:26] [02miraheze/mw-config] 07paladox 03530afcb - Add notice of maintenance [21:38:28] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbvb3 [21:38:29] [02mw-config] 07paladox opened pull request 03#3110: Add notice of maintenance - 13https://git.io/Jf9Ma [21:42:14] [02mw-config] 07paladox synchronize pull request 03#3110: Add notice of maintenance - 13https://git.io/Jf9Ma [21:42:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Jf9MS [21:42:17] [02miraheze/mw-config] 07paladox 0370a34fb - Update Sitenotice.php [21:43:03] [02mw-config] 07paladox closed pull request 03#3110: Add notice of maintenance - 13https://git.io/Jf9Ma [21:43:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf9MQ [21:43:06] [02miraheze/mw-config] 07paladox 038d9e67a - Add notice of maintenance (#3110) * Add notice of maintenance * Update Sitenotice.php [21:43:08] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [21:43:09] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbvb3 [21:46:40] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [21:56:52] paladox: good luck [22:06:40] thanks [22:07:18] * RhinosF1 will likely be alseep soon [22:10:21] what's this maintenance for? and what notifications were put out for it? [22:10:50] a sitenotice, and something to do with the dbt1 failure according to Rhinos [22:13:20] Voidwalker: the db failure the other day afaik and sitenotice [22:15:28] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_34 [+0/-0/±2] 13https://git.io/Jf9yC [22:15:29] [02miraheze/mediawiki] 07paladox 030dc0d87 - Update Cargo [22:15:31] [02miraheze/mediawiki] 07paladox 0368fde05 - Update PageForm [22:15:45] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2111 MB (10% inode=93%); [22:16:10] Voidwalker site notice and also we're falling back to dbt1 (now known as db9). [22:17:22] cool, dropped announcement on discord [22:17:36] paladox: db6 to stay fine or? [22:17:48] yes, db6 is on the dedicated server [22:30:31] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:30:34] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:31:13] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:31:44] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:32:10] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:32:44] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [22:33:28] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:36:22] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [22:36:50] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:37:15] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:37:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf9SJ [22:37:46] [02miraheze/mw-config] 07paladox 036065862 - Fix [23:00:55] !log stopping mysql on db8, moving block storage to db9 and start mysql up on db9 [23:00:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:05:07] * Texas is going off now [23:05:52] * RhinosF1 too [23:06:04] Texas: I'm going as crazy with bots as you [23:11:08] [02mw-config] 07paladox closed pull request 03#3109: db8 -> db9 - 13https://git.io/Jf9K1 [23:11:10] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf99K [23:11:11] [02miraheze/mw-config] 07paladox 03121f992 - db8 -> db9 (#3109) [23:11:13] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [23:11:14] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [23:35:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:35:40] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [23:36:18] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf9QT [23:36:20] [02miraheze/mw-config] 07paladox 03195170f - Remove first site notice [23:38:54] [02puppet] 07paladox closed pull request 03#1404: dbt1 -> db9 - 13https://git.io/Jf9Ky [23:38:55] [02miraheze/puppet] 07paladox pushed 035 commits to 03master [+0/-0/±8] 13https://git.io/Jf9Qm [23:38:57] [02miraheze/puppet] 07paladox 0317ea9b0 - Merge pull request #1404 from miraheze/paladox-patch-8 dbt1 -> db9 [23:38:58] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS [23:39:00] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [23:39:46] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf9QO [23:39:48] [02miraheze/dns] 07paladox 03f37ba58 - Remove db8 from dns [23:44:36] PROBLEM - ping4 on db8 is CRITICAL: PING CRITICAL - Packet loss = 100% [23:44:39] PROBLEM - ping6 on db8 is CRITICAL: PING CRITICAL - Packet loss = 100% [23:44:42] PROBLEM - bacula2 Bacula Databases db9 on bacula2 is CRITICAL: CRITICAL: no terminated jobs [23:44:51] PROBLEM - db8 Current Load on db8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:45:05] PROBLEM - db8 APT on db8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:45:16] PROBLEM - db8 Disk Space on db8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:46:48] PROBLEM - db8 SSH on db8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:48:02] PROBLEM - Host db8 is DOWN: PING CRITICAL - Packet loss = 100% [23:50:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf9Q0 [23:50:40] [02miraheze/puppet] 07paladox 03adbf10c - varnish: Update bootstrap to 4.5.0 [23:50:41] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS [23:50:43] [02puppet] 07paladox opened pull request 03#1405: varnish: Update bootstrap to 4.5.0 - 13https://git.io/Jf9QE [23:51:40] [02puppet] 07paladox synchronize pull request 03#1405: varnish: Update bootstrap to 4.5.0 - 13https://git.io/Jf9QE [23:51:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf9Qa [23:51:43] [02miraheze/puppet] 07paladox 0311aef9b - Update default.vcl [23:52:24] [02puppet] 07paladox synchronize pull request 03#1405: varnish: Update bootstrap to 4.5.0 - 13https://git.io/Jf9QE [23:52:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf9Qo [23:52:27] [02miraheze/puppet] 07paladox 03e41354d - Update default.vcl [23:53:12] [02puppet] 07paladox synchronize pull request 03#1405: varnish: Update bootstrap to 4.5.0 - 13https://git.io/Jf9QE [23:53:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jf9Qi [23:53:15] [02miraheze/puppet] 07paladox 0371b3096 - Update default.vcl [23:55:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf9Q1 [23:55:16] [02miraheze/services] 07MirahezeSSLBot 0350b4884 - BOT: Updating services config for wikis [23:55:29] [02puppet] 07paladox closed pull request 03#1405: varnish: Update bootstrap to 4.5.0 - 13https://git.io/Jf9QE [23:55:30] [02miraheze/puppet] 07paladox pushed 035 commits to 03master [+0/-0/±5] 13https://git.io/Jf9QD [23:55:32] [02miraheze/puppet] 07paladox 03a0bc8bc - Merge pull request #1405 from miraheze/paladox-patch-8 varnish: Update bootstrap to 4.5.0 [23:55:33] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [23:55:35] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS