[00:02:36] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 34.11, 26.90, 18.57 [00:02:41] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3973 MB (20% inode=93%); [00:04:31] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 15.53, 22.13, 17.79 [00:06:27] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 12.58, 19.47, 17.36 [00:47:53] PROBLEM - spcodex.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'spcodex.wiki' expires in 15 day(s) (Wed 08 Jul 2020 00:45:23 GMT +0000). [00:49:24] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfNSQ [00:49:25] [02miraheze/ssl] 07MirahezeSSLBot 0310cd329 - Bot: Update SSL cert for spcodex.wiki [01:03:54] RECOVERY - spcodex.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'spcodex.wiki' will expire on Sat 19 Sep 2020 23:49:17 GMT +0000. [02:33:57] PROBLEM - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:35:57] RECOVERY - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Thu 20 Aug 2020 14:59:34 GMT +0000. [09:43:40] PROBLEM - bacula2 Bacula Databases db6 on bacula2 is WARNING: WARNING: Full, 22675 files, 97.02GB, 2020-06-07 09:40:00 (2.1 weeks ago) [10:13:13] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:13:42] ^ fine [10:13:54] They are heavy on false positives [10:15:12] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Thu 20 Aug 2020 14:54:45 GMT +0000. [11:35:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfAKq [11:35:16] [02miraheze/services] 07MirahezeSSLBot 0302ab28c - BOT: Updating services config for wikis [12:18:29] PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:19:28] paladox: that's a false positive [12:19:36] That check is dodgy [12:20:29] RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Sun 16 Aug 2020 23:10:01 GMT +0000. [14:57:36] Hi [15:06:48] Welcome back [15:22:42] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is WARNING: WARNING: Full, 82779 files, 3.261GB, 2020-06-07 15:19:00 (2.1 weeks ago) [15:26:18] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JfA7U [15:26:19] [02miraheze/CreateWiki] 07translatewiki 0320e7c4f - Localisation updates from https://translatewiki.net. [15:26:20] [ Main page - translatewiki.net ] - translatewiki.net [15:26:21] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/JfA7T [15:26:23] [02miraheze/MirahezeMagic] 07translatewiki 0357e7f57 - Localisation updates from https://translatewiki.net. [15:26:24] [ Main page - translatewiki.net ] - translatewiki.net [15:26:24] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfA7k [15:26:26] [02miraheze/ManageWiki] 07translatewiki 03cc0ad5b - Localisation updates from https://translatewiki.net. [15:26:27] [ Main page - translatewiki.net ] - translatewiki.net [15:56:25] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:57:46] Reception123: ^ false positive again [16:04:57] RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Thu 20 Aug 2020 14:59:34 GMT +0000. [16:25:16] :( [16:40:53] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JfAdh [16:40:55] [02miraheze/mw-config] 07Reception123 03bebd61a - add politics as a wiki category [16:40:56] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [16:41:22] [02mw-config] 07Reception123 opened pull request 03#3121: add politics as a wiki category - 13https://git.io/JfAFU [18:42:20] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.55, 2.07, 1.43 [18:44:15] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.49, 1.85, 1.43 [18:46:13] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.22, 1.66, 1.41 [19:01:17] Hey JohnLewis [19:02:03] hi [19:02:13] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.51, 2.11, 1.59 [19:02:22] icinga-miraheze: relax [19:02:42] why chat when we can listen to the stupid bots [19:03:41] Naleksuh I think it's very offensive that you insult me this way, us bots are very useful! [19:04:08] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.88, 1.69, 1.50 [19:04:37] /ignore icinga-miraheze [19:05:16] if it doesn't interest you feel free to ignore it yeah [19:05:47] * RhinosF1 wonders who was hiding behind icinga-miraheze [19:06:01] * RhinosF1 stares at that perso [19:06:06] n* [19:06:23] RhinosF1: if you mean hiding behind ZppixBot :P [19:06:36] I do [19:06:39] reception123 : the bots should be removed from this channel and put into #miraheze-bot channel instead [19:07:25] that could be in consideration, but until now it wasn't really because there isn't that much traffic in this channel to justify it [19:07:54] maybe there's not much traffic *because* of the bots [19:08:07] channels like #wikipedia-en dont have bots going off every 3 seconds [19:08:10] #miraheze-bots is quite now SigmaBot isn't flooding [19:08:37] i meant like why can't *all* bots be moved there [19:09:30] well an argument for the bots staying here would be that it's more visible to sysadmins [19:09:56] and I wouldn't really say there's no traffic because of the bots, I haven't really seen instances of users trying to talk and being unable to because of the bots [19:13:46] there can certainly be disadvantages [19:13:47] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_phabricator-extensions] [19:14:31] Hey Guys [19:14:49] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core],Exec[git_pull_landing] [19:14:52] insource:"anything" doesn't work on Meta? [19:14:54] * hispano76 greetings [19:15:05] Hi [19:15:24] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config],Exec[git_pull_MediaWiki core],Exec[git_pull_landing] [19:16:37] Or how is produced lock image behind page title if is page locked? [19:16:58] Reception123: why's puppet angry? [19:17:18] RhinosF1: I really don't get puppet sometimes [19:17:20] No one did anything [19:17:53] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core],Exec[git_pull_landing] [19:18:40] Reception123: puppet agent randomly having some fun? [19:18:46] Jun 22 19:16:01 mw4 puppet-agent[13060]: (/Stage[main]/Mediawiki/Git::Clone[MediaWiki core]/Exec[git_pull_MediaWiki core]/returns) fatal: unable to access 'https://github.com/Hutchy68/pivot.git/': SSL: certificate subject name (*.miraheze.org) does not match target host name 'github.com' [19:18:47] [ GitHub - Hutchy68/pivot: A MediaWiki mobile skin which "Pivots" seamlessly to any size display. ] - github.com [19:18:47] yup seems so [19:19:14] paladox: GitHub down! [19:19:19] Of course [19:19:20] yup [19:19:21] same for me [19:19:25] so for once puppet was right to blab [19:19:29] Reception123: DNS fail [19:19:32] yes [19:19:46] how fun when the site hosting all of our code decides to go down [19:20:00] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core],Exec[git_pull_landing] [19:20:17] paladox: maybe downtime for an hour? [19:20:31] https://www.githubstatus.com/ not updated [19:20:31] [ GitHub Status ] - www.githubstatus.com [19:20:32] Reception123: seems to be a localised dns failure [19:20:41] why downtime? i don't understand why you would downtime something if theirs an outage [19:20:54] you would downtime something if like there is planned downtime tho [19:20:55] if we want to silence icinga for a bit probably just a +q here [19:21:02] *theres [19:21:11] Something to reduce spam until it's fixed [19:21:17] +q :) [19:21:22] It's a known global issue we can't control [19:21:57] silencing monitoring is heavily discouraged [19:22:12] you owe it to your users to make them aware of any incident, irrespective of who's fault it is [19:22:24] eth01: well this incident doesn't affect users in any way [19:22:44] eth01: No reason for any user impact [19:22:45] i guess it's like what if there is an incident when we quieted things [19:22:46] JohnLewis: well then maybe Naleksuh is right about moving bots from here, as users who want to discuss things or ask questions shouldn't have to be spammed with icinga notifs [19:23:01] paladox: all staff are opped and would see [19:23:07] no [19:23:13] Or can [19:23:15] this channel has to be moderated for us to see [19:23:15] RhinosF1: uh I don't see +qs when I'm opped [19:23:22] you would need +z [19:23:26] Reception123: +z [19:23:27] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:23:32] Reception123: I'm not commenting on it as frankly I've never seen it as an issue and only a very few number of people over 5 years has ever brought it up (few as in <10 I believe) [19:23:51] GitHub Recovered now it seems [19:24:01] If people want to move it, they can [19:24:03] I can't access it yet [19:24:07] Oh I thought it was impacting my bad [19:24:34] ok perhaps we need to ask more people and get everyone's opinion on the matter, with priority to sysadmins as they are the ones responding to the alerts [19:24:46] Or not [19:24:54] Seemed to come back for a minute [19:25:22] eth01: GitHub's down so puppet couldn't check for updates to deploy [19:28:27] * RhinosF1 doesn't have strong thoughts but would be happy for it in -bots or a -SRE channel [19:29:34] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core],Exec[git_pull_landing] [19:30:55] Anything about my question? How is produced lock image behind page title if is page locked? Like this https://i.nuuls.com/x4P9d.png [19:31:01] And sorry for the annoying question [19:31:22] Not sure [19:32:03] Due to bad category "Category:Wikipedia semi-protected templates" [19:33:11] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [19:33:47] RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:34:04] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:37:08] Can be "Facebook" in the wiki name? [19:38:24] I'm not sure [19:38:34] Depends on context [19:38:42] And copyright rules [19:39:45] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config],Exec[git_pull_MediaWiki core] [19:40:18] And whether or not you comply with the terms of use [19:41:06] (of Miraheze) [19:42:15] https://www.githubstatus.com/ [19:42:15] [ GitHub Status ] - www.githubstatus.com [19:42:22] Now updated [19:42:37] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [19:43:46] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:43:49] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:56:10] Reception123: look what SPF|Cloud said https://phabricator.miraheze.org/T5782#113014 [19:56:10] [ ⚓ T5782 [Access Request] RhinosF1 for Mediawiki-Admins ] - phabricator.miraheze.org [20:11:13] > Anything about my question? How is produced lock image behind page title if is page locked? > Like this https://i.nuuls.com/x4P9d.png @MrJaroslavik Is there something in the MediaWiki: interface namespace that needs to be changed, or is it really just that simple of setting the icon position to appear in the upper most right corner? [20:12:14] adding on an individual page bases [20:13:39] RhinosF1: https://www.githubstatus.com/incidents/shdhxntw519h?utm_ts=1592856013 heh [20:13:39] [ GitHub Status - Incident on 2020-06-22 19:52 UTC ] - www.githubstatus.com [20:14:12] Reception123: just seen, loads for me when you're ready [20:14:28] Or not [20:14:31] RhinosF1: still doesn't work for me [20:14:41] Up and down [20:14:55] only been down for me [20:15:39] Seems to last for 30s every so often [20:22:49] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.31, 18.23, 16.82 [20:24:46] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 12.23, 16.17, 16.26 [20:26:52] I got it > [[Module:Protection banner/config]] [20:29:19] [02puppet] 07Reception123 closed pull request 03#1408: Access: Restore rhinosf1 [mw-admin] - 13https://git.io/JfdpO [20:29:20] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/Jfxf4 [20:29:22] [02miraheze/puppet] 07RhinosF1 03598cf53 - Access: Restore rhinosf1 [mw-admin] (#1408) * Restore rhinosf1 https://phabricator.miraheze.org/T5782 * Update aliases * Update data.yaml * Update users.conf * Update data.yaml Add it to the right person [20:29:22] [ ⚓ T5782 [Access Request] RhinosF1 for Mediawiki-Admins ] - phabricator.miraheze.org [20:29:27] Yey [20:29:39] * RhinosF1 will set ssh back up soon [20:30:40] RhinosF1: check your email for the GH invite [20:30:57] !log added RhinosF1 to @miraheze GitHub project as mw-admins [20:31:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:31:36] Reception123: done [20:39:20] !log created mail account for rhinosf1@ [20:39:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:42:01] :) [20:42:56] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2116 MB (10% inode=93%); [20:49:42] !log root@misc1:/var/mail# chown -R rhinosf1:rhinosf1 /home/rhinosf1/ [20:49:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:50:38] Reception123: try emailing now [20:51:15] [02mw-config] 07paladox closed pull request 03#3121: add politics as a wiki category - 13https://git.io/JfAFU [20:51:17] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfxJc [20:51:18] [02miraheze/mw-config] 07Reception123 033e83cd3 - add politics as a wiki category (#3121) [20:51:23] [02miraheze/mw-config] 07paladox deleted branch 03Reception123-patch-1 [20:51:24] [02mw-config] 07paladox deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [20:52:50] Reception123: got it [20:54:38] RhinosF1: great! [20:54:41] paladox: thanks [21:14:43] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.76, 1.71, 1.28 [21:16:44] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.88, 1.37, 1.20 [21:20:47] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.86, 2.06, 1.54 [21:22:48] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.08, 1.72, 1.48 [21:24:49] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.92, 1.43, 1.40 [21:36:13] !log run rhinos@mw4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/purgeList.php --namespace=2 --wiki=rf1botwiki to test my access (without --purged and only one page) [21:36:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:39:54] [02ssl] 07RhinosF1 opened pull request 03#326: Delete wiki.edhel.online.crt - 13https://git.io/Jfxkg [21:40:25] [02ssl] 07RhinosF1 synchronize pull request 03#326: Delete wiki.edhel.online.crt - 13https://git.io/Jfxkg [21:40:26] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jfxka [21:40:28] [02miraheze/services] 07MirahezeSSLBot 03341a718 - BOT: Updating services config for wikis [21:41:26] Reception123, paladox: merge pls ^ [21:41:56] [02ssl] 07paladox closed pull request 03#326: Delete wiki.edhel.online.crt - 13https://git.io/Jfxkg [21:41:58] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/Jfxki [21:41:59] [02miraheze/ssl] 07RhinosF1 03014cd0b - Delete wiki.edhel.online.crt (#326) * Delete wiki.edhel.online.crt No longer needed per https://phabricator.miraheze.org/T5787 * Update certs.yaml [21:42:00] [ ⚓ T5787 Request for CNAME and Let's Encrypt Configuration Reversal ] - phabricator.miraheze.org [21:43:01] .in 10mins edhel.mh.o [21:43:01] RhinosF1: Okay, will remind at 2020-06-22 - 22:53:01BST [21:44:06] That was requested days ago [21:53:01] RhinosF1: edhel.mh.o [22:23:36] !log rhinos@mw4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --r="Requested on Discord" --i=2 --wiki=thefinalrumblewiki /home/rhinos/catvid.txt [22:23:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:28:36] .in 7mins check ^^ [22:28:37] RhinosF1: Okay, will remind at 2020-06-22 - 23:35:36BST [22:35:36] RhinosF1: check ^^ [22:36:10] at least 290s left [22:37:27] .in 450s ^^ [22:37:28] RhinosF1: Okay, will remind at 2020-06-22 - 23:44:57BST [22:44:58] RhinosF1: ^^ [22:45:12] 42 left [22:50:46] !log rhinos@mw4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --namespace=6 --category=Videos --wiki=thefinalrumblewiki [22:50:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:51:31] !log both done [22:51:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:49:15] !log sudo -u www-data php /srv/mediawiki/w/maintenance/changePassword.php --user=ZppixBot --password= --wiki=loginwiki (per request of MacFan) [23:49:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master