[01:48:09] <icinga-miraheze>	 PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2115 MB (10% inode=93%);
[02:19:29] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jv8i0
[02:19:30] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03c67acd4 - Mariadb: Add extra-port
[02:19:32] <Not-5a0a>	 [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS
[02:19:33] <Not-5a0a>	 [02puppet] 07paladox opened pull request 03#1246: Mariadb: Add extra-port - 13https://git.io/Jv8iE
[05:01:43] <icinga-miraheze>	 RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3383 MB (17% inode=93%);
[06:25:35] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2709 MB (11% inode=94%);
[07:55:03] <RhinosF1>	 Reception123: ping
[08:24:57] <Reception123>	 RhinosF1: yes?
[08:29:52] <RhinosF1>	 PM’d
[11:26:18] <icinga-miraheze>	 PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins]
[11:30:08] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8dh
[11:30:10] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03d499113 - BOT: Updating services config for wikis
[11:34:17] <icinga-miraheze>	 RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[12:39:41] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/Jv8FX
[12:39:41] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 039fd3409 - add wiki.nvda-nl.org cert
[12:39:41] <Not-5a0a>	 [02ssl] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vxP9L
[12:39:41] <Not-5a0a>	 [02ssl] 07Reception123 opened pull request 03#265: add wiki.nvda-nl.org cert - 13https://git.io/Jv8F1
[12:39:41] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+1/-0/±0] 13https://git.io/Jv8Fy
[12:39:41] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 03b5d8fa3 - add wiki.nvda-nl.org cert
[12:39:41] <Not-5a0a>	 [02ssl] 07Reception123 synchronize pull request 03#265: add wiki.nvda-nl.org cert - 13https://git.io/Jv8F1
[12:39:41] <Not-5a0a>	 [02ssl] 07Reception123 closed pull request 03#265: add wiki.nvda-nl.org cert - 13https://git.io/Jv8F1
[12:39:41] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jv8FS
[12:39:41] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 03b41591b - add wiki.nvda-nl.org cert (#265) * add wiki.nvda-nl.org cert * add wiki.nvda-nl.org cert
[12:39:41] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 deleted branch 03Reception123-patch-1
[12:39:41] <Not-5a0a>	 [02ssl] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vxP9L
[12:39:41] <RhinosF1>	 Reception123: can you do the open wiki rename request
[12:39:41] <Reception123>	 RhinosF1: I just opened that task and was doing the config for read-only :)
[12:39:42] <RhinosF1>	 Reception123: great!
[12:39:42] <Not-5a0a>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8Fb
[12:39:42] <Not-5a0a>	 [02miraheze/mw-config] 07Reception123 039c2d4e9 - ungamewiki readonly for maint
[12:39:43] <Reception123>	 !log renamed ungamewiki to libertygamewiki (db + script)
[12:39:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[12:39:44] <Not-5a0a>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv8bl
[12:39:44] <Not-5a0a>	 [02miraheze/mw-config] 07Reception123 036281d77 - rm readonly for ungamewiki
[12:50:27] <Reception123>	 !log disabled PlavorSeol  on Phabricator
[12:50:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:18:38] <ZppixBot>	 Hello Peterl65! If you have any questions, feel free to ask and someone should answer soon.
[13:21:23] <Peterl65>	 I asked two questions on the Stewards noticeboard but may be here I get a faster answer. My first is https://wiki.arcolinuxclub.org/ does not work as expeced
[13:24:03] <RhinosF1>	 Peterl65: one moment
[13:24:55] <RhinosF1>	 Peterl65: have you requested a custom domain on phabricator?
[13:28:41] <Peterl65>	 I think I used the normal wiki request process where I was asked a subdomain and my wanted custom domain. Phabricator I used only for a Let's Encrypt Cert.
[13:29:40] <Reception123>	 Peterl65: Hi! I think I remember your domain, let me take a look
[13:38:31] <Reception123>	 Peterl65: Oh, your domain doesn't work because it was never configured by us (pointing is not enough)
[13:38:43] <Reception123>	 and your Phabricator request is for wiki.endeavouros.de not wiki.arcolinuxclub.org
[13:46:17] <Peterl65>	 I had some requests in the past: https://arcolinuxclub.miraheze.org/wiki/Main_Page was at the 1. December 2019 created, while https://endeavourosde.miraheze.org/wiki/Hauptseite was at the 13. February 2020 created.
[13:46:21] <ZppixBot>	 [ archman-o ] - arcolinuxclub.miraheze.org
[13:46:22] <ZppixBot>	 [ EndeavourOS Users Wiki ] - endeavourosde.miraheze.org
[13:52:11] <RhinosF1>	 Peterl65: please fill out a phabricator ssl request
[13:52:34] <RhinosF1>	 What was the other issue?
[13:55:25] <Peterl65>	 For both cases? And should I request a new wiki for this archlinuxfamily domain also or will it be possible to rename the endeavouros wiki?
[14:02:22] <RhinosF1>	 Peterl65: both in seperate tasks, please state change x to y clearly.
[14:43:17] <Peterl65>	 I had lost the connection, sorry, is my order for a Let's Encrypt Certificate arrived?
[14:43:44] <Zppix>	 Let me check
[14:44:25] <Zppix>	 Yes your task was created it will be gotten to whenever we get the chance Peterl65
[14:45:33] <Peterl65>	 What will be the normal handling time? When I may ask that.
[14:46:16] <Zppix>	 Peterl65: usually not too long
[14:46:42] <Zppix>	 Peterl65: i dont have an exact eta however
[14:47:21] <Peterl65>	 ;-)  Thats's like Debian: It's ready when it's ready...
[14:48:00] <Zppix>	 Peterl65: heh sorry, we’re all volunteers if i wasnt away from my pc i would do it right now but im not home
[14:53:51] <Peterl65>	 YES, no problem I'm not in a hurry but I know this behavior from Linux Distributions where they are also are driven by enthusiastic voluteers... What's about my other wish to rename endeavouros by archlinuxfamily? Shoulld I make a new wiki request?
[14:54:16] <Zppix>	 Peterl65: no if you create another task for that we can rename it
[14:56:50] <RhinosF1>	 Reception123: https://phabricator.miraheze.org/T5228#99580
[14:56:53] <ZppixBot>	 [ ⚓ T5228 Enable edit rights for everyone - no requirement to login ] - phabricator.miraheze.org
[14:56:58] <Peterl65>	 Thanks a lot! You are very helpful.
[15:00:39] <RhinosF1>	 Peterl65: no problem
[15:29:42] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2648 MB (10% inode=94%);
[15:31:45] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2607:5300:205:200::17f6/cpweb
[15:33:42] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[15:34:38] <Peterl65>	 Please take a look at T5240 and T5241 if I filled them orrectly
[15:37:51] <RhinosF1>	 .task T5240
[15:37:53] <ZppixBot>	 https://phabricator.miraheze.org/T5240 - Please rename endeavourosde to archlinuxfamilyde [Open] authored by Peter_Littmann, assigned to None
[15:38:50] <RhinosF1>	 Peterl65: https://phabricator.miraheze.org/T5240 doesn't make sense. What needs renaming from what to what?
[15:38:52] <ZppixBot>	 [ ⚓ T5240 Please rename endeavourosde to archlinuxfamilyde ] - phabricator.miraheze.org
[15:39:09] <RhinosF1>	 neither does T5421
[15:39:11] <paladox>	 !log sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki=pcsdscotlandwiki --overwrite
[15:39:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:40:27] <RhinosF1>	 paladox: thx
[15:41:10] <paladox>	 it didn't fix the issue
[15:41:27] <RhinosF1>	 paladox: oh. I'll look
[15:45:17] <RhinosF1>	 paladox: anon has perms, error message logged out when creating pages looks custom
[15:46:00] <paladox>	 Even if custom, there's still a problem.
[15:47:18] <RhinosF1>	 paladox: I'm trying to work out what's controlling it though
[15:48:24] <paladox>	 RhinosF1 https://www.mediawiki.org/wiki/Extension:EditSubpages
[15:48:24] <ZppixBot>	 [ Extension:EditSubpages - MediaWiki ] - www.mediawiki.org
[15:48:49] <Peterl65>	 T5240: Which name do you need? Actual subdomain: https://endeavourosde.miraheze.org, to new subdomain https://archlinuxfamily.miraheze.org, and afterwads place a Certificate please.
[15:48:51] <ZppixBot>	 https://phabricator.miraheze.org/T5240 - Please rename endeavourosde to archlinuxfamilyde [Open] authored by Peter_Littmann, assigned to None
[15:49:14] <paladox>	 RhinosF1 fixed!
[15:49:21] <RhinosF1>	 Peterl65: thanks. What about for T4521?
[15:49:26] <RhinosF1>	 paladox: that's it
[15:56:46] <paladox>	 RhinosF1 https://phabricator.miraheze.org/T5242 huh
[15:56:49] <ZppixBot>	 [ ⚓ T5242 Rename archlinuxfamily to endeavourosde ] - phabricator.miraheze.org
[15:56:51] <paladox>	 that wiki does not exist
[15:57:05] <paladox>	 so how can i rename a nonexistent wiki to a wiki that exists?
[15:59:58] <RhinosF1>	 paladox: it does - https://endeavourosde.miraheze.org/
[16:00:01] <ZppixBot>	 [ EndeavourOS Users Wiki ] - endeavourosde.miraheze.org
[16:00:05] <paladox>	 archlinuxfamily does not
[16:00:20] <paladox>	 your asking archlinuxfamily to rename to the wiki you linked to
[16:00:31] <RhinosF1>	 paladox: 15:48:49 <Peterl65> T5240: Which name do you need? Actual subdomain: https://endeavourosde.miraheze.org, to new subdomain https://archlinuxfamily.miraheze.org, and afterwads place a Certificate please.
[16:00:45] <RhinosF1>	 that's how I've asked it based on
[16:00:45] <paladox>	 RhinosF1 "Rename archlinuxfamily to endeavourosde "
[16:00:53] <paladox>	 "archlinuxfamily to endeavourosde"
[16:01:21] <paladox>	 That user says it correctly
[16:01:31] <RhinosF1>	 paladox: what do you mean? check now :)
[16:01:46] <paladox>	 they want to rename endeavourosde to archlinuxfamily
[16:02:00] <paladox>	 now it's correct
[16:04:03] <paladox>	 !log renaming endeavourosdewiki to archlinuxfamilywiki per T5242
[16:04:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:04:36] <Peterl65>	 T5241 old: kujum , new name archlinuxfamilyorg
[16:04:39] <ZppixBot>	 https://phabricator.miraheze.org/T5241 - Please rename kujum to archlinuxfamilyorg [Open] authored by Peter_Littmann, assigned to None
[16:06:10] <Peterl65>	 Is this now better understandable?
[16:07:56] <RhinosF1>	 Peterl65: move the rename and custom domain to separate tasks
[16:08:26] <paladox>	 So i need to remove the custom domain from kujum, rename it to archlinuxfamilyorg and add a new domain
[16:09:20] <RhinosF1>	 paladox: yes
[16:12:35] <paladox>	 !log renaming kujumwiki to archlinuxfamilyorgwiki per T5241
[16:12:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:13:49] <Peterl65>	 So you keep these T5240 and T5241 for the remove of the old domains and request me to fill out two new tasks-requests for connecting, right?
[16:15:21] <Not-5a0a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jv4vH
[16:15:23] <Not-5a0a>	 [02miraheze/ssl] 07paladox 0342fb869 - Remove wiki.kujum.org
[16:15:24] <Not-5a0a>	 [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L
[16:15:26] <Not-5a0a>	 [02ssl] 07paladox opened pull request 03#266: Remove wiki.kujum.org - 13https://git.io/Jv4vQ
[16:15:38] <Not-5a0a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/Jv4v7
[16:15:39] <Not-5a0a>	 [02miraheze/ssl] 07paladox 0330539f4 - Delete wiki.kujum.org.crt
[16:15:41] <Not-5a0a>	 [02ssl] 07paladox synchronize pull request 03#266: Remove wiki.kujum.org - 13https://git.io/Jv4vQ
[16:15:50] <Not-5a0a>	 [02ssl] 07paladox closed pull request 03#266: Remove wiki.kujum.org - 13https://git.io/Jv4vQ
[16:15:51] <Not-5a0a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/Jv4vd
[16:15:53] <Not-5a0a>	 [02miraheze/ssl] 07paladox 03e86bfb6 - Remove wiki.kujum.org (#266) * Remove wiki.kujum.org * Delete wiki.kujum.org.crt
[16:15:54] <Not-5a0a>	 [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L
[16:15:56] <Not-5a0a>	 [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1
[16:17:14] <Not-5a0a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jv4vN
[16:17:16] <Not-5a0a>	 [02miraheze/ssl] 07paladox 03c8cd561 - Add wiki.archlinuxfamily.org
[16:17:17] <Not-5a0a>	 [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L
[16:17:19] <Not-5a0a>	 [02ssl] 07paladox opened pull request 03#267: Add wiki.archlinuxfamily.org - 13https://git.io/Jv4vA
[16:17:31] <Not-5a0a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/Jv4vx
[16:17:33] <Not-5a0a>	 [02miraheze/ssl] 07paladox 039e86a81 - Create wiki.archlinuxfamily.org.crt
[16:17:34] <Not-5a0a>	 [02ssl] 07paladox synchronize pull request 03#267: Add wiki.archlinuxfamily.org - 13https://git.io/Jv4vA
[16:17:48] <Not-5a0a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jv4vp
[16:17:50] <Not-5a0a>	 [02miraheze/ssl] 07paladox 0388d398e - Update wiki.archlinuxfamily.org.crt
[16:17:51] <Not-5a0a>	 [02ssl] 07paladox synchronize pull request 03#267: Add wiki.archlinuxfamily.org - 13https://git.io/Jv4vA
[16:18:33] <Not-5a0a>	 [02ssl] 07paladox closed pull request 03#267: Add wiki.archlinuxfamily.org - 13https://git.io/Jv4vA
[16:18:35] <Not-5a0a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jv4fe
[16:18:36] <Not-5a0a>	 [02miraheze/ssl] 07paladox 03608239d - Add wiki.archlinuxfamily.org (#267)
[16:18:38] <Not-5a0a>	 [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L
[16:18:39] <Not-5a0a>	 [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1
[16:20:12] <RhinosF1>	 Peterl65: paladox has done it anyway
[16:24:43] <paladox>	 !log ALTER TABLE /*_*/oathauth_users DROP COLUMN secret, DROP COLUMN scratch_tokens;
[16:24:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:25:21] <RhinosF1>	 paladox: what does that do? and close your tasks
[16:25:39] <paladox>	 RhinosF1 it does exactly what it says.
[16:26:07] <Zppix>	 Looks like it alerts something and drops something else
[16:26:14] <Zppix>	 Alters*
[16:26:29] <Peterl65>	 I have seen the output but my browser still shows your mainpage not my wiki and the certificat is also missing. The german domaain is also not working Do you need further Takss?
[16:26:56] <Zppix>	 Peterl65: give it a few minutes to apply
[16:27:17] <RhinosF1>	 paladox: why though?
[16:27:30] <Zppix>	 Our servers check for changes approx. 10 minutes
[16:27:32] <paladox>	 Because that is was the extension does?
[16:27:43] <Peterl65>	 ok, thanks, willl check later.
[16:34:49] <Peterl65>	 https://wiki.archlinuxfamily.org/wiki/ shows error 404
[16:34:52] <ZppixBot>	 [ 404 Not Found ] - wiki.archlinuxfamily.org
[16:35:31] <Zppix>	 paladox: ^
[16:35:50] <paladox>	 yeh, i need to attach a custom domain
[16:36:01] <paladox>	 i'm having issues logging into that wiki...
[16:36:07] <paladox>	 it's failing with 2fa for some reason
[16:36:09] <Zppix>	 paladox: want me to try
[16:36:15] <paladox>	 yes please
[16:36:22] <Zppix>	 paladox: your setting it to wiki.?
[16:36:44] <paladox>	 Needs to be https://wiki.archlinuxfamily.org
[16:36:45] <ZppixBot>	 [ Wiki not Found ] - wiki.archlinuxfamily.org
[16:36:56] <Zppix>	 paladox: which subdomain is it?
[16:37:13] <paladox>	 archlinuxfamilyorg
[16:37:20] <Zppix>	 Ok
[16:38:39] <Zppix>	 paladox: {{done}}
[16:39:29] <Zppix>	 paladox: still 404s?
[16:39:35] <paladox>	 takes a minute
[16:39:45] <paladox>	 works for me
[16:39:54] <Zppix>	 paladox: helps if i clear cache :D
[16:40:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv4f9
[16:40:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 035b152b3 - BOT: Updating services config for wikis
[16:40:30] <Zppix>	 Peterl65: it should work now
[16:41:56] <Zppix>	 paladox: it seems to intermittenly  404
[16:42:01] <paladox>	 hmm
[16:42:34] <Zppix>	 paladox: weird thing is its just a standard nginx 404
[16:45:34] <Peterl65>	 yay archlinuxfamily.de is still your mainpage and archlinuxfamily.org is error 404 from nginx
[16:45:54] <Zppix>	 Peterl65: were looking into the wiki domain
[16:46:06] <Zppix>	 paladox: ^ any ideas?
[16:46:12] <paladox>	 i'm looking
[16:46:47] <Peterl65>	 anything I can do?
[16:46:54] <paladox>	 !log restart nginx on cp4
[16:47:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[16:51:22] <Zppix>	 Doesnt appear to have fixed it paladox
[16:51:54] <Zppix>	 And im mobile so idk what cp is serving me
[16:59:26] <icinga-miraheze>	 PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4212 bytes in 0.072 second response time
[16:59:28] <JohnLewis>	 Zppix: the same CP as always
[16:59:39] <icinga-miraheze>	 PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4212 bytes in 0.023 second response time
[16:59:44] <icinga-miraheze>	 PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4210 bytes in 0.022 second response time
[17:00:08] <paladox>	 huh
[17:00:11] <paladox>	 why is everything down?
[17:00:13] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[17:00:26] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[17:01:01] <icinga-miraheze>	 PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:01:22] <icinga-miraheze>	 RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 20541 bytes in 1.030 second response time
[17:01:33] <icinga-miraheze>	 RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 20540 bytes in 0.811 second response time
[17:01:34] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2
[17:01:40] <SPF|Cloud>	 wut
[17:01:49] <icinga-miraheze>	 PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:01:59] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[17:02:04] <RhinosF1>	 what just happened?
[17:02:06] <paladox>	 !log restart nginx on mw2
[17:04:59] <icinga-miraheze>	 PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 3 backends are down. mw1 mw3 lizardfs6
[17:05:36] <icinga-miraheze>	 PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is WARNING: WARNING - NGINX Error Rate is 46%
[17:07:03] <SPF|Cloud>	 seems to be php-fpm
[17:07:17] <paladox>	 SPF|Cloud php-fpm is a symptom not the cause
[17:07:35] <icinga-miraheze>	 PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is CRITICAL: CRITICAL - NGINX Error Rate is 77%
[17:07:50] <SPF|Cloud>	 well whatever is causing php-fpm to behave like this is also causing the issue
[17:08:23] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 88%
[17:08:51] <Not-5a0a>	 [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jv4J0
[17:08:53] <Not-5a0a>	 [02miraheze/dns] 07paladox 037ec62bc - mw[123] disable ipv6 temporarily
[17:08:54] <Not-5a0a>	 [02dns] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbQXl
[17:08:55] <Zppix>	 RhinosF1: were looking into it
[17:08:56] <Not-5a0a>	 [02dns] 07paladox opened pull request 03#128: mw[123] disable ipv6 temporarily - 13https://git.io/Jv4JE
[17:09:03] <RhinosF1>	 Zppix: I can see
[17:09:21] <icinga-miraheze>	 PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4212 bytes in 0.045 second response time
[17:09:33] <icinga-miraheze>	 PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 62%
[17:09:36] <icinga-miraheze>	 PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is WARNING: WARNING - NGINX Error Rate is 49%
[17:10:18] <icinga-miraheze>	 PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 48%
[17:11:35] <icinga-miraheze>	 RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 34%
[17:11:38] <icinga-miraheze>	 RECOVERY - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is OK: OK - NGINX Error Rate is 7%
[17:11:45] <icinga-miraheze>	 RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 20541 bytes in 0.815 second response time
[17:12:06] <icinga-miraheze>	 RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 20542 bytes in 6.947 second response time
[17:12:15] <icinga-miraheze>	 RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 19%
[17:12:59] <icinga-miraheze>	 RECOVERY - cp8 Varnish Backends on cp8 is OK: All 6 backends are healthy
[17:13:10] <icinga-miraheze>	 RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 20540 bytes in 0.341 second response time
[17:13:22] <icinga-miraheze>	 RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 20541 bytes in 0.313 second response time
[17:13:34] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[17:13:51] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy
[17:14:13] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[17:14:37] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[17:22:06] <Peterl65>	 This was amazing! I hope all is okay now again? wiki.archlinuxfamily.de shows your Mainpage as before...
[17:25:26] <Peterl65>	 archlinuxfamily.org shows: Error 503 Backend fetch failed, forwarded for 2003:ec:cf11:b800::1b9, 127.0.0.1(Varnish XID 17498770) via cp4.miraheze.org at Mon, 17 Feb 2020 17:23:15 GMT.
[17:26:37] <Not-5a0a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jv4UT
[17:26:39] <Not-5a0a>	 [02miraheze/ssl] 07paladox 036f5492e - Add wiki.archlinuxfamily.de
[17:26:40] <Not-5a0a>	 [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L
[17:26:42] <Not-5a0a>	 [02ssl] 07paladox opened pull request 03#268: Add wiki.archlinuxfamily.de - 13https://git.io/Jv4Uk
[17:27:04] <Not-5a0a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/Jv4UI
[17:27:06] <Not-5a0a>	 [02miraheze/ssl] 07paladox 03ab38580 - Create wiki.archlinuxfamily.de.crt
[17:27:07] <Not-5a0a>	 [02ssl] 07paladox synchronize pull request 03#268: Add wiki.archlinuxfamily.de - 13https://git.io/Jv4Uk
[17:27:38] <Not-5a0a>	 [02ssl] 07paladox closed pull request 03#268: Add wiki.archlinuxfamily.de - 13https://git.io/Jv4Uk
[17:27:40] <Not-5a0a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jv4Ut
[17:27:42] <Not-5a0a>	 [02miraheze/ssl] 07paladox 03a04579a - Add wiki.archlinuxfamily.de (#268) * Add wiki.archlinuxfamily.de * Create wiki.archlinuxfamily.de.crt
[17:27:43] <Not-5a0a>	 [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1
[17:27:45] <Not-5a0a>	 [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L
[17:29:46] <Not-5a0a>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-1/±0] 13https://git.io/Jv4Um
[17:29:48] <Not-5a0a>	 [02miraheze/ssl] 07paladox 034b6b062 - fix
[17:33:23] <paladox>	 Peterl65 should work now
[17:34:49] <Peterl65>	 moment please
[17:34:53] <Not-5a0a>	 [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbQXl
[17:34:54] <Not-5a0a>	 [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1
[17:34:56] <Not-5a0a>	 [02dns] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbQXl
[17:34:57] <Not-5a0a>	 [02miraheze/dns] 07paladox deleted branch 03paladox-patch-2
[17:39:56] <Peterl65>	 Could you please check my account?  I get on both wikis the failure that my account information could not transmitted and so can not login.
[17:42:06] <paladox>	 SPF|Cloud: ^
[17:42:07] <paladox>	 (I’m now mobile)
[17:46:39] <paladox>	 Peterl65: what does it say?
[17:48:36] <Peterl65>	 There was a problem with the transfer of your user data. This action was aborted for security reasons to prevent a wrong assignment of your changes to another user. Please resubmit the form. Translate by DEEPL
[17:49:40] <Peterl65>	 On META and wiki.mxlinuxusers.de I am logged in without problem.
[17:50:08] <RhinosF1>	 Peterl65: go to meta, log out, close all tabs, log back in. Then it should work
[17:50:52] <paladox>	 Try logging out then log in on the wiki that’s having problems to log into 
[17:51:55] <RhinosF1>	 paladox: it’s an issue with renamed wikis. You have to log out/in again to be able to act on them again.
[17:54:59] <paladox>	 Yes, I know :)
[17:58:52] <Peterl65>	 Thanks, now I am logged in!
[17:59:40] <Peterl65>	 Good evening!
[18:04:40] <RhinosF1>	 Peterl65: great
[18:05:31] <RhinosF1>	 Peterl65: need anything else?
[18:07:11] <Peterl65>	 No, not for now, thanks for your assistence!
[18:07:20] <RhinosF1>	 No problem
[18:40:03] <RhinosF1>	 Zppix, Reception123: https://phabricator.miraheze.org/T5202
[18:40:06] <ZppixBot>	 [ ⚓ T5202 Please allow GNU FDL ] - phabricator.miraheze.org
[19:49:06] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://git.io/Jv4L7
[19:49:08] <Not-5a0a>	 [02miraheze/puppet] 07paladox 0326190b8 - varnish: Use mw[4567] only on cp7
[19:49:09] <Not-5a0a>	 [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://git.io/vbiAS
[19:49:11] <Not-5a0a>	 [02puppet] 07paladox opened pull request 03#1247: varnish: Use mw[4567] only on cp7 - 13https://git.io/Jv4L5
[19:49:42] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1247: varnish: Use mw[4567] only on cp7 - 13https://git.io/Jv4L5
[19:49:44] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://git.io/Jv4Lb
[19:49:45] <Not-5a0a>	 [02miraheze/puppet] 07paladox 037a75e52 - Update default.vcl
[20:05:32] <icinga-miraheze>	 PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2111 MB (10% inode=93%);
[20:11:08] <Zppix>	 paladox:  disk warning? we've already used alot of cp8's disk?
[20:11:13] <paladox>	 yup
[20:11:22] <paladox>	 it only has 20gb
[20:11:27] <Zppix>	 paladox:  the host isnt a week old yet lmao
[20:11:44] <paladox>	 well we're using 10gb of varnish cache files i think?
[20:17:14] <RhinosF1>	 Zppix: u have access?
[20:20:26] <Zppix>	 RhinosF1:  to what?
[20:21:21] <Zppix>	 RhinosF1:  i have access to mw*, test[12], and jobrunner1
[20:22:54] <RhinosF1>	 Zppix: a computer to finish a pr & deploy
[20:23:04] <Zppix>	 RhinosF1:  depenmds what it is
[20:23:06] <Not-5a0a>	 [02mw-config] 07RhinosF1 opened pull request 03#2888: Add GPLv3 to WikiLicenses - 13https://git.io/Jv4qT
[20:23:33] <Zppix>	 I wouldnt know how to deply ^
[20:23:42] <Zppix>	 and i dont have much time cause i go to work at 3pm utc-5
[20:23:45] <Zppix>	 leave for work*
[20:25:14] <RhinosF1>	 Zppix: I’ll finish the PR, tidy it up, and then merge and puppet it
[20:27:16] <Zppix>	 RhinosF1:  thats the issue with the new servers I dont know if i should deploy it to them as well and if so how, so i think Reception123  or paladox  should handle it til then
[20:27:44] <paladox>	 Zppix what issue?
[20:29:10] <RhinosF1>	 paladox: can you deploy & tidy #2888 if I finish most of the work
[20:29:18] <Zppix>	 paladox:  RhinosF1  wants his PR deployed when its ready, and im not around in about 30 mins
[20:31:44] <JohnLewis>	 Zppix: you don't deploy changes manually though, puppet does it
[20:32:23] <paladox>	 i can merge
[20:32:33] <Zppix>	 JohnLewis:  i wasnt sure... if that was still the case
[20:32:57] <JohnLewis>	 Puppet has always done that, since 2015. There's no reason it will ever change
[20:33:44] <RhinosF1>	 JohnLewis: deploy = merge and run puppet if needed
[20:34:16] <JohnLewis>	 RhinosF1: Not going to lie, that adds literally no value to my comment :)
[20:35:29] <Not-5a0a>	 [02puppet] 07paladox closed pull request 03#1247: varnish: Use mw[4567] only on cp7 - 13https://git.io/Jv4L5
[20:35:46] <Not-5a0a>	 [02mw-config] 07RhinosF1 synchronize pull request 03#2888: Add GPLv3 to WikiLicenses - 13https://git.io/Jv4qT
[20:35:47] <Not-5a0a>	 [02puppet] 07paladox closed pull request 03#1231: db6: Downgrade to mariadb 10.2 - 13https://git.io/Jvl2K
[20:35:49] <Not-5a0a>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8
[20:35:51] <Not-5a0a>	 [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS
[20:35:58] <Zppix>	 RhinosF1:  is your pr readyu
[20:36:06] <Not-5a0a>	 [02puppet] 07paladox closed pull request 03#1232: Redirect all wikis to miraheze.org - 13https://git.io/Jvla3
[20:36:08] <Not-5a0a>	 [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://git.io/vbiAS
[20:36:10] <Not-5a0a>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9
[20:38:19] <Not-5a0a>	 [02mw-config] 07RhinosF1 synchronize pull request 03#2888: Add GPLv3 to WikiLicenses - 13https://git.io/Jv4qT
[20:39:16] <Not-5a0a>	 [02mw-config] 07RhinosF1 commented on pull request 03#2888: Add GPLv3 to WikiLicenses - 13https://git.io/Jv4qu
[20:39:48] <RhinosF1>	 Zppix: is now but check syntax. I’m mobile so tabbing might be messy.
[20:39:58] <Zppix>	 paladox:  ^ i think its ready but i cant deploy it as im not around for long and if it breaks i wont have time to fix
[20:41:24] <RhinosF1>	 Zppix: if that is merged as-is, I’m 99.99% it will fail due to indentation being odd
[20:41:28] <Not-5a0a>	 [02mw-config] 07JohnFLewis reviewed pull request 03#2888 commit - 13https://git.io/Jv4q2
[20:41:48] <Not-5a0a>	 [02mw-config] 07JohnFLewis reviewed pull request 03#2888 commit - 13https://git.io/Jv4qa
[20:43:12] <Not-5a0a>	 [02mw-config] 07RhinosF1 synchronize pull request 03#2888: Add GPLv3 to WikiLicenses - 13https://git.io/Jv4qT
[20:44:19] <Not-5a0a>	 [02mw-config] 07RhinosF1 synchronize pull request 03#2888: Add GPLv3 to WikiLicenses - 13https://git.io/Jv4qT
[20:44:39] <RhinosF1>	 JohnLewis: switched
[20:44:51] <JohnLewis>	 thanks, it's good for me
[20:45:13] <RhinosF1>	 JohnLewis: as long as everything is indented right then it should be but be careful
[20:46:08] <JohnLewis>	 I mean the indentation is wrong, but PHP isn't one of those languages
[20:47:12] <Zppix>	 I wish python was like PHP when it came to indention
[20:47:26] <Zppix>	 in python you have extra pixel of space and your whole program is screwed
[20:47:35] <RhinosF1>	 JohnLewis: then it’ll be good. I can fix that later but I told the user on twitter I’d do the task today and I’m
[20:47:42] <RhinosF1>	 Not getting back on pc
[20:48:40] <JohnLewis>	 It's fine I'll get it merged in a few
[20:49:50] <Zppix>	 JohnLewis:  i can merge it if you think it's fine
[20:50:04] <JohnLewis>	 no I'm going to fix the indentation
[20:50:07] <Zppix>	 oh ok
[20:55:07] <Not-5a0a>	 [02miraheze/mw-config] 07JohnFLewis pushed 036 commits to 03master [+0/-0/±8] 13https://git.io/Jv4qd
[20:55:09] <Not-5a0a>	 [02miraheze/mw-config] 07RhinosF1 035016b40 - Add GPLv3 to WikiLicenses Fixes T5202
[20:55:10] <Not-5a0a>	 [02miraheze/mw-config] 07RhinosF1 035bab680 - Update LocalWiki.php
[20:55:12] <Not-5a0a>	 [02miraheze/mw-config] 07RhinosF1 0336f4881 - Update LocalWiki.php
[20:55:13] <Not-5a0a>	 [02miraheze/mw-config] ... and 3 more commits.
[20:55:41] <Not-5a0a>	 [02mw-config] 07JohnFLewis closed pull request 03#2888: Add GPLv3 to WikiLicenses - 13https://git.io/Jv4qT
[21:00:06] <RhinosF1>	 JohnLewis: let me know once puppet has run
[21:02:31] <JohnLewis>	 Puppet should be running now
[21:03:10] <RhinosF1>	 Ok
[21:04:34] <RhinosF1>	 JohnLewis: the image doesn’t load
[21:05:38] <paladox>	 needs to be added to the csp policy
[21:05:54] <RhinosF1>	 Thx paladox
[22:12:41] <icinga-miraheze>	 PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1351 bytes in 0.052 second response time
[22:12:46] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[22:13:11] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[22:13:18] <icinga-miraheze>	 PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:13:20] <icinga-miraheze>	 PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 4 backends are down. mw1 mw2 mw3 lizardfs6
[22:13:21] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[22:13:51] <icinga-miraheze>	 PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:13:54] <icinga-miraheze>	 PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4210 bytes in 0.021 second response time
[22:14:05] <RhinosF1>	 paladox:
[22:14:12] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. mw1 mw2 mw3 lizardfs6
[22:14:17] <paladox>	 hmm
[22:15:16] <icinga-miraheze>	 RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 20539 bytes in 3.314 second response time
[22:15:17] <icinga-miraheze>	 RECOVERY - cp8 Varnish Backends on cp8 is OK: All 6 backends are healthy
[22:15:19] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy
[22:15:29] <paladox>	 recovered it seems
[22:15:36] <RhinosF1>	 Good
[22:15:45] <icinga-miraheze>	 RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 20541 bytes in 0.441 second response time
[22:15:53] <icinga-miraheze>	 RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 20541 bytes in 0.413 second response time
[22:16:08] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[22:16:38] <icinga-miraheze>	 RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 20541 bytes in 0.911 second response time
[22:16:41] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[22:17:02] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[23:40:53] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb
[23:42:47] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online