[00:37:36] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:38:57] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:39:51] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:29] PROBLEM - db4 Current Load on db4 is CRITICAL: CRITICAL - load average: 15.62, 9.84, 5.40 [00:40:38] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw2 [00:40:45] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18599 bytes in 5.963 second response time [00:40:52] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2 [00:41:51] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24646 bytes in 0.005 second response time [00:42:25] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18600 bytes in 0.639 second response time [00:42:54] RECOVERY - db4 Current Load on db4 is OK: OK - load average: 2.61, 6.73, 4.88 [00:43:06] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [00:43:13] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [00:57:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+4/-0/±1] 13https://git.io/JeF6t [00:57:45] [02miraheze/puppet] 07paladox 03a3e9edf - Add nutcracker module Based on https://github.com/wikimedia/puppet/blob/production/modules/nutcracker [00:57:46] [ puppet/modules/nutcracker at production · wikimedia/puppet · GitHub ] - github.com [01:04:18] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeF63 [01:04:20] [02miraheze/mw-config] 07paladox 03b20041d - Redis: migrate to use nutcracker proxy [01:11:37] !log restarted redis-server to free up disk space [01:11:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:30:18] SantaRhino: I have to go now, so if you need more clarification please poke me tomorrow, just wanted to get this out before I leave. The Tech:Access page defines what SRE is. 'System administrators' are all people holding one or more roles described on that same page. Subgroups are teams that fall under the SRE flag, since they are delegated by SRE. [01:31:29] !log /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/PageTriage/sql/PageTriageTagsPatch-recreated.sql [01:31:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:31:41] !log /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/Wikibase/repo/sql/AddNormalizedTermsTablesDDL.sql [01:31:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:32:36] !log /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/OATHAuth/sql/mysql/patch-remove_module_specific_fields.sql [01:32:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:42:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeF6Q [01:42:07] [02miraheze/puppet] 07paladox 03c932457 - Jobrunner: Use nutcracker [02:05:03] !log sudo service jobrunner restart && sudo service jobchron restart (lizardfs6) [02:05:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:05:33] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:15:08] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:51:22] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1438 MB (5% inode=94%); [02:54:14] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 1515 MB (6% inode=94%); [02:56:29] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:58:50] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24668 bytes in 0.519 second response time [04:38:53] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1444 MB (5% inode=94%); [05:05:45] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.86, 1.95, 1.68 [05:08:09] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.11, 0.93, 1.32 [06:25:48] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2452 MB (10% inode=94%); [07:24:56] SPF|Cloud: so what we call just the main team rather than any subgroup? As we’ve all taken SRE as a direct replacement of the operations teams? [08:31:28] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 1.24, 2.22, 2.25 [08:33:54] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.77, 1.26, 1.83 [08:36:15] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.32, 1.18, 1.62 [08:44:01] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 7.74, 3.66, 2.32 [09:01:05] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.36, 0.59, 1.71 [09:03:17] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.20, 0.38, 1.34 [09:07:40] Reception123: an is only used when it's before a,e,i,o,u - fixed phabricator - also no one can spell Reliability - fixed as well [09:10:01] SantaRhino: I know :P did I make a typo somewhere? [09:10:14] and lol reliability is one of those words with too many i's :D [09:13:51] Reception123: grammar on the project board [09:17:25] SantaRhino: ohh, that's because of my careless copy/paste [09:17:36] didn't realize there was an 'an' before I just switched ops with SRE [09:17:45] Reception123: we all do it [09:18:08] * SantaRhino is laughing at what past impersonators have done to impersonate him [09:18:16] SantaRhino: impersonators? [09:19:09] Reception123: testwiki.wiki/wiki/Special:Contributions/RhinosF1 and meta.miraheze.org/wiki/Special:CentralAuth/RhinosF2 [09:19:13] neither are me [09:19:33] sorry https://www.thetestwiki.org/wiki/The_Test_Wiki:Administrators#RhinosF1 [09:19:34] [ The Test Wiki:Administrators - The Test Wiki ] - www.thetestwiki.org [09:19:35] ah [09:19:40] ignore testwiki.wiki [09:19:45] I've had quite a few as well from LTAs on Meta [09:20:04] https://meta.miraheze.org/wiki/User:Reception :P [09:20:09] [ User:Reception - Miraheze Meta ] - meta.miraheze.org [09:20:13] Reception123: I remember one of yours [09:21:00] Reception123: it was 321noitpecer [09:21:05] lol [09:21:17] they took your whole userpage and wrote everything backwards [09:21:34] I admit I don't remember that [09:21:52] Reception123: that was old System Administrator [09:22:33] it was why SPF|Cloud finally locked him [09:24:26] ah [09:24:57] Reception123: do you remeber when we started taking the mick out of icinga-miraheze [09:25:09] s/remeber/remember [09:25:09] SantaRhino meant to say: Reception123: do you remember when we started taking the mick out of icinga-miraheze [09:27:32] SantaRhino: oh yeah lol [09:27:57] ZppixBot joined in too :P [09:28:03] Hello blackaniello! If you have any questions, feel free to ask and someone should answer soon. [09:28:08] Yeah, uh-huh, you know what it is [09:28:12] Black and yellow, black and yellow [09:28:13] Black and yellow, black and yellow [09:28:18] Yeah, uh-huh, you know what it is [09:28:22] Yeah, uh-huh, you know what it is [09:28:25] blackaniello: hi [09:28:27] Everything I do, yeah, I do it big [09:28:30] Reception123: yep [09:28:32] Yeah, uh-huh, screaming: "That's nothin'!" [09:28:37] When I pulled off the lot, that's stuntin' [09:28:37] blackaniello: how can we help? [09:28:43] Reppin' my town, when you see me you know everything [09:28:46] Black and yellow, black and yellow [09:28:46] Black and yellow, black and yellow [09:28:50] I put it down from the whip to my diamonds, I'm in [09:28:54] Black and yellow, black and yellow [09:28:54] Black and yellow, black and yellow [09:29:26] Sigyn: you worked! [09:29:31] merry christmas [09:32:20] Hello mobamba! If you have any questions, feel free to ask and someone should answer soon. [09:32:33] mobamba: hi [09:32:41] back biatches paladox PuppyKun Reception123 SPF|Cloud Zppix SantaRhino [09:32:54] thanks [09:32:55] :P [09:33:54] Reception123: maybe stick a quiet on *!?id41127?@* for a few hours and op you and sigyn up with +z [09:34:16] should hold off a at least the next few [09:35:08] they can't even spell either [09:35:44] yeah good idea [09:37:33] Reception123: mode +z or they won't come through and op sigyn so it can still see them [09:38:28] SantaRhino: what does +z do again actually? [09:38:35] it's been a while since we've had spammers around [09:38:51] Reception123: lets quited users' messages still be seen by ops [09:38:59] s/s'/'s [09:38:59] SantaRhino meant to say: Reception123: lets quited user's messages still be seen by ops [09:39:10] quieted [09:40:52] ah ok [09:41:54] Reception123: quiet *!*@gateway/web/irccloud.com/x-kclncvljbqnhuvie as well as that's their session and it was same on both [09:42:32] SantaRhino: ok, are we sure that doesn't block all irccloud users though? [09:42:51] Reception123: nope, just their session [09:52:42] [02mw-config] 07autoresponder[bot] commented on issue 03#2833: biathcesss I AM FATHER CHRISTMAS HERES UR GIFT - 13https://git.io/JeFyX [09:52:56] [02mw-config] 07mobambo11 commented on issue 03#2833: biathcesss I AM FATHER CHRISTMAS HERES UR GIFT - 13https://git.io/JeFyD [09:52:59] [02mw-config] 07mobambo11 commented on issue 03#2833: biathcesss I AM FATHER CHRISTMAS HERES UR GIFT - 13https://git.io/JeFyy [09:53:02] [02mw-config] 07mobambo11 commented on issue 03#2833: biathcesss I AM FATHER CHRISTMAS HERES UR GIFT - 13https://git.io/JeFyS [09:53:05] [02mw-config] 07mobambo11 commented on issue 03#2833: biathcesss I AM FATHER CHRISTMAS HERES UR GIFT - 13https://git.io/JeFy9 [09:53:08] [02mw-config] 07mobambo11 commented on issue 03#2833: biathcesss I AM FATHER CHRISTMAS HERES UR GIFT - 13https://git.io/JeFyH [09:54:10] [02mw-config] 07autoresponder[bot] commented on issue 03#2834: I got hoes calling a young nigga phone Where's Ali with the motherfucking dope? (bitch, bitch) I be ballin' like a mo'fuckin' pro (like a, huh, like a, huh) I be ballin' like my nigga Mo (Bamba, bitch) Sheck Wes, I ain't a mo'fuckin' joke (haha, hahahahaha) - 13https://git.io/JeFyQ [09:54:22] [02mw-config] 07autoresponder[bot] commented on issue 03#2835: I got hoes calling a young nigga phone Where's Ali with the motherfucking dope? (bitch, bitch) I be ballin' like a mo'fuckin' pro (like a, huh, like a, huh) I be ballin' like my nigga Mo (Bamba, bitch) Sheck Wes, I ain't a mo'fuckin' joke (haha, hahahahaha) - 13https://git.io/JeFy7 [09:54:47] [02mw-config] 07autoresponder[bot] commented on issue 03#2836: irc is funnnnnn hahaha paladox SPF|Cloud SantaRhino Reception123 Zppix everrrione - 13https://git.io/JeFy5 [09:56:13] [02mw-config] 07autoresponder[bot] commented on issue 03#2837: I got hoes calling a young nigga phone Where's Ali with the motherfucking dope? (bitch, bitch) I be ballin' like a mo'fuckin' pro (like a, huh, like a, huh) I be ballin' like my nigga Mo (Bamba, bitch) Sheck Wes, I ain't a mo'fuckin' joke (haha, hahahahaha) - 13https://git.io/JeFyA [09:56:20] [02mw-config] 07autoresponder[bot] commented on issue 03#2838: I got hoes calling a young nigga phone Where's Ali with the motherfucking dope? (bitch, bitch) I be ballin' like a mo'fuckin' pro (like a, huh, like a, huh) I be ballin' like my nigga Mo (Bamba, bitch) Sheck Wes, I ain't a mo'fuckin' joke (haha, hahahahaha) - 13https://git.io/JeFyx [09:56:29] [02mw-config] 07autoresponder[bot] commented on issue 03#2839: I got hoes calling a young nigga phone Where's Ali with the motherfucking dope? (bitch, bitch) I be ballin' like a mo'fuckin' pro (like a, huh, like a, huh) I be ballin' like my nigga Mo (Bamba, bitch) Sheck Wes, I ain't a mo'fuckin' joke (haha, hahahahaha) - 13https://git.io/JeFyp [10:29:30] Hello outkast! If you have any questions, feel free to ask and someone should answer soon. [10:29:40] Hey ya! Hey ya! [10:29:42] outkast: hi [10:29:50] Hey ya! Hey ya! [10:29:55] Reception123: ffs [10:29:55] You think you've got it [10:29:58] Oh, you think you've got it [10:30:02] But got it just don't get it when there's nothin' at all [10:30:26] We get together [10:30:26] Oh, we get together [10:30:26] But separate's always better when there's feelings involved [10:30:26] Know what they say -its: [10:30:26] Nothing lasts forever! [10:30:34] Reception123: kill pls [10:30:36] We get together [10:30:36] Oh, we get together [10:30:36] But separate's always better when there's feelings involved [10:30:36] Know what they say -its: [10:30:36] Nothing lasts forever! [10:30:40] what's Sigyn doing... [10:31:01] Reception123: I don't want to go much wider on the ban [10:31:25] why didn't Sigyn K-line though, strange... [10:31:42] Reception123: we might have to +mz and op a few people temp to watch [10:31:58] it'll be destructive but it's the best for now [10:32:07] Yeah, if it continues we will [10:32:30] Reception123: or use a unidentified quiet [10:58:03] Hello madness! If you have any questions, feel free to ask and someone should answer soon. [10:58:21] Our house, in the middle of our street [10:58:25] Our house, in the middle of our [10:58:36] Our house it has a crowd [10:58:53] madness: nice song. happy christmas but go somewhere else [10:59:06] https://www.irccloud.com/pastebin/GhQoDVrC/ [10:59:06] [ Snippet | IRCCloud ] - www.irccloud.com [10:59:09] Nothing ever slows her down [10:59:09] And a mess is not allowed [10:59:09] Our house, in the middle of our street [10:59:09] Our house, in the middle of our ... [10:59:09] Our house, in the middle of our street [10:59:09] Our house, in the middle of our ... [11:16:58] Reception123: setting up a test mediawiki [11:19:01] SantaRhino: oh, for what purpose? [11:19:33] Reception123: testing locally [11:19:59] ok [11:23:13] Reception123: getting it to replicate MH production (if composer will work [11:23:15] ) [11:23:33] ok [11:30:19] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 3.06, 2.27, 1.81 [11:32:46] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.75, 1.49, 1.60 [12:24:08] * grumble grumbles [12:25:37] grumble: hi [12:25:48] they seem to have been gone for a bit [12:25:52] ye [12:26:21] grumble: very christmassy of them [12:28:38] they liked that Rhinos said it was a nice song xD [13:02:14] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 6.70, 3.62, 2.26 [13:10:52] looking @ cp4 [13:12:00] !log restart varnish on cp4 [13:12:19] !log restart stunnel4 on cp4 [13:12:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:12:59] !log apt-get upgrade on cp4 [13:13:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:15:39] !log [13:12:00] <+paladox> !log restart varnish on cp4 [13:15:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:33:45] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.31, 1.25, 1.99 [13:35:09] paladox: ^ lower [13:35:18] yup [13:35:55] paladox: what was up with it? [13:36:15] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 7.53, 2.63, 2.21 [13:36:18] I'm not sure, i took a look and decided to restart varnish/stunnel4 which seemed to fix it [13:36:20] though [13:36:24] it seems back [13:36:29] paladox: meta is still slower than a snail [13:36:39] And we cursed it - critical load [13:36:41] it's fast for me [13:36:50] paladox: wasn’t just then [13:37:01] that be because your going through lizardfs6 [13:37:03] paladox: back [13:37:21] paladox: is there anyway to tell / force? [13:37:36] Not really i doin't think [13:37:43] we should really add that [13:38:14] paladox: phab goal for new year? Task? [13:39:30] yes please though goal wise, it's not decided yet. [13:39:50] paladox: tag and debate [13:40:04] RECOVERY - Host cp2 is UP: PING OK - Packet loss = 28%, RTA = 97.53 ms [13:40:07] PROBLEM - cp2 SSH on cp2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:07] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: connect to address 107.191.126.23 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [13:40:08] RECOVERY - cp2 Disk Space on cp2 is OK: DISK OK - free space: / 18186 MB (44% inode=99%); [13:40:08] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:40:12] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet last ran 2 days ago [13:40:15] RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.548 second response time [13:40:26] paladox: can you set a goal 2020 tag up? [13:40:28] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [13:40:39] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [13:40:39] And icinga-miraheze looks very happy [13:40:47] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24646 bytes in 0.392 second response time [13:41:03] RECOVERY - cp2 Stunnel Http for test1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24645 bytes in 0.492 second response time [13:41:05] RECOVERY - cp2 Current Load on cp2 is OK: OK - load average: 0.48, 0.21, 0.08 [13:41:12] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 20% [13:41:14] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24645 bytes in 0.391 second response time [13:41:25] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:41:41] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24646 bytes in 0.390 second response time [13:41:51] there we go [13:41:53] RECOVERY - cp2 SSH on cp2 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u7 (protocol 2.0) [13:42:02] so me asking ramnode was not a waste of my time then :P [13:42:07] RECOVERY - wiki.counterculturelabs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.counterculturelabs.org' will expire on Sat 18 Jan 2020 05:39:44 AM GMT +0000. [13:42:08] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1515 bytes in 0.493 second response time [13:42:33] paladox: ? [13:42:55] !log apt-get upgrade on cp2 [13:43:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:43:22] SantaRhino well rn said we had to buy additional bandwidth from now on. [13:43:29] paladox: and? [13:43:35] But cp2 bill was only paid a few days ago. [13:43:40] And? [13:43:42] Oh [13:43:49] They restored it anyway [13:44:01] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [13:44:06] yup [13:46:28] paladox: great, can you create a goal project (https://phabricator.miraheze.org/search/query/cwv5BwEUcO5n/#R) [13:46:29] [ Query: Advanced Search ] - phabricator.miraheze.org [13:46:35] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.56, 0.93, 1.71 [13:47:00] not till the end of december [13:48:04] paladox: a goal can be created before then and planned surely? [13:48:35] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.17, 0.66, 1.43 [13:48:36] Jul-Dec was created Jun 3 by SPF|Cloud [13:52:26] Yeah, it can be created but should only be applied after [13:52:28] I'll create it [13:53:22] Reception123: not what’s been done before - you can start it in a week and a day but tag before then (see goal for 2019H2) [13:53:55] yeah, I'm just going to create the tag for now and then it can be switched when the time comes is what I mean [13:54:45] Reception123: ok [13:55:01] https://phabricator.miraheze.org/project/view/50/ [13:55:02] [ Goal-2020-Jan-Jun · Manage ] - phabricator.miraheze.org [13:56:25] ok [13:57:06] though what would be planned? [13:58:11] paladox: setting a way to view/pick which servers you go through [13:58:23] And whatever else we come up with [14:02:15] Which servers we go through? [14:02:39] paladox: like I asked moments ago [14:02:58] Forcing your connection to go through a specific server [14:08:00] paladox: you mean the mw servers? if yes, that already is a thing [14:08:07] *SantaRhino [14:08:09] wrong ping :P [14:08:47] Reception123: for anyone? Maybe tell other SRE members [14:09:23] SantaRhino: well technically everyone can do it you just need to change the IP in the host file on your PC [14:09:35] Reception123: maybe a nicer method [14:09:38] and afaik paladox also had this chrome extension at one point [14:09:40] ^ SantaRhino this one :) [14:09:41] * SantaRhino is also mobile [14:10:02] I remember there being an extension for it [14:10:07] Reception123: what is it’s name? Could we have a more inter grated MH provided one though [14:10:19] there's a link, paladox would have it [14:10:27] I don't remember [14:22:06] SantaRhino: I'm curious why such a thing should be made easy [14:22:30] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 4.28, 2.80, 1.84 [14:23:09] https://phabricator.miraheze.org/T5019 [14:23:10] I ment create a way to tell users which backend there going through [14:23:10] [ ⚓ T5019 Create a way to check and change servers proxied through ] - phabricator.miraheze.org [14:23:11] We technically allow that already but it should not be used by non developers [14:23:37] By we technically allow, I mean changing the server we proxy through [14:24:08] changing the cache proxy? that is done by DNS and you cannot just change that [14:24:31] SPF|Cloud: he means backend [14:24:37] I think? [14:24:39] SPF|Cloud: I’d like it to be simple (might be useful for those on discord who wanted to use upgraded MediaWiki but at the moment it’s guess of the server) [14:24:45] "both the cache proxy and MediaWiki service" [14:24:57] Cp isn’t too much but MediaWiki would be great [14:25:29] I fail to see the need for non-system administrators [14:26:21] SPF|Cloud: for cp not so much but MediaWiki, seeing upgrades mid slowly cause eager, getting around slow servers [14:26:24] And also [14:26:26] 07:24:56 SPF|Cloud: so what we call just the main team rather than any subgroup? As we’ve all taken SRE as a direct replacement of the operations teams? [14:26:27] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.94, 1.33, 1.49 [14:26:30] if a mediawiki version is ready for production, it will be put in production accordingly. if you have a good reason to go through another server, sure, but we already have x-miraheze-debug for that [14:27:13] SPF|Cloud: x-Miraheze-debug means little to me (also said via mobile as well would be good if that matters) [14:27:14] ^ [14:27:28] a mobile device is not a sufficient solution for debugging or work arounds [14:27:52] and people technically aware enough to recognise the need for using non-supported methods should also know how to inject custom headers into their web requests [14:28:16] now - if there are really issues caused by upgrades, that should be our problem to solve, that is not the fault of users [14:28:30] ^ [14:28:55] SPF|Cloud: not issues but when the first server was running 1.34 (read only) quite a few were eager to use it [14:29:06] And also would allow me round the slow lfs6 [14:29:10] you mean test1? that is a debugging server [14:29:24] SPF|Cloud: no, during rollout to production [14:30:22] I fail to understand the need to do that, again, the newer version will be put in production as soon as that is deemed viable [14:30:57] SPF|Cloud: eagerness [14:31:21] we take care of all requirements and procedures as needed, and for the convenience of users we also deploy the versions whenever they're ready [14:31:27] It’s in production but you could get either 1.33 or 1.34 during the upgrade based on where they hit [14:31:41] That’s not a reason? [14:32:00] eagerness is fine, but that is not a supported solution [14:32:19] paladox: it’s something people asked whether they could get 1.33 or 1.34 and how to set it during the update the other day [14:32:35] and thus not worth spending time into knowing the upgrade will be in place whenever fine [14:32:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeF5L [14:32:44] [02miraheze/puppet] 07paladox 03430e28a - Default new installs to 1.34 [14:34:27] if you want to force usage of a specific mediawiki server (through http headers). sure thing, but after all it was meant to be a debugging method for system administrators and as such, a method that is easy enough to implement [14:34:55] SPF|Cloud: can you send an instruction guide on it? [14:35:21] SantaRhino: ok [14:35:38] https://meta.miraheze.org/wiki/Tech:Varnish#X-Miraheze-Debug [14:35:43] [ Tech:Varnish - Miraheze Meta ] - meta.miraheze.org [14:36:10] SPF|Cloud: thx , see PM as well [14:36:37] SantaRhino: alright, looking [14:44:55] paladox: can you see if you just got a connection to lixardfs6.miraheze.org, if so, that’s the slow thing - you’ll know by the User Agent [14:46:44] I'm sure he won't get a connection to 'lixardfs6' :P [14:48:40] Reception123: In access.log [14:49:13] s/lixardfs6.miraheze.org/lizardfs6 [14:49:16] that :D [14:50:11] Oh [14:52:25] SantaRhino: I know lizardfs6 is slow that’s what I’ve been saying :) [14:52:26] It has latency to the db in the netherlands [14:52:27] Also I’m mobile [14:53:45] paladox: I was comparing it and let me know when you’re on [14:58:10] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 9.61, 4.05, 2.34 [15:02:12] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.77, 1.74, 1.91 [15:04:13] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.22, 0.92, 1.53 [15:30:07] Reception123: have you removed the quiets? [15:34:47] SantaRhino: see PM [15:37:34] Seen replied [16:00:34] SantaRhino i'm back [16:06:06] paladox: ok, then did I 100% go through lfs6 at that point [16:06:12] You’ll see my UA [16:06:18] What is your UA? [16:06:23] The access log is quite long [16:08:55] SantaRhino ^ [16:17:45] * hispano76 greentings [16:18:38] hi hispano76 [16:19:00] hi Reception123 [16:21:48] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.53, 1.84, 1.14 [16:23:04] [ANNOUNCEMENT] on behalf of all ZppixBot developers merry christmas and happy holidays [16:23:32] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeFbI [16:23:34] [02miraheze/dns] 07paladox 03803c6ca - Depool cp4 [16:23:47] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 1.95, 2.22, 1.48 [16:25:10] !log turn off TUN on cp4 [16:25:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:25:50] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.59, 0.28, 0.11 [16:26:46] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeFbO [16:26:47] [02miraheze/dns] 07paladox 032b8a15c - Revert "Depool cp4" This reverts commit 803c6ca15e9b4c0f58d507287f43c6469f49b6d0. [16:52:01] paladox, JohnLewis: https://phabricator.miraheze.org/T5016#95533 [16:52:02] [ ⚓ T5016 Renew mail alias config ] - phabricator.miraheze.org [16:52:03] !log depool lizardfs6 [16:52:21] If no objections, I can create a PR tonight [16:52:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:53:14] !log repool lizardfs6 [16:53:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:54:12] !log depool and repool mw3, restart php7.3-fpm and nginx [16:54:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:54:43] ok [16:54:47] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.28, 1.82, 1.65 [16:56:47] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.78, 1.02, 1.35 [16:57:50] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeFb5 [16:57:51] [02miraheze/dns] 07paladox 0389a82f0 - Depool cp4 [16:58:58] !log reimage cp4 a debian 10 [16:59:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:02:22] PROBLEM - cp4 Stunnel Http for test1 on cp4 is CRITICAL: connect to address 81.4.109.133 port 5666: Connection refusedconnect to host 81.4.109.133 port 5666: Connection refused [17:02:23] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: connect to address 81.4.109.133 port 5666: Connection refusedconnect to host 81.4.109.133 port 5666: Connection refused [17:02:36] PROBLEM - cp4 Stunnel Http for misc2 on cp4 is CRITICAL: connect to address 81.4.109.133 port 5666: Connection refusedconnect to host 81.4.109.133 port 5666: Connection refused [17:02:38] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: connect to address 81.4.109.133 port 5666: Connection refusedconnect to host 81.4.109.133 port 5666: Connection refused [17:02:41] PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address thesimswiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [17:02:59] PROBLEM - cp4 Puppet on cp4 is CRITICAL: connect to address 81.4.109.133 port 5666: Connection refusedconnect to host 81.4.109.133 port 5666: Connection refused [17:03:10] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: connect to address 81.4.109.133 port 5666: Connection refusedconnect to host 81.4.109.133 port 5666: Connection refused [17:03:10] PROBLEM - cp4 Current Load on cp4 is CRITICAL: connect to address 81.4.109.133 port 5666: Connection refusedconnect to host 81.4.109.133 port 5666: Connection refused [17:03:27] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: connect to address enc.for.uz and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [17:03:42] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: connect to address 81.4.109.133 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [17:03:52] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:03:55] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: connect to address 81.4.109.133 port 5666: Connection refusedconnect to host 81.4.109.133 port 5666: Connection refused [17:04:03] PROBLEM - guiasdobrasil.com.br - LetsEncrypt on sslhost is CRITICAL: connect to address guiasdobrasil.com.br and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [17:04:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:04:11] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: connect to address 81.4.109.133 port 5666: Connection refusedconnect to host 81.4.109.133 port 5666: Connection refused [17:04:13] PROBLEM - cp4 Disk Space on cp4 is CRITICAL: connect to address 81.4.109.133 port 5666: Connection refusedconnect to host 81.4.109.133 port 5666: Connection refused [17:10:53] PROBLEM - cp4 Puppet on cp4 is UNKNOWN: NRPE: Unable to read output [17:11:19] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.86, 0.91, 0.50 [17:11:46] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is UNKNOWN: NRPE: Unable to read output [17:12:11] RECOVERY - cp4 Disk Space on cp4 is OK: DISK OK - free space: / 41594 MB (96% inode=97%); [17:12:11] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.021 second response time [17:12:13] PROBLEM - cp4 Varnish Backends on cp4 is UNKNOWN: NRPE: Unable to read output [17:12:19] RECOVERY - cp4 Stunnel Http for test1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24645 bytes in 0.974 second response time [17:12:20] RECOVERY - cp4 Stunnel Http for misc2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.081 second response time [17:12:31] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24645 bytes in 0.005 second response time [17:13:04]