[00:33:20] PROBLEM - Puppet errors on tools-exec-1409 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:08:18] RECOVERY - Puppet errors on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [02:22:23] PROBLEM - Puppet errors on tools-exec-1419 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [02:27:07] (03CR) 10Legoktm: [V: 032 C: 032] Major revamp [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/374095 (owner: 10Legoktm) [03:25:19] 10Data-Services, 10DC-Ops, 10Operations, 10ops-codfw: Split up labstore external shelf storage available in codfw between labstore2001 and 2 - https://phabricator.wikimedia.org/T171623#3560941 (10Papaul) @madhuvishy I took a quick look at labstore2001 the H800 controller doesn't allow me to create a RAID... [03:27:21] RECOVERY - Puppet errors on tools-exec-1419 is OK: OK: Less than 1.00% above the threshold [0.0] [03:42:43] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:49:12] PROBLEM - Puppet errors on tools-exec-1405 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [04:17:45] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [04:19:13] RECOVERY - Puppet errors on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [05:33:54] 10Data-Services, 10DC-Ops, 10Operations, 10ops-codfw: Split up labstore external shelf storage available in codfw between labstore2001 and 2 - https://phabricator.wikimedia.org/T171623#3561111 (10madhuvishy) @Papaul, Hardware RAID 10 on both labstore2001 and 2002, with 6 or 8 disks per logical/virtual RAID... [06:26:02] 10Cloud-Services, 10DBA: Prepare and check storage layer for electcomwiki - https://phabricator.wikimedia.org/T174385#3561138 (10Marostegui) 05Open>03Resolved a:03Marostegui I have merged Reedy's patch (https://gerrit.wikimedia.org/r/374384) and restarted MySQL on both sanitarium db1095 and db1102 and th... [06:35:40] PROBLEM - Puppet errors on tools-exec-1417 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:36:59] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1420 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:10:42] RECOVERY - Puppet errors on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [07:11:40] bd808: -bash: /usr/bin/svn: No such file or directory [07:12:00] From /usr/bin/jsub -N cron-tools.iabot-1 -once -quiet svn export --force https://github.com/cyberpower678/Cyberbot_II/trunk/IABot/www/i18n /mnt/nfs/labstore-secondary-tools-project/iabot/public_html/i18n [07:16:59] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [08:50:50] 10wikitech.wikimedia.org, 10Wikimedia-Extension-setup, 10I18n: Install Translate extension on wikitech - https://phabricator.wikimedia.org/T100313#3561312 (10Ladsgroup) a:05Ladsgroup>03None [09:07:58] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:43:00] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [11:48:17] 10cloud-services-team, 10DBA, 10Operations, 10Scoring-platform-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3561691 (10Marostegui) How do you guys want to proceed with this in the end? Is it worth the risk? [12:24:43] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:32:53] PROBLEM - Puppet errors on tools-exec-1428 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:56:45] 10Tool-Attribution-Generator, 10TCB-Team: [AG] MediaViewer links not working - https://phabricator.wikimedia.org/T130614#2140734 (10Katja_Ullrich_WMDE) seems to be solved [12:58:23] 10Tool-Attribution-Generator, 10TCB-Team: [AG] Possibility to get attribution in MediaWiki-Syntax - https://phabricator.wikimedia.org/T130613#2140719 (10Katja_Ullrich_WMDE) same as T160347 [12:58:59] 10Tool-Attribution-Generator, 10TCB-Team: [AG] Possibility to create a permanent link to full attribution - https://phabricator.wikimedia.org/T130620#2140870 (10Katja_Ullrich_WMDE) what is a full attribution? [12:59:44] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [13:03:46] 10Tool-Attribution-Generator, 10TCB-Team, 10User-Addshore: [AG] MediaViewer links not working - https://phabricator.wikimedia.org/T130614#3561904 (10Addshore) 05Open>03Resolved a:03Addshore +1 [13:04:43] 10Tool-Attribution-Generator, 10TCB-Team: Attribution in Wikitext - https://phabricator.wikimedia.org/T160347#3561914 (10Addshore) [13:04:46] 10Tool-Attribution-Generator, 10TCB-Team: [AG] Possibility to get attribution in MediaWiki-Syntax - https://phabricator.wikimedia.org/T130613#3561916 (10Addshore) [13:05:33] 10Tool-Attribution-Generator, 10TCB-Team: [AG] MediaViewer links not working - https://phabricator.wikimedia.org/T130614#3561923 (10Addshore) a:05Addshore>03None [13:07:51] RECOVERY - Puppet errors on tools-exec-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [13:20:01] 10cloud-services-team (FY2017-18), 10Wikimedia-Blog-Content: Publish a blog post covering the Cloud Services rebranding process - https://phabricator.wikimedia.org/T170288#3561963 (10MelodyKramer) @bd808 Here is a draft: https://docs.google.com/a/wikimedia.org/document/d/1DL_j4ScxbLkzYzPmfUYIVaCMQKddqjfkXpfJ... [13:33:53] bd808: I looked at Striker. I note there's no selection for "GPL 2.0 or any later version" in the license dropdown. [13:34:47] I also couldn't put CC0 as the license on oauth-hello-world, but that's not too important. [13:43:22] !help No module named 'pymysql' [13:43:22] JarBot: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [13:43:40] in python3 [13:44:15] just in crontab [13:45:29] 10Data-Services, 10DC-Ops, 10Operations, 10ops-codfw: Split up labstore external shelf storage available in codfw between labstore2001 and 2 - https://phabricator.wikimedia.org/T171623#3562003 (10Papaul) @madhuvishy here is what I am about to setup on Labstore2001 3xRAID10 of 8 disks per logical/virtua... [13:56:41] JarBot: it looks to me like python3-pymysql is installed on the tools bastion but not on exec nodes. So that would produce the behavior you're seeing. [13:57:27] I don't know the reason for that inconsistency though. The quick solution (?) would be to use python 2, otherwise you can open a ticket requesting addition of python3 on exec nodes and we'll discuss/research the possibility. [13:59:29] Thank you [13:59:56] JarBot: if you open a phab ticket let me know and I'll make sure the tags get set up correctly [14:00:42] https://phabricator.wikimedia.org/maniphest/task/edit/174439/ [14:01:50] 10Toolforge, 10cloud-services-team (Kanban): No module named 'pymysql' -- python3-mysql installed on toolforge bastions but not on exec nodes - https://phabricator.wikimedia.org/T174439#3562064 (10Andrew) [14:24:30] 10PAWS: Debugging notebook cell action/state - https://phabricator.wikimedia.org/T173416#3562109 (10Jprorama) I have a test environment with a 1G memory limit and can reliably make the jupyter kernel fail in this environment at the gridsearch call as a result of limited memory (VM with 1G and running Jupyter in... [14:35:39] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3562150 (10Cmjohnson) HP requested the AHS log, uploaded the log to their system. Waiting on their response. Only working with 1006 at the moment. [14:38:28] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1418 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:08:28] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [15:08:43] anomie: legoktm mentioned the GPL+ issue too. I didn't want to make the list crazy to look at, but we probably should add them in. [15:09:06] CC0 isn't in the list because it's not an OSI approved license for software [15:13:52] hmmmm.... [[Module:Extension]] doesn't cover "+" variants either... [15:14:51] 10Striker: Add GPL+ license variants - https://phabricator.wikimedia.org/T174444#3562251 (10bd808) [15:17:11] 10Data-Services, 10DC-Ops, 10Operations, 10ops-codfw: Split up labstore external shelf storage available in codfw between labstore2001 and 2 - https://phabricator.wikimedia.org/T171623#3562267 (10madhuvishy) @Papaul Yup that's perfect, thanks! [15:40:10] 10Cloud-VPS (Project-requests): Request creation of suggestbot VPS project - https://phabricator.wikimedia.org/T174306#3562333 (10chasemp) p:05Triage>03Normal [15:52:16] 10Cloud-VPS (Project-requests): Request creation of suggestbot VPS project - https://phabricator.wikimedia.org/T174306#3557478 (10chasemp) +1 [16:12:37] 10cloud-services-team, 10DBA, 10Operations, 10Scoring-platform-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3562459 (10chasemp) a:03madhuvishy [16:34:59] 10cloud-services-team, 10DBA, 10Operations, 10Scoring-platform-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3562523 (10madhuvishy) @Marostegui We talked about this today in our meeting, and think that since we don't have significant user traffic moved over from 1001... [16:38:07] 10cloud-services-team, 10DBA, 10Operations, 10Scoring-platform-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3562543 (10bd808) I vote we close this as "resolved" with a note that 1001/3 have not been rebooted because of the fear of catastrophic hardware failure and t... [16:40:01] 10Cloud-VPS (Project-requests): Request creation of suggestbot VPS project - https://phabricator.wikimedia.org/T174306#3562545 (10Andrew) 05Open>03Resolved a:03Andrew I've created the 'suggestbot' project with initial project admin Nettrom. @nettrom, you can add additional members or admins via https://wi... [16:51:45] 10Data-Services, 10DC-Ops, 10Operations, 10ops-codfw: Split up labstore external shelf storage available in codfw between labstore2001 and 2 - https://phabricator.wikimedia.org/T171623#3562604 (10Papaul) @madhuvishy done [16:54:11] 10cloud-services-team, 10DBA, 10Operations, 10Scoring-platform-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3562639 (10Marostegui) >>! In T168584#3562523, @madhuvishy wrote: > @Marostegui We talked about this today in our meeting, and think that since we don't have... [17:00:38] !help Hi, I am interested in creating my own tool at https://tools.wmflabs.org for a non-WMF wiki. Is this possible? [17:00:38] Catalan: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [17:02:09] Catalan: can you explain what your tool does a bit more? [17:02:27] madhuvishy: the tool doesn't exist yet [17:02:29] I want to create it [17:02:45] Catalan: okay, what will the tool do? [17:04:04] The wiki that I am interested in developing it for is running MediaWiki version 1.25, which is before blocked users were able to edit their own talk pages. Therefore, I'm looking to create a tool based off of English Wikipedia UTRS but much, much less complicated [17:04:31] I've found that relying on email appeals are not adequate [17:06:22] Catalan: and which wiki is this? [17:06:46] I'd prefer not to give out the wiki name in a public channel as there has been a LOT of spam/vandalism recently [17:09:34] Catalan: I see, okay. Please fill out a membership request at https://toolsadmin.wikimedia.org/tools/membership/apply with these details. This data is viewed by admins only. In general, non-wmf wiki tools seems fine to me, as long as it's broadly movement related. [17:10:11] Can you define "movement related" [17:10:20] And I already explained what I want my tool to be [17:12:20] Catalan: "movement related" means does this wiki and tool promote Wikimedia's open knowledge mission [17:12:29] "non-WMF wiki" sounds like maybe not [17:12:49] Well, it's a tool for a wiki that is not associated with WMF but the tool itself will be based off of one that is used by WMF [17:12:52] Namely the English Wikipedia [17:13:11] our servers and support time are paid for by Wikimedia donations, so we really only want them to be used to help Wikimedia projects [17:13:57] madhuvishy: the applications for toolforge membership are public [17:14:14] ah I see, I assumed it goes into striker db [17:14:17] Okay, do you know of any other location where I can create my own tools for my independent needs? [17:14:20] I don't mean to "well actually" you on that, but its good to know [17:14:33] nope, that's good to know, thank you! [17:15:41] Catalan: Heroku has a free tier -- https://www.heroku.com/free -- and there are many relatively low cost VPS providers on the internet [17:29:27] 10cloud-services-team, 10Operations: Reboots of cloud servers - https://phabricator.wikimedia.org/T168445#3562862 (10madhuvishy) [17:29:29] 10cloud-services-team, 10DBA, 10Operations, 10Scoring-platform-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3562860 (10madhuvishy) 05Open>03Resolved 1001/3 have not been rebooted because of the fear of catastrophic hardware failure and their impending decomm. [17:44:21] 10Cloud-VPS: VPS Project wikidumpparse is using 795G at /home on NFS - https://phabricator.wikimedia.org/T174467#3562977 (10madhuvishy) [17:47:41] 10Cloud-VPS: VPS Project dumps is using 1.7T at /data/project on NFS - https://phabricator.wikimedia.org/T174468#3562988 (10madhuvishy) [17:48:19] 10Cloud-VPS: VPS Project dumps is using 1.7T at /data/project on NFS - https://phabricator.wikimedia.org/T174468#3563006 (10madhuvishy) p:05Triage>03Normal [17:48:31] 10Cloud-VPS: VPS Project wikidumpparse is using 795G at /home on NFS - https://phabricator.wikimedia.org/T174467#3563007 (10madhuvishy) p:05Triage>03Normal [17:52:06] 10Cloud-Services, 10Operations: Register to Wikitech - https://phabricator.wikimedia.org/T174469#3563019 (10Vacio) [17:57:18] 10Cloud-Services, 10Operations: Register to Wikitech - https://phabricator.wikimedia.org/T174469#3563019 (10madhuvishy) @Vacio Could you please elaborate on what the problem is? Did you try signing up to wikitech and did you run into an error? If so what? You can create a wikitech account at https://wikitech.w... [17:58:27] 10wikitech.wikimedia.org: Register to Wikitech - https://phabricator.wikimedia.org/T174469#3563058 (10bd808) @Vacio Wikitech uses a separate user database from the normal Wikimedia wikis. You will need to create a new account for use on wikitech. You can do this in either of two ways: * Go to https://wikitech.wi... [18:03:28] (03PS1) 10Legoktm: Renable bug stuff, I didn't mean to commit this [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/374595 [18:04:12] (03CR) 10Legoktm: [V: 032 C: 032] Renable bug stuff, I didn't mean to commit this [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/374595 (owner: 10Legoktm) [18:12:31] (03PS1) 10Legoktm: Get full list of archived extensions [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/374598 [18:12:39] (03CR) 10Legoktm: [V: 032 C: 032] Get full list of archived extensions [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/374598 (owner: 10Legoktm) [18:14:03] (03PS1) 10Legoktm: Look for skin.json for skins [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/374599 [18:14:11] (03CR) 10Legoktm: [V: 032 C: 032] Look for skin.json for skins [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/374599 (owner: 10Legoktm) [18:15:02] 10wikitech.wikimedia.org: Register to Wikitech - https://phabricator.wikimedia.org/T174469#3563187 (10Vacio) 05Open>03Resolved Okay sorry, for not being clear enough. If I try to create an account with my Wikimedia accout name (Vacio) I get the message that the username is already in use. When I try to log... [18:15:21] 10wikitech.wikimedia.org: Register to Wikitech - https://phabricator.wikimedia.org/T174469#3563191 (10Vacio) 05Resolved>03Open [18:20:53] (03PS1) 10Legoktm: Link to Phabricator in toolinfo.json [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/374600 [18:21:04] (03CR) 10Legoktm: [V: 032 C: 032] Link to Phabricator in toolinfo.json [labs/tools/extreg-wos] - 10https://gerrit.wikimedia.org/r/374600 (owner: 10Legoktm) [18:27:30] 10wikitech.wikimedia.org, 10Wikimedia-Video: Video thumbnails not working at wikitech - https://phabricator.wikimedia.org/T174476#3563246 (10Quiddity) [18:32:05] 10wikitech.wikimedia.org: Register to Wikitech - https://phabricator.wikimedia.org/T174469#3563266 (10madhuvishy) @Vacio You are in the right place! If you can hop on to the #wikimedia-cloud IRC channel sometime, we can help you figure this out easier real time :) [18:33:12] 10wikitech.wikimedia.org: Register to Wikitech - https://phabricator.wikimedia.org/T174469#3563269 (10Vacio) I will right now, thanks a lot (: [18:33:25] 10wikitech.wikimedia.org: Register to Wikitech - https://phabricator.wikimedia.org/T174469#3563270 (10Vacio) 05Open>03Resolved [18:34:01] Hi, everyone :) [18:34:08] hello! [18:34:47] I am writing a bot that feeds from Eventstreams, worked fine the past few weeks, but it seems today Eventstream isn't working [18:35:07] did anyone try to use it recently? [18:35:27] imandes: you can ping ottomata on #wikimedia-analytics who works on Eventstream [18:35:39] great tankx! [18:36:09] PROBLEM - Puppet errors on tools-exec-1404 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:36:55] An another issue. I want to register on wikitech.wikimedia.org with my unified wikimedia login (Vacio), bit I get the message that the username is already in use. When I try to login with the same username, I am told that it doesn't exist. If anyone has the time, can you please help me? :) [18:37:27] imandes: right, so I checked that you do already have an account (with a riseup email associated) [18:38:03] 10Cloud-Services, 10Wikimania-Hackathon-2017: History & Purpose of Wikimedia Cloud Services @ Wikimania Hackathon - https://phabricator.wikimedia.org/T170833#3563274 (10Quiddity) Note, this is on Commons at https://commons.wikimedia.org/wiki/File:Introduction_to_Wikimedia_Cloud_Services.webm I've added it to... [18:39:45] imandes: I'm not sure why it won't accept your login yet, I'm hoping when bd808 is back from lunch he can help look into the wikitech logs [18:40:14] great, thanks. I tried to restore password by email, but it didn't work. [18:41:06] imandes: just to check, can you try logging into https://toolsadmin.wikimedia.org/ with the same account and see if it has the same behavior? [18:43:18] yes, it doesn't recognize my username/pssw [18:46:15] imandes: okay, thanks for checking [18:47:36] @madhuvishy thank you :) [18:50:44] * bd808 looks at ticket and server logs [18:52:03] imandes: You are the Vacio LDAP user? [18:53:10] vacio: :) So I see in the LDAP logs that your password doesn't match what the LDAP directory thinks it should be. There are 11 failed password attempts [18:53:40] can I change my password? [18:54:34] vacio: yes, with https://wikitech.wikimedia.org/wiki/Special:PasswordReset [18:56:06] madhuvishy: the way I found this is by searching LDAP for cn=Vacio and asking ldapsearch to show the system data ('+'). With my ldapsearch alias this looks like `ldap cn=Vacio +` [18:56:20] I tried, it thinks that my username doesn't exist [18:56:28] "There is no user by the name "Vacio". Usernames are case sensitive. Check your spelling, or create a new account." [18:56:57] Did you create your account using toolsadmin.wikimedia.org? [18:57:16] if so this may be a really dumb bug on my part :/ [18:57:32] I don't remember frankly [18:57:57] but when I try to login there, it says "Wikimedia unified account is already in use." [18:58:28] yeah. I see you have a record in toolsadmin's local db [18:59:35] So the problem is that you do have an LDAP account but that account has not been attached to wikitech's MediaWiki user database [18:59:39] bd808: cool, didn't know about the + thing [18:59:43] and of course I haven [18:59:56] haven't added password recovery support to Striker yet :/ [19:00:27] so ... we need to attach your account to wikitech so that you can use the password recovery there to fix your password [19:01:02] 10wikitech.wikimedia.org: Register to Wikitech - https://phabricator.wikimedia.org/T174469#3563356 (10bd808) 05Resolved>03Open [19:01:06] bd808: should creating the ldap account through Toolsadmin have already done that association? [19:01:31] bd808 hope that's not much hassle for you [19:11:12] RECOVERY - Puppet errors on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [19:11:41] 10wikitech.wikimedia.org: LDAP account that is not attached on wikitech has no means for password reset - https://phabricator.wikimedia.org/T174469#3563462 (10bd808) p:05Triage>03High [19:12:29] 10wikitech.wikimedia.org: LDAP account that is not attached on wikitech has no means for password reset - https://phabricator.wikimedia.org/T174469#3563467 (10Framawiki) a:05Vacio>03None [19:16:42] 10wikitech.wikimedia.org: LDAP account that is not attached on wikitech has no means for password reset - https://phabricator.wikimedia.org/T174469#3563491 (10bd808) Task description has been updated to reflect debugging done over irc. We need to: * Figure out how to attach this LDAP account to Wikitech so that... [19:34:46] 10Cloud-Services, 10cloud-services-team: Request for additional edit permissions on wikitech.wikimedia.org for dr0ptp4kt - https://phabricator.wikimedia.org/T174488#3563595 (10dr0ptp4kt) [19:35:19] 10cloud-services-team, 10wikitech.wikimedia.org: Request for additional edit permissions on wikitech.wikimedia.org for dr0ptp4kt - https://phabricator.wikimedia.org/T174488#3563610 (10dr0ptp4kt) [19:35:48] 10Cloud-Services, 10cloud-services-team: Request for additional edit permissions on wikitech.wikimedia.org for dr0ptp4kt - https://phabricator.wikimedia.org/T174488#3563595 (10dr0ptp4kt) [19:39:31] 10Cloud-Services, 10cloud-services-team: Request for additional edit permissions on wikitech.wikimedia.org for dr0ptp4kt - https://phabricator.wikimedia.org/T174488#3563595 (10Krenair) What kind of edits would you like to make? [19:45:56] 10Cloud-Services, 10cloud-services-team: Request for additional edit permissions on wikitech.wikimedia.org for dr0ptp4kt - https://phabricator.wikimedia.org/T174488#3563649 (10Quiddity) Editing protected pages like https://wikitech.wikimedia.org/w/index.php?title=Help:SSH_Fingerprints&diff=prev&oldid=1768984 [19:58:56] bd808: is mail forwarding already configured for VPS projects, or just toolforge? [20:07:31] 10PAWS: HTTP 404 error (or sometimes 502 error) when trying to access the Control Panel of PAWS - https://phabricator.wikimedia.org/T140525#3563731 (10yuvipanda) Is this still the case with paws.wmflabs.org now? It's running on a completely different backend! [20:08:45] legoktm: I actually don't know. [20:09:46] hm [20:12:24] I guess I could just try emailing admin@library-upgrader.wmflabs.org? [20:13:50] An error occurred while sending mail. The mail server responded: [20:13:51] 4.1.2 : Recipient address rejected: Domain not found. [20:13:51] Please check the message recipient "admin@library-upgrader.wmflabs.org" and try again. [20:14:02] I guess that's a nope. [20:23:01] PROBLEM - Puppet errors on tools-exec-1433 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:27:05] 10PAWS: HTTP 404 error (or sometimes 502 error) when trying to access the Control Panel of PAWS - https://phabricator.wikimedia.org/T140525#3563852 (10Strainu) @yuvipanda: Yes, absolutely, I could just reproduce it. I lauched Strainu's server, then tried to log in as Strainubot and I get a "500 : Internal Serve... [20:57:59] RECOVERY - Puppet errors on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [20:59:26] legoktm: I *think* that the mail magic for toolforge is handled by a project local mail server. [20:59:45] pretty sure there isn't an equivalent for the rest of the projects out of the box [21:00:01] not sure at all what it would take to set one up in a project of globally [21:00:36] *in a project OR globally [21:01:21] 10Cloud-Services, 10cloud-services-team: Request for additional edit permissions on wikitech.wikimedia.org for dr0ptp4kt - https://phabricator.wikimedia.org/T174488#3563995 (10dr0ptp4kt) Yeah, the line should probably say: ``` If that doesn't work, you might try using ssh -o FingerprintHash=md5 a... [21:28:51] PROBLEM - Puppet errors on tools-exec-1428 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [22:08:51] RECOVERY - Puppet errors on tools-exec-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [22:27:10] 10wikitech.wikimedia.org, 10Wikimedia-Video: Video thumbnails not working at wikitech - https://phabricator.wikimedia.org/T174476#3563246 (10bd808) Related, but maybe not root cause: ``` [072ca6f7ea0e4d7ec4234ae8] /w/index.php?title=File:Introduction_to_Wikimedia_Cloud_Services.webm&action=purge ErrorExcepti... [22:33:14] 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10User-bd808: LDAP account that is not attached on wikitech has no means for password reset - https://phabricator.wikimedia.org/T174469#3564381 (10bd808) a:03bd808 [23:29:46] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3564554 (10Cmjohnson) Spoke with HP support, not very helpful. They will not send anyone to help unless we want to pay for it. Going to try and talk...