[00:30:41] PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [00:31:05] PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [00:50:54] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labvirt10(19|20).eqiad.wmnet - https://phabricator.wikimedia.org/T172538#3564686 (10Cmjohnson) [01:06:09] RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0] [01:26:11] 10Cloud-VPS: VPS Project dumps is using 1.7T at /data/project on NFS - https://phabricator.wikimedia.org/T174468#3564713 (10Hydriz) Thank you for the notice. This issue was caused by an archiving job for the CirrusSearch dumps which did not exit cleanly. I am in the midst of resolving this issue and will update... [01:28:14] 10Cloud-VPS: VPS Project dumps is using 1.7T at /data/project on NFS - https://phabricator.wikimedia.org/T174468#3562988 (10EBernhardson) Is there something we could do to make it more clear when the dumps are succesfull or not, perhaps emitting some sort of tag file next to the dump when its succesfull? [01:54:04] PROBLEM - Puppet errors on tools-exec-1413 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:57:04] 10Cloud-VPS: VPS Project dumps is using 1.7T at /data/project on NFS - https://phabricator.wikimedia.org/T174468#3564721 (10Hydriz) @EBernhardson Having a `status.txt` file with a simple text of "done" or "in progress" would be very ideal for us to know if a dump is successfully completed. If you think this is a... [02:34:03] RECOVERY - Puppet errors on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [02:40:42] RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [03:06:39] PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [03:41:40] RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [03:58:30] PROBLEM - Puppet errors on tools-exec-1435 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [04:02:41] PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [04:07:55] bd808: I'm just going to create a tool for the mail account then [04:08:18] that works :) [04:18:31] RECOVERY - Puppet errors on tools-exec-1435 is OK: OK: Less than 1.00% above the threshold [0.0] [04:42:41] RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [04:58:41] PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [05:31:03] (03PS1) 10Legoktm: Add git config [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374678 [05:31:05] (03PS1) 10Legoktm: Move COPY line down so steps after it are cacheable [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374679 [05:31:07] (03PS1) 10Legoktm: Port thing.sh to Python [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374680 [05:31:58] (03CR) 10Legoktm: [C: 032] Add git config [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374678 (owner: 10Legoktm) [05:32:02] (03CR) 10Legoktm: [C: 032] Move COPY line down so steps after it are cacheable [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374679 (owner: 10Legoktm) [05:32:04] (03CR) 10Legoktm: [C: 032] Port thing.sh to Python [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374680 (owner: 10Legoktm) [05:32:16] (03Merged) 10jenkins-bot: Add git config [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374678 (owner: 10Legoktm) [05:32:20] (03Merged) 10jenkins-bot: Move COPY line down so steps after it are cacheable [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374679 (owner: 10Legoktm) [05:32:22] (03Merged) 10jenkins-bot: Port thing.sh to Python [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374680 (owner: 10Legoktm) [05:37:46] (03PS1) 10Legoktm: Remove unnecessary newline [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374681 [05:38:06] (03CR) 10Legoktm: [C: 032] Remove unnecessary newline [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374681 (owner: 10Legoktm) [05:38:26] (03Merged) 10jenkins-bot: Remove unnecessary newline [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374681 (owner: 10Legoktm) [05:43:37] 10Data-Services, 10Operations, 10Tracking: overhaul labstore setup [tracking] - https://phabricator.wikimedia.org/T126083#3564855 (10madhuvishy) [05:43:39] 10Data-Services, 10DC-Ops, 10Operations, 10ops-codfw: Split up labstore external shelf storage available in codfw between labstore2001 and 2 - https://phabricator.wikimedia.org/T171623#3564853 (10madhuvishy) 05Open>03Resolved Thank you so much, that all looks right. Closing this as resolved! [06:38:13] (03PS1) 10Legoktm: Introduce concept of "modes" [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374686 [06:38:16] (03PS1) 10Legoktm: Update codesniffer versions being pre-cached [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374687 [06:38:17] (03PS1) 10Legoktm: Add some lib functions to thing from tools-ci [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374688 [06:38:45] RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [06:39:25] (03PS1) 10Legoktm: Tag the docker image as "libraryupgrader' [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374689 [06:40:47] (03CR) 10Legoktm: [C: 032] Introduce concept of "modes" [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374686 (owner: 10Legoktm) [06:40:51] (03CR) 10Legoktm: [C: 032] Update codesniffer versions being pre-cached [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374687 (owner: 10Legoktm) [06:40:56] (03CR) 10Legoktm: [C: 032] Add some lib functions to thing from tools-ci [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374688 (owner: 10Legoktm) [06:40:59] (03CR) 10Legoktm: [C: 032] Tag the docker image as "libraryupgrader' [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374689 (owner: 10Legoktm) [06:41:06] (03Merged) 10jenkins-bot: Introduce concept of "modes" [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374686 (owner: 10Legoktm) [06:41:10] (03Merged) 10jenkins-bot: Update codesniffer versions being pre-cached [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374687 (owner: 10Legoktm) [06:41:14] (03Merged) 10jenkins-bot: Add some lib functions to thing from tools-ci [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374688 (owner: 10Legoktm) [06:41:18] (03Merged) 10jenkins-bot: Tag the docker image as "libraryupgrader' [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374689 (owner: 10Legoktm) [06:49:37] 10Cloud-Services, 10Tools, 10Community-Tech-Tool-Labs, 10Developer-Relations, and 3 others: Create an authoritative and well promoted catalog of Wikimedia tools - https://phabricator.wikimedia.org/T115650#3564924 (10Qgil) I wonder whether the progress on [[ https://lists.wikimedia.org/pipermail/labs-l/2017... [06:54:36] (03PS1) 10Legoktm: Run composer require before composer update to avoid lock issues [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374699 [06:55:57] (03CR) 10Legoktm: [C: 032] Run composer require before composer update to avoid lock issues [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374699 (owner: 10Legoktm) [06:56:16] (03Merged) 10jenkins-bot: Run composer require before composer update to avoid lock issues [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374699 (owner: 10Legoktm) [06:57:36] (03PS1) 10Legoktm: Add shortcut for deleting all docker containers [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374700 [06:57:44] (03CR) 10Legoktm: [C: 032] Add shortcut for deleting all docker containers [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374700 (owner: 10Legoktm) [06:58:02] (03Merged) 10jenkins-bot: Add shortcut for deleting all docker containers [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/374700 (owner: 10Legoktm) [08:13:58] (03PS1) 10Reception123: Add myself to contacts for adding ZppixBot to channels. [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/374710 [08:14:15] (03CR) 10Reception123: [C: 032] Add myself to contacts for adding ZppixBot to channels. [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/374710 (owner: 10Reception123) [08:14:23] (03Merged) 10jenkins-bot: Add myself to contacts for adding ZppixBot to channels. [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/374710 (owner: 10Reception123) [08:34:42] PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:14:42] RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [09:22:45] 10Data-Services, 10DBA: Design a method for keeping user-created tables in sync across labsDBs - https://phabricator.wikimedia.org/T156869#3565126 (10Marostegui) I agree with @bd808 too - let's continue the discussion on T173511. [09:30:40] PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:59:52] PROBLEM - Puppet errors on tools-worker-1019 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:05:41] RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [10:39:52] RECOVERY - Puppet errors on tools-worker-1019 is OK: OK: Less than 1.00% above the threshold [0.0] [11:19:06] 10cloud-services-team, 10DBA, 10Operations, 10Scoring-platform-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3369055 (10faidon) JFTR, since I didn't see it mentioned neither here nor in T142807, how impending is that decomm? Days/weeks/months? [11:25:30] 10Quarry: CHAR_LENGTH does not return the character count - https://phabricator.wikimedia.org/T174543#3565363 (10WhitePhosphorus) [11:33:49] 10Cloud-VPS (Quota-requests), 10Recommendation-API: Request custom instance for recommendation-api labs project - https://phabricator.wikimedia.org/T169766#3565383 (10schana) [13:24:24] (03PS1) 10Melos: Convert local temp user rights string as utf-8 string [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/374812 [13:24:49] (03CR) 10jerkins-bot: [V: 04-1] Convert local temp user rights string as utf-8 string [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/374812 (owner: 10Melos) [13:25:33] Technical Advice IRC meeting starting at 3 pm UTC/5 pm CEST in channel #wikimedia-tech, hosts: @addshore & @CFisch_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [13:30:29] (03PS2) 10Melos: Convert local temp user rights string as utf-8 string [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/374812 [13:32:19] (03CR) 10Melos: [C: 032] Convert local temp user rights string as utf-8 string [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/374812 (owner: 10Melos) [13:32:42] (03Merged) 10jenkins-bot: Convert local temp user rights string as utf-8 string [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/374812 (owner: 10Melos) [13:39:50] 10Cloud-VPS: VPS Project dumps is using 1.7T at /data/project on NFS - https://phabricator.wikimedia.org/T174468#3565708 (10Nemo_bis) Thanks madhuvishy for the notification. Hydriz, how long will it take to recover the backlog? If bandwidth is the bottleneck, do we need our bandwidth cap to be temporarily lifted? [13:52:43] 10Cloud-VPS: VPS Project dumps is using 1.7T at /data/project on NFS - https://phabricator.wikimedia.org/T174468#3565760 (10Hydriz) I should be able to free up at least 1 TB by the end of this week, which is about 3 of the CirrusSearch dumps. The remaining 2 dumps will take a while to complete, so the storage sp... [14:40:16] 10Cloud-VPS, 10Mail: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#3566009 (10faidon) a:03herron This is a WMCS task, but since this use case is currently supported by the production mailservers and that's a long-standing problem (and risk) for us, perhaps it's still worth it for pr... [14:40:37] 10Cloud-Services, 10Mail, 10Operations: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#3566017 (10faidon) [14:58:41] !log wikilabels Created edit quality campaign for hrwiki (T172047) [14:58:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [14:58:45] T172047: Edit quality campaign for hr.wiki - https://phabricator.wikimedia.org/T172047 [15:06:46] 10Cloud-Services, 10Tools, 10Community-Tech-Tool-Labs, 10Developer-Relations, and 3 others: Create an authoritative and well promoted catalog of Wikimedia tools - https://phabricator.wikimedia.org/T115650#3566101 (10bd808) >>! In T115650#3564924, @Qgil wrote: > I wonder whether the progress on [[ https://l... [15:09:14] 10Cloud-Services, 10Tools, 10Community-Tech-Tool-Labs, 10Developer-Relations, and 3 others: Create an authoritative and well promoted catalog of Wikimedia tools - https://phabricator.wikimedia.org/T115650#3566118 (10MelodyKramer) I volunteer to help with any kind of content needs for this project, whether... [15:11:36] !log wikilabels bac73b9 is going to staging [15:11:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [15:11:39] 10Striker, 10User-bd808: Can’t log into Striker, potentially because of accents in logins - https://phabricator.wikimedia.org/T172949#3566129 (10JeanFred) So I tried a couple of days ago (can’t remember when sorry :-( ) and it was still failing ; I tried just now and it worked. Looks like this can be closed t... [15:13:05] works just fine, going to prod [15:13:36] !log wikilabels deploying bac73b9 to prod (T174343) [15:13:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [15:13:39] T174343: Wikilabels translations not updated and cannot load old worksets - https://phabricator.wikimedia.org/T174343 [15:31:42] 10Cloud-Services, 10Tools, 10Community-Tech-Tool-Labs, 10Developer-Relations, and 3 others: Create an authoritative and well promoted catalog of Wikimedia tools - https://phabricator.wikimedia.org/T115650#3566291 (10bd808) Cool directories for external collections that @MelodyKramer showed me: * https://di... [15:32:28] 10Data-Services, 10DBA, 10Dumps-Generation, 10MediaWiki-Platform-Team, 10Blocked-on-schema-change: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3566292 (10jcrespo) I am going to assume that dumps generation is backwards compatible, but adding it here as a heads... [15:41:21] 10Data-Services, 10DBA: LabsDB infrastructure pending work - https://phabricator.wikimedia.org/T153058#3566345 (10bd808) [15:41:24] 10Data-Services, 10DBA: Design a method for keeping user-created tables in sync across labsDBs - https://phabricator.wikimedia.org/T156869#3566343 (10bd808) 05Open>03declined Per rationale explained in T156869#3559945, this feature request is being denied. Write access will **not** be granted to end-user a... [15:43:43] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Promote beta test of new Wiki Replica servers - https://phabricator.wikimedia.org/T172704#3566365 (10bd808) [15:44:14] 10Cloud-Services, 10Mail, 10Operations: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#3566374 (10herron) For sure! Maybe a pair of instances in different locations for durability? Regarding handling root@ mail for individual instances. If it would be useful for some projects to h... [16:08:11] 10Cloud-Services, 10Mail, 10Operations: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#3566527 (10faidon) Indeed! Note that ToolForge already has something like that for tool authors that does LDAP calls etc. if I recall correctly, so perhaps these two efforts could complement each o... [16:17:37] 10Cloud-Services, 10Mail, 10Operations: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#434985 (10chasemp) That Toolforge mail server is a real mess. It maybe the remaining holdover from the early days of un-puppetized things and we have been kicking that can down the road for a good... [16:19:06] legoktm: hey look! That project email thing you were asking about has a task -- T41785 [16:19:07] T41785: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785 [16:43:00] 10cloud-services-team, 10DBA, 10Operations, 10Scoring-platform-team: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3369055 (10chasemp) No definite date has been set as we are working on T173511 as the precursor to moving over quarry (and probably PAWS). I think we are tal... [17:20:40] bd808: oh, sweet [17:22:59] legoktm: if you have requirements I bet Keith would read them at least :) [17:29:53] 10PAWS, 10Documentation: Document reasons to use stat machines rather than PAWS for analysis - https://phabricator.wikimedia.org/T146949#3566936 (10Quiddity) [17:49:29] 10Cloud-Services, 10Mail, 10Operations: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#434985 (10Legoktm) I was actually just asking @bd808 about the status of mail access for Cloud VPS projects yesterday. At least for my use case, it would be great if mail to @ {{done}} [17:49:47] 10Cloud-VPS: VPS Project dumps is using 1.7T at /data/project on NFS - https://phabricator.wikimedia.org/T174468#3567035 (10madhuvishy) Thanks @Hydriz @Nemo_bis! Do keep the ticket updated as the storage space gets freed up. [17:58:12] 10Striker, 10User-bd808: Can’t log into Striker, potentially because of accents in logins - https://phabricator.wikimedia.org/T172949#3567071 (10bd808) 05Open>03Invalid Solved by ... unknown. [18:04:52] 10Cloud-Services, 10cloud-services-team (FY2017-18): Revert patch that adds a temporary exception to the block-for-export check for the testlabs project - https://phabricator.wikimedia.org/T174590#3567105 (10madhuvishy) [18:17:21] 10Cloud-VPS, 10Operations, 10netops: dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour - https://phabricator.wikimedia.org/T174596#3567207 (10Krenair) [18:18:45] 10Cloud-VPS, 10Operations, 10netops: dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour - https://phabricator.wikimedia.org/T174596#3567234 (10Krenair) [18:19:56] 10Cloud-VPS, 10Operations, 10netops: dmz_cidr only includes some wikimedia public IP ranges, leading to some very strange behaviour - https://phabricator.wikimedia.org/T174596#3567207 (10Krenair) See also T167357 where this task will probably become obsolete, I just wanted to document the effect of this really. [18:21:52] 10Cloud-Services, 10Mail, 10Operations: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#3567268 (10bd808) >>! In T41785#3567033, @Legoktm wrote: > it would be great if mail to @.wmflabs.org (where something is fixed string(s) or any string) just worked for forw... [18:25:07] 10Cloud-Services, 10Mail, 10Operations: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#3567277 (10Krenair) >>! In T41785#3566374, @herron wrote: > For sure! Maybe a pair of instances in different locations for durability? You mean on separate physical hosts, right? I think we're st... [18:29:14] 10Cloud-Services, 10Mail, 10Operations: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#3567298 (10Krenair) >>! In T41785#3567268, @bd808 wrote: > Next tricky step is that `.wmflabs.org` does not exist in DNS by default. We could just say these projects can't get mail un... [18:40:51] 10Data-Services, 10cloud-services-team, 10wikitech.wikimedia.org, 10DBA: move wikitech and labstestwiki to s3 (needs discussion) - https://phabricator.wikimedia.org/T167973#3351594 (10Krenair) > I cannot think any reason why wikitech's database (and the similar labstestwiki) is not part of s3 Historically,... [18:41:27] 10cloud-services-team, 10wikitech.wikimedia.org, 10DBA: Rename database labswiki to wikitech - https://phabricator.wikimedia.org/T171570#3469261 (10Krenair) Surely it'd be wikitechwiki anyway :p [18:44:03] 10Cloud-Services, 10Cloud-VPS, 10cloud-services-team (Kanban): monitor some things on all Cloud instances (discussion) - https://phabricator.wikimedia.org/T166845#3309520 (10Krenair) Like shinken? [19:05:29] 10Cloud-Services, 10Mail, 10Operations: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#3567441 (10herron) >! In T41785#3567277, @Krenair wrote: > > You mean on separate physical hosts, right? I think we're still limited to eqiad if it is to be a Cloud VPS instance :) Nah, this does... [19:06:30] 10Cloud-Services, 10Mail, 10Operations: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#3567458 (10Krenair) >>! In T41785#3567441, @herron wrote: >>! In T41785#3567277, @Krenair wrote: >> >> You mean on separate physical hosts, right? I think we're still limited to eqiad if it is to... [19:08:41] 10Cloud-Services, 10Mail, 10Operations: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#3567462 (10faidon) Also see T47827, T47828, T47829 and T61142. This task is supposed to be for the smarthost which sounds like a good first step. I'd recommend keeping separate instances for inboun... [19:15:59] 10Cloud-Services, 10DBA, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#3567484 (10Base) Any news? [19:18:36] 10Horizon: Add a note to the launch instance window about instance names being global - https://phabricator.wikimedia.org/T147859#3567499 (10Krenair) upstream? [19:27:28] 10Data-Services, 10DBA, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#3567557 (10Reedy) [19:28:26] 10Data-Services, 10DBA, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Replicate CentralNotice tables to Labs - https://phabricator.wikimedia.org/T135405#2297843 (10Reedy) [19:30:26] (03CR) 10Jean-Frédéric: [C: 032] Allow skipping some configs during full harvest. [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/374403 (https://phabricator.wikimedia.org/T174334) (owner: 10Lokal Profil) [19:31:27] (03CR) 10Jean-Frédéric: [C: 032] "> Do we document allowed monuments_config variables or command line" [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/374403 (https://phabricator.wikimedia.org/T174334) (owner: 10Lokal Profil) [19:32:17] (03Merged) 10jenkins-bot: Allow skipping some configs during full harvest. [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/374403 (https://phabricator.wikimedia.org/T174334) (owner: 10Lokal Profil) [19:33:48] (03CR) 10Jean-Frédéric: "> Would it be an idea to dump the report to a log?" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/373880 (owner: 10Jean-Frédéric) [19:34:49] (03CR) 10Jean-Frédéric: "> Would it be an idea to dump the report to a log?" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/373880 (owner: 10Jean-Frédéric) [19:35:20] 10Cloud-Services, 10Epic: Improve SMTP inbound/outbound services - https://phabricator.wikimedia.org/T174608#3567587 (10bd808) [19:36:18] 10Cloud-Services, 10Cloud-VPS: Labs: Let mail (from cron and perhaps other defaults) go to project forwarder by default - https://phabricator.wikimedia.org/T47829#3567605 (10bd808) [19:36:21] 10Cloud-Services, 10Cloud-VPS: Labs: Implement mail aliases for projects (@wmflabs.org) - https://phabricator.wikimedia.org/T47828#3567606 (10bd808) [19:36:22] 10Cloud-Services, 10Cloud-VPS: Labs: Mails from cron should not go to ops-wikimedia - https://phabricator.wikimedia.org/T47827#3567607 (10bd808) [19:36:24] 10Cloud-Services, 10Epic: Improve SMTP inbound/outbound services - https://phabricator.wikimedia.org/T174608#3567603 (10bd808) [19:36:26] 10Cloud-Services, 10Mail, 10Operations: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#3567608 (10bd808) [19:37:42] 10Data-Services, 10DBA: enwiki_p logging vs logging_userindex returning dramatically different results - https://phabricator.wikimedia.org/T168349#3567619 (10Krenair) [19:37:59] 10Cloud-Services, 10Cloud-VPS: Labs: Implement mail aliases for projects (@wmflabs.org) - https://phabricator.wikimedia.org/T47828#503968 (10bd808) >>! In T41785#3567033, @Legoktm wrote: > I was actually just asking @bd808 about the status of mail access for Cloud VPS projects yesterday. At least... [19:38:54] (03CR) 10jenkins-bot: Allow skipping some configs during full harvest. [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/374403 (https://phabricator.wikimedia.org/T174334) (owner: 10Lokal Profil) [19:39:23] 10Cloud-Services, 10Mail: Support per project user email address $project-admins@wmflabs.org - https://phabricator.wikimedia.org/T166349#3567632 (10Krenair) [19:40:39] 10Cloud-Services, 10Mail, 10Operations: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#3567639 (10bd808) Should we start a #cloud-vps project in the spirit of [[https://tools.wmflabs.org/openstack-browser/project/bastion|bastion ]] to host VMs for inbound/outbound SMTP services? [19:43:07] 10Cloud-Services, 10Gerrit: Deactivate repository labs/invisible-unicorn - https://phabricator.wikimedia.org/T154099#3567645 (10Krenair) [19:46:27] 10Cloud-Services, 10Puppet: Migrate references from $instance.eqiad.wmflabs to $instance.$project.eqiad.wmflabs - https://phabricator.wikimedia.org/T153608#3567672 (10Krenair) [19:47:39] 10Cloud-Services, 10Operations: Missing Labs hiera entry in labs-private repo - https://phabricator.wikimedia.org/T152767#3567673 (10Krenair) [19:47:46] 10cloud-services-team, 10Operations: update *.wmflabs.org by 2017-10-16 - https://phabricator.wikimedia.org/T174611#3567674 (10RobH) [19:48:02] bd808: ^ so i think that is the best way to hand over the cert update to cloud team =] [19:48:26] i have it all staged just ready for someone in cloud team and with +2 in operations and puppetmaster access to merge them live. [19:48:40] since it will affect your services and all, seemed something to hand off for implementation. [19:50:10] 10Cloud-Services: Determine need and replacement for dmz_cidr configuration in nova-network - https://phabricator.wikimedia.org/T167357#3567690 (10chasemp) @ayounsi and I spoke about this for a few minutes today. General agreement that the allowance here are too broad and even if we wanted to keep them incomple... [19:52:04] robh: awesome! I'll get it on our tracking board [19:52:11] 10cloud-services-team, 10Analytics: Remove logging from labs for schema https://meta.wikimedia.org/wiki/Schema:CommandInvocation - https://phabricator.wikimedia.org/T166712#3305259 (10Krenair) Ping @Nuria [19:52:28] 10cloud-services-team (Kanban), 10Operations: update *.wmflabs.org by 2017-10-16 - https://phabricator.wikimedia.org/T174611#3567702 (10bd808) [19:56:31] bd808: was just tagging with #cloud-services-team (not kanban) the right thing to do? [19:56:46] 10Cloud-Services, 10Mail, 10Operations: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#3567720 (10Krenair) I think so [19:56:48] just making sure i triage stuff to the team correctly =] [19:57:16] robh: yes. It's nicest for us to pull things into the "do this shit soon" board ourselves. :) [19:57:17] since the notes of th eother project (kanban) state its for your team to add to things [19:57:19] cool [19:57:28] just wanted to make sure! [19:57:53] traffic, cloud, and db team all have a project for triage in the same way, its nice. [19:58:05] robh: all things you do are awesome and the irc pings are most helpful of all [19:58:07] (so does dc ops for each site, but its been that way forever) [19:58:14] 10Cloud-Services, 10Mail: Support per project user email address $project-admins@wmflabs.org - https://phabricator.wikimedia.org/T166349#3567723 (10Krenair) [19:58:23] ok, i just dont wanna be bugging you guys too much, nt everyone likes irc pings, heh [19:58:40] also updating clinic docs so others follow procedure. [20:01:57] 10Cloud-Services, 10Cloud-VPS: Labs: Implement mail aliases for projects (@wmflabs.org) - https://phabricator.wikimedia.org/T47828#3567742 (10bd808) [20:01:59] 10Cloud-Services, 10Mail: Support per project user email address $project-admins@wmflabs.org - https://phabricator.wikimedia.org/T166349#3567744 (10bd808) [20:03:23] (03CR) 10Lokal Profil: [C: 031] "Looks fine but I'm to tired to understand if you meant the flipping mechanism will go in a separate review or as another patch set on this" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/373880 (owner: 10Jean-Frédéric) [20:06:56] 10Horizon: Horizon prefix puppet dialog puts you in wrong prefix after you create a new prefix - https://phabricator.wikimedia.org/T150828#3567762 (10Krenair) [20:08:03] (03CR) 10Jean-Frédéric: [C: 032] "> Looks fine but I'm to tired to understand if you meant the flipping" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/373880 (owner: 10Jean-Frédéric) [20:09:01] (03Merged) 10jenkins-bot: Do not crash when unkwnown_fields report cannot be created [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/373880 (owner: 10Jean-Frédéric) [20:09:52] (03CR) 10jenkins-bot: Do not crash when unkwnown_fields report cannot be created [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/373880 (owner: 10Jean-Frédéric) [20:15:40] 10Cloud-VPS: Implement mail aliases for Cloud-VPS projects (@wmflabs.org) - https://phabricator.wikimedia.org/T47828#3567801 (10bd808) [20:21:45] 10Cloud-VPS, 10cloud-services-team: dss keys disabled prematurely - https://phabricator.wikimedia.org/T171136#3455186 (10Krenair) I just tested this and it seems to work on all the non-stretch (stretch has a newer sshd that disables it by default) instances that I tried. Where did you test? [20:28:14] 10Cloud-Services, 10Cloud-VPS, 10cloud-services-team (Kanban): monitor some things on all Cloud instances (discussion) - https://phabricator.wikimedia.org/T166845#3567890 (10Andrew) >>! In T166845#3567349, @Krenair wrote: > Like shinken? Yes! But as I understand it Shinken currently only monitors select pr... [20:29:17] 10Cloud-Services, 10Cloud-VPS, 10cloud-services-team (Kanban): monitor some things on all Cloud instances (discussion) - https://phabricator.wikimedia.org/T166845#3309520 (10chasemp) SSH availability via whatever new and fancy Cumin things get setup seems idea. [21:14:37] 10Cloud-VPS, 10Toolforge, 10cloud-services-team (Kanban): Deprecate DSA (ssh-dss) SSH keys for Labs users - https://phabricator.wikimedia.org/T168433#3568023 (10Krenair) Audit script: ```lang=python import ldap ldap_conn = ldap.initialize('ldap://ldap-labs.eqiad.wikimedia.org:389') lc = ldap.controls.libldap... [21:15:18] 10Cloud-VPS, 10Toolforge, 10cloud-services-team (Kanban): Deprecate DSA (ssh-dss) SSH keys for Labs users - https://phabricator.wikimedia.org/T168433#3568024 (10Krenair) Currently 178 users with these [21:22:06] 10Cloud-Services, 10Cloud-VPS, 10cloud-services-team (Kanban): monitor some things on all Cloud instances (discussion) - https://phabricator.wikimedia.org/T166845#3568033 (10Krenair) >>! In T166845#3567890, @Andrew wrote: >>>! In T166845#3567349, @Krenair wrote: >> Like shinken? > > Yes! But as I understan... [21:26:51] 10Cloud-VPS (Project-requests): Request creation of VPS project - https://phabricator.wikimedia.org/T174618#3568042 (10bd808) [21:27:33] 10Cloud-VPS (Project-requests): Request creation of vpsmail VPS project - https://phabricator.wikimedia.org/T174618#3568054 (10bd808) [21:27:43] 10Cloud-Services, 10Mail, 10Operations: Create a labs SMTP smarthost - https://phabricator.wikimedia.org/T41785#3568057 (10bd808) I created {T174618} for the project. [21:34:26] !log testlabs Add Osstackcanary user as project admin [21:34:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Testlabs/SAL [21:55:53] 10Cloud-Services: Add new Cloud Services domains to public suffix list - https://phabricator.wikimedia.org/T168677#3568230 (10Krenair) I just checked and that domain doesn't even have the right nameservers (it's set to ns[0-2].wikimedia.org), so this might be a bit premature :) [22:05:36] (03PS1) 10Lokal Profil: [WIP]Merge Wikidata branch into master [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374899 (https://phabricator.wikimedia.org/T174146) [22:08:13] (03CR) 10Lokal Profil: "Marked this as [WIP] for three reasons" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374899 (https://phabricator.wikimedia.org/T174146) (owner: 10Lokal Profil) [22:28:46] (03PS1) 10Lokal Profil: Make handling of wikidata urls more compatible with other handlers [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374901 [22:29:29] (03CR) 10Lokal Profil: "this is the "diff" mentioned in the merge patch" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374901 (owner: 10Lokal Profil) [22:29:48] (03CR) 10jerkins-bot: [V: 04-1] Make handling of wikidata urls more compatible with other handlers [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374901 (owner: 10Lokal Profil) [22:30:22] (03CR) 10Lokal Profil: "> 3) I have a diff from a previously attempted rebase which likely" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374899 (https://phabricator.wikimedia.org/T174146) (owner: 10Lokal Profil) [22:31:29] (03CR) 10Lokal Profil: "Whenever this is deployed we probably want to make sure the `skip_wd` flag is set in our cron job." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374899 (https://phabricator.wikimedia.org/T174146) (owner: 10Lokal Profil) [22:50:34] (03PS2) 10Lokal Profil: Make handling of wikidata urls more compatible with other handlers [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/374901 [23:58:54] 10Cloud-Services, 10wikitech.wikimedia.org, 10Phabricator, 10LDAP, and 2 others: Blocking an account on wikitech should disable LDAP logins - https://phabricator.wikimedia.org/T168692#3373658 (10bd808) There is probably some hook that we can attach code to to make some adjustment to the LDAP object for a u...