[00:30:19] PROBLEM - Puppet errors on tools-exec-1409 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [00:41:55] 10Tools: Tool "list" redirects to GitHub without consent - https://phabricator.wikimedia.org/T172658#3526860 (10Krinkle) p:05Triage>03Normal [00:43:45] PROBLEM - Puppet errors on tools-exec-1427 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [01:10:19] RECOVERY - Puppet errors on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [01:18:45] RECOVERY - Puppet errors on tools-exec-1427 is OK: OK: Less than 1.00% above the threshold [0.0] [03:57:42] PROBLEM - Puppet errors on tools-exec-1415 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [04:22:42] RECOVERY - Puppet errors on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [06:34:27] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1418 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:09:28] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [07:41:08] 10cloud-services-team: labsdb1003 BBU failing - https://phabricator.wikimedia.org/T173402#3527057 (10Marostegui) [07:41:46] 10cloud-services-team: labsdb1003 BBU failing - https://phabricator.wikimedia.org/T173402#3527069 (10Marostegui) The impact of the RAID being WT instead of WB is, long story short, performance. I would not spend much time on replacing its BBU as this host will go away soon. [10:35:20] 10Tools: Tool "ifttt-testing" loads assets from many sites, mixed http/https - https://phabricator.wikimedia.org/T172609#3527280 (10D3r1ck01) a:03D3r1ck01 [10:44:01] 10Tools: Tool "wikipedia-fetch-content" loads jquery and bootstrap from code.jquery.com and bootstrapcdn - https://phabricator.wikimedia.org/T173067#3527304 (10D3r1ck01) @zhuyifei1999, thanks again for reporting this. I will go ahead and fix this once and or all :). I need to keep this link in my head: https://t... [11:58:20] 10cloud-services-team: labsdb1003 BBU failing - https://phabricator.wikimedia.org/T173402#3527392 (10Marostegui) 05Open>03Resolved a:03Marostegui And the re-learn worked for now and raid back in WB ``` ˜/icinga-wm 13:47> RECOVERY - MegaRAID on labsdb1003 is OK: OK: optimal, 1 logical, 2 physical, WriteBac... [12:00:06] PROBLEM - Puppet errors on tools-worker-1014 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:40:08] RECOVERY - Puppet errors on tools-worker-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [13:26:41] 10PAWS: Debugging notebook cell action/state - https://phabricator.wikimedia.org/T173416#3527551 (10Jprorama) [13:33:00] 10cloud-services-team: labsdb1003 BBU failing - https://phabricator.wikimedia.org/T173402#3527057 (10chasemp) Thanks man, buys us a bit more time [13:39:19] 10PAWS: Debugging notebook cell action/state - https://phabricator.wikimedia.org/T173416#3527576 (10Jprorama) [15:12:45] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:47:39] 10Tools: Tool "wikipedia-fetch-content" loads jquery and bootstrap from code.jquery.com and bootstrapcdn - https://phabricator.wikimedia.org/T173067#3527944 (10D3r1ck01) This patch solves the issue: [[https://github.com/ch3nkula/Wikipedia-Fetch-Content/commit/a36d015d9822482c2831a73e6e483dfa197aacd0|Patch Set 1]... [15:47:44] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [15:48:56] 10Tools: Tool "wikipedia-fetch-content" loads jquery and bootstrap from code.jquery.com and bootstrapcdn - https://phabricator.wikimedia.org/T173067#3527950 (10D3r1ck01) [15:49:26] 10Tools, 10Patch-For-Review: Tool "wikipedia-fetch-content" loads jquery and bootstrap from code.jquery.com and bootstrapcdn - https://phabricator.wikimedia.org/T173067#3517709 (10D3r1ck01) [17:17:23] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labmon1002 - https://phabricator.wikimedia.org/T165784#3528256 (10Cmjohnson) @robh I must've confused this with one of the other lab servers..no controller card present on labmon1002.....only 4 disk couldn't do a Raid10 if... [17:30:36] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labmon1002 - https://phabricator.wikimedia.org/T165784#3528336 (10Cmjohnson) a:05Cmjohnson>03RobH [18:19:28] chasemp: I was wondering, would it in theory be possible to have the WikiReplica revision deletion redaction happen in a mariadb virtual column instead of happening in the view, so that it would be possible to index those fields? [18:21:14] bawolff: I'm trying to think of why not and I don't have a good reason other than adding a layer (of which we have too many) and manageability [18:21:48] jaime and manuel will have their own thoughts but it's an interesting proposition [18:22:14] It does indeed add an extra layer of stuff, and from what I understand altering tables to add virtual columns is not a free operations, so it makes quickly changing the redactions more annoying [18:23:11] Ideal situation would be for mariadb to just be smart enough to optimize IF(rev_deleted&1=1,rev_user,null) = 'Foo' into something sane, but that's probably not going to happen any time soon [18:24:02] It would be really nice to not require users to worry about if they should be using revision vs revision_userindex [18:24:52] that is true, I'm not sure if this is worth it for human time on our end but let's see what jynus thinks [19:33:49] bd808: for my cloud vps project, should I name the gerrit repo labs/ (old convention) or cloud/ ? [19:36:18] we didn't talk gerrit rebranding yet :) legoktm for now let's do labs/ only because we may do wmcloud in some places and that is likely to be one of them but I'm unsure. this way it's still consistent for migration later [19:36:27] legoktm: ^ (bryan is on vaca) [19:36:44] sounds good, thanks :) [19:38:32] The phab project default is vps-project-*. Like chasemp said we haven't talked [19:38:44] About gerrit [19:40:59] legoktm: my advice would be to find something that doesn't make RainbowSprinkles puke and run with it if you want to brake the labs/foo model [19:41:43] Herp? [19:42:18] Gerrit repo naming questions from legoktm [19:42:26] Names are pointless in gerrit [19:42:32] :) [19:42:39] I stopped caring wtf people name their repos like 3 years ago [19:42:52] bd808: go back to vacation! [19:42:53] And parents can be anything, so foo/bar inheriting from foo is just convention, not required. [19:42:57] The fake hierarchy fooled people [19:43:17] #mistakesweremade [19:43:26] #truethat [19:43:32] * bd808 slinks back into the mist [19:43:43] But to answer legoktm's question: idgafos what you call your repo :) [19:44:06] I'm sticking with labs/ [19:44:15] legoktm if you're bored I have ... T173419 for you :D [19:44:17] T173419: Unblock stuck global renames at Meta-Wiki - https://phabricator.wikimedia.org/T173419 [19:44:46] TabbyCat: I wasn't bored, but I'll look after lunch [19:46:04] legoktm: thanks. Just to clarify: it is not that those renames are stuck, which they are, but that every global rename is becoming stuck now and then at metawiki for some reason we don't know. [19:46:34] bon appetit [22:02:22] PROBLEM - Puppet errors on tools-paws-worker-1016 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [22:37:22] RECOVERY - Puppet errors on tools-paws-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [22:52:57] bd808: currently here? [22:53:32] it looks like the timestamps at https://tools.wmflabs.org/versions/ are broken [22:53:38] each entry has "2017-08-16 22:53" there [22:53:49] looks like it is using the current time [22:57:11] chasemp: is there a name for the cloud VPSes with 300 GB of storage? [23:23:54] 10Cloud-Services, 10DBA, 10User-Urbanecm: Prepare and check storage layer for bawikibooks - https://phabricator.wikimedia.org/T173473#3529530 (10Urbanecm) [23:26:04] !log rcm Neon: Running update, new patch was released [23:26:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [23:26:34] 10Cloud-Services, 10DBA, 10User-Urbanecm: Prepare and check storage layer for bawikibooks - https://phabricator.wikimedia.org/T173473#3529558 (10Urbanecm) 05Open>03Invalid Seems it is about reopening, not creation. No DBA attention should be needed then. [23:27:28] !log rcm Tin: Running update of jenkins [23:27:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [23:30:40] (03PS1) 10Legoktm: Add tox and fix flake8 issues [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/372206 [23:30:55] (03CR) 10Legoktm: [V: 032 C: 032] Add tox and fix flake8 issues [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/372206 (owner: 10Legoktm) [23:32:29] !log rcm Xenon: Running update [23:32:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [23:33:11] !log rcm CAC: running vagrant git-update [23:33:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [23:35:17] 10Cloud-Services: Set up designate-dashboard - https://phabricator.wikimedia.org/T93089#3529572 (10Krenair) 05Open>03Resolved a:03Andrew Yep, March 2016: https://gerrit.wikimedia.org/r/#/c/275854/ [23:47:31] (03PS1) 10Legoktm: Update link to find source code [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/372209 [23:48:02] (03CR) 10Legoktm: [C: 032] Update link to find source code [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/372209 (owner: 10Legoktm) [23:48:55] (03CR) 10Legoktm: [C: 032] Update link to find source code [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/372209 (owner: 10Legoktm) [23:49:13] (03Merged) 10jenkins-bot: Update link to find source code [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/372209 (owner: 10Legoktm) [23:50:59] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3529617 (10zhuyifei1999) [23:51:00] 10Tools, 10Patch-For-Review: Tool "wikipedia-fetch-content" loads jquery and bootstrap from code.jquery.com and bootstrapcdn - https://phabricator.wikimedia.org/T173067#3529615 (10zhuyifei1999) 05Open>03Resolved Thanks, LGTM. [23:51:27] harej, which ones have 300GB storage? [23:51:45] I was told they exist but you have to ask for them. [23:52:22] m1.xlarge is 150Gish I think [23:53:12] well [23:53:13] it's 160 GB [23:53:19] they can create custom flavours [23:53:54] oh there was m1.gigantic used in the video project [23:54:33] hm no, that's 80GB storage [23:54:34] m1.gigantic is only 80GB [23:55:10] harej, do you know the name of an instance with 300GB? [23:55:17] nope [23:57:48] I wish I could be more helpful, but I only know about this type of instance because halfak said something about it [23:58:03] o/ [23:58:11] wikibrain-embeddings-01/02 [23:58:32] https://phabricator.wikimedia.org/T161554 [23:58:32] https://tools.wmflabs.org/openstack-browser/server/wikibrain-embeddings-01.wikibrain.eqiad.wmflabs [23:58:38] type=bigdisk [23:58:44] * halfak runs away again [23:59:02] oh also halfak, what is a query like https://quarry.wmflabs.org/query/20931 but for all of enwiki? [23:59:09] and why does it only go to October 2016?