[00:34:09] PROBLEM - Puppet errors on tools-exec-1409 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:09:10] RECOVERY - Puppet errors on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [01:29:32] PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [02:33:50] 10Tool-Labs-tools-Pageviews: Typeahead should be case insensitive - https://phabricator.wikimedia.org/T167488#3334569 (10MusikAnimal) [03:09:32] RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0] [04:39:19] 10Tool-Labs-tools-stewardbots: Let stewardbot display temporary rightchanges too - https://phabricator.wikimedia.org/T164204#3334618 (10Linedwell) I made some tries this morning and it seems that it isn't yet possible. StewardBot gets his informations from the RC feed on #meta.wikimedia@irc.wikimedia.org and thi... [08:03:48] 10Labs, 10Tool-Labs, 10DBA: s51127__dewiki_lists (merlbot) database using 13G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133325#2228208 (10Marostegui) Thanks @Luke081515 - we expanded the partitions some months ago so disk space isn't an issue here anymore, but it is always good to clean up... [08:47:53] (03PS1) 10Ayounsi: Remove Rancid passwdrd [labs/private] - 10https://gerrit.wikimedia.org/r/357977 [08:49:04] (03PS2) 10Ayounsi: Remove Rancid password [labs/private] - 10https://gerrit.wikimedia.org/r/357977 [08:49:20] (03CR) 10Ayounsi: [V: 032 C: 032] Remove Rancid password [labs/private] - 10https://gerrit.wikimedia.org/r/357977 (owner: 10Ayounsi) [09:22:41] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:25:28] PROBLEM - Puppet errors on tools-exec-1433 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:57:40] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [10:00:26] RECOVERY - Puppet errors on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [10:13:04] hey! [10:13:10] is anyone online at the moment? [12:09:50] PROBLEM - Puppet errors on tools-exec-1431 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:13:18] PROBLEM - High iowait on tools-exec-1431 is CRITICAL: CRITICAL: tools.tools-exec-1431.cpu.total.iowait (>12.50%) [12:33:17] PROBLEM - High iowait on tools-exec-1431 is CRITICAL: CRITICAL: tools.tools-exec-1431.cpu.total.iowait (>33.33%) [13:13:17] RECOVERY - High iowait on tools-exec-1431 is OK: OK: All targets OK [13:34:19] 10Labs, 10Labs-Infrastructure, 10DBA: Database upgrade MariaDB 10: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127#731912 (10Marostegui) For the record, this is the current status of the discrepancies of this table (based on tables.sql: https://phabricator.wik... [13:39:50] RECOVERY - Puppet errors on tools-exec-1431 is OK: OK: Less than 1.00% above the threshold [0.0] [13:47:12] 10Labs, 10DBA: Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" - https://phabricator.wikimedia.org/T153213#2872990 (10Marostegui) @Halfak what do you want to do with this ticket. The views are failing because the tables do not exist anymore as per: T153213#2873200 Thanks! [14:46:35] 10Labs, 10DBA: Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" - https://phabricator.wikimedia.org/T153213#3335483 (10Halfak) Seems like the views for tables that do not exist should get cleaned up, I suppose. [14:54:58] 10Wikibugs, 10Anti-Harassment: Update Wikibugs to Point Anti-Harassment to #wikimedia-anti-harassment-tools - https://phabricator.wikimedia.org/T167516#3335495 (10dbarratt) [14:55:25] 10Wikibugs, 10Anti-Harassment: Update Wikibugs to Point Anti-Harassment to #wikimedia-anti-harassment-tools - https://phabricator.wikimedia.org/T167516#3335508 (10dbarratt) [14:57:59] 10Labs, 10DBA: Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" - https://phabricator.wikimedia.org/T153213#3335515 (10Marostegui) @chasemp is that something you guys can do? [15:51:08] 10Labs: update queue_server.pp for user/pass setup or one-time setup instructions - https://phabricator.wikimedia.org/T167518#3335581 (10chasemp) [15:51:55] Im unable to query any of the labs dbstores from my tool any idea why, i was able to yesterday [15:52:48] Zppix: that's a very vague report none of us could recreate or action, if you could create a paste demonstrating the issue and ping madhuvishy I bet she could help [15:52:52] Zppix: what is the error message? [15:53:16] Cannot connect to, address [15:53:49] what is the command you are running? can you paste both from the terminal please [15:53:55] Im trying to query the rev-comment for explicit terms i rather not paate [15:53:58] Paste* [15:54:26] try a simple select * from limit 1 type query? [15:54:27] madhuvishy: let me get the cmd one sec [15:54:31] Zppix: you can creaete a phab paste and permission it to just you and madhu [15:55:36] madhuvishy: i tried that no luck [15:56:19] Zppix: I mean, try this - connecting to the database, running a simple query, and pasting what you run from the terminal here [15:56:26] Oh wait there it goes i logout from shell then logged back in [15:56:38] 10Labs: update queue_server.pp for user/pass setup or one-time setup instructions - https://phabricator.wikimedia.org/T167518#3335608 (10chasemp) [15:57:35] Zppix: are you saying it's resolved? [15:58:20] Yes [15:59:21] okay! [16:11:11] it was probably a bad connection to bastion or something [16:18:30] (03PS1) 10Framawiki: Update git repo link [labs/toollabs] - 10https://gerrit.wikimedia.org/r/358039 [16:48:39] !log git moving gerrit-test3 back to gerrit 2.14.1 for testing after finding a possible fix for T152640 [16:48:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [16:48:43] T152640: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640 [16:52:28] wikibugs maintainers, is the latest phab config running on the bot if so, wikibugs isnt join wikimedia-ai even though i know for a fact i've updated tasks that should of triggered the join to that chan [16:59:12] I'm about to do some big DNS cleanups. Should be safe, but… let me know if anything breaks! [17:01:14] andrewbogott: good luck :D [17:40:09] Zppix: they probably don't ping on "wikibugs" since that is the bot name. :) [17:41:50] wikibugs: join #wikimedia-ai [17:42:51] i cant remember who all is a maintainer :P [17:43:52] candidates could be in https://www.mediawiki.org/w/index.php?title=Wikibugs&action=history [17:45:27] mutante: i wish wikibugs had that command :P] [17:45:32] (03CR) 10BryanDavis: [C: 04-1] Update git repo link (031 comment) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/358039 (owner: 10Framawiki) [17:46:10] Zppix: i was about to joke about it.. wikibugs: implement "wall" command to send message to maintainers [17:46:33] mutante: i could write a command to do that for wikibugs but i doubt they would merge it [17:46:50] Zppix: https://tools.wmflabs.org/?tool=wikibugs [17:47:32] see also https://toolsadmin.wikimedia.org/tools/id/wikibugs [17:48:00] let me retest it maybe it will work (i could just be me screwing up not the bot lord knows thats possible) [17:48:33] bd808: toolsadmin is coming along good btw [17:49:32] :) thanks. I have a bunch of work on progress to add toolinfo style description editing. Once I get that working well I'll be able to move new tool account creation over to it [17:49:51] If I had a month of time just to work on that app I have a bunch of ideas that will be easy. Just hard to find the time [17:51:42] what lang is it again? i may be able to lend a hand [17:55:18] Its a Django framework Python app [17:55:30] Zppix: git clone https://phabricator.wikimedia.org/source/wikibugs.git | https://phabricator.wikimedia.org/source/wikibugs/ [17:55:39] it's on diffusion [18:12:31] mutante: i know wikibugs is written in sorry ( i forgot to mention who i was talking to :P) [18:12:47] bd808: ill try to help out where i can (im no python expert) [18:49:48] bd808: https://phabricator.wikimedia.org/T166985#3324440 sounds super sad :/ it's like "a year ago we were ambitious (positively) to fix everything, but now we are stuck with all the debts and creating new debts, and are out of manpower" [18:53:14] zhuyifei1999_: look at it the other way. a year ago we were still ignoring the debt that we knew was due in 2013 and now we are going to fix things before we build more fragile towers on top of it! [18:53:51] um okay [18:54:24] I agree that things could be better. just trying not to wallow in my own sadness ;) [18:54:52] I want a k8s based PaaS very badly [18:55:10] actually any PaaS but we have k8s now so ... yup [18:55:15] let me try a workaround for this task [18:55:43] i.e. building it in k8s with a lot of environment variables set [18:57:03] (03CR) 10Framawiki: Update git repo link (031 comment) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/358039 (owner: 10Framawiki) [18:58:53] (03CR) 10BryanDavis: [C: 04-1] Update git repo link (031 comment) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/358039 (owner: 10Framawiki) [19:02:48] 10Tool-Labs-tools-Other: Tool Bothasava writes too many cache files to /tmp on the bastion - https://phabricator.wikimedia.org/T167542#3336196 (10madhuvishy) [19:21:26] PROBLEM - Puppet errors on tools-exec-1433 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:38:14] (03PS2) 10Framawiki: Update git repo link [labs/toollabs] - 10https://gerrit.wikimedia.org/r/358039 [19:39:16] (03CR) 10Framawiki: Update git repo link (031 comment) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/358039 (owner: 10Framawiki) [19:42:48] 10Tool-Labs-tools-Other, 10User-bd808: Source code link on tools.wmflabs.org must be more discreet - https://phabricator.wikimedia.org/T167545#3336268 (10Framawiki) [19:43:15] 10Tool-Labs-tools-Other, 10User-bd808: Source code link on tools.wmflabs.org must be more discreet - https://phabricator.wikimedia.org/T167545#3336286 (10Framawiki) a:05bd808>03None [19:54:08] 10Labs, 10Tool-Labs, 10Tools-Kubernetes, 10cloud-services-team (Kanban): Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs - https://phabricator.wikimedia.org/T166985#3336313 (10zhuyifei1999) @Xelgen If you desire you may compile the two utilities locally for your tool, instead of... [19:54:43] tada [19:59:58] 10Labs, 10Tool-Labs: get_current_user() returns empty string - https://phabricator.wikimedia.org/T167546#3336324 (10Jogo.obb) [20:01:25] RECOVERY - Puppet errors on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [20:02:56] 10Tool-Labs-tools-Other: get_current_user() returns empty string - https://phabricator.wikimedia.org/T167546#3336352 (10zhuyifei1999) [20:21:49] 10Tool-Labs-tools-Other, 10User-bd808: The link of the source code of this repo that is present on tools.wmflabs.org must be more discreet - https://phabricator.wikimedia.org/T167545#3336414 (10Framawiki) [20:23:05] 10Tool-Labs-tools-Other, 10User-bd808: The link of the source code of this repo that is present on tools.wmflabs.org could be more discreet - https://phabricator.wikimedia.org/T167545#3336417 (10bd808) [20:24:48] 10Tool-Labs-tools-Other, 10User-bd808: The link of the source code of the repo that is present on tools.wmflabs.org could be more discreet - https://phabricator.wikimedia.org/T167545#3336425 (10Framawiki) [20:25:18] 10Tool-Labs-tools-Other, 10User-bd808: The link of the source code of this repo that is present on tools.wmflabs.org could be more discreet - https://phabricator.wikimedia.org/T167545#3336268 (10Framawiki) [21:49:12] 10Labs, 10Operations, 10Patch-For-Review: (don't) decom promethium - https://phabricator.wikimedia.org/T164395#3336547 (10Dzahn) @Andrew @MoritzMuehlenhoff Should i just close it now? I don't have anything specific to do for me, but do we still need to classify it as prod or labs? [21:49:31] 10Labs, 10Operations, 10ops-eqiad, 10Patch-For-Review: setup promethium in eqiad in support of T95185 - https://phabricator.wikimedia.org/T120262#3336550 (10Dzahn) [21:49:35] 10Labs, 10Operations, 10Patch-For-Review: (don't) decom promethium - https://phabricator.wikimedia.org/T164395#3336548 (10Dzahn) 05Open>03stalled p:05Normal>03Low [22:52:27] 10Labs, 10DBA: Missing data on labs replica database - https://phabricator.wikimedia.org/T167554#3336634 (10Ankry) [23:14:49] 10Labs, 10DBA: Missing plwikisource_p.revision data on labs replica database - https://phabricator.wikimedia.org/T167554#3336667 (10bd808) [23:17:47] 10Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3336668 (10bd808) Copied from {T167554} by @Ankry: > As this query shows (https://quarry.wmflabs.org/query/19383) > Revisions from 1147609 to 1147764 are missing from plwikisource_p.revision table in labs database. > > They... [23:18:00] 10Labs, 10DBA: Missing plwikisource_p.revision data on labs replica database - https://phabricator.wikimedia.org/T167554#3336634 (10bd808) [23:18:04] 10Labs, 10DBA: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3336674 (10bd808)