[02:40:16] kaldari, ping [02:47:54] (03PS62) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [02:50:17] (03CR) 10Ricordisamoa: "PS62 updates grunt-jscs from ~2.6.0 to ~2.7.0" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [02:52:11] (03PS17) 10Ricordisamoa: Initial commit [labs/tools/faces] - 10https://gerrit.wikimedia.org/r/192096 [02:56:09] (03CR) 10Ricordisamoa: "PS17 updates grunt-jscs from ~2.6.0 to ~2.7.0" [labs/tools/faces] - 10https://gerrit.wikimedia.org/r/192096 (owner: 10Ricordisamoa) [03:23:19] 6Labs, 10Tool-Labs: Set versions in man pages from debian/changelog - https://phabricator.wikimedia.org/T87344#1985550 (10scfc) On the other hand, if the version is inserted, and it should be because having differences between the man page and the package meta data is confusing, then inherently the man page wa... [03:35:31] (03PS1) 10Tim Landscheidt: Set versions and dates in man pages from debian/changelog [labs/toollabs] - 10https://gerrit.wikimedia.org/r/267631 (https://phabricator.wikimedia.org/T87344) [03:38:48] (03PS2) 10Tim Landscheidt: Set versions and dates in man pages from debian/changelog [labs/toollabs] - 10https://gerrit.wikimedia.org/r/267631 (https://phabricator.wikimedia.org/T87344) [03:43:31] (03CR) 10Tim Landscheidt: [C: 032] "(As I'm probably the only one who cares :-).)" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/267631 (https://phabricator.wikimedia.org/T87344) (owner: 10Tim Landscheidt) [03:46:19] (03PS1) 10Tim Landscheidt: Add src/.deps to src/.gitignore [labs/toollabs] - 10https://gerrit.wikimedia.org/r/267632 [03:47:03] 6Labs, 10Tool-Labs, 5Patch-For-Review: Set versions in man pages from debian/changelog - https://phabricator.wikimedia.org/T87344#1985556 (10scfc) 5Open>3Resolved [03:48:30] (03CR) 10Tim Landscheidt: [C: 032] Add src/.deps to src/.gitignore [labs/toollabs] - 10https://gerrit.wikimedia.org/r/267632 (owner: 10Tim Landscheidt) [03:56:40] (03CR) 10Tim Landscheidt: Add list-user-databases command [labs/toollabs] - 10https://gerrit.wikimedia.org/r/234934 (https://phabricator.wikimedia.org/T91231) (owner: 10Tim Landscheidt) [04:10:28] (03PS3) 10Tim Landscheidt: Add list-user-databases command [labs/toollabs] - 10https://gerrit.wikimedia.org/r/234934 (https://phabricator.wikimedia.org/T91231) [04:11:25] 6Labs, 10Tool-Labs, 5Patch-For-Review: Create a utility that lists all databases of a user - https://phabricator.wikimedia.org/T91231#1985563 (10scfc) [04:12:55] 6Labs, 10Tool-Labs, 5Patch-For-Review: Create a utility that lists all databases of a user - https://phabricator.wikimedia.org/T91231#1077206 (10scfc) Narrowed the scope down to listing all databases of a user; this is my real-life use case. [04:13:41] (03CR) 10Tim Landscheidt: [C: 04-2] "python3-mysql.connector is not available on Precise, so I have to use Python 2 instead." [labs/toollabs] - 10https://gerrit.wikimedia.org/r/234934 (https://phabricator.wikimedia.org/T91231) (owner: 10Tim Landscheidt) [04:15:50] 6Labs, 10Labs-Infrastructure, 10Tool-Labs: toolsbeta-puppetmaster3 instance has "BUG: soft lockup - CPU#0 stuck for 23s! [git:1366]" - https://phabricator.wikimedia.org/T125360#1985568 (10scfc) [04:15:54] 6Labs, 10Labs-Infrastructure, 7Tracking: Labs instances sometimes freeze - https://phabricator.wikimedia.org/T124133#1985567 (10scfc) [04:17:48] 6Labs, 10Labs-Infrastructure: Instance console does not gives output / keystroke access - https://phabricator.wikimedia.org/T64847#1985570 (10scfc) 5Open>3Resolved a:3scfc Both instances have been deleted in the mean time, so resolving this task. [04:17:57] 6Labs, 10Labs-Infrastructure: Instance console does not gives output / keystroke access - https://phabricator.wikimedia.org/T64847#1985573 (10scfc) a:5scfc>3None [06:18:00] 6Labs: Separate flannel's etcd from k8s' etcd - https://phabricator.wikimedia.org/T125371#1985595 (10yuvipanda) 3NEW a:3yuvipanda [06:26:39] 10Tool-Labs-tools-Matthewrbowker's-tools: Convert to PDO - https://phabricator.wikimedia.org/T61691#1985605 (10Matthewrbowker) 5Open>3Resolved Converted to mysqli_ functions in https://github.com/Matthewrbowker/labs/commit/3a79ec11f302e0fd4c077d5f65496cf8d3cb5b3e, we're good here now. [06:27:36] 10Tool-Labs-tools-Matthewrbowker's-tools: Convert Matthewrbowker's tools to Bootstrap - https://phabricator.wikimedia.org/T61072#1985607 (10Matthewrbowker) 5Open>3Resolved Done in https://github.com/Matthewrbowker/labs/commit/3a79ec11f302e0fd4c077d5f65496cf8d3cb5b3e, it's in staging now. With a few tweaks i... [07:19:41] How to see to which project instances I belong to? [08:40:57] Tanvir: run groups on any host, and look for groups starting with project- [08:43:41] valhallasw`cloud, I get this: http://pastebin.com/raw/x4HFcVtP [08:43:56] What's that project? o.O [08:46:12] So: bastion and tools [08:47:32] valhallasw`cloud, is that enough to run pywikipedbabot? [08:51:35] So, I just want to run the bot to run some scripts time to time with the help of corn. That's all. [09:35:22] (03CR) 10Merlijn van Deen: "Alternatively, dh-virtualenv could work (might be an idea in general, given that we have a whole bunch of python tools)." (031 comment) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/234934 (https://phabricator.wikimedia.org/T91231) (owner: 10Tim Landscheidt) [09:36:40] Tanvir: yes. See the help pages [09:37:00] Although creating a tool account is preferable [09:37:15] As you can then give others access to maintain it [09:39:39] Yes, I do need something like that as I will also run a welcome bot. [15:52:08] Hey guys, would you please have a look? https://gerrit.wikimedia.org/r/#/c/267513/ I don't that would so risky to apply but can unblock our work. Thank you [15:52:32] *believe [15:53:31] ops, I didn't notice it is still weekend there. [16:01:28] ebraminio: I'm pretty sure you can do those per-tool, if it's urgent [16:02:54] zhuyifei1999_: This was just done before https://gerrit.wikimedia.org/r/#/c/263229/ but the one the package and python binding seems was the issue [16:03:40] It is actually a less troublemaking portion of a landed before patch [16:57:29] bd808, bummer, it works not :( [16:57:58] it might have been that i had to reboot vagrant after provisioning [17:17:21] kaldari, piung [17:19:30] is there a problem with authentication right now? I'm not able to log in to phab-01.eqiad.wmflabs [17:20:31] just that instance? [17:20:39] I can get into deployment-bastion.deployment-prep.eqiad.wmflabs [17:24:18] I can log into my instaces too [17:25:17] twentyafterfour: Other instances of that project unreachable too? [17:29:33] Permission denied (publickey,keyboard-interactive). [17:29:37] same thing on phab-02 [17:29:54] I get this too, but I'm not a member of that project ;) [17:29:57] strange [17:30:01] what project is this? [17:30:06] Phabricator [17:30:08] Krenair: I can get into deployment-bastion also [17:30:09] https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator [17:30:18] so it's just phabricator project [17:30:33] twentyafterfour: yes, I can't get into them either [17:30:36] as root [17:30:52] my key is denied [17:30:56] hmm [17:30:58] yeah [17:31:00] mine too [17:31:13] so how does a whole project get f***ed [17:31:25] good question [17:31:28] or better yet how does one fix it? ;) [17:31:40] what's the name of the project? [17:31:42] phabricator? [17:31:44] ok [17:31:45] yes [17:32:02] am trying [17:32:05] yuvipanda@labcontrol1001:~$ sudo salt 'phab-02.phabricator.eqiad.wmflabs' cmd.run hostname [17:32:08] let's see if that runs [17:32:10] yes [17:33:49] let me try running puppet there now [17:51:02] thcipriani: hi [17:51:47] any problem with my swat request? https://wikitech.wikimedia.org/wiki/Deployments https://phabricator.wikimedia.org/T123109 [17:52:27] Kelson: it looks fine, I ran out of time in my window so I bumped it. [17:52:45] twentyafterfour: /usr/bin/mandb: can't write to /var/cache/man/4752: No space left on device [17:52:49] on -02 [17:53:01] /dev/vda1 18G 18G 0 100% / [17:53:25] Kelson: if you're around for evening SWAT, could you move it down there? Note that you should be in wikimedia-operations during the SWAT window. [17:53:27] thcipriani: OK. What does that mean concretly? At which date will this be put online? [17:54:24] Kelson: Or, if it's easier for your, you could post it for morning SWAT tomorrow, and be in #wikimedia-operations at that time. [17:54:36] thcipriani: I can't probably? Why do you need me? [17:54:50] YuviPanda: :-/ [17:55:03] Kelson: can you join #wikimedia-releng and we can talk there? [17:55:11] twentyafterfour: 14G /srv [17:55:35] 13G /srv/phab/repos [17:55:39] twentyafterfour: ^ [17:56:10] which instance? I'm unable to log in to any instance in that project it seems [17:56:12] 6Labs, 10Labs-Infrastructure, 7Tracking: Labs instances sometimes freeze - https://phabricator.wikimedia.org/T124133#1986867 (10scfc) [17:56:14] twentyafterfour: 13G /srv/phab/repos/MWM [17:56:17] twentyafterfour: this is phab-02 [17:56:24] twentyafterfour: I'm using salt which miraculously works [17:56:43] hmm actually I can log in to phab-03, but 01 and 02 don't work [17:56:46] twentyafterfour: can I delete that folder? [17:56:51] I think they both have full disks [17:57:57] YuviPanda: I suppose /srv/phab/repos is non-critical on the labs instances [17:58:02] twentyafterfour: ok [17:58:05] twentyafterfour: so I'll wipe that [17:58:23] twentyafterfour: try sshing in? [17:58:39] denied [17:59:07] on both 01 and 02 [17:59:45] does a full disk normally cause ssh to fail? I thought there was a workaround for that in sshd [18:01:34] twentyafterfour: good question. not sure [18:01:37] twentyafterfour: let me restart sshd [18:01:51] try now? [18:03:51] same problem [18:04:06] YuviPanda: should I try restarting one of the instances? [18:04:27] twentyafterfour: yeah [18:04:31] twentyafterfour: actually [18:04:35] twentyafterfour: let me try to read auth.log [18:07:06] YuviPanda, Hi :) [18:08:20] YuviPanda, i can open a private message? it's about the wikibugs... [18:09:12] tatoo: hello! I don't really do much wikibugs work, so I'm unlikely to be able to help you... :( [18:09:16] try valhallasw`cloud or legoktm [18:09:23] tatoo: Feb 1 17:58:31 phab-02 sshd[4959]: User twentyafterfour from phab-03.phabricator.eqiad.wmflabs not allowed because not listed in AllowUsers [18:09:58] tatoo: I'm available [18:10:01] YuviPanda: hmm... weird [18:10:10] YuviPanda: ping meeting :) [18:11:20] YuviPanda: thanks for trying, I can try restarting things [18:11:39] twentyafterfour: ok! [18:12:50] YuviPanda, oh, i understand, anyway thanks [18:15:37] well rebooting phab-01 didn't work [18:16:06] valhallasw`cloud, is on IRCNOTIFIER_KEY in the config says YuviPanda ask, what should i put there? simply I do not understand [18:16:20] ask YuviPanda * [18:18:04] hmm I guess I have to rebuild it ... [18:18:26] interesting... [18:18:36] someone filling up test repos [18:21:14] kaldari, ping agan [18:54:47] 6Labs, 10Labs-Infrastructure: Not possible to switch projects in horizon.wikimedia.org - https://phabricator.wikimedia.org/T125417#1987080 (10scfc) 3NEW [18:59:15] 6Labs, 10Labs-Infrastructure: Not possible to switch projects in horizon.wikimedia.org - https://phabricator.wikimedia.org/T125417#1987102 (10scfc) [18:59:17] 6Labs: Unable to change projects in horizon - https://phabricator.wikimedia.org/T123310#1987104 (10scfc) [19:48:02] hola: how do i do to reboot deployment-eventlogging03.eqiad.wmflabs? [19:48:24] Since it is not on the analytics project I am not sure i have permits [19:48:48] nuria_: release engineering manages the deployment-prep project, so they would know who has rights / can give them to you if you need them [19:48:54] #wikimedia-releng :D [19:49:11] k Thank youuuu [19:49:18] nuria_: np! [19:49:35] you're a deployment-prep project admin [19:49:47] you should be able to restart it [20:03:00] PROBLEM - Puppet failure on tools-flannel-etcd-03 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [0.0] [20:05:22] ^ is me [20:50:00] 6Labs, 10Labs-Infrastructure, 10Tool-Labs, 6operations: tools-exec: automatic php upgrade - puppet fail - https://phabricator.wikimedia.org/T125438#1987548 (10Dzahn) 3NEW [21:02:22] 6Labs, 10Labs-Infrastructure, 10Tool-Labs, 6operations: tools-exec: automatic php upgrade - puppet fail - https://phabricator.wikimedia.org/T125438#1987599 (10scfc) 5Open>3Resolved a:3scfc Fixed by `apt-get install php5-gd` and downgrading. [21:03:18] 6Labs, 10Labs-Infrastructure, 10Tool-Labs, 6operations: tools-exec: automatic php upgrade - puppet fail - https://phabricator.wikimedia.org/T125438#1987605 (10Dzahn) thanks @scfc :) i uploaded https://gerrit.wikimedia.org/r/#/c/267778/ [21:24:16] 6Labs, 5Patch-For-Review: Separate flannel's etcd from k8s' etcd - https://phabricator.wikimedia.org/T125371#1987683 (10yuvipanda) Wondering if I should dump and reload the data or just recreate from scratch. [21:44:18] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:46:04] well, it's up for me [21:46:58] yeah [21:47:03] and no icinga page [22:02:28] 6Labs, 10Labs-Infrastructure, 5Patch-For-Review: Install hunspell on tools - https://phabricator.wikimedia.org/T125193#1987832 (10Ebraminio) 5Open>3Resolved Thank you. Also we've started using it on our tools also and it is working great. [22:03:24] (03PS19) 10ArthurPSmith: Added a Wikidata-based "chart of the nuclides" under /nuclides [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/245591 [22:04:42] (03CR) 10ArthurPSmith: [C: 031] "Following comments from ricordisamoa - indentation should now be fixed. The sequence of 'if' statements was replaced by a dict mapping the" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/245591 (owner: 10ArthurPSmith) [22:05:21] 6Labs, 10Tool-Labs, 5Patch-For-Review: Install python-hunspell (and dictionaries?) - https://phabricator.wikimedia.org/T123192#1987839 (10Ebraminio) 5Open>3Resolved a:3Ebraminio Reza and I were working on same thing and with the fix of T125193 we used hunspell on our tool and actually we no longer need... [22:17:53] heya [22:17:53] qq [22:18:04] anyone know for sure if $::labsproject is supposed to evalulate to [22:18:09] say [22:18:11] 'analytics' [22:18:11] or [22:18:14] 'project-analytics' [22:18:14] ? [22:18:55] ottomata: the former [22:19:06] ahh ok [22:19:08] makes sense then [22:19:08] thank you [22:19:46] you can tell because of uses like [22:19:47] modules/eventlogging/manifests/deployment/source.pp: 'labs' => "project-${::labsproject}", [22:19:56] and modules/wmflib/lib/hiera/backend/mwyaml_backend.rb: # "labs/%{::labsproject}/common". We map the former [22:57:58] MusikAnimal, when does MusikBot handle PC tags? [23:03:45] 6Labs, 5Patch-For-Review: Separate flannel's etcd from k8s' etcd - https://phabricator.wikimedia.org/T125371#1988036 (10yuvipanda) Ok, done for flannel! Now to set it up for k8s... [23:04:44] CP678: any protection template [23:04:52] but it only removes/repairs them, does not add them [23:04:59] I see. [23:05:28] Well PCBot is pretty sound and quick. If any issues arise, it [23:05:42] *it's usually a caching issue. [23:05:53] sorry you said "when", the answer to that would be when the page shows up in Category:Wikipedia_pages_with_incorrect_protection_templates [23:06:01] yeah, I had to work around that [23:06:37] It actually looks at the direct list of PC protected pages [23:06:54] It might have still been in the list at the time Cyberbot was looking at it. [23:07:02] Hence the caching issue. [23:07:13] yeah that might be cached, because sometimes the API says a page is still protected when it is not [23:07:13] yeah [23:07:46] MusikBot checks if it's protected via the API, if it isn't, it removes the template. If it is still protected, it will look at the protect log to get around potential caching issues [23:09:40] Alternatively, you could call action=purge. :P0 [23:09:40] Before the API call [23:09:40] nice [23:09:40] that will purge the API response? [23:09:40] It will purge everything cached about the page [23:09:41] gotcha [23:09:41] should probably do that anyway, sometimes that's all that needs to be done to remove the page from the maintenance category [23:11:47] err wait, I lied. It does not check the protect log, just looks at current protect info and compares timestamps. If the API is up-to-date there would be no current protect info [23:12:06] so the timestamp comparison gets around the caching issue [23:12:24] so no need to purge [23:25:32] RECOVERY - Puppet failure on tools-k8s-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:27:32] RECOVERY - Puppet failure on tools-k8s-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [23:32:14] PROBLEM - Puppet failure on tools-k8s-etcd-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]