[00:29:27] (03PS2) 10BryanDavis: Add pretty error pages [labs/striker] - 10https://gerrit.wikimedia.org/r/307107 (https://phabricator.wikimedia.org/T143949) [00:38:49] (03CR) 10BryanDavis: "Cherry-picked to https://striker.wmflabs.org/ for testing." [labs/striker] - 10https://gerrit.wikimedia.org/r/307107 (https://phabricator.wikimedia.org/T143949) (owner: 10BryanDavis) [01:04:23] PROBLEM - Puppet run on tools-exec-1202 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:21:32] PROBLEM - Free space - all mounts on tools-web-static-02 is CRITICAL: CRITICAL: tools.tools-web-static-02.diskspace._srv.byte_percentfree (<11.11%) [01:34:05] 06Labs, 10Beta-Cluster-Infrastructure: puppet::self hosts now have two servers set - https://phabricator.wikimedia.org/T144108#2588952 (10mmodell) a:0520after4>03Andrew @andrew: `puppet::self` is very prone to race conditions and it needs to be completely refactored. I don't know if this was caused by {8a... [01:44:24] RECOVERY - Puppet run on tools-exec-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [03:51:33] PROBLEM - Free space - all mounts on tools-web-static-02 is CRITICAL: CRITICAL: tools.tools-web-static-02.diskspace._srv.byte_percentfree (<11.11%) [04:49:05] (03PS1) 10BryanDavis: Catch and log database errors while saving models [labs/striker] - 10https://gerrit.wikimedia.org/r/307110 (https://phabricator.wikimedia.org/T144082) [04:51:32] PROBLEM - Free space - all mounts on tools-web-static-02 is CRITICAL: CRITICAL: tools.tools-web-static-02.diskspace._srv.byte_percentfree (<10.00%) [04:56:13] !log tools sudo git gc --aggressive on tools-web-static-02.tools:/srv/cdnjs [04:56:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [05:21:31] PROBLEM - Free space - all mounts on tools-web-static-02 is CRITICAL: CRITICAL: tools.tools-web-static-02.diskspace._srv.byte_percentfree (<20.00%) [05:25:24] !log tools sudo git gc --aggressive on tools-web-static-01.tools:/srv/cdnjs [05:25:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [05:34:11] !log tools After git gc on web-static-02.tools:/srv/cdnjs: /dev/mapper/vd-cdnjs--disk 61G 54G 3.3G 95% /srv [05:34:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [05:59:29] PROBLEM - Free space - all mounts on tools-web-static-01 is CRITICAL: CRITICAL: tools.tools-web-static-01.diskspace._srv.byte_percentfree (<20.00%) [06:01:11] PROBLEM - Puppet staleness on tools-exec-1204 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [06:02:59] PROBLEM - Puppet staleness on tools-exec-1213 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [06:58:05] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Not Available - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 531 bytes in 0.035 second response time [07:01:13] PROBLEM - Puppet run on tools-webgrid-lighttpd-1414 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [07:22:20] 06Labs, 10Beta-Cluster-Infrastructure: puppet::self hosts now have two servers set - https://phabricator.wikimedia.org/T144108#2589029 (10hashar) Keep rOPUP8a4cc8790f ! Else we are back to T132689 which has was quite annoying. [07:23:05] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.047 second response time [07:36:12] RECOVERY - Puppet run on tools-webgrid-lighttpd-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [08:16:52] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [08:34:34] PROBLEM - Puppet run on tools-static-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:51:51] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:19:33] RECOVERY - Puppet run on tools-static-01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:34:40] (03CR) 10Alex Monk: Catch and log database errors while saving models (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/307110 (https://phabricator.wikimedia.org/T144082) (owner: 10BryanDavis) [11:39:02] (03CR) 10Alex Monk: Add pretty error pages (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/307107 (https://phabricator.wikimedia.org/T143949) (owner: 10BryanDavis) [11:51:14] (03CR) 10Merlijn van Deen: "lgtm overall, but I don't really like the way Django handles error pages (see inline comment)." (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/307107 (https://phabricator.wikimedia.org/T143949) (owner: 10BryanDavis) [12:02:38] 06Labs, 10Phabricator: Update phabricator role to support labs - https://phabricator.wikimedia.org/T144112#2589198 (10Paladox) This task is also for tracking the fixes we do to the role and when we discontinue the use of phabricator labs role :) [14:57:37] (03CR) 10BryanDavis: Add pretty error pages (032 comments) [labs/striker] - 10https://gerrit.wikimedia.org/r/307107 (https://phabricator.wikimedia.org/T143949) (owner: 10BryanDavis) [15:09:18] (03PS3) 10BryanDavis: Add pretty error pages [labs/striker] - 10https://gerrit.wikimedia.org/r/307107 (https://phabricator.wikimedia.org/T143949) [15:10:37] (03CR) 10Merlijn van Deen: [C: 031] Add pretty error pages [labs/striker] - 10https://gerrit.wikimedia.org/r/307107 (https://phabricator.wikimedia.org/T143949) (owner: 10BryanDavis) [15:19:27] PROBLEM - Free space - all mounts on tools-web-static-01 is CRITICAL: CRITICAL: tools.tools-web-static-01.diskspace._srv.byte_percentfree (<11.11%) [15:43:09] (03CR) 10BryanDavis: Catch and log database errors while saving models (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/307110 (https://phabricator.wikimedia.org/T144082) (owner: 10BryanDavis) [16:55:33] (03PS1) 10BryanDavis: Add license, source and docs links to footer [labs/striker] - 10https://gerrit.wikimedia.org/r/307134 (https://phabricator.wikimedia.org/T143972) [17:08:39] (03PS1) 10BryanDavis: Update site.css [labs/striker/staticfiles] - 10https://gerrit.wikimedia.org/r/307136 (https://phabricator.wikimedia.org/T143972) [17:12:28] (03CR) 10BryanDavis: "Cherry-picked to https://striker.wmflabs.org/ for testing" [labs/striker] - 10https://gerrit.wikimedia.org/r/307134 (https://phabricator.wikimedia.org/T143972) (owner: 10BryanDavis) [17:32:23] PROBLEM - Free space - all mounts on tools-static-02 is CRITICAL: CRITICAL: tools.tools-static-02.diskspace._srv.byte_percentfree (<100.00%) [17:54:14] (03PS1) 10BryanDavis: Explicitly register admin modules [labs/striker] - 10https://gerrit.wikimedia.org/r/307139 (https://phabricator.wikimedia.org/T144134) [17:57:47] (03CR) 10BryanDavis: [V: 031] "Cherry-picked to https://striker.wmflabs.org/ for testing and verified that this enables access to the admin interface for models that reg" [labs/striker] - 10https://gerrit.wikimedia.org/r/307139 (https://phabricator.wikimedia.org/T144134) (owner: 10BryanDavis) [18:01:53] 10PAWS, 06Research-and-Data-Backlog, 07Epic, 03Research-and-Data-2017-Q1: Launch Open Notebooks Infrastructure - https://phabricator.wikimedia.org/T140430#2589512 (10Capt_Swing) @DarTar @yuvipanda Saw your meeting notes--thanks for posting. Do y'all want some help with end-user documentation for PAWS? I'v... [18:09:07] 10PAWS: Non-notebook files don't redirect to paws-public when URL is changed - https://phabricator.wikimedia.org/T143459#2589516 (10Capt_Swing) [18:09:10] 10PAWS: download datafiles generated in PAWS - https://phabricator.wikimedia.org/T137251#2589518 (10Capt_Swing) [18:09:53] 10PAWS: Non-notebook files don't redirect to paws-public when URL is changed - https://phabricator.wikimedia.org/T143459#2568964 (10Capt_Swing) @staeiou I just merged in an old request of mine, which is essentially the same, but less detailed ;) [18:16:34] 10PAWS: add ui button or menu option to view public version of a notebook - https://phabricator.wikimedia.org/T144135#2589522 (10Capt_Swing) [18:16:46] 10PAWS: add ui button or menu option to view public version of a notebook - https://phabricator.wikimedia.org/T144135#2589522 (10Capt_Swing) p:05Triage>03Low [18:57:35] (03CR) 10Merlijn van Deen: [C: 031] Explicitly register admin modules [labs/striker] - 10https://gerrit.wikimedia.org/r/307139 (https://phabricator.wikimedia.org/T144134) (owner: 10BryanDavis) [19:37:18] PROBLEM - Puppet run on tools-docker-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:51:29] 10PAWS: add ui button or menu option to view public version of a notebook - https://phabricator.wikimedia.org/T144135#2589601 (10yuvipanda) p:05Low>03Normal I agree! Although note that you can change just *one* (either one) and it works. [19:52:39] hi all, re: recitation-bot, can someone advise on how I install libxml2 in the kubernetes image? [19:53:46] yuvipanda, andrewbogott ? ^ [19:56:32] dfko, create a task in the tracking task [19:56:39] Lemme find it... [19:57:20] dfko, T140110 [19:57:20] T140110: Packages to be installed in Tool Labs Kubernetes Images (Tracking) - https://phabricator.wikimedia.org/T140110 [19:58:17] Why do you need that package installed? It says basically if you can install it via a language specific package manager, then the package won't be installe.d [20:01:20] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 07Tracking: Packages to be installed in Tool Labs Kubernetes Images (Tracking) - https://phabricator.wikimedia.org/T140110#2453218 (10Dfko) I need libxml2 (probably also libxml2-dev in kubernetes for recitation-bot since it uses the command-line xsltproc to run xslt... [20:01:53] python package wraps the system library [20:02:16] http://lxml.de/xpathxslt.html [20:09:11] Would be excellent if I could get it installed before 14h Pacific [20:10:19] It's Sunday, and yuvi doesn't appear to be around unfortunately :/ [20:10:52] happens :( [21:10:54] !log striker Added Krenair as project admin [21:10:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Striker/SAL, Master [21:11:18] !log striker Added Merlijn van Deen as project admin [21:11:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Striker/SAL, Master [21:11:33] !log striker Added Andrew Bogott as project admin [21:11:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Striker/SAL, Master [21:12:47] bd808: the 'striker' project is the staging area, right? [21:13:18] yeah, it's my test project for the prod like setup [22:00:45] (03CR) 10Alex Monk: [C: 031] Add pretty error pages [labs/striker] - 10https://gerrit.wikimedia.org/r/307107 (https://phabricator.wikimedia.org/T143949) (owner: 10BryanDavis) [22:16:40] (03CR) 10Alex Monk: [C: 031] Catch and log database errors while saving models [labs/striker] - 10https://gerrit.wikimedia.org/r/307110 (https://phabricator.wikimedia.org/T144082) (owner: 10BryanDavis) [22:21:40] (03CR) 10Alex Monk: [C: 031] Add license, source and docs links to footer [labs/striker] - 10https://gerrit.wikimedia.org/r/307134 (https://phabricator.wikimedia.org/T143972) (owner: 10BryanDavis) [22:51:10] (03CR) 10Alex Monk: [C: 031] Explicitly register admin modules [labs/striker] - 10https://gerrit.wikimedia.org/r/307139 (https://phabricator.wikimedia.org/T144134) (owner: 10BryanDavis) [23:19:05] (03CR) 10Alex Monk: [C: 032] Add pretty error pages [labs/striker] - 10https://gerrit.wikimedia.org/r/307107 (https://phabricator.wikimedia.org/T143949) (owner: 10BryanDavis) [23:19:11] (03CR) 10Alex Monk: [C: 032] Catch and log database errors while saving models [labs/striker] - 10https://gerrit.wikimedia.org/r/307110 (https://phabricator.wikimedia.org/T144082) (owner: 10BryanDavis) [23:19:16] (03CR) 10Alex Monk: [C: 032] Add license, source and docs links to footer [labs/striker] - 10https://gerrit.wikimedia.org/r/307134 (https://phabricator.wikimedia.org/T143972) (owner: 10BryanDavis) [23:19:20] (03CR) 10Alex Monk: [C: 032] Explicitly register admin modules [labs/striker] - 10https://gerrit.wikimedia.org/r/307139 (https://phabricator.wikimedia.org/T144134) (owner: 10BryanDavis) [23:20:27] (03Merged) 10jenkins-bot: Add pretty error pages [labs/striker] - 10https://gerrit.wikimedia.org/r/307107 (https://phabricator.wikimedia.org/T143949) (owner: 10BryanDavis) [23:20:52] (03CR) 10Alex Monk: "not gonna delete the existing site.css version?" [labs/striker/staticfiles] - 10https://gerrit.wikimedia.org/r/307136 (https://phabricator.wikimedia.org/T143972) (owner: 10BryanDavis) [23:21:55] (03Merged) 10jenkins-bot: Add license, source and docs links to footer [labs/striker] - 10https://gerrit.wikimedia.org/r/307134 (https://phabricator.wikimedia.org/T143972) (owner: 10BryanDavis) [23:21:57] (03Merged) 10jenkins-bot: Explicitly register admin modules [labs/striker] - 10https://gerrit.wikimedia.org/r/307139 (https://phabricator.wikimedia.org/T144134) (owner: 10BryanDavis) [23:27:08] (03CR) 10BryanDavis: "> not gonna delete the existing site.css version?" [labs/striker/staticfiles] - 10https://gerrit.wikimedia.org/r/307136 (https://phabricator.wikimedia.org/T143972) (owner: 10BryanDavis)