[13:23:13] !help [13:23:13] petan: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [13:23:32] @dump [13:24:02] oh yes, ignore me :) just testing if wm-bot works as expected, not in need of help [13:24:16] @db [13:26:34] petan: psal jsem ti vedle [13:32:59] PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:08:01] RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [14:12:06] 10Tool-Labs-tools-Pageviews: Add Mediaviews to Pageviews suite - https://phabricator.wikimedia.org/T149642#3370538 (10Sadads) @Qgil : I think so, and there are already some technical modules which do the main behaviour here, but aren't integrated into this suite. For example, https://meta.wikimedia.org/wiki/GLAM... [14:13:10] 10Labs, 10Tool-Labs, 10cloud-services-team (Kanban): Cron Spam: Package 'sudo-ldap' has conffile prompt and needs to be upgraded manually - https://phabricator.wikimedia.org/T168094#3370555 (10Andrew) The way to bypass this is ``` apt-get -o Dpkg::Options::="--force-confold" install sudo-ldap ``` I've run... [14:50:25] 10Tool-Labs-tools-Pageviews, 10Possible-Tech-Projects: Add Mediaviews to Pageviews suite - https://phabricator.wikimedia.org/T149642#3370724 (10Qgil) [14:55:44] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install labtestpuppetmaster2001 - https://phabricator.wikimedia.org/T167157#3370746 (10Papaul) a:05Papaul>03Andrew [15:34:20] PROBLEM - Puppet errors on tools-worker-1019 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:44:31] !log wikilabels running apt-get update and apt-get upgrade on wikilabels-02 [15:44:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [15:57:09] 10Labs, 10Labs-Infrastructure, 10Operations, 10cloud-services-team (Kanban): Puppet CA: virt1000.wikimedia.org' will expire on 2017-08-15 - https://phabricator.wikimedia.org/T168110#3370989 (10Andrew) We're close to setting up new puppetmaster hardware, as per T167905. So I'm going to let this slide in ho... [15:57:28] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-eqiad: rack/setup/install labpuppetmaster100[12].wikimedia.org - https://phabricator.wikimedia.org/T167905#3349071 (10Andrew) [15:57:30] 10Labs, 10Labs-Infrastructure, 10Operations, 10cloud-services-team (Kanban): Puppet CA: virt1000.wikimedia.org' will expire on 2017-08-15 - https://phabricator.wikimedia.org/T168110#3370991 (10Andrew) [15:59:37] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-eqiad: rack/setup/install labpuppetmaster100[12].wikimedia.org - https://phabricator.wikimedia.org/T167905#3371009 (10Andrew) p:05Normal>03High @Cmjohnson, @RobH, the cert for the existing puppetmaster is expiring on July 15th, so I'd like to move ever... [16:00:06] 10Tool-Labs-tools-Other, 10User-bd808, 10cloud-services-team (Kanban): grid-jobs tool broken; loads forever with no actual response - https://phabricator.wikimedia.org/T168653#3371013 (10bd808) [16:00:38] 10Labs, 10Labs-Infrastructure, 10Operations, 10cloud-services-team (Kanban): Puppet CA: virt1000.wikimedia.org' will expire on 2017-08-15 - https://phabricator.wikimedia.org/T168110#3371028 (10Andrew) 05Open>03stalled [16:01:07] !log tools.grid-jobs Disabled hourly purge cron job while debugging T168653 [16:01:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.grid-jobs/SAL [16:01:10] T168653: grid-jobs tool broken; loads forever with no actual response - https://phabricator.wikimedia.org/T168653 [16:04:05] 10Tool-Labs-tools-Other, 10User-bd808, 10cloud-services-team (Kanban): grid-jobs tool broken; loads forever with no actual response - https://phabricator.wikimedia.org/T168653#3371047 (10bd808) ``` [2017-06-22 12:49:54,451] ERROR in app: Exception on / [GET] Traceback (most recent call last): File "/data/p... [16:19:56] !log tools Backed up elasticsearch indexes to personal laptop using elasticdump incase T164842 goes horribly wrong [16:20:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:20:00] T164842: Upgrade Tool Labs elasticsearch to 5.x - https://phabricator.wikimedia.org/T164842 [16:23:22] :) [16:24:09] 10cloud-services-team, 10DBA, 10Operations: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3371135 (10Andrew) [16:25:48] 10cloud-services-team, 10DBA, 10Operations: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3369055 (10Andrew) [16:30:41] bd808: Quick question, I'm about to merge a patch with a schema change - will it be automatically applied on beta or do I need to run the db patch file manually? (it has no effect if not applied, so no worries either way) [16:31:11] marktraceur: there is a jenkins job that runs update.php occasionally... [16:31:22] Well then I'm probably good. Thanks! [16:31:30] https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad/ [16:31:36] 10Labs, 10Patch-For-Review, 10cloud-services-team (Kanban): labmon1001 disk filling up - https://phabricator.wikimedia.org/T168344#3371146 (10Andrew) I've installed a cleanup cron on labmon1001. I'm going to give it a day and make sure that things are properly cleaned up, then this can be closed. Today, us... [16:35:34] ebernhardson: I've getting "Error: encountered environment variables that are no longer supported. Use jvm.options or ES_JAVA_OPTS to configure the JVM" when trying to start the new elasticsearch package. [16:35:43] I have forced a puppet run too. [16:35:55] bd808: hmm, sec looking [16:37:19] bd808: is elasticsearch::version set to 5 in puppet? [16:37:31] probably not! [16:37:56] it defaults to 5, but it supports 2 or 5 and generates different config files depending on the setting. I think that might be it [16:38:32] * bd808 pokes around in horizon [16:39:02] bd808: i suppose the easy way to tell would be look at the top of /etc/default/elasticsearch, if it starts with `# Run elasticsearch as this user ID and group ID` then you have v2 configs [16:39:24] RECOVERY - Puppet errors on tools-worker-1019 is OK: OK: Less than 1.00% above the threshold [0.0] [16:39:27] yup. that is the case [16:40:07] hieradata/labs.yaml is pinned to version: 2 [16:40:20] I'll change in project hiera [16:42:48] closer. now complaining that I'm not running java8 [16:48:00] ugh. The index [[bash/8j-7PJe8TFWSfT5pbViVWg]] was created with version [1.7.1] but the minimum compatible version is [2.0.0-beta1]. It should be re-indexed in Elasticsearch 2.x before upgrading to 5.3.2. [16:48:30] missed some steps in the past apparently :/ [16:49:08] :( yea. best is to use the elasticsearch migration plugin and run it against the cluster, it reports all those things (albeit a bit verbosely ... giving me the same errors 2000 times because we have 200 indices with roughly the same schema...) [16:49:14] s/200 indices/2000 indices/ [16:56:20] andrewbogott: I cannot access http://etytree-virtuoso.wmflabs.org/ [16:57:19] Ester: is that your project? [16:57:24] yes [16:57:29] https://tools.wmflabs.org/openstack-browser/project/etytree [16:57:58] I can ssh [16:58:04] Ester: Have you verified that your server (whatever it is) is up and running on the host? [16:58:16] I haven't turned it off [16:58:19] didn't check [16:58:26] that's probably the first thing to check :) [16:58:31] everything was rebooted yesterday [16:58:32] :) [16:58:37] oh ok [16:58:53] thanks! [17:03:07] !log tools Rolled back attempt at Elasticsearch upgrade. Indices need to be rebuilt with 2.x before 5.x can be installed. T164842 [17:03:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:03:12] T164842: Upgrade Tool Labs elasticsearch to 5.x - https://phabricator.wikimedia.org/T164842 [17:05:59] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-eqiad: rack/setup/install labpuppetmaster100[12].wikimedia.org - https://phabricator.wikimedia.org/T167905#3371358 (10RobH) https://gerrit.wikimedia.org/r/#/c/360874/ [17:08:21] 10cloud-services-team, 10DBA, 10Operations: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3371363 (10Andrew) [17:13:49] 10Labs, 10Tool-Labs, 10User-bd808, 10cloud-services-team (Kanban): Upgrade Tool Labs elasticsearch to 5.x - https://phabricator.wikimedia.org/T164842#3371384 (10bd808) Aborted upgrade because of this error: ``` Caused by: java.lang.IllegalStateException: The index [[bash/8j-7PJe8TFWSfT5pbViVWg]] was create... [17:19:22] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-eqiad: rack/setup/install labpuppetmaster100[12].wikimedia.org - https://phabricator.wikimedia.org/T167905#3371397 (10Cmjohnson) Connected labpuppetmaster1001 b8 ge-8/0/11 labpuppetmaster1002 d6 ge-6/0/1 [17:19:41] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-eqiad: rack/setup/install labpuppetmaster100[12].wikimedia.org - https://phabricator.wikimedia.org/T167905#3371398 (10Cmjohnson) [18:10:33] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-eqiad, and 2 others: rack/setup/install labvirt101[5-8] - https://phabricator.wikimedia.org/T165531#3371629 (10Cmjohnson) 2nd ethernet connection...not setup on switch yet Labvirt1015 2/0/21 Labvirt1016 3/0/12 labvirt1017. 7/0/11 Labvirt1018 8/0/13 [18:11:40] Hello, I have an tool commons-mass-description at toollabs. In its ~/www there is a directory named static and its content should appear at https://tools-static.wmflabs.org/commons-mass-description. After every update of ~/www/static/js/main.js the tools-static host throw 403 Forbidden error. Does anybody know what am I doing wrong? [18:12:53] Fascinating is that if I go to the directory-listing url (so https://tools-static.wmflabs.org/commons-mass-description/js/ in this case and click at main.js, no 403 error is shown. [18:13:16] This seems to be a rule. I've updated the file again and 403 Forrbidden was thrown until I didn't visit the https://tools-static.wmflabs.org/commons-mass-description/js/ URL. [18:13:20] What's wrong? [18:13:24] !log planet - apply 'project puppet' role::planet_server, create fresh instance "planet-hotdog" for stretch testing of the "rawdog" package. adding paladox as member. https://gerrit.wikimedia.org/r/#/q/topic:planet-stretch+(status:open+OR+status:merged) [18:13:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Planet/SAL [18:37:39] Urbanecm: what exact url ends up giving you the 403? [18:38:21] Urbanecm___: so https://tools-static.wmflabs.org/commons-mass-description/js/main.js give you a 403 until you visit https://tools-static.wmflabs.org/commons-mass-description/js/ ? [18:41:41] I'm getting more "backend fetch failed" on deployment-cache-upload04 when trying to get some thumbnails for STL files on beta commons - is there any reason this would be happening that's not my fault? :P [18:42:01] Figured I'd ask since yesterday's weird errors weren't expected based on what I saw in this channel beforehand [18:42:47] marktraceur: no maintenance that we are doing for the cluster today. You might check in -releng to see if they know if anything is up [18:56:47] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Bogus "not a valid project" errors - https://phabricator.wikimedia.org/T168676#3371738 (10kaldari) [18:57:04] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Bogus "not a valid project" errors - https://phabricator.wikimedia.org/T168676#3371750 (10kaldari) p:05Triage>03High [18:58:23] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Bogus "not a valid project" errors - https://phabricator.wikimedia.org/T168676#3371738 (10kaldari) This is while not logged in via OAuth. [19:02:12] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Restrict access to users' edit stats unless opted-in - https://phabricator.wikimedia.org/T165401#3371767 (10kaldari) Was not able to test due to T168676. [19:05:25] 10Labs: Add new Cloud Services domains to public suffix list - https://phabricator.wikimedia.org/T168677#3371779 (10valhallasw) [19:27:42] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-eqiad: rack/setup/install labcontrol100[34] - https://phabricator.wikimedia.org/T165781#3371884 (10Cmjohnson) [19:28:12] 10Tool-Labs-tools-Xtools, 10CSS: The fields in X!'s tools are too small to type in mobile - https://phabricator.wikimedia.org/T168680#3371885 (10wassan.anmol117) [19:50:08] 10Tool-Labs-tools-Xtools, 10CSS: The fields in X!'s tools are too small to type in mobile - https://phabricator.wikimedia.org/T168680#3371953 (10Matthewrbowker) [19:50:10] 10Tool-Labs-tools-Xtools: Ensure xTools Rebirth is fully responsive - https://phabricator.wikimedia.org/T165706#3371952 (10Matthewrbowker) [20:00:42] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Bogus "not a valid project" errors - https://phabricator.wikimedia.org/T168676#3371977 (10Matthewrbowker) 05Open>03Invalid Works for me. Queries need to be optimized but that's outside of the scope of this task. [20:12:47] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1422 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:47:44] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1422 is OK: OK: Less than 1.00% above the threshold [0.0] [20:50:09] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Bogus "not a valid project" errors - https://phabricator.wikimedia.org/T168676#3372095 (10kaldari) 05Invalid>03Open Still broken for me. Steps to reproduce: * In Firefox, go to http://xtools.wmflabs.org/ec * Fill in "fr.wikipedia.org" as the project * Fill... [21:05:10] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Bogus "not a valid project" errors - https://phabricator.wikimedia.org/T168676#3372145 (10Matthewrbowker) Ahhhh, you were testing on the new "live" instance. I was testing on the "dev" instance, which succeeded. I wonder what the difference is between the in... [21:06:24] 10Labs-Vagrant, 10Fundraising-Backlog, 10MediaWiki-Vagrant, 10Easy: Make it easier to use the fundraising puppet role on labs - https://phabricator.wikimedia.org/T102304#3372172 (10DStrine) a:05awight>03None [21:13:53] 10Labs: Use wmcloud.org domain for PAWS - https://phabricator.wikimedia.org/T168686#3372279 (10yuvipanda) [21:33:08] 10cloud-services-team, 10DBA, 10Operations: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3372376 (10madhuvishy) Hi all, So current status is: - labsdb1001 and 1003: Cloud team needs to announce user maintenance, and handle dns switchover during reboots(I'm not sure what t... [22:24:01] 10Labs-project-Extdist, 10Patch-For-Review: Migrate extdist.wmflabs.org to Debian stretch - https://phabricator.wikimedia.org/T168456#3373557 (10Legoktm) Currently set up a new instance at https://extdist-stretch.wmflabs.org/dist/ [22:38:40] 10Labs, 10Labs-Infrastructure, 10Operations, 10cloud-services-team (Kanban): Puppet CA: virt1000.wikimedia.org' will expire on 2017-08-15 - https://phabricator.wikimedia.org/T168110#3373570 (10akosiaris) The difficult part will be getting all the old clients (old VMs practically) getting to trust the new p... [23:12:52] 10Tool-Labs-tools-Other, 10User-bd808, 10cloud-services-team (Kanban): grid-jobs tool broken; loads forever with no actual response - https://phabricator.wikimedia.org/T168653#3371013 (10zhuyifei1999) ``` >>> sorted(['', None]) Traceback (most recent call last): File "", line 1, in TypeErro... [23:14:08] 10Labs, 10Phabricator, 10wikitech.wikimedia.org, 10LDAP, and 2 others: Blocking an account on wikitech should disable LDAP logins - https://phabricator.wikimedia.org/T168692#3373658 (10mmodell) [23:20:58] 10Tool-Labs-tools-Other, 10User-bd808, 10cloud-services-team (Kanban): grid-jobs tool broken; loads forever with no actual response - https://phabricator.wikimedia.org/T168653#3373695 (10zhuyifei1999) [[https://phabricator.wikimedia.org/source/tool-grid-jobs/browse/master/grid_jobs/__init__.py;15634c3b2d5dcc... [23:27:03] 10Tool-Labs-tools-Other, 10User-bd808, 10cloud-services-team (Kanban): grid-jobs tool broken; loads forever with no actual response - https://phabricator.wikimedia.org/T168653#3373704 (10zhuyifei1999) Test running the same code shows two instances of `None`: * `(None, 'backlogupdater', 'tools-exec-1417')` *... [23:32:22] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Bogus "not a valid project" errors - https://phabricator.wikimedia.org/T168676#3373711 (10MusikAnimal) The API that it is using works at the time of writing: http://xtools.wmflabs.org/api/namespaces/fr.wikipedia.org Compare to http://xtools.wmflabs.org/api/na... [23:43:35] bd808, I have trouble figuring out why 'gpy' tool quit the last time, was it also the memory? "qacct -j gpy" output doesn't fit a screenfull [23:48:12] 10Labs-project-Wikistats: wikistats: add new wikipedias: kbp, khw, dty and pt.wikimedia - https://phabricator.wikimedia.org/T160947#3373730 (10Dzahn) kbp added per T160868 MariaDB [wikistats]> insert into wikipedias (prefix,lang,loclang,method) values ("kbp","Kabiye","Kabɩyɛ","8"); [23:55:00] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Bogus "not a valid project" errors - https://phabricator.wikimedia.org/T168676#3373743 (10Samwilson) I think it works now (I fixed a bug in prod deployment relating to permissions on the `var` directory, and usage of Redis). [23:59:43] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Bogus "not a valid project" errors - https://phabricator.wikimedia.org/T168676#3373753 (10kaldari) @Samwilson: If there are some clean-up tasks that need to be done each time the code is updated, you might want to implement a [[ https://getcomposer.org/doc/art...