[00:49:45] PROBLEM - Puppet errors on tools-exec-1416 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [00:57:59] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Convert xtools intuition to its own repository - https://phabricator.wikimedia.org/T165708#3274486 (10Krinkle) If you run into anything odd or confusing.. let me know, anything I can do to help! [00:59:35] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3330053 (10Krinkle) >>! In T167345#3380933, @Ricordisamoa wrote: > With Overpass font: > {F8533593} I like the curving better on this one (xtools5 with Overpass font). The previous one has a mo... [01:16:04] PROBLEM - Puppet errors on tools-exec-1412 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:23:37] PROBLEM - Puppet errors on tools-worker-1008 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:24:43] RECOVERY - Puppet errors on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [01:28:11] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Update all of the Xtools with new backend - https://phabricator.wikimedia.org/T165399#3381072 (10Samwilson) The Xtools/AppBundle differentiation is about XTools-only vs Symfony-specific classes. Not that it's all that strickt, so as you say they could probably... [01:56:04] RECOVERY - Puppet errors on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [01:58:37] RECOVERY - Puppet errors on tools-worker-1008 is OK: OK: Less than 1.00% above the threshold [0.0] [02:09:17] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Update all of the Xtools with new backend - https://phabricator.wikimedia.org/T165399#3381089 (10kaldari) 05Open>03Resolved [02:09:20] 10Tool-Labs-tools-Xtools, 10Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#3381091 (10kaldari) [02:34:29] (03CR) 10Minhtq15: [C: 031] first commit [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/361020 (owner: 10Minhtq15) [02:52:23] (03CR) 10Minhtq15: [C: 031] second [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/361021 (owner: 10Minhtq15) [03:27:57] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [03:37:27] PROBLEM - Free space - all mounts on tools-bastion-02 is CRITICAL: CRITICAL: tools.tools-bastion-02.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-bastion-02.diskspace.root.byte_percentfree (<10.00%) [03:57:28] PROBLEM - Free space - all mounts on tools-bastion-02 is CRITICAL: CRITICAL: tools.tools-bastion-02.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-bastion-02.diskspace.root.byte_percentfree (<30.00%) [03:58:02] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [05:37:24] 10Tool-Labs-tools-Other: get_current_user() returns empty string - https://phabricator.wikimedia.org/T167546#3336324 (10Betateschter) The issue seems to be solved. At least the tools of @Magnus are working fine again. [06:24:00] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:25:58] (03PS1) 10Minhtq15: asd [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/361626 [06:41:47] (03CR) 10Minhtq15: [C: 031] asd [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/361626 (owner: 10Minhtq15) [06:59:01] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [06:59:31] PROBLEM - Puppet errors on tools-bastion-05 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [07:18:56] (03PS1) 10Minhtq15: fourth [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/361628 [07:28:04] (03PS1) 10Minhtq15: a [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/361629 [07:39:29] RECOVERY - Puppet errors on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [07:49:36] PROBLEM - Puppet errors on tools-worker-1008 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [08:26:48] 10Labs, 10Labs-Infrastructure, 10LDAP-Access-Requests, 10Operations, 10Patch-For-Review: Make all ldap users have a sane shell (/bin/bash) - https://phabricator.wikimedia.org/T86668#3381412 (10hashar) Thank you @bd808 ! [08:59:37] RECOVERY - Puppet errors on tools-worker-1008 is OK: OK: Less than 1.00% above the threshold [0.0] [14:06:00] (03PS1) 10Gilles: Move wikibugs performance updates to #wikimedia-perf-bots [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/361672 [14:27:12] 10Labs, 10DBA, 10Patch-For-Review: Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" - https://phabricator.wikimedia.org/T153213#3382376 (10chasemp) >>! In T153213#3377614, @Marostegui wrote: > I have cleaned up the views, the only pending thing would be to merge: https://g... [14:28:31] 10Labs, 10DBA, 10Patch-For-Review: Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" - https://phabricator.wikimedia.org/T153213#3382386 (10Marostegui) Oh thank you @chasemp! I didn't know that - next time I will clean it up with the script :-) [14:37:47] 10Labs, 10DBA, 10Patch-For-Review: Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" - https://phabricator.wikimedia.org/T153213#3382413 (10chasemp) A run post merge on labsdb1009: > sudo puppet agent --test (updates the repo) > time maintain-views --all-databases --clean... [15:14:19] 10Labs, 10Operations, 10wikitech.wikimedia.org: wikitech-static sync check shouldn't happen so often - https://phabricator.wikimedia.org/T168962#3382632 (10Andrew) [15:46:47] (03CR) 10Merlijn van Deen: [C: 032] Move wikibugs performance updates to #wikimedia-perf-bots [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/361672 (owner: 10Gilles) [15:47:08] (03Merged) 10jenkins-bot: Move wikibugs performance updates to #wikimedia-perf-bots [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/361672 (owner: 10Gilles) [15:47:17] (03CR) 10jenkins-bot: Move wikibugs performance updates to #wikimedia-perf-bots [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/361672 (owner: 10Gilles) [15:47:33] !log tools.wikibugs Updated channels.yaml to: 883245b221a820da7b660519747d994dd544840e Move wikibugs performance updates to #wikimedia-perf-bots [16:11:10] (03CR) 10Gilles: "The bot hasn't joined the new channel, is there a special step for that beyond this change?" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/361672 (owner: 10Gilles) [16:19:17] (03CR) 10Paladox: "> The bot hasn't joined the new channel, is there a special step for" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/361672 (owner: 10Gilles) [16:30:57] (03CR) 10Gilles: "That's the case. Maybe the bot didn't like the fact that it was a channel renaming?" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/361672 (owner: 10Gilles) [16:33:51] (03CR) 10Gilles: "Ah, it works now. I guess it was waiting for the first task update to join the channel." [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/361672 (owner: 10Gilles) [16:50:28] hi, i would like to check how do i use this option [16:50:28] Fabiorahamim/EditCounterGlobalOptIn.js (metawiki) [16:52:10] !help, i would like to check how do i use this option Fabiorahamim/EditCounterGlobalOptIn.js (metawiki) [16:52:47] Guest61884: is that a gadget on metawiki? [16:53:05] hi, thanks for response, yes [16:54:44] I'm not exactly sure what help you are needing. Can you give me a link to a page that you are seeing this on? [16:55:13] This channel is mainly for discussing building tools and running them on Wikimedia servers [16:55:50] ok I understand [16:56:07] sorry for disturb, thanks for cooperation :) [16:56:29] I'm glad to try and help you but your description of the problem is not clear [16:59:05] I would like to configure this gadget in my page [16:59:07] https://tools.wmflabs.org/xtools-ec/?user=Fabiorahamim&lang=he&wiki=wikipedia&uselang=en [16:59:46] under Top edited pages [17:00:39] Guest61884: ok. so you want to enable the feature that is described in the "Time card" section for xtools-ec [17:03:11] Guest61884: It looks like you just need to make a blank page under your user page space to enable it [17:04:38] Guest61884: so you need to create a page at https://meta.wikimedia.org/wiki/User:____/EditCounterGlobalOptIn.js where ____ is replaced with your username [17:08:35] םל [17:08:40] ok, Thanks a lot! [17:12:03] 10Labs, 10Datasets-General-or-Unknown, 10Dumps-Generation, 10Operations, 10hardware-requests: Eqiad: Hardware request for labstore1006/7, dataset1002/3 - https://phabricator.wikimedia.org/T161311#3383469 (10RobH) 05stalled>03Resolved This request has been fulfilled, and systems are being setup on T16... [17:12:27] 10Labs, 10Operations, 10hardware-requests: eqiad: (1) hardware access request for labnodepool1002 - https://phabricator.wikimedia.org/T161753#3383474 (10RobH) 05stalled>03Resolved This has been ordered and setup is tracked via T168407. [17:12:46] 10Labs, 10Operations, 10hardware-requests: Codfw: (1) hardware access request for labtestvirt2003 [region 2] - https://phabricator.wikimedia.org/T161765#3383478 (10RobH) 05Open>03Resolved Ordered and setup via T166564 [17:21:41] 10cloud-services-team, 10DBA, 10Operations: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3383543 (10madhuvishy) Thanks for the detailed explanation @jcrespo. For labsdb1001 and 1003, I'll check with Chris and schedule the dns switchover and the reboots to happen this week/e... [17:23:31] 10Labs, 10Labs-Infrastructure, 10LDAP-Access-Requests, 10Operations, 10Patch-For-Review: Make all ldap users have a sane shell (/bin/bash) - https://phabricator.wikimedia.org/T86668#3383568 (10bd808) Just for the historical record, here's what I did to edit the `loginShell` entries. Commands were run fro... [17:24:14] 10cloud-services-team, 10DBA, 10Operations: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3383577 (10madhuvishy) @Cmjohnson Hi! We are looking at rebooting labsdb1001 and 1003, and it seems like these boxes may not come up automatically on reboot. Jaime recommended that it w... [18:24:11] 10Quarry: Login to somebody's account - https://phabricator.wikimedia.org/T120988#1866469 (10Milimetric) This happened again, after I rebooted all the quarry instances. Must be some shared Flask state or something. No time to look into it, but posting for future reference. [18:25:18] hi, puppet is breaking in labs for me now [18:25:25] Warning: Unable to fetch my node definition, but the agent run will continue: [18:25:25] Warning: Find /production/node/jenkins-slave-01.git.eqiad.wmflabs?transaction_uuid=2a269e6b-83f1-4b3b-82f3-bedb... resulted in 404 with the message: [18:25:26] [18:25:26] 404 Not Found [18:25:26] [18:25:26]

Not Found

[18:25:27]

The requested URL /production/node/jenkins-slave-01.git.eqiad.wmflabs was not found on this server.

[18:25:28] [18:25:30] im getting alot of ^^ [18:25:36] and on the puppet master too [18:26:45] 'on the puppet master too'? i.e. you're using your own puppetmaster, not the central labs one? [18:27:27] yes [18:27:33] the error sounds like a misconfiguration of the puppetmaster host -- e.g. starting a webserver instead of the puppetmaster [18:27:38] nope [18:27:45] i git pulled [18:27:54] and it started failing after doing puppet agent -tv [18:28:33] it's happening to the ores project too [18:28:40] which does not use the puppet master i use [18:29:44] andrewbogott: ^ reports of puppet weirdness. Maybe related to your cleanup patch? [18:29:51] yeah, I have a fix [18:30:44] valhallasw`cloud: btw, all modern puppetmasters should use apache/passenger instead of the puppetmaster service. We standardized on that a while back. [18:31:11] Ahhh [18:31:20] That explains why the error message looks apache-y [18:33:25] (03Draft1) 10Paladox: Up interval level for puppet master due to outage in labs [labs/icinga2] - 10https://gerrit.wikimedia.org/r/361714 [18:33:27] (03PS2) 10Paladox: Up interval level for puppet master due to outage in labs [labs/icinga2] - 10https://gerrit.wikimedia.org/r/361714 [18:33:39] (03CR) 10Paladox: [V: 032 C: 032] Up interval level for puppet master due to outage in labs [labs/icinga2] - 10https://gerrit.wikimedia.org/r/361714 (owner: 10Paladox) [18:34:54] just thought to my self that ^^ wont even work lol until puppet is fixed. [18:34:58] will need to do it manually [18:35:50] puppet should be fixed by now [18:37:21] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1428 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:37:22] PROBLEM - Puppet errors on tools-exec-1426 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:37:26] PROBLEM - Puppet errors on tools-exec-1421 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:37:26] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1405 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:37:30] PROBLEM - Puppet errors on tools-k8s-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:37:32] PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:37:34] still broken [18:37:38] PROBLEM - Puppet errors on tools-static-10 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:37:42] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:37:42] PROBLEM - Puppet errors on tools-exec-1419 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:37:42] PROBLEM - Puppet errors on tools-services-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:37:44] PROBLEM - Puppet errors on tools-exec-1416 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:37:44] PROBLEM - Puppet errors on tools-cron-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:37:46] PROBLEM - Puppet errors on tools-docker-builder-05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:37:51] PROBLEM - Puppet errors on tools-worker-1026 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:37:55] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:37:55] andrewbogott ^^ [18:37:59] PROBLEM - Puppet errors on tools-redis-1002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:01] PROBLEM - Puppet errors on tools-exec-gift-trusty-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:03] PROBLEM - Puppet errors on tools-worker-1003 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:03] PROBLEM - Puppet errors on tools-paws-master-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:05] PROBLEM - Puppet errors on tools-exec-1412 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:11] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1414 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:11] PROBLEM - Puppet errors on tools-exec-1432 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:14] PROBLEM - Puppet errors on tools-exec-1415 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:17] PROBLEM - Puppet errors on tools-exec-1410 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:34] PROBLEM - Puppet errors on tools-exec-1404 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:36] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:37] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:42] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1425 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:48] PROBLEM - Puppet errors on tools-worker-1013 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:50] PROBLEM - Puppet errors on tools-exec-1418 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:38:56] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1426 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:38:58] PROBLEM - Puppet errors on tools-worker-1016 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:38:58] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:00] PROBLEM - Puppet errors on tools-logs-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:39:02] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [18:39:05] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:05] PROBLEM - Puppet errors on tools-exec-1440 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:07] PROBLEM - Puppet errors on tools-exec-1407 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:07] PROBLEM - Puppet errors on tools-worker-1010 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:11] PROBLEM - Puppet errors on tools-worker-1017 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:13] PROBLEM - Puppet errors on tools-prometheus-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:13] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1415 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:39:13] PROBLEM - Puppet errors on tools-worker-1021 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:21] PROBLEM - Puppet errors on tools-proxy-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:23] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:23] PROBLEM - Puppet errors on tools-exec-1406 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:27] PROBLEM - Puppet errors on tools-worker-1002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:29] PROBLEM - Puppet errors on tools-exec-1424 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:39:30] PROBLEM - Puppet errors on tools-exec-1413 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:34] PROBLEM - Puppet errors on tools-flannel-etcd-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:36] PROBLEM - Puppet errors on tools-checker-01 is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [0.0] [18:39:36] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1417 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:40] PROBLEM - Puppet errors on tools-worker-1022 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:40] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:40] PROBLEM - Puppet errors on tools-exec-1402 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:46] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1422 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:46] PROBLEM - Puppet errors on tools-elastic-03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:53] this is such a tangle :( [18:39:55] PROBLEM - Puppet errors on tools-exec-1417 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:59] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:03] PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:05] PROBLEM - Puppet errors on tools-worker-1018 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:05] PROBLEM - Puppet errors on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:09] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1407 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:09] PROBLEM - Puppet errors on tools-bastion-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:40:11] PROBLEM - Puppet errors on tools-flannel-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:11] PROBLEM - Puppet errors on tools-worker-1015 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:15] PROBLEM - Puppet errors on tools-exec-1431 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:15] PROBLEM - Puppet errors on tools-flannel-etcd-03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:20] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1421 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:40:20] PROBLEM - Puppet errors on tools-worker-1014 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:22] PROBLEM - Puppet errors on tools-redis-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:24] PROBLEM - Puppet errors on tools-worker-1005 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:28] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:30] PROBLEM - Puppet errors on tools-bastion-05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:30] PROBLEM - Puppet errors on tools-puppetmaster-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:40:36] PROBLEM - Puppet errors on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:36] PROBLEM - Puppet errors on tools-worker-1008 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:38] PROBLEM - Puppet errors on tools-exec-1429 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:49] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [0.0] [18:40:49] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:49] PROBLEM - Puppet errors on tools-exec-1441 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:51] PROBLEM - Puppet errors on tools-grid-master is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:40:51] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:51] PROBLEM - Puppet errors on tools-exec-1433 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:55] PROBLEM - Puppet errors on tools-docker-registry-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:55] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1419 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [18:40:55] PROBLEM - Puppet errors on tools-exec-1423 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:41:00] PROBLEM - Puppet errors on tools-worker-1012 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [18:41:02] PROBLEM - Puppet errors on tools-exec-1409 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:41:04] PROBLEM - Puppet errors on tools-exec-1442 is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [0.0] [18:41:10] andrewbogott: ^ [18:41:11] ? [18:41:14] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1424 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:41:14] PROBLEM - Puppet errors on tools-exec-1435 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:41:20] andrewbogott: i am in a meeting, what's up? [18:41:23] are you on it [18:41:38] It's fine, I'm just fighting with puppet-on-puppet-on-puppet [18:41:42] ok man thanks [18:42:02] PROBLEM - Puppet errors on tools-exec-1428 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:04] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:42:05] PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:11] it's puppet all the way down [18:42:18] PROBLEM - Puppet errors on tools-package-builder-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:20] PROBLEM - Puppet errors on tools-services-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:21] PROBLEM - Puppet errors on tools-exec-1430 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:23] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:23] PROBLEM - Puppet errors on tools-worker-1019 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:23] PROBLEM - Puppet errors on tools-exec-1408 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:25] PROBLEM - Puppet errors on tools-proxy-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:25] PROBLEM - Puppet errors on tools-worker-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:25] PROBLEM - Puppet errors on tools-exec-1414 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:29] PROBLEM - Puppet errors on tools-checker-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:29] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:30] PROBLEM - Puppet errors on tools-exec-1438 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:32] PROBLEM - Puppet errors on tools-exec-1405 is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [0.0] [18:42:32] PROBLEM - Puppet errors on tools-exec-1401 is CRITICAL: CRITICAL: 90.00% of data above the critical threshold [0.0] [18:42:36] PROBLEM - Puppet errors on tools-mail is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:38] PROBLEM - Puppet errors on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:38] PROBLEM - Puppet errors on tools-exec-1425 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:40] PROBLEM - Puppet errors on tools-grid-shadow is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [18:42:41] PROBLEM - Puppet errors on tools-k8s-etcd-03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:41] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:41] PROBLEM - Puppet errors on tools-exec-1422 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:43] PROBLEM - Puppet errors on tools-worker-1027 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:43] PROBLEM - Puppet errors on tools-paws-worker-1001 is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [0.0] [18:42:45] PROBLEM - Puppet errors on tools-worker-1006 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:45] PROBLEM - Puppet errors on tools-worker-1004 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:48] PROBLEM - Puppet errors on tools-elastic-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:48] PROBLEM - Puppet errors on tools-docker-registry-01 is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [0.0] [18:42:48] PROBLEM - Puppet errors on tools-static-11 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:53] PROBLEM - Puppet errors on tools-exec-1403 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:53] PROBLEM - Puppet errors on tools-worker-1025 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:53] PROBLEM - Puppet errors on tools-worker-1023 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:42:56] PROBLEM - Puppet errors on tools-k8s-etcd-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:43:00] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1420 is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [0.0] [18:43:01] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1418 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:43:01] PROBLEM - Puppet errors on tools-elastic-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:43:03] PROBLEM - Puppet errors on tools-prometheus-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:43:07] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1427 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [18:43:07] PROBLEM - Puppet errors on tools-exec-1427 is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [0.0] [18:43:13] PROBLEM - Puppet errors on tools-worker-1011 is CRITICAL: CRITICAL: 77.78% of data above the critical threshold [0.0] [18:43:13] PROBLEM - Puppet errors on tools-k8s-master-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:02:49] (03PS1) 10Paladox: Revert "Up interval level for puppet master due to outage in labs" [labs/icinga2] - 10https://gerrit.wikimedia.org/r/361727 [19:02:53] (03CR) 10Paladox: [V: 032 C: 032] Revert "Up interval level for puppet master due to outage in labs" [labs/icinga2] - 10https://gerrit.wikimedia.org/r/361727 (owner: 10Paladox) [19:04:22] it seems to be recovering for ores, but the git and phabricator project that use a custom puppet master seem to still be failing. [19:04:47] andrewbogott ^^ [19:04:56] paladox: please stop pinging me [19:05:03] ok sorry. [19:05:25] I'm working on it [19:06:49] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:07:19] PROBLEM - Puppet errors on tools-worker-1009 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:12:13] mobrovac: beta issue might be related to all the puppet errors on tools labs [19:12:20] eg puppetmasters being broken someho [19:12:21] w [19:13:27] oh ok [19:13:32] thnx hashar [19:14:51] 10cloud-services-team, 10DBA, 10Operations: Labsdb* servers need to be rebooted - https://phabricator.wikimedia.org/T168584#3384109 (10jcrespo) > it would be easier to just reboot the boxes I am ok with that if you are ok with that. Announcement should be done, though- on last upgrade people got upset even... [19:32:10] hi! I am looking for help about python3 UWSGI web services [19:32:33] I run https://tools.wmflabs.org/openrefine-wikidata/ which uses that [19:33:12] today I pushed a rather benign code change and I cannot get the web service back up [19:33:23] (I tried again with the previous code and nothing changes) [19:33:44] (the previous code does not work either) [19:33:52] all I get is a 502 bad gateway error [19:34:45] the UWSGI logs look fine: http://pintoch.ulminfo.fr/3c469a6629/uwsgi.txt [19:35:04] running the app with "python app.py" also works fine [19:35:11] any idea where to look for errors? [19:36:02] the command I use to restart the web service is "webservice --backend=kubernetes python restart" [19:36:48] pintoch: ok, so the long wait before the 502 suggests the webservice is timing out [19:37:01] kubectl get pods shows there is a pod running [19:37:24] oh I get an error now: http://pintoch.ulminfo.fr/9b4de1dba8/kubernetes.txt [19:37:24] tools.openrefine-wikidata@tools-bastion-03:~$ kubectl logs openrefine-wikidata-66450673-wkuot [19:37:24] open("/usr/lib/uwsgi/plugins/python_plugin.so"): No such file or directory [core/utils.c line 3659] [19:37:27] hmmm, that's not good [19:37:45] that error is just the script that starts the service, not an error from the service itself [19:37:51] yeah [19:37:59] bd808: did we recently rebuild the base images for k8s? [19:38:40] valhallasw`cloud: it's been quite a while I think... like 4-5 weeks? [19:39:24] hm, it seems the plugin was maybe renamed? ls shows /usr/lib/uwsgi/plugins/python3_plugin.so does exist [19:41:34] hmm interesting… I have no idea where this can come from [19:42:21] I could try rebuild my virtualenv [19:42:55] the error suggests it's not the virtualenv, but the uwsgi plugin. I'm not sure how to get it to load the right plugin though [19:45:09] huuuuuh [19:45:17] but the webservice *is* running! [19:45:22] try telnet 192.168.206.8 8000 [19:45:52] indeed [19:46:06] 192.168.155.6 now [19:46:25] (I just restarted it with a fresh venv, but indeed it does not change anything) [19:48:34] valhallasw`cloud: I'll have some time in 30 minutes or so to look [19:49:23] Ok. The pod is now running as 192.168.155.6, but http://192.168.155.6:8000 just hangs [19:50:03] usually when I have an error like that, it is because I have added a new dependency and forgot to install it in the virtualenv. In that case, running "python app.py" fails with an error [19:51:19] 'error like that' = HTTP/502 from the proxy? [19:52:11] yes [19:52:59] Sure. That error only says 'the backend webservice is not responding in time', which would also happen for issues like that. [19:53:38] The weird thing here is that the webservice /was/ running (I could connect to it on the internal network), but the external proxy was unable to do so [19:55:28] the pod seems to be hanging altogether -- even "kubectl exec openrefine-wikidata-66450673-pqgla -- ls" hangs [19:58:48] ok, killing & restarting pod helps; now webservice is running as http://192.168.234.10:8000/openrefine-wikidata [19:59:08] and https://tools.wmflabs.org/openrefine-wikidata is also back up [19:59:10] ...hrm. [19:59:19] you're my hero ❤ [19:59:30] how did you restart the pod? [19:59:42] kubectl delete pod [19:59:51] but webservice stop would probably also have worked [20:00:31] okay, I have never touched kubernetes directly so I need to get into that [20:03:10] bd808: short recap at https://wikitech.wikimedia.org/wiki/User:Merlijn_van_Deen/k8s_notes [20:14:09] webservice restart does indeed kill the pod and spawn a new one [20:25:29] RECOVERY - Puppet errors on tools-puppetmaster-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:28:03] RECOVERY - Puppet errors on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [20:28:06] madhuvishy: cool, good to know :-) [20:32:24] RECOVERY - Puppet errors on tools-proxy-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:24] RECOVERY - Puppet errors on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:24] RECOVERY - Puppet errors on tools-exec-1421 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:26] RECOVERY - Puppet errors on tools-checker-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:33] RECOVERY - Puppet errors on tools-k8s-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:34] RECOVERY - Puppet errors on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:36] RECOVERY - Puppet errors on tools-static-10 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:39] RECOVERY - Puppet errors on tools-k8s-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:39] RECOVERY - Puppet errors on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:39] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:44] RECOVERY - Puppet errors on tools-paws-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:45] RECOVERY - Puppet errors on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:45] RECOVERY - Puppet errors on tools-elastic-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:45] RECOVERY - Puppet errors on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:47] RECOVERY - Puppet errors on tools-docker-builder-05 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:47] RECOVERY - Puppet errors on tools-docker-registry-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:49] RECOVERY - Puppet errors on tools-worker-1023 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:49] RECOVERY - Puppet errors on tools-worker-1026 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:51] RECOVERY - Puppet errors on tools-worker-1025 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:55] RECOVERY - Puppet errors on tools-k8s-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:57] RECOVERY - Puppet errors on tools-redis-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:58] RECOVERY - Puppet errors on tools-exec-gift-trusty-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:02] RECOVERY - Puppet errors on tools-paws-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:02] RECOVERY - Puppet errors on tools-prometheus-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:02] RECOVERY - Puppet errors on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:08] RECOVERY - Puppet errors on tools-exec-1427 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:10] RECOVERY - Puppet errors on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:10] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:12] RECOVERY - Puppet errors on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:12] RECOVERY - Puppet errors on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:12] RECOVERY - Puppet errors on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:42] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1425 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:49] RECOVERY - Puppet errors on tools-worker-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:51] RECOVERY - Puppet errors on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:55] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1426 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:59] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:01] RECOVERY - Puppet errors on tools-logs-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:01] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:01] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:13] RECOVERY - Puppet errors on tools-prometheus-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:14] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:21] RECOVERY - Puppet errors on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:23] RECOVERY - Puppet errors on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:25] RECOVERY - Puppet errors on tools-worker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:29] RECOVERY - Puppet errors on tools-exec-1424 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:31] RECOVERY - Puppet errors on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:34] RECOVERY - Puppet errors on tools-flannel-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:36] RECOVERY - Puppet errors on tools-checker-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:42] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:42] RECOVERY - Puppet errors on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:46] RECOVERY - Puppet errors on tools-elastic-03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:58] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [20:35:04] RECOVERY - Puppet errors on tools-worker-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [20:35:10] RECOVERY - Puppet errors on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:35:12] RECOVERY - Puppet errors on tools-flannel-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:35:16] RECOVERY - Puppet errors on tools-flannel-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:35:24] RECOVERY - Puppet errors on tools-worker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [20:35:28] RECOVERY - Puppet errors on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [20:35:36] RECOVERY - Puppet errors on tools-worker-1008 is OK: OK: Less than 1.00% above the threshold [0.0] [20:35:38] RECOVERY - Puppet errors on tools-exec-1429 is OK: OK: Less than 1.00% above the threshold [0.0] [20:35:47] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [20:35:49] RECOVERY - Puppet errors on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [20:35:53] RECOVERY - Puppet errors on tools-grid-master is OK: OK: Less than 1.00% above the threshold [0.0] [20:35:53] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [20:35:55] RECOVERY - Puppet errors on tools-docker-registry-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:35:55] RECOVERY - Puppet errors on tools-exec-1423 is OK: OK: Less than 1.00% above the threshold [0.0] [20:35:59] RECOVERY - Puppet errors on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:05] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:05] RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:17] RECOVERY - Puppet errors on tools-package-builder-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:19] RECOVERY - Puppet errors on tools-exec-1426 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:23] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:24] RECOVERY - Puppet errors on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:27] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:31] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:38] RECOVERY - Puppet errors on tools-exec-1425 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:42] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:42] RECOVERY - Puppet errors on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:44] RECOVERY - Puppet errors on tools-exec-1419 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:44] RECOVERY - Puppet errors on tools-cron-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:54] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:56] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [20:38:00] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [20:42:39] thanks for helping with that valhallasw`cloud. I"ll see if I can find anything out. I have another py3 pod that has been acting strangely for the last week. [20:53:30] andrewbogott: since the lab* machines have been rebooted on June 21th, they looks less loaded :] [20:53:35] https://grafana.wikimedia.org/dashboard/db/labs-capacity-planning?panelId=91&fullscreen&orgId=1&from=now-7d&to=now ! [20:55:51] 10Labs, 10Labs-Infrastructure: labvirt1006 super busy right now - https://phabricator.wikimedia.org/T165753#3384406 (10hashar) 05Open>03Resolved Seems load went down on June 21th in the afternoon (UTC) which is when the lab* hosts have been rebooted and I possibly instance reshuffled around. That show up... [20:58:53] 10Labs, 10Labs-Infrastructure: Investigate instances with high "steal" CPU - https://phabricator.wikimedia.org/T161118#3384411 (10hashar) The steal CPU definitely reflects CPU over usage on a labvirt machine (was seen via T165753 as well). Maybe the metric could be used to build a metric about the labs perfo... [21:04:23] \o/ [21:11:29] PROBLEM - Puppet errors on tools-puppetmaster-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:11:47] 10Labs, 10Labs-Infrastructure, 10Release-Engineering-Team (Kanban): labnet-users group can no more access labnet1001 / labnet1002 - https://phabricator.wikimedia.org/T169018#3384480 (10hashar) [21:13:14]