[00:16:09] <wikibugs>	 10Tool-Labs-tools-Pageviews, 07I18n: massviews-category-description lego for "category" - https://phabricator.wikimedia.org/T146973#2911786 (10MusikAnimal) >>! In T146973#2910964, @Nikerabbit wrote: > It seems this message has multiple problems: > * Lego > * Message re-use > * Not properly escaped >  > Unfortu...
[03:27:10] <wikibugs>	 06Labs, 10OOjs-UI: Abandon (remove) outdated WMFLabs OOjs UI builds and demo instance - https://phabricator.wikimedia.org/T154454#2911859 (10Paladox)
[03:27:27] <wikibugs>	 06Labs, 10Tool-Labs, 10OOjs-UI: Abandon (remove) outdated WMFLabs OOjs UI builds and demo instance - https://phabricator.wikimedia.org/T154454#2911861 (10Paladox)
[04:00:16] <wikibugs>	 06Labs, 10Tool-Labs, 10PageImages, 06Reading-Web-Backlog, and 2 others: Data disappeared from labs replica in cswiki_p.page_props - https://phabricator.wikimedia.org/T153888#2911872 (10Jdlrobson) >>! In T153888#2897492, @Blahma wrote: > I dare consider it bad design that an existing property has been assig...
[04:05:02] <wikibugs>	 06Labs, 10Tool-Labs, 10OOjs-UI: Abandon (remove) outdated WMFLabs OOjs UI builds and demo instance - https://phabricator.wikimedia.org/T154454#2911874 (10matmarex) @rillke runs that. Or used to, since it seems pretty dead now.
[06:38:04] <shinken-wm>	 PROBLEM - Puppet run on tools-exec-1221 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[07:13:05] <shinken-wm>	 RECOVERY - Puppet run on tools-exec-1221 is OK: OK: Less than 1.00% above the threshold [0.0]
[07:38:42] <shinken-wm>	 PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[09:04:00] <wikibugs>	 10Tool-Labs-tools-Pageviews, 07I18n: massviews-category-description lego for "category" - https://phabricator.wikimedia.org/T146973#2912148 (10Nikerabbit) Flexibility required for grammatical correctness trumps over extra work. The fact that you need to programmatically alter the other message is a sign that i...
[09:14:36] <wikibugs>	 10Tool-Labs-tools-Pageviews, 07I18n: massviews-category-description lego for "category" - https://phabricator.wikimedia.org/T146973#2912152 (10Nikerabbit) Not relevant at all, but thanks to you I finally pushed my experiments to https://github.com/Nikerabbit/monkey-i18n before I lose them. It has a prototype f...
[09:47:02] <shinken-wm>	 PROBLEM - Free space - all mounts on tools-worker-1003 is CRITICAL: CRITICAL: tools.tools-worker-1003.diskspace._var_lib_docker.byte_percentfree (No valid datapoints found) tools.tools-worker-1003.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-worker-1003.diskspace.root.byte_percentfree (<100.00%)
[11:00:55] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth was modified, changed by Shirayuki link https://www.mediawiki.org/w/index.php?diff=2337611 edit summary: translation tweaks
[11:01:13] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth was modified, changed by Shirayuki link https://www.mediawiki.org/w/index.php?diff=2337612 edit summary: Marked this version for translation
[11:01:34] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/en was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2337617 edit summary: Updating to match new version of source page
[11:01:34] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/ca was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2337618 edit summary: Updating to match new version of source page
[11:01:35] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/es was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2337619 edit summary: Updating to match new version of source page
[11:01:35] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/fr was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2337620 edit summary: Updating to match new version of source page
[11:01:35] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/pt-br was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2337621 edit summary: Updating to match new version of source page
[11:01:36] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/it was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2337622 edit summary: Updating to match new version of source page
[11:01:36] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/scn was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2337623 edit summary: Updating to match new version of source page
[11:01:36] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/ja was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2337624 edit summary: Updating to match new version of source page
[11:01:37] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/pl was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2337625 edit summary: Updating to match new version of source page
[11:01:37] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/de was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2337626 edit summary: Updating to match new version of source page
[11:01:38] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/ms was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2337628 edit summary: Updating to match new version of source page
[11:01:38] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/pt was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2337627 edit summary: Updating to match new version of source page
[11:01:38] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/ru was modified, changed by FuzzyBot link https://www.mediawiki.org/w/index.php?diff=2337629 edit summary: Updating to match new version of source page
[11:01:52] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/ja was modified, changed by Shirayuki link https://www.mediawiki.org/w/index.php?diff=2337631 edit summary: update
[11:02:11] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/es was modified, changed by Shirayuki link https://www.mediawiki.org/w/index.php?diff=2337633 edit summary: update
[11:02:24] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/ru was modified, changed by Shirayuki link https://www.mediawiki.org/w/index.php?diff=2337635 edit summary: update
[11:02:59] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/ja was modified, changed by Shirayuki link https://www.mediawiki.org/w/index.php?diff=2337637 edit summary: update
[11:03:15] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/es was modified, changed by Shirayuki link https://www.mediawiki.org/w/index.php?diff=2337639 edit summary: update
[11:03:28] <wm-bot>	 Change on 12www.mediawiki.org a page OAuth/ru was modified, changed by Shirayuki link https://www.mediawiki.org/w/index.php?diff=2337641 edit summary: update
[12:20:38] <wikibugs>	 06Labs, 10Tool-Labs, 10OOjs-UI: Abandon (remove) outdated WMFLabs OOjs UI builds and demo instance - https://phabricator.wikimedia.org/T154454#2912412 (10Rillke) 05Open>03Resolved a:03Rillke Done. Thanks for the reminder.
[14:20:54] <wikibugs>	 06Labs, 10Labs-Infrastructure, 06Operations, 07Wikimedia-Incident: labservices1001 down, suspected overheating - https://phabricator.wikimedia.org/T152340#2845101 (10chasemp) A few things that happen that should not when labservices1001 dies (we do not see these same failures when labservices1002 is down):...
[14:21:22] <wikibugs>	 06Labs, 10Labs-Infrastructure, 06Operations, 07Wikimedia-Incident: labservices1001 down, suspected overheating - https://phabricator.wikimedia.org/T152340#2912706 (10chasemp) p:05Triage>03High
[15:11:52] <wikibugs>	 06Labs, 10Tool-Labs, 10OOjs-UI: Abandon (remove) outdated WMFLabs OOjs UI builds and demo instance - https://phabricator.wikimedia.org/T154454#2912828 (10Volker_E) @Rillke Awesome, thanks!
[15:57:00] <ThatAndromeda>	 HI y'all - I'm trying to add an instance to a project I administer following the instructions at https://wikitech.wikimedia.org/wiki/Help:Instances . I've followed the instructions under Add Instance link to get to the correct page, but step 6 is 'Click on the “Add instance” link.' and there is no such link on the page I'm looking at. ???
[15:58:42] <tom29739>	 ThatAndromeda, that page is wrong
[15:59:07] <ThatAndromeda>	 Okay...what should I actually be doing, then?
[15:59:26] <tom29739>	 ThatAndromeda, you need to log into Horizon at https://horizon.wikimedia.org/
[15:59:53] <tom29739>	 Two-factor authentication will need to be enabled on your account before you can log in.
[16:00:40] <ThatAndromeda>	 2fa is enabled for me
[16:01:11] <ThatAndromeda>	 Horizon's never let me log in in the past. I wonder if today will be different!
[16:01:44] <tom29739>	 ThatAndromeda, so once you've logged in, check that the project you want to create an instance for is selected,
[16:01:50] <ThatAndromeda>	 ...nope. Is this where I ask for help getting Horizon to log me in, or is that somewhere else?
[16:02:06] <ThatAndromeda>	 (It rejects me with 'invalid credentials' and no further i nfo)
[16:03:09] <tom29739>	 ThatAndromeda, your username, password, or authentication code are wrong.
[16:03:19] <tom29739>	 At least that;s what I think that error means.
[16:04:36] <ThatAndromeda>	 There's no link for resetting my password on this page and my credentials are in lastpass and authenticator; I don't have any further actions to take here. Is there horizon-specific help I can contact?
[16:05:06] <tom29739>	 ThatAndromeda, you can log into wikitech with the same credentials?
[16:05:20] <ThatAndromeda>	 Yes.
[16:05:45] <tom29739>	 You perhaps need to ask for help from one of the labs ops then.
[16:10:57] <chasemp>	 ThatAndromeda: do you have 2fa setup?
[16:11:05] <ThatAndromeda>	 Yes
[16:12:46] <chasemp>	 I don't have an explanation for why it would work in the wikitech case and not horizon, andrewbogott is the best person to look into it and he should be around later on today
[16:13:03] <chasemp>	 ThatAndromeda: file a task outlining what you are trying to do and what you have done to that end so far?
[16:13:48] <ThatAndromeda>	 ok, thanks
[16:16:13] <chasemp>	 ThatAndromeda: one small thought is I had a similar issue when my time wasn't syncing w/ the mobile towers correctly (it's an option buried under android menus), worth checking
[16:17:13] <ThatAndromeda>	 whooa, fascinating. I'll check that, thanks
[16:17:42] <ThatAndromeda>	 hm, looks right for me
[16:19:03] <wikibugs>	 06Labs: Request creation of twl-staging labs project - https://phabricator.wikimedia.org/T153549#2913031 (10ThatAndromeda) I suppose so!  I worked on it a bit this morning and ran into the following problems:   * I tried to follow the instance creation instructions at https://wikitech.wikimedia.org/wiki/Help:Ins...
[16:57:06] <wikibugs>	 06Labs, 10wikitech.wikimedia.org, 07Technical-Debt: Cleanup ContentHandler deprecated calls in Wikitech specific extensions - https://phabricator.wikimedia.org/T147924#2913185 (10Reedy)
[16:57:42] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs - https://phabricator.wikimedia.org/T143349#2913187 (10chasemp)
[16:58:58] <wikibugs>	 06Labs, 10wikitech.wikimedia.org, 03Google-Code-In-2016, 07Technical-Debt: Identify/Cleanup ContentHandler deprecated calls (and hook subscribers) in Wikitech specific extensions branches - https://phabricator.wikimedia.org/T147924#2708611 (10Reedy)
[17:00:49] <wikibugs>	 06Labs: Request creation of twl-staging labs project - https://phabricator.wikimedia.org/T153549#2913207 (10Andrew) The keystone log says 'DEBUG OATH: user 'Thatandromeda' does not have 2FA enabled'  Since you think you have it enabled, something interesting is probably happening... it might be worth you disabli...
[17:01:13] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Tools - https://phabricator.wikimedia.org/T154495#2913209 (10chasemp)
[17:01:44] <wikibugs>	 06Labs, 10wikitech.wikimedia.org, 03Google-Code-In-2016, 07Technical-Debt: Identify/Cleanup ContentHandler deprecated calls (and hook subscribers) in Wikitech specific extensions branches - https://phabricator.wikimedia.org/T147924#2913222 (10Reedy)
[17:03:38] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Tools - https://phabricator.wikimedia.org/T154495#2913228 (10chasemp)
[17:07:39] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, and 2 others: labs DHCP server gives only a single DNS resolver (was: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedi... - https://phabricator.wikimedia.org/T137460#2913237
[17:09:57] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, and 2 others: labs DHCP server gives only a single DNS resolver (was: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedi... - https://phabricator.wikimedia.org/T137460#2913241
[17:11:50] <wikibugs>	 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from toollabs - https://phabricator.wikimedia.org/T94790#2913253 (10bd808)
[17:11:53] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Tools - https://phabricator.wikimedia.org/T154495#2913252 (10bd808)
[17:23:28] <wikibugs>	 10Labs-project-other, 10Wikimedia-Mailing-lists: Wikimedia-l <-> Discourse synchronization (PRELIMINARY) - https://phabricator.wikimedia.org/T126458#2913281 (10Dzahn) @EBernhardson Could you add the (new) user @Austin created to the Labs project for Discourse?
[17:23:39] <wikibugs>	 10Labs-project-other, 10Wikimedia-Mailing-lists: Wikimedia-l <-> Discourse synchronization (PRELIMINARY) - https://phabricator.wikimedia.org/T126458#2913282 (10Dzahn) a:03EBernhardson
[17:27:48] <wikibugs>	 06Labs, 10Tool-Labs, 10DBA: Spatial database for tool-labs - https://phabricator.wikimedia.org/T154497#2913296 (10Tobias1984)
[17:27:52] <Nemo_bis>	 ugh
[17:29:29] <wikibugs>	 06Labs, 10wikitech.wikimedia.org, 03Google-Code-In-2016, 07Technical-Debt: Identify/Cleanup ContentHandler deprecated calls (and hook subscribers) in Wikitech specific extensions branches - https://phabricator.wikimedia.org/T147924#2708611 (10Reedy)
[17:30:28] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs - https://phabricator.wikimedia.org/T143349#2913343 (10chasemp)
[17:30:30] <wikibugs>	 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from toollabs - https://phabricator.wikimedia.org/T94790#2913342 (10chasemp)
[17:32:44] <wikibugs>	 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from toollabs - https://phabricator.wikimedia.org/T94790#2913352 (10chasemp)
[17:33:16] <wikibugs>	 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from Tool Labs - https://phabricator.wikimedia.org/T94790#1172717 (10chasemp)
[17:33:58] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Tools - https://phabricator.wikimedia.org/T154495#2913356 (10chasemp)
[17:34:02] <wikibugs>	 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from Tool Labs - https://phabricator.wikimedia.org/T94790#1172717 (10chasemp)
[17:46:37] <wikibugs>	 10Tool-Labs-tools-Pageviews, 07I18n: massviews-category-description lego for "category" - https://phabricator.wikimedia.org/T146973#2913403 (10MusikAnimal) The issue is I'm using raw HTML and not markdown or wikitext. So here I have `<a href="URL" target="_blank">category</a>` at its simplest, but sometimes th...
[17:50:37] <wikibugs>	 06Labs, 10Tool-Labs, 10DBA: Spatial database for tool-labs - https://phabricator.wikimedia.org/T154497#2913411 (10scfc) There is a PostgreSQL database that replicates data from OSM and is used by some tools; currently accounts for that database are managed manually in `role::osm::master`, mainly by @akosiari...
[17:55:02] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, and 2 others: labs DHCP server gives only a single DNS resolver (was: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedi... - https://phabricator.wikimedia.org/T137460#2368900
[17:59:04] <wikibugs>	 06Labs, 10Tool-Labs, 10DBA: Spatial database for tool-labs - https://phabricator.wikimedia.org/T154497#2913296 (10bd808) We have an [[https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Elasticsearch|Elasticsearch cluster in Tool Labs]]. Could you achieve the same end result by using the [[https://www.elastic...
[18:12:35] <wikibugs>	 06Labs, 10Tool-Labs, 10DBA: Spatial database for tool-labs - https://phabricator.wikimedia.org/T154497#2913296 (10EBernhardson) Can this be accomplished with the geo keywords available on commons for search?  https://www.mediawiki.org/wiki/Help:CirrusSearch#Geo_Search https://commons.wikimedia.org/w/index.ph...
[18:12:57] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, and 2 others: labs DHCP server gives only a single DNS resolver (was: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedi... - https://phabricator.wikimedia.org/T137460#2913504
[18:17:11] <wikibugs>	 10Labs-project-other, 10Wikimedia-Mailing-lists: Wikimedia-l <-> Discourse synchronization (PRELIMINARY) - https://phabricator.wikimedia.org/T126458#2913539 (10EBernhardson) Tried again and wikitech was happy this time, so just some hiccup. Austin is now an admin for the Discourse project.
[18:19:03] <wikibugs>	 06Labs, 15User-bd808: Migrate projects using ::role::deprecated::mediawiki::install to ::role::labs::mediawiki_vagrant - https://phabricator.wikimedia.org/T121476#2913543 (10Mattflaschen-WMF)
[18:26:48] <wikibugs>	 06Labs, 15User-bd808: Migrate projects using ::role::deprecated::labsvagrant to ::role::labs::mediawiki_vagrant - https://phabricator.wikimedia.org/T121477#2913577 (10Mattflaschen-WMF)
[18:31:01] <wikibugs>	 06Labs, 10Labs-Infrastructure: Deprecate precise instances in Labs - https://phabricator.wikimedia.org/T143349#2913585 (10hashar)
[18:32:22] <wikibugs>	 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Make webservice backend default to kubernetes - https://phabricator.wikimedia.org/T154504#2913589 (10yuvipanda)
[18:39:02] <wikibugs>	 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 07Tracking: Make webservice backend default to kubernetes - https://phabricator.wikimedia.org/T154504#2913642 (10yuvipanda)
[18:46:54] <wikibugs>	 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Make webservice backend for lighttpd default to kubernetes - https://phabricator.wikimedia.org/T154506#2913676 (10yuvipanda)
[18:55:03] <wikibugs>	 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#2913707 (10yuvipanda) @chasemp let's plan and do this this month sometime?
[19:01:03] <wikibugs>	 06Labs, 10Labs-Infrastructure: secondary.bastion.wmflabs.org is dead - https://phabricator.wikimedia.org/T150896#2913731 (10yuvipanda) 05Open>03Resolved a:03yuvipanda Works fine now.
[19:05:16] <wikibugs>	 06Labs, 10Tool-Labs: bigbrother doesn't stop - https://phabricator.wikimedia.org/T94500#2913762 (10bd808) This looks like something that did not get addressed in the rewrite. My my reading of the code, `update_db` will check for configuration for each running job's owner by calling `read_config(owner)`. `read_...
[19:07:32] <wikibugs>	 06Labs, 10Tool-Labs: bigbrother doesn't stop - https://phabricator.wikimedia.org/T94500#2913770 (10bd808) >>! In T94500#2913762, @bd808 wrote: > This actually has a bug that needs to be fixed where ~/.bigbrotherrc won't be read on the first pass Poor reading of the code on my part. The exit condition is `now <...
[19:26:30] <wikibugs>	 06Labs, 10Tool-Labs: bigbrother only watches users jobs if they already have a job running - https://phabricator.wikimedia.org/T88122#2913836 (10bd808) >>! In T88122#2885401, @bd808 wrote: >>>! In T88122#2885398, @scfc wrote: >> @bd808, does the new `bigbrother` start jobs listed in `.bigbrotherrc` for users t...
[19:29:29] <wikibugs>	 06Labs, 10Tool-Labs: Monitor bigbrother - https://phabricator.wikimedia.org/T90850#1069215 (10bd808) The timestamp on the scoreboard file should change on each pass through bigbrother.py's run loop. Can we setup an icinga alert if that timestamp is more than N minutes old?
[19:36:24] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, and 2 others: labs DHCP server gives only a single DNS resolver (was: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedi... - https://phabricator.wikimedia.org/T137460#2913874
[19:39:14] <paladox>	 Hi trying to access https://tools.wmflabs.org/watroles is returning a not found error
[19:40:00] <paladox>	 yuvipanda ^^
[19:40:02] <mutante>	 yea, watroles is gone
[19:40:08] <mutante>	 i used it a lot
[19:40:29] <mutante>	 would be unfortunate if permanent
[19:40:39] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, and 2 others: labs DHCP server gives only a single DNS resolver (was: CI jobs failing with DNS resolution errors such as "Could not resolve host: gerrit.wikimedi... - https://phabricator.wikimedia.org/T137460#2913884
[19:41:18] <paladox>	 chasemp ^^
[19:42:19] <chasemp>	 mutante: iirc someone has to dig in and update it to reflect new role sources of DB instead of LDAP but I'm not sure
[19:42:42] <chasemp>	 there may or may not be a task but I think breakage is known and it being a yuvi side project (?) I'm not sure the status
[19:44:09] <mutante>	 aww :( that was the only way to tell which instances use something when making a puppet change
[19:44:50] <chasemp>	 mutante: I believe there is a rest api now
[19:45:03] <chasemp>	 and possibly andrew or yuvi have a cli utility 
[19:45:37] <mutante>	 ok, that would be great. i hope it doesn't require shell 
[19:45:47] <mutante>	 that would limit to ops
[19:48:06] <mutante>	 rest api would be better
[20:00:07] <akoopal>	 any problems with bigbrother?
[20:09:46] <wikibugs>	 10Labs-project-other, 10Wikimedia-Mailing-lists: Wikimedia-l <-> Discourse synchronization (PRELIMINARY) - https://phabricator.wikimedia.org/T126458#2914111 (10Dzahn) a:05EBernhardson>03Austin Cool, thanks.  Is this resolved now, @Austin?
[20:12:40] <wikibugs>	 10Tool-Labs-tools-Other: watroles is misleading and should be disabled - https://phabricator.wikimedia.org/T153216#2914136 (10scfc) 05Open>03Resolved a:03scfc I have `webservice stop`ped `watroles`.
[20:17:44] <wikibugs>	 10Tool-Labs-tools-Other: watroles is misleading and should be disabled - https://phabricator.wikimedia.org/T153216#2873080 (10Dzahn) Is there anything that can replace the functionality of watroles? It was used a lot. basically every time you rename or delete a puppet class this was the way to check which instan...
[20:21:19] <wikibugs>	 10Tool-Labs-tools-Other: watroles is misleading and should be disabled - https://phabricator.wikimedia.org/T153216#2914188 (10Dzahn) really nice would be a redirect to the new place when deleting (popular) tools. was surprised by a sudden 404 and first thought it's a bug.
[20:26:52] <wikibugs>	 10Tool-Labs-tools-Other: watroles has no index page - https://phabricator.wikimedia.org/T135053#2914220 (10scfc)
[20:28:14] <paladox>	 chasemp hi, i have a puppet master but one of my instances wont connect to it.
[20:28:17] <paladox>	 i get this error
[20:28:18] <paladox>	 root@gerrit-test3:/home/paladox# sudo -i puppet agent --test --verbose
[20:28:18] <paladox>	 Info: Creating a new SSL key for gerrit-test3.git.eqiad.wmflabs
[20:28:18] <paladox>	 Error: Could not request certificate: Connection timed out - connect(2) for "puppet-paladox.git.eqiad.wmflabs" port 8140
[20:28:19] <paladox>	 Exiting; failed to retrieve certificate and waitforcert is disabled
[20:29:06] <mutante>	 paladox: you should point out the networking part
[20:29:16] <mutante>	 this is about connecting to port 8140
[20:29:24] <mutante>	 it's one level below the puppet cert signing part
[20:29:36] <mutante>	 afaict because you used "nc" to check that
[20:29:40] <paladox>	 Oh, i enabled that ^^ port but it dosent seem to work on the instance (client connecting to master)
[20:29:47] <mutante>	 otherwise it looks like you are talking about cert issues
[20:29:54] <mutante>	 which it's apparently not.. 
[20:30:41] <mutante>	 like i said earlier, you should look at why the security group changes dont seem to be applied
[20:30:45] <mutante>	 or what might be wrong with them
[20:30:47] <paladox>	 ok
[20:30:58] <mutante>	 running puppet won't do a thing
[20:31:08] <mutante>	 if the whole thing is "timed out" 
[20:31:33] <mutante>	 try to connect to _any_ other port
[20:31:37] <mutante>	 just to see if you can do that
[20:31:51] <mutante>	 try 443 and 80, try adding those to the groups, try again
[20:32:05] <mutante>	 see if you can confirm that makes a difference or not
[20:32:43] <paladox>	 Ok
[20:33:12] <mutante>	 just to confirm in general that you can edit security groups succesfully
[20:33:35] <paladox>	 mutante port 80 works now
[20:33:38] <mutante>	 then get back to port 8140 
[20:33:38] <paladox>	 but 8140 dosent
[20:33:48] <mutante>	 did you add both rules in the same security group?
[20:33:53] <wikibugs>	 10Tool-Labs-tools-Other: watroles is misleading and should be disabled - https://phabricator.wikimedia.org/T153216#2914236 (10scfc) @Dzahn: T151522.  (I also saw some WIP in the `watroles` tool to access data with `keystoneclient`; source code for `watroles` is at https://github.com/yuvipanda/watroles.)
[20:33:57] <mutante>	 do they look identical except the port?
[20:34:34] <paladox>	 mutante nope, one rule is in one group
[20:34:39] <paladox>	 and the other in another
[20:34:52] <paladox>	 as the default one is full up i had to create another group
[20:34:55] <mutante>	 paladox: try using the default group 
[20:35:00] <paladox>	 ok
[20:35:02] <mutante>	 i think that is why it fails
[20:35:18] <mutante>	 i wasn't aware at what point it's "ful;"
[20:35:19] <mutante>	 full
[20:35:30] <mutante>	 but i remember how adding a new group would not work
[20:35:35] <mutante>	 while adding a rule to the default group did
[20:37:20] <wikibugs>	 06Labs, 10Tool-Labs, 07Epic: Phase out precise instances from Tool Labs - https://phabricator.wikimedia.org/T94790#2914264 (10chasemp)
[20:37:23] <wikibugs>	 06Labs, 10Horizon: Provide watroles functionality in Horizon - https://phabricator.wikimedia.org/T151522#2820072 (10Dzahn) Yes please, watroles was used regularly and is being missed.
[20:37:25] <paladox>	 oh, adding it to the default group and trying again dosent work
[20:37:33] <paladox>	 mutante ^^
[20:38:05] <wikibugs>	 10Tool-Labs-tools-Other: watroles is misleading and should be disabled - https://phabricator.wikimedia.org/T153216#2873080 (10Dzahn) thanks @scfc i see, subscribed there
[20:38:53] <mutante>	 paladox: so you are saying you were able to add a rule for port 80 and confirm that, but when you did the same port port 8140 it fails?
[20:39:19] <paladox>	 Ive added the rule for 8140, what im saying is it is still failing to connect
[20:39:35] <paladox>	 root@gerrit-test3:/var/lib/puppet/state# telnet puppet-paladox.git.eqiad.wmflabs 8140
[20:39:36] <paladox>	 Trying 10.68.19.229...
[20:39:45] <mutante>	 but you also added a rule for port 80 and you could connect to port 80?
[20:39:50] <paladox>	 yep
[20:39:57] <mutante>	 did you add a rule for port 80 that did not exist before?
[20:40:02] <mutante>	 just now?
[20:40:10] <paladox>	 nope, that was added a few months
[20:40:11] <paladox>	 ago
[20:40:27] <mutante>	 so you did not actually did that
[20:40:38] <mutante>	 do
[20:40:44] <mutante>	 please do it
[20:40:55] <paladox>	 nope, i only added it for 8140 as port 80 is already there
[20:41:32] <mutante>	 yes, so i said try another port
[20:41:41] <paladox>	 ok
[20:41:48] <mutante>	 12:34 < mutante> just to confirm in general that you can edit security groups succesfully
[20:42:10] <paladox>	 port 443 doesent work
[20:42:16] <paladox>	 root@gerrit-test3:/var/lib/puppet/state# telnet puppet-paladox.git.eqiad.wmflabs 443
[20:42:16] <paladox>	 Trying 10.68.19.229...
[20:42:40] <mutante>	 ok, so you confirmed it doesnt work, good
[20:42:45] <mutante>	 now add the rule to allow 443
[20:42:56] <yuvipanda>	 !log tools.hennalabs move simple PHP webservice to k8s
[20:42:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.hennalabs/SAL
[20:42:58] <paladox>	 the rule 443 is already there
[20:43:20] <paladox>	 Ingress	-	TCP	443 (HTTPS)	0.0.0.0/0
[20:43:28] <madhuvishy>	 !log tools Silenced tools checker on icinga to test labservices1001 failure causing toolschecker to flake out  T152369
[20:43:32] <mutante>	 paladox: you know, i had a reason to say add something new
[20:43:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:43:35] <stashbot>	 T152369: toolschecker fell to pieces when labs-ns0 went down - https://phabricator.wikimedia.org/T152369
[20:43:42] <mutante>	 paladox: i dont want to test stuff that is already there
[20:43:47] <paladox>	 oh ok
[20:44:31] <paladox>	 should i remove the port so we can try it without it in the firewall?
[20:44:33] <paladox>	 mutante ^^
[20:44:53] <yuvipanda>	 !log tools.wmf-task-samtar moved simple php webservice to k8s
[20:44:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wmf-task-samtar/SAL
[20:45:18] <yuvipanda>	 !log tools.common-app-stats moved simple static site to k8s
[20:45:19] <stashbot>	 Unknown project "tools.common-app-stats"
[20:46:22] <yuvipanda>	 !log tools.yellowbot moved simple PHP site to k8s
[20:46:23] <yuvipanda>	 !log tools.blog moved simple redirect to k8s
[20:46:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.yellowbot/SAL
[20:46:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.blog/SAL
[20:46:29] <yuvipanda>	 !log tools.file-reuse moved simple redirect to k8s
[20:46:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.file-reuse/SAL
[20:47:00] <yuvipanda>	 !log tools.file-reuse-test moved PHP application to k8s
[20:47:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.file-reuse-test/SAL
[20:47:11] <yuvipanda>	 addshore: ^ I've moved file-reuse-test to k8s, which gives it a small PHP version bump. everything else should be fine, let me know if it is not
[20:47:12] <yuvipanda>	 !log tools.mzmcbride moved simple static application to k8s
[20:47:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.mzmcbride/SAL
[20:47:33] <yuvipanda>	 !log tools.wikiradio moved simple PHP app to k8s
[20:47:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiradio/SAL
[20:48:41] <mutante>	 paladox: no, let's do this:  on the master:  nc -l 4444    on the client:  nc puppet-paladox.git.eqiad.wmflabs 4444       then type something, confirm it does _not_ work.   go to security groups, add a rule for port 4444, repeat the same thing, confirm it now works
[20:48:50] <paladox>	 ok
[20:48:59] <mutante>	 if that doesnt work, then open ticket
[20:51:27] <madhuvishy>	 !log tools Adding iptables rule to block outgoing connections to labservices1001 on tools-checker-01
[20:51:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:51:51] <paladox>	 mutante nope dosent work
[20:52:34] <mutante>	 paladox: then open ticket about security groups and horizon i guess.. or i dont know
[20:52:40] <yuvipanda>	 https://phabricator.wikimedia.org/T153216
[20:52:42] <paladox>	 ok
[20:52:57] <yuvipanda>	 yeah, it was a side project and I'm not sure of status either
[20:52:57] <yuvipanda>	 I wrote code that makes it all available as a YAML file, I guess I could just spend time deploying it at some point
[20:52:58] <yuvipanda>	 instance-info-dumper.py
[20:53:30] <mutante>	 that would be great
[20:53:32] <yuvipanda>	 the REST API only does instance -> roles/hiera mapping
[20:53:34] <paladox>	 mutante i belive it takes a while for port changes to take effect
[20:53:45] <yuvipanda>	 not the reverse
[20:56:08] <yuvipanda>	 anyway, help welcome :) I don't know when I'll get to it, but will try to
[20:56:09] <yuvipanda>	 (since most of the code is already written)
[20:56:16] <mutante>	 to me it was the most important tool. that's how it goes with side projects heh
[20:56:28] <mutante>	 basically every time you make a change to a class used in labs
[20:57:34] <chasemp>	 yuvipanda: where is instance-info-dumper.py now? github?
[20:57:40] <paladox>	 yuvipanda hi, im wondering do you know why the changes i made to the security groups arn't being reflected on the instances
[20:58:10] <paladox>	 i've opened port 8140 but i carn't access it
[21:01:55] <addshore>	 yuvipanda: cool!
[21:16:15] <yuvipanda>	 akoopal: you tell us :)
[21:21:10] <akoopal>	 yuvipanda: a project I go-maintain used to be restarted by bigbrother but doesn't seem to work anymore
[21:21:38] <akoopal>	 should probably first read the docs on it and check if the config is still ok
[21:24:52] <yuvipanda>	 chasemp: nope, in ops/puppet
[21:28:19] <akoopal>	 yuvipanda: hmm, if I understand correctly, if I submit the job with jstart it should start automatically?
[21:28:55] <akoopal>	 this is about wikilinkbot btw
[21:36:49] <yuvipanda>	 jstart should just start your job and doesn't touch bigbrother at all
[21:38:33] <akoopal>	 yep, understood now
[21:38:47] <akoopal>	 but bigbrother is also setup, and looks correct
[21:39:09] <akoopal>	 but had to restart the bot again today, bigbrother didn't seem to have done it
[21:40:14] <bd808>	 akoopal: there is a bug in bigbrother that can keep it from starting a job for the first time
[21:40:26] * bd808 looks for the bug number
[21:41:00] <bd808>	 akoopal: T88122
[21:41:01] <stashbot>	 T88122: bigbrother only watches users jobs if they already have a job running - https://phabricator.wikimedia.org/T88122
[21:41:18] <bd808>	 its possible that caused you issues
[21:42:09] <akoopal>	 hmmm, so if there is no job running, the crashed job also doesn't start?
[21:42:51] <akoopal>	 but this tool has the webservice running as well
[21:42:57] <akoopal>	 and that was still running
[21:43:03] <madhuvishy>	 !log tools Adding iptables rule to drop incoming connections from toolschecker on labservices1001
[21:43:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[21:43:34] <bd808>	 akoopal: is the webservice using the job grid or kubernetes?
[21:43:56] <bd808>	 if it is using the job grid then yeah that should make bigbrother read the config file
[21:44:06] <akoopal>	 not sure?
[21:44:33] <akoopal>	 it starts it with jstats (yes, somebody else all set it up, and is less active now)
[21:47:10] <bd808>	 akoopal: I think I see the problem
[21:48:05] <bd808>	 the directory permissions on /data/project/wikilinkbot are very restrictive and may be tricking bigbrother into thinking that your tool doesn't really exist
[21:48:34] <akoopal>	 hmmm
[21:49:10] <bd808>	 I think that o+r is needed
[21:54:13] <bd808>	 akoopal: nope. I think the permissions are a red herring. os.path.isdir('/data/project/wikilinkbot') returns true even for an unprivledged account
[22:02:23] <akoopal>	 ok, so that is then not the problem, then I am not going to test
[22:02:29] <akoopal>	 (sorry, call in between)
[22:02:51] <akoopal>	 bd808: ^^
[22:03:35] <bd808>	 akoopal: if you'd like to open a bug we can try to remember to play with it sometime and see if we can figure the root problem out
[22:04:08] <akoopal>	 ok, just under toolserver?
[22:04:31] <akoopal>	 ehh, tool-labs
[22:08:40] <wikibugs>	 06Labs, 10Tool-Labs: Bigbrother doesn't restart wikilinkbot - https://phabricator.wikimedia.org/T154527#2914733 (10Akoopal)
[22:08:51] <akoopal>	 done
[22:18:12] <wikibugs>	 06Labs, 10Tool-Labs: Bigbrother doesn't restart wikilinkbot - https://phabricator.wikimedia.org/T154527#2914733 (10bd808) What we aren't sure of at this point is if bigbrother is even reading the .bigbrotherrc file or not. It's possible that the failures are related to {T88122}
[22:36:42] <mutante>	 on shinken, it says "UNKNOWN: execution of the check script exited with exception 'UNKNOWN' " for a bunch of instances
[22:36:51] <mutante>	 double unknown
[22:54:40] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Shinken: shinken checks broken with "UNKNOWN"/no valid datapoints - https://phabricator.wikimedia.org/T154533#2914921 (10Dzahn)
[22:58:05] <wikibugs>	 06Labs, 10Tool-Labs: Install opencv-data on toollabs - https://phabricator.wikimedia.org/T142321#2914949 (10Dzahn) I would confirm this but currently i can't seem to SSH to any tools-exec instance (or the right ones are not in shinken)
[23:11:34] <madhuvishy>	 !log tools Disabled puppet on tools-checker-01 (T152369)
[23:11:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:11:38] <stashbot>	 T152369: toolschecker fell to pieces when labs-ns0 went down - https://phabricator.wikimedia.org/T152369
[23:21:22] <wikibugs>	 06Labs, 10Tool-Labs: Reduce Precise OGE exec hosts to 5 - https://phabricator.wikimedia.org/T154539#2915068 (10bd808)
[23:24:41] <bd808>	 !log tools drained tools-exec-1212 (T154539)
[23:24:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:24:44] <stashbot>	 T154539: Reduce Precise OGE exec hosts to 5 - https://phabricator.wikimedia.org/T154539
[23:25:15] <bd808>	 !log tools drained tools-exec-1213 (T154539)
[23:25:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:25:52] <bd808>	 !log tools drained tools-exec-1214 (T154539)
[23:25:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:26:36] <bd808>	 !log tools drained tools-exec-1215 (T154539)
[23:26:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:27:18] <bd808>	 !log tools drained tools-exec-1216 (T154539)
[23:27:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:30:13] <wikibugs>	 06Labs, 10Tool-Labs: Reduce Precise OGE exec hosts to 5 - https://phabricator.wikimedia.org/T154539#2915103 (10bd808) The queues are disabled on tools-exec-1212 through tools-exec-1216. All continuous jobs have been rescheduled using qmod -rj. The nodes are now empty and ready for steps 2-7 from https://wikite...
[23:37:04] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-generic-1402 is OK: line 390
[23:37:04] <shinken-wm>	 RECOVERY - High iowait on tools-checker-02 is OK: line 390
[23:37:06] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1207 is OK: line 390
[23:37:08] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1213 is OK: line 390
[23:37:10] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1410 is OK: line 390
[23:37:10] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1019 is OK: line 390
[23:37:10] <shinken-wm>	 RECOVERY - High iowait on tools-bastion-05 is OK: line 390
[23:37:12] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1005 is OK: line 390
[23:37:14] <shinken-wm>	 RECOVERY - High iowait on tools-static-10 is OK: line 390
[23:37:16] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1013 is OK: line 390
[23:37:18] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1411 is OK: line 390
[23:37:22] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-generic-1404 is OK: line 390
[23:37:24] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1022 is OK: line 390
[23:37:30] <shinken-wm>	 RECOVERY - High iowait on tools-prometheus-01 is OK: line 390
[23:37:32] <shinken-wm>	 RECOVERY - High iowait on tools-cron-01 is OK: line 390
[23:37:39] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Shinken: shinken checks broken with "UNKNOWN"/no valid datapoints - https://phabricator.wikimedia.org/T154533#2915112 (10Krenair) a:03Krenair I think I'm responsible for some of these
[23:37:42] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1216 is OK: line 390
[23:37:44] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1004 is OK: line 390
[23:37:46] <shinken-wm>	 RECOVERY - High iowait on tools-elastic-02 is OK: line 390
[23:37:53] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1003 is OK: OK: All targets OK
[23:38:01] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1010 is OK: OK: All targets OK
[23:38:01] <shinken-wm>	 RECOVERY - High iowait on tools-k8s-etcd-02 is OK: OK: All targets OK
[23:38:03] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1408 is OK: OK: All targets OK
[23:38:03] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1404 is OK: OK: All targets OK
[23:38:03] <shinken-wm>	 RECOVERY - High iowait on tools-checker-01 is OK: OK: All targets OK
[23:38:05] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1408 is OK: OK: All targets OK
[23:38:09] <shinken-wm>	 RECOVERY - High iowait on tools-flannel-etcd-01 is OK: OK: All targets OK
[23:38:11] <shinken-wm>	 RECOVERY - High iowait on tools-elastic-01 is OK: OK: All targets OK
[23:38:13] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1405 is OK: OK: All targets OK
[23:38:17] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1217 is OK: OK: All targets OK
[23:38:21] <shinken-wm>	 RECOVERY - High iowait on tools-elastic-03 is OK: OK: All targets OK
[23:38:23] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1011 is OK: OK: All targets OK
[23:38:25] <shinken-wm>	 RECOVERY - High iowait on tools-flannel-etcd-02 is OK: OK: All targets OK
[23:38:25] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1409 is OK: OK: All targets OK
[23:38:26] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1403 is OK: OK: All targets OK
[23:38:29] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1404 is OK: OK: All targets OK
[23:38:30] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-generic-1401 is OK: OK: All targets OK
[23:38:37] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1007 is OK: OK: All targets OK
[23:38:39] <shinken-wm>	 RECOVERY - High iowait on tools-k8s-master-01 is OK: OK: All targets OK
[23:38:40] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1015 is OK: OK: All targets OK
[23:38:42] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1001 is OK: OK: All targets OK
[23:38:42] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1205 is OK: OK: All targets OK
[23:38:44] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1401 is OK: OK: All targets OK
[23:38:54] <shinken-wm>	 RECOVERY - High iowait on tools-grid-shadow is OK: OK: All targets OK
[23:39:04] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1218 is OK: OK: All targets OK
[23:39:04] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1202 is OK: OK: All targets OK
[23:39:12] <shinken-wm>	 RECOVERY - High iowait on tools-grid-master is OK: OK: All targets OK
[23:39:14] <shinken-wm>	 RECOVERY - High iowait on tools-logs-02 is OK: OK: All targets OK
[23:39:16] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1025 is OK: OK: All targets OK
[23:39:16] <shinken-wm>	 RECOVERY - High iowait on tools-flannel-etcd-03 is OK: OK: All targets OK
[23:39:22] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1008 is OK: OK: All targets OK
[23:39:24] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1409 is OK: OK: All targets OK
[23:39:25] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1406 is OK: OK: All targets OK
[23:39:25] <Krenair>	 ^ that's me... looks like I broke the iowait check
[23:39:26] <shinken-wm>	 RECOVERY - High iowait on tools-precise-dev is OK: OK: All targets OK
[23:39:31] <Krenair>	 but just now fixed it with a live hack
[23:39:31] <shinken-wm>	 RECOVERY - High iowait on tools-exec-gift is OK: OK: All targets OK
[23:39:39] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1407 is OK: OK: All targets OK
[23:39:40] <Krenair>	 that puppet will probably pull soon. I have a patch incoming
[23:39:42] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1406 is OK: OK: All targets OK
[23:39:42] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1401 is OK: OK: All targets OK
[23:39:44] <shinken-wm>	 RECOVERY - High iowait on tools-k8s-etcd-03 is OK: OK: All targets OK
[23:39:50] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1209 is OK: OK: All targets OK
[23:39:50] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1414 is OK: OK: All targets OK
[23:39:52] <shinken-wm>	 RECOVERY - High iowait on tools-redis-1001 is OK: OK: All targets OK
[23:39:56] <shinken-wm>	 RECOVERY - High iowait on tools-proxy-01 is OK: OK: All targets OK
[23:39:58] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1405 is OK: OK: All targets OK
[23:39:58] <shinken-wm>	 RECOVERY - High iowait on tools-static-11 is OK: OK: All targets OK
[23:40:08] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1412 is OK: OK: All targets OK
[23:40:12] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1212 is OK: OK: All targets OK
[23:40:24] <shinken-wm>	 RECOVERY - High iowait on tools-mail-01 is OK: OK: All targets OK
[23:40:24] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1018 is OK: OK: All targets OK
[23:40:34] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1215 is OK: OK: All targets OK
[23:40:36] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1210 is OK: OK: All targets OK
[23:40:40] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1402 is OK: OK: All targets OK
[23:40:41] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1220 is OK: OK: All targets OK
[23:40:42] <shinken-wm>	 RECOVERY - High iowait on tools-services-02 is OK: OK: All targets OK
[23:40:43] <shinken-wm>	 RECOVERY - High iowait on tools-bastion-02 is OK: OK: All targets OK
[23:40:43] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1214 is OK: OK: All targets OK
[23:40:43] <shinken-wm>	 RECOVERY - High iowait on tools-services-01 is OK: OK: All targets OK
[23:40:45] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1206 is OK: OK: All targets OK
[23:40:45] <shinken-wm>	 RECOVERY - High iowait on tools-redis-1002 is OK: OK: All targets OK
[23:40:51] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1219 is OK: OK: All targets OK
[23:40:54] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1204 is OK: OK: All targets OK
[23:40:56] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1020 is OK: OK: All targets OK
[23:40:56] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1009 is OK: OK: All targets OK
[23:40:58] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1021 is OK: OK: All targets OK
[23:41:04] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1413 is OK: OK: All targets OK
[23:41:06] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1002 is OK: OK: All targets OK
[23:41:16] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-generic-1403 is OK: OK: All targets OK
[23:41:16] <shinken-wm>	 RECOVERY - High iowait on tools-docker-registry-01 is OK: OK: All targets OK
[23:41:17] <shinken-wm>	 RECOVERY - High iowait on tools-proxy-02 is OK: OK: All targets OK
[23:41:17] <shinken-wm>	 RECOVERY - High iowait on tools-k8s-etcd-01 is OK: OK: All targets OK
[23:41:22] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1203 is OK: OK: All targets OK
[23:41:26] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1201 is OK: OK: All targets OK
[23:41:28] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1221 is OK: OK: All targets OK
[23:41:29] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1407 is OK: OK: All targets OK
[23:41:29] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1016 is OK: OK: All targets OK
[23:41:32] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1006 is OK: OK: All targets OK
[23:41:34] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1403 is OK: OK: All targets OK
[23:41:35] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1208 is OK: OK: All targets OK
[23:41:35] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1012 is OK: OK: All targets OK
[23:41:41] <shinken-wm>	 RECOVERY - High iowait on tools-prometheus-02 is OK: OK: All targets OK
[23:41:43] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1014 is OK: OK: All targets OK
[23:41:53] <shinken-wm>	 RECOVERY - High iowait on tools-exec-1410 is OK: OK: All targets OK
[23:41:53] <shinken-wm>	 RECOVERY - High iowait on tools-webgrid-lighttpd-1402 is OK: OK: All targets OK
[23:41:57] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1023 is OK: OK: All targets OK
[23:41:57] <shinken-wm>	 RECOVERY - High iowait on tools-mail is OK: OK: All targets OK
[23:41:58] <shinken-wm>	 RECOVERY - High iowait on tools-worker-1017 is OK: OK: All targets OK
[23:42:01] <shinken-wm>	 RECOVERY - High iowait on tools-bastion-03 is OK: OK: All targets OK
[23:43:14] <Krenair>	 mutante, ^ that cut the list shown to guests down from 5 pages to 2
[23:43:41] <shinken-wm>	 RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[23:43:42] <Krenair>	 Nothing we can do about "No valid datapoints found"
[23:43:53] <Krenair>	 that'd be an issue for the target projectadmins to investigate
[23:44:12] <Krenair>	 probably same for "100.00% of data above the critical threshold [0.0]"
[23:56:08] <bd808>	 !log tools Removed tools-exec-12[12-16] from gridengine (T154539)
[23:56:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:56:12] <stashbot>	 T154539: Reduce Precise OGE exec hosts to 5 - https://phabricator.wikimedia.org/T154539
[23:57:18] <wikibugs>	 06Labs, 10Labs-Infrastructure, 10Shinken, 13Patch-For-Review: shinken checks broken with "UNKNOWN"/no valid datapoints - https://phabricator.wikimedia.org/T154533#2915211 (10Krenair) The first patch handles #1, #2 are errors to be dealt with by project admins, #3 should be investigated by project admins (f...