[02:17:53] PROBLEM - Puppet errors on tools-exec-1435 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:49:13] 10Tool-Labs-tools-Xtools, 06Community-Tech: XTools: Clean up "Pages created" tool - https://phabricator.wikimedia.org/T162752#3208650 (10DannyH) [02:49:43] 10Tool-Labs-tools-Xtools, 06Community-Tech: Remove references to "Range Contributions" and "Autoblock" within xTools code - https://phabricator.wikimedia.org/T163374#3208651 (10DannyH) [02:51:58] 10Tool-Labs-tools-Xtools, 06Community-Tech: Fix caching problems with XTools - https://phabricator.wikimedia.org/T162753#3208658 (10DannyH) [02:52:54] RECOVERY - Puppet errors on tools-exec-1435 is OK: OK: Less than 1.00% above the threshold [0.0] [02:52:56] 10Tool-Labs-tools-Xtools, 06Community-Tech: Bugs section on articleinfo returns incorrect results - https://phabricator.wikimedia.org/T148046#3208660 (10DannyH) [04:11:29] 06Labs: List of leaked dns precords for Andrew to clean up - https://phabricator.wikimedia.org/T163737#3208710 (10Andrew) 05Open>03Resolved All gone! [04:34:14] 06Labs, 10Labs-Infrastructure: When an instance is deleted, remove proxy records that point to it - https://phabricator.wikimedia.org/T163765#3208713 (10Andrew) [06:32:05] 10Tool-Labs-tools-Xtools: Update Goan Konkani namespace names in X!'s Tools edit counter - https://phabricator.wikimedia.org/T156641#3208803 (10The_Discoverer) Thank you :) [06:44:08] PROBLEM - Puppet errors on tools-exec-1410 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:50:00] PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:59:06] PROBLEM - Puppet errors on tools-exec-1408 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:04:12] PROBLEM - Puppet errors on tools-exec-1401 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:24:09] RECOVERY - Puppet errors on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [07:24:57] RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [07:34:04] RECOVERY - Puppet errors on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [07:44:13] RECOVERY - Puppet errors on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [08:09:35] 10Tool-Labs-tools-Attribution-Generator, 06TCB-Team: Use different colour on image outline in article images preview - https://phabricator.wikimedia.org/T121073#3208937 (10jhsoby) 05Open>03Resolved a:03jhsoby [Fixed](https://github.com/wmde/Lizenzhinweisgenerator/pull/240). :-) [08:24:41] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:38:52] 10PAWS, 10Pywikibot-General: PAWS: API error mwoauth-invalid-authorization: - https://phabricator.wikimedia.org/T163772#3208977 (10MarcoAurelio) [08:59:41] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [11:53:06] 06Labs, 10DBA: archive/archive_userindex is not filled in eswiki_p - https://phabricator.wikimedia.org/T133251#3209362 (10Superzerocool) After one year, is there any solution or just is a minor/lowest issue? [12:07:46] Change on 12www.mediawiki.org a page Wikimedia Labs/Puppet learning mode was modified, changed by 172.58.104.235 link https://www.mediawiki.org/w/index.php?diff=2454031 edit summary: [12:47:41] Change on 12www.mediawiki.org a page Wikimedia Labs/Puppet learning mode was modified, changed by Zhuyifei1999 link https://www.mediawiki.org/w/index.php?diff=2454046 edit summary: Undo revision 2454031 by [[Special:Contributions/172.58.104.235|172.58.104.235]] ([[User talk:172.58.104.235|talk]]) [13:31:36] 06Labs, 06Operations: Investigate ceasing new Trusty instance creation in Labs - https://phabricator.wikimedia.org/T161899#3209811 (10Andrew) As soon as we disable Trusty we'll also be violating 'cattle, not pets' for most of our users. It will mean that anytime they need to recreate an instance they will als... [13:38:38] 06Labs, 06Operations: Ensure we can survive a loss of labservices1001 - https://phabricator.wikimedia.org/T163402#3209846 (10Andrew) [13:48:01] andrewbogott: re: T161899 what do you mean by the "cattle, not pets" rule? [13:48:02] T161899: Investigate ceasing new Trusty instance creation in Labs - https://phabricator.wikimedia.org/T161899 [13:52:13] 06Labs, 06Operations: Investigate ceasing new Trusty instance creation in Labs - https://phabricator.wikimedia.org/T161899#3209873 (10chasemp) >>! In T161899#3209811, @Andrew wrote: > As soon as we disable Trusty we'll also be violating 'cattle, not pets' for most of our users. It will mean that anytime they... [13:52:16] paravoid: I mean that it turns instances that are a reproducible collective cattle into unique beloved and irreplaceable pets. Or at least pushes our users slightly further in that direction. [13:52:41] not entirely sure how that works in labs, but ok [13:52:45] what do you propose instead? [13:53:25] trusty is deprecated as the default instance in prod for almost a year now [13:54:06] trusty hosts are going to be < 10% of prod by end of Q4, but I understand the landscape (and %) in labs is very different [13:54:39] so I think this task was an attempt to avoid making the trusty deprecation a big deal where we need to contact all users etc. a year down the line [13:56:40] I see you and chasemp have been discussing on-task, so maybe I should just shut up here :) [13:57:14] I really don't have any strong opinion on what you guys should be doing in labs/wmcs fwiw [13:57:52] but I do want us to refresh distros a little more aggressively than we have done in the past in prod [13:58:01] and so far, to do that, we need to do that in tandem with labs as well :) [13:58:29] Well, we don't actually /have/ to keep labs in sync. The only cost is maintaining some version switching in the puppet repo. [13:58:59] If we just regarded that as the cost of doing business/being kind to Labs users rather than indulging our overwhelming urge to purge... [13:58:59] not just version switching [13:59:19] there are maintenance overheads in backports etc. as well [13:59:37] and holdbacks in all kinds of other areas, like ruby versions which have an effect in puppet versions and puppet facts and stuff like that [14:01:05] Sure, I'm not saying it's without a cost. Just that when we say we 'cannot' support those old distros in labs, what we're really saying is that we aren't willing to. As long as a distro is supported by upstream (as Trusty will be for almost 2 more years) we /can/ support it. [14:01:27] sure [14:02:03] we could support it past the EOL date too, by providing security updates for Ubuntu's main internally :P [14:04:07] there are some larger policy questions here -- is the role of labs to replicate what's in production or to provide a generic VM/IaaS product [14:04:51] if it's the former, it should be following what prod has, otherwise there's little point in e.g. playing with phabricator in a trusty instance in labs when prod runs phab only on jessie [14:05:21] I think the status quo and/or the vision is to be increasingly the latter [14:05:40] in which case and for those cases perhaps it doesn't make sense to be using prod's puppet tree and apt repo in the first place? [14:06:44] I wonder about that some time. There are cases where it's useful for labs instances to use the prod puppet tree, but maybe it makes sense to have a dual-tree approach. [14:06:44] 06Labs, 06Operations: Investigate ceasing new Trusty instance creation in Labs - https://phabricator.wikimedia.org/T161899#3209922 (10chasemp) For fwiw's the second is what I intended, a better title here would be `Investigate ceasing self-service new Trusty instance creation in Labs`. That's on me, I thought... [14:08:29] expanding on this thought further, if the vision is to offer a public cloud for the wikimedia movement, there's nothing stopping you from offering fedora or centos or whatever too [14:08:50] and of course to continue offering ubuntu too (LTS like 16.04 or even non-LTS) [14:09:06] so it really depends on how you guys view this product of yours :) [14:09:11] I think sanity is stopping us, or I hope it does [14:09:21] haha [14:09:42] well it depends, if you treat those VMs as just user VMs that you have nothing to do with, don't even get SSH access to etc. [14:09:52] "cattle, not pets" as andrew mentioned :) [14:22:18] yeah it's a fair point, I think for the foreseeable future we take enough responsibility (even where not managed service) for the state of most instances than makes that really a pipedream [14:22:26] but it's essentially a policy question [14:22:52] but all attempts to fully enforce a hands off we are strictly here for the pipes and wires approach has failed with "but if we just did x for them" etc [14:23:00] lots to be figured out [14:43:42] 06Labs, 06Operations: Investigate ceasing self-service new Trusty instance creation in Labs - https://phabricator.wikimedia.org/T161899#3210003 (10chasemp) [14:44:18] 06Labs, 06Operations: Investigate ceasing self-service new Trusty instance creation in Labs - https://phabricator.wikimedia.org/T161899#3146739 (10chasemp) >>! In T161899#3209922, @chasemp wrote: > fwiw's the second is what I intended, a better title here would be `Investigate ceasing self-service new Trusty i... [14:50:08] paravoid: regarding "role of labs", you might be interested in https://wikitech.wikimedia.org/wiki/Labs_labs_labs/future#Defining_Classes_of_Service [14:50:50] that's pretty old and written by me at a different time :) [14:51:21] but still, it means something [14:51:44] yeah I've seen this [14:52:00] yeah it's the beginnings of a thought but shouldn't be considered canonical in any sense [15:14:45] 06Labs, 10Labs-Infrastructure: When an instance is deleted, remove proxy records that point to it - https://phabricator.wikimedia.org/T163765#3210129 (10Andrew) This might turn out to be hard -- sink gets notified of instance deletion but not until after the instance's IP has been freed, so we don't have a goo... [15:33:06] 06Labs: Request creation of Wikidata Concepts labs project - https://phabricator.wikimedia.org/T163672#3205312 (10Andrew) Unless there's a public interface for e.g. Hadoop you won't be able to route there from a Labs VM. Having a public IP doesn't help with that. Are you sure you want a scratch start for this... [15:34:35] 06Labs: Request creation of Wikidata Concepts labs project - https://phabricator.wikimedia.org/T163672#3210222 (10Andrew) (No real objection to granting this resource request, if you're sure it's actually going to be useful to you) [15:43:33] 06Labs, 10Labs-Infrastructure: Audit disk usage on labvirts - https://phabricator.wikimedia.org/T163796#3210291 (10Andrew) [15:55:12] 06Labs, 10Labs-Infrastructure: Audit disk usage on labvirts - https://phabricator.wikimedia.org/T163796#3210350 (10chasemp) p:05Triage>03High [16:17:37] 06Labs: Request creation of Wikidata Concepts labs project - https://phabricator.wikimedia.org/T163672#3210402 (10Addshore) > In the near future the instance will also have to be able to connect to Spark from RStudio using {sparklyr} on our Hadoop cluster (production). Connecting from labs to production hadoop... [16:41:07] 06Labs, 10Tool-Labs, 07Documentation, 07Epic: Re-organize Tool Labs documentation - https://phabricator.wikimedia.org/T91509#3210573 (10bd808) [16:42:29] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1427 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:42:37] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1417 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:42:49] PROBLEM - Puppet errors on tools-exec-1429 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:43:09] PROBLEM - Puppet errors on tools-exec-1409 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:43:19] PROBLEM - Puppet errors on tools-prometheus-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:43:23] PROBLEM - Puppet errors on tools-flannel-etcd-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:43:29] PROBLEM - Puppet errors on tools-k8s-master-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:43:54] PROBLEM - Puppet errors on tools-exec-1435 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:44:06] PROBLEM - Puppet errors on tools-worker-1016 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:44:11] andrewbogott: ^ is this puppet spam you getting started or something else? [16:44:20] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:44:24] bd808: probably me [16:44:36] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:44:44] PROBLEM - Puppet errors on tools-worker-1017 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:44:44] PROBLEM - Puppet errors on tools-k8s-etcd-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:45:02] PROBLEM - Puppet errors on tools-mail is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:45:08] PROBLEM - Puppet errors on tools-exec-1410 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:45:14] PROBLEM - Puppet errors on tools-exec-1412 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:45:43] PROBLEM - Puppet errors on tools-worker-1011 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:45:59] PROBLEM - Puppet errors on tools-worker-1009 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:46:01] PROBLEM - Puppet errors on tools-worker-1018 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:46:19] PROBLEM - Puppet errors on tools-bastion-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:47:19] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:47:21] PROBLEM - Puppet errors on tools-grid-master is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:47:23] PROBLEM - Puppet errors on tools-exec-1418 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:47:34] andrewbogott its failing with Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to determine $::labsproject at /etc/puppet/manifests/realm.pp:41 on node puppet-paladox3.git.eqiad.wmflabs [16:47:36] PROBLEM - Puppet errors on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:47:42] PROBLEM - Puppet errors on tools-exec-gift-trusty-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:47:46] PROBLEM - Puppet errors on tools-elastic-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:47:50] PROBLEM - Puppet errors on tools-exec-1442 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:47:52] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:47:53] paladox: It's known maintenance. Thanks for looking! [16:47:58] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1415 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:47:59] Ok [16:48:01] thanks [16:48:14] PROBLEM - Puppet errors on tools-k8s-etcd-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:48:14] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1426 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:49:09] ha ha [16:49:11] okay [16:49:17] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:49:19] PROBLEM - Puppet errors on tools-worker-1006 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:49:23] madhuvishy: merge conflict! :) [16:49:29] :) [16:49:31] PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:49:31] PROBLEM - Puppet errors on tools-exec-1424 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:49:45] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:49:55] PROBLEM - Puppet errors on tools-worker-1012 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:49:57] andrewbogott: I'm going to kill shinken if that's okay - i can turn it back on whenever you want [16:49:57] PROBLEM - Puppet errors on tools-grid-shadow is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:49:58] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:50:13] (the irc echo stuff( [16:50:17] madhuvishy: sure. It should be getting recoveries now, so you can re-enable in 20 or so [16:50:17] PROBLEM - Puppet errors on tools-docker-builder-05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [16:50:21] okay [16:50:29] Or just let it yell :) [16:50:46] PROBLEM - Puppet errors on tools-checker-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:51:00] PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:51:02] PROBLEM - Puppet errors on tools-worker-1028 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:18:20] RECOVERY - Puppet errors on tools-prometheus-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:18:20] RECOVERY - Puppet errors on tools-prometheus-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:20:10] RECOVERY - Puppet errors on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [17:20:12] RECOVERY - Puppet errors on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [17:21:16] RECOVERY - Puppet errors on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:21:17] RECOVERY - Puppet errors on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:27] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1427 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:27] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1427 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:37] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:37] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:41] RECOVERY - Puppet errors on tools-exec-gift-trusty-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:41] RECOVERY - Puppet errors on tools-exec-gift-trusty-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:47] RECOVERY - Puppet errors on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:47] RECOVERY - Puppet errors on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:49] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:49] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:50] RECOVERY - Puppet errors on tools-exec-1429 is OK: OK: Less than 1.00% above the threshold [0.0] [17:22:50] RECOVERY - Puppet errors on tools-exec-1429 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:11] RECOVERY - Puppet errors on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:12] RECOVERY - Puppet errors on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:15] RECOVERY - Puppet errors on tools-k8s-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:16] RECOVERY - Puppet errors on tools-k8s-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:23] RECOVERY - Puppet errors on tools-flannel-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:24] RECOVERY - Puppet errors on tools-flannel-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:30] RECOVERY - Puppet errors on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:30] RECOVERY - Puppet errors on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:56] RECOVERY - Puppet errors on tools-exec-1435 is OK: OK: Less than 1.00% above the threshold [0.0] [17:23:56] RECOVERY - Puppet errors on tools-exec-1435 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:06] RECOVERY - Puppet errors on tools-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:06] RECOVERY - Puppet errors on tools-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:20] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:20] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:36] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:36] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:44] RECOVERY - Puppet errors on tools-k8s-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:44] RECOVERY - Puppet errors on tools-k8s-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:46] RECOVERY - Puppet errors on tools-worker-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:48] RECOVERY - Puppet errors on tools-worker-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:58] RECOVERY - Puppet errors on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [17:24:59] RECOVERY - Puppet errors on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:02] RECOVERY - Puppet errors on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:03] RECOVERY - Puppet errors on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:15] RECOVERY - Puppet errors on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:15] RECOVERY - Puppet errors on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:19] RECOVERY - Puppet errors on tools-docker-builder-05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:19] RECOVERY - Puppet errors on tools-docker-builder-05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:43] RECOVERY - Puppet errors on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:43] RECOVERY - Puppet errors on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:58] RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:58] RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:00] RECOVERY - Puppet errors on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:00] RECOVERY - Puppet errors on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:01] RECOVERY - Puppet errors on tools-worker-1028 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:01] RECOVERY - Puppet errors on tools-worker-1028 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:02] RECOVERY - Puppet errors on tools-worker-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:03] RECOVERY - Puppet errors on tools-worker-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:16] RECOVERY - Puppet errors on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:17] RECOVERY - Puppet errors on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:37] 06Labs, 10Labs-Infrastructure: When an instance is deleted, remove proxy records that point to it - https://phabricator.wikimedia.org/T163765#3208713 (10bd808) It looks like it would fairly trivial to add the instance id to the `Backend` tracked in the canonical sqlite database. The complete change would proba... [17:27:09] RECOVERY - Puppet errors on tools-cron-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:09] RECOVERY - Puppet errors on tools-cron-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:19] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:19] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:19] RECOVERY - Puppet errors on tools-grid-master is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:20] RECOVERY - Puppet errors on tools-grid-master is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:23] RECOVERY - Puppet errors on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:23] RECOVERY - Puppet errors on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:29] RECOVERY - Puppet errors on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:30] RECOVERY - Puppet errors on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:35] RECOVERY - Puppet errors on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:35] RECOVERY - Puppet errors on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:49] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:49] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:50] RECOVERY - Puppet errors on tools-exec-1442 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:50] RECOVERY - Puppet errors on tools-exec-1442 is OK: OK: Less than 1.00% above the threshold [0.0] [17:27:59] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [17:28:00] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [17:28:18] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1426 is OK: OK: Less than 1.00% above the threshold [0.0] [17:28:18] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1426 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:16] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:16] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:18] RECOVERY - Puppet errors on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:18] RECOVERY - Puppet errors on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:30] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:30] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:34] RECOVERY - Puppet errors on tools-exec-1424 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:34] RECOVERY - Puppet errors on tools-exec-1424 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:44] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:44] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:56] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:57] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:58] RECOVERY - Puppet errors on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:59] RECOVERY - Puppet errors on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [17:30:44] RECOVERY - Puppet errors on tools-checker-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:30:45] RECOVERY - Puppet errors on tools-checker-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:32:13] RECOVERY - Puppet errors on tools-exec-1422 is OK: OK: Less than 1.00% above the threshold [0.0] [17:33:03] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [17:36:05] sorry for the double alerts, fixed [17:37:20] 06Labs: Something should clean up dangling dynamicproxy records - https://phabricator.wikimedia.org/T163651#3210797 (10bd808) 05Open>03Resolved a:03Andrew @Andrew did a one time cleanup and is now looking at {T163765} [17:42:18] PROBLEM - Puppet errors on tools-services-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [18:17:19] RECOVERY - Puppet errors on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [18:17:34] 06Labs, 06Operations, 10hardware-requests: eqiad: (1) hardware access request for dedicated labmon1002 - https://phabricator.wikimedia.org/T161750#3210919 (10RobH) @chasemp: You list 32 cores, but labmon1001 has dual 8 core CPUs, for a total of 16 actual cores. It then has hyperthreading enabled, and shows... [18:30:12] PROBLEM - High iowait on tools-grid-master is CRITICAL: CRITICAL: tools.tools-grid-master.cpu.total.iowait (>22.22%) [18:45:31] * paladox has icinga notifications and is getting puppet errors now [18:55:21] 06Labs, 06Operations, 10hardware-requests: Eqiad: (2) hardware access request for labnet1003/1004 - https://phabricator.wikimedia.org/T158204#3211117 (10RobH) a:05RobH>03chasemp We don't do 4 CPU options. So we can toss in dual Intel CPUs with more cores, but we don't have anything that has that many ac... [19:10:30] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:17:21] Im getting a few failed jsub messages? [19:19:09] RECOVERY - Puppet errors on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [19:19:18] Betacommand: maintenance period but the grid should be ok atm [19:19:24] and hopefully stay that way [19:19:54] seems ok now [19:20:19] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [19:21:08] RECOVERY - Puppet errors on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [19:22:18] RECOVERY - Puppet errors on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:23:18] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [19:23:38] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [19:23:42] RECOVERY - Puppet errors on tools-exec-gift-trusty-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:23:48] RECOVERY - Puppet errors on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:23:50] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:15] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1426 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:15] RECOVERY - Puppet errors on tools-k8s-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:25] RECOVERY - Puppet errors on tools-flannel-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:29] RECOVERY - Puppet errors on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:24:53] RECOVERY - Puppet errors on tools-exec-1435 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:07] RECOVERY - Puppet errors on tools-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:17] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:19] RECOVERY - Puppet errors on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:39] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:45] RECOVERY - Puppet errors on tools-k8s-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:45] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:48] RECOVERY - Puppet errors on tools-worker-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:56] RECOVERY - Puppet errors on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:02] RECOVERY - Puppet errors on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:16] RECOVERY - Puppet errors on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:18] RECOVERY - Puppet errors on tools-docker-builder-05 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:42] RECOVERY - Puppet errors on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:58] RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:00] RECOVERY - Puppet errors on tools-worker-1028 is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:01] RECOVERY - Puppet errors on tools-worker-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [19:27:14] RECOVERY - Puppet errors on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:07] RECOVERY - Puppet errors on tools-cron-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:13] RECOVERY - Puppet errors on tools-exec-1422 is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:19] RECOVERY - Puppet errors on tools-grid-master is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:23] RECOVERY - Puppet errors on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:29] RECOVERY - Puppet errors on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:29] RECOVERY - Puppet errors on tools-worker-1029 is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:37] RECOVERY - Puppet errors on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:43] o.O [19:28:51] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:52] RECOVERY - Puppet errors on tools-exec-1442 is OK: OK: Less than 1.00% above the threshold [0.0] [19:28:59] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [19:29:14] RECOVERY - Puppet errors on tools-elastic-02 is OK: OK: Less than 1.00% above the threshold [0.0] [19:30:32] RECOVERY - Puppet errors on tools-exec-1424 is OK: OK: Less than 1.00% above the threshold [0.0] [19:30:42] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [19:30:56] RECOVERY - Puppet errors on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [19:30:56] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [19:31:12] RECOVERY - Puppet errors on tools-exec-1426 is OK: OK: Less than 1.00% above the threshold [0.0] [19:31:44] RECOVERY - Puppet errors on tools-checker-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:31:48] Sagan: it looks worse than it is [19:31:54] RECOVERY - Puppet errors on tools-worker-1027 is OK: OK: Less than 1.00% above the threshold [0.0] [19:31:58] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [19:32:31] RECOVERY - Puppet errors on tools-k8s-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:34:01] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [19:34:57] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [19:39:22] the hostname now dosent work [19:39:39] * paladox is going to have so many pings now :) [19:42:53] hmm is this expected [19:42:53] -id: jenkins-slave-01.git.eqiad.wmflabs [19:42:54] +id: jenkins-slave-01.eqiad.wmflabs [19:44:30] 10PAWS, 10Pywikibot-General: PAWS: API error mwoauth-invalid-authorization: - https://phabricator.wikimedia.org/T163772#3211372 (10Framawiki) [19:44:33] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login - https://phabricator.wikimedia.org/T136114#3211375 (10Framawiki) [19:45:54] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login, OAuth error: API error mwoauth-invalid-authorization - https://phabricator.wikimedia.org/T136114#2323722 (10Framawiki) [19:46:33] PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:47:15] PROBLEM - Puppet errors on tools-exec-1412 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:47:17] PROBLEM - Puppet errors on tools-docker-builder-05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:47:20] ^ andrewbogott see the message on dn vs fqdn from paladox [19:47:43] that seems totally normal to me [19:47:48] k [19:47:59] PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:49:07] PROBLEM - Puppet errors on tools-cron-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:49:23] PROBLEM - Puppet errors on tools-exec-1418 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:49:27] PROBLEM - Puppet errors on tools-exec-1433 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:49:35] PROBLEM - Puppet errors on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:49:51] PROBLEM - Puppet errors on tools-exec-1442 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:50:18] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1426 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:50:18] paladox: what is that paste from? [19:50:58] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:51:40] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [19:52:46] PROBLEM - Puppet errors on tools-checker-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:53:38] PROBLEM - Puppet errors on tools-docker-registry-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:54:56] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1421 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [19:55:33] PROBLEM - Puppet errors on tools-exec-1415 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:10:34] 06Labs, 06Operations, 10hardware-requests: Eqiad: (2) hardware access request for labcontrol1003/1004 - https://phabricator.wikimedia.org/T158207#3211444 (10RobH) [20:10:35] andrewbogott from jenkins-slave-01 [20:10:54] pinging it works [20:11:19] paladox: from the syslog? A puppet run? A web page? [20:11:25] puppet run [20:11:35] did it just now switch itself back again? [20:11:50] Not sure [20:11:52] i will check [20:12:01] though it generated a new certificate for puppet [20:12:01] Info: Caching certificate for jenkins-slave-01.eqiad.wmflabs [20:13:11] andrewbogott: not the only place I saw that [20:13:13] root@tools-docker-registry-02:~# puppet agent --test [20:13:14] Info: Creating a new SSL key for tools-docker-registry-02.eqiad.wmflabs [20:13:22] any ideas? [20:13:42] yep it switched back id: jenkins-slave-01.git.eqiad.wmflabs [20:13:59] paladox: and no problems with the cert? [20:14:05] Nope [20:14:15] Trying tools-docker-registry-02, will see if it recovers the same [20:14:19] seems fine [20:14:23] why would fqdn suddenly haves issues? [20:14:34] though pinging jenkins-slave-01.eqiad.wmflabs still works [20:14:51] paladox: it always did [20:14:55] oh [20:15:18] chasemp: I'm not sure. It might be that it falls back on dnsmasq if the dns servers aren't reachable or return nxdomain [20:15:31] andrewbogott: I have some more proof for you hang on [20:16:04] andrewbogott: https://phabricator.wikimedia.org/P5330 [20:16:08] 06Labs, 06Operations, 10hardware-requests: Eqiad: (2) hardware access request for labnet1003/1004 - https://phabricator.wikimedia.org/T158204#3211463 (10RobH) I chatted with Chase about this, and the dual 12 core for a total presented (in /etc/cpuinfo) will show 48, since 12 (cores per cpu) * 2 (cpus) * 2 (e... [20:16:08] a run during the issue [20:16:26] every puppet instance of foo.project.eqiad.wmflabs changed to foo.eqiad.wmflabs [20:16:34] $fqdn I'm assuming [20:16:49] https://phabricator.wikimedia.org/P5330$72-77 [20:16:51] etc [20:16:57] that seems really not good [20:17:05] but I don't get why [20:17:39] makes me wonder if that's not the cause of puppet flapping and who knows what it can cause [20:19:32] chasemp: back soon... [20:24:43] 06Labs, 06Operations: Investigate ceasing self-service new Trusty instance creation in Labs - https://phabricator.wikimedia.org/T161899#3211512 (10Andrew) Another thing I'd suggest is that we get Stretch available to users before we start pushing them off Trusty. Jessie isn't in support for much longer than T... [20:24:52] chasemp: lets make a ticket and let andrew poke at that a bit [20:25:09] yep I'll make that now [20:25:57] andrewbogott: now you are gone :/ [20:39:03] 06Labs, 06Operations: During labservices1001 failover fqdn changed from foo.project.eqiad.wmflabs to foo.eqiad.wmflabs - https://phabricator.wikimedia.org/T163823#3211561 (10chasemp) [20:43:13] 06Labs, 06Operations: During labservices1001 failover fqdn changed from foo.project.eqiad.wmflabs to foo.eqiad.wmflabs - https://phabricator.wikimedia.org/T163823#3211590 (10chasemp) I see a few requested certs for the foo.eqiad.wmflabs pattern on the Tools puppet master: ```root@tools-puppetmaster-02:~# pupp... [20:43:44] 06Labs, 06Operations: Investigate ceasing self-service new Trusty instance creation in Labs - https://phabricator.wikimedia.org/T161899#3211603 (10Paladox) +1 to stretch. [20:45:40] 06Labs, 10Tool-Labs, 15User-Urbanecm: Allow tool's maintainers to force HTTPS for their tool - https://phabricator.wikimedia.org/T163019#3211627 (10dpatrick) This seems to be a non-Security issue, and one which is best handled by another team, so I'm untagging the Security project. [20:49:47] 06Labs, 06Operations: During labservices1001 failover fqdn changed from foo.project.eqiad.wmflabs to foo.eqiad.wmflabs - https://phabricator.wikimedia.org/T163823#3211653 (10chasemp) [20:59:56] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login, OAuth error: API error mwoauth-invalid-authorization - https://phabricator.wikimedia.org/T136114#3211702 (10Dvorapa) @MarcoAurelio Hint: reinstalling pywikibot inside paws terminal was a solution for me ;) [21:30:27] !log staged ores-wmflabs-deploy:dc934e8 [21:30:28] halfak: Unknown project "staged" [21:30:29] Woops [21:30:34] !log ores staged ores-wmflabs-deploy:dc934e8 [21:30:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [21:33:24] !log ores deployed ores-wmflabs-deploy:dc934e8 [21:33:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [21:47:26] 06Labs, 10Labs-Infrastructure, 10Tool-Labs-tools-Other: Automatically updated list of all configured domains - https://phabricator.wikimedia.org/T45580#3212140 (10bd808) [21:59:54] 06Labs, 06Operations: During labservices1001 failover fqdn changed from foo.project.eqiad.wmflabs to foo.eqiad.wmflabs - https://phabricator.wikimedia.org/T163823#3212177 (10Andrew) Just turning off various dns services (including mdns) does not reproduce this issue. The change is probably in the puppet 'fqdn... [22:03:40] 10Striker, 07Epic: Manage shared tool accounts via Striker - https://phabricator.wikimedia.org/T149458#3212185 (10bd808) [22:36:49] 06Labs, 10Labs-Infrastructure: bootstrap_vz: Move firstboot.sh out of the base image? - https://phabricator.wikimedia.org/T161327#3128899 (10chasemp) I made two rough pitches for this during our convo: 1. We provide some metadata that is the sha1 of the intended script through safe means (metadata service)... [22:42:55] 10PAWS, 10MediaWiki-extensions-OAuth, 10Pywikibot-OAuth: PAWS can not login, OAuth error: API error mwoauth-invalid-authorization - https://phabricator.wikimedia.org/T136114#3212343 (10MarcoAurelio) @Dvorapa: would you please tell me how to do that? I am new to the PAWS platform. Thanks. [23:15:30] brought back shinken [23:29:25] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Optimize edit count queries in XTools - https://phabricator.wikimedia.org/T163284#3212475 (10DannyH) [23:38:55] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: XTools: Trying to get info about a talk page gives 500 internal server error - https://phabricator.wikimedia.org/T163508#3212517 (10DannyH)