[01:28:03] 10Tool-Labs-tools-Xtools, 03Community-Tech-Sprint: Unknown project should not cause an Internal Server Error - https://phabricator.wikimedia.org/T166332#3306618 (10Matthewrbowker) https://github.com/x-tools/xtools-rebirth/pull/33 [02:36:45] 10Tool-Labs-tools-Pageviews: [Bug] Querying for articles without page view stats produces weird results - https://phabricator.wikimedia.org/T166692#3306647 (10MusikAnimal) 05Open>03Resolved a:03MusikAnimal Should be all good now! Thanks for the report :) [02:38:16] 10Tool-Labs-tools-Pageviews, 07I18n: "Query for:" and "Individual projects" don't appear translated - https://phabricator.wikimedia.org/T164664#3306651 (10MusikAnimal) [02:53:08] 10Tool-Labs-tools-Pageviews, 07I18n: "Query for:" and "Individual projects" don't appear translated - https://phabricator.wikimedia.org/T164664#3306655 (10MusikAnimal) 05Open>03Resolved a:03MusikAnimal The usual developer negligence... apologies! The translations for "Query for:" should now be visible, a... [04:30:14] where should I put my request key and secret during a OAuth handshake? client-side cookies, server-side cookies, or somewhere else? [04:38:00] silly question nvm [06:23:30] 10Labs-project-other, 10MediaWiki-extensions-OAuth: Add OAuth 2.0 support to MediaWiki - https://phabricator.wikimedia.org/T125337#3306766 (10jayvdb) [06:38:57] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1421 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:18:57] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1421 is OK: OK: Less than 1.00% above the threshold [0.0] [07:54:03] 10Tool-Labs-tools-stewardbots, 06Stewards-and-global-tools, 07WorkType-Maintenance: Cleanup of the project stewardbots - https://phabricator.wikimedia.org/T130031#3306810 (10MarcoAurelio) [09:21:56] 06Labs, 10Quarry, 10Community-Wikimetrics, 10DBA, and 2 others: Evaluate future of wmf puppet module "mysql" - https://phabricator.wikimedia.org/T165625#3306961 (10jcrespo) Some modules maybe should install wmf-mariadb101-client ? [09:26:03] (03PS1) 10Elukey: Revert "Add zookeeper.yaml to hieradata common" [labs/private] - 10https://gerrit.wikimedia.org/r/356556 [09:26:10] (03CR) 10Elukey: [V: 032 C: 032] Revert "Add zookeeper.yaml to hieradata common" [labs/private] - 10https://gerrit.wikimedia.org/r/356556 (owner: 10Elukey) [10:19:57] (03CR) 10Jean-Frédéric: [C: 032] Moving sql template files out of sql dir into the template dir [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/356084 (owner: 10Lokal Profil) [10:21:06] (03Merged) 10jenkins-bot: Moving sql template files out of sql dir into the template dir [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/356084 (owner: 10Lokal Profil) [10:21:56] (03CR) 10jenkins-bot: Moving sql template files out of sql dir into the template dir [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/356084 (owner: 10Lokal Profil) [10:31:38] (03CR) 10Jean-Frédéric: [C: 032] Ensure sql transactions are committed [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/356089 (owner: 10Lokal Profil) [10:32:07] (03CR) 10Jean-Frédéric: [C: 032] "Good job figuring that one out :)" [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/356089 (owner: 10Lokal Profil) [10:33:20] (03Merged) 10jenkins-bot: Ensure sql transactions are committed [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/356089 (owner: 10Lokal Profil) [10:34:27] (03CR) 10jenkins-bot: Ensure sql transactions are committed [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/356089 (owner: 10Lokal Profil) [10:43:12] 10Tool-Labs-tools-Pageviews: Add SPARQL as input for Massviews - https://phabricator.wikimedia.org/T164181#3307072 (10Edgars2007) Sorry for late reply. Some "documentation". * Users are executing queries [[https://query.wikidata.org/]] * You can get results in JSON (in browser or with bots/tools) via https://qu... [10:49:45] (03CR) 10Jean-Frédéric: [C: 032] Don't crash just because we cannot save [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/356094 (owner: 10Lokal Profil) [10:50:03] (03CR) 10Jean-Frédéric: [C: 032] "That makes sense. Was also bothering me for a while when running locally." [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/356094 (owner: 10Lokal Profil) [10:50:56] (03Merged) 10jenkins-bot: Don't crash just because we cannot save [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/356094 (owner: 10Lokal Profil) [10:51:46] (03CR) 10jenkins-bot: Don't crash just because we cannot save [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/356094 (owner: 10Lokal Profil) [10:53:29] (03CR) 10Jean-Frédéric: [C: 032] Add address to handled wikidata fields. [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/356149 (owner: 10Lokal Profil) [10:54:33] (03Merged) 10jenkins-bot: Add address to handled wikidata fields. [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/356149 (owner: 10Lokal Profil) [10:55:33] (03CR) 10jenkins-bot: Add address to handled wikidata fields. [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/356149 (owner: 10Lokal Profil) [11:08:53] (03CR) 10Jean-Frédéric: [C: 032] Rename -lang argument -langcode [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/356152 (https://phabricator.wikimedia.org/T166528) (owner: 10Lokal Profil) [11:11:42] (03Merged) 10jenkins-bot: Rename -lang argument -langcode [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/356152 (https://phabricator.wikimedia.org/T166528) (owner: 10Lokal Profil) [11:16:15] (03CR) 10jenkins-bot: Rename -lang argument -langcode [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/356152 (https://phabricator.wikimedia.org/T166528) (owner: 10Lokal Profil) [12:20:12] 06Labs, 10Labs-Infrastructure: labvirt1006 super busy right now - https://phabricator.wikimedia.org/T165753#3307250 (10hashar) Slightly less loaded indeed. Thank you for the rebalance. [12:23:47] 06Labs, 10Labs-Infrastructure, 06Operations, 10ops-eqiad, 13Patch-For-Review: rack/setup/install labvirt101[5-8] - https://phabricator.wikimedia.org/T165531#3307252 (10chasemp) @cmjohnson @robh the `row b` requirement for labvirts and labnets is unfortunately still real as of now. We are working on it t... [12:43:07] 06Labs, 10Labs-Infrastructure, 10Tool-Labs: Rollout prometheus-node-exporter 0.14 in labs - https://phabricator.wikimedia.org/T166561#3307332 (10chasemp) p:05Triage>03Normal [12:43:38] 06Labs, 10Labs-Infrastructure, 10Tool-Labs: Rollout prometheus-node-exporter 0.14 in labs - https://phabricator.wikimedia.org/T166561#3300186 (10chasemp) sounds good to me, let us know when you hit the Tools road block with aptly and one of us can untangle (i.e. [[ https://phabricator.wikimedia.org/T152580#2... [12:45:59] 06Labs: nova-fullstack is losing instances on creation - https://phabricator.wikimedia.org/T165555#3307336 (10chasemp) >>! In T165555#3300129, @Andrew wrote: > Still no failures. of course, as soon as you said this we lost one on 5/31 :) ```+--------------------------------------+-----------------------+------... [14:00:29] PROBLEM - Puppet errors on tools-exec-1403 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:15:31] hey yall, i'm having an issue with puppet hiera and self hosted puppetmaster [14:15:43] lemme know if someone has a sec [14:15:49] ottomata: what's happening? [14:17:54] undefined method `[]' for nil:NilClass at /etc/puppet/modules/profile/manifests/kafka/broker.pp:22 on node kafkatest02.analytics.eqiad.wmflabs [14:18:06] i'm pretty sure that is happening inside of a puppet parser function [14:18:17] it works totally fine on non self hosted puppet masters [14:18:59] also, in addition to that, it seems hiera values are not being looked up properly in the local puppet master's hiera directory [14:19:16] i had to put them directly in the horizon interface to get rid of errors about missing hiera vars [14:19:56] the line in question (22 in this error), is for [14:19:57] kafka_config($kafka_cluster_name) [14:20:01] which is in kafka_config.rb [14:20:09] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Move wikitech-static to Chicago - https://phabricator.wikimedia.org/T164271#3307581 (10Andrew) wikitech-static-ord is now updating properly! There's some fancy automatic cert stuff on wikitech-static, so I'm hoping to refer the next steps to whoever set... [14:20:14] i've been trying to debug that, but no matter what I do I can't get any output [14:20:30] so i have a hunch it isn't running the code when I edit it, but another version of it somewhere [14:23:34] I'll take a look at that instance and see if I can find anything obvious... [14:25:07] where are you editing? in /var/lib/git/operations/puppet? [14:25:10] yes [14:25:25] so in that i've got modules/profile/manifests/kafka/broker.pp [14:25:29] and also [14:25:39] role::kafka::simple::broker [14:25:43] which includes that profile [14:25:48] and [14:25:50] so... [14:25:50] $config = kafka_config($kafka_cluster_name) [14:26:00] where does kafka_config come from? [14:26:06] hieradata/role/common/kafka/simple/broker.yaml [14:26:13] kafka_config is a pupppet parser function [14:26:27] kafka_config.rb in role module [14:26:42] hmm, maybe i should move it to profile! ? [14:27:05] its only in role because it was created before we had profiles [14:27:26] looks like it's getting installed properly [14:27:51] but that failure message sure isn't helpful [14:28:30] yeah, hence why i tried to put tons of debugs all over that thing [14:28:36] i even tried to write directly to a tempfile [14:28:38] coudlnt' get anything [14:30:17] what is up with git here? I can't do 'git log' in /var/lib/git/operations/puppet [14:30:37] ohhh, could that be a problem? iv'e been rsyncing my code in, and .git is big, so i don't rsync it [14:30:43] i will rsync the whole thing, hang on [14:35:46] andrewbogott: .git restored [14:35:49] same error [14:36:02] can I do a submodule update? [14:36:17] sure [14:39:45] So I tried hardcoding the cluster name and nothing changed. So presumably the error is somewhere deep in kafka_config() [14:40:04] which is a kind of thing I've never worked on. I guess I'd do a binary search stubbing out bits of that script to find where it's failing [14:40:12] unless you've already done that and know which line is unhappy? [14:52:45] andrewbogott: yeah [14:52:47] i've done that [14:52:48] but [14:52:52] i can't get any output [14:53:01] i modify kafka_config.rb directly [14:53:10] add logging statements, and have even tried writing directly to a tmp file [14:53:16] nothing comes out [14:53:25] which indicates to me that puppet is not actually running my code, but some cached version somewhere [14:53:32] i'll add a simple file output to the function [14:53:36] so we can test... [14:53:37] The cached version is in /var/lib/puppet/lib [14:53:51] PROBLEM - Puppet errors on tools-exec-1407 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:53:54] I'm not sure what forces a refresh, possibly 'service apache2 graceful' [14:54:06] but by watching that file hopefully you can figure out how to actually edit it :) [14:54:11] I have found that the puppetmaster needs to be restarted when changing ruby code [14:54:15] (Well, and double-check that that is actually the version getting called) [14:54:30] bd808: 'service apache2 graceful' == restarting the puppetmaster, right? [14:54:41] yeah I think so in the modern setup [14:54:49] oh really? [14:54:51] hmm ok [14:56:00] anytime I get weird responses from the puppetmaster my first debug attempt is to restart it. Maybe primitive but often helpful [14:56:20] ok, apache 2 restart worked, ok cool [14:56:22] i can debug now! [14:56:35] one more for ya though... [14:57:16] i have hieradata/role/common/kafka/simple/broker.yaml [14:57:18] and [14:57:20] in horizon [14:57:20] i've applied [14:57:27] role::kafka::simple::broker [14:57:28] in other classes [14:57:51] but [14:57:52] Could not find data item profile::kafka::broker::plaintext in any Hiera data file and no default supplied at /etc/puppet/modules/profile/manifests/kafka/broker.pp:63 [14:58:02] PROBLEM - Puppet errors on tools-mail is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:58:19] (gotta go, sorry, back sometime) [14:58:20] does adding a role in other classes load local hiera data role scope like i expect it to? [14:58:22] ok [14:58:23] thanks [14:58:32] i can at least work now, thank you andrewbogott! :) [14:58:40] The short answer is: I have no idea what decides what hiera to load [14:59:19] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:00:43] PROBLEM - Puppet errors on tools-exec-1406 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:01:03] PROBLEM - Puppet errors on tools-exec-1408 is CRITICAL: CRITICAL: 62.50% of data above the critical threshold [0.0] [15:01:10] PROBLEM - Puppet errors on tools-exec-1409 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:04:39] PROBLEM - Puppet errors on tools-exec-1404 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:09:35] ^ seems real [15:10:30] PROBLEM - Puppet errors on tools-exec-1405 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:10:41] Those are me, sorry [15:10:46] Doing more apt purges [15:10:50] ah [15:10:51] ok [15:10:59] I was deep into wtf is locking that file so cool [15:11:14] PROBLEM - Puppet errors on tools-exec-1401 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:11:59] chasemp: When faidon was updating facter yesterday we noticed that lots of VMs have full / which led me down a rabbit hole of purging old kernels left over from unattended upgrade. [15:12:09] gotcha [15:12:17] For whatever reason it takes like 10 minutes per kernel to purge the files. [15:14:09] PROBLEM - Puppet errors on tools-exec-1410 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:15:44] !log tools depooling/rebooting/repooling tools-exec-1403 as part of old kernel-purge testing [15:15:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:19:22] PROBLEM - Puppet errors on tools-exec-1402 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:22:24] 06Labs, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-LdapAuthentication, 10wikitech.wikimedia.org: Ldap auth extension vs. ldap vs. username Case - https://phabricator.wikimedia.org/T165795#3307884 (10Andrew) Bump! @Reedy or @Anomie can one of you propose a patch for this or do I n... [15:35:32] RECOVERY - Puppet errors on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [15:38:01] RECOVERY - Puppet errors on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [15:39:19] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [15:39:49] 06Labs, 10Labs-Infrastructure, 10DBA: Decide whether back-compat views for upcoming major schema changes will be provided in the Labs replicas - https://phabricator.wikimedia.org/T166798#3307956 (10Anomie) [15:39:58] 06Labs, 10Labs-Infrastructure, 10DBA: Decide whether back-compat views for upcoming major schema changes will be provided in the Labs replicas - https://phabricator.wikimedia.org/T166798#3307956 (10Anomie) a:05Anomie>03None [15:40:45] RECOVERY - Puppet errors on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [15:45:30] RECOVERY - Puppet errors on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [15:49:06] RECOVERY - Puppet errors on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [15:54:23] RECOVERY - Puppet errors on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [16:03:55] RECOVERY - Puppet errors on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [16:06:05] RECOVERY - Puppet errors on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [16:06:11] RECOVERY - Puppet errors on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [16:09:39] RECOVERY - Puppet errors on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [16:13:59] 06Labs, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-LdapAuthentication, 10wikitech.wikimedia.org: Ldap auth extension vs. ldap vs. username Case - https://phabricator.wikimedia.org/T165795#3308066 (10Anomie) I think I'll pass. Some notes: * There's a config option (`$wgLDAPLowerCas... [16:16:14] RECOVERY - Puppet errors on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [16:16:52] 06Labs, 10Tool-Labs, 15User-bd808, 06cloud-services-team (Kanban): webservice stop says service not running but service.manifest not cleared - https://phabricator.wikimedia.org/T163355#3308070 (10bd808) 05Open>03Resolved [16:19:22] 06Labs, 15User-bd808, 06cloud-services-team (Kanban): Consult with technical community on Cloud Services rebranding plan - https://phabricator.wikimedia.org/T165094#3308074 (10bd808) The announced time window for discussion has passed and feedback seems to have slowed down, so I have marked the initial talk... [16:20:08] 06Labs, 10Tool-Labs, 15User-bd808, 06cloud-services-team (Kanban): Intro to Cloud Services (Labs, Tool Labs, database replicas, oh my) - https://phabricator.wikimedia.org/T164814#3308075 (10bd808) 05Open>03Resolved a:03bd808 [17:01:06] PROBLEM - Host tools-exec-1403 is DOWN: CRITICAL - Host Unreachable (10.68.17.239) [17:06:55] RECOVERY - Host tools-exec-1403 is UP: PING OK - Packet loss = 0%, RTA = 0.74 ms [17:08:03] andrewbogott: tools-exec-1403 blip is that you? pretty unusual to see something totally disappear [17:08:13] yeah, still me tinkering [17:08:24] I'm going to restart it at least one more time [17:09:17] kk [17:13:41] Trying to figure out if there's any systematic way to really reclaim this disk space from the labvirt file system. I suspect the answer will be "possible but terrible" [17:14:57] andrewbogott: as in: virtual drive was filled, now cleared, but drive still takes all that space on the physical server? [17:15:23] valhallasw`cloud: right. So the process to reclaim is to zero-fill the running VM, stop it, recompress it, restart it [17:15:31] Which I'm doing now just to see if it works, and how slow/annoying it is [17:16:08] yes, that sounds about right. The only better option I can think of is if the virtualized host has a feature to do filesystem replication to a second drive [17:17:28] that sounds even harder/slower :) [17:18:00] possibly, but doesn't require creepy things like 'zero-filling a drive while mounted' ;-) [17:18:06] true [17:18:26] well, right now I'm avoiding that by zero-filling 90% of available space rather than 100% [17:39:11] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Move wikitech-static to Chicago - https://phabricator.wikimedia.org/T164271#3308434 (10Dzahn) @Andrew No, that wasn't me. That was Krenair. But i can take a look anyways once i get back.. (kind of off work for right now). [17:39:52] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Move wikitech-static to Chicago - https://phabricator.wikimedia.org/T164271#3308435 (10Dzahn) pretty sure that is LE's certbot and i can fix it. [17:40:40] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Can we search namespaces on wikitech? - https://phabricator.wikimedia.org/T165725#3308451 (10Andrew) @dcausse nothing has changed on the actual wikitech, right? Because this is still pending the above patch? If so, I'm totally fine with merging the patc... [17:41:56] PROBLEM - Host tools-exec-1403 is DOWN: CRITICAL - Host Unreachable (10.68.17.239) [17:43:20] RECOVERY - Host tools-exec-1403 is UP: PING OK - Packet loss = 0%, RTA = 3.65 ms [17:44:27] well, that was a waste of time, only got me 3Gb of space reclaimed [17:46:44] 06Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: Move wikitech-static to Chicago - https://phabricator.wikimedia.org/T164271#3308477 (10Andrew) Great, thank you! To clarify -- I want wikitech-static-ord to (eventually) be at https://wikitech-static.wikimedia.org so it should use the cert that the old w... [17:51:21] andrewbogott: :/ [17:51:55] andrewbogott: it might require a defrag beforehand, but that makes life difficult again (I assume the underlying representation will only cut of NULs at the end of the drive, not in the middle) [17:52:30] valhallasw`cloud: yeah, in general I think this isn't going to be worth the trouble and downtime. [17:59:56] *nod* delete and rebuild is probably a fair solution [18:00:23] or more disk space ;-) [18:03:11] there's plenty of disk space, I'm just being compulsive :) We can totally delete/rebuild tools nodes if it ever matters. [18:18:47] chasemp, andrewbogott: https://gerrit.wikimedia.org/r/356644 [18:19:16] could you guys let us know what you want to do with this [18:19:23] and potentially merge and babysit it? :) [18:20:51] chasemp: madhuvishy: also https://gerrit.wikimedia.org/r/#/q/topic:labstore-eth0 [18:21:10] paravoid: yeah i saw that - will review today [18:21:15] thx :) [18:21:23] not super important or urgent [18:21:25] thanks for the patches :) [18:21:29] yup okay [18:21:40] I'm just hunting down all of the hardcoded ethNs across the tree [18:24:39] were the interface_* facts already there or are they new? [18:28:36] interface_primary is new [18:28:56] "new", I pushed it on Feb 22nd :) [18:28:57] abf0e49c32acbf99a993e47dd482e4d194d23318 [18:29:17] facter 3 has $networking['primary'] but we don't have facter 3 yet [18:29:24] it's not even in Debian, I was working on it a few months back [18:29:53] facter 3 was essentially rewritten from Ruby to C, and uses a couple of (C) libraries puppetlabs wrote, so it's a bit complicated :) [18:30:12] so interface_primary will do for now and we can just do a search and replace when the time comes [18:30:42] I also fixed ipaddress/ipaddress6, their values are always what you'd expect them to be now [18:30:52] so we fixed all kinds of cases where we had $ipaddress_eth0 etc. [18:44:40] paravoid: nice, thanks for all the details :) [18:49:40] madhuvishy: you'll go through https://gerrit.wikimedia.org/r/#/q/status:open+project:operations/puppet+branch:production+topic:labstore-eth0 then? [18:49:53] chasemp: yup :) [18:49:55] k [18:53:12] chasemp: https://gerrit.wikimedia.org/r/356644 is probably for you though :) [18:54:53] paravoid: if that's ready to roll do you mind if I merge it first thing tomorrow my morning, or monday? read up a bit on the setting and seems in order but I have a thing in a bit and couldn't hang out in case of chaos today [18:55:07] fine with me :) [18:55:22] it's really labs-only at this point, not blocking us in anything [18:55:39] but it'd be nice to have the two environments be consistent again :) [18:56:01] yep, fyi then I'll plan on first thing my day and a ping to let you know [18:56:21] the divergence (and the disgusting "if realm" guard) was meant to be temporary in order to speed things along [19:01:47] thanks paravoid [19:02:25] thank you :) [19:44:59] (03PS1) 10BryanDavis: Add #wikimedia-cloud handling [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/356662 (https://phabricator.wikimedia.org/T166420) [19:45:42] (03CR) 10BryanDavis: [C: 032] Add #wikimedia-cloud handling [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/356662 (https://phabricator.wikimedia.org/T166420) (owner: 10BryanDavis) [19:46:17] (03Merged) 10jenkins-bot: Add #wikimedia-cloud handling [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/356662 (https://phabricator.wikimedia.org/T166420) (owner: 10BryanDavis) [20:26:11] Change on 12www.mediawiki.org a page OAuth/For Developers was modified, changed by Enterprisey link https://www.mediawiki.org/w/index.php?diff=2479390 edit summary: /* Example code */ added a full Python tutorial [20:46:40] (03PS1) 10BryanDavis: Add bot to #wikimedia-cloud [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/356670 (https://phabricator.wikimedia.org/T166420) [21:00:29] (03CR) 10Paladox: [C: 031] Add bot to #wikimedia-cloud [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/356670 (https://phabricator.wikimedia.org/T166420) (owner: 10BryanDavis) [21:01:22] looks good to me ^^ [23:12:55] 06Labs, 10Labs-Infrastructure: monitor some things on all Cloud instances (discussion) - https://phabricator.wikimedia.org/T166845#3309532 (10Andrew) p:05Triage>03Normal