As an anti-spam measure, you are limited from performing this action too many times in a short space of time, and you have exceeded this limit. [10:24:24] Please try again in a few minutes. [10:25:04] :) [10:25:21] if ( $wgUser->pingLimiter() || $wgUser->pingLimiter( 'linkpurge', 0 ) ) { [10:25:24] $status->fatal( 'actionthrottledtext' ); [10:25:26] $status->value = self::AS_RATE_LIMITED; [10:27:52] case self::AS_RATE_LIMITED: [10:27:52] throw new ThrottledError(); [10:28:00] which [10:28:00] $wgOut->setStatusCode( 503 ); [10:29:14] I think it should be a 403 instead [10:29:50] There is a more specific header too [10:30:11] 429 Too Many Requests [10:30:12] what do you mean Nemo_bis? [10:30:41] If you're going to make that more specific you may also use 429 or similar [10:30:55] 418 also seems appropriate:) [10:31:10] hmm, 429 is very new, I wasn't aware of it [10:31:16] it's rfc6585, apr 2012 [10:31:24] I've seen some mediawikis using it [10:31:34] note that this isn't an API request though [10:31:37] (at webserver level I suppose, and very few) [10:31:40] it's a regular edit page [10:31:45] so I wonder how browsers use it [10:31:52] s/use/display/ [10:31:53] Ah [10:32:41] and the web says that Apache doesn't understand it and converts 429 to 500... [10:33:15] -> https://issues.apache.org/bugzilla/show_bug.cgi?id=44995 [10:33:42] (03PS8) 10Matanya: Torrus: add torrus to netmon1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/108314 [10:33:47] :( [10:34:02] I think 403 is the safest choice here [10:34:13] I'll submit a patch, and we can discuss it there. sounds sensible? [10:35:16] yeah, better not lose track of it [10:35:30] * Nemo_bis was just reporting something seen in the wild [10:35:39] yup, it was very useful [10:35:39] thanks [10:41:51] dependency tree : ship manutius to eqiad --> move torrus from manutius to netmon1001 --> make torrus a module [10:44:18] (03PS4) 10Matanya: torrus: move into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/108498 [10:44:38] (03PS1) 10ArielGlenn: wikiqueries: pep8 and fixup of usage message [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/120764 [10:45:02] apergos: ohhi [10:45:14] apergos: the fighter planes are driving me nuts [10:45:37] F35? [10:45:46] something like that [10:48:08] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC [10:48:22] apergos: https://rt.wikimedia.org/Ticket/Display.html?id=6789 do you need help chasing people? [10:48:47] yes they are getting on my last nerve too [10:48:50] https://gerrit.wikimedia.org/r/120765 <- throttlederror [10:48:54] ( paravoid ) [10:49:11] and matanya sure that would be great [10:49:51] which group you want me to handle? [10:50:24] pick a grup, any group :-) [10:50:30] (I'll make a note so we don't overlap) [10:51:17] I'll take A and B [10:51:33] if i finish quickly, i'll move on :) [10:52:01] k, this does mean followup before just removing though [10:52:25] because someone on stat1002 may not be actively using it or mayhave different jobs on one host than the other (for example) [10:52:32] users do crazy things [10:53:55] (03CR) 10ArielGlenn: [C: 032] wikiqueries: pep8 and fixup of usage message [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/120764 (owner: 10ArielGlenn) [10:54:04] sure, not working in shouting mode: i.e lets close and see who shouts :) [10:59:45] (03PS1) 10ArielGlenn: refactor generation of per project media lists [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/120766 [10:59:52] :-) [11:11:15] (03CR) 10ArielGlenn: [C: 032] refactor generation of per project media lists [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/120766 (owner: 10ArielGlenn) [11:15:42] (03PS1) 10ArielGlenn: rename wmfgetremoteimages.py to what it does (listmediaperproject.py) [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/120769 [11:18:19] (03PS2) 10ArielGlenn: rename wmfgetremoteimages.py to what it does (listmediaperproject.py) [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/120769 [11:19:18] (03CR) 10ArielGlenn: [C: 032] rename wmfgetremoteimages.py to what it does (listmediaperproject.py) [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/120769 (owner: 10ArielGlenn) [11:20:05] (03PS4) 10Tim Landscheidt: Tools: Unify Tools and Toolsbeta configuration [operations/puppet] - 10https://gerrit.wikimedia.org/r/102385 [11:23:14] apergos: one questions, users that are already disabled, but not removed, should i push a patch to remove them ? [11:23:48] no, we keep the user [11:26:19] thanks [11:33:54] (03PS1) 10Faidon Liambotis: twmeproxy: monitor the memcache port as well [operations/puppet] - 10https://gerrit.wikimedia.org/r/120774 [11:35:27] (03CR) 10Faidon Liambotis: [C: 032] twmeproxy: monitor the memcache port as well [operations/puppet] - 10https://gerrit.wikimedia.org/r/120774 (owner: 10Faidon Liambotis) [12:10:31] (03PS1) 10ArielGlenn: puppetize generation of media lists cron job on snapshot hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/120778 [12:24:32] (03PS2) 10ArielGlenn: puppetize generation of media lists cron job on snapshot hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/120778 [12:27:06] (03CR) 10ArielGlenn: [C: 032] puppetize generation of media lists cron job on snapshot hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/120778 (owner: 10ArielGlenn) [12:36:51] (03PS1) 10ArielGlenn: fix paths for media lists generation on snapshots [operations/puppet] - 10https://gerrit.wikimedia.org/r/120780 [12:39:00] (03CR) 10ArielGlenn: [C: 032] fix paths for media lists generation on snapshots [operations/puppet] - 10https://gerrit.wikimedia.org/r/120780 (owner: 10ArielGlenn) [13:36:57] (03PS1) 10Mark Bergsma: Assign new subnet private1-esams [operations/dns] - 10https://gerrit.wikimedia.org/r/120792 [13:48:57] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC [14:00:58] (03CR) 10Mark Bergsma: [C: 032] Assign new subnet private1-esams [operations/dns] - 10https://gerrit.wikimedia.org/r/120792 (owner: 10Mark Bergsma) [14:09:53] (03CR) 10Hashar: "> wouldn't it be better to backport a newer version? gem2deb is your friend :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120498 (owner: 10Hashar) [14:18:11] heya cmjohnson1, which rack are the analytics nodes in now? [14:18:20] the three we moved yesterday? [14:25:39] (03CR) 10Ottomata: "Yup! templates/mariadb/ would be a great place for WMF specific stuff. Then each role class in manifests/roles can specify the WMF speci" (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119930 (owner: 10Springle) [14:26:02] ottomata: i started talking to folks about moving to stat1002 [14:26:04] hashar_: next time i claim jenkins does anything wrong and it's just me adding a dependency, i'll send you beer .. to your house:) [14:26:45] can you help with communication in th analytics-l ? [14:27:18] hashar_: do you want me to backport puppet-lint ? [14:27:30] yeah matanya, for sure, i have two emails I am going to respond to after the meeting im' in [14:27:35] which thread do are you referring to? [14:27:41] matanya: sure [14:28:05] matanya: would not get puppet-lint on our slaves anytime soon though unless you manage to get the package reviewed/upload to apt.wikimedia.org quickly :-] [14:28:10] ottomata: ticket 64789, i can't post to that list [14:28:28] matanya: I use forks hosted on gerrit nowadays and deploy them with git-deploy or git::clone. [14:28:37] mutante: sold :-] [14:28:39] hashar_: give me a few days, i guess [14:28:48] matanya: thanks :] [14:29:17] some carzy stuff the last few days at my job, slows me down a bit [14:29:32] ok, i'll be back later [14:30:22] matanya: ticket 64789? in what? [14:30:22] * 6789 in RT [14:41:06] matanya should not have to help us in finding if paid employees are still using our prod systems [14:42:42] (03PS1) 10Mark Bergsma: Add subnet private1-esams, remove storage1-esams [operations/puppet] - 10https://gerrit.wikimedia.org/r/120800 [14:43:45] haha, mutante, matanya volunteered to help! [14:46:09] (03PS1) 10ArielGlenn: add config file for media list generation on snapshots [operations/puppet] - 10https://gerrit.wikimedia.org/r/120801 [14:48:02] (03CR) 10ArielGlenn: [C: 032] add config file for media list generation on snapshots [operations/puppet] - 10https://gerrit.wikimedia.org/r/120801 (owner: 10ArielGlenn) [14:48:14] ottomata: d2...thx have to move in racktables [14:50:00] ja i have to move in puppet to, hadoop is rack aware [14:50:11] cmjohnson1: we going to do the remaining node today? [14:51:02] ottomata: let's do tomorrow. I am working on the tampa stuff today [14:51:46] mutante: sorry to have you subscribed to yet another list :-D [14:52:11] ottomata: if you want to move today we can do around 2:30...will that work? [14:53:10] ok cool, no probs cmjohnson1 [14:53:14] naw, tomorrow is better [14:54:33] can't do the kafka network test move today ottomata :( [14:54:33] (03PS1) 10Ottomata: Updating Hadoop net-topology config with new rack locations of moved datanodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/120803 [14:54:50] ok [15:02:08] (03PS3) 10ArielGlenn: decom snapshot1-4 [operations/puppet] - 10https://gerrit.wikimedia.org/r/120615 (owner: 10Dzahn) [15:03:48] if I wanted to figure out what hosts had a specific module, can I discover that in the environment w/ salt? is there another standard way? [15:04:47] (03CR) 10ArielGlenn: [C: 032] "thanks mutante!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120615 (owner: 10Dzahn) [15:06:46] apergos: cool!:) kill all the tampa [15:06:58] sstomp stomp [15:08:09] (03CR) 10Mark Bergsma: [C: 032] Add subnet private1-esams, remove storage1-esams [operations/puppet] - 10https://gerrit.wikimedia.org/r/120800 (owner: 10Mark Bergsma) [15:09:07] chasemp: i believe you could find out with salt if we had gerrit change 107831, but for now i'd just look in site.pp [15:09:31] chasemp: specific module you're interested in? [15:09:52] hashar_: it will work if i start using more "mute" feature for some other threads:) [15:10:11] mutante: I guess each list in its own folder, so I can easily skip a list entirely :] [15:10:15] well I'm looking to modify some, and something I have done in teh past is dynamic discovery of who has it in the env, and then pick a host at randon and confirm a good puppet run. it's a nice manual sanity check if I'm feeling risky [15:10:43] so I'm trying to figure out if I modify a module how do I confirm puppet is ok where it is exists [15:13:39] you could see if classes from that module are applied on the host (/var/lib/puppet/state/classes.txt) and then manually parse the last part of the log for good entries (I do that part clunkily n one of my scripts)... is this what you mean, verifying the run is good? [15:15:52] that is an option once I find the hosts I think. So not advocating it, but was hoping salt had something similar to the mcollective integration, where you can do '/usr/sbin/mc-find-hosts --with-class | sort' and get a returned list of all hosts w/ a class applied [15:16:07] and then pick one out programmatically and run puppet on it and collect the output [15:16:19] ah [15:16:57] reading up on salt grains now, unsure if it's similar [15:20:46] that's the intention of that gerrit change [15:20:56] to use the puppet role to add a salt grain [15:21:12] so then you can select "--with-role" [15:23:34] heya, what is the difference between wikitech-l and enginneering@? [15:23:42] do you know mutante? [15:24:10] also, apergos, I am about to send an email out asking about the accoutns under Follow Up: here [15:24:11] http://etherpad.wikimedia.org/p/stat1_accounts [15:24:11] ottomata: not really, it's a good question [15:24:15] any thoughts before I do that? [15:24:20] mutante: thanks man, roles are good but I'm looking a little more granular I think, although I may need to phrase my question better [15:24:39] uh can you give me sorry about this bout about 15 mins? [15:24:44] then I can look [15:26:09] ottomata: strictly per definiton: wikitech-l = "discussing the technical organization of the Wikimedia projects", engineering = "WMF Engineering staff and contractors (only!) list." so the difference is who get subscribed .. [15:26:51] so you can't get on engineering as a volunteer [15:26:57] if you want to tell community you need wikitech-l [15:27:31] but i totally agree there is soo much overlap.. they could likely be merged? [15:29:20] hm, i think that probably analytics-l + engineering is what I want [15:29:52] ottomata: if it's for those accounts, yes, agree [15:30:02] ottomata: well, there is a catch-22 there [15:30:16] if people get unsubscribed from engineering after they are not contractors... [15:30:25] but we want to find out people who are not anymore.. [15:31:00] then it's getting tricky without asking HR [15:31:28] i have a pretty good idea of those folks, i think mutante [15:31:35] !log snapshot1-4 in pmtpa powered off, decom :-) [15:31:38] but there could be some on my Follow Up list, for sure [15:31:39] that I don't know about [15:31:40] Logged the message, Master [15:33:05] apergos: did you work on labs ganglia at all? [15:33:18] (03PS2) 10Ottomata: Updating Hadoop net-topology config with new rack locations of moved datanodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/120803 [15:33:24] (03CR) 10Ottomata: [C: 032 V: 032] Updating Hadoop net-topology config with new rack locations of moved datanodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/120803 (owner: 10Ottomata) [15:33:25] no [15:33:28] ottomata: engineering sounds best bet... yea [15:33:31] not even for a seconf [15:33:33] second [15:33:45] ah now I can look at that email though [15:34:18] apergos: ok, it must've just been sara smollet. I'm just pinging everyone on the admin list. [15:34:34] which, hm, csteipp is on there too... [15:35:29] I did what? [15:35:42] apergos: https://gerrit.wikimedia.org/r/#/c/120060/ [15:36:03] killing from icinga first though [15:36:06] you're fast [15:36:12] I already have it gone form icinga [15:36:26] hosts are powerd down; that implies gone from icinga [15:36:45] just saw :) [15:37:10] only thing that was left was dns but I was going to look at otto's email first [15:37:17] well, not that fast, just prepared before needing it and voted down with "wait" :) [15:37:26] uh [15:37:31] how do I look at that email [15:37:31] you read the email [15:37:54] oh. I was just being asked abut the list, not the email [15:38:37] so uh ottomata, I would just explain the rationale in the email (why we are asking them about removal) and I don't have any other thoughts beyod that, except [15:38:55] as bad as it sounds, copies on the ticket so we have a record [15:38:58] chasemp: maybe /var/lib/puppet/ (client_data, client_yaml...) on palladium, the puppetmaster is another option to find that out [15:39:37] good thought, I haven't culled the master logs before for client state, unsure of what I'll find. I'll go digging [15:40:11] apergos: ottomata: or just say "whoever did not login in the last X month, needs to say something or it gets removed"? not sure [15:40:18] this assumes that reporting made it back to the master [15:40:22] yeah not a bad idea [15:40:24] you can't alays be sure of that [15:40:39] mutante: there are a few different categories is all (of user) [15:46:28] (03PS2) 10Dzahn: decom: remove snapshot1-4 [operations/dns] - 10https://gerrit.wikimedia.org/r/120060 [15:54:05] RoanKattouw: I see you started already [15:55:12] I started some prep work yeah [15:59:19] (03CR) 10Dzahn: [C: 032] decom: remove snapshot1-4 [operations/dns] - 10https://gerrit.wikimedia.org/r/120060 (owner: 10Dzahn) [15:59:56] !log DNS update - removing snapshots [16:00:02] Logged the message, Master [16:00:55] mutante: will ya'll ceremoniously shutdown -h now the last server in tampa? [16:01:17] like, live stream it and pop champagne? [16:01:57] Nikerabbit: Did you do any work on documentation and config changes? [16:02:28] greg-g: i wish.. but it will be hard to determine because some stuff is still just moving to different floor [16:02:43] i was about to offer something similar to apergos though [16:02:52] RoanKattouw: config changes I submitted patches for [16:02:54] because of that nice uptime:) [16:03:01] apergos: ms5...:) [16:03:04] :-D [16:03:10] how long? [16:03:39] uptime is 1091 days [16:03:56] RoanKattouw: Kartik is working on the extension page update: https://www.mediawiki.org/wiki/User:KartikMistry/Extension:LocalisationUpdate [16:04:04] Nikerabbit: Alright. The config change is harmless without the code using it but not vice versa, so I'm gonna deploy that first [16:04:10] (03CR) 10Catrope: [C: 032] New LocalisationUpdate config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119945 (owner: 10Nikerabbit) [16:04:23] (03Merged) 10jenkins-bot: New LocalisationUpdate config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119945 (owner: 10Nikerabbit) [16:04:26] oh wow over 1000 days [16:04:27] nice [16:04:30] apergos: i'm checking right now why it's still in icinga :) [16:04:40] already removed it from some tings [16:04:43] well I see it's gone from salt but I can still log int [16:04:48] guess you are in the middle of that process [16:04:51] * apergos gets off [16:04:53] that's cause i deleted yesterday [16:05:04] i'm a bit surprised it's still in icinga [16:05:12] because i deleted from puppetstoredconfigs etc [16:05:29] apergos: check out the screen process from 2011 :) [16:05:38] sec [16:05:46] mutante: oh right [16:05:52] SCREEN -S qps_measurement [16:05:58] 2011 1:18 [16:06:18] RoanKattouw: yup, the puppet change is mostly about removing outdated params and comments [16:06:19] oh wow... old manual thumb removal [16:06:21] man that sucked [16:06:28] 2011 179:56 /bin/bash :) [16:18:04] RoanKattouw: how far are you? [16:19:13] (03CR) 10Mark Bergsma: [C: 031] Adding archiva module and role, applying on titanium [operations/puppet] - 10https://gerrit.wikimedia.org/r/117024 (owner: 10Ottomata) [16:19:54] great, thanks mark! [16:20:12] !log catrope synchronized wmf-config/CommonSettings.php 'New LocalisationUpdate config' [16:20:18] Logged the message, Master [16:20:27] Nikerabbit: Sorry, got confused because git claimed there was an undeployed commit, while the logs claimed it was deployed and it did appear to be deployed [16:20:34] So it took some time to convince myself it was safe to proceed [16:21:48] rright [16:21:52] I've had that as well [16:22:30] long live svn [16:24:12] (03PS4) 10Dzahn: add salt grains automatically in system::role [operations/puppet] - 10https://gerrit.wikimedia.org/r/107831 [16:25:23] (03CR) 10Dzahn: "per Mark's comment and discussion with Chase, now using $name. (as it meanwhile already is used in deployment.pp as well)." [operations/puppet] - 10https://gerrit.wikimedia.org/r/107831 (owner: 10Dzahn) [16:25:29] !log catrope synchronized php-1.23wmf19/extensions/LocalisationUpdate 'Deploy rewrite for 1.23wmf19' [16:25:32] (03CR) 10Dzahn: add salt grains automatically in system::role (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/107831 (owner: 10Dzahn) [16:25:36] Logged the message, Master [16:26:26] !log restarting hadoop namenodes to bring in new net topology layout [16:26:32] Logged the message, Master [16:26:40] (03CR) 10Dzahn: "compare: modules/deployment/manifests/target.pp: salt::grain { "deployment_target_${name}":" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107831 (owner: 10Dzahn) [16:27:00] oh? row-aware namenode topology? [16:29:24] (03CR) 10Rush: [C: 032] "seems good" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107831 (owner: 10Dzahn) [16:30:20] paravoid, yup! [16:30:25] well, datanode, yeah [16:30:52] Nikerabbit: Hmm, are you sure that the code in l10nupdate-1 to do with CDB-to-JSON conversion doesn't need changing? [16:31:10] hadoop will use that to try to send work for data that is in the same row to the same nodes, in order to minimize cross row traffic [16:31:25] RoanKattouw: can you point me to the lines? [16:31:48] paravoid: https://gist.github.com/ottomata/9765738 [16:32:16] Nikerabbit: Never mind, I'm an idiot [16:32:30] ottomata: awesome [16:32:55] RoanKattouw: you mean the Localisation*Cache* files? Those haven't changed ;) [16:33:00] has to be manually maintained though :/ would be super cool if I could hook that up to an api or something somewhere [16:33:17] we have some vague plans about that [16:33:20] but nothing concrete for now [16:33:25] https://github.com/wikimedia/operations-puppet/blob/production/templates/hadoop/net-topology.py.erb [16:33:35] Nikerabbit: Yeah exactly [16:34:27] LU parses PHP and JSON to JSON, LC parsers PHP, JSON and LU JSON to CDB, the sync script converts CDB to JSON and rebuilds CDB files from JSON on the target servers [16:34:42] parses* [16:34:58] lol [16:35:02] Yay complexity! :) [16:35:14] I thought I knew how it worked but after this summary I feel confused [16:35:36] I did change LU to store files in JSON as opposed to serialized files [16:37:15] bd808: see the craziness which is above :) [16:37:23] Hmmm [16:37:30] Nikerabbit: I ran update.php but I can't find the .json files now [16:37:31] * bd808 reads backscroll [16:37:37] just the last 6 or so lines [16:37:50] I think they're supposed to be in /var/lib/l10nupdate/cache-1.23wmf19 but that directory has no *.json files [16:38:18] bd808: I hope I haven't given you a headache :) [16:38:20] RoanKattouw: did you check the output of update.php? [16:38:31] Nikerabbit: It was one line [16:38:36] "N things updated" or whatever [16:38:55] if N > 0, then it hopefully stored some files somewhere! [16:39:04] Yeah N was in the 2000s [16:39:51] Trying to use eval.php to track down the save path [16:39:55] Which files are we looking for? The cdbs or the json that is built on tin from the cdbs to send to the nodes so they can build local cdbs? [16:40:14] bd808: json build by LocalisationUpdate [16:40:17] "/var/lib/l10nupdate/cache-$wmfVersionNumber" [16:40:53] that matches where you were looking for them [16:41:34] On 3rd reading, the summary above is very simple ^^ [16:41:52] Hmm… I actually have managed to avoid looking at that bit of code. I know how scap manages things in /a/common/php-*/cache/l10n [16:43:43] Oh, hold on [16:43:47] Permissions [16:43:58] hah [16:45:21] Hmm [16:45:28] Trying to figure out how this even worked in the first place [16:45:36] It seems like the permissions problems should already be a problem [16:45:38] I hear that way too often... [16:45:51] /var/lib/l10nupdate/cache-* is owned by l10nupdate [16:45:59] But mwscript forces scripts to run as apache [16:46:01] * bd808 nods [16:46:18] wut [16:46:56] can you check dates of those files? [16:47:20] scap (mw-update-l10n) does `sudo -u l10nupdate $BINDIR/mwscript rebuildLocalisationCache.php …` [16:47:28] They were updated recently, some on Mar 25 [16:47:30] So clearly it does work [16:47:33] *somehow* [16:47:42] bd808: Sure, but mwscript does sudo -u apache, doesn't it? [16:47:49] Unless it has a weird $BINDIR ... [16:48:33] Oooooooh [16:48:34] if groups | grep -Ewq 'sudo|wikidev|root'; then [16:48:35] RoanKattouw: It does the sudo one when groups | grep -Ewq 'sudo|wikidev|root' [16:48:36] OK [16:48:37] I see [16:49:34] (03CR) 10Andrew Bogott: [C: 032] Fix error: timidity service can not be stopped [operations/puppet] - 10https://gerrit.wikimedia.org/r/118709 (owner: 10Hashar) [16:49:41] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC [16:50:32] Sweet! [16:50:35] OK I have JSON files now [16:50:51] mutante: did you remove the labstores in tampa? see the latest icinga message? [16:51:18] RoanKattouw: nice!! [16:51:22] anything more to do? [16:52:18] Now going to sync those .json files to wmf19 [16:52:21] And gonna check if they work [16:52:27] I found a message in nl that changed [16:52:36] https://www.mediawiki.org/wiki/MediaWiki:Mobile-frontend-nearby-to-page/nl [16:52:42] greg-g: no, labstore2 puppet run has been doing this for a long time, it's more a question for labs migration [16:52:59] mutante: gotcha [16:53:26] RoanKattouw: with sync you mean rebuilding l10n cache and syncing that? [16:54:08] Yes [16:54:08] andrewbogott, do you have time for a quick question re labs migration? [16:54:11] ok [16:54:12] Hmm, the cache refused to rebuild [16:54:17] It said it was fresh [16:54:20] gwicke: sort of… what's up? [16:54:35] andrewbogott, I was wondering if the home dirs are migrated too [16:54:46] so far that does not seem to be the case yet [16:54:49] gwicke: It depends on if you asked me to or not. [16:55:01] RoanKattouw: hmm, I think I've seen that... the first time might need --force [16:55:09] If you didn't ask, then an async copy of the home dirs should be in /home/glustercopy or /home/pmtpa-nfs-copy [16:55:14] andrewbogott, it would be great if the home dirs of the VE project would be migrated [16:55:16] It may or may not be up-to-date. [16:55:32] ah, ok [16:55:35] Nikerabbit: OK I'll --force [16:55:37] RoanKattouw: because the hook hasn't run yet, it has no knowledge of the new json files [16:56:26] gwicke: let me know if what you have is sufficient or if you need me to do an rsync [16:57:02] andrewbogott, there were no important changes in the last days, so if that copy is a few days old then we should be fine [16:57:22] currently rsyncing my home dir back [16:57:32] gwicke: depends on if it was gluster or nfs. If gluster it might be a couple of weeks back. [16:57:55] they were on gluster [16:58:10] in any case the worst is that I'll have to re-upload a deb [16:58:15] so no problem [16:58:28] thanks! [17:00:27] (03PS3) 10Hashar: Make syslog-ng basepath a parameter [operations/puppet] - 10https://gerrit.wikimedia.org/r/119256 [17:00:42] (03CR) 10Hashar: "rebased to fix conflicts" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119256 (owner: 10Hashar) [17:01:40] (03PS3) 10Hashar: Create roles for syslog-ng [operations/puppet] - 10https://gerrit.wikimedia.org/r/119257 [17:02:15] !jenkins beta-parsoid-update-eqiad [17:03:43] (03CR) 10Andrew Bogott: Make syslog-ng basepath a parameter (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119256 (owner: 10Hashar) [17:03:57] ottomata: what's with all the varnishkafka warnings? [17:04:08] and packetloss avg [17:07:05] why does ms5 not wanna die..eh..i mean disappear from icinga, i've done the same for a bunch of hosts without problems [17:07:07] hume: Failed to add the RSA host key for IP address '2620:0:860:2:21d:9ff:fe33:f235' to the list of known hosts (/home/l10nupdate/.ssh/known_ho). [17:07:09] fenari: Failed to add the RSA host key for IP address '2620:0:860:2:208:80:152:165' to the list of known hosts (/home/l10nupdate/.ssh/known_ho). [17:07:11] Yay for errors no one pays attention to [17:07:32] paravoid, looking [17:08:09] !log Syncing rebuilt l10ncache for 1.23wmf19, built with new LocalisationUpdate version [17:08:14] Logged the message, Mr. Obvious [17:08:50] hmm, ganglia down? [17:09:11] (03CR) 10Andrew Bogott: [C: 032] Create roles for syslog-ng [operations/puppet] - 10https://gerrit.wikimedia.org/r/119257 (owner: 10Hashar) [17:10:06] paravoid, ganglia is being weird [17:10:14] those icinga alerts get their data from ganglia [17:10:22] kafka looks ok [17:10:52] where is packet_loss_avg alert (that also comes from ganglia, i think) [17:12:46] andrewbogott: fwiw, virt1001-1009, they have pending unaccepted salt keys, if they are fine, they should probably be accepted so that salt works on them [17:13:06] mutante: hm, that would explain some things :) [17:13:20] I don't think I know how to accept a salt key, is that something done on palladium? [17:13:22] andrewbogott: on palladium, salt-key -L lists them all [17:13:26] ok [17:13:33] thanks! [17:13:37] salt-key -a virt1001.eqiad.wmnet [17:13:39] should work [17:13:54] so far i've been deleting all the decom stuff.. but yea [17:13:58] that's why i see it [17:14:17] (03PS1) 10Hashar: role::parsoid::beta needs contint slave scripts [operations/puppet] - 10https://gerrit.wikimedia.org/r/120823 [17:14:45] ok, I left 1009 because I need to rebuild that one anyway. [17:14:46] andrewbogott: yep, looks good ! [17:15:12] yea, as long as a host is running it will try to readd itself there automatically [17:15:18] but just not be accepted [17:15:37] so you can only really delete one from that as well after a host is physically shutdown [17:15:43] (03PS2) 10Ottomata: access: remove erosen [operations/puppet] - 10https://gerrit.wikimedia.org/r/120759 (owner: 10Matanya) [17:15:49] (03CR) 10Ottomata: [C: 032 V: 032] access: remove erosen [operations/puppet] - 10https://gerrit.wikimedia.org/r/120759 (owner: 10Matanya) [17:15:54] paravoid, mark, any further objections to this, or can we start experimenting? http://www.mediawiki.org/wiki/Talk:Requests_for_comment/Reducing_image_quality_for_mobile#Impact_on_infrastructure [17:16:08] not sure about [17:16:09] labnet1001.eqiad.wmnet [17:16:09] labsdb1004.eqiad.wmnet [17:16:10] (03CR) 10Hashar: [V: 032] "deployed manually by doing the git clone on deployment-parsoid04.eqiad.wmflabs which is the sole node using the role::parsoid::beta class." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120823 (owner: 10Hashar) [17:16:12] andrewbogott: [17:16:25] oh, labnet is real... [17:16:31] dunno about labsdb, I think that's Coren or springle [17:17:04] labsdb100[45] are real. [17:17:24] Although, by design, only one of them is turned on at any one time. [17:17:42] Sweetness! [17:17:45] Nikerabbit: LU works :) [17:17:52] Now gonna upgrade it on wmf18 and rerun everything [17:18:01] interesting that palladium has not signed the key for palladium [17:18:10] RoanKattouw: thanks! [17:18:21] andrewbogott: same thought, i dunno what's up with salt master signing itself [17:19:31] (03PS14) 10Ottomata: Adding archiva module and role, applying on titanium [operations/puppet] - 10https://gerrit.wikimedia.org/r/117024 [17:19:39] (03CR) 10Ottomata: [C: 032 V: 032] Adding archiva module and role, applying on titanium [operations/puppet] - 10https://gerrit.wikimedia.org/r/117024 (owner: 10Ottomata) [17:20:02] (03CR) 10Hashar: Make syslog-ng basepath a parameter (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119256 (owner: 10Hashar) [17:20:41] andrewbogott: i could probably ask at salt user group tomorrow [17:21:07] apergos: something is weird about ms5 and icinga, one puppet run removes it partly, another one adds it again... [17:21:25] mutante: It seems straightforward to me -- if a salt command contains a wildcard that encompases palladium, then salt should act on palladium. [17:21:29] So I think it should be signed. [17:21:46] that is: palladium should act as client as well as master. [17:21:59] andrewbogott: i was more about "why is it trying to add itself" [17:22:05] but agreed, it's just both [17:22:16] (03CR) 10Hashar: Make syslog-ng basepath a parameter (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119256 (owner: 10Hashar) [17:22:36] (03PS1) 10Ottomata: Setting archiva port properly [operations/puppet] - 10https://gerrit.wikimedia.org/r/120825 [17:22:46] andrewbogott: fixed [17:22:48] (03CR) 10Ottomata: [C: 032 V: 032] Setting archiva port properly [operations/puppet] - 10https://gerrit.wikimedia.org/r/120825 (owner: 10Ottomata) [17:23:08] (03PS4) 10Andrew Bogott: Make syslog-ng basepath a parameter [operations/puppet] - 10https://gerrit.wikimedia.org/r/119256 (owner: 10Hashar) [17:23:12] !log salt-keys: removed snapshots1-4, signed palladiums own salt key [17:23:13] mutante: I'll look in 2 minutes [17:23:18] Logged the message, Master [17:23:25] I had removed the snapshot keys [17:23:30] (03PS4) 10Andrew Bogott: Create roles for syslog-ng [operations/puppet] - 10https://gerrit.wikimedia.org/r/119257 (owner: 10Hashar) [17:23:34] apergos: there's 2 kinds of "remove" [17:23:44] ? [17:24:14] apergos: salt key can be one of 3 states: existing and accepted, existing but not accepted, gone [17:24:30] salt-key -d should toss an accepted key or even a rejected key [17:24:57] back in 2 min must finish this [17:25:02] it makes an accepted one unaccapted, but you will have it reappear as a pending one [17:25:14] as long as the host is still running [17:25:36] because it will keep adding itself.. that part is just cruft ..not important but still cruft [17:26:58] (03PS1) 10ArielGlenn: generate alternate index html of dumps sorted by wiki name [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/120826 [17:27:24] it is cruft, so that means salt key shoudl be removed after host shutdown [17:27:38] can you update the server lifecycle page? (or I can) [17:28:35] (03CR) 10ArielGlenn: [C: 032] generate alternate index html of dumps sorted by wiki name [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/120826 (owner: 10ArielGlenn) [17:28:47] apergos: yes, it means one should run salt-key -d after the actual shutdown command [17:29:41] Alright, let's finish this LU thing [17:29:54] apergos: that page merely links to the salt page [17:30:00] let me check there [17:30:05] just move the step down [17:30:11] its listed as before shutdown [17:32:15] apergos: done [17:32:28] awesome [17:32:28] !log catrope synchronized php-1.23wmf18/extensions/LocalisationUpdate 'LU rewrite' [17:32:34] Logged the message, Master [17:33:53] hashar: Is Beta Labs back to normal speed now? [17:33:55] (03PS1) 10ArielGlenn: update dumps template to allow multiple index html pages [operations/puppet] - 10https://gerrit.wikimedia.org/r/120828 [17:34:47] hashar: (Its slowness was blamed on the transfer from pmtmp to eqiad.) [17:35:43] (03CR) 10ArielGlenn: [C: 032] update dumps template to allow multiple index html pages [operations/puppet] - 10https://gerrit.wikimedia.org/r/120828 (owner: 10ArielGlenn) [17:35:48] James_F: cache bust yesterday [17:36:00] not sure why ec [17:36:03] +t [17:36:05] greg-g: Ah, just that? It seems normal-ish speed now. [17:36:07] * James_F nods. [17:37:07] James_F: hello [17:37:15] hashar: Hey. :-) [17:37:17] James_F: beta is still on the pmtpa cluster. The eqiad version is not active yet [17:37:31] hashar: Ah, right. [17:37:37] James_F: the slowdown I have no idea. Yesterday around midnight I got all the varnishes upgraded and puppet run on them [17:37:43] James_F: might have cleared the caches. [17:37:47] Midnight UTC? [17:37:57] Midnight GMT+1 [17:38:04] ~ 3pm PST ? [17:38:07] or 4 [17:38:17] yurik: the RFC isn't accepted, so no [17:38:32] hashar: Right, yeah, that could explain. [17:39:21] yurik: also, I'm pretty much against moving forward with this unless the simplify RFC gets implemented, as I said [17:39:27] apergos: don't look at the ms5 thing anymore,..fixed [17:39:57] ok [17:40:41] yurik: finally, this is one of the things that as we discussed, we need to plan ahead in quarterlies etc.; you can't just drop that on us and start experimenting, sorry [17:40:46] RoanKattouw: I'm no longer needed, correct? [17:41:03] Nikerabbit: Yeah I think that's right [17:41:11] It seems to work on wmf19 [17:41:18] So I'm just rolling it out to wmf18 now [17:41:28] It's just that all these build steps are so slow :( [17:42:15] RoanKattouw: I know.. prolly among the slowest steps [17:43:22] greg-g: can we sneak in to deploy couple changes to test.wikidata, in about 20 minutes from now? (or when?) [17:43:40] aude: depends :) [17:43:42] we'd like to confirm the patches fix the issues before deploy to wikidata [17:44:04] ah, so before reedy does the update today [17:44:08] (03PS2) 10Ori.livneh: Setting recurse_submodules => true on labs self hosted puppet clones [operations/puppet] - 10https://gerrit.wikimedia.org/r/119766 (owner: 10Ottomata) [17:44:08] right [17:44:10] (03CR) 10Hashar: [C: 04-1] Make syslog-ng basepath a parameter (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119256 (owner: 10Hashar) [17:44:22] (workaround is bump parser cache epoch again but would like to avoid, and think we have fix) [17:44:24] aude: so you'll need two backports? [17:44:25] (03CR) 10Ori.livneh: [C: 032] Setting recurse_submodules => true on labs self hosted puppet clones [operations/puppet] - 10https://gerrit.wikimedia.org/r/119766 (owner: 10Ottomata) [17:44:33] James_F: I am off sorry :-/ [17:44:39] we'll package it as one, but it's two wikibase changes [17:44:41] g'night hashar ! [17:44:41] hashar: No worries, it seems faster now. [17:44:44] hashar: good night [17:44:46] hashar: Good night1 [17:44:48] James_F: if there are slowdownns, I guess folks should be pointed to beta and fix it up! [17:44:49] !log ms5 - 1091 days of uptime, but this was the last, shutdown -h now [17:44:50] ori: :-] [17:44:55] Logged the message, Master [17:44:56] * aude waiting for jenkins, so it will take 15-20 minutes before it is ready [17:45:35] (03CR) 10Ori.livneh: "Ryan, ping! Should this be filed as a GH issue?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119232 (owner: 10BryanDavis) [17:45:43] aude: if you're ready now, go ahead and do it to testwikidata, but that basically means it'll automatically go out to wikidata without you doinug anything else (ie: if you merge to wmf20) [17:45:44] ori: hello [17:45:49] hi matanya [17:45:50] what's the change? [17:46:12] wait, wmf20? [17:46:20] ori: what is the fate of olivne account? sould it be delete/removed/kept ? [17:46:38] * aude thinks wmf19 [17:47:24] greg-g: give me 15 minutes to have jenkins merge things [17:47:27] k [17:47:32] matanya: i'm not sure -- what's the policy these days? it used to be at least that we don't delete old accounts [17:47:36] just disable them [17:47:38] aude: er, wmf19, sorry [17:47:42] well, ori, i'm deleting [17:47:44] just not migrating [17:47:49] to new stat1003 server from stat1 [17:47:55] ori: i think we still keep them [17:48:01] meaning, not copying homedirs [17:48:03] or including accounts [17:48:07] in puppet on the new server [17:48:08] ottomata: yeah definitely no need to copy [17:48:11] ok cool [17:48:20] ottomata: for how much longer will stat1 be accessible? [17:48:33] i should probably take a quick peek to see if i have anything of value my old home dir [17:49:43] apr 1 ori [17:49:51] * ori nods [17:49:54] but please do it before :) [17:52:08] ori, a week or two [17:53:18] oh, why is gerrit inaccessible [17:53:26] just me? [17:53:31] (03PS1) 10Reedy: Non Wikipedias to 1.23wmf19 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120830 [17:53:59] aude: reedy's prepping ^ :) [17:54:04] i see [17:54:15] Reedy: aude has a change to go out with wmf19 if it works on testwikidata [17:54:22] aude: web interface seems to be down [17:54:22] * aude can't access gerrit [17:54:33] ^d ^ [17:54:38] qchris_away and ^demon|away are both away [17:54:49] :( [17:55:02] icinga hasn't complained yet [17:55:18] do we have alerts on gerrit (not just the machine)? [17:55:21] Anyone from ops around mind poking gerrit? [17:55:34] I thought we got webserver down type warnings [17:55:41] fetching seems not to work also [17:56:02] fetch WFM [17:56:18] maybe not from outside [17:56:24] tru [17:56:25] e [17:57:03] pull works remotely [17:57:09] gerrit is back [17:57:20] damnt [17:57:28] * Reedy blames YuviPanda [17:57:29] slow [17:57:45] I'm in no rush to deploy as with time changing it's now foood time [17:58:06] i am fast as gerrit and jenkins allow [17:58:14] mutante: you decom'ed pc1-3, right? they're still in icinga [17:58:33] I think springle actually did it last night [17:58:40] mutante might've made the changeset [17:59:14] paravoid: no, i asked springle about the mysql replication to eqiad to pc100x and he made a pending change, getting to that next [17:59:47] mutante: Reedy you didn't fing the purge checkuer on hume ? [17:59:55] fing? [17:59:56] find? [17:59:57] fix? [17:59:59] did snapshots [18:00:00] find [18:00:16] https://wikitech.wikimedia.org/wiki/Gerrit#Details says manganese still not ytterbium [18:00:30] matanya: "not puppetized" means in this case "it's not running at all" it seems [18:00:31] not logged into wikitech [18:01:06] back in half an hour or so [18:01:09] k [18:01:20] mutante: so you can merge, in that sense i think, though this job must be running somewhere, i think [18:01:20] matanya: that cron doesn't appear to exist anymore [18:01:31] ok [18:01:42] if you merge, i push hume decom, deal? [18:02:27] matanya: the puppet code looks good to me, and i already voted, but as long as i know nothing about the script it's starting and why it was removed.. no [18:02:59] so we will wait, the question is who to ask [18:03:23] suggests platform [18:03:37] name ? ^ [18:04:39] cajoel: hi, any news on formey ldap ? [18:04:59] PROBLEM - Host ms5 is DOWN: PING CRITICAL - Packet loss = 100% [18:06:01] what ..the ..heck [18:06:16] i just checked that 5 times.. we does it keep coming back unlike the other hosts [18:06:20] :%s/formey/sanger [18:06:38] salt key mutante ? [18:07:04] no, puppet stored configs, but i know it was gone [18:08:34] * aude running out of patience [18:09:56] greg-g: OK, LU deployment all done [18:10:13] RoanKattouw: wow :) [18:10:18] Sorry for the slowness [18:10:26] Cache rebuild scripts are slow [18:10:29] It seems to be working fine [18:10:29] * greg-g nods [18:10:33] stuff is slow today [18:10:35] great [18:10:36] But we'll find out for real in the next few days [18:10:42] (03PS1) 10Spage: Enable Flow on Compact Personal Bar BF talk page [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120833 [18:13:26] (03PS1) 10Ottomata: Setting up simple nginx proxy to archiva on port 8080 [operations/puppet] - 10https://gerrit.wikimedia.org/r/120837 [18:14:29] (03PS4) 10coren: Redirect pywikipedia.org to Tools [operations/apache-config] - 10https://gerrit.wikimedia.org/r/109237 (owner: 10Tim Landscheidt) [18:14:46] (03PS2) 10Ottomata: Setting up simple nginx proxy to archiva on port 8080 [operations/puppet] - 10https://gerrit.wikimedia.org/r/120837 [18:15:26] (03CR) 10coren: [C: 032] "Redirect seems correct." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/109237 (owner: 10Tim Landscheidt) [18:16:11] Tim-away: when you are back, please ping me [18:17:14] regarding CU purge script [18:19:17] (03PS1) 10Catrope: Enable math VE plugin on labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120838 [18:19:57] still waiting for jenkins for the last thing [18:20:19] (03PS3) 10Ottomata: Setting up simple nginx proxy to archiva on port 8080 [operations/puppet] - 10https://gerrit.wikimedia.org/r/120837 [18:20:26] (03CR) 10Ottomata: [C: 032 V: 032] Setting up simple nginx proxy to archiva on port 8080 [operations/puppet] - 10https://gerrit.wikimedia.org/r/120837 (owner: 10Ottomata) [18:22:34] (03PS1) 10Ottomata: Fixing erb variable for nginx simple-proxy.erb [operations/puppet] - 10https://gerrit.wikimedia.org/r/120840 [18:22:40] (03PS2) 10Ottomata: Fixing erb variable for nginx simple-proxy.erb [operations/puppet] - 10https://gerrit.wikimedia.org/r/120840 [18:22:45] (03CR) 10Ottomata: [C: 032 V: 032] Fixing erb variable for nginx simple-proxy.erb [operations/puppet] - 10https://gerrit.wikimedia.org/r/120840 (owner: 10Ottomata) [18:24:37] (03CR) 10Jforrester: [C: 031] Enable math VE plugin on labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120838 (owner: 10Catrope) [18:29:23] (03CR) 10Catrope: [C: 031] New LocalisationUpdate config [operations/puppet] - 10https://gerrit.wikimedia.org/r/119946 (owner: 10Nikerabbit) [18:29:37] (03PS1) 10Ottomata: Adding CNAME archiva.wikimedia.org -> titanium.wikimedia.org [operations/dns] - 10https://gerrit.wikimedia.org/r/120843 [18:29:39] robh: Hey could https://gerrit.wikimedia.org/r/#/c/119946/ be merged & deployed please? [18:30:52] (Not very urgent but would be nice to get done today) [18:31:03] RoanKattouw: yea np [18:31:26] RoanKattouw: do i need to force puppet runs for affected hosts or just let it merge and done? [18:31:52] robh deserves a cookie for doing RT duty well. [18:31:52] Force run isn't really needed [18:31:56] Yes he does [18:32:06] (03CR) 10RobH: [C: 032] New LocalisationUpdate config [operations/puppet] - 10https://gerrit.wikimedia.org/r/119946 (owner: 10Nikerabbit) [18:32:27] =] [18:32:35] robh: you going to zurich or london? [18:32:37] RoanKattouw: its live [18:32:43] london, not zurich [18:32:53] I'll buy you a beer there, then [18:33:02] greg-g: we are ready [18:33:44] aude: k, have hoo deploy to wmf19 (testwikis) and test before Reedy gets back :) [18:33:52] ok [18:33:57] hoo: ^ [18:34:00] you want to do [18:34:39] https://gerrit.wikimedia.org/r/#/c/120845/ [18:34:53] can do, if needed [18:35:02] waiting for approval [18:35:05] PROBLEM - check_mysql on lutetium is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [18:35:07] or aude, I guess [18:35:14] doesn't matter [18:35:21] usually better i don't deploy my own stuff [18:35:34] :) [18:35:42] although know that happens all the time [18:35:57] people deploying own stuff (config changes etc) [18:36:02] greg-g: if we're good to go, I can do it [18:36:05] I'm already here :P [18:36:07] go ahead [18:36:20] jenkins... [18:38:06] ok, done [18:38:11] :) [18:40:01] deployed? [18:40:05] PROBLEM - check_mysql on lutetium is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 967 [18:40:06] I see no !log [18:40:09] (03CR) 10RobH: "I'm not sure I understand why this group is being made, when restricted is supposed to be just bastions." [operations/puppet] - 10https://gerrit.wikimedia.org/r/116019 (owner: 10Hoo man) [18:40:14] !log hoo synchronized php-1.23wmf19/extensions/Wikidata/ 'Update Wikidata, patch for Wikibase js config and revert entity selector patch' [18:40:17] there [18:40:17] greg-g: There we go ;) [18:40:20] Logged the message, Master [18:40:21] heh [18:40:32] does it work? [18:40:33] and the item works [18:40:35] yay [18:40:37] \o/ [18:40:38] checkign more [18:40:39] why are folks putting restricted user group on private data hosts ;_; [18:40:45] 'restricted' is the name! [18:40:47] * aude loves caching [18:41:05] robh: Reviewing my change? :P [18:41:10] all seems good [18:41:13] (03PS5) 10ArielGlenn: add salt grains automatically in system::role [operations/puppet] - 10https://gerrit.wikimedia.org/r/107831 (owner: 10Dzahn) [18:41:24] robh: Thanks man! Also, another random question: does the "you must have a different key for the cluster vs Gerrit/labs" also apply to non-roots? [18:41:25] aude: Yay :) [18:41:35] hoo: im a bit confused about it [18:41:48] In other words, if I'm arranging a shell request for someone, do I get them to generate a new key or grab their existing one? [18:42:06] RoanKattouw: New one, apergo.s will enforce that :P [18:42:07] RoanKattouw: yep, cuz non roots may have deployment based off production key [18:42:11] its not new! [18:42:15] its always been this way =] [18:42:28] we're just now making folks do it more ;] [18:43:04] hoo: So im confused about the change, cuz it seems to be to make a more restrictive group than restricted. If folks are putting that bastion restricted group on private data hsots [18:43:10] they need to stop and make groups for that host [18:43:28] rather than suddenly decide to promote what should be restricted, by its very name, heh [18:43:43] so i understand why the change was done, and it should be done, but it should be done in the opposite direction (imo) [18:44:03] robh: well that's because our restricted is as restricted as ... well it's terribly unrestricted [18:44:09] yep [18:44:18] OK [18:44:22] folks got lazy and just said 'restricted allowed' [18:44:25] I'll get him to create a new one [18:44:25] which is shame on them! [18:44:34] So the idea is to have more specific groups for all service functions, so that everyone get's the access they need (and not more) [18:44:37] RoanKattouw: cool, if he can toss on officewiki it proves its him [18:44:37] This is explicitly for deployment rights [18:44:42] k [18:45:00] RoanKattouw: Well, deployment can be granted at a later date and we dont make folks do a new key when that happens [18:45:05] PROBLEM - check_mysql on lutetium is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1267 [18:45:18] hoo: hrmm, i can see your point [18:45:27] (03CR) 10ArielGlenn: "I have verified that: the bugfix for ':' is in fact in our installed salt version, that the grains get added as advertised and multiple gr" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107831 (owner: 10Dzahn) [18:45:44] (03CR) 10ArielGlenn: [C: 032] add salt grains automatically in system::role [operations/puppet] - 10https://gerrit.wikimedia.org/r/107831 (owner: 10Dzahn) [18:45:51] robh: Long-term restricted should probably be renamed (or killed), but we're just not there yet [18:45:55] hoo: hey atleast I came back to it like promised! [18:46:01] Reedy: so, yeah, all good with the wikidata update to wmf19 [18:46:06] yea, admins is in horrible shape [18:46:15] see the TODO at the top :) [18:46:15] though i think our newest opsen is reviewing it actually [18:46:35] i wonder if i shouldnt turf this to him cuz he may be doing something like it already (im not trying to just palm this off,i promise!) [18:46:39] Yep and mutant.e also has a rewrite up in gerrit [18:46:47] i jsut dont wanna merge and piss folks off cuz they think i should have done something else. [18:47:13] If I only had more time for this... would be nice if you could poke him [18:47:24] i dont see him online right now, so i'll make a note to track him down and ask when i see him [18:47:31] +1 [18:47:33] i expect sometime today, but if not then tomorrow, or until i do [18:47:35] rt duty wooooo [18:47:37] =] [18:47:42] :) [18:47:43] (its still on my radar) [18:48:01] I'm going to try keeping an eye on security / access stuffs [18:48:22] (03CR) 10RobH: "I think we have some opsen working on re-factoring how admins.pp is handled. I'll check in with them later today/tomorrow and pass this a" [operations/puppet] - 10https://gerrit.wikimedia.org/r/116019 (owner: 10Hoo man) [18:48:43] aude: As the world didn't explode and I'm hungry, I'm going to leave for food in a second... [18:48:52] ok [18:49:36] food sounds like a reasonable plan... but the tuesday air raid hasnt started yet in SF [18:49:40] cannot eat until the air raid siren [18:49:50] greg-g: What TODO? [18:49:51] it’s lunchhorn! [18:50:05] PROBLEM - check_mysql on lutetium is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [18:50:10] if i eat before the horn [18:50:15] when it goes off i'll just be hungry again. [18:50:16] Reedy: oh, sorry, not for you, that was for robh, the TODO at the top of admins.pp [18:50:34] greg-g: heh, yep [18:50:38] i think chase took that on as new dude [18:50:42] poor sucker. [18:50:47] :) [18:51:59] robh: Hmm, so I'm doing this puppet change for a new shell user, and I see we have set UIDs now. They seem to follow no pattern whatsoever. How do I determine the UID for a new user? [18:52:12] hehe [18:52:20] there is no pattern [18:52:29] greps [18:52:38] Come up with a number, grep to verify it's not used? [18:52:47] i'd append to the highest one normally [18:52:50] though seems folks have gone nuts. [18:52:54] Yeah well they're not ordreed [18:53:14] yea well 1232 was highest of the lower #s [18:53:24] but then jtwo folks made themselves ik 2k and 4k range [18:53:25] grrr [18:53:28] Right [18:53:31] changing it later may be problematic [18:53:32] rush is in the 4k range [18:53:38] I can do 1233 [18:53:44] i have no clue why they are so far off [18:53:47] please do yes [18:54:07] hrmm, there are a bunch in 3k [18:54:09] RoanKattouw: i dunno! [18:54:32] who the heck decided to just jump up thousands and go [18:54:33] I thought you used whatever their uid in labs is? [18:54:35] =/ [18:54:42] thats 100% new to me [18:54:48] I read that recently, like yesterday [18:54:51] though explains why some are so far off [18:54:54] (03PS1) 10Catrope: Add esanders shell account [operations/puppet] - 10https://gerrit.wikimedia.org/r/120851 [18:54:57] greg-g: link? [18:55:04] * greg-g looks [18:55:05] PROBLEM - check_mysql on lutetium is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [18:55:09] cuz if so then we should match but i've not heard this. [18:55:13] hmm that does sound reasonable I suppose [18:55:16] Should be the same shell name too [18:55:28] yep [18:55:35] # NOTE: To choose the UID for a new user please lookup [18:55:35] # the existing UID in (labs) LDAP and use that. [18:55:36] # currently you do this on formey, example: [18:55:47] top o' admins.pp [18:55:47] that rt is invalid on that ps RoanKattouw [18:55:56] ? [18:56:03] OK [18:56:07] I wonder how I look that up in LDAP [18:56:09] 7160 isnt valid ticket [18:56:22] Argh it's 7120 [18:56:43] greg-g: who reads the config file comments?!? [18:56:45] heh [18:56:52] so yea, its in file, if folks are doing that, it has some method of sanity [18:56:54] i'd do that. [18:56:55] (03CR) 10ArielGlenn: "note that this fails on hosts running hardy (dobson, mchenry, pdf2,3) because no salt. puppet is pretty broken on those boxes though, and" [operations/puppet] - 10https://gerrit.wikimedia.org/r/107831 (owner: 10Dzahn) [18:56:57] RoanKattouw: i stand corrected. [18:57:03] RoanKattouw: 'how' is the next line I didn't quote, line 11 [18:57:08] OK [18:57:13] # ldaplist -l passwd someuser [18:57:14] seems a lot more sane than 'find free uid and apply' [18:57:19] (03PS2) 10Matanya: Add esanders shell account [operations/puppet] - 10https://gerrit.wikimedia.org/r/120851 (owner: 10Catrope) [18:57:27] greg-g: nice catch, thanks [18:57:27] # advantages: no more duplicate UIDs that needed fixing, [18:57:27] # matching UID across production and labs, [18:57:27] # no need to grep|sort for the latest free UID anymore [18:57:27] # almost every user who gets prod. shell already has a [18:57:27] # labs user. if not, ask them nicely to make one first [18:57:29] blame [18:57:32] -3 [18:57:34] -e, that is [18:58:04] part of me says i should put that on wikitech [18:58:11] (03PS3) 10Catrope: Add esanders shell account [operations/puppet] - 10https://gerrit.wikimedia.org/r/120851 [18:58:14] * robh makes note to do so later. [18:59:19] OK, that should be all set up now [18:59:55] (03CR) 10RobH: [C: 04-1] "No problem with actual change, just -1 while the RT ticket has its pending processes." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120851 (owner: 10Catrope) [19:00:05] PROBLEM - check_mysql on lutetium is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [19:00:11] RoanKattouw_away: thx for making my week easier =] [19:00:16] lunchhorn! [19:01:09] wow, lunch is an hour early this week [19:01:45] (03CR) 10Reedy: [C: 032] Non Wikipedias to 1.23wmf19 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120830 (owner: 10Reedy) [19:01:50] the DST change is hard on our AU based opsen for meetings. [19:01:59] springle just likes 4am meetings. [19:01:59] totally confuses me [19:02:06] (03Merged) 10jenkins-bot: Non Wikipedias to 1.23wmf19 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120830 (owner: 10Reedy) [19:03:24] robh we use the lab uid for the shell prod uid, consistently now and if that was not documented and advertised widely then that is our bad (in part me since I knew) [19:03:58] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.23wmf19 [19:04:01] i never read the header of the file, heh [19:04:03] Logged the message, Master [19:04:18] i like that there is now some sensible way though [19:04:20] so +1 to change [19:04:42] and log one needs to doc it [19:05:05] PROBLEM - check_mysql on lutetium is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [19:06:02] !log reedy synchronized docroot and w [19:06:07] Logged the message, Master [19:07:28] wikidata looks good and the logs quiet [19:07:44] robh: frank just replied to me [19:07:56] removing his account and logging his mail [19:07:58] except https://bugzilla.wikimedia.org/show_bug.cgi?id=62547 which is not new and we will look at [19:08:11] (and can't reproduce, no reports of problems) [19:09:52] matanya: mind forwarding his reply into the rt ticket for me, audit trail =] rt 7117 [19:10:01] sure [19:10:05] PROBLEM - check_mysql on lutetium is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [19:10:13] thx =] [19:11:31] sent, i'm pushing a patch for this atm [19:15:05] PROBLEM - check_mysql on lutetium is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3067 [19:16:45] (03PS1) 10Matanya: access: remove fschulenburg [operations/puppet] - 10https://gerrit.wikimedia.org/r/120855 [19:16:50] robh: ^ [19:17:59] firefox + gerrit is killing me, i need to change default browser i guess [19:18:16] (03CR) 10RobH: [C: 032] access: remove fschulenburg [operations/puppet] - 10https://gerrit.wikimedia.org/r/120855 (owner: 10Matanya) [19:18:24] hurry up zuul! [19:20:04] robh: can you please verify mgrover is abent too ? [19:20:05] PROBLEM - check_mysql on lutetium is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3367 [19:21:49] will do, going to go eat my lunch now (its heated up) [19:21:52] so back in a bit! [19:22:05] frank's account removal patch is live on cluster [19:22:18] sure, bon apetite and thanks :) [19:24:18] * robh noms on leftovers, back in about 20-30 [19:25:05] RECOVERY - check_mysql on lutetium is OK: Uptime: 6389096 Threads: 3 Questions: 48257017 Slow queries: 155447 Opens: 44784 Flush tables: 2 Open tables: 63 Queries per second avg: 7.553 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [19:32:20] gerrit seems down [19:32:42] qchris_away: ^ [19:33:27] ok, just *really* slow, but not entirely down it seems [19:36:09] hoo: welcome to java [19:37:07] ... [19:37:09] Reedy: who would know why the purge CU script isn't on hume ? [19:37:34] No idea [19:37:46] I don't know who even added the lines to puppet [19:39:12] thanks [19:39:58] The script is there [19:39:59] -r-xr-xr-x 1 root root 323 Jan 9 2013 /usr/local/bin/purge-checkuser [19:47:10] mutante: ^ [19:47:35] I think it's called by another script [19:47:44] And that script may or may not have a cron attached [19:47:54] either way, it's a mess... [19:48:01] !log Gerrit 503 :-( [19:48:07] Logged the message, Master [19:48:11] It's currently being restarted [19:48:11] !log Gerrit back [19:48:12] matanya: i could not find a cron using it [19:48:16] Logged the message, Master [19:48:17] I am not sure why I bother logging that :D [19:48:18] sorry [19:48:21] !log restarted gerrit on ytterbium [19:48:27] Logged the message, Master [19:48:29] hoo: aude ^^ [19:48:31] oh, i thought you didn't find the file itself [19:48:45] matanya: all i looked for was cron jobs [19:49:17] it's a mess [19:49:29] do we really care what is on hume if hume is going away? [19:49:45] thanks, Reedy [19:49:47] we do, if we don't know what is going on [19:49:48] hah, it's the other way around [19:49:58] this is the blocker that keeps us from shutting it down [19:50:15] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC [19:50:36] the request is "run this script on mediawiki that has not been running since X because somebody removed it" [19:50:43] while i +1 the puppet code to add it [19:50:58] i don't want to add it without knowing why it was removed [19:55:15] !log pc1-3 - remove from puppet,salt,icinga,.. [19:55:21] Logged the message, Master [20:01:26] I am getting a fatal on Commons wiki for my watchlist [20:01:33] [c22c108a] 2014-03-25 20:00:30: Fatal exception of type MWException [20:02:37] (03CR) 10Dzahn: [C: 032] "Sean, thanks for shutting them down and merging those things. this part (install-server) isn't a problem. actually you just needed extra t" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120748 (owner: 10Springle) [20:03:34] bd808: is Raymond's error ^ in logstash? [20:04:12] * bd808 looks [20:04:46] (03CR) 10Dzahn: "paravoid: pc1-3 gone now" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120748 (owner: 10Springle) [20:06:29] (03CR) 10Dzahn: [C: 032] decom - remove ms5 [operations/dns] - 10https://gerrit.wikimedia.org/r/120705 (owner: 10Dzahn) [20:06:48] robla, Raymond_ : It's probably "PHP Fatal error: Call to a member function getValue() on a non-object in /usr/local/apache/common-local/php-1.23wmf19/extensions/Wikidata/extensions/Wikibase/client/includes/hooks/SpecialWatchlistQueryHandler.php on line 49" [20:07:01] !log DNS update - killing ms5 [20:07:06] Logged the message, Master [20:07:30] * bd808 blames aude and hoo [20:08:31] remembers a "just for beta" comment on that [20:10:19] seems like you found the reason [20:10:49] "$hideWikibase = $opts->getValue( 'hideWikibase');" in SpecialWatchlistQueryHandler::addWikibaseConditions [20:11:14] I'd say that $opts === null in this case and that's not expected by the code [20:11:40] matanya: feel like fixing 127058 path conflict? [20:12:12] i did git review -d on that, rebase origin/production.. and it's just "noop"? [20:13:51] There are other non-fatal exceptions for the same line as well "Invalid option hideWikibase" [20:16:19] Where do bugs against Wikibase go in bugzilla? [20:16:28] github? [20:16:49] bd808: it is an extension in mediawiki extensions [20:17:21] bd808: either WikidataClient or WikidataRepo [20:17:58] I see "client" in the path to the file so … got it [20:21:41] hey greg-g thanks for your email earlier. regarding a possible backport for gwtoolset on wmf20; i don't understand yet what that would mean. as far as i know the commit should be abel to be placed as it is on wmf20, but i must be missing something. [20:22:16] matanya: Can I bribe you to review https://gerrit.wikimedia.org/r/113755 ? :P [20:22:42] Raymond_: I filed https://bugzilla.wikimedia.org/show_bug.cgi?id=63087 about your likely error [20:22:59] bd808: thanks [20:23:27] (03CR) 10Anomie: [C: 04-1] Add contact pages for legal to testwiki (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119873 (owner: 10Reedy) [20:24:21] dan-nl: heya! so, I mis-stated and meant wmf19 (wmf19 is the branch that just went to Commons today). If you're fine with wmf20, ie waiting until next tuesday, all is fine. [20:25:02] greg-g: ah, next tuesday deploy is fine, thanks :) [20:25:06] (03PS2) 10Dzahn: access: remove giovanni [operations/puppet] - 10https://gerrit.wikimedia.org/r/120758 (owner: 10Matanya) [20:25:10] dan-nl: np! [20:25:13] sorry for the confusion! [20:25:21] no worries [20:25:21] * greg-g had that mixed up in his head all morning for some reason [20:28:52] !jenkins beta-recompile-math-texvc [20:28:58] (03CR) 10Dzahn: [C: 032] "confirmed by Dario on ticket" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120758 (owner: 10Matanya) [20:30:42] what are we doing w/ the lldpd service in prod?r [20:39:00] (03PS1) 10RobH: adding user gdubuc to stat1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/120868 [20:40:17] gotta undo all of andrew o's auditing ;] [20:40:22] (just kidding he is aware of this) [20:40:51] (03CR) 10RobH: [C: 032] adding user gdubuc to stat1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/120868 (owner: 10RobH) [20:41:40] Reedy: hi, any updates on https://gerrit.wikimedia.org/r/#/c/119990/ ? thx [20:53:30] yurikR: if you want to enforce it, it likely needs another redirect in apache config [21:02:25] mutante: not sure i understood. Which file/line are you refering? [21:04:09] yurikR: your comment asking if that enforces https or not [21:04:33] i think if that means you want it to enforce then we'd have to add Apache config to cluster [21:05:58] mutante: https://gerrit.wikimedia.org/r/#/c/119985/ ? [21:07:40] wikibase is causing exceptions and I'm aware of that [21:07:45] just sayin' [21:11:34] (03CR) 10Hashar: Add scap-recompile to puppet instead of wikimedia-task-appserver (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/109951 (owner: 10Reedy) [21:16:45] yurikR: no, line 1304 in https://gerrit.wikimedia.org/r/#/c/119990/1/wmf-config/InitialiseSettings.php [21:28:57] greg-g: FYI, I need to deploy another Wikidata fix tonight [21:29:25] hoo: what's up? [21:29:45] greg-g: I see a lot of these: 2014-03-25 21:03:13 mw1167 commonswiki: [25190252] /wiki/Special:Watchlist Exception from line 141 of /usr/local/apache/common-local/php-1.23wmf19/includes/FormOptions.php: Invalid option hideWikibase [21:29:53] in the exception logs and users are complaining [21:30:06] yuck, got the fix ready? [21:30:13] the fix is easy, it just changes the order we check parameters [21:30:32] Yeah, I got the fix, but waiting for CR [21:32:53] * greg-g nods [21:32:53] https://bugzilla.wikimedia.org/show_bug.cgi?id=63087 is the bug [21:40:08] greg-g: Sorry, still waiting for CR [21:40:15] and there it is :P [21:40:24] coincidence, I guess :D [21:56:29] (03CR) 10Matanya: "please keep the diff review clean from tabs etc, for easier reviewing." [operations/puppet] - 10https://gerrit.wikimedia.org/r/113755 (owner: 10Hoo man) [21:58:31] greg-g: I'm ready [22:01:25] Flow deploy starting [22:01:37] uh [22:02:04] hoo: yeah, flow's from 2-4/21-23 :) [22:02:18] Ok [22:02:46] https://commons.wikimedia.org/wiki/Commons:Village_pump#Fatal_exception_of_type_MWException_when_trying_to_view_Special:Watchlist [22:02:48] they'll probably be quick though, but the SWAT deploy should be fine for this [22:02:53] so I'd really like to shoot this out [22:03:42] hoo, fine by me, whatever "this" is. ebernhardson have you started? [22:05:01] (03CR) 10Spage: [C: 032] Enable Flow on Compact Personal Bar BF talk page [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120833 (owner: 10Spage) [22:05:15] (03Merged) 10jenkins-bot: Enable Flow on Compact Personal Bar BF talk page [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120833 (owner: 10Spage) [22:07:23] !log spage updated /a/common to {{Gerrit|I7ca3051f6}}: Enable Flow on Compact Personal Bar BF talk page [22:07:28] Logged the message, Master [22:07:56] wow, new logging on tin! [22:08:53] heh... the git hook is there for quite some time now... not sure why it's not firing all the time, though [22:09:35] !log spage synchronized wmf-config/InitialiseSettings.php 'Enable Flow on mediawiki.org Compact Personal Bar BF talk page' [22:09:41] Logged the message, Master [22:10:07] quiddity: the Flow has spiced [22:10:25] !log Reloading Zuul to deploy Ib2abe3a000300 [22:10:31] Logged the message, Master [22:12:51] ty [22:12:58] spagewmf: Are you done? [22:13:54] hoo: no, patch to 1.23wmf19/extensions/Flow coming. No scap, should take 15 minutes? [22:14:23] spagewmf: Scap still takes a few minutes [22:14:27] not sure what you mean [22:14:34] I don't need a scap, just sync a dir [22:15:09] hoo, I mean the Flow fix won't require a scap. If you want to do a sync-dir of your own right now, that's fine by me if greg-g OKs [22:15:30] Oh right :) [22:15:59] ah, yeah, go ahead and do that after spagewmf is done, hoo [22:16:38] ok, doing then... let's wait for jenkins [22:16:55] greg-g: hoo can go now, I'm not ready [22:17:09] * greg-g nods [22:17:15] gotcha, now we're all on the same page (sorry) [22:18:28] hoo's on first [22:18:37] bada ching [22:18:53] I've waited years to say that! [22:19:07] :) [22:19:17] :D [22:22:25] !log hoo synchronized php-1.23wmf19/extensions/Wikidata/ 'Update Wikidata to fix an exception within WikibaseClient (bug 63087)' [22:22:30] Logged the message, Master [22:22:38] works: https://commons.wikimedia.org/wiki/Special:Watchlist?hideWikibase=1&enhanced=true \o/ [22:22:48] I'm done [22:23:29] unless further catastrophes arise, but let's hope not [22:26:13] * greg-g knocks on wood [22:26:35] hoo, Watchlist/RecentChanges/Log/IRC feed integration are the bane of Flow too [22:27:00] doesn't surprise me [22:29:11] (03PS1) 10Dzahn: add missing system roles [operations/puppet] - 10https://gerrit.wikimedia.org/r/120956 [22:29:54] greg-g: thanks for your coordination affords, the error count graph really looks silent now :) [22:31:47] (03PS2) 10Dzahn: add missing system roles [operations/puppet] - 10https://gerrit.wikimedia.org/r/120956 [22:32:35] hoo: woohoo [22:36:03] (03CR) 10Rush: [C: 031] "I feel at last qualified to say something is ok. benign changes is where I'm at my best." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120956 (owner: 10Dzahn) [22:40:27] !log spage synchronized php-1.23wmf19/extensions/Flow/Hooks.php 'Fix Flow notification preference in 1.23wmf19' [22:40:33] Logged the message, Master [22:43:08] greg-g: we are done, thanks bye y'all [22:43:36] :) [22:44:03] (03CR) 10RobH: "The review on refactoring that file is scheduled for a couple opsen tomorrow. I'll check back in with them late tomorrow or early Thursda" [operations/puppet] - 10https://gerrit.wikimedia.org/r/116019 (owner: 10Hoo man) [22:44:33] thanks, robh [22:45:19] quite welcome, i figure even 'i dont have an answer but expect one at X' is better than silence ;] [22:45:52] I'm happy to get some review already [22:50:32] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC [23:02:05] (03CR) 10Matanya: "I'd prefer if you can break it to multi-line like fenari." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120956 (owner: 10Dzahn) [23:05:40] (03PS2) 10Rush: shell for proposed admin module [operations/puppet] - 10https://gerrit.wikimedia.org/r/120724 [23:09:14] greg-g: Who's doing the SWAT today? [23:09:23] :) [23:09:27] Is spagewmf just doing his own thing and ignoring calendared deployments? [23:09:42] ? [23:10:14] he moved a flow thing from the SWAT window to the Flow window, which is ok [23:10:41] Oh, sorry [23:10:45] * RoanKattouw needs to look at timestamps [23:11:15] :) [23:11:26] I assume you can do the math and ve one? :) [23:11:36] Sure [23:11:52] Although I would appreciate it if there was a time, even once, where I didn't end up deploying my own SWAT changes [23:12:05] This goes back to having to formalize who's on SWAT duty on what day [23:12:11] I'm not even *normally in the office* on Tuesdays [23:12:16] So I kind of feel like I'm picking up all the slack here [23:13:43] fair. [23:14:23] Roan, I'll be more involved in the future. [23:15:25] Plus, I've done a few already. [23:15:47] I think we're about even, no? [23:16:34] That's possible [23:16:42] I guess I just don't notice the days I don't have a commit in [23:16:56] :) [23:17:01] Either way, formalizing who's on duty when is an easy solution [23:17:09] It makes fairness verifiable [23:17:15] also, thanks for removing your name from today's SWAT window as you're normally not here (that helps me) [23:17:41] I didn't remove my name [23:17:43] It was never there [23:17:46] (I hope) [23:18:23] oh, that's probably right :) [23:22:31] OK doing the math change now [23:27:39] !log catrope synchronized php-1.23wmf19/extensions/Math/modules/VisualEditor/ve.ui.MWMathInspector.js 'Fix VE math inspector title' [23:27:45] Logged the message, Master [23:27:56] greg-g: All dne [23:29:11] rock, thanks [23:37:14] (03PS1) 10Jkrauska: Remove jdavis admin user. (no longer wmf employee, long ago deactivated/disabled) [operations/puppet] - 10https://gerrit.wikimedia.org/r/120965 [23:39:19] (03CR) 10Reedy: [C: 04-1] "IIRC we don't ever remove accounts from admins.pp" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120965 (owner: 10Jkrauska) [23:40:11] (03CR) 10Matanya: [C: 04-1] "what reedy said. + we don't want to reuse uid's." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120965 (owner: 10Jkrauska) [23:40:42] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:41:03] (03CR) 10Jkrauska: "Hrm: Well what about the reference in site.pp?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120965 (owner: 10Jkrauska) [23:42:36] (03CR) 10Matanya: "no issue with site,pp as long as the key is set to absent, we are ok." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120965 (owner: 10Jkrauska) [23:42:42] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 246193 bytes in 7.858 second response time [23:44:32] (03CR) 10Jkrauska: "Seemed beneficial to clean up the old record to me, but happy to cancel." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120965 (owner: 10Jkrauska) [23:46:23] (03Abandoned) 10Jkrauska: Remove jdavis admin user. (no longer wmf employee, long ago deactivated/disabled) [operations/puppet] - 10https://gerrit.wikimedia.org/r/120965 (owner: 10Jkrauska) [23:50:42] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:42] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 242668 bytes in 7.730 second response time