[00:00:59] !log updated the paypal IPN listener on aluminium to r1450 [00:01:03] Logged the message, Master [00:07:26] New patchset: Asher; "pretty graphs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3002 [00:07:37] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3002 [00:08:17] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3002 [00:08:20] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3002 [00:11:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:19:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 3.963 seconds [00:26:38] !log preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/opensearch.min.js 'fixes to code push' [00:26:58] !log preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/opensearch.js 'fixes to code push' [00:27:18] !log preilly synchronized php-1.19/extensions/MobileFrontend/templates/ApplicationTemplate.php 'fixes to code push' [00:27:53] Logged the message, Master [00:27:56] Logged the message, Master [00:27:59] Logged the message, Master [00:32:27] !log preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/beta_opensearch.min.js 'fixes to code push' [00:32:30] Logged the message, Master [00:32:56] !log preilly synchronized php-1.19/extensions/MobileFrontend/javascripts/beta_opensearch.js 'fixes to code push' [00:32:57] Logged the message, Master [00:38:44] !log preilly synchronized wmf-config/InitialiseSettings.php 'add ZeroRatedMobileAccess extension to mywiki' [00:38:48] Logged the message, Master [00:45:14] New patchset: Dzahn; "add nagios::monitor::snmp to spence" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3003 [00:45:25] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3003 [00:46:22] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3003 [00:46:25] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3003 [00:54:04] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:55:07] PROBLEM - Puppet freshness on db1022 is CRITICAL: Puppet has not run in the last 10 hours [00:59:55] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 1.306 seconds [01:24:19] !log tstarling synchronized wmf-config/InitialiseSettings.php '$wgBug34832TransitionalRollback=false on wikis other than zhwiki' [01:26:56] !log aaron synchronized php-1.19/includes/filerepo 'deployed r113443' [01:29:25] New review: Lcarr; "I actually did this because nagios/icinga was breaking since no routers group was showing up/being a..." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/2956 [01:33:58] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:41] !log awjrichards synchronizing Wikimedia installation... : Fixing show/hide button translations in mobile view, r113445 [01:39:30] sync done. [01:39:49] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 4.685 seconds [01:45:16] !log awjrichards synchronized php-1.19/extensions/MobileFrontend/templates/ApplicationTemplate.php 'r113449' [01:46:18] !log tstarling synchronized php-1.19/includes/Skin.php [01:53:27] PROBLEM - udp2log log age on oxygen is CRITICAL: NRPE: Command check_udp2log_log not defined [01:53:45] PROBLEM - udp2log processes on oxygen is CRITICAL: NRPE: Command check_udp2log_procs not defined [02:00:37] !log LocalisationUpdate failed: SVN update of extensions failed [02:02:16] wtf [02:02:36] U /var/lib/l10nupdate/trunk/extensions/SideBarMenu/includes/MenuItem.php [02:02:38] svn: Unrecognized line ending style [02:02:40] Updating extensions FAILED. [02:02:42] Whoa [02:02:52] Did the person committing SideBarMenu just break a production service? [02:03:24] a linefeed o' doom? [02:03:45] https://www.mediawiki.org/wiki/Special:Code/MediaWiki/113362#c31969 [02:03:51] Trying the remedy described there [02:04:31] That seems to work [02:04:37] !log Rerunning LocalisationUpdate manually [02:10:55] svn:eol-style = native *.eot = svn:mime-type=application/vnd.ms-fontobject *.gif = svn:mime-type=image/gif *.ico = svn:mime-type=image/vnd.microsoft.icon *.inc = svn:eol-style=native *.info = svn:eol-style=native *.install=svn:eol-style=native *.java = svn:eol-style=native *.jpeg = svn:mime-type=image/jpeg *.jpg = svn:mime-type=image/jpeg *.js = svn:eol-style=native *.jsp = svn:eol-style=native *.module = svn:eol-style=native *.php5 [02:10:55] = svn:eol-style=native *.php = svn:eol-style=native *.pl = svn:eol-style=native *.pm = svn:eol-style=native *.png = svn:mime-type=image/png *.py = svn:eol-style=native *.sh = svn:eol-style=native *.sql = svn:eol-style=native *.svg = svn:mime-type=image/svg+xml *.tpl = svn:eol-style=native *.ttf = svn:mime-type=application/x-font-ttf *.txt = svn:eol-style=native *.woff = svn:mime-type=application/x-woff *.xcf = svn:mime-type=image/x-x [02:11:06] cf Makefile = svn:eol-style=native SConscript = svn:eol-style=native SConstruct = svn:eol-style=native USERINFO = svn:eol-style=native [02:11:06] oops, didn't realise it was quite that long [02:11:14] thanks xchat for breaking that line instead of truncating it [02:13:17] * AaronSchulz waits for TimStarling to boot himself ;) [02:16:15] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:18:48] PROBLEM - Puppet freshness on db1033 is CRITICAL: Puppet has not run in the last 10 hours [02:20:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 8.429 seconds [02:21:05] !log LocalisationUpdate completed (1.19) at Fri Mar 9 02:21:05 UTC 2012 [02:23:49] hi - on nl: there is a new feature what is called in dutch 'terugkoppelingslogboek' to give one's opinion about wikipedia. this opinion is mentioned in the RC's. I protest this strongely to mention the opinion in the RC's because it opresses my privacy rights [02:25:30] where can i comply and who can clean this edit of my in this log? - not thats is that interesting but for the record [02:28:32] !mod [02:28:57] aleichem: That would be what we call MoodBar in English [02:29:37] ok - but if i said - werdna is a little strange in the moodbar - i dont want that in the RC's [02:29:57] I'm not sure what you mean by "it opresses your privacy rights" [02:30:04] why do you think that? [02:30:18] you ask me what is my opinion of A [02:30:33] i say it is shit because ..... [02:30:55] i dont this ...... in the RC's [02:31:01] +want [02:31:50] in a file for someone - ok - but for the whole world? [02:32:09] your username is aleichem? [02:32:35] yes and alot others - just mentioned something as michiel043 on nl: [02:33:00] At the bottom of the little pop up that comes up, it would have said this: Door op te slaan gaat u akkoord met transparantie onder deze voorwaarden -- that means your feedback will be transparent and public [02:33:03] it's not just this one line - its just the point - do you see the point or not? [02:33:13] "Users with the right moodbar-view can view feedback sent through the interface on Special:MoodBar" [02:33:32] did you see that? [02:34:01] yes = f--- off - like - whats that called on the bottom of every page - you have to click this to agree with our conditions [02:34:17] It's not particularly onerous [02:34:38] it's not like we said "You agree to our terms and conditions" and the fact that it was made public was buried in clause 27 [02:34:38] and i didnt seee i - and even when i deed see it - i didnt expected it to be in the RC's [02:35:00] The thing specifically says "transparency" -- surely if you were posting something you'd be worried about other people seeing, you would pause for a second [02:35:01] so you want *all* of these to disappear? https://nl.wikipedia.org/w/index.php?title=Speciaal%3ALogboeken&type=moodbar [02:35:25] no - maybe just - xxx made an entry on the moodbar? [02:35:31] look, I can remove your specific feedback if there's something particularly private in there, but you're not going to convince anybody to turn the feature off because you didn't read what you were doing. [02:35:33] i dunno who reads this log [02:35:51] thats not the point [02:36:01] then what is? [02:36:05] I don't see you in there [02:36:13] in where? [02:36:18] you posted a comment publicly. Is there something we could do to make it more clear that it's public? [02:36:27] in https://nl.wikipedia.org/w/index.php?title=Speciaal%3ALogboeken&type=moodbar [02:36:32] http://nl.wikipedia.org/wiki/Speciaal:Logboeken/moodbar [02:36:58] i mentioned a name [02:37:05] yeah, I'm not sure if it's a proxy, but it's got a ton of places complaining about spam from it [02:37:08] er wrong channel [02:37:26] ok [02:37:37] as michiel043 [02:38:31] yes I see it [02:38:39] but the question is : where do i complain about this? [02:39:22] Well, I'm the lead developer on that project, but that won't help you very much. [02:39:38] ok I suppressed the edit summary for you [02:39:39] You haven't really explained what your problem is yet [02:39:51] no - you're a developer - but who says you too do things this way? [02:40:21] I can't help you if you don't tell me what your problem is [02:40:37] the problem is the privacy [02:40:38] 9 mrt 2012 03:08 Michiel043 (Overleg | bijdragen) MoodBar-terugkoppeling Speciaal:DashboardTerugkoppeling/569 (bewerkingssamenvatting verwijderd) [02:40:50] yep ok timS - tnx - [02:40:57] I'll leave you to battle with werdna about the wider issues but I hope that helps [02:41:13] no - i dont know who werdna is? [02:41:14] aleichem: You didn't realise that you were making a public comment when you submitted feedback? [02:41:21] nope [02:41:22] looks like this would be a place for discussion about design issue, no? http://www.mediawiki.org/wiki/Talk:MoodBar/Design [02:41:23] aleichem: that would be me, the person with the nickname 'werdna' [02:41:38] aleichem: What could we do to make that clearer to you? [02:41:39] 'public' - like in with some mods or other nerds [02:42:05] Maybe the problem is the translation. In English 'transparency' means public [02:42:10] but not 'public' like in with al eskimos [02:42:37] maybe it doesn't have that implication in Dutch, I don't know. I'm trying to learn some dutch, but all I have so far is "Wil jouw een bier" :) [02:42:39] RECOVERY - Puppet freshness on mw1010 is OK: puppet ran at Fri Mar 9 02:42:28 UTC 2012 [02:42:41] no - it's not the translation - its the system [02:43:04] where are delphine and the other old guys? [02:43:09] still around? [02:43:17] Delphine's around, but did you hear she had a baby? [02:43:20] or two now? [02:43:26] 3 i thought [02:43:28] I don't know, anyway I think she's looking after the baby [02:43:32] some number of babies [02:43:47] I think a lot of people don't hang around on IRC as much [02:44:09] anyway, the intention of the system is to be in line with Wikipedia's values. That is, everything is transparent and public [02:44:26] If you don't like your comments being public, then you should submit them some other way I guess [02:44:31] the question is [02:44:36] how can we avoid surprising people with the fact that their comments will be public [02:44:51] nope - its a 'valkuil' the word in dutch - lets see whats the translation [02:44:53] I don't want people like you to not realise that what you're posting is public and then finding out once everyone can see it [02:45:11] this one? http://nl.wikipedia.org/wiki/Valkuil_%28psychologie%29 [02:45:30] i'm not a first day wikipedian [02:45:36] yep [02:46:03] I didn't suspect you were [02:46:18] pitfall [02:46:21] is the word in english [02:46:26] a trap, I guess, you might say [02:46:47] so weird - i now made a new username [02:46:55] and everything is so different [02:47:03] can I ask you then, what could we have done to make it obvious to you that the comment was public? [02:48:26] i don't think you have to tell me it is public - because it is not - it is a message to wikimedia - nad nobody has nothing to do with it [02:48:49] well, you know that wikimedia doesn't have a great big army of staff to answer these things, right? [02:49:07] we can't look into this stuff without using volunteers to process them [02:49:29] ok - the chapter here has some chap to analyse the stuff [02:50:25] but thats something entirely different than to publish every answer on a question of a individual [02:50:26] not nearly enough for the number of comments we get [02:50:41] what if i called you [02:51:06] i want to ask you for a survey - do you like cocoa cola? [02:51:38] and the next day i put a sign in your front yard -> hey this guy like cola cola [02:52:06] /win 4 [02:52:12] sry [02:52:23] What if I asked you "Do you want to tell me if you like coca cola? We're planning on putting signs up outside people's houses saying they like coca cola." [02:52:37] the issue is not the sign, it's the violation of expectations [02:53:48] and the basic expectation of a human is trust, i suppose? [02:55:50] but maybe we could talk about this whole (night right here) , who or where is the right entrance for this subject? whe developed the idea? and not executed it like tomas? [02:56:09] something's up with the edit window buttons [02:56:14] half of them don't show up immediately [02:56:56] like I go to edit a page and only half of the buttons show up [02:57:21] and the worst thing is that some guys in north-korea like you messes in my wiki [02:58:15] I'm sorry I don't believe I give a fuck [03:00:47] this is what my edit options look when first editing http://i42.tinypic.com/dqnk1e.jpg [03:01:02] and this is after I refresh the edit page http://i40.tinypic.com/11ca0co.jpg [03:01:06] what's going on [03:01:16] Ryulong: what browser? [03:01:30] latest version of firefox [03:01:35] also monobook [03:01:41] have you got firebug? [03:01:46] no [03:02:37] btw aleichem was talking to werdna [03:02:44] yep [03:02:50] oh [03:02:54] then I apologize [03:03:18] no need [03:03:25] TimStarling: are my js modifications screwing everything up again [03:03:38] or is it some sort of lost code in Aaron's old scripts that need fixing [03:04:14] I can reproduce it [03:05:09] it's probably a monobook problem [03:05:44] hmm, maybe not, just hit refresh on vector and saw it there too [03:06:22] why do we need to fix things that aren't broken [03:07:51] because when you fix things that are broken, you inevitably need to touch some code used by things that aren't broken [03:08:10] @werdna - and - when the idea of the log is so great and it needs to be seen for everybody - do you analyze it? who does? you ask me for my opinion and you throw it in the wastepaperbasket? [03:08:32] the shorter toolbar seems to be the same as a default MW install, I wonder where the extra buttons come from [03:10:09] I guess it is mwCustomEditButtons from https://en.wikipedia.org/wiki/MediaWiki:Common.js/edit.js [03:20:43] I can see why it doesn't work [03:22:02] aleichem: there are volunteers who do the work on that [03:23:03] I think you should talk to Steven or Maryana in the community department, or Oliver Keyes, the community liaison for the tech department [03:23:13] but I will warn you, they will say exactly what I said [03:25:03] i just anserwerd my own feedback and it appeard on my talkpage - thats neat [03:25:29] thats neat like as 'thats great' [03:25:42] PROBLEM - check_minfraud_primary on payments4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:26:19] @werdna - privacy issues are different over there than over here [03:26:34] ? [03:26:36] over where? [03:26:46] you usa? uk? [03:27:07] australia [03:27:10] ok [03:27:28] I understand that there are cultural differences. But you'll be interested to know we have a fairly international team working on this [03:27:33] and we're tying to make it work for everyone [03:28:29] but in nl: we dont like this kund of things (at least i do not). we have prizes for the least privacy friendly companys etc [03:29:21] prizes were given last night lets find the link for you [03:30:04] https://www.bof.nl/ bof = bits of freedom [03:30:12] RECOVERY - check_minfraud_primary on payments4 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 118 bytes in 0.198 second response time [03:31:24] https://www.bof.nl/2012/03/08/winnaars-big-brother-awards-bekend/ prize winner : the police - hey came and accepted the award [03:46:25] New patchset: Dzahn; "nagios,snmp,iptables: allow private IP ranges, but not labs 10.4.0.0/24 and 10.4.16.0/24" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3004 [03:46:36] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3004 [04:15:01] !log switching php symlink over to point to php-1.19 instead of php-1.18 [04:16:30] !log also switching over wmf-deployment symlink [04:16:49] no bot? [04:18:17] !log restarted morebots [04:18:21] Logged the message, Master [04:18:39] !log switched php and wmf-deployment symlinks over to php-1.19 instead of php-1.18 [04:18:42] Logged the message, Master [04:30:23] TimStarling: is there anything still using the php symlink... I remember a few gadgets for images but I think they got updated to commons [04:30:31] *?... [04:31:09] yes, ExtensionDistributor for instance [04:46:05] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3004 [04:46:08] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3004 [04:58:34] New patchset: Dzahn; "can't use Iptables_add_service[lo_all] etc. twice, duplicate definition, so merging nsca and nrpe iptables rules into one nagios::monitoring::firewall class instead of repeating everything all over with different names, also disallow nsca from labs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3005 [04:58:46] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3005 [05:01:16] New patchset: Dzahn; "can't use Iptables_add_service[lo_all] etc. twice, duplicate definition, so merging nsca and nrpe iptables rules into one nagios::monitoring::firewall class instead of repeating everything all over with different names, also disallow nsca from labs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3005 [05:01:28] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3005 [05:02:53] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3005 [05:02:56] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3005 [05:07:50] New patchset: Dzahn; "iptables_add_exec{ "${hostname} would also be a duplicate .. we need 2 of them though, right?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3006 [05:08:01] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3006 [05:08:37] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3006 [05:08:40] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3006 [05:14:14] Can I get some help with bugzilla, please? [05:22:42] PROBLEM - Puppet freshness on mw1020 is CRITICAL: Puppet has not run in the last 10 hours [05:22:42] PROBLEM - Puppet freshness on mw1110 is CRITICAL: Puppet has not run in the last 10 hours [06:07:22] New patchset: Ryan Lane; "Up the version of user-management scripts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3007 [06:07:33] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3007 [06:07:55] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3007 [06:07:57] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3007 [06:09:27] New patchset: Ryan Lane; "Adding manage-volumes script link" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3008 [06:09:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3008 [06:09:50] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3008 [06:09:53] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3008 [06:13:10] !log running svn cleanup on extdist trunk [06:13:13] Logged the message, Master [06:35:49] New patchset: Dzahn; "typo: snmaptrap :p we aren't portscanning here .. and let's remove all "-" just in case" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3009 [06:36:00] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3009 [06:38:38] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3009 [06:38:41] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3009 [06:53:43] PROBLEM - Puppet freshness on db1004 is CRITICAL: Puppet has not run in the last 10 hours [07:02:52] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [07:02:52] New patchset: Ryan Lane; "Add manage-volume cron on labstore2, and add an ircecho bot for the logfile" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3010 [07:03:04] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3010 [07:03:16] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3010 [07:03:18] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3010 [07:04:58] New patchset: Ryan Lane; "Placing the script properly on labstore2" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3011 [07:05:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3011 [07:05:14] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3011 [07:05:17] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3011 [07:11:52] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [07:11:52] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [07:28:12] New patchset: Dzahn; "nagios monitoring for mw profiling collector and graphite (RT-2367)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3012 [07:28:24] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3012 [07:30:20] New patchset: Dzahn; "nagios monitoring for mw profiling collector and graphite (RT-2367)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3012 [07:30:31] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3012 [07:32:17] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3012 [07:32:20] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3012 [07:36:45] PROBLEM - Disk space on ms1004 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=95%): /var/lib/ureadahead/debugfs 0 MB (0% inode=95%): [07:37:48] PROBLEM - DPKG on snapshot1 is CRITICAL: Connection refused by host [07:38:15] PROBLEM - Disk space on snapshot1 is CRITICAL: Connection refused by host [07:38:42] PROBLEM - DPKG on virt1 is CRITICAL: Connection refused by host [07:38:42] PROBLEM - RAID on ms5 is CRITICAL: Connection refused by host [07:39:00] PROBLEM - RAID on snapshot1 is CRITICAL: Connection refused by host [07:39:09] PROBLEM - DPKG on cp1010 is CRITICAL: Connection refused by host [07:39:09] PROBLEM - RAID on mw8 is CRITICAL: Connection refused by host [07:39:18] PROBLEM - MySQL disk space on db25 is CRITICAL: Connection refused by host [07:39:18] PROBLEM - DPKG on db25 is CRITICAL: Connection refused by host [07:39:27] PROBLEM - Full LVS Snapshot on db25 is CRITICAL: Connection refused by host [07:39:27] PROBLEM - RAID on mw1051 is CRITICAL: Connection refused by host [07:39:36] PROBLEM - Disk space on ms5 is CRITICAL: Connection refused by host [07:39:45] PROBLEM - Disk space on cp1010 is CRITICAL: Connection refused by host [07:39:45] PROBLEM - MySQL Idle Transactions on db12 is CRITICAL: Connection refused by host [07:39:45] PROBLEM - RAID on mw12 is CRITICAL: Connection refused by host [07:39:45] PROBLEM - MySQL Idle Transactions on db25 is CRITICAL: Connection refused by host [07:39:45] PROBLEM - DPKG on ms5 is CRITICAL: Connection refused by host [07:39:54] PROBLEM - DPKG on mw8 is CRITICAL: Connection refused by host [07:39:54] PROBLEM - MySQL Recent Restart on db12 is CRITICAL: Connection refused by host [07:40:03] PROBLEM - MySQL Recent Restart on db25 is CRITICAL: Connection refused by host [07:40:12] PROBLEM - DPKG on mw1051 is CRITICAL: Connection refused by host [07:40:21] PROBLEM - MySQL Replication Heartbeat on db1018 is CRITICAL: Connection refused by host [07:40:21] PROBLEM - RAID on srv255 is CRITICAL: Connection refused by host [07:40:21] PROBLEM - Disk space on virt1 is CRITICAL: Connection refused by host [07:40:21] PROBLEM - MySQL Replication Heartbeat on db25 is CRITICAL: Connection refused by host [07:40:30] PROBLEM - DPKG on db1018 is CRITICAL: Connection refused by host [07:40:30] PROBLEM - Disk space on mw8 is CRITICAL: Connection refused by host [07:40:30] PROBLEM - MySQL Slave Delay on db1018 is CRITICAL: Connection refused by host [07:40:39] PROBLEM - MySQL Replication Heartbeat on db12 is CRITICAL: Connection refused by host [07:40:39] PROBLEM - Disk space on db1018 is CRITICAL: Connection refused by host [07:40:39] PROBLEM - MySQL Slave Running on db1018 is CRITICAL: Connection refused by host [07:40:39] PROBLEM - Disk space on mw1051 is CRITICAL: Connection refused by host [07:40:39] PROBLEM - RAID on cp1010 is CRITICAL: Connection refused by host [07:40:48] PROBLEM - MySQL Slave Delay on db12 is CRITICAL: Connection refused by host [07:40:48] PROBLEM - MySQL Slave Delay on db25 is CRITICAL: Connection refused by host [07:40:48] PROBLEM - mysqld processes on db25 is CRITICAL: Connection refused by host [07:40:48] PROBLEM - mysqld processes on db1018 is CRITICAL: Connection refused by host [07:40:48] PROBLEM - DPKG on mw12 is CRITICAL: Connection refused by host [07:40:57] PROBLEM - Disk space on mw12 is CRITICAL: Connection refused by host [07:40:57] PROBLEM - Disk space on db25 is CRITICAL: Connection refused by host [07:40:57] PROBLEM - RAID on db25 is CRITICAL: Connection refused by host [07:40:57] PROBLEM - RAID on mw1127 is CRITICAL: Connection refused by host [07:40:57] PROBLEM - MySQL Slave Running on db25 is CRITICAL: Connection refused by host [07:40:58] PROBLEM - Full LVS Snapshot on db1018 is CRITICAL: Connection refused by host [07:40:58] PROBLEM - RAID on db12 is CRITICAL: Connection refused by host [07:41:06] PROBLEM - DPKG on srv255 is CRITICAL: Connection refused by host [07:41:06] PROBLEM - Disk space on db12 is CRITICAL: Connection refused by host [07:41:06] PROBLEM - MySQL Slave Running on db12 is CRITICAL: Connection refused by host [07:41:06] PROBLEM - Disk space on srv221 is CRITICAL: Connection refused by host [07:41:15] PROBLEM - mysqld processes on db12 is CRITICAL: Connection refused by host [07:41:15] PROBLEM - DPKG on db12 is CRITICAL: Connection refused by host [07:41:15] PROBLEM - Full LVS Snapshot on db12 is CRITICAL: Connection refused by host [07:41:24] PROBLEM - DPKG on mw1083 is CRITICAL: Connection refused by host [07:41:24] PROBLEM - DPKG on srv221 is CRITICAL: Connection refused by host [07:41:24] PROBLEM - MySQL Idle Transactions on db1018 is CRITICAL: Connection refused by host [07:41:24] PROBLEM - Disk space on srv255 is CRITICAL: Connection refused by host [07:41:24] PROBLEM - MySQL disk space on db12 is CRITICAL: Connection refused by host [07:41:25] PROBLEM - MySQL disk space on db1018 is CRITICAL: Connection refused by host [07:41:25] PROBLEM - DPKG on mw1127 is CRITICAL: Connection refused by host [07:48:10] PROBLEM - MySQL Recent Restart on db1018 is CRITICAL: Connection refused by host [07:48:10] PROBLEM - DPKG on mw43 is CRITICAL: Connection refused by host [07:48:10] PROBLEM - Disk space on mw1127 is CRITICAL: Connection refused by host [07:48:19] PROBLEM - Full LVS Snapshot on db54 is CRITICAL: Connection refused by host [07:48:28] PROBLEM - Full LVS Snapshot on db22 is CRITICAL: Connection refused by host [07:48:28] PROBLEM - MySQL disk space on db22 is CRITICAL: Connection refused by host [07:48:28] PROBLEM - MySQL disk space on db32 is CRITICAL: Connection refused by host [07:48:28] PROBLEM - Full LVS Snapshot on db32 is CRITICAL: Connection refused by host [07:48:37] PROBLEM - RAID on mw43 is CRITICAL: Connection refused by host [08:01:52] PROBLEM - RAID on cp1016 is CRITICAL: Connection refused by host [08:01:52] PROBLEM - Disk space on cp1007 is CRITICAL: Connection refused by host [08:01:52] PROBLEM - DPKG on cp1008 is CRITICAL: Connection refused by host [08:01:52] PROBLEM - Disk space on mw1106 is CRITICAL: Connection refused by host [08:01:52] PROBLEM - Disk space on srv259 is CRITICAL: Connection refused by host [08:01:53] PROBLEM - DPKG on cp1041 is CRITICAL: Connection refused by host [08:01:53] PROBLEM - Disk space on srv203 is CRITICAL: Connection refused by host [08:02:01] PROBLEM - Disk space on mw1084 is CRITICAL: Connection refused by host [08:02:10] PROBLEM - RAID on srv233 is CRITICAL: Connection refused by host [08:02:31] PROBLEM - RAID on cp1001 is CRITICAL: Connection refused by host [08:02:31] PROBLEM - MySQL Slave Running on db43 is CRITICAL: Connection refused by host [08:02:31] PROBLEM - MySQL Recent Restart on db52 is CRITICAL: Connection refused by host [08:02:31] PROBLEM - Disk space on db43 is CRITICAL: Connection refused by host [08:02:31] PROBLEM - RAID on db52 is CRITICAL: Connection refused by host [08:02:37] PROBLEM - DPKG on nfs1 is CRITICAL: Connection refused by host [08:02:37] PROBLEM - MySQL Idle Transactions on db24 is CRITICAL: Connection refused by host [08:02:37] PROBLEM - mailman on sodium is CRITICAL: Connection refused by host [08:02:37] PROBLEM - Disk space on sodium is CRITICAL: Connection refused by host [08:02:37] PROBLEM - DPKG on srv233 is CRITICAL: Connection refused by host [08:02:46] PROBLEM - Disk space on srv275 is CRITICAL: Connection refused by host [08:02:46] PROBLEM - RAID on nfs1 is CRITICAL: Connection refused by host [08:02:46] PROBLEM - RAID on mw1084 is CRITICAL: Connection refused by host [08:02:46] PROBLEM - MySQL Recent Restart on db24 is CRITICAL: Connection refused by host [08:02:46] PROBLEM - RAID on db24 is CRITICAL: Connection refused by host [08:02:47] PROBLEM - MySQL Replication Heartbeat on db52 is CRITICAL: Connection refused by host [08:36:16] RECOVERY - RAID on cp1002 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [09:02:38] RECOVERY - RAID on cp1015 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [09:03:32] RECOVERY - DPKG on cp1015 is OK: All packages OK [09:03:41] RECOVERY - Disk space on cp1015 is OK: DISK OK [10:28:09] Very slow this morning, You guys fixing something? [10:30:10] Speed seems fineish for me, But I may be hitting the other cluster group [10:56:23] PROBLEM - Puppet freshness on db1022 is CRITICAL: Puppet has not run in the last 10 hours [11:19:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:19:51] New patchset: Mark Bergsma; "Create a Varnish VCL consisting of all Wikimedia networks" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3013 [11:20:02] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3013 [11:20:40] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3013 [11:20:43] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3013 [11:23:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 1.425 seconds [11:25:04] New patchset: Mark Bergsma; "Add missing ;" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3014 [11:25:15] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3014 [11:25:22] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3014 [11:25:25] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3014 [11:29:05] PROBLEM - Varnish HTTP bits on sq67 is CRITICAL: Connection refused [11:29:53] New patchset: Mark Bergsma; "Hate Varnish persisting usage of ACLs and backends" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3015 [11:30:05] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3015 [11:30:33] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3015 [11:30:36] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3015 [11:33:36] New patchset: Mark Bergsma; "Hate Puppet's assignment rules too" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3016 [11:33:47] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3016 [11:34:04] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3016 [11:34:06] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3016 [11:35:29] New patchset: Mark Bergsma; "Syntax" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3017 [11:35:41] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3017 [11:35:47] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3017 [11:35:50] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3017 [11:38:14] New patchset: Mark Bergsma; "Change parameter defaults" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3018 [11:38:26] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3018 [11:38:35] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3018 [11:38:38] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3018 [11:40:16] RECOVERY - Varnish HTTP bits on sq67 is OK: HTTP OK HTTP/1.1 200 OK - 627 bytes in 0.004 seconds [11:43:11] New patchset: Mark Bergsma; "Make varnish work for upload thumbs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3019 [11:43:22] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3019 [11:43:51] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3019 [11:43:53] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3019 [11:47:08]