[00:04:24] PROBLEM - Check status of defined EventLogging jobs on vanadium is CRITICAL: CRITICAL: Stopped EventLogging jobs: consumer/mysql-db1047 [00:06:41] Is there an issue with lists.wikimedia.org? Trying to change a list's 'owner' field and mailman is timing out on the request… [00:07:08] James_F: try Firefox instdead of Chrome [00:07:23] mutante: Really? Wow. [00:07:38] yes, Chrome timeout [00:07:41] black magic [00:07:53] i dont even know what to blame, but it only happens with that :p [00:07:56] Yeah, it works. [00:07:58] * James_F sighs. [00:10:12] mutante: Thanks hugely, BTW. :-) [00:13:06] James_F: np! i guess you recently created a bunch of lists since THO was gone? [00:13:47] mutante: No, Jamesofur did, but I was asked in a personal capacity to wield a ban-hammer. [00:14:08] got it! [00:14:42] (yes, sadly chrome and mailman do not get along) [00:14:45] James_F: when they ask about spam https://wikitech.wikimedia.org/wiki/Mailman#Fighting_spam_in_mailman [00:14:54] mutante: yeah, they were getting backed up and so I went 'PSH' [00:15:17] thank you, i actually had a message from THO about them but couldnt find the tickets [00:15:22] mutante: Nice. [00:20:03] * AaronSchulz strangles http://osdir.com/ml/repo-discuss/2013-07/msg00088.html [00:30:34] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [00:32:44] springle, looks like s1-analytics is down now. "ERROR 2003 (HY000): Can't connect to MySQL server on 's1-analytics-slave.eqiad.wmnet' (111)" [00:32:58] should I just keep waiting? [00:33:03] halfak: yep :) [00:33:08] OK. [00:33:13] trying something [00:35:28] halfak: should be back. the txn and locks are gone, but undo list must still be processed. would be good not to hammer the box for a day or so [00:35:51] so at least event logging can continue and replication start to catch up [00:36:26] Thanks. I'll spread the word. [00:39:04] springle, once lag is caught up, should it be good to take the regular abuse? [00:42:27] halfak: yep. worth rethinking those leizi_temp* tables though; at least drop the ORDER BY, and consider breaking it up into batches [00:43:36] Oh yeah. Not sure what was up with those. Will have a look. :) [00:44:34] maybe even use ENGINE=ARIA (like MyISAM) for the temp tables. would make it easier to recover if a transaction wasn't logging masses of history for rollback which could be as simple as dropping a few temp tables [02:13:12] !log LocalisationUpdate completed (1.23wmf17) at 2014-03-18 02:13:11+00:00 [02:16:40] morebots is dead again [02:17:09] Likely the Labs maintenance. [02:17:47] andrewbogott: morebots? :-) [02:17:56] I see grrrit-wm is back, yay. [02:18:48] Gloria: I think all bets are off while the bots migrate to ashburn [02:19:16] andrewbogott: Just as long as we eventually log the entries in the server admin log. [02:19:29] Can always do it by hand while morebots is down. :-) [02:19:38] grrrit-wm seems connected, but quiet. [02:24:11] !log LocalisationUpdate completed (1.23wmf18) at 2014-03-18 02:24:11+00:00 [02:56:18] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Mar 18 02:56:15 UTC 2014 (duration 56m 14s) [03:16:24] bd808|BUFFER: ori yay, the directory permissions fixed the l10n update :) [03:31:34] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [03:38:53] greg-g: :) [04:00:05] ori: Do you know anything about the 'eventlogging' labs project? [04:00:12] Specifically, whether it's good for anything anymore? [04:01:28] there's an eventlogging setup on the beta cluster that is the canonical place for working on eventlogging in labs, and i'm almost certain that the dedicated eventlogging project is therefore a dupe [04:01:30] but let me make sure [04:02:13] thanks [04:39:58] andrewbogott_afk: I don't think I see an eventlogging labs project [06:18:14] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [06:32:34] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [06:48:14] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [06:51:14] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [06:55:14] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [06:56:14] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [07:32:14] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [08:33:23] !log 5xx resps spiked between 6:15 and 6:35 UTC; lvs1001 SSH check flapped between 6:48 and 6:56 UTC. [08:33:29] Logged the message, Master [08:47:00] aude, Tobi_WMDE: 2014-03-18 08:44:30 mw1196 wikidatawiki: [588a2739] /w/api.php?action=wbgetentities&props=sitelinks&format=json&ids=Q18559 Exception from line 98 of /usr/local/apache/common-local/php-1.23wmf17/extensions/Wikidata/extensions/Wikibase/lib/includes/store/sql/WikiPageEntityLookup.php: No such revision found for Q18559: 116886079 [08:47:38] ori: so that revision has just vanished? [08:48:03] maybe it was oversighted? [08:48:23] dunno, I'm on my way to bed, not brain power left to debug :) [08:48:32] *no [08:48:42] * Jasper_Deng wants to high-five ori w/ respect to that but can't atm :/ [08:49:10] can't high-five? [08:49:14] ori: thx! will tell it to the team once they show up.. [08:49:40] Tobi_WMDE: np, thank you [08:50:28] good night / day! *waves* [08:57:06] ah ori heading to bed [08:57:07] that is good [08:58:59] ori: b/c I can't go to bed atm [08:59:01] >.< [09:02:59] ori: https://bugzilla.wikimedia.org/show_bug.cgi?id=62547 [09:11:18] aude: any idea how I can pull from the puppet-jenkins repo on the new jenkins instance? [09:12:08] sorry, wrong channel [09:33:34] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [09:57:55] apergos: please https://gerrit.wikimedia.org/r/#/c/119212/1 [10:27:10] matanya: sorry, doing cleanup from yesterday, it's going to be awhile... [10:37:34] hashar: i don't understand what you are trying to do [10:38:46] matanya: on the beta cluster, I need the syslog-ng central server to write to /data/project/syslog instead of /home/wikipedia/syslog [10:38:46] :D [10:39:01] that part i got [10:39:10] you can just add a simple variable [10:39:31] and I need a role as well [10:39:53] if ::relam = producation write syslog to foo, else to bar [10:40:27] why do you need a role? [10:41:49] cause I had previous complaints about adding if realm / else [10:41:59] so I add some new roles [10:42:16] though that probably depends on the ops ultimately merging the change [10:47:55] i see [11:29:04] akosiaris: quick question: in modules/authdns/manifests/init.pp there is a variable ipaddress, is it equal to ::ipaddress ? [11:29:51] no, you can clearly see that the class has a parameter named ipaddress at the top [11:30:28] ah how nice. A local scope variable shadowing a global one ? [11:31:11] it's not shadowing it, you're not supposed to use the global one unprefixed anyway :) [11:31:24] unprefixed ? [11:31:34] unscoped [11:32:02] aah. Yeah in puppet3, but in puppet2 it is shadowing it [11:32:38] ? [11:32:45] since when it puppet sees the $ipaddress it will start searching every scope up to global [11:32:56] and throwing errors [11:32:58] which was bad bad bad and is no longer happening [11:33:01] er, warnings [11:33:07] yeah yeah sure [11:33:21] but technically it is shadowing, that is what i wanted to point out [11:33:25] you're not supposed to use $ipaddress to fetch the global variable even with puppet 2.7 [11:47:24] Puppet's starting to show bad heritage, just like PHP:P [11:48:31] anyway akosiaris and paravoid i guess the best way would be to rename that variable, correct? [11:48:56] * matanya got into every egde case of puppet in this puppet 3 migration project :) [11:49:13] why? [11:49:17] it is not technically needed. Scoping and all seems to be correct [11:49:18] it works fine [11:49:20] it will work fine [11:49:34] Dynamic lookup of $ipaddress is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes. [11:49:47] i would rename it simply for git grep to have better output [11:49:50] but it is not needed [11:50:27] matanya: is that warning for authdns? [11:50:48] that is the only left call to ipaddress i see when grepping [11:51:07] execpt site.pp [11:51:17] which i changes locally to test [11:51:20] *d [11:52:05] I see a bunch more [11:52:22] like manifests/realm.pp [11:52:28] also site.pp would trigger this error [11:52:46] realm is also fixed locally [11:53:03] have you reproduced this warning locally? [11:53:33] i have, but not sure it is related to authdns [11:53:48] since puppet doesn't bother to tell me *where* is the problem [11:53:53] if you have reproduced it, try modifying authdns and then retrying? [11:53:57] yeah, it's annoying [11:54:01] good point [11:54:05] i'll try that [11:54:06] it could be a false positive too [11:56:18] and now i can't reproduce [11:56:23] sigh [12:00:33] matanya: thanks for all the reviews :-] [12:01:21] my pleasure [12:03:14] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [12:05:22] lunch [12:05:38] the old tools login is throwing me out on login... brilliant [12:05:49] bon appetite [12:32:27] matanya: I'd also use suggestions :) https://gerrit.wikimedia.org/r/119216 [12:33:40] is doesn't even build Nemo_bis [12:34:16] Nemo_bis: you are missing } in line 26 [12:34:34] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [12:35:49] MZMcBride: Is there a way to get a statistic how many users use the math rendering option source png mathjax and default [12:36:08] matanya: I know but that's not the main problem [12:36:12] The historic values would be interesting as well [12:36:18] physikerwelt: Is that a user preference? [12:36:23] it is hoo [12:36:23] yes [12:36:35] Nemo_bis: what is your main problem ? [12:37:09] matanya: see inline comments [12:37:13] ok [12:37:20] I found a script https://en.wikipedia.org/wiki/Wikipedia:Database_reports/User_preferences/Configuration that generates user preference statistics but does not include the math rendering option [12:37:58] contact the author physikerwelt [12:38:40] physikerwelt: that user preference uses numerical values? Yikes... [12:38:53] yes 1-7 [12:38:56] I can probably get you numbers if you tell me which wiki you're interested i [12:38:57] n [12:39:57] physikerwelt: ^ [12:40:03] I would be interested in the distribution of the numbers [12:40:45] hoo: I would be interested in the distribution of the numbers [12:40:59] physikerwelt: On which wiki? [12:41:18] enwiki would be sufficient [12:41:33] can you differntiate between 0 and NULL [12:42:04] 0 means PNG [12:44:24] physikerwelt: Ok, back with irssi now... my pidgin went crazy [12:45:30] physikerwelt: http://fpaste.org/86302/39514671/ [12:46:11] thank you. Is SPACE = null [12:47:07] no, empty string [12:49:00] caffeineexcess: ok thank you. Thus if users don't change the default setting there will be no entry [12:49:28] That's how the preferences are supposed to work :) [12:49:59] Nemo_bis: not very familiar with this stuff, what is it all about? [12:50:25] caffeineexcess: I'd like to mention you if I publish this numbers unfortunately I don't remember your name. Should I refere to user hoo man? [12:50:37] physikerwelt: Yeah :) [12:50:47] Pidgin seems to also work again... [12:50:49] ok thank you [12:50:55] wonder where these malloc failures came form [12:50:57] * from [12:54:19] matanya: it's about what the commit message says :) [12:54:43] Do you what mediawiki-vagrant is? It's used by MediaWiki devs for local machines for testing, and also for quick virtual server setups [12:57:55] i do know that Nemo_bis [12:58:08] i'm not familiar with the setting and configs [13:18:30] matanya: those are not important, they can be argued over later by MediaWiki devs :) [13:19:04] ok [13:19:14] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [13:26:45] hoo: another question. Can I determine how manny users did not set anything. Is that the rest to 20 954 494 [13:27:18] or are just the active users taken into account 129,542 [13:27:56] I'm wondering that most common setting is MODERN which has been marked as deprecated in 2011 [13:28:12] and is no longer selectable from the ui [13:29:02] physikerwelt: All other users don't have that setting set, yes [13:29:07] What?? https://integration.wikimedia.org/ci/job/operations-puppet-validate/12493/console [13:29:28] so that would be most of the almost 21 million [13:29:51] I guess so, yes [13:30:39] ugh, spurious self-pasting things [13:31:27] !log hoo synchronized wmf-config/InitialiseSettings.php 'Enable GuidedTours on Wikidata' [13:31:33] Logged the message, Master [13:35:16] ottomata: http://www.debian.org/doc/debian-policy/ch-docs.html#s-copyrightfile [13:36:01] akosiaris: because it's your week :) I think https://gerrit.wikimedia.org/r/117250 is ready [13:36:12] ottomata: http://dep.debian.net/deps/dep5/#license-specification (see syntax for example) [13:37:09] that's what I meant by "use common licenses" [13:37:42] hoo|irssi: Thank you [13:38:17] You're welcome :) [13:38:34] RECOVERY - Squid on carbon is OK: TCP OK - 0.000 second response time on port 8080 [13:45:46] AzaToth: ok cool, I was able to shorten the copyright file by referencing BSD and LGPL-2.1 [13:46:22] yea, but you should include the "header" though [13:46:30] for the LGPL one [13:47:19] ha, it was not clear what the header is [13:47:23] that one is really verbose [13:47:39] is the header the preamble? [13:47:52] I usually copies /usr/share/debhelper/dh_make/licenses/lgpl2 [13:47:54] yea [13:48:06] oh great [13:48:07] thanks [13:49:21] offcourse if it's a common license + exception, you can combine it as it was shown in the example [14:10:22] hashar: I think your proxy issue is resolved :-) [14:10:50] akosiaris: carbon is back up ? :-] [14:10:57] yes [14:11:01] that was fast! [14:11:34] carbon is easy to find ;) [14:14:43] squid3 eh? [14:14:56] (I was looking at it when I saw someone removing the package... I figured it was you) [14:15:24] akosiaris: you can close https://rt.wikimedia.org/Ticket/Display.html?id=7069 :] [14:15:29] ah it is closed alredy [14:16:58] :-) [14:29:44] !log Create EducationProgram and Translate tables on legalteamwiki [14:29:49] akosiaris: regarding the syslog-ng puppet changes, do you want to schedule that ahead of time? [14:29:49] Logged the message, Master [14:29:58] akosiaris: since it might cause issues on nfs1/nfs2 servers [14:30:31] I haven't had the time to look at the change yet. I 'll go through it and let you know [14:31:47] great : [14:32:37] """'Calling package builder 'cowbuilder' on distributions 'lucidprecise'""" *sigh* [14:32:44] poor Lucid [14:34:10] cmjohnson1: ok i guess I should take an18 offline? [14:34:34] ottomata: no don't, I am not there at that moment. [14:34:47] got hung up with getting inventory from Tampa [14:35:01] ok cool [14:35:12] didn't put it on my calendar [14:35:36] ottomata: wanna do after lunch? [14:36:30] err, today is a bad day for that, there are a bunch of analytics showcase meetings and I leave early on tuesdays to go teach [14:36:37] tomorrow? [14:37:18] !log reedy synchronized wmf-config/ [14:37:23] Logged the message, Master [14:38:41] ottomata: tomorrow is better [14:38:50] k cool [14:44:04] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 382.833344 [14:46:24] heyyYYYyyyy shh [14:48:34] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 983.700012 [14:49:03] :-@ [14:51:04] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [14:51:14] manybubbles, MaxSem: Looks like no changes for this morning's SWAT window. [14:51:45] there were some cirrus changes [14:51:50] but it looks like someone moved them? [14:52:23] https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=104947&oldid=104772 [14:52:52] hey [14:52:53] yep [14:53:25] I don't think those cirrus changes were deployed yet [14:53:29] I mean, no one asked me about them [14:56:10] manybubbles: oh, I thought you did those yesterday morning [14:56:19] manybubbles: I thought you just put them on the wrong day [14:56:28] my bad [14:56:45] manybubbles: the email thread confused me. [14:57:02] ah, sorry! yeah, I scheduled them for this morning [14:57:25] no worries, happens, thank you, go forth! [14:57:27] paravoid, any word on that unbalanced networking stuff to/from esams/eqiad? [14:58:04] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 85.73333 [14:58:34] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [15:01:13] who today? [15:01:41] ottomata: no, maybe mark wants to have a look instead? :) [15:03:07] hi [15:03:21] greg-g: manybubbles we are deploying a config change [15:03:33] uh? [15:03:57] looks like guided tour was already done [15:04:01] !log reedy updated /a/common to {{Gerrit|I72a7751f8}}: Let AbuseFilter block users on Spanish Wikivoyage [15:04:06] Logged the message, Master [15:04:18] manybubbles: really? [15:04:28] that is what https://wikitech.wikimedia.org/wiki/Deployments says [15:05:09] Yeah, already did that earlier [15:05:17] oh [15:05:30] Didn't see that one... mh [15:05:45] what's SWAT anyway... did we move over to killing people? :P [15:06:08] probably good to have these on the calendar, but ok this time [15:06:32] nobody told me that we scheduled that, so I wasn't aware and just used a silent spot [15:06:39] that's fine [15:07:42] hoo: aude , just fyi (you did nothing wrong), but when asked if guidedtours on wikidata was ok for a SWAT deploy, I responded thusly: [15:07:45] 10:55 < greg-g> anomie: yeah, good question. That one is a blurry line (since it was enabled other places, and was on test wikidata already). I almost blocked it last night when I saw it, but... didn't. I think it's fine. [15:07:59] aude: hoo just so you get a feel of what the line is :) [15:08:17] it is blurry [15:08:23] * greg-g nods [15:08:49] anyway, look forward to the tours on wikidata :) [15:08:58] well tested on testwikidata [15:09:00] Yep, that's the main thing :) [15:09:12] we tested on both testwikidata and beta labs [15:09:13] and overwhelming consensus on project chat [15:10:03] yeah, all fine, but if it was a new extension, it'd need a separate window so it gets announced correctly [15:10:06] :) [15:11:18] Oh sure [15:11:59] anywho, that's all! [15:14:06] greg-g: anyhoo [15:15:04] Reedy: :P [15:17:46] hoo: oh, wait, when did you do this? I thought during a SWAT window? Next time SWAT window please :) [15:20:11] greg-g: what's that SWAT thing after all? It says I'm not even allowed to deploy there as I'm not on that list, right? [15:20:22] we can deploy [15:20:23] list of people [15:20:32] it's just there are more experienced people around at the time [15:20:43] i would just list our thing and the person who is deploying [15:20:55] * aude should ahve poked you that i put it on there [15:21:03] bah, I guess I just emailed engineering@, you're on ops@ list right, hoo? [15:21:10] https://wikitech.wikimedia.org/wiki/SWAT_deploys [15:21:12] greg-g: Yep, only on ops [15:21:12] greg-g: you emailed us [15:21:17] * aude got email [15:21:48] ah, it was in "Roadmap and deployment highlights - week of March 17th" on wikitech [15:22:19] yeah, I'll be sure to cc ops@ about it in the future, too. [15:22:34] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 787.799988 [15:22:45] when I first started, I assumed/was told ops was a subset of engineering@, so I didn't always email both [15:22:59] but, since we have more non-WMF deployers now who are on ops@... no longer true! [15:23:01] mh, maybe I'm supposed to be on engineering as well [15:23:24] * aude thinks it's a staff list [15:23:36] ops@ is the requirement for all deployers, WMF or not, so the original assumption was probably that they already were on engineering@, since they were a WMF engineer [15:23:50] yeah, it's pretty quiet [15:24:09] purpose is staff only, I believe /me looks at archives [15:24:31] so, who is on point for swat today? [15:24:35] greg-g: ^? [15:24:48] i think no one [15:24:59] nothing today since ours is done [15:25:02] manybubbles: who ever in the list of names steps up :) [15:25:12] so, me then? [15:25:18] guess so! [15:26:21] hoo: aude yeah, engineering is mostly "here's this meeting on TDD on this date" and "don't forget to fill out the form to see who goes to the hackathon" and such [15:27:04] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [15:27:37] * manybubbles has the conch [15:29:04] James_F|Away: btw https://bugzilla.wikimedia.org/show_bug.cgi?id=56842 [15:33:04] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 125.133331 [15:33:34] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [15:35:14] !log manybubbles synchronized php-1.23wmf18/extensions/CirrusSearch/ [15:35:21] Logged the message, Master [15:35:34] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [15:36:19] mutante|away: Thanks. [15:36:25] * manybubbles puts the conch down [15:37:14] * greg-g wipes the spit off the conch [15:46:49] ah puppet killed me :-((( [15:47:51] * ^d revives hashar [15:52:09] hiiiii [15:52:25] ^d: just in time, I got a question for you. If I am to report hhvm issue should I do them upstream ? [15:52:37] or would it makes sense to create a hhvm component in our bugzilla? [15:52:45] <^d> hashar: We've got a keyword! [15:53:08] ^d: are you actually triaging it ? :-] [15:53:36] will fill in bugs under Wikimedia > continuous integration and mark them [15:53:51] <^d> I'll at least look at it :) [15:53:58] <^d> And see if an upstream bug needs filing. [15:55:30] ^d: also you might want to enroll ebernhardson on the hhvm front. He proposed at least one hhvm configuration change for mw/core :] [15:58:34] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 855.56665 [16:03:04] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [16:03:22] paravoid: (and Jeff_Green's ghost): after discussion with the apparent sender I looked more closely at the headers and http://lists.wikimedia.org/pipermail/wikimania-l/2014-March/005536.html looks spoofed. any thoughts about that? idk what yahoo supports but are we doing any checks on the way to mailman? (dkim/spf/etc.) [16:04:04] we can contact the guilty relay to fix the compromised account. but then presumably they'll find a new relay [16:04:54] ^d: https://bugzilla.wikimedia.org/show_bug.cgi?id=62787 and https://bugzilla.wikimedia.org/show_bug.cgi?id=62788 . You are on CC and I pushed you back an email by Timo from Sunday :-] [16:05:06] i was considering telling him to try to use a different address for posting. (there are other addresses he's already regularly using. idk how he chooses which to use for a given mail) [16:09:34] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [16:10:04] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 136.666672 [16:13:42] basically, https://xkcd.com/1254/ [16:21:10] grrrit-wm is broken [16:21:11] fwiw [16:24:34] paravoid: ahhhh tabs :) [16:25:40] it's amazing how useful that (grrrit-wm) was for me [16:26:07] greg-g: he will be back [16:26:15] and i agree it is useful [16:32:34] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 714.466675 [16:37:06] matanya: tabs because the file had tabs already and my changeset wasn't whitespace-related [16:38:04] i disagree, since it is a add and not rename, and i would like if you can fix it but i'm out. bye [16:39:03] it is a rename [16:39:04] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [16:40:05] akosiaris: https://gerrit.wikimedia.org/r/#/c/118794/4/manifests/admins.pp [16:42:34] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [16:45:04] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 139.899994 [16:47:52] akosiaris, thanks for rt 6961! [16:48:13] gwicke: you are welcome :-) [16:54:33] Someone broke the sudoers. [16:54:47] Is anyone on it? [16:54:55] more info ? [16:55:24] i merged a sudo change not so long ago [16:55:34] but only on wtp* [16:57:15] its what is emailing [16:57:27] all the wtps are sending security warning emails, but only those [16:57:33] oh, and tin. [16:57:42] (but thats different thing) [16:58:00] akosiaris: "parse error near line 5" [16:58:01] i don't think so [16:58:07] I just saw it [16:58:11] damn cron emails.. [16:58:34] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 3952.133301 [16:58:52] Gah, there's a tab at the beginning of lines. [16:59:02] yeah [16:59:06] it does not matter [16:59:18] other files have it too [16:59:26] look into it since I broke it [17:00:04] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [17:00:34] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [17:03:12] akosiaris: space after NOPASSWD: ? [17:03:22] fully path to command [17:03:24] and space [17:03:30] full* [17:06:45] Yeah, I just commited. [17:07:12] Coren: you beat me to it [17:07:14] thanks [17:07:41] https://gerrit.wikimedia.org/r/#/c/119308/ if you want to quick +2 [17:08:27] merged and puppet-merged [17:09:46] also triggered a puppet run through salt [17:09:58] * Coren hopes google won't throttle his inbox again. :-) [17:10:02] ahahahaha [17:10:10] I had forgotten about that [17:10:41] I didn't. It took /days/ to recover. [17:11:11] arg, rebase number 1000 [17:12:07]