[00:08:31] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 00:08:28 UTC 2013 [00:09:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:41] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 00:09:37 UTC 2013 [00:10:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 00:10:40 UTC 2013 [00:11:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:41] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 00:11:37 UTC 2013 [00:12:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:31] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 00:12:27 UTC 2013 [00:13:08] TimStarling: ping [00:13:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:13:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 00:13:41 UTC 2013 [00:13:54] TimStarling: I've got a HHVM question for you [00:14:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 00:14:44 UTC 2013 [00:15:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:30:26] preilly: If you just ask the question, I'm sure TimStarling will respond when hes sees it [00:43:42] RECOVERY - Puppet freshness on mc15 is OK: puppet ran at Fri May 17 00:43:36 UTC 2013 [00:44:52] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 00:44:43 UTC 2013 [00:45:12] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:49:47] drwxr-xr-x 2 aaron wikidev 4096 May 14 20:12 5a [00:49:47] drwxr-xr-x 2 aaron wikidev 4096 May 14 20:14 67 [00:49:51] on 1.22wmf4 [00:50:37] !log reedy synchronized php-1.22wmf3/maintenance/checkUsernames.php [00:50:46] Logged the message, Master [00:52:01] Can someone add group write recursively to /a/common/php-1.22wmf4/.git/objects/5a and /a/common/php-1.22wmf4/.git/objects/67 on tin please? [01:14:49] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 01:14:48 UTC 2013 [01:15:09] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [02:07:15] !log LocalisationUpdate completed (1.22wmf4) at Fri May 17 02:07:14 UTC 2013 [02:07:24] Logged the message, Master [02:11:42] New patchset: Diederik; "Set cwd for git log when determining version info" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64252 [02:12:45] !log LocalisationUpdate completed (1.22wmf3) at Fri May 17 02:12:44 UTC 2013 [02:12:54] Logged the message, Master [02:13:19] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:19] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [02:17:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:18:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [02:33:21] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri May 17 02:33:21 UTC 2013 [02:33:30] Logged the message, Master [02:39:03] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [02:39:09] New review: MZMcBride; "Heh, I had a tingling feeling about 'wikipedia' =>, but I couldn't figure it out. I guess there's no..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64109 [02:39:22] New review: MZMcBride; "This is a follow-up to I2d571028c3fbf5a9f5a27c7a68e5048db49ab122." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64109 [02:40:31] New review: MZMcBride; "Follow-up changeset: Idd62b43535cf1b19127421c2f10d3caee4c38f79." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/63877 [02:40:42] New review: MZMcBride; "... in Idd62b43535cf1b19127421c2f10d3caee4c38f79." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64110 [03:17:39] hi, could i push ext/zero out now? should fix the bug we are seeing in logs [03:20:07] greg-g, ^ [03:22:54] TimStarling, ^? [04:02:01] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:01] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 04:07:55 UTC 2013 [04:08:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 04:08:50 UTC 2013 [04:09:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:09:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 04:09:40 UTC 2013 [04:10:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:31] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 04:10:20 UTC 2013 [04:11:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 04:14:49 UTC 2013 [04:15:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:47:55] yurik: i think as long as no one here has said anything about a conflicting deploy then you can iff you can also stay for a bit (an hour at least I guess) to watch for new problems and clean up whatever you broke [04:48:45] jeremyb, heh, sounds like a good plan :) [04:49:44] but i really would like to get someone in ops to ok this [04:50:10] yurik: heya, what jeremyb said is probaby good (but don't tell anyone that it is Friday where you live ;) ) [04:50:19] what do you need reviewed/ [04:50:20] ? [04:50:32] (not that I can really review it) [04:50:55] greg-g, nothing reviewed - i just cehcked, it seems the mw-config with my change has gone live already, so i'm all good to go - just my extension [04:51:04] * greg-g nods [04:51:16] ok, here i go [04:51:18] yeah, pgehres|away did a deploy this afternoon [04:51:30] good :) [04:51:43] i was worried about config thing [04:51:52] (added a new debug log entry "zero" [04:52:03] for the file named zero [04:52:12] sounds reasonable [04:52:21] hope i don't have to do anything crazy like set up dir permissions for the new log [04:52:29] * greg-g doesn't know [04:52:39] i seriously doubt it :) [04:56:02] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [05:06:06] !log yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php [05:06:14] Logged the message, Master [05:09:37] greg-g, seems like its up and running. One problem though - I tried to sync up wmf4, but I can't do git pull in that dir [05:09:54] error: insufficient permission for adding an object to repository database .git/objects [05:13:30] someone didn't set their umode before deploying [05:13:42] you need a root i thinks [05:14:20] (not sure exactly how it's laid out. and maybe there are other ways. but Reedy usually just hunts down a root) [05:17:08] thx jeremyb ! will ask Reedy to reset perms on wmf4 dir or something [05:17:24] not a big deal - wmf4 is not heavily live yet [06:27:39] PROBLEM - Puppet freshness on colby is CRITICAL: No successful Puppet run in the last 10 hours [06:29:29] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 06:29:24 UTC 2013 [06:30:19] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:30:49] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 06:30:48 UTC 2013 [06:31:19] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:31:19] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 06:31:18 UTC 2013 [06:32:19] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:02:10] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [08:02:10] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [08:02:10] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [08:07:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 08:07:46 UTC 2013 [08:08:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:31] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 08:08:27 UTC 2013 [08:09:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:15:01] PROBLEM - Puppet freshness on gallium is CRITICAL: No successful Puppet run in the last 10 hours [08:16:01] PROBLEM - Puppet freshness on db1017 is CRITICAL: No successful Puppet run in the last 10 hours [08:16:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 08:16:48 UTC 2013 [08:17:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:18:38] New patchset: ArielGlenn; "adapt redis template for labs use, update labs redis role settings" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64267 [08:44:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 08:44:45 UTC 2013 [08:45:11] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:52:53] apergos: finally got an internet connection :-] [08:59:13] New patchset: ArielGlenn; "adapt redis role and redis template for lab use" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64267 [09:00:10] hey there [09:02:01] hmm overcast again, wonder if it will rain [09:04:05] ah puppetization nice [09:09:55] well we shall see if is nice :-D [09:14:52] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 09:14:44 UTC 2013 [09:15:12] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:58] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [10:44:13] [06:14:20] (not sure exactly how it's laid out. and maybe there are other ways. but Reedy usually just hunts down a root) [10:44:13] [06:17:08] thx jeremyb ! will ask Reedy to reset perms on wmf4 dir or something [10:44:26] Presumably the same ones I was complaining about a few hours before [10:52:07] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63866 [10:56:25] LordOfLight: why are you supporting stennis ? [10:56:47] eh? [10:57:09] what is stennis? [10:57:33] oh, with a nick like that i was thinking of game of thrones [11:01:45] no spoilers! [11:03:21] haha, Reedy [11:04:02] apparently you added the milionth article to the Spanish Wikipedia [11:05:35] New patchset: Hashar; "tweak jenkins slave authorization key" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64272 [11:24:35] New patchset: Hashar; "tweak jenkins slave definition" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64272 [11:42:17] LeslieCarr: hi! can you help with some bad output from wikidata.org being stuck in squid/varnish? [11:42:27] ...and perhaps in the process enlighten my about cache control headers? [11:42:52] paravoid: or you, maybe? [11:44:29] <^demon> DanielK_WMDE already asked me, but I'm a bit at a loss tbh :) [11:44:37] PROBLEM - DPKG on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:45:27] RECOVERY - DPKG on mc15 is OK: All packages OK [11:47:34] can I help? [11:49:19] mark: oh hey! [11:49:45] possibly. basically, https://www.wikidata.org/wiki/Special:EntityData/Q60.rdf and https://www.wikidata.org/wiki/Special:EntityData/Q60 have bad output stuck in squid/varnish [11:50:06] interestingly, get get the correct result in firefox. but with wget, the result is empty [11:50:13] adding ?foo to the url gets me the correct reult in wget too [11:50:47] adding maxage=0 gets the right response, but doesn't seem to purge the bad entry from the cache [11:50:51] or at least not from all the caches [11:51:06] of course not, it's a different url :) [11:51:21] ugh :/ [11:51:32] so, what's the best way to purge special page output? [11:51:55] note that that special page sets the cache control header. currently, to $wgSquidMaxAge == 31 days. [11:52:05] (i'm abotu to add a config variable there) [11:52:10] and it also sets a vary header on Accept-Encoding [11:52:16] which is probably why firefox works, wget doesn't [11:52:18] (gzip encoding) [11:52:38] mediawiki may set that, the special page doesn't [11:52:41] but yea, makes sense [11:53:09] did you try ?action=purge ? [11:53:14] i don't think it works for special pages [11:53:16] but I'm not sure [11:53:28] hm... no, didn't try. didn't think it would work :) [11:53:51] looks like it didn't work [11:53:52] nope [11:53:56] still broken [11:54:57] how about new Title("Special:EntityData/Q60")->purgeSquid()? [11:55:05] i could patch that in [11:55:17] actually, i could patch that in as a reaction of action=purge being passed to the special page [11:55:35] sounds hacky [11:55:40] why? [11:56:16] hm, i'm curious what Title::getSquidUrls will return for a special page with subpage syntax :) [11:57:45] mark: eventually, we'd want to purge these thigns from squid whenever the data changes - but that means purging .../Q123, .../Q123.json, .../Q123.xml, .../Q123.rdf, .../Q123.n3, etc etc... [11:58:06] but that's for later- for now, i'd be already happy if i could purge one particular rendering [11:58:13] then you better come up with a really solid and scaleable solution for that instead of hoping it will magically fix it self like ULS :) [11:58:41] you should do that first, not last [11:59:01] mark: in the case of ULS, we supposed that it's a foundation project, and they were Doing It Right. [11:59:17] there are far fewer serialization formats than there are languages, so i don't think it's that much of a problem [11:59:18] anyway [12:00:13] mark: the purging when the data is updated isn't critical. that's just an aside. but now we have broken output stuck there. and it's stuck for far longer than I though it would be ($wgSquidMaxAge == 31 days) [12:00:19] ULS magically fix self? [12:00:29] Nemo_bis: nope [12:00:35] http://www.wikidata.org/wiki/Special:EntityData/Q60.rdf [12:00:39] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64272 [12:00:43] Date: Thu, 16 May 2013 19:26:34 GMT [12:00:48] this was served by mediawiki/apache yesterday [12:00:53] the empty response [12:01:19] yes [12:01:26] the code was broken (premature flush) [12:01:30] we fixed that [12:01:40] now we want the broken responses gone from the cache [12:02:01] it'S not totally critical, it's not really used yet. it just randomly interferes with testing [12:02:28] and i just don't know how many broken responses are cached. i just know of two versions of Q60. there may be many more. [12:02:55] mark: to me it's realyl a generaly question - if I have a URL, can I purge it? Or rather, can you? who can? [12:03:11] I believe I just purged the Q60 url [12:04:04] there's a maintenance script for purging [12:04:06] mark: looks like it! awesome! can you do that again without the .rdf at the end? [12:04:08] purgeList.php [12:04:11] just did [12:04:15] thanks [12:04:48] ok, so, as long as we know the exact urls to purge, that's doable. but finding all urls that need purging is going to be hard [12:06:10] yup [12:06:37] mark: right. thanks again! [12:06:49] <^demon> Yay :) [12:06:54] * DanielK_WMDE gallops off to make this more flexible [12:07:55] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 12:07:53 UTC 2013 [12:08:15] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:55] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 12:08:50 UTC 2013 [12:09:15] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:45] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 12:09:40 UTC 2013 [12:10:15] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:26] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 12:10:24 UTC 2013 [12:11:15] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:45] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:45] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:05] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 12:15:01 UTC 2013 [12:15:15] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:24:55] RECOVERY - Puppet freshness on gallium is OK: puppet ran at Fri May 17 12:24:51 UTC 2013 [12:25:10] Can someone add group write recursively to /a/common/php-1.22wmf4/.git/objects/5a and /a/common/php-1.22wmf4/.git/objects/67 on tin please? [12:25:19] i'll have a look [12:25:38] AaronSchulz seems to have a bad umask [12:26:09] done [12:26:29] Thanks [12:26:30] Reedy: perhaps some git hook could check for that? ;) [12:26:37] heh [12:26:44] ^demon: It's getting worse... [12:26:46] Fetching submodule extensions/ZeroRatedMobileAccess [12:26:46] error: The requested URL returned error: 403 while accessing https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ZeroRatedMobileAccess.git/info/refs [12:26:46] fatal: HTTP request failed [12:26:57] Meh, worked 2nd time [12:27:14] <^demon> I still say it's not gerrit's fault. [12:27:53] Hmm, no. Zero is doing it on demand [12:28:09] Why is it only a couple of repos that seem to be borked? [12:28:46] <^demon> Nothing's wrong with those repos :\ [12:28:56] !log reedy synchronized php-1.22wmf4/maintenance/checkUsernames.php [12:29:04] Logged the message, Master [12:30:40] New patchset: Hashar; "contint: fix openjdk packages names" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64280 [12:31:24] reedy@terbium:~$ foreachwiki [12:31:24] /usr/local/bin/foreachwikiindblist: line 4: /a/common/all.dblist: No such file or directory [12:31:38] It's only in /a/common on tin... [12:37:24] Reedy: can you merge https://gerrit.wikimedia.org/r/#/c/64221/ so that I fix what I broke? [12:39:52] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [12:43:06] New patchset: Hashar; "contint: jenkins slave user had no home" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64282 [12:45:02] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 12:45:00 UTC 2013 [12:45:12] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:48:59] New patchset: Petrb; "motd is now project wide" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64285 [12:53:27] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64221 [12:55:00] ^demon: Won't let me update ProofreadPage now.. [12:55:15] !log reedy synchronized wmf-config/InitialiseSettings.php [12:55:23] Logged the message, Master [12:56:08] <^demon> I'm wondering if we could do all the submodules as ssh. [12:56:21] <^demon> Wouldn't hurt, and I know ssh tin -> manganese works. [12:57:09] wget gerrit.wikimedia.org works fine ;) [12:57:14] <^demon> Yeah ;-) [12:57:34] New review: coren; "profile.d can mess with noninteractive sessions if it outputs stuff; this would be better done in up..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/64285 [12:58:00] <^demon> Adjusting make-wmf-branch to create the submodules with ssh would be trivial. [12:58:07] <^demon> Adjusting the current clone would be a bit more annoying. [12:58:36] move it [12:58:38] reclone [12:58:41] move l10n back in [12:58:45] <^demon> Or that. [12:58:47] push for consistency [12:59:12] <^demon> I was thinking something like edit .gitmodules then `git submodule foreach git remote set-url origin ` `git remote update --init` [12:59:22] <^demon> But your way is easier. [12:59:46] won't take too long to do either :D [13:18:57] New patchset: Petrb; "motd is now project wide" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64285 [13:20:40] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64285 [13:27:13] Reedy, hi, i couldn't push out wmf4 yesterday because of an issue with the git repo. I didn't want to break it, so left it as is. git pull was giving perm error [13:27:21] I saw [13:27:31] do you know what caused it? [13:27:37] bad umask by AaronSchulz [13:27:42] I reported the same problem in here a few hours before you [13:27:50] mark fixed it an hour or so ago [13:27:57] But you can't currently update Zero either [13:29:01] that's ok - wmf4 has not been pushed to wikipedias yet [13:30:01] Reedy, i am more worried about the 76 Warning: Recursion detected in RequestContext::getLanguage in /usr/local/apache/common-local/php-1.22wmf3/includes/context/RequestContext.php on lin [13:30:02] e 281 [13:30:09] Why? [13:30:18] It's been happening for a while [13:30:19] !log Zuul: applying project templates {{gerrit|63674}} [13:30:28] Logged the message, Master [13:30:36] i'm wondering if that's my call that's causing it [13:30:57] doubtful, but since i don't see the callstack... [13:32:38] We should probably try and set PHP up to log the callstack somewhere [13:33:56] yes, that's what grownup devs have always wanted for their birthdays :) [13:34:45] Reedy, one question though - wmf4 is scheduled to go live on monday. Will it pick up the latest wmf4, including submodules, or will they simply deploy whatever is in wmf4 dir on tin? [13:34:59] I usually make sure things are all upto date [13:35:59] ok, pls make sure the zero submodule is updated - i have already comited the wmf4 ver. Thx! [13:36:17] PROBLEM - RAID on analytics1016 is CRITICAL: Timeout while attempting connection [13:37:47] PROBLEM - Host analytics1016 is DOWN: PING CRITICAL - Packet loss = 100% [13:37:54] As I said, it's currently broken [13:38:27] reedy@tin:/a/common/php-1.22wmf4$ git submodule update extensions/ZeroRatedMobileAccess [13:38:27] error: The requested URL returned error: 403 while accessing https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ZeroRatedMobileAccess.git/info/refs [13:38:28] fatal: HTTP request failed [13:38:28] Unable to fetch in submodule path 'extensions/ZeroRatedMobileAccess' [13:38:28] reedy@tin:/a/common/php-1.22wmf4$ [13:38:37] RECOVERY - Host analytics1016 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [13:39:27] PROBLEM - RAID on analytics1017 is CRITICAL: Timeout while attempting connection [13:39:36] Reedy, i understand, but i thought you were fixing perms there? [13:39:52] It's not permissions [13:39:55] Read the error? [13:40:02] And I said, the permission issue is already fixed [13:40:57] PROBLEM - Host analytics1017 is DOWN: PING CRITICAL - Packet loss = 100% [13:41:57] RECOVERY - Host analytics1017 is UP: PING OK - Packet loss = 0%, RTA = 0.71 ms [13:43:30] ohh. oops, sorry, didn't read it... and Reedy, i 'm getting http 406 when i click git link (i wonder if that's authentication). Regardless, I am not sure what the cause of the error is - git is down again? [13:45:35] yurik: a change was merged ~10m ago [13:48:23] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64280 [13:49:28] p858snake|l, thanks! [13:49:51] my concern is that i don't know what to do in case i see such errors during deployment [13:50:19] best: scream loudly [13:51:42] thx mark! I know how to do that well! :) [13:56:51] get a mat out and start sending the smoke signals towards Reedy [13:58:10] yurik: then open up a tab account at the closest stroopwafel provider near Reedy [13:58:40] * yurik googles stroopwafel [13:58:51] oooo! [13:58:52] yam [13:59:56] I suspect WMNL will have sourced many for next week [14:01:58] New patchset: Hashar; "contint: jenkins slave user had no home" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64282 [14:02:47] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [14:10:45] Change abandoned: Andrew Bogott; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64105 [14:11:17] New review: Milimetric; "This will fix the problem we had getting the version to show in prod." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/64252 [14:14:14] preilly: I have made jenkins to block sartoris changes whenever pyflakes fail [14:14:39] Reedy, i will git update the zero submodule on tin, but won't push out anything [14:15:02] will let you do the honors on monday :) [14:22:06] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64282 [14:22:59] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64252 [14:45:57] New patchset: Faidon; "Varnish: remove send_timeout=30, rely on the default (600s)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64314 [14:47:04] New patchset: Faidon; "Varnish: remove send_timeout=30, rely on the default (600s)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64314 [14:48:04] New review: Faidon; "Troubleshooted with mark, implies at least a +1 :)" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/64314 [14:48:04] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64314 [14:56:17] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [14:56:19] New review: Anomie; "It seems the _SOURCE names won't work on terbium, for example. But it doesn't work now either, so I'..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55059 [14:58:14] New patchset: Ottomata; "Making misc::limn::instance more configurable" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64316 [14:58:38] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64316 [14:59:24] !log setting Varnish send_timeout to 600 on upload, mobile, bits (eqiad), upload, bits (esams) for both frontend & backend [14:59:29] did I miss anything? [14:59:33] Logged the message, Master [15:04:16] New patchset: Ottomata; "Fixing typo in variable name" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64318 [15:04:29] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64318 [15:26:14] New patchset: Ottomata; "Piping stderr to limn log file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64319 [15:26:30] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64319 [15:41:34] New patchset: Ottomata; "Ensuring mod proxy_http is enabled for statistics apache" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64321 [15:41:42] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64321 [16:05:48] New patchset: Ottomata; "Not redefining $base_directory in limn::instance define" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64324 [16:06:48] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/64324 [16:08:00] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 16:07:58 UTC 2013 [16:08:40] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:10] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 16:09:00 UTC 2013 [16:09:40] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:00] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 16:09:57 UTC 2013 [16:10:40] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:50] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 16:10:47 UTC 2013 [16:11:40] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:10] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 16:12:04 UTC 2013 [16:12:40] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:00] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Fri May 17 16:14:54 UTC 2013 [16:15:40] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:24:56] say you have a screen session on a server with an always on irssi client. you connect via ssh and `screen -R freenode` to enter the session, any suggestions for how to get ping's (highlights) forwarded into the desktop linux machine your usings notification system? [16:26:32] it would seem it might need a few pieces, perhaps when you connect you forward a particular port from the server back to your machine. irssi could send the notification over that port and a server locally could receive it and generate the notification, i could write that up but was thinking theres probably a simpler solution [16:26:46] s/server locally/daemon locally/ [16:27:35] I know on Windows and Mac there are tools for doing this [16:28:00] PROBLEM - Puppet freshness on colby is CRITICAL: No successful Puppet run in the last 10 hours [16:28:13] doh i just realized i'm asking in the wrong channel, meant to ask in #irssi :P [16:28:14] ebernhardson: Did you try googling? http://jonathanbeluch.com/blog/2011/03/remote-notify-irssi-screen/ [16:28:25] but thanks i will check that out :) [16:35:11] RECOVERY - Disk space on ms-be9 is OK: DISK OK [16:36:07] sudo -u apache mwscript checkUsernames.php zhwiktionary | tee ~/checkUsernames.log [16:36:25] sudo -u apache ./foreachwiki checkUsernames.php | tee ~/checkUsernames.log [16:36:26] even [16:36:50] Any idea why all that gets appended to the log is the output from the top foreachwiki script? Both appear in the log [16:37:14] reedy@terbium:~$ sudo -u apache mwscript checkUsernames.php zhwiktionary | tee ~/checkUsernames.log [16:37:14] zhwiktionary: 120: '約翰　可比西都羅農' [16:37:14] reedy@terbium:~$ cat ~/checkUsernames.log [16:37:14] reedy@terbium:~$ [16:39:17] I'm guessing it's due to where the output is being written, or not, in this case [16:40:28] Hmm, 2>&1 fixes it [16:40:36] sudo -u apache mwscript checkUsernames.php zhwiktionary 2>&1 | tee ~/checkUsernames.log [16:40:55] Reedy: having a nice conversation with yourself? [16:40:59] Yup [16:41:21] * Reedy high fives himself [16:41:24] <^demon> I assume most people are pretty interested in what they have to say ;-) [16:41:44] * pgehres calls a physciatrist for Reedy [16:41:52] mutante: is this the security issue being fixed (and are you ok/not ok with me linking to it in my message)? https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1179943 [16:42:13] that must be it, it was released on the 15th [16:42:13] you've just posted it in public.. [16:42:28] yeah, it's a public bug, if anyone knows we use Ubuntu, they know what the issue is [16:42:43] or, really, if they know we use the linux kernel :) [16:42:59] It was just confusing as you were asking if you could post it somewhere else at the same time [16:43:23] yeah, true, nevermind :) [16:43:50] eh, i think Reedy already answered what i thought:) [16:45:07] commonswiki: 10712: 'Ff02::3' [16:45:10] We have some awesome usernames [16:46:01] huh [16:46:15] I'm running a script to get a list of all invalid usernames [16:46:26] That looks like a variant of an IPv6 address [16:46:55] IPv6 multicast [16:47:11] commonswiki: 1287235: 'ɑdmins eating elephant poo' [16:47:13] * Reedy grins [16:49:07] <^demon> !log rebooting antimony [16:49:15] Logged the message, Master [16:50:51] PROBLEM - Host antimony is DOWN: CRITICAL - Host Unreachable (208.80.154.7) [16:51:20] Reedy: all from commons so far, heh [16:51:34] There's more than that [16:51:36] They were just amusing ;) [16:51:41] * greg-g nods [16:51:41] RECOVERY - Host antimony is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [16:52:06] 54 and we're onto dewiki [16:54:24] dewiki: 364940: 'WP:CU' [16:55:57] heh [16:55:58] Reedy: what are you planning to do with this list of usernames? [16:56:04] DELETED! [16:56:20] pgehres: Post them on a bug [16:56:29] And decide if I care enough to work out a plan to "fix" them [16:56:42] https://bugzilla.wikimedia.org/show_bug.cgi?id=3507 [16:56:56] Ah, k. I'll be interested to see how many of them are targets of the SUL finalisation [16:57:48] Reedy: http://en.wikipedia.org/wiki/User:Recentchanges :) [16:58:14] typo squatter in user space,, hehe [16:59:30] nice [16:59:45] 377 [16:59:47] We're onto enwiki [17:00:27] pgehres: CA bug for you in -tech [17:04:00] New patchset: Odder; "(bug 48308) Change namespace settings for ukwikisource" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64336 [17:04:38] gah, i search for africa (looking for south african chapter website) and all i get is RT 1081 :-P [17:12:12] robh: have time to walk through Cerium disk addition? [17:23:24] cmjohnson1: sorry, was moving into office [17:23:26]