[00:20:33] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [00:27:36] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [00:27:36] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [00:28:03] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:33:54] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.024 seconds [00:58:44] Can anyone tell me what would happen if you edit an template that was being used on 200,000+ pages? [01:02:16] techman224: From my understanding you'd add a large number of operations to the job queue, but you wouldn't kill the site. [01:04:35] techman224: It also might be a cache nightmare... [01:07:07] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:09:43] techman224: In general, try to make edits to a copy of the template first (e.g. Template:Foobar/sandbox) and on a few pages where Foobar is used, change it from Foobar to Foobar/sandbox to try it out [01:10:02] would be a waste if you change it and turns out to break something [01:10:22] discussion probably on a talk page as well. But as far as -tech is concerned, should be just fine. [01:12:58] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 2.942 seconds [01:47:01] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:58] PROBLEM - Disk space on mw61 is CRITICAL: DISK CRITICAL - free space: / 284 MB (3% inode=63%): /var/lib/ureadahead/debugfs 284 MB (3% inode=63%): [01:52:52] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 2.303 seconds [02:17:32] !log LocalisationUpdate completed (1.18) at Mon Feb 27 02:17:32 UTC 2012 [02:17:38] Logged the message, Master [02:25:13] is something wrong? [02:25:21] with bits.wikimedia [02:26:13] Waiting for bits.wikimedia ... [02:28:06] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:28:33] PROBLEM - Varnish HTTP bits on niobium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:29:23] * PiRSquared17 waits [02:29:45] PROBLEM - Varnish HTTP bits on arsenic is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:30:09] i'm sure they're aware [02:30:38] ok [02:31:19] they probably have some kind of beeper hooked up to the servers that pages them wheneever there's trouble [02:31:24] :P [02:31:42] It would depend on what the trouble is [02:31:52] it's just not loading [02:31:58] stuck on Waiting for bits.. [02:32:32] and when it finally does, there's no CSS [02:32:48] Right - the actual site is loading, the issue seems to be the css server [02:32:54] <^demon> All bits does is make your pages pretty. Use a text browser and you won't notice a difference ;-) [02:33:17] Huggle is working okay [02:33:20] ^demon: I am getting segfaults from w3m actually, which I find very interesting [02:34:02] !log LocalisationUpdate completed (1.19) at Mon Feb 27 02:34:02 UTC 2012 [02:34:04] Logged the message, Master [02:34:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.023 seconds [02:39:15] <[Haekchen]> http://bits.wikimedia.org/de.wikipedia.org/load.php?debug=false&lang=de&modules=ext.gadget.Personendaten%2CSuchfokus-Hauptseite%7Cjquery.checkboxShiftClick%2Ccookie%2CmakeCollapsible%2CmessageBox%2CmwPrototypes%2Cplaceholder%7Cmediawiki.action.watch.ajax%7Cmediawiki.language%2Cuser%2Cutil%7Cmediawiki.legacy.ajax%2Cmwsuggest%2Cwikibits%7Cmediawiki.page.ready&skin=monobook&version=20120227T021754Z&* gave me 503 ??? [02:40:19] <^demon> Yes, it's being looked at. [02:40:51] bits is unhappy, yes [02:44:09] RECOVERY - Varnish HTTP bits on arsenic is OK: HTTP OK HTTP/1.1 200 OK - 625 bytes in 0.267 seconds [02:44:45] RECOVERY - Varnish HTTP bits on niobium is OK: HTTP OK HTTP/1.1 200 OK - 627 bytes in 0.143 seconds [02:45:45] commons still seems to be having some problems [02:49:02] ^demon: Do you have time to deploy r112464 ? [02:49:09] please don't deploy right now [02:49:34] A revision to CodeReview was deployed an hour ago, with a bug in it that is fixed by r112464 [02:49:48] we're in the middle of an outage [02:49:53] again, don't deploy [02:50:14] k, I'm not away of any outage though, but ok in that case no rush :) [02:50:18] PROBLEM - Varnish HTTP bits on arsenic is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:04:15] RECOVERY - Varnish HTTP bits on arsenic is OK: HTTP OK HTTP/1.1 200 OK - 627 bytes in 0.062 seconds [03:05:00] Krinkle-away: also, it isn't very nice to deploy on a sunday ;) [03:05:17] Ryan_Lane: yeah [03:05:36] Ryan_Lane: I blame ^demon though, he suggested to deploy it since the bug annoyed him as well [03:05:44] and then we discovered there was a bug in it when it went live [03:05:49] minor thing though [03:05:50] I gave him crap about it too [03:05:56] k [03:06:15] its merged to 1.19wmf1 [03:06:58] Ryan_Lane: Outage you speak of is about bits? [03:07:08] yep [03:07:11] k [03:18:21] PROBLEM - MySQL disk space on db12 is CRITICAL: DISK CRITICAL - free space: / 287 MB (3% inode=90%): [03:19:33] PROBLEM - Disk space on db12 is CRITICAL: DISK CRITICAL - free space: / 287 MB (3% inode=90%): [03:34:05] PROBLEM - Disk space on srv249 is CRITICAL: DISK CRITICAL - free space: / 284 MB (3% inode=63%): /var/lib/ureadahead/debugfs 284 MB (3% inode=63%): [05:03:40] RECOVERY - Disk space on db12 is OK: DISK OK [05:03:49] RECOVERY - MySQL disk space on db12 is OK: DISK OK [05:10:17] Anyone with arcane knowledge about [[MediaWiki:Asksqltext]] and [[MediaWiki:Sqlislogged]] that were used (still used?) at enWP at some point in time. Can they be formally retired? Or do they need to hang around for hysterical/historical reasons? [05:18:49] the former belongs to AskSQL which is now a extension and I doubt is enabled on the cluster [05:20:31] *is not enabled on the cluster so that can be deleted [05:22:30] before I had shell access, I ran a poorly constructed query with that feature and brought down wikipedia for hours [05:23:13] I couldn't really believe at the time that it was my fault, and nobody was complaining about it being down because it was down so often back then [05:23:29] but I realised later when I learnt more about mysql that it was definitely my fault [05:25:34] there was no way to kill a query once it started [05:26:29] O_O [05:26:38] that must have been fun [05:27:45] OTRS 2012022510008857 has reported an issue with the links that google shows for the search "wikipedia": http://dpaste.com/hold/708360/ [05:28:01] anyone know how that works? do they use sitemaps? [05:28:09] and do we generate them? [05:28:18] * jeremyb is too tired to investigate [05:29:24] moving the ticket to tech issues. i can respond in the morning if no one else has by then. (the full text of the problem description is in the paste i linked. at a glance it seems to match my experience when i did the same search) [05:29:28] works for me [05:29:29] * jeremyb sleeps! [05:29:39] it links to http://en.wikipedia.org/wiki/English_language [05:29:50] huh [05:30:19] it might be location-dependent [05:32:00] maybe... [05:32:41] hmm it took me to ampersand... [05:33:04] ... [05:33:14] i currently see: Wikipedia -> [Wikipedia, How I Met Your Mother, English, Lady Gaga, Wikipedia:Wikipedia (which also links to the wrong place), Steve Jobs] [05:33:21] TimStarling: what do you see? [05:33:35] (when reading across first and then down) [05:35:01] can someone please fix https://meta.wikimedia.org/wiki/Main_Page?useskin=monobook ? [05:35:03] English, History Portal, Deaths in 2011, Contents [05:35:07] bye [05:35:59] TimStarling: you think we should ask google to investigate on their end? [05:36:18] I see Amperstand as number 5 on there [05:36:34] Bsadowski1: it says ampersand or links to ampersand? [05:36:43] er [05:37:12] anyway it's their fault, we don't generate those links [05:37:42] TimStarling: do we expose a sitemap of some kind? [05:37:59] not really [05:38:00] also, i thought maybe someone knew how google generates those [05:40:44] we used to generate site maps but I'm pretty sure they stopped using them a long time ago [05:41:44] * jeremyb is attempting to raise some googlers informally... [05:47:25] Bsadowski1: which one? [05:49:05] PiRSquaredBai: More information than just "fix that" would be helpful [05:56:13] TimStarling: a google guy has emailed a few other google guys and cc'd that ticket. FYI, in case you (and any others who might be interested) want to subscribe to the ticket [05:56:23] (for when google replies) [05:57:09] jeremyb: You're talking about the "sublinks" in Google search results? [05:57:22] They're not controlled by site admins. They're Google auto-detect magic. [05:57:27] Joan: idk what they're called. i've been calling them subordinate [05:57:45] Joan: i thought they were (sometimes) derived from sitemaps [05:57:55] http://support.google.com/webmasters/bin/answer.py?hl=en&answer=47334 [05:58:11] > At the moment, sitelinks are automated. We're always working to improve our sitelinks algorithms, and we may incorporate webmaster input in the future. [05:58:19] :-) [05:58:46] i don't necessarily see that language in conflict with mine [05:58:49] anyway, sleep [06:00:06] I provided a citation from a reliable source that definitively answers your question. [06:00:14] And then you quibble. [06:45:20] RECOVERY - Disk space on srv249 is OK: DISK OK [06:48:02] RECOVERY - Disk space on mw61 is OK: DISK OK [08:07:05] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 16.3681021053 (gt 8.0) [08:07:42] PROBLEM - udp2log processes on locke is CRITICAL: CRITICAL: filters absent: /a/squid/urjc.awk, [08:12:08] RECOVERY - udp2log processes on locke is OK: OK: all filters present [08:24:08] PROBLEM - udp2log processes on locke is CRITICAL: CRITICAL: filters absent: /a/squid/urjc.awk, [08:30:08] RECOVERY - udp2log processes on locke is OK: OK: all filters present [08:32:59] PROBLEM - Puppet freshness on mw1010 is CRITICAL: Puppet has not run in the last 10 hours [08:38:59] PROBLEM - Puppet freshness on mw1110 is CRITICAL: Puppet has not run in the last 10 hours [08:38:59] PROBLEM - Puppet freshness on mw1020 is CRITICAL: Puppet has not run in the last 10 hours [08:56:41] RECOVERY - Packetloss_Average on locke is OK: OK: packet_loss_average is 1.90896460177 [09:29:09] PROBLEM - Puppet freshness on lvs1002 is CRITICAL: Puppet has not run in the last 10 hours [10:21:35] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [10:23:56] !log hashar synchronized php-1.19/includes/Pager.php '(bug 34736) empty limit on special pages causes navigation issues' [10:24:01] Logged the message, Master [10:29:32] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [10:29:32] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [11:15:50] what does it mean when the API gives me a null replag? [11:15:58] https://it.wikiquote.org/w/api.php?action=query&meta=siteinfo&siprop=dbrepllag&sishowalldb [11:16:31] treat it as a problem. (rather than 0) [13:32:05] PROBLEM - Disk space on mw61 is CRITICAL: DISK CRITICAL - free space: / 284 MB (3% inode=63%): /var/lib/ureadahead/debugfs 284 MB (3% inode=63%): [14:19:56] RECOVERY - Disk space on mw61 is OK: DISK OK [14:29:09] I've got a user on en.wp who claims she can't log in to her account through an anon-only block. She claims to try to log in, and it gives her the account creation blocked page. Any idea what that might be? [14:36:17] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 10.8520634211 (gt 8.0) [14:43:10] RECOVERY - Packetloss_Average on locke is OK: OK: packet_loss_average is 3.23771982301 [14:49:01] New patchset: Hashar; "send Swift syslogs to their own file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2673 [14:57:29] I'm having an interesting problem; I requested a rename for SUL unification on frwiki, it was done, and [[User:Madman]] doesn't exist there anymore. But when I go to Special:MergeAccount it doesn't attach and when I go to frwiki it shows not logged in. It doesn't show as unattached anywhere, it just doesn't show up. What else could block unification? [15:03:17] AMadman, what's the new user name? [15:03:28] On fr? They renamed Madman to Madman_old. [15:03:47] Ah [15:03:50] Now there's no user named Madman, so I'm trying to add it to my global account. [15:03:54] AMadman, can't you login there? [15:04:24] Ohhh. [15:04:26] in other words if you are already logged in somewhere else and you go to fr wiki... aren't you automatically [15:04:26] My IP is blocked there. [15:04:30] woooops [15:04:53] (I'm proxying through my dedicated server for security reasons as usual.) [15:04:55] But that's fixable. [15:05:08] That makes sense; if account creation is blocked... :x [15:05:11] This explains it. :) Request unblock and retry [15:05:25] Yay! It's merged. [15:05:29] sweet! [15:05:33] ok [15:05:47] Thanks! (<-- fail) [15:05:54] Cheers! [15:07:04] FooBarMartijn, problem solved? [15:07:09] did you check the block number? [15:07:25] ugh, I missed the whole convo, thanks for the ping [15:07:44] actually nobody replied you :p [15:07:53] oh, in that case, my problem is not solved [15:07:54] haha [15:08:14] FooBarMartijn, I assume you checked the block number? [15:08:53] it came in through the mailinglist, but the useraccount is not blocked, and the IP only shows up in special:BlockList as an anon-only block [15:08:59] so that shouldn't be autoblock, right? [15:09:30] anyway, that shouldn't direct to the account creation blocked page, should it? [15:09:50] so what's the IP [15:10:05] could still be a global block [15:10:18] I don't know how it works from the user's perspective. [15:11:02] I'll check for that. I don't think our oh so useful privacy policy allows me to disclose the IP [15:11:22] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 8.13589707965 (gt 8.0) [15:11:56] not globally blocked either [15:12:26] well, if it was sent to a mailing list... and you don't link it to a person [15:13:18] well, the mailinglist unblock-en-l is presumed to be fairly private. It's not logged for exactly that reason. But then, if I don't link to a person [15:13:21] ok, so let's imagine there's a magical connection between concurrent problems: does the user actually have an account? :) [15:13:26] yes [15:13:34] or is him/her trying to autocreate one [15:14:09] I gave a direct link to the login page, instructed to fill it out, and manually press the login button [15:14:22] the user *claims* that it redirects to account creation blocked [15:14:47] on the one hand, I have the strong feeling that the problem is between the keyboard and the wall [15:15:05] hehe [15:15:22] on the other, how would the user get to the account creation blocked page, if she is positive that she manually clicked the login button [15:15:23] I don't know, if it's true you need a real expert, I can't help you :p [15:16:30] oh well [15:16:52] 'you're the only user in the world with this problem. Tough cookie, get a different hobby' [15:20:37] Ugh, I think I got it. [15:20:53] In case she got her username wrong in the form, it will try to autocreate [15:21:08] and in that case, she will get an ACB message [16:12:45] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 10.1901229204 (gt 8.0) [16:36:45] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 10.2623781579 (gt 8.0) [16:57:03] New patchset: Hashar; "modifying testfile again" [test/mediawiki/core2] (master) - https://gerrit.wikimedia.org/r/2791 [16:58:27] New review: jenkins-bot; "Build Started https://integration.mediawiki.org/ci/job/MediaWiki-GIT-Fetching/27/ (1/2)" [test/mediawiki/core2] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/2791 [16:58:28] New review: jenkins-bot; "Build Started https://integration.mediawiki.org/ci/job/MediaWiki-GIT-Fetching/39/ (2/2)" [test/mediawiki/core2] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/2791 [17:00:30] Can someone please fix https://meta.wikimedia.org/wiki/Stewards/Elections_2012?useskin=monobook ? [17:01:23] bits is still broken? [17:01:50] New patchset: RobH; "require unzip for blog server" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2792 [17:03:02] New review: RobH; "need unzip package to update blog plugins" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2792 [17:03:03] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2792 [17:04:39] New review: Hashar; "(no comment)" [test/mediawiki/core2] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2791 [17:04:39] Change merged: Hashar; [test/mediawiki/core2] (master) - https://gerrit.wikimedia.org/r/2791 [17:05:57] New patchset: Hashar; "add 'topic' feature" [test/mediawiki/core2] (master) - https://gerrit.wikimedia.org/r/2793 [17:06:58] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 10.3481768421 (gt 8.0) [17:07:07] New review: jenkins-bot; "Build Started https://integration.mediawiki.org/ci/job/MediaWiki-GIT-Fetching/40/ (1/2)" [test/mediawiki/core2] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/2793 [17:07:07] New review: jenkins-bot; "Build Started https://integration.mediawiki.org/ci/job/MediaWiki-GIT-Fetching/28/ (2/2)" [test/mediawiki/core2] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/2793 [17:07:07] New review: Hashar; "(no comment)" [test/mediawiki/core2] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2793 [17:07:08] Change merged: Hashar; [test/mediawiki/core2] (master) - https://gerrit.wikimedia.org/r/2793 [17:08:33] hmmm [17:08:33] Request: GET http://it.wikiquote.org/w/api.php?action=query&meta=siteinfo&siprop=dbrepllag&sishowalldb, from 91.198.174.56 via sq72.wikimedia.org (squid/2.7.STABLE9) to () [17:08:33] Error: ERR_SOCKET_FAILURE, errno (98) Address already in use at Mon, 27 Feb 2012 17:08:00 GMT [17:08:40] same here [17:08:59] ... and on meta [17:09:13] blub error blub [17:09:49] mediawikiwiki not responding [17:10:17] add simplewiki to that list [17:10:22] S3 ? [17:10:29] @info mediawikiwik [17:10:29] (the not responding wikis) [17:10:29] Krinkle: Unknown identifier (mediawikiwik) [17:10:30] @info mediawikiwiki [17:10:30] Krinkle: [mediawikiwiki: DEFAULT (s3)] db39: 10.0.6.49, db34: 10.0.6.44, db25: 10.0.6.35, db11: 10.0.6.21 [17:10:38] @info simplewiki [17:10:38] Krinkle: [simplewiki: DEFAULT (s3)] db39: 10.0.6.49, db34: 10.0.6.44, db25: 10.0.6.35, db11: 10.0.6.21 [17:10:42] indeed [17:10:45] @info meta [17:10:46] Krenair: Unknown identifier (meta) [17:10:58] @info metawiki [17:10:58] Saibo: [metawiki: s7] db37: 10.0.6.47, db16: 10.0.6.26, db18: 10.0.6.28, db26: 10.0.6.36 [17:11:02] still [17:11:03] @replag all [17:11:03] after a long wait got [17:11:03] <[Haekchen]> @info dewiki [17:11:20] hm.. [17:11:27] PROBLEM - Apache HTTP on mw22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:11:27] PROBLEM - Apache HTTP on srv199 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:11:27] PROBLEM - Apache HTTP on mw53 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:11:27] PROBLEM - Apache HTTP on mw29 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:11:27] PROBLEM - Apache HTTP on mw4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:11:28] PROBLEM - Apache HTTP on mw14 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:11:28] PROBLEM - Apache HTTP on srv267 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:11:29] PROBLEM - Apache HTTP on srv226 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:11:33] if the server isn't responding it can't get @replag [17:11:35] "Error 'Unknown column 'rev_sha1' in 'field list'' on query. Default database: 'testwiki'" [17:11:36] there we go ;) [17:11:43] it simply doesn't even run replication [17:11:57] the bot makes an API request, so it'll probably time out [17:12:03] PROBLEM - Apache HTTP on mw25 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:03] PROBLEM - Apache HTTP on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:03] PROBLEM - Apache HTTP on srv225 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:03] PROBLEM - Apache HTTP on mw48 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:03] PROBLEM - Apache HTTP on srv288 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:04] PROBLEM - Apache HTTP on mw19 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:04] PROBLEM - Apache HTTP on mw6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:05] PROBLEM - Apache HTTP on srv289 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:05] PROBLEM - Apache HTTP on srv194 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:06] PROBLEM - Apache HTTP on srv270 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:06] PROBLEM - Apache HTTP on srv261 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:07] PROBLEM - Apache HTTP on mw54 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:07] PROBLEM - Apache HTTP on mw26 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:08] PROBLEM - Apache HTTP on mw31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:08] PROBLEM - Apache HTTP on srv200 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:09] PROBLEM - Apache HTTP on srv233 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:09] PROBLEM - Apache HTTP on srv212 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:10] PROBLEM - Apache HTTP on srv276 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:10] PROBLEM - Apache HTTP on srv208 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:11] PROBLEM - Apache HTTP on srv283 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:11] PROBLEM - Apache HTTP on mw20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:11] yep [17:12:12] PROBLEM - Apache HTTP on srv244 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:18] Here it is. Now. [17:12:21] ouch [17:12:21] RECOVERY - Apache HTTP on mw53 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.028 second response time [17:12:22] PROBLEM - LVS HTTP on appservers.svc.pmtpa.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:22] PROBLEM - Apache HTTP on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:23] Krinkle: i hoped it had some special knowledge ;) [17:12:30] PROBLEM - Apache HTTP on mw30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:30] PROBLEM - Apache HTTP on mw36 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:30] PROBLEM - Apache HTTP on mw41 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:31] PROBLEM - Apache HTTP on mw47 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:31] PROBLEM - Apache HTTP on srv207 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:31] PROBLEM - Apache HTTP on srv213 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:31] PROBLEM - Apache HTTP on srv232 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:31] PROBLEM - Apache HTTP on srv238 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:32] Oh, seems like there's a problem. ;) [17:12:48] PROBLEM - Apache HTTP on srv243 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:48] PROBLEM - Apache HTTP on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:48] PROBLEM - Apache HTTP on mw37 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:48] PROBLEM - Apache HTTP on srv262 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:48] PROBLEM - Apache HTTP on srv275 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:48] PROBLEM - Apache HTTP on srv269 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:57] PROBLEM - Apache HTTP on srv201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:57] PROBLEM - Apache HTTP on srv227 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:58] PROBLEM - Apache HTTP on srv239 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:58] PROBLEM - Apache HTTP on srv245 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:58] PROBLEM - Apache HTTP on srv263 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:58] PROBLEM - Apache HTTP on srv282 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:01] Krenair: something like that [17:13:04] lol [17:13:06] PROBLEM - Apache HTTP on mw15 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:06] PROBLEM - Apache HTTP on mw27 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:06] PROBLEM - Apache HTTP on mw32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:06] PROBLEM - Apache HTTP on mw38 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:06] PROBLEM - Apache HTTP on mw43 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:07] PROBLEM - Apache HTTP on mw49 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:11] slave was stopped because schema change was in process [17:13:15] PROBLEM - Apache HTTP on srv195 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:15] PROBLEM - Apache HTTP on srv196 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:15] PROBLEM - Apache HTTP on srv202 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:15] PROBLEM - Apache HTTP on srv209 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:15] PROBLEM - Apache HTTP on srv228 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:16] PROBLEM - Apache HTTP on srv234 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:16] PROBLEM - Apache HTTP on srv240 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:17] PROBLEM - Apache HTTP on srv246 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:17] PROBLEM - Apache HTTP on srv258 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:18] PROBLEM - Apache HTTP on srv264 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:18] PROBLEM - Apache HTTP on srv271 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:19] PROBLEM - Apache HTTP on srv277 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:19] PROBLEM - Apache HTTP on srv285 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:20] lol [17:13:27] domas: but that's not supposed to bring the wiki down? [17:13:31] @info enwiki [17:13:33] PROBLEM - Apache HTTP on mw28 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:33] PROBLEM - Apache HTTP on mw33 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:33] PROBLEM - Apache HTTP on mw39 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:33] PROBLEM - Apache HTTP on mw7 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:33] PROBLEM - Apache HTTP on mw44 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:34] PROBLEM - Apache HTTP on mw5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:34] PROBLEM - Apache HTTP on mw56 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:35] PROBLEM - Apache HTTP on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:35] PROBLEM - Apache HTTP on srv190 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:36] PROBLEM - Apache HTTP on srv197 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:36] PROBLEM - Apache HTTP on srv203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:37] PROBLEM - Apache HTTP on srv210 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:37] PROBLEM - Apache HTTP on srv229 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:38] PROBLEM - Apache HTTP on srv235 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:38] PROBLEM - Apache HTTP on srv247 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:39] PROBLEM - Apache HTTP on srv259 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:39] PROBLEM - Apache HTTP on srv265 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:40] no, it started happening before, probably [17:13:40] PROBLEM - Apache HTTP on srv272 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:40] PROBLEM - Apache HTTP on srv286 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:51] PROBLEM - Apache HTTP on mw17 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:51] PROBLEM - Apache HTTP on mw34 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:51] PROBLEM - Apache HTTP on mw45 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:51] PROBLEM - Apache HTTP on mw16 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:51] PROBLEM - Apache HTTP on srv198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:56] nagios-wm: quiet [17:13:57] of course, it would be better to depool a box that is broken for months [17:13:57] domas: ah, schema change while server was already unhappy? [17:14:00] PROBLEM - Apache HTTP on srv230 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:00] PROBLEM - Apache HTTP on srv236 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:00] PROBLEM - Apache HTTP on srv242 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:00] PROBLEM - Apache HTTP on srv260 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:00] PROBLEM - Apache HTTP on srv273 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:00] PROBLEM - Frontend Squid HTTP on cp1018 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:01] no [17:14:06] this may not be related [17:14:09] New patchset: RobH; "hooper is no longer a blog server" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2794 [17:14:09] PROBLEM - Apache HTTP on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:09] k [17:14:10] PROBLEM - Apache HTTP on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:10] PROBLEM - Apache HTTP on mw18 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:10] PROBLEM - Apache HTTP on mw24 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:10] PROBLEM - Apache HTTP on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:10] PROBLEM - Apache HTTP on mw40 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:10] PROBLEM - Apache HTTP on mw46 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:11] PROBLEM - Apache HTTP on mw52 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:11] PROBLEM - Apache HTTP on mw58 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:18] PROBLEM - Apache HTTP on srv204 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:19] PROBLEM - Apache HTTP on srv205 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:19] PROBLEM - Apache HTTP on srv231 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:19] PROBLEM - Apache HTTP on srv237 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:19] PROBLEM - Apache HTTP on srv268 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:19] PROBLEM - Apache HTTP on srv274 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:19] PROBLEM - Apache HTTP on srv280 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:27] PROBLEM - Apache HTTP on srv211 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:36] midom cleared profiling data [17:15:07] schema change was long over [17:15:23] error reported when saving the page is Request: GET http://www.mediawiki.org/w/index.php?title=Talk:ResourceLoader/Default_modules/Archive&action=submit, from 91.198.174.36 via sq65.wikimedia.org (squid/2.7.STABLE9) to () Error: ERR_SOCKET_FAILURE, errno (98) Address already in use at Mon, 27 Feb 2012 17:07:57 GMT [17:15:27] New review: RobH; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2794 [17:15:27] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2794 [17:15:28] yeh, site is toast [17:15:41] domas: how can schema change possibly cause apaches to go down? [17:15:48] do we know if this is all clusters? [17:15:49] there was no schema change _now_ [17:15:55] it was long ago [17:15:58] vvv: not able to talk to slaves? [17:16:06] i.e. what should i change /topic s to? [17:16:19] equest: POST http://de.wikipedia.org/w/index.php?title=Informationsfreiheit&action=submit, from 208.80.152.84 via sq60.wikimedia.org (squid/2.7.STABLE9) to () [17:16:22] Error: ERR_CANNOT_FORWARD, errno [No Error] at Mon, 27 Feb 2012 17:16:08 GMT [17:16:24] PROBLEM - Router interfaces on br1-knams is CRITICAL: CRITICAL: No response from remote host 91.198.174.245 for 1.3.6.1.2.1.2.2.1.7 with snmp version 2 [17:16:42] now, bgp too? [17:17:06] this is not even in PHP [17:17:08] it's just ifAdminStatus from IF-MIB [17:17:11] @info enwiki [17:17:15] jeremyb: [enwiki: s1] db36: 10.0.6.46, db12: 10.0.6.22, db32: 10.0.6.42, db38: 10.0.6.48, db52: 10.0.6.62, db53: 10.0.6.63 [17:17:18] PROBLEM - Apache HTTP on mw55 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:24] woot [17:17:30] @replag all [17:17:36] DDOS or so [17:17:36] PROBLEM - Apache HTTP on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:36] PROBLEM - Apache HTTP on mw21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:45] PROBLEM - Apache HTTP on srv241 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:03] PROBLEM - Apache HTTP on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:03] PROBLEM - Apache HTTP on mw51 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:03] PROBLEM - Apache HTTP on mw57 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:03] RECOVERY - Apache HTTP on srv236 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 8.794 second response time [17:18:12] PROBLEM - Apache HTTP on srv287 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:21] PROBLEM - Apache HTTP on mw35 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:21] RECOVERY - Router interfaces on br1-knams is OK: OK: host 91.198.174.245, interfaces up: 10, down: 0, dormant: 0, excluded: 0, unused: 0 [17:18:39] PROBLEM - Apache HTTP on mw53 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:57] wat [17:18:57] RECOVERY - Apache HTTP on srv208 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 5.582 second response time [17:18:57] RECOVERY - Apache HTTP on srv283 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 7.301 second response time [17:19:06] RECOVERY - Apache HTTP on srv289 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 8.766 second response time [17:19:15] PROBLEM - Apache HTTP on mw42 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:42] RECOVERY - Apache HTTP on srv235 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 7.894 second response time [17:19:51] jeremyb: [s1] db36: 0s, db12: 0s, db32: 0s, db38: 0s, db52: 0s, db53: 0s; [s2] db13: 0s, db30: 0s, db24: 0s, db54: 0s; [s3] db39: 0s, db34: s, db25: 0s, db11: 0s [17:19:52] jeremyb: [s4] db22: 0s, db31: 0s, db33: 0s, db51: 0s; [s5] db45: 0s, db35: 0s, db44: 0s, db55: 0s; [s6] db47: 0s, db43: 0s, db46: 0s, db50: 0s; [s7] db37: 0s, db16: 0s, db18: 0s, db26: 0s [17:20:42] domas: were you talking about db34? (half schema change) [17:20:45] RECOVERY - Apache HTTP on srv226 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 6.934 second response time [17:20:45] RECOVERY - Apache HTTP on srv207 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 8.416 second response time [17:20:57] DDoS in progress [17:20:59] keep silent please [17:21:03] RECOVERY - Apache HTTP on srv263 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 8.181 second response time [17:21:12] RECOVERY - Apache HTTP on srv245 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 9.978 second response time [17:21:30] RECOVERY - Apache HTTP on srv264 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 7.047 second response time [17:21:39] RECOVERY - Apache HTTP on srv203 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.031 second response time [17:21:39] RECOVERY - Apache HTTP on srv190 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 2.656 second response time [17:21:39] RECOVERY - Apache HTTP on srv277 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 4.156 second response time [17:21:39] RECOVERY - Apache HTTP on srv241 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 4.356 second response time [17:21:39] RECOVERY - Apache HTTP on srv286 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 4.815 second response time [17:21:40] RECOVERY - Apache HTTP on srv209 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 6.663 second response time [17:21:48] RECOVERY - Apache HTTP on srv197 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 9.700 second response time [17:21:57] RECOVERY - Apache HTTP on srv198 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.825 second response time [17:22:06] RECOVERY - Apache HTTP on srv287 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 5.171 second response time [17:22:24] RECOVERY - Apache HTTP on srv280 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 3.655 second response time [17:22:25] RECOVERY - Apache HTTP on srv211 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 2.624 second response time [17:22:25] RECOVERY - Apache HTTP on srv274 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 4.842 second response time [17:22:25] RECOVERY - Apache HTTP on srv268 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 5.554 second response time [17:22:33] ... [17:22:33] RECOVERY - Apache HTTP on srv205 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 6.467 second response time [17:22:42] RECOVERY - Apache HTTP on srv232 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [17:22:42] RECOVERY - Apache HTTP on srv238 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.023 second response time [17:22:42] RECOVERY - Apache HTTP on srv194 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [17:22:42] RECOVERY - Apache HTTP on srv244 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 6.982 second response time [17:22:42] RECOVERY - Apache HTTP on srv213 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 7.289 second response time [17:22:51] RECOVERY - Apache HTTP on srv288 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [17:23:00] RECOVERY - Apache HTTP on srv262 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.020 second response time [17:23:00] RECOVERY - Apache HTTP on srv243 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.038 second response time [17:23:00] RECOVERY - Apache HTTP on srv270 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [17:23:00] RECOVERY - Apache HTTP on srv269 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.019 second response time [17:23:00] RECOVERY - Apache HTTP on srv261 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.467 second response time [17:23:01] RECOVERY - Apache HTTP on srv201 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.030 second response time [17:23:01] RECOVERY - Apache HTTP on srv227 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.022 second response time [17:23:01] RECOVERY - Apache HTTP on srv239 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.026 second response time [17:23:02] RECOVERY - Apache HTTP on srv276 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.018 second response time [17:23:02] RECOVERY - Apache HTTP on srv233 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [17:23:04] RECOVERY - Apache HTTP on srv282 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.025 second response time [17:23:04] RECOVERY - Apache HTTP on srv275 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 3.242 second response time [17:23:04] RECOVERY - Apache HTTP on srv225 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 5.849 second response time [17:23:18] RECOVERY - Apache HTTP on srv212 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.037 second response time [17:23:36] RECOVERY - Apache HTTP on srv246 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [17:23:36] RECOVERY - Apache HTTP on srv271 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.023 second response time [17:23:36] RECOVERY - Apache HTTP on srv258 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.022 second response time [17:23:36] RECOVERY - Apache HTTP on srv228 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [17:23:36] RECOVERY - Apache HTTP on srv196 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.023 second response time [17:23:37] RECOVERY - Apache HTTP on srv195 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.025 second response time [17:23:37] RECOVERY - Apache HTTP on srv234 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 4.130 second response time [17:23:45] RECOVERY - Apache HTTP on srv240 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [17:23:46] RECOVERY - Apache HTTP on srv265 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.020 second response time [17:23:46] RECOVERY - Apache HTTP on srv229 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [17:23:46] RECOVERY - Apache HTTP on srv210 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.023 second response time [17:23:46] RECOVERY - Apache HTTP on srv272 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.021 second response time [17:23:46] RECOVERY - Apache HTTP on srv247 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.030 second response time [17:23:46] RECOVERY - Apache HTTP on srv259 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.021 second response time [17:23:54] RECOVERY - Apache HTTP on srv285 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.018 second response time [17:23:54] RECOVERY - Apache HTTP on srv202 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.033 second response time [17:24:03] RECOVERY - Apache HTTP on mw22 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 2.188 second response time [17:24:03] RECOVERY - Apache HTTP on mw9 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 6.874 second response time [17:24:03] RECOVERY - Apache HTTP on mw16 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 7.585 second response time [17:24:12] RECOVERY - Apache HTTP on srv260 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.019 second response time [17:24:12] RECOVERY - Apache HTTP on srv230 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [17:24:13] RECOVERY - Apache HTTP on srv242 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [17:24:13] RECOVERY - Apache HTTP on srv273 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.034 second response time [17:24:13] RECOVERY - Apache HTTP on srv267 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [17:24:21] RECOVERY - Apache HTTP on mw12 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 3.757 second response time [17:24:21] RECOVERY - Apache HTTP on mw24 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 3.880 second response time [17:24:22] RECOVERY - Apache HTTP on mw35 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 3.992 second response time [17:24:30] RECOVERY - LVS HTTP on appservers.svc.pmtpa.wmnet is OK: HTTP OK HTTP/1.1 200 OK - 64220 bytes in 0.114 seconds [17:24:31] RECOVERY - Apache HTTP on srv199 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.026 second response time [17:24:31] RECOVERY - Apache HTTP on srv237 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.163 second response time [17:24:31] RECOVERY - Apache HTTP on srv204 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.173 second response time [17:24:39] RECOVERY - Apache HTTP on mw46 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 7.049 second response time [17:24:40] RECOVERY - Apache HTTP on mw13 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.025 second response time [17:24:40] RECOVERY - Apache HTTP on srv231 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 2.883 second response time [17:24:40] RECOVERY - Apache HTTP on mw30 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 4.244 second response time [17:24:40] RECOVERY - Apache HTTP on mw36 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 8.234 second response time [17:24:48] RECOVERY - Apache HTTP on srv200 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.868 second response time [17:24:58] RECOVERY - Apache HTTP on mw26 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.045 second response time [17:24:58] RECOVERY - Apache HTTP on mw37 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.362 second response time [17:24:58] RECOVERY - Apache HTTP on mw31 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.467 second response time [17:24:58] RECOVERY - Apache HTTP on mw6 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 2.805 second response time [17:24:58] RECOVERY - Apache HTTP on mw54 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 3.315 second response time [17:25:06] RECOVERY - Apache HTTP on mw10 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.562 second response time [17:25:06] RECOVERY - Apache HTTP on mw14 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 9.117 second response time [17:25:15] RECOVERY - Apache HTTP on mw38 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.031 second response time [17:25:15] RECOVERY - Apache HTTP on mw32 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.035 second response time [17:25:15] RECOVERY - Apache HTTP on mw15 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 3.021 second response time [17:25:15] RECOVERY - Apache HTTP on mw27 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 3.358 second response time [17:25:15] RECOVERY - Apache HTTP on mw55 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 3.736 second response time [17:25:16] RECOVERY - Apache HTTP on mw42 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 6.673 second response time [17:25:24] RECOVERY - Apache HTTP on mw49 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 2.954 second response time [17:25:24] RECOVERY - Apache HTTP on mw43 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 3.047 second response time [17:25:33] RECOVERY - Apache HTTP on mw33 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.896 second response time [17:25:34] RECOVERY - Apache HTTP on mw39 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 1.013 second response time [17:25:34] RECOVERY - Apache HTTP on mw11 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 1.887 second response time [17:25:34] RECOVERY - Apache HTTP on mw7 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 5.928 second response time [17:25:42] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.025 second response time [17:25:42] RECOVERY - Apache HTTP on mw28 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.037 second response time [17:25:42] RECOVERY - Apache HTTP on mw5 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.984 second response time [17:25:42] RECOVERY - Apache HTTP on mw44 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 4.284 second response time [17:25:51] RECOVERY - Apache HTTP on mw21 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.023 second response time [17:25:52] RECOVERY - Apache HTTP on mw56 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.028 second response time [17:26:00] RECOVERY - Apache HTTP on mw45 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.036 second response time [17:26:00] RECOVERY - Apache HTTP on mw17 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.028 second response time [17:26:00] RECOVERY - Apache HTTP on mw51 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.043 second response time [17:26:00] RECOVERY - Apache HTTP on mw29 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 1.165 second response time [17:26:00] RECOVERY - Apache HTTP on mw57 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 2.510 second response time [17:26:00] RECOVERY - Apache HTTP on mw4 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 5.202 second response time [17:26:09] RECOVERY - Packetloss_Average on locke is OK: OK: packet_loss_average is 3.02932451327 [17:26:09] RECOVERY - Apache HTTP on mw1 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.032 second response time [17:26:09] RECOVERY - Frontend Squid HTTP on cp1018 is OK: HTTP OK HTTP/1.0 200 OK - 27672 bytes in 0.161 seconds [17:26:18] RECOVERY - Apache HTTP on mw34 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 8.477 second response time [17:26:36] RECOVERY - Apache HTTP on mw58 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.022 second response time [17:26:36] RECOVERY - Apache HTTP on mw52 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.036 second response time [17:26:36] RECOVERY - Apache HTTP on mw40 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.163 second response time [17:26:36] RECOVERY - Apache HTTP on mw3 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.498 second response time [17:26:36] RECOVERY - Apache HTTP on mw25 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.038 second response time [17:26:37] RECOVERY - Apache HTTP on mw41 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.036 second response time [17:26:37] RECOVERY - Apache HTTP on mw47 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [17:26:38] RECOVERY - Apache HTTP on mw53 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.040 second response time [17:26:38] RECOVERY - Apache HTTP on mw19 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.852 second response time [17:26:39] RECOVERY - Apache HTTP on mw18 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 2.123 second response time [17:27:03] RECOVERY - Apache HTTP on mw2 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 3.529 second response time [17:27:12] RECOVERY - Apache HTTP on mw48 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.034 second response time [17:27:21] RECOVERY - Apache HTTP on mw20 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.176 second response time [17:27:56] nagios-wm: come on, go flood somewhere else [17:29:51] <[Haekchen]> Is that a complete restart? [17:35:26] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2673 [17:35:27] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2673 [17:36:32] [Haekchen]: of what? i don't think much was restarted. (maybe just dbbot-wm) [17:37:16] are we more or less back up? [17:37:22] Fluffernutter: should be [17:40:54] maplebed: http://en.wikipedia.org/wiki/Second_Battle_of_Bull_Run <-- two half thumbs [17:42:36] AaronSchulz: needs an outage to trigger? ;) [17:44:17] does that make a whole thumb? [17:44:38] no. they're not exact halves. just partial [17:44:38] ;) [17:44:45] yep [17:44:49] both seem to be more than half [17:45:13] plus they're from different fingers [17:52:06] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 9.15823333333 (gt 8.0) [17:53:09] PROBLEM - Disk space on srv192 is CRITICAL: DISK CRITICAL - free space: / 284 MB (3% inode=63%): /var/lib/ureadahead/debugfs 284 MB (3% inode=63%): [17:53:23] New review: Mark Bergsma; "Please just make that whole list one single file resource, with "recurse => remote"... and use mode ..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2782 [17:55:25] https://meta.wikimedia.org/wiki/?diff=prev&oldid=3513992&useskin=monobook looks "old-fashioned" to me still (no CSS/JS)... is this for everyone? [17:55:48] New review: Ottomata; "(no comment)" [analytics/reportcard] (andre/mobile); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2770 [17:56:10] New review: Ottomata; "(no comment)" [analytics/reportcard] (andre/mobile); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2770 [18:00:12] RECOVERY - MySQL Slave Running on db34 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [18:01:05] !log midom synchronized wmf-config/db.php [18:01:07] Logged the message, Master [18:01:36] New review: Aaron Schulz; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/2788 [18:03:11] * AaronSchulz sees domas sleuthing about [18:03:34] hehe [18:03:42] aaronschulz: had to deal with stupid ddos too [18:03:56] you're to big to sneak around! [18:03:59] *too [18:04:11] hehe [18:04:15] PROBLEM - MySQL Slave Delay on db34 is CRITICAL: CRIT replication delay 1102134 seconds [18:04:19] but I'm tiny! [18:10:42] does anyone else have that problem? or is it just me? [18:12:02] just you I guess [18:12:28] did you look? [18:13:46] PiRSquared17: what's old fashioned? [18:14:00] no CSS/JS [18:14:06] ... [18:14:10] like when bits.wikimedia is down [18:14:15] did you hard refresh? [18:14:25] looks quite normal to me [18:14:43] * PiRSquared17 tries [18:14:44] did you tell us which bits cluster you're using? [18:15:10] no, but I will if it's still brokwn [18:15:13] *e [18:15:17] hi jeremyb [18:15:24] hi Nikerabbit [18:16:15] looks ok now [18:16:17] thank you [18:17:15] !log nikerabbit synchronized php-1.18/languages/messages/ 'i18ndeploy r112497' [18:17:18] Logged the message, Master [18:18:13] !log nikerabbit synchronized php-1.18/extensions/Narayam/ 'i18ndeploy r112497' [18:18:15] Logged the message, Master [18:18:51] !log nikerabbit synchronized php-1.18/extensions/WebFonts/ 'i18ndeploy r112497' [18:18:54] Logged the message, Master [18:21:28] Nikerabbit: just for 1.18 not 1.19? [18:24:14] jeremyb: doing that now [18:24:43] jeremyb: although I run into one problem [18:24:55] ran [18:25:09] !log nikerabbit synchronized php-1.19/languages/messages/ 'i18ndeploy r112498' [18:25:12] Logged the message, Master [18:26:04] jeremyb: you don't happen to know about how to trigger l10n cache rebuilt? [18:26:26] !log nikerabbit synchronized php-1.19/extensions/Narayam/ 'i18ndeploy r112498' [18:26:28] Logged the message, Master [18:26:50] !log nikerabbit synchronized php-1.19/extensions/WebFonts/ 'i18ndeploy r112498' [18:26:53] Logged the message, Master [18:27:48] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 9.55816752212 (gt 8.0) [18:32:26] !log nikerabbit synchronized wmf-config/InitialiseSettings.php 'I18ndeploy config changes: bug 33423 bug 34591' [18:32:29] Logged the message, Master [18:34:37] PROBLEM - Puppet freshness on mw1010 is CRITICAL: Puppet has not run in the last 10 hours [18:40:37] PROBLEM - Puppet freshness on mw1110 is CRITICAL: Puppet has not run in the last 10 hours [18:40:37] PROBLEM - Puppet freshness on mw1020 is CRITICAL: Puppet has not run in the last 10 hours [18:41:33] Nikerabbit: not a clue [18:42:04] Nikerabbit: does that mean message cache? [18:43:37] Nikerabbit: http://wikitech.wikimedia.org/view/MessageCache [18:44:11] * jeremyb loves when stuff happens to be sufficiently documented before you need it ;) [18:45:59] jeremyb: that's not it [18:47:13] Nikerabbit: then what if not msg cache? [18:48:05] localisation cache [18:50:31] Nikerabbit: maybe http://wikitech.wikimedia.org/view/LocalisationUpdate is relevant? [18:50:55] jeremyb: that's yet another thing :) [18:51:09] Nikerabbit: it talks about caches... [18:51:25] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 13.3243047368 (gt 8.0) [18:54:26] Nikerabbit: https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/core.git;a=blob;f=includes/LocalisationCache.php#l546 ? [18:55:20] jeremyb: I know Tim and others talked about localisation cache few days ago, but I do not know if they made any changes [18:58:27] Nikerabbit: idk either... [19:07:08] New patchset: Asher; "s5 master changing from db45 to db35" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2795 [19:20:54] anyone familiar with our memcached cluster around? what's its total memory pool? [19:23:22] RECOVERY - Packetloss_Average on locke is OK: OK: packet_loss_average is 3.85951903509 [19:24:09] maxsem: 158G [19:24:53] so, 2G each [19:25:46] domas, thanks. so if we throw an extra gig per day into it, there will be no serious impact? [19:26:13] huh? [19:26:44] i think throw needs definition [19:27:08] $wgMemc->set() ;) [19:27:52] i still don't get the question [19:27:52] (/me analyses possible impact of a new feature he develops) [19:28:20] not much, no [19:28:25] "per day" is odd concept [19:28:27] t has ttl of few hours [19:28:52] hmm, ok [19:29:25] I said per day because I have per-day estimates [19:30:34] PROBLEM - Puppet freshness on lvs1002 is CRITICAL: Puppet has not run in the last 10 hours [19:31:19] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 10.3746984211 (gt 8.0) [19:42:29] hi guys. how much time does it take usually to establish a new wiki? [19:44:57] wizardist: a couple minutes? you should be asking #wikimedia if you want *approval* to do so. unless it's a special wiki [19:45:38] nope, the wiki was approved 16 days ago. I just can't wait anymore :)) [19:46:05] wizardist: https://meta.wikimedia.org/wiki/Proposals_for_new_projects https://meta.wikimedia.org/wiki/Requests_for_new_languages [19:46:10] wizardist: which one? [19:46:16] 34351 [19:46:27] !b 34351 [19:46:27] https://bugzilla.wikimedia.org/show_bug.cgi?id=34351 [19:47:43] wizardist: i guess a straightforward bug. filer is langcom iirc [19:47:45] i'm just curious, i don't want to drive anyone on. [19:48:20] wizardist: now you just need someone willing and free to do it [19:48:22] ;) [19:48:57] Yes, Milosh was a bit away for some time. [19:49:53] I think he comes with 2 L's [19:50:05] yeah [19:54:06] wizardist: We're in the process of upgrading the wikis to 1.19 one by one, so I could imagine wiki creations being on hold until that's done (the last deployment, enwiki, is on Wednesday) [19:54:20] RoanKattouw_away: yes, I was thinking of that a week ago [19:54:36] unreasonable to set a new wiki and then upgrade it the day after :) [19:59:08] btw, should have wikisource.org been already updated? [19:59:36] wizardist: yes [19:59:49] seems like it is not yet [20:00:09] wizardist: maybe they rolled it back.... let me check SAL [20:00:29] wizardist: you mean oldwikisource, right? [20:00:34] yep! [20:01:12] https://en.wikisource.org/wiki/Special:Version is, but, yeah ows isn't [20:01:34] SAL doesn't say anything except all have been changed [20:01:47] Reedy: you forgot oldwikisource [20:01:59] heh [20:02:50] reedy@fenari:/home/wikipedia/common$ grep oldwikisource *.dblist [20:02:50] reedy@fenari:/home/wikipedia/common$ [20:03:03] um... ? [20:03:30] :) [20:04:05] What dbname actually is it? [20:04:43] reedy, I don't actually know [20:04:49] and I'm stepping away [20:04:50] heh [20:05:51] It's not oldwikisource... [20:05:52] just wikisource? [20:07:37] oldwikisource is used in import lists [20:08:02] When I check there the database name is sourceswiki (if that's any help). [20:09:50] Cheers [20:09:56] I knew it was a weird exception [20:09:59] * Reedy grumbles [20:10:04] np [20:10:17] hexmode: XXwikisource vs sourceswiki [20:10:27] :D [20:10:55] I should probably though up a maintenance notice first if I'm gonna do it [20:11:05] robla: mind if I push sourceswiki across to 1.19wmf1? [20:11:42] whu? [20:11:48] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 9.57119877193 (gt 8.0) [20:12:15] Reedy: what's sourceswiki? [20:12:26] https://wikisource.org [20:13:14] * robla looks around that wiki a little bit [20:13:47] IIRC it was originally a multilingual project, but then we did them seperately [20:13:52] oh...that isn't just a landing page wiki [20:14:04] wikipedia.dblist:sourceswiki [20:14:10] Cause that's obvious [20:16:54] Reedy: go for it [20:18:07] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Put oldwikisource/sourceswiki on 1.19wmf1 [20:18:09] Logged the message, Master [20:18:15] hello :) [20:22:24] !log hashar synchronized php-1.19/extensions/CodeReview/backend/DiffHighlighter.php 'Special:CodeReview fix HTML entities showing up in diff output r112459 r112464' [20:22:27] Logged the message, Master [20:22:54] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [20:25:37] !log asher synchronized wmf-config/db.php 'swapping s5 master to db35, setting read-only' [20:25:39] Logged the message, Master [20:27:16] !log asher synchronized wmf-config/db.php 'done s5 master swap' [20:27:18] Logged the message, Master [20:30:14] RECOVERY - Packetloss_Average on locke is OK: OK: packet_loss_average is 2.4255640708 [20:31:52] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [20:31:53] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [20:32:37] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2795 [20:32:37] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2795 [20:33:20] PROBLEM - Disk space on srv249 is CRITICAL: DISK CRITICAL - free space: / 284 MB (3% inode=63%): /var/lib/ureadahead/debugfs 284 MB (3% inode=63%): [20:34:37] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2746 [20:34:39] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2746 [20:34:55] PROBLEM - MySQL Replication Heartbeat on db1021 is CRITICAL: CRIT replication delay 513 seconds [20:35:05] PROBLEM - MySQL Replication Heartbeat on db35 is CRITICAL: CRIT replication delay 517 seconds [20:35:05] PROBLEM - MySQL Replication Heartbeat on db44 is CRITICAL: CRIT replication delay 518 seconds [20:35:19] PROBLEM - MySQL Replication Heartbeat on db55 is CRITICAL: CRIT replication delay 529 seconds [20:37:37] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2748 [20:37:38] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2748 [20:38:01] RECOVERY - MySQL Replication Heartbeat on db1021 is OK: OK replication delay seconds [20:38:01] RECOVERY - MySQL Replication Heartbeat on db35 is OK: OK replication delay seconds [20:38:09] RECOVERY - MySQL Replication Heartbeat on db44 is OK: OK replication delay seconds [20:38:18] RECOVERY - MySQL Replication Heartbeat on db55 is OK: OK replication delay 0 seconds [20:39:39] RECOVERY - MySQL Recent Restart on db1006 is OK: OK seconds since restart [20:39:48] RECOVERY - RAID on db1006 is OK: OK: State is Optimal, checked 2 logical device(s) [20:39:48] RECOVERY - Host db1006 is UP: PING OK - Packet loss = 0%, RTA = 31.11 ms [20:39:57] RECOVERY - SSH on db1006 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [20:39:58] RECOVERY - MySQL Replication Heartbeat on db1006 is OK: OK replication delay seconds [20:40:06] RECOVERY - MySQL Slave Delay on db1006 is OK: OK replication delay seconds [20:40:07] RECOVERY - DPKG on db1006 is OK: All packages OK [20:40:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:40:42] RECOVERY - MySQL Slave Running on db1006 is OK: OK replication [20:40:42] RECOVERY - Disk space on db1006 is OK: DISK OK [20:40:51] RECOVERY - Full LVS Snapshot on db1006 is OK: OK no full LVM snapshot volumes [20:40:52] RECOVERY - MySQL disk space on db1006 is OK: DISK OK [20:41:04] !log hashar synchronized php-1.19/extensions/CodeReview/backend/DiffHighlighter.php 'Special:CodeReview merge r112513 fix bug 27375' [20:41:06] Logged the message, Master [20:41:18] RECOVERY - MySQL Idle Transactions on db1006 is OK: OK longest blocking idle transaction sleeps for seconds [20:42:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 4.031 seconds [20:43:31] Any clue if http://www.infodisiac.com/Wikipedia/ScanMail/_PowerPosters.html is up to date? [20:43:39] (was linked from stats.wikimedia.org) [20:43:53] oh nvm says generated today, I guess it is :O [20:45:06] New patchset: Siebrand; "Break long line." [mediawiki/core] (master) - https://gerrit.wikimedia.org/r/2796 [20:45:08] Snowolf, yes, that's updated very frequently [20:45:26] Yeah I realized it a bit late :D [20:45:33] * siebrand looks innocent... [20:45:52] Pushing that only took 3+ minutes. [20:46:06] PROBLEM - MySQL Slave Delay on db1006 is CRITICAL: CRIT replication delay 646809 seconds [20:46:06] PROBLEM - MySQL Replication Heartbeat on db1006 is CRITICAL: CRIT replication delay 646808 seconds [20:46:52] I'm getting the following error: "Request: POST http://pt.wikibooks.org/w/index.php?title=Teoria_de_n%C3%BAmeros/Equa%C3%A7%C3%B5es_diofantinas&action=submit, from 208.80.154.133 via cp1020.eqiad.wmnet (squid/2.7.STABLE9) to 10.64.0.134 (10.64.0.134) Error: ERR_READ_TIMEOUT, errno [No Error] at Mon, 27 Feb 2012 20:46:24 GMT" [20:47:05] after I save edits such as this: [https://pt.wikibooks.org/w/index.php?diff=prev&oldid=233856] [20:47:22] is this expected? [20:47:39] siebrand: have you managed to install git-review ? [20:47:49] hashar: doesn't exist for Windows? [20:48:02] given it is a python script, that might just work [20:48:16] helderwiki: but it saves fine? [20:48:27] hashar: I did the manual install, with a little magic, i think (added the alias to the global config file, instead of the project config, because I expect to get a lot of repos that use it...) [20:48:28] it seems so [20:48:36] Reedy: ^ [20:48:45] Yeah [20:48:56] It's not a small diff, but it isn't very large either [20:48:57] hashar: I need some confirmation that my patch arrived as expected, though. [20:49:05] siebrand: don't forget you can create a local repo which references over repos (as submodules) [20:49:07] Reedy: but it happened four times (so far) [20:49:16] Yeah, 22k page is small too [20:49:19] siebrand: patch is at https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/core.git;a=blobdiff;f=RELEASE-NOTES-1.20;h=2aec7e158f50359e32178f9c30df606baa150705;hp=7c25fefc1c5039d184cb7c6ac6fe3db3e62473af;hb=fee30c37b2aed3b6602ee2c01ebd87df49bc56b8;hpb=14b767762a7a08f5c17692561458a4390d043916 [20:49:21] hashar: I committed using TortoiseGIT and pushed from commandline. [20:49:25] (gonna love long URls) [20:50:16] Did someone just kill gerrit? [20:50:22] (poor gerrit) [20:50:28] well it seems slow [20:50:36] ^demon|away is upset that I was able to push to core. [20:50:55] helderwiki: i just test edited and reverted myself on that page, but that rally doesn't say much [20:51:02] Is there a repo yet that contains all extensions? [20:51:09] <^demon|away> siebrand: So I sabotaged gerrit for everyone ;-) [20:51:19] <^demon|away> siebrand: Not just yet, will be soon. [20:51:21] ^demon|away: what is the mediawiki/core repo ? [20:51:30] ^demon|away: k. [20:51:30] Reedy: I'll probably cause it again in a sec.. [20:51:32] <^demon|away> siebrand: It *will* be mediawiki/extensions.git [20:51:37] <^demon|away> hashar: phase3. [20:51:41] ^demon|away: sweet. [20:51:43] Reedy: done [20:51:50] Request: POST http://pt.wikibooks.org/w/index.php?title=Teoria_de_n%C3%BAmeros/Congru%C3%AAncias&action=submit, from 208.80.154.133 via cp1018.eqiad.wmnet (squid/2.7.STABLE9) to 10.64.0.132 (10.64.0.132) Error: ERR_READ_TIMEOUT, errno [No Error] at Mon, 27 Feb 2012 20:51:34 GMT [20:51:52] ^demon|away: oh the final one so ? [20:51:57] <^demon|away> Not yet, no. [20:52:00] hashar: https://gerrit.wikimedia.org/r/#change,2796 looks shorter than what you pasted? [20:52:09] <^demon|away> It's the borked copy from last week, but siebrand submitted to it :p [20:52:19] helderwiki: that's 4 different eqiad squids... Where are you in the world? [20:52:23] Just checking: there will be a git/gerrit hangout by Sumana in an hour right? [20:52:32] siebrand: yeah your link is Gerrit change, my link is the related gitweb link [20:52:32] Sao Paulo, Brazil [20:52:43] <^demon|away> siebrand: Ahaha, it's because you're in 'wmf' I believe. Mortals can't push. [20:52:47] * siebrand wants his CR back. [20:53:07] Reedy: each of these edits caused the same kind of error: https://pt.wikibooks.org/w/index.php?title=Especial:Contribui%C3%A7%C3%B5es/Helder.wiki&dir=prev&offset=20120227202848&limit=5&target=Helder.wiki [20:54:14] <^demon|away> hashar: Although, we could drop the core.git part and just do mediawiki.git, with extensions in mediawiki/extensions/foo.git [20:54:22] <^demon|away> The core part is kinda superfluous. [20:54:43] New patchset: QChris; "Migration to upstream git" [operations/dumps/test] (master) - https://gerrit.wikimedia.org/r/2797 [20:55:07] ^demon|away: I am in WMF? https://gerrit.wikimedia.org/r/#settings,group-memberships only shows anon and reg. [20:55:22] <^demon|away> siebrand: https://gerrit.wikimedia.org/r/#admin,group,6 ? [20:55:39] <^demon|away> It's an auto-ldap group. [20:55:50] ^demon|away: sure. But how can I see that *I* am in that group. I guess I can't? [20:55:53] ^demon|away: I still have to find out a good way to cleanup the remaining objects [20:56:03] ^demon|away: err the video objects [20:56:09] <^demon|away> I don't think it's that big a deal. [20:56:12] <^demon|away> We'll live. [20:56:14] ^demon|away: haven't took the time to seriously look at them [20:56:36] <^demon|away> siebrand: Hrm, well I can't see any users in it either, since it's ldap. I'm just guessing. [20:56:55] ^demon|away: sounds plausible to use a myth busters term... [20:57:07] :) [20:57:41] <^demon|away> siebrand: Ah, checking ldap it looks like you're not. So it's probably my fault on the permissions. [20:59:14] New patchset: Ottomata; "Made Dygraphs loader smarter. Observations now will not record instances of a trait_set if any of the trait properties are None. This allows transform callbacks to reject a trait based on its value." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2798 [20:59:47] !log asher synchronized wmf-config/db.php 'swapping s6 master to db43, setting read-only' [20:59:50] Logged the message, Master [21:01:03] !log asher synchronized wmf-config/db.php 'done s6 master swap' [21:01:05] Logged the message, Master [21:02:09] PROBLEM - Disk space on srv248 is CRITICAL: DISK CRITICAL - free space: / 283 MB (3% inode=63%): /var/lib/ureadahead/debugfs 283 MB (3% inode=63%): [21:03:46] New patchset: Asher; "new s6 master" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2799 [21:04:27] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2799 [21:04:29] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2799 [21:05:10] New review: Siebrand; "Blah." [mediawiki/core] (master) C: 1; - https://gerrit.wikimedia.org/r/2796 [21:06:03] PROBLEM - MySQL Replication Heartbeat on db50 is CRITICAL: CRIT replication delay 333 seconds [21:06:12] PROBLEM - MySQL Replication Heartbeat on db43 is CRITICAL: CRIT replication delay 340 seconds [21:06:23] New patchset: Ottomata; "Adding page_views_pipeline.py." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2800 [21:06:38] New patchset: Bhartshorne; "removing aotto from stat1 so it doesn't barf when it hits the account ensure => absent" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2801 [21:06:48] PROBLEM - MySQL Replication Heartbeat on db46 is CRITICAL: CRIT replication delay 379 seconds [21:07:15] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2801 [21:07:15] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2801 [21:07:18] New review: Diederik; "OK." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2800 [21:07:47] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2798 [21:07:48] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2800 [21:07:48] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2798 [21:08:54] RECOVERY - MySQL Replication Heartbeat on db46 is OK: OK replication delay seconds [21:09:30] RECOVERY - MySQL Replication Heartbeat on db1040 is OK: OK replication delay seconds [21:09:57] RECOVERY - MySQL Replication Heartbeat on db1006 is OK: OK replication delay seconds [21:10:01] New patchset: Bhartshorne; "Revert "removing aotto from stat1 so it doesn't barf when it hits the account ensure => absent"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2802 [21:10:06] RECOVERY - MySQL Replication Heartbeat on db50 is OK: OK replication delay seconds [21:10:06] RECOVERY - MySQL Replication Heartbeat on db43 is OK: OK replication delay 0 seconds [21:10:06] RECOVERY - MySQL Replication Heartbeat on db1022 is OK: OK replication delay seconds [21:15:19] New patchset: Asher; "pt-heartbeat should use REPLACE instead of UPDATE" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2803 [21:15:25] !log nikerabbit synchronized php-1.18/resources/startup.js 'touch' [21:15:28] Logged the message, Master [21:15:42] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2802 [21:15:42] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2802 [21:15:47] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2803 [21:15:48] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2803 [21:16:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:20:09] RECOVERY - Disk space on srv248 is OK: DISK OK [21:21:34] Change merged: QChris; [operations/dumps/test] (master) - https://gerrit.wikimedia.org/r/2797 [21:22:33] RECOVERY - Disk space on srv249 is OK: DISK OK [21:24:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.021 seconds [21:29:45] New patchset: Bhartshorne; "giving andrew a different real name so his two user accounts don't conflict." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2804 [21:30:53] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2804 [21:30:54] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2804 [21:49:37] !log catrope synchronized php-1.19/extensions/ProofreadPage/proofread.js 'r112522' [21:49:39] Logged the message, Master [21:52:46] !log catrope synchronized php-1.19/extensions/ProofreadPage/proofread.js 'r112522' [21:52:49] Logged the message, Master [21:55:45] !log asher synchronized wmf-config/db.php 'swapping s1 enwiki master to db38, setting read-only' [21:55:47] Logged the message, Master [21:56:45] !log asher synchronized wmf-config/db.php 'done s1 switch' [21:56:48] Logged the message, Master [21:57:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:02] New patchset: Asher; "s1 master swap" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2805 [21:59:42] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2805 [21:59:43] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2805 [22:00:54] PROBLEM - MySQL Replication Heartbeat on db12 is CRITICAL: CRIT replication delay 272 seconds [22:00:54] PROBLEM - MySQL Replication Heartbeat on db1033 is CRITICAL: CRIT replication delay 274 seconds [22:02:51] RECOVERY - MySQL Replication Heartbeat on db12 is OK: OK replication delay 0 seconds [22:03:00] RECOVERY - MySQL Replication Heartbeat on db1033 is OK: OK replication delay 0 seconds [22:03:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 7.368 seconds [22:19:30] RECOVERY - Host cadmium is UP: PING OK - Packet loss = 0%, RTA = 31.07 ms [22:22:21] PROBLEM - MySQL Slave Delay on db45 is CRITICAL: CRIT replication delay 284 seconds [22:23:51] PROBLEM - MySQL Replication Heartbeat on db45 is CRITICAL: CRIT replication delay 375 seconds [22:24:42] New patchset: Siebrand; "Add magic word translations for Dutch." [test/mediawiki/extensions/examples] (master) - https://gerrit.wikimedia.org/r/2806 [22:26:25] !log reedy synchronized php-1.19/extensions/UploadWizard/UploadWizard.i18n.php 'r112531' [22:26:28] Logged the message, Master [22:29:23] !log asher synchronized wmf-config/db.php 'switching s4 master to db31, setting read-only' [22:29:26] Logged the message, Master [22:29:44] New patchset: Asher; "switching s4 master to db31" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2807 [22:30:51] !log asher synchronized wmf-config/db.php 'done s4 switch' [22:30:54] Logged the message, Master [22:31:38] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2807 [22:31:39] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2807 [22:33:22] !log reedy synchronized php-1.19/includes/ 'r112532' [22:33:24] Logged the message, Master [22:39:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:23] New patchset: Nikerabbit; "Breaking stuff" [test/mediawiki/extensions/examples] (master) - https://gerrit.wikimedia.org/r/2808 [22:43:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 6.358 seconds [22:51:00] New review: Hashar; "You are breaking stuff! :-D" [test/mediawiki/extensions/examples] (master) C: -1; - https://gerrit.wikimedia.org/r/2808 [22:54:58] !log reedy synchronized php-1.19/includes/MessageBlobStore.php 'r112536' [22:55:01] Logged the message, Master [22:56:54] Change abandoned: Siebrand; "This sux." [test/mediawiki/extensions/examples] (master) - https://gerrit.wikimedia.org/r/2806 [22:57:16] hashar: yay [22:58:16] Change restored: Hashar; "Restoring. Just amend your change!" [test/mediawiki/extensions/examples] (master) - https://gerrit.wikimedia.org/r/2806 [22:58:55] New review: Nikerabbit; "(no comment)" [test/mediawiki/extensions/examples] (master) C: 1; - https://gerrit.wikimedia.org/r/2806 [22:59:20] Nikerabbit: siebrand: if you have any git question after tonight training, be bold and ask me tomorrow morning :-) [22:59:40] will try to answer them or at least find out the answer with you [22:59:54] hashar: clearly lint check does not catch syntax errors [23:00:03] hashar: Im mostly interested in the review workflow. [23:00:15] Nikerabbit: there is no lint check right now :/ [23:00:16] hashar: I guess I get the branches and commits… [23:00:28] hashar: my core change *was* lint-ed [23:00:34] ohhh [23:00:53] maybe an internal lint check by gerrit [23:01:06] hashar: and yet I get email about that? [23:01:32] hashar: yep. Also on the examples repo: SimpleAntiSpam [23:02:19] New review: Varnent; "It's possible this change will bring about the end of humankind as we know it..." [test/mediawiki/extensions/examples] (master) C: -1; - https://gerrit.wikimedia.org/r/2808 [23:03:32] New review: Siebrand; "Stubborn me. I'll approve anyway." [test/mediawiki/extensions/examples] (master) C: 1; - https://gerrit.wikimedia.org/r/2808 [23:03:40] Nikerabbit: siebrand: ahh it is linted!!! [23:03:44] by puppet :-) [23:03:49] New review: Siebrand; "Stubborn me. I'll approve anyway." [test/mediawiki/extensions/examples] (master) C: 2; - https://gerrit.wikimedia.org/r/2808 [23:03:51] New review: Varnent; "Maybe this is terrible after all" [test/mediawiki/extensions/examples] (master) C: -1; - https://gerrit.wikimedia.org/r/2808 [23:03:53] Change merged: Varnent; [test/mediawiki/extensions/examples] (master) - https://gerrit.wikimedia.org/r/2808 [23:03:57] so your changes are safe according to: puppet parser validate [23:04:25] the hooks are in operations/puppet repo in /files/gerrit/hooks/ [23:08:21] TimStarling: so bug 34755 is a result of the "files with wrong sha1 bug" [23:08:30] nothing separate [23:08:48] right [23:08:54] is anyone able to tell me the use of [[MediaWiki:History copyright]] and how we find out whether it is used? the mw.org manual is of no help [23:09:49] New patchset: Lcarr; "Commenting out dual definitions" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2809 [23:10:05] * robla put up site notice on nlwiki and plwiki [23:10:14] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2809 [23:10:47] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2809 [23:10:47] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2809 [23:14:46] Reedy: TimStarling: AaronSchulz: we ready to deploy to one of nlwiki or plwiki? [23:14:57] (and which one should we do first?) [23:15:19] * robla gets out coin to flip [23:15:31] Is Roan, siebrand and Krinkle about? [23:15:37] If so, NL might be a good choice [23:15:38] I am [23:15:39] *are [23:15:42] * AaronSchulz looks at Roan [23:15:50] he seems alive and about [23:16:06] I'm ready [23:16:15] I'm here [23:16:17] alright, let's just do nlwiki now [23:16:45] binasher: ^ [23:17:00] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: nlwiki to 1.19 [23:17:02] Logged the message, Master [23:18:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:19:36] robla: RoanKattouw user scripts appear to be not loading on nl.wikipedia [23:19:38] at least not for me [23:19:40] checking out now [23:19:49] my common.js [23:19:54] hrmph [23:19:58] Uncaught TypeError: Object # has no method 'anonymous' [23:20:00] aha [23:20:04] Yeah [23:20:10] Someone's using mw.user with an undeclared dependency [23:20:47] site script [23:21:14] function loggedOutTalkPage() { [23:21:14] if ( mw.user.anonymous() ) { [23:21:18] $(loggedOutTalkPage); [23:21:43] ruwikisource was using $j [23:21:55] RoanKattouw: Why would that be a problem ? [23:22:03] I... don't know. But it was [23:23:05] RoanKattouw: If that beaks, we've got a bigger problem [23:23:13] $j = jQuery in mediawiki.js, right ? [23:23:19] PHP Warning: array_merge() [function.array-merge]: Argument #2 is not an array in /usr/local/apache/common-local/php-1.19/includes/api/ApiQueryLogEvents.php on line 248 [23:23:21] / Alias $j to jQuery for backwards compatibility [23:23:22] window.$j = jQuery; [23:23:25] yeah [23:23:38] Hmm [23:23:54] * chrismcmahon starts over looking for the error [23:24:07] Well whatever was wrong with that script, it's fixed now [23:24:16] https://ru.wikisource.org/w/index.php?title=MediaWiki%3AEditpage.js&diff=689433&oldid=683473 [23:24:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 3.959 seconds [23:25:22] nlwiki issue solved [23:25:46] We kind of overlooked a changed that should have been merged in 1.18/1.19wmf1 for Narayam. 2 input methods stayed at beta, where they shouldn't have, so now they're not available. Is there any objection to them being merged into those branches now? (given the current deployment activity) [23:26:06] RoanKattouw: Weird though, that TypeError in the