[00:31:25] !ops I am pleased to announce that meepsheep will release all IRC logs for #wikipedia-en-help #wikipedia-en #wikipedia #wikimedia-ops all starting jan 25 with full 24 hours logs if derp is not unbanned. [01:41:20] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 212 seconds [01:42:59] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 279 seconds [01:49:44] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 682s [01:54:32] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 10 seconds [01:57:23] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 0 seconds [01:58:44] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 22s [02:21:44] !log LocalisationUpdate completed (1.20wmf8) at Wed Aug 8 02:21:44 UTC 2012 [02:21:56] Logged the message, Master [02:40:06] !log LocalisationUpdate completed (1.20wmf9) at Wed Aug 8 02:40:06 UTC 2012 [02:40:17] Logged the message, Master [03:11:30] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [03:25:27] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [03:25:27] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [03:25:27] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [04:02:30] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [04:53:29] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [05:24:32] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [05:37:27] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [06:00:33] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [06:30:51] PROBLEM - SSH on labstore3 is CRITICAL: Server answer: [06:32:12] RECOVERY - SSH on labstore3 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [06:40:33] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [06:55:33] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [07:10:32] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [07:25:32] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [07:27:29] RECOVERY - Puppet freshness on srv281 is OK: puppet ran at Wed Aug 8 07:27:19 UTC 2012 [07:35:47] when I try to go to sv.wikipedia.org, my browser (Firefox) redirects to sv.wikipedsv.wikipedia.org (which doesn't resolve in DNS, naturally) [07:36:49] but no.wikipedia.org works fine [07:37:31] After you unsing any browser extensions that may interfer with your browsing experience? [07:37:42] HTTPS Everywhere might be a example of one [07:37:57] nope [07:38:08] now it works, as if it was just a temporary error [07:38:14] very odd [08:30:48] <_et> anybody here involved in setting up swift with swauth? [08:40:33] PROBLEM - Puppet freshness on virt0 is CRITICAL: Puppet has not run in the last 10 hours [08:41:27] Hello. [08:54:05] Hello Dereckson. [08:54:30] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [08:54:30] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [10:00:38] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [10:07:38] mutante: are you working on this? http://wikitech.wikimedia.org/view/Special:RecentChanges [10:29:53] https://www.mediawiki.org/w/index.php?oldid=570066&rcid=646932 <-- an IP can create it but nobody can edit nor delete [10:33:27] Error loading data from server: timeout. Would you like to retry? < Oh how I love javascript alerts [10:34:38] maurelio: I can delete it ;) I just did [10:35:03] Oh. Probably I need ve-edit for that. [10:36:38] "Edit pages in the VisualEditor namespace with the original wikitext editor instead of the visual editor (ve-edit)" what the... [10:36:47] why is that even a right... [10:37:17] I just have the 'delete' right via the global steward flag and looks it's not enough ;) [13:12:29] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [13:26:36] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [13:26:36] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [13:26:36] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [14:03:30] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [14:43:06] !log reedy synchronized wmf-config/ [14:43:15] Logged the message, Master [15:21:20] !log catrope synchronized php-1.20wmf9/extensions/VisualEditor/.git 'Sync .git metadata so Special:Version is accurate' [15:21:30] Logged the message, Master [15:25:32] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [15:38:35] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [16:01:33] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [16:18:30] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Puppet has not run in the last 10 hours [16:41:36] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [16:45:21] !log preilly synchronized php-1.20wmf8/extensions/MobileFrontend 'update for zero' [16:45:31] Logged the message, Master [16:46:11] !log preilly synchronized php-1.20wmf9/extensions/MobileFrontend 'update for zero' [16:46:20] Logged the message, Master [16:52:57] !log preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for zero' [16:53:07] Logged the message, Master [16:53:22] !log preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for zero' [16:53:30] Logged the message, Master [16:56:36] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [16:58:28] !log freeing disk space on wikitech [16:58:40] Logged the message, Master [16:58:45] Nemo_bis: fixed, it ran out of disk space :/ [16:59:06] mutante: thanks [17:03:53] is University of robla accredited? [17:04:16] <^demon> I think so, but I don't know if the credits transfer. [17:04:17] everywhere except Minnesota [17:04:39] robla: picking on andrewbogott? [17:11:35] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [17:20:41] !log preilly synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess 'update for zero' [17:20:50] Logged the message, Master [17:21:15] !log preilly synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess 'update for zero' [17:21:24] Logged the message, Master [17:26:35] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [17:28:32] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [18:15:19] Deploy times [18:15:25] \o/ [18:18:21] Reedy: where are you starting? [18:19:33] PROBLEM - Host ms10 is DOWN: PING CRITICAL - Packet loss = 100% [18:20:10] yeah and it's going to stay down til tomorrow morning my time [18:20:14] when I will be back on the clock [18:20:35] apergos: which is that? [18:20:43] ms10 [18:20:45] * jeremyb was splitted i guess [18:20:50] ah [18:20:55] not splitted for this one [18:20:57] ;-) [18:21:13] you were just talking to a non-human [18:21:55] I seeee [18:22:51] RECOVERY - Host search32 is UP: PING OK - Packet loss = 0%, RTA = 0.55 ms [18:24:57] Hmm, why didn't sync-dblist just !log [18:25:06] !log special, fishbowl, closed and private wikis to 1.20wmf9 [18:25:17] Logged the message, Master [18:25:51] Or apparently not [18:26:18] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, fishbowl, closed and private wikis to 1.20wmf9 [18:26:26] Logged the message, Master [18:26:45] PROBLEM - Lucene disk space on search32 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:27:12] PROBLEM - SSH on search32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:29:04] Not a lot happenging [18:29:23] binasher: You should be able to run that CentralNotice schema update now for meta [18:29:54] kaldari: thanks, will do a minute [18:30:03] RECOVERY - SSH on search32 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [18:31:05] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikisource to 1.20wmf9 [18:31:14] Logged the message, Master [18:34:27] kaldari: done [18:34:35] binasher: Thank you! [18:36:52] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikiquote over to 1.20wmf9 [18:37:01] Logged the message, Master [18:38:27] PROBLEM - Lucene on search32 is CRITICAL: Connection refused [18:41:36] PROBLEM - Puppet freshness on virt0 is CRITICAL: Puppet has not run in the last 10 hours [18:44:53] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikibooks to 1.20wmf9 [18:45:02] Logged the message, Master [18:45:38] !log catrope synchronized wmf-config/CommonSettings.php 'Fix $wgVisualEditorParsoidPrefix' [18:45:47] Logged the message, Master [18:46:48] Reedy: did we deploy updates to jquery over the past weeks ? [18:47:08] Nope [18:47:18] There hasn't been a jquery update in ages [18:47:23] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia to 1.20wmf9 [18:47:23] hmm. [18:47:27] Maybe some individual modules [18:47:32] Logged the message, Master [18:47:33] There was a few jquery.ui changes [18:47:49] Reedy: Whoops sorry about that CommonSettings change, didn't realize you guys were deploying [18:48:00] lol, don't worry about it [18:48:01] there have been ie8 and ie9 users complaining about broken preferences (blank page) and broken diff pages (blank page).. [18:48:04] nothing is happening [18:48:23] perhaps it's the max on styles in the head ..... [18:48:39] RoanKattouw: i guess that's still not solved right ? [18:49:11] No it's not [18:49:16] Timo has plans for it [18:49:26] I think he was writing code for it a few weeks ago [18:49:31] though you'd expect more pages than diff and prefs to be affected by that. [18:50:06] they say they have the problem on the other wiki's as well. [18:51:00] I have to say so, i'm secretly happy that IE users now feel like I used to feel when using my mac to browse the web in 2003 (outcasts) :P [18:51:30] PROBLEM - NTP on search32 is CRITICAL: NTP CRITICAL: No response from NTP server [18:53:29] everything but wikipedia is done now, right? [18:53:34] Reedy: ^ [18:53:47] Yup [18:55:33] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [18:55:33] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [18:55:34] Reedy: My CommonSettings change is a live hack to stop VE from breaking, the proper change is here https://gerrit.wikimedia.org/r/18125 [18:55:48] (Again, sorry for doing this in the middle of your deployment window) [18:56:10] I wasn't touching any php config, so there was little chance of any conflicts happening! ;) [19:00:04] Reedy: don't ruin a perfectly good guilt trip I've been laying on him [19:00:06] * robla sighs [19:01:27] haha [19:01:31] I deservedi t [19:04:41] robla: let me help you [19:04:43] RoanKattouw: tsk tsk [19:05:03] golly thanks [19:05:06] heh [19:05:08] :) [19:08:08] RECOVERY - Lucene disk space on search32 is OK: DISK OK [19:26:26] RECOVERY - NTP on search32 is OK: NTP OK: Offset -0.01722049713 secs [20:01:32] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [20:16:11] I have a sort of complicated deployment to do today. Anyone want to help me make sure I'm doing it correctly? [20:18:50] specifically I want to update wmf8 to the master version of CentralNotice, but without the latest change that was merged in. Is there an easy way to do that? [20:19:05] Yes [20:19:22] kaldari: It's not master precisely but it's a historic version of master, right? [20:19:27] yes [20:19:58] Right, so do the dance for a regular master update [20:20:12] Except in the step where you do 'git checkout origin/master' , instead you do 'git checkout origin/master~1' [20:20:22] Then use 'git log' to check the result [20:20:29] wow, that's much easier than I thought :) [20:20:48] Roan: Does that work ~2, ~3, etc? [20:21:03] Yes [20:21:05] * aude thinks so [20:21:15] Oooh, cool. [20:21:18] * aude <3 git [20:21:22] foo~n means "n commits ago from foo" [20:21:29] So foo~1 is the parent/predecessor of foo [20:21:33] or HEAD~2 [20:21:38] Yeah [20:21:53] Sounds oddly sane. [20:22:26] Well, I should probably say "left parent", because merge commits have multiple parents (of course everything in git has to be complicated :D) [20:27:33] about to run scap if no one objects [20:29:34] kaldari: did you send it up to gerrit yet? [20:29:46] yeah [20:29:51] * jeremyb takes a look [20:29:57] https://gerrit.wikimedia.org/r/#/c/18183/ [20:30:12] and all the merged changes to CentralNotice except the last one [20:30:42] The last one is another big update, so saving that one for later [20:31:09] this one is just changing how campaign priorities work [20:31:24] interesi [20:31:30] interesting. [20:31:51] * aude awaits more fine tuned geotargeting [20:32:54] some day :) [20:33:32] alright, here goes nothing... [20:33:55] heh [20:34:07] lemme know if Wikipedia catches on fire [20:34:22] :) [20:36:13] well it does look like what's on gerrit matches what you described [20:37:21] the next update should speed up banner loading slightly, but that won't be for a couple more weeks [20:38:11] 2.1? [20:40:59] RECOVERY - Lucene on search32 is OK: TCP OK - 0.001 second response time on port 8123 [20:43:43] yeah, 2.1 [20:49:23] !log kaldari Started syncing Wikimedia installation... : [20:49:32] Logged the message, Master [20:53:38] scap really likes to build up the anticipation [20:53:52] And upto an hours worth of boredom ;) [21:15:23] almost there.... [21:20:23] huh, i think i just hit a bug in bugzilla... [21:21:42] cc'd on bug 35578 and the mail generated said http://dpaste.com/783440/plain/ [21:22:03] bugzilla says it mailed everyone except matanya [21:22:43] including wikibugs-l [21:23:29] http://www.mail-archive.com/wikibugs-l@lists.wikimedia.org/msg152480.html [21:29:27] almost there... [21:29:45] kaldari: that was 15 mins ago [21:30:00] geez, when did scap go from taking 30 minutes to an hour? [21:30:03] !log kaldari Finished syncing Wikimedia installation... : [21:30:14] Logged the message, Master [21:30:19] you were warned! [21:32:24] kaldari: https://gerrit.wikimedia.org/r/#/q/project:operations/puppet+topic:forkforkfork,n,z [21:32:47] Based on the NFS usage on monday.. [21:33:06] http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=load_one&s=by+name&c=Miscellaneous+pmtpa&h=nfs1.pmtpa.wmnet&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 [21:33:14] And today, doubling it won't get much of a gain.. [21:39:15] jeremyb, I think that happens when there's an edit conflict [21:39:31] It also screws with the wikibugs bot [21:39:59] It wrongly attributed a comment by someone else to me the other day [21:40:15] and I received an email just like that [21:41:10] Krenair: errr, did you see the dates? [21:41:21] that seems unpossible [21:41:34] oh. right. [21:41:34] i read matanya's comment before submit [21:41:36] lol [21:41:39] it wasn't an edit conflict [21:50:57] Hi jeremyb. [21:51:06] what lag [22:45:47] * Nemo_bis slaps Brooke [22:45:54] stopi spoiling my bugs [22:46:00] Now the other cheek. [22:46:18] * Nemo_bis executes [22:46:21] any other place? [22:46:46] Nemo_bis: did you see my bug? [22:46:59] jeremyb: which one? [22:47:16] right now I'm getting rid of a 900 emails backlog after OS reinstall [22:48:06] Nemo_bis: /msg [22:48:47] PROBLEM - Router interfaces on cr1-sdtpa is CRITICAL: CRITICAL: host 208.80.152.196, interfaces up: 76, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-1/1/0: down - Core: cr2-eqiad:xe-5/2/1 (FPL/Level3, CV71028) [10Gbps wave]BR [22:54:52] Brooke: I'm now digging my archives to source my statement that the new reportcard is meant as a replacement of Wikistats, just because it's you [22:55:12] but your comment wasn't helpful in any way [22:55:17] ;) [23:13:33] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [23:27:30] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [23:27:30] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [23:27:30] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours