[00:10:29] New patchset: Pyoungmeister; "swithcing to bash logic for binasher" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1682 [00:10:59] !log awjrichards synchronizing Wikimedia installation... : r107015 [00:11:08] Logged the message, Master [00:13:16] sync done. [00:18:57] !log synchronized payments cluster to r107018 [00:19:06] Logged the message, Master [00:19:41] New patchset: Pyoungmeister; "swithcing to bash logic for binasher" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1682 [00:21:23] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1682 [00:21:23] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1682 [00:57:02] gn8 folks [01:08:21] New review: Demon; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/1669 [01:09:31] New review: Demon; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/1680 [01:26:52] good night =_= [01:59:48] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 918s [02:02:16] !log started db9 maintenance phase 1 (getting it replicated to db10 again) [02:02:25] Logged the message, Master [02:04:05] !log LocalisationUpdate completed (1.18) at Thu Dec 22 02:04:05 UTC 2011 [02:04:14] Logged the message, Master [02:07:58] !log db9 is writable again [02:08:06] Logged the message, Master [02:12:38] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1688s [02:22:40] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:24:10] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:35:50] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [02:47:50] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [02:55:50] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [04:35:37] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [04:43:13] !log upping version of OpenStackManager on virt1 to match development. configuration and creation of instances should be avoided. [04:43:23] Logged the message, Master [05:13:13] !log finished creating all puppet configuration groups, classes, and variables. It's safe to configure and create instances again. [05:13:22] Logged the message, Master [05:55:15] RECOVERY - Misc_Db_Slave on db10 is OK: OK: [06:16:32] RECOVERY - Disk space on db9 is OK: DISK OK [06:20:42] RECOVERY - MySQL disk space on db9 is OK: DISK OK [09:08:50] PROBLEM - Puppet freshness on amssq53 is CRITICAL: Puppet has not run in the last 10 hours [09:35:40] RECOVERY - MySQL slave status on es1004 is OK: OK: [09:48:24] Hi all! I was wondering if my site could send about 200000 request/day to the wikipedia api [09:49:38] that is not a low number, what are you trying to do? [09:51:21] In my wiki I wanted to insert a code that would display the initial words of the article of wikipedia correlated to the page (sorry if i can't describe better the problem, I'm not good at english) [09:52:44] The code is in javascript [09:52:55] So is the visitor that send the request [09:53:47] there's no good policy what you can do with API at the moment [09:53:49] hehe [09:54:05] you could download dump and use it ;-) [09:55:35] Yes, is an option that I wanted to avoid, but I will use it [09:55:40] Thanks! [10:09:25] there is the abstracts dump which is probably closest to what you want [10:14:31] New review: ArielGlenn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1680 [10:14:32] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1680 [10:14:38] New review: ArielGlenn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1673 [10:14:39] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1673 [11:46:16] PROBLEM - Puppet freshness on knsq27 is CRITICAL: Puppet has not run in the last 10 hours [11:58:35] New review: Dzahn; "of course this makes sense, but the cert should also be installed on the host using "install_certifi..." [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/1669 [12:00:23] New review: Dzahn; "nevermind that, it does. looks good" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1669 [12:00:23] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1669 [12:02:32] New review: Dzahn; "it's just HTML .." [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1657 [12:02:32] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1657 [12:05:13] Hello [12:05:24] I'm having problem with generating pdf from a page... [12:05:30] http://en.wikipedia.org/w/index.php?title=Special:Book&bookcmd=rendering&return_to=Emoji&collection_id=02505e9913e9b350&writer=rl [12:05:39] goes to 75%... and crashes [12:22:37] New patchset: Mark Bergsma; "Try if fetching from the squids instead of ms5 is faster" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1684 [12:23:01] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1684 [12:23:02] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1684 [12:31:39] New patchset: Mark Bergsma; "upload doesn't accept domain upload.pmtpa.wikimedia.org, so use upload.wmorg instead" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1685 [12:31:52] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1685 [12:31:57] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1685 [12:31:57] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1685 [12:45:02] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [12:51:41] Ryan_Lane: hI [12:51:46] Hi [12:52:17] brb [12:57:02] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [12:57:35] Ryan_Lane: I am back. [13:05:02] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [13:16:53] bwahahaha [13:17:44] https://docs.google.com/spreadsheet/oimg?key=0AtHDNfVx0WNhdFZkSUxYSTJSNFFFVFNNUXhSOWhpbXc&oid=2&zx=jtcu0u1ez3nn [13:21:39] :-/ [13:21:45] well thank god for that [13:22:12] now that we know it's perfect we can move on to other articles [13:32:36] apergos: a sizeable minority disagrees, says it is very bad! [13:32:57] great, let em work on it [13:33:03] the rest can do something else. [13:44:22] anyway, that average value is completely bogus, as pretty much every vote on that page :) [13:44:46] but I'm sure it will skew any analytics done ;-D [13:55:25] New patchset: Dzahn; "additional generic check_procs with -C option & fix "mobile traffic logger" checks" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1687 [13:55:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1687 [14:02:20] New patchset: Dzahn; "additional generic check_procs with -C option & fix "mobile traffic logger" checks" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1687 [14:19:01] <31NAAAO2O> !log hashar: testswarm: deleted mobile jobs requests since they are now disabled [14:19:10] Logged the message, Master [14:21:40] 31NAAAO2O <--- that is a creative name [14:35:41] New review: Hashar; "Works for me!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1669 [15:00:09] PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 122 MB (1% inode=60%): /var/lib/ureadahead/debugfs 122 MB (1% inode=60%): [15:09:50] RECOVERY - Disk space on srv219 is OK: DISK OK [15:10:45] New review: Dzahn; "better than before for sure but i was still missing an "ok" here:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1669 [15:24:38] PROBLEM - Disk space on srv222 is CRITICAL: DISK CRITICAL - free space: / 210 MB (2% inode=60%): /var/lib/ureadahead/debugfs 210 MB (2% inode=60%): [15:34:18] RECOVERY - Disk space on srv222 is OK: DISK OK [19:13:20] PROBLEM - Exim SMTP on grosley is CRITICAL: Connection refused [19:19:23] RECOVERY - Exim SMTP on grosley is OK: SMTP OK - 0.007 sec. response time [19:23:42] PROBLEM - Puppet freshness on amssq53 is CRITICAL: Puppet has not run in the last 10 hours [19:31:17] New patchset: Asher; "fix db writeable assignment for research db's" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1689 [19:39:43] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1689 [19:39:43] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1689 [19:58:08] New patchset: Mark Bergsma; "Disable reporting because Puppet Dashboard ain't web-scale." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1690 [19:58:20] New patchset: Catrope; "Logrotate doesn't work with a missing olddir." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1691 [19:58:33] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1690 [19:58:33] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1691 [19:58:50] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1690 [19:58:51] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1690 [20:27:54] !log catrope synchronized wmf-config/InitialiseSettings.php 'Re-enable Contest extension' [20:28:02] Logged the message, Master [20:34:50] New patchset: Jgreen; "added aggressive transport to aluminium/grosley exim config" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1692 [20:35:02] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1692 [20:36:13] New review: Jgreen; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1692 [20:36:13] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1692 [20:59:13] !log neilk synchronized p/resources/mediawiki/mediawiki.feedback.js [20:59:22] Logged the message, Master [21:55:28] PROBLEM - Puppet freshness on knsq27 is CRITICAL: Puppet has not run in the last 10 hours [22:04:38] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Thu Dec 22 22:04:10 UTC 2011 [22:28:40] TimStarling, would you happen to be around? [22:47:21] hello [22:47:47] Howdy. [22:48:11] Snowing heavy here, how is your day? [22:48:19] TimStarling: hey [22:48:35] I think I need to get in touch with you about a second password to arbcom-l's archives? [22:49:06] yes, ok [22:49:29] I just posted my public key here... (link coming) [22:49:37] https://secure.wikimedia.org/wikipedia/en/wiki/User:Hersfold/hash [22:50:49] you know we have proper production HTTPS now, you don't have to use that rubbish hack secure.wikimedia.org anymore [22:50:59] oh [22:51:02] oh well [22:51:09] https://en.wikipedia.org/wiki/User:Hersfold/hash [22:51:17] * Hersfold doens't feel like updating all his bookmarks again [22:54:11] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [22:59:43] looks like I have 2 out of 4 keys [22:59:53] that's a start at least [23:01:24] :-) [23:06:06] gn8 folks [23:06:11] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [23:08:40] thanks, TimStarling [23:09:32] did it work? [23:13:51] I don't know... [23:14:02] I click on the archives link, and it asks for a username and password [23:14:10] am I supposed to paste that whole thing in there? [23:14:11] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [23:14:48] you're meant to decrypt it first [23:15:12] :-/ [23:15:17] * Hersfold|Skype is new to this whole PGP thing [23:16:02] ah, found the directions on your guide [23:16:28] yeah under "receiving encrypted email" [23:17:29] course, it would be nice if Kleopatra would start back up... [23:21:33] and no, it's not working [23:21:44] what happens? [23:21:57] I just get the same prompt back up, for username and password [23:22:16] ok, so you decrypted the email I sent you? [23:22:31] yep [23:22:36] and there was a password in it? [23:22:41] and it was legible, so I know it worked, yes [23:23:03] I enter the username and copy/paste the password in and nothing happens [23:23:27] what username? [23:24:07] did you use an upper case "H"? [23:24:09] Hersfold [23:24:10] yes [23:24:12] because that's how I wrote it [23:24:28] just tried with lowercase too, didn't work either [23:24:37] I'll reload lighttpd, I forget if that was needed [23:24:38] New patchset: Bhartshorne; "making owa storage bricks for containers to test" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1694 [23:24:50] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1694 [23:24:51] try now [23:25:10] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1694 [23:25:11] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1694 [23:25:56] nope [23:26:11] neither upper or lower-case h [23:26:20] are you typing it or copy and pasting? [23:26:46] I tried both [23:26:58] right, I see the problem, my fault [23:27:36] this is the problem with only doing this once every 6 months, you forget things [23:27:42] lol [23:27:47] try now [23:29:03] still not working [23:29:38] New patchset: Lcarr; "Moving all logging types of servers in new file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1695 [23:30:43] ok try again, I reckon there's a 75% chance it will work this time [23:30:53] maybe 76 [23:31:42] there we go [23:31:44] :-) [23:32:39] that was easy wasn't it? [23:33:01] heh [23:33:16] * Hersfold|Skype kinda wonders what the odds on the first four attempts were ;-) [23:34:08] I had the wrong "realm" [23:34:17] arbcom-l instead of "arbcom-l archive" [23:34:22] ah [23:34:22] so I used a text editor to change it [23:34:46] do I need one of these password to access the mail administration page as well? [23:34:50] but actually the realm is included in the password hash so changing the realm means you have to generate the password hash again [23:34:51] if so, I may need both... [23:35:24] yeah, you need both passwords, mailman isn't smart enough to know that you logged in the other way [23:36:40] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 1; - https://gerrit.wikimedia.org/r/1695 [23:37:21] mmm yakisoba [23:38:46] there's a restaurant in brazil that's spammed security@wikimedia.org twice with an ad for yakisoba [23:38:49] New patchset: Lcarr; "Moving all logging types of servers in new file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1695 [23:39:02] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1695 [23:39:07] if it wasn't for the airfares I'd so be into that [23:39:36] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 1; - https://gerrit.wikimedia.org/r/1695 [23:40:28] it's in Belém [23:40:58] anyway, thanks for the help, Tim. [23:41:01] i'm off [23:41:12] wonderful how internet technology has evolved to the point where the easiest way to reach the residents of Belém is not by a letterbox drop but by spamming everyone in the world [23:41:56] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1695 [23:41:57] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1695 [23:57:33] New patchset: Lcarr; "adding in UDP iptables service" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1696 [23:57:46] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1696