[01:40:20] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 236 seconds
[01:42:26] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 261 seconds
[01:44:05] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds
[01:46:56] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 10 seconds
[01:59:44] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 235 seconds
[02:00:11] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 263 seconds
[02:22:06] <logmsgbot_>	 !log LocalisationUpdate completed (1.21wmf3) at Tue Nov 13 02:22:06 UTC 2012
[02:22:14] <morebots>	 Logged the message, Master
[02:39:21] <logmsgbot_>	 !log LocalisationUpdate completed (1.21wmf4) at Tue Nov 13 02:39:21 UTC 2012
[02:39:27] <morebots>	 Logged the message, Master
[03:03:47] <nagios-wm>	 PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours
[03:25:02] <Phil_Bordelon>	 Does anyone have access to (or can point me to) account creations by date?  I know I was within the first 27, but a family member (okay, my mom) wanted to know the exact number, presumably for weird work bragging rights.
[03:29:01] <jeremyb>	 Phil_Bordelon: which wiki ?
[03:29:06] <Phil_Bordelon>	 en
[03:30:33] <jeremyb>	 and what's your username?
[03:30:35] <Phil_Bordelon>	 (may not be 27th, since I think that's tracking people with >10 edits, but, yeah, the old stats page is all I know to be able to go on.)
[03:30:38] <Phil_Bordelon>	 Phil_Bordelon
[03:31:08] <Phil_Bordelon>	 Well, a space, not an underscore.
[03:31:14] <jeremyb>	 i don't see you at http://en.wikipedia.org/w/index.php?title=Special:ListUsers&offset=&limit=500&username=&group=&creationSort=1
[03:31:28] <jeremyb>	 but i'm not certain how accurate the dates are there
[03:31:46] <Phil_Bordelon>	 Those dates are totally bogus, since my first edits were in March 2001
[03:31:49] <jeremyb>	 and obviously some have no dates at all. but still somehow have an order in the list
[03:33:11] <Phil_Bordelon>	 http://en.wikipedia.org/w/index.php?title=Special:Contributions/Phil_Bordelon&offset=20030611043634&limit=20&target=Phil+Bordelon
[03:33:20] <jeremyb>	 yeah, i saw
[03:33:37] <jeremyb>	 idk what to tell you besides what I already said: idk how accurate the dates are
[03:33:59] <Phil_Bordelon>	 Pretty sure that's an artefact from the move to the proto-mediawiki from the previous Wiki software I don't remember the name of.
[03:34:22] <Phil_Bordelon>	 Okay.  So that data is as good as it gets.  Thanks anyhow :)
[03:36:23] <jeremyb>	 grrrrr, lots of packet loss here ;(
[03:36:26] <jeremyb>	 idk what to tell you besides what I already said: idk how accurate the dates are
[03:37:19] <Phil_Bordelon>	 I appreciate the help anyhow.
[03:38:22] <jeremyb>	 hah, https://en.wikipedia.org/wiki/Wikipedia#cite_note-WikipediaHome-129
[03:39:03] <Phil_Bordelon>	 I remember that.
[03:39:10] <Phil_Bordelon>	 And our insane attempt at categorization :p
[03:39:17] <Phil_Bordelon>	 Note Games/Die Siedler von Catan
[03:40:20] <Phil_Bordelon>	 (also an early choice to use native names, later reverted)
[03:40:38] <Phil_Bordelon>	 wacky.
[03:40:41] <Phil_Bordelon>	 Anyhow, thanks again.
[03:44:44] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds
[03:51:33] <jeremyb>	 http://article.gmane.org/gmane.science.linguistics.wikipedia.technical/10844 is funny ;)
[03:52:04] <jeremyb>	 http://www.gossamer-threads.com/lists/wiki/wikitech/249466#249466
[03:53:06] <Phil_Bordelon>	 Huh.  Tim is still around.  Good on him.
[03:53:53] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 3 seconds
[03:55:11] <jeremyb>	 and here's what I was actually looking for this whole time: http://lists.wikimedia.org/pipermail/foundation-l/2010-December/063088.html
[03:59:37] <Phil_Bordelon>	 Whoa.
[03:59:51] <Phil_Bordelon>	 8.4M.  Haha.
[04:02:11] <Phil_Bordelon>	 Wow.  I edited The Rock/Spoler
[04:02:14] <Phil_Bordelon>	 er, The Rock/Spoiler
[04:02:22] <Phil_Bordelon>	 Dunno how I feel about that :|
[04:02:49] <Phil_Bordelon>	 Glad it's not in my edit history, I guess?
[04:04:48] <Phil_Bordelon>	 Okay, I think I can do some shenanigans with the rclog to get the unique entries and see how far down I am.
[04:13:25] <Phil_Bordelon>	 According to that file I'm #158, but it looks like IPs were getting assigned random values.  Anyway.  Yeah.  Fascinating.  Thanks for that, jeremyb.
[04:13:41] <Phil_Bordelon>	 Plus there's duplicates like Jwales and JimboWales
[04:15:04] <jeremyb>	 sure
[04:16:39] <jeremyb>	 (btw, for future reference the right channel for this question is #wikimedia-tech not here. not a big deal because it's so quiet. and I know someone else did direct you here from #-ops)
[04:18:03] <Phil_Bordelon>	 Yeah.
[04:18:06] <Phil_Bordelon>	 Sorry about that, then.
[04:18:27] <jeremyb>	 Phil_Bordelon: idk what city you're in, there's periodic meetings of wikimedians around the country/world. maybe you'd like to check out a local one wherever you are. ([[WP:MEET]])
[04:18:46] <jeremyb>	 do you remember ed poor? i met him in person about 5 years ago at a picnic
[04:18:59] <Phil_Bordelon>	 I certainly saw his edits
[04:20:17] <jeremyb>	 his earliest on record in prod right now is november 2001
[04:20:24] <Phil_Bordelon>	 Anyhow, I apologize for cluttering up the channel with unnecessary guff.
[04:20:31] <jeremyb>	 np ;)
[04:20:37] <jeremyb>	 enjoy showing your mom!
[04:21:01] <Phil_Bordelon>	 Except for the part where I edited an article on "The Rock," :| I will :)
[04:35:44] <nagios-wm>	 PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours
[04:35:44] <nagios-wm>	 PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours
[04:56:18] <nagios-wm>	 PROBLEM - Puppet freshness on cp1042 is CRITICAL: Puppet has not run in the last 10 hours
[06:44:39] <nagios-wm>	 PROBLEM - Squid on brewster is CRITICAL: Connection refused
[06:44:47] <nagios-wm>	 PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours
[06:59:38] <nagios-wm>	 RECOVERY - Squid on brewster is OK: TCP OK - 0.005 second response time on port 8080
[08:14:56] <nagios-wm>	 PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours
[08:14:56] <nagios-wm>	 PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours
[08:14:56] <nagios-wm>	 PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours
[08:56:34] <logmsgbot_>	 !log reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler
[08:56:41] <morebots>	 Logged the message, Master
[08:57:15] <logmsgbot_>	 !log reedy synchronized php-1.21wmf4/extensions/TimedMediaHandler
[08:57:21] <morebots>	 Logged the message, Master
[09:32:15] <nagios-wm>	 PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours
[11:25:56] <paravoid>	 j^: around? :)
[12:05:39] <j^>	 paravoid: pong
[12:07:35] <paravoid>	 hi!
[12:08:16] <paravoid>	 https://bugzilla.wikimedia.org/show_bug.cgi?id=40070
[12:08:26] <paravoid>	 this was solved a few months back
[12:09:04] <paravoid>	 but I'm not sure it's actually configured
[12:09:32] <paravoid>	 I tried to check but with browser autodetection and everything I only get an HTML5 <video> :)
[12:10:53] <andre__>	 paravoid, could you add a comment to that bug report asking for clarification?
[12:17:50] <j^>	 paravoid: will have a look at it
[12:17:59] <paravoid>	 andre__: well, technically that bug is actually solved
[12:18:23] <paravoid>	 I could open a new one, but thought to just ask here first
[12:19:09] <andre__>	 paravoid, I'd be lazy and ask on the ticket itself. If somebody complains, blame me ;)
[12:24:52] <j^>	 paravoid: the bug is fixed. you can overwrite cortado location with $wgCortadoJarFile (this is currently not done)
[12:25:26] <j^>	 paravoid: if you have a working java setup you can select java in the menu
[12:27:15] <paravoid>	 okay. do you ship cortado in bits?
[12:27:47] <j^>	 paravoid: https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/extensions/TimedMediaHandler.git;a=blob;f=MwEmbedModules/EmbedPlayer/resources/mw.EmbedPlayerJava.js we also set cortado location for upload to 'http://upload.wikimedia.org/jars/cortado.jar
[12:27:56] <j^>	 cortado has to be on the same domain as the video
[12:28:00] <paravoid>	 no it hasn't
[12:28:13] <paravoid>	 I've already refuted that argument
[12:28:21] <paravoid>	 http://upload.wikimedia.org/crossdomain.xml
[12:28:45] <j^>	 does java implement that now?
[12:28:55] <j^>	 they did not for a long time
[12:28:59] <paravoid>	 for about 4 years yes :)
[12:29:31] <j^>	 around the time it stopped working for me :)
[12:30:09] <paravoid>	 since  Java SE 6 update 10
[12:30:14] <paravoid>	 released 2008-10-15
[12:30:36] <paravoid>	 not implemented in icedtea though, but cortado doesn't work with icedtea anyway
[12:31:41] <j^>	 it should
[12:32:00] <j^>	 anyway dont think people on linux need it since they can use html5 browsers
[12:32:21] <j^>	 its only used as a fallback for IE users with java
[12:33:09] <j^>	 regarding bits, its in extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/binPlayers/cortado/cortado-ovt-stripped-0.6.0.jar
[12:33:18] <j^>	 not sure how to translate that to a bits url
[12:33:30] <paravoid>	 no idea either :)
[12:35:02] <j^>	 here we go http://bits.wikimedia.org/w/extensions/TimedMediaHandler/MwEmbedModules/EmbedPlayer/binPlayers/cortado/cortado-ovt-stripped-0.6.0.jar
[12:38:49] <j^>	 paravoid: i dont see where the cortado url was set for OggHandler in wmf-config
[12:39:23] <paravoid>	 https://gerrit.wikimedia.org/r/#/c/23036/
[12:40:29] <paravoid>	 the comment is probably more useful than the code diff
[12:46:01] <andre__>	 mutante: There will be a Bugzilla update to 4.0.9 in the next 6 hours fixing five security issues. Could you plan to spare some time for applying it?
[12:49:28] <paravoid>	 a bit early for SF :)
[12:57:23] <j^>	 paravoid: ah ok, so TMH also uses wgExtensionAssetsPath and gets the right bits url but its overwritten for upload, so we can take out that overwrite
[12:57:38] <andre__>	 paravoid, I know :)
[12:57:58] <paravoid>	 j^: cool! want me to open a bug report or...?
[12:58:07] <paravoid>	 andre__: what will you need?
[12:58:23] <andre__>	 paravoid, really nothing huge, just apply some patches. They are not out yet.
[12:58:42] <andre__>	 paravoid: was more like a warning. not urgent either.
[12:58:54] <paravoid>	 "five security issues" sounded like urgent :)
[13:00:06] <paravoid>	 Daniel is probably the right person for this, but if it's something urgent/something you need help in this timezone, I can make myself available too.
[13:03:06] <j^>	 paravoid: https://gerrit.wikimedia.org/r/#/c/33173
[13:04:54] <nagios-wm>	 PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours
[13:52:45] <nagios-wm>	 PROBLEM - LVS Lucene on search-pool4.svc.eqiad.wmnet is CRITICAL: Connection timed out
[13:53:12] <notpeter>	 woo
[13:54:15] <nagios-wm>	 RECOVERY - LVS Lucene on search-pool4.svc.eqiad.wmnet is OK: TCP OK - 9.026 second response time on port 8123
[13:57:07] <paravoid>	 did you fix it?
[13:57:55] <notpeter>	 what is "fix" anyway?
[13:58:11] <notpeter>	 I got a recovery just as I was restarting lucene on 1015
[13:58:20] <notpeter>	 so I think it actually fixed itself
[13:58:57] <notpeter>	 but no, it's not really fixed, as its still running lucene
[13:59:02] <paravoid>	 hahaha
[14:37:14] <nagios-wm>	 PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours
[14:37:14] <nagios-wm>	 PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours
[14:37:40] <paravoid>	 j^: merged I see. you rock!
[14:56:17] <nagios-wm>	 PROBLEM - Puppet freshness on cp1042 is CRITICAL: Puppet has not run in the last 10 hours
[15:10:26] <gerrit-wm>	 New review: Cmcmahon; "please!" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32061
[15:14:57] <hashar>	 !log restarting jenkins on gallium
[15:15:04] <morebots>	 Logged the message, Master
[15:23:54] <average_drifter>	 hello guys
[15:23:57] <average_drifter>	 I'm Stefan Petrea, external contractor working on WMF Analytics team
[15:24:00] <average_drifter>	 is there a reliable way to get hostname of stat1 and all prod machines ?
[15:30:16] <paravoid>	 what do you mean "to get hostname"?
[15:32:56] <cmjohnson1>	 !log provisioning labsdb2
[15:33:03] <morebots>	 Logged the message, Master
[15:33:25] <Hydriz>	 labsdb2? Sounds cool
[15:39:55] <Nemo_bis>	 average_drifter: ^^
[15:50:47] <Jeff_Green>	 robh can you move silicon today?
[15:52:35] <gerrit-wm>	 New patchset: Cmjohnson; "Adding labsdb3 to dhcpd file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33192
[15:53:22] <RobH>	 Jeff_Green: yep, can even do now
[15:53:28] <RobH>	 its down and all, so i'll just pull it
[15:55:03] <cmjohnson1>	 lesliecarr: could you look at this https://gerrit.wikimedia.org/r/33192
[16:01:41] <Jeff_Green>	 RobH: thanks!
[16:03:04] <gerrit-wm>	 Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33192
[16:03:54] <paravoid>	 cmjohnson1, LeslieCarr: merged, running puppet on brewster now
[16:04:07] <cmjohnson1>	 thx paravoid!
[16:05:15] <nagios-wm>	 PROBLEM - Host silicon is DOWN: PING CRITICAL - Packet loss = 100%
[16:08:38] <cmjohnson1>	 !log provision labsdb3
[16:08:45] <morebots>	 Logged the message, Master
[16:10:34] <paravoid>	 !log creating media1:thumbs volume at nas1-a and setting up /etc/export rules for it
[16:10:43] <morebots>	 Logged the message, Master
[16:30:03] <apergos>	 !log rebooting cp1042 (XFS: possible memory allocation deadlock in kmem_alloc), lots of stuck /bin/ps form nagios, load average ridiculous
[16:30:10] <morebots>	 Logged the message, Master
[16:32:15] <nagios-wm>	 RECOVERY - Varnish HTCP daemon on cp1042 is OK: PROCS OK: 1 process with UID = 997 (varnishhtcpd), args varnishhtcpd worker
[16:32:42] <nagios-wm>	 RECOVERY - Varnish HTTP mobile-backend on cp1042 is OK: HTTP OK HTTP/1.1 200 OK - 698 bytes in 0.053 seconds
[16:35:51] <nagios-wm>	 PROBLEM - Varnish HTTP mobile-frontend on cp1042 is CRITICAL: Connection refused
[16:45:45] <nagios-wm>	 RECOVERY - Varnish HTTP mobile-frontend on cp1042 is OK: HTTP OK HTTP/1.1 200 OK - 641 bytes in 0.053 seconds
[16:46:16] <apergos>	 !log had to restart varnish frontend by hand after reboot, also the varnishncsa for emery, locke, multicast
[16:46:21] <nagios-wm>	 PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours
[16:46:22] <morebots>	 Logged the message, Master
[16:46:48] <nagios-wm>	 RECOVERY - Varnish traffic logger on cp1042 is OK: PROCS OK: 3 processes with command name varnishncsa
[16:50:53] <paravoid>	 !log running "cf giveback" on nas1001-b to restore nas1001-a; had to "partner snapmirror off" first.
[16:51:00] <morebots>	 Logged the message, Master
[16:51:00] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:52:30] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.300 seconds
[16:54:09] <nagios-wm>	 RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Tue Nov 13 16:53:51 UTC 2012
[16:54:34] <paravoid>	 !log snapmirror between nas1-a/nas1001-a for all volumes in-sync again
[16:54:40] <morebots>	 Logged the message, Master
[17:24:24] <Jeff_Green>	 !log wikimedia.org zone update, new DKIM keys for silverpop
[17:24:30] <morebots>	 Logged the message, Master
[17:27:27] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:35:42] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.755 seconds
[17:53:25] <paravoid>	 !log setting up media1:thumbs on nas1001-a and semi-sync snapmirror from nas1-a
[17:53:31] <morebots>	 Logged the message, Master
[18:05:57] <gerrit-wm>	 New patchset: Faidon; "Add nfs::netapp::thumbs for the new thumbs volume" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33201
[18:05:57] <gerrit-wm>	 New patchset: Faidon; "Unmount /mnt/thumbs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33202
[18:06:25] <gerrit-wm>	 Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33201
[18:10:51] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:16:24] <nagios-wm>	 PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours
[18:16:24] <nagios-wm>	 PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours
[18:16:24] <nagios-wm>	 PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours
[18:18:07] <paravoid>	 friggin puppet
[18:28:51] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.034 seconds
[18:33:31] <RobH>	 !log testing analytics1002, pxe boots no problem
[18:33:38] <morebots>	 Logged the message, RobH
[18:34:51] <nagios-wm>	 RECOVERY - Host analytics1002 is UP: PING OK - Packet loss = 0%, RTA = 31.02 ms
[18:35:10] <nagios-wm>	 PROBLEM - check_mysql on payments1004 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null)
[18:35:44] <Jeff_Green>	 that ^^^ payments1004 alert is me testing nagios
[18:40:15] <nagios-wm>	 RECOVERY - check_mysql on payments1004 is OK: Uptime: 1571207  Threads: 1  Questions: 2063533  Slow queries: 959  Opens: 1341  Flush tables: 1  Open tables: 64  Queries per second avg: 1.313 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0
[18:41:56] <RobH>	 !log analytics1007.mgmt locked up, had to remove all power to fix, now working
[18:42:02] <morebots>	 Logged the message, RobH
[18:43:49] <RobH>	 !log analytics1007 now pxe boots
[18:43:55] <morebots>	 Logged the message, RobH
[18:46:51] <nagios-wm>	 RECOVERY - Host analytics1007 is UP: PING OK - Packet loss = 0%, RTA = 30.96 ms
[19:00:13] <nagios-wm>	 PROBLEM - check_mysql on payments4 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null)
[19:01:24] <Jeff_Green>	 mysqlbarf from payments4 is me testing nagios...
[19:01:42] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:03:56] <apergos>	 anyone knowo what blondel serves? (some db or other but I have no idea what "m1" is
[19:03:57] <apergos>	 )
[19:04:08] <apergos>	 guess it's starting to get short of space
[19:05:09] <nagios-wm>	 RECOVERY - check_mysql on payments4 is OK: Uptime: 1571469  Threads: 1  Questions: 2292791  Slow queries: 0  Opens: 38  Flush tables: 1  Open tables: 31  Queries per second avg: 1.459 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0
[19:07:37] <apergos>	 Jeff_Green: I saw you touched blondel once.. any ide what dbs it serves?
[19:07:40] <apergos>	 :-P
[19:07:46] <apergos>	 I could not get much from looking at puppet
[19:07:54] <Jeff_Green>	 hahahah you're funny
[19:08:08] <Jeff_Green>	 trying to remember ....
[19:08:22] <Jeff_Green>	 wild guess is that it's an otrs slave, but that's really a wild guess
[19:08:45] <apergos>	 uhhh
[19:09:08] <Jeff_Green>	 from the databases it looks like a misc db
[19:09:17] <Jeff_Green>	 master is db9
[19:09:28] <apergos>	 hmm ok
[19:09:37] <apergos>	 well I guess I might have to clean up some binlogs on it
[19:09:43] <Jeff_Green>	 ooh, /a 96% yeah
[19:09:59] <Jeff_Green>	 binlogs go back to 10/14
[19:10:17] <Jeff_Green>	 expire_logs_days = 30
[19:10:23] <apergos>	 ugh
[19:10:25] <Jeff_Green>	 it's doing the right things
[19:10:44] <apergos>	 ok, should I shorten that then and if so, can you point me to how to do that?
[19:12:41] <Jeff_Green>	 uhh
[19:12:48] <Jeff_Green>	 this box is set up RAID0? wtf?
[19:13:06] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 2.864 seconds
[19:13:49] <Jeff_Green>	 apergos: I have serious questions about this box as a hardware choice...
[19:13:50] <apergos>	 ah I see it's in the conf file
[19:14:01] <Jeff_Green>	 yeah that's written by puppet though
[19:14:07] <apergos>	 oohhh
[19:14:41] <apergos>	 defaut template
[19:14:42] <apergos>	 meh
[19:16:11] <ori-l>	 hey guys, which version of squid do we run in prod?
[19:17:32] <apergos>	 so according to https://rt.wikimedia.org/Ticket/Display.html?id=3889  this got to 99%, prolly before it tossed the most recent binlog
[19:18:09] <apergos>	 but neither "let's make a special conf file for this one host" nor "let's keep less logs everywhere" seems like a good solution
[19:18:53] <Jeff_Green>	 apergos: I think it's parameterized so you can override the setting per host, right?
[19:18:54] <LeslieCarr>	 ori-l: 2.7.9-7wm2
[19:19:03] <ori-l>	 LeslieCarr: thanks!
[19:19:03] <apergos>	 ah is it? lemme look
[19:19:10] <Jeff_Green>	 apergos: my vote is for replacing that host with something that is not a joke
[19:19:27] <apergos>	 yeah but in the meantime...
[19:24:45] <Jeff_Green>	 apergos: might be able to extend the lvm volume
[19:26:06] <nagios-wm>	 PROBLEM - Host analytics1002 is DOWN: PING CRITICAL - Packet loss = 100%
[19:28:08] <Jeff_Green>	 apergos: looks like there's ~30GB unallocated space available
[19:28:17] <apergos>	 yeah I was just looking at the numbers
[19:28:20] <apergos>	 that will do for awhile
[19:28:36] <Jeff_Green>	 maybe until one or the other disk dies :-P
[19:28:49] <apergos>	 :-/
[19:31:48] <nagios-wm>	 RECOVERY - Host analytics1002 is UP: PING OK - Packet loss = 0%, RTA = 30.91 ms
[19:33:31] <apergos>	 added 20gb
[19:33:41] <apergos>	 so we can play this game again later when we forget about the issue :-/
[19:33:45] <nagios-wm>	 PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours
[19:33:49] <apergos>	 thank you for looking at it with me
[19:44:47] <woosters>	 jeff_green ping
[19:44:59] <Jeff_Green>	 woosters: pong
[19:45:09] <woosters>	 u running 24-hr stress test on Thursday
[19:45:18] <woosters>	 starting ...what time?
[19:45:28] <Jeff_Green>	 afaik they haven't stated yet
[19:45:36] <woosters>	 http://wikitech.wikimedia.org/view/Deployments
[19:45:51] <woosters>	 2 items are scheduled - E2 & E3
[19:45:54] <Jeff_Green>	 woosters: looking
[19:46:18] <Jeff_Green>	 yup I see it. i'll let them know
[19:46:31] <woosters>	 asher is going to switch over to dedicated mc servers this week
[19:46:52] <woosters>	 and notpeter to deploy the new mw app servers on precise
[19:46:56] <ori-l>	 Jeff_Green: what are the implications for E3/E2?
[19:47:04] <woosters>	 those 2 are not scheduled yet
[19:48:09] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:48:38] <Jeff_Green>	 ori-l: I'm entirely not sure--banners would be up everywhere though
[19:49:12] <ori-l>	 Jeff_Green: yeah, that's OK. would you prefer it if we tried to reschedule for wednesday or something?
[19:50:29] <Jeff_Green>	 I'll point out the deployments to zack and start there
[19:50:38] <ori-l>	 k
[19:51:00] <nagios-wm>	 PROBLEM - Host analytics1002 is DOWN: PING CRITICAL - Packet loss = 100%
[19:51:05] <Jeff_Green>	 the email subject says thursday, but the body actually talks about starting on weds afternoon
[19:55:05] <ottomata>	 Jeff_Green, drdee wants to finally push through some of the web access log format changes soon
[19:55:13] <ottomata>	 we're adding two headers at the end of the line
[19:55:22] <ottomata>	 and switching field separator to tab instead of space
[19:55:46] <ottomata>	 this would require modifying the logging configs for squid, nginx and varnish
[19:55:53] <ottomata>	 i've tested these changes out on a labs instance
[19:56:20] <ottomata>	 do you think we can do this soon?  will it affect your stuff?
[19:57:02] <Jeff_Green>	 ottomata: is this the E3 deploy? or is this unrelated?
[19:57:07] <ottomata>	 unrelated
[19:57:16] <ottomata>	 this is a long standing desired change to the web log formats
[19:57:22] <ottomata>	 adding accept-language and x-carrier headers
[19:57:39] <ottomata>	 and then changing separator to tab (there are sometimes unencoded spaces in user-agent, which causes parsing problems)
[19:58:03] <Jeff_Green>	 can you email this to fr-tech?
[19:58:03] <ottomata>	 its been on the todo for a very long time, but was halted because we didn't want to break erik zachte's wikistats stuff
[19:58:14] <ottomata>	 but now drdee has stefan working on wikistats, and says it should be ok
[19:58:21] <nagios-wm>	 RECOVERY - SSH on analytics1002 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0)
[19:58:22] <Jeff_Green>	 peter g may have to adjust his parser script
[19:58:30] <nagios-wm>	 RECOVERY - Host analytics1002 is UP: PING OK - Packet loss = 0%, RTA = 31.11 ms
[19:58:34] <Jeff_Green>	 depending on how he's detecting delimiters
[19:59:17] <ottomata>	 ok
[19:59:30] <ottomata>	 fr-tech@wikimedia.org?
[19:59:32] <ottomata>	 can I just email that?
[19:59:34] <Jeff_Green>	 yep
[20:01:00] <Jeff_Green>	 ottomata: thanks
[20:03:00] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.037 seconds
[20:05:29] <Jeff_Green>	 ori-l: can you tell me more (or point me to info on) the E2 deployment?
[20:05:58] <ori-l>	 i'm not sure what (if anything) E2 has planned. kaldari or bsitu are probably the people to ask
[20:06:00] <nagios-wm>	 PROBLEM - Host analytics1007 is DOWN: PING CRITICAL - Packet loss = 100%
[20:06:30] <ori-l>	 E3 has a few extensions to sync, but i can ask whomever's deploying on wednesday to ride shotgun
[20:07:09] * Reedy  eyes ori-l

[20:07:20] <ori-l>	 ...?
[20:07:23] <ori-l>	 what'd i do?
[20:08:11] <Jeff_Green>	 oh sorry I meant E3 yeah
[20:08:45] <Jeff_Green>	 I'm just looking for any reason it would tangle with the proposed fundraising test
[20:09:32] <ori-l>	 dunno, i'll ask S (the other E3 dev) if he expects to push anything that could conflict
[20:10:32] <ori-l>	 spagewmf: ^
[20:12:30] <spagewmf>	 I have no E3 deployment items for this week... yet
[20:12:40] <cmjohnson1>	 lesliecarr: can you look at rt 3906 plz
[20:12:52] <LeslieCarr>	 cmjohnson1: sure
[20:13:02] <cmjohnson1>	 thx
[20:14:03] <LeslieCarr>	 cmjohnson1: in the labs subnet ?
[20:14:17] <Jeff_Green>	 ori-l/spagewmf: ok thank you
[20:14:19] <cmjohnson1>	 oh..yes plz
[20:14:28] <cmjohnson1>	 lesliecarr: ^
[20:14:38] <LeslieCarr>	 cmjohnson1: done (once it commits in about 30 seconds
[20:15:05] <LeslieCarr>	 if i ever make a switch that has seamless upgrades and commits instantly, i'll be a multi billionaire
[20:15:11] <apergos>	 heh
[20:15:13] <apergos>	 you would
[20:15:33] <LeslieCarr>	 lunchtime
[20:15:46] <apergos>	 enjoy
[20:23:00] <gerrit-wm>	 New patchset: Kaldari; "Turning on wgCopyUploadsFromSpecialUpload on testwiki for testing" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33217
[20:23:48] <gerrit-wm>	 Change merged: Kaldari; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33217
[20:34:32] <gerrit-wm>	 New review: Hashar; "You can also get rid of it in nfs::upload::labs , it is not in use either." [operations/puppet] (production); V: 0 C: -1;  - https://gerrit.wikimedia.org/r/33202
[20:35:51] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:48:54] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.804 seconds
[21:24:12] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:34:18] <gerrit-wm>	 New patchset: Reedy; "Remove some old comments/useless settings and stuff" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33272
[21:35:52] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33272
[21:40:33] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.030 seconds
[21:57:25] <gerrit-wm>	 New patchset: Cmjohnson; "Addin spage to site.pp for Vanadium RT3884" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33276
[22:00:16] <gerrit-wm>	 New review: Dzahn; "You will also have to add an account, like accounts::spage above the sudo line." [operations/puppet] (production); V: 0 C: 0;  - https://gerrit.wikimedia.org/r/33276
[22:01:25] <gerrit-wm>	 New patchset: Kaldari; "Adding flickrApiUrl to UploadWizard config settings" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33277
[22:02:05] <gerrit-wm>	 Change merged: Kaldari; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33277
[22:06:27] <logmsgbot_>	 !log kaldari synchronized wmf-config/PrivateSettings.php  'syncing PrivateSettings for Flickr API key'
[22:06:36] <morebots>	 Logged the message, Master
[22:07:20] <gerrit-wm>	 New patchset: Lcarr; "upgrading to new varnish.py for RT 3842 sourced from https://github.com/ganglia/gmond_python_modules/tree/master/varnish/python_modules" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33278
[22:07:29] <mutante>	 !log mailman lists: renaming wikimedia-th to wikimedia-th-private
[22:07:35] <morebots>	 Logged the message, Master
[22:10:04] <gerrit-wm>	 Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33278
[22:12:21] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:15:31] <gerrit-wm>	 New patchset: Cmjohnson; "Addin spage to site.pp for Vanadium RT3884" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33276
[22:15:58] <cmjohnson1>	 mutante: is that what you are talking about ? ^
[22:17:43] <mutante>	 cmjohnson1: yes, it is. well technically the request was just "access" to even sudo, but from the context of the ticket one can guess they meant sudo .. i think
[22:18:02] <mutante>	 just "access" without even sudo
[22:18:50] <cmjohnson1>	 ah..yeah, i just assumed sudo based on the ticket
[22:18:59] <gerrit-wm>	 New patchset: Reedy; "Add OAI/lucene schema" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33280
[22:19:12] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33280
[22:19:27] <cmjohnson1>	 so would i just remove the sudo line? (mutante)
[22:21:49] <mutante>	 cmjohnson1: leave it it, it says to "administer" it.
[22:22:04] <gerrit-wm>	 New patchset: Reedy; "Add index.html for namespace too" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33281
[22:22:20] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33281
[22:22:43] <Reedy>	 !log Publishing schema for lucene/oai on mw.o....
[22:22:49] <morebots>	 Logged the message, Master
[22:23:04] <cmjohnson1>	 mutante: cool thx for looking
[22:27:03] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.582 seconds
[22:27:16] <gerrit-wm>	 New patchset: Reedy; "Fixup whitespace" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33282
[22:27:31] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33282
[22:30:10] <gerrit-wm>	 New patchset: Reedy; "Fix chmod" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33284
[22:30:23] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33284
[22:32:32] <gerrit-wm>	 New patchset: Reedy; "Ignore ganglia.dat" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33285
[22:32:43] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33285
[22:41:17] <binasher>	 !log converted puppet.updates_update table to innodb - received the most number of of any puppet db table and was myisam, yikes. this might help with concurrency.
[22:41:24] <morebots>	 Logged the message, Master
[22:45:56] <gerrit-wm>	 New patchset: Lcarr; "Varnish could not find the stats" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33286
[22:46:11] <gerrit-wm>	 Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33286
[22:49:02] <gerrit-wm>	 New review: Dzahn; "yep, confirmed with S and Ori, this was for full sudo access. granted." [operations/puppet] (production); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/33276
[22:49:03] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33276
[22:56:44] <logmsgbot_>	 !log kaldari synchronized wmf-config/PrivateSettings.php  'syncing PrivateSettings back to previous version'
[22:56:50] <morebots>	 Logged the message, Master
[23:01:06] <gerrit-wm>	 New patchset: Asher; "For https-only wikis, the redirect from http to https for bits assets should occur in varnish instead of apache, since the apache redirect and mediawiki doesn't vary by protocol. This can result in a redirect loop and assets not loading." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32676
[23:01:29] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:01:36] <gerrit-wm>	 Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32676
[23:06:02] <gerrit-wm>	 New patchset: Asher; "disable use of MemcachedPhpBagOStuff and memcached on apache servers" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33289
[23:06:17] <nagios-wm>	 PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours
[23:12:01] <AaronSchulz>	 paravoid: http://upload.wikimedia.org/wikipedia/commons/thumb/1/1a/%C2%BFQu%C3%A9_es_Wikipedia%3F.ogv/%C2%BFQu%C3%A9_es_Wikipedia?.ogv.360p.webm
[23:12:17] <AaronSchulz>	 would be nice to kill that double escaping :)
[23:14:11] <binasher>	 AaronSchulz: https://gerrit.wikimedia.org/r/33289
[23:14:17] <hashar>	 binasher: do you happen to know why we were to both ?
[23:14:38] <hashar>	 that puzzled me earlier this morning when reading that mc.php file
[23:15:00] <binasher>	 to avoid migrating to an empty cache
[23:18:17] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.034 seconds
[23:18:20] <hashar>	 ohh
[23:18:36] <hashar>	 makes more sense. Thanks binasher :-]
[23:18:45] <hashar>	 and congrats on getting commit access on ishmael !
[23:19:02] <binasher>	 thanks :)
[23:19:12] <hashar>	 off to bed for now and thanks for wikivoyage!
[23:19:33] <binasher>	 good night!
[23:19:56] <AaronSchulz>	 binasher: you didn't git rid of $wgObjectCaches['memcached-multiwrite'] = ...
[23:20:22] <AaronSchulz>	 no need to leave that code
[23:20:35] <binasher>	 good point
[23:21:39] <gerrit-wm>	 New patchset: Kaldari; "Adding Flickr settings for UploadWizard" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33291
[23:23:25] <gerrit-wm>	 New patchset: Asher; "ssl-only redirect logic shouldn't be entangled within an optional geoip config block" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33292
[23:23:55] <gerrit-wm>	 Change merged: Kaldari; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33291
[23:24:04] <gerrit-wm>	 Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33292
[23:24:44] <nagios-wm>	 PROBLEM - MySQL Replication Heartbeat on db39 is CRITICAL: CRIT replication delay 228 seconds
[23:25:02] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db11 is CRITICAL: CRIT replication delay 246 seconds
[23:25:20] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db39 is CRITICAL: CRIT replication delay 263 seconds
[23:25:29] <nagios-wm>	 PROBLEM - MySQL Replication Heartbeat on db64 is CRITICAL: CRIT replication delay 272 seconds
[23:25:29] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db64 is CRITICAL: CRIT replication delay 272 seconds
[23:25:29] <nagios-wm>	 PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 272 seconds
[23:25:29] <nagios-wm>	 PROBLEM - MySQL Replication Heartbeat on db1019 is CRITICAL: CRIT replication delay 274 seconds
[23:25:47] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db1019 is CRITICAL: CRIT replication delay 290 seconds
[23:25:56] <nagios-wm>	 PROBLEM - MySQL Replication Heartbeat on db11 is CRITICAL: CRIT replication delay 300 seconds
[23:26:14] <nagios-wm>	 PROBLEM - MySQL Replication Heartbeat on db1010 is CRITICAL: CRIT replication delay 317 seconds
[23:26:23] <nagios-wm>	 PROBLEM - MySQL Replication Heartbeat on db1003 is CRITICAL: CRIT replication delay 326 seconds
[23:28:33] <Reedy>	 That would be me...
[23:30:50] <AaronSchulz>	 binasher: are you amending?
[23:31:20] <binasher>	 Reedy: under control?
[23:31:32] <Reedy>	 Yeah, I just killed what I was doing
[23:31:36] <binasher>	 AaronSchulz: when i'm done what i'm working on
[23:31:41] <Reedy>	 Killing 37 million rows of crap doesn't seem to replicate so well
[23:33:17] <Reedy>	 I'm guessing because said table has proper PK
[23:37:02] <nagios-wm>	 RECOVERY - MySQL Replication Heartbeat on db1019 is OK: OK replication delay 1 seconds
[23:37:11] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db64 is OK: OK replication delay 0 seconds
[23:37:11] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db1019 is OK: OK replication delay 0 seconds
[23:37:56] <nagios-wm>	 RECOVERY - MySQL Replication Heartbeat on db39 is OK: OK replication delay 0 seconds
[23:38:32] <nagios-wm>	 RECOVERY - MySQL Replication Heartbeat on db64 is OK: OK replication delay 0 seconds
[23:38:41] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db39 is OK: OK replication delay 0 seconds
[23:39:44] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db1010 is CRITICAL: CRIT replication delay 244 seconds
[23:40:47] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db1003 is CRITICAL: CRIT replication delay 306 seconds
[23:40:56] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 317 seconds
[23:43:36] <Reedy>	 just down to the eqiad slaves now
[23:45:53] <nagios-wm>	 RECOVERY - MySQL Replication Heartbeat on db11 is OK: OK replication delay 0 seconds
[23:46:20] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db11 is OK: OK replication delay 0 seconds
[23:46:47] <gerrit-wm>	 New review: Erik Zachte; "I talked to Diederik and the basic idea seems good to me. So it boils down to adequate testing now. ..." [operations/puppet] (production) C: 1;  - https://gerrit.wikimedia.org/r/12188
[23:48:39] <gerrit-wm>	 New patchset: Asher; "disable use of MemcachedPhpBagOStuff and memcached on apache servers" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33289
[23:48:53] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db1003 is OK: OK replication delay 0 seconds
[23:49:11] <nagios-wm>	 RECOVERY - MySQL Replication Heartbeat on db1010 is OK: OK replication delay 0 seconds
[23:49:18] <binasher>	 AaronSchulz: ^^ still leaving the config for the apache servers in place for now, but removed the mutliwrite clause entirely
[23:49:29] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db1010 is OK: OK replication delay 0 seconds
[23:49:47] <nagios-wm>	 RECOVERY - MySQL Replication Heartbeat on db1003 is OK: OK replication delay 0 seconds
[23:51:17] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:51:58] <paravoid>	 AaronSchulz: hi
[23:52:31] <paravoid>	 AaronSchulz: I'll have a look at it tomorrow
[23:53:18] <gerrit-wm>	 New review: Diederik; "Fundraising has explicitly requested us to wait until January with rolling out this feature. We will..." [operations/puppet] (production) C: 0;  - https://gerrit.wikimedia.org/r/12188
[23:53:41] <nagios-wm>	 RECOVERY - MySQL Replication Heartbeat on db1035 is OK: OK replication delay 0 seconds
[23:54:08] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 0 seconds
[23:54:17] <Nemo_bis>	 paravoid: do you know where this HTML page is stored/defined? http://dumps.wikimedia.org/other/pagecounts-raw/
[23:55:30] <Reedy>	 Probably more of a question for Ariel when it's not past midnight
[23:55:37] <paravoid>	 yeah
[23:55:53] <paravoid>	 seems to be on dataset2, but apergos knows the ins and outs of dumps
[23:56:00] <paravoid>	 combined with how I know nothing about them
[23:56:39] <paravoid>	 AaronSchulz: did you see /mnt/thumbs2?
[23:57:48] <AaronSchulz>	 no
[23:57:51] <Nemo_bis>	 thanks
[23:58:03] <paravoid>	 yeah, there was an RT for you, to enable multiwrite
[23:58:13] <paravoid>	 we now have /mnt/thumbs2, mounted from the NetApp
[23:58:23] <paravoid>	 synced semi-asynchronously to eqiad
[23:59:10] <Reedy>	 awesome naming ;)
[23:59:51] <paravoid>	 heh