[00:44:41] hmm [00:44:50] I was reading about 2013 [01:06:49] gn8 folks [02:06:03] !log LocalisationUpdate completed (1.18) at Sat Dec 24 02:06:03 UTC 2011 [02:06:15] Logged the message, Master [02:16:56] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1284s [02:22:26] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 712s [02:26:36] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:32:06] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [03:59:55] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [04:40:15] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [05:20:40] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [05:32:57] nighty [05:37:14] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [06:12:34] PROBLEM - Disk space on hume is CRITICAL: DISK CRITICAL - free space: /a/static/uncompressed 35276 MB (3% inode=99%): [07:54:43] !log removed manually log files olders than 14 days from /a/static/uncompressed on hume; looks like there's a problem with the script /usr/local/bin/offhost_backups on storage3 which was modified recently, guess the rsync delete option isn't being picked up [07:54:53] Logged the message, Master [07:55:11] RECOVERY - Disk space on hume is OK: DISK OK [09:07:54] apergos: what's up with upload.wikimedia.org :o it's terribly slow in here [09:08:02] no idea [09:08:20] ok [09:08:23] I can't stick around though: I set the water for a shower and it's time for me to go do that [09:08:30] :) [09:08:31] ok [09:08:42] petan: for thumbs or full size? [09:09:05] just in general when I open any page it stuck at loading something from upload [09:09:06] apergos: do you have your own personal water heater or something? [09:13:38] petan: well i was able to sustain 11mbit from upload.wm and [[special:newfiles]] @ commons looks ok [09:13:42] petan: http://commons.wikimedia.org/wiki/Special:NewFiles?showbots=1&limit=&offset= [09:13:54] ok... probably something with me [09:16:02] petan: well, try [[special:newfiles]]? or firebug's traffic analysis thing? or do you have another connection to try? (both same ISP and others) [09:16:16] I will [09:16:32] problem is not on commons but on all sites [09:16:44] when I open wikipedia I stuck on loading stuff from upload [09:16:56] but if it's only me I nvm [09:17:07] I thought there is some problem with that [09:18:29] there might be. e.g. europe vs. us [09:19:09] also, are you ssl'd [09:23:18] woot, i went to try torrus and it's not a 500! :) [09:26:40] torrus is less friendly for looking at lots of stuff at a glance [09:26:58] anyway, ganglia looked fairly normal [09:27:11] for the 2 upload squid clusters and the img scalers [09:28:42] petan: where art thou geographically? [09:30:28] wtf, http://ganglia.wikimedia.org/?c=Miscellaneous%20pmtpa&h=en.wikipedia.org&m=network_report&r=hour&s=by%20name&hc=3&mc=3 [09:31:42] anyway, i'm about to give up on upload slowness [09:44:07] RECOVERY - MySQL slave status on es1004 is OK: OK: [10:07:41] jeremyb: yes, people do [10:07:52] you turn it on 20 min to half an hour before you need hot water [10:08:15] this building doesn't have the solar ones on the roof so it's not as good but whatever [10:08:23] apergos: makes sense just never heard of an actual implementation [10:08:51] it's pretty much standard here except in hotels (the more expensive ones anyways) [10:09:39] apergos: i recently saw a bathroom in NY with a device on the wall near the sink that was plugged in to power and that made the hot water for the sink [10:10:25] well here there's just a circuit breaker of some kind (with the rest of them for the apartment) or sometimes a dial of sorts [10:10:58] here (i guess you know already) it's mostly one big shared unit for the whole building [10:11:38] was in a restaurant... hard to believe because they would presumably need hot water for e.g. dishwashing [10:11:58] how do you figure out the bills? [10:13:11] you don't. it's just all included [10:13:14] in the rent [10:13:25] ah, right [10:13:29] once in a very rare while you'll find a place that doesn't have hot water included [10:13:31] I guess they have to do it that way [10:14:01] and then i've seen (in the basement) individual boilers for each place (they didn't include heat either) [10:15:56] right [10:16:47] apergos: anyway, did you see which ganglia cats i checked? missing anything? [10:16:57] sorry? [10:17:01] I didn't see any of the backread [10:17:21] I just sat down for a couple of minutes so my hair gets drier before going out to get bread and stuff [10:17:34] for the 2 upload squid clusters and the img scalers [10:17:59] look at ms5 and ms6 also [10:18:02] also, not really relevant but < jeremyb> wtf, http://ganglia.wikimedia.org/?c=Miscellaneous%20pmtpa&h=en.wikipedia.org&m=network_report&r=hour&s=by%20name&hc=3&mc=3 [10:18:06] oh, right good idea [10:18:14] deending on whether they are europe you might be able to leave out ms6 [10:20:07] ok I'm gonna do this while it's sunny [10:20:08] see folks later [10:20:17] apergos: ganglia's broken on ms6? [10:20:19] apergos: enjoy [10:20:31] later [14:02:00] Any idea why this isn't working? https://outreach.wikimedia.org/wiki/Sandbox [14:09:27] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [15:29:45] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [15:49:39] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [16:01:29] PROBLEM - mobile traffic loggers on cp1044 is CRITICAL: PROCS CRITICAL: 5 processes with args varnishncsa [16:11:09] RECOVERY - mobile traffic loggers on cp1044 is OK: PROCS OK: 2 processes with args varnishncsa [16:46:06] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33042 - Set wgBabelMainCategory to false for nlwiki' [16:46:15] Logged the message, Master [16:49:11] !log reedy synchronized closed.dblist 'Locking readerfeedback.l.w.o per bg 33229' [16:49:20] Logged the message, Master [16:53:05] !log reedy synchronized wmf-config/InitialiseSettings.php 'Part of bug 33137 Enable GoogleNewsSitemap on ptwikinews' [16:53:14] Logged the message, Master [17:05:21] !log reedy synchronized wmf-config/InitialiseSettings.php 'bug 32641' [17:05:29] Logged the message, Master [17:08:01] !log reedy synchronized closed.dblist 'Add tenwiki to closed.dblist per bug 33185' [17:08:10] Logged the message, Master [20:12:59] PROBLEM - mobile traffic loggers on cp1041 is CRITICAL: PROCS CRITICAL: 7 processes with args varnishncsa [20:12:59] PROBLEM - mobile traffic loggers on cp1042 is CRITICAL: PROCS CRITICAL: 7 processes with args varnishncsa [20:20:07] zzz =_= [20:22:39] RECOVERY - mobile traffic loggers on cp1042 is OK: PROCS OK: 4 processes with args varnishncsa [20:41:59] RECOVERY - mobile traffic loggers on cp1041 is OK: PROCS OK: 1 process with args varnishncsa [21:08:13] PROBLEM - mobile traffic loggers on cp1043 is CRITICAL: PROCS CRITICAL: 5 processes with args varnishncsa [21:12:23] PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100% [21:13:04] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 1.90 ms [21:30:43] RECOVERY - mobile traffic loggers on cp1043 is OK: PROCS OK: 1 process with args varnishncsa [21:49:24] !log reedy synchronized wmf-config/InitialiseSettings.php 'kill tenwiki setting' [21:49:34] Logged the message, Master [21:50:43] Reedy: which tenwiki setting? [22:02:40] !log reedy synchronized docroot/wikipedia.org/favicon-si.ico [22:02:49] Logged the message, Master [22:03:34] !log Created flaggedrevs tables on ukwiki [22:03:42] Logged the message, Master [22:18:35] !log reedy synchronized wmf-config/InitialiseSettings.php 'bug 33137 adding import sources for ptwikinews' [22:18:44] Logged the message, Master [22:20:30] !log reedy synchronized wmf-config/InitialiseSettings.php 'bug 33137 Add transwiki namespace' [22:20:35] new wiki on the way? [22:20:39] Logged the message, Master [22:24:14] DaBPunkt: erm? [22:24:29] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33291 Add autopatrolled flag right on sa.wikipedia' [22:24:38] Logged the message, Master [22:27:18] DaBPunkt, nope, siwiki want a new favico [22:27:25] was trying to work out how to get it to play ball [22:28:04] solved? [22:28:13] ah ok. [22:28:38] nope [22:30:25] Reedy: can't it be just a clone of the way the site logo is handled? [22:30:35] * jeremyb will bbiab [22:32:52] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33200 - Enable Autopatrol user group and RC patol for sa wiki' [22:33:01] Logged the message, Master