[00:18:50] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:44] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 6.959 seconds [00:54:05] PROBLEM - Puppet freshness on cadmium is CRITICAL: Puppet has not run in the last 10 hours [00:57:59] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:03:50] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.386 seconds [01:37:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:43:44] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.043 seconds [02:17:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:17:55] !log LocalisationUpdate completed (1.18) at Sun Feb 26 02:17:55 UTC 2012 [02:18:00] Logged the message, Master [02:23:40] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 3.748 seconds [02:28:28] PROBLEM - Puppet freshness on mw1010 is CRITICAL: Puppet has not run in the last 10 hours [02:28:28] PROBLEM - Disk space on srv249 is CRITICAL: DISK CRITICAL - free space: / 284 MB (3% inode=63%): /var/lib/ureadahead/debugfs 284 MB (3% inode=63%): [02:34:28] PROBLEM - Puppet freshness on mw1020 is CRITICAL: Puppet has not run in the last 10 hours [02:34:28] PROBLEM - Puppet freshness on mw1110 is CRITICAL: Puppet has not run in the last 10 hours [02:34:55] !log LocalisationUpdate completed (1.19) at Sun Feb 26 02:34:55 UTC 2012 [02:34:58] Logged the message, Master [03:30:28] PROBLEM - Disk space on mw61 is CRITICAL: DISK CRITICAL - free space: / 284 MB (3% inode=63%): /var/lib/ureadahead/debugfs 284 MB (3% inode=63%): [04:17:16] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [04:23:16] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [04:23:16] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [04:48:55] RECOVERY - Disk space on search1018 is OK: DISK OK [04:50:43] RECOVERY - Disk space on search1017 is OK: DISK OK [05:54:18] PROBLEM - Disk space on search1017 is CRITICAL: DISK CRITICAL - free space: /a 4282 MB (3% inode=99%): [05:54:27] PROBLEM - Disk space on search1018 is CRITICAL: DISK CRITICAL - free space: /a 4286 MB (3% inode=99%): [06:42:34] RECOVERY - Disk space on mw61 is OK: DISK OK [06:46:10] PROBLEM - Squid on brewster is CRITICAL: Connection refused [06:50:13] RECOVERY - Squid on brewster is OK: TCP OK - 0.010 second response time on port 8080 [06:53:49] RECOVERY - Disk space on srv249 is OK: DISK OK [07:57:09] PROBLEM - Host cadmium is DOWN: PING CRITICAL - Packet loss = 100% [08:13:35] sigh, Roan actually killed it [08:21:24] Anyone awake? I've got an image problem. http://commons.wikimedia.org/wiki/File:Wichita,_Portraits-Kiowa_women_-_NARA_-_523844repairedCropped.tif - the orininal displays fine but the .jpg thumbs have errors... Is that expected? [08:24:53] no, it's not expected. [08:25:02] please try purging the File page [08:25:18] this will force the thumbs to be removed and regenerated [08:25:53] regeneration of thumbs not on the File page itself might be delyed for 1-2 minutes, so if you're looking at a thumb in some page you'll want to give it tht long [08:26:00] then see if the problem is still there. [08:26:22] if it's still there I'll ask you to report it as a bug in bugzilla [08:26:38] could be https://bugzilla.wikimedia.org/show_bug.cgi?id=34718 again? [08:26:54] apergos: No change [08:28:13] I assume you have ctrl-f5ed the page with the thumb too, yeah? [08:28:46] yeah. Nothing. Still has errors [08:28:49] ok [08:29:05] please open a new bug for it [08:29:24] ah, these don't show up at all, I see [08:29:53] Yes. And the right-click, view image returns "this image has errors and cannot be displayed" [08:30:06] someone will have to look at it to find out if it's memory, a problem with tiff conversion, a problem with the original for some reason, etc [08:30:26] OK. So you still want me to file a bug? [08:30:55] yes please [08:31:07] it might turn out to be a dup but that's ok [08:31:15] best to have it investigated as something new [08:37:01] apergos: Filed: https://bugzilla.wikimedia.org/show_bug.cgi?id=34724 [08:38:41] thanks, that's enough for someone to be able to look at it [08:39:17] Excellent! Thank you. I hope we get it sorted out soon. [10:34:42] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:36:39] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [11:00:18] New review: Hashar; "(no comment)" [test/mediawiki/extensions/examples] (master) C: 2; - https://gerrit.wikimedia.org/r/2776 [11:00:18] Change merged: Hashar; [test/mediawiki/extensions/examples] (master) - https://gerrit.wikimedia.org/r/2776 [11:58:59] hi. how to make the Babel extensio add categories to the user page in some format? [11:59:48] malafaya: there are some configuration options for that [12:00:11] http://www.mediawiki.org/wiki/Extension:Babel#Configuration [12:00:12] the ones at https://www.mediawiki.org/wiki/Extension:Babel do not seems to be available for me [12:00:22] what do you mean? [12:00:40] maybe it should be changed in $wgBabelCategoryNames ? [12:00:50] all others seem to be related to auto-creation [12:05:08] 0 [12:05:11] wce [12:05:15] Nikerabbit, I found out: https://bugzilla.wikimedia.org/show_bug.cgi?id=33408 [12:05:23] a bug is needed I suppose [12:13:24] I added bug 34728. Probably it will be Reedy who will check what's going on :) [12:29:40] PROBLEM - Puppet freshness on mw1010 is CRITICAL: Puppet has not run in the last 10 hours [12:35:40] PROBLEM - Puppet freshness on mw1020 is CRITICAL: Puppet has not run in the last 10 hours [12:35:40] PROBLEM - Puppet freshness on mw1110 is CRITICAL: Puppet has not run in the last 10 hours [13:26:07] PROBLEM - Puppet freshness on lvs1002 is CRITICAL: Puppet has not run in the last 10 hours [14:19:13] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [14:25:49] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [14:25:49] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [15:50:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:42] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 4.242 seconds [15:56:52] could I ask someone to explain me why Bug 32077 is considered as unimportant? [16:05:42] 'cause you filed it [16:05:43] :P [16:26:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:32:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.024 seconds [17:05:08] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 2.963 seconds [17:44:53] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:50:44] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 2.772 seconds [17:51:29] PROBLEM - Apache HTTP on mw37 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:29] PROBLEM - Apache HTTP on mw53 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:38] PROBLEM - Apache HTTP on mw38 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:47] PROBLEM - Apache HTTP on mw15 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:56] PROBLEM - Apache HTTP on mw21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:05] PROBLEM - Apache HTTP on mw49 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:05] PROBLEM - Apache HTTP on mw43 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:12] The site is running very slowly [17:52:14] PROBLEM - Apache HTTP on srv246 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:14] PROBLEM - Apache HTTP on srv235 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:23] PROBLEM - Apache HTTP on mw4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:23] PROBLEM - Apache HTTP on mw56 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:23] PROBLEM - Apache HTTP on srv241 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:27] hmm not for me [17:52:32] PROBLEM - Apache HTTP on srv242 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:41] PROBLEM - Apache HTTP on mw5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:41] PROBLEM - Apache HTTP on mw22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:50] PROBLEM - Apache HTTP on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:50] PROBLEM - Apache HTTP on mw35 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:50] PROBLEM - Apache HTTP on mw45 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:50] PROBLEM - Apache HTTP on mw25 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:59] PROBLEM - Apache HTTP on mw47 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:59] PROBLEM - Apache HTTP on mw52 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:00] PROBLEM - Apache HTTP on srv243 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:08] PROBLEM - Apache HTTP on mw19 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:08] PROBLEM - Apache HTTP on mw41 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:08] PROBLEM - Apache HTTP on mw40 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:08] PROBLEM - Apache HTTP on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:17] PROBLEM - Apache HTTP on mw6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:18] PROBLEM - Apache HTTP on mw54 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:18] PROBLEM - Apache HTTP on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:18] PROBLEM - Apache HTTP on mw26 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:18] PROBLEM - Apache HTTP on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:20] TBloemink, maybe your accessing from somewhere other than Tampa [17:53:26] Yep [17:53:26] PROBLEM - Apache HTTP on mw14 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:26] PROBLEM - Apache HTTP on mw31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:26] PROBLEM - Apache HTTP on mw42 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:26] PROBLEM - Apache HTTP on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:35] oh, nvm [17:53:35] PROBLEM - Apache HTTP on mw7 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:35] PROBLEM - Apache HTTP on mw48 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:35] PROBLEM - Apache HTTP on mw27 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:35] PROBLEM - Apache HTTP on mw20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:38] for me too [17:53:44] PROBLEM - Apache HTTP on mw51 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:44] PROBLEM - Apache HTTP on mw46 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:02] PROBLEM - Apache HTTP on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:03] PROBLEM - Apache HTTP on mw55 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:03] RECOVERY - Apache HTTP on srv246 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 2.402 second response time [17:54:03] PROBLEM - Apache HTTP on mw28 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:11] RECOVERY - Apache HTTP on srv235 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.026 second response time [17:54:11] PROBLEM - Apache HTTP on mw39 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:20] RECOVERY - Apache HTTP on srv241 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.025 second response time [17:54:21] PROBLEM - Apache HTTP on mw34 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:21] PROBLEM - Apache HTTP on mw29 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:21] PROBLEM - Apache HTTP on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:29] RECOVERY - Apache HTTP on srv242 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.037 second response time [17:54:29] PROBLEM - Apache HTTP on mw33 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:35] Looks like every apache server crashed [17:54:38] PROBLEM - Apache HTTP on mw24 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:38] PROBLEM - Apache HTTP on mw57 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:42] great [17:54:45] Or most of them [17:54:48] PROBLEM - Apache HTTP on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:48] PROBLEM - Apache HTTP on mw17 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:56] RECOVERY - Apache HTTP on srv243 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.030 second response time [17:55:06] PROBLEM - Apache HTTP on mw18 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:14] PROBLEM - Apache HTTP on mw36 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:41] PROBLEM - Apache HTTP on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:46] apache genocide!!! [17:55:59] PROBLEM - Apache HTTP on mw32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:56:09] Looks like a cascading failure [17:56:26] PROBLEM - Apache HTTP on mw44 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:56:31] Any sysadmins here? [17:56:53] PROBLEM - Apache HTTP on mw58 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:11] PROBLEM - Apache HTTP on mw30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:57:11] RECOVERY - Apache HTTP on mw1 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 6.160 second response time [17:57:20] RECOVERY - Apache HTTP on mw6 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 9.402 second response time [17:57:21] RECOVERY - Apache HTTP on mw31 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.897 second response time [17:57:21] RECOVERY - Apache HTTP on mw42 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 5.115 second response time [17:57:29] RECOVERY - Apache HTTP on mw27 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 5.583 second response time [17:57:38] RECOVERY - Apache HTTP on mw53 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 8.477 second response time [17:57:38] RECOVERY - Apache HTTP on mw2 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 8.067 second response time [17:57:39] RECOVERY - Apache HTTP on mw48 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 9.358 second response time [17:57:47] RECOVERY - Apache HTTP on mw46 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.028 second response time [17:57:48] RECOVERY - Apache HTTP on mw15 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 5.455 second response time [17:57:56] RECOVERY - Apache HTTP on mw32 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.030 second response time [17:57:57] RECOVERY - Apache HTTP on mw21 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.043 second response time [17:57:57] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.038 second response time [17:57:57] RECOVERY - Apache HTTP on mw55 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.044 second response time [17:57:57] RECOVERY - Apache HTTP on mw49 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.028 second response time [17:57:57] RECOVERY - Apache HTTP on mw28 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.018 second response time [17:58:00] whoa [17:58:05] RECOVERY - Apache HTTP on mw39 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [17:58:05] RECOVERY - Apache HTTP on mw43 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.019 second response time [17:58:05] RECOVERY - Apache HTTP on mw38 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.033 second response time [17:58:05] RECOVERY - Apache HTTP on mw51 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.030 second response time [17:58:14] RECOVERY - Apache HTTP on mw9 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.021 second response time [17:58:14] RECOVERY - Apache HTTP on mw56 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.028 second response time [17:58:23] RECOVERY - Apache HTTP on mw44 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.025 second response time [17:58:23] RECOVERY - Apache HTTP on mw33 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [17:58:23] RECOVERY - Apache HTTP on mw34 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.035 second response time [17:58:23] RECOVERY - Apache HTTP on mw29 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.032 second response time [17:58:23] RECOVERY - Apache HTTP on mw4 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.039 second response time [17:58:32] RECOVERY - Apache HTTP on mw57 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.028 second response time [17:58:33] RECOVERY - Apache HTTP on mw24 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.037 second response time [17:58:37] They fix themselves :) [17:58:41] RECOVERY - Apache HTTP on mw58 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [17:58:41] RECOVERY - Apache HTTP on mw3 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [17:58:41] RECOVERY - Apache HTTP on mw17 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.030 second response time [17:58:41] RECOVERY - Apache HTTP on mw35 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [17:58:50] RECOVERY - Apache HTTP on mw25 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.023 second response time [17:58:51] RECOVERY - Apache HTTP on mw45 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.029 second response time [17:58:51] RECOVERY - Apache HTTP on mw47 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.041 second response time [17:58:51] RECOVERY - Apache HTTP on mw52 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.040 second response time [17:58:51] RECOVERY - Apache HTTP on mw5 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.041 second response time [17:58:59] RECOVERY - Apache HTTP on mw22 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.026 second response time [17:58:59] RECOVERY - Apache HTTP on mw19 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.036 second response time [17:58:59] RECOVERY - Apache HTTP on mw41 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.032 second response time [17:58:59] RECOVERY - Apache HTTP on mw30 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.033 second response time [17:58:59] RECOVERY - Apache HTTP on mw18 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.037 second response time [17:59:08] RECOVERY - Apache HTTP on mw12 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [17:59:09] RECOVERY - Apache HTTP on mw40 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.032 second response time [17:59:09] RECOVERY - Apache HTTP on mw36 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.026 second response time [17:59:09] RECOVERY - Apache HTTP on mw11 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.023 second response time [17:59:09] RECOVERY - Apache HTTP on mw54 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [17:59:09] RECOVERY - Apache HTTP on mw13 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [17:59:10] RECOVERY - Apache HTTP on mw26 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [17:59:17] RECOVERY - Apache HTTP on mw14 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [17:59:18] RECOVERY - Apache HTTP on mw10 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.020 second response time [17:59:26] RECOVERY - Apache HTTP on mw37 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.026 second response time [17:59:35] RECOVERY - Apache HTTP on mw20 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [17:59:36] RECOVERY - Apache HTTP on mw7 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [18:05:34] bah, nagios-wm should spam on separate channel [18:18:20] it seems to be finish for the recoverys .. [18:24:56] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:30:47] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 3.324 seconds [19:06:13] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:12:04] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 4.332 seconds [19:46:16] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:49:45] !ops [19:50:01] too late, queen [19:50:02] :P [19:50:04] :D [19:50:10] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 6.394 seconds [19:50:22] first time I use that thingy [20:03:55] nighty~ [20:06:01] the idea of a caldav install on WMF cluster, vs. current use of Google Calendars by some devo activities, came up in conversation last night [20:06:12] before it goes anywhere - wanted to see if any ops folks see it as a dead in the water issue [20:06:28] as in "uh..no f'ing way" :) [20:25:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:31:39] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.746 seconds [21:05:51] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:11:42] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.986 seconds [21:11:47] https://upload.wikimedia.org/wikipedia/commons/thumb/f/fb/718smiley.svg/120px-718smiley.svg.png seems broken. [21:18:51] Interesting, https://upload.wikimedia.org/wikipedia/commons/thumb/f/fb/718smiley.svg/240px-718smiley.svg.png isn't [21:20:33] hexmode: what do you mean? [21:21:19] Akoopal: I mean it is interesting that the scalars have trouble with 120px versions of the file, but not 240px [21:21:28] ahh [21:22:08] both look ok to me? [21:22:21] both are okay for me (FF 10) [21:23:01] Akoopal: you see a thumnail here: https://commons.wikimedia.org/wiki/File:718smiley.svg [21:23:11] Quedel: I meant you [21:23:36] oh, nm [21:23:50] all thumbs and pictures look like they should be [21:23:59] it is there now... I did ?action=purge on the URL [21:24:06] you mean in the history? yes [21:24:11] ok :-) [21:24:18] but didn't see the first one in history [21:24:46] I see thumb from first version of 2008 April 23th [21:24:51] seems the purge helped [21:25:08] Oh. [21:25:15] Quedel: I meant "last" which is the URL (I think) that Joan gave [21:25:28] Yeah, it seems to have purged itself after enough poking. [21:25:40] The real question is how it stored a corrupt thumbnail in the first place. [21:25:45] agreed [21:25:46] you mean from 2011-09-04? but this is fine for me, too [21:26:16] Quedel: as Joan said "it seems to have purged itself after enough poking." [21:26:52] It really was broken when I filed the bug, I swear. ;-) [21:27:06] yeah, the servers aren't running as well as they should be. There are many caching-problems in different situations [21:27:08] I saw it! [21:27:17] we believe you, Joan [21:27:41] I'm relieved. [21:28:31] yeah, I still have one ticket open as well, which seems to be wrong on the squid side ... [21:40:46] Akoopal: ticket? [21:41:02] bugzilla [21:41:13] 34718 [21:44:08] Akoopal: ah, I already commented. was there an image in https://nl.wikipedia.org/wiki/Xiongnu ?? [21:45:02] hexmode: commented on that one already, that was a false report indeed [21:45:05] Akoopal: nm... reading the bug shows that you said there wasn't, but what issue did that one have? [21:45:16] the first one is still wrong [21:45:49] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:45:59] https://upload.wikimedia.org/wikipedia/commons/thumb/1/11/Ftacnik_lubomir_20081025_berlin_bundesliga.jpg/264px-Ftacnik_lubomir_20081025_berlin_bundesliga.jpg [21:46:05] that one fails still [21:46:28] Akoopal: right, got that, and Ben is looking at it. But what was wrong with the Xiongnu article? you said it was "something else" [21:46:46] indeed error in the template code [21:46:52] kk [21:47:36] as I saw both issues reported about the same time, investigated the first one, and reported both [21:48:13] but didn't deeply look in the second one unfortunately ... [21:49:11] o well, time for a episode of flying wild alaska now :-) [21:51:40] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 3.031 seconds [21:55:14] * hexmode needs to find a non-1.19 wiki with LQT [21:56:03] hrm... huwiki? [22:25:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:31:34] PROBLEM - Puppet freshness on mw1010 is CRITICAL: Puppet has not run in the last 10 hours [22:31:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 5.922 seconds [22:37:34] PROBLEM - Puppet freshness on mw1020 is CRITICAL: Puppet has not run in the last 10 hours [22:37:34] PROBLEM - Puppet freshness on mw1110 is CRITICAL: Puppet has not run in the last 10 hours [22:46:06] https://bugzilla.wikimedia.org/show_bug.cgi?id=34412 When is it processed? [23:07:48] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:11:51] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 9.061 seconds [23:27:36] PROBLEM - Puppet freshness on lvs1002 is CRITICAL: Puppet has not run in the last 10 hours [23:48:00] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:51:55] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 6.846 seconds [23:52:20] gn8 folks