[00:16:15] tstarling cleared profiling data [01:15:09] PROBLEM - Disk space on stafford is CRITICAL: DISK CRITICAL - free space: /var/lib/puppet 748 MB (3% inode=92%): [02:03:10] aaron cleared profiling data [02:17:21] !log LocalisationUpdate completed (1.19) at Sun Mar 25 02:17:21 UTC 2012 [02:17:25] Logged the message, Master [03:25:12] PROBLEM - Disk space on srv224 is CRITICAL: DISK CRITICAL - free space: / 106 MB (1% inode=61%): /var/lib/ureadahead/debugfs 106 MB (1% inode=61%): [03:40:40] RECOVERY - Disk space on srv224 is OK: DISK OK [04:03:28] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [04:05:25] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [04:15:28] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [04:15:28] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [04:27:28] RECOVERY - Disk space on stafford is OK: DISK OK [04:35:17] PROBLEM - Disk space on search1022 is CRITICAL: DISK CRITICAL - free space: /a 4009 MB (3% inode=99%): [04:35:17] PROBLEM - Disk space on search1021 is CRITICAL: DISK CRITICAL - free space: /a 4009 MB (3% inode=99%): [04:56:26] PROBLEM - Disk space on search1021 is CRITICAL: DISK CRITICAL - free space: /a 4277 MB (3% inode=99%): [04:58:41] PROBLEM - Disk space on search1022 is CRITICAL: DISK CRITICAL - free space: /a 4033 MB (3% inode=99%): [06:02:34] PROBLEM - Disk space on search1021 is CRITICAL: DISK CRITICAL - free space: /a 4012 MB (3% inode=99%): [06:02:34] PROBLEM - Disk space on search1022 is CRITICAL: DISK CRITICAL - free space: /a 4013 MB (3% inode=99%): [06:15:19] PROBLEM - Disk space on search1022 is CRITICAL: DISK CRITICAL - free space: /a 4013 MB (3% inode=99%): [06:37:53] PROBLEM - Disk space on search1022 is CRITICAL: DISK CRITICAL - free space: /a 4166 MB (3% inode=99%): [06:55:52] PROBLEM - Squid on brewster is CRITICAL: Connection refused [07:21:31] PROBLEM - Puppet freshness on db59 is CRITICAL: Puppet has not run in the last 10 hours [07:26:29] PROBLEM - Disk space on search1022 is CRITICAL: DISK CRITICAL - free space: /a 3815 MB (3% inode=99%): [07:36:59] PROBLEM - Disk space on search1022 is CRITICAL: DISK CRITICAL - free space: /a 3914 MB (3% inode=99%): [07:59:56] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [11:40:02] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:41:59] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [12:28:41] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [12:44:26] PROBLEM - check_job_queue on spence is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 9,999 jobs: , commonswiki (11362) [12:53:56] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.056 second response time [13:25:35] RECOVERY - check_job_queue on spence is OK: JOBQUEUE OK - all job queues below 10,000 [13:29:11] PROBLEM - Puppet freshness on mw19 is CRITICAL: Puppet has not run in the last 10 hours [13:31:17] PROBLEM - Puppet freshness on ssl2 is CRITICAL: Puppet has not run in the last 10 hours [13:32:11] PROBLEM - Puppet freshness on mw33 is CRITICAL: Puppet has not run in the last 10 hours [13:35:11] PROBLEM - Puppet freshness on mw27 is CRITICAL: Puppet has not run in the last 10 hours [13:37:17] PROBLEM - Puppet freshness on ms-be3 is CRITICAL: Puppet has not run in the last 10 hours [13:42:14] PROBLEM - Puppet freshness on search1016 is CRITICAL: Puppet has not run in the last 10 hours [13:42:54] Krinkle, is TestSwarm among the things to fix after git conversion? [13:43:01] Yes [13:43:26] Among the things being fixed, rather [13:43:29] I'm working on it [13:44:11] PROBLEM - Puppet freshness on mw59 is CRITICAL: Puppet has not run in the last 10 hours [13:44:23] There's a fair amount of things involved due some secondary components failing indirectly now, so it's not just swapping 'svn checkout' for 'git clone' [13:45:42] Krinkle, yep, I guessed so :) [13:46:17] PROBLEM - Puppet freshness on mw1073 is CRITICAL: Puppet has not run in the last 10 hours [13:49:17] PROBLEM - Puppet freshness on mw45 is CRITICAL: Puppet has not run in the last 10 hours [13:55:17] PROBLEM - Puppet freshness on mw30 is CRITICAL: Puppet has not run in the last 10 hours [13:55:17] PROBLEM - Puppet freshness on mw72 is CRITICAL: Puppet has not run in the last 10 hours [13:55:17] PROBLEM - Puppet freshness on search1006 is CRITICAL: Puppet has not run in the last 10 hours [13:56:48] could someone explain why this svg isnt rendering? [13:56:49] http://en.wikipedia.org/wiki/Template:Ranks_and_insignia_of_NATO/Generic/Army/OR-08 [14:07:16] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [14:07:16] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [14:17:37] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [14:17:37] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [14:56:10] PROBLEM - Packetloss_Average on emery is CRITICAL: CRITICAL: packet_loss_average is 14.0073177586 (gt 8.0) [15:02:28] RECOVERY - Packetloss_Average on emery is OK: OK: packet_loss_average is 1.96235805085 [15:35:48] PROBLEM - Packetloss_Average on emery is CRITICAL: CRITICAL: packet_loss_average is 8.15920394737 (gt 8.0) [15:42:06] RECOVERY - Packetloss_Average on emery is OK: OK: packet_loss_average is 0.6930475 [16:18:24] PROBLEM - Host ps1-a1-sdtpa is DOWN: PING CRITICAL - Packet loss = 0%, RTA = 2060.24 ms [16:21:06] RECOVERY - Host ps1-a1-sdtpa is UP: PING OK - Packet loss = 0%, RTA = 2.95 ms [16:26:30] PROBLEM - Packetloss_Average on emery is CRITICAL: CRITICAL: packet_loss_average is 9.88808470085 (gt 8.0) [16:32:48] RECOVERY - Packetloss_Average on emery is OK: OK: packet_loss_average is 0.604784869565 [16:45:04] PROBLEM - Puppet freshness on brewster is CRITICAL: Puppet has not run in the last 10 hours [17:23:01] PROBLEM - Puppet freshness on db59 is CRITICAL: Puppet has not run in the last 10 hours [18:01:32] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [19:06:29] RECOVERY - MySQL Replication Heartbeat on db52 is OK: OK replication delay 0 seconds [19:07:23] RECOVERY - MySQL Slave Delay on db52 is OK: OK replication delay 0 seconds [19:38:05] a [19:38:11] hi [19:39:00] hi [19:39:14] hi [19:41:16] hi [19:41:25] ola [19:41:28] hi [19:41:56] k tl? [19:44:43] brion: I basically said in the email that even if SVG/PNG fallback works perfectly, SVG writers suddenly have five slightly different renderers to cater to. [19:45:02] Jarry1250: that is a true fact! [19:45:16] most of the time things work reasonably consistently, but when they break, oh boy can they break ;) [19:45:27] Also I don't know about fonts. [19:45:42] Which browsers allow webfonts in SVGs, that is the question... [19:45:46] fonts is one issue to consider; may be some overlap with webfonts [19:45:47] *nod* [19:46:22] .......................... [19:46:26] I mean, we've also got the issue that loads of our SVG images are going to well, break. [19:46:35] I don't care so much about that [19:46:47] bye [19:46:56] because at some point we're going to ditch SVG->PNG conversion anyway [19:47:17] So we're always going to experience that problem at some point [19:47:36] (Oooh, look at me using "we" there like I own the place...) [19:53:26] :) [19:53:34] we're very inclusive here :) [19:54:59] :P [20:01:57] we are inclusive? [20:02:23] I see no evidence of inclusion :p [20:12:56] PROBLEM - Host ps1-a4-eqiad is DOWN: PING CRITICAL - Packet loss = 44%, RTA = 2303.25 ms [20:14:44] RECOVERY - Host ps1-a4-eqiad is UP: PING OK - Packet loss = 0%, RTA = 29.23 ms [21:33:47] PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 281 MB (3% inode=61%): /var/lib/ureadahead/debugfs 281 MB (3% inode=61%): [21:46:23] RECOVERY - Disk space on srv221 is OK: DISK OK [21:47:59] Joan, are we sure that gerrit is a good prefix for https://bugzilla.wikimedia.org/show_bug.cgi?id=35409 ? [21:48:07] It's twice as long as rev: [21:48:17] dunno what else to use [21:48:41] It's shorter than "bugzilla". [21:48:55] Which is probably used much more often. [21:49:14] And ti's shorter than "https://gerrit.wikimedia.org/r/". [21:49:49] At least as an alias, it has to exist. But maybe something shorter is possible in the future. [21:49:55] I was just hoping that someone smart could come up with something genial. [21:50:06] Dunno, we don't add shortcuts usually. [21:50:39] We could use mwr: [21:51:01] It's used somewhere for the current svn revisions, and could be recycled. [21:52:10] Also, what will happen of svn: [21:53:27] it's * [21:53:39] Every interwiki link on that list is a shortcut. [21:55:31] Joan, I mean that we don't add duplicate entries (as in, where second column equals). [21:55:50] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [21:55:50] Of course it can change. [21:57:49] http://p.defau.lt/?Nw1_UyheeqNChhjTTsG0qg [21:58:09] > eĉei [21:58:15] Heh, is that legal? [21:59:02] I like how dictionary and dict don't go to Wiktionary. [22:00:11] :) [22:00:30] even interwikis with ! work afaik [22:01:00] Hmm, yeah, works. [22:01:08] Odd. [22:01:33] Space even [22:01:39] IIRC [22:01:50] And hyphens. [22:03:22] 12 article feedback tables... [22:16:33] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.054 second response time [22:24:12] PROBLEM - Host ps1-b3-eqiad is DOWN: PING CRITICAL - Packet loss = 37%, RTA = 2238.17 ms [22:26:18] RECOVERY - Host ps1-b3-eqiad is UP: PING OK - Packet loss = 0%, RTA = 29.42 ms [22:46:54] nighty~ [23:31:15] PROBLEM - Puppet freshness on mw19 is CRITICAL: Puppet has not run in the last 10 hours [23:35:22] PROBLEM - Puppet freshness on mw33 is CRITICAL: Puppet has not run in the last 10 hours [23:35:22] PROBLEM - Puppet freshness on ssl2 is CRITICAL: Puppet has not run in the last 10 hours [23:36:34] PROBLEM - Puppet freshness on mw27 is CRITICAL: Puppet has not run in the last 10 hours [23:38:40] PROBLEM - Puppet freshness on ms-be3 is CRITICAL: Puppet has not run in the last 10 hours [23:43:46] PROBLEM - Puppet freshness on search1016 is CRITICAL: Puppet has not run in the last 10 hours [23:45:34] PROBLEM - Puppet freshness on mw59 is CRITICAL: Puppet has not run in the last 10 hours [23:47:40] PROBLEM - Puppet freshness on mw1073 is CRITICAL: Puppet has not run in the last 10 hours [23:50:40] PROBLEM - Puppet freshness on mw45 is CRITICAL: Puppet has not run in the last 10 hours [23:56:40] PROBLEM - Puppet freshness on mw30 is CRITICAL: Puppet has not run in the last 10 hours [23:56:40] PROBLEM - Puppet freshness on search1006 is CRITICAL: Puppet has not run in the last 10 hours [23:56:40] PROBLEM - Puppet freshness on mw72 is CRITICAL: Puppet has not run in the last 10 hours