[02:05:05] !log LocalisationUpdate completed (1.18) at Mon Jan 16 02:05:05 UTC 2012 [02:05:09] Logged the message, Master [02:08:14] Hmmm... anyone on? [02:18:28] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1536s [02:19:55] matthewrbowker: ? [02:21:20] Ryan_Lane: http://test.wikipedia.org/?banner=blackout Banner doesn't go all the way to the bottom. [02:21:36] ah [02:21:37] I see [02:21:53] kaldari is working on this [02:22:01] we're doing a sprint tomorrow [02:22:06] Is it supposed to float? (it does on my iPod Touch) [02:22:06] I'll make sure to point this out [02:22:13] Thanks :) [02:22:47] yw [02:24:08] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1876s [02:28:18] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:33:58] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:46:38] PROBLEM - Puppet freshness on brewster is CRITICAL: Puppet has not run in the last 10 hours [02:53:38] PROBLEM - Puppet freshness on sodium is CRITICAL: Puppet has not run in the last 10 hours [04:15:18] been a very quiet day... [04:15:47] PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [04:16:32] wtf, just because i said something? damn murphy [04:18:57] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 111 MB (1% inode=60%): /var/lib/ureadahead/debugfs 111 MB (1% inode=60%): [04:20:37] RECOVERY - Disk space on es1004 is OK: DISK OK [04:20:57] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:44:17] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [04:45:37] RECOVERY - Disk space on srv219 is OK: DISK OK [04:48:57] RECOVERY - Disk space on srv220 is OK: DISK OK [09:11:56] PROBLEM - Disk space on srv223 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [09:12:14] morening, any dev awake? apergos perhaps? [09:12:40] morning [09:12:43] rats. [09:12:52] that's what I get for being on line... what's up? [09:13:11] a warning, if it's not an emergency I'll probably leave it til tomorrow (but go ahead and ask) [09:15:36] !log cleaned up /tmp on srv223... seems like cleanup once an hour by cron isn't often enough any more, scalers are doing too much work [09:15:39] Logged the message, Master [09:22:09] RECOVERY - Disk space on srv223 is OK: DISK OK [09:22:22] matanya: ? [09:22:29] yes [09:22:47] I have an issue [09:23:21] in firefox from the diff screen, no "edit" links appear anymor [09:23:23] *e [09:23:43] what about chrome? [09:23:54] works [09:23:58] and IE as well [09:24:17] on he.wiki only, for now [09:27:00] If its only one wiki, It may be due to some local javascript or css they are running [09:28:11] I thought so too, but we disabled it all, and still. [09:28:48] give me one of the diff links please? [09:29:10] to correct - arwiki as well. might be RTL issue. [09:29:16] https://ar.wikipedia.org/w/index.php?title=%D9%88%D9%8A%D9%83%D9%8A%D8%A8%D9%8A%D8%AF%D9%8A%D8%A7:%D8%AA%D8%B1%D8%B4%D9%8A%D8%AD/%D8%B9%D8%A8%D8%A7%D8%AF_%D8%AF%D9%8A%D8%B1%D8%A7%D9%86%D9%8A%D8%A9&curid=1206804&diff=8017549&oldid=8016791 [09:29:30] https://he.wikipedia.org/w/index.php?title=%D7%A7%D7%95%D7%A0%D7%A6%27%D7%A8%D7%98%D7%95_%D7%9C%D7%A4%D7%A1%D7%A0%D7%AA%D7%A8_%D7%9E%D7%A1%27_2_%28%D7%A8%D7%97%D7%9E%D7%A0%D7%99%D7%A0%D7%95%D7%91%29&curid=970156&diff=11818090&oldid=11818079 [09:30:42] now, I see on some pages in ar there are links. for example - https://he.wikipedia.org/w/index.php?title=%D7%A7%D7%95%D7%A0%D7%A6%27%D7%A8%D7%98%D7%95_%D7%9C%D7%A4%D7%A1%D7%A0%D7%AA%D7%A8_%D7%9E%D7%A1%27_2_%28%D7%A8%D7%97%D7%9E%D7%A0%D7%99%D7%A0%D7%95%D7%91%29&curid=970156&diff=11818090&oldid=11818079 [09:30:42] I see an edit link tab [09:30:48] on the link you gave me [09:30:55] not the tab [09:30:57] usin their content language, not mine [09:31:02] where? [09:31:13] the side of a pharagrph [09:32:55] umm where are the edit links on the third link? [09:34:09] I don't know. now they disappeared [09:34:26] so you confirm this? [09:35:16] in IE9 they are missing as well [09:35:48] I don't see them in a regular article with a diff, without rtl [09:36:06] so a general issue. [09:36:27] I don't know if we were supposed to see them [09:36:30] but eg [09:36:34] http://en.wikipedia.org/w/index.php?title=Press_Gang&action=historysubmit&diff=471646076&oldid=471642319 [09:37:14] we did, until 2 days ago [09:37:34] then I would check bz and if it's not already reported, put in a report [09:38:25] thanks [09:38:50] thanks for reporting it [09:43:26] apergos: just to prove it: https://commons.wikimedia.org/w/index.php?title=User_talk:Matanya&diff=cur [09:44:10] to prove which? [09:44:21] we used to see. [09:44:29] I don't see them there either [09:44:39] I do :( [09:44:43] :-D [09:44:44] nice [09:44:53] screenshot it [09:44:54] :-P [09:47:07] https://commons.wikimedia.org/wiki/File:My_userpage,_for_a_bug_report.jpg.png [09:47:12] apergos: ^ [09:47:30] nice! [09:49:30] https://en.wikipedia.org/w/index.php?title=Danube&curid=21209639&diff=471652359&oldid=471459304 [09:50:19] no edit links for me [09:50:26] I just don't think I'm going to be shown any [09:51:17] no consistency. [09:51:30] heh for me there is [09:51:35] no links on any of them :-D [09:51:56] how can I discribe this? [09:52:14] like you did to me, mention the browsers that show the links and the ones that don't [09:52:20] point out whether you are logged in or not [09:52:33] try it with and without uselang=en so we can see that it's not rtl issue [09:52:47] thanks, will do [09:52:50] and mention when you started seeing pages without the links [09:53:01] sure [09:54:20] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 456098 MB (3% inode=99%): [09:59:57] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 432021 MB (3% inode=99%): [10:03:37] apergos: https://bugzilla.wikimedia.org/show_bug.cgi?id=33671 [10:04:30] ok great [10:32:34] RECOVERY - MySQL slave status on es1004 is OK: OK: [12:07:45] New patchset: Hashar; "testswarm: fix mw.org hostname" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1923 [12:38:43] RECOVERY - Puppet freshness on sodium is OK: puppet ran at Mon Jan 16 12:38:25 UTC 2012 [12:39:43] RECOVERY - HTTP on sodium is OK: HTTP OK HTTP/1.0 200 OK - 3827 bytes in 0.116 seconds [12:50:15] New patchset: Mark Bergsma; "Fix nrpe.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1924 [12:50:35] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1924 [12:50:36] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1924 [12:57:07] PROBLEM - Puppet freshness on brewster is CRITICAL: Puppet has not run in the last 10 hours [13:00:50] RECOVERY - mailman on sodium is OK: PROCS OK: 10 processes with args mailman [13:02:36] RECOVERY - spamassassin on sodium is OK: PROCS OK: 4 processes with args spamd [13:07:36] RECOVERY - DPKG on sodium is OK: All packages OK [13:16:15] New patchset: Mark Bergsma; "Disable grub2's hidden timeout, so we'll know to press ESC (RT#1914)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1925 [13:16:30] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1925 [13:18:15] RECOVERY - RAID on sodium is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [13:19:55] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1925 [13:19:55] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1925 [13:21:11] RECOVERY - HTTPS on sodium is OK: OK - Certificate will expire on 08/22/2015 22:23. [13:23:36] PROBLEM - RAID on nfs1 is CRITICAL: Connection refused by host [13:23:40] PROBLEM - Disk space on mw1084 is CRITICAL: Connection refused by host [13:23:47] PROBLEM - RAID on db1015 is CRITICAL: Connection refused by host [13:23:48] PROBLEM - Disk space on mw18 is CRITICAL: Connection refused by host [13:23:55] PROBLEM - Disk space on mw1092 is CRITICAL: Connection refused by host [13:23:55] PROBLEM - Disk space on mw1011 is CRITICAL: Connection refused by host [13:24:43] PROBLEM - DPKG on db1044 is CRITICAL: Connection refused by host [13:24:44] PROBLEM - RAID on db1044 is CRITICAL: Connection refused by host [13:25:24] PROBLEM - Disk space on mw1059 is CRITICAL: Connection refused by host [13:26:03] PROBLEM - Disk space on db1022 is CRITICAL: Connection refused by host [13:26:03] PROBLEM - DPKG on srv257 is CRITICAL: Connection refused by host [13:26:03] PROBLEM - Disk space on db1016 is CRITICAL: Connection refused by host [13:26:13] PROBLEM - MySQL disk space on db1015 is CRITICAL: Connection refused by host [13:26:23] PROBLEM - Disk space on locke is CRITICAL: Connection refused by host [13:26:46] PROBLEM - Disk space on db1044 is CRITICAL: Connection refused by host [13:27:03] PROBLEM - DPKG on mw67 is CRITICAL: Connection refused by host [13:27:13] PROBLEM - DPKG on mw1134 is CRITICAL: Connection refused by host [13:28:15] PROBLEM - DPKG on srv222 is CRITICAL: Connection refused by host [13:28:38] PROBLEM - Disk space on mw1101 is CRITICAL: Connection refused by host [13:28:38] PROBLEM - Disk space on mw1134 is CRITICAL: Connection refused by host [13:29:04] PROBLEM - Disk space on virt2 is CRITICAL: Connection refused by host [13:29:23] PROBLEM - RAID on mw67 is CRITICAL: Connection refused by host [13:29:23] PROBLEM - DPKG on mw1101 is CRITICAL: Connection refused by host [13:31:11] PROBLEM - Disk space on db1041 is CRITICAL: Connection refused by host [13:31:43] PROBLEM - Disk space on ms1002 is CRITICAL: Connection refused by host [13:31:53] PROBLEM - RAID on db51 is CRITICAL: Connection refused by host [13:32:15] PROBLEM - RAID on srv225 is CRITICAL: Connection refused by host [13:32:25] PROBLEM - mobile traffic loggers on cp1044 is CRITICAL: Connection refused by host [13:32:25] RECOVERY - RAID on srv278 is OK: OK: no RAID installed [13:32:47] PROBLEM - RAID on srv227 is CRITICAL: Connection refused by host [13:32:56] PROBLEM - RAID on cp1042 is CRITICAL: Connection refused by host [13:33:16] PROBLEM - DPKG on mw1044 is CRITICAL: Connection refused by host [13:33:16] PROBLEM - RAID on mw1104 is CRITICAL: Connection refused by host [13:33:16] PROBLEM - DPKG on mw1073 is CRITICAL: Connection refused by host [13:33:16] PROBLEM - DPKG on es1002 is CRITICAL: Connection refused by host [13:33:26] PROBLEM - DPKG on srv227 is CRITICAL: Connection refused by host [13:33:26] PROBLEM - RAID on mw1129 is CRITICAL: Connection refused by host [13:34:52] PROBLEM - Disk space on mw69 is CRITICAL: Connection refused by host [13:35:39] PROBLEM - DPKG on snapshot2 is CRITICAL: Connection refused by host [13:35:57] PROBLEM - Disk space on db1048 is CRITICAL: Connection refused by host [13:35:57] PROBLEM - RAID on srv253 is CRITICAL: Connection refused by host [13:35:58] RECOVERY - RAID on nfs1 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [13:36:15] RECOVERY - Disk space on mw1084 is OK: DISK OK [13:36:15] RECOVERY - RAID on db1015 is OK: OK: State is Optimal, checked 2 logical device(s) [13:36:29] PROBLEM - DPKG on db11 is CRITICAL: Connection refused by host [13:36:29] RECOVERY - Disk space on mw1092 is OK: DISK OK [13:36:29] RECOVERY - Disk space on mw1011 is OK: DISK OK [13:36:40] RECOVERY - Disk space on mw18 is OK: DISK OK [13:36:49] PROBLEM - Disk space on aluminium is CRITICAL: Connection refused by host [13:36:49] RECOVERY - DPKG on db1044 is OK: All packages OK [13:36:49] PROBLEM - RAID on srv200 is CRITICAL: Connection refused by host [13:36:49] RECOVERY - RAID on db1044 is OK: OK: State is Optimal, checked 2 logical device(s) [13:37:19] PROBLEM - RAID on db1040 is CRITICAL: Connection refused by host [13:37:19] PROBLEM - MySQL disk space on db1020 is CRITICAL: Connection refused by host [13:37:39] PROBLEM - DPKG on virt3 is CRITICAL: Connection refused by host [13:37:39] RECOVERY - Disk space on mw1059 is OK: DISK OK [13:38:01] RECOVERY - Disk space on db1022 is OK: DISK OK [13:38:11] PROBLEM - DPKG on db1040 is CRITICAL: Connection refused by host [13:38:12] RECOVERY - DPKG on srv257 is OK: All packages OK [13:38:23] RECOVERY - Disk space on db1016 is OK: DISK OK [13:38:33] RECOVERY - Disk space on locke is OK: DISK OK [13:38:41] RECOVERY - MySQL disk space on db1015 is OK: DISK OK [13:38:50] RECOVERY - Disk space on db1044 is OK: DISK OK [13:39:10] RECOVERY - DPKG on mw67 is OK: All packages OK [13:39:10] PROBLEM - DPKG on srv285 is CRITICAL: Connection refused by host [13:39:19] RECOVERY - DPKG on mw1134 is OK: All packages OK [13:39:42] PROBLEM - DPKG on mw70 is CRITICAL: Connection refused by host [13:39:45] PROBLEM - mobile traffic loggers on cp1043 is CRITICAL: Connection refused by host [13:40:00] PROBLEM - DPKG on mw1126 is CRITICAL: Connection refused by host [13:40:00] PROBLEM - DPKG on db34 is CRITICAL: Connection refused by host [13:40:19] PROBLEM - RAID on cp1043 is CRITICAL: Connection refused by host [13:40:31] PROBLEM - RAID on mw1035 is CRITICAL: Connection refused by host [13:40:31] PROBLEM - Disk space on virt3 is CRITICAL: Connection refused by host [13:40:31] RECOVERY - DPKG on srv222 is OK: All packages OK [13:40:31] PROBLEM - RAID on mw1029 is CRITICAL: Connection refused by host [13:40:53] PROBLEM - DPKG on srv251 is CRITICAL: Connection refused by host [13:40:53] RECOVERY - Disk space on mw1134 is OK: DISK OK [13:40:53] RECOVERY - Disk space on mw1101 is OK: DISK OK [13:41:11] RECOVERY - Disk space on virt2 is OK: DISK OK [13:41:41] PROBLEM - RAID on mw1015 is CRITICAL: Connection refused by host [13:41:41] RECOVERY - RAID on mw67 is OK: OK: no RAID installed [13:42:11] RECOVERY - DPKG on mw1101 is OK: All packages OK [13:42:24] PROBLEM - RAID on db1019 is CRITICAL: Connection refused by host [13:43:04] PROBLEM - Disk space on mw8 is CRITICAL: Connection refused by host [13:43:04] PROBLEM - Disk space on srv246 is CRITICAL: Connection refused by host [13:43:14] PROBLEM - DPKG on srv215 is CRITICAL: Connection refused by host [13:43:14] PROBLEM - RAID on db45 is CRITICAL: Connection refused by host [13:43:14] PROBLEM - RAID on mw1067 is CRITICAL: Connection refused by host [13:43:14] PROBLEM - Disk space on mw1159 is CRITICAL: Connection refused by host [13:43:24] PROBLEM - DPKG on mw1118 is CRITICAL: Connection refused by host [13:43:24] PROBLEM - MySQL disk space on db1033 is CRITICAL: Connection refused by host [13:43:24] RECOVERY - Disk space on db1041 is OK: DISK OK [13:43:24] PROBLEM - DPKG on es2 is CRITICAL: Connection refused by host [13:43:37] PROBLEM - Disk space on db45 is CRITICAL: Connection refused by host [13:44:07] RECOVERY - RAID on db51 is OK: OK: State is Optimal, checked 2 logical device(s) [13:44:25] PROBLEM - Disk space on es2 is CRITICAL: Connection refused by host [13:44:25] PROBLEM - DPKG on stafford is CRITICAL: Connection refused by host [13:44:35] RECOVERY - RAID on srv225 is OK: OK: no RAID installed [13:44:54] PROBLEM - DPKG on db25 is CRITICAL: Connection refused by host [13:45:13] RECOVERY - RAID on srv227 is OK: OK: no RAID installed [13:45:25] PROBLEM - RAID on srv262 is CRITICAL: Connection refused by host [13:45:25] PROBLEM - RAID on db1003 is CRITICAL: Connection refused by host [13:45:25] PROBLEM - DPKG on mw1072 is CRITICAL: Connection refused by host [13:45:25] PROBLEM - DPKG on mw1042 is CRITICAL: Connection refused by host [13:45:39] RECOVERY - RAID on cp1042 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [13:45:56] RECOVERY - DPKG on mw1044 is OK: All packages OK [13:45:56] RECOVERY - DPKG on mw1073 is OK: All packages OK [13:45:56] RECOVERY - RAID on mw1104 is OK: OK: no RAID installed [13:45:56] RECOVERY - DPKG on es1002 is OK: All packages OK [13:46:05] RECOVERY - DPKG on srv227 is OK: All packages OK [13:46:06] RECOVERY - RAID on mw1129 is OK: OK: no RAID installed [13:46:44] PROBLEM - RAID on srv221 is CRITICAL: Connection refused by host [13:47:05] PROBLEM - RAID on mw1158 is CRITICAL: Connection refused by host [13:47:14] RECOVERY - Disk space on mw69 is OK: DISK OK [13:47:36] RECOVERY - DPKG on snapshot2 is OK: All packages OK [13:47:55] PROBLEM - RAID on srv214 is CRITICAL: Connection refused by host [13:47:55] RECOVERY - Disk space on db1048 is OK: DISK OK [13:47:55] PROBLEM - Disk space on srv187 is CRITICAL: Connection refused by host [13:47:55] RECOVERY - RAID on srv253 is OK: OK: no RAID installed [13:48:04] PROBLEM - Disk space on mw1090 is CRITICAL: Connection refused by host [13:48:05] PROBLEM - MySQL disk space on db1035 is CRITICAL: Connection refused by host [13:48:14] RECOVERY - DPKG on db11 is OK: All packages OK [13:48:14] PROBLEM - Disk space on mw1070 is CRITICAL: Connection refused by host [13:48:24] RECOVERY - RAID on srv200 is OK: OK: no RAID installed [13:48:24] RECOVERY - Disk space on aluminium is OK: DISK OK [13:48:54] PROBLEM - Disk space on mw1075 is CRITICAL: Connection refused by host [13:49:06] RECOVERY - MySQL disk space on db1020 is OK: DISK OK [13:49:08] PROBLEM - DPKG on srv193 is CRITICAL: Connection refused by host [13:49:15] PROBLEM - DPKG on srv220 is CRITICAL: Connection refused by host [13:49:15] RECOVERY - RAID on db1040 is OK: OK: State is Optimal, checked 2 logical device(s) [13:49:16] PROBLEM - Disk space on mw1060 is CRITICAL: Connection refused by host [13:49:25] RECOVERY - DPKG on virt3 is OK: All packages OK [13:49:25] PROBLEM - RAID on srv276 is CRITICAL: Connection refused by host [13:49:34] PROBLEM - DPKG on srv276 is CRITICAL: Connection refused by host [13:49:34] PROBLEM - DPKG on db1002 is CRITICAL: Connection refused by host [13:49:34] PROBLEM - MySQL disk space on db1038 is CRITICAL: Connection refused by host [13:49:34] PROBLEM - Disk space on srv220 is CRITICAL: Connection refused by host [13:49:34] PROBLEM - Disk space on srv294 is CRITICAL: Connection refused by host [13:49:35] PROBLEM - DPKG on srv243 is CRITICAL: Connection refused by host [13:50:55] RECOVERY - DPKG on db1040 is OK: All packages OK [13:50:56] PROBLEM - DPKG on db1004 is CRITICAL: Connection refused by host [13:51:05] PROBLEM - RAID on ms1002 is CRITICAL: Connection refused by host [13:51:47] PROBLEM - DPKG on db1026 is CRITICAL: Connection refused by host [13:51:47] PROBLEM - Disk space on mw1048 is CRITICAL: Connection refused by host [13:51:58] PROBLEM - RAID on emery is CRITICAL: Connection refused by host [13:51:58] PROBLEM - MySQL disk space on db1002 is CRITICAL: Connection refused by host [13:52:16] RECOVERY - DPKG on srv285 is OK: All packages OK [13:52:22] PROBLEM - DPKG on mw1147 is CRITICAL: Connection refused by host [13:52:39] PROBLEM - Disk space on db1026 is CRITICAL: Connection refused by host [13:52:49] RECOVERY - mobile traffic loggers on cp1043 is OK: PROCS OK: 2 processes with command name varnishncsa [13:52:49] RECOVERY - DPKG on mw70 is OK: All packages OK [13:53:01] O.o [13:53:01] RECOVERY - DPKG on mw1126 is OK: All packages OK [13:53:04] RECOVERY - DPKG on db34 is OK: All packages OK [13:53:18] RECOVERY - RAID on cp1043 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [13:53:18] RECOVERY - RAID on mw1035 is OK: OK: no RAID installed [13:53:18] RECOVERY - Disk space on virt3 is OK: DISK OK [13:53:18] RECOVERY - RAID on mw1029 is OK: OK: no RAID installed [13:53:28] RECOVERY - DPKG on srv251 is OK: All packages OK [13:53:38] PROBLEM - DPKG on mw1141 is CRITICAL: Connection refused by host [13:53:48] PROBLEM - Disk space on srv243 is CRITICAL: Connection refused by host [13:54:08] PROBLEM - RAID on db1006 is CRITICAL: Connection refused by host [13:54:08] RECOVERY - RAID on mw1015 is OK: OK: no RAID installed [13:54:48] PROBLEM - MySQL disk space on db1026 is CRITICAL: Connection refused by host [13:54:48] PROBLEM - Disk space on mw1147 is CRITICAL: Connection refused by host [13:54:58] PROBLEM - Disk space on mw1141 is CRITICAL: Connection refused by host [13:55:19] PROBLEM - Disk space on db1017 is CRITICAL: Connection refused by host [13:55:21] RECOVERY - RAID on db1019 is OK: OK: State is Optimal, checked 2 logical device(s) [13:55:21] RECOVERY - Disk space on mw8 is OK: DISK OK [13:55:21] RECOVERY - Disk space on srv246 is OK: DISK OK [13:55:21] PROBLEM - Disk space on srv193 is CRITICAL: Connection refused by host [13:55:43] PROBLEM - Disk space on emery is CRITICAL: Connection refused by host [13:55:43] RECOVERY - DPKG on srv215 is OK: All packages OK [13:55:43] RECOVERY - RAID on db45 is OK: OK: State is Optimal, checked 2 logical device(s) [13:55:43] RECOVERY - Disk space on mw1159 is OK: DISK OK [13:55:43] RECOVERY - RAID on mw1067 is OK: OK: no RAID installed [13:55:43] RECOVERY - DPKG on mw1118 is OK: All packages OK [13:55:55] RECOVERY - MySQL disk space on db1033 is OK: DISK OK [13:55:55] RECOVERY - DPKG on es2 is OK: All packages OK [13:56:24] RECOVERY - Disk space on db45 is OK: DISK OK [13:57:02] RECOVERY - DPKG on stafford is OK: All packages OK [13:57:02] RECOVERY - Disk space on es2 is OK: DISK OK [13:57:21] RECOVERY - DPKG on db25 is OK: All packages OK [13:57:41] RECOVERY - RAID on srv262 is OK: OK: no RAID installed [13:57:41] RECOVERY - DPKG on mw1072 is OK: All packages OK [13:57:41] RECOVERY - RAID on db1003 is OK: OK: State is Optimal, checked 2 logical device(s) [13:57:52] RECOVERY - DPKG on mw1042 is OK: All packages OK [13:59:01] RECOVERY - RAID on srv221 is OK: OK: no RAID installed [13:59:31] RECOVERY - RAID on mw1158 is OK: OK: no RAID installed [14:00:01] RECOVERY - RAID on srv214 is OK: OK: no RAID installed [14:00:14] RECOVERY - Disk space on srv187 is OK: DISK OK [14:00:14] RECOVERY - Disk space on mw1090 is OK: DISK OK [14:00:33] RECOVERY - MySQL disk space on db1035 is OK: DISK OK [14:00:46] RECOVERY - Disk space on mw1070 is OK: DISK OK [14:01:31] RECOVERY - Disk space on mw1075 is OK: DISK OK [14:01:51] RECOVERY - DPKG on srv193 is OK: All packages OK [14:01:51] RECOVERY - DPKG on srv220 is OK: All packages OK [14:02:11] RECOVERY - RAID on srv276 is OK: OK: no RAID installed [14:02:11] RECOVERY - DPKG on srv276 is OK: All packages OK [14:02:11] RECOVERY - DPKG on db1002 is OK: All packages OK [14:02:11] RECOVERY - MySQL disk space on db1038 is OK: DISK OK [14:02:11] RECOVERY - Disk space on srv220 is OK: DISK OK [14:02:11] RECOVERY - Disk space on srv294 is OK: DISK OK [14:02:21] RECOVERY - DPKG on srv243 is OK: All packages OK [14:02:33] RECOVERY - Disk space on mw1060 is OK: DISK OK [14:02:44] RECOVERY - DPKG on db1004 is OK: All packages OK [14:02:52] RECOVERY - RAID on ms1002 is OK: OK: State is Optimal, checked 2 logical device(s) [14:03:23] RECOVERY - Disk space on mw1048 is OK: DISK OK [14:03:23] RECOVERY - DPKG on db1026 is OK: All packages OK [14:03:23] RECOVERY - RAID on emery is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [14:03:24] RECOVERY - MySQL disk space on db1002 is OK: DISK OK [14:03:42] RECOVERY - DPKG on mw1147 is OK: All packages OK [14:04:01] RECOVERY - Disk space on db1026 is OK: DISK OK [14:05:14] RECOVERY - DPKG on mw1141 is OK: All packages OK [14:05:15] RECOVERY - Disk space on srv243 is OK: DISK OK [14:05:42] RECOVERY - RAID on db1006 is OK: OK: State is Optimal, checked 2 logical device(s) [14:06:11] RECOVERY - Disk space on mw1147 is OK: DISK OK [14:06:12] RECOVERY - Disk space on mw1141 is OK: DISK OK [14:06:22] RECOVERY - MySQL disk space on db1026 is OK: DISK OK [14:06:22] RECOVERY - Disk space on db1017 is OK: DISK OK [14:06:36] RECOVERY - Disk space on srv193 is OK: DISK OK [14:07:03] RECOVERY - Disk space on emery is OK: DISK OK [14:35:26] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1923 [14:35:26] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1923 [15:09:41] !log torrus was broken ([[RT:2279]]) and did not start due to corrupted berkeleydb, used db4.8_recover, service started again [15:09:42] Logged the message, Master [15:56:05] RECOVERY - Squid on brewster is OK: TCP OK - 0.010 second response time on port 8080 [16:06:53] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33758 - Arabic numerals on Arabic Wiktionary' [16:06:54] Logged the message, Master [16:26:08] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33708 - Add alias to fa wikisource' [16:26:10] Logged the message, Master [16:28:13] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33708 - Add alias to fa wikisource' [16:28:15] Logged the message, Master [16:31:05] PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 271 MB (3% inode=60%): /var/lib/ureadahead/debugfs 271 MB (3% inode=60%): [16:34:46] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33662 - Change project namespace for lb.wiktionary' [16:34:48] Logged the message, Master [16:35:25] 1 [16:35:32] !log Ran namespaceDupes on fawikisource [16:35:34] Logged the message, Master [16:37:15] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33662 - Change project namespace for lb.wiktionary' [16:37:16] Logged the message, Master [16:40:48] !log reedy synchronized wmf-config/flaggedrevs.php 'Kill labs setting stuff' [16:40:49] Logged the message, Master [16:41:01] !log reedy synchronized wmf-config/flaggedrevs.php 'Kill labs setting stuff' [16:41:02] Logged the message, Master [16:41:05] RECOVERY - Disk space on srv219 is OK: DISK OK [16:55:58] Platonides: hey, aswiki's still broke i think... did you want to take a look? ;) [16:56:16] Platonides: (the last change was good but then there was something else still wrong) [16:56:28] (NS names) [16:57:42] oh, i see now Reedy's made a comment [16:58:36] Reedy: did you try the talk page @ http://as.wikipedia.org/wiki/%E0%A7%B1%E0%A6%BF%E0%A6%95%E0%A6%BF%E0%A6%AA%E0%A6%BF%E0%A6%A1%E0%A6%BF%E0%A6%AF%E0%A6%BC%E0%A6%BE:Meetup/GAU1 ? [16:58:44] something's definitely going on there [16:59:01] when you click talk then you're taken to a different subject space page [16:59:04] Yeah [16:59:10] and then click talk again and then that goes to a talk page [16:59:20] The characters are hard to compare [16:59:21] just based on which tab is hilighted [16:59:30] i'm not reading chars at all [16:59:37] yeah [16:59:38] i don't even have the font [16:59:39] I saw that [16:59:43] but still [16:59:47] it doesn't make life any easier :p [17:00:02] !log reedy synchronized wmf-config/InitialiseSettings.php 'Disable autoconfirmed reupload on incubatorwiki' [17:00:04] Logged the message, Master [17:00:20] !log srv221 and srv222 are out of space on / [17:00:22] Logged the message, Master [17:01:57] huh, why were there no recent alerts about either of those? (that you just logged as full /) [17:02:20] PROBLEM - Disk space on srv222 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [17:02:36] I r faster than nagios ;) [17:02:48] though, warnings, ont sure [17:03:21] RECOVERY - Disk space on srv222 is OK: DISK OK [17:04:07] Reedy: so, fwiw, based on 's class the first link is ns-4 (Project), clicking talk goes to ns-0, talk again to ns-1 [17:05:42] !log reedy synchronized wmf-config/InitialiseSettings.php 'Disable autoconfirmed reupload on incubatorwiki' [17:05:44] Logged the message, Master [17:17:35] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33508 - Enable Rollback group on id.wiki' [17:17:36] Logged the message, Master [18:44:41] !log nikerabbit synchronized php-1.18/skins/common/shared.css 'i18ndeploy' [18:44:42] Logged the message, Master [18:45:41] !log nikerabbit synchronized php-1.18/extensions/Narayam/resources/ext.narayam.rules.as.js 'i19ndeploy' [18:45:43] Logged the message, Master [18:47:51] !log nikerabbit synchronized php-1.18/extensions/WebFonts/ 'i18ndeploy' [18:47:52]