[00:00:13] gn8 folks [00:42:04] zzz =_= [00:46:40] hola a todos voy a pedirles una gran ayuda a todos ustedes estoy realisando una encuenta y necesito gente que pueda llenar esa encuenta la puedne encontrar en esta direccion http://www.encuestafacil.com/RespWeb/Qn.aspx?EID=1158938 [01:11:39] PROBLEM - Disk space on srv224 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [01:28:49] anyone around with a bit of time on their hands? [01:29:10] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 152 MB (2% inode=60%): /var/lib/ureadahead/debugfs 152 MB (2% inode=60%): [01:33:20] PROBLEM - Disk space on srv222 is CRITICAL: DISK CRITICAL - free space: / 110 MB (1% inode=60%): /var/lib/ureadahead/debugfs 110 MB (1% inode=60%): [01:34:32] Roan fixed 220 earlier today [01:34:42] I've got a feeeling just need /tmp clearing on most of them... [01:37:10] RECOVERY - Disk space on srv224 is OK: DISK OK [01:39:20] RECOVERY - Disk space on srv220 is OK: DISK OK [01:53:00] RECOVERY - Disk space on srv222 is OK: DISK OK [02:05:09] !log LocalisationUpdate completed (1.18) at Sat Jan 7 02:05:09 UTC 2012 [02:05:14] Logged the message, Master [02:36:41] PROBLEM - MySQL replication status on db1025 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1272s [03:32:52] RECOVERY - MySQL replication status on db1025 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [04:19:57] RECOVERY - Disk space on es1004 is OK: DISK OK [04:21:07] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:44:37] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [05:10:48] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [06:55:36] PROBLEM - Puppet freshness on cp1043 is CRITICAL: Puppet has not run in the last 10 hours [07:03:16] PROBLEM - Puppet freshness on cp1044 is CRITICAL: Puppet has not run in the last 10 hours [07:16:16] PROBLEM - Puppet freshness on db22 is CRITICAL: Puppet has not run in the last 10 hours [09:04:19] PROBLEM - Puppet freshness on db1003 is CRITICAL: Puppet has not run in the last 10 hours [09:56:30] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 451393 MB (3% inode=99%): [10:00:20] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 424095 MB (3% inode=99%): [10:10:23] RECOVERY - MySQL slave status on es1004 is OK: OK: [12:02:31] zzz [15:20:11] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [15:42:57] I didn't manage to upload this file over HTTPS: http://commons.wikimedia.org/wiki/File:Mujeres_c%C3%A9lebres_III.djvu [15:43:06] unless it's FF's fault, which seems unlikely [15:44:19] and it's horribly slow, can't go over 200 KB/s right now [17:04:26] PROBLEM - Puppet freshness on cp1043 is CRITICAL: Puppet has not run in the last 10 hours [17:11:59] PROBLEM - Puppet freshness on cp1044 is CRITICAL: Puppet has not run in the last 10 hours [17:25:49] PROBLEM - Puppet freshness on db22 is CRITICAL: Puppet has not run in the last 10 hours [18:42:04] Is it me or is enwiki slow as hell? [18:58:27] PROBLEM - Apache HTTP on srv282 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:58:47] PROBLEM - Disk space on srv282 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:06:37] PROBLEM - SSH on srv282 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:06:37] PROBLEM - RAID on srv282 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:06:37] PROBLEM - DPKG on srv282 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:15:11] RECOVERY - Disk space on srv282 is OK: DISK OK [19:20:51] RECOVERY - Apache HTTP on srv282 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.020 second response time [19:20:51] RECOVERY - RAID on srv282 is OK: OK: no RAID installed [19:21:01] PROBLEM - Puppet freshness on db1003 is CRITICAL: Puppet has not run in the last 10 hours [19:22:41] RECOVERY - SSH on srv282 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [19:22:41] RECOVERY - DPKG on srv282 is OK: All packages OK [20:06:59] @db enwiki [20:08:58] @info enwiki [20:08:58] jeremyb: [enwiki: s1] db36: 10.0.6.46, db32: 10.0.6.42, db12: 10.0.6.22, db26: 10.0.6.36, db38: 10.0.6.48 [21:56:09] Hello. What is the URL to the settings again, please? [22:02:39] Hazard-SJ: you mean noc.wikimedia.org/conf ? [22:02:49] PROBLEM - Disk space on srv276 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:03:05] Hazard-SJ: or https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git;a=summary ? or tell me what you mean [22:05:05] jeremyb: noc.wikimedia.org/conf thanks :D [22:06:19] PROBLEM - RAID on srv262 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:06:59] PROBLEM - Apache HTTP on srv262 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:19] PROBLEM - SSH on srv262 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:19] PROBLEM - DPKG on srv262 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:08:39] PROBLEM - RAID on srv276 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:09:39] PROBLEM - Apache HTTP on srv276 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:10:19] PROBLEM - SSH on srv276 is CRITICAL: Server answer: [22:11:56] Not sure if it's relevant, but it feels as if edits are real slow at the moment. [22:12:15] (bot from tool server, as well as interactive) [22:13:29] PROBLEM - DPKG on srv276 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:13:37] siebrand: ppl have been saying that for hours. idk if anyone's investigated [22:13:45] (above in this channel) [22:14:09] PROBLEM - Disk space on srv262 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:27:54] also had a real error on submitting an edit some time ago: [22:27:54] Request: POST http://commons.wikimedia.org/w/index.php?title=Commons:Freedom_of_panorama&action=submit, from 208.80.152.19 via sq60.wikimedia.org (squid/2.7.STABLE9) to 208.80.152.83 (208.80.152.83) [22:27:54] Error: ERR_READ_TIMEOUT, errno [No Error] at Sat, 07 Jan 2012 21:58:52 GMT [22:27:54] however, F5 solved it [22:28:19] yes, edits seem to be slow [22:29:19] RECOVERY - Apache HTTP on srv276 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [22:29:23] Saibo: it says no error! explicitly! :) [22:29:31] rofl [22:29:40] server humor ;) [22:30:10] hmm, there's a few errors in the logs [22:30:19] RECOVERY - SSH on srv276 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [22:33:29] RECOVERY - Disk space on srv276 is OK: DISK OK [22:33:29] RECOVERY - DPKG on srv276 is OK: All packages OK [22:33:30] !log Users reporting slowness while editing. dberror.log shows a few mysql errors for enwiki master and slaves. Few errors on other wikis, mainly enwiki [22:33:31] Logged the message, Master [22:35:42] now fast at commons for me ;) [22:36:16] lol [22:36:33] well, sorry - I have no testing environment.. just an edit every now and then :D [22:36:56] Ping me if it starts getting worse [22:37:00] okay [22:37:02] thx [22:38:29] RECOVERY - RAID on srv276 is OK: OK: no RAID installed [22:46:20] RECOVERY - Apache HTTP on srv262 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.020 second response time [22:46:20] RECOVERY - RAID on srv262 is OK: OK: no RAID installed [22:47:26] Has nagios been flapping a lot all day? [22:50:10] RECOVERY - Disk space on srv262 is OK: DISK OK [22:50:50] RECOVERY - SSH on srv262 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [22:51:42] Reedy: Lol [22:51:50] RECOVERY - DPKG on srv262 is OK: All packages OK [23:31:10] Reedy: just 282 and 262 i think (flapping) [23:35:50] PROBLEM - Apache HTTP on srv262 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:35:50] PROBLEM - RAID on srv262 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:39:50] PROBLEM - Disk space on srv262 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:40:10] PROBLEM - SSH on srv262 is CRITICAL: Server answer: [23:41:30] PROBLEM - DPKG on srv262 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:48:33] zzz [23:49:00] PROBLEM - RAID on srv276 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:49:30] RECOVERY - Disk space on srv262 is OK: DISK OK [23:49:50] PROBLEM - DPKG on srv276 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:50:00] RECOVERY - SSH on srv262 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [23:53:20] PROBLEM - Disk space on srv276 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:56:40] RECOVERY - Apache HTTP on srv262 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.023 second response time [23:56:40] RECOVERY - RAID on srv262 is OK: OK: no RAID installed [23:58:50] PROBLEM - Apache HTTP on srv276 is CRITICAL: CRITICAL - Socket timeout after 10 seconds