[00:04:22] * AaronSchulz summons maplebed [00:05:04] and so I have been summoned. [00:06:09] * maplebed looks around, bored... [00:17:12] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:21] maplebed: sooo...what now? [00:24:50] oh, I can do one of the 1.5 ones, duh ;) [00:25:05] have you reliably confirmed that the tests fail agaist 1.5 and succeed against 1.4.3? [00:25:06] maplebed: you said magnesium or something? [00:25:16] mg and zi. [00:25:35] (are the two counterparts to cu.) [00:26:01] sorry. Zn. [00:26:54] mg works fine [00:27:08] 3 runs [00:27:20] mg is v1.5. [00:28:32] 3 runs ok on zn too [00:28:53] and cu? [00:29:23] copper? I already did that, it worked [00:29:40] ok, so that means that we've had successful runs against both 1.4.3 and 1.5 [00:30:06] so whatever was going wrong is now fixed. [00:30:24] that kinda sucks, since I don't think we fixed anything. [00:32:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.033 seconds [00:33:28] maplebed: perfFileOpTest also works with zinc/copper from my laptop [00:33:38] maybe it's just the regular unit tests somehow [00:34:08] well, it can't just be the unit tests, since manual file restores gave errors [01:04:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:16:47] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.988 seconds [01:41:50] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 235 seconds [01:42:26] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 277 seconds [01:48:44] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 650s [01:51:08] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: Puppet has not run in the last 10 hours [01:51:08] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: Puppet has not run in the last 10 hours [01:51:08] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: Puppet has not run in the last 10 hours [01:51:08] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [01:51:08] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [01:51:08] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: Puppet has not run in the last 10 hours [01:51:09] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [01:51:09] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [01:51:09] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [01:51:10] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [01:51:11] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [01:51:11] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [01:51:11] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [01:51:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:57:26] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 5 seconds [01:58:20] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 17 seconds [01:59:14] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 18s [02:03:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.068 seconds [03:42:37] PROBLEM - Host virt1005 is DOWN: CRITICAL - Network Unreachable (10.4.125.3) [04:36:00] PROBLEM - Puppet freshness on ms-be6 is CRITICAL: Puppet has not run in the last 10 hours [05:54:39] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [05:54:39] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [05:54:39] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [05:54:39] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [06:00:39] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [07:22:51] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [07:22:51] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [07:57:45] PROBLEM - NTP on ms-be6 is CRITICAL: NTP CRITICAL: No response from NTP server [08:54:02] PROBLEM - Host search32 is DOWN: PING CRITICAL - Packet loss = 100% [09:24:02] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [09:50:02] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.036 second response time [11:52:26] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: Puppet has not run in the last 10 hours [11:52:26] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: Puppet has not run in the last 10 hours [11:52:26] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [11:52:26] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: Puppet has not run in the last 10 hours [11:52:26] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: Puppet has not run in the last 10 hours [11:52:26] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [11:52:26] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [11:52:27] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [11:52:27] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [11:52:28] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [11:52:28] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [11:52:29] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [11:52:29] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [12:38:29] PROBLEM - Puppet freshness on lvs5 is CRITICAL: Puppet has not run in the last 10 hours [12:48:58] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.755 seconds [13:23:55] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:26:46] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.728 seconds [14:01:06] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.022 seconds [14:36:57] PROBLEM - Puppet freshness on ms-be6 is CRITICAL: Puppet has not run in the last 10 hours [14:45:48] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [14:47:00] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:54] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.39 ms [14:51:57] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [15:02:00] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.087 seconds [15:27:38] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.030 second response time [15:33:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:36:50] Anyone got a Windows VM for testing? [15:37:00] Or isn't worried about his system? [15:37:10] * hoo doesn't want to mess his system [15:41:39] hoo, did you find the extension? [15:41:47] http://www.complitly.com/ [15:42:00] I've found a user with some technical clue who had issues :) [15:42:34] :) [15:42:56] but I couldn't yet verify it [15:43:16] I don't have a Windows VM over here, so I can't verify it [15:45:56] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.873 seconds [15:55:14] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [15:55:14] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [15:55:14] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [15:55:14] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [16:01:15] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [16:01:32] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [16:06:30] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.41 ms [16:09:47] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [16:20:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:32:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.873 seconds [16:39:47] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.030 second response time [17:07:14] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:07:59] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [17:08:12] hoo|away: Platonides: is that from OTRS? [17:08:23] does it really need windows? [17:09:05] jeremyb, that's a .exe [17:09:21] > Work on all browsers and web sites [17:09:26] i see that now ;/ [17:12:11] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [17:15:56] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [17:20:53] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.062 seconds [17:24:20] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [17:24:20] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [17:41:26] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [17:42:20] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.66 ms [17:52:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:04:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.194 seconds [18:07:36] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.026 second response time [18:15:42] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [18:17:03] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms [18:21:15] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [18:39:15] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:51:06] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 2.453 seconds [18:54:33] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.031 second response time [19:26:03] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:39:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.028 seconds [20:11:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:24:01] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 9.364 seconds [20:59:07] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:11:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.850 seconds [21:45:52] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:53:04] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: Puppet has not run in the last 10 hours [21:53:04] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: Puppet has not run in the last 10 hours [21:53:04] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [21:53:04] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: Puppet has not run in the last 10 hours [21:53:04] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [21:53:04] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: Puppet has not run in the last 10 hours [21:53:04] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [21:53:05] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [21:53:05] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [21:53:06] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [21:53:06] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [21:53:07] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [21:53:07] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [21:56:09] thanks nagios-wm, very helpful ;) [21:57:05] lol [21:58:01] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.416 seconds [22:02:25] !!log this is a test... [22:02:33] huh [22:11:04] morebots: you don't like emphasis do you [22:13:28] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [22:14:22] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.71 ms [22:14:54] no, i wasn't talking to morebots [22:14:59] i was talking to wm-bot [22:16:12] * Nemo_bis doesn't like him [22:17:49] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [22:21:10] Nemo_bis: http://wikitech.wikimedia.org/index.php?title=Server_admin_log&diff=50452&oldid=50416 [22:25:52] * Nemo_bis nods [22:32:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:38:58] PROBLEM - Puppet freshness on lvs5 is CRITICAL: Puppet has not run in the last 10 hours [22:43:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.164 seconds [22:48:16] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.030 second response time [23:18:50] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:26:20] PROBLEM - Host mw8 is DOWN: PING CRITICAL - Packet loss = 100% [23:26:56] RECOVERY - Host mw8 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [23:30:23] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [23:30:50] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.023 seconds [23:48:23] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.025 second response time