[00:00:01] in https://gerrit.wikimedia.org/r/#/q/status:merged,n,z [00:00:26] because it didn't go through gerrit [00:01:30] "It's always confusing if you guys change without code review" that's what I wanted to say with that [00:01:58] There's a 404 on gitweb as well, btw *hoo hides* [00:02:11] hoo: where? [00:02:19] https://gerrit.wikimedia.org/r/static/openstack-page-bkg.jpg [00:02:34] from the skin updates [00:04:35] Now I think I finally know enough about gerrit to use it without getting insane :P Good night and thanks ;) [00:04:48] that's not gitweb [00:05:48] jeremyb: https://gerrit.wikimedia.org/r/gitweb?p=operations/mediawiki-config.git;a=commitdiff;h=597d5e8a578a74cbc9670c114a5b2f8db098d51b that's gitweb, no? [00:06:02] hoo: it's not a 404? [00:06:06] 04 00:01:58 < hoo> There's a 404 on gitweb as well, btw *hoo hides* [00:06:21] my browser showed it once [00:06:21] jeremyb: Only https://gerrit.wikimedia.org/r/static/openstack-page-bkg.jpg [00:06:30] It's not important [00:06:34] it's a broken image [00:06:36] which is *not* *gitweb* [00:07:30] jeremyb: But it's on gitweb [00:07:51] (the image is used on gitweb) [00:08:07] ok... /me will look [00:09:48] jeremyb: Thanks :D I know, I'm confusing today :P [00:14:17] New review: Demon; "Remind me later to track down what the real class is here, so we can mark it @external for future re..." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/17610 [00:43:18] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [00:43:18] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [01:15:23] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [01:15:24] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [01:16:47] New patchset: Dzahn; "add account for krinkle to mortals and jenkins admins per RT-2997" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17633 [01:17:28] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/17633 [01:21:58] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17633 [01:27:23] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [01:27:24] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [01:42:14] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 265 seconds [01:42:14] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 265 seconds [01:42:23] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 277 seconds [01:42:23] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 277 seconds [01:48:32] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 644s [01:48:32] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 644s [01:51:23] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [01:51:23] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [01:54:14] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 7 seconds [01:54:14] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 7 seconds [01:54:14] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 8s [01:54:14] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 8s [01:55:44] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 24 seconds [01:55:44] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 24 seconds [02:00:16] RoanKattouw_away: https://github.com/JerrySievert/node-judy/blob/master/judy.js [02:31:34] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [02:31:34] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [02:46:34] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [02:46:35] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [03:00:40] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [03:00:41] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [03:15:21] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [03:15:22] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [04:44:07] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: Puppet has not run in the last 10 hours [04:44:07] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [04:44:07] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [04:44:07] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: Puppet has not run in the last 10 hours [04:44:07] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [04:44:07] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [08:31:29] New patchset: Ori.livneh; "(RT 1829) add pre-commit hook to lint manifests" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/17640 [08:32:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/17640 [08:36:28] hey ops, what's the practice with rt -- do you uncheck people so that comments on a ticket don't go to *everyone*, or do you let them all percolate to all admins? [10:44:21] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [10:44:21] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [11:16:18] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [11:16:18] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [11:28:18] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [11:28:18] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [11:52:50] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [11:52:50] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [12:32:44] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [12:32:45] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [12:47:08] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [12:47:09] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [13:01:14] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [13:01:14] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [13:16:05] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [13:16:05] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [13:30:37] New patchset: J; "set $wgTimedTextNS on commons/TMH" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/17650 [14:45:17] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [14:45:17] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: Puppet has not run in the last 10 hours [14:45:17] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [14:45:17] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [14:45:18] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: Puppet has not run in the last 10 hours [14:45:18] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [16:13:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:13:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:14:54] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.628 seconds [16:14:54] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.628 seconds [16:48:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:52:42] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.680 seconds [16:52:42] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.680 seconds [17:13:49] puppetmaster??? [17:18:52] aude: ? [17:21:03] PROBLEM - Host search23 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:04] PROBLEM - Host search23 is DOWN: PING CRITICAL - Packet loss = 100% [17:26:07] is puppetmaster ok? [17:26:31] * aude hasn't been able to login to my instance that i created yesterday [17:26:35] aude: it recovered ;) [17:26:41] that's not your puppetmaster anyway [17:27:05] that check needs to be fixed so 400 is not OK (and so that the path it checks is a 200) [17:27:09] 400 bad request is okay? [17:27:10] but 400 is normal for that check [17:27:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:21] anyway, < jeremyb> that's not your puppetmaster anyway [17:27:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:23] if you say so.... [17:27:26] i do! [17:27:34] why can't i login? [17:27:37] -> #-labs [17:41:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.029 seconds [17:41:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.029 seconds [18:12:52] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:12:52] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:01] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.028 seconds [18:25:01] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.028 seconds [18:57:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:07:46] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.058 seconds [19:07:46] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.058 seconds [19:42:54] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:42:55] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:57] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.252 seconds [19:52:58] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.252 seconds [20:27:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:39:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.048 seconds [20:39:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.048 seconds [20:44:51] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [20:44:51] PROBLEM - Puppet freshness on srv281 is CRITICAL: Puppet has not run in the last 10 hours [21:11:15] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:11:15] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:57] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [21:16:57] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [21:22:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.485 seconds [21:22:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.485 seconds [21:29:22] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [21:29:22] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [21:53:22] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [21:53:23] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [21:56:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.568 seconds [22:08:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.568 seconds [22:33:25] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [22:33:25] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [22:41:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:48:25] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [22:48:25] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [22:55:10] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.038 seconds [22:55:10] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.038 seconds [23:01:55] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [23:01:55] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [23:06:54] !replag [23:16:55] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [23:16:55] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [23:26:49] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:26:49] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:38:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.489 seconds [23:38:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.489 seconds