[00:09:24] Coren: trivial toollabs patch! ^ [00:09:29] hmm, my network is slow [00:09:31] wait for it! [00:09:36] (03PS1) 10Yuvipanda: Disable MONITOR command for tools redis [operations/puppet] - 10https://gerrit.wikimedia.org/r/77561 [00:09:40] there! [00:10:05] scfc_de: we'll need to restart Redis at some point after Coren merges ^ [00:10:11] (since redis doesn't autorestart) [00:11:48] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [00:12:30] I'm heading to sleep, but hopefully Coren can merge that before he becomes Coren|Travel :) [00:12:35] night [00:14:10] YuviPanda|sleep: k [00:14:41] scfc_de: do look at the semantics of MONITOR too, I am sortof ashamed to have missed something that obvious [00:14:42] grr [00:14:44] now to really sleep [00:15:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:16:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [00:32:48] PROBLEM - Puppet freshness on sq41 is CRITICAL: No successful Puppet run in the last 10 hours [00:32:58] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 00:32:53 UTC 2013 [00:33:48] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [01:06:15] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [01:08:55] PROBLEM - RAID on analytics1010 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:12:55] RECOVERY - RAID on analytics1010 is OK: OK: Active: 6, Working: 6, Failed: 0, Spare: 0 [01:32:55] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 01:32:45 UTC 2013 [01:33:15] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [02:04:59] !log LocalisationUpdate completed (1.22wmf12) at Sun Aug 4 02:04:59 UTC 2013 [02:05:11] Logged the message, Master [02:07:29] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [02:13:09] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Aug 4 02:13:09 UTC 2013 [02:13:21] Logged the message, Master [02:26:39] RECOVERY - Puppet freshness on mchenry is OK: puppet ran at Sun Aug 4 02:26:29 UTC 2013 [02:32:59] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 02:32:54 UTC 2013 [02:33:29] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [03:07:56] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [03:32:46] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 03:32:41 UTC 2013 [03:32:56] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [03:58:09] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [03:58:09] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [03:58:09] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [03:58:09] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [03:58:09] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [03:58:09] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [04:09:45] (03CR) 10Tim Landscheidt: [C: 031] "@Yuvipanda I *think* Redis is restarted automatically when the configuration file changes as IIRC this was the concern with your initial c" [operations/puppet] - 10https://gerrit.wikimedia.org/r/77561 (owner: 10Yuvipanda) [05:03:59] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: No successful Puppet run in the last 10 hours [05:06:49] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last [06:33:40] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 06:33:32 UTC 2013 [06:34:20] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [07:08:09] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [07:10:59] PROBLEM - Puppet freshness on holmium is CRITICAL: No successful Puppet run in the last 10 hours [07:31:17] (03PS6) 10TTO: Clean up headers in CommonSettings and InitialiseSettings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76342 [07:33:22] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 07:33:14 UTC 2013 [07:34:22] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [08:01:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:03:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [08:11:08] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [08:17:58] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [08:21:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:23:08] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.139 second response time [08:32:58] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 08:32:54 UTC 2013 [08:33:08] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [08:52:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:53:08] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [08:56:58] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [09:03:38] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 09:03:28 UTC 2013 [09:04:08] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [09:14:31] (03CR) 10Yuvipanda: "modules/redis/manifests/init.pp, line 53:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/77561 (owner: 10Yuvipanda) [09:20:40] (03CR) 10Yuvipanda: "And yeah, my initial concerns were unfounded then :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/77561 (owner: 10Yuvipanda) [09:21:19] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:23:08] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [09:33:18] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 09:33:08 UTC 2013 [09:34:08] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [09:51:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [10:06:48] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [10:21:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:08] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [10:32:48] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 10:32:41 UTC 2013 [10:32:58] PROBLEM - Puppet freshness on sq41 is CRITICAL: No successful Puppet run in the last 10 hours [10:33:48] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [10:51:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:53:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [10:53:57] paravoid: is it okay to bug operations if something rather cruical on tools appears broken and noone tools specific is around to fix it? ;p [11:01:07] ie. no webtools being accessible :> [11:01:58] (but http://tools-webserver-01.proxy.wmflabs.org/ works) [11:06:26] yes YuviPanda :P But unfortunately thats not where the traffic is going ;p [11:06:42] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [11:21:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:24:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.142 second response time [11:32:52] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 11:32:45 UTC 2013 [11:33:42] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [11:41:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.139 second response time [11:51:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.135 second response time [12:06:20] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [12:24:24] (03PS1) 10Yuvipanda: Add golang compiler / runtime environment to toollabs [operations/puppet] - 10https://gerrit.wikimedia.org/r/77567 [12:32:49] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 12:32:40 UTC 2013 [12:33:19] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [12:53:07] PROBLEM - Puppet freshness on mchenry is CRITICAL: No successful Puppet run in the last 10 hours [13:08:19] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [13:32:49] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 13:32:45 UTC 2013 [13:33:19] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [13:58:10] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [13:58:10] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [13:58:10] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [13:58:10] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [13:58:10] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [13:58:10] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [14:06:10] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [14:32:50] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 14:32:40 UTC 2013 [14:33:10] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [14:56:24] (03PS2) 10Yuvipanda: role::eqiad-proxy: file[] -> File[] [operations/puppet] - 10https://gerrit.wikimedia.org/r/77121 (owner: 10Hashar) [14:56:31] (03CR) 10Yuvipanda: [C: 031] role::eqiad-proxy: file[] -> File[] [operations/puppet] - 10https://gerrit.wikimedia.org/r/77121 (owner: 10Hashar) [15:04:54] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: No successful Puppet run in the last 10 hours [15:06:00] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [15:32:40] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 15:32:38 UTC 2013 [15:33:00] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [15:52:55] PROBLEM - Puppet freshness on pdf3 is CRITICAL: No successful Puppet run in the last 10 hours [15:57:15] PROBLEM - Disk space on ms-be1008 is CRITICAL: DISK CRITICAL - /var/lib/ceph/osd/ceph-93 is not accessible: Input/output error [16:07:56] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [16:32:46] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 16:32:44 UTC 2013 [16:32:56] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [16:43:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [16:52:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [17:09:02] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [17:11:02] PROBLEM - Puppet freshness on holmium is CRITICAL: No successful Puppet run in the last 10 hours [17:22:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [17:32:52] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 17:32:46 UTC 2013 [17:33:02] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [17:51:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [18:07:32] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [18:18:12] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [18:28:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:29:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [18:33:32] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 18:33:30 UTC 2013 [18:34:32] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [18:44:25] (03CR) 10Ori.livneh: [C: 031] "go yuvi!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/77567 (owner: 10Yuvipanda) [18:51:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:52:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.151 second response time [18:57:53] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [19:06:38] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [19:21:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:08] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [19:32:48] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 19:32:43 UTC 2013 [19:33:38] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [19:51:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:08] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.137 second response time [20:06:40] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [20:21:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:10] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [20:33:00] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 20:32:50 UTC 2013 [20:33:00] PROBLEM - Puppet freshness on sq41 is CRITICAL: No successful Puppet run in the last 10 hours [20:33:40] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [20:36:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:40:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 4.378 second response time [20:41:33] Could someone block User:Larygloria01 on wikitech, please? [20:43:57] Ryan_Lane, Wikimedia's SSL cert covers a lot of domains, but I noticed that it has both *.domain namd and m.domain [20:44:14] Aren't the m. entries redundant given the *. ones? [20:44:19] Krenair: nope [20:44:24] en.m, for instance [20:44:33] it also has *.m [20:44:42] Those are under *.m subdomains [20:45:09] *.wikipedia.org, wikipedia.org, m.wikipedia.org, *.m.wikipedia.org [20:45:16] how is the m. useful when you have the *. one? [20:45:21] you can also hit m.wikipedia.org directly [20:45:31] *.m wouldn't cover that [20:45:50] *.wikipedia.org should [20:46:07] ah. right [20:46:11] Ryan_Lane: can you +2 https://gerrit.wikimedia.org/r/#/c/77561/? trivial, but having it on completely negates toollabs redis 'securtity' [20:46:20] yeah, likely unnecessary then [20:46:37] s/on/off/ [20:46:44] (03CR) 10Ryan Lane: [C: 032] Disable MONITOR command for tools redis [operations/puppet] - 10https://gerrit.wikimedia.org/r/77561 (owner: 10Yuvipanda) [20:46:47] find the keys that other tools use? [20:46:52] seems like obscurity to me [20:47:03] it is, indeed, security by obscurity :) [20:47:07] YuviPanda: done [20:47:21] with the idea being if people use long enough (and random enough!) keys, you can't really guess those [20:47:43] Ryan_Lane: equally trivial, but not as much of an issue... https://gerrit.wikimedia.org/r/#/c/77567/ [20:48:16] technically lots of things are security by obscurity, but the obscurity involves a secret ;) [20:48:33] and not the fact that you are hiding the implementation [20:48:35] indeed, in this case too. I am mad at myself for missing that command before. [20:48:50] (03CR) 10Ryan Lane: [C: 032] Add golang compiler / runtime environment to toollabs [operations/puppet] - 10https://gerrit.wikimedia.org/r/77567 (owner: 10Yuvipanda) [20:48:55] Ryan_Lane: also https://gerrit.wikimedia.org/r/#/c/77454/? :) [20:49:00] -_- [20:49:54] sorry, I was being greedy :) [20:50:20] on a sunday even [20:50:35] (03CR) 10Ryan Lane: [C: 032] Add labsproxy module for replacing instanceproxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/77454 (owner: 10Yuvipanda) [20:50:43] Ryan_Lane: \o/ ty! [20:50:47] yw [20:51:04] none of them affect anything running, really, so it's somewhat ok [20:51:06] greg-g: what is a guy who now suddenly has an extra day to do? [20:51:12] though I really shouldn't be online ;) [20:51:19] me neither [20:51:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:28] I thought I should be waiting for my flight to take off right now [20:51:29] grr [20:51:33] I need to do laundry, pack, do some errands, etc [20:51:40] YuviPanda: I think there's a movie about this, Groundhog Day ;) [20:51:45] oh? [20:51:49] * greg-g jokes [20:51:59] ah :P [20:52:02] i've not seen that one [20:52:20] https://en.wikipedia.org/wiki/Groundhog_Day_%28film%29 [20:53:40] greg-g: ah! [20:53:47] * YuviPanda re-examines his priorities in life [20:53:57] hmm, 'understand dates' is at the top of it now [20:54:02] :) [20:54:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 5.020 second response time [21:08:09] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [21:21:19] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:22:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [21:24:53] (03PS1) 10Yuvipanda: Add nginx service to labsproxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/77640 [21:25:02] Ryan_Lane: ^ another trivial one... :) [21:30:29] (03PS2) 10Yuvipanda: Add nginx service to labsproxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/77640 [21:32:49] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 21:32:45 UTC 2013 [21:33:09] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [21:51:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:52:08] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [22:01:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:08] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.142 second response time [22:06:36] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [22:21:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:22:43] YuviPanda: shouldn't you be using puppetmaster::self? :) [22:22:59] (03CR) 10Ryan Lane: [C: 032] Add nginx service to labsproxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/77640 (owner: 10Yuvipanda) [22:23:07] Ryan_Lane: yes, yes, yes! I will, after this one :) [22:23:11] :D [22:23:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.152 second response time [22:27:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:28:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.144 second response time [22:28:35] Ryan_Lane: that patch actually made everything work! :D http://tools-webserver-01.proxy.wmflabs.org/ not bad for untested, blind puppet writing [22:28:44] now I make it puppetmaster::self and actually test the things [22:30:32] hrh [22:30:34] err [22:30:34] heh [22:30:35] cool [22:30:49] YuviPanda, why test if it works? [22:31:01] MaxSem: heh :D [22:31:13] MaxSem: well, I did the configurations by hand, then I puppetized it [22:31:15] so in ways, I am testing it [22:31:21] I'm testing the actual configs [22:31:24] just not the puppetization [22:31:28] so not as 'blind' [22:32:46] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 22:32:44 UTC 2013 [22:33:36] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [22:43:58] The "News" section here is stored in a text file...we usually have to poke in here to get it changed by the relevant ops person...Anybody have time to fix that? Just want to add a note about the upcoming upgrade [22:44:05] @ https://ticket.wikimedia.org/otrs/index.pl [22:44:27] I am an OTRS admin [22:48:52] "The "News" messages on the main OTRS login screen can be editing by modifying /opt/otrs/Kernel/Output/HTML/Standard/Motd.dtl." according to wikitech [22:54:08] PROBLEM - Puppet freshness on mchenry is CRITICAL: No successful Puppet run in the last 10 hours [23:01:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:02:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 6.138 second response time [23:09:03] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [23:21:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:22:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [23:33:23] RECOVERY - Puppet freshness on mexia is OK: puppet ran at Sun Aug 4 23:33:16 UTC 2013 [23:34:03] PROBLEM - Puppet freshness on mexia is CRITICAL: No successful Puppet run in the last 10 hours [23:51:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:52:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [23:58:58] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [23:58:58] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [23:58:58] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [23:58:58] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [23:58:58] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [23:58:58] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours