[00:23:26] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:36:38] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.883 seconds
[00:53:22] <nagios-wm>	 PROBLEM - Puppet freshness on sq59 is CRITICAL: Puppet has not run in the last 10 hours
[00:59:22] <nagios-wm>	 PROBLEM - Puppet freshness on sq72 is CRITICAL: Puppet has not run in the last 10 hours
[01:00:25] <nagios-wm>	 PROBLEM - Puppet freshness on kaulen is CRITICAL: Puppet has not run in the last 10 hours
[01:00:25] <nagios-wm>	 PROBLEM - Puppet freshness on mc16 is CRITICAL: Puppet has not run in the last 10 hours
[01:00:25] <nagios-wm>	 PROBLEM - Puppet freshness on mc12 is CRITICAL: Puppet has not run in the last 10 hours
[01:00:25] <nagios-wm>	 PROBLEM - Puppet freshness on ms-fe2 is CRITICAL: Puppet has not run in the last 10 hours
[01:00:25] <nagios-wm>	 PROBLEM - Puppet freshness on search1004 is CRITICAL: Puppet has not run in the last 10 hours
[01:00:26] <nagios-wm>	 PROBLEM - Puppet freshness on search1016 is CRITICAL: Puppet has not run in the last 10 hours
[01:00:26] <nagios-wm>	 PROBLEM - Puppet freshness on snapshot1 is CRITICAL: Puppet has not run in the last 10 hours
[01:00:27] <nagios-wm>	 PROBLEM - Puppet freshness on snapshot1003 is CRITICAL: Puppet has not run in the last 10 hours
[01:00:27] <nagios-wm>	 PROBLEM - Puppet freshness on sq73 is CRITICAL: Puppet has not run in the last 10 hours
[01:00:28] <nagios-wm>	 PROBLEM - Puppet freshness on sq65 is CRITICAL: Puppet has not run in the last 10 hours
[01:00:28] <nagios-wm>	 PROBLEM - Puppet freshness on sq79 is CRITICAL: Puppet has not run in the last 10 hours
[01:00:29] <nagios-wm>	 PROBLEM - Puppet freshness on sq84 is CRITICAL: Puppet has not run in the last 10 hours
[01:00:29] <nagios-wm>	 PROBLEM - Puppet freshness on sq86 is CRITICAL: Puppet has not run in the last 10 hours
[01:00:30] <nagios-wm>	 PROBLEM - Puppet freshness on sq80 is CRITICAL: Puppet has not run in the last 10 hours
[01:00:30] <nagios-wm>	 PROBLEM - Puppet freshness on sq77 is CRITICAL: Puppet has not run in the last 10 hours
[01:00:31] <nagios-wm>	 PROBLEM - Puppet freshness on virt2 is CRITICAL: Puppet has not run in the last 10 hours
[01:00:31] <nagios-wm>	 PROBLEM - Puppet freshness on virt1007 is CRITICAL: Puppet has not run in the last 10 hours
[01:00:32] <nagios-wm>	 PROBLEM - Puppet freshness on virt1008 is CRITICAL: Puppet has not run in the last 10 hours
[01:01:19] <nagios-wm>	 PROBLEM - Puppet freshness on cadmium is CRITICAL: Puppet has not run in the last 10 hours
[01:01:19] <nagios-wm>	 PROBLEM - Puppet freshness on mw3 is CRITICAL: Puppet has not run in the last 10 hours
[01:01:19] <nagios-wm>	 PROBLEM - Puppet freshness on search1007 is CRITICAL: Puppet has not run in the last 10 hours
[01:01:19] <nagios-wm>	 PROBLEM - Puppet freshness on sq70 is CRITICAL: Puppet has not run in the last 10 hours
[01:02:22] <nagios-wm>	 PROBLEM - Puppet freshness on pdf2 is CRITICAL: Puppet has not run in the last 10 hours
[01:10:55] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:22:19] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.401 seconds
[01:40:19] <nagios-wm>	 PROBLEM - Puppet freshness on ms-be3002 is CRITICAL: Puppet has not run in the last 10 hours
[01:58:01] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[02:11:04] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.017 seconds
[02:22:43] <logmsgbot>	 !log LocalisationUpdate completed (1.21wmf5) at Sun Dec  9 02:22:43 UTC 2012
[02:23:36] <nagios-wm>	 PROBLEM - Puppet freshness on constable is CRITICAL: Puppet has not run in the last 10 hours
[02:29:36] <nagios-wm>	 PROBLEM - Puppet freshness on search1001 is CRITICAL: Puppet has not run in the last 10 hours
[02:36:39] <nagios-wm>	 PROBLEM - Puppet freshness on dataset2 is CRITICAL: Puppet has not run in the last 10 hours
[02:36:39] <nagios-wm>	 PROBLEM - Puppet freshness on search14 is CRITICAL: Puppet has not run in the last 10 hours
[02:36:39] <nagios-wm>	 PROBLEM - Puppet freshness on search18 is CRITICAL: Puppet has not run in the last 10 hours
[02:36:39] <nagios-wm>	 PROBLEM - Puppet freshness on sq44 is CRITICAL: Puppet has not run in the last 10 hours
[02:36:39] <nagios-wm>	 PROBLEM - Puppet freshness on sq81 is CRITICAL: Puppet has not run in the last 10 hours
[02:37:33] <nagios-wm>	 PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours
[02:37:33] <nagios-wm>	 PROBLEM - Puppet freshness on labstore2 is CRITICAL: Puppet has not run in the last 10 hours
[02:37:33] <nagios-wm>	 PROBLEM - Puppet freshness on search1018 is CRITICAL: Puppet has not run in the last 10 hours
[02:37:33] <nagios-wm>	 PROBLEM - Puppet freshness on search1024 is CRITICAL: Puppet has not run in the last 10 hours
[02:37:33] <nagios-wm>	 PROBLEM - Puppet freshness on search13 is CRITICAL: Puppet has not run in the last 10 hours
[02:37:34] <nagios-wm>	 PROBLEM - Puppet freshness on search19 is CRITICAL: Puppet has not run in the last 10 hours
[02:37:34] <nagios-wm>	 PROBLEM - Puppet freshness on search17 is CRITICAL: Puppet has not run in the last 10 hours
[02:37:35] <nagios-wm>	 PROBLEM - Puppet freshness on search16 is CRITICAL: Puppet has not run in the last 10 hours
[02:37:35] <nagios-wm>	 PROBLEM - Puppet freshness on search21 is CRITICAL: Puppet has not run in the last 10 hours
[02:37:36] <nagios-wm>	 PROBLEM - Puppet freshness on search20 is CRITICAL: Puppet has not run in the last 10 hours
[02:37:36] <nagios-wm>	 PROBLEM - Puppet freshness on search31 is CRITICAL: Puppet has not run in the last 10 hours
[02:37:37] <nagios-wm>	 PROBLEM - Puppet freshness on snapshot3 is CRITICAL: Puppet has not run in the last 10 hours
[02:37:37] <nagios-wm>	 PROBLEM - Puppet freshness on snapshot2 is CRITICAL: Puppet has not run in the last 10 hours
[02:37:38] <nagios-wm>	 PROBLEM - Puppet freshness on sq36 is CRITICAL: Puppet has not run in the last 10 hours
[02:37:51] <nagios-wm>	 RECOVERY - Puppet freshness on search27 is OK: puppet ran at Sun Dec 9 02:37:42 UTC 2012
[02:37:51] <nagios-wm>	 RECOVERY - Puppet freshness on search14 is OK: puppet ran at Sun Dec 9 02:37:48 UTC 2012
[02:38:36] <nagios-wm>	 PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours
[02:38:36] <nagios-wm>	 PROBLEM - Puppet freshness on mw10 is CRITICAL: Puppet has not run in the last 10 hours
[02:38:45] <nagios-wm>	 RECOVERY - Puppet freshness on sq73 is OK: puppet ran at Sun Dec 9 02:38:30 UTC 2012
[02:39:57] <nagios-wm>	 RECOVERY - Puppet freshness on sq74 is OK: puppet ran at Sun Dec 9 02:39:35 UTC 2012
[02:39:57] <nagios-wm>	 RECOVERY - Puppet freshness on search30 is OK: puppet ran at Sun Dec 9 02:39:38 UTC 2012
[02:39:57] <nagios-wm>	 RECOVERY - Puppet freshness on sq77 is OK: puppet ran at Sun Dec 9 02:39:39 UTC 2012
[02:40:24] <nagios-wm>	 RECOVERY - Puppet freshness on search1024 is OK: puppet ran at Sun Dec 9 02:40:01 UTC 2012
[02:40:24] <nagios-wm>	 RECOVERY - Puppet freshness on sq86 is OK: puppet ran at Sun Dec 9 02:40:20 UTC 2012
[02:40:51] <nagios-wm>	 RECOVERY - Puppet freshness on search21 is OK: puppet ran at Sun Dec 9 02:40:40 UTC 2012
[02:41:27] <nagios-wm>	 RECOVERY - Puppet freshness on sq72 is OK: puppet ran at Sun Dec 9 02:41:11 UTC 2012
[02:42:21] <nagios-wm>	 RECOVERY - Puppet freshness on dataset2 is OK: puppet ran at Sun Dec 9 02:42:05 UTC 2012
[02:43:24] <nagios-wm>	 RECOVERY - Puppet freshness on kaulen is OK: puppet ran at Sun Dec 9 02:43:09 UTC 2012
[02:43:24] <nagios-wm>	 RECOVERY - Puppet freshness on ms-fe2 is OK: puppet ran at Sun Dec 9 02:43:12 UTC 2012
[02:43:24] <nagios-wm>	 RECOVERY - Puppet freshness on search18 is OK: puppet ran at Sun Dec 9 02:43:14 UTC 2012
[02:44:27] <nagios-wm>	 RECOVERY - Puppet freshness on search1007 is OK: puppet ran at Sun Dec 9 02:44:19 UTC 2012
[02:44:36] <nagios-wm>	 RECOVERY - Puppet freshness on sq70 is OK: puppet ran at Sun Dec 9 02:44:24 UTC 2012
[02:45:21] <nagios-wm>	 RECOVERY - Puppet freshness on snapshot3 is OK: puppet ran at Sun Dec 9 02:45:03 UTC 2012
[02:45:21] <nagios-wm>	 RECOVERY - Puppet freshness on lardner is OK: puppet ran at Sun Dec 9 02:45:05 UTC 2012
[02:45:57] <nagios-wm>	 RECOVERY - Puppet freshness on mw3 is OK: puppet ran at Sun Dec 9 02:45:25 UTC 2012
[02:46:51] <nagios-wm>	 RECOVERY - Puppet freshness on search1016 is OK: puppet ran at Sun Dec 9 02:46:30 UTC 2012
[02:46:51] <nagios-wm>	 RECOVERY - Puppet freshness on search28 is OK: puppet ran at Sun Dec 9 02:46:49 UTC 2012
[02:47:54] <nagios-wm>	 RECOVERY - Puppet freshness on mw10 is OK: puppet ran at Sun Dec 9 02:47:42 UTC 2012
[02:47:54] <nagios-wm>	 RECOVERY - Puppet freshness on labstore1 is OK: puppet ran at Sun Dec 9 02:47:46 UTC 2012
[02:48:12] <nagios-wm>	 RECOVERY - Puppet freshness on mc15 is OK: puppet ran at Sun Dec 9 02:48:01 UTC 2012
[02:49:25] <nagios-wm>	 RECOVERY - Puppet freshness on sq79 is OK: puppet ran at Sun Dec 9 02:49:11 UTC 2012
[02:49:51] <nagios-wm>	 RECOVERY - Puppet freshness on analytics1013 is OK: puppet ran at Sun Dec 9 02:49:37 UTC 2012
[02:50:27] <nagios-wm>	 RECOVERY - Puppet freshness on search1004 is OK: puppet ran at Sun Dec 9 02:49:54 UTC 2012
[02:50:27] <nagios-wm>	 RECOVERY - Puppet freshness on virt1007 is OK: puppet ran at Sun Dec 9 02:50:02 UTC 2012
[02:50:27] <nagios-wm>	 RECOVERY - Puppet freshness on constable is OK: puppet ran at Sun Dec 9 02:50:06 UTC 2012
[02:50:27] <nagios-wm>	 RECOVERY - Puppet freshness on snapshot2 is OK: puppet ran at Sun Dec 9 02:50:13 UTC 2012
[02:50:54] <nagios-wm>	 RECOVERY - Puppet freshness on sq80 is OK: puppet ran at Sun Dec 9 02:50:51 UTC 2012
[02:51:57] <nagios-wm>	 RECOVERY - Puppet freshness on search1018 is OK: puppet ran at Sun Dec 9 02:51:39 UTC 2012
[02:52:24] <nagios-wm>	 RECOVERY - Puppet freshness on analytics1020 is OK: puppet ran at Sun Dec 9 02:52:00 UTC 2012
[02:52:24] <nagios-wm>	 RECOVERY - Puppet freshness on mc12 is OK: puppet ran at Sun Dec 9 02:52:04 UTC 2012
[02:52:51] <nagios-wm>	 RECOVERY - Puppet freshness on mc7 is OK: puppet ran at Sun Dec 9 02:52:46 UTC 2012
[02:53:18] <nagios-wm>	 RECOVERY - Puppet freshness on analytics1019 is OK: puppet ran at Sun Dec 9 02:52:55 UTC 2012
[02:53:27] <nagios-wm>	 RECOVERY - Puppet freshness on sq81 is OK: puppet ran at Sun Dec 9 02:53:10 UTC 2012
[02:53:54] <nagios-wm>	 RECOVERY - Puppet freshness on search31 is OK: puppet ran at Sun Dec 9 02:53:29 UTC 2012
[02:54:21] <nagios-wm>	 RECOVERY - Puppet freshness on tola is OK: puppet ran at Sun Dec 9 02:54:06 UTC 2012
[02:54:57] <nagios-wm>	 RECOVERY - Puppet freshness on search24 is OK: puppet ran at Sun Dec 9 02:54:29 UTC 2012
[02:56:27] <nagios-wm>	 RECOVERY - Puppet freshness on search1005 is OK: puppet ran at Sun Dec 9 02:56:19 UTC 2012
[02:56:27] <nagios-wm>	 RECOVERY - Puppet freshness on analytics1017 is OK: puppet ran at Sun Dec 9 02:56:20 UTC 2012
[02:57:21] <nagios-wm>	 RECOVERY - Puppet freshness on search1001 is OK: puppet ran at Sun Dec 9 02:56:59 UTC 2012
[02:57:21] <nagios-wm>	 RECOVERY - Puppet freshness on analytics1018 is OK: puppet ran at Sun Dec 9 02:57:15 UTC 2012
[02:58:25] <nagios-wm>	 RECOVERY - Puppet freshness on analytics1022 is OK: puppet ran at Sun Dec 9 02:58:00 UTC 2012
[02:58:25] <nagios-wm>	 RECOVERY - Puppet freshness on search13 is OK: puppet ran at Sun Dec 9 02:58:01 UTC 2012
[02:58:51] <nagios-wm>	 RECOVERY - Puppet freshness on analytics1014 is OK: puppet ran at Sun Dec 9 02:58:32 UTC 2012
[02:58:51] <nagios-wm>	 RECOVERY - Puppet freshness on labstore2 is OK: puppet ran at Sun Dec 9 02:58:39 UTC 2012
[02:58:51] <nagios-wm>	 RECOVERY - Puppet freshness on sq65 is OK: puppet ran at Sun Dec 9 02:58:46 UTC 2012
[02:59:54] <nagios-wm>	 RECOVERY - Puppet freshness on sq84 is OK: puppet ran at Sun Dec 9 02:59:24 UTC 2012
[02:59:54] <nagios-wm>	 RECOVERY - Puppet freshness on pdf2 is OK: puppet ran at Sun Dec 9 02:59:29 UTC 2012
[02:59:54] <nagios-wm>	 RECOVERY - Puppet freshness on analytics1010 is OK: puppet ran at Sun Dec 9 02:59:40 UTC 2012
[03:00:57] <nagios-wm>	 RECOVERY - Puppet freshness on snapshot1003 is OK: puppet ran at Sun Dec 9 03:00:27 UTC 2012
[03:00:57] <nagios-wm>	 RECOVERY - Puppet freshness on search16 is OK: puppet ran at Sun Dec 9 03:00:40 UTC 2012
[03:00:57] <nagios-wm>	 RECOVERY - Puppet freshness on sq76 is OK: puppet ran at Sun Dec 9 03:00:41 UTC 2012
[03:00:57] <nagios-wm>	 RECOVERY - Puppet freshness on search34 is OK: puppet ran at Sun Dec 9 03:00:42 UTC 2012
[03:00:57] <nagios-wm>	 RECOVERY - Puppet freshness on sq36 is OK: puppet ran at Sun Dec 9 03:00:46 UTC 2012
[03:01:24] <nagios-wm>	 RECOVERY - Puppet freshness on mc16 is OK: puppet ran at Sun Dec 9 03:01:01 UTC 2012
[03:01:24] <nagios-wm>	 RECOVERY - Puppet freshness on search19 is OK: puppet ran at Sun Dec 9 03:01:08 UTC 2012
[03:01:24] <nagios-wm>	 RECOVERY - Puppet freshness on search20 is OK: puppet ran at Sun Dec 9 03:01:17 UTC 2012
[03:02:27] <nagios-wm>	 RECOVERY - Puppet freshness on search33 is OK: puppet ran at Sun Dec 9 03:02:14 UTC 2012
[03:03:21] <nagios-wm>	 RECOVERY - Puppet freshness on sq44 is OK: puppet ran at Sun Dec 9 03:02:53 UTC 2012
[03:03:21] <nagios-wm>	 RECOVERY - Puppet freshness on sq59 is OK: puppet ran at Sun Dec 9 03:03:01 UTC 2012
[03:03:21] <nagios-wm>	 RECOVERY - Puppet freshness on search1017 is OK: puppet ran at Sun Dec 9 03:03:09 UTC 2012
[03:03:58] <nagios-wm>	 RECOVERY - Puppet freshness on virt2 is OK: puppet ran at Sun Dec 9 03:03:44 UTC 2012
[03:04:24] <nagios-wm>	 RECOVERY - Puppet freshness on mc10 is OK: puppet ran at Sun Dec 9 03:03:57 UTC 2012
[03:04:51] <nagios-wm>	 RECOVERY - Puppet freshness on snapshot1 is OK: puppet ran at Sun Dec 9 03:04:35 UTC 2012
[03:05:27] <nagios-wm>	 RECOVERY - Puppet freshness on search17 is OK: puppet ran at Sun Dec 9 03:05:08 UTC 2012
[03:05:54] <nagios-wm>	 RECOVERY - Puppet freshness on search1015 is OK: puppet ran at Sun Dec 9 03:05:33 UTC 2012
[03:06:22] <nagios-wm>	 RECOVERY - Puppet freshness on analytics1004 is OK: puppet ran at Sun Dec 9 03:06:14 UTC 2012
[03:06:57] <nagios-wm>	 RECOVERY - Puppet freshness on db29 is OK: puppet ran at Sun Dec 9 03:06:25 UTC 2012
[03:07:06] <nagios-wm>	 RECOVERY - Puppet freshness on virt1008 is OK: puppet ran at Sun Dec 9 03:06:51 UTC 2012
[03:07:24] <nagios-wm>	 RECOVERY - Puppet freshness on cadmium is OK: puppet ran at Sun Dec 9 03:07:17 UTC 2012
[03:08:27] <nagios-wm>	 RECOVERY - Puppet freshness on db59 is OK: puppet ran at Sun Dec 9 03:07:57 UTC 2012
[03:12:21] <nagios-wm>	 RECOVERY - Puppet freshness on db9 is OK: puppet ran at Sun Dec 9 03:12:14 UTC 2012
[04:05:52] <nagios-wm>	 PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours
[04:05:52] <nagios-wm>	 PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours
[04:05:52] <nagios-wm>	 PROBLEM - Puppet freshness on sq48 is CRITICAL: Puppet has not run in the last 10 hours
[05:07:52] <Ryan_Lane>	 !log clearing apache logs older than 7 days on wikitech (/var is full)
[05:08:07] <Ryan_Lane>	 ah. right. the bot is dead too
[05:08:51] <Ryan_Lane>	 !log restarting apache on wikitech
[05:08:55] <Ryan_Lane>	 ....
[05:08:57] <Ryan_Lane>	 I'm stupid
[05:09:59] <ori-l>	 Logged the message, master.
[05:10:14] <Ryan_Lane>	 wtf
[05:10:52] <Ryan_Lane>	 wikitech has 424 mb of ram?
[05:10:54] <Ryan_Lane>	 seriously?
[05:12:40] <Matthew_>	 That's like ... nothing.
[05:13:39] <Ryan_Lane>	 yes
[05:13:41] <Ryan_Lane>	 it's absurd
[05:14:22] <jeremyb>	              total       used       free     shared    buffers     cached
[05:14:25] <jeremyb>	 Mem:           496        462         34          0         58        326
[05:14:37] <jeremyb>	 i wonder why you have 424 and i have 496 ;)
[05:14:58] <jeremyb>	 maybe i'm 32 bit and you're 64
[05:15:18] <jeremyb>	 are you saying it needs more than that?
[05:15:23] <Ryan_Lane>	 yes
[05:15:28] <Ryan_Lane>	 [Sun Dec  9 05:04:39 2012] [apc-warning] Unable to allocate memory for pool. in /srv/org/wikimedia/wikitech-1.19.2/includes/AutoLoader.php on line 1007.
[05:15:37] <Ryan_Lane>	 the apache error log is *full* of that
[05:15:43] <jeremyb>	 aha
[05:15:47] <Ryan_Lane>	 in fact, that's what filled up var
[05:15:54] <jeremyb>	 maybe it can just run it's own varnish?
[05:16:10] <Ryan_Lane>	 with what memory?
[05:16:11] <jeremyb>	 or is load not the problem?
[05:16:18] <Ryan_Lane>	 memory is the issue
[05:16:29] <Ryan_Lane>	 it's swapping, a lot
[05:17:42] <jeremyb>	 well to give you an idea: the wikimediadc.org box on the same service where wikitech runs has twice the specs of wikitech. it runs more than a couple wikis and wordpress and a few other things. and varnish. and load is very rarely >0.1. even in the peak of the WLM campaign (when we were getting traffic from WMF banners)
[05:18:11] <jeremyb>	 we've not recently tried running with the same specs as wikitech. maybe we could. idk
[05:18:25] <jeremyb>	 but we're also not getting load from banners
[05:18:52] <Ryan_Lane>	 again, load isn't an issue
[05:19:17] <Ryan_Lane>	 the box simply doesn't have enough memory to run without swapping
[05:19:27] <jeremyb>	 hrmmm
[05:19:49] <jeremyb>	 well, i'm not sure what's running or how it's set up
[05:20:10] <Ryan_Lane>	 IMO linux takes a minimum of 1GB to properly run
[05:20:30] <Ryan_Lane>	 unless you don't really want to do anything with it
[05:20:34] <jeremyb>	 maybe it would be worth using nginx. anyway, you can also resize to 1G or whatever you like without too much hassle
[05:20:45] <Ryan_Lane>	 I'd prefer to move off of linode
[05:21:12] <Ryan_Lane>	 linode is slow, and shitty
[05:21:14] <jeremyb>	 to where?
[05:21:25] <Ryan_Lane>	 rackspace or dreamhost
[05:21:27] <jeremyb>	 and how about puppetizing it?
[05:21:29] <jeremyb>	 huh
[05:21:35] <jeremyb>	 i guess those are both nova ;P
[05:21:35] <Ryan_Lane>	 (cloud services)
[05:21:48] <jeremyb>	 hi morebots!
[05:21:51] <Ryan_Lane>	 they also both have support
[05:22:03] <Ryan_Lane>	 !log rebooted wikitech and cleared some logs to free up /var (which was full)
[05:22:10] <Ryan_Lane>	 ec2 for instance has no support
[05:22:13] <morebots>	 Logged the message, Master
[05:22:33] <nagios-wm>	 PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours
[05:24:02] <jeremyb>	 Ryan_Lane: and what about the puppetizing question?
[05:24:17] <Ryan_Lane>	 once we have everything in modules that's easy
[05:24:22] <Ryan_Lane>	 until then I don't think it's likely
[05:24:42] <Ryan_Lane>	 we don't really need everything in modules, I guess
[05:24:50] <sumanah>	 yay puppetization. Ryan_Lane I'm working through that aspect of the Ops blog post I'm writing
[05:24:51] <Ryan_Lane>	 I have a mediawiki puppet module I wrote for openstack
[05:24:55] <jeremyb>	 right. modules are important and there was kind of a religion against them until recently
[05:25:01] <Ryan_Lane>	 we're using that now
[05:25:10] <Ryan_Lane>	 I'm switching openstack's wiki to mediawiki on monday
[05:25:15] <jeremyb>	 Ryan_Lane: where is that module?
[05:25:24] <Ryan_Lane>	 in openstack's ci repo
[05:25:29] <jeremyb>	 k
[05:26:00] <Ryan_Lane>	 https://review.openstack.org/p/openstack/openstack-ci-puppet.git
[05:26:18] <Ryan_Lane>	 it's not an amazing module, but it gets the job done
[05:26:29] <Ryan_Lane>	 it doesn't really set up mediawiki, either
[05:26:30] <jeremyb>	 Ryan_Lane: err, 404?
[05:26:49] <jeremyb>	 404 && a white screen of death
[05:26:52] <Ryan_Lane>	 hm
[05:28:20] <Ryan_Lane>	 https://review.openstack.org/gitweb?p=openstack%2Fopenstack-ci-puppet.git;a=shortlog;h=HEAD
[05:33:48] <jeremyb>	 hrmmm, that works
[05:39:48] <nagios-wm>	 PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours
[05:39:48] <nagios-wm>	 PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours
[05:48:30] <jeremyb>	 Ryan_Lane: are there plans for openstack to use notgitwweb?
[05:48:43] <Ryan_Lane>	 openstack's gerrit?
[05:48:48] <jeremyb>	 yeah
[05:48:51] <Ryan_Lane>	 dunno
[05:49:07] <Ryan_Lane>	 maybe I'll ask them when we switch
[05:49:08] <jeremyb>	 just because i hear so much about that for WMF gerrit ;)
[05:49:36] <jeremyb>	 maybe we can steal some of their modules? what i've (briefly) read so far looks nice
[05:49:46] <Ryan_Lane>	 we're fans of sharing infrastructure, so maybe for once we can do something first :D
[05:49:52] <Ryan_Lane>	 modules?
[05:49:54] <Ryan_Lane>	 which ones?
[05:49:55] <jeremyb>	 puppet
[05:50:17] <Ryan_Lane>	 they mostly use upstream ones
[05:50:27] <jeremyb>	 for now i was looking at one that definitely not be prod. openstack_project::eavesdrop
[05:50:40] <Ryan_Lane>	 what's that?
[05:50:44] <jeremyb>	 meetbot
[05:50:52] <Ryan_Lane>	 ah
[05:50:55] <jeremyb>	 that would*
[05:51:07] <Ryan_Lane>	 I like meetbot a lot
[05:51:12] <jeremyb>	 do they copy/paste or do use submodules or?
[05:51:14] <Ryan_Lane>	 and I'd like us to do irc meetings
[05:51:15] <Ryan_Lane>	 and use it
[05:51:32] <Ryan_Lane>	 they use the puppet way of installing them somehow
[05:51:49] <jeremyb>	 i had dinner with it's primary driver for the last few years (AIUI) last night
[05:51:56] <jeremyb>	 oh, ewww ;(
[05:52:03] <Ryan_Lane>	 man apple did a really good job with the new itunes
[05:52:16] <Ryan_Lane>	 the store is incredibly better
[06:57:31] <nagios-wm>	 PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours
[07:40:53] <nagios-wm>	 PROBLEM - Puppet freshness on ms-be3 is CRITICAL: Puppet has not run in the last 10 hours
[09:50:22] <nagios-wm>	 PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours
[09:55:03] <MaxSem>	 do we need security training, too? http://forums.thedailywtf.com/forums/p/26967/310301.aspx#310301
[09:58:19] <nagios-wm>	 PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours
[11:41:33] <nagios-wm>	 PROBLEM - Puppet freshness on ms-be3002 is CRITICAL: Puppet has not run in the last 10 hours
[13:31:10] <nagios-wm>	 PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100%
[13:31:37] <nagios-wm>	 RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms
[13:35:22] <nagios-wm>	 PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused
[13:53:22] <nagios-wm>	 RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.052 second response time
[14:07:28] <nagios-wm>	 PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours
[14:07:29] <nagios-wm>	 PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours
[14:07:29] <nagios-wm>	 PROBLEM - Puppet freshness on sq48 is CRITICAL: Puppet has not run in the last 10 hours
[15:23:35] <nagios-wm>	 PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours
[15:23:36] <nagios-wm>	 PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours
[15:40:32] <nagios-wm>	 PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours
[15:40:32] <nagios-wm>	 PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours
[16:58:36] <nagios-wm>	 PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours
[17:43:13] <nagios-wm>	 PROBLEM - Puppet freshness on ms-be3 is CRITICAL: Puppet has not run in the last 10 hours
[17:47:50] <gerrit-wm>	 New review: Alex Monk; "I39e532ce is simpler, I suggest we just use that until the extension can be fixed." [operations/mediawiki-config] (master) C: 0;  - https://gerrit.wikimedia.org/r/37650
[17:50:32] <gerrit-wm>	 New review: Krinkle; "-1 for archive.org url breakage." [operations/apache-config] (master) C: -1;  - https://gerrit.wikimedia.org/r/13293
[18:07:33] <Krenair>	 Hi paravoid
[18:09:47] <Krenair>	 Basically what needs to happen is I2c6ab07d be rebased against master and fixed (per Krinkle's comment)
[18:10:15] <Krenair>	 Then we can rebase I2a7a5b66 on top of that
[19:48:09] <nagios-wm>	 PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100%
[19:50:42] <nagios-wm>	 RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms
[19:51:18] <nagios-wm>	 PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours
[19:54:27] <nagios-wm>	 PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused
[19:59:15] <nagios-wm>	 PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours
[20:09:00] <nagios-wm>	 RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.048 second response time
[21:43:03] <nagios-wm>	 PROBLEM - Puppet freshness on ms-be3002 is CRITICAL: Puppet has not run in the last 10 hours