[00:01:32] PROBLEM - SSH on es1004 is CRITICAL: Server answer: [00:08:13] !log rebooting ssl1004 [00:08:15] Logged the message, Master [00:08:21] no clue what happened to it [00:08:27] did someone push out a change to nginx? [00:08:55] that's usually what triggers this [00:10:50] RECOVERY - SSH on ssl1004 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [00:21:02] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:27:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 8.950 seconds [01:01:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:08:53] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 8.898 seconds [01:43:14] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:44:08] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 284 seconds [01:46:59] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 20 seconds [01:49:05] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 4.547 seconds [02:23:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:29:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 6.673 seconds [02:48:33] New patchset: Ryan Lane; "Change to move projects into another OU rename groups" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4835 [02:48:49] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4835 [02:49:41] PROBLEM - Puppet freshness on sq34 is CRITICAL: Puppet has not run in the last 10 hours [02:51:16] New review: Ryan Lane; "This requires LDAP changes too, and a lot of things need to be synched for this to work. Please don'..." [operations/puppet] (production); V: 0 C: -1; - https://gerrit.wikimedia.org/r/4835 [02:59:44] PROBLEM - udp2log log age on emery is CRITICAL: CRITICAL: log files /var/log/squid/e3_necromancy_idle1year.log, /var/log/squid/e3_necromancy_idle3month.log, have not been written to in 6 hours [03:02:44] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [05:15:22] PROBLEM - Puppet freshness on gilman is CRITICAL: Puppet has not run in the last 10 hours [06:11:51] PROBLEM - MySQL Replication Heartbeat on db1018 is CRITICAL: CRIT replication delay 193 seconds [06:12:18] PROBLEM - MySQL Slave Delay on db1018 is CRITICAL: CRIT replication delay 213 seconds [06:12:54] PROBLEM - MySQL Replication Heartbeat on db24 is CRITICAL: CRIT replication delay 182 seconds [06:13:12] PROBLEM - MySQL Slave Delay on db24 is CRITICAL: CRIT replication delay 193 seconds [06:40:07] PROBLEM - Host bellin is DOWN: PING CRITICAL - Packet loss = 100% [06:41:10] RECOVERY - Host bellin is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [06:41:46] RECOVERY - udp2log log age on emery is OK: OK: all log files active [06:49:07] PROBLEM - Host bellin is DOWN: PING CRITICAL - Packet loss = 100% [06:50:10] RECOVERY - Host bellin is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [06:51:58] PROBLEM - MySQL Idle Transactions on db13 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:57:06] RECOVERY - MySQL Idle Transactions on db13 is OK: OK longest blocking idle transaction sleeps for 0 seconds [07:04:36] PROBLEM - MySQL Idle Transactions on db13 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:05:48] RECOVERY - MySQL Idle Transactions on db13 is OK: OK longest blocking idle transaction sleeps for 0 seconds [08:23:58] New patchset: Dzahn; "initial commit - test pushing to new repo and add index.php" [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/4845 [08:24:24] RECOVERY - MySQL Slave Delay on db24 is OK: OK replication delay 0 seconds [08:25:09] RECOVERY - MySQL Replication Heartbeat on db24 is OK: OK replication delay 0 seconds [08:35:05] New review: Dzahn; "test review" [operations/debs/wikistats] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4845 [08:35:07] Change merged: Dzahn; [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/4845 [09:05:12] PROBLEM - Router interfaces on cr1-sdtpa is CRITICAL: CRITICAL: host 208.80.152.196, interfaces up: 73, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-0/0/1: down - Core: cr1-eqiad:xe-5/2/1 (FPL/GBLX, CV71026) [10Gbps wave]BR [09:08:03] RECOVERY - Router interfaces on cr1-sdtpa is OK: OK: host 208.80.152.196, interfaces up: 75, down: 0, dormant: 0, excluded: 0, unused: 0 [09:12:07] PROBLEM - Puppet freshness on es1004 is CRITICAL: Puppet has not run in the last 10 hours [09:13:09] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [09:13:09] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [09:19:24] RECOVERY - MySQL Replication Heartbeat on db1018 is OK: OK replication delay 0 seconds [09:19:51] RECOVERY - MySQL Slave Delay on db1018 is OK: OK replication delay 0 seconds [09:21:12] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [09:21:12] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [11:39:43] New patchset: Mark Bergsma; "Log multiple requests per udp log packet" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4856 [11:39:44] New patchset: Mark Bergsma; "New version 3.0.2-2wm4: * Log multiple requests per udp log packet" [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4857 [11:40:12] hi mark, can you look at https://gerrit.wikimedia.org/r/4261 , it documents existing symbolic links on fenari ( the /home/w/c/p ones ) [11:40:14] New review: Mark Bergsma; "(no comment)" [operations/debs/varnish] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4857 [11:40:29] New review: Mark Bergsma; "(no comment)" [operations/debs/varnish] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4857 [11:41:10] New review: Mark Bergsma; "(no comment)" [operations/debs/varnish] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4856 [11:41:18] New review: Mark Bergsma; "(no comment)" [operations/debs/varnish] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4856 [11:41:20] Change merged: Mark Bergsma; [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4857 [11:41:21] Change merged: Mark Bergsma; [operations/debs/varnish] (master) - https://gerrit.wikimedia.org/r/4856 [11:42:04] hashar: I think that should be in a different class, bastion hosts and deployment hosts are gonna be separate soon [11:43:16] New review: Mark Bergsma; "I think this should be put in a different class for MediaWiki deployment purposes. We're going to sp..." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/4261 [11:43:46] mark: any class name suggestion ? :D [11:44:18] misc::deployment-host perhaps [11:45:11] will deployments hosts have /home/wikipedia/logs ? [11:45:22] yes [11:46:16] !log Imported varnish 3.0.2-2wm4 into the Wikimedia APT repository [11:46:20] Logged the message, Master [11:51:14] New patchset: Hashar; "document fenari symbolic links" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4261 [11:51:30] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4261 [11:51:46] mark: I have moved the fenari symbolics links into the new class misc::deployment-host [11:54:15] can you use full paths for targets? [11:56:28] mark: the current ones are relative. That makes ls -l nicer too. [11:56:37] but will surely change them to full if you insist :-] [11:57:43] rereading it, looks like I made a huge mistake too :-D [11:57:59] /home/b should be /home/wikipedia/b [12:00:37] New patchset: Mark Bergsma; "Automatically restart varnish on package upgrades The package scripts do this for the main instance anyway" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4858 [12:00:53] New patchset: Mark Bergsma; "Enable persistent storage on all even numbered hosts, and upgrade Varnish" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4859 [12:01:08] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4858 [12:01:08] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/4859 [12:01:22] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/4858 [12:01:35] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4858 [12:01:38] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4858 [12:03:06] New patchset: Mark Bergsma; "Enable persistent storage on all even numbered hosts, and upgrade Varnish" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4859 [12:03:22] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4859 [12:04:57] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4859 [12:05:00] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4859 [12:05:58] New patchset: Hashar; "document fenari symbolic links" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4261 [12:06:12] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/4261 [12:07:14] New review: Hashar; "Patchset 3:" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/4261 [12:07:35] mark: I have updated the symlinks https://gerrit.wikimedia.org/r/4261 [12:08:51] ! [12:09:22] !log Deploying varnish 3.0.2-2wm4 and enabling persistent storage on all even numbered eqiad upload varnish hosts [12:09:24] Logged the message, Master [12:12:52] New patchset: Hashar; "document fenari symbolic links" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4261 [12:13:00] of course running puppet validation on the wrong file is not very helpfull [12:13:08] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4261 [12:19:55] mark updated :D [12:21:08] you want ${wp} [12:21:12] not {$wp} :) [12:21:18] also [12:21:26] you can put all that in one file resource [12:21:37] (not important though) [12:26:15] New patchset: Hashar; "document fenari symbolic links" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4261 [12:26:33] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4261 [12:27:03] and here is the fifth patchset https://gerrit.wikimedia.org/r/#patch,unified,4261,5,manifests/misc/deployment-host.pp [12:27:03] :D [12:27:17] now the indentation is wrong ;) [12:27:20] OH NO [12:27:22] 6th! [12:28:47] New patchset: Hashar; "document fenari symbolic links" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4261 [12:28:59] never trust automatic replacement [12:29:04] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4261 [12:29:07] https://gerrit.wikimedia.org/r/#patch,unified,4261,6,manifests/misc/deployment-host.pp [12:31:49] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4261 [12:31:52] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4261 [12:34:22] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 202 seconds [12:34:40] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 211 seconds [12:35:57] hashar: err: Could not retrieve catalog from remote server: Error 400 on SERVER: left operand of + is not a number at /var/lib/git/operations/puppet/manifests/misc/deployment-host.pp:14 on node fenari.wikimedia.org [12:35:59] alas ;) [12:36:13] you need target => "${wp}/bla" [12:36:55] oh no [12:39:08] New patchset: Hashar; "fix variable concatenation" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4860 [12:39:24] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4860 [12:40:08] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4860 [12:40:11] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4860 [12:42:35] Afternoon guys. Hate to be the bearer of possibly bad news, but I am unable to get into english wikipedia - 3 attempts have each given me a clear message informing me that my account does not exist. [12:43:15] It suggests to check for case sensitivity, etc - just wondering if we have any db errors or anything noticed at all. [12:43:31] PROBLEM - udp2log log age on emery is CRITICAL: CRITICAL: log files /var/log/squid/e3_necromancy_idle1year.log, /var/log/squid/e3_necromancy_idle3month.log, have not been written to in 6 hours [12:43:56] I am out to grab some food [12:44:03] not seeing any db errors [12:44:04] dank je wel mark [12:44:12] and just logged in fine [12:44:20] hm. [12:44:54] I've had issues similar to this in the past, binasher - but that's when I have been operating from my linux kit at home - I'm away at the moment on a totally different machine and OS [12:45:51] the last time I had this, I had to clear my entire cache out, and mess with stuff. I don't know why I'm getting this where I am. Nothing similar at all, different browser, OS and ISP [12:46:55] Your user name there is the same as your irc nick? [12:47:11] This is all I get, binasher - "There is no user by the name "BarkingFish". User names are case sensitive. Please check your spelling, or use the link below to create a new user account." [12:47:24] apergos: yes, same spelling, same case [12:47:46] for me as anon trying to log in as you it tells me the password is wrong (as expected) [12:48:16] if you try a different browser on that host what do you get? [12:48:31] so you're not getting the same error as me? Hm. [12:49:07] This is a locked machine, apergos - I'm not authorised to install any other software on here without sysadmin perms - and the only browser on here is IE [12:49:21] bleah [12:49:35] well next is tossing cache and all cookies [12:50:22] I hate to tell you this, but I can't do that either - "This operation has been cancelled due to restrictions in effect on this computer. Please contact an administrator" [12:50:32] user_name: BarkingFish, user_touched: 20120412235800, user_newpass_time: 20110118045829 [12:50:39] you look ok [12:50:43] PROBLEM - Puppet freshness on sq34 is CRITICAL: Puppet has not run in the last 10 hours [12:50:47] this thing is tighter than a ducks bottom underwater, apergos [12:51:13] except that it undoubtedly writes all kinds of cookies and other sorts of browsing information without letting the user remove any of it [12:51:15] nice administration [12:51:22] so I last touched at 23:58 yesterday evening [12:51:31] New patchset: Pyoungmeister; "udp2log cleanup, new instance" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4782 [12:51:48] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4782 [12:52:24] are you going http or https? just out of curiosity [12:52:34] I'm gonna drop out and contact the helpdesk, get one of their "bods" to come up and fix this. [12:52:39] http, apergos [12:52:53] you could try https (unlikely to make a difference but what the heck right?) [12:53:02] otherwise, yeah get someone there to be useful [12:53:07] yeah, possibly :) [12:53:11] !log restopping puppet on locke/emery [12:53:14] Logged the message, notpeter [12:53:23] I'll see what they can do. Thanks anyway apergos / binasher [12:53:29] good luck [12:53:42] * BarkingFish gets his truncheon and sets off for the IT helpdesk office [12:53:45] bye [12:53:47] :) [12:53:50] later [12:54:08] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4782 [12:54:10] RECOVERY - MySQL Replication Heartbeat on db1035 is OK: OK replication delay 0 seconds [12:54:11] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4782 [12:55:04] RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 0 seconds [12:58:04] New patchset: Pyoungmeister; "needed this too... :/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4865 [12:58:20] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4865 [12:59:31] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4865 [12:59:33] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4865 [13:00:24] I was going to recommend https. [13:00:28] But apergos beat me. [13:01:51] * apergos beats Joan with whips and chains [13:02:01] and immediately pamphlets self [13:02:52] bah, 4gb for my latop is not enough [13:03:52] New patchset: Mark Bergsma; "Upgrade varnish on the noneven eqiad upload hosts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4866 [13:03:53] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [13:04:08] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4866 [13:04:26] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4866 [13:04:29] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4866 [13:18:09] New patchset: Dzahn; "initial import of old wikistats PHP stuff" [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/4871 [13:36:44] New patchset: Pyoungmeister; "will it blend?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4872 [13:36:57] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/4872 [13:46:48] New review: Dzahn; "(no comment)" [operations/debs/wikistats] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4871 [13:46:51] Change merged: Dzahn; [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/4871 [13:47:32] New patchset: Mark Bergsma; "Upgrade all varnish instances to 3.0.2-2wm4" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4873 [13:47:49] New patchset: Dzahn; "first get rid of red stuff in gerrit. go trailing whitespace: :%s/\s\+$// away leading spaces: :%s-^\( \)\+-\t" [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/4874 [13:47:49] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4873 [13:48:04] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4873 [13:48:07] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4873 [13:49:24] New patchset: Pyoungmeister; "will it blend?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4872 [13:49:38] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/4872 [13:50:51] New review: Dzahn; "(no comment)" [operations/debs/wikistats] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4874 [13:50:53] Change merged: Dzahn; [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/4874 [13:58:38] New patchset: Pyoungmeister; "will it blend?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4872 [13:58:52] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/4872 [14:01:10] New patchset: Pyoungmeister; "will it blend?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4872 [14:01:26] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4872 [14:02:02] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4872 [14:02:05] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4872 [14:02:23] PROBLEM - MySQL Slave Delay on db24 is CRITICAL: CRIT replication delay 187 seconds [14:02:32] PROBLEM - MySQL Replication Heartbeat on db24 is CRITICAL: CRIT replication delay 193 seconds [14:03:15] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 189 seconds [14:03:33] PROBLEM - MySQL Replication Heartbeat on db1018 is CRITICAL: CRIT replication delay 237 seconds [14:04:00] PROBLEM - MySQL Slave Delay on db1018 is CRITICAL: CRIT replication delay 253 seconds [14:04:18] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 220 seconds [14:07:37] New patchset: Pyoungmeister; ""one last cleanup" he sez..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4877 [14:07:53] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4877 [14:08:03] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4877 [14:08:06] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4877 [14:10:52] New patchset: Pyoungmeister; "what kind kind of error is that?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4878 [14:11:08] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4878 [14:11:21] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [14:11:39] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [14:12:37] New patchset: Pyoungmeister; "what kind kind of error is that?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4878 [14:12:54] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4878 [14:13:01] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4878 [14:13:04] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4878 [14:17:14] New patchset: Pyoungmeister; "gotta type it" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4879 [14:17:31] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4879 [14:18:00] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4879 [14:18:03] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4879 [14:19:09] RECOVERY - MySQL Replication Heartbeat on db24 is OK: OK replication delay 0 seconds [14:19:18] RECOVERY - MySQL Replication Heartbeat on db1018 is OK: OK replication delay 0 seconds [14:19:27] RECOVERY - MySQL Slave Delay on db24 is OK: OK replication delay 0 seconds [14:19:36] RECOVERY - MySQL Slave Delay on db1018 is OK: OK replication delay 0 seconds [14:22:21] New patchset: Pyoungmeister; "back to built in type;" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4880 [14:22:36] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/4880 [14:26:24] New patchset: Dzahn; "add debian directories and rudimentary control and copyright file" [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/4881 [14:27:08] New patchset: Pyoungmeister; "back to built in type;" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4880 [14:27:24] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4880 [14:27:48] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4880 [14:27:50] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4880 [14:28:02] New review: Dzahn; "(no comment)" [operations/debs/wikistats] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4881 [14:28:04] Change merged: Dzahn; [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/4881 [14:40:45] PROBLEM - Host bellin is DOWN: PING CRITICAL - Packet loss = 100% [14:43:37] New patchset: Pyoungmeister; "wrong place?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4882 [14:43:51] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/4882 [14:51:28] New patchset: Mark Bergsma; "Make backend upload varnish servers talk directly to Swift again" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4883 [14:51:44] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4883 [14:52:06] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4883 [14:52:10] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4883 [14:56:52] New patchset: Pyoungmeister; "wrong place?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4882 [14:57:10] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4882 [14:57:26] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4882 [14:57:28] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4882 [15:10:53] New patchset: Dzahn; "move files around so that a .deb can be built right out of the repo" [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/4886 [15:11:21] New patchset: Pyoungmeister; "will title work?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4887 [15:11:37] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4887 [15:11:38] New review: Dzahn; "(no comment)" [operations/debs/wikistats] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4886 [15:11:40] Change merged: Dzahn; [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/4886 [15:12:00] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/4887 [15:12:08] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4887 [15:12:10] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4887 [15:15:55] New patchset: Pyoungmeister; "regression testing" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4888 [15:15:58] PROBLEM - Puppet freshness on gilman is CRITICAL: Puppet has not run in the last 10 hours [15:16:12] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4888 [15:16:30] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4888 [15:16:33] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4888 [15:22:06] New patchset: Pyoungmeister; "I am so upset with myself right now..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4889 [15:22:22] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4889 [15:22:29] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4889 [15:22:32] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4889 [15:27:04] New patchset: Pyoungmeister; "once more to the breach" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4890 [15:27:20] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4890 [15:27:53] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4890 [15:27:56] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4890 [15:36:03] New patchset: Pyoungmeister; "one final test before letting it loose" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4891 [15:36:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4891 [15:36:33] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4891 [15:36:37] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4891 [15:47:18] diederik: you there? [15:47:42] yeapp [15:47:57] I'm trying to set up oxygen for you properly [15:48:11] what do you knwo about the host? [15:48:24] like, what is already using port 8420? [15:48:58] PROBLEM - LVS HTTP on bits.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:54] hmm [15:51:40] RECOVERY - LVS HTTP on bits.esams.wikimedia.org is OK: HTTP OK HTTP/1.1 200 OK - 3963 bytes in 1.265 seconds [15:55:43] PROBLEM - Varnish HTTP bits on cp3001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:55] RECOVERY - Varnish HTTP bits on cp3001 is OK: HTTP OK HTTP/1.1 200 OK - 632 bytes in 3.489 seconds [15:59:37] PROBLEM - Varnish HTTP bits on cp3002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:41] reports of slowness in wm tech [16:02:46] PROBLEM - Varnish HTTP bits on cp3001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:13] RECOVERY - Varnish HTTP bits on cp3001 is OK: HTTP OK HTTP/1.1 200 OK - 632 bytes in 9.064 seconds [16:07:00] notpeter: what i know is that oxygen has been setup as a multicast proxy, that's it for now (AFAIK) [16:07:12] i have no clue what is already using port 8420 [16:07:56] question: would it be okay to install php5 on bayes? (if yes, then i'll file a ticket) [16:07:59] okie dokie. I'm going to restart it real quick [16:08:07] RECOVERY - Varnish HTTP bits on cp3002 is OK: HTTP OK HTTP/1.1 200 OK - 634 bytes in 3.160 seconds [16:08:39] I don't know what bayes is doing right now, so I can't really answer that, but feel free to file a ticket, and if it's not ok, for some odd reason, then someone will say so [16:10:27] okidoki [16:11:40] notpeter: this is also a relevant ticket for oxygen: http://rt.wikimedia.org/Ticket/Display.html?id=2454 [16:11:45] not sure if you have seen it [16:12:28] yeah, they're kind duplicates. I might merge them [16:12:46] it's really just the "get this working the way diederik wants it" ticket :) [16:12:57] :D [16:13:19] not really, to be honest, it's just an additional filter box that also does multicasting [16:13:30] gotcha [16:15:35] god damnit [16:15:41] I found the varnish instability problem [16:15:45] we simply ran out of sessions [16:15:47] 100k max [16:16:18] that's a lot of sessions [16:16:49] cp3002 kernel: [13023076.306683] possible SYN flooding on port 80. Sending cookies. [16:16:56] some of these in the last hour [16:17:00] not anymore [16:23:34] Speed seems pretty much back to normal for me in the UK in last few mins. [16:24:00] Ooops sorry wrong channel, meant that for wm-tech [16:26:00] New patchset: Mark Bergsma; "Double session_max" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4893 [16:26:16] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4893 [16:26:52] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4893 [16:26:55] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4893 [16:28:30] diederik: FYI, I changed logging on the varnish servers today [16:28:43] those are now sending multiple log lines per udp packet, similar to how the squids always did [16:29:23] mark: okay, does it change how udp2log consumes the log lines? [16:29:31] shouldn't [16:29:42] just making you aware in case you notice something's wrong [16:29:57] squid has always done this, varnish didn't until today [16:30:09] okay, thanks for the headsup :) [16:30:13] it should be a little more efficient for locke/udp2log to process, since there are less packets [16:30:21] cool [16:39:37] PROBLEM - Varnish traffic logger on cp1031 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:39:37] PROBLEM - Varnish traffic logger on cp1021 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:39:37] PROBLEM - Varnish traffic logger on cp1023 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:39:46] PROBLEM - Varnish traffic logger on cp1043 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:39:46] PROBLEM - Varnish traffic logger on cp1022 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:39:46] PROBLEM - Varnish traffic logger on cp1025 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:39:55] PROBLEM - Varnish traffic logger on cp1024 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:40:04] PROBLEM - Varnish traffic logger on cp1034 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:40:04] PROBLEM - Varnish traffic logger on cp1042 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:40:04] PROBLEM - Varnish traffic logger on cp1026 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:40:13] PROBLEM - Varnish traffic logger on cp1036 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:40:13] PROBLEM - Varnish traffic logger on cp1028 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:40:22] PROBLEM - Varnish traffic logger on cp1030 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:40:22] PROBLEM - Varnish traffic logger on cp1044 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:40:31] PROBLEM - Varnish traffic logger on cp1032 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:40:40] PROBLEM - Varnish traffic logger on cp1035 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:40:49] PROBLEM - Varnish traffic logger on cp1033 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:40:58] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [16:42:23] we've got a severe change in bits throughput: http://ganglia.wikimedia.org/latest/graph.php?c=Bits%20caches%20esams&m=load_one&r=hour&s=by%20name&hc=4&mc=2&st=1334335290&g=network_report&z=medium&r=hour [16:42:31] vs. http://ganglia.wikimedia.org/latest/graph.php?c=Bits%20caches%20pmtpa&m=load_one&r=hour&s=by%20name&hc=4&mc=2&st=1334335291&g=network_report&z=medium&r=hour [16:43:18] yeah [16:44:16] RECOVERY - Varnish traffic logger on cp1024 is OK: PROCS OK: 2 processes with command name varnishncsa [16:44:43] RECOVERY - Varnish traffic logger on cp1028 is OK: PROCS OK: 2 processes with command name varnishncsa [16:44:52] RECOVERY - Varnish traffic logger on cp1035 is OK: PROCS OK: 2 processes with command name varnishncsa [16:45:19] RECOVERY - Varnish traffic logger on cp1021 is OK: PROCS OK: 2 processes with command name varnishncsa [16:46:49] RECOVERY - Varnish traffic logger on cp1031 is OK: PROCS OK: 2 processes with command name varnishncsa [16:47:07] RECOVERY - Varnish traffic logger on cp1022 is OK: PROCS OK: 2 processes with command name varnishncsa [16:47:07] RECOVERY - Varnish traffic logger on cp1025 is OK: PROCS OK: 2 processes with command name varnishncsa [16:47:07] RECOVERY - Varnish traffic logger on cp1026 is OK: PROCS OK: 2 processes with command name varnishncsa [16:47:25] RECOVERY - Varnish traffic logger on cp1030 is OK: PROCS OK: 2 processes with command name varnishncsa [16:48:01] RECOVERY - Varnish traffic logger on cp1033 is OK: PROCS OK: 2 processes with command name varnishncsa [16:48:19] RECOVERY - Varnish traffic logger on cp1023 is OK: PROCS OK: 2 processes with command name varnishncsa [16:48:46] RECOVERY - Varnish traffic logger on cp1034 is OK: PROCS OK: 2 processes with command name varnishncsa [16:48:55] RECOVERY - Varnish traffic logger on cp1036 is OK: PROCS OK: 2 processes with command name varnishncsa [16:48:55] RECOVERY - Varnish traffic logger on cp1032 is OK: PROCS OK: 2 processes with command name varnishncsa [16:55:40] RECOVERY - Varnish traffic logger on cp1042 is OK: PROCS OK: 2 processes with command name varnishncsa [16:58:49] RECOVERY - Varnish traffic logger on cp1044 is OK: PROCS OK: 2 processes with command name varnishncsa [17:04:04] RECOVERY - Varnish traffic logger on cp1043 is OK: PROCS OK: 2 processes with command name varnishncsa [17:07:28] New patchset: ArielGlenn; "cleanup for use by other programs:" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/4895 [17:07:53] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 2 processes with command name varnishncsa [17:09:06] New review: ArielGlenn; "(no comment)" [operations/dumps] (ariel); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4895 [17:09:08] Change merged: ArielGlenn; [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/4895 [17:13:37] binasher: Hey, you around? Nikerabbit wants to deploy this MW extension feature that needs a shared database à la CentralAuth, and if this were 2009 I would just create a database for it on s7, but since it's now 2012 and we have a DBA, I figured I'd check with you :) [17:13:50] diederik: is oxygen actaully supposed to be collecting any logs? [17:14:01] notpeter:yes [17:14:27] gotcha [17:14:39] the relay and udp are trying to use the same port [17:14:41] hurray [17:14:44] New patchset: ArielGlenn; "list images per project local and on commons" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/4896 [17:16:09] can't have the multicast relay listen on *:8420 and do the same with udplog [17:16:22] binasher: sounds like quitter talk [17:16:43] need to make socat bing to $external-ip:8420 and make udplog listen on the multicast stream socat outputs [17:17:04] !log restarting udp2log on emery [17:17:06] Logged the message, notpeter [17:17:57] which also depends on wether or not socat sets IP_MULTICAST_LOOP on its sending socket fd [17:18:25] it has to, or nothing on that host will see what its sending if it binds to the mcast address [17:18:44] enjoy :) [17:18:49] hurray [17:20:03] binasher: ping? [17:20:14] beer pong [17:20:15] (re question above) [17:20:30] oh god [17:20:50] New patchset: Pyoungmeister; "script update" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4897 [17:21:06] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4897 [17:21:20] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4897 [17:21:23] PROBLEM - Varnish traffic logger on cp1026 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:21:23] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4897 [17:21:23] PROBLEM - Varnish traffic logger on cp1036 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:21:23] PROBLEM - Varnish traffic logger on cp1044 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:21:23] PROBLEM - Varnish traffic logger on cp1028 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:21:32] PROBLEM - Varnish traffic logger on cp1024 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:21:41] PROBLEM - Varnish traffic logger on cp1030 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:21:41] PROBLEM - Varnish traffic logger on cp1034 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:21:41] PROBLEM - Varnish traffic logger on cp1032 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:21:59] PROBLEM - Varnish traffic logger on cp1035 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:21:59] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:21:59] PROBLEM - Varnish traffic logger on cp1029 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:21:59] PROBLEM - Varnish traffic logger on cp1033 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:21:59] PROBLEM - Varnish traffic logger on cp1043 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:22:00] PROBLEM - Varnish traffic logger on cp1025 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:22:08] PROBLEM - Varnish traffic logger on cp1027 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:22:08] PROBLEM - Varnish traffic logger on cp1031 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:22:08] PROBLEM - Varnish traffic logger on cp1021 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:22:08] PROBLEM - Varnish traffic logger on cp1023 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:22:35] PROBLEM - Varnish traffic logger on cp1042 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:22:44] PROBLEM - Varnish traffic logger on cp1022 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:23:09] RoanKattouw: my answer, since it isn't 2009.. and well, it would have been the same in 2009, is add support for data store sharded by some common key, and spread it over everywhere. something that is shared across all our wikis probably deserves to be shared across db shards. then please rewrite the rest of mw's data access layer :) [17:23:29] lol [17:23:46] who is this binasher? [17:23:54] does he work for facebook? ;) [17:24:09] hey binasher, do you work for facebook? [17:24:15] I'm going to run iwth that rumor. [17:24:30] domas: speaking of facebook, i heard you met darrin (from google) yesterday! [17:24:32] RECOVERY - Varnish traffic logger on cp1032 is OK: PROCS OK: 2 processes with command name varnishncsa [17:24:34] they still haven't offered to pre-date options to 2008, so.. no. [17:24:45] diederik: can you take a look at emery and make sure that everything is runing properly? [17:24:51] lesliecarr: hehe, yes [17:24:53] binasher: So you want us to rewrite this stuff so it can split its data over all 7 clusters, then create a global table on each cluster [17:24:54] LeslieCarr: devin? [17:25:00] he's my secondary :) [17:25:03] *split over 7 DBs [17:25:11] he mentioned a tall facebook mysql person and i was like "domas?" [17:25:26] RECOVERY - Varnish traffic logger on cp1042 is OK: PROCS OK: 2 processes with command name varnishncsa [17:25:30] binasher: who is devin ? [17:25:43] lesliecarr: there're two tall facebook mysql persons [17:25:52] notpeter: afaik, emery is just running fine, there are some warnings about 2 filters but that's because they get so little traffic [17:25:59] :) [17:26:00] domas: are you hoarding them? :) [17:26:14] domas: who's the other? ma rk? [17:26:25] yeh [17:26:29] he is quite tall too [17:26:35] diederik: well, I just did... stuff... to it. it should have had no impact. but I'd like you to check just ot make sure [17:26:51] before I moce onto locke :) [17:26:57] RECOVERY - Varnish traffic logger on cp1022 is OK: PROCS OK: 2 processes with command name varnishncsa [17:27:00] *move [17:27:02] !log created new operations/debs/search-qa repo for packaging search qa scripts [17:27:04] Logged the message, Master [17:27:19] notpeter: the filters are monitored on emery and i haven't seen any new complaints since.. [17:27:30] Jeff_Green: please stick to wikimedia-* for our packages [17:27:31] cool [17:27:33] wikimedia-search-qa [17:27:33] but i'll check quickly [17:27:43] * jeremyb awaits binasher's answer for RoanKattouw [17:27:56] scyouryour [17:28:05] mark: I wish I had known that an hour ago [17:28:07] your [17:28:08] RECOVERY - Varnish traffic logger on cp1023 is OK: PROCS OK: 2 processes with command name varnishncsa [17:28:12] RoanKattouw: if we did that, i'd probably want to build new db clusters for data that is sharded. such that maybe s1 would include s1.{1..4} and data shared across wikis would get its own shard set [17:28:13] afaik we can't rename [17:28:23] diederik: so, I was going to upgrade udplog on locke to 1.8-2, which is what emery is running. any reason to not do that? [17:28:25] just delete and make again [17:28:26] RECOVERY - Varnish traffic logger on cp1044 is OK: PROCS OK: 2 processes with command name varnishncsa [17:28:32] you can maybe hide in the somewhere near future [17:28:39] i thought there was no delete? [17:28:47] notpeter: not that i am aware off, maybe maplebed has an opinion about this [17:28:55] ? [17:29:11] RECOVERY - Varnish traffic logger on cp1034 is OK: PROCS OK: 2 processes with command name varnishncsa [17:29:13] iirc, I didn't upgrade it when I upgraded emery, becuase it was during the fundraiser [17:29:22] maplebed: notpeter: diederik: so, I was going to upgrade udplog on locke to 1.8-2, which is what emery is running. any reason to not do that? [17:29:27] binasher: OK well I'll just tell Niklas to do configurable sharding then I guess [17:29:36] But... wait [17:29:38] hmm [17:29:44] This may or may not be feasible given the nature of the data [17:29:46] maplebed: part of the udp2log puppet cleanup [17:30:05] mark: delete it? [17:30:30] notpeter: +1 upgrade, but be prepared to downgrade if shit blows up. [17:30:31] ;) [17:30:42] :D [17:30:51] maplebed: my path forward shall be illuminated by the bridges buring behind me [17:30:55] *burning [17:31:18] ok, going to upgrade and such [17:31:35] RECOVERY - Varnish traffic logger on cp1026 is OK: PROCS OK: 2 processes with command name varnishncsa [17:31:43] !log upgrading udplog on locke to 1.8-2 and restarting, etc [17:31:45] Logged the message, notpeter [17:32:09] mark: are you referring to the name of the git project or just the package name? [17:32:15] both [17:32:19] since they should be the same [17:32:26] ok. so this will be the first project in git with a name wikimedia-* [17:32:30] yes [17:32:37] none of the existing two are wikimedia native packages [17:32:45] but in svn there are several [17:32:46] wikistats? [17:32:53] LeslieCarr: nm, devin is a googler i used to work with who some peeps here have met climbing [17:32:58] that one is debatable [17:33:43] ok. I'm fine renaming but I have no idea how to do that with git/gerrit and so far have been advised it's not feasible "yet" [17:34:53] RECOVERY - Varnish traffic logger on cp1033 is OK: PROCS OK: 2 processes with command name varnishncsa [17:35:11] PROBLEM - Varnish traffic logger on cp1023 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:35:20] PROBLEM - Varnish traffic logger on cp1042 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:35:29] PROBLEM - Varnish traffic logger on cp1022 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:35:38] PROBLEM - Varnish traffic logger on cp1044 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:35:47] PROBLEM - Varnish traffic logger on cp1032 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:35:56] PROBLEM - Varnish traffic logger on cp1026 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:35:59] Jeff_Green: You can't rename and you can't delete. In 2.3 (which we're upgrading to soon) you can hide [17:36:14] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 2 processes with command name varnishncsa [17:36:14] PROBLEM - Varnish traffic logger on cp1034 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:36:47] so we are at an impasse [17:37:04] i need to drive to new jersey now. yay! back on later today [17:37:08] I guess for now you can just create a new repo with the right name and lock down the old one [17:37:10] shall I create another repo and leave the first one as cruft? [17:37:54] Probably best [17:37:58] We can hide it once 2.3 rolls around [17:38:10] you can't delete? that's silly [17:38:14] And even now we can set the permissions so no one's allowed to do anything with it [17:38:16] * mark gets out mysql-client and rm -rf [17:39:05] RECOVERY - Varnish traffic logger on cp1029 is OK: PROCS OK: 2 processes with command name varnishncsa [17:39:21] heh [17:39:23] RECOVERY - Varnish traffic logger on cp1027 is OK: PROCS OK: 2 processes with command name varnishncsa [17:39:33] going the other direction--seeing a bunch of operations/debs/wikimedia-* seems like a waste of perfectly good pixesl [17:39:35] pixels [17:39:49] what if we just leave the project names short but name the packages appropriately? [17:39:57] Anyway for more discussion re Gerrit renaming/deleting/whatever, talk to ^demon [17:40:10] then what if we have overlapping ubuntu/debian packages? [17:40:13] or is that even possible with debs--to mismatch a directory name with the project name? [17:40:13] you could at least push a note to cruft with a pointer to the new one ;) [17:40:17] can we just keep it simple? package name == project name please [17:40:31] sure [17:40:35] PROBLEM - Varnish traffic logger on cp1033 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:40:35] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 1 process with command name varnishncsa [17:40:58] creating new project... [17:41:26] <^demon> mark: Project deletion is like the most-requested upstream feature. [17:41:54] is that missing from git or missing from gerrit? [17:42:07] deleting a git repo is as simple as rm -rf [17:42:42] <^demon> That's the easy part. But there's more than one table to delete rows from and I don't know how to do it right. [17:42:52] <^demon> Workarounds are all for old 2.1 installs :\ [17:43:15] <^demon> Anyway, with 2.3 we can just click "Hide" and it won't show up except to admins (and then at the bottom of the list) [17:44:03] https://tools.ietf.org/html/rfc6540 [17:44:15] IPv6 Support Required for All IP-Capable Nodes [17:44:16] yay [17:44:29] RECOVERY - Varnish traffic logger on cp1024 is OK: PROCS OK: 2 processes with command name varnishncsa [17:45:05] RECOVERY - Varnish traffic logger on cp1021 is OK: PROCS OK: 2 processes with command name varnishncsa [17:45:59] RECOVERY - Varnish traffic logger on cp1035 is OK: PROCS OK: 2 processes with command name varnishncsa [17:46:30] paravoid: how us centric, all 4 authors are in US and 2 share the same company [17:46:35] RECOVERY - Varnish traffic logger on cp1031 is OK: PROCS OK: 2 processes with command name varnishncsa [17:47:01] it's not like they're defining anything important [17:47:07] they just stated the obvious [17:47:16] but made it into a best practices rfc [17:47:42] still reading [17:51:31] notpeter: i am back in 30 minutes in case of any questions [17:52:00] New patchset: Pyoungmeister; "got a little over zealous there" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4900 [17:52:10] drdee_: ok, cool. I think things are in a good state now [17:52:17] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4900 [17:52:35] RECOVERY - Varnish traffic logger on cp1028 is OK: PROCS OK: 2 processes with command name varnishncsa [17:52:47] New patchset: Jgreen; "new package wikimedia-search-tools, with a new name" [operations/debs/wikimedia-search-qa] (master) - https://gerrit.wikimedia.org/r/4901 [17:53:11] how gender-centeric jeremyb and paravoid, they're all men ! [17:53:11] RECOVERY - Varnish traffic logger on cp1030 is OK: PROCS OK: 2 processes with command name varnishncsa [17:53:31] New review: Jgreen; "(no comment)" [operations/debs/wikimedia-search-qa] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4901 [17:53:33] Change merged: Jgreen; [operations/debs/wikimedia-search-qa] (master) - https://gerrit.wikimedia.org/r/4901 [17:53:38] New patchset: Pyoungmeister; "got a little over zealous there" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4900 [17:53:53] LeslieCarr: haha, good point [17:53:54] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4900 [17:54:15] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4900 [17:54:17] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4900 [17:54:41] !log created new repo operations/debs/wikimedia-search-qa to stay within package naming conventions [17:54:43] Logged the message, Master [17:55:54] * jeremyb thinks those are prolly unrelated problems [17:55:58] New patchset: Jgreen; "grr, wrong package name" [operations/debs/search-qa] (master) - https://gerrit.wikimedia.org/r/4902 [17:56:13] New review: Jgreen; "(no comment)" [operations/debs/search-qa] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4902 [17:56:15] Change merged: Jgreen; [operations/debs/search-qa] (master) - https://gerrit.wikimedia.org/r/4902 [17:56:47] ok, new repo created, content deleted from the mis-named repo, is there anything else to do? [18:18:04] New patchset: Jgreen; "tweaked version number and copyright to better fit wmf traditions" [operations/debs/wikimedia-search-qa] (master) - https://gerrit.wikimedia.org/r/4903 [18:18:28] New review: Jgreen; "(no comment)" [operations/debs/wikimedia-search-qa] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4903 [18:18:30] Change merged: Jgreen; [operations/debs/wikimedia-search-qa] (master) - https://gerrit.wikimedia.org/r/4903 [18:44:08] New patchset: Lcarr; "Fixing /etc/puppet/ to /var/lib/git/operations for puppet files" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4905 [18:44:24] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4905 [19:02:06] New patchset: Lcarr; "Fixing git directories and geoip" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4905 [19:02:23] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4905 [19:02:58] Lcarr: just curious, where are you using geoip? [19:04:53] geoiplookup.wikimedia.org for example [19:05:20] New patchset: Lcarr; "Fixing git directories and geoip" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4905 [19:05:37] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4905 [19:07:09] mark: do you use the free or paid version? [19:07:30] free [19:07:43] mark: we have a license for the paid, let me know if you need it [19:07:46] New patchset: Lcarr; "Fixing git directories and geoip" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4905 [19:07:55] that might be good yes [19:07:56] check 20 :) [19:08:01] drdee: ooo [19:08:04] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4905 [19:08:19] oooo? [19:08:46] ooo paid geoip which i am assuming has much more fine grained knobs to twiddle around [19:08:55] LeslieCarr: include, not require puppetmaster::config [19:09:07] require introduces dependencies which can lead to loops and in any case make puppet slow [19:09:37] you only need to refer to that variable, so include is enough [19:09:43] mark, LeslieCarr: maybe we can also fix this ticket http://rt.wikimedia.org/Ticket/Display.html?id=2385 (automatic updating of the geoip db) [19:09:51] New patchset: Lcarr; "Fixing git directories and geoip" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4905 [19:09:57] I can share api key [19:10:07] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4905 [19:10:14] yeah we can do that [19:10:22] we already have automatic updating of the free version on many servers [19:10:42] that's what leslie is now working on [19:10:46] do you see a downside of replacing the free version witht the paid version mark ? [19:10:54] I don't [19:11:02] as long as it's easy to retrieve that file [19:11:15] and of course we need to hide the license key [19:11:17] so not in public puppet [19:11:26] yeah [19:11:58] drdee: can you email the key and i'll put it in our private repo ? [19:12:29] first i'll commit these changes and make sure they don't break everything, then i'll fix up so we are auto updating with the paid ver everywhere [19:12:52] PROBLEM - Puppet freshness on es1004 is CRITICAL: Puppet has not run in the last 10 hours [19:13:55] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [19:13:55] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [19:15:09] LesLieCarr, yep i'll do that [19:15:26] who can create me an RT account? [19:15:34] mark? [19:15:41] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4905 [19:15:45] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4905 [19:15:54] CT likes doing that [19:16:02] ah [19:22:05] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [19:22:05] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [19:23:09] New patchset: Lcarr; ""true" does not parse" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4909 [19:23:21] LeslieCarr: ideally, this should be a standalone puppet so I can also use it on Locke,Emery and Oxygen [19:23:27] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4909 [19:23:53] yes and no , so if i pull it to stafford, we should be able to push it then to the other machines ... [19:24:03] unless you want to only do it on demand or something ? [19:24:19] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4909 [19:24:22] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4909 [19:24:38] okay sounds good, i think we always want the mos up-to-date version [19:24:48] :) [19:24:54] curious, what data does this one have ? [19:25:07] like there's the big chart of all the different databases [19:25:12] https://www.maxmind.com/app/geoip_features [19:25:21] we have licenses for country and city level [19:25:28] cool :) [19:26:04] :) when we get the caching center should be very helpful ! [19:26:06] maybe we should try it first in labs? (we also need to rename the downloaded file to a canonical geoip db name) [19:29:28] ah [19:29:37] so each one needs to be unique ? [19:29:42] or just to a certain name ? [19:30:38] Hello, has anyone seen Asher today please? [19:31:21] he's off this week, IIRC [19:31:24] hashar: yeah, he was on [19:31:36] oh so he must be blinking :-] [19:31:40] on off on off on off [19:31:42] UTC: 13 17:37:04 < binasher> i need to drive to new jersey now. yay! back on later today [19:31:47] \o/ [19:31:59] so I will catch him monday. Thanks! [19:32:37] Lesliecarr: no just always the same name, but when you download the update i think you will always get a version that contains the date in the filename [19:33:31] cool, it's called "update" [19:33:34] (checked this out) [19:36:47] New patchset: Mark Bergsma; "Don't up or downgrade varnish right now" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4911 [19:37:03] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4911 [19:37:10] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4911 [19:37:13] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4911 [19:37:22] LeslieCarr: do you have privileges to create me an observium account? [19:37:42] paravoid: if you ask nice :) [19:39:47] LeslieCarr, can I help test it? [19:39:52] or do something useful? [19:47:40] nothing right now, need to make an account for paravoid on another system then i need some foods …. [19:47:52] and then i need to fix the broken puppet job [19:47:57] and then, we can fix this :) [19:48:17] assuming nothing breaks in between now and then [19:57:12] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4835 [19:57:15] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4835 [20:25:53] New patchset: preilly; "Add ACL for carriers and redirect support for carriers landing page" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4915 [20:26:09] New patchset: preilly; "Add new mobile puppet manifest with simple vumi class * provides ussd application server" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4483 [20:26:24] Change abandoned: preilly; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4915 [20:33:13] New patchset: preilly; "Add ACL for carriers and redirect support for carriers landing page" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4916 [20:34:50] New patchset: preilly; "Add ACL for carriers and redirect support for carriers landing page" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4916 [20:38:31] New review: preilly; "See the Partner IP Ranges located at:" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/4916 [20:39:52] maplebed, LeslieCarr: would oneo f you mind flushing the mobile varnish cache? [20:40:05] or any other capable opsen ^ [20:41:13] one outage is enough, thank you [20:41:31] mark: ha ha ha [20:42:38] haha [20:42:45] awjr: still need someone to get it ? [20:42:53] LeslieCarr yes please [20:43:04] i'll pour you a scotch next time im in the office. [20:43:09] you do know we're having some varnish issues thoguh, right ? just warning you that it is higher risk than usual [20:43:16] awww, you know the way straight to my heart :) [20:43:22] what is going on with varnish? [20:43:33] hey if clearing those caches causes any issues, don't count on my help tonight [20:43:35] it's a bad idea [20:43:49] jebus. [20:43:56] memory leaks [20:44:10] crap sandwich [20:45:15] yeah [20:50:54] ok, i'll be back, cupcakes [20:51:24] <^demon> awjr: Do you need me any more? [20:51:41] ^demon nope i'm all set thanks for your help [20:52:01] <^demon> Okie dokie. It's almost 5 so I'm going to call it a day. Have a good weekend everyone. [20:52:12] thanks ^demon|away [20:52:27] (and have a great weekend!) [20:53:05] !log Rebooting cp3002 [20:53:07] Logged the message, Master [20:53:50] one box is now handling 20k requests/s hehe [20:56:32] back [21:05:57] LeslieCarr: did you say cupcakes? http://commons.wikimedia.org/wiki/File:WikiXDC_Cupcakes_b.jpg [21:06:59] cute!! [21:15:06] New patchset: Lcarr; "trying provider => shell" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4921 [21:19:07] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4921 [21:19:54] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4921 [21:19:56] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4921 [21:31:40] New patchset: Pyoungmeister; "add diederik to oxygen, remove arthur's sudo, per rt 2430" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4922 [21:36:36] New review: Pyoungmeister; "forcing verified" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4922 [21:36:38] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4922 [21:42:46] New patchset: Pyoungmeister; "cleaning up stuff that is no longer needed/was no longer in use after revamping udp2log configs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4923 [21:49:01] paravoid: so the puppetstoredconfigclean script doesn't clean exported variables maybe ? (my puppet knowledge has worn thin) [21:49:22] let me check [21:49:23] where is that? [21:50:00] /usr/sbin/puppetstoredconfigclean [21:50:02] in ruby [21:51:41] hrmmm, paravoid knows ruby? i wonder if he's been roped into penta hacking yet ;) [21:52:10] gerrit refuses to lint check for me :( [21:52:36] now, granted, it's 6pm on a friday, so I'm not heartbroken or anything [21:52:43] but still.... [21:52:46] think it's borked again [21:53:13] this might just be a feature ;) [21:53:15] I'm not sure [22:06:12] drdee: hey [22:06:28] i can work on getting the geodata db up for the next 20 minutes [22:16:27] drdee: fyi, you now have shell access to oxygen [22:27:29] PROBLEM - Puppet freshness on stat1 is CRITICAL: Puppet has not run in the last 10 hours [22:48:00] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [22:49:43] New patchset: Ryan Lane; "Up version of ldap scripts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4934 [22:51:54] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [22:51:54] PROBLEM - Puppet freshness on sq34 is CRITICAL: Puppet has not run in the last 10 hours [23:04:57] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [23:26:08] * AaronSchulz gets his twinview working [23:26:28] hrm, now why did the font get smaller? [23:56:00] AaronSchulz: would you come look at code for me?