[00:06:37] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: Puppet has not run in the last 10 hours [00:28:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:32:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 4.936 seconds [01:08:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:12:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 1.381 seconds [01:48:57] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:06] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.020 seconds [02:19:42] PROBLEM - Puppet freshness on brewster is CRITICAL: Puppet has not run in the last 10 hours [02:25:42] PROBLEM - Puppet freshness on db59 is CRITICAL: Puppet has not run in the last 10 hours [02:28:51] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:32:54] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.811 seconds [02:35:36] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [02:35:36] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [02:50:36] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [02:50:36] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [03:34:42] PROBLEM - Puppet freshness on search1001 is CRITICAL: Puppet has not run in the last 10 hours [04:53:07] PROBLEM - Disk space on search1022 is CRITICAL: DISK CRITICAL - free space: /a 2949 MB (2% inode=99%): [04:53:07] PROBLEM - Disk space on search1021 is CRITICAL: DISK CRITICAL - free space: /a 2962 MB (2% inode=99%): [05:33:22] PROBLEM - Puppet freshness on sq34 is CRITICAL: Puppet has not run in the last 10 hours [05:56:46] PROBLEM - Disk space on search1022 is CRITICAL: DISK CRITICAL - free space: /a 3319 MB (2% inode=99%): [06:32:13] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [07:06:49] PROBLEM - Puppet freshness on search1023 is CRITICAL: Puppet has not run in the last 10 hours [07:06:49] PROBLEM - Puppet freshness on search1013 is CRITICAL: Puppet has not run in the last 10 hours [07:09:49] PROBLEM - Puppet freshness on search1011 is CRITICAL: Puppet has not run in the last 10 hours [07:12:49] PROBLEM - Puppet freshness on search1010 is CRITICAL: Puppet has not run in the last 10 hours [07:14:46] PROBLEM - Puppet freshness on search1016 is CRITICAL: Puppet has not run in the last 10 hours [07:15:49] PROBLEM - Puppet freshness on search1002 is CRITICAL: Puppet has not run in the last 10 hours [07:18:49] PROBLEM - Puppet freshness on search1008 is CRITICAL: Puppet has not run in the last 10 hours [07:18:49] PROBLEM - Puppet freshness on search1004 is CRITICAL: Puppet has not run in the last 10 hours [07:18:49] PROBLEM - Puppet freshness on search1014 is CRITICAL: Puppet has not run in the last 10 hours [07:21:49] PROBLEM - Puppet freshness on search1003 is CRITICAL: Puppet has not run in the last 10 hours [07:21:49] PROBLEM - Puppet freshness on search1007 is CRITICAL: Puppet has not run in the last 10 hours [07:21:49] PROBLEM - Puppet freshness on search1009 is CRITICAL: Puppet has not run in the last 10 hours [07:21:49] PROBLEM - Puppet freshness on search1005 is CRITICAL: Puppet has not run in the last 10 hours [07:21:49] PROBLEM - Puppet freshness on search1024 is CRITICAL: Puppet has not run in the last 10 hours [07:21:50] PROBLEM - Puppet freshness on search1017 is CRITICAL: Puppet has not run in the last 10 hours [07:23:46] PROBLEM - Puppet freshness on search1012 is CRITICAL: Puppet has not run in the last 10 hours [07:25:52] PROBLEM - Puppet freshness on search1006 is CRITICAL: Puppet has not run in the last 10 hours [07:25:52] PROBLEM - Puppet freshness on search1019 is CRITICAL: Puppet has not run in the last 10 hours [07:25:52] PROBLEM - Puppet freshness on search1020 is CRITICAL: Puppet has not run in the last 10 hours [07:28:13] PROBLEM - Puppet freshness on search1021 is CRITICAL: Puppet has not run in the last 10 hours [07:28:13] PROBLEM - Puppet freshness on search1018 is CRITICAL: Puppet has not run in the last 10 hours [07:31:31] PROBLEM - Puppet freshness on search1015 is CRITICAL: Puppet has not run in the last 10 hours [07:31:31] PROBLEM - Puppet freshness on search1022 is CRITICAL: Puppet has not run in the last 10 hours [10:08:02] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: Puppet has not run in the last 10 hours [12:21:13] PROBLEM - Puppet freshness on brewster is CRITICAL: Puppet has not run in the last 10 hours [12:27:13] PROBLEM - Puppet freshness on db59 is CRITICAL: Puppet has not run in the last 10 hours [12:37:07] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [12:37:07] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [12:52:10] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [12:52:10] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [13:36:16] PROBLEM - Puppet freshness on search1001 is CRITICAL: Puppet has not run in the last 10 hours [14:04:10] New patchset: Pyoungmeister; "perhaps this be the proper way of throwing in this var?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4081 [14:04:24] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4081 [14:04:33] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4081 [14:04:36] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4081 [14:06:05] hashar ^demon someone. I'm merging in some crons/execs in svn.pp. is this ok? [14:06:21] <^demon> Diff? [14:06:23] looking [14:06:24] <^demon> Probably ok :) [14:06:32] notpeter: isn't it like 5am for you? [14:06:41] hello chad [14:06:46] hashar: 10am, so sysadmin 5am ;) [14:07:01] hashar: Chad and Peter are actually in the same timezone :) [14:07:01] hashar: I'm in US eastern standard time [14:07:12] ohh yet another east coaster [14:07:13] cool [14:07:26] well, michigan is hardly the east coast... but yeah ;) [14:07:34] * hashar hereby bless notpeter with the “merge stuff for hashar“ badge :-) [14:07:37] Yeah, notpeter is a TZ edge case [14:08:03] ok, I'm merging this stuff. I thought I laready merged this... [14:08:42] notpeter: which changes were you talking about anyway? [14:08:49] cause 4081 is unrelated to svn.pp [14:09:01] cron { [14:09:01] doc_generation: [14:09:01] - command => "(cd /home/mwdocs/phase3 && svn up && php maintenance/mwdocgen.php --all) >> /var/log/mwdocs.log 2>&1", [14:09:04] + command => "(cd /var/mwdocs/phase3 && git pull && php maintenance/mwdocgen.php --all) >> /var/log/mwdocs.log 2>&1", [14:09:07] + require => Exec["clone mediawiki for doc"], [14:09:14] that and another 4 lines [14:09:18] sure go ahead [14:09:29] cool [14:09:30] the only thing it can break is the doc generation [14:09:43] which is like very low priority and has no impact at all on production [14:09:58] <^demon> I thought we already merged that. [14:10:02] I suppose ^demon has already updated /var/mwdocs/phase3 to use git :-) [14:10:05] documentation is for the craven. [14:10:09] that's our motto, right? [14:10:19] <^demon> hashar: No I can't, because it's not group-writable. [14:10:19] yeah [14:10:20] ^demon: you know, me too [14:10:39] we use bug driven development [14:10:50] <^demon> The clone's not going to work since the directory already has contents. [14:10:58] THAT IS ALL BECAUSE OF root USER AGAIN [14:11:10] that guy keep causing troubles [14:11:25] <^demon> Actually this stuff is owned by mwdocs:svn. [14:11:33] <^demon> But it's only user-writable, not group writable. [14:11:36] <^demon> So I can't touch it :p [14:11:53] we can ask puppet to make sure it is mwdocs:svn and group writable [14:12:29] though I am not sure we want puppet to do permission check on a full tree [14:12:37] <^demon> No, that'd be silly :) [14:13:50] we need all of that to be rethought anyway mwdocs home should be /var/lib/mwdocs [14:15:06] notpeter: can you possibly fix some permissions on formey please? [14:15:19] chown -Rv mwdocs:svn /var/mwdocs/phase3 && chmod -Rv g+w /var/mwdocs/phase3 [14:16:51] notpeter: or just drop the phase3 directory [14:17:02] the "clone mediawiki for doc" cron is supposed to create it for us [14:20:19] I can run that chown on formey [14:21:01] <^demon> Dropping the phase3 directory should cause it to re-clone when the cron is run. [14:21:10] <^demon> I'd say drop it, then run the clone/pull [14:21:32] chmod/chown done [14:22:51] notpeter: sorry you will have to rename the directory :-D [14:23:07] notpeter: the cronjob has a dependency that will clone the directory from git to /var/mwdocs/phase3 [14:23:41] reanme what to what? [14:23:43] <^demon> Does it mean it requires that dependency to execute before the pull is run? [14:23:55] mv /var/mwdocs/phase3 /var/mwdocs/phase3-svn [14:23:57] <^demon> Because I'm afraid the clone will exit with a non-0 and the pull won't happen. [14:25:12] New patchset: Hashar; "let svn group modify formey:/var/mwdocs/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4083 [14:25:13] so we will need to fix it [14:25:26] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4083 [14:25:30] change 4083 makes /var/mwdocs/ g+w [14:40:19] I have done the mv as requested [14:51:05] New patchset: Pyoungmeister; "ruby can be... frustrating..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4084 [14:51:19] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/4084 [14:53:02] New patchset: Pyoungmeister; "ruby can be... frustrating..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4084 [14:53:16] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4084 [14:53:29] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4084 [14:53:31] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4084 [14:55:54] New patchset: Pyoungmeister; "so... frustrating..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4085 [14:56:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4085 [14:56:16] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4085 [14:56:20] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4085 [14:57:28] I still have to fix the linter message for good [14:57:38] still looks cryptic [15:00:41] New patchset: Pyoungmeister; "RAWR" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4087 [15:00:55] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4087 [15:01:06] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4087 [15:01:08] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4087 [15:02:37] New patchset: Pyoungmeister; "sweet mother of..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4089 [15:02:52] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4089 [15:02:58] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4089 [15:03:01] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4089 [15:06:29] New patchset: Hashar; "make puppet parser validate monospaced" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4090 [15:06:43] New patchset: Hashar; "let svn group modify formey:/var/mwdocs/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4083 [15:06:58] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4090 [15:06:58] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4083 [15:34:39] PROBLEM - Puppet freshness on sq34 is CRITICAL: Puppet has not run in the last 10 hours [15:52:40] New patchset: ArielGlenn; "rsync project dump files with multiple threads" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/4096 [15:58:19] New patchset: Pyoungmeister; "now with more correct syntax thanks to mark!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4098 [15:58:33] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4098 [15:59:28] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4098 [15:59:31] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4098 [16:33:55] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [17:08:52] PROBLEM - Puppet freshness on search1013 is CRITICAL: Puppet has not run in the last 10 hours [17:08:52] PROBLEM - Puppet freshness on search1023 is CRITICAL: Puppet has not run in the last 10 hours [17:11:52] PROBLEM - Puppet freshness on search1011 is CRITICAL: Puppet has not run in the last 10 hours [17:14:52] PROBLEM - Puppet freshness on search1010 is CRITICAL: Puppet has not run in the last 10 hours [17:15:47] New patchset: Pyoungmeister; "typo" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4108 [17:16:02] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4108 [17:16:20] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4108 [17:16:22] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4108 [17:16:49] PROBLEM - Puppet freshness on search1016 is CRITICAL: Puppet has not run in the last 10 hours [17:17:52] PROBLEM - Puppet freshness on search1002 is CRITICAL: Puppet has not run in the last 10 hours [17:20:52] PROBLEM - Puppet freshness on search1004 is CRITICAL: Puppet has not run in the last 10 hours [17:20:52] PROBLEM - Puppet freshness on search1008 is CRITICAL: Puppet has not run in the last 10 hours [17:20:52] PROBLEM - Puppet freshness on search1014 is CRITICAL: Puppet has not run in the last 10 hours [17:23:52] PROBLEM - Puppet freshness on search1003 is CRITICAL: Puppet has not run in the last 10 hours [17:23:52] PROBLEM - Puppet freshness on search1007 is CRITICAL: Puppet has not run in the last 10 hours [17:23:52] PROBLEM - Puppet freshness on search1005 is CRITICAL: Puppet has not run in the last 10 hours [17:23:52] PROBLEM - Puppet freshness on search1017 is CRITICAL: Puppet has not run in the last 10 hours [17:23:52] PROBLEM - Puppet freshness on search1009 is CRITICAL: Puppet has not run in the last 10 hours [17:23:53] PROBLEM - Puppet freshness on search1024 is CRITICAL: Puppet has not run in the last 10 hours [17:25:49] PROBLEM - Puppet freshness on search1012 is CRITICAL: Puppet has not run in the last 10 hours [17:27:46] PROBLEM - Puppet freshness on search1020 is CRITICAL: Puppet has not run in the last 10 hours [17:27:46] PROBLEM - Puppet freshness on search1006 is CRITICAL: Puppet has not run in the last 10 hours [17:27:46] PROBLEM - Puppet freshness on search1019 is CRITICAL: Puppet has not run in the last 10 hours [17:29:52] PROBLEM - Puppet freshness on search1021 is CRITICAL: Puppet has not run in the last 10 hours [17:29:52] PROBLEM - Puppet freshness on search1018 is CRITICAL: Puppet has not run in the last 10 hours [17:32:52] PROBLEM - Puppet freshness on search1022 is CRITICAL: Puppet has not run in the last 10 hours [17:32:52] PROBLEM - Puppet freshness on search1015 is CRITICAL: Puppet has not run in the last 10 hours [18:17:54] !log removed old bin files on db1004 and prolly borked it by removing the wrong files [18:17:56] Logged the message, RobH [18:26:24] New patchset: Pyoungmeister; "trailing commas?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4116 [18:26:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4116 [18:26:55] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4116 [18:26:58] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4116 [18:30:16] !log brewster puppet daemon stopped, doing local hacks [18:30:18] Logged the message, RobH [18:43:48] New patchset: Pyoungmeister; "read more, different approach" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4117 [18:44:02] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4117 [18:45:32] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4117 [18:45:35] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4117 [18:48:50] New patchset: Pyoungmeister; "need local var" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4118 [18:49:04] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4118 [18:49:16] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4118 [18:49:18] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4118 [18:49:35] New patchset: Lcarr; "reenabling contact groups" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4119 [18:49:50] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4119 [18:50:18] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4119 [18:50:21] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4119 [18:50:45] RECOVERY - Puppet freshness on search1001 is OK: puppet ran at Mon Apr 2 18:50:13 UTC 2012 [18:52:15] RECOVERY - Puppet freshness on search1011 is OK: puppet ran at Mon Apr 2 18:51:55 UTC 2012 [18:54:12] RECOVERY - Puppet freshness on search1007 is OK: puppet ran at Mon Apr 2 18:53:44 UTC 2012 [18:54:12] RECOVERY - Puppet freshness on search1018 is OK: puppet ran at Mon Apr 2 18:53:55 UTC 2012 [18:57:13] RECOVERY - Puppet freshness on search1004 is OK: puppet ran at Mon Apr 2 18:56:43 UTC 2012 [18:57:13] RECOVERY - Puppet freshness on search1016 is OK: puppet ran at Mon Apr 2 18:57:06 UTC 2012 [18:57:48] RECOVERY - Puppet freshness on search1006 is OK: puppet ran at Mon Apr 2 18:57:37 UTC 2012 [18:58:42] RECOVERY - Puppet freshness on search1009 is OK: puppet ran at Mon Apr 2 18:58:26 UTC 2012 [19:01:03] cmjohnson1 / RobH where is professor ? racktables search came up empty ? [19:02:18] RECOVERY - Puppet freshness on search1021 is OK: puppet ran at Mon Apr 2 19:02:01 UTC 2012 [19:02:18] RECOVERY - Puppet freshness on search1002 is OK: puppet ran at Mon Apr 2 19:02:06 UTC 2012 [19:02:45] RECOVERY - Puppet freshness on search1008 is OK: puppet ran at Mon Apr 2 19:02:22 UTC 2012 [19:02:45] RECOVERY - Puppet freshness on search1019 is OK: puppet ran at Mon Apr 2 19:02:40 UTC 2012 [19:02:47] i dunno, i iasked that once, someone renamed a server and didnt tell me [19:02:48] lesliecarr: never heard of it [19:02:49] so no clue [19:02:58] whoever did it needs to have their ass kicked. [19:03:02] that's asher's graphite server [19:03:15] so he just renamed a server without telling the folks who keep inventory. [19:03:17] =/ [19:03:28] so it would seem [19:03:32] notpeter: any idea what it used to be? [19:04:01] sigh [19:04:07] renaming db20 to professor -asher [19:04:11] svn log sez [19:04:11] thanks notpeter [19:04:19] didn't actually look at the diff [19:04:25] grrrrrrrrrr good thing asher's not sitting here [19:04:29] heh [19:04:42] RECOVERY - Puppet freshness on search1003 is OK: puppet ran at Mon Apr 2 19:04:32 UTC 2012 [19:04:42] RECOVERY - Puppet freshness on search1017 is OK: puppet ran at Mon Apr 2 19:04:38 UTC 2012 [19:04:44] when's he supposed to be back? [19:04:44] LeslieCarr: when he gets in tomorrow kick him in the shins for me and demand to know what the server is really named ;] [19:04:47] tomorrow [19:04:54] he is flying now i think. [19:04:54] hehe [19:05:09] ah [19:05:18] RECOVERY - Puppet freshness on search1024 is OK: puppet ran at Mon Apr 2 19:04:46 UTC 2012 [19:06:13] RECOVERY - Puppet freshness on search1013 is OK: puppet ran at Mon Apr 2 19:05:42 UTC 2012 [19:07:26] ok, going to try to check mac address vs where db20 looks like it should be vs what professor is [19:08:09] yep, that's it [19:08:12] grrr asher [19:08:26] i'll get the port names and racktables - cmjohnson1 do you want to rename the machien in any other locations ? [19:08:56] sure! [19:11:45] RECOVERY - Puppet freshness on search1020 is OK: puppet ran at Mon Apr 2 19:11:37 UTC 2012 [19:13:57] RECOVERY - Puppet freshness on search1005 is OK: puppet ran at Mon Apr 2 19:13:45 UTC 2012 [19:15:01] RECOVERY - Puppet freshness on search1012 is OK: puppet ran at Mon Apr 2 19:14:43 UTC 2012 [19:19:12] RECOVERY - Puppet freshness on search1023 is OK: puppet ran at Mon Apr 2 19:18:57 UTC 2012 [19:19:21] RECOVERY - Puppet freshness on search1014 is OK: puppet ran at Mon Apr 2 19:19:08 UTC 2012 [19:19:31] RECOVERY - Puppet freshness on search1015 is OK: puppet ran at Mon Apr 2 19:19:23 UTC 2012 [19:19:33] if db20 is renaming [19:19:39] it need to have a new label, and be updated in racktables [19:19:43] so someone needs to do that. [19:19:51] someone also needs to kick asher in the ass for me [19:19:57] RECOVERY - Puppet freshness on search1010 is OK: puppet ran at Mon Apr 2 19:19:31 UTC 2012 [19:19:57] RECOVERY - Puppet freshness on search1022 is OK: puppet ran at Mon Apr 2 19:19:40 UTC 2012 [19:19:57] updated racktables [19:20:01] renaming servers and not telling me is not ok. [19:20:06] can you put in a ticket for the ass kicking ? [19:22:51] to be clear...db20 is not professor? [19:22:58] db20 is professor [19:23:01] now professor [19:23:01] is now professor [19:23:04] hehehe [19:23:05] :) [19:23:18] k [19:33:02] New patchset: Diederik; "Deploy Wikipedia Zero and Teahouse filters, upgrade package to 0.2.0" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4120 [19:33:17] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4120 [19:42:14] * schoolcraftT slaps Thehelpfulone with a large smelly trout [19:42:52] New patchset: Diederik; "Deploy Wikipedia Zero and Teahouse filters, upgrade package to 0.2.0 Uninstall old package Change-Id: Ie4951a567b5bb9d7b5c06c3dc17f217f1caa59ba" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4120 [19:43:06] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4120 [19:44:31] * schoolcraftT slaps Laaknor with a large smelly trout [19:47:17] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4120 [19:47:19] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4120 [19:51:19] !log running authdns-update on dobson [19:51:21] Logged the message, notpeter [19:51:57] !log deploying new udp-filter to emery rt-2501 gerrit/r4120 [19:51:59] Logged the message, Master [19:55:57] PROBLEM - udp2log processes on emery is CRITICAL: CRITICAL: filters absent: /usr/bin/udp-filter, /usr/bin/udp-filter, /usr/bin/udp-filter, [19:59:28] New patchset: Lcarr; "this broke something Revert "Deploy Wikipedia Zero and Teahouse filters, upgrade package to 0.2.0 Uninstall old package Change-Id: Ie4951a567b5bb9d7b5c06c3dc17f217f1caa59ba"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4123 [19:59:42] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4123 [20:09:36] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: Puppet has not run in the last 10 hours [20:14:26] hey folks [20:14:33] got a quick git issue [20:14:51] just did git commit --amend -a [20:14:54] then git-review [20:15:20] and then this happens: Errors running git rebase -i remotes/gerrit/production [20:15:29] cannot cherry-pick blablabla [20:15:42] what is the magic word? [20:18:09] <^demon> Use `git review -R` when submitting amended changes. [20:19:29] nope, but i think i understand the problem [20:19:42] the original commit has been closed [20:19:59] so i assume i have to commit this as a new changeset? [20:21:30] <^demon> Yep, won't be able to add a new patchset if the old one's already submitted. [20:21:53] but the old one was merged and then reverted [20:23:00] <^demon> Sounds like the best thing to do is submit a new patchset there. [20:33:08] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: -1; - https://gerrit.wikimedia.org/r/4020 [20:34:12] PROBLEM - BGP status on cr2-pmtpa is CRITICAL: CRITICAL: No response from remote host 208.80.152.197, [20:34:39] PROBLEM - Disk space on srv258 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:34:39] PROBLEM - Disk space on mw49 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:34:42] ^demon: I did a hard reset, made my changes from scratch but still getting rebase merge errors [20:34:48] PROBLEM - Disk space on locke is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:34:55] ^demon: do you have any suggestions? [20:34:57] <^demon> dvanliere: Nothing should be rebasing if you're using `git review -R` [20:35:15] PROBLEM - Disk space on mw44 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:35:24] PROBLEM - Disk space on mw33 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:35:24] PROBLEM - Disk space on mw4 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:35:24] New patchset: Diederik; "Added Wikipedia Zero and Teahouse filters, upgraded package to 0.2" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4127 [20:35:37] thx [20:35:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4127 [20:35:42] PROBLEM - Disk space on mw12 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:35:51] PROBLEM - Disk space on srv276 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=61%): /var/lib/ureadahead/debugfs 0 MB (0% inode=61%): [20:36:00] PROBLEM - RAID on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:36:09] PROBLEM - Disk space on mw24 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:36:09] PROBLEM - Disk space on mw25 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:36:09] PROBLEM - Disk space on mw46 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:36:18] PROBLEM - Disk space on mw51 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:36:27] PROBLEM - Disk space on mw41 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:36:27] PROBLEM - Disk space on mw13 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:36:27] PROBLEM - Disk space on srv285 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=56%): /var/lib/ureadahead/debugfs 0 MB (0% inode=56%): [20:36:27] PROBLEM - Disk space on mw31 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:36:27] PROBLEM - Disk space on mw3 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:36:36] PROBLEM - Disk space on srv246 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:36:36] PROBLEM - Disk space on mw27 is CRITICAL: DISK CRITICAL - free space: / 55 MB (0% inode=63%): /var/lib/ureadahead/debugfs 55 MB (0% inode=63%): [20:36:40] ewwwwwwwwwwwww [20:36:45] PROBLEM - Disk space on srv264 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:36:45] PROBLEM - Disk space on srv286 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=61%): /var/lib/ureadahead/debugfs 0 MB (0% inode=61%): [20:36:45] RECOVERY - Disk space on locke is OK: DISK OK [20:36:45] PROBLEM - Disk space on srv259 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:36:54] PROBLEM - Disk space on mw38 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:36:54] PROBLEM - Disk space on mw48 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:37:12] PROBLEM - Disk space on mw5 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:37:12] oh nooo [20:37:30] PROBLEM - Disk space on srv260 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:37:30] PROBLEM - Disk space on srv282 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=61%): /var/lib/ureadahead/debugfs 0 MB (0% inode=61%): [20:37:40] Change abandoned: Lcarr; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4123 [20:37:57] PROBLEM - Disk space on mw21 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:37:57] PROBLEM - Disk space on srv232 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:37:57] PROBLEM - Disk space on srv208 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:38:06] PROBLEM - Disk space on srv226 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:38:06] PROBLEM - Disk space on srv227 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:38:06] PROBLEM - Disk space on srv270 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:38:06] RECOVERY - RAID on hume is OK: OK: State is Optimal, checked 6 logical device(s) [20:38:15] PROBLEM - Disk space on mw35 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:38:15] RECOVERY - BGP status on cr2-pmtpa is OK: OK: host 208.80.152.197, sessions up: 7, down: 0, shutdown: 0 [20:38:42] PROBLEM - Disk space on srv229 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:38:51] PROBLEM - Disk space on mw37 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:39:00] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:39:09] PROBLEM - Disk space on srv237 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=63%): /var/lib/ureadahead/debugfs 0 MB (0% inode=63%): [20:39:19] <^demon> :\ [20:39:27] PROBLEM - Disk space on srv225 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=64%): /var/lib/ureadahead/debugfs 0 MB (0% inode=64%): [20:40:57] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.026 seconds [20:44:27] New patchset: Diederik; "Add sampling rate" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4128 [20:44:41] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4128 [20:45:11] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4128 [20:45:14] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4128 [20:45:46] New patchset: Demon; "Create hook for l10n-autoreview in gerrit." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4020 [20:46:00] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4020 [20:52:29] PROBLEM - udp2log log age on emery is CRITICAL: CRITICAL: log files /var/log/squid/teahouse.log, /var/log/squid/orange-ivory-coast.log, /var/log/squid/digi-malaysia.log, have not been written to in 6 hours [20:54:23] RECOVERY - udp2log log age on emery is OK: OK: all log files active [20:54:41] RECOVERY - Squid on brewster is OK: TCP OK - 0.002 second response time on port 8080 [21:07:28] New patchset: Pyoungmeister; "adding some lvs IPs to the lvs servers as well..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4131 [21:07:42] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4131 [21:09:59] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4131 [21:10:02] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4131 [21:14:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.024 seconds [21:24:55] hi leslie, maplebed: so the problem seems to be with the debian package [21:24:58] -bash: /usr/bin/udp-filter: cannot execute binary file [21:25:06] or is it a permission thing? [21:25:12] also, logrotate doesn't appear to be working [21:25:24] it has a+x [21:25:41] (i am trying to run udp-filter myself to see what's going on but i cannot even start it) [21:26:13] file udp-filter gives: [21:26:14] udp-filter: Mach-O 64-bit executable [21:26:24] did you try with pipe before it ? [21:26:34] no [21:27:19] pipe: command not found ???? [21:29:27] hi. [21:29:55] sorry to hear it's not working. [21:30:09] i can only blame me :) [21:30:17] !log turned down ms7's secondary ethernet port to prevent the flapping (stupid sun boxes) [21:30:20] Logged the message, Mistress of the network gear. [21:30:26] dvanliere: where did you build the package? [21:30:36] on a ubuntu 64 10.10 machine [21:30:56] same machine as the previous version [21:31:29] Mach-O sounds wrong. [21:31:41] that's the mac architecture. [21:32:14] mmmmmmmm [21:32:17] that's very strange [21:32:31] New patchset: Pyoungmeister; "a bit much on the quotes front." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4134 [21:32:37] in the debian control fie [21:32:44] i think we're 10.04 not 10.10 ? [21:32:45] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4134 [21:32:46] how do you specify ubuntu 64 architecture? [21:33:00] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/4134 [21:33:03] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4134 [21:33:46] Architecture: x86_64-linux-gnu? [21:33:48] /usr/bin/udp_filter: ELF 64-bit LSB executable, <--- that's what you want to see. [21:33:58] i think i got it [21:34:19] New review: Hashar; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/4020 [21:35:39] it's probably best to revert the changes for now [21:35:44] I treat it as a place ok. [21:35:48] i have to recreate the debian package [21:35:49] whoops. [21:35:49] ok. [21:35:51] I'll revert. [21:36:14] New patchset: Bhartshorne; "Revert "Deploy Wikipedia Zero and Teahouse filters, upgrade package to 0.2.0 Uninstall old package Change-Id: Ie4951a567b5bb9d7b5c06c3dc17f217f1caa59ba"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4135 [21:36:28] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4135 [21:36:45] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4135 [21:37:17] dvanliere: did you make any changes to emery that my revert and puppet will wipe out? [21:37:42] no: didn't touch emery [21:38:59] https://gerrit.wikimedia.org/r/#change,4128 [21:39:51] New patchset: Bhartshorne; "Revert "Add sampling rate"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4136 [21:40:06] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/4136 [21:40:14] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4136 [21:40:17] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4136 [21:40:27] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/4135 [21:40:28] maplebed: how do you specify linux 64 amd architecture in a debian control file [21:40:29] ? [21:40:30] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/4135 [21:40:44] PROBLEM - LVS Lucene on search-prefix.svc.eqiad.wmnet is CRITICAL: Connection refused [21:41:44] dvanliere: if you're building it in the right place you sholudn't have to do anything. [21:41:57]