[00:05:35] Could I get someone to make a change to the mediawiki svn repo post commit hook in puppet, and push the change out please? [00:05:49] Reedy: (continuing conversation from #mediawiki) ...the ops folks here: Ryan_Lane, LeslieCarr [00:05:57] "--smtp smtp.pmtpa.wikimedia.org" needs to become "--smtp smtp.pmtpa.wmnet" [00:06:16] LeslieCarr: were you involved in the mailman migration? [00:06:48] AFAIK m ark did it all [00:08:12] Reedy: do you not have a labs account? [00:08:19] I do [00:08:26] it still needs merging to production [00:08:29] sure [00:08:31] putting out to puppet [00:08:36] and I haven't got a working git setup [00:08:47] so in theory, getting Ryan to do it would be a lot quicker [00:08:48] i need to get my local gerrit post commit set up [00:09:04] sure [00:09:26] Hmm [00:10:13] That's somewhat concerning [00:10:22] the file in puppet has "--smtp lily.knams.wikimedia.org \" [00:10:50] So doesn't even match what's in production... [00:12:45] robla: was not [00:13:15] so what file do you need me to fix? :) [00:14:05] Ideally /svnroot/mediawiki/hooks/post-commit on formey [00:14:05] but it's puppet managed.. [00:14:05] puppet/files/svn/hooks/post-commit [00:14:13] And the file in production vs the one on formey don't match [00:14:53] okay [00:14:54] hah [00:14:57] svn.pp looks sane [00:15:02] so are you sure it's puppet managed ? :) [00:15:09] well, chad changed it yesterday [00:15:12] now it's changed back [00:15:20] so *something* is at least changing... [00:15:35] okay, lemme check it out [00:15:48] aren't i not supposed to do normal work during the hackathon? ;) [00:16:49] Hah! "The site is down!!!!" "Yeah, and? I'm not allowed to do normal work!" :p [00:17:42] exactly! [00:17:52] =D [00:23:26] just switched "--smtp smtp.pmtpa.wikimedia.org" needs to become "--smtp smtp.pmtpa.wmnet" in the file [00:23:48] New patchset: Lcarr; "correcting smtp server for formey post-commit hook" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2005 [00:24:05] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2005 [00:24:22] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2005 [00:24:23] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2005 [00:26:58] Reedy: robla it's fixed [00:27:02] fixed in puppet [00:27:18] excellent, thanks! [00:27:20] Great, thankyou :) [00:27:21] siebrand, ^ [00:27:24] and updated [00:28:22] I'll go grab a drink then see about generating the rest of the missing emails [01:11:32] New patchset: pugmajere; "Rework the JunOS firewall creation tools." [operations/software] (master) - https://gerrit.wikimedia.org/r/2009 [01:11:33] New review: gerrit2; "Lint check passed." [operations/software] (master); V: 1 - https://gerrit.wikimedia.org/r/2009 [01:16:57] New patchset: pugmajere; "Clean up some bad whitespace unintentionally introduced in 15e696f" [operations/software] (master) - https://gerrit.wikimedia.org/r/2010 [01:19:34] New review: Lcarr; "whitespaces fixed in change-id I95f19aa6" [operations/software] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2009 [01:19:34] Change merged: Lcarr; [operations/software] (master) - https://gerrit.wikimedia.org/r/2009 [01:19:44] New review: Lcarr; "(no comment)" [operations/software] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2010 [01:19:44] Change merged: Lcarr; [operations/software] (master) - https://gerrit.wikimedia.org/r/2010 [02:21:50] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1630s [02:25:40] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1861s [02:35:46] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:35:56] PROBLEM - Puppet freshness on knsq9 is CRITICAL: Puppet has not run in the last 10 hours [02:41:56] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [04:19:03] RECOVERY - Disk space on es1004 is OK: DISK OK [04:24:53] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:36:43] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [06:24:18] PROBLEM - Router interfaces on cr1-eqiad is CRITICAL: CRITICAL: host 208.80.154.196, interfaces up: 86, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-5/2/1: down - Core: cr1-sdtpa:xe-0/0/1 (Level3/FPL, CV71026) {#2008} [10Gbps wave]BR [06:28:08] PROBLEM - Router interfaces on cr1-sdtpa is CRITICAL: CRITICAL: host 208.80.152.196, interfaces up: 76, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-0/0/1: down - Core: cr1-eqiad:xe-5/2/1 (FPL/GBLX, CV71026) [10Gbps wave]BR [07:00:54] New patchset: Catrope; "Adding .gitreview file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2014 [07:01:13] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2014 [10:05:35] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 389336 MB (3% inode=99%): [10:06:55] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 385133 MB (3% inode=99%): [10:15:45] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [10:25:45] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.031 second response time [10:51:05] RECOVERY - MySQL slave status on es1004 is OK: OK: [12:45:22] PROBLEM - Puppet freshness on knsq9 is CRITICAL: Puppet has not run in the last 10 hours [12:53:27] PROBLEM - Puppet freshness on cp1044 is CRITICAL: Puppet has not run in the last 10 hours [13:25:47] RECOVERY - Router interfaces on cr1-sdtpa is OK: OK: host 208.80.152.196, interfaces up: 78, down: 0, dormant: 0, excluded: 0, unused: 0 [13:25:56] huh [13:25:59] ? [13:26:20] oh well, must run errands [13:28:17] RECOVERY - Router interfaces on cr1-eqiad is OK: OK: host 208.80.154.196, interfaces up: 88, down: 0, dormant: 0, excluded: 0, unused: 0 [17:09:37] PROBLEM - Puppet freshness on ssl4 is CRITICAL: Puppet has not run in the last 10 hours [18:10:37] PROBLEM - Router interfaces on cr1-eqiad is CRITICAL: CRITICAL: host 208.80.154.196, interfaces up: 86, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-5/2/1: down - Core: cr1-sdtpa:xe-0/0/1 (Level3/FPL, CV71026) {#2008} [10Gbps wave]BR [18:20:37] RECOVERY - Router interfaces on cr1-eqiad is OK: OK: host 208.80.154.196, interfaces up: 88, down: 0, dormant: 0, excluded: 0, unused: 0 [18:39:31] New patchset: Mark Bergsma; "Explicitly pin the Wikimedia version of php-wikidiff2 to prevent problems in the future" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2016 [18:41:00] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2016 [18:41:01] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2016 [18:56:22] New patchset: Catrope; "Adding .gitreview file" [labs/private] (master) - https://gerrit.wikimedia.org/r/2017 [19:23:01] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2014 [19:23:01] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2014 [19:23:10] New review: Ryan Lane; "(no comment)" [labs/private] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2017 [19:23:11] Change merged: Ryan Lane; [labs/private] (master) - https://gerrit.wikimedia.org/r/2017 [19:24:15] New patchset: Catrope; "Add .gitreview file" [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/2018 [19:24:20] Ryan_Lane: --^^ [19:24:39] New review: Ryan Lane; "(no comment)" [integration/jenkins] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2018 [19:24:40] Change merged: Ryan Lane; [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/2018 [19:24:45] RoanKattouw: ^^ :D [20:12:34] New patchset: Domas; "PEP8 sans line lengths, fixed the 'no database' condition to be somewhat nicer" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2021 [20:12:51] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2021 [20:14:20] welcome PugMajere [20:14:53] New patchset: Domas; "uh oh, tabs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2022 [20:16:57] New review: Domas; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2021 [20:16:57] Change merged: Domas; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2021 [20:17:13] New review: Domas; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2022 [20:17:13] Change merged: Domas; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2022 [22:19:17] i'm getting intermittent 503 errors for en.m.wikipedia.org [22:28:31] !log Restarted varnish backend on cp1041 and cp1042 [22:28:33] Logged the message, Master [22:30:34] New patchset: pugmajere; "Upate firewall creation tools to support protocol." [operations/software] (master) - https://gerrit.wikimedia.org/r/2032 [22:30:35] New review: gerrit2; "Lint check passed." [operations/software] (master); V: 1 - https://gerrit.wikimedia.org/r/2032 [22:33:56] New patchset: pugmajere; "Upate firewall creation tools to support protocol." [operations/software] (master) - https://gerrit.wikimedia.org/r/2032 [22:33:57] New review: gerrit2; "Lint check passed." [operations/software] (master); V: 1 - https://gerrit.wikimedia.org/r/2032 [22:35:24] New review: Lcarr; "Looks better than good, looks awesome!!!" [operations/software] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2032 [22:35:24] Change merged: Lcarr; [operations/software] (master) - https://gerrit.wikimedia.org/r/2032 [22:41:18] !log cp1042 stuck on disk i/o, rebooting [22:41:19] Logged the message, Master [22:53:12] New patchset: Domas; "Add atop to all servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2034 [22:53:28] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2034 [22:54:17] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2034 [22:54:18] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2034 [22:54:32] PROBLEM - Puppet freshness on knsq9 is CRITICAL: Puppet has not run in the last 10 hours [23:02:42] PROBLEM - Puppet freshness on cp1044 is CRITICAL: Puppet has not run in the last 10 hours [23:07:33] New patchset: Domas; "use all servers in srv250-257 range for memcached" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2035 [23:07:49] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2035 [23:08:04] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2035 [23:08:04] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2035 [23:11:22] PROBLEM - mobile traffic loggers on cp1042 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [23:33:54] New patchset: Lcarr; "Add a script to run a local parse validate of the puppet configs." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2040 [23:34:11] New patchset: Lcarr; "Hide warnings about storeconfigs when running a local lint." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2041 [23:34:26] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2040 [23:34:26] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2040 [23:34:26] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2040 [23:34:27] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2040 [23:34:29] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2041 [23:34:29] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2041 [23:35:26] hey who did the "- if $hostname =~ /^srv25[0-3]$/ { [23:35:26] - include memcached [23:35:26] - } change ? [23:35:32] is it good to merge ? [23:35:33] domas [23:35:35] yes [23:35:51] he's trying to piggyback on you huh [23:36:09] RECOVERY - mobile traffic loggers on cp1042 is OK: PROCS OK: 2 processes with command name varnishncsa [23:36:17] domas: always trying to steal my swagger [23:36:30] merging it [23:39:27] New patchset: Catrope; "ROCK!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2042 [23:39:42] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2042 [23:41:19] Change abandoned: Catrope; "Demo change" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2042 [23:47:22] New patchset: Ryan Lane; "Making https servers use the new encoded user agent" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2043 [23:47:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2043 [23:47:44] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2043 [23:47:45] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2043 [23:48:59] RECOVERY - Puppet freshness on ssl4 is OK: puppet ran at Sun Jan 22 23:48:58 UTC 2012