[00:01:21] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [00:08:31] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [00:13:31] PROBLEM - DPKG on helium is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:14:32] New patchset: Andrew Bogott; "Use instance-proxy for the default hostname." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57443 [00:17:31] RECOVERY - DPKG on helium is OK: All packages OK [00:18:00] lesliecarr: rt4870 is for rdb1/2 [00:22:01] PROBLEM - Host mw27 is DOWN: PING CRITICAL - Packet loss = 100% [00:23:41] RECOVERY - Host mw27 is UP: PING OK - Packet loss = 0%, RTA = 26.55 ms [00:26:21] PROBLEM - Apache HTTP on mw27 is CRITICAL: Connection refused [00:29:58] New patchset: coren; "Add ssh_restrict_network variable to SSH" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57447 [00:33:18] LeslieCarr: you got your dns issue fixed i hope? [00:33:31] if not you can invalidate cached entries with rec_control wipe-cache fqdn [00:35:21] RECOVERY - Apache HTTP on mw27 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.141 second response time [00:35:48] robh: last check leslie was in the middle of install and all was good [00:47:55] cool [00:52:03] New patchset: coren; "Add ssh_restrict_network variable to SSH" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57447 [01:05:00] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [01:08:43] New review: coren; "Tested and works as intended." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/57447 [01:56:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:57:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [02:04:30] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [02:13:31] !log LocalisationUpdate completed (1.22wmf1) at Thu Apr 4 02:13:31 UTC 2013 [02:13:40] Logged the message, Master [02:21:01] !log LocalisationUpdate completed (1.21wmf12) at Thu Apr 4 02:21:01 UTC 2013 [02:21:09] New review: Lcarr; "This will put this on every machine - I don't think that's a good idea" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57447 [02:21:09] Logged the message, Master [02:45:45] New review: coren; "It actually is a no-op unless you set the appropriate puppet variable since it's guarded in a condit..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57447 [03:08:59] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [03:12:42] New review: Ryan Lane; "Please check with puppetmaster::self to ensure this won't break every system." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57447 [03:26:09] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [03:56:05] !log upgrading opendj on virt1000 [03:56:18] Logged the message, Master [03:59:06] PROBLEM - Host mw1085 is DOWN: PING CRITICAL - Packet loss = 100% [04:01:06] RECOVERY - Host mw1085 is UP: PING OK - Packet loss = 0%, RTA = 1.35 ms [04:05:53] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:08:03] PROBLEM - LDAP on virt1000 is CRITICAL: Connection refused [04:08:13] PROBLEM - LDAPS on virt1000 is CRITICAL: Connection refused [04:09:03] RECOVERY - LDAP on virt1000 is OK: TCP OK - 0.000 second response time on port 389 [04:09:13] RECOVERY - LDAPS on virt1000 is OK: TCP OK - 0.000 second response time on port 636 [04:09:23] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 04:09:14 UTC 2013 [04:09:53] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:11:13] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 04:11:03 UTC 2013 [04:11:53] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:12:53] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 04:12:48 UTC 2013 [04:13:53] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:14:33] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 04:14:26 UTC 2013 [04:14:53] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:16:03] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [04:16:03] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 04:15:59 UTC 2013 [04:16:34] !log opendj upgrade on virt1000 complete [04:16:42] Logged the message, Master [04:16:53] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:17:33] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 04:17:25 UTC 2013 [04:17:53] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:18:53] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 04:18:46 UTC 2013 [04:18:53] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:20:03] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 04:20:01 UTC 2013 [04:20:53] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:21:13] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 04:21:10 UTC 2013 [04:21:53] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:22:23] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 04:22:12 UTC 2013 [04:22:53] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:23:13] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 04:23:07 UTC 2013 [04:23:53] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:24:03] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 04:23:54 UTC 2013 [04:24:53] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:25:13] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 04:25:04 UTC 2013 [04:25:53] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:30:03] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [04:32:53] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 04:32:46 UTC 2013 [04:32:53] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [04:43:57] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [04:59:57] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [05:04:25] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [05:38:10] PROBLEM - Squid on brewster is CRITICAL: Connection refused [05:42:51] !log clearing squid logs and restarting squid on brewster [05:42:58] Logged the message, Master [05:43:36] RECOVERY - Squid on brewster is OK: TCP OK - 0.027 second response time on port 8080 [05:57:42] New patchset: Jeremyb; "fix comment on udp2log sample rate" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57471 [05:57:54] Ryan_Lane: filled up again? [05:57:58] :( [06:00:19] yes [06:03:26] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [06:04:46] PROBLEM - Varnish HTCP daemon on cp1041 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:04:46] PROBLEM - Varnish HTTP mobile-backend on cp1041 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:04:46] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:16:11] New review: Stefan.petrea; "Looks good" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/57471 [06:22:36] RECOVERY - Varnish HTCP daemon on cp1041 is OK: PROCS OK: 1 process with UID = 997 (varnishhtcpd), args varnishhtcpd worker [06:22:36] RECOVERY - Varnish HTTP mobile-backend on cp1041 is OK: HTTP OK: HTTP/1.1 200 OK - 634 bytes in 6.417 second response time [06:22:46] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [06:30:26] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 06:30:21 UTC 2013 [06:31:26] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [06:32:56] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 06:32:49 UTC 2013 [06:33:26] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [07:06:38] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [07:44:58] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [07:44:58] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [07:44:58] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [07:54:28] New patchset: Rfaulk; "mod. use get_project_host_map method to generate map for project to host key." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56576 [08:04:59] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:08:19] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 08:08:10 UTC 2013 [08:08:59] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:09:19] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 08:09:16 UTC 2013 [08:09:59] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [08:20:32] https://doc.wikimedia.org/mediawiki-core/master/php/html/ is gone [08:30:09] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [08:32:59] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 08:32:57 UTC 2013 [08:33:59] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [09:06:39] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [09:32:48] Krinkle: 11:20 < ori-l> https://doc.wikimedia.org/mediawiki-core/master/php/html/ is gone [09:33:09] paravoid: I already know, he told me in #wikimedia-tech [09:33:26] the job has been disabled for a while, so it wasn't being updated. [09:33:34] I'm trying to get it working again [09:33:46] In doing so I cleared the existing dir [09:34:05] (rsync cleared it when syncing) [09:34:15] I don't care that much, I just thought you should know :) [09:34:25] I know :) [10:05:43] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [10:10:29] New patchset: Dereckson; "(bug 43359) Enable WebFonts on Javanese projects" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/39578 [10:10:39] New review: Dereckson; "PS3: rebased" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/39578 [10:19:46] New patchset: Mark Bergsma; "Temporarily remove cp3003 from the cache pool" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57490 [10:20:46] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57490 [10:29:51] was mailman changed recently? [10:31:22] following the steps to subscribe to certain private ml [10:31:33] - The moderator doesn't get an email [10:31:53] - The request doesn't appear in the list of pending requests [10:32:07] (and the user isn't subscribed, which is correct) [10:33:33] more likely that someone changed privacy options for that list and all requests are rejected automatically [10:36:17] PROBLEM - Varnish HTTP upload-backend on cp3003 is CRITICAL: Connection refused [10:36:22] then that "someone" did so in global mailman preferences [10:36:38] in that case I would expect an email saying that it was automatically rejected [10:36:47] PROBLEM - Varnish HTTP upload-frontend on cp3003 is CRITICAL: Connection refused [10:37:18] RECOVERY - Varnish HTTP upload-backend on cp3003 is OK: HTTP OK: HTTP/1.1 200 OK - 634 bytes in 0.177 second response time [10:37:47] RECOVERY - Varnish HTTP upload-frontend on cp3003 is OK: HTTP OK: HTTP/1.1 200 OK - 675 bytes in 0.176 second response time [10:37:49] the only similar config option I see is ban_list) [10:37:49] chapin.thomasovnta@batterydream.com [10:38:02] that's the only banned email [10:40:27] trying to follow https://wikitech.wikimedia.org/wiki/Git-buildpackage to build a deb for libvpx-1.1.0+patch i have the repo ready but can not push to gerrit since create-project requires the right permissions [10:40:57] who can run ssh USER@gerrit.wikimedia.org -p 29418 'gerrit create-project' -d "Package libvpx" -n operations/debs/libvpx -o ops -p operations/debs [10:41:01] and allow me to push changes [10:43:47] j^, poke ^demon [10:44:23] New patchset: MaxSem; "Check mobile site's HTTP status" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57419 [10:44:37] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [10:52:27] PROBLEM - Host cp3003 is DOWN: PING CRITICAL - Packet loss = 100% [11:02:37] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [11:04:25] Any ideas why gerrit might complain about lack of Change-ID when it clearly /is/ included in a commit message? [11:04:32] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [11:04:50] odder, that's the commit message? [11:04:54] *what's the commit message? [11:06:33] odder, is the commit id in the middle of the message? [11:06:36] git log -1 shows Change-Id: I92f841efc6eb10c3c46990a27b5e599d1da8f411 in the last line [11:08:27] New review: Faidon; "Class name "builder" is far too generic. Most Debian tools call everything pbuilder even if it's usi..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/56382 [11:09:13] Platonides: http://tools.wikimedia.pl/~odder/showHEAD is what I get when I do git show HEAD [11:13:20] odder, looks good to me [11:16:24] It looks good to me too, but still – doesn't work :) [11:18:37] New patchset: Platonides; "(bug 46828) Add patroller and autopatrolled groups on viwiki Added patroller and autopatrolled user groups on viwiki, and modified wgAddGroups, wgRemoveGroups according to the bug." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57497 [11:19:01] that's your patchset, odder [11:19:05] New patchset: Nikerabbit; "(bug 46840) Update ttmservr solr schema" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57498 [11:19:07] :) [11:20:22] New review: Nikerabbit; "Needed for Special:SearchTranslations." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57498 [11:20:39] New patchset: Nikerabbit; "(bug 46840) Update TTMServer Solr schema" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57498 [11:28:32] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [11:32:32] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [11:37:31] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [11:40:22] New review: MaxSem; "It's a sync with the master schema at mediawiki/extensions/Translate which was reviewed by me and Or..." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/57498 [12:02:31] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [12:02:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.153 second response time [12:07:08] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:08:48] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 12:08:38 UTC 2013 [12:09:08] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:09:48] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 12:09:41 UTC 2013 [12:10:08] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:10:38] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 12:10:35 UTC 2013 [12:11:08] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:11:38] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 12:11:29 UTC 2013 [12:12:08] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:12:18] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 12:12:09 UTC 2013 [12:13:08] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:13:18] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 12:13:15 UTC 2013 [12:14:08] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [12:15:28] RECOVERY - search indices - check lucene status page on search1017 is OK: HTTP OK: HTTP/1.1 200 OK - 62829 bytes in 0.010 second response time [12:17:38] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [12:33:28] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 12:33:25 UTC 2013 [12:33:38] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [12:45:35] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [13:01:28] New patchset: Mark Bergsma; "Temporarily depool cp1035 for persistent debugging" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57502 [13:02:05] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57502 [13:06:32] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [13:26:20] <^demon> !log running jgit gc on all repos [13:26:27] Logged the message, Master [13:26:32] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [13:26:42] thanks for that demon ;) [13:26:51] <^demon> You're welcome. [13:26:51] the puppet repo was getting unbearably slow [13:27:18] i believe there are some large binaries in the history, perhaps at some point we should just remove that, even if it breaks people's clones [13:27:42] then again, that's painful too [13:27:43] maybe not [13:28:09] speaking of that [13:28:17] we talked a bit about making puppet a fast-forward only repo [13:28:27] then totally forgot about it [13:28:31] maybe we should do it now? [13:29:07] <^demon> Considering it'll automatically rebase for you before merging, the workflow ends up being pretty much identical. [13:29:16] <^demon> Either way, you have to resolve conflicts if they exist. [13:31:13] yeah that would be good [13:31:48] <^demon> https://gerrit.wikimedia.org/r/#/admin/projects/operations/puppet - easily changed, just swap the merge type. [13:32:17] done [13:32:32] i've done that for some of the operations/debs repos previously as well [13:32:42] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [13:35:53] <^demon> There's actually a new experimental merge type as well, just not exposed in the UI. Jgit now supports recursive merges :) [13:36:36] I think fast-forward is better [13:37:12] <^demon> Yeah, I prefer fast-forward too. [13:37:24] <^demon> But for something like core, where there's lots of potential points for conflict, it may be useful [13:38:14] and maybe at some point it might make sense to rewrite history to fix authors, make it more sequential, remove binary files etc [13:39:00] <^demon> I want to do that with mediawiki core, but people seemed to shy away from the idea when I brought it up. [13:39:08] <^demon> "Everybody has to re-clone? The horror!" [13:42:33] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [13:43:33] PROBLEM - Varnish HTTP upload-frontend on cp1035 is CRITICAL: Connection refused [13:43:43] PROBLEM - Varnish HTCP daemon on cp1035 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 997 (varnishhtcpd), args varnishhtcpd worker [13:44:03] PROBLEM - Varnish HTTP upload-backend on cp1035 is CRITICAL: Connection refused [13:45:00] that's me [13:45:16] New patchset: Ottomata; "Now syncing sampled-1000 logs from gadolinium instead of emery." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57504 [13:45:39] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57504 [13:49:17] New patchset: Jeremyb; "fix comment on udp2log sample rate" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57471 [13:49:44] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57471 [13:53:33] RECOVERY - Varnish HTTP upload-frontend on cp1035 is OK: HTTP OK: HTTP/1.1 200 OK - 675 bytes in 0.001 second response time [13:53:43] RECOVERY - Varnish HTCP daemon on cp1035 is OK: PROCS OK: 1 process with UID = 997 (varnishhtcpd), args varnishhtcpd worker [13:54:06] <^demon> Heh, you can see the small spike from repacking: http://ganglia.wikimedia.org/latest/?c=Miscellaneous%20eqiad&h=manganese.wikimedia.org&m=cpu_report&r=hour&s=descending&hc=4&mc=2 [13:57:15] New patchset: Demon; "Set up weekly jgit gc operations for all repositories" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57327 [14:00:07] New review: Demon; "PS2 tosses the (very noisy) output to /dev/null. This will run every Saturday at 2am. Should take ~5..." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/57327 [14:00:39] New patchset: Mark Bergsma; "Revert "Temporarily depool cp1035 for persistent debugging"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57505 [14:01:04] New patchset: Odder; "(bug 46882) Namespace 100 to be searched by default on sewikimedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57506 [14:01:10] New patchset: Mark Bergsma; "Revert "Temporarily depool cp1035 for persistent debugging"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57505 [14:01:21] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57505 [14:03:33] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [14:05:05] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [14:11:44] New patchset: Ottomata; "Installing webstatscollector gzip cron on gadolinium." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57507 [14:12:19] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57507 [14:19:51] New patchset: Demon; "Updating gerrit to 2.6-rc0-144-gb1dadd2" [operations/debs/gerrit] (master) - https://gerrit.wikimedia.org/r/57508 [14:20:18] New review: Demon; "war is available at: https://integration.wikimedia.org/nightly/gerrit/wmf/gerrit-2.6-rc0-144-gb1dadd..." [operations/debs/gerrit] (master) C: 1; - https://gerrit.wikimedia.org/r/57508 [14:22:06] and where is available peace? [14:22:30] heyaaa apergos, you there? [14:25:07] PROBLEM - Puppet freshness on virt1000 is CRITICAL: No successful Puppet run in the last 10 hours [14:26:46] <^demon> MaxSem: I didn't create any peace. [14:27:08] how typical! [14:28:20] Change abandoned: Demon; "Not going to do this as a giant change. We can do it as needed." [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/52892 [14:32:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:19] New patchset: Andrew Bogott; "Do a full MW clone instead of a shallow one." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57325 [14:33:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.198 second response time [14:33:34] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57325 [14:35:49] New patchset: Demon; "Updating gerrit to 2.6-rc0-144-gb1dadd2" [operations/debs/gerrit] (master) - https://gerrit.wikimedia.org/r/57508 [14:43:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:43:40] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [14:44:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [14:46:24] ottomata: [14:46:29] what's up? [14:50:05] heya [14:50:13] do you know how the webstats dumps files are removed from locke? [14:50:27] i know your script on snapshot1 copies them to dumps.wm.o [14:50:37] but the logs on locke are about only 11 or 12 days old [14:50:54] i was about to add a find -mtime +12 -delete cron on gadolinium [14:50:59] unless you know a better way / how locke does it now [14:51:20] apergos^ [14:51:35] hmm [14:51:51] there used to be a job over there that cleaned em up I thought [14:52:10] OH [14:52:12] you know there is [14:52:14] sorry, i missed that [14:52:25] /a/webstats/scripts/purge [14:52:32] yeah [14:52:42] for file in $(find ./ -maxdepth 1 -type f -mtime +10) [14:52:42] do [14:52:42] # date=$(echo $file | cut -d- -f2) [14:52:42] # newdir=./archive/${date:0:4}/${date:4:2}/ [14:52:42] # mkdir -p $newdir [14:52:43] # mv $file $newdir/ [14:52:43] rm $file [14:52:44] done [14:52:45] basically find --delete [14:52:53] i guess it used to do some fancy rotating [14:53:01] k, i'll stick with my delete cron then [14:53:06] all righty [14:53:09] danke! [14:55:07] sure [14:55:49] i'm soon going to edit your cron on snapshot1 to copy webstats dumps from gadolinium [14:55:50] s'ok? [14:55:53] is there anything I should know? [14:58:10] New patchset: Ottomata; "Setting up cron to delete old webstats dumps on gadolinium." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57511 [15:00:49] heya chad, mark, now that action = fast forward only [15:01:07] what should we do differently when committing changes? [15:01:17] ^demon [15:01:52] New patchset: Matthias Mullie; "Update frwiki config" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55946 [15:02:12] <^demon> Everything has to be rebased on top of HEAD before submitting. Gerrit's supposed to do this implicitly, but if it doesn't there's always the rebase button. [15:02:21] <^demon> If there's conflicts, you'll still have to resolve, so no change there. [15:02:40] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [15:04:10] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [15:04:17] yeah, hm, what about patchsets though [15:04:23] amended ones [15:04:26] every time rebase? [15:05:42] New patchset: Ottomata; "Setting up cron to delete old webstats dumps on gadolinium." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57511 [15:05:52] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57511 [15:06:33] New review: coren; "I have, there is no added configuration to sshd_config when the ssh_restrict_network has no value, w..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57447 [15:09:17] New patchset: Ottomata; "Missing a /" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57512 [15:09:26] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57512 [15:10:10] <^demon> You don't have to rebase until right before submission. [15:10:28] <^demon> Individual patches can be based on whatever parent makes sense. [15:10:39] until merge? [15:10:40] hmm, i see [15:10:45] well, no longer merge [15:10:46] i get it [15:10:46] k [15:10:51] <^demon> :) [15:10:53] same diff anyway, right? [15:10:57] <^demon> Basically, yeah [15:11:00] if there was a conflict it wouldn't merge a nyway [15:11:10] usually rebase will jsut work? [15:11:17] <^demon> Yeah, if it's a clean rebase. [15:11:53] <^demon> Ideally I'd like to see it use the recursive merge for rebasing, but meh, still experimental. [15:12:03] <^demon> So a rebase could at least solve some conflicts. [15:27:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [15:32:07] New patchset: Odder; "(bug 46154) Add forgotten config for thwiki Add abusefilter-log-detail for autoconfirmed on thwiki per bug." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57515 [15:33:38] New review: Andrew Bogott; "This should probably wait for https://gerrit.wikimedia.org/r/#/c/43886/ since that renames the proxy." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57443 [15:35:57] ^demon: if you can't approve this small patch, what would be best way to get it in quickly? https://gerrit.wikimedia.org/r/#/c/56879/ [15:36:24] (that's to halve the wikibugs spam) [15:37:04] <^demon> I can't. [15:37:09] <^demon> It's not puppetized. [15:37:24] <^demon> https://gerrit.wikimedia.org/r/#/c/53973/ needs to be merged. [15:39:45] PROBLEM - DPKG on cp1041 is CRITICAL: Timeout while attempting connection [15:40:35] RECOVERY - DPKG on cp1041 is OK: All packages OK [15:44:17] ^demon: but that's only the bugzilla part of the fix. Or "It's not puppetized" referred to bugzilla? [15:45:09] <^demon> Oh, I misread the repo name. [15:45:19] <^demon> Nothing relating to bugzilla or wikibugs is puppetized :( [15:45:58] Right, I suspected that [15:46:32] still, CR would help (and the commit message quotes you :p) [15:47:10] PROBLEM - Varnish HTCP daemon on cp1041 is CRITICAL: Timeout while attempting connection [15:48:05] RECOVERY - Varnish HTCP daemon on cp1041 is OK: PROCS OK: 1 process with UID = 997 (varnishhtcpd), args varnishhtcpd worker [15:48:06] <^demon> Isn't REL_CC what it used to be? [15:50:35] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [15:54:38] !log now generating webstatscollector dumps on gadolinium. snapshot1 syncs dumps from there over to dataset2 for dumps.wikimedia.org [15:54:45] Logged the message, Master [15:54:58] New patchset: Odder; "(bug 45638) Add patrol right to autopatrolled on itwikivoyage This change removes patrol right added to autoconfirmed group in I198651d7, and moves it to the autopatrolled group instead, per comment 9 in bug 45638." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57519 [16:00:35] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [16:05:38] RECOVERY - Host cp3003 is UP: PING OK - Packet loss = 0%, RTA = 87.83 ms [16:05:38] PROBLEM - Varnish HTTP upload-frontend on cp3003 is CRITICAL: Connection refused [16:05:39] PROBLEM - Varnish HTCP daemon on cp3003 is CRITICAL: Connection refused by host [16:05:48] PROBLEM - NTP on cp3003 is CRITICAL: NTP CRITICAL: No response from NTP server [16:05:48] PROBLEM - SSH on cp3003 is CRITICAL: Connection refused [16:05:58] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:08:18] PROBLEM - Varnish traffic logger on cp3003 is CRITICAL: Connection refused by host [16:08:19] apergos: Hey. Is that slave not so lagged now? :D [16:08:28] PROBLEM - Varnish HTTP upload-backend on cp3003 is CRITICAL: Connection refused [16:08:48] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 16:08:43 UTC 2013 [16:08:58] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:09:18] PROBLEM - DPKG on tmh2 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [16:09:28] PROBLEM - DPKG on tmh1 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [16:09:48] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 16:09:46 UTC 2013 [16:09:58] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:10:18] RECOVERY - DPKG on tmh2 is OK: All packages OK [16:10:28] RECOVERY - DPKG on tmh1 is OK: All packages OK [16:10:48] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 16:10:43 UTC 2013 [16:10:58] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:11:38] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 16:11:33 UTC 2013 [16:11:58] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:12:18] PROBLEM - Host cp3003 is DOWN: PING CRITICAL - Packet loss = 100% [16:12:18] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 16:12:16 UTC 2013 [16:12:58] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:12:58] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 16:12:55 UTC 2013 [16:13:58] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:14:06] Looks to be.. [16:15:41] !log upgrading image & video scalers for libav [16:15:49] Logged the message, Master [16:17:19] PROBLEM - DPKG on mw78 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [16:17:28] PROBLEM - DPKG on mw80 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [16:18:18] PROBLEM - Apache HTTP on mw75 is CRITICAL: Connection refused [16:18:18] RECOVERY - DPKG on mw78 is OK: All packages OK [16:18:28] PROBLEM - Apache HTTP on mw78 is CRITICAL: Connection refused [16:18:28] PROBLEM - Apache HTTP on mw79 is CRITICAL: Connection refused [16:18:28] RECOVERY - DPKG on mw80 is OK: All packages OK [16:18:39] PROBLEM - Apache HTTP on mw80 is CRITICAL: Connection refused [16:18:39] PROBLEM - Apache HTTP on mw76 is CRITICAL: Connection refused [16:18:39] PROBLEM - Apache HTTP on mw77 is CRITICAL: Connection refused [16:18:48] PROBLEM - LVS HTTP IPv4 on rendering.svc.pmtpa.wmnet is CRITICAL: Connection refused [16:20:04] let's shutdown tampa ;) [16:20:34] But then we wouldn't have slaves to run silly queries on! [16:20:45] <^demon> Or fenari! [16:21:42] or srv193/testwiki! [16:22:36] paravoid: hello :) [16:22:43] hi [16:22:48] RECOVERY - Host cp3003 is UP: PING OK - Packet loss = 0%, RTA = 88.13 ms [16:25:52] Reedy: we still don't have such queries :( [16:26:08] I've used a few [16:26:33] selfish [16:26:35] oh crap [16:26:40] I'm for the query equality https://gerrit.wikimedia.org/r/#/c/33713/ [16:26:59] just saw the lvs, fixing [16:27:09] you again!?! [16:27:14] yes :) [16:28:18] RECOVERY - Apache HTTP on mw75 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.129 second response time [16:28:28] RECOVERY - Apache HTTP on mw78 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.146 second response time [16:28:38] RECOVERY - Apache HTTP on mw80 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.234 second response time [16:28:38] RECOVERY - Apache HTTP on mw77 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.231 second response time [16:28:44] why the f** did apache stop [16:28:50] oh well, it was pmtpa so no harm [16:29:28] RECOVERY - Apache HTTP on mw79 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.133 second response time [16:29:28] PROBLEM - Host cp3003 is DOWN: PING CRITICAL - Packet loss = 100% [16:29:48] RECOVERY - LVS HTTP IPv4 on rendering.svc.pmtpa.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 62001 bytes in 0.812 second response time [16:30:10] New review: Nemo bis; "Ping! Tim, your last comment in short is that this may not load Tampa servers *enough* (now that the..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33713 [16:31:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:32:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.135 second response time [16:33:28] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Thu Apr 4 16:33:17 UTC 2013 [16:33:58] PROBLEM - Puppet freshness on xenon is CRITICAL: No successful Puppet run in the last 10 hours [16:35:25] New review: Matthias Mullie; "Approved by Luke!" [operations/mediawiki-config] (master) C: 2; - https://gerrit.wikimedia.org/r/55946 [16:37:19] RECOVERY - SSH on cp3003 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [16:37:29] RECOVERY - Host cp3003 is UP: PING OK - Packet loss = 0%, RTA = 88.12 ms [16:37:41] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55946 [16:38:53] New patchset: Se4598; "fix AFTv5 oversight e-mail for frwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57522 [16:40:27] Change abandoned: Se4598; "fixed by https://gerrit.wikimedia.org/r/55946" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57522 [16:45:11] mlitn: heya, just checking if you were doing the AFTv5 deploy yet? [16:45:48] greg-g: I am :) [16:46:39] mlitn: cool, just didn't see any server-side activity in the log :) [16:46:53] so Reedy unfortunately my job is still running :-( [16:47:10] I noticed the 84306 lag :( [16:47:45] and the log file is a lot larger, it grew a giant amount in only well less than two weeks [16:47:49] this is not good [16:48:40] RECOVERY - Apache HTTP on mw76 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.133 second response time [16:48:43] ugh all the autocreated users are in there of course [16:48:43] man, everyone wants db55 today [16:49:13]