[00:01:17] <^d> !log manganese back up [00:01:25] Logged the message, Master [00:04:00] <^d> Ryan_Lane: We're dandy now, thanks. [00:04:23] \o/ [00:04:25] yw [00:06:32] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [00:06:52] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [00:07:12] New patchset: Dzahn; "include exim::aliases::private on mchenry to puppetize mchenry alias files (class and files in private repo) RT #5358" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71745 [00:08:02] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Wed Jul 3 00:07:57 UTC 2013 [00:08:32] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Wed Jul 3 00:08:30 UTC 2013 [00:08:53] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:12] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Wed Jul 3 00:09:09 UTC 2013 [00:09:32] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [00:09:42] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Wed Jul 3 00:09:39 UTC 2013 [00:09:53] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:22] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Wed Jul 3 00:10:13 UTC 2013 [00:10:32] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [00:10:42] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Wed Jul 3 00:10:40 UTC 2013 [00:10:52] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:22] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Wed Jul 3 00:11:12 UTC 2013 [00:11:32] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [00:11:42] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Wed Jul 3 00:11:36 UTC 2013 [00:11:52] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:12] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Wed Jul 3 00:12:03 UTC 2013 [00:12:32] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [00:12:32] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Wed Jul 3 00:12:24 UTC 2013 [00:12:52] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:52] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Wed Jul 3 00:12:48 UTC 2013 [00:13:03] New review: Dzahn; "tested this on local computer with puppet apply" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/71745 [00:13:04] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71745 [00:13:32] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [00:13:42] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Wed Jul 3 00:13:38 UTC 2013 [00:13:52] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:32] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [00:16:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:17:12] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Wed Jul 3 00:17:07 UTC 2013 [00:17:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [00:17:52] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [00:30:02] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Wed Jul 3 00:30:01 UTC 2013 [00:30:32] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [01:01:45] New patchset: Jdlrobson; "Only special case main pages on projects which have set it up" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71748 [01:08:28] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [01:08:28] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [01:11:04] Ryan_Lane: hah! [01:11:12] there should be 'vagrant wtf' subcommand for reporting issues [01:11:48] :D [01:16:48] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [01:16:48] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [01:16:48] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [01:16:48] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [01:16:48] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [01:16:48] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [01:16:48] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [01:16:49] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [01:31:58] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset 0.0007344484329 secs [01:32:58] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -4.208087921e-05 secs [02:07:22] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [02:07:42] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [02:15:00] !log LocalisationUpdate completed (1.22wmf8) at Wed Jul 3 02:15:00 UTC 2013 [02:15:10] Logged the message, Master [13:17:12] <^d> apergos: Mind taking a look at https://gerrit.wikimedia.org/r/#/c/66665/ again? I revised it based on yours and Aaron's input. [13:18:04] sec [13:18:10] hi ^d [13:18:15] <^d> Hi [13:18:17] i am looking at https://bugzilla.wikimedia.org/show_bug.cgi?id=47656 [13:18:22] which has a resource tracker thing there [13:18:30] which i can't see, so no idea the status [13:18:37] 1) can someone check? [13:18:59] 2) maybe it's warranted for me or some of the wikidata team (e.g. denny / daniel) have access to rt [13:19:07] 3) who is in charge of it? [13:19:12] * MatmaRex still has no idea why rt isn't public [13:19:18] so we can stop poking people :) [13:19:33] it has stuff like pricing or such about hardward [13:19:36] hardware [13:19:43] <^d> 1) Doesn't look like anyone's done anything [13:19:45] or such stuff [13:19:50] ^d: ok :/ [13:19:51] bleh [13:19:52] <^d> 2) That's worth asking, but I'm not the one to answer [13:20:02] ok, who do i ask? [13:20:05] "working" [13:20:07] wikimedia trade secrets [13:20:15] <^d> aude: I can help, this is trivial. [13:20:23] ^d: cool [13:20:58] rmeoved the trailing slash and it loaded, wtf [13:21:24] <^d> aude: So, we're just going to copy the file from the Wikibase extension, and when it needs updating on the site, we'll just copy again? [13:21:38] <^d> Or do we want an alias just pointing at that file? [13:21:47] i think an apache thing would be preferred [13:21:59] * aude thinks alias would work [13:22:12] <^d> Ok, let's have a look [13:22:15] k [13:22:16] ^d, I'm fine with it, should I give other people the opportunity to review (i.e. just +1 it)? [13:23:01] <^d> I think it's an improvement over now. We're falling back to defaults, which is like 65/70 (instead of the proposed 100/120) [13:23:30] yep [13:25:55] so was that a "just +2 it" or not? :-D [13:25:59] I'm fine to deploy it [13:28:06] New patchset: Demon; "Adding alias for Wikidata ontology" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/71790 [13:28:39] <^d> apergos: Just needs merging on sockpuppet, I can run puppet on manganese. [13:28:43] great [13:28:49] <^d> aude: 71790 ^ [13:28:56] looks good to me [13:29:19] must need rebase ^d [13:29:37] New patchset: Demon; "Set max commit summary lengths for Gerrit" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66665 [13:29:50] <^d> I <3 the rebase button [13:30:40] <^d> aude: Daniel usually takes a look at those sorts of apache changes (and filed the RT), so I guess bug him when he gets around. [13:31:01] * apergos waits for gerrit to get around to it [13:31:31] ^d: ok [13:32:11] thanks ^d [13:32:12] :) [13:32:15] <^d> yw. [13:32:22] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/70557 [13:32:30] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/66665 [13:33:02] ^d: also, not sure if Reedy is around but we have some config changes [13:33:12] https://gerrit.wikimedia.org/r/#/c/71549/ and https://gerrit.wikimedia.org/r/#/c/71547/ [13:33:20] ^d: your turn [13:33:35] * ^d takes the baton and runs [13:33:38] and.... one sec [13:36:18] <^d> aude: Does 71547 rely on anything being deploy, or is it already on all wikis and this is just back-compat cleanup? [13:36:31] the settings requires nothing [13:36:39] the propagate changes one requires a cherry pick [13:36:41] !log maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/70557/' [13:36:44] is mw80 known to be b0rked? [13:36:50] Logged the message, Master [13:37:34] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71547 [13:38:01] New patchset: Petr Onderka; "starting: reading from dumpBackup using XMLIO library" [operations/dumps/incremental] (master) - https://gerrit.wikimedia.org/r/71793 [13:38:25] woo hoo! [13:38:45] New review: Petr Onderka; "For now, there is just a Visual Studio solution. I will add a makefile later." [operations/dumps/incremental] (master) C: 1; - https://gerrit.wikimedia.org/r/71793 [13:39:03] hmm in branch master. heh [13:39:12] !log demon synchronized wmf-config/CommonSettings.php 'Wikibase: I2e9d03b2' [13:39:21] Logged the message, Master [13:39:48] <^d> aude: First one out, please verify :) [13:39:59] ok [13:41:04] all seems good [13:41:38] <^d> Ok, and https://gerrit.wikimedia.org/r/#/c/71549/ requires a cherry-pick? [13:41:41] <^d> What, and to what branches? [13:41:44] i'm doing [13:42:22] it's for wmf9 to update wikibase [13:42:25] <^d> MaxSem: Known? Dunno, but I just timed out to it too when sync'ing. [13:42:32] i can make a patch for it [13:43:00] <^d> That'd be great. [13:43:43] <^d> MaxSem: icinga says mw80 and mw1173 are down [13:45:05] <^d> aude: I'll be back in 5, raiding the fridge. [13:45:08] ok [13:45:15] * aude updating the submodules [13:45:18] icinga says something? [13:45:30] ah I see it's back [13:45:52] <^d> https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?hostgroup=all&style=hostdetail&hoststatustypes=4&nostatusheader - mw80 and 1173 [13:45:57] <^d> Anyway, brb [13:47:10] mw80: bad memory [13:47:25] New patchset: Petr Onderka; "starting: reading from dumpBackup using XMLIO library" [operations/dumps/incremental] (gsoc) - https://gerrit.wikimedia.org/r/71796 [13:47:32] mw1173: also memory issue (rt) [13:47:48] ah the right branch this time :-) [13:48:04] New review: Petr Onderka; "For now, there is just a Visual Studio solution. I will add a makefile later." [operations/dumps/incremental] (gsoc); V: 2 C: 2; - https://gerrit.wikimedia.org/r/71796 [13:48:38] Change merged: Petr Onderka; [operations/dumps/incremental] (gsoc) - https://gerrit.wikimedia.org/r/71796 [13:48:41] ^d: when you are back https://gerrit.wikimedia.org/r/#/c/71797/ [13:52:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [13:53:49] <^d> apergos: Ok, MaxSem and I were just wondering since we couldn't sync to them from tin :) [13:54:45] that would be a good reason ;-) [13:57:52] <^d> aude: +2'd, waiting on jenkins. [13:58:05] k [13:58:09] so hm, ^d, https://github.com/wikimedia/puppet-cdh4 doesn't look replicated yet :/ [13:58:23] <^d> Yeah, I noticed late last night it was failing again. [13:58:32] <^d> But I had a headache and didn't have the energy to figure it out. [13:58:41] <^d> Lemme finish up what I'm doing for aude and I'll have a look-see. [13:58:44] then we will need to do a settings change, so that moving a page on test2 does *not* change site links on wikidata [13:58:44] can I help? where do I look? [13:58:47] mangese log somewhere? [13:59:02] at some point soon, we'll configure test.wikidata to be test2 repo [13:59:05] not today.... [13:59:14] <^d> ottomata: manganese:/var/lib/gerrit2/review_site/logs/error_log, grep for the repo name. [13:59:59] errrr, stupid [14:00:04] * aude already did the settings change :) [14:01:28] hmm, missing commit sha? [14:01:28] hm [14:02:56] New patchset: Demon; "I'm lazy and don't like typing" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71801 [14:03:31] ok hm, ^d, this commit is from a branch on github that doesn't exist on gerrit [14:03:38] i don't need this branch, its old and won't be used [14:03:43] hm, but i guess that doesn't matter [14:03:44] if the commit is there [14:03:53] should I just push the branch and its commits to gerrit? [14:04:20] <^d> I guess that'd work. [14:04:32] k will try [14:05:47] !log demon synchronized php-1.22wmf9/extensions/Wikibase/ 'Updating wikibase to 5a6511760562dc5b8fae033492af704599141640' [14:05:56] Logged the message, Master [14:06:16] <^d> aude: Wikibase updated, please verify :) [14:06:27] ok [14:06:43] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [14:06:53] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [14:07:38] do i dare move a page on test2 now :) [14:08:18] * aude satisfied that everything is okay [14:08:40] <^d> Okie dokie, so let's have a look at this setting now [14:08:53] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [14:09:07] i actually think wikibase (deployed in wmf8 / wikidata) would not understand what a UpdateRepoOnPageMove job class is [14:09:25] the job runner would probably throw a fatal if i tried moving a page [14:09:31] * aude won't try [14:09:44] and the setting will prevent a job being issued [14:10:21] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71549 [14:11:33] !log demon synchronized wmf-config/CommonSettings.php 'Wikibase changes' [14:11:42] Logged the message, Master [14:12:07] !log demon synchronized wmf-config/InitialiseSettings.php 'Wikibase changes' [14:12:18] Logged the message, Master [14:12:40] <^d> aude: Okie dokie, all done. [14:13:32] thanks ^d [14:13:36] <^d> yw [14:13:38] everything is good :) [14:13:42] <^d> yay [14:15:27] New patchset: ArielGlenn; "set up the dataset* servers for simultaneous rsyncs between hosts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71802 [14:16:43] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Wed Jul 3 14:16:32 UTC 2013 [14:16:53] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [14:19:52] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71802 [14:22:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [14:26:13] heyaaaa great! thanks ^d, replication is working! [14:26:23] <^d> Yay, glad you figured it out. [14:26:46] Hm, this replication doesn't override the header message on github like the default one does [14:26:51] Github mirror of "operations/puppet/cdh4" [14:27:02] it'd be nice if it said that too, is that something in the replication settings? [14:27:04] <^d> That's just done at creation time. [14:27:13] <^d> You just need to edit that on the github side. [14:27:44] hm ok [14:27:47] i see cool [14:27:56] <^d> I've got a hacky wmf-specific plugin that handles github creation at project creation time, it does that then :) [14:29:53] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Wed Jul 3 14:29:46 UTC 2013 [14:30:36] New patchset: Ottomata; "Updating README.md" [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/71804 [14:30:43] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [14:30:44] Change merged: Ottomata; [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/71804 [14:31:10] ok, and ^d, which file do I add new repositories to replicate to? [14:31:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:41] <^d> For things that are one-off named like this? Or generally? [14:31:58] one off i guess [14:32:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [14:32:34] <^d> Ah, well they're in the same place. manifests/roles/gerrit.pp [14:33:29] <^d> After making changes and getting live you have to reload the replication plugin. `ssh -p 29418 gerrit.wikimedia.org gerrit plugin reload replication` [14:34:47] ah ok, cool [14:34:48] danke [14:50:54] New patchset: BBlack; "Multiple DBs per VCL; removed fatal startup errors" [operations/software/varnish/libvmod-netmapper] (master) - https://gerrit.wikimedia.org/r/71806 [14:50:54] New patchset: BBlack; "version bump to 1.1" [operations/software/varnish/libvmod-netmapper] (master) - https://gerrit.wikimedia.org/r/71807 [14:51:17] Change merged: BBlack; [operations/software/varnish/libvmod-netmapper] (master) - https://gerrit.wikimedia.org/r/71806 [14:51:27] Change merged: BBlack; [operations/software/varnish/libvmod-netmapper] (master) - https://gerrit.wikimedia.org/r/71807 [14:52:16] New patchset: BBlack; "Merge branch 'master' into debian" [operations/software/varnish/libvmod-netmapper] (debian) - https://gerrit.wikimedia.org/r/71808 [14:52:17] New patchset: BBlack; "update upstream version" [operations/software/varnish/libvmod-netmapper] (debian) - https://gerrit.wikimedia.org/r/71809 [14:52:27] Change merged: BBlack; [operations/software/varnish/libvmod-netmapper] (debian) - https://gerrit.wikimedia.org/r/71808 [14:52:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:36] Change merged: BBlack; [operations/software/varnish/libvmod-netmapper] (debian) - https://gerrit.wikimedia.org/r/71809 [14:53:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.146 second response time [15:01:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:02:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.150 second response time [15:06:42] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [15:06:42] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [15:07:25] New patchset: ArielGlenn; "no comment on rest of config line in rsync apparently" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71812 [15:08:41] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71812 [15:11:32] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:12:22] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [15:12:49] New review: Anomie; "(1 comment)" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/71775 [15:16:43] New review: Anomie; "Works for me." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/70322 [15:18:16] New patchset: MaxSem; "Enable new wmfSetupMobileLoadScript hooks everywhere" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71813 [15:19:52] New review: Anomie; "(1 comment)" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/71774 [15:20:50] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71813 [15:22:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:24:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [15:25:06] !log maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/71813/' [15:25:15] Logged the message, Master [15:26:32] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/70597 [15:29:26] !log maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/70597/' [15:29:35] Logged the message, Master [15:32:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:33:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [15:36:50] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71375 [15:38:48] !log maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/71375/' [15:38:57] Logged the message, Master [15:40:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:41:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [15:46:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.145 second response time [15:50:59] RECOVERY - Packetloss_Average on analytics1006 is OK: OK: packet_loss_average is 0.636164897025 [15:51:07] New patchset: Reedy; "I'm lazy and don't like typing" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71801 [15:51:24] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71801 [15:52:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:39] PROBLEM - SSH on mc15 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:54:14] New patchset: Ottomata; "Fixing PacketLossLogtailer all_roles metric, also limiting the number of roles reported to ganglia via rolematcher.py" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71817 [15:54:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.169 second response time [15:54:29] RECOVERY - SSH on mc15 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [15:55:27] New review: Diederik; "ok." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/71817 [15:55:57] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71817 [15:57:13] New review: MaxSem; "FAIL" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71801 [15:57:23] Reedy, ^_^ [15:58:00] ohlol [15:59:17] New patchset: Reedy; "$res -> $ret" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71818 [15:59:59] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71818 [16:02:03] New patchset: Ottomata; "Using new names for udp2log_kafka_producer metrics in Kafka ganglia view" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71819 [16:02:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:02:34] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71819 [16:02:39] ottomata: there was an RT about geoip, did you see that? [16:02:55] nope [16:03:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [16:03:37] #5411 [16:04:25] k danke looking [16:06:16] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:06:26] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:07:56] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Wed Jul 3 16:07:52 UTC 2013 [16:08:16] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:16] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Wed Jul 3 16:08:07 UTC 2013 [16:08:26] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:08:36] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Wed Jul 3 16:08:32 UTC 2013 [16:08:56] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Wed Jul 3 16:08:48 UTC 2013 [16:09:16] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:16] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Wed Jul 3 16:09:08 UTC 2013 [16:09:26] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:09:26] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Wed Jul 3 16:09:22 UTC 2013 [16:10:16] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:26] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:12:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:13:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [16:14:32] New patchset: Ottomata; "Updating geoip module usage on puppetmaster" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71822 [16:14:48] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71822 [16:17:16] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Wed Jul 3 16:17:10 UTC 2013 [16:17:36] Wikipedia is sooo sloooow [16:18:20] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:46] New patchset: Ottomata; "Fully qualifying geoip module class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71825 [16:19:19] hashar, can you have a look at that beta search box and see if the rsync went down properly? [16:19:22] I don't know where to look. [16:19:36] neither do i :-D [16:19:57] lets dig under /a/search :-) [16:20:30] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71825 [16:20:32] andrewbogott: /a/search/indexes/update/enwikiquote.prefix/20130703092515/ [16:20:42] andrewbogott: on deployment-search01.pmtpa.wmflabs [16:20:54] andrewbogott: so I guess the rsync has run, it might even be working properly [16:21:41] how would we know if it's working properly? [16:21:55] enwiki.prefix -> /a/search/indexes/update/enwiki.prefix/20130703092515/ [16:22:10] so potentially look for a wiki which had a new page created [16:22:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:22:59] andrewbogott: the selenium tests are creating a bunch of new pages everyday http://en.wikipedia.beta.wmflabs.org/wiki/Special:NewPages [16:23:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.404 second response time [16:23:38] andrewbogott: [[‎0.1821400206458863]] got created yesterday and is available in the search box [16:23:49] you're ahead of me [16:23:53] New patchset: Ottomata; "Need passwords::geoip for puppetmaster::geoip" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71826 [16:23:57] but, ok, sounds good -- I declare victory [16:24:04] andrewbogott: yeah that seems fine :-] [16:24:06] RECOVERY - Packetloss_Average on analytics1008 is OK: OK: packet_loss_average is 1.11864984848 [16:24:12] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71826 [16:24:17] andrewbogott: but then tomorrow is an holiday day, so you might want to postpone till monday :-] [16:24:47] hm… yeah. [16:25:27] andrewbogott: I am off, ping me by email I might follow up later this evening (eu time) [16:25:32] New review: Andrew Bogott; "Yep, verified that this works fine. I'm waiting until Monday to merge, though, just in case." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71106 [16:25:35] ok, thanks! [16:26:18] off *waves* [16:30:06] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Wed Jul 3 16:29:55 UTC 2013 [16:30:06] RECOVERY - Packetloss_Average on analytics1009 is OK: OK: packet_loss_average is 0.759576335878 [16:30:26] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:32:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:32:47] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71719 [16:33:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [16:36:48] New patchset: Ottomata; "Adding an analytics-data ganglia view" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71828 [16:36:59] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71828 [16:42:30] @notice hashar https://gerrit.wikimedia.org/r/#/c/71245/ [16:43:55] AzaToth: i am about to upload buck deb to apt.w.o [16:44:27] akosiaris: even though it contains 6 jars? [16:44:55] akosiaris: whay do you think about the above one? [16:45:18] AzaToth: a yes... we should check for any license issues about these first [16:45:47] i suppose validity (a CRC or GPG sig) has been checked ? [16:46:01] akosiaris: I never did such thing [16:46:15] akosiaris: I was waiting for some indication how we should proceed [16:46:40] ok i will do it then. Other than that i see no other hurdle [16:46:54] ok [16:47:00] so we could probably deploy it and wait for all hell to break loose [16:48:22] so jenkins building packages ? [16:53:25] akosiaris: yup [16:54:14] i like the idea. [16:55:09] !log reedy synchronized php-1.22wmf9/extensions/Wikibase [16:55:18] Logged the message, Master [16:59:03] AzaToth: damn... needs some more work ... jython license.... figuring out the rest [17:01:18] akosiaris: https://gerrit.wikimedia.org/r/#/c/71245/ updated it to be less changy [17:01:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:01:30] did some bad restructuring before [17:02:11] akosiaris: /usr/share/doc/jython/copyright [17:02:17] akosiaris: I couldn't use supplied jython [17:03:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [17:03:49] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [17:05:24] paravoid: hi [17:05:32] paravoid: can you please have another look at https://gerrit.wikimedia.org/r/#/c/68711 ? [17:06:10] paravoid: I took into consideration all your comments, and marked as done or replied on gerrit [17:06:22] paravoid: I also took into consideration Andrew's comments [17:07:21] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [17:07:39] Ryan_Lane: I was hoping hasar would have looked into that one by now... [17:07:39] https://gerrit.wikimedia.org/r/#/c/71245/ [17:08:11] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [17:08:26] AzaToth: heh. I have no clue what that does :) [17:09:34] !log authdns-update rt 4547 [17:09:44] Logged the message, RobH [17:10:07] bleh. SFO's wifi sucks [17:10:10] bleh, ns1 didnt like that. [17:10:15] Ryan_Lane1: traveling? [17:10:32] RobH: going to NYC till the 10th [17:10:39] who is the person to ask about bugzilla rights? [17:10:40] will be working except for the 4th and 5th [17:10:49] gwicke: andre__ [17:10:52] Ryan_Lane: nice, have a good time =] [17:10:57] Ryan_Lane: you sure go to NYC a lot :P [17:11:00] eh? [17:11:01] have fun riding transit without striking. [17:11:02] andre__: ping ;) [17:11:05] paravoid: thanks! [17:11:06] pong [17:11:11] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:11] AzaToth: I don't know what the debian-glue stuff does. I'd imagine it's for making jenkins build the packages? [17:11:20] paravoid: well, I'm dating someone there ;) [17:11:44] RobH: yep. and transit that works 24/7 and fast and actually goes everywhere [17:11:48] andre__: it would be great if mail@marcoil.org had the right to take over bugs in the Parsoid product [17:11:49] I'm not that stupid you know :P [17:11:53] :) [17:11:55] Ryan_Lane: *G train notwithstanding [17:12:00] just teasing you [17:12:03] ori-l: true [17:12:14] gwicke, done [17:12:26] paravoid: :D well, NYC is also awesome, so there's that too [17:12:30] andre__: awesome, thx! [17:12:39] np [17:13:14] Ryan_Lane: aye [17:13:36] Ryan_Lane: it's or building, testing, making sure it's 100% perfect [17:13:42] is* [17:13:48] it would be awesome to be able to define this kind of stuff outside of the jenkins repo [17:14:17] it would be nice for jenkins to build all of our packages :) [17:14:32] Ryan_Lane: read the diff :--P [17:14:47] I am. gerrit and buck are defined there [17:15:00] yea [17:15:18] you just need to make a such definition for each of the "project" to be built [17:15:25] I'm going to let hashar +1/2 this [17:15:33] and also zuul must be setup to forward gerrit [17:15:45] * Ryan_Lane nods [17:15:53] he never did :-( [17:16:09] AzaToth: so ... all jars apl 2.0, apart from jython which is psfl. Minor changes to copyright and good to go on that front [17:16:19] I don't know jenkins well enough to know if this is ok, so I'm probably not a great reviewer for it [17:16:21] psfl? [17:16:28] about to start zero deployment [17:16:37] Ryan_Lane, greg-g [17:16:38] python software foundation license [17:16:53] BSD-style [17:16:54] yurik: don't break sh!t [17:16:59] Ryan_Lane: piuparts is a system which tries to install and uninstall the package to make sure it behaves properly [17:17:05] greg-g: :D [17:17:08] akosiaris: okidoki [17:17:08] !log deploying change 60783 to virt0 [17:17:18] Logged the message, Master [17:17:25] greg-g, then how would i feel important?! [17:17:45] yurik: by -1ing changes for whitespace violations [17:17:50] akosiaris: have you tried to use buck yourself building gerrit? [17:17:51] (my flight has wifi, so if something goes bad with my deployment, I'll revert ;) ) [17:17:52] yurik: there's plenty of ways to do that [17:18:03] yurik: see what ori-l said, that sounds like a great alternative [17:18:14] while I've done it on my local comp, I can't guarantee it's 100% functional as I've not made a chroot [17:18:24] AzaToth: not realy... must I? [17:18:44] akosiaris: "must" is such a hard word [17:19:09] you can if you feel it's relevant to test before pushing it [17:19:29]