[00:00:02] hrm, and trying to figure out if Roan is in the wmf ldap group [00:00:04] "not working as specified since 23:49:16" [00:00:09] because he's getting not authorized errors [00:00:10] So it took 5 minutes to get to my phone?wtf [00:00:30] sounds like sms template expansion [00:01:02] Oooooh here we go [00:01:08] It's paging me for Parsoid Varnish on cerium [00:01:17] That's actually a Watchmouse page [00:01:26] It's not paging for the 3 LVS services we killed [00:02:03] LeslieCarr: on formey: ldaplist -l group wmf [00:04:27] New patchset: Lcarr; "giving roan command access" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59976 [00:05:05] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59976 [00:06:25] LeslieCarr: could you possibly do https://gerrit.wikimedia.org/r/#/c/59972/ as well? I stopped puppet and moved the directory in preparation. [00:07:16] si [00:07:21] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59972 [00:07:31] done [00:07:35] I've used up like a month's worth of my puppet allowance in like 24 hours but I'm pretty much done, I promise [00:07:37] thanks :)) [00:11:13] Hah, looks like there's something wrong with the pybal health checks or something [00:11:21] It thinks everything is down [00:11:27] Nagios thinks the same [00:11:35] GET / seems to have broken somehow but everything else is still working [00:17:58] Hah, looks like Parsoid broke it [00:18:04] They didn't close the connection when serving / [00:18:07] So the monitors timed out [00:18:29] Which is why pybal depooled half the cluster, and why Nagios believed things were down. But they were actually up, every URL except / worked ^^ [00:20:37] I'm deploying a fix [00:20:59] RECOVERY - Parsoid on wtp1002 is OK: HTTP OK: HTTP/1.1 200 OK - 1368 bytes in 3.471 second response time [00:21:00] RECOVERY - LVS HTTP IPv4 on parsoid.svc.pmtpa.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 1368 bytes in 3.093 second response time [00:21:03] RECOVERY - Parsoid on wtp1001 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.006 second response time [00:21:09] there we go [00:21:10] RECOVERY - Parsoid on titanium is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.004 second response time [00:21:10] RECOVERY - LVS HTTP IPv4 on parsoid.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.012 second response time [00:21:12] RECOVERY - LVS HTTP IPv4 on parsoidcache.svc.pmtpa.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 1357 bytes in 0.057 second response time [00:21:19] RECOVERY - Parsoid Varnish on celsus is OK: HTTP OK: HTTP/1.1 200 OK - 1357 bytes in 0.058 second response time [00:21:39] RECOVERY - Parsoid Varnish on titanium is OK: HTTP OK: HTTP/1.1 200 OK - 1357 bytes in 0.008 second response time [00:21:40] RECOVERY - Parsoid on wtp1 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.056 second response time [00:21:49] RECOVERY - Parsoid on wtp1003 is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.003 second response time [00:21:49] RECOVERY - Parsoid on mexia is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.054 second response time [00:21:50] RECOVERY - Parsoid on tola is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.054 second response time [00:21:50] RECOVERY - Parsoid Varnish on constable is OK: HTTP OK: HTTP/1.1 200 OK - 1358 bytes in 0.055 second response time [00:21:50] RECOVERY - Parsoid on lardner is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.055 second response time [00:21:50] RECOVERY - Parsoid Varnish on cerium is OK: HTTP OK: HTTP/1.1 200 OK - 1357 bytes in 0.003 second response time [00:21:59] Whee [00:21:59] RECOVERY - Parsoid on cerium is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.008 second response time [00:22:28] \o/ [00:22:42] All so exciting. [00:22:50] RECOVERY - Parsoid on kuo is OK: HTTP OK: HTTP/1.1 200 OK - 1373 bytes in 0.088 second response time [00:25:28] New patchset: Ori.livneh; "Set MPLCONFIGDIR env var for Matplotlib" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59981 [00:27:12] OK, if get that merged I get graphs and then I can stop jumping in my chair. [00:27:55] (I'm not asking, just being declarative, like puppet.) [00:28:05] ensure => 'merged', etc. [00:28:35] !log catrope synchronized php-1.22wmf1/extensions/VisualEditor 'Update VisualEditor to master' [00:28:36] LeslieCarr: James_F points out I am the only one with a capital letter in that file, I hope that wasn't a mistake. I know I used Catrope with a capital C to log into icinga, is all [00:28:42] Logged the message, Master [00:28:57] !log catrope synchronized php-1.22wmf2/extensions/VisualEditor 'Update VisualEditor to master' [00:29:04] Logged the message, Master [00:29:47] LeslieCarr: Neeeever mind silencing works now. Thanks :) [00:30:36] ACKNOWLEDGEMENT - Parsoid on wtp1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds Catrope Deliberately sabotaged to serve as a testing/benchmarking ground for Gabriel [00:30:48] heh [00:32:02] ACKNOWLEDGEMENT - Parsoid on constable is CRITICAL: Connection refused Catrope Known breakage, not pooled we should really remove Parsoid from this box [00:32:27] ACKNOWLEDGEMENT - Parsoid on celsus is CRITICAL: Connection refused Catrope Known breakage, not pooled we should really remove Parsoid from this box [00:32:56] ori-l: I think you missed a sudo joke in there somewhere ;) [00:33:41] Writing a postmortem for the ops list [00:34:27] i sudidn't! (er. best i could come up with.) [00:57:40] RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds [01:08:55] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 18 seconds [01:13:55] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 198 seconds [01:15:55] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 15 seconds [01:28:55] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [01:30:55] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 0 seconds [01:42:08] New patchset: Ori.livneh; "Set MPLCONFIGDIR env var for Matplotlib" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59981 [01:43:57] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [01:50:57] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 27 seconds [01:51:36] New patchset: Aaron Schulz; "Revert "Enabled 1:1 profiling for cli scripts and put "cli" in the profile ID."" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59982 [01:51:46] Change merged: Aaron Schulz; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59982 [01:53:43] !log aaron synchronized wmf-config/StartProfiler.php 'Removed cli profiling for now.' [01:53:51] Logged the message, Master [01:54:12] !log aaron cleared profiling data [01:54:20] Logged the message, Master [02:16:42] !log LocalisationUpdate completed (1.22wmf2) at Fri Apr 19 02:16:42 UTC 2013 [02:16:49] Logged the message, Master [02:23:57] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 182 seconds [02:26:27] !log aaron cleared profiling data [02:26:34] Logged the message, Master [02:27:18] !log LocalisationUpdate completed (1.22wmf1) at Fri Apr 19 02:27:17 UTC 2013 [02:27:25] Logged the message, Master [02:28:58] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 232 seconds [02:31:37] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [02:33:57] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 18 seconds [02:40:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:41:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [02:44:04] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 218 seconds [02:53:04] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 24 seconds [03:00:04] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 218 seconds [03:03:04] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 14 seconds [03:32:40] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 19 03:32:40 UTC 2013 [03:32:48] Logged the message, Master [03:46:08] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [03:48:28] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 198 seconds [03:49:28] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 14 seconds [03:52:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:53:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [03:58:28] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 198 seconds [04:00:28] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 18 seconds [04:48:21] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 198 seconds [04:50:21] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 17 seconds [04:56:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:57:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [05:18:29] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 199 seconds [05:20:30] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 10 seconds [05:23:09] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [05:23:45] wow, apergos is the fastest RT gun in the east [05:23:54] maybe. i have no citations [05:23:55] no, you were just lucky [05:24:18] but let's see if you were lucky and the rename worked out [05:24:20] hehe :) [05:24:23] right [05:28:30] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 199 seconds [05:30:29] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 19 seconds [05:44:22] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 191 seconds [05:45:22] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 19 seconds [05:52:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:53:04] RD: scroll up [05:53:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [05:53:32] apergos? [05:53:36] yes [05:53:43] Ty for handling that OTRS RT :-) [05:53:44] yes? [05:53:49] ah it works? [05:53:52] Yup [05:53:58] great I'll close that then [05:54:07] * jeremyb_ was going to close [05:54:14] It turns out, after all these years of not being able to access it, it was just a dummy filter - it is deleted now! [05:54:19] ok you can [05:54:25] yes a test filter :-D [05:54:31] I admit I snickered when I saw that [05:55:09] closed [05:55:22] sweet [05:55:50] I was sort of mad! [05:55:58] hah [05:55:59] It has been bothering me for years ;-) [05:56:14] well, not any more. must be a good year :-D [05:57:07] ho ho [05:57:13] how is milan ? [05:58:22] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 199 seconds [06:00:22] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 19 seconds [06:01:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:02:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [06:06:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:07:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [06:09:22] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 184 seconds [06:10:22] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 10 seconds [06:13:01] PROBLEM - Puppet freshness on gallium is CRITICAL: No successful Puppet run in the last 10 hours [06:14:22] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 226 seconds [06:15:22] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 19 seconds [06:18:22] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 199 seconds [06:19:22] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 12 seconds [06:22:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [06:25:53] New patchset: Tim Starling; "(bug 45005) Redirect wikidata.org to www.wikidata.org" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49069 [06:26:38] New review: Tim Starling; "PS12: rebase." [operations/apache-config] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/49069 [06:26:43] Change merged: Tim Starling; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49069 [06:27:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:27:58] TimStarling: If I'm correct, that should mean http://wikidata.org/w/index.php?title=Special:Watchlist should redirect to www? [06:28:10] yes [06:28:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [06:28:18] it isn't yet for me [06:28:24] but that might be just my cahcing [06:28:40] hold your horses [06:28:52] sorry just wanted to use that expression [06:29:05] I only merged it, I haven't finished deploying it yet [06:29:18] * Jasper_Deng_busy forgot there wasn't a !log yet [06:31:26] !log deploying apache conf change for www.wikidata.org redirect (I7bb872fd) [06:31:34] Logged the message, Master [06:32:45] and it now works [06:33:20] are you sure you're busy? [06:41:57] New patchset: Tim Starling; "Basic puppetization of dsh" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56107 [06:45:52] New patchset: Tim Starling; "Basic puppetization of dsh" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56107 [06:47:08] New review: Tim Starling; "PS5: wrong parent, please ignore" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56107 [06:52:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:53:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [06:58:37] New patchset: Tim Starling; "In sync-dir, actually perform the syntax check" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56105 [06:58:37] New patchset: Tim Starling; "Move scap source location from fenari to tin" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56104 [06:58:38] New patchset: Tim Starling; "Basic puppetization of dsh" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56107 [07:00:11] New review: Tim Starling; "PS5: rebase including conflict resolution with I037a1f5e" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56104 [07:01:02] New patchset: Tim Starling; "Remove some node lists" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56108 [07:01:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:02:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.140 second response time [07:05:19] New review: Tim Starling; "-1 per Ryan's comment, ircecho from tin to Freenode won't work." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/56104 [07:21:47] New review: Tim Starling; "I think the simplest solution would be to use socat as a TCP relay, from ircecho on tin to Freenode...." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56104 [07:40:35] binasher, thanks! [07:52:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:53:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.135 second response time [08:05:57] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [08:05:57] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [08:05:57] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:11:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [08:18:55] New review: Krinkle; "(1 comment)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59810 [08:32:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:33:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [09:08:02] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [09:10:32] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [09:11:32] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [10:50:16] New patchset: Hashar; "lucene-jobs: convert java opts to shell variables" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59995 [10:50:16] New patchset: Hashar; "conf file for lucene.jobs.sh (not used yet)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59996 [10:52:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:53:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [11:00:17] New patchset: Hashar; "conf file for lucene.jobs.sh (not used yet)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59996 [11:00:50] New review: Hashar; "fixed invalid template call (source => content)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59996 [11:01:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [11:15:06] do I understand correctly that if we have a '+wiktionary' => array( 'Wiktionary' => NS_PROJECT, ) alias in wgNamespaceAliases then it's unnecessary to repeat this in wiki-specific aliases? [11:17:02] New patchset: Hashar; "conf file for lucene.jobs.sh (not used yet)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59996 [11:22:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [11:41:19] New patchset: Odder; "(bug 46846) Localise project namespaces for dv.wiktionary" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59998 [11:41:36] New patchset: Mark Bergsma; "Support per-backend options" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59999 [11:44:44] so hashar [11:44:47] New patchset: Hashar; "lucene-jobs: enable conf file loading" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60000 [11:44:57] how do I test this in beta labs? ;) [11:44:58] https://gerrit.wikimedia.org/r/#/c/59999/1 [11:44:59] mark: hello [11:46:09] New review: Hashar; "The conf sourcing is enabled in another change to prevent disruption https://gerrit.wikimedia.org/r/..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59996 [11:47:21] mark: sorry was doing some paperwork [11:47:27] no worries [11:47:31] New patchset: Odder; "(bug 44899) Namespace setup for Korean Wikiversity" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59786 [11:47:35] you're not my secretary after all ;-) [11:47:57] so beta has cache instances for upload, mobile, bits [11:48:05] some of them run from the production branch [11:48:17] mobile uses puppetmaster:self [11:49:11] I would love to have puppetmaster self self update [11:49:33] how do I log in? ;-) [11:49:45] ah [11:50:04] do you have a labs account ? [11:50:24] haha [11:50:25] yes [11:51:07] the labsconsole UI is so horrible [11:51:12] i'm trying to find the beta project there ;p [11:51:22] ahh https://wikitech.wikimedia.org/wiki/Special:Contributions/Mark_Bergsma [11:51:30] it is named 'deployment-prep' [11:51:38] oh [11:51:45] New patchset: Odder; "(bug 44899) Namespace setup for Korean Wikiversity" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59786 [11:52:02] Successfully added Mark Bergsma to deployment-prep. [11:52:02] !! [11:52:28] deployment-cache-mobile01 [11:53:06] that one runs out of the tip of production [11:53:14] i need one with puppetmaster self [11:53:21] oh god. :-( [11:53:23] currently mobile is pointing to another one : deployment-cache-varnish-t3 [11:53:51] I would like the instance to self update which is not yet possible with puppetmaster::self [11:54:04] so I wanted to migrate the mobile site to the deployment-cache-mobile01 instance [11:54:10] but then that means not being able to test out changes [11:54:17] so I should update puppetmaster:self :) [11:54:21] i don't see that instance [11:54:26] ah [11:54:31] deployment-varnish-t3 [11:55:02] how do I import that change now? [11:55:09] just fetch from gerrit? [11:56:16] why the fuck people write wrong doc :( [11:57:20] New patchset: Odder; "(bug 44899) Namespace setup for Korean Wikiversity" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59786 [11:57:39] so as root: [11:57:39] export GIT_SSH=/var/lib/git/ssh [11:57:40] cd /var/lib/git/operations/puppet [11:57:42] then fetch the change [11:57:55] I usually copy paste the 'checkout' line from the gerrit change [11:57:59] right [11:58:01] thanks :) [11:58:04] and add a -b 12345/12 [11:58:14] to craft a local branch named after the change + patchset [11:58:22] then puppetd -tv and the rest as usual [11:59:02] brb [12:00:07] awesome [12:00:09] New review: Odder; "Doing things the right way, i.e. removing NS_PROJECT definition from $wgExtraNamespaces and moving i..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59786 [12:00:09] it didn't do anything [12:00:19] that's a good start ;) [12:02:20] it is good to see you having interest in labs instances :D [12:02:46] heh [12:02:49] zeljkof runs selenium tests against beta. So that let us catch mediawiki issues before they got deployed [12:02:51] i have an interest in not breaking the site [12:03:02] and I almost rigged up something by doing my changes only in a large comment section [12:03:04] but that would be unfair [12:03:06] this is what beta is for ;) [12:03:37] exactly :-] [12:14:46] New patchset: Mark Bergsma; "Support per-backend options" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59999 [12:15:11] bbl [12:15:26] oh you made it after all [12:15:37] New patchset: Odder; "(bug 46534) Add namespace aliases for uz.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60002 [12:18:03] New patchset: Odder; "(bug 46846) Localise project namespaces for dv.wiktionary" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59998 [12:18:44] Geez, you can't do anything here without the whole channel knowing about it immediately. [12:18:55] :D [12:20:56] morning paravoid :-] [12:21:25] paravoid: so how would I get you to sponsor my packages ? :-] I am not sure what the process is or what you expect from me [12:22:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [12:24:55] the process is you give a slight hint about wanting to package something new and faidon comes enthusiastically running at you [12:25:10] back in an hour or so ;) [12:32:19] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [12:32:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:33:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [12:47:59] hashar: looking at statsd now [12:48:13] you know you can do --download-current-version ? [12:48:14] paravoid: the up to date changes are in svn [12:48:26] and get rid of the DEB_UPSTREAM_VERSION hacks? :) [12:48:27] paravoid: with uscan yeah [12:48:35] svn build package has an option to do it too [12:48:48] but it does not rename the tarball in build-area :( [12:48:51] yeah I'm talking about uscan [12:48:56] rename? [12:48:57] it symlinks [12:48:58] wfm [12:49:02] in tarballs [12:49:05] but not in build-area :( [12:49:09] at least on a precise instance [12:49:21] so when building out the package it can't find the tar ball :( [12:49:35] huh? [12:50:18] ohh [12:50:47] I see what you mean now [12:51:02] my rules get-orig-source: target needs a tweak [12:52:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.142 second response time [12:54:29] uscan --download-current-version --rename --destdir ../tarballs [13:00:18] paravoid: I have no idea how svn-buildpackage fetch the sources [13:00:23] apparently we have to uscan first [13:00:58] that's what I mean [13:01:16] so apparently I have to do: [13:01:21] ./debian/rules get-orig-source && svn-buildpackage [13:01:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:02:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [13:03:05] that works [13:03:12] I usually do uscan manually [13:03:16] anyway, are you commiting that? [13:03:28] ditching DEB_UPSTREAM_VERSION and using --download-current-version [13:03:44] also, on d/copyright, the Copyright (c) 2012-2013, James Socol [13:03:51] isn't needed, as you have it two lines above [13:04:17] and while at it, I tend to license debian/* with the same license as upstream to be a good citizen, but that's entirely you decision of course [13:06:09] yeah [13:06:52] otherwise looks good, do those and I'll upload [13:07:22] paravoid: isn't the copyright part of the license ? [13:07:30] no [13:08:33] paravoid: diff review : http://paste.openstack.org/show/36385/ ;) [13:09:12] with colors http://paste.openstack.org/show/36386/ [13:26:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [13:32:59] New patchset: Faidon; "icinga: authorize faidon for info/commands" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60007 [13:33:04] hashar: ack [13:33:12] paravoid: sending to svn [13:33:49] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60007 [13:34:11] New patchset: Mark Bergsma; "Support per-backend options" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59999 [13:34:28] paravoid: sent http://anonscm.debian.org/viewvc/python-modules?view=revision&revision=23970 [13:35:09] New patchset: Mark Bergsma; "Support per-backend options" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59999 [13:35:25] and there is zero lintian issue :-] [13:36:29] I know [13:36:33] this isn't my first upload you know :) [13:37:47] quick mailbox count shows ~320 [13:38:49] maybe I should subscribe to the list of debian-python commits [13:39:27] voluptuous has a different get-orig-source [13:39:35] yeah haven't fixed that one [13:41:18] can it make it into wheezy? [13:41:48] no [13:42:05] wheezy is frozen since July [13:42:16] so no new packages since then [13:42:16] july? holy fuck [13:42:33] wheezy gets released on May 4th/5th [13:43:15] New patchset: Mark Bergsma; "Support per-backend options" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59999 [13:43:30] paravoid: I have fixed the voluptuous target for orig-get-source [13:43:40] get-orig-source [13:43:41] rhgr [13:44:03] New patchset: Mark Bergsma; "Support per-backend options" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59999 [13:44:54] hashar: statsd ftbfs, it needs setuptools to build but it's not in B-D [13:45:12] warn: can't parse acronym ftbfs [13:45:20] warn: can't parse acronym B-D :-] [13:45:24] haha [13:45:30] ah hmm [13:45:31] fails to build from source [13:45:34] Build-Depends [13:45:46] let's make up our own acronyms too hashar [13:45:52] and then annoy paravoid with it [13:45:54] ftbfs is quite common [13:45:55] annoying debianisms ;p [13:46:03] common in the grand world of debian [13:46:06] there's even a wikipedia article about it! [13:46:09] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [13:46:11] oh then [13:47:21] paravoid: I did not catch that issue while building in my debian/unstablve vbox [13:47:55] ah B-D for setuptools is set in voluptuous [13:47:59] must have build that one first [13:48:04] * hashar should use a clean chroot [13:48:11] yes you should :) [13:48:14] pbuilder [13:50:14] B-D added with r23972 [13:51:53] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59854 [13:52:15] New patchset: Mark Bergsma; "Support per-backend options" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59999 [13:55:03] New patchset: Mark Bergsma; "Support per-backend options" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59999 [13:56:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:57] New patchset: Mark Bergsma; "Support per-backend options" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59999 [13:57:06] hashar: both uploaded [13:57:09] so, [13:57:14] what happens next? [13:57:19] these are new packages, i.e. both new source & binaries [13:57:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [13:57:24] this means that they get into a queue called NEW [13:57:38]