[00:00:48] (03PS1) 10Dr0ptp4kt: Add an extra header for cache variance of W0 banners for proxies. [operations/puppet] - 10https://gerrit.wikimedia.org/r/88261 [00:02:39] (03PS2) 10Dr0ptp4kt: Add an extra header for cache variance of W0 banners for proxies. [operations/puppet] - 10https://gerrit.wikimedia.org/r/88261 [00:08:02] (03PS3) 10Dr0ptp4kt: Add an extra header for cache variance of W0 banners for proxies. [operations/puppet] - 10https://gerrit.wikimedia.org/r/88261 [00:10:41] paravoid: link? [00:10:50] graphite.wm.org :) [00:11:10] !log all ams-ix traffic is now on cr2-esams [00:11:23] Logged the message, Mistress of the network gear. [00:11:33] under the "stats" cetegory [00:13:03] I see [00:16:10] PROBLEM - Puppet freshness on osm-cp1001 is CRITICAL: No successful Puppet run in the last 10 hours [02:06:00] Meh, hopefully one day this will be easier and doesn't require mocking parts of core and wmf-config: https://gist.github.com/Krinkle/6878261 [02:06:12] get site confifguration values for all (or some) wmf wikis [02:06:34] https://gist.github.com/Krinkle/6878261#file-usage-log [02:07:40] there is a getConfiguration.php maintenance script though. Which can build a cache per wiki (though that contains private data, so needs to be masked) [02:07:51] and there is sitematrix API which provides some of this data [02:08:32] what private data? [02:08:44] it exports all global variables [02:08:53] that match /^wm?g/ [02:09:03] so there's some passwords, certain tokens/salts and whatnot [02:09:52] hah [02:13:54] !log LocalisationUpdate completed (1.22wmf20) at Tue Oct 8 02:13:54 UTC 2013 [02:14:14] Logged the message, Master [02:18:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [02:19:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 13.172 second response time [02:23:04] (03CR) 10Springle: [C: 031] Labs DB: Add labsdb-side second line of defense [operations/software] - 10https://gerrit.wikimedia.org/r/88149 (owner: 10coren) [02:24:37] (03CR) 10coren: [C: 032 V: 032] "A +1 from Sean is like unto a touch from God, and good enough for me. :-)" [operations/software] - 10https://gerrit.wikimedia.org/r/88149 (owner: 10coren) [02:25:27] !log LocalisationUpdate completed (1.22wmf19) at Tue Oct 8 02:25:27 UTC 2013 [02:25:38] Logged the message, Master [02:27:13] PROBLEM - Puppet freshness on srv193 is CRITICAL: No successful Puppet run in the last 10 hours [02:31:53] !log start online optimize logging indexes s4 & s5 [02:32:04] Logged the message, Master [02:32:30] (03PS2) 10coren: Add templateeditor right, group, and restriction [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88196 [02:32:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [02:33:43] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 27.151 second response time [02:35:56] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Oct 8 02:35:56 UTC 2013 [02:36:09] Logged the message, Master [02:36:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [02:37:43] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 29.596 second response time [03:20:27] !log rmmod nf_conntrack on db1002 causing mass mysql connect failure [03:20:41] Logged the message, Master [03:23:19] !log xtrabackup clone s6 db1039 to db1022 [03:23:35] Logged the message, Master [03:28:35] (03PS1) 10Springle: s6 db1015 to full steam [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88400 [03:29:04] (03CR) 10Springle: [C: 032] s6 db1015 to full steam [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88400 (owner: 10Springle) [03:30:07] !log springle synchronized wmf-config/db-eqiad.php 's6 db1015 to full steam' [03:30:19] Logged the message, Master [04:12:58] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [04:44:53] what's up with the slaves of Wikidata? [04:47:23] which ones? [04:48:00] https://www.wikidata.org/w/api.php?action=query&meta=siteinfo&siprop=dbrepllag&sishowalldb= [04:49:57] * jeremyb suddenly wants ishmael... [04:50:18] i wonder if springle's there? [04:50:24] is he AEST? [04:50:28] no more lag [04:53:43] s5 (db1021 mainly) has been having issues. there's a optimize process running on logging tables right now, and i'm arranging for a spare slave [04:53:49] jeremyb: Jasper_Deng ^ [04:54:16] oh, the optimize [04:54:23] i was focusing on the xtrabackup :P [04:54:53] springle: s5 is much better now in dbtree [04:54:55] issues == slow queries causing max connections to be hit. related to https://bugzilla.wikimedia.org/show_bug.cgi?id=54876 [04:55:12] *click* [04:55:20] springle: is it AEST? [04:55:34] my tz? yes [04:56:28] huh. there's no A according to my system (wheezy) [04:56:47] $ for TZ in America/Los_Angeles Europe/Athens Australia/Sydney; do zdump -v "$TZ" | fgrep ' 2013 ' | tail -n 1; done | while read TZ line; do printf '%-23s %s\n' "$TZ" "$line"; done [04:56:51] America/Los_Angeles Sun Nov 3 09:00:00 2013 UTC = Sun Nov 3 01:00:00 2013 PST isdst=0 gmtoff=-28800 [04:56:54] Europe/Athens Sun Oct 27 01:00:00 2013 UTC = Sun Oct 27 03:00:00 2013 EET isdst=0 gmtoff=7200 [04:56:57] Australia/Sydney Sat Oct 5 16:00:00 2013 UTC = Sun Oct 6 03:00:00 2013 EST isdst=1 gmtoff=39600 [04:59:07] i use Australia/Brisbane which is EST +1000. AEST is only an abbreviation i think http://www.timeanddate.com/library/abbreviations/timezones/au/est.html [05:01:56] ok, so brisbane hasn't had DST 1992 [05:11:04] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [05:28:04] RECOVERY - Puppet freshness on srv193 is OK: puppet ran at Tue Oct 8 05:27:54 UTC 2013 [05:37:04] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 8 05:36:57 UTC 2013 [05:38:04] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [06:06:24] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 8 06:06:17 UTC 2013 [06:07:04] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [06:36:24] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 8 06:36:15 UTC 2013 [06:37:04] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [06:37:10] !log fixed up salt on mw1046, mw1072, mw1173 [06:37:26] Logged the message, Master [06:45:34] PROBLEM - Host wikiversity-lb.esams.wikimedia.org_ipv6 is DOWN: PING CRITICAL - Packet loss = 100% [06:45:36] PROBLEM - Host wikiquote-lb.esams.wikimedia.org_ipv6 is DOWN: PING CRITICAL - Packet loss = 100% [06:45:37] PROBLEM - Host bits-lb.esams.wikimedia.org_ipv6 is DOWN: PING CRITICAL - Packet loss = 100% [06:46:24] RECOVERY - Host wikiquote-lb.esams.wikimedia.org_ipv6 is UP: PING OK - Packet loss = 0%, RTA = 86.62 ms [06:46:26] RECOVERY - Host wikiversity-lb.esams.wikimedia.org_ipv6 is UP: PING OK - Packet loss = 0%, RTA = 86.62 ms [06:46:28] RECOVERY - Host bits-lb.esams.wikimedia.org_ipv6 is UP: PING OK - Packet loss = 0%, RTA = 86.24 ms [06:47:30] hm [07:05:54] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 8 07:05:44 UTC 2013 [07:06:04] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [07:11:48] (03PS1) 10ArielGlenn: depool db1040 for upgrade, conversion to mariadb [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88411 [07:16:52] (03CR) 10ArielGlenn: [C: 032] depool db1040 for upgrade, conversion to mariadb [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88411 (owner: 10ArielGlenn) [07:20:20] !log ariel synchronized wmf-config/db-eqiad.php 'depool db1040 for upgrade/conversion to mariadb' [07:20:34] Logged the message, Master [07:37:02] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 8 07:37:00 UTC 2013 [07:38:02] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [07:39:02] PROBLEM - mysqld processes on db1040 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [07:50:53] (03PS1) 10Springle: warm up db1039 in s6 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88413 [07:51:22] (03CR) 10Springle: [C: 032] warm up db1039 in s6 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88413 (owner: 10Springle) [07:52:02] !log springle synchronized wmf-config/db-eqiad.php 'warm up db1039 in s6' [07:52:15] Logged the message, Master [08:08:35] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [08:21:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [08:22:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 10.891 second response time [08:28:40] I wish PHP would stop segfaulting on terbium [08:28:41] Importing Svepmanschett_-_Livrustkammaren_-_34386-negative.tif.../usr/local/bin/mwscript: line 18: 906 Segmentation fault php "$MW_COMMON_DIR_USE/multiversion/MWScript.php" "$@" [08:35:55] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 8 08:35:51 UTC 2013 [08:36:35] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [08:41:17] (03PS1) 10ArielGlenn: fix up nagios contact name for ariel [operations/puppet] - 10https://gerrit.wikimedia.org/r/88418 [08:42:11] (03CR) 10ArielGlenn: [C: 032] fix up nagios contact name for ariel [operations/puppet] - 10https://gerrit.wikimedia.org/r/88418 (owner: 10ArielGlenn) [08:46:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [08:49:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 14.948 second response time [09:08:15] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 8 09:08:06 UTC 2013 [09:08:35] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [09:08:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [09:09:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 24.027 second response time [09:11:25] !log installing package upgrades on calcium (cameras) [09:11:38] Logged the message, Master [09:14:41] mutante: ??? [09:14:49] cameras ? [09:14:57] DC cameras ? [09:15:15] yes [09:15:37] i started going through a list of servers where it's not very obvious from site.pp what their status is [09:15:52] and this turned out to be the camera server setup by Rob [09:16:14] aha... please document :-) :-) :-) [09:16:30] yep ,ok:) [09:28:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [09:31:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 16.226 second response time [09:31:51] (03PS1) 10Odder: (bug 48480) Remove EmailCapture extension settings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88424 [09:37:35] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 8 09:37:30 UTC 2013 [09:38:35] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [09:39:48] !log puppetstoredconflicean .. killiam williams.wikimedia.org .. done [09:40:00] Logged the message, Master [09:40:36] arg cant type on this [09:40:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [09:42:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 13.113 second response time [09:43:51] !log installing package upgrades on antimony (gitblit, gerrit repl.) [09:44:02] Logged the message, Master [09:57:23] (03PS1) 10Reedy: Add ttf-sinhala-lklug to imagescalers [operations/puppet] - 10https://gerrit.wikimedia.org/r/88430 [09:57:41] mutante: ^ want an easy one? :) [09:58:17] Reedy_: ah a font .. yea [09:58:52] easy bug is easy [09:59:57] Reedy: N: Can't select versions from package 'ttf-sinhala-lklug' as it is purely virtual [10:00:12] fonts-lklug-sinhala - Unicode Sinhala font by Lanka Linux User Group [10:00:32] http://packages.ubuntu.com/precise/ttf-sinhala-lklug [10:00:38] Hmm [10:00:40] Are the others? [10:00:42] * Reedy looks [10:01:10] http://packages.ubuntu.com/precise/fonts-lklug-sinhala [10:01:32] Ahh [10:01:41] It would look like there's a few like this http://packages.ubuntu.com/precise/ttf-sil-yi [10:01:49] Which is a transitional package [10:02:32] Inst defoma (0.11.12ubuntu1 Ubuntu:12.04/precise [all]) [10:02:32] Inst fonts-lklug-sinhala (0.5.4-1 Ubuntu:12.04/precise [all]) [10:02:32] Inst x-ttcidfont-conf (32+nmu2 Ubuntu:12.04/precise [all]) [10:02:41] <- that's what it would do [10:02:53] for install ttf-sinhala-lklug [10:03:37] I'll modify that commit [10:03:47] eh and exactly the same for install fonts-lklug-sinhala [10:03:48] and see about cleaning the others up then to the newer packages [10:03:52] (03CR) 10Akosiaris: [C: 04-1] "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/88430 (owner: 10Reedy) [10:04:04] * Reedy eyes akosiaris [10:04:05] ah [10:04:06] but did we expect we still need defoma [10:04:13] so... same talk here :-) [10:04:13] i mean, we already have all those fonts [10:04:34] looked on mw75 [10:04:50] We can just stick defoma at the top of the list ;) [10:07:09] waits for gerrit to publish a comment [10:07:18] isn't defoma deprecated ? [10:07:37] i have no such package on wheezy... [10:07:44] http://packages.ubuntu.com/precise/defoma [10:07:50] Doesn't seem to be in ubuntu land [10:07:54] Maintainer: Ubuntu Developers [10:07:57] Original-Maintainer: Debian QA Group [10:08:02] http://packages.ubuntu.com/saucy/defoma [10:08:05] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 8 10:08:01 UTC 2013 [10:08:10] so... not in saucy... [10:08:35] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [10:08:38] which means that in the next LTS no defoma [10:08:40] disappeared in raring [10:08:53] nice... [10:08:54] (03CR) 10Dzahn: "root@mw75:~# apt-get -s install ttf-sinhala-lklug" [operations/puppet] - 10https://gerrit.wikimedia.org/r/88430 (owner: 10Reedy) [10:08:58] let's please avoid it :-) [10:09:20] why would it want defoma for this specific package [10:09:28] while all the other packages are already installed [10:09:33] and didnt pull it [10:09:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [10:10:15] The defoma dependancy just disappears [10:10:16] http://packages.ubuntu.com/raring/ttf-sinhala-lklug [10:11:18] Reedy: easy bug turns into a general cleanup again :) [10:11:28] just install the virtual package!!!!! [10:11:29] :p [10:11:57] https://wiki.debian.org/OldPkgRemovals [10:12:16] Could dump the list into etherpad and update them all [10:12:52] ttf-sinhala-lkmug ? [10:13:17] mutante: maybe because of recommends x-ttcidfont-conf [10:13:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 12.472 second response time [10:14:00] try a --no-install-recommends (we should have it in apt though) [10:14:27] (03CR) 10Dzahn: "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/88430 (owner: 10Reedy) [10:15:21] root@mw75:~# apt-get install -s --no-install-recommends ttf-sinhala-lkmug [10:15:27] Inst fonts-lklug-sinhala (0.5.4-1 Ubuntu:12.04/precise [all]) [10:15:27] Inst ttf-sinhala-lkmug (0.5.4-1 Ubuntu:12.04/precise [all]) [10:15:32] <- that looks good [10:15:53] same with "lklug" vs. "lkmug" [10:16:03] Inst fonts-lklug-sinhala (0.5.4-1 Ubuntu:12.04/precise [all]) [10:16:14] gets you that but not the ttf- package [10:16:15] PROBLEM - Puppet freshness on osm-cp1001 is CRITICAL: No successful Puppet run in the last 10 hours [10:18:16] I am afraid that list needs cleanup... even worse... I am afraid that for every font it will be a different name in 10.04 and 12.04 ... :-( [10:18:33] (03CR) 10Dzahn: "< akosiaris> mutante: maybe because of recommends x-ttcidfont-conf" [operations/puppet] - 10https://gerrit.wikimedia.org/r/88430 (owner: 10Reedy) [10:18:52] I don't think we've any machines on 10.04 this would be a problem for... [10:19:11] i was hoping to hear that :-) [10:19:26] but i am surprised i actually did [10:19:38] I know we do have some 10.04 machines around still [10:19:46] But nothing that's mass installed [10:19:54] If we did, mutante would be finding a big stick to hit them with [10:20:24] don't suppose we have a nice way to setting no recommends in the manifest? [10:20:35] not in the manifest [10:20:36] but [10:21:18] $ cat apt.conf.d/90no-recommends.conf [10:21:18] APT::Install-Recommends "0"; [10:21:28] http://serverfault.com/questions/280405/installing-open-vm-tools-in-ubuntu-via-puppet-whats-the-lesser-evil [10:22:00] stabby [10:23:58] http://projects.puppetlabs.com/issues/1766 [10:24:04] Control of recommends installation with aptitude provider [10:24:09] we should have no prolem anyway [10:24:11] Added by Nick Phillips almost 5 years ago. Updated almost 3 years ago. [10:24:17] we don't use the aptitude provider [10:24:32] and the apt provider does not install recommends ... problem solved :-) [10:24:33] ah true [10:24:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [10:24:50] 2 levels of dupe http://projects.puppetlabs.com/issues/4113 [10:24:52] wait, i didn't use aptitude to test though [10:24:57] just apt-get install [10:25:24] which is not what puppet does... it run apt-get install -R (or --no-recommends or smt)... [10:25:31] sweet [10:25:39] then problem solved indeed [10:26:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 16.790 second response time [10:27:33] so..it's ttf-sinhala-lkmug [10:27:37] Reedy: [10:29:53] Ya? [10:30:27] hai [10:31:29] Reedy: just change it to -lkmug instead of -lklug and we can merge it as puppet will not install the recommends [10:33:56] (03PS2) 10Reedy: Add ttf-sinhala-lklug to imagescalers [operations/puppet] - 10https://gerrit.wikimedia.org/r/88430 [10:34:15] Should I make a commit to update and bypass all those virtual packages? [10:34:25] Aghh, commit summary [10:35:02] (03PS3) 10Reedy: Add ttf-sinhala-lkmug to imagescalers [operations/puppet] - 10https://gerrit.wikimedia.org/r/88430 [10:35:10] (03PS1) 10ArielGlenn: db1040 -> mariadb and file_per_table [operations/puppet] - 10https://gerrit.wikimedia.org/r/88436 [10:35:50] (03CR) 10Dzahn: [C: 031] Add ttf-sinhala-lkmug to imagescalers [operations/puppet] - 10https://gerrit.wikimedia.org/r/88430 (owner: 10Reedy) [10:36:13] (03CR) 10Dzahn: [C: 032] Add ttf-sinhala-lkmug to imagescalers [operations/puppet] - 10https://gerrit.wikimedia.org/r/88430 (owner: 10Reedy) [10:36:24] (03CR) 10ArielGlenn: [C: 032] db1040 -> mariadb and file_per_table [operations/puppet] - 10https://gerrit.wikimedia.org/r/88436 (owner: 10ArielGlenn) [10:39:22] Reedy: re: patchset to update all. i think akosiaris would like that (some time) [10:39:34] looks at puppet on an imagescaler [10:43:11] They're not all transitional [10:43:11] http://packages.ubuntu.com/precise/ttf-alee [10:43:12] :/ [10:44:24] * Reedy starts looking [10:46:15] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 8 10:46:13 UTC 2013 [10:46:35] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [10:47:27] notice: /Stage[main]/Imagescaler::Packages::Fonts/Package[ttf-sinhala-lkmug]/ensure: ensure changed 'purged' to 'latest' [10:52:55] akosiaris: arr.. we still got defoma [10:53:00] the actual commandline from puppet: [10:53:04] Commandline: /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install ttf-sinhala-lkmug [10:53:23] Coren: https://bugzilla.wikimedia.org/show_bug.cgi?id=49350#c4 [11:01:45] (03PS1) 10Reedy: Update font packages to not use virtual packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/88441 [11:02:06] (03CR) 10jenkins-bot: [V: 04-1] Update font packages to not use virtual packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/88441 (owner: 10Reedy) [11:03:30] (03PS2) 10Reedy: Update font packages to not use virtual packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/88441 [11:06:17] (03PS3) 10Reedy: Update font packages to not use virtual packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/88441 [11:06:30] bloody whitespace [11:06:40] (03CR) 10jenkins-bot: [V: 04-1] Update font packages to not use virtual packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/88441 (owner: 10Reedy) [11:07:25] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 8 11:07:22 UTC 2013 [11:07:32] (03PS4) 10Reedy: Update font packages to not use virtual packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/88441 [11:07:34] the package is on all scalers for now they can test if bug is fixed [11:07:35] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [11:11:27] Thanks [11:20:34] what is wrong with the virtual packages ? :( [11:35:52] mutante: arg... [11:37:15] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 8 11:37:05 UTC 2013 [11:37:35] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [11:38:22] mutante: could we use salt to remove defoma ? [11:38:36] Wouldn't puppet just end up installing it again? [11:38:54] yes but i am writing a patch to avoid that next time [11:38:59] aha :) [11:39:19] Out of interest, what would be the easiest way to find out how much out traffic dataset2 has a month? [11:39:22] akosiaris: yes, dsh is also easy enough since there is an existing group for image_scalers [11:39:33] while i dont think we have grains for that [11:39:49] and the hostnames are just mw's [11:39:50] mutante: ok... dsh it is [11:40:12] I 'll handle it [11:40:19] thx [11:41:07] bash history of fenari root [11:51:03] :-) [11:51:25] (03PS1) 10Akosiaris: apt-get should not install recommends [operations/puppet] - 10https://gerrit.wikimedia.org/r/88446 [11:57:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [11:58:41] Reedy: er ganglia graphs? [11:59:26] maybe not, that's going to count other network traffic [12:00:00] no... /modules/wikidata_singlenode/files/simple-elements.xml must die [12:00:23] reason: all of a sudden i cant easily grep for our server names in puppet repo :p [12:00:38] because it's an unrelated file full of element names [12:02:29] paravoid: are you a user on host "copper"? (swiftrepl) [12:08:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 27.420 second response time [12:13:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [12:14:08] | grep -v simple-elements :-P [12:14:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 12.844 second response time [12:14:55] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 8 12:14:46 UTC 2013 [12:15:35] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [12:17:45] !log installing package upgrades on ekrem (IRC) [12:18:00] Logged the message, Master [12:18:18] (03CR) 10Akosiaris: [C: 032] apt-get should not install recommends [operations/puppet] - 10https://gerrit.wikimedia.org/r/88446 (owner: 10Akosiaris) [12:20:20] created new wikitech pages for ekrem, chromium, capella, carbon, calcium, .. [12:21:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [12:22:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 29.688 second response time [12:23:55] mutante: yippi [12:25:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [12:30:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 18.767 second response time [12:33:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: Connection timed out [12:34:59] !log apt-get purge defoma on image_scalers dsh group [12:35:14] Logged the message, Master [12:35:21] !log install package upgrades on gallium (jenkins) [12:35:34] Logged the message, Master [12:35:42] mutante: argh [12:35:46] mutante: are you upgrading jenkins ? :D [12:36:10] ah no [12:36:27] hashar: no, just python, linux-tolls-common, some libs [12:36:34] w3m :p [12:36:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 23.349 second response time [12:36:43] :D [12:37:48] pfff [12:37:52] I restarted jenkins by mistake [12:37:58] !log restarted Jenkins by mistake :-( [12:38:14] Logged the message, Master [12:38:19] sorry if i should have asked, they seemed harmless and unrelated [12:39:05] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 8 12:38:57 UTC 2013 [12:39:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: Connection timed out [12:39:35] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [12:40:38] mutante: yeah yeah that is harmless usually :-] [12:40:51] mutante: just wanted to make sure jenkins was not going to restart [12:40:55] but I did kill it by mistake :-(((((((((((( [12:41:17] :p no worries, i wouldn't touch jenkins itself without pinging you at least [12:43:51] actually.. instead of those wikitech templates for servers we could use wikidata right away :p [12:49:36] !log installing package upgrades on formey [12:49:46] Logged the message, Master [12:52:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 20.908 second response time [12:52:38] hashar: ^ while we're at it, that upgraded and reloaded apache on formey (svn/gerrit) [12:52:48] status OK [12:59:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [13:00:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 23.961 second response time [13:03:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [13:07:47] (03PS1) 10Akosiaris: elasticsearch plugins in git-deploy [operations/puppet] - 10https://gerrit.wikimedia.org/r/88455 [13:09:05] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 8 13:08:59 UTC 2013 [13:09:35] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [13:13:11] https://wikitech.wikimedia.org/w/index.php?title=Template%3AServer&diff=85340&oldid=70064 [13:13:49] https://wikitech.wikimedia.org/wiki/Special:NewPages [13:15:38] (03CR) 10Odder: "See I3aee7b08501435f1248037eba66912436720bf4d for a follow-up." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88424 (owner: 10Odder) [13:20:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 11.391 second response time [13:22:16] mutante: I am not sure what is left on formey [13:22:37] mutante: ahhh svn [13:22:37] hashar: svn and gerrit repl. destination [13:23:10] mutante: it might be a preproduction gerrit install. [13:23:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [13:23:46] mutante: we will have to migrate it to eqiad eventually :-D [13:24:22] role::deployment::test ? [13:25:19] hashar: at least the last svn users are gone :P [13:25:25] pywiki [13:25:29] yup [13:25:35] mutante: yes that exists too [13:25:44] you might want to follow up with Chad. I get access on formey but I merely use it to query ldap [13:25:56] Ryan created it while giving me a crash course on git-deploy [13:27:36] akosiaris: helium could use use an apt-get upgrade (it's the bacula box) [13:27:45] mutante: go for it [13:27:49] kk [13:28:09] !log installing package upgrades on helium [13:28:20] Logged the message, Master [13:29:10] mutante: we can upgrade carbon anytime since brewster is still primary tftp server [13:30:04] cmjohnson1: alright, cool so i guess reinstall ? [13:30:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 25.261 second response time [13:30:53] I don't see why we couldn't [13:31:02] ok [13:32:43] mark: do you want me to move dysprosium to c8? or wait for new 10g rack in D? [13:33:06] is it in the way of anything right now? [13:33:16] no [13:33:17] we haven't really assigned it to anything new yet [13:33:26] i'd say leave it for now [13:33:52] kk [13:35:03] (03PS2) 10Akosiaris: elasticsearch plugins in git-deploy [operations/puppet] - 10https://gerrit.wikimedia.org/r/88455 [13:37:45] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 8 13:37:40 UTC 2013 [13:38:35] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [13:38:48] !log installing package upgrades on hooper [13:39:00] Logged the message, Master [13:42:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [13:46:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 25.218 second response time [13:52:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [13:53:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 19.202 second response time [13:58:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [14:00:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 26.354 second response time [14:08:55] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 8 14:08:45 UTC 2013 [14:09:35] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [14:20:49] where is mark? [14:21:59] Snaps: need him in particular? [14:22:26] also, there's 3 different marks. you mean the one that goes by just "mark"? [14:22:37] Mr Bergsma [14:22:58] Im mostly curious, but I wouldnt mind him reviewing some stuff. [14:26:53] Is it in gerrit? [14:27:03]