[00:00:06] New patchset: awjrichards; "Revert "Revert "Revert "Enable CentralNotice on mobile everywhere CentralNotice is enabled"""" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60611 [00:01:20] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60611 [00:02:43] !log awjrichards synchronized wmf-config/InitialiseSettings.php 'Disable mobile sitenotice everywhere but testwiki' [00:02:49] Logged the message, Master [00:03:06] !log awjrichards synchronized wmf-config/mobile.php 'Disable mobile sitenotice everywhere but testwiki' [00:03:13] Logged the message, Master [00:03:59] New patchset: Dzahn; "delete outdated apache configs for outreachcivi (RT-5002) and ocs.wikimania2009 they still resolve(d) to singer, but they were not enabled apache sites anyways" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60612 [00:07:34] RECOVERY - Disk space on ms-be1009 is OK: DISK OK [00:08:54] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 224 seconds [00:09:24] RECOVERY - Disk space on ms-be1011 is OK: DISK OK [00:14:54] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 230 seconds [00:15:54] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 5 seconds [00:23:54] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 223 seconds [00:26:54] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 14 seconds [00:33:54] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 224 seconds [00:35:31] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 19 seconds [00:38:31] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 199 seconds [00:40:31] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 21 seconds [00:43:33] Change abandoned: Spage; "I didn't LevelUp. #wikimedia-e3 has a bot logging thanks to the work of others." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/46672 [01:58:29] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 184 seconds [02:00:29] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 19 seconds [02:18:59] !log LocalisationUpdate completed (1.22wmf2) at Wed Apr 24 02:18:59 UTC 2013 [02:19:07] Logged the message, Master [02:31:44] !log LocalisationUpdate completed (1.22wmf1) at Wed Apr 24 02:31:43 UTC 2013 [02:31:52] Logged the message, Master [02:54:13] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 193 seconds [02:55:13] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 6 seconds [03:42:13] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 24 03:42:12 UTC 2013 [03:42:20] Logged the message, Master [03:44:10] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 238 seconds [03:45:10] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 6 seconds [04:13:47] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 217 seconds [04:17:47] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 21 seconds [04:28:47] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 217 seconds [04:31:47] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 19 seconds [04:34:47] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 199 seconds [04:38:30] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [04:38:30] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [04:38:41] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 216 seconds [04:43:41] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 18 seconds [04:49:41] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 189 seconds [04:53:41] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 217 seconds [04:54:41] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 18 seconds [05:13:48] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 217 seconds [05:14:48] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 24 seconds [05:29:18] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [05:34:50] New patchset: Asher; "every-15s stat collection + every-60s reporting = under reporting rate based stats by 4x" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60626 [05:35:30] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60626 [06:01:00] New patchset: Tim Starling; "Set $wgTorBlockProxy to url-downloader.wikimedia.org:8080" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60424 [06:01:06] Change merged: Tim Starling; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60424 [06:03:43] !log tstarling synchronized wmf-config/CommonSettings.php [06:03:51] Logged the message, Master [06:07:37] New patchset: Tim Starling; "Move tor_exit_node from hume to terbium now proxy config exists" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60427 [06:07:52] New review: Tim Starling; "Tested." [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/60427 [06:07:53] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60427 [06:09:14] PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 10 hours [06:10:14] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [06:12:58] New patchset: Tim Starling; "Fix typo in parameter name" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60628 [06:13:43] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60628 [06:29:35] RECOVERY - Puppet freshness on terbium is OK: puppet ran at Wed Apr 24 06:29:29 UTC 2013 [06:34:24] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 218 seconds [07:09:28] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 19 seconds [07:17:28] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 192 seconds [07:17:28] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 193 seconds [07:19:28] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 194 seconds [07:19:28] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 195 seconds [07:20:28] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 3 seconds [07:20:28] RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds [07:28:28] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 198 seconds [07:29:28] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 259 seconds [08:06:05] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 210 seconds [08:06:05] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 211 seconds [08:17:05] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 233 seconds [08:24:05] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 236 seconds [08:39:10] mark: I could use a simple merge, changing a package version for contint : https://gerrit.wikimedia.org/r/#/c/60435/ [08:39:18] oh and hello :-) [08:42:47] RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay NULL seconds [08:44:07] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:07] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:08] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [08:44:17] PROBLEM - MySQL Slave Running on db78 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: No Last_Error: Query partially completed on the master (error on master: 1317) and w [08:47:20] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60435 [08:48:01] hashar: ^^ [08:48:13] apergos: will run puppet on gallium. thx! [08:48:38] yw [08:50:07] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay NULL seconds [08:52:07] PROBLEM - MySQL Slave Running on db1025 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: No Last_Error: Query partially completed on the master (error on master: 1317) and w [08:57:52] !log upgrading Zuul [08:58:01] Logged the message, Master [09:05:06] !log restarted Zuul. Upgraded to e403579 (latest master) [09:05:14] Logged the message, Master [09:13:05] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [10:50:40] New patchset: ArielGlenn; "ms-be9 to correct stanza with new disk layout" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60652 [10:56:22] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60652 [11:32:10] Hello, I am trying to find out how core dumps work in wikimedia. I would like to ask how I should compare 2 xml wikimidia source files I have downloaded from 2 different days? (When I open the xml via a browser,FF,Chromium they crash even with 25mb) [11:36:11] sib_: what xml files? [11:38:20] elwikisource-20130326-stub-meta-history for example [11:38:45] and a later version for comparison [11:38:53] ok well opening them in a web browser is going to kill the browser [11:39:00] these are plain text files (after uncompression) [11:39:05] well in XML markup [11:39:15] what do you want to comapre in these files? [11:39:22] *compare [11:39:58] I want to open two different dates and see the differences in the xml dumps [11:40:14] ah [11:40:18] I have found various tools for this but imagined that you all use something by defualt [11:40:29] *default [11:40:46] we don't really [11:41:13] as long as you undersand the format of them, I would just do a diff and look at things moved or added or whatever [11:41:59] Oook, will try a few methods and post back if I need any help [11:42:24] sure [11:51:01] New review: Demon; "PositionalScorer: Eclipse was yelling about dead code, and I figured it's basically harmless." [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/60444 [11:59:34] apergos: swift? [12:00:09] paravoid: trying to get puppet or me to quit being stupid about two of these partitions [12:00:34] I swear this used to work and yet right now I cannot get the swift labelling to take [12:00:59] I ran the test by hand that's required for the labelling, and it works fine [12:01:54] I don't understand a thing :) [12:03:02] :-D [12:03:09] node /^ms-be(9|11|12)\.pmtpa\.wmnet$/ { [12:03:17] swift::label_filesystem{ '/dev/sda3': } [12:03:22] those (in site.pp) [12:04:22] root@ms-be9:~# /usr/sbin/xfs_admin -l /dev/sda3 [12:04:22] label = "" [12:04:32] you can see it didnt do them [12:06:22] and the other fun fact is I have to go ask the lawyer (now) to handle the next paperwork, the people at the offices don't want to give me the time of day [12:08:45] New patchset: Mark Bergsma; "Do not cap the frontend cache" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60655 [12:09:01] New patchset: Mark Bergsma; "Do not cap the frontend cache TTL" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60655 [12:09:56] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60655 [12:10:23] all right let me go do this thing with the paperwork and be right back, maybe I'll see the problem with a fresh look in a little. [12:10:26] >_< [12:14:51] it now does the right thing [12:14:59] I think you didn't merge puppet properly [12:15:11] it was there on sockpuppet but maybe it wasn't on stafford [12:15:24] I logged in to see, but I was too late and it worked, probably mark merged his :) [12:18:04] yeah looks like I merged a site change [12:23:55] morning cmjohnson1 [12:24:06] good morning [12:40:54] ok. I did merge it with puppet-merge and it *said* that it was ok, I even backscrolled to double check after I ran into the issue [12:40:56] who knows [12:41:04] but yay for it now "just working" [12:41:13] did you forward your agent? [12:41:23] yes [12:42:13] whatever, if it's fixed now good [12:43:08] tip: you can do puppetd -vt --debug --trace [12:43:10] and see what it does [12:44:03] I did debug (but not trace) and didn't see any references to the label check [12:44:16] that was where I was feeling like it was a brick wall [12:59:42] New patchset: Hashar; "(bug 41285) adapt `foreachwiki` for labs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55059 [13:05:15] New patchset: Hashar; "(bug 41285) adapt `foreachwiki` for labs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55059 [13:06:11] New review: Hashar; "Finally came back to that bug and updated the foreachwiki script to uses getRealmSpecificFilename." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/55059 [13:19:55] New patchset: Odder; "(bug 47383) Update Apple Touch icon for Wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60658 [13:45:35] odder: also see https://bugzilla.wikimedia.org/show_bug.cgi?id=27911 [13:46:59] Oh noes. [13:47:05] :-) [13:47:48] !b 44680 | Nemo_bis [13:47:48] Nemo_bis: https://bugzilla.wikimedia.org/44680 [13:48:13] You want to have that fixed for Meta, or to have the general format changed? :-) [13:48:26] foundationwiki is also broken in the same way [13:48:55] odder: general format [13:49:08] ok, wrote that in bz [13:51:01] paravoid: adding per-wiki icons is one option [13:51:08] for meta and foundation [13:51:29] the other wikis shouldn't be broken that way [13:53:20] 'wiktionary' => '//$lang.$site.org/apple-touch-icon.png', hmmm [13:54:15] commons isn't broken but it's wrong too [13:54:26] it links to commons.wikipedia.org, which redirects [13:55:01] http://commons.wikimedia.org/apple-touch-icon.png looks fine [13:55:15] [13:55:29] commons.wikiPedia.org, but it redirects [13:55:43] all the rest *.wikimedia.org wikis are broken [13:55:46] :-( [13:56:05] officewiki etc. [13:56:16] have fun! [13:56:18] noone cares about that one [13:56:18] :P [13:57:24] that's besides the point though [13:57:40] the fact that there are dozens of *.wikimedia.org wikis means that the logic there is flawed [13:58:09] I'm not saying someone should create touch icons for everything, just to stop giving link rels to non-existent URLs [13:59:54] 'wikimedia' => '//$lang.wikimedia.org/apple-touch-icon.png' [14:00:46] would this work for, say, Outreach? [14:02:56] I don't think so [14:03:05] http://noc.wikimedia.org/conf/highlight.php?file=langlist [14:03:08] apparently not. [14:03:27] Actuall,y maybe [14:03:28] 'wikimedia' => '//$lang.wikimedia.org', [14:07:42] Ah yes. [14:08:04] Then there is no reason not to set Apple Touch icons the same way that favicons are set. [14:22:10] Hm. So what is the result of the discussion about the role vs module vs module roles discussion? I'm pretty sure we didn't agree on anything. :-) [14:23:49] paravoid: can this be fixed by linking to non-existing files instead? :) [14:26:34] paravoid: hi :) I have upgraded Zuul this morning thanks to all the packaging work! [14:39:20] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [14:39:20] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [14:42:39] andrewbogott_afk: Ping me when you get back. [15:05:00] !log maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/skins/MinervaTemplate.php 'bug 47597' [15:05:08] Logged the message, Master [15:09:13] !log maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/skins/MinervaTemplate.php 'bug 47597' [15:09:20] Logged the message, Master [15:18:31] New review: coren; "(1 comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59969 [15:29:20] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [15:45:09] New patchset: Odder; "(bug 27911) Standardise Apple Touch icons for all projects" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60670 [15:45:30] paravoid: ^^ [15:48:06] Change abandoned: Odder; "This will be fixed with Ib5554c547295d9b53b4f554e2773a4d7c14f8c6a" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60658 [15:55:20] New patchset: Odder; "(bug 27911) Standardise Apple Touch icons for all projects" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60670 [15:58:26] New review: Faidon; "Please move the Wiktionary one to bits too." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/60670 [15:58:43] New patchset: Lcarr; "rfaulk is no longer with company" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60674 [16:00:38] paravoid: ? [16:00:42] ? [16:01:14] odder: what? [16:01:22] Please move the Wiktionary one to bits too. [16:01:25] yes [16:01:30] don't use upload [16:01:34] docroot/bits/favicon/apple-touch/wiktionary.png [16:01:35] renamed from docroot/wiktionary.org/apple-touch-icon.png [16:01:56] enwiktionary, sorry. [16:02:01] my bad [16:02:01] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60674 [16:02:07] hmmm [16:02:55] ideally they should have one for all languages, but I don't particularly care [16:02:57] paravoid: so you're suggesting we should not be loading stuff from Commons? [16:03:03] I am [16:03:07] not for this anyway [16:03:09] paravoid: ah yes, that's the problem with Wiktionary [16:03:39] they are very serious about their logos. And almost every major Wiktionary has a different logo... [16:04:14] people like being a pain I guess :) [16:04:32] add it to bits as /enwiktionary.png? [16:04:52] anyway, I'm no developer nor do I have any say in this policy or mediawiki matters usually [16:05:13] but having every icon in bits but this is just wrong :) [16:05:30] paravoid: you /do/ know that we're loading wgLogo from Commons too? :) [16:05:35] I do [16:06:38] I don't really care where this particular logo will be, but am trying to look a few steps forward... [16:06:54] why not move it where everything else? [16:07:01] i.e. enwiktionary.png? [16:08:03] also, http://upload.wikimedia.org/wikipedia/test/apple-touch-icon.png doesn't exist :) [16:08:31] LOL [16:09:04] that's one of the problems with using upload [16:09:14] Why defining it then? :-) [16:09:21] it used to exist [16:10:30] used to be this http://noc.wikimedia.org/testwiki-apple-touch-icon.png [16:11:14] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:09] ah well. [16:19:21] Since they're not using it now, I'll let them define it when they feel like it. [16:19:37] thanks for the help, paravoid [16:20:27] New patchset: Odder; "(bug 27911) Standardise Apple Touch icons for all projects" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60670 [16:31:03] New review: Odder; "The English Wiktionary icon's been moved to bits per Faidon's comment above." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60670 [17:13:08] RECOVERY - MySQL Slave Running on db1025 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [17:14:08] RECOVERY - MySQL Slave Running on db78 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [17:15:18] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 31020 seconds [17:16:18] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 30964 seconds [17:21:38] New patchset: RobH; "added magnesium netboot info" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60679 [17:23:21] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60679 [17:24:37] LeslieCarr: speaking of private keys, I had a Debian/Ubuntu buddy who would revoke his gpg signature on someone's key if he ever saw them leave their laptop unattended and unlocked at a conference. Should we do the same here at the office? ;) [17:24:51] or, he at least threatened to [17:25:01] we dont sign keys [17:25:02] heh [17:25:27] but, you do have the ability to revoke someon's access based on the assumption of a compromised private key [17:26:04] better said: "should we do similar with private keys here at the office?" :) [17:28:13] hehehe [17:28:38] eh, so far i have just tried to set their homepage and/or background to rick astley [17:28:57] haha [17:28:57] PROBLEM - Host ms-be2 is DOWN: PING CRITICAL - Packet loss = 100% [17:30:57] !log taking down ms-be2 for hw replacement [17:31:05] Logged the message, Master [17:31:07] PROBLEM - SSH on magnesium is CRITICAL: Connection refused [17:31:10] sbernardin: is that you taking ms-be2 down? [17:31:19] okay cool [17:31:24] yes [17:31:34] New one is up [17:31:49] sbernardin: cool [17:31:54] Putting in ticket for network now [17:32:26] thx [17:36:49] New patchset: MaxSem; "Create role::logging::eventlogging" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60680 [17:43:33] PROBLEM - NTP on magnesium is CRITICAL: NTP CRITICAL: No response from NTP server [17:45:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:45:46] greg-g: who was that? :) [17:46:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [17:46:15] your buddy, if I may ask [17:46:30] paravoid: dholbach [17:52:23] RECOVERY - SSH on magnesium is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [17:54:12] New patchset: Aude; "Enable data transclusion and site link widget on all wikipedias" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60682 [18:00:07] sbernardin: okay i wanted to check and make sure db59 was not a master. in case it is we would probably wait until we had blessing from asher or peter. but in this case we are replacing a single failed disk so not too worried [18:00:12] which disk are you going to replace? [18:01:02] sbernardin ^ [18:05:31] cmjohnson1: slot 11 [18:05:37] Last disk [18:06:37] okay...i see it slot 11...so log that you are going to replace failed disk on db59 slot 11 and then swap..ping me after so i can verify the rebuild. [18:06:44] sbernardin ^ [18:06:46] cmjohnson1: according to the megacli [18:06:58] yep [18:08:09] sbernardin: the disk is hot swap so it's okay to remove the failed disk and replace with a new one [18:08:35] OK [18:08:56] !log replacing bad disk on db59 [18:09:04] Logged the message, Master [18:09:16] !log DNS update - removing unused remnants [18:09:24] Logged the message, Master [18:13:23] cmjohnson1: replacement drive has been installed [18:15:33] sbernardin: cool...i see Firmware state: Rebuild [18:20:51] New review: Dzahn; "remove --purge :)" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/60612 [18:20:52] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60612 [18:28:27] New patchset: RobH; "adding git and gerrit wikimedia.org.pem files" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60687 [18:29:51] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60687 [18:32:33] New patchset: Demon; "Initial puppetization for git.wikimedia.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59857 [18:32:33] New patchset: Demon; "Remove old github detection hack, that's what remoteNameStyle is for" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59928 [18:32:33] New patchset: Demon; "Begin replicating all repositories to antimony" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59927 [18:32:34] New patchset: Demon; "Swap gerrit to new gerrit.wikimedia.org certificate" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60688 [18:33:25] ^demon: So the keys are in the private repo and the pems are in the public [18:33:30] you should be all set [18:34:05] <^demon> Thanks so much. I think my puppet manifests are all ready now too. [18:38:23] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non enwiki to 1.22wmf2 [18:38:31] Logged the message, Master [18:39:28] RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds [18:41:08] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60682 [18:44:18] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [18:44:18] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [18:44:19] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [18:45:24] !log reedy synchronized wmf-config/ 'wikidata stuffs' [18:45:31] Logged the message, Master [18:47:55] New patchset: Demon; "Don't bother restarting Gerrit for replication.config changes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60690 [18:48:11] binasher: yurik and i are trying to track down why a 301 is happening. when someone comes from a zero partner network (or forges the X-Carrier and X-Subdomain headers) and goes to http://en.zero.wikipedia.org a *301* is issued to redirect the user to http://en.m.wikipedia.org. do you know of anything in varnish or apache or general wikipedia.org configuration that would make this happen? haven't been able to pinpoint it in [18:48:12] mobile-frontend.inc.vcl.erb or main.conf…and grep '301' in MobileFrontEnd and ZeroRatedMobileAccess doesn't seem to help much, either :( [18:50:18] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 9 seconds [19:07:04] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50044 [19:13:12] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [19:16:21] !log racktables going offline a moment for upgrade [19:16:28] Logged the message, RobH [19:16:44] !log deploying Bugzilla wikimedia extension changes for improved wikilinking [19:16:51] Logged the message, Master [19:21:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [19:39:49] New review: Demon; "(1 comment)" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/60447 [19:40:44] <^demon> mutante: While you're on the subject of bots...https://gerrit.wikimedia.org/r/#/c/60317/ is pretty trivial. [19:47:20] RECOVERY - RAID on db59 is OK: OK: State is Optimal, checked 2 logical device(s) [19:50:27] Change abandoned: Umherirrender; "Has problems with UTF-8 characters (maybe a IDE problem), so this patch has bad side effects." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60447 [19:51:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:53:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [19:55:44] New patchset: Demon; "Turn mwsearch debug log back on" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60703 [19:56:48] !log aaron synchronized php-1.22wmf2/includes/filerepo/file/LocalFile.php 'deployed 0e42bdadd9349fe5b6119ff65322e6c52d813a6e' [19:56:56] Logged the message, Master [19:57:18] <^demon> Any root about who would mind doing `echo '' > /a/mw-log/mwsearch.log` on fluorine? We changed what we're logging and I'll be re-enabling it in a bit. [19:57:44] * AaronSchulz still can't view http://commons.wikimedia.org/wiki/Category:1877_books [19:59:23] AaronSchulz: saw you mention that before, is that reported in bz? part of something bigger? [19:59:49] I think it's just the usual djvu file BS [20:00:00] heh, ok (not that I know anything about that) [20:00:07] I don't think that category has a bug for itself [20:00:13] * greg-g nods [20:00:52] https://bugzilla.wikimedia.org/buglist.cgi?quicksearch=djvu%20file%20BS&list_id=196855 [20:02:50] !log demon synchronized php-1.22wmf2/includes/api/ApiQuerySearch.php 'Deploying Ib7abe8d5' [20:02:57] Logged the message, Master [20:03:58] !log demon synchronized php-1.22wmf2/extensions/MWSearch 'Deploying I48480cb6' [20:04:05] Logged the message, Master [20:06:19] New patchset: Matmarex; "(bug 45979) Set $wgCategoryCollation to 'uca-vi' on all Vietnamese-language wikis" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60705 [20:07:17] New review: Demon; "This can be merged now." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/58523 [20:07:30] New patchset: Demon; "Remove notice since the skins are now gone" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58523 [20:15:38] !log demon synchronized php-1.22wmf2/extensions/MWSearch [20:15:45] Logged the message, Master [20:19:07] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 237 seconds [20:21:07] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 4 seconds [20:21:53] New review: Dzahn; "sure, looks harmless, and is just about IRC output" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/60317 [20:21:54] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60317 [20:22:21] ^demon: done [20:22:34] <^demon> Ah, thanks. [20:24:07] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 184 seconds [20:25:32] !log racktables flap again, working on it [20:25:39] Logged the message, RobH [20:28:07] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 20 seconds [20:28:10] ^demon: re: 52251, i see it's merged, just fyi it also needs deployment, it's not auto-cloned by puppet [20:28:20] deploying it.. (CVE linker) [20:28:27] <^demon> Ok. [20:29:29] -our $VERSION = '0.01'; [20:29:29] +our $VERSION = '0.1'; [20:29:40] ehm, i don't see that change in that patch set [20:29:46] but seems really minor [20:30:35] sees more changes to link formats [20:32:22] !log racktables on hooper upgraded to recent stable [20:32:30] Logged the message, RobH [20:32:33] cmjohnson1: take alook, its a bit odd [20:32:38] slightly different [20:33:30] it is faster robh [20:34:05] looks a bit odd...sort of stripped down [20:34:12] ^demon: how does this look to you? http://paste.debian.net/107/ it's the diff between gerrit and prod currently [20:35:22] <^demon> lgtm [20:35:42] <^demon> AaronSchulz: I'm stumped on this stacktrace: http://p.defau.lt/?bCUbLpwJAmUpUYrX6PsKYQ [20:35:59] cmjohnson1: So I am going to puppetize and move it, so first step was ensuring we were able to run the most recent version [20:36:06] as i'll install recent version on new server [20:36:15] (you'll have to do this too for observium) [20:36:20] <^demon> I can't see any reason why $workStatus shouldn't be a Status. [20:36:40] robh: so can you walk it through w/me [20:36:50] andre__: ^ [20:36:57] * MaxSem pokes notpeter [20:38:19] mutante: Hello. What did I do wrong this time? [20:38:23] * andre__ too lazy to go through the backlog [20:38:56] andre__: not saying you did wrong:) we just have a diff between what is on prod and what is in gerrit in one case [20:38:59] Extension.pm [20:39:09] http://paste.debian.net/107/ [20:39:18] this is what i'd change right now if i copy the gerrit version over [20:41:18] can we just sync it and then see if anything needs additional fixes [20:41:21] ^demon: call_user_func_array can return null and give a warning on fatals [20:41:38] <^demon> Gotcha... [20:41:52] cmjohnson1: walk through what i do for racktables? [20:41:53] <^demon> So if a fatal happened inside the called function, we'd be hiding it with a null. [20:42:04] i plan to document the actual setup oon wikitech as well [20:42:19] robh: cool...than i will go off wikitech [20:42:34] since we are going to migrate some services without packaging them (like wordpress & racktables) [20:42:41] the trade off is if they arent inpuppet [20:42:42] ^demon: I bet that doesn't play well with wfSuppressWarnings() [20:42:45] they must be on wikitech [20:42:52] since puppet is somewhat self documenting. [20:43:30] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 181 seconds [20:43:49] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 211 seconds [20:44:59] New review: Krinkle; "Rename enwiktionary.png to wiktionary/en.png for consistency with bits.wm.o/favicon/ and that way th..." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/60670 [20:45:12] <^demon> AaronSchulz: Ideas on best way to handle? [20:45:27] * AaronSchulz was thinking [20:45:29] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 193 seconds [20:47:14] ^demon: testing locally, fatals seem to cause actually fatals [20:47:36] there is some case where it can be null though, trying to figure that out [20:50:05] no that's not even it [20:50:06] return $this->error( $status ); [20:50:37]