[07:40:54] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [07:49:02] PROBLEM - ircecho_service_running on neon is CRITICAL: Connection refused by host [07:49:12] PROBLEM - MySQL disk space on neon is CRITICAL: Connection refused by host [08:10:42] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: Puppet has not run in the last 10 hours [08:10:42] PROBLEM - Puppet freshness on amssq46 is CRITICAL: Puppet has not run in the last 10 hours [08:10:42] PROBLEM - Puppet freshness on ms6 is CRITICAL: Puppet has not run in the last 10 hours [08:10:42] PROBLEM - Puppet freshness on ssl3003 is CRITICAL: Puppet has not run in the last 10 hours [08:11:42] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [08:11:42] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: Puppet has not run in the last 10 hours [08:11:42] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [08:11:42] PROBLEM - Puppet freshness on amssq32 is CRITICAL: Puppet has not run in the last 10 hours [08:11:42] PROBLEM - Puppet freshness on amssq36 is CRITICAL: Puppet has not run in the last 10 hours [10:35:28] New patchset: Ori.livneh; "Fix use of deprecated configs for Extension:RSS" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52221 [13:33:46] New patchset: Silke Meyer; "Install Solr and Solarium on Wikidata test repos." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52043 [13:34:09] MaxSem: ^^ [13:35:13] mhm [13:35:45] what errors do you see, Silke_WMDE [13:35:46] ? [13:36:45] puppet tries do find the schema before downloading WikibaseSolr. It looks like this: err: /Stage[main]/Solr::Config/File[schema]: Could not evaluate: Could not retrieve information from environment production source(s) file:/srv/mediawiki/extensions/WikibaseSolr/schema.solr3.xml at /etc/puppet/modules/solr/manifests/init.pp:52 [13:38:25] mmm, have you tried applying that schema the same way other solr users do? [13:39:00] how do others apply it? [13:39:15] from a puppet module [13:39:46] that's what I'm doing (at least I think it is what I'm doing) [13:40:01] no, you're trying to do it from a local file [13:40:08] ah [13:42:06] you mean I should try to put the schema file as a file into puppet? [13:42:14] yes [13:42:40] after all, nobody will permit you storing configuration in another repo in production [13:43:39] ah ok! I'll try that. Thanks. (These puppet files for Wikidata are not used in real production they are for testing Wikidata in Labs.) [13:45:01] if you just want to try some stuff, is puppetization required? [13:47:19] We had discussions about this and then decided to do it ot be able to "click" test instances with little manual setup work. [14:05:12] MaxSem: \o/ yay, the error is gone! [14:05:54] aha;) [14:18:57] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50454 [15:09:13] New patchset: Silke Meyer; "Install Solr and Solarium on Wikidata test repos." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52043 [15:10:52] New review: Silke Meyer; "Do not merge. Now this works from the puppet side but the update script fails to create some Wikibas..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/52043 [15:11:41] New patchset: MaxSem; "Postgres module for OSM" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36155 [15:11:58] New patchset: Ottomata; "mod. update config info for E3 user metrics deployment." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/51615 [15:13:35] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/51615 [15:15:03] New patchset: Ottomata; "Removing unused variables in metrics_api in statistics.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52242 [15:16:11] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52242 [16:02:58] ori-l: that ALTER didn't change the constituent tables, right? (maybe you want to change them too?) [16:09:50] Can someone check with the value of $wgAutopromoteOnce['onEdit']['autoreview'] is on trwiki? [16:17:10] what the value* [16:42:18] New patchset: Matthias Mullie; "Remove ArticleFeedback from enwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52246 [16:47:47] New patchset: Ottomata; "Adding puppet Limn module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49710 [16:55:37] !log mw1085 powering off to troubleshoot DIMM error [16:55:42] Logged the message, Master [16:56:52] New patchset: Ottomata; "Adding puppet Limn module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49710 [16:58:40] I'm an OTRS admin - I have a user who lost access to their otrs-wiki account - no longer has access to the address in the prefs. Is there any way I can have an op change it? :) [17:13:11] RD: yeah, sure. update it on LOA? [17:13:24] erm [17:13:29] I need it changed in their preferences [17:13:33] i know [17:13:48] but so you don't have to state their address in public in a logged channel [17:13:48] It is updated on the LOA, and on OTRS [17:13:56] ok [17:14:01] of course I would PM the op, jeremyb_ [17:14:09] * RD isn't new to privacy :P [17:14:17] doesn't need an op [17:14:19] just shell [17:14:26] Jeff_Green: If you have a minute, my request ^^ up there [17:14:44] ah, yeah, jeff may be the right TZ by now [17:15:11] RD I think this is a job for philippe [17:15:51] Jeff_Green: needs a cluster shell user. you mean to approve it? [17:16:11] to approve it [17:16:20] IMHO, it doesn't [17:16:33] but it can't hurt too much [17:16:47] (to ask philippe) [17:16:57] We've done it before w/o approvals, but it's ok I just poked Maggie who'll poke Philippe or whatever needs to be done :) [17:17:54] k. I'm just not at all involved with user admin with OTRS or otrs-wiki, I have no idea where I would be overstepping [17:19:06] sure [17:19:07] np [17:19:33] > Rjd0060 (Talk | contribs)‏‎ (bureaucrat, administrator, transwiki importer) (Created on 2008-04-21 at 13:41:03) [17:19:36] fwiw :-) [17:20:36] jeremyb_: I understand, but I can't say I entirely understand what the roles mean :-P [17:20:50] Jeff_Green: right [17:22:50] "All public logs" has a weird ring to it on a fishbowl :-P [17:38:03] New patchset: Cmjohnson; "changing to eth1 to test network port" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52250 [17:42:18] Change merged: Cmjohnson; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52250 [17:45:07] Change abandoned: Matthias Mullie; "Duplicate of https://gerrit.wikimedia.org/r/#/c/47551/ & https://gerrit.wikimedia.org/r/#/c/51341/" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52246 [17:47:17] Can someone check what the value of $wgAutopromoteOnce['onEdit']['autoreview'] is on trwiki? [17:50:45] Jeff_Green: Philippe says that your (ops team/whoever) time is too valuable for such a minor task. So, interesting. But there's the answer I guess.... (my actual thoughts, "wtf...") [17:50:48] jeremyb_: ^ [17:51:19] So, very sorry to trouble you and waste time. ;-) [17:51:27] well, it does seem as though the system should be such that ops isn't necessary for day-to-day user admin [17:51:38] RD no problem! [17:51:42] lol [17:51:59] ops is not necessary. just needs shell. i.e. ori-l could do it [17:52:02] (AIUI) [17:53:16] Ya sure he aint too busy too? [17:53:19] :P [17:54:49] RD: tbh, more time was spent deciding whether to do it than it would take to do it anyway [17:55:00] RD: whoever fills Krenair's req can do yours at the same time [17:55:27] I know. We've had this done about a half dozen times or so without issue. [18:04:13] !log restarting dhcpd3 service brewster [18:04:18] Logged the message, Master [18:13:31] New patchset: Reedy; "Add CVE linker" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52253 [18:35:36] Change merged: Matthias Mullie; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47551 [18:35:44] Change merged: Matthias Mullie; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/51341 [18:36:31] robh: confirmed on the 6 HPM servers [18:42:45] !log mlitn synchronized wmf-config/InitialiseSettings.php 'Disable AFTv4 (entirely) and AFTv5 (leaving only opt-in) on enwiki' [18:42:51] Logged the message, Master [18:52:50] Can someone please check what the value of $wgAutopromoteOnce['onEdit']['autoreview'] is on trwiki? [18:54:39] NULL [18:56:52] Reedy: want to do the otrs-wiki email reset while you're at it? (RD, still need that?) [18:57:06] I can't [18:57:13] I've not got DB access to RT [18:57:24] i'm confused [18:57:30] oh, wiki [18:57:33] this is for a wiki on the cluster [18:58:00] Yeah [19:01:21] New patchset: Cmjohnson; "Going back to nic1 on db1032" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52262 [19:01:51] Done [19:03:51] Change merged: Cmjohnson; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52262 [19:23:50] New patchset: Ottomata; "Adding puppet Limn module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49710 [19:39:23] Can an irc.wikimedia.org oper deal with this quit-flooder? [19:40:31] apparently not if you ask petan. i haven't witnessed the flooding myself. but probably can be done. (you're now the second person to ask) [19:40:43] Danny_B: ^ same guy still i assume? [19:41:14] 'snatch!snerk@anonymous.user' in #mediawiki.wikipedia and probably other channels [19:42:55] yes [19:43:22] Krenair: looks like on many wp channels and wikidata [19:43:30] perhaps trying to run some bot [19:43:37] but it's very annoyig [19:43:57] the entire screen can roll out with his rejoining between regular posts [19:44:08] why not just not send nick lists/joins/parts to other users at all? [19:44:19] idk how hard that is to not do [19:46:36] /ignore in my client doesn't seem to be working for joins/quits [19:48:23] New review: Dzahn; "RT-4566" [operations/apache-config] (master) C: 2; - https://gerrit.wikimedia.org/r/52182 [19:48:23] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/52182 [19:49:18] "/ignore -channels #chan1,#chan2,#chan3 * JOINS PARTS QUITS NICKS" [19:49:36] irssi.org [19:51:28] Reedy: I hate those 'Warning: the RSA host key for 'hume.wikimedia.org' differs from the key for the IP address '208.80.152.190'' messages [19:51:29] New review: Dzahn; "apache-fast-test wikimedia.ee.url mw1044" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/52182 [19:51:47] Remove them from your ~/.ssh/known_hosts ? [19:52:16] ssh-keygen -R [19:52:26] DELETE ALL THE KEYS [19:52:46] * AaronSchulz dislikes nuking all of them ;) [19:52:49] ssh-keygen -R hostname [-f known_hosts_file] [19:54:29] yeah, so that gives me the "new fingerprint" message as normal and then the same error afterwards [19:54:48] you gotta say y to save it once [19:55:31] dzahn is doing a graceful restart of all apaches [19:56:12] !log dzahn gracefulled all apaches [19:56:17] Logged the message, Master [19:56:26] !log gracefulling eqiad Apaches, push wiki(m|p)edia.ee redirects [19:56:31] Logged the message, Master [19:56:34] LeslieCarr: Cautious poke about https://gerrit.wikimedia.org/r/#/c/38457/ ; feel free to tell me to beat it if you're still dealing with the nagios->icinga stuff [19:56:37] mutante: still there [19:57:07] meh, it's just hume [19:57:32] this is from fenari? hmm, don't get a message when connecting to hume [19:57:41] no from my laptop [19:58:11] works from hume (using forwarding at least, ugh) [19:58:17] s/hume/fenari [19:58:33] hmm, are you root or normal user [19:58:40] * AaronSchulz is a mortal [19:58:49] on your laptop.. check /root/.ssh/known_hosts? [20:00:00] !log wikipedia.ee now redirects to et.wikipedia, wikimedia.ee now redirects to et.wikimedia [20:00:03] Krenair: i meant on the server side [20:00:05] Logged the message, Master [20:00:09] mutante: empty [20:05:04] resolved [20:12:00] hello binasher [20:12:13] hello Platonides [20:12:45] New patchset: Ottomata; "Adding puppet Limn module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49710 [20:13:01] I was precisely being questioned about a rebel mysql [20:13:18] maybe you can give some idea? [20:15:23] a raid disk failed, and after restoring the db, the sql INSERT INTO `filearchive` (fa_storage, ...) failed with «1062: Duplicate entry '0' for key 'PRIMARY'» [20:15:28] the table explain is http://es.wikipedia.org/w/index.php?title=Usuario_discusi%C3%B3n:Platonides&redirect=no#Como_siempre_vengo_a_pedir_XD [20:15:40] notice the missing auto_increment for fa_id [20:15:49] so I suggested doing ALTER TABLE filearchive MODIFY COLUMN fa_id int NOT NULL AUTO_INCREMENT; [20:16:00] but it fails with ERROR 1062 (23000): ALTER TABLE causes auto_increment resequencing, resulting in duplicate entry '1' for key 'PRIMARY' [20:16:01] 21:09:38 yo probé el comando en mi bbdd y funcionaba :P [20:22:23] Can someone please check what the value of $wgAutopromoteOnce['onEdit']['autoreview'] is on trwiki? [20:22:23] NULL [20:22:26] sorry, just saw this reply [20:22:33] !log installing package upgrades on hume [20:22:39] Logged the message, Master [20:22:49] That's odd, because this change was supposed to define that... https://gerrit.wikimedia.org/r/#/c/49685/2/wmf-config/flaggedrevs.php [20:26:47] !log reedy synchronized wmf-config/ [20:26:48] Logged the message, Master [20:28:44] It definitely gets set [20:29:49] Krenair: I bet.. it's wmgAutopromoteOnceonEdit [20:29:54] 'wmgAutopromoteOnceonEdit' => array( [20:29:54] 'default' => array(), [20:29:54] ), [20:30:11] Yup [20:30:12] $wgAutopromoteOnce = array( [20:30:12] 'onEdit' => $wmgAutopromoteOnceonEdit, [20:30:12] 'onView' => $wmgAutopromoteOnceonView, [20:30:13] ); [20:30:48] Krenair: I guess, as it's not FR specific config, it should go in InitialiseSettings.php [20:30:48] disconnected from wi-fi once again [20:31:03] okay, I'll make a change to move it over [20:35:24] !log installing package upgrades on hooper [20:35:30] Logged the message, Master [20:36:38] New patchset: Alex Monk; "Try to fix trwiki autopromotion config by moving $wgAutopromoteOnce['onEdit']['autoreview'] to wmgAutopromoteOnceonEdit" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52273 [20:46:39] New patchset: Reedy; "Bug 45744 - Please create ombudsmen.wikimedia.org" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/52275 [20:46:57] mutante: ^ Can you review/merge that etc, and also add a DNS entry for ombudsmen.wikimedia.org ? [20:47:03] Thanks [20:47:06] I'm gonna go and create the docroot now [20:48:25] New patchset: Reedy; "Bug 45744 - Please create ombudsmen.wikimedia.org" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52276 [20:48:57] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52276 [20:49:35] !log reedy synchronized docroot [20:49:40] Logged the message, Master [20:59:49] New patchset: Reedy; "Bug 45744 - Please create ombudsmen.wikimedia.org" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52277 [21:02:15] Reedy, do you still have stuff to deploy? [21:02:52] Nope [21:03:00] I just reset head to remove that revision from mediawiki-config [21:03:07] Can't do anything till it's got a DNS entry [21:03:11] ok, thanks [21:03:20] mobile fun! [21:03:37] Should probably push 52273 out later [21:04:50] New patchset: MaxSem; "Set template to append to mobile photo upload desc" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/50387 [21:05:26] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/50387 [21:20:42] RoanKattouw: you around ? [21:20:50] LeslieCarr: Just got back [21:21:28] wanna do this ? [21:21:39] Yeah let's do it [21:22:14] I am, however, eating lunch (and trying to get it to not completely fall apart) so I'll stay at my desk [21:23:02] okay [21:23:04] eat lunch [21:23:07] i'll give you 5 minutes :) [21:23:12] and then i can move to the hammock [21:23:18] k [21:24:13] The WoW Leeroy Jenkins video seems quite apt here [21:26:35] the hammock o.0 [21:30:32] [21:21:28] wanna do this ? [21:30:32] [21:21:36] Yeah let's do it [21:30:51] https://www.youtube.com/watch?v=LkCNJRfSZBU [21:30:52] glhf [21:31:19] hahaha [21:32:03] New patchset: Lcarr; "LVS for Parsoid Varnish" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/38457 [21:33:57] !log DNS update - add ombudsmen (bug 45744) [21:34:03] Logged the message, Master [21:35:42] !log restarting pdns on ns2 [21:35:47] Logged the message, Master [21:39:31] New review: Dzahn; "apache-fast-test ombudsmen.url mw1044" [operations/apache-config] (master) C: 2; - https://gerrit.wikimedia.org/r/52275 [21:39:32] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/52275 [21:39:39] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/38457 [21:40:55] dzahn is doing a graceful restart of all apaches [21:41:36] !log dzahn gracefulled all apaches [21:41:41] Logged the message, Master [21:42:04] !log gracefulling eqiad Apaches - add ombudsmen.wm (bug 45744) [21:42:10] Logged the message, Master [21:43:02] !log adding parsoidcache service to lvs1006 [21:43:07] Logged the message, Mistress of the network gear. [21:44:48] !log tools rejiggered the webproxy config to be smarter about paths not leading to specific tools [21:44:53] Logged the message, Master [21:46:03] Erm. Wrong log. [21:46:09] hah [21:46:24] New patchset: Lcarr; "Adding parsoidcache ip to lvs and fixing ip on parsoidcache hosts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52284 [21:47:41] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52284 [21:50:21] LeslieCarr: /home/wikipedia/common/docroot/noc/pybal/eqiad# [21:50:32] LeslieCarr: /home/wikipedia/common/docroot/noc/pybal/eqiad [21:50:43] LeslieCarr: Also, http://noc.wikimedia.org/pybal/eqiad/parsoidcache [21:50:44] thanks [21:53:53] Reedy: done [21:53:58] Missing wiki [21:54:14] Yeah, I've not pushed any code out for it, or run the script ;) [21:54:27] yep, just to confirm we get the expected [21:54:48] http://ombudsmen.wikimedia.org [21:54:49] * 301 Moved Permanently https://ombudsmen.wikimedia.org/ [21:54:49] https://ombudsmen.wikimedia.org [21:54:49] * 302 Found https://meta.wikimedia.org/wiki/Missing_wiki [21:54:56] incl. the https redirect [21:55:50] !log restarting pybal on lvs1006 [21:55:56] Logged the message, Mistress of the network gear. [21:55:57] office dns seems to have it cached :( [21:56:13] worked straight away on a vm at home [21:56:16] ack, i hacked my etc/hosts to confirm :p [21:56:53] lols [21:57:05] 208.80.154.224 ombudsmen.wikimedia.org [21:57:16] Probably going to be the simplest way [21:57:18] also works fine on dobson [21:57:48] !log restarting pybal on lvs1003 [21:57:53] Logged the message, Mistress of the network gear. [21:59:27] huzzah [21:59:30] parsercache is alive! [22:03:53] New patchset: Reedy; "Bug 45744 - Please create ombudsmen.wikimedia.org" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52277 [22:04:00] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52277 [22:05:04] New patchset: Reedy; "Try to fix trwiki autopromotion config by moving $wgAutopromoteOnce['onEdit']['autoreview'] to wmgAutopromoteOnceonEdit" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52273 [22:05:10] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52273 [22:05:44] !log reedy synchronized wmf-config/ [22:05:49] Logged the message, Master [22:07:27] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: [22:07:33] Logged the message, Master [22:09:06] New patchset: Reedy; "Trailing config for ombudsmenwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52289 [22:09:20] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52289 [22:13:40] !log restarting pybal [22:13:45] Logged the message, Mistress of the network gear. [22:14:15] !log restarting pybal on lvs4 [22:14:19] !log restarting pybal on lvs3 [22:14:20] Logged the message, Mistress of the network gear. [22:14:24] Logged the message, Mistress of the network gear. [22:16:52] !log reedy synchronized wmf-config/ [22:16:58] Logged the message, Master [22:17:38] New patchset: Reedy; "'ombudsmen' => 'ombudsmenwiki'" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52291 [22:17:53] !rt 4646 | Jeff_Green [22:17:53] Jeff_Green: http://rt.wikimedia.org/Ticket/Display.html?id=4646 [22:17:53] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52291 [22:18:24] !log reedy synchronized wmf-config/InitialiseSettings.php [22:18:29] Logged the message, Master [22:20:52] * MaxSem is scapping [22:30:56] New patchset: Pyoungmeister; "adding back in db1009 at low weight" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52293 [22:33:26] New patchset: ArielGlenn; "add interwiki cdb to noc downloads" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52294 [22:33:39] cute [22:33:42] how did I not ever know that was in git? [22:33:47] Change merged: Pyoungmeister; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52293 [22:33:51] I just had changed it manually on fenari [22:36:41] !log py synchronized wmf-config/db-eqiad.php 'repooling db1009 at 25% weight' [22:36:46] Logged the message, Master [22:40:36] !log maxsem Started syncing Wikimedia installation... : Weekly mobile deployment [22:40:41] Logged the message, Master [22:47:48] i think icinga-wm broke again [22:51:35] is parsoid down? [22:51:40] rawr [22:51:48] fucking ircecho [22:52:38] well, let me rephrase: fucking python-irclib [22:53:42] PROBLEM - LVS HTTP IPv4 on parsoidcache.svc.pmtpa.wmnet is CRITICAL: Connection timed out [22:54:01] RoanKattouw, gwicke: ^^ [22:54:02] ? [22:54:14] Yeah LeslieCarr is on that I think [22:54:19] yeah [22:54:22] i'm trying to figure out why [22:54:23] grr [22:54:28] also wtf ircecho [22:54:45] though, my ryan offered to check it out if i will do the laundry tonight [22:54:55] still deciding if that trade is worth it [22:54:55] hahaha [22:55:13] we just need to rewrite it as a supybot plugin [22:55:14] #geeklove [22:55:22] irclib doesn't reconnect properly [22:55:33] * marktraceur wonders if someone in here would be interested in coming to the Etherpad Lite meetup next week to discuss deployment-related things [22:55:55] it's incredibly simple. it just registers to files via inotify and echos them to irc [22:56:19] solution, cron job to restart ircecho every 5 minutes ? [22:56:22] ;) [22:56:25] heh [22:56:33] I'm not sure why it's disconnecting so often [22:56:40] marktraceur: yeah, i'm game. is it in NY? :-) [22:56:54] jeremyb_: If only! [22:57:03] It's at the Mozilla office in SF, sorry :) [22:57:34] Oh, hm, that's closer than I thought it was. [22:58:19] Ok, so why the hell won't it let me login to that wiki :( [22:58:52] yeah it's frustrating, silent fail [22:59:05] Reedy: which wiki? [22:59:15] https://ombudsmen.wikimedia.org which I just created [22:59:19] ah [23:00:10] I suspect it's something being cached somewhere [23:00:31] !log maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment [23:00:36] Logged the message, Master [23:01:19] oh pdns-recurser in the office had a crazy long negative cache [23:01:25] i think office is using bind now though [23:01:33] but might still have timeout [23:02:01] I've added it to my hosts. It seems to not want to accept my password for some stupid reason [23:02:09] even after I've just reset it via shell [23:02:27] Reedy: is it maybe redirecting you to https://ombudsmen.wikipedia.org ? [23:02:32] try https://ombudsmen.wikimedia.org/wiki/Main_Page [23:02:42] I am [23:02:45] hrm [23:03:00] I've seen it doing that if you visit it via http, sometimes [23:03:43] the new wiki text is funny [23:03:49] creation of a "wikimedia" [23:04:07] and it's being imported from incubator! [23:04:49] Yeah, sometimes funky with the caching [23:04:58] just created a new account with a 2 appended and logged me in first time [23:06:47] One would've hoped User::newFromName( 'Reedy' )->invalidateCache(); would've been enough [23:07:03] Reedy: https://gerrit.wikimedia.org/r/#/c/52298/ [23:13:38] !log stopping slave and starting mysql dumps on dbs 71, 57, 66, 65, 55, 50, and 68 [23:13:44] Logged the message, notpeter [23:14:54] PROBLEM - Solr on vanadium is CRITICAL: Average request time is 460.66974 (gt 400) [23:15:16] !log reedy synchronized wmf-config/InitialiseSettings.php [23:15:22] Logged the message, Master [23:16:39] RoanKattouw: damnit i figured out the problem [23:16:46] LeslieCarr: ? [23:16:53] whee, my voodoo works! [23:16:54] RoanKattouw: the stupid system doesn't have an interface in that vlan [23:16:55] stupid lvs [23:16:59] stupid sdtpa [23:17:02] ? [23:17:04] i hate you sdtpa [23:17:17] Wait so .1.28 works but .1.29 doesn't? [23:17:24] our lb system requires the balancer to have an address in the backend [23:17:27] it's the backends [23:17:38] .1.28's backends are in a different subnet [23:17:53] Oh [23:17:54] PROBLEM - MySQL Replication Heartbeat on db71 is CRITICAL: CRIT replication delay 224 seconds [23:17:55] Right [23:18:08] !log reedy synchronized wmf-config/InitialiseSettings.php [23:18:13] Logged the message, Master [23:18:24] PROBLEM - MySQL Replication Heartbeat on db57 is CRITICAL: CRIT replication delay 205 seconds [23:18:44] PROBLEM - MySQL Replication Heartbeat on db66 is CRITICAL: CRIT replication delay 209 seconds [23:18:51] I see [23:18:55] PROBLEM - MySQL Replication Heartbeat on db65 is CRITICAL: CRIT replication delay 199 seconds [23:19:05] LeslieCarr: ACtually, if you just wanna move em to the right subnet... [23:19:14] PROBLEM - MySQL Replication Heartbeat on db50 is CRITICAL: CRIT replication delay 196 seconds [23:19:14] PROBLEM - MySQL Replication Heartbeat on db55 is CRITICAL: CRIT replication delay 207 seconds [23:19:40] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [23:19:42] I mean the backends have public IPs right now [23:19:42] i don't think we have ip's [23:19:45] and I want them to be private [23:19:47]