[02:12:03] !log LocalisationUpdate completed (1.23wmf10) at 2014-01-27 02:12:03+00:00 [02:12:13] Logged the message, Master [02:22:28] !log LocalisationUpdate completed (1.23wmf11) at 2014-01-27 02:22:27+00:00 [02:22:36] Logged the message, Master [02:44:23] !log LocalisationUpdate ResourceLoader cache refresh completed at 2014-01-27 02:44:23+00:00 [02:44:31] Logged the message, Master [03:09:24] !log I'm about to do a bunch of experimental bullshit in the production puppet repo, because I can't test network topology in labs. [03:09:32] Logged the message, Master [03:14:11] (03PS1) 10Springle: repool db1010 depool db1023 for schema changes revert vslow load to db1022 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109612 [03:15:30] (03CR) 10Springle: [C: 032] repool db1010 depool db1023 for schema changes revert vslow load to db1022 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109612 (owner: 10Springle) [03:15:38] (03Merged) 10jenkins-bot: repool db1010 depool db1023 for schema changes revert vslow load to db1022 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109612 (owner: 10Springle) [03:16:49] !log springle synchronized wmf-config/db-eqiad.php 'repool db1010, depool db1023' [03:16:58] Logged the message, Master [03:20:30] springle: G'day! I'm seeing intermittent connect failures with labsdb* ("Could not connect to s4: Lost connection to MySQL server at 'reading authorization packet', system error: 0") with no discernible pattern (not constant, different servers). Do you know what causes this? [03:24:52] scfc_de: not immediately. many possibilities. definitely all three labsdb boxes? [03:25:46] all shards too? [03:30:23] (03PS1) 10Andrew Bogott: Import the stackforge openstack module for testing. [operations/puppet] - 10https://gerrit.wikimedia.org/r/109613 [03:31:28] springle: s1, s2, s4 and s5 from a cursory look at my logs. [03:31:38] scfc_de: also, when did they start, approx? [03:32:08] * andrewbogott counts to ten before self-merging [03:34:48] springle: It was quiet since Thu, 23 Jan 2014 05:55:09, and started about Sun, 26 Jan 2014 19:41:09. I run a script once per minute, and it had 55 failures in those roughly eight hours. [03:37:01] (03CR) 10Andrew Bogott: [C: 032] Import the stackforge openstack module for testing. [operations/puppet] - 10https://gerrit.wikimedia.org/r/109613 (owner: 10Andrew Bogott) [03:38:29] scfc_de: thanks. digging... [03:45:38] scfc_de: no obvious server-side issues but i do see a spike in aborted connections. i'm setting mariadb log_warnings=2 on labsdb1002. will need to watch it for an hour or so [03:49:39] springle: Thanks. [03:50:31] (03CR) 10Ori.livneh: "Daniel, just do it -- it shouldn't take a dozen confirmations. I trust your judgment, etc." [operations/puppet] - 10https://gerrit.wikimedia.org/r/109285 (owner: 10Dzahn) [03:56:55] andrewbogott: one second per 860 lines of code? [03:57:11] Yep! [03:57:22] Most of those lines are tests :) [03:58:04] it looks non-awful, at a glance [03:58:09] godspeed and good luck [03:58:39] Yeah, that module was endorsed by some folks in #openstack so I have high hopes, soon to be dashed. [04:10:32] !log xtrabackup clone db1022 to db1023 [04:10:40] Logged the message, Master [04:11:50] mark, if awake: Is there a separate IP pool for labs in eqiad, or is it the same set that we're using in tampa? [06:13:43] andrewbogott: one of your focus areas is labs, right? can you help me debug a network issue there? [06:13:54] maybe, what's up? [06:14:56] andrewbogott: separate subnet for sure, but I don't remember which one; DNS doesn't have it, right? [06:15:14] (03PS1) 10Andrew Bogott: Simple Openstack test cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/109615 [06:15:39] all three labsdb boxes started seeing aborted tcp connections around 2014-01-26 19:30:00 utc. suggests a network issue. but i don't know where to look in librenms [06:15:50] (03CR) 10jenkins-bot: [V: 04-1] Simple Openstack test cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/109615 (owner: 10Andrew Bogott) [06:16:11] wow, this openstack module is not that good isn't it [06:16:38] paravoid: I don't know yet… it looks somewhat promising to me, although I'm not clear if it has actual neutron support yet [06:16:41] springle: connections between labsdb and what? [06:16:52] paravoid, springle, I just updated https://wikitech.wikimedia.org/wiki/IP [06:17:02] paravoid: various clients, including each other [06:17:06] andrewbogott: the swift stuff is really bad for sure [06:17:25] i don't see a pattern yet for affected clients, though several of the most frequent are pmtpa [06:17:38] paravoid: I haven't looked at those bits :( I assume the nova sections get the most attention. [06:18:37] springle: I believe that all labs traffic passes through virt2. So might start by looking at its icinga page [06:18:39] * andrewbogott does this now [06:18:56] springle: production hosts or labs instances? [06:19:20] paravoid: no production hosts at all yet. only labs instances and each other [06:19:28] anyone want to look at https://gerrit.wikimedia.org/r/#/c/109074/ ? [06:19:36] hm, virt2 says 0% loss so that's not it [06:19:59] springle: default labs instances are heavily firewalled. Could it just be that? [06:20:19] springle: each other? so between labsdb1001 and, say, labsdb1002? [06:20:47] * andrewbogott curses autocorrect [06:20:59] paravoid: correct. though those are infrequent. tools-exec-01.pmtpa.wmflabs is a big offender [06:21:01] I need to restart so I can stop my client from s/springle/sprinkle/ brb [06:21:37] springle, what labs project do these boxes live in? [06:21:56] labsdb are production hosts [06:22:17] tools-exec-01 is toollabs [06:22:25] so "tools" I suppose [06:22:46] Is this something that worked until recently, or never worked? [06:23:34] mariadb doesn't know why the connection abort. "Unknown error". usually means an external glitch [06:24:02] I don't see anything wrong with e.g. labsdb1001's port [06:24:26] andrewbogott: it spiked at 2014-01-26 19:3x:xx on all three labsdb. prectically non-existent before that [06:24:49] there's a bunch of [06:24:49] Jan 24 00:02:09 virt2 kernel: [5113381.225710] nf_conntrack: table full, dropping packet. [06:24:50] so, 11h ago, or so [06:24:53] Jan 24 00:02:09 virt2 kernel: [5113381.447881] nf_conntrack: table full, dropping packet. [06:24:59] but none after the 24th [06:25:05] (on virt2) [06:25:54] andrewbogott: on an unrelated note since I'm looking at icinga, there's a bunch of unrelated with each other alerts for virt hosts [06:26:25] paravoid: I'm after the DC folks to get a new drive into virt5. [06:27:00] disk space is a constant juggle [06:27:14] And, dammit, I just fussed with the certs on Friday, they must've been changed again. [06:27:15] * andrewbogott scowls [06:28:27] springle, 'it spiked' = dropped packets? [06:28:40] Or by 'spiked' to you mean 'stopped working entirely'? [06:28:49] What port would mariadb be using? [06:29:06] andrewbogott: paravoid: http://aerosuidae.net/aborted_connections.png [06:29:27] sorry that's just an image. i hacked it up from some queries [06:29:35] wow [06:30:04] labsdb1001 is enwiki. it will be much larger always [06:31:00] Is tools-db in the mix here? (I thought that db queries from exec nodes generally happened via tools-db, but I don't know much.) [06:31:44] * springle doesn't know about tools-db [06:32:16] So this is ~10 hours ago, yes? [06:32:23] roughly, yes [06:32:53] (03PS1) 10Faidon Liambotis: Fix gitblit duplicate definition [operations/puppet] - 10https://gerrit.wikimedia.org/r/109617 [06:33:09] (03CR) 10Faidon Liambotis: [C: 032] Fix gitblit duplicate definition [operations/puppet] - 10https://gerrit.wikimedia.org/r/109617 (owner: 10Faidon Liambotis) [06:34:51] springle: Does that correspond to… this? http://ganglia.wmflabs.org/latest/graph.php?r=day&z=xlarge&me=wmflabs&m=load_one&s=by+name&mc=2&g=network_report [06:34:59] RECOVERY - Puppet freshness on antimony is OK: puppet ran at Mon Jan 27 06:34:52 UTC 2014 [06:35:19] PROBLEM - Puppet freshness on antimony is CRITICAL: Last successful Puppet run was Mon 27 Jan 2014 06:34:52 AM UTC [06:35:37] this would explain it, this is saturating a gigabit [06:35:46] andrewbogott: looks like it [06:36:19] 08:20 < springle> paravoid: correct. though those are infrequent. tools-exec-01.pmtpa.wmflabs is a big offender [06:36:21] er [06:36:25] http://ganglia.wikimedia.org/latest/?c=Virtualization%20cluster%20pmtpa&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2 [06:36:31] this is... broken [06:36:56] ganglia for virt hosts is broken, which explains why production ganglia looks okay [06:37:28] springle: (unrelated) db1057 broken LD, I found no RT, just making sure you know :) [06:37:29] can you find ganglia graphs for tampa virt hosts at all? i sure can't [06:37:50] andrewbogott: no, that's my point :) [06:38:18] yep [06:38:28] Is this a 'recently stopped working' or 'never tracked to begin with'? [06:38:36] dunno, I wasn't looking [06:39:22] paravoid: ah yes, thanks. forgot about that one [06:39:26] And I guess my next question is… is that crazy network graph because the connection failures are causing bad behavior, or does it show the bad behavior that is causing the connect failures? [06:39:44] the latter, probably [06:39:48] something is generating traffic [06:39:59] saturating some gigabit somewhere [06:40:08] which means lost packets and in turn dropped TCP connections [06:40:40] surely ganglia can break this down by host [06:40:46] brb, I'm overdue for a coffee dosage :) [06:40:50] jetlag and everything :) [06:42:24] springle: Looks like a misbehaving tool. Still digging... [06:42:48] andrewbogott: thank you! [06:43:32] Maybe not, we have two different projs showing that same graph shape. http://ganglia.wmflabs.org/latest/?c=visualeditor&m=load_one&r=day&s=by%20name&hc=4&mc=2 and http://ganglia.wmflabs.org/latest/?c=tools&m=load_one&r=day&s=by%20name&hc=4&mc=2 [06:43:42] So I bet one is cause and one is symptom… [06:44:47] oh, well, now that I read the axes... [06:44:53] ACKNOWLEDGEMENT - RAID on db1057 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) Sean Pringle RT 6711 [06:45:45] :) [06:46:59] RECOVERY - Puppet freshness on antimony is OK: puppet ran at Mon Jan 27 06:46:52 UTC 2014 [06:47:19] PROBLEM - Puppet freshness on antimony is CRITICAL: Last successful Puppet run was Mon 27 Jan 2014 06:46:52 AM UTC [06:50:58] grrr, why can't I log into these nodes today? [06:52:10] springle, can you log into any exec node? [06:56:57] andrewbogott: no, but then i've never tried before either [06:59:09] (03PS1) 10Springle: repool db1023 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109620 [06:59:35] (03CR) 10Springle: [C: 032] repool db1023 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109620 (owner: 10Springle) [06:59:41] (03Merged) 10jenkins-bot: repool db1023 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109620 (owner: 10Springle) [07:00:33] !log springle synchronized wmf-config/db-eqiad.php 'repool db1023' [07:00:42] Logged the message, Master [07:01:06] well. [07:01:23] springle, I don't know why my keys don't work on the exec nodes, but it's hard to detect and kill the bad tool without [07:01:49] wait, spoke too soon [07:02:08] my root keys don't work but my user does and I have sudo [07:08:23] springle, I don't know quite what I'm looking at. I've run netstat -t -u and one process has a gigantic 'Send-Q'. [07:08:30] But I don't know if that corresponds to heavy network use or not. [07:08:39] Any suggestions how to locate the offending tool? [07:09:42] ntop? or netstat [07:11:10] don't have ntop. can you advise how to read netstat? [07:11:17] for example, ^^^^ [07:13:44] iftop on there? [07:14:11] nope [07:15:30] i'd try to identify the port send/recv the most. then use netsta to get a pid [07:16:36] And how do I know what send/recvs the most? [07:16:58] iftop is a standard debian pkg. can you just apt-get it? [07:17:33] RECOVERY - Puppet freshness on antimony is OK: puppet ran at Mon Jan 27 07:17:24 UTC 2014 [07:17:45] Bad admin practice! But, yes :( [07:18:36] :) maybe paravoid knows a better way [07:19:38] could tcpdump instead i guess. laborious [07:20:18] so… netstat ? [07:21:29] grr, iftop doesn't show me port or process anyway [07:21:39] err.. something like netstat -p [07:21:39] just tools-exec-02.pmtpa.wmflabs => 10.0.0.45 [07:21:42] or -tp [07:21:45] Whcih I knew already [07:22:15] netstat 949 [07:23:49] iftop can show both source and dest ports. S and D keys iirc [07:26:08] ok, I continue to think that 949 is the offender but… still no idea as to pid or tool [07:27:16] talking to 10.0.0.45 [07:27:21] lsof -i tcp | grep 949 [07:28:19] that returns nothing at all [07:30:20] 949 is a privileged port. i wouldn't have expected something that low [07:31:35] thanks matanya, now how to find the mchenry-only classes :) https://bugzilla.wikimedia.org/show_bug.cgi?id=57890#c5 [07:31:53] springle: https://dpaste.de/TrWo [07:32:05] Nemo_bis: look in manifests/site.pp [07:32:45] matanya: yes but when I did it was a mess; let's see [07:33:41] (03PS1) 10Matanya: swift: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109625 [07:36:12] mogge guillom [07:36:34] andrewbogott: netstat -p ? [07:36:59] matanya: my problem with mchenry is that in site.pp it doesn't have *anything* with mail-sounding names, except privateexim::aliases::private (and that's not it) [07:38:19] I think I had grepped all the other classes' names too, but couldn't find anything sensible [07:38:51] Nemo_bis: you will have to ask one that really knows what is going on mchenry, it might even be it is not puppetized [07:39:14] hmm it's the only one including privateexim::aliases::private [07:39:26] matanya: you'd think! But... [07:39:27] tcp 0 0 tools-exec-02.pmtpa:949 10.0.0.45:38467 ESTABLISHED - [07:39:37] That last '-' is where the pid and program should be [07:39:43] yeah [07:40:46] * andrewbogott has /no/ idea what that means [07:41:25] andrewbogott: did you use sudo ? [07:41:29] or are you root? [07:41:40] I'm root [07:42:02] hmm, under 1024 doesn't show unless your are root, but you are :/ [07:42:45] I'm also not clear on what 10.0.0.45 and why labs can reach it, or thinks it can reach it [07:42:45] andrewbogott: just for the sake of the test, please do with sudo [07:43:43] nah, same. [07:44:31] can you telnet/nmap it? [07:44:49] ? An outbound port? [07:45:00] 949 [07:45:15] (03PS1) 10Springle: depool db1015 for schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109626 [07:45:20] you mean like 'telnet localhost 949' [07:45:40] yes andrewbogott [07:45:44] hi mutante [07:45:47] ah, that's just refused. [07:46:02] so i guess the port isn't open? [07:46:45] (03CR) 10Springle: [C: 032] depool db1015 for schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109626 (owner: 10Springle) [07:46:50] and i guess you tried with lsof andrewbogott ? [07:46:52] (03Merged) 10jenkins-bot: depool db1015 for schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109626 (owner: 10Springle) [07:46:53] matanya: the private mchenry stuff is in private/modules/privateexim/ [07:46:57] andrewbogott: [07:47:00] yeah, lsof knows nothing [07:47:19] Nemo_bis: see mutante's reply [07:48:34] !log springle synchronized wmf-config/db-eqiad.php 'depool db1015' [07:48:42] Logged the message, Master [07:49:31] andrewbogott: found your culprit [07:49:34] i think [07:49:41] yeah? How? [07:49:51] it is nfs, and nfs is provided by the kernel, so no PID [07:49:56] i grep 10.0.0.45 in dns [07:50:06] and it is labnfs [07:50:40] So the network is saturated by something doing file access... [07:50:44] yes [07:50:57] now you need to find that [07:51:01] any way for us to know who? [07:51:14] either iostat [07:51:17] or lsof [07:51:35] (03CR) 10Dzahn: "add some reviewers from mobile team" [operations/puppet] - 10https://gerrit.wikimedia.org/r/109511 (owner: 10Matanya) [07:51:58] for a lint change? [07:52:00] (03PS1) 10Springle: switch db1015 to innodb_file_per_table [operations/puppet] - 10https://gerrit.wikimedia.org/r/109627 [07:52:05] doesn't lsof require us to know /which/ file? [07:52:17] no, you can look by rate [07:52:28] mutante: just review it and merge, mobile has nothing to do with all that [07:52:56] paravoid: is mobile::vumi even still used? [07:53:01] no idea [07:53:14] (03CR) 10Springle: [C: 032] switch db1015 to innodb_file_per_table [operations/puppet] - 10https://gerrit.wikimedia.org/r/109627 (owner: 10Springle) [07:53:16] that was more what i wanted confirmed by adding them [07:53:18] but ok [07:53:31] ok, it wasn't obvious :) [07:53:53] also all that package version pinning from the manifests seems ugly to me [07:54:07] yea,i'll review it,no worries, the queues is just really long and this is like the latest of them all [07:54:12] I'd rather prefer a package { [ 'python-redis', 'python-smpp', ... ]: ... } [07:54:55] paravoid: i think matanya is splitting all the files and packages [07:55:00] for style reasons [07:55:01] i do [07:55:03] um, matanya, sorry I do not at all see how to do that. The lsof man page is long... [07:55:07] per style guide [07:55:33] andrewbogott: lsof /name/of/device [07:55:34] (03CR) 10Faidon Liambotis: [C: 032] "I don't see any issue with that or why it would be controversial." [operations/puppet] - 10https://gerrit.wikimedia.org/r/109325 (owner: 10Ottomata) [07:57:39] matanya: I mean, where is the rate? [07:57:46] And, anyway, unclear what the device is in this context... [07:57:49] mutante: if you can please link rt 6634 to 6684, i don't have permission [07:58:19] andrewbogott: i meant combining lsof and iostat, sorry i wasn't clear [07:58:19] !log xtrabackup clone db1022 to db1015 [07:58:26] Logged the message, Master [07:58:45] the device is the nfs mount [07:58:56] what ever it is called in df -h [07:58:57] matanya: done [07:59:01] thanks [08:00:35] paravoid: https://gerrit.wikimedia.org/r/#/c/83768/ uncontroversial ? [08:01:21] the reply to your comment was starting WIP https://gerrit.wikimedia.org/r/#/c/107831/ for discussion how to do it automatically, but that isnt ready like this [08:01:55] matanya: sorry to be dim, I do not follow at all. I need to iostat each of the 100+ files mounted on each of the 3 nfs volumes? That doesn't narrow it down much [08:03:57] andrewbogott: ah, so that isn't very helpful mmethod. do you know what nfs is being queried ? [08:04:23] (03CR) 10Dzahn: "that was more to confirm mobile::vumi is still used and will be" [operations/puppet] - 10https://gerrit.wikimedia.org/r/109511 (owner: 10Matanya) [08:04:24] matanya: please pretend like I don't know anything that I haven't already told you :) [08:04:41] sorry, i will [08:07:17] ok, andrewbogott better approache, rpcdebug [08:07:26] man there are some seriously big error logs in progress here [08:08:07] 2345298511 <- am I counting digits right? 2tb? [08:09:02] yeah, god [08:09:13] That's not even the only one, there's another that's 2.9 [08:09:19] I guess those are good candidates for iostat [08:09:42] (03CR) 10Dzahn: [C: 031] imagescaler: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109502 (owner: 10Matanya) [08:09:53] Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn [08:09:54] vda 1.38 1.29 137.38 6313385 673611120 [08:10:33] I guess between the two of them that's… maybe enough traffic to break things, huh? [08:10:49] guess so [08:11:51] well, wait, no matter what process I type in I get that same 137.38 [08:11:55] so I guess that's per device rather than per file or process :( [08:13:52] (03CR) 10Dzahn: search: lint (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/109485 (owner: 10Matanya) [08:14:07] andrewbogott: can you log on the nfs server and check there? [08:14:30] I can… what will I be checking? [08:15:29] iostat [08:16:20] (03CR) 10Matanya: search: lint (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/109485 (owner: 10Matanya) [08:18:14] iostat of what? [08:21:51] andrewbogott: correction : nfsstat -rc [08:22:29] nfsstat -rc ? [08:22:53] no, just that as a start [08:23:04] Ah, that's emtpy. Just 0s [08:23:13] sigh [08:23:45] I may drop this and leave it to Coren when he wakes up. Wasting lots of time, and I"m not sure springle still cares anyway :) [08:24:30] one last shot andrewbogott [08:24:36] 'k [08:24:38] on the toolnode [08:24:52] nfsiostat -h -m -t 1 4 > size.out [08:25:29] that got me a usage statement in a file :) [08:25:50] what ? [08:25:58] any broken paramter? [08:26:37] -m is not mentioned in the usage out. Neither is -t [08:27:02] ok use: nfsiostat -h [08:27:48] http://manpages.ubuntu.com/manpages/oneiric/man1/nfsiostat.1.html [08:27:48] that's the same usage [08:28:14] the man page clearly stats it is there :/ ok, wasted too much of your time [08:28:22] From oneiric though [08:28:49] I'm willing to keep poking if you have more ideas, just frustrated [08:29:04] And figuring that at any minute the offending process will complete and I'll never learn anything :( [08:29:05] http://manpages.ubuntu.com/manpages/precise/man1/nfsiostat.1.html ok, here for precise [08:30:49] (03PS2) 10Matanya: search: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109485 [08:31:21] so no nfsiostat andrewbogott ? [08:32:03] https://dpaste.de/CpzE [08:32:57] not the keys :) [08:32:58] hm, I don't have the version in that last link either, since -h is '-help' in my version [08:33:33] that explians this, -h should be humen readble [08:34:11] none of those kb/s look very big to me, must be missing something [08:35:19] because the network is still crazy busy… http://ganglia.wmflabs.org/latest/?r=day&cs=&ce=&c=tools&h=tools-exec-02&tab=m&vn=&mc=2&z=small&metric_group=ALLGROUPS [08:35:57] andrewbogott: let is run for some time. nfsiostat 1 4 [08:38:08] https://dpaste.de/v23t [08:41:44] this is a bit more suspected [08:45:23] hello Nemo_bis [08:48:07] (03PS2) 10Andrew Bogott: Simple Openstack test cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/109615 [08:49:13] back soon... [09:04:48] (03CR) 10Ori.livneh: [C: 032] "Thanks!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/108831 (owner: 10Tim Landscheidt) [09:06:21] (03PS1) 10Matanya: dns: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109632 [09:23:52] (03CR) 10Dzahn: [C: 032] "looked at the diff minus tabs one more time, should really just be lint change, can't see anything that would be a functional change" [operations/puppet] - 10https://gerrit.wikimedia.org/r/109485 (owner: 10Matanya) [09:27:37] (03CR) 10Dzahn: "nothing new to see on puppet run on searchidx1001 or elastic1001, that's good" [operations/puppet] - 10https://gerrit.wikimedia.org/r/109485 (owner: 10Matanya) [09:37:14] (03PS1) 10Dzahn: change search monitor_group description to eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/109635 [09:38:33] (03CR) 10Dzahn: "switched to eqiad already, right? but is the description string of the group really all?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/109635 (owner: 10Dzahn) [09:39:49] (03CR) 10Dzahn: search: lint (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/109485 (owner: 10Matanya) [09:44:33] (03CR) 10Dzahn: [C: 04-1] mobile: lint (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/109511 (owner: 10Matanya) [09:48:16] (03PS2) 10Matanya: mobile: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109511 [10:17:13] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [10:17:36] (03PS1) 10TTO: Move VisualEditor to secondary status on eswiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109639 [10:18:05] (03CR) 10Dzahn: [C: 031] mobile: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109511 (owner: 10Matanya) [10:18:38] matanya: will merge that too, last sanity check by yuvi as well [10:18:47] thanks [10:20:28] i usually retab the existing thing and then see what other diffs you have, but looks all good to me, noop [10:20:40] (03CR) 10Yuvipanda: [C: 031] "Seems ok to me" [operations/puppet] - 10https://gerrit.wikimedia.org/r/109511 (owner: 10Matanya) [10:20:57] (03CR) 10Dzahn: [C: 032] mobile: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109511 (owner: 10Matanya) [10:21:32] mutante: seems ok :) [10:21:48] confirming on silver/zhen, thx YuviPanda [10:22:09] YuviPanda: we still use vumi? [10:22:13] yup [10:22:28] well, I know we still use Vumi [10:22:34] not sure if it is this instance or some other one :P [10:22:40] I know that we have partners in Africa who are using it [10:23:35] (03CR) 10Dzahn: "@silver: notice: Finished catalog run in 33.50 seconds" [operations/puppet] - 10https://gerrit.wikimedia.org/r/109511 (owner: 10Matanya) [10:24:09] oh, not good [10:24:29] (03PS3) 10Andrew Bogott: Simple Openstack test cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/109615 [10:24:34] matanya: what? [10:24:46] 10 seconds between zhen and silver [10:25:12] andrewbogott: thanks for digging around tools-exec-02 earlier. sorry to frustrate you :) [10:25:22] eh, i don't think you can expect puppet runs to be the exact same length on 2 hosts [10:25:37] yeah, but 10 seconds? [10:25:53] springle: no problem. I'm sure that Coren will solve the problem in 5 minutes, but this way I'll be more likely to pay attention to his explanation :) [10:27:50] andrewbogott: want to +2 https://gerrit.wikimedia.org/r/#/c/109291/? [10:27:51] ;) [10:28:14] YuviPanda: silver and zhen have mobile::vumi in site.pp, that's why [10:28:39] mutante: i would most appricate if you review site.pp lint, it is moving forward and i'll have hard time rebasing [10:28:42] matanya: i'm not sure if they are even the same hardware, likely not [10:28:53] yeah, well. [10:29:05] YuviPanda: I suppose redis.lua inherits that license from a different dev? [10:29:12] andrewbogott: yup [10:29:17] 'k [10:29:29] andrewbogott: I didn't add headers to the API stuff, because you wrote those [10:29:37] weird use of -- as comment marker [10:30:07] andrewbogott: heh, I think it's not as weird as starting array indices from 1 [10:30:13] SQL also uses that [10:30:23] Does lua count from 1? [10:30:24] andrewbogott: still better than vimscript's choice of " as comment marker [10:30:27] andrewbogott: it does! [10:30:32] well, arrays at least [10:30:34] hm [10:30:42] (03CR) 10Andrew Bogott: [C: 032] dyanmicproxy: Add license headers to files [operations/puppet] - 10https://gerrit.wikimedia.org/r/109291 (owner: 10Yuvipanda) [10:31:00] +1091, -746 gaaah, sorrry matanya, merged 2 bigger lint changes, now i'd like to go back to fixing my own planet module first [10:31:22] andrewbogott: you should add a license header to the files you wrote :D [10:31:35] np mutante thanks for those two :) [10:31:40] I'll be sure to use a different license from you just to keep things interesting [10:31:58] andrewbogott: WTFPL FOR LYF! [10:32:16] lol, i was about to say Yuvi will license it was WTFPL [10:32:18] * andrewbogott is old school, likes gpl [10:32:53] andrewbogott: heh :D I don't mind licensing my stuff as GPL too. I can +1 if you want to make the entire thing GPL [10:32:59] add licensing@fsf.org as default reviewer on topic branch licensing:) fixed [10:33:01] andrewbogott: but I guess it has to be v3 or later than v2 [10:33:08] (03CR) 10Andrew Bogott: [C: 032] "This is totally whimsical, shouldn't touch any active machines." [operations/puppet] - 10https://gerrit.wikimedia.org/r/109615 (owner: 10Andrew Bogott) [10:34:35] "shouldn't touch any active machines" <- how you know I'm about to cause an outage on en.wikipedia [10:37:27] jenkins lint: detected use of "shouldn't" in message ?>:) [10:38:06] also 'safe' and 'no-op' [10:38:26] :) [10:39:02] well, those manifests don't work at all, which means it's probably safe for me to go to dinner. [10:40:16] andrewbogott: dinner? are you in japan? [10:40:29] Singapore, utc+8 [10:40:51] all around the globe :) [10:40:59] andrewbogott: class { 'openstack::compute': [10:41:06] inside role/openstack.pp [10:41:10] i think you want [10:41:25] the class from the module [10:41:45] but that would look for role::openstack::computer [10:42:11] Actually it's finding the right classes [10:42:16] But there's a naming conflict deeper down [10:42:18] among other things [10:42:31] (03PS1) 10Springle: repool db1015, warm up [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109641 [10:42:32] isnt thtat class { '::openstack::compute'" [10:42:37] matanya: I usually live in the midwest US, just here waiting out the winter. [10:42:38] in the role class [10:42:47] mutante: yeah, :: would be more correct. It happens to work as is [10:43:04] interesting, because i ran into that issue before and it would not [10:43:17] just looked in role:: itself.. ok [10:43:19] (03CR) 10Springle: [C: 032] repool db1015, warm up [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109641 (owner: 10Springle) [10:43:24] (03Merged) 10jenkins-bot: repool db1015, warm up [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109641 (owner: 10Springle) [10:43:33] matanya: whereas mutante is waiting out the California winter in Germany, which seems an odd choice [10:43:51] lol, it is, you normally would not pick January to be in .de [10:44:33] !log springle synchronized wmf-config/db-eqiad.php 'repool db1015, warm up' [10:44:41] Logged the message, Master [10:45:23]