[00:13:07] (03PS1) 10Sitic: Add cross-wiki notifications [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/222222 (https://phabricator.wikimedia.org/T103678) [00:19:18] (03PS2) 10Sitic: Add cross-wiki notifications [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/222222 (https://phabricator.wikimedia.org/T103678) [00:19:21] 6Labs, 10Labs-Infrastructure: Switch to horizon/designate/pdns/mysql for labs public dns - https://phabricator.wikimedia.org/T104520#1419432 (10Andrew) 3NEW [00:20:01] (03CR) 10Sitic: [C: 032 V: 032] Add cross-wiki notifications [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/222222 (https://phabricator.wikimedia.org/T103678) (owner: 10Sitic) [00:20:33] 6Labs, 10Labs-Infrastructure: Support reverse dns for public labs IPs - https://phabricator.wikimedia.org/T104521#1419440 (10Andrew) 3NEW [01:22:03] would it be an issue to install the "libosp-dev" package on tools.wmflabs ? [01:29:44] what are URLs to see the "relic Toolserver files"? [01:30:23] hmm.. this [01:30:27] manifests/role/relic.pp: system::role { 'relic': description => 'Toolserver legacy server' } [01:30:32] where is that server [01:32:44] maybe http://toolserver.org/ ? [01:32:57] that seems to be served by wikimedia [01:40:18] Krenair: indeed. that was too easy. thx [01:40:47] hmmm, is it just me or does the logo on the left not load [01:40:55] it's just me [02:10:44] 10Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1419535 (10Legoktm) 1 [02:15:30] rillke: file a bug - should be mostly OK [02:28:19] greg-g: are you in charge of wikiphabot? [02:52:47] Negative24: wikiphabot is dead. [02:53:16] legoktm: was there a reason? (read: can I take a shot?) [02:53:39] Negative24: we use wikibugs instead? [02:53:47] take a shot at what? [02:54:12] at implementing something similar [02:54:35] we already have wikibugs...? [02:55:15] legoktm: something like this: [02:55:20] T8271 [02:55:21] T8271: Allow multiple classes of footnotes on the same page - https://phabricator.wikimedia.org/T8271 [02:55:34] oh, please no :( [02:55:36] https://phabricator.wikimedia.org/T8271 [02:55:43] kinda like phabot [02:55:59] too spammy? [02:56:04] yes. [02:56:14] if people need to know the title of a bug, they can open their web browser. [02:56:35] well at least it didn't take much time [02:56:38] Bot24: stop [02:56:38] Stopping... [03:05:00] gasp, le bot :o [03:05:18] anyways, is there a copy of enwiki dump on tool labs so I don't have to redownload? [03:05:30] as in, the bot doesn't have to redownload when updating dump [03:10:14] nevermind, found em :) /public/dumps [03:46:17] which instance uses "role::relic"? [03:46:32] or how do i connect to relay.toolserver.org [03:55:06] 6Labs, 10Tool-Labs: role::relic - changes not applied by puppet? where is it applied? - https://phabricator.wikimedia.org/T104537#1419713 (10Dzahn) 3NEW [03:55:34] 6Labs, 10Tool-Labs: role::relic - changes not applied by puppet? on which node or instance is it? - https://phabricator.wikimedia.org/T104537#1419721 (10Dzahn) [03:56:21] 6Labs, 10Tool-Labs: role::relic - changes not applied by puppet? on which node or instance is it? - https://phabricator.wikimedia.org/T104537#1419713 (10Dzahn) asking this because we want the changes from: https://gerrit.wikimedia.org/r/#/c/221067/ https://gerrit.wikimedia.org/r/#/c/222234/ [04:09:19] (03CR) 10Ricordisamoa: ""3822 Errors, 6489 warning(s)"" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/221091 (owner: 10Ricordisamoa) [05:07:24] This whole filesystem desaster on labs confuses me a bit. [05:07:37] I created a new instance in one of my projects [05:08:10] and its /data/project is empty [05:08:25] isn't that supposed to be shared between instance?! [05:08:44] do we still have a shared space across instances? [05:09:01] or do we need to set up our own NFS now? [05:30:53] dschwen: depend on the project, but I think the shared NFS has been disabled for most projects [06:47:22] PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL 50.00% of data above the critical threshold [0.0] [06:47:53] is there a way to get additional space on a labs instance? (not NFS) [06:52:47] SMalyshev: depends on the type of instance, but for most of them there's unallocated space [06:53:12] create a partition there, format, use [06:53:55] valhallasw`cloud: is there anywhere I could read about that unallocated space? [06:54:07] how I get to it, etc.? [06:54:24] and how big it is [06:54:36] SMalyshev: that depends on the instance you created [06:55:22] see the instance info page for how much disk space there is available [06:55:23] valhallasw`cloud: this is the instance: https://wikitech.wikimedia.org/wiki/Nova_Resource:Wdq-beta.wikidata-query.eqiad.wmflabs [06:55:35] 160GB allocated [06:55:54] so you have a 20GB /, and then 140GB unpartitioned space on the drive [06:56:02] valhallasw`cloud: yeah, I may need a little more [06:56:20] valhallasw`cloud: that one is already partitioned :) [06:56:37] under role::labs::lvm::srv [06:56:43] https://wikitech.wikimedia.org/wiki/LVM [06:57:19] hmm... so I can just grow the partition as I like? [06:57:52] well, no, not as you like, as that page explains =p [06:58:03] but yes, you can grow /, I think [06:58:40] VG Size 141.40 GiB [06:58:41] Alloc PE / Size 36199 / 141.40 GiB [06:59:04] I don't think this VG has any more... [06:59:32] looks like it [06:59:42] so you'll have to create a new instance [06:59:48] so is there some way to get more space there? :) [07:12:21] RECOVERY - Puppet failure on tools-exec-catscan is OK Less than 1.00% above the threshold [0.0] [07:15:09] SMalyshev: hm, wait, 140GB is already xlarge [07:15:22] yes, I know [07:16:15] it's not possible without human intervention, then. andrewbogott_afk should know what's possible [07:16:34] ok, thanks, I'll try to ping him when he's not afk :) [07:17:49] the problem is I have a service with 75G DB, and I need to reload it. So I either can shut down the old one and have service down for 2 days (ok, but sucks) or I need to find a way to store both, at least for a short while [07:18:29] SMalyshev: I'd just spin up a second instance as new database server, load the data there, then change config to point to the new server [07:18:31] it _may_ happen that new DB is like 55 G instead of 75 then I still fit into 140G but I can't say... [07:19:24] valhallasw`cloud: unfortunately, this is not fully puppetized yet, so there's a bunch of manual work in creating new server. But yeah that's an option too [07:20:44] maybe actually even better one because it would force me to automate more things... :) [07:23:13] 6Labs, 10Labs-Infrastructure: replica.my.cnf creation broken - https://phabricator.wikimedia.org/T104453#1419905 (10Legoktm) [07:23:14] 6Labs, 7Database: Provision a labsdb useraccount that can be used to run replica-addusers.pl - https://phabricator.wikimedia.org/T104476#1419906 (10Legoktm) [07:25:12] SMalyshev: practicing/documenting recovery steps is always good, but often not something one has time for :-) [07:26:45] valhallasw`cloud: surprisingly, it's documented... it's just that I have to actually do all that... :) [07:29:50] 6Labs, 10Tool-Labs: toolsbeta login broken - https://phabricator.wikimedia.org/T104544#1419911 (10valhallasw) 3NEW a:3yuvipanda [07:30:28] 6Labs, 10Tool-Labs: toolsbeta login broken - https://phabricator.wikimedia.org/T104544#1419920 (10valhallasw) [07:43:41] mount.nfs: mounting labstore.svc.eqiad.wmnet:/project/wikidata-query/home failed, reason given by server: No such file or directory [07:44:07] why would that be? on other instances it is mounter with the same path [07:48:12] 6Labs, 7Database: Provision a labsdb useraccount that can be used to run replica-addusers.pl - https://phabricator.wikimedia.org/T104476#1419950 (10jcrespo) p:5Triage>3Normal [08:21:50] 6Labs, 7Database: Provision a labsdb useraccount that can be used to run replica-addusers.pl - https://phabricator.wikimedia.org/T104476#1419989 (10jcrespo) This SUPER user already exists and is already on the private repo (as the previous script worked). I will not put its name publicly, for obvious reasons.... [08:47:42] 6Labs, 6operations, 10wikitech.wikimedia.org, 7Database: labswiki DB is inaccessible from tin, terbium, etc. - https://phabricator.wikimedia.org/T98682#1420009 (10jcrespo) Maybe dumps of labswiki are failing, too. Can you confirm it @ArielGlenn? I just saw `:real_connect(): (HY000/2003): Can't connect to... [09:05:42] 6Labs, 10Math, 10Mathoid, 10MediaWiki-extensions-MathSearch, 7Tracking: Set up a stable math demo instance - https://phabricator.wikimedia.org/T104548#1420036 (10Physikerwelt) 3NEW a:3Physikerwelt [09:05:45] valhallasw`cloud: hi! [09:07:20] still the same errors with the mysql...which is rather slower than yesterday :( [09:07:40] 6Labs, 10Math, 10Mathoid, 10MediaWiki-extensions-MathSearch, 7Tracking: Fix 503 on Help:Forumla purge - https://phabricator.wikimedia.org/T104549#1420057 (10Physikerwelt) 3NEW a:3Physikerwelt [09:07:59] as far as it seems, when there is a "MySQL gone away", it points the last line of the code block. Even though it has nothing to do with the line in which it is stuck. [09:09:55] would it be possible to increase the packet size for my mysql connections? [09:11:20] YuviPanda|zzz: Coren|Away :) [09:11:24] marmick: the error happens in the __exit__ block [09:12:13] 6Labs, 10Math, 10Mathoid, 10MediaWiki-extensions-MathSearch, 7Tracking: Fix 503 on Help:Forumla purge - https://phabricator.wikimedia.org/T104549#1420084 (10Physikerwelt) [09:12:57] valhallasw`cloud: it seems so. i changed the code structure several times and the error always points at the exit somehow [09:13:21] even if the exit is unrelated to mysql [09:14:08] marmick: the __exit__ block is *always* run, also if you just get to the end of the with block [09:15:28] ? [09:15:42] https://www.python.org/dev/peps/pep-0343/ [09:17:28] marmick: I don't have the time or energy to explain how python works, sorry. [09:18:00] valhallasw`cloud: i appreciate your efforts, but i'm not asking that. [09:18:44] what i tried new was using a fetchmany(array) for the mysql. and it is stuck in the middle. [09:22:22] valhallasw`cloud: I see what you meant with the exit block. the problem is in the mysql management [09:22:33] the connection [09:22:51] maybe i should try to change to pymysql [09:23:05] and it won't die. what do u think? [10:03:25] hi, my new projekt is missing the replica conf [10:13:11] Steinsplitter: yes, I emailed labs-l (and labs-announce) about it yesterday. still working on it [10:14:04] ok :( [10:14:32] Steinsplitter: hopefully in a couple of hours! [10:17:04] 6Labs, 7Database: Provision a labsdb useraccount that can be used to run replica-addusers.pl - https://phabricator.wikimedia.org/T104476#1420229 (10yuvipanda) @jcrespo nope, labstore1001/2 is not accessible to non-ops. I've fixed the grants escaping problem. [11:12:42] 6Labs, 3Labs-Sprint-104: Recover files from old corrupted file system (Tracking) - https://phabricator.wikimedia.org/T104334#1420356 (10yuvipanda) [11:12:43] 6Labs, 10Incident-20150617-LabsNFSOutage: Recover data file /data/project/phetools/public_html/data/new_stats.py - https://phabricator.wikimedia.org/T104239#1420353 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Done [11:14:36] 6Labs, 3Labs-Sprint-104: Recover files from old corrupted file system (Tracking) - https://phabricator.wikimedia.org/T104334#1413628 (10yuvipanda) [11:19:53] 6Labs, 3Labs-Sprint-104: Recover files from old corrupted file system (Tracking) - https://phabricator.wikimedia.org/T104334#1420387 (10yuvipanda) [11:19:54] 6Labs, 3Labs-Sprint-104: Recover GND bot from wikidata-todo - https://phabricator.wikimedia.org/T104336#1420385 (10yuvipanda) 5Open>3Resolved I've copied over the entire scripts folder into wikidata-todo's homedir and called it scripts.recovered. You can take ownership of it via 'take scripts.recovered' [12:35:02] YuviPanda: could you take a look at toolsbeta? it's blocking me on the mailrelay stuff [12:35:10] (login issues) [12:35:17] valhallasw`cloud: oh, right. have you rebooted the machines already? [12:35:20] yeah [12:35:26] so it's not just NFS, I guess [12:35:49] valhallasw`cloud: ok, so which instnaces do you want access to? [12:35:56] can you get into puppetmaster3? [12:36:08] YuviPanda: toolsbeta-exec-01 and toolsbeta-mail [12:36:11] I can login to puppetmster3 [12:36:16] strangely enough [12:36:33] yeah, let me fix [12:40:34] valhallasw`cloud: -exec-01 should be available now [12:41:05] YuviPanda: nope :( [12:41:11] valhallasw`cloud: try now? [12:41:23] yup [12:41:24] yep, works now [12:41:31] valhallasw`cloud: yeah, needed puppet to run [12:41:37] valhallasw`cloud: toolsbeta-mail should also work now [12:41:48] YuviPanda: huh. Was puppet not running every 20 mins, then? [12:42:00] I force-pulled puppetmaster yesterday, but that didn't solve it [12:42:08] so on tools-exec-01 it still had old puppetmaster cert [12:42:16] I changed it (edited /etc/puppet/puppet.conf) [12:42:20] yeah, mail is also fine now [12:42:23] and also edited resolv.conf [12:43:08] valhallasw`cloud: and latest puppetmaster was failing for beta, because someone (apparently me!) disabled NFS for toolsbeta... [12:43:20] I fixed that in a local diff on the puppetmaster, I should commit and merge that [12:43:43] YuviPanda: can you use your root powahz to push my pubkey to /etc/ssh/userkeys/valhallasw on all toolsbeta hosts? then I can take a look at the rest [12:44:04] valhallasw`cloud: yeah, which key? [12:44:12] ldaplist -l passwd valhallasw [12:44:35] * valhallasw`cloud is guessing IRC will just cut if off halfway =p [12:44:46] valhallasw`cloud: actually, let me just add you to the root keysetup temporarily for toolsbeta [12:44:55] hmm, actually no that's harder [12:44:59] YuviPanda: except puppet doesn't run =p [12:45:03] yeah [12:45:06] so I can use salt [12:45:09] but then I'll hate my life [12:45:10] so let me psssh [12:45:22] I actually have a patch to submit that just adds our pubkeys to /etc/ssh/userkeys/ [12:45:37] although I'm wondering whether I should use hiera for the actual keys [12:45:50] easier to revoke [12:46:25] valhallasw`cloud: our as in admins'? [12:46:31] yeah [12:46:45] valhallasw`cloud: yeah, fair enough. [12:46:53] valhallasw`cloud: ok, so I submitted [12:47:01] well, did it for all instances, I mean [12:47:25] (y) [12:47:35] valhallasw`cloud: I'm rewriting the script that does mysql grants now. [12:48:10] valhallasw`cloud: I'm going to modify it to make it: 1. make it opt-in for projects, 2. opt-in only tools to begin with, 3. do it only for service groups to begin with [12:50:42] why opt-in? [12:52:26] valhallasw`cloud: 99% of projects don't need a replica.my.cnf - currently it creates one for every user in every project.... [12:52:40] editor-engagement project doesn't need replica.my.cnf for their vagrant user... [12:52:54] YuviPanda: huh. I thought it only did ldap users [12:52:55] neither does revscoring, rcm, etc... [12:53:05] valhallasw`cloud: vagrant is an LDAP user :) [12:53:11] oh =p [12:53:24] anyway, it's a lot of cruft, and also assumes all projects have NFS :P [13:08:50] YuviPanda hello again from the math cluster... I created a new instance math-ru but running puppet agent -tv returned the error that mount.nfs: mounting labstore.svc.eqiad.wmnet:/project/math/project failed, reason given by server: No such file or directory [13:09:26] physikerwelt: indeed, that still isn't fixed :( https://phabricator.wikimedia.org/T102782 - I hope to finish that today [13:09:35] we're a bit short handed this week. apologies for the service issues [13:10:12] no worries [13:10:29] if it helps feel free to use the math-ru instance for testing... there is nothing on it [13:10:34] ok :) [13:38:17] 6Labs, 10Labs-Infrastructure: No reverse DNS entries for public IPs - https://phabricator.wikimedia.org/T64887#1420802 (10scfc) [13:38:18] 6Labs, 10Labs-Infrastructure: Support reverse dns for public labs IPs - https://phabricator.wikimedia.org/T104521#1420803 (10scfc) [14:03:05] YuviPanda: can I re-enable nfs for toolsbeta? [14:03:17] it's still i nthe hiera config [14:03:18] valhallasw`cloud: make it a local commit? [14:03:22] valhallasw`cloud: the hiera config is ignored now. [14:03:29] errr? [14:03:31] valhallasw`cloud: it's in modules/labstore/files/projects-config.yaml [14:03:35] oh, ok [14:03:58] valhallasw`cloud: make a local commit there? I didn't want to merge and commit one since I'm working on https://gerrit.wikimedia.org/r/#/c/221872/8 and parent and touching that file will make my life rebasehell :) [14:08:08] 6Labs, 10Tool-Labs: role::relic - changes not applied by puppet? on which node or instance is it? - https://phabricator.wikimedia.org/T104537#1420985 (10scfc) a:3coren The public IP points to https://wikitech.wikimedia.org/wiki/Nova_Resource:Relic.toolserver-legacy.eqiad.wmflabs and that does only have the P... [14:09:23] YuviPanda: too bad puppet doesn't do for loops :/ [14:09:36] well, newpuppet does, but I don't think we can use that easily [14:22:02] PROBLEM - Puppet failure on tools-exec-1402 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:22:16] PROBLEM - Puppet failure on tools-webgrid-generic-1404 is CRITICAL 33.33% of data above the critical threshold [0.0] [14:22:28] woooo [14:22:28] PROBLEM - Puppet failure on tools-exec-1207 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:22:42] PROBLEM - Puppet failure on tools-exec-1202 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:23:01] looks not good.... [14:23:07] it's ok [14:23:11] no explicit failures [14:23:24] PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL 20.00% of data above the critical threshold [0.0] [14:23:36] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1204 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:23:38] PROBLEM - Puppet failure on tools-exec-1405 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:23:53] PROBLEM - Puppet failure on tools-exec-1208 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:23:59] PROBLEM - Puppet failure on tools-bastion-02 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:24:16] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1210 is CRITICAL 22.22% of data above the critical threshold [0.0] [14:24:57] but a lot of problems.... [14:25:37] describes my life of late a bit too accurately. [14:26:13] PROBLEM - Puppet failure on tools-bastion-01 is CRITICAL 22.22% of data above the critical threshold [0.0] [14:26:49] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1409 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:27:01] PROBLEM - Puppet failure on tools-exec-1212 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:27:11] PROBLEM - Puppet failure on tools-exec-1407 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:27:21] PROBLEM - Puppet failure on tools-exec-1401 is CRITICAL 60.00% of data above the critical threshold [0.0] [14:27:34] 6Labs, 10Labs-Infrastructure: Don't rely on wikitech API for production services - https://phabricator.wikimedia.org/T104575#1421040 (10Andrew) 3NEW [14:27:38] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1207 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:27:38] PROBLEM - Puppet failure on tools-services-01 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:27:40] PROBLEM - Puppet failure on tools-shadow is CRITICAL 50.00% of data above the critical threshold [0.0] [14:27:46] PROBLEM - Puppet failure on tools-webgrid-generic-1401 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:28:17] 6Labs, 10Labs-Infrastructure: Don't rely on wikitech API for production services - https://phabricator.wikimedia.org/T104575#1421048 (10yuvipanda) +1 - we need a readonly nova api account, I think. [14:28:20] PROBLEM - Puppet failure on tools-mail is CRITICAL 55.56% of data above the critical threshold [0.0] [14:28:28] PROBLEM - Puppet failure on tools-exec-1216 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:28:29] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1206 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:28:30] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1410 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:28:33] yeah, that's fixed now [14:28:36] PROBLEM - Puppet failure on tools-precise-dev is CRITICAL 50.00% of data above the critical threshold [0.0] [14:29:00] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1405 is CRITICAL 60.00% of data above the critical threshold [0.0] [14:29:11] PROBLEM - Puppet failure on tools-webproxy-01 is CRITICAL 44.44% of data above the critical threshold [0.0] [14:31:37] PROBLEM - Puppet failure on tools-exec-1211 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:31:53] PROBLEM - Puppet failure on tools-master is CRITICAL 60.00% of data above the critical threshold [0.0] [14:32:00] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1406 is CRITICAL 60.00% of data above the critical threshold [0.0] [14:32:13] andrewbogott_afk: ^ no need to panic, the recovery has already started [14:33:21] PROBLEM - Puppet failure on tools-webgrid-generic-1403 is CRITICAL 66.67% of data above the critical threshold [0.0] [14:33:35] PROBLEM - Puppet failure on tools-exec-1406 is CRITICAL 60.00% of data above the critical threshold [0.0] [14:34:01] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1403 is CRITICAL 70.00% of data above the critical threshold [0.0] [14:36:55] PROBLEM - Puppet failure on tools-exec-1210 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:37:29] RECOVERY - Puppet failure on tools-exec-1207 is OK Less than 1.00% above the threshold [0.0] [14:38:19] PROBLEM - Puppet failure on tools-services-02 is CRITICAL 22.22% of data above the critical threshold [0.0] [14:38:55] PROBLEM - Puppet failure on tools-exec-1201 is CRITICAL 20.00% of data above the critical threshold [0.0] [14:39:03] PROBLEM - Puppet failure on tools-exec-1204 is CRITICAL 33.33% of data above the critical threshold [0.0] [14:39:26] PROBLEM - Puppet failure on tools-webgrid-generic-1402 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:41:00] PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL 50.00% of data above the critical threshold [0.0] [14:41:14] PROBLEM - Puppet failure on tools-exec-1404 is CRITICAL 66.67% of data above the critical threshold [0.0] [14:41:14] RECOVERY - Puppet failure on tools-bastion-01 is OK Less than 1.00% above the threshold [0.0] [14:41:52] what is the job, which shinken should do? [14:42:02] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1202 is CRITICAL 50.00% of data above the critical threshold [0.0] [14:42:09] Luke081515: alert us when things go wrong. [14:42:15] Luke081515: it alerted, I founda nd fixed the problem. [14:42:25] ok [14:43:33] PROBLEM - Puppet failure on tools-exec-1207 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:44:56] ugh git, why do you change the committer on --amend [14:44:56] :< [14:45:22] and ugh gerrit, why do you complain that hte committer email is not registered for me [14:48:00] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL 20.00% of data above the critical threshold [0.0] [14:49:41] PROBLEM - Puppet failure on tools-exec-1209 is CRITICAL 30.00% of data above the critical threshold [0.0] [14:52:11] PROBLEM - Puppet failure on tools-bastion-01 is CRITICAL 22.22% of data above the critical threshold [0.0] [14:52:19] YuviPanda: unfortunately, the userkeys trick didn't work. Puppet automagically removes keys. Sane from a security perspective, though... [14:54:39] YuviPanda: but I can login to the hosts I need, so it's not an issue [14:55:39] !log toolsbeta toolsbeta-webproxy does not respond at all to SSH; rebooting [14:55:44] Logged the message, Master [15:01:51] 6Labs, 10Tool-Labs: Audit redis usage on toollabs - https://phabricator.wikimedia.org/T91979#1421116 (10valhallasw) 5Open>3Resolved [15:02:12] RECOVERY - Puppet failure on tools-bastion-01 is OK Less than 1.00% above the threshold [0.0] [15:03:49] YuviPanda: so, what’s the state of nfs exports now? Would you expect them to be working? [15:04:22] RECOVERY - Puppet failure on tools-webgrid-generic-1402 is OK Less than 1.00% above the threshold [0.0] [15:06:13] RECOVERY - Puppet failure on tools-exec-1404 is OK Less than 1.00% above the threshold [0.0] [15:06:57] RECOVERY - Puppet failure on tools-exec-1210 is OK Less than 1.00% above the threshold [0.0] [15:08:18] RECOVERY - Puppet failure on tools-services-02 is OK Less than 1.00% above the threshold [0.0] [15:08:58] RECOVERY - Puppet failure on tools-exec-1201 is OK Less than 1.00% above the threshold [0.0] [15:09:02] RECOVERY - Puppet failure on tools-exec-1204 is OK Less than 1.00% above the threshold [0.0] [15:10:58] RECOVERY - Puppet failure on tools-exec-wmt is OK Less than 1.00% above the threshold [0.0] [15:12:00] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1202 is OK Less than 1.00% above the threshold [0.0] [15:12:02] RECOVERY - Puppet failure on tools-exec-1402 is OK Less than 1.00% above the threshold [0.0] [15:12:20] RECOVERY - Puppet failure on tools-exec-1401 is OK Less than 1.00% above the threshold [0.0] [15:12:21] RECOVERY - Puppet failure on tools-webgrid-generic-1404 is OK Less than 1.00% above the threshold [0.0] [15:12:38] RECOVERY - Puppet failure on tools-shadow is OK Less than 1.00% above the threshold [0.0] [15:12:41] RECOVERY - Puppet failure on tools-exec-1202 is OK Less than 1.00% above the threshold [0.0] [15:13:21] RECOVERY - Puppet failure on tools-mail is OK Less than 1.00% above the threshold [0.0] [15:13:25] RECOVERY - Puppet failure on tools-exec-catscan is OK Less than 1.00% above the threshold [0.0] [15:13:29] RECOVERY - Puppet failure on tools-exec-1216 is OK Less than 1.00% above the threshold [0.0] [15:13:29] RECOVERY - Puppet failure on tools-exec-1207 is OK Less than 1.00% above the threshold [0.0] [15:13:37] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1204 is OK Less than 1.00% above the threshold [0.0] [15:13:37] RECOVERY - Puppet failure on tools-precise-dev is OK Less than 1.00% above the threshold [0.0] [15:13:37] RECOVERY - Puppet failure on tools-exec-1405 is OK Less than 1.00% above the threshold [0.0] [15:13:51] RECOVERY - Puppet failure on tools-exec-1208 is OK Less than 1.00% above the threshold [0.0] [15:13:59] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1405 is OK Less than 1.00% above the threshold [0.0] [15:13:59] RECOVERY - Puppet failure on tools-bastion-02 is OK Less than 1.00% above the threshold [0.0] [15:14:11] RECOVERY - Puppet failure on tools-webproxy-01 is OK Less than 1.00% above the threshold [0.0] [15:14:17] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1210 is OK Less than 1.00% above the threshold [0.0] [15:16:23] 6Labs: Remove NFS mounts from project cephtest - https://phabricator.wikimedia.org/T102381#1421137 (10Andrew) yep, and/or delete the project. [15:16:45] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1409 is OK Less than 1.00% above the threshold [0.0] [15:16:53] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-102, 3Labs-Sprint-103, 3Labs-Sprint-104: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1421146 (10yuvipanda) [15:16:53] RECOVERY - Puppet failure on tools-master is OK Less than 1.00% above the threshold [0.0] [15:16:55] 6Labs: Remove NFS mounts from project cephtest - https://phabricator.wikimedia.org/T102381#1421143 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Ah, cool. Can you just delete it then? [15:16:58] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1406 is OK Less than 1.00% above the threshold [0.0] [15:17:00] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Sprint-102, 3Labs-Sprint-103, 3Labs-Sprint-104: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#1360119 (10yuvipanda) [15:17:02] 6Labs: Remove NFS mounts from project cephtest - https://phabricator.wikimedia.org/T102381#1421149 (10yuvipanda) 5Resolved>3Open [15:17:03] RECOVERY - Puppet failure on tools-exec-1212 is OK Less than 1.00% above the threshold [0.0] [15:17:11] RECOVERY - Puppet failure on tools-exec-1407 is OK Less than 1.00% above the threshold [0.0] [15:17:39] RECOVERY - Puppet failure on tools-services-01 is OK Less than 1.00% above the threshold [0.0] [15:17:39] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1207 is OK Less than 1.00% above the threshold [0.0] [15:17:45] RECOVERY - Puppet failure on tools-webgrid-generic-1401 is OK Less than 1.00% above the threshold [0.0] [15:18:01] RECOVERY - Puppet failure on tools-exec-cyberbot is OK Less than 1.00% above the threshold [0.0] [15:18:23] RECOVERY - Puppet failure on tools-webgrid-generic-1403 is OK Less than 1.00% above the threshold [0.0] [15:18:31] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1206 is OK Less than 1.00% above the threshold [0.0] [15:18:31] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1410 is OK Less than 1.00% above the threshold [0.0] [15:18:35] RECOVERY - Puppet failure on tools-exec-1406 is OK Less than 1.00% above the threshold [0.0] [15:19:00] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK Less than 1.00% above the threshold [0.0] [15:19:42] RECOVERY - Puppet failure on tools-exec-1209 is OK Less than 1.00% above the threshold [0.0] [15:21:38] RECOVERY - Puppet failure on tools-exec-1211 is OK Less than 1.00% above the threshold [0.0] [15:26:15] hello, can someone help me? I tried to access my instance per WinSCP, with a tunnel, but the open of the tunnel failed. [15:26:49] which instance? [15:27:01] rcm-1 [15:27:18] my own istance, it create it yesterday [15:27:20] can you login over ssh normally? [15:27:28] yes, with putty [15:27:37] 6Labs, 10Tool-Labs: toolsbeta-exec-01 is not accessible - https://phabricator.wikimedia.org/T101805#1421187 (10scfc) [15:27:39] 6Labs, 10Tool-Labs: toolsbeta login broken - https://phabricator.wikimedia.org/T104544#1421189 (10scfc) [15:28:11] 6Labs, 10Tool-Labs: toolsbeta login broken - https://phabricator.wikimedia.org/T104544#1421191 (10valhallasw) 5Open>3Resolved [15:32:23] YuviPanda Hi, I saw that your patch had been submitted... does it mean that the problem should be fixed now? [15:32:40] physikerwelt: nope, not yet. still on it. [15:32:47] testing to make sure we don't blow everything up :) [15:32:51] before actually turning it on [15:33:04] ok [15:36:10] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Gabriel Birke was created, changed by Gabriel Birke link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Gabriel_Birke edit summary: Created page with "{{Tools Access Request |Justification=Development projects for Wikimedia DE |Completed=false |User Name=Gabriel Birke }}" [15:37:06] :| why is role::mail applied to toolsbeta hosts.... [15:37:25] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Gabriel Birke was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=168740 edit summary: [15:40:27] oh, hiera. Right. [16:03:19] * CP678|away is away: This is a manual computer virus. Please copy paste me in your away message. I'm not here right now. [16:03:46] CP678|away: please turn off that script [16:09:30] bd808: the cirrus role in mwv has an onlyif => 'mwscript ....' that needs to become a foreachwiki. Any good ideas on how to make that work right though? the script being run exits with 0 if it needs to run, 1 if nothing needs to happen. It seems iffy to just adjust foreachwiki to base its retval on this semantic tough [16:09:56] i could make foreachwiki change its return value based on an env variable? [16:10:43] basically foreachwiki would exit 0 at the end if any one wiki exit'd with a 0 retval [16:11:56] (which also feels backwards) [16:12:20] !log tools stopping puppet on tools-login and tools-mail to check for changes in deploying https://gerrit.wikimedia.org/r/#/c/205914/ [16:12:22] YuviPanda: ^ [16:12:25] Logged the message, Master [16:12:36] valhallasw`cloud: tools-login isn't a real host. [16:12:54] !log tools I mean tools-bastion-01 [16:12:58] Logged the message, Master [16:17:59] toollabs: how is "www/content/list.php" deployed [16:19:10] mutante: build a debian package, deploy on /data/project/.system/something, ???, PROFIT [16:19:37] (03CR) 10Dzahn: [C: 032] Fix some HTML validation errors [labs/toollabs] - 10https://gerrit.wikimedia.org/r/221091 (owner: 10Ricordisamoa) [16:20:06] 6Labs, 10Tool-Labs, 5Patch-For-Review: Multiple queue runners on tools-mail - https://phabricator.wikimedia.org/T74867#1421365 (10valhallasw) 5Open>3Resolved a:3valhallasw [16:20:44] valhallasw`cloud: .. wow.. eh.. thanks.. [16:21:43] mutante: yeah, not that great. I think building the package is not that hard (just dpkg-buildpackage, but I always forget the magic incantations) [16:22:18] debuild -us -uc .. i would hope [16:22:37] valhallasw`cloud: did you test mail (incoming / outgoing) to see if it still works fine? [16:22:43] mutante: valhallasw`cloud nope, the php stuff is just a git pull :) [16:22:46] no packages needed. [16:22:49] oh, good [16:22:50] that repository is crap anyway [16:22:56] needs to be split up and fixed... [16:22:59] so many things to do.. [16:23:00] YuviPanda: nice, where should i pull [16:23:01] YuviPanda: no, not really, but the config file didn't change =p [16:23:17] mutante: it should automatically get pulled on puppet runs [16:23:25] YuviPanda: yay :) thx [16:23:27] mutante: if not, /data/project/admin/public_html [16:23:44] 'k [16:24:15] it's not listening on port 25 for external connections though O_o [16:25:19] but that seems to be a firewall issue? [16:25:19] :/ [16:27:21] what on earth. it has security group MTA, which allows incoming port 25, I can connect to 10.68.16.27:25 (internally) but not 208.80.155.162:25 (externally) [16:28:37] the only thing I can think of is my provider blocking it, but that would be odd :P [16:29:22] valhallasw`cloud: i can connect to that [16:29:33] Connected to 208.80.155.162. [16:29:33] Escape character is '^]'. [16:29:39] 220 mail.tools.wmflabs.org [16:30:29] mutante: I guess my provider /is/ blocking the connection then O_o or there's some kind of routing issue [16:32:38] port 6666 works [16:32:51] * valhallasw`cloud hits his provider [16:36:17] valhallasw`cloud: it probably thinks if you are talking directly to port 25 out there it means your PC is owned and spamming [16:36:32] which is reasonable [16:36:48] but annoying :-p [16:37:52] PROBLEM - Puppet failure on tools-webproxy-02 is CRITICAL 60.00% of data above the critical threshold [0.0] [16:38:33] Error: /Stage[main]/Toollabs::Proxy/Git::Clone[labs/toollabs]/Exec[git_pull_labs/toollabs]/returns: change from notrun to 0 failed: /usr/bin/git pull --quiet returned 1 instead of one of [0] [16:38:36] let me ssh in... [16:41:34] and it magically solved itself. Bah :P [16:41:46] probably two servers pulling at the same time [16:57:50] got a good question [16:57:53] RECOVERY - Puppet failure on tools-webproxy-02 is OK Less than 1.00% above the threshold [0.0] [16:57:57] we've moved from #xtools to #wikimedia-xtools [16:58:03] I see this phab ticket: https://phabricator.wikimedia.org/rTWBT4500101f021b8eec83899848932edaee98bd680a [16:58:05] is this important? [16:59:41] elee: that's a wikibug config [16:59:44] 6Labs, 10Labs-Infrastructure: upgrade to Openstack Kilo - https://phabricator.wikimedia.org/T104586#1421635 (10Andrew) 3NEW [16:59:52] it should probably be updated with your new channel [16:59:56] 6Labs, 10Labs-Infrastructure: Upgrade Labs to Openstack Juno - https://phabricator.wikimedia.org/T104587#1421642 (10Andrew) 3NEW [16:59:58] Negative24: I also see https://phabricator.wikimedia.org/rTWBT72718ffe7bc6e4a03b7bb1c3f561664bc17a78ae is relevant [17:00:06] 6Labs, 10Labs-Infrastructure: Upgrade Labs to Openstack Juno - https://phabricator.wikimedia.org/T104587#1421642 (10Andrew) [17:00:07]