[00:18:27] $ dsh -g apaches -cM 'test -e /sys/fs/cgroup/memory || echo help' [00:18:27] mw1035: help [00:18:27] mw1145: help [00:18:27] mw1078: help [00:18:27] mw1152: help [00:18:27] mw1150: help [00:21:35] !log some apaches are missing /sys/fs/cgroup/* so shell execution from MW fails. Stopped apache on mw1035 during investigation [00:21:45] Logged the message, Master [00:23:39] PROBLEM - Apache HTTP on mw1035 is CRITICAL: Connection refused [00:24:00] it's not mounted [00:25:06] (03CR) 10Reedy: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 (owner: 10Reedy) [00:29:15] !log on mw1035,mw1145,mw1078,mw1152,mw1150: running "start cgconfig" to fix cgroups [00:29:26] Logged the message, Master [00:31:06] (03PS1) 10Reedy: Rename www portals files to drop wikipedia from name [operations/apache-config] - 10https://gerrit.wikimedia.org/r/89760 [00:35:12] I love our apache configs [00:35:14] #arbcom_nlwiki Wiki [00:35:14] [00:35:14] DocumentRoot "/usr/local/apache/common/docroot/arbcom_nlwiki" [00:35:14] ServerName arbcom.de.wikimedia.org [00:36:15] Already removed in https://gerrit.wikimedia.org/r/#/c/89329/ [00:37:19] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 15 00:37:16 UTC 2013 [00:37:24] more cgroup breakage: http://paste.tstarling.com/p/PBaCGB.html [00:37:39] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [00:41:34] !log manually created MW cgroups on all apaches since apparently the init script is totally broken [00:41:47] Logged the message, Master [00:59:39] RECOVERY - Apache HTTP on mw1035 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.296 second response time [01:03:46] springle, awake? [01:05:27] andrewbogott: yup [01:05:36] can you help me troubleshoot a broken mysql install? [01:05:46] ok [01:05:55] on labs instance wikimetrics [01:06:10] I've modified things a bit and now mysql won't start. I think it's an apparmor issue but I can't pin it down. [01:06:35] hm… probably I need to give you access to the system; what's your username on wikitech? [01:06:49] Springle [01:07:12] ok -- let me know if you can't access or can't sudo or anything. [01:07:28] I think it has a public ip… probably you can just ssh wikimetrics.wmflabs.org [01:08:02] hm, maybe not, I guess you need to go via bastion [01:12:15] where is mysqld err going? syslog? [01:12:33] /var/log/mysql [01:20:13] definitely apparmor [01:21:18] andrewbogott: what was the recent modification you made? my.cnf or apparmor mysqld profile? [01:21:40] Well... [01:21:59] Switching over to the mysql module. It didn't manage apparmor formerly, so I'm trying to get things caught up. [01:22:02] So, both... [01:22:20] But at this point it's the necessary apparmor changes that I'm curious about. [01:28:01] springle, it seems like apparmor is protecting the datadir, but the setting looks right to me... [01:28:54] i just added the /mnt/mysql/** rwk, line [01:28:59] seems happy now [01:29:17] andrewbogott: ^ [01:30:55] hm, ok… I must be confused about where the proper datadir is. [01:31:12] Anyway, thanks, I can sort it out. [01:31:13] proper? [01:32:22] /etc/mysql/my.cnf:20 is what i assumed the datadir to be :) [01:32:24] (03PS1) 10Ryan Lane: Fix docroot ownership for sartoris project [operations/puppet] - 10https://gerrit.wikimedia.org/r/89761 [01:33:24] springle: yeah, I was distracted by /var/lib/mysql [01:34:32] :) [01:37:29] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 15 01:37:22 UTC 2013 [01:37:39] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [01:47:44] (03CR) 10Ryan Lane: [C: 032] Fix docroot ownership for sartoris project [operations/puppet] - 10https://gerrit.wikimedia.org/r/89761 (owner: 10Ryan Lane) [01:53:33] !log rebooting mw1035 for init testing [01:53:51] Logged the message, Master [01:54:56] (03PS1) 10Andrew Bogott: Further attempts at mysql module apparmor. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89762 [01:56:49] PROBLEM - Host mw1035 is DOWN: PING CRITICAL - Packet loss = 100% [01:57:01] (03CR) 10Andrew Bogott: [C: 032] Further attempts at mysql module apparmor. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89762 (owner: 10Andrew Bogott) [01:57:19] RECOVERY - Host mw1035 is UP: PING OK - Packet loss = 0%, RTA = 2.43 ms [01:59:39] PROBLEM - Apache HTTP on mw1035 is CRITICAL: Connection refused [02:01:39] RECOVERY - Apache HTTP on mw1035 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.093 second response time [02:01:50] (03PS1) 10Andrew Bogott: More app-armor floundering! [operations/puppet] - 10https://gerrit.wikimedia.org/r/89763 [02:02:56] (03CR) 10Andrew Bogott: [C: 032] More app-armor floundering! [operations/puppet] - 10https://gerrit.wikimedia.org/r/89763 (owner: 10Andrew Bogott) [02:14:53] !log LocalisationUpdate completed (1.22wmf20) at Tue Oct 15 02:14:53 UTC 2013 [02:15:10] Logged the message, Master [02:20:18] !log LocalisationUpdate completed (1.22wmf21) at Tue Oct 15 02:20:18 UTC 2013 [02:20:33] Logged the message, Master [02:25:32] (03PS1) 10Andrew Bogott: Switch labs instances to use the mysql module. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89764 [02:37:25] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 15 02:37:21 UTC 2013 [02:37:39] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Oct 15 02:37:39 UTC 2013 [02:37:45] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [02:37:51] Logged the message, Master [02:48:10] (03PS1) 10MZMcBride: Minor tweaks to admins.pp. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89765 [03:27:47] do we have generic names for the newer & older branches in production? like 'unstable' / 'stable', or something like that. [03:28:43] there was some talk of having names for git-deploy to use. but those would be alternating. you couldn't know from the name which was newer [03:29:26] hmm, thanks. [03:30:55] i suppose you could do "which does the "php" symlink point to?" programatically [03:31:11] and it's also easy to get a list of all current versions [03:31:14] ori-l [03:37:31] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 15 03:37:21 UTC 2013 [03:37:41] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [03:39:38] jeremyb: yes, i was just fishing for a less clumsy locution for "the newer of the two branches deployed to production at a given point in time" [03:44:04] ori-l: oh, you're talking to a human not a machine, huh [03:44:24] yes [04:37:27] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 15 04:37:18 UTC 2013 [04:37:47] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [05:16:56] RECOVERY - Host cp1052 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [05:18:53] !log powercycled cp1052, it was unresponsive and unreachable via mgmt [05:19:12] Logged the message, Master [05:37:26] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 15 05:37:17 UTC 2013 [05:37:46] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [05:46:57] (03PS4) 10TTO: Miscellaneous cleanup of InitialiseSettings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86415 [06:37:26] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 15 06:37:18 UTC 2013 [06:37:46] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [06:53:12] (03PS1) 10ArielGlenn: depool db1022 to use as source for clone of other db [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89772 [06:55:06] (03CR) 10ArielGlenn: [C: 032] depool db1022 to use as source for clone of other db [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89772 (owner: 10ArielGlenn) [06:56:51] !log ariel synchronized wmf-config/db-eqiad.php 'db1022 (s6) depooled to use as clone source' [06:57:05] Logged the message, Master [07:24:15] cgroups again, sigh [07:37:30] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 15 07:37:22 UTC 2013 [07:37:40] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [07:56:40] PROBLEM - RAID on ms-be1008 is CRITICAL: CRITICAL: 1 failed logical drive(s) (Offline) [07:57:10] PROBLEM - Disk space on ms-be1008 is CRITICAL: DISK CRITICAL - /srv/swift-storage/sdc1 is not accessible: Input/output error [08:00:12] RECOVERY - Disk space on ms-be1008 is OK: DISK OK [08:11:01] (03CR) 10Akosiaris: [C: 032] Minor tweaks to admins.pp. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89765 (owner: 10MZMcBride) [08:14:07] (03CR) 10Akosiaris: [V: 032] Minor tweaks to admins.pp. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89765 (owner: 10MZMcBride) [08:14:53] (03PS1) 10ArielGlenn: db1006 (s6) -> file_per_table, mariadb [operations/puppet] - 10https://gerrit.wikimedia.org/r/89776 [08:15:54] (03CR) 10ArielGlenn: [C: 032] db1006 (s6) -> file_per_table, mariadb [operations/puppet] - 10https://gerrit.wikimedia.org/r/89776 (owner: 10ArielGlenn) [08:17:50] who's got the changes to modules/mysql/templates/apparmor.template.usr.sbin.mysqld.erb unmerged? [08:18:00] seems like andrewbogott [08:18:06] i am looking at the same thing now [08:18:22] worst thing is: git show 9e3493dc [08:18:39] says it is not just those 2 lines ... but 5 in total [08:18:42] andrewbogott_afk: [08:19:04] yep [08:19:54] i think we don't use the module however... [08:20:01] at least not yet... [08:23:42] so ... it's two different patchsets [08:23:54] 9e3493dca52bdef910ca89801e2a7ea09fd67ecc and 1ed783d5aea635c6e6cce867f6110517db29f5ba [08:24:06] I'm going to merge it (including the admin.pp changes) [08:24:25] he is doing something and then partly undoing it... [08:24:43] it probably won't hurt but i am unsure of what he was trying to achieve [08:25:22] since it's not going to affect the production myqls (afaict) [08:25:39] merged [08:25:52] and I'll just check on a live db host anyways [08:28:20] looks great [08:28:29] seems like it will affect neon. For some reason icinga::monitor includes mysql, probably to have virtual resources realized [08:28:51] mmmm [08:29:02] well let's see what it does over there [08:29:52] mm puppet takes a while to run there (but it does complete eventually) [08:30:03] sometimes [08:30:13] cause others it just says execution expired [08:35:07] actually since reducing run frequency (and getting errors in the manifest fixed) neon has been completing runs consistently [08:36:28] naggen neeeds a rewrite [08:36:40] but maybe we can just ditch it and go with puppetdb [08:37:29] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 15 08:37:22 UTC 2013 [08:49:04] anyways seems like that change didn't touch neon [08:49:14] :-) [08:51:36] do you guys have experience with puppetdb? if so, how was it? [09:03:15] i don't have any... But it is something we should look at [09:06:03] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [09:29:45] hello [09:34:51] hashar: hello \O/ [09:36:10] (03PS1) 10Ori.livneh: NavigationTiming: aggregate data by bits cache host; add 'totalPageLoadTime' [operations/puppet] - 10https://gerrit.wikimedia.org/r/89785 [09:36:16] anyone available to review? [09:37:23] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 15 09:37:21 UTC 2013 [09:37:51] ori-l: stat = 'browser.%s.%s.%s:%s|ms' % (metric, bits_cache, value) [09:38:03] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [09:38:07] format string says 4, tuple is 3 elements [09:38:07] god damn it, good eye [09:38:21] ok, you pass the test [09:38:32] :-) [09:38:59] (03PS2) 10Ori.livneh: NavigationTiming: aggregate data by bits cache host; add 'totalPageLoadTime' [operations/puppet] - 10https://gerrit.wikimedia.org/r/89785 [09:39:07] now the real commit [09:41:00] just out of curiosity... [09:41:01] bits_cache = meta.get('recvFrom', '').split('.')[0] [09:41:09] this line will always succeed ? [09:41:20] or can it be that there will be an indexError ? [09:41:28] not sure what is does btw... :-) [09:42:21] it can't be an index error [09:42:53] ''.split('nonexistent separator')[0] = '' [09:43:18] it gets the first part of the hostname [09:44:24] ok, thanks. LGTM then. Want me to merge now ? [09:44:38] sure, thanks! [09:45:55] akosiaris: meta is a has, the get() methods attempt to retrieve the value for the key 'recvFrom', if there the key does not exist it returns an empty string. [09:45:56] BUT [09:46:02] if the key contains None, you get an error :-] [09:46:22] >>> {'key': None}.get('key','').split('foo') [09:46:22] hashar: SyntaxError: Unexpected token : [09:46:23] AttributeError: 'NoneType' object has no attribute 'split' [09:46:37] ecmabot-wm: learn python [09:46:37] hashar: There is no command: learn python [09:47:32] hashar: lol... yeah it makes sense. I suppose though that scenario will not show up ? [09:48:02] cause you need code to explicitly set recvFrom to None [09:48:33] which sounds bad from an architectural point of view... Just not having the key in the hash is safer [09:50:29] (03CR) 10Akosiaris: [C: 032] "After a quick chat with Ori on IRC about a question I had LGTM." [operations/puppet] - 10https://gerrit.wikimedia.org/r/89785 (owner: 10Ori.livneh) [09:51:24] yes, it's not going to be None [09:52:20] akosiaris: if it was for the space shuttle, I would catch the exception and run some fallback code :-] [09:52:24] but he [09:52:31] we are not in rocket science [09:52:55] so yeah, as long as the 'recvFrom' is set to something having a split() method, we should be fine [09:52:55] i it was a space shuttle i would require you to mathematically prove every line of code :P [09:53:02] this is consumed from the validated event stream [09:53:08] it is a string or its absent [09:53:20] I have met someone working for Airbus [09:53:49] I can't recall the detail, but basically they are using some language that let you mathematically prove that all possible code paths combinaisons are valid [09:53:53] you can still have bugs though [09:54:15] ATS or Haskell, usually [09:54:18] but at least they are feature, not that much unexpected one [09:54:42] sometimes ML [09:54:48] ah ADA [09:55:00] I think it was that one [09:55:14] we got someone at the WMF that used to do avionic programming, not sure who though [09:55:29] mwalker used to do embedded stuff IIRC [09:58:13] well, puppet was pretty much blazing fast [09:58:19] and i see that cpu usage on stafford is down too [09:58:27] what was it? just the change to 60 mins rather than 30/ [10:00:40] I guess so [10:00:52] I am not sure how puppet client /server works [10:01:08] but it seems the client request a catalog using the unix epoch [10:01:16] which would trigger a catalog recompilation on the server [10:02:01] with 600 servers we have like 10 recompilations per minutes now instead of 20 [10:02:11] (pure speculations) [10:03:10] I have seen some installations using something like `git rev-parse HEAD` for the catalog version [10:03:27] and a friend just installed a varnish in front of puppet to cache catalogs :D [10:03:45] some others are rsyncing the puppet files to all servers which run their own local puppetmaster [10:03:48] but that is evil [10:05:35] akosiaris: thank you for the merge of my evil git::clone( integration/jenkins ) ( https://gerrit.wikimedia.org/r/87058 ) [10:05:49] hashar: :-) [10:06:06] /sbin/iptables -A tftp -s 198.35.26.0/22 -j ACCEPT [10:06:11] I will eventually migrate it to Sartoris whenever I found out how to use it :-] [10:06:13] * akosiaris sad.... [10:06:21] we don't even own /22... we own /23... [10:06:52] we need to put all those things in one single place. That info is scattered all over the place.... :-( [10:08:19] yes, just recuding the puppet run frequency had that impact on stafford [10:08:22] *reducing [10:13:57] ori-l: if still around I was wondering whether you were still maintaining deployment-fluoride.pmtpa.wmflabs . It is a receiver for some log2udp on ports 8423 and 8324 but there are no process listening on deployment-fluoride [10:14:18] ori-l: not really important though [10:26:16] (03PS1) 10ArielGlenn: repool db1006 (s6) warming up, after upgrade/convert to mariadb [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89790 [10:27:43] (03CR) 10ArielGlenn: [C: 032] repool db1006 (s6) warming up, after upgrade/convert to mariadb [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89790 (owner: 10ArielGlenn) [10:29:18] !log ariel synchronized wmf-config/db-eqiad.php 'db1006 (s6) repooled, warming up after upgrade/conversion' [10:29:34] Logged the message, Master [10:33:27] (03PS1) 10Akosiaris: Adding IPv4/6 networks in ferms defs [operations/puppet] - 10https://gerrit.wikimedia.org/r/89791 [10:37:24] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 15 10:37:20 UTC 2013 [10:37:54] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [10:47:44] RECOVERY - HTTPS on cp4003 is OK: OK - Certificate will expire on 01/20/2016 12:00. [10:47:54] RECOVERY - HTTPS on cp4004 is OK: OK - Certificate will expire on 01/20/2016 12:00. [10:48:04] RECOVERY - HTTPS on cp4002 is OK: OK - Certificate will expire on 01/20/2016 12:00. [10:50:28] (03CR) 10Mark Bergsma: [C: 04-1] "I don't like doing this in a temporary and not very well defined way, because it will get used all over the place before we ever get aroun" [operations/puppet] - 10https://gerrit.wikimedia.org/r/89791 (owner: 10Akosiaris) [11:29:09] mark: You are right about this https://gerrit.wikimedia.org/r/#/c/89791. It's just that i needed it in an install-server module rewrite and it was an even worse idea to stick it in there. I suppose I can stall the module till I get the time to chase all this down and clear it up. Aside from this, any objections on the approach of using network.pp to document all networks and using it to populate firewall rules ? [11:29:26] that's exactly what we planned [11:29:36] that's the thing; we want ferm to finally do this the right way and fix all this [11:29:43] by putting temporary measures in there too, it will never happen :) [11:30:02] I'd much rather have yet another temp crappy old iptables rule in puppet than to do almost the same thing in ferm :) [11:30:46] ok. Stalling the install-server module until we find the time to clean that thing up then [11:30:54] thanks [11:36:45] going to the ripe venue now [11:37:24] !log jenkins refreshing all mediawiki extensions jobs so they log Junit results under /logs/ [11:37:25] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 15 11:37:20 UTC 2013 [11:37:37] Logged the message, Master [11:37:55] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [12:09:26] (03PS1) 10Lcarr: fixing rancid file with new routers [operations/puppet] - 10https://gerrit.wikimedia.org/r/89808 [12:11:10] (03CR) 10Lcarr: [C: 032] fixing rancid file with new routers [operations/puppet] - 10https://gerrit.wikimedia.org/r/89808 (owner: 10Lcarr) [12:37:25] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 15 12:37:19 UTC 2013 [12:37:54] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [13:23:10] (03PS1) 10Lcarr: IPv6 blocks for ulsfo interconnects and loopbacks [operations/dns] - 10https://gerrit.wikimedia.org/r/89822 [13:23:57] (03CR) 10Lcarr: [C: 032] IPv6 blocks for ulsfo interconnects and loopbacks [operations/dns] - 10https://gerrit.wikimedia.org/r/89822 (owner: 10Lcarr) [13:26:34] (03PS12) 10Reedy: WIP don't deduce sites based on docroot stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 [13:26:38] (03PS13) 10Reedy: Don't deduce sites based on docroot stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 [13:28:56] (03PS1) 10ArielGlenn: db1006 (s6) back to full weight in the pool [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89824 [13:29:45] (03CR) 10ArielGlenn: [C: 032] db1006 (s6) back to full weight in the pool [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89824 (owner: 10ArielGlenn) [13:31:27] !log ariel synchronized wmf-config/db-eqiad.php 'db1006 (s6) back to full weight in pool' [13:31:40] Logged the message, Master [13:36:12] (03PS1) 10Reedy: All non wikipedias to 1.22wmf21 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89825 [13:37:25] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 15 13:37:23 UTC 2013 [13:37:55] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [13:59:21] (03PS1) 10Reedy: Add urls file for testing with apache-fast-test etc [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89828 [13:59:47] (03CR) 10Reedy: [C: 032] Add urls file for testing with apache-fast-test etc [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89828 (owner: 10Reedy) [14:00:17] (03Merged) 10jenkins-bot: Add urls file for testing with apache-fast-test etc [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89828 (owner: 10Reedy) [14:22:54] (03PS14) 10Reedy: Don't deduce sites based on docroot stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 [14:23:20] (03CR) 10Reedy: [C: 032] Don't deduce sites based on docroot stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 (owner: 10Reedy) [14:23:29] (03Merged) 10jenkins-bot: Don't deduce sites based on docroot stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 (owner: 10Reedy) [14:23:45] (03PS7) 10Reedy: Update tests to remove docroot setting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85172 [14:23:50] (03CR) 10Reedy: [C: 032] Update tests to remove docroot setting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85172 (owner: 10Reedy) [14:24:01] (03Merged) 10jenkins-bot: Update tests to remove docroot setting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85172 (owner: 10Reedy) [14:29:50] (03PS1) 10Reedy: Correct a few minor url issues to skip 404s etc [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89830 [14:30:33] (03CR) 10Reedy: [C: 032] Correct a few minor url issues to skip 404s etc [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89830 (owner: 10Reedy) [14:32:01] (03Merged) 10jenkins-bot: Correct a few minor url issues to skip 404s etc [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89830 (owner: 10Reedy) [14:35:29] !log reedy synchronized multiversion/ 'Changing away from docroot based site deduction' [14:35:41] Logged the message, Master [14:35:57] Well, enwiki isn't broken! :D [14:35:59] aude: ^^ [14:36:45] :) [14:37:06] brave to refactor that [14:37:30] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 15 14:37:22 UTC 2013 [14:38:00] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [14:47:06] (03CR) 10Reedy: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 (owner: 10Reedy) [14:53:59] (03CR) 10Reedy: "I've deployed the PHP code and that seems to be working fine, so this should be good to be retested for deployment :)" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/84707 (owner: 10Reedy) [14:57:23] Anyone around who fancies merging and pushing some apache config cleanup for me please? [15:04:51] !log reedy synchronized php-1.22wmf21/extensions/Wikibase [15:05:03] Logged the message, Master [15:08:36] Reedy: do you have +2 on puppets? :) [15:08:45] I don't, sorry :( [15:08:56] (03PS1) 10Yurik: Allow ESI only if enable_esi flag is set in the puppet cluster options [operations/puppet] - 10https://gerrit.wikimedia.org/r/89836 [15:08:56] its ok :) [15:09:11] Reedy: but do you think you could explain how extract2 works for me? :) [15:09:23] i want to make something similar for zero, need your expertise [15:09:37] Uh [15:09:38] i think you were the one who wrote that thing in the beginning [15:09:41] yeah [15:10:06] Haha, no [15:10:17] If it all blames to me, then I probably imported it into git [15:10:31] mark or paravoid, could you +2 https://gerrit.wikimedia.org/r/89836 - its for beta cluster testing of ESI [15:10:52] Reedy: do you know who is the one to blame (explain) that thing? [15:10:55] it looks scarry [15:11:31] hashar: actually that thing was your suggestion :) ^ [15:11:44] What's difficult about it? It's relatively simple.. [15:13:29] q [15:13:31] gah [15:14:24] yurik_: that's the thing that makes the main page of www.wikipedia.org? [15:14:33] aude: yep [15:14:38] that thing scares me [15:14:49] i am trying to figure out how to make it so that m.wiki and zero.wiki could have a custom startup page that is generated by apache instead of being a hardcoded wiki page on meta [15:14:52] it gets the templates from meta wiki [15:14:57] exactly [15:15:09] parse / get content and displays it [15:15:17] amazingly simple and scary :) [15:15:19] but i need to run some code to generate that page if the X-CS header is set and if the user comes to m. or zero. [15:15:30] oh, no images [15:15:43] aude: no images is fine [15:15:51] is that the requirement? [15:15:59] or more? [15:15:59] but there are lots of weird dirs in the config [15:16:19] ? [15:16:20] (03CR) 10Mark Bergsma: [C: 04-1] "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/89836 (owner: 10Yurik) [15:18:01] mark, but that feature is only for the beta cluster testing [15:18:10] and nothing bad would happen if I manually set that header [15:18:21] without coming from the -TEST range [15:18:47] yurik_: not sure i really understand completely what you need [15:19:35] yurik_: yup that would be fine :-] [15:19:46] yurik_: this way you can enable_esi on the staging instance for mobile [15:19:47] hashar: mark -1 it :( [15:19:53] see his comment :-] [15:19:56] and he is not here to answer [15:19:59] he is the actual varnish expert hehe [15:20:23] yeah, that's true [15:20:34] one of those cases when its easier just do as asked :( [15:23:29] yes [15:23:31] (03PS2) 10Yurik: Allow ESI only if enable_esi flag is set in the puppet cluster options [operations/puppet] - 10https://gerrit.wikimedia.org/r/89836 [15:24:24] (03CR) 10Mark Bergsma: [C: 032] Allow ESI only if enable_esi flag is set in the puppet cluster options [operations/puppet] - 10https://gerrit.wikimedia.org/r/89836 (owner: 10Yurik) [15:24:36] (03CR) 10Yurik: "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/89836 (owner: 10Yurik) [15:27:04] aude: in case the X-CS header is set (meaning that the user comes from a known mobile partner network), any visit to m.wikipedia.org or zero.wikipedia.org (or any other m.* actually) should see a custom-generated page [15:27:52] (03CR) 10Mark Bergsma: "You're mixing tabs and spaces here" [operations/dns] - 10https://gerrit.wikimedia.org/r/89822 (owner: 10Lcarr) [15:29:22] yurik_: ah ok [15:29:27] that page is currently generated at en.m.wikipedia.org/wiki/Special:ZeroRatedMobileAccess but you can't view it unless you set header on your browser: X-CS:250-99 [15:29:40] but it shouldn't be part of "en" [15:29:49] it should be in the language of the carrier [15:31:16] yurik_: so have it on meta wiki? [15:31:32] aude: what do you mean? its not a static page [15:31:33] * aude really doesn't know the best place [15:31:42] it's a special page [15:31:54] have one on all wikipedias? or it just one place? [15:32:21] meta contains the raw html of the page that extract2 yields in the place of "www.wikipedia.org" [15:32:34] right [15:32:43] what i need is something similar to extract2 that will yield a dynamic page [15:32:49] instead of m & zero subdomains [15:33:00] oh [15:33:47] and instead of the special page? [15:34:00] correct [15:35:59] * aude sees apache rewrite rule for zero :) [15:37:01] instead i assume it would redirect to the docroot page [15:37:23] RECOVERY - Puppet freshness on williams is OK: puppet ran at Tue Oct 15 15:37:18 UTC 2013 [15:37:24] and then not quite sure how to find out the languages [15:38:03] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [15:38:11] or be an alias or something [15:38:40] (03CR) 10Reedy: "Depends on https://gerrit.wikimedia.org/r/84707" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84721 (owner: 10Reedy) [15:38:47] aude: sorry, got disconnected, please repeat [15:39:10] yurik_2: i see things are handled with apache redirect to the special page [15:39:28] would it redirect instead (or alias) to something in docroot? [15:40:10] * aude would like to do fancy lang redirects for wikidata (btw) although it's complicated [15:42:14] yurik_: or maybe it's possible to obfusicate (w/ apache) the fact that users are being redirected to the special page? [15:47:02] mark: ping [15:47:09] pong [15:47:55] mark, re cassandra testing: labs is very IO-limited, does not have SSDs etc [15:47:55] you don't need that for functional testing [15:47:55] so would not actually provide useful benchmark data [15:48:00] I have done functional testing on my laptop [15:48:13] which happens to have an SSD, which helps [15:49:03] we actually want to push cassandra to see if it holds up, and how it performs with a big data set [15:50:31] what dataset are you going to put in? [15:50:55] initially a wikitext dump, then some old revisions and HTML [15:51:15] if i remember right from when i last investigated cassandra, the compaction where it merges sstables is where its most likely to run into issues(especially with IO) so will want to make sure enough data is going in to simulate regular compaction [15:51:34] yup [15:51:57] in local testing I imported a part of a wikitext times times [15:51:57] gwicke: who on Ops is working with you on that? [15:51:57] *a wikitext dump [15:52:17] mark: Ryan_Lane is providing moral support [15:53:24] otherwise it is mainly a matter of trying it [15:53:24] sounds like good stuff :) I liked cassandra, but was just leery about their php support and current unfinished transition to CQL3, but from an eng standpoint its a nice system [15:53:24] installing cassandra is rather easy [15:53:25] * mark sighs [15:53:51] mark: I also offered to do the apt-get install via puppet, but without sudo tweaking the configs and simulating failures will be slower [15:54:21] it might make more sense to figure that out without puppet, and then puppetize when the first testing was successful [15:54:52] yes [15:56:40] if anybody in ops is interested to join the experiment then that would be great too of course [15:56:40] that would be a good idea [15:56:40] or this will turn into another failed adoption [15:56:41] i'll discuss it in our meetings later today [15:57:33] mark: is that the Monday meeting moved to Tuesday? [15:57:34] the internal ops meeting [15:57:34] yes [15:57:34] there's also a scrum of scrums before that [15:57:34] ah, not the Monday morning Dev/Ops meeting [15:57:34] it would be good to discuss it there too [15:57:41] plenty of meetings today [15:57:46] apparently [15:58:09] my monday calls have been moved today [15:59:00]