[00:00:00] (03CR) 10TTO: "I thought about this the other day. I still think it should be done, but perhaps using names like "Test Namespace 1", "Test Namespace 2", " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78016 (owner: 10TTO) [00:23:23] (03PS1) 10Springle: depool pc1001 for package upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98139 [00:24:17] (03CR) 10Springle: [C: 032] depool pc1001 for package upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98139 (owner: 10Springle) [00:25:36] !log springle synchronized wmf-config/db-eqiad.php 'depool pc1001 for package upgrade' [00:25:52] Logged the message, Master [00:34:35] (03CR) 10Springle: [C: 032] pc1001 upgrade to mariadb [operations/puppet] - 10https://gerrit.wikimedia.org/r/98140 (owner: 10Springle) [01:00:51] (03PS1) 10Springle: remove mysql 5.1 fb specific settings. needs further review for mariadb 5.5 once we know how it performs. [operations/puppet] - 10https://gerrit.wikimedia.org/r/98142 [01:02:03] (03CR) 10Springle: [C: 032] remove mysql 5.1 fb specific settings. needs further review for mariadb 5.5 once we know how it performs. [operations/puppet] - 10https://gerrit.wikimedia.org/r/98142 (owner: 10Springle) [01:20:36] (03PS1) 10Springle: repool pc1001 after upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98144 [01:21:00] (03CR) 10Springle: [C: 032] repool pc1001 after upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98144 (owner: 10Springle) [01:22:19] !log springle synchronized wmf-config/db-eqiad.php 'repool pc1001 after upgrade, max_connections lowered during warm up' [01:22:33] Logged the message, Master [02:01:44] !log LocalisationUpdate completed (1.23wmf4) at Fri Nov 29 02:01:44 UTC 2013 [02:02:01] Logged the message, Master [02:02:25] !log LocalisationUpdate completed (1.23wmf5) at Fri Nov 29 02:02:25 UTC 2013 [02:02:41] Logged the message, Master [02:07:55] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Nov 29 02:07:55 UTC 2013 [02:08:12] Logged the message, Master [02:23:51] the varnish metrics in ganglia are all wrong, aren't they? [02:26:45] I mean, what is this? http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&c=Text+caches+eqiad&h=cp1054.eqiad.wmnet&jr=&js=&v=2604995227&m=varnish.s_req&vl=N%2Fs&ti=Total+Requests [02:26:56] that looks like garbage to me [03:27:45] TimStarling: it really is the fault of Ganglia; new-style metric modules are crap [03:28:07] so you know what the bug is? [03:28:39] it's variants on a theme: gmond is all like, "don't call me, i'll call you" [03:28:52] but it calls you for individual metrics, so you have to keep your own means of determining when a set of metrics needs to be refreshed [03:29:13] plus, the logic for when gmond calls you is nuts, there are multiple variables that determine its behavior [03:29:48] so what inevitably happens is that your metric update logic and gmond's polling logic get slightly out of sync [03:30:44] whenever gmond attempts to get metric values more often than the metric values are refreshed, slope=positive metrics are broken, because the reporting of the same value twice makes the rate of change plummet to 0 [03:31:29] oh, well that is easily fixed [03:32:07] no one should ever use slope=positive; one should just use the gauge type (which graphs whatever value you send it) [03:32:16] and do the delta calculation in-module [03:33:17] I thought it was suspicious that of all the modules bundled in gmond/python_modules, only one uses slope=positive [03:35:56] the slope is a parameter to rrdtool create, so presumably it would be necessary to delete all the old RRDs in order to change the slope [03:36:44] they're cached in gmetad as well. there's a trick to getting rid of them, and it's truly awful. i hope you're ready for this. [03:37:07] * TimStarling braces [03:37:41] gmetric --spoof="foohost:foohost" -n foo.blah.metric -v expiring -t string -d 10 [03:38:02] interesting [03:38:25] basically, you report it as a string metric with a dmax of 10 [03:38:29] and ten seconds later it falls out of ganglia [03:39:05] (03CR) 10Jeremyb: [C: 04-1] "What's the latest spec for this? (i.e. desired functionality)" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/65443 (owner: 10Dzahn) [03:42:57] gmond is a bad solution to the problem they were trying to solve, IMO [03:43:12] it's common to have metrics that are somewhat expensive to update [03:44:27] in which case you want to ensure that you don't update too frequently [03:45:04] presumably that is what the collect_every parameter is for [03:45:06] probably enforcing a delay between the *end* of each metric refresh [03:46:05] yes, but consider that gmond calls you for an individual metric, not the whole set [03:50:32] it basically shows up at your door with a gun and demands that you refresh a metric *right now* or it'll kill your graph; it doesn't care that you're not ready to update it again yet [03:51:54] there's something else going on [03:52:04] packet loss, maybe [03:52:05] strong language [03:52:50] I watched UDP packets coming out of cp1054, and simultaneously watched the RRD on nickel [03:53:09] and many of the updates which were sent out were missed [03:55:49] well [03:56:14] ganglia sets the rrd step size when it first encounters the metric, based on its update interval [03:56:30] if you change the update interval, it doesn't update the rrd file [04:00:52] perhaps it drops values when a value has already been received during the current interval [04:02:28] http://paste.tstarling.com/p/duwuJw.html [04:04:46] you see, the gmond on cp1052 is missing updates [04:05:24] the packets are arriving regularly [04:08:01] anyway, missing updates would not be a problem if it was using GAUGE [04:12:56] # while true; do netstat -s | grep 'packet receive errors'; sleep 10; done [04:12:56] 564837446 packet receive errors [04:12:56] 564839480 packet receive errors [04:12:56] 564840427 packet receive errors [04:16:34] gross [04:21:18] it stalls for 85ms or so to calculate and send its own varnish metrics, but that's not quite enough to explain the effect [04:23:04] it may just be temporary CPU exhaustion [04:23:18] during the recvfrom() loop [04:25:50] what do you mean, "the gmond on cp1052 is missing updates"? [04:27:08] well, you know that every gmond sends metrics to multicast [04:27:22] yes [04:27:24] and some gmonds (in listen mode) join that multicast channel [04:27:40] oh, and act as relays? no, i didn't know that [04:28:00] yes, a complex sort of relay I suppose [04:28:15] they maintain the current state (all metric values) for all hosts that they receive metrics for [04:28:52] they also listen on TCP for connections from gmetad [04:29:16] gmetad has a configured list of data sources, usually 2 or 3 per cluster [04:29:34] gmetad connects to the first configured gmond in the cluster using TCP [04:29:45] the gmond writes out its current state in XML format [04:30:06] gmetad writes that data to the relevant RRD [04:30:33] gmetad itself also has state information, similar to the state held by each gmond [04:30:47] but it doesn't listen on any multicast channels [04:31:15] I wonder how long the XML write takes [04:44:59] ah, actually gmetad is selecting cp1053 for this cluster [04:46:28] doesn't seem to matter much [04:47:01] cp1053 has 41% packet loss, cp1052 has 42% packet loss [04:49:15] not sure why, in strace, receiving takes about 200µs per packet, and it should only be getting roughly one packet per 3ms [04:52:03] TCP is done in a separate thread, so it wouldn't cause stalls [04:53:58] well, those percentages are assuming errors are included in "packets received" [04:54:21] if not, it is more like 30% loss [05:51:44] !log on cp1052 and cp1053: tweaked /proc/sys/net/core/rmem_default to see if that fixes the observed massive gmond packet loss [05:51:58] Logged the message, Master [05:52:55] ori-l: http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&c=Text+caches+eqiad&h=cp1054.eqiad.wmnet&jr=&js=&v=2604995227&m=varnish.s_req&vl=N%2Fs&ti=Total+Requests [05:53:23] answer: yes [05:53:38] what was it before? [05:56:42] 229376 [05:57:08] i'm glad that the usage of COUNTER types made the problem apparent [05:57:28] but the fact that GAUGES are more resilient is to their credit [06:01:31] gotta run, but i enjoyed following along -- thanks [06:03:19] there's actually an option in gmond.conf for this, it's not necessary to set the sysctl [06:26:43] (03PS1) 10Tim Starling: Set SO_RCVBUF on gmond listeners to a sensible value [operations/puppet] - 10https://gerrit.wikimedia.org/r/98150 [07:42:47] (03CR) 10Faidon Liambotis: [C: 031] "Awesome, thanks!" (031 comment) [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/98135 (owner: 10Edenhill) [07:47:12] (03CR) 10Faidon Liambotis: [C: 032] "Amazing. Awesome investigation :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/98150 (owner: 10Tim Starling) [07:49:22] (03CR) 10Faidon Liambotis: Correctly calculate escape buffer size (031 comment) [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/98134 (owner: 10Edenhill) [08:39:33] (03CR) 10Faidon Liambotis: [C: 032] parsoid: role class for beta and factor out common code [operations/puppet] - 10https://gerrit.wikimedia.org/r/98014 (owner: 10Hashar) [09:23:23] paravoid: thank you for the merge of the role::parsoid::* [09:23:37] not sure whom to ask since I have no clue whether parsoid is ops or features [09:53:32] (03CR) 10Faidon Liambotis: "The only thing that wlm.wm.org does is redirect to a toolserver URL with no real content but an Apache directory index." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/98057 (owner: 10Faidon Liambotis) [09:54:38] hashar: for things in ops/puppet, default to ops :) [09:59:47] (03Abandoned) 10Reedy: Wrap extract2.php in ob_start()/ob_flush() [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92828 (owner: 10Reedy) [10:03:54] (03PS1) 10Hashar: zuul: let us change branch of zuul-config repo [operations/puppet] - 10https://gerrit.wikimedia.org/r/98155 [10:04:29] (03CR) 10Edenhill: [C: 031] Added %{VCL_Log:key}x support (031 comment) [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/98135 (owner: 10Edenhill) [10:05:35] (03CR) 10Edenhill: [C: 031] Correctly calculate escape buffer size (031 comment) [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/98134 (owner: 10Edenhill) [10:06:18] (03PS1) 10Edenhill: escaper: Comment magical constant numbers [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/98156 [10:06:37] Snaps: yup, which is why I +1ed it :) [10:07:13] Since I dont use one branch per commit there is another review for the *5 commenting. [10:07:23] Maybe I will learn one day, but until then... [10:11:44] git checkout -b foobar [10:16:58] (03CR) 10Reedy: "I'm presuming it's present on other projects due to the match of *.PROJECTNAME.org and then the RewriteRule ^/favicon\.ico$ /w/favicon.php" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/91209 (owner: 10Reedy) [10:18:02] I typically dont want to use a branch for each commit. In this case it would've been okay, but when making larger changes that are related multiple commits should really be used, and that wont fly well with multi branches [10:21:06] They're not full branches [10:21:10] they're feature branches [10:21:26] a branch is a branch [10:21:42] s/feature/topic/ [10:22:12] It's easier for some people, as they can just checkout master [10:22:26] Im not saying gerrit is wrong, its just disruptive to the standard git workflow Im used to :) [10:22:31] rather than having to git reset HEAD~1 --hard or similar [10:29:27] yes, depending on my mood I switch between reset --hard and topic branches, can't seem to settle into just one apporach [10:29:29] approach [10:30:03] I tend to be "lazy" and reset [10:30:34] if I think about it in advance and expect to be working on a few things at once then I'll do the branch [11:07:05] (03CR) 10Faidon Liambotis: [C: 04-1] "Sounds very useful, thanks for packaging this :) See inline for comments. Let me know if you'd like to upload this to Debian." (0315 comments) [operations/debs/python-kafka] (debian) - 10https://gerrit.wikimedia.org/r/97848 (owner: 10Ottomata) [11:12:46] !log Created EducationProgram tables on arwiki [11:13:02] Logged the message, Master [11:24:51] (03CR) 10Faidon Liambotis: [C: 04-1] "Besides inline comments, missing gbp.conf." (0312 comments) [operations/debs/logster] (debian) - 10https://gerrit.wikimedia.org/r/95556 (owner: 10Ottomata) [11:43:30] (03PS6) 10Faidon Liambotis: varnish: whitespace & lint cleanups [operations/puppet] - 10https://gerrit.wikimedia.org/r/97910 (owner: 10Matanya) [11:43:45] (03PS7) 10Faidon Liambotis: varnish: whitespace & lint cleanups [operations/puppet] - 10https://gerrit.wikimedia.org/r/97910 (owner: 10Matanya) [11:44:38] (03CR) 10jenkins-bot: [V: 04-1] varnish: whitespace & lint cleanups [operations/puppet] - 10https://gerrit.wikimedia.org/r/97910 (owner: 10Matanya) [11:45:57] (03PS8) 10Faidon Liambotis: varnish: whitespace & lint cleanups [operations/puppet] - 10https://gerrit.wikimedia.org/r/97910 (owner: 10Matanya) [11:49:28] (03CR) 10Faidon Liambotis: [C: 031] "I made some changes myself and it's good from my side. Considering how critical these manifests are for site operations, though, I'd like " [operations/puppet] - 10https://gerrit.wikimedia.org/r/97910 (owner: 10Matanya) [13:14:55] (03CR) 10Akosiaris: [C: 04-1] "Minor stuff mostly but seems like the best time to fix them." (039 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/97910 (owner: 10Matanya) [13:44:11] (03CR) 10Yuvipanda: [C: 031] "Kill it. The old app is no longer on the store (IIRC?) and the new app uses the toollabs URL[" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/98057 (owner: 10Faidon Liambotis) [13:52:34] (03CR) 10Akosiaris: [C: 031] "LGTM" [operations/software/varnish/varnishkafka] (debian) - 10https://gerrit.wikimedia.org/r/78782 (owner: 10Faidon Liambotis) [14:02:11] (03PS2) 10Edenhill: Tag column reader was used incorrectly [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/98136 [14:02:12] (03PS2) 10Edenhill: Added %{VCL_Log:key}x support [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/98135 [14:02:13] (03PS2) 10Edenhill: Correctly calculate escape buffer size [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/98134 [14:02:35] (03Abandoned) 10Edenhill: escaper: Comment magical constant numbers [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/98156 (owner: 10Edenhill) [14:05:11] (03CR) 10Akosiaris: [C: 032] Kill old sub-labs.wikimedia.org domains [operations/apache-config] - 10https://gerrit.wikimedia.org/r/98056 (owner: 10Faidon Liambotis) [14:07:40] (03PS1) 10Faidon Liambotis: Varnish: add German version of CentralAutoLogin [operations/puppet] - 10https://gerrit.wikimedia.org/r/98166 [14:07:41] (03PS1) 10Faidon Liambotis: Varnish: set hash_ignore_busy for CentralAutoLogin [operations/puppet] - 10https://gerrit.wikimedia.org/r/98167 [14:13:23] (03CR) 10Faidon Liambotis: [C: 032] Varnish: add German version of CentralAutoLogin [operations/puppet] - 10https://gerrit.wikimedia.org/r/98166 (owner: 10Faidon Liambotis) [14:13:47] (03CR) 10Faidon Liambotis: [C: 032] Varnish: set hash_ignore_busy for CentralAutoLogin [operations/puppet] - 10https://gerrit.wikimedia.org/r/98167 (owner: 10Faidon Liambotis) [14:16:58] (03CR) 10Akosiaris: [C: 032] Kill wlm.wikimedia.org [operations/apache-config] - 10https://gerrit.wikimedia.org/r/98057 (owner: 10Faidon Liambotis) [14:49:26] !log restarted gmond on ms-fe1001/2, both were stuck 6h ago and we lost all swift eqiad's metrics for that period [14:49:42] Logged the message, Master [14:55:26] (03PS1) 10Faidon Liambotis: Varnish: fix regexp for German CentralAutoLogin [operations/puppet] - 10https://gerrit.wikimedia.org/r/98169 [14:56:07] (03CR) 10Faidon Liambotis: [C: 032] Varnish: fix regexp for German CentralAutoLogin [operations/puppet] - 10https://gerrit.wikimedia.org/r/98169 (owner: 10Faidon Liambotis) [15:59:48] (03PS3) 10John F. Lewis: Enable AbuseFilter block option on Wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98002 [16:25:24] (03CR) 10Yuvipanda: "OMG TABS!!!1 4spaces plz :)" (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/96552 (owner: 10Addshore) [16:28:25] (03CR) 10Odder: [C: 031] Enable AbuseFilter block option on Wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98002 (owner: 10John F. Lewis) [16:51:38] (03PS7) 10Addshore: Start wikidata puppet module for builder [operations/puppet] - 10https://gerrit.wikimedia.org/r/96552 [16:51:53] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [17:06:38] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run for 7d 7h 48m 17s [17:08:38] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run for 0d 0h 2m 0s [17:10:38] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run for 0d 0h 4m 0s [17:12:38] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run for 0d 0h 6m 0s [17:14:38] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run for 0d 0h 8m 0s [17:16:38] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run for 0d 0h 10m 0s [17:18:38] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run for 0d 0h 12m 0s [17:20:38] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run for 0d 0h 14m 0s [17:22:38] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run for 0d 0h 16m 0s [17:24:38] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run for 0d 0h 18m 0s [17:26:38] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run for 0d 0h 20m 0s [17:28:38] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run for 0d 0h 22m 0s [17:29:08] RECOVERY - Puppet freshness on cp1004 is OK: puppet ran at Fri Nov 29 17:28:58 UTC 2013 [17:30:38] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run for 0d 0h 1m 30s [17:32:38] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run for 0d 0h 2m 0s [17:34:38] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run for 0d 0h 4m 0s [17:59:10] RECOVERY - Puppet freshness on cp1004 is OK: puppet ran at Fri Nov 29 17:59:05 UTC 2013 [19:26:17] !log "swapoff -a" on all cache_text to deal with strange kernel issue with kswapd dropping the whole page cache on memory pressure [19:26:32] Logged the message, Master [19:34:55] damn, that didn't do it [20:17:08] PROBLEM - Varnish HTTP text-frontend on cp1065 is CRITICAL: Connection refused [20:18:30] !log rebooting cp1065 with new kernel [20:18:45] Logged the message, Master [20:19:08] PROBLEM - Host cp1065 is DOWN: PING CRITICAL - Packet loss = 100% [20:20:28] RECOVERY - Host cp1065 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [20:20:28] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [20:21:08] RECOVERY - Varnish HTTP text-frontend on cp1065 is OK: HTTP OK: HTTP/1.1 200 OK - 198 bytes in 0.001 second response time [21:02:17] (03PS1) 10Ori.livneh: collector: allow listen port to be specified via COLLECTOR_PORT env var [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/98247 [21:02:34] (03CR) 10Ori.livneh: [C: 032 V: 032] collector: allow listen port to be specified via COLLECTOR_PORT env var [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/98247 (owner: 10Ori.livneh) [21:06:54] (03PS2) 10Ori.livneh: Add a MassMessage-related user group on Meta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97790 (owner: 10Odder) [21:07:46] (03CR) 10Ori.livneh: [C: 032] Add a MassMessage-related user group on Meta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/97790 (owner: 10Odder) [21:08:26] !log ori updated /a/common to {{Gerrit|Id9c7321b8}}: Add a MassMessage-related user group on Meta [21:08:42] Logged the message, Master [21:09:19] !log ori synchronized wmf-config/InitialiseSettings.php 'Id9c7321b8: Add a MassMessage-related user group on Meta' [21:09:35] Logged the message, Master [21:15:29] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [21:39:35] (03PS1) 10Ori.livneh: parametrize profiler-to-carbon [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/98248 [21:39:51] (03CR) 10Ori.livneh: [C: 032 V: 032] parametrize profiler-to-carbon [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/98248 (owner: 10Ori.livneh) [22:08:55] PROBLEM - Disk space on tungsten is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=96%): [22:09:55] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [22:10:35] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 35.38 ms [22:28:12] ugh [22:35:52] RECOVERY - Disk space on tungsten is OK: DISK OK [23:02:58] ori-l, http://en.m.wikipedia.org/w/api.php?action=zeroconfig gives 500 ??? [23:03:27] oh, i think that's our bug [23:03:28] bleh :( [23:08:34] heh [23:19:14] greg-g, welcome back, there shall be lots of depls early next week [23:19:40] greg-g, can we push something early? [23:19:58] there are tons of depls next week, including new m.wikipedia.org landing page [23:20:03] lots of coordinations with ops [23:20:11] so i want to get backend ready as early as possible [23:20:38] yurik: still have today off, send me an email with your proposal (taking [[wikitech:Deployments]] into account) [23:20:41] and btw, the above is not a bug, its a feature... Just need to change it at some point now that ops have been depls [23:20:45] :) [23:20:54] yeah yeah [23:21:01] everyday here is a day off [23:21:05] oki, will do [23:23:54] yurik: I had/have the entire week off [23:24:04] enjoy :)) [23:24:05] * greg-g goes back to ignoring work irc :) [23:24:09] thanks [23:45:28] yurik: you should rename the extension to 'Zero' [23:45:39] 'ZeroRatedMobileAccess' is awful [23:45:55] it's like Supercalifragilisticexpialidocious [23:53:18] addshoretravel: https://github.com/yuvipanda/ansible-config I'm trying out ansible :) [23:55:30] YuviPanda: *stalks the repo :P* [23:56:46] addshoretravel: :D just learning. Spent about 3 hours trying to get a small puppet repo setup for that VPS of mine, and gave up. At least I don't feel like smashing my face on an iron spike now [23:57:03] :D