[00:00:22] mwalker: mostly, if the nightly l10n update fails or not, it's been failing the last 3 days or so [00:00:44] alright, I'm out [00:01:11] kk, cya greg-g! [00:02:38] mwalker: From SAL, yes [00:03:32] I certainly appreciate that ori made it automatically log to IRC [00:06:39] mwalker: Ahm, so frwiktionary isn't working yet; could you try touch wmf-config/InitialiseSettings.php && sync-file wmf-config/InitialiseSettings.php ? [00:08:30] !log mwalker synchronized wmf-config/InitialiseSettings.php 'Attempting to poke frwikitionary into using VE' [00:08:36] Logged the message, Master [00:11:53] mwalker: Sweet! It's working now [00:11:55] Thanks man [00:11:59] np [00:12:07] mwalker: Thanks. :-) [00:12:13] this has been an interesting experience [00:12:56] mwalker: I owe you coffee/chocolate/etc. tomorrow. :-) [00:13:02] Sorry, I didn't realize that just touching dblist files isn't enough due to config cache [00:13:20] The config cache somewhat stupidly only pays attention to InitialiseSettings and CommonSettings [00:14:19] I didn't realize we had a configuration cache actually; it makes sense, but I'd never thought about it [00:32:18] (03PS2) 10PiRSquared17: Remove C: namespace alias (for categories) from hiwiki config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/113656 (owner: 10Gerrit Patch Uploader) [01:04:05] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [01:55:52] !log springle synchronized wmf-config/db-eqiad.php 's7 repool db1034' [01:56:01] Logged the message, Master [02:05:25] PROBLEM - Apache HTTP on mw1044 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:25] PROBLEM - Apache HTTP on mw1113 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:35] PROBLEM - Apache HTTP on mw1060 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:35] PROBLEM - Apache HTTP on mw1173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:35] PROBLEM - Apache HTTP on mw1043 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:35] PROBLEM - Apache HTTP on mw1072 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:35] PROBLEM - Apache HTTP on mw1218 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:36] PROBLEM - Apache HTTP on mw1174 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:36] PROBLEM - Apache HTTP on mw1070 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:37] PROBLEM - Apache HTTP on mw1211 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:37] PROBLEM - Apache HTTP on mw1183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:38] PROBLEM - Apache HTTP on mw1166 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:38] PROBLEM - Apache HTTP on mw1162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:42] !log springle synchronized wmf-config/db-eqiad.php 's7 adjust loads' [02:05:45] PROBLEM - Apache HTTP on mw1019 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:45] PROBLEM - Apache HTTP on mw1107 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:45] PROBLEM - Apache HTTP on mw1063 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:45] PROBLEM - Apache HTTP on mw1097 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:45] PROBLEM - Apache HTTP on mw1077 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:46] PROBLEM - Apache HTTP on mw1086 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:46] PROBLEM - Apache HTTP on mw1029 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:47] PROBLEM - Apache HTTP on mw1046 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:47] PROBLEM - Apache HTTP on mw1177 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:48] Logged the message, Master [02:05:55] PROBLEM - Apache HTTP on mw1079 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:05] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [02:06:12] !log springle synchronized wmf-config/db-eqiad.php 's7 adjust loads' [02:06:15] RECOVERY - Apache HTTP on mw1044 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.090 second response time [02:06:25] PROBLEM - Apache HTTP on mw1161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:25] RECOVERY - Apache HTTP on mw1060 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.077 second response time [02:06:25] RECOVERY - Apache HTTP on mw1072 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.080 second response time [02:06:25] RECOVERY - Apache HTTP on mw1043 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.079 second response time [02:06:25] PROBLEM - Apache HTTP on mw1219 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:26] PROBLEM - Apache HTTP on mw1178 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:26] PROBLEM - Apache HTTP on mw1164 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:27] PROBLEM - Apache HTTP on mw1180 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:27] RECOVERY - Apache HTTP on mw1070 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.059 second response time [02:06:28] PROBLEM - Apache HTTP on mw1054 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:28] PROBLEM - Apache HTTP on mw1179 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:29] PROBLEM - Apache HTTP on mw1109 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:29] RECOVERY - Apache HTTP on mw1174 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 3.921 second response time [02:06:35] RECOVERY - Apache HTTP on mw1211 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.777 second response time [02:06:35] RECOVERY - Apache HTTP on mw1019 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.070 second response time [02:06:35] PROBLEM - Apache HTTP on mw1104 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:35] RECOVERY - Apache HTTP on mw1077 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.071 second response time [02:06:35] RECOVERY - Apache HTTP on mw1029 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.078 second response time [02:06:36] RECOVERY - Apache HTTP on mw1097 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.095 second response time [02:06:36] PROBLEM - Apache HTTP on mw1215 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:37] RECOVERY - Apache HTTP on mw1162 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 6.559 second response time [02:06:37] RECOVERY - Apache HTTP on mw1046 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.058 second response time [02:06:38] PROBLEM - Apache HTTP on mw1209 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:38] RECOVERY - Apache HTTP on mw1107 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 6.257 second response time [02:06:39] RECOVERY - Apache HTTP on mw1063 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 6.319 second response time [02:06:39] PROBLEM - Apache HTTP on mw1169 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:40] RECOVERY - Apache HTTP on mw1177 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.740 second response time [02:06:40] RECOVERY - Apache HTTP on mw1166 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 9.323 second response time [02:06:41] PROBLEM - Apache HTTP on mw1188 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:41] PROBLEM - Apache HTTP on mw1170 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:45] RECOVERY - Apache HTTP on mw1086 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 8.894 second response time [02:06:45] RECOVERY - Apache HTTP on mw1079 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.068 second response time [02:06:54] That is a lot of apache alerts, guessing someone is actively doing maintain? [02:07:15] RECOVERY - Apache HTTP on mw1161 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.060 second response time [02:07:15] RECOVERY - Apache HTTP on mw1178 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.050 second response time [02:07:15] RECOVERY - Apache HTTP on mw1164 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.057 second response time [02:07:15] RECOVERY - Apache HTTP on mw1180 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.059 second response time [02:07:15] RECOVERY - Apache HTTP on mw1219 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.059 second response time [02:07:16] RECOVERY - Apache HTTP on mw1113 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.048 second response time [02:07:16] RECOVERY - Apache HTTP on mw1179 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.050 second response time [02:07:17] RECOVERY - Apache HTTP on mw1109 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.071 second response time [02:07:17] RECOVERY - Apache HTTP on mw1054 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.073 second response time [02:07:25] RECOVERY - Apache HTTP on mw1104 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.096 second response time [02:07:25] RECOVERY - Apache HTTP on mw1173 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.060 second response time [02:07:25] RECOVERY - Apache HTTP on mw1215 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.069 second response time [02:07:25] RECOVERY - Apache HTTP on mw1218 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.048 second response time [02:07:25] RECOVERY - Apache HTTP on mw1209 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.059 second response time [02:07:26] RECOVERY - Apache HTTP on mw1169 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.056 second response time [02:07:26] RECOVERY - Apache HTTP on mw1183 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.054 second response time [02:07:27] RECOVERY - Apache HTTP on mw1188 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.056 second response time [02:07:27] RECOVERY - Apache HTTP on mw1170 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.059 second response time [02:19:40] !log LocalisationUpdate completed (1.23wmf19) at 2014-03-28 02:19:40+00:00 [02:19:48] Logged the message, Master [02:33:57] !log springle synchronized wmf-config/db-eqiad.php 's7 increase db1007 temporarily' [02:34:03] Logged the message, Master [02:35:23] wth was that [02:35:38] greg-g: http://ganglia.wikimedia.org/latest/?c=Application%20servers%20eqiad&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2 [02:35:50] https://gdash.wikimedia.org/dashboards/reqerror/ ??? [02:36:15] 02:05 logmsgbot: springle synchronized wmf-config/db-eqiad.php 's7 adjust loads' is it okay now? [02:38:29] huh: not really. that is, the site is ok, but something happened on db1028 that i didn't expect, triggered by my repooling another box [02:38:36] still digging [02:41:41] springle: https://logstash.wikimedia.org/#dashboard/temp/VF35KQ5XSDqCZGV86oZaGg [02:42:13] Does that need WMF password? [02:46:14] huh: yeah, logstash can contain non-public data [02:51:49] !log LocalisationUpdate completed (1.23wmf20) at 2014-03-28 02:51:49+00:00 [02:51:55] Logged the message, Master [02:53:49] !log BetaFeatures is missing message "betafeatures-toplink" on every page on mediawiki.org [02:53:55] Logged the message, Master [02:57:29] ^ likely means the recent change to LocalisationUpdate and/or the switch to JSON for extension messages broke stuff [02:57:32] and is going to break more stuff [02:57:32] greg-g: [02:59:45] Isn't front-end caching either. eval.php --wiki mediawikiwiki [02:59:46] > return wfMessage('betafeatures-toplink')->text(); [02:59:46] <betafeatures-toplink> [03:00:12] RoanKattouw_away: Reedy: Nikerabbit: mwalker|away: [03:02:43] Krinkle: can you fix it? [03:03:01] I could drop everything and try and find out, but I'm not familiar with this stuff. [03:03:12] Would be more efficient for someone else to do it. I just noticed it. [03:03:14] also <betafeatures-section-desc> <betafeatures-about-betafeatures> | <betafeatures-discuss-betafeatures> [03:03:27] Yeah, the whole thing. Anything that uses JSON instead of .i18n.php [03:03:31] wow [03:03:34] O_o [03:03:36] other extensions as well [03:04:30] Did you report it on Bugzilla, or are you hoping someone will fix this very soon? [03:04:52] (03PS1) 10Springle: Disable slow_digest job on coredb s[1-7] [operations/puppet] - 10https://gerrit.wikimedia.org/r/121569 [03:06:52] !log All interface messages for of any extension in wmf20 that uses json is missing (mw-ui-feature-user-count, betafeatures-toplink, wikimediamessages-desc, )) [03:06:58] Logged the message, Master [03:07:00] RoanKattouw: https://www.mediawiki.org/wiki/Special:Version look for -desc> [03:07:05] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [03:08:03] Only BetaFeatures and WikimediaMessages [03:08:05] (03CR) 10Springle: [C: 032] Disable slow_digest job on coredb s[1-7] [operations/puppet] - 10https://gerrit.wikimedia.org/r/121569 (owner: 10Springle) [03:08:07] !log Rerunning LocalisationUpdate in an attempt to figure out what's going on [03:08:13] Logged the message, Mr. Obvious [03:08:32] Looking for /a/common/php-1.23wmf20/extensions/*/i18n/en.json on tin.wmnet gives me many more extensions than just those two [03:08:38] so there must be specific about those two [03:08:57] * Jasper_Deng_busy remembers a time when it apparently completed in like 10 minutes and Reedy (or someone) didn't believe it [03:09:50] Warning: LU_Updater::readMessages: Unable to parse messages from /var/lib/l10nupdate/mediawiki/extensions/GoogleNewsSitemap/GoogleNewsSitemap.i18n.php in /a/common/php-1.23wmf19/extensions/LocalisationUpdate/Updater.php on line 63 [03:09:54] Lots and lots of those [03:09:58] They're probably JSON extensions though [03:11:02] Hmm, /var/lib/l10nupdate/cache-1.23wmf20/*.json are like 2x smaller than their wmf19 counterparts [03:11:06] Not in wmf19 [03:11:07] https://github.com/wikimedia/mediawiki-extensions-GoogleNewsSitemap/tree/wmf/1.23wmf19 [03:11:16] Oooh [03:11:17] hah [03:11:24] But did it move to JSON in wmf20? [03:11:26] Cause LU uses master [03:12:00] RoanKattouw: https://gist.github.com/Krinkle/34317862e57e7606cab6 [03:12:04] I think most extensions will have [03:12:29] https://github.com/wikimedia/mediawiki-extensions-GoogleNewsSitemap/tree/wmf/1.23wmf20 [03:12:35] (siebrand was planning on) [03:13:25] RoanKattouw: GoogleNewsSitemap doesn't appear broken though on test/test2/mediawiki [03:13:34] It's only BetaFeatures and WikimediaMessages [03:13:36] Hmm [03:13:54] I'll investigate those then [03:19:47] !log LocalisationUpdate completed (1.23wmf19) at 2014-03-28 03:19:47+00:00 [03:19:54] Logged the message, Master [03:27:44] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Mar 28 03:27:41 UTC 2014 (duration 27m 40s) [03:27:50] Logged the message, Master [03:33:49] !log LocalisationUpdate completed (1.23wmf20) at 2014-03-28 03:33:49+00:00 [03:33:54] Logged the message, Master [03:34:05] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [03:34:16] RoanKattouw and I are debugging it on tin [03:34:26] looks like something is messing with $wgMessagesDirs in the global scope [03:34:34] dumping it on eval.php shows BetaFeatures isn't even set in there [03:34:44] Eventhough BetaFeatures.php is most certainly setting it [03:34:58] (03PS1) 10Ryan Lane: Fix repos with checked-in .gitmodules [operations/puppet] - 10https://gerrit.wikimedia.org/r/121574 [03:35:03] What BetaFeatures and WikimediaMessages have in common is that they are the ones that do not have a pre-1.23 fallback i18n.php file [03:35:35] So looks like something is causing proper i18n parsing to not work, and we're actually relying polyfil it seems [03:35:50] Yeah most extensions are falling back to the polyfill [03:37:39] https://gist.github.com/Krinkle/34317862e57e7606cab6 [03:38:00] RoanKattouw: [03:38:04] i18n parsing works fine, it's just that $wgMessagesDirs is almost empty [03:38:18] Many extensions aren't in there, for some mysterious reason [03:41:19] Seems to be a bug in mergeMessagesFileList.php [03:41:25] Somehow not merging $wgMessagesDirs correctly [03:42:05] (03PS1) 10Springle: Disable slow query logging while coredb::slow_digest is disabled for s[1-7], so the log file does not grow. Has also been SET GLOBAL. [operations/puppet] - 10https://gerrit.wikimedia.org/r/121576 [03:45:54] !log ExtensionMessages-1.23wmf20.php is out of date, regenerated [03:46:01] Logged the message, Mr. Obvious [03:48:45] !log So scap/mw-update-l10n (at branch cut) or LocalisationUpdate must've run mergeMessagesFileList.php wrongly somehow [03:48:51] Logged the message, Master [03:48:58] !log Maybe the new scap scripts regenerate ExtensionMessages-$version.php at the wrong time (i.e. do things in the wrong order)? [03:49:03] !log catrope synchronized wmf-config/ExtensionMessages-1.23wmf20.php 'Regenerated to add new JSON dirs that were missing' [03:49:04] Logged the message, Mr. Obvious [03:49:10] Logged the message, Master [03:54:25] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Mar 28 03:54:22 UTC 2014 (duration 46m 33s) [03:54:32] Logged the message, Master [03:55:05] !log Running mw-update-l10n. Skipped wmf19, is now rebuilding l10ncache for wmf20 because I updated ExtensionMessages-1.23wmf20.php [03:55:11] Logged the message, Mr. Obvious [03:56:07] !log Rebuilding i18n using mw-update-l10n, syncing ExtensionMessages-1.23wmf20.php doesn't actually regenerate the cache [03:56:12] Logged the message, Master [03:56:41] Hello. Late to the party; let me know if I can help. [04:01:35] !log Running sync-l10nupdate to sync l10n cache [04:01:42] Logged the message, Mr. Obvious [04:01:43] ori: I need to drive a bunch of people home right now [04:01:51] But I'm running sync-l10nupdate in a screen on tin [04:02:02] Once that's done everything *should* be fine [04:02:05] RoanKattouw: And mw-update-l10n in the other screen? [04:02:10] No that finished [04:02:19] OK [04:02:27] Symptoms: Ctrl+F betafeatures on Special:Version, observe broken messages [04:03:15] that is find "-desc>" [04:03:16] kk, I'll keep an eye out. [04:03:22] on Special:Version [04:03:29] or betafeatures-toplink on any page [04:03:33] if it doesn't work, what would you have tried next? [04:03:34] in group0 wikis [04:03:41] (he just detached) [04:03:45] (in rl) [04:03:58] ori: Not sure, probably verify it works via eval.php [04:04:10] * ori nods. [04:04:13] return wfMessage('betafeatures-toplink')->text() [04:04:20] should not be [04:08:59] roan's sync-l10nupdate is pid 5279 on tin [04:09:07] for those following along [04:16:58] https://www.mediawiki.org/wiki/Special:Preferences#mw-prefsection-betafeatures looks right now [04:23:11] almost there [04:25:57] !log sync-l10nupdate ran to completion. messages that were previously missing now render. [04:26:04] Logged the message, Master [04:29:09] was the apache freakout earlier related to the missing messages? doesn't seem like it would be [04:30:53] i'm assuming that piece is the db1028 issue springle is looking at. (springle, i think it's worth a !log even if you haven't yet traced it exactly.) [04:31:05] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [04:31:50] anyways, looks ok, i'm going to walk away and have some dinner. [04:39:03] ori: true. edited my earlier SAL entry [06:09:12] ori: morning [06:23:15] hi Nikerabbit [06:23:22] i'm not really here, calling it an early night [06:23:36] but hello and good morning to you nonetheless :) [06:47:15] Reedy: ping [06:47:20] when you awake :) [07:13:40] (03CR) 10Nikerabbit: applicationserver::hhvm: add /var/run/hhvm (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/121547 (owner: 10Ori.livneh) [07:25:56] (03CR) 10Nikerabbit: hhvm on beta: use upstart (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/121451 (owner: 10Ori.livneh) [08:54:32] (03PS1) 10Hashar: contint: migrate slaves to use role pbuilder class [operations/puppet] - 10https://gerrit.wikimedia.org/r/121602 [08:56:26] hey apergos :-] [08:56:32] hello [08:56:43] * hashar apergos: could use some merge to finish the migration of some instances in labs please :D https://gerrit.wikimedia.org/r/#/c/121602/ [08:56:47] grr [08:56:58] not sure why my client send my sentences as notices from time to time [08:57:09] change just switch from some misc:: to a role:: class :] [08:58:15] also python linted one of your snapshot scripts modules/snapshot/files/dumps/list-last-n-good-dumps.py https://gerrit.wikimedia.org/r/#/c/120499/ :d [09:06:09] I hav been rying to pep8/pylint som of those scripts as I puppetize them or update them, but still missing some things, thanks for that [09:07:35] (03CR) 10ArielGlenn: [C: 032] contint: migrate slaves to use role pbuilder class [operations/puppet] - 10https://gerrit.wikimedia.org/r/121602 (owner: 10Hashar) [09:09:00] (03PS3) 10ArielGlenn: snapshot: lint list-last-n-good-dumps.py [operations/puppet] - 10https://gerrit.wikimedia.org/r/120499 (owner: 10Hashar) [09:10:42] (03CR) 10ArielGlenn: [C: 032] snapshot: lint list-last-n-good-dumps.py [operations/puppet] - 10https://gerrit.wikimedia.org/r/120499 (owner: 10Hashar) [09:20:41] apergos: I am randomly running pep8/pylint from time to time :D [09:20:53] apergos: there is also flake8 which is a wrapper around pep8 and pylint [09:20:59] only one command to handle everything [09:25:20] (03PS1) 10Hashar: pbuilder root on labs need to be a directory [operations/puppet] - 10https://gerrit.wikimedia.org/r/121605 [09:26:18] apergos: and a follow up which is that file { ensure => present } creates a file when I really needed a directory. sorry https://gerrit.wikimedia.org/r/121605 [09:26:40] indeed it does [09:26:59] and I broke the class meanwhile, damn puppet [09:27:05] package-builder.pp is a crazy class [09:27:12] (03CR) 10ArielGlenn: [C: 032] pbuilder root on labs need to be a directory [operations/puppet] - 10https://gerrit.wikimedia.org/r/121605 (owner: 10Hashar) [09:27:20] springle: ohai [09:27:32] ohhh a reedy [09:27:42] id you see my pings in wm dev? [09:28:09] I have now [09:28:22] I thought AdminSettings remnants had been removed from core... [09:28:33] yes from core but that is for the release [09:28:44] for wmf use we have the wikiadmin user for long running jobs [09:28:47] Does the file even exist anymore? [09:28:54] if someone wanted to remove that, we'd have had a chat about it [09:29:01] yes it exists all right [09:29:34] indeed [09:29:46] have a look on tin in /apache/common/wmf-config [09:30:54] !log reedy synchronized wmf-config/PrivateSettings.php 'Uncomment AdminSettings.php' [09:31:00] Logged the message, Master [09:31:05] Could not open input file: /a/common/PrivateSettings.php [09:31:05] Aborted due to syntax errors [09:31:06] Amusing [09:31:12] oh joy [09:31:16] so I was about to say [09:31:29] I don't know that this commenting out was the change made on the 21st [09:31:34] since no repo, no ~ file [09:32:01] but it seems very likely, it had to be something picked up by the scap, applicable to all mw versions, not viible in my tree n laptop [09:33:02] Sounds likely [09:33:57] (03CR) 10Faidon Liambotis: [C: 032] Remove language subdomains for wikidata.org [operations/dns] - 10https://gerrit.wikimedia.org/r/119032 (owner: 10Faidon Liambotis) [09:35:11] it would be very nice if: that file were in a repo somplace, and [09:35:44] we had a conversation about how folks want to handle wikiadminuser (and anything else in adminsettings) going forward [09:36:31] email (to whom)? or another venue? [09:38:59] springle: I see queries onn db1042 as wikiadmin now so it seems pt-kill could be re-enabled [09:39:04] Reedy: thanks for that [09:43:39] ignore the page [09:43:46] !log removing language subdomains for wikidata [09:43:52] Logged the message, Master [09:43:52] yep [09:43:53] the page is broken, I'll fix it [09:45:34] I hate watchmouse [09:49:24] matanya: around by any chance ? :D [10:06:36] paravoid: would you have sometime to debug a puppet scope issue in the infamous package-builder class please? :( [10:06:49] the file is https://github.com/wikimedia/operations-puppet/blob/production/manifests/misc/package-builder.pp [10:07:13] and the $pbuilder_root is not properly passed down the defines in the class :/ [10:13:00] Reedy: apergos: thanks for sorting out wikiuser/wikiadmin [10:13:12] hey everyone [10:13:20] icinga -> all unhandled problems [10:13:54] ah now there's a PDU warning [10:13:57] it will go away [10:14:08] gone [10:14:44] very nnice [10:15:24] yeah, I've been hunting them down one by one [10:15:46] now we need to shorten the list of handled problems :) [10:16:58] paravoid: apergos found my scoping issue :-] [10:17:32] (03PS1) 10Hashar: package-builder.pp: pass $pbuilder_root to image{} [operations/puppet] - 10https://gerrit.wikimedia.org/r/121622 [10:17:50] apergos: passing the values to the image{} calls would most probably fix the issue I have encountered https://gerrit.wikimedia.org/r/121622 [10:17:50] :D [10:18:05] orilly :-D [10:18:08] I have added the pbuilder_root parameter and forgot to update all the calls to pass the variable down to the image define [10:18:09] damn [10:18:50] I have updated the examples as well [10:21:28] (03CR) 10ArielGlenn: [C: 032] package-builder.pp: pass $pbuilder_root to image{} [operations/puppet] - 10https://gerrit.wikimedia.org/r/121622 (owner: 10Hashar) [10:21:47] * hashar updates his list of owed beers [10:22:09] I am probably going to be broke after the next face to face meetup [10:22:12] that's not a beer, that's one swallow of a beer [10:22:34] I only order pints. [10:22:38] heh [10:22:48] I got to maintain some level of classiness [10:22:49] also it's cancelled out by the instance move fiasco [10:23:32] which fiasco? [10:23:52] the media server instance in deployment prep [10:23:58] ahh [10:24:01] it went well :D [10:24:05] see, yer lucky [10:24:07] :-D [10:24:13] just had to copy paste the instance to eqiad [10:24:16] and it seems to be working [10:24:41] sh -c pbuilder --create --distribution lucid --basetgz /mnt/pbuilder/lucid.tgz [10:24:43] \O/ [10:24:46] proper path [10:24:57] I will be able to phase out yet another instance from pmtpa [10:25:04] yay [10:27:59] debootstrap failed [10:27:59] removing directory /var/cache/pbuilder/build//17560 and its subdir [10:28:01] bah [10:28:02] it never ends [10:29:32] gotta pass --buildplace [location of build] [10:35:40] and aptcache [10:35:41] grr [10:35:49] (03PS1) 10Hashar: package-builder.pp: set buildplace/aptcache [operations/puppet] - 10https://gerrit.wikimedia.org/r/121624 [10:35:59] apergos: and got another follow up https://gerrit.wikimedia.org/r/121624 [10:36:09] pbuilder has a bunch of parameters [10:43:28] (03CR) 10ArielGlenn: [C: 032] package-builder.pp: set buildplace/aptcache [operations/puppet] - 10https://gerrit.wikimedia.org/r/121624 (owner: 10Hashar) [10:43:59] hashar: now i am [10:44:14] matanya: apergos fixed up the issue 8) [10:52:16] :-/ [10:52:17] (03PS1) 10Hashar: package-builder: duplicate $build_place [operations/puppet] - 10https://gerrit.wikimedia.org/r/121631 [10:52:46] I give up will follow up next week it is taking too long :D [10:55:59] !log Jenkins: depooling ntegration-debian-builder pmtpa slave. Jobs should be build on the eqiad slaves now. debian-glue jobs might be broken for a while until I figure out how to get cow builder setup properly. [10:56:06] Logged the message, Master [11:00:00] hashar: i see now, sorry i caused you trouble [11:00:22] matanya: nothing to worry. It i just that I have not been careful enough :/ [11:01:43] paravoid: hashar wanted to backport puppet-lint from trusty to presice, it has the same dep's and I tested it a bit on labs. what is the process to get it to apt.wikimedia.org ? [11:16:39] (03PS1) 10Hashar: contint: include browsertest role on labs slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/121645 [11:54:26] (03Abandoned) 10Thiemo Mättig (WMDE): Redirect all *.wikidata.org subdomains to www.wikidata.org [operations/apache-config] - 10https://gerrit.wikimedia.org/r/113972 (owner: 10Thiemo Mättig (WMDE)) [12:11:31] (03PS2) 10Dzahn: Add config file for bing webmaster tools. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121297 (owner: 10Jalexander) [12:11:40] (03CR) 10Dzahn: [C: 031] Add config file for bing webmaster tools. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121297 (owner: 10Jalexander) [12:26:08] !log Jenkins: migrating both labs slaves puppet master to integration-puppetmaster.eqiad.wmflabs [12:26:15] Logged the message, Master [12:31:40] (03PS2) 10Hashar: contint: include browsertest role on labs slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/121645 [12:33:29] (03PS1) 10Dzahn: decom db40, remove from dsh,dhcp [operations/puppet] - 10https://gerrit.wikimedia.org/r/121655 [12:34:44] (03PS3) 10Hashar: contint: include browsertest role on labs slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/121645 [12:34:59] (03CR) 10Dzahn: [C: 04-1] "no, the glance.pp stuff doesn't belong here" [operations/puppet] - 10https://gerrit.wikimedia.org/r/121655 (owner: 10Dzahn) [12:36:47] (03PS2) 10Dzahn: decom db40, remove from dsh,dhcp [operations/puppet] - 10https://gerrit.wikimedia.org/r/121655 [12:38:30] (03PS1) 10Dzahn: decom db62, remove from dhcp [operations/puppet] - 10https://gerrit.wikimedia.org/r/121656 [12:42:58] (03PS4) 10Hashar: contint: include browsertest role on labs slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/121645 [12:45:46] (03PS5) 10Hashar: contint: include browsertest role on labs slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/121645 [12:47:51] (03PS6) 10Hashar: contint: include browsertest role on labs slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/121645 [12:50:16] (03PS7) 10Hashar: contint: include browsertest role on labs slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/121645 [12:54:41] (03PS1) 10Hashar: elasticsearch: prevent dupe packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/121659 [12:55:49] (03CR) 10Dzahn: [C: 031] Fix repos with checked-in .gitmodules [operations/puppet] - 10https://gerrit.wikimedia.org/r/121574 (owner: 10Ryan Lane) [12:57:31] (03PS2) 10Hashar: elasticsearch: prevent dupe packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/121659 [12:57:33] (03PS8) 10Hashar: contint: include browsertest role on labs slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/121645 [12:57:59] (03CR) 10Aklapper: [C: 04-1] "I'll give -1 here, not for technical reasons but because I'd like to see more social agreement first. See https://bugzilla.wikimedia.org/s" [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/106761 (owner: 1001tonythomas) [13:05:14] (03CR) 10Dzahn: "bump, are we still going to do this? linked ticket says needs DBA feedback" [operations/puppet] - 10https://gerrit.wikimedia.org/r/111152 (owner: 10Diederik) [13:28:16] (03PS3) 10Dzahn: add missing system roles [operations/puppet] - 10https://gerrit.wikimedia.org/r/120956 [13:30:07] (03CR) 10Manybubbles: [C: 031] "Fine by me but I'd run it by ottomata because he is my puppet oracle." [operations/puppet] - 10https://gerrit.wikimedia.org/r/121659 (owner: 10Hashar) [13:31:26] (03CR) 10Ottomata: [C: 031] elasticsearch: prevent dupe packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/121659 (owner: 10Hashar) [13:32:49] (03PS4) 10Dzahn: add missing system roles [operations/puppet] - 10https://gerrit.wikimedia.org/r/120956 [13:33:29] (03PS3) 10Hashar: elasticsearch: prevent dupe packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/121659 [13:33:31] (03PS2) 10Hashar: package-builder: duplicate $build_place [operations/puppet] - 10https://gerrit.wikimedia.org/r/121631 [13:33:33] (03PS9) 10Hashar: contint: include browsertest role on labs slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/121645 [13:33:35] (03PS1) 10Hashar: package-builder.pp: create aptcache dir [operations/puppet] - 10https://gerrit.wikimedia.org/r/121663 [13:34:47] (03PS5) 10Dzahn: add missing system roles [operations/puppet] - 10https://gerrit.wikimedia.org/r/120956 [13:35:50] (03PS1) 10Dzahn: retab role/glance [operations/puppet] - 10https://gerrit.wikimedia.org/r/121667 [13:40:52] (03CR) 10Dzahn: [C: 032] add missing system roles [operations/puppet] - 10https://gerrit.wikimedia.org/r/120956 (owner: 10Dzahn) [13:42:37] (03CR) 10Dzahn: "matanya: done" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120956 (owner: 10Dzahn) [13:46:10] apergos: hi.. what is the java stuff on db62? [13:46:29] huh? [13:46:37] let's shut down? no mysql there (as opposed to db40 where ticket says no mysql, but in fact it runs) [13:46:44] oh you are on the decom ticket [13:46:50] /usr/lib/jvm/jdk1.7.0/bin/java -Dproc_namenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx [13:46:54] lemme look [13:46:56] < stuff like that on db62 [13:46:58] yes, i am [13:50:11] looks like old hadoop from feb 2013 [13:50:21] ah [13:51:47] root@db62:/var/log/hadoop/root# grep 'Number of transactions' *namenode* | grep -v 'Number of transactions: 0' [13:51:47] hadoop-root-namenode-db62.log.2013-02-05:2013-02-05 17:11:06,885 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 96 Total time for transactions(ms): 7Number of transactions batched in Syncs: 0 Number of syncs: 46 SyncTimes(ms): 10 [13:51:48] hadoop-root-namenode-db62.log.2013-02-05:2013-02-05 17:14:44,961 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 215 Total time for transactions(ms): 15Number of transactions batched in Syncs: 2 Number of syncs: 122 SyncTimes(ms): 31 [13:51:58] think that can be shot :-D [13:53:10] thanks, lemme get it out of icinga [13:53:41] must be stored resources though, not in site [13:54:18] Killing db62.pmtpa.wmnet...done. [13:54:54] CUSTOM - Host db62 is UP: PING OK - Packet loss = 0%, RTA = 35.44 ms [13:55:33] as far as db40, the mysqld over there hasn't logged anything since July 2012, no updates to any tables or the ib_logfiles so [13:55:35] it too can go [13:56:40] !log db62 - revoke puppet cert,salt key,remove from monitoring [13:56:46] Logged the message, Master [13:57:05] !log db40 - stopping mysql [13:57:14] Logged the message, Master [13:58:05] apergos: great, thanks [13:58:21] sure! glad to see em going going goooone [13:59:05] !log db40 - revoke puppet cert,salt key,remove from monitoring [13:59:10] Logged the message, Master [14:00:02] (03PS4) 10Hashar: elasticsearch: prevent dupe packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/121659 [14:00:04] (03PS3) 10Hashar: package-builder: duplicate $build_place [operations/puppet] - 10https://gerrit.wikimedia.org/r/121631 [14:00:06] (03PS10) 10Hashar: contint: include browsertest role on labs slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/121645 [14:00:08] (03PS2) 10Hashar: package-builder.pp: create aptcache dir [operations/puppet] - 10https://gerrit.wikimedia.org/r/121663 [14:00:36] (03CR) 10Dzahn: [C: 032] decom db40, remove from dsh,dhcp [operations/puppet] - 10https://gerrit.wikimedia.org/r/121655 (owner: 10Dzahn) [14:01:23] (03PS2) 10Dzahn: decom db62, remove from dhcp [operations/puppet] - 10https://gerrit.wikimedia.org/r/121656 [14:02:24] (03CR) 10Dzahn: [C: 032] decom db62, remove from dhcp [operations/puppet] - 10https://gerrit.wikimedia.org/r/121656 (owner: 10Dzahn) [14:02:43] (03CR) 10Dzahn: [V: 032] decom db62, remove from dhcp [operations/puppet] - 10https://gerrit.wikimedia.org/r/121656 (owner: 10Dzahn) [14:07:30] (03PS5) 10Hashar: elasticsearch: prevent dupe packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/121659 [14:07:32] (03PS4) 10Hashar: package-builder.pp: duplicate $build_place [operations/puppet] - 10https://gerrit.wikimedia.org/r/121631 [14:07:34] (03PS11) 10Hashar: contint: include browsertest role on labs slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/121645 [14:07:36] (03PS1) 10Hashar: package-builder.pp: --create must be 1st arg to pbuilder [operations/puppet] - 10https://gerrit.wikimedia.org/r/121671 [14:11:37] !log shut down db62 [14:11:42] Logged the message, Master [14:13:09] (03CR) 10Hashar: [C: 031 V: 032] "Works on labs once though it needs follow up change https://gerrit.wikimedia.org/r/#/c/121671/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/121663 (owner: 10Hashar) [14:13:27] (03CR) 10Hashar: [C: 031 V: 032] "Fix part of the pbuilder issue on labs." [operations/puppet] - 10https://gerrit.wikimedia.org/r/121671 (owner: 10Hashar) [14:13:49] (03CR) 10Hashar: [C: 031 V: 032] "That is a bit messy for now but managed to get the browsertests packages on the labs instance." [operations/puppet] - 10https://gerrit.wikimedia.org/r/121645 (owner: 10Hashar) [14:14:27] (03PS1) 10Dzahn: remove db62, decom [operations/dns] - 10https://gerrit.wikimedia.org/r/121672 [14:14:50] (03CR) 10Hashar: [C: 031 V: 032] "That is a bit ugly but I need that on the Jenkins slaves in labs which install openjdk and also include elastic search." [operations/puppet] - 10https://gerrit.wikimedia.org/r/121659 (owner: 10Hashar) [14:20:51] (03CR) 10Ottomata: [C: 031] "Totally fine. This is pretty normal for common packages." [operations/puppet] - 10https://gerrit.wikimedia.org/r/121659 (owner: 10Hashar) [14:23:14] (03CR) 10Hashar: [C: 031 V: 032] "Fix the duplicate issue in labs." [operations/puppet] - 10https://gerrit.wikimedia.org/r/121631 (owner: 10Hashar) [14:34:41] PROBLEM - Host db62 is DOWN: PING CRITICAL - Packet loss = 100% [14:42:05] (03PS2) 10Springle: Disable slow query logging while coredb::slow_digest is disabled for s[1-7], so the log file does not grow. Has also been SET GLOBAL. [operations/puppet] - 10https://gerrit.wikimedia.org/r/121576 [14:44:20] (03CR) 10Springle: [C: 032] Disable slow query logging while coredb::slow_digest is disabled for s[1-7], so the log file does not grow. Has also been SET GLOBAL. [operations/puppet] - 10https://gerrit.wikimedia.org/r/121576 (owner: 10Springle) [14:44:45] (03PS1) 10Hashar: contint: remove hhvm from slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/121675 [14:45:25] (03PS1) 10Springle: MariaDB monitoring for disk and process(es) [operations/puppet] - 10https://gerrit.wikimedia.org/r/121676 [14:46:37] (03CR) 10Hashar: "I have to disable hhvm because it depends on libmemcached6 whereas wikimedia-task-appserver (which i really need) depends on libmemcached1" [operations/puppet] - 10https://gerrit.wikimedia.org/r/121675 (owner: 10Hashar) [14:49:36] (03CR) 10Springle: [C: 04-2] MariaDB monitoring for disk and process(es) [operations/puppet] - 10https://gerrit.wikimedia.org/r/121676 (owner: 10Springle) [15:04:35] (03PS1) 10Hashar: Puppet masters in labs needs puppet::self::geoip [operations/puppet] - 10https://gerrit.wikimedia.org/r/121677 [15:05:15] (03PS2) 10Hashar: Puppet masters in labs needs puppet::self::geoip [operations/puppet] - 10https://gerrit.wikimedia.org/r/121677 [15:06:21] (03CR) 10Hashar: "Try!ing to import it on role::puppet::self for labs instance: https://gerrit.wikimedia.org/r/#/c/121677/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119555 (owner: 10BryanDavis) [15:09:01] (03CR) 10Hashar: "Ottomata, Bryan, I got some puppet failures on instances having solely role::puppet::self applied on them :(" [operations/puppet] - 10https://gerrit.wikimedia.org/r/121677 (owner: 10Hashar) [15:10:35] (03CR) 10Ottomata: Puppet masters in labs needs puppet::self::geoip (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/121677 (owner: 10Hashar) [15:12:01] (03PS2) 10Hashar: Fix repos with checked-in .gitmodules [operations/puppet] - 10https://gerrit.wikimedia.org/r/121574 (owner: 10Ryan Lane) [15:12:29] (03CR) 10Hashar: "Added pointer to bug 63029" [operations/puppet] - 10https://gerrit.wikimedia.org/r/121574 (owner: 10Ryan Lane) [15:14:29] * bd808 reads a lot of l10n horror in backscroll [15:16:43] (03CR) 10Hashar: "Additionally /var/lib/puppet/volatile directory does not exist :" [operations/puppet] - 10https://gerrit.wikimedia.org/r/121677 (owner: 10Hashar) [15:16:49] For what it's worth, mw-update-l10n on a new branch has had problems since long before I started messing with things, but I had hoped it was finally fixed -- https://bugzilla.wikimedia.org/show_bug.cgi?id=51174 [15:25:50] bd808: yeah, fun night [15:30:58] There's a bad chicken and egg problem with building the l10n cache for a new branch. The code to create the list of all extensions needs to have the list of all extensions already present due to a require from the top level config file. [15:31:28] Once that is worked around the builder needs an english l10n file to make the list of extenstions [15:32:19] So I generate an en l10n file *before* we have the list of extensions and then --force them to rebuild later [15:32:24] bd808: is extensions-list changing with the json stuff? [15:32:47] aude: That I don't know [15:32:55] ok, phew [15:34:13] wmf-config/ExtensionMessages-1.XwmfY.php is the list file I'm blathering about [15:35:18] (03PS6) 10Hashar: elasticsearch: prevent dupe packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/121659 [15:35:30] Which I guess is the "list of all places that l10n files might be found" [15:36:11] (03CR) 10Hashar: "Cherry picked to tip of the production branch." [operations/puppet] - 10https://gerrit.wikimedia.org/r/121659 (owner: 10Hashar) [15:37:02] (03CR) 10Andrew Bogott: [C: 032] elasticsearch: prevent dupe packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/121659 (owner: 10Hashar) [15:37:21] (03PS12) 10Hashar: contint: include browsertest role on labs slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/121645 [15:39:43] (03CR) 10Andrew Bogott: [C: 032] contint: include browsertest role on labs slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/121645 (owner: 10Hashar) [15:40:28] I know I looked for obvious things like the betafeatures l10n on testwiki and mw.o when I pushed out 1.23wmf20 yesterday. I'll take a look at what the LocalisationUpdate cron job is running to see if there are obvious differences to the process that runs during a full scap. [15:43:37] !log Jenkins removing pbuilder images from gallium.wikimedia.org and lanthanum.eqiad.wmnet production slaves. Packages should now be build on the labs slaves. [15:48:05] (03PS5) 10Hashar: package-builder.pp: duplicate $build_place [operations/puppet] - 10https://gerrit.wikimedia.org/r/121631 [15:48:07] (03PS3) 10Hashar: package-builder.pp: create aptcache dir [operations/puppet] - 10https://gerrit.wikimedia.org/r/121663 [15:48:09] (03PS2) 10Hashar: package-builder.pp: --create must be 1st arg to pbuilder [operations/puppet] - 10https://gerrit.wikimedia.org/r/121671 [15:48:11] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [15:50:05] (03PS2) 10Hashar: contint: remove hhvm from slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/121675 [15:50:21] (03CR) 10Hashar: "Cherry picked to tip of production. Already deployed on the integration project." [operations/puppet] - 10https://gerrit.wikimedia.org/r/121675 (owner: 10Hashar) [15:50:43] (03CR) 10Andrew Bogott: [C: 032] package-builder.pp: create aptcache dir [operations/puppet] - 10https://gerrit.wikimedia.org/r/121663 (owner: 10Hashar) [15:51:46] (03CR) 10BryanDavis: "I think Andrew is right that puppet::self::geoip should either be called from or rolled into puppet::self::master. I was trying to isolate" [operations/puppet] - 10https://gerrit.wikimedia.org/r/121677 (owner: 10Hashar) [15:53:13] to get this merged, who do I need to provide sexual favors to? https://gerrit.wikimedia.org/r/#/c/120617/ [15:54:50] o_O [15:54:58] gi11es: If I were you I'd ping paravoid and bblack ^^ [15:55:12] (03CR) 10Andrew Bogott: [C: 032] package-builder.pp: duplicate $build_place [operations/puppet] - 10https://gerrit.wikimedia.org/r/121631 (owner: 10Hashar) [15:55:25] And rescind the threat of sexual abuse ;) [15:55:46] * Nemo_bis thought the word meant something else [15:56:23] (03CR) 10Hashar: Puppet masters in labs needs puppet::self::geoip (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/121677 (owner: 10Hashar) [15:56:28] (03PS3) 10Hashar: Puppet masters in labs needs puppet::self::geoip [operations/puppet] - 10https://gerrit.wikimedia.org/r/121677 [15:56:52] (03CR) 10jenkins-bot: [V: 04-1] Puppet masters in labs needs puppet::self::geoip [operations/puppet] - 10https://gerrit.wikimedia.org/r/121677 (owner: 10Hashar) [15:56:54] (03PS4) 10Hashar: Puppet masters in labs needs puppet::self::geoip [operations/puppet] - 10https://gerrit.wikimedia.org/r/121677 [15:57:10] (03CR) 10Hashar: "Cherry picked to tip of production branch" [operations/puppet] - 10https://gerrit.wikimedia.org/r/121677 (owner: 10Hashar) [16:02:09] (03CR) 10Andrew Bogott: [C: 032] package-builder.pp: --create must be 1st arg to pbuilder [operations/puppet] - 10https://gerrit.wikimedia.org/r/121671 (owner: 10Hashar) [16:09:47] (03PS1) 10Hashar: beta: vary $wgGeoDataSolr* by datacenter [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121682 [16:10:17] (03CR) 10Hashar: [C: 032] beta: vary $wgGeoDataSolr* by datacenter [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121682 (owner: 10Hashar) [16:10:32] (03Merged) 10jenkins-bot: beta: vary $wgGeoDataSolr* by datacenter [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121682 (owner: 10Hashar) [16:10:34] (03CR) 10Hashar: "Added MaxSem for information" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121682 (owner: 10Hashar) [16:25:37] (03CR) 10coren: [C: 032] "I don't think we'll be able to do better until/unless we overhaul the labs DNS away from dnsmasq." [operations/dns] - 10https://gerrit.wikimedia.org/r/121416 (owner: 10Tim Landscheidt) [16:38:35] (03CR) 10Hoo man: "Both affected users confirmed per mail that the access given here is enough or their tasks." [operations/puppet] - 10https://gerrit.wikimedia.org/r/116019 (owner: 10Hoo man) [16:46:06] (03PS1) 10MaxSem: Prepare GeoData config for Elasticsearch switchover [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121691 [16:47:13] MaxSem: you're awesome. [16:50:24] manybubbles: i'm writing docs and organizing archiva a bit more [16:50:30] cool! [16:50:32] so [16:50:37] i'm thinking 3 repositories: [16:50:46] external, releases, snapshots [16:50:53] archiva comes with internal and snapshots by default [16:51:10] i think external should be for 3rd party artifacts, releases for wmf releases, and snapshots for wmf snapshots [16:51:15] that way 3rd party stuff is isolated from our own [16:51:24] i'm using 'internal' for that now [16:51:32] but i think it would make more sense if this was renamed to 'external'? [16:51:34] whatcha think? [16:53:55] makes sense [16:53:59] actually pretty standard [16:54:06] nexus does something like that [16:54:33] a place for mirrored stuff, a place for stuff you aren't mirroring but you didn't make, a place for stuff you made, a place for SNAPSHOTs you made, etc [16:54:58] right, but i'm not sure if it is normal to call the mirrored stuff 'external', sicne archiva comes with that called 'internal' by default [16:55:21] what would "a place for stuff you aren't mirroring" be? [16:55:43] like, artifacts that we add that aren't in a public maven repo somehwere? [16:55:47] deps that we've manually added? [16:55:53] oh, maybe like your elasticsaerch plugins? [16:56:35] !log shut down db40 [16:56:41] Logged the message, Master [17:00:14] (03PS1) 10Dzahn: remove db40, decom [operations/dns] - 10https://gerrit.wikimedia.org/r/121693 [17:01:31] manybubbles: so external makes sense instead of internal [17:01:32] ? [17:02:28] (03CR) 10Dzahn: [C: 032] remove db62, decom [operations/dns] - 10https://gerrit.wikimedia.org/r/121672 (owner: 10Dzahn) [17:02:59] (03CR) 10Dzahn: [C: 032] remove db40, decom [operations/dns] - 10https://gerrit.wikimedia.org/r/121693 (owner: 10Dzahn) [17:03:23] !log DNS update - remove db40,db62 [17:03:29] Logged the message, Master [17:05:42] ottomata: why not just call it mirrored? [17:05:50] I mean, we're not acting as a trasparent proxy [17:05:58] but we're mirroring it if we deem it ok [17:10:54] (03PS4) 10Gilles: Add ?download parameter to images [operations/puppet] - 10https://gerrit.wikimedia.org/r/120617 [17:14:16] hmmm [17:14:51] ok manybubbles, so you think we shoudl have 4 then? [17:15:05] +1 for non mirrored but still 3rd party artifacts? [17:15:18] not sure there is such a thing, really [17:15:32] if it isn't in maven central I wonder why [17:16:41] well, i mean, what about the elasticsearch plugin artifacts you added [17:16:56] all of them are in central [17:17:07] oh ok hmm [17:17:08] but I added them to "bless" them as being unmodified [17:17:22] aye right [17:17:53] there are a couple of non-central places taht are like central that are generally considered ok to mirror too [17:17:54] hmm, since we're talking about it, is there a reason to have separate releases and snapshots repos? [17:17:56] like scala-tools [17:18:00] which is only scala stuff [17:18:17] yeah, i'm fine with mirrored as long as it exists in a maven-like repo somewhere [17:18:18] yeah mostly because we don't want to deploy snapshots to the cluster [17:18:20] which could be used as a proxy [17:18:26] yeah but, what difference does it make? [17:18:29] a blessed proxy [17:18:32] we don't deploy using archiva, but git-fat [17:18:36] yeah [17:18:41] so maybe not much difference [17:18:44] tradition? [17:18:45] should we just have a single wikimedia repo? [17:18:58] maybe it will make a differnce in the future [17:19:10] I'd do mirrored, snapshot, release [17:19:16] like, snapshots would have relaxed deploy permissions [17:19:16] ok [17:19:19] ok will do [17:19:24] am I forgetting something? [17:19:27] i hope 'mirrored' doesn't bite us though [17:19:29] if so, we can add it later I think [17:19:33] so [17:19:36] why it bite us? [17:19:40] i mean, what if there is some awkward dep that isn't 'mirrored' [17:19:56] like, aw man I really want to use X.jar [17:20:03] but twitter hasn't released it yet [17:20:16] hmmmmmmm [17:20:30] maybe we have another one just for those [17:20:35] if we called it 'external' then i wouldn't care if it was in the same repo as mirrored stuff :p [17:20:36] it should mostly stay empty though [17:21:07] but having one just for mirrored would be convenient because we'd know everything in it is mirrored from somewhere and blessed [17:21:19] hm ok ok [17:21:22] i'll just do the 3 then [17:21:28] if we need the 4th we'll make it when we have to [17:21:31] hey, also, I was wondering if we should set up sync from our release repository to maven central [17:21:33] one day [17:21:38] sounds good [17:21:39] maybe one day, yeah [17:21:42] on making the fourth [17:21:45] i don't think we'll have that much generic stuff in there [17:23:34] (03PS1) 10Chad: Use proper php.ini comment format [operations/puppet] - 10https://gerrit.wikimedia.org/r/121695 [17:24:02] (03CR) 10Hoo man: [C: 031] Use proper php.ini comment format [operations/puppet] - 10https://gerrit.wikimedia.org/r/121695 (owner: 10Chad) [17:33:39] Reedy: ping [17:47:38] (03CR) 10Chad: [C: 032] Prepare GeoData config for Elasticsearch switchover [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121691 (owner: 10MaxSem) [17:48:22] (03Merged) 10jenkins-bot: Prepare GeoData config for Elasticsearch switchover [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121691 (owner: 10MaxSem) [17:48:55] !log demon synchronized wmf-config/InitialiseSettings.php 'Prepping GeoData config for Elasticsearch support. No-op' [17:49:02] Logged the message, Master [17:49:07] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [17:49:45] !log demon synchronized wmf-config/InitialiseSettings-labs.php 'Prepping GeoData config for Elasticsearch support. No-op' [17:49:51] Logged the message, Master [17:50:01] !log demon synchronized wmf-config/CommonSettings.php 'Prepping GeoData config for Elasticsearch support. No-op' [17:50:07] Logged the message, Master [17:51:15] (03PS1) 10MaxSem: Enable GeoData as a CirrusSearch plugin on labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121699 [17:52:18] (03CR) 10MaxSem: [C: 032] Enable GeoData as a CirrusSearch plugin on labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121699 (owner: 10MaxSem) [17:52:25] (03Merged) 10jenkins-bot: Enable GeoData as a CirrusSearch plugin on labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121699 (owner: 10MaxSem) [18:05:27] (03PS1) 10Nuria: Adding scheduler mode for wikimetrics in puppet [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/121700 [18:08:18] (03PS1) 10JGonera: Enable Compact Personal Bar and Winter on beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121702 [18:08:52] MaxSem, is this the right way to do it? ^ [18:09:51] jgonera, no. these variables will be overwritten after the extension's inclusion [18:10:24] MaxSem, so I need to create a new $wmg... var and set those vars in CommonSetting-labs.php? [18:10:38] set wmg variables in that array, then $wg = $wmg after require(VectorBeta) [18:10:50] right [18:13:25] (03PS2) 10JGonera: Enable Compact Personal Bar and Winter on beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121702 [18:13:33] MaxSem, ^ [18:14:18] jgonera, wouldn't these vars be undefined in prod? [18:14:51] undefined? wouldn't they be false which is what VectorBeta sets them to? [18:15:24] I mean, I assume that by modifying only *-labs.php files I'm not touching prod [18:15:31] (03CR) 10Ottomata: Adding scheduler mode for wikimetrics in puppet (033 comments) [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/121700 (owner: 10Nuria) [18:16:18] ah, it's CommonSettings-*labs* [18:16:24] then LGTM [18:20:41] (03PS1) 10MaxSem: Switch beta GeoData to Elasticsearch [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121704 [18:21:02] (03CR) 10MaxSem: [C: 032] Switch beta GeoData to Elasticsearch [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121704 (owner: 10MaxSem) [18:21:08] (03Merged) 10jenkins-bot: Switch beta GeoData to Elasticsearch [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121704 (owner: 10MaxSem) [18:27:49] MaxSem, if you can +1, that would be great ;) do I need someone else to approve? PM, ops? [18:28:16] jgonera, labs-only, JFDI:) [18:28:21] hoo: pong [18:28:35] then +2 MaxSem :) [18:28:48] Reedy: What do you know about the arbcom wikis? :P [18:28:55] they fail [18:28:59] also, does this needs to be deployed or will get picked up automatically? [18:29:01] Am already talking with Chri [18:29:01] s [18:29:04] need* [18:29:09] no. you're now a deployer yourself, get used to it :evilgrin: [18:29:13] hoo: what's the question? [18:29:31] MaxSem, to what, self-merge? [18:29:43] self merge is my life [18:29:48] Reedy: Why their wgServer is "wrong" ssl-wise [18:30:02] what is it? http less? [18:30:05] yes [18:30:07] eg. arbcom-nl.wiki... vs. arbcom.nl.wiki... [18:30:19] MaxSem, it wouldn't hurt if you did +1 though ;) [18:31:17] 'wgServer' => 'advisorywiki' => '//advisory.wikimedia.org', [18:31:19] uh [18:31:25] 'arbcom_enwiki' => '//arbcom.en.wikipedia.org', [18:31:31] Reedy: Will probably push a patch to gerrit [18:31:34] (03CR) 10MaxSem: [C: 031] Enable Compact Personal Bar and Winter on beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121702 (owner: 10JGonera) [18:31:37] To do what? [18:32:04] Reedy: To change the wgServer [18:32:09] To what? [18:32:22] Do you mean wgCanonicalServer? [18:32:27] Reedy: Both [18:32:38] I suspect it's due to the ssl cert being broken [18:32:40] well [18:32:43] "broken" [18:32:44] not right [18:32:56] haha MaxSem, I don't have a +2 on this repo ;) [18:33:15] eh? [18:33:35] jgonera, that is a bug - you should file a RT request [18:33:57] (03CR) 10Kaldari: [C: 032] Enable Compact Personal Bar and Winter on beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121702 (owner: 10JGonera) [18:33:59] (03CR) 10MaxSem: [C: 032] Enable Compact Personal Bar and Winter on beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121702 (owner: 10JGonera) [18:34:09] (03Merged) 10jenkins-bot: Enable Compact Personal Bar and Winter on beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121702 (owner: 10JGonera) [18:34:57] hoo: springle is looking at renaming stuffs in external storage... [18:35:14] Reedy: Renaming is probably not needed [18:35:30] the db names etc. are already arbcom_foo... [18:35:34] AFAIS [18:36:21] MaxSem, jgonera: i'll just add him; it doesn't require RT since he already has the access [18:36:35] arbcom_enwiki [18:36:41] thanks ori [18:36:48] en-arbcom.wikipedia [18:36:50] I already forgot how to file an RT ticket [18:36:52] arbcom-en.wikipedia [18:37:01] https://arbcom-fi.wikipedia.org/ etc. already work [18:37:02] ori, fixing an oversight is also a RTable issue;) [18:37:11] jgonera: if in doubt just mail the requests address in topic [18:37:31] !log added jgonera to wmf-deployment; was already a deployer but not in relevant gerrit group [18:37:36] hoo: o_0 [18:37:38] Logged the message, Master [18:37:44] Reedy: ? :P [18:37:48] since when? :| [18:37:54] Reedy: Dunno... [18:37:57] wiki*P*edia.org???! [18:38:01] mutante|away, thanks, handy [18:38:07] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [18:38:15] MaxSem: That's a Wikipedia related thing... so that's somehwat legit, I guess [18:38:23] what's up with the 5xxs? [18:38:27] wikiPMedia.org [18:39:15] Reedy: :D That one really is registered... [18:39:22] doesn't look active, though [18:40:00] ori, spambot again? logstash looks ok [18:41:17] Reedy: So... what's your opinion about changing the wg(Canonical)?Server ? [18:43:01] Does arbcom-en deal with non Wikipedia? [18:43:09] no [18:43:44] It's a project specific thing, really [18:43:50] (03CR) 10Milimetric: Adding scheduler mode for wikimetrics in puppet (031 comment) [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/121700 (owner: 10Nuria) [18:45:42] (03PS1) 10Hoo man: Change wgServer and wgCanonicalServer for arbcom wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121709 [18:46:00] Reedy: ^ [18:52:30] was rsvg updated recently? [18:52:53] if yes, then there's a regression in the new version; if not, then this is even more interesting. [18:53:04] https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#MW_SVG_to_PNG_conversion_issue [18:54:00] MatmaRex: has it had any security release anytime soon? [18:54:03] *recently [18:54:19] no idea, i'm asking here because i assume ops would know [18:58:13] http://changelogs.ubuntu.com/changelogs/pool/main/libr/librsvg/librsvg_2.36.1-0ubuntu1.1/changelog [18:58:20] Fri, 14 Mar 2014 [18:58:24] Let me check what's on the scalers [18:58:51] 2.36.1-1wm1 [18:58:59] yes, we have a patched librsvg [18:59:06] it wasn't changed, no [19:07:05] (03PS2) 10Nuria: Adding scheduler mode for wikimetrics in puppet [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/121700 [19:08:56] (03CR) 10Nuria: Adding scheduler mode for wikimetrics in puppet (032 comments) [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/121700 (owner: 10Nuria) [19:12:20] (03PS4) 10RobH: Add esanders shell account [operations/puppet] - 10https://gerrit.wikimedia.org/r/120851 (owner: 10Catrope) [19:12:36] not sure it really needs rebase, but cannot hurt. [19:13:19] MatmaRex: log link? [19:13:47] [19:52] was rsvg updated recently? [19:13:47] [19:58] http://changelogs.ubuntu.com/changelogs/pool/main/libr/librsvg/librsvg_2.36.1-0ubuntu1.1/changelog [19:13:47] [19:58] Fri, 14 Mar 2014 [19:13:47] [19:58] Let me check what's on the scalers [19:13:47] [19:58] 2.36.1-1wm1 [19:13:48] [19:59] yes, we have a patched librsvg [19:13:48] [19:59] it wasn't changed, no [19:13:50] PiRSquared: ^ [19:14:04] thanks [19:15:20] So is it a bug in the main librsvg or Wikimedia's patchy version? [19:16:26] (03CR) 10RobH: [C: 032] Add esanders shell account [operations/puppet] - 10https://gerrit.wikimedia.org/r/120851 (owner: 10Catrope) [19:19:15] (03CR) 10Ottomata: Adding scheduler mode for wikimetrics in puppet (032 comments) [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/121700 (owner: 10Nuria) [19:24:11] (03PS3) 10Nuria: Adding scheduler mode for wikimetrics in puppet [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/121700 [19:56:19] (03CR) 10Ottomata: [C: 032] Adding scheduler mode for wikimetrics in puppet [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/121700 (owner: 10Nuria) [19:56:23] (03CR) 10Ottomata: [V: 032] Adding scheduler mode for wikimetrics in puppet [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/121700 (owner: 10Nuria) [20:16:42] (03PS1) 10Nknudsen: Bug 34897 I4bdaa1b4c679356e6355987b31d1dce04ae85bd3 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121834 [21:06:45] (03PS1) 10Ottomata: Adding bacula backup::set for /var/lib/archiva. [operations/puppet] - 10https://gerrit.wikimedia.org/r/121856 [21:27:30] (03CR) 10Jkrauska: "patch set 3 coming -- addresses concerns in notes." (0310 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/115345 (owner: 10Jkrauska) [21:30:01] (03PS7) 10BryanDavis: Manage scap proxy rsync config in puppet [operations/puppet] - 10https://gerrit.wikimedia.org/r/119677 (owner: 10Reedy) [21:30:14] (03CR) 10jenkins-bot: [V: 04-1] Manage scap proxy rsync config in puppet [operations/puppet] - 10https://gerrit.wikimedia.org/r/119677 (owner: 10Reedy) [21:33:16] (03PS8) 10BryanDavis: Manage scap proxy rsync config in puppet [operations/puppet] - 10https://gerrit.wikimedia.org/r/119677 (owner: 10Reedy) [21:34:26] (03PS1) 10MaxSem: Back out Elasticsearch for beta GeoData for now [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121866 [21:38:35] (03CR) 10MaxSem: [C: 032] Back out Elasticsearch for beta GeoData for now [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121866 (owner: 10MaxSem) [21:39:09] (03Merged) 10jenkins-bot: Back out Elasticsearch for beta GeoData for now [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121866 (owner: 10MaxSem) [21:39:21] (03CR) 10BryanDavis: "Manually rebased, changed the variable name to be more consistent with other network ranges and added usage of new consolidated range vari" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119677 (owner: 10Reedy) [21:40:10] (03PS4) 10BryanDavis: Add mw1161 and mw1201 as scap proxies for EQIAD row C and D [operations/puppet] - 10https://gerrit.wikimedia.org/r/119686 (owner: 10Reedy) [21:41:03] (03PS1) 10MaxSem: Put a safeguard on GeoData's usage of CirrusSearch [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121874 [21:48:20] (03PS3) 10Jkrauska: Initial commit of pmacct module and role [operations/puppet] - 10https://gerrit.wikimedia.org/r/115345 [22:16:37] (03PS4) 10Jkrauska: Initial commit of pmacct module and role [operations/puppet] - 10https://gerrit.wikimedia.org/r/115345 [22:31:06] (03PS5) 10Jkrauska: Initial commit of pmacct module and role [operations/puppet] - 10https://gerrit.wikimedia.org/r/115345 [23:02:50] (03PS1) 10MaxSem: Revert "Back out Elasticsearch for beta GeoData for now" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121887 [23:03:03] (03CR) 10MaxSem: [C: 032] Revert "Back out Elasticsearch for beta GeoData for now" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121887 (owner: 10MaxSem) [23:14:12] (03CR) 10MaxSem: [V: 032] Revert "Back out Elasticsearch for beta GeoData for now" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121887 (owner: 10MaxSem) [23:25:37] Coren, andrewbogott_afk: how do I allocate an IP to a project? (I tried it via Special:NovaAddress and got "Failed to allocate new public IP address. ") [23:26:01] ori: The default quota for IPs is 0; so you probably need it raised. What project is this? [23:26:11] Coren: quality-assurance [23:26:56] ori: U can haz eyepee [23:27:00] <3 [23:28:00] thanks [23:31:28] (03PS1) 10Jforrester: Use the BetaFeatures whitelist for production to avoid accidental deploys [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121892 [23:31:38] (03CR) 10jenkins-bot: [V: 04-1] Use the BetaFeatures whitelist for production to avoid accidental deploys [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121892 (owner: 10Jforrester) [23:32:23] (03PS2) 10Jforrester: Use the BetaFeatures whitelist for production to avoid accidental deploys [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121892 [23:34:38] (03CR) 10Jforrester: [C: 04-1] "Uses code that won't go out until wmf21, so better to merge when its effects can be observed in production." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121892 (owner: 10Jforrester) [23:35:20] greg-g: FYI https://gerrit.wikimedia.org/r/#/c/121892/ name-checks you as an authority for BF deployment.