[00:00:38] Vogone: standard deadline for translation is around a week right? [00:01:13] actually I'll go without a deadline and see what happens? [00:01:19] (03PS1) 10Dzahn: remove HTTPS config from gitblit template [operations/puppet] - 10https://gerrit.wikimedia.org/r/154973 [00:01:53] (03CR) 10Dzahn: "git.wikimedia.org is an alias for misc-web-lb.eqiad.wikimedia.org." [operations/puppet] - 10https://gerrit.wikimedia.org/r/154973 (owner: 10Dzahn) [00:02:00] Thehelpfulone: I believe if de and es are complete it is fine [00:02:09] de is almost done [00:02:48] I just wonder how to translate "Based on a design by", as in German the subject would be in the middle of that sentence :p [00:03:14] "Basiert auf einem Design von" [00:03:27] then a colon would be required, no? [00:03:36] hm [00:04:13] egal [00:04:17] i'm more wondering if "Design" is really German [00:04:27] but yes, http://en.wiktionary.org/wiki/Design [00:05:03] so de is complete then :) [00:05:08] i should have slept already instead of waiting because i am "MUST be available" :/ [00:06:07] Thehelpfulone: https://meta.wikimedia.org/wiki/Mailing_lists/Old_list_info -- made a few changes as you can see [00:06:25] yep, happy with that [00:06:37] Thehelpfulone: have you made changes to that page while I was translating? a section failed to save :p [00:07:10] oic [00:07:16] the translate thing was moved [00:07:19] fine then [00:07:49] Thehelpfulone: https://meta.wikimedia.org/wiki/Mailing_lists/List_info/de voila [00:07:54] (03CR) 10Dzahn: [C: 032] "we have already done for stats.wm , so metrics.wm should really follow and be ok because it's the same host" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154315 (owner: 10Dzahn) [00:10:17] (03PS6) 10John F. Lewis: mailman: use a new default theme (prettier mailman) [operations/puppet] - 10https://gerrit.wikimedia.org/r/154964 (https://bugzilla.wikimedia.org/61283) [00:10:21] Vogone: ^ [00:11:01] PROBLEM - puppet last run on stat1001 is CRITICAL: CRITICAL: Epic puppet fail [00:11:17] yea, ^ fixing that [00:12:33] Vogone: does it look good on https://mailman.wmflabs.org/mailman/listinfo/default ? [00:12:41] PROBLEM - Disk space on lanthanum is CRITICAL: DISK CRITICAL - free space: /var/lib/jenkins-slave/tmpfs 15 MB (3% inode=99%): [00:12:59] JohnFLewis: can you sync the default with github again please? the translate link should be on there now too [00:14:22] Ok [00:14:27] JohnFLewis: nope [00:14:29] the Umlauts aren't displayed properly [00:14:30] "WÃ¼rdest" [00:14:32] Unicode support required!! :p [00:15:12] "Ãœber Wikimedia" :p [00:15:57] Thehelpfulone: ^ should just be a tag to add so google it :p [00:15:58] JohnFLewis: where to translate "View this page in"? [00:16:02] Vogone: that's mailman itself [00:16:13] it's completely wrong [00:16:16] in German [00:16:23] Vogone: mailman :p [00:16:29] go complain to them :D [00:16:56] anyway, the button is quite large [00:16:57] (03PS7) 10John F. Lewis: mailman: use a new default theme (prettier mailman) [operations/puppet] - 10https://gerrit.wikimedia.org/r/154964 (https://bugzilla.wikimedia.org/61283) [00:17:21] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There are 2 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [00:17:58] Vogone: the translator probably did German / English [00:17:58] yes [00:17:58] now it looks ugly :p [00:18:02] (03CR) 10Bartosz Dziewoński: "The issue has just been mentioned again, on VPT: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Protection_level_selecti" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [00:18:04] (03PS1) 10Dzahn: metrics: remove broken dependency on Apache site [operations/puppet] - 10https://gerrit.wikimedia.org/r/154976 [00:18:42] JohnFLewis: and why is the no checkbox before the yes checkbox? [00:19:08] the way the theme was designed I assume [00:19:19] blame Quim and Thehelpfulone :p I'm applying their work to puppet [00:19:40] Vogone: that's how it was originally IIRC [00:19:54] yep, originally [00:19:55] hm, looks strange at least to me [00:19:58] JohnFLewis: so I tried to add The page you saved contains suspicious HTML that could potentially expose your users to cross-site scripting attacks. This change has therefore been rejected. If you still want to make these changes, you must have shell access to your Mailman server. [00:20:12] when I tried to add [00:20:14] (03CR) 10Dzahn: [C: 032] "Could not find dependency File[/etc/apache2/sites-enabled/metrics.wikimedia.org] for File[/var/log/apache2/access.metrics.log] at /etc/pup" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154976 (owner: 10Dzahn) [00:20:39] hm? [00:20:45] so can you add that to the default one below html lang line [00:20:57] and sync it through shell on labs? [00:21:02] ok [00:21:03] Thehelpfulone: I think I can't add the wm-bot in my channel. Can't use @part either. [00:21:04] JohnFLewis: could you update the de file as well? [00:21:06] thanks [00:21:12] I changed a translation [00:21:17] Vogone: will do [00:21:20] k [00:21:22] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [00:21:23] Romtam: which channel? [00:21:58] Thehelpfulone: ##Romtam [00:21:58] Thehelpfulone: {{done}} [00:21:58] thanks [00:22:01] RECOVERY - puppet last run on stat1001 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [00:22:02] Bot is probably banned or stuck in some way. For example if you invite a bot and then use freenode's /remove command (which isn't compliant with irc standards anyway so it makes sense the bot doesn't know it) the bot will still think it's in the channel. You can try to make it part the channel from #wm-bot by calling @part #channel [00:22:09] @part ##romtam [00:22:13] (03CR) 10Dzahn: "<+icinga-wm> RECOVERY - puppet last run on stat1001 is OK" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154976 (owner: 10Dzahn) [00:22:15] Romtam: now try to make it join? [00:22:41] PROBLEM - Disk space on lanthanum is CRITICAL: DISK CRITICAL - free space: /var/lib/jenkins-slave/tmpfs 15 MB (3% inode=99%): [00:22:55] Works. Thanks for the help! [00:23:07] (03CR) 10Dzahn: "Info: /Stage[main]/Misc::Statistics::Sites::Metrics/Apache::Site[metrics.wikimedia.org]/Apache::Conf[metrics.wikimedia.org]/File[/etc/apac" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154315 (owner: 10Dzahn) [00:23:46] JohnFLewis: need a sync with https://github.com/quimgil/mailman-templates/edit/master/mailman-template.html please (then add the tag) [00:23:51] Romtam: you're welcome :) [00:25:28] Thehelpfulone: Also, big thanks for the fast cloak delivery and the autovoice in the help channel :D [00:25:38] Thehelpfulone: done [00:26:07] Romtam: heh cloak delivery is not done by me, but the Group Contacts (as soon as you get a cloak you're autovoiced) but you're welcome ;) [00:26:09] thanks JohnFLewis [00:26:31] Ok bye! [00:28:30] (03PS8) 10John F. Lewis: mailman: use a new default theme (prettier mailman) [operations/puppet] - 10https://gerrit.wikimedia.org/r/154964 (https://bugzilla.wikimedia.org/61283) [00:28:40] JohnFLewis: heh [00:28:41] Error Line 1, Column 77: Internal encoding declaration utf-8 disagrees with the actual encoding of the document (us-ascii). [00:28:41] [00:28:45] http://validator.w3.org/check?uri=https%3A%2F%2Fmailman.wmflabs.org%2Fmailman%2Flistinfo%2Fdefault&charset=%28detect+automatically%29&doctype=Inline&group=0&user-agent=W3C_Validator%2F1.3+http%3A%2F%2Fvalidator.w3.org%2Fservices [00:29:21] is that the way the document is saved? [00:29:31] (03PS1) 10Dzahn: bugzilla - use ssl_ciphersuite to add HSTS [operations/puppet] - 10https://gerrit.wikimedia.org/r/154978 [00:29:43] https://stackoverflow.com/questions/10158743/how-do-i-change-the-actual-encoding-of-my-html-document [00:30:56] Thehelpfulone: $myeditor - how to enabled UTF-8 [00:31:17] well, it can be OS and shell and editor, but start with editor [00:34:10] (03CR) 10Dzahn: [C: 031] OTRS - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153998 (owner: 10Dzahn) [00:34:20] (03PS5) 10Dzahn: OTRS - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153998 [00:37:17] JohnFLewis: ^ you'd need to change that on the gerrit patch too right? [00:37:46] Thehelpfulone: it depends how you change it :p [00:37:58] I'll let you do it then ;) [00:45:31] (03PS1) 10Dzahn: put racktables behind misc. varnish [operations/puppet] - 10https://gerrit.wikimedia.org/r/154980 [00:46:09] (03CR) 10Dzahn: [C: 04-2] "needs DNS change" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154980 (owner: 10Dzahn) [01:00:21] PROBLEM - Puppet freshness on db1010 is CRITICAL: Last successful Puppet run was Mon 18 Aug 2014 22:59:39 UTC [01:19:52] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Tue Aug 19 01:19:50 UTC 2014 [02:16:18] !log LocalisationUpdate completed (1.24wmf16) at 2014-08-19 02:15:14+00:00 [02:16:28] Logged the message, Master [02:37:25] !log LocalisationUpdate completed (1.24wmf17) at 2014-08-19 02:36:21+00:00 [02:37:29] Logged the message, Master [02:57:11] (03CR) 10Legoktm: "This broke the RSS feed on because you can't use the proxy to access internal things, b" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/152917 (owner: 10Legoktm) [02:57:21] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Tue 19 Aug 2014 00:56:57 UTC [02:57:31] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Tue Aug 19 02:57:23 UTC 2014 [03:21:27] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Aug 19 03:20:21 UTC 2014 (duration 20m 20s) [03:21:33] Logged the message, Master [03:25:19] (03CR) 10Chmarkine: [C: 031] bugzilla - use ssl_ciphersuite to add HSTS [operations/puppet] - 10https://gerrit.wikimedia.org/r/154978 (owner: 10Dzahn) [04:26:56] (03PS3) 10Springle: tendril - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153984 (owner: 10Dzahn) [04:28:04] (03CR) 10Springle: [C: 032] tendril - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153984 (owner: 10Dzahn) [05:00:21] PROBLEM - Puppet freshness on db1010 is CRITICAL: Last successful Puppet run was Tue 19 Aug 2014 02:59:35 UTC [06:28:51] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:51] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:01] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:02] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:11] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 1 failures [06:39:51] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Tue Aug 19 06:39:50 UTC 2014 [06:40:28] (03PS1) 10Springle: Maintenance reports limit incremental increase. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154986 [06:45:51] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:45:51] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:46:01] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:46:02] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:47:11] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:50:01] PROBLEM - puppet last run on db1043 is CRITICAL: CRITICAL: Puppet has 1 failures [07:05:20] (03PS1) 10BBlack: reduce passenger max reqs back to 2K [operations/puppet] - 10https://gerrit.wikimedia.org/r/154989 [07:05:30] (03CR) 10BBlack: [C: 032 V: 032] reduce passenger max reqs back to 2K [operations/puppet] - 10https://gerrit.wikimedia.org/r/154989 (owner: 10BBlack) [07:07:42] !log restarted apache2 service on strontium/palladium, expect another small spike of puppet fail->ok [07:07:49] Logged the message, Master [07:09:12] PROBLEM - puppet last run on ms-be3001 is CRITICAL: CRITICAL: Epic puppet fail [07:09:21] PROBLEM - puppet last run on labmon1001 is CRITICAL: CRITICAL: Epic puppet fail [07:09:55] !log ... and strontium passenger is failing to start up correctly again. icinga-wm disabled to avoid spam [07:10:01] Logged the message, Master [07:17:50] !log Jenkins: manually cleared out a tmpfs partition on lanthanum.eqiad.wmnet which was causing all MediaWiki / extensions jobs to fail completely. {{bug|69731}}. We need disk space monitoring which is {{bug|69733}}. [07:17:55] Logged the message, Master [07:20:22] so with the strontium thing - after a few failed attempts to start it from the commandline, I left apache2 shut off and went to google some things, and 2 minutes later the puppet agent ran and started it successfully [07:20:46] still no real indicators or log outputs or anything on what causes it fail to start [07:21:31] just waiting now for ~20 mins for all of the "Puppet is ok" spam to end before turning on icinga-wm again [07:39:49] !log strontium ok, icinga-wm back [07:39:56] Logged the message, Master [08:25:21] PROBLEM - puppet last run on amssq53 is CRITICAL: CRITICAL: Epic puppet fail [08:44:21] RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [09:16:06] (03CR) 10Filippo Giunchedi: beta: Change auth realmmessage for logstash-beta (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/152932 (https://bugzilla.wikimedia.org/69267) (owner: 10BryanDavis) [09:22:13] (03CR) 10Filippo Giunchedi: [C: 031] HHVM: improvements to logging [operations/puppet] - 10https://gerrit.wikimedia.org/r/154414 (owner: 10Ori.livneh) [09:32:07] (03CR) 10Lokal Profil: "just spotted that the kmb white-listing is superfluous to an existing one. Sorry about that." (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154718 (https://bugzilla.wikimedia.org/69439) (owner: 10John F. Lewis) [09:32:34] (03CR) 10Filippo Giunchedi: [C: 031] Provision role::jobrunner on osmium [operations/puppet] - 10https://gerrit.wikimedia.org/r/154378 (owner: 10Ori.livneh) [09:35:41] (03CR) 10Filippo Giunchedi: exim templates - deprecated variable syntax (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/154371 (owner: 10Dzahn) [09:48:35] (03CR) 10Ricordisamoa: "The more patches get merged in that part of code, the harder will be to revert the original 'superprotect' introduction, which (in my hope" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [09:58:41] !log depool mw1019 from appservers, testing trusty+hhvm reinstall RT #8153 [09:58:47] Logged the message, Master [09:59:58] (03CR) 10TTO: "Could you please tell me exactly what to do here, or else do the change yourself? Sorry to be blunt but I'm very busy at the moment..." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154448 (https://bugzilla.wikimedia.org/69055) (owner: 10TTO) [10:00:43] (03CR) 10Ori.livneh: [C: 032] HHVM: improvements to logging (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/154414 (owner: 10Ori.livneh) [10:07:01] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: Puppet has 1 failures [10:09:03] (03PS1) 10Ori.livneh: Correct erronuous file references from I11b85c87a [operations/puppet] - 10https://gerrit.wikimedia.org/r/154998 [10:09:24] (03CR) 10Ori.livneh: [C: 032 V: 032] Correct erronuous file references from I11b85c87a [operations/puppet] - 10https://gerrit.wikimedia.org/r/154998 (owner: 10Ori.livneh) [10:10:52] PROBLEM - puppet last run on osmium is CRITICAL: CRITICAL: Puppet has 1 failures [10:14:27] (03PS1) 10Ori.livneh: Crreot another file reference from I11b85c87a [operations/puppet] - 10https://gerrit.wikimedia.org/r/154999 [10:14:55] (03CR) 10Ori.livneh: [C: 032 V: 032] Crreot another file reference from I11b85c87a [operations/puppet] - 10https://gerrit.wikimedia.org/r/154999 (owner: 10Ori.livneh) [10:17:02] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [10:19:09] !log Jenkins: bringing back irc bot wmf-insecte in #wikimedia-qa . Will be used to notify failures/fixe of the beta cluster jenkins jobs [10:19:14] Logged the message, Master [10:19:15] ACKNOWLEDGEMENT - puppet last run on ms-be1005 is CRITICAL: CRITICAL: Puppet has 1 failures Filippo Giunchedi broken disk [10:22:43] (03PS1) 10Filippo Giunchedi: hhvm: reimage mw1019 with trusty and hhvm [operations/puppet] - 10https://gerrit.wikimedia.org/r/155000 [10:28:19] (03PS2) 10Filippo Giunchedi: hhvm: reimage mw1019 with trusty and hhvm [operations/puppet] - 10https://gerrit.wikimedia.org/r/155000 [10:29:51] RECOVERY - puppet last run on osmium is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [10:33:27] what's on mw1019? [10:35:07] a generic appserver [10:36:51] ok, better line up for my connecting flight [10:36:58] back in 5-6 hours [10:37:04] hmmm, so like testing hhvm with one of the wikipedias? [10:37:22] the app servers can server all wikis [10:37:35] so we're not selecting a wiki to be hhvm-exclusive [10:37:47] rather, we're selecting a small subset ot traffic pan-wikis [10:37:51] to be hhvm specific [10:37:56] intiially only via opt in [10:38:19] ori: safe flights! [10:38:24] thanks :) ttyl [10:38:48] I wonder if I should also remove mw1019 from some dsh groups to avoid surprises during deployments ori bd808|BUFFER [10:39:09] you could, yes [10:39:31] after reimaging, you have to: [10:39:40] which groups though? [10:39:46] "mediawiki-installation" [10:39:55] i assumed you removed it from pybal? [10:40:01] yep [10:40:13] ack so mediawiki-installation [10:40:22] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 7.14% of data above the critical threshold [500.0] [10:40:42] (a) run giuseppe's script to re-sync keys between minion and saltmaster [10:41:12] would be nice to have wikibase tests passing before wikidata gets hhvm [10:41:12] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Puppet has 1 failures [10:41:20] i think we are almost there [10:41:30] yep I'm looking at http://etherpad.wikimedia.org/p/app-server-upgrade essentially ori [10:41:50] if wikidata can opt-out at this point and only do test wikidata, we'd much prefer that [10:42:04] and hope our "client" parts work ok [10:42:50] there's one thing missing from it [10:42:53] * ori digs it up [10:44:01] godog: under "Symptom: Apache can't start because its docroot is missing. " [10:44:04] i added a sixth item [10:44:16] sudo -u mwdeploy -n -- scap-rebuild-cdbs [10:45:07] we have to make mediawiki deployment be less of a pile of shit one of these days [10:45:13] (03PS3) 10Filippo Giunchedi: hhvm: reimage mw1019 with trusty and hhvm [operations/puppet] - 10https://gerrit.wikimedia.org/r/155000 [10:45:19] haha thanks ori ! [10:46:28] I can harass Reedy too for deployment-related things! [10:46:40] godog: note that you'll meed to re-add it to the mediaiwki-installation dsh-group if you want it to continue receiving rolling deployments [10:46:42] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] hhvm: reimage mw1019 with trusty and hhvm [operations/puppet] - 10https://gerrit.wikimedia.org/r/155000 (owner: 10Filippo Giunchedi) [10:46:45] ok/ me runs [10:46:52] tty folks [10:47:30] ori: yep, thanks! [10:49:12] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Epic puppet fail [10:49:21] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: Puppet has 1 failures [10:58:12] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [11:01:22] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [11:07:22] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [11:08:12] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [11:54:04] hey - I get Unknown error in PHP's mail() function. at http://en.wikipedia.beta.wmflabs.org/wiki/Special:EmailUser [11:56:34] tonythomas: This is just on the beta cluster right and works on production correct? [11:56:52] JohnLewis: let me check on production [11:57:31] works well on https://en.wikipedia.org/wiki/Special:EmailUser [11:57:57] Ok. [12:01:02] The beta cluster has been iffy lately anyway as a result of HHVM (or the correct acronym :p) [12:03:10] JohnLewis: hope so. was working well yesterday though [12:03:28] yeah but things change :) [12:03:43] I would look in the logs but I'm currently being a mindless copy and paster so [12:04:56] JohnLewis:: just as an FYI I made some tweaks to the design last night [12:05:00] https://mailman.wmflabs.org/mailman/listinfo/test [12:05:03] "list members" [12:05:27] so many tweaks; so many outdated translations :( but ok :p [12:05:38] https://meta.wikimedia.org/w/index.php?title=Mailing_lists/List_info&diff=9567083&oldid=9565953 [12:05:51] yeah it was quite a few to get that new section added :P [12:06:37] Thehelpfulone: when I upload the new translations; look for any non-1.5s and poke translators for me will you? :) [12:07:55] then after this I'm going to see what we can do about .15 [12:08:37] sure [12:14:34] Thehelpfulone: just wanted to wave and say it was nice meeting you in person :) [12:14:49] JohnLewis: appropriate acronym now is HAT (HHVM Apache Trusty) :) [12:15:10] (03PS1) 10Chmarkine: ssl_ciphersuite - change Header add to Header set [operations/puppet] - 10https://gerrit.wikimedia.org/r/155016 [12:15:12] YuviPanda: alright :) [12:15:35] JohnLewis: since we're doing all three at once :) HHVM with Apache FastCGI on Ubuntu Trusty [12:15:42] what could possibly go wrong... :) [12:16:11] I have no idea :D [12:18:34] <^d> YuviPanda: is fastcgi fast? [12:18:48] ^d: Is HHVM Hip? [12:19:05] <^d> not anymore. [12:19:44] and again - One of our patch to bouncehandler extension which merged yestereday, is not yet there in beta yet [12:19:49] it can take that long ? [12:20:07] this one https://gerrit.wikimedia.org/r/#/c/154860/ [12:21:50] (03PS9) 10John F. Lewis: mailman: use a new default theme (prettier mailman) [operations/puppet] - 10https://gerrit.wikimedia.org/r/154964 (https://bugzilla.wikimedia.org/61283) [12:22:15] Thehelpfulone ^ [12:26:49] (03PS1) 10John F. Lewis: mailman: add 4 new languages [operations/puppet] - 10https://gerrit.wikimedia.org/r/155017 [12:27:36] Thehelpfulone ^ look at that please quickly :) [12:31:28] (03CR) 10John F. Lewis: "I6b0de11b1f6b6ea1ad851b0457480847b3dfe679 should be merged before this is." [operations/puppet] - 10https://gerrit.wikimedia.org/r/154964 (https://bugzilla.wikimedia.org/61283) (owner: 10John F. Lewis) [12:34:32] (03CR) 10Chmarkine: puppetmaster - use ssl_ciphersuite (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/153986 (owner: 10Dzahn) [12:38:53] <^d> aude: About? [12:40:24] <^d> I'm looking at the train deploy today and it mentions some Wikidata things. Anything special I need to do for that or should it just go with the train? [12:43:44] (03CR) 10Mark Bergsma: [C: 031] "Minor: I think it would be good to explicitly unset the X-Use-HHVM request header if we don't intend to set it to 1." [operations/puppet] - 10https://gerrit.wikimedia.org/r/152903 (owner: 10Mark Bergsma) [12:43:47] ^d: Lydia says she is in a meeting at the minute. She'll be back in ~1 hour? :) [12:44:04] <^d> No worries, got several hours until deploy. [12:44:10] <^d> Thanks for relaying :) [12:48:43] (03CR) 10Yuvipanda: [C: 031] "Tested to work on precise and trusty." [operations/puppet] - 10https://gerrit.wikimedia.org/r/154712 (owner: 10Yuvipanda) [12:49:38] (03PS1) 10Filippo Giunchedi: puppetmaster: fix reimage.sh [operations/puppet] - 10https://gerrit.wikimedia.org/r/155019 [12:50:14] JohnLewis:: that patch, will that break anything in an update? [12:50:30] Thehelpfulone: which one? [12:50:33] brb, lunch [12:50:35] the 4 new languages [12:50:51] nope [12:50:56] if nothing else exists for those languages in terms of the other files, does it default to english? [12:51:04] Yes [12:51:22] it defaults to SEVER_LANUAGE which is en here :) [12:52:00] ok cool [12:53:15] LANUAGE? [12:53:22] a typo? [12:53:30] presumably, SERVER too ;) [12:53:46] hehe, didn't notice that one even :D [12:55:32] :p [12:56:24] No - mailman just doesn't like the word SERVER and LANGUAGE :p [13:10:08] (03CR) 10Hashar: "@Ricordisamoa stop instilling FUD ( https://en.wikipedia.org/wiki/Fear,_uncertainty_and_doubt ) in the community . Changes to user rights " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [13:10:17] (03PS3) 10Hashar: Fix $wgRestrictionLevels ordering [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [13:10:53] (03CR) 10Hashar: [C: 032] Fix $wgRestrictionLevels ordering [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [13:10:57] (03Merged) 10jenkins-bot: Fix $wgRestrictionLevels ordering [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154376 (https://bugzilla.wikimedia.org/69640) (owner: 10Anomie) [13:12:38] !log hashar Synchronized wmf-config/InitialiseSettings.php: Fix $wgRestrictionLevels ordering {{bug|69640}} (duration: 00m 04s) [13:12:44] Logged the message, Master [13:13:02] ^d: nothing special [13:13:12] when you are done, i can do config changes [13:13:41] <^d> aude: Okie dokie, thx [13:14:59] my tiny change is now deployed :D [13:39:44] anomie: hi ! I am finding that changes committed in gerrit under extensions/BounceHandler ( a night ago ) is not yet deployed in beta. Do it wait this long ? [13:39:47] !log Jenkins upgrading hhvm on the Trusty Jenkins slave integration-slave1006-trusty : Unpacking hhvm (3.3-dev+20140728+wmf4) over (3.3-dev+20140728+wmf3) [13:39:52] Logged the message, Master [13:40:13] tonythomas: Beta should deploy in a few minutes. hashar_, any idea? [13:40:45] anomie: yeah. thats what I too was thinking. this one https://gerrit.wikimedia.org/r/#/c/154860/ [13:41:15] I find the file same in /srv/common-local/php-master/extensions/BounceHandler [13:42:22] anomie: deploy what? [13:42:46] hashar_: tonythomas is observing that https://gerrit.wikimedia.org/r/#/c/154860/ seems to not have been auto-deployed on Beta. [13:43:32] yeah. ( if /srv/common-local/php-master/extensions/BounceHandler ) is the extension folder for en.wikipedia.beta.wmflabs.org [13:45:32] anomie: tonythomas it deploys using the repo mediawiki/extensions.git which is autoupdated by Gerrit itself [13:45:59] hashar_: It looks like things are not happening -- even after a day [13:46:00] :( [13:46:41] on deployment-bastion there is * 715b466 - (HEAD, origin/master, origin/HEAD) Use my real name in the credits (6 hours ago) [13:46:56] hashar_: that seems right ! [13:47:02] looking looking [13:47:21] but I dont see the change in BounceHandlerHooks.php or BounceHandler.php too [13:47:51] and it seems up to date on deployment-mediawiki02 which serves the traffic [13:48:34] so /srv/common-local/php-master/extensions/BounceHandler is not the place to look for ? [13:49:01] not sure what it is [13:49:40] tonythomas: yeah that is the proper path [13:49:52] hashar_: and you are able to see that change in there ? [13:50:17] Aug 18 17:23 BounceHandlerHooks.php [13:50:22] UTC [13:50:43] and yeah the change is there [13:51:07] hashar_: looks like there is sth wrong with my shell :\ [13:51:28] I see the changes still unaffected there in BounceHandlerHooks.php [13:51:43] might be due to some permission issue ? [13:52:05] on which host ? [13:52:12] eployment-bastion [13:52:14] I am on deployment-mediawiki02 which serves the traffic [13:52:16] *deployment-bastion [13:52:18] oh. [13:52:32] looks like I am entirely in the wrong host [13:52:42] thanks hashar_ :) [13:52:47] on bastion the files are fetched in /srv/scap-stage-dir [13:53:12] oh. [13:53:21] ah found an issue on bastion [13:53:49] ah. looks like the patch is merged in here :) [13:54:14] (03PS1) 10Chad: Configure swift-backed snapshots for Cirrus in beta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/155030 [13:54:33] (03CR) 10Chad: [C: 032] "Happy fun times. Gogogo!" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/155030 (owner: 10Chad) [13:55:00] (03Merged) 10jenkins-bot: Configure swift-backed snapshots for Cirrus in beta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/155030 (owner: 10Chad) [13:55:14] (03CR) 10Yuvipanda: [C: 031] "@Mutante: No, this was sorted to be closer to tungsten, our prod graphite/txstatsd host :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154854 (owner: 10Yuvipanda) [13:56:07] (03CR) 10Yuvipanda: "It also puts data in /srv by default, which is ok with the way labmon is configured." [operations/puppet] - 10https://gerrit.wikimedia.org/r/154854 (owner: 10Yuvipanda) [13:58:39] <^d> Somebody forgot beta again. [13:58:42] <^d> Notice: Undefined index: wmgExtraLanguageNames in /mnt/srv/scap-stage-dir/php-master/includes/SiteConfiguration.php on line 305 [13:59:12] ^d: everyone forgets beta [13:59:13] poor beta [13:59:23] heh, 'beta' is close to 'son' in Hindi [13:59:26] haven't noticed that before [13:59:42] (03PS2) 10Aude: Add good and featured badges to config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149918 (https://bugzilla.wikimedia.org/40810) (owner: 10Bene) [13:59:44] (03PS1) 10Aude: Enable redirects on Wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/155034 [13:59:46] (03PS1) 10Aude: Enable adding site links to wikidata (non-entity) pages on Wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/155035 [13:59:48] (03PS1) 10Aude: Enable WikibaseClient on Wikidata, with arbitrary data access [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/155036 [13:59:50] (03PS1) 10Aude: Enable WikibaseClient on Wikinews [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/155037 [14:00:42] (03CR) 10Aude: [C: 04-2] "not to be deployed until after today after the 'train', later today" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/149918 (https://bugzilla.wikimedia.org/40810) (owner: 10Bene) [14:01:55] ^d: even I got a php error in http://en.wikipedia.beta.wmflabs.org/wiki/Special:EmailUser :D [14:04:30] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: There is one unmerged change in mediawiki_config (dir /a/common/). [14:04:42] <^d> Yes yes icinga. [14:05:28] !log demon Synchronized wmf-config/CirrusSearch-labs.php: beta swift config, no-op (duration: 00m 04s) [14:05:29] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge. [14:05:34] Logged the message, Master [14:18:58] (03CR) 10Yuvipanda: [C: 04-1] "ugh, no, it won't put them on /srv since $::realm will point to prod and not to labs." [operations/puppet] - 10https://gerrit.wikimedia.org/r/154854 (owner: 10Yuvipanda) [14:29:50] (03CR) 10Andrew Bogott: graphite: Fixes to work with trusty (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/154712 (owner: 10Yuvipanda) [14:35:33] manybubbles: I'll do the SWAT today, unless you have your heart set on doing it. [14:35:41] anomie: have fun! [14:35:56] That's a no then :p [14:37:17] (03PS1) 10Chad: Group1 wikis to 1.24wmf17 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/155043 [14:37:43] !log demon updated /a/common to {{Gerrit|I035cebe20}}: Configure swift-backed snapshots for Cirrus in beta [14:37:49] Logged the message, Master [14:37:53] (03CR) 10Chad: [C: 04-2] "Prep for today's train. Will merge later." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/155043 (owner: 10Chad) [14:38:42] (03CR) 10Hoo man: [C: 04-1] "Looks ok despite" (034 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/155035 (owner: 10Aude) [14:39:39] (03PS1) 10coren: Labs: give labstore1003 a (barebones) manifest [operations/puppet] - 10https://gerrit.wikimedia.org/r/155045 [14:39:53] apergos: ^^ [14:40:07] (03CR) 10Anomie: [C: 04-1] Add two commonswiki CopyUploadDomains (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154718 (https://bugzilla.wikimedia.org/69439) (owner: 10John F. Lewis) [14:40:26] JohnLewis: Fix the comments there, please ^ [14:40:41] oh I forgot. will do [14:41:20] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: Epic puppet fail [14:42:16] d you need the ldap includes, Coren? [14:42:47] apergos: Not for that box; there's no write access from labs so no need to know its userids, etc. [14:42:53] ok [14:44:11] it's not an rsync server then? [14:44:42] (03PS2) 10John F. Lewis: Add a commonswiki CopyUploadDomains [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154718 (https://bugzilla.wikimedia.org/69439) [14:45:02] apergos: Not really. The entire box only exists to provide a filesystem to export. :-) [14:45:29] apergos: Or did we do pagecount by /fetching/ by rsync rather than pushing from dataset*? I don't remember. [14:47:27] well it's your cronjob at this point so :- [14:47:29] :-P [14:48:10] Let's get baseline working first; we'll worry about the rest later. [14:49:45] /usr/bin/rsync from a local filesystem to a local (i.e. nfs mounted) filesystem [14:49:48] so looks like you're ok [14:49:51] PROBLEM - DPKG on stat1003 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [14:49:52] (03PS2) 10Yuvipanda: labmon: Add labmon role to labmon1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/154854 [14:49:59] andrewbogott: Coren ^ [14:50:20] JohnLewis, MatmaRex: ping for SWAT in 10 minutes [14:50:28] !log starting stat1003 upgrade to trusty [14:50:33] Logged the message, Master [14:50:38] anomie: kk [14:50:48] i'm here [14:51:30] (03CR) 10coren: [C: 031] "Straightforward enough." [operations/puppet] - 10https://gerrit.wikimedia.org/r/154854 (owner: 10Yuvipanda) [14:51:45] YuviPanda: +1'ed, but no merge because dependencies. [14:51:50] RECOVERY - DPKG on stat1003 is OK: All packages OK [14:51:50] Coren: right. [14:51:54] Coren: I'll bug andrewbogott [14:51:58] andrewbogott: merge both? [14:52:25] (03CR) 10Andrew Bogott: [C: 032] labmon: Add labmon role to labmon1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/154854 (owner: 10Yuvipanda) [14:52:35] andrewbogott: dependency :) [14:52:55] yeah, can you set cwd in the graphite patch? [14:53:21] andrewbogott: no need, since I use the full path instead [14:53:41] andrewbogott: and nothing else is relative to current dir there, from what I could see [14:54:03] that is… not necessarily true? Unless you know for sure that the script and all of its dependencies nothing looks at cwd [14:55:08] andrewbogott: I do, since I also looked at the script. It refers to all python modules via global imports [14:55:25] apergos: +2 the thing? [14:55:26] ok [14:55:35] +2 ALL THE THINGS! [14:55:38] (03CR) 10Andrew Bogott: [C: 032] graphite: Fixes to work with trusty [operations/puppet] - 10https://gerrit.wikimedia.org/r/154712 (owner: 10Yuvipanda) [14:55:39] hahaha [14:55:54] (03PS3) 10Andrew Bogott: labmon: Add labmon role to labmon1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/154854 (owner: 10Yuvipanda) [14:56:17]