[00:00:00] or maybe it needs a mailman training in a hangout [00:02:31] lol [00:02:40] yeah I think people just need to be made aware of its existence maybe [00:03:29] it's a common WMF problem not specific to mailman [00:03:41] we have stuff on wiki, people will say but there you dont see it [00:03:44] then we make a new list [00:04:14] then others say too many lists, then we make an IRC channel.. and so on [00:04:35] i still say "list of list admins" feels natural though [00:08:03] Thehelpfulone, maybe we can put a link to the wikitech docs right in the admin interface? [00:08:09] and the admindb interface too [00:08:50] good one [00:09:45] !log ori Started scap: SWAT: cherry picks for TMH and Echo [00:09:51] Logged the message, Master [00:10:37] (03PS15) 10Dzahn: turn RT from misc/* into puppet module [operations/puppet] - 10https://gerrit.wikimedia.org/r/116064 [00:11:01] what should we do with wiki-mail.wm.o ? [00:11:06] move to an eqiad IP? [00:12:21] legoktm: stand by to confirm things are ok post-deployment, please [00:12:26] i see there's also a wiki-mail-eqiad... [00:12:54] ori: aye aye captain [00:14:47] Thehelpfulone: or maybe you want to turn "mailman" into that meta list? [00:16:20] Invalid resource type apache_module, getting closer:) [00:16:30] mutante: hmm if people then email that list then every list admin would get spammed [00:17:08] jeremyb: yes that sounds like a good idea - I also wanted to test out making the new list design default, IIRC 2 years ago or so we had a mailman instance on labs where we were testing things? [00:23:52] Thehelpfulone: it's just HTML, right, you could also just test it on a list called "test" [00:24:04] to save you the labs work in this case [00:24:16] since it's really just copy/pasting HTML [00:24:23] !log ori Finished scap: SWAT: cherry picks for TMH and Echo (duration: 14m 38s) [00:24:29] Logged the message, Master [00:24:36] legoktm: ^ [00:24:37] * legoktm tests [00:24:38] and it wont be puppetized [00:25:08] Echo looks good [00:25:25] TMH looks better! [00:25:27] thanks ori [00:25:44] mutante: so https://lists.wikimedia.org/mailman/listinfo/wikitech-announce actually does already work [00:25:58] Thehelpfulone: looks way nicer than default :) [00:26:29] there's only one tweak that needs to be made per list IIRC - else the rest of the code is generic [00:27:19] https://github.com/quimgil/mailman-templates/blob/master/mailman-template.html [00:27:36] Hmm. TMH still looks slightly wrong to me, but probably there's caching somewhere there [00:27:42] I think it's just enabling the multilingual selector if you want it [00:28:30] Thehelpfulone: so it's just a matter of telling all admins about it again [00:28:32] bawolff: I'm now seeing 404s instead of 403s on mw.o, which is what was expected right? [00:29:12] (404 because of the bad namespace) [00:29:13] yep - I mean I just copied it straight onto https://lists.wikimedia.org/mailman/listinfo/test [00:29:27] the only thing this does that some may not like is remove the password box [00:29:43] legoktm: Hmm, it fixed the use index.php [00:29:56] but there's still + signs in the url for some reason [00:30:05] but I'm sure I could tweak the code and then just add that back in, but comment it out [00:30:06] Thehelpfulone: lol, i'm admin of test? dang :) [00:30:11] Well that fixes your use case anyways, the other thing is less critical [00:30:13] you created it IIRC :p [00:30:17] yeah [00:30:42] Thehelpfulone: sounds like we had the entire discussion 2 years ago :) [00:31:47] Thehelpfulone: you could create a mass mail without even making a new list, you know they are all reachable as foo-owner@lists [00:32:01] and you have the list of lists in wiki table [00:32:24] yeah well I was thinking of asking you [or someone else] to run that report again so I can check through all the lists [00:32:48] I'll poke someone tomorrow about it (should I create another RT ticket, or leave a comment on the old one to re-open it)? [00:32:54] reopen the old ticket? [00:32:59] should be easiest [00:33:05] yep [00:35:21] (03PS1) 10Jeremyb: fix cert mismatch on mail.wikipedia.org [operations/dns] - 10https://gerrit.wikimedia.org/r/154222 (https://bugzilla.wikimedia.org/44731) [00:35:34] (03PS1) 10Jeremyb: fix cert mismatch on mail.wikipedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/154223 (https://bugzilla.wikimedia.org/44731) [00:35:54] hmm, actually the + signs are fine now since its part of the url parameter [00:36:31] (03PS16) 10Dzahn: turn RT from misc/* into puppet module [operations/puppet] - 10https://gerrit.wikimedia.org/r/116064 [00:38:52] (03CR) 10JanZerebecki: "Certificate revocation works as usual, because STS (as currently used) doesn't concern itself which certificate is used, just that one tha" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148289 (https://bugzilla.wikimedia.org/38516) (owner: 10Dzahn) [00:39:03] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 20:37:55 UTC [00:39:32] Thehelpfulone, do you want to review? :) [00:49:02] (03PS2) 10Jeremyb: fix cert mismatch on mail.wikipedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/154223 (https://bugzilla.wikimedia.org/44731) [00:49:50] (03CR) 10Jeremyb: "PS2: on second thought, now that I3511f4b0d0185d1e4d35166c13f2104c7805f737 is done, we might as well force HTTPS here too instead of doing" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154223 (https://bugzilla.wikimedia.org/44731) (owner: 10Jeremyb) [00:51:00] did we lose the refreshDomainRedirects check in jenkins on the way from apache-config -> puppet repo? [00:52:24] ori: Anything about the testwikidata Specia:Version thing? [00:52:31] (just saw your mail to the list) [00:52:57] hoo|busy: doh, i forgot [00:53:23] hoo|busy: how urgent is it? any chance i could look tomorrow? it's almost 4 AM here and i am purportedly on vacation [00:53:39] So am I :D [00:53:53] Yeah, that's ok, I'm not going to look into anything anyway today [00:53:57] flying tomorrow [00:54:03] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 22:53:31 UTC [00:54:04] and should probably already be sleeping [00:54:29] thanks :) i will definitely look tomorrow. [00:55:20] ori: https://github.com/facebook/hhvm/pull/3404 [00:55:42] would fix most of our test failures on wikibase and then want to see what else, if anything is still broken [00:56:05] aude: :) \o/ thanks! i'll nudge upstream if it isn't merged by monday [00:56:07] can look tomomrrow [00:56:14] * aude can wait [00:56:40] thank you :) i appreciate it. i'll work on aggregating logs as well. [00:56:51] thanks [00:56:58] * ori sleeps [00:58:32] * aude too [01:11:55] labs: channel 0: open failed: administratively prohibited: open failed [01:13:03] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Fri Aug 15 01:12:55 UTC 2014 [01:13:08] no, taking it back, wrong instance name [01:13:17] heh [01:13:52] (03CR) 10Dzahn: [C: 032] wikistats - use apache::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/153828 (owner: 10Dzahn) [01:14:24] (03CR) 10Chmarkine: [C: 031] StrictTransportSecurity for lists.wm.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/145500 (https://bugzilla.wikimedia.org/38516) (owner: 10Dzahn) [01:17:34] RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge. [01:18:51] nslcd : error writing to client: Broken pipe [01:18:52] hrmm [01:19:00] jeremyb: sorry for cross-posting :) [01:19:09] heh I think mutante called that patch earlier [01:19:12] (STS) [01:19:18] mutante, it's ok :-) [01:19:45] hah, yea, they were only making it https-only for STS [01:20:03] and the nslcd stuff is labs [01:21:15] "We should probably wait a week after that before enabling STS." in the comments though [01:21:39] true [01:22:46] what's the concern with https://gerrit.wikimedia.org/r/148289 ? [01:26:41] jeremyb: well it's a very long TTL there [01:26:54] someone needs to be sure they really understand the implications before setting it [01:27:16] if 4 months later we decide to do something that doesn't work with that, it could break a bunch of users. [01:27:17] we did it for Bugzilla itself so far [01:27:36] I can't claim to understand all the possible down-the-road interactions [01:43:44] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Epic puppet fail [02:02:43] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [02:28:49] heya /dev/vda1 is full on deploy-bastion.eqiad.wmflabs so rsync is failing, bug 69590. Nobody's around in #wikimedia-labs ... [02:29:05] *deployment-bastion [02:34:24] !log LocalisationUpdate completed (1.24wmf16) at 2014-08-15 02:33:21+00:00 [02:34:32] Logged the message, Master [02:34:44] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [02:40:03] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 20:37:55 UTC [03:03:23] (03PS2) 10Jeremyb: StrictTransportSecurity for lists.wm.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/145500 (https://bugzilla.wikimedia.org/38516) (owner: 10Dzahn) [03:04:52] !log LocalisationUpdate completed (1.24wmf17) at 2014-08-15 03:03:49+00:00 [03:04:59] Logged the message, Master [03:05:01] (03CR) 10Jeremyb: "rebased" [operations/puppet] - 10https://gerrit.wikimedia.org/r/145500 (https://bugzilla.wikimedia.org/38516) (owner: 10Dzahn) [03:51:29] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Aug 15 03:50:23 UTC 2014 (duration 50m 22s) [03:51:34] Logged the message, Master [04:41:03] PROBLEM - Puppet freshness on db1010 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 02:40:32 UTC [04:41:03] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 20:37:55 UTC [04:55:23] any admins around? i need a perm reset on stat1002 - synced files incorrectly :( [04:55:43] stat1002 /a/zerosms should have the same perms as /a/zerosms [04:56:50] sorry, on server stat1002, /a/zerosms should have the same perms as /a/zero-sms [04:56:58] yurikR: /a/zero-sms ? [04:57:03] springle, yep [04:57:04] ah [04:57:05] thx! [04:57:57] done [04:58:32] thanks!! [04:58:36] got it [05:20:33] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Fri Aug 15 05:20:23 UTC 2014 [05:33:55] (03PS1) 10Tim Starling: Remove bits.wikimedia.org/robots.txt [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154234 [06:28:33] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:44] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:46] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:13] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:23] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:43] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:43] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:03] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: Puppet has 1 failures [06:42:03] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 20:37:55 UTC [06:45:03] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:45:33] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:45:43] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:46:13] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:46:23] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:46:44] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:46:44] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:46:47] is there a way in puppet to force newer repository? stat1002 seems to be using a very old ubuntu repo, so when i added "python-requests" package, a 3 year-old package was installed (0.8.2). http://packages.ubuntu.com/search?keywords=requests+python&searchon=names&suite=all§ion=all [06:46:53] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:47:43] yurikR, well is it a precise box? [06:47:52] you could use a trusty box instead [06:47:53] :) [06:48:07] you may also want to do: [06:48:16] apt-cache policy python-requests [06:48:21] jeremyb, no idea realy, but i have to use stat1002 as that's the only secure statistics server :) [06:48:39] well do [06:48:39] lsb_release -a [06:48:43] what does that say? [06:48:48] plus i'm root. [06:49:36] yurik@stat1002:~/zero-sms/scripts$ apt-cache policy python-requests [06:49:36] python-requests: [06:49:36] Installed: 0.8.2-1 [06:49:36] Candidate: 0.8.2-1 [06:49:36] Version table: [06:49:36] *** 0.8.2-1 0 [06:49:39] 500 http://ubuntu.wikimedia.org/ubuntu/ precise/universe amd64 Packages [06:49:41] 100 /var/lib/dpkg/status [06:49:42] yurik@stat1002:~/zero-sms/scripts$ lsb_release -a [06:49:45] No LSB modules are available. [06:49:47] Distributor ID: Ubuntu [06:49:49] Description: Ubuntu 12.04.2 LTS [06:49:51] Release: 12.04 [06:49:53] Codename: precise [06:50:14] right. so it's a precise box and doesn't seem to have trusty in apt sources [06:50:38] i have no idea offhand if any WMF boxes have a hybrid of distro releases [06:50:46] e.g. with apt pins [06:51:02] i wouldn't want to break the box - it has a lot of stats running on it :) [06:51:53] i would assume it can be configured somehow in puppet/manifests/misc/statistics.pp [06:52:07] the answer might be to backport a newer python-requests into the wikimedia section of precise [06:52:56] jeremyb, any pointers on that? [06:53:05] so you don't need extra apt sources, you just have the newer version on brewster (or current equivalent) for all boxen to use if they install that package on precise [06:53:07] never done any packaging [06:53:07] yeah [06:53:32] https://wikitech.wikimedia.org/wiki/Backport_packages [06:56:24] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 0.077 second response time [06:56:28] so, maybe also need to backport python-urllib3 [06:57:00] sigh :( [06:57:21] might have to rewrite my script to use ancient requests library :( [06:57:48] it's not lucid... [06:57:57] * jeremyb runs away [06:58:00] :D [06:58:42] i might try to taunt hashar tomorrow, since he's the one who wrote the guide :D [07:04:24] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.005 second response time [07:04:43] PROBLEM - Number of mediawiki jobs running on tungsten is CRITICAL: CRITICAL: Anomaly detected: 32 data above and 0 below the confidence bounds [07:04:43] PROBLEM - Number of mediawiki jobs queued on tungsten is CRITICAL: CRITICAL: Anomaly detected: 32 data above and 0 below the confidence bounds [07:05:03] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 7.14% of data above the critical threshold [500.0] [07:16:03] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [07:55:03] (03PS1) 10Calak: Add namespace alias on ckbwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154239 (https://bugzilla.wikimedia.org/69594) [08:03:53] PROBLEM - puppet last run on mw1040 is CRITICAL: CRITICAL: Puppet has 1 failures [08:20:54] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [08:35:03] PROBLEM - check_mysql on lutetium is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 833 [08:40:04] RECOVERY - check_mysql on lutetium is OK: Uptime: 4365213 Threads: 2 Questions: 23246251 Slow queries: 34354 Opens: 31292 Flush tables: 2 Open tables: 64 Queries per second avg: 5.325 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [08:43:03] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 20:37:55 UTC [09:08:29] (03CR) 10QChris: [C: 031] "> QChris: yea, so far i expected we delete them manually and" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153832 (owner: 10Dzahn) [09:12:25] andrewbogott_afk: what do you think about the last PS on https://gerrit.wikimedia.org/r/#/c/153584/ ? [09:18:57] (03PS4) 10Nuria: Make hourly backup keep around known-good full backups in case of issues [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/153568 (https://bugzilla.wikimedia.org/68731) (owner: 10QChris) [09:47:31] (03PS1) 10Filippo Giunchedi: hhvm: add error logging to file [operations/puppet] - 10https://gerrit.wikimedia.org/r/154243 [09:48:58] ori: apparently we don't log hhvm anywhere, I got https://gerrit.wikimedia.org/r/#/c/154243/ out to get things going but let me know if I'm off base or there were other thoughts [10:03:10] (03CR) 10QChris: [C: 031] gerrit - use apache::site (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/153849 (owner: 10Dzahn) [10:05:17] godog: fcgi-error is the log to look for? [10:11:04] aude: not sure what you mean, look for where? [10:11:11] hhvm errors? [10:11:47] like why https://test.wikidata.org/wiki/Special:Version is unavailable [10:11:52] test.wikidata is on hhvm? [10:12:18] and http://wikidata.beta.wmflabs.org/wiki/Q2558 unavailable [10:12:33] don't know, but that patch above isn't yet in puppet or anywhere so it won't log there, no [10:13:16] if we go with that then yes that would be the file eventually [10:13:21] ok [10:15:12] (03CR) 10QChris: "> Not needed, but I'm not sure what to do with it. Its a good script!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/151095 (owner: 10Ottomata) [10:32:49] (03CR) 10Phuedx: "^ Erm… @Reedy?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/151639 (https://bugzilla.wikimedia.org/69103) (owner: 10Phuedx) [10:39:33] (03CR) 10JanZerebecki: gerrit - use apache::site (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/153849 (owner: 10Dzahn) [10:41:15] PROBLEM - Puppet freshness on analytics1029 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 10:38:25 UTC [10:43:15] PROBLEM - Puppet freshness on analytics1029 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 10:38:25 UTC [10:43:15] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 20:37:55 UTC [10:45:15] PROBLEM - Puppet freshness on analytics1029 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 10:38:25 UTC [10:47:15] PROBLEM - Puppet freshness on analytics1029 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 10:38:25 UTC [10:49:15] PROBLEM - Puppet freshness on analytics1029 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 10:38:25 UTC [10:51:15] PROBLEM - Puppet freshness on analytics1029 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 10:38:25 UTC [10:53:15] PROBLEM - Puppet freshness on analytics1029 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 10:38:25 UTC [10:55:15] PROBLEM - Puppet freshness on analytics1029 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 10:38:25 UTC [10:57:15] PROBLEM - Puppet freshness on analytics1029 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 10:38:25 UTC [10:58:16] RECOVERY - Puppet freshness on analytics1029 is OK: puppet ran at Fri Aug 15 10:58:13 UTC 2014 [11:10:40] (03CR) 10JanZerebecki: [C: 031] stats.wm.org - use apache::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/153832 (owner: 10Dzahn) [11:11:31] (03CR) 10JanZerebecki: [C: 031] StrictTransportSecurity for lists.wm.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/145500 (https://bugzilla.wikimedia.org/38516) (owner: 10Dzahn) [11:27:46] (03PS1) 10Ori.livneh: HHVM: use syslog; route to fluorine [operations/puppet] - 10https://gerrit.wikimedia.org/r/154253 [11:29:34] godog: ^ [11:35:04] (03CR) 10Filippo Giunchedi: [C: 031] HHVM: use syslog; route to fluorine [operations/puppet] - 10https://gerrit.wikimedia.org/r/154253 (owner: 10Ori.livneh) [11:35:14] ori: LGTM, what happens if we turn on sth more verbose btw? [11:39:26] what do you mean? [11:40:34] back in ~30 [11:42:07] ori: temporarily e.g. a very verbose logging level for testing or debugging [11:56:27] (03CR) 10Ori.livneh: [C: 032] HHVM: use syslog; route to fluorine [operations/puppet] - 10https://gerrit.wikimedia.org/r/154253 (owner: 10Ori.livneh) [11:59:07] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [12:19:15] (03Abandoned) 10Filippo Giunchedi: hhvm: add error logging to file [operations/puppet] - 10https://gerrit.wikimedia.org/r/154243 (owner: 10Filippo Giunchedi) [12:43:57] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 20:37:55 UTC [13:22:39] (03PS1) 10Springle: Need REPEATABLE-READ so that specific tables can remain InnoDB and row-based binlogging still works. [operations/puppet] - 10https://gerrit.wikimedia.org/r/154264 [13:25:01] (03CR) 10Springle: [C: 032] Need REPEATABLE-READ so that specific tables can remain InnoDB and row-based binlogging still works. [operations/puppet] - 10https://gerrit.wikimedia.org/r/154264 (owner: 10Springle) [13:33:50] !log disabling puppet on mw1017 to test rsyslog config [13:33:54] Logged the message, Master [14:00:57] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 12:00:02 UTC [14:00:57] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Fri Aug 15 14:00:53 UTC 2014 [14:17:43] (03PS1) 10Ori.livneh: MediaWiki/HHVM: log fatals to fluorine:/a/mw-log/hhvm-fatal.log [operations/puppet] - 10https://gerrit.wikimedia.org/r/154271 [14:32:35] (03CR) 10Filippo Giunchedi: MediaWiki/HHVM: log fatals to fluorine:/a/mw-log/hhvm-fatal.log (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/154271 (owner: 10Ori.livneh) [14:41:18] (03PS2) 10Ori.livneh: MediaWiki/HHVM: log fatals to fluorine:/a/mw-log/hhvm-fatal.log [operations/puppet] - 10https://gerrit.wikimedia.org/r/154271 [14:44:57] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 20:37:55 UTC [14:47:06] (03CR) 10Filippo Giunchedi: [C: 031] MediaWiki/HHVM: log fatals to fluorine:/a/mw-log/hhvm-fatal.log [operations/puppet] - 10https://gerrit.wikimedia.org/r/154271 (owner: 10Ori.livneh) [14:47:15] ori: ^ [14:52:09] (03CR) 10Andrew Bogott: "Can I get some context for this?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153975 (owner: 10Dzahn) [14:59:10] (03PS3) 10Ori.livneh: MediaWiki/HHVM: log fatals to fluorine:/a/mw-log/hhvm-fatal.log [operations/puppet] - 10https://gerrit.wikimedia.org/r/154271 [14:59:25] (03CR) 10Ori.livneh: [C: 032 V: 032] MediaWiki/HHVM: log fatals to fluorine:/a/mw-log/hhvm-fatal.log [operations/puppet] - 10https://gerrit.wikimedia.org/r/154271 (owner: 10Ori.livneh) [15:00:03] !log re-enabled puppet on mw1017 [15:00:08] Logged the message, Master [15:08:48] (03PS1) 10Ori.livneh: Fix update-alternatives for HHVM [operations/puppet] - 10https://gerrit.wikimedia.org/r/154275 [15:28:34] (03CR) 10Greg Grossmeier: "Reedy: feel free to do this one when/if it makes sense." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/147922 (https://bugzilla.wikimedia.org/67287) (owner: 10Bartosz Dziewoński) [15:30:11] greg-g: test.wikidata is spamming the exception logs about a missing column and many statement values are displayed as 'invalid' [15:30:20] we would like to update with small fix for these https://gerrit.wikimedia.org/r/#/c/154274/ [15:30:36] so users have a few days to test [15:30:52] right now, test.wikidata is rather not editable [15:31:37] ori: would that be ok with you? [15:32:23] basically, our feature flag for not using the column was broken [15:32:31] aude: are the changes only those needed to fix the db error and statement values? [15:32:34] yes [15:32:37] * greg-g nods [15:32:40] small as possible [15:32:47] * greg-g nods [15:33:15] is it running on beta? [15:33:22] good question! [15:33:26] beta is broken for us [15:33:29] heh [15:33:33] * greg-g sighs [15:33:37] bug? [15:33:39] due to hiphop but tested locally [15:33:58] oooh, i get a page today http://wikidata.beta.wmflabs.org/wiki/Q2558 [15:35:00] i assume beta has the column though, so wouldn't have caught this [15:35:04] aude: btw, did you know that wikidatabeta is calling out to wikidata.org? [15:35:12] right, update.php runs and all that [15:35:14] hmmmm [15:35:23] just for a gadget and something [15:35:26] just fy [15:35:27] i [15:35:28] oh [15:36:00] anywho, sure, backport :/ [15:36:12] http://wikidata.beta.wmflabs.org/wiki/MediaWiki:Common.js [15:36:35] we wanted to test the gadgets, so should be ok hopefully [15:37:11] would be nice to have an extra day between test.wikidata and firday [15:37:17] friday* for these issues [15:37:44] "it's just testwiki" [15:38:02] yes, but it's valuable testing time [15:38:28] yes yes, do it, fix it [15:38:31] ok [15:42:07] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 2 below the confidence bounds [15:46:08] * aude waits [15:48:38] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Epic puppet fail [15:50:24] !log aude Synchronized php-1.24wmf17/extensions/Wikidata: Fix database error and snak value display on test wikidata (duration: 00m 09s) [15:50:30] Logged the message, Master [15:50:31] * aude verifies [15:52:58] (03CR) 10Chmarkine: [C: 031] StrictTransportSecurity for lists.wm.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/145500 (https://bugzilla.wikimedia.org/38516) (owner: 10Dzahn) [15:53:06] !log aude Synchronized php-1.24wmf17/extensions/Wikidata: Update test.wikidata (duration: 00m 07s) [15:53:10] ok, for real [15:53:11] Logged the message, Master [15:53:37] looks good! [15:57:57] aude: :) [16:02:13] aude: was that expected to fix https://test.wikidata.org/wiki/Special:Version ? [16:04:42] aude: now i see: [16:04:43] 2014-08-15 15:53:56 mw1017 testwikidatawiki: [34bd4407] /w/api.php?action=wbparsevalue&format=json&parser=monolingualtext&values=meo&options=%7B%22valuelang%22%3A%22%22%7D Exception from line 46 of /usr/local/apache/common-local/php-1.24wmf17/extensions/Wikidata/vendor/data-values/common/src/DataValues/MonolingualTextValue.php: Can only construct MonolingualTextValue with a language code of non-zero length [16:07:47] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [16:45:57] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 20:37:55 UTC [16:46:17] (03CR) 10Nuria: [C: 031] Make hourly backup keep around known-good full backups in case of issues [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/153568 (https://bugzilla.wikimedia.org/68731) (owner: 10QChris) [16:52:54] (03CR) 10Ottomata: [C: 032 V: 032] Make hourly backup keep around known-good full backups in case of issues [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/153568 (https://bugzilla.wikimedia.org/68731) (owner: 10QChris) [16:55:30] (03PS4) 10Nuria: Force redis dump before backing up [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/153395 (https://bugzilla.wikimedia.org/68731) (owner: 10QChris) [16:56:44] (03CR) 10Filippo Giunchedi: [C: 04-1] "it'd be better to use update-alternative's --query since it is designed for that, I can see how at least the config part could be useful i" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154275 (owner: 10Ori.livneh) [17:07:19] milimetric, the mysql pool (in wikimetrics at least) is hared for all celery processes seems like as it does not go beyond 32 [17:07:51] I just lied, show it going to 35 [17:08:10] :) [17:08:20] you still might not have lied nuria [17:08:23] let's talk in -analytics [17:08:40] arg! [17:12:28] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 1.118 second response time [17:18:23] (03CR) 10CSteipp: [C: 031] "You're right, I thought HSTS did some pinning, which it doesn't. So yeah, no problem deploying this now." [operations/puppet] - 10https://gerrit.wikimedia.org/r/148289 (https://bugzilla.wikimedia.org/38516) (owner: 10Dzahn) [17:24:12] ori: we know about that one and have a patch for swat on monday for that [17:33:38] (03CR) 10Dzahn: "meanwhile i moved the config from apache/files to templates, so can't be merged like this. fixing" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148289 (https://bugzilla.wikimedia.org/38516) (owner: 10Dzahn) [17:35:17] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [17:35:28] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.004 second response time [17:35:35] (03PS3) 10Dzahn: OTRS - raise max-age for STS to 1 year [operations/puppet] - 10https://gerrit.wikimedia.org/r/148289 (https://bugzilla.wikimedia.org/38516) [17:35:49] PROBLEM - Number of mediawiki jobs queued on tungsten is CRITICAL: CRITICAL: Anomaly detected: 33 data above and 0 below the confidence bounds [17:35:49] PROBLEM - Number of mediawiki jobs running on tungsten is CRITICAL: CRITICAL: Anomaly detected: 33 data above and 0 below the confidence bounds [17:42:57] (03CR) 10Nuria: [C: 031] "Tested on labs development instance. a 2.5 G file of redis takes about a second to dump, so an interval of 15 secs should be more than suf" [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/153395 (https://bugzilla.wikimedia.org/68731) (owner: 10QChris) [17:45:28] (03CR) 10Ottomata: [C: 032 V: 032] Force redis dump before backing up [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/153395 (https://bugzilla.wikimedia.org/68731) (owner: 10QChris) [17:49:22] (03PS1) 10Nuria: Bumping up wikimetrics module. New and improved backup [operations/puppet] - 10https://gerrit.wikimedia.org/r/154290 [17:52:53] (03PS2) 10Ottomata: Bumping up wikimetrics module. New and improved backup [operations/puppet] - 10https://gerrit.wikimedia.org/r/154290 (owner: 10Nuria) [17:52:58] (03CR) 10Ottomata: [C: 032 V: 032] Bumping up wikimetrics module. New and improved backup [operations/puppet] - 10https://gerrit.wikimedia.org/r/154290 (owner: 10Nuria) [17:53:30] (03PS1) 10Aude: Enable use of epp_redirect_target column on test.wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154292 [17:56:01] (03CR) 10Aude: [C: 032] "per lydia who has asked greg :)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154292 (owner: 10Aude) [17:56:24] (03Merged) 10jenkins-bot: Enable use of epp_redirect_target column on test.wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154292 (owner: 10Aude) [17:58:02] !log aude Synchronized wmf-config/Wikibase.php: Enable redirects on test.wikidata (duration: 00m 07s) [17:58:08] Logged the message, Master [17:58:13] * aude hides :) [17:58:42] (03PS1) 10Ottomata: cdh::hadoop::users now supports multiple groups [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/154294 [17:58:44] (03CR) 10jenkins-bot: [V: 04-1] cdh::hadoop::users now supports multiple groups [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/154294 (owner: 10Ottomata) [17:58:49] (03PS2) 10Ottomata: cdh::hadoop::users now supports multiple groups [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/154294 [17:58:51] (03CR) 10jenkins-bot: [V: 04-1] cdh::hadoop::users now supports multiple groups [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/154294 (owner: 10Ottomata) [17:59:20] (03PS3) 10Ottomata: cdh::hadoop::users now supports multiple groups [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/154294 [18:00:25] (03PS4) 10Ottomata: cdh::hadoop::users now supports multiple groups [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/154294 [18:01:01] (03CR) 10Ottomata: [C: 032 V: 032] cdh::hadoop::users now supports multiple groups [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/154294 (owner: 10Ottomata) [18:03:32] (03PS1) 10Ottomata: Manage automatically HDFS user directories in multiple groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/154295 [18:03:40] (03PS2) 10Ottomata: Manage automatically HDFS user directories in multiple groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/154295 [18:03:42] (03CR) 10jenkins-bot: [V: 04-1] Manage automatically HDFS user directories in multiple groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/154295 (owner: 10Ottomata) [18:03:55] (03CR) 10Ottomata: [C: 032 V: 032] Manage automatically HDFS user directories in multiple groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/154295 (owner: 10Ottomata) [18:05:43] (03PS1) 10Ottomata: Fix group parameter on cdh::hadoop::users class [operations/puppet] - 10https://gerrit.wikimedia.org/r/154297 [18:05:50] (03CR) 10jenkins-bot: [V: 04-1] Fix group parameter on cdh::hadoop::users class [operations/puppet] - 10https://gerrit.wikimedia.org/r/154297 (owner: 10Ottomata) [18:05:52] (03PS2) 10Ottomata: Fix group parameter on cdh::hadoop::users class [operations/puppet] - 10https://gerrit.wikimedia.org/r/154297 [18:06:09] (03CR) 10Ottomata: [C: 032 V: 032] Fix group parameter on cdh::hadoop::users class [operations/puppet] - 10https://gerrit.wikimedia.org/r/154297 (owner: 10Ottomata) [18:06:57] PROBLEM - puppet last run on analytics1010 is CRITICAL: CRITICAL: Epic puppet fail [18:08:28] (03CR) 10Dzahn: [C: 032] OTRS - raise max-age for STS to 1 year [operations/puppet] - 10https://gerrit.wikimedia.org/r/148289 (https://bugzilla.wikimedia.org/38516) (owner: 10Dzahn) [18:08:58] RECOVERY - puppet last run on analytics1010 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [18:18:45] (03CR) 10Dzahn: "" This server supports HTTP Strict Transport Security with long duration." :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/148289 (https://bugzilla.wikimedia.org/38516) (owner: 10Dzahn) [18:24:34] (03PS1) 10Manybubbles: Make permanent some elasticsearch settings [operations/puppet] - 10https://gerrit.wikimedia.org/r/154301 [18:25:21] (03CR) 10jenkins-bot: [V: 04-1] Make permanent some elasticsearch settings [operations/puppet] - 10https://gerrit.wikimedia.org/r/154301 (owner: 10Manybubbles) [18:26:01] (03CR) 10Manybubbles: "I no longer have an environment in which I can easily test puppet code so I wrote this blind. The configuration is already applied to the" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154301 (owner: 10Manybubbles) [18:26:56] (03PS2) 10Manybubbles: Make permanent some elasticsearch settings [operations/puppet] - 10https://gerrit.wikimedia.org/r/154301 [18:40:55] (03CR) 10Dzahn: "AndrewBogott, background is this: In the past we changed the SSL settings to support newer ciphers[1] and had a separate change for each s" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153975 (owner: 10Dzahn) [18:44:37] (03CR) 10Dzahn: "part of this series, topic branch "ssl-ciphersuite":" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153975 (owner: 10Dzahn) [18:46:57] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 20:37:55 UTC [18:51:48] (03PS5) 10Dzahn: stats.wm.org - use apache::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/153832 [18:52:55] (03CR) 10Dzahn: [C: 032] stats.wm.org - use apache::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/153832 (owner: 10Dzahn) [18:55:45] ottomata1: ^ doing that, and seeing some unrelated things on stat1001, rsync denied on module a from stat1003.wikimedia.org [18:59:57] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 16:59:44 UTC [19:00:06] (03CR) 10Andrew Bogott: [C: 032] wikitech - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153975 (owner: 10Dzahn) [19:01:22] (03CR) 10Dzahn: "SUCCESS:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153832 (owner: 10Dzahn) [19:03:10] ottomata1: community-analytics.wm.org ? is that an upcoming thing or an old thing? because it's not in DNS but there is apache conf [19:10:39] mutante: old thing [19:10:42] ask dartar [19:10:58] ottomata: ok, will do, thx [19:20:52] (03CR) 10Ottomata: [C: 031] "Let's merge it on Monday, JUST IN CASE!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154301 (owner: 10Manybubbles) [19:27:30] (03PS1) 10Manybubbles: Enable a Cirrus optimization on group0 wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154307 [19:40:07] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Fri Aug 15 19:39:57 UTC 2014 [19:44:48] (03PS1) 10Dzahn: metrics.wikimedia.org - use apache::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/154315 [19:57:05] (03CR) 10Ottomata: "Or, maybe we should merge this and then remove it, to have the history in git instead of just in gerrit." [operations/puppet] - 10https://gerrit.wikimedia.org/r/151095 (owner: 10Ottomata) [20:01:23] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 19:58:17 UTC [20:03:23] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 19:58:17 UTC [20:05:23] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 19:58:17 UTC [20:07:23] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 19:58:17 UTC [20:09:23] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 19:58:17 UTC [20:10:47] (03PS1) 10Dzahn: stats.wm.org - revert port config via apache::conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/154319 [20:11:23] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 19:58:17 UTC [20:13:23] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 19:58:17 UTC [20:15:05] (03CR) 10Dzahn: [C: 032] "for now, so that it cant break on restarts, then compare to wikitech" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154319 (owner: 10Dzahn) [20:15:23] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 19:58:17 UTC [20:17:23] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 19:58:17 UTC [20:18:13] RECOVERY - Puppet freshness on lvs4001 is OK: puppet ran at Fri Aug 15 20:18:07 UTC 2014 [20:20:23] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Fri 15 Aug 2014 20:18:07 UTC [20:24:56] (03CR) 10Dzahn: "Ori, how should we fix? it's similar to what you did in Change-Id: Ib2dcd16c81ebd" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154319 (owner: 10Dzahn) [20:33:18] (03CR) 10Ottomata: [C: 031] metrics.wikimedia.org - use apache::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/154315 (owner: 10Dzahn) [20:38:02] (03CR) 10Hashar: "The change cause two issues:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153807 (owner: 10Ori.livneh) [20:38:38] RECOVERY - Puppet freshness on lvs4001 is OK: puppet ran at Fri Aug 15 20:38:34 UTC 2014 [20:43:40] (03PS1) 10Ori.livneh: alternatives::config -> alternatives::select [operations/puppet] - 10https://gerrit.wikimedia.org/r/154328 [20:45:30] (03PS1) 10Hashar: Revert "mediawiki: create common-local directory" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154329 (https://bugzilla.wikimedia.org/69590) [20:46:51] (03PS2) 10Hashar: Revert "mediawiki: create common-local directory" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154329 (https://bugzilla.wikimedia.org/69590) [20:47:05] ^^do not merge :-D [20:47:58] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 20:37:55 UTC [20:50:23] godog: someone should probably ack that ^^ [20:50:37] or fix it, but I assume ack since it's it since yesterday [20:51:30] (03CR) 10Hashar: [C: 04-1] "I agree with Ori that our paths and symbolic links (on beta) are totally crap and cause my headaches each time I have to deal with them (s" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154329 (https://bugzilla.wikimedia.org/69590) (owner: 10Hashar) [20:52:52] (03CR) 10Dzahn: [C: 031] racktables - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153973 (owner: 10Dzahn) [20:54:40] (03CR) 10Dzahn: "thanks! i'll update all the other commit messages with some copypasta" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153975 (owner: 10Dzahn) [20:59:07] !log kaldari Synchronized php-1.24wmf16/extensions/MobileFrontend/less: fixing iOS search bug (duration: 00m 05s) [20:59:11] Logged the message, Master [20:59:52] (03PS2) 10Dzahn: racktables - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153973 [21:00:10] (03PS3) 10Dzahn: racktables - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153973 [21:00:47] (03PS3) 10Hashar: Revert "mediawiki: create common-local directory" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154329 (https://bugzilla.wikimedia.org/69590) [21:00:49] (03PS2) 10Dzahn: subversion - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153991 [21:02:31] (03PS2) 10Dzahn: puppetmaster - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153986 [21:03:17] (03PS2) 10Dzahn: gitblit - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153985 [21:03:29] (03PS2) 10Dzahn: tendril - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153984 [21:03:50] (03PS2) 10Dzahn: ishmael - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153982 [21:04:13] (03PS3) 10Dzahn: etherpad - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153978 [21:04:23] (03PS2) 10Dzahn: rt - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153981 [21:07:45] mutante: I have seen your apache::site change for contint servers [21:08:00] mutante: I guess I will get it applied during european morning and see what happens :] [21:10:22] hashSpeleology: apache::site should be fine, the issues we had were with apache::conf replacing puppetized ports.conf [21:11:00] and thanks! (ran compiler this time) [21:11:33] someone wants a bigdelete [21:11:38] just so the ops people know [21:11:40] (-stewards) [21:12:31] Jasper_Deng: friday night isn't really a great time for that, communcation to ops-wise :/ [21:12:50] Jasper_Deng: not sure the right thing to do in this case, though, honestly [21:12:58] testwikidatawiki Wikibase\Lib\Store\Sql\SqlEntityInfoBuilder::getPageInfoForType 10.64.16.16 1054 Unknown column 'epp_redirect_target' in 'field list' (10.64.16.16) [21:13:02] (first time I've heard of that kind of problem) [21:13:07] Reedy: missing change? [21:13:28] (03PS2) 10Dzahn: stats.wm.org - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153977 [21:13:36] (03CR) 10jenkins-bot: [V: 04-1] stats.wm.org - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153977 (owner: 10Dzahn) [21:13:53] mutante: do you still need me for the apache::conf thing? [21:15:03] (03CR) 10Ori.livneh: [C: 04-2] "This won't get fixed until someone throws a tantrum." [operations/puppet] - 10https://gerrit.wikimedia.org/r/154329 (https://bugzilla.wikimedia.org/69590) (owner: 10Hashar) [21:15:24] ori: if you want to, remove file { '/etc/apache2/ports.conf': from misc/statistics.pp and then i'll use it as an example for others [21:15:39] but it's not urgent [21:15:53] i'd be happy to, give me a few [21:15:54] i partly reverted it because it would have broken stats.wm on apache restart [21:15:57] thanks [21:16:22] AaronS: the errors should have stopped a few hours ago [21:16:48] so what we usually want when puppetizing ports.conf is adding the NameVirtualHost for 443 [21:16:52] there still are exceptions about monolingual text, for which we'll have patch on monday [21:17:00] and then we overwrite the file from distro package [21:17:14] which then conflicts with the one in conf-available/ [21:17:32] and apache then "port already in use" [21:17:40] bbiab [21:17:41] (03CR) 10Ori.livneh: [C: 032] alternatives::config -> alternatives::select [operations/puppet] - 10https://gerrit.wikimedia.org/r/154328 (owner: 10Ori.livneh) [21:18:17] SELECT page_id,page_title,page_namespace,page_is_redirect,iwl_from,iwl_prefix,iwl_title FROM `iwlinks`,`page` WHERE (iwl_from = page_id) ORDER BY iwl_prefix,iwl_title,iwl_from LIMIT 11 [21:18:33] [rows] => 33347611 [21:18:35] [Extra] => Using temporary; Using filesort [21:19:41] Yeah no wonder [21:20:02] It can either use the index to sort by (prefix,title,from), or to resolve the join condition on from, but not both [21:20:52] what's 33 mil rows between friends [21:22:30] * AaronS waits for domas to troll [21:22:46] AaronS: Where did you get that query from? [21:22:58] ApiQueryIWBacklinks ... it's in dberror.log [21:26:56] Ouch [21:27:09] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [21:27:09] Maybe I shouldn't have been so quick to point out the query's flaws given that it's probably my code :D [21:27:24] well there should be few bogus entries in iwlinks right? [21:27:56] so one would think iwl_prefix_title_from would work, unless there is some selectivity issue [21:28:02] It seems to me like changing the ORDER BY would best [21:28:06] To order by from first [21:28:45] in fact, using FORCE INDEX makes it fast [21:28:54] maybe the index stats are just off [21:28:59] Forcing which index? [21:29:14] SELECT page_id,page_title,page_namespace,page_is_redirect,iwl_from,iwl_prefix,iwl_title FROM `iwlinks` FORCE INDEX(iwl_prefix_title_from),`page` WHERE (iwl_from = page_id) ORDER BY iwl_prefix,iwl_title,iwl_from LIMIT 11 ; [21:29:17] Oooh hold on, I see [21:29:20] The join is in the other direction [21:29:30] From iwl to page [21:32:58] (03PS2) 10Dzahn: noc.wm.org - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153976 [21:33:14] (03PS2) 10Dzahn: ganglia - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153972 [21:35:11] hashSpeleology: how/where do I put a secret configuration variable for an extension to deploy on beta labs? [21:35:18] springle: seeing a database error on beta labs during one of mobile's browser tests: table 'enwiki.echo_target_page' doesn't exist [21:35:39] marxarelli: beta labs only creates new db tables every 30 minutes iirc [21:35:48] marxarelli: oh, maybe they just didn't merge the needed create table to update.php? [21:35:48] ah, ok [21:35:58] or that [21:37:52] (03PS3) 10Dzahn: stats.wm.org - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153977 [21:37:59] greg-g: yeah, how does that work? does each extension maintain its own schema migrations? [21:39:16] (03PS4) 10Dzahn: stats.wm.org - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153977 [21:39:38] arg [21:39:44] greg-g: oh wait, i see... a directory called db_patches [21:40:09] (03PS5) 10Dzahn: stats.wm.org - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153977 [21:40:15] marxarelli: there's a LoadDatabaseSchemaUpdates (or something) hook which registers the table when update.php is run [21:42:44] legoktm: got it [21:43:55] legoktm: secret ? [21:44:09] like, $wgVERPsecret [21:44:12] legoktm: let me dig in. I think we have some kind of SecretSettings.php or something [21:44:24] if it was public, people could reverse the hashes or something. [21:45:11] bah I can't remember the canonical path [21:45:19] PrivateSettings.php [21:45:30] Also there's $wgSecretKey that's used for some things [21:45:34] (03CR) 10Mattflaschen: "Phuedx, I should have -1'ed it per https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Step_3:_configuration_and_other_prep_work ." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/151639 (https://bugzilla.wikimedia.org/69103) (owner: 10Phuedx) [21:46:14] legoktm: possibly deployment-bastion:/srv/scap-stage-dir/private (which is a local git repo) [21:46:31] (03PS1) 10Dzahn: what's left in Tampa [operations/puppet] - 10https://gerrit.wikimedia.org/r/154340 [21:46:36] legoktm: the PrivateSettings.php there is symlinked in mediawiki-config [21:46:50] ah great [21:46:55] legoktm: but the edit must be done somewhere else [21:47:34] so can I do it in the /srcv/scap-stage-dir/private or do I need it somewhere else? [21:47:48] (03PS2) 10Dzahn: what's left in Tampa [operations/puppet] - 10https://gerrit.wikimedia.org/r/154340 [21:47:55] legoktm: just there in PrivateSettings.php [21:48:21] legoktm: if the extension ever land to production, it must not fatal /exception etc if the parameter is missing [21:48:35] legoktm: in case we forget to add it in production PrivateSettings.php before the code get synced [21:48:46] that is just paranoia really [21:49:09] I think we can have the extension fallback to $wgSecretKey or something if the verp specific key isn't set [21:51:46] legoktm: though you dont want the $wgSecretKey to be leaked, so better have to make sure the extension is never going to leak it somehow :-D [21:51:54] it won't! [21:52:50] ok, commited the change [21:57:16] (03PS1) 10Legoktm: Add BounceHandler extension in log only mode to beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154342 [21:59:43] (03PS2) 10Legoktm: Add BounceHandler extension in log only mode to beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154342 (https://bugzilla.wikimedia.org/69621) [22:00:51] no! [22:00:54] (03CR) 10Chmarkine: [C: 031] racktables - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153973 (owner: 10Dzahn) [22:01:58] (03PS2) 10Chmarkine: OTRS - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153998 (owner: 10Dzahn) [22:03:25] (03PS3) 10Dzahn: OTRS - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153998 [22:03:42] :o [22:07:34] (03CR) 10Chmarkine: "According to I9bc1104b, ssl_ciphersuite also supports adding HSTS header. So how about we use this function to add HSTS?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153998 (owner: 10Dzahn) [22:07:42] (03PS17) 10Dzahn: turn RT from misc/* into puppet module [operations/puppet] - 10https://gerrit.wikimedia.org/r/116064 [22:09:03] (03CR) 10Dzahn: "ah, yes yes, absolutely right, we should do that for these 3 cases now: Wikitech, Bugzilla, OTRS" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153998 (owner: 10Dzahn) [22:09:11] domas: kas atsitiko? :O [22:09:48] kazko zmones nori is manes! [22:10:37] ah [22:12:05] * p858snake|l spins Carmela's blamewheel ans watches it not land on domas [22:12:52] yup [22:12:56] happens [22:12:57] sometimes [22:13:47] how's wikipedia doing nowadays? [22:13:53] I realize that I don't really use it that much! [22:14:01] Wiki BROKEN. DOMAS HELP US [22:14:02] domas: as tai nemaciau jus Wikimanioje :S [22:14:13] Vogone: buvau pora valandu [22:14:14] :) [22:14:52] DerHexer man pasakojo [22:15:11] * domas is on SF-London-Brussels-Amsterdam-Vilnius-Paris-London-SF trip [22:15:23] * Nemo_bis scratches head [22:15:25] took a train today from London to Brussels [22:16:16] (03PS3) 10Chmarkine: gitblit - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153985 (owner: 10Dzahn) [22:16:42] (03CR) 10Chmarkine: [C: 031] gitblit - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153985 (owner: 10Dzahn) [22:17:30] damn taxi driver screwed up with my "no cash" streak [22:17:52] (03PS1) 10Dzahn: salt - minion.erb - fix compiler warnings [operations/puppet] - 10https://gerrit.wikimedia.org/r/154347 [22:18:00] I was on my 8th day of this trip without any cash [22:18:22] speaking of which [22:18:29] (03CR) 10Chmarkine: [C: 031] ishmael - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153982 (owner: 10Dzahn) [22:19:05] https://en.wikipedia.org/w/index.php?title=User%3AMidom&diff=621412670&oldid=619363546 [22:19:05] :) [22:20:25] (03CR) 10Chmarkine: [C: 031] rt - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153981 (owner: 10Dzahn) [22:20:26] domas: you're only a year older than me? wtf [22:20:31] hehe [22:20:49] hax [22:21:18] hehe, wiki version of http://www.whereivebeen.com/ [22:21:30] (03CR) 10Chmarkine: [C: 031] ganglia - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153972 (owner: 10Dzahn) [22:21:42] yeah, have to tag in tripadvisor too [22:21:48] I maintain a tripadvisor map [22:22:25] (03CR) 10Chmarkine: [C: 031] noc.wm.org - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153976 (owner: 10Dzahn) [22:22:38] domas: :) and once it was also 43 Places, but seems dead now [22:25:33] 208 cities, 36 countries on tripadvisor map, hm [22:26:48] which countries? [22:26:52] (03CR) 10Chmarkine: [C: 031] stats.wm.org - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153977 (owner: 10Dzahn) [22:27:06] Danny_B: https://en.wikipedia.org/w/index.php?title=User%3AMidom&diff=621412670&oldid=619363546 [22:27:13] I guess there's an overlap [22:28:51] I wonder if they have a better embeddable image than http://www.tripadvisor.com/CommunityMapImage?id=21869208&type=TRIPADVISOR&size=LARGE [22:30:04] domas: http://tinyurl.com/mox64wz <- interesting interview, BTW ;) [22:30:11] domas: i would say use foursquare for check-ins, then export the resulting .kml, go to Google maps, make personal layer, import file.. but they are splitting it into 2 apps, and i guess it's FB competition :p [22:30:28] Vogone: recorder went out of battery at 33% of it [22:30:38] mutante: I uninstalled 4sq [22:30:50] (03CR) 10Jeremyb: [C: 04-1] "Please make the commit msg conform to https://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines#Auto-linking_and_cross-referencing" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154239 (https://bugzilla.wikimedia.org/69594) (owner: 10Calak) [22:30:52] sadly i think that is what is going to happen, yea [22:31:07] xD [22:31:32] and nothing on alternative.to for sq if you check "open source or free" either [22:31:37] 36, hmmm, let's see, if i can approach or even beat it :-P [22:33:32] good luck! [22:33:52] wikivoyage app should do this stuff some day :) [22:34:02] when i see " Photo credit: wikitravel.org " on whereivebeen [22:34:19] riiiight, wikivoyage.... [22:34:42] alright, time to sleep [22:34:44] maybe [22:35:16] meanwhile Google has Ingress [22:35:49] hashSpeleology: how do you go online in a cave? [22:36:13] sounds like no network down there [22:36:24] oh we are not that deep under the ground [22:36:34] maybe 300 meters [22:36:49] hah, i see, probably spliced into some fiber :) [22:37:07] :) [22:37:09] "just" 300 meters of rock [22:37:46] http://www.anatoliamed.com/wp-content/uploads/2013/02/Krubera-Mağarası-Gürcistan2.jpg [22:37:58] :o wow [22:38:08] that pit is roughly 150m [22:38:22] full map and some story at http://www.theblaze.com/stories/2014/03/28/underground-explorers-and-the-shocking-dimensions-of-the-worlds-deepest-cave-how-far-could-you-make-it/ [22:38:40] that caves goes down to 2200m :-/ [22:39:22] domas: labanakt ;) [22:39:28] hashSpeleology: Mr. Bean in last pic [22:39:30] we should have a grant to send Wikipedian / OpenStreetMapper folks down there [22:39:46] so, my roaming/edge did not work at the restaurant, I had to take a taxi [22:39:49] felt like a cave man [22:39:53] could not get uber [22:39:53] haha, yea, OSM is a great idea [22:40:07] uber price for hotel->restaurant - 11eur, taxi back - 27eur [22:40:16] DISRUPT ALL THE THINGS [22:40:17] Uber started in Berlin like 2 months ago, local taxi drivers hate it, law suite.. as usual :p [22:40:31] (03PS2) 10Calak: Add namespace alias on ckbwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154239 (https://bugzilla.wikimedia.org/69594) [22:42:11] well, for me in bay area, Taxi home - ~90-100usd, uberblack (limo) - 80usd, uberx - 40usd. [22:42:31] (03CR) 10Calak: "@Jeremyb: Thank you, done." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154239 (https://bugzilla.wikimedia.org/69594) (owner: 10Calak) [22:43:04] gotta rush out. Have a good week-end [22:43:18] uberx is cheaper than getting a frickin supershuttle [22:44:12] hashSpeleology: enjoy the underground, cya [22:44:24] domas: but cant go to airport(s)? [22:44:32] can go just fine [22:44:35] hm [22:44:42] or not pick up or so? [22:44:54] not sure about picking up, I used uber black lately [22:44:58] (03PS2) 10Calak: Add botadmin user group on fa.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154126 (https://bugzilla.wikimedia.org/69411) [22:45:00] there was something that made uberx different from black car when going to SFO [22:45:06] right [22:46:39] http://blog.uber.com/SFO-update [22:47:11] "SFO has taken an aggressive stance against uberX and has begun citing some drivers. We believe that all Uber rides to and from SFO are legal and that airport officials are acting without the proper authority " :p that [22:48:58] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Thu 14 Aug 2014 20:37:55 UTC [22:52:35] (03CR) 10Jeremyb: "thanks, commit msg now lgtm" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154239 (https://bugzilla.wikimedia.org/69594) (owner: 10Calak) [23:04:18] PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Puppet has 19 failures [23:04:29] PROBLEM - puppet last run on wtp1006 is CRITICAL: CRITICAL: Epic puppet fail [23:04:39] PROBLEM - puppet last run on db73 is CRITICAL: CRITICAL: Epic puppet fail [23:04:39] PROBLEM - puppet last run on lvs4002 is CRITICAL: CRITICAL: Epic puppet fail [23:04:39] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: Puppet has 12 failures [23:04:48] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: Puppet has 6 failures [23:04:49] PROBLEM - puppet last run on cp1059 is CRITICAL: CRITICAL: Puppet has 19 failures [23:04:50] PROBLEM - puppet last run on mw1031 is CRITICAL: CRITICAL: Puppet has 5 failures [23:04:51] PROBLEM - puppet last run on amslvs2 is CRITICAL: CRITICAL: Epic puppet fail [23:04:58] PROBLEM - puppet last run on magnesium is CRITICAL: CRITICAL: Epic puppet fail [23:04:58] PROBLEM - puppet last run on caesium is CRITICAL: CRITICAL: Epic puppet fail [23:05:10] PROBLEM - puppet last run on db1065 is CRITICAL: CRITICAL: Puppet has 14 failures [23:05:10] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: Puppet has 8 failures [23:05:10] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: Puppet has 9 failures [23:05:10] PROBLEM - puppet last run on mw1115 is CRITICAL: CRITICAL: Puppet has 5 failures [23:05:10] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: Puppet has 13 failures [23:05:10] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Epic puppet fail [23:05:10] PROBLEM - puppet last run on amssq43 is CRITICAL: CRITICAL: Puppet has 4 failures [23:05:10] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: Puppet has 5 failures [23:05:10] PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: Epic puppet fail [23:05:11] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: Puppet has 15 failures [23:05:18] PROBLEM - puppet last run on dbproxy1002 is CRITICAL: CRITICAL: Puppet has 11 failures [23:05:18] PROBLEM - puppet last run on db1029 is CRITICAL: CRITICAL: Puppet has 7 failures [23:05:18] PROBLEM - puppet last run on amssq54 is CRITICAL: CRITICAL: Puppet has 4 failures [23:05:18] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: Puppet has 2 failures [23:05:28] "Epic puppet fail" always gets me [23:05:28] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: Puppet has 1 failures [23:05:29] PROBLEM - puppet last run on analytics1017 is CRITICAL: CRITICAL: Puppet has 8 failures [23:05:29] PROBLEM - puppet last run on mw1048 is CRITICAL: CRITICAL: Puppet has 31 failures [23:05:29] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: Puppet has 7 failures [23:05:39] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: Puppet has 12 failures [23:05:39] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: Puppet has 10 failures [23:05:48] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: Puppet has 1 failures [23:05:48] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Puppet has 5 failures [23:05:49] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Puppet has 4 failures [23:05:49] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: Puppet has 5 failures [23:06:08] PROBLEM - puppet last run on mw1007 is CRITICAL: CRITICAL: Puppet has 1 failures [23:06:09] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Puppet has 14 failures [23:07:56] ignore that, it's just puppetmaster tuning gone badly, I think [23:09:08] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:09:39] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: Epic puppet fail [23:09:48] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: Epic puppet fail [23:09:48] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CRITICAL: Epic puppet fail [23:09:48] PROBLEM - puppet last run on nfs1 is CRITICAL: CRITICAL: Epic puppet fail [23:09:58] PROBLEM - puppet last run on mw1084 is CRITICAL: CRITICAL: Epic puppet fail [23:10:08] PROBLEM - puppet last run on cp4005 is CRITICAL: CRITICAL: Epic puppet fail [23:10:08] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: Puppet has 2 failures [23:10:08] PROBLEM - puppet last run on mw1208 is CRITICAL: CRITICAL: Puppet has 45 failures [23:10:09] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: Epic puppet fail [23:10:09] PROBLEM - puppet last run on search1002 is CRITICAL: CRITICAL: Epic puppet fail [23:10:09] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: Epic puppet fail [23:10:09] PROBLEM - puppet last run on amssq51 is CRITICAL: CRITICAL: Epic puppet fail [23:10:18] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: Epic puppet fail [23:10:29] PROBLEM - puppet last run on mc1005 is CRITICAL: CRITICAL: Puppet has 10 failures [23:10:29] PROBLEM - puppet last run on gadolinium is CRITICAL: CRITICAL: Epic puppet fail [23:10:29] PROBLEM - puppet last run on analytics1013 is CRITICAL: CRITICAL: Epic puppet fail [23:10:39] PROBLEM - puppet last run on amssq56 is CRITICAL: CRITICAL: Epic puppet fail [23:10:49] PROBLEM - puppet last run on amssq34 is CRITICAL: CRITICAL: Puppet has 2 failures [23:10:58] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: Puppet has 4 failures [23:19:24] (03PS4) 10Chmarkine: OTRS - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153998 (owner: 10Dzahn) [23:19:35] win 35 [23:21:48] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [23:21:48] RECOVERY - puppet last run on mw1031 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [23:22:09] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [23:22:29] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [23:22:39] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:22:48] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [23:22:49] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:22:50] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [23:22:58] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [23:22:58] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [23:23:08] RECOVERY - puppet last run on db1065 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [23:23:09] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [23:23:09] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:23:18] RECOVERY - puppet last run on dbproxy1002 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:23:18] RECOVERY - puppet last run on db1029 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:23:29] RECOVERY - puppet last run on wtp1006 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [23:23:29] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:23:39] RECOVERY - puppet last run on db73 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:23:39] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:23:39] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:23:48] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:23:48] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [23:23:48] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [23:24:08] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [23:24:08] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [23:24:08] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [23:24:08] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:24:09] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:24:09] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:24:09] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:24:18] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [23:24:18] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [23:24:18] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [23:24:28] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:24:29] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [23:24:39] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [23:28:08] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:28:48] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:28:49] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:28:58] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [23:29:08] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [23:29:29] RECOVERY - puppet last run on gadolinium is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [23:29:29] RECOVERY - puppet last run on mc1005 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [23:29:29] RECOVERY - puppet last run on analytics1013 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [23:29:39] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [23:29:48] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [23:29:48] RECOVERY - puppet last run on nfs1 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:29:58] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:30:09] RECOVERY - puppet last run on mw1151 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [23:30:09] RECOVERY - puppet last run on search1002 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [23:30:09] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:30:09] RECOVERY - puppet last run on amssq51 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [23:30:18] RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [23:30:39] RECOVERY - puppet last run on amssq56 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [23:31:08] RECOVERY - puppet last run on cp4005 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:47:06] (03PS18) 10Dzahn: turn RT from misc/* into puppet module [operations/puppet] - 10https://gerrit.wikimedia.org/r/116064 [23:53:19] (03PS19) 10Dzahn: turn RT from misc/* into puppet module [operations/puppet] - 10https://gerrit.wikimedia.org/r/116064 [23:56:12] (03PS1) 10Ori.livneh: ordered_yaml(): fix for Ruby 1.8 [operations/puppet] - 10https://gerrit.wikimedia.org/r/154366 [23:59:18] (03CR) 10Ori.livneh: "Easy to test on 1.8 with this snippet: https://dpaste.de/itPT/raw" [operations/puppet] - 10https://gerrit.wikimedia.org/r/154366 (owner: 10Ori.livneh)