[00:11:58] PROBLEM - udp2log log age for oxygen on oxygen is CRITICAL: CRITICAL: log files /var/log/udp2log/packet-loss.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [00:19:19] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [00:22:19] PROBLEM - ElasticSearch health check on elastic1012 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 17: number_of_data_nodes: 17: active_primary_shards: 2037: active_shards: 6110: relocating_shards: 0: initializing_shards: 1: unassigned_shards: 1 [00:23:18] RECOVERY - ElasticSearch health check on elastic1012 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 17: number_of_data_nodes: 17: active_primary_shards: 2038: active_shards: 6113: relocating_shards: 0: initializing_shards: 0: unassigned_shards: 0 [00:33:09] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sun Aug 17 00:33:07 UTC 2014 [00:42:48] PROBLEM - MySQL Processlist on db1056 is CRITICAL: CRIT 88 unauthenticated, 0 locked, 0 copy to table, 0 statistics [00:43:28] PROBLEM - MySQL Processlist on db1068 is CRITICAL: CRIT 68 unauthenticated, 0 locked, 0 copy to table, 0 statistics [00:43:48] RECOVERY - MySQL Processlist on db1056 is OK: OK 1 unauthenticated, 0 locked, 0 copy to table, 0 statistics [00:44:28] RECOVERY - MySQL Processlist on db1068 is OK: OK 1 unauthenticated, 0 locked, 0 copy to table, 1 statistics [01:55:25] (03PS1) 10PleaseStand: CommonSettings.php: Remove some dated cruft [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154442 [02:14:39] !log LocalisationUpdate completed (1.24wmf16) at 2014-08-17 02:13:35+00:00 [02:14:48] Logged the message, Master [02:24:12] !log LocalisationUpdate completed (1.24wmf17) at 2014-08-17 02:23:08+00:00 [02:24:21] Logged the message, Master [02:40:39] PROBLEM - Disk space on labsdb1002 is CRITICAL: DISK CRITICAL - free space: /srv 121890 MB (3% inode=99%): [02:45:34] ugh [02:48:39] RECOVERY - Disk space on labsdb1002 is OK: DISK OK [02:49:52] !log killed stuff on labsdb1003 using all disk for temp tables. investigating [02:49:58] Logged the message, Master [02:56:33] YuviPanda|zzz: bunch of hour-long queries from Quarry coincided with that labsdb1002 disk usage spike. is your query killer active? [03:00:24] legoktm: noooo... we're still seeing unattached accounts [03:01:15] springle, yuvi went to sleep a few minutes ago. However, see his last !log msg in -labs which might shed light? [03:01:49] Those were attached successfully (according to row count) and we're not in a transaction, but still they aren't really being attached [03:01:50] * hoo wtfs [03:05:28] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Aug 17 03:04:22 UTC 2014 (duration 4m 21s) [03:05:34] Logged the message, Master [03:12:28] PROBLEM - check if dhclient is running on rhenium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:13:19] RECOVERY - check if dhclient is running on rhenium is OK: PROCS OK: 0 processes with command name dhclient [03:32:04] hoo: awwww crap [03:32:36] legoktm: API URL that time [03:32:46] so just fixing TMH wont help [03:32:56] at least wont fix it, might help, though [03:34:04] should we just wrap CentralAuthUser::attach in a transaction? [03:34:15] so we know it gets commited [03:42:01] legoktm: I guess we could do that [03:42:07] not sure it has *any* value, though [03:42:11] but also I'm tired [03:42:17] jet lag -.- [03:42:22] :/ [03:42:34] well, it would force it to get committed right? so nothing could roll it back [03:45:06] legoktm: mh, true [03:45:15] if that's the actual problem [03:45:26] do it [03:45:33] do it now and I'll even review it [03:45:35] :P [04:34:54] !log restarted udp2log on oxygen [04:34:58] RECOVERY - udp2log log age for oxygen on oxygen is OK: OK: all log files active [04:34:59] Logged the message, Master [04:51:08] RECOVERY - Packetloss_Average on oxygen is OK: packet_loss_average OKAY: 0.0 [05:42:58] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Epic puppet fail [05:51:58] PROBLEM - udp2log log age for oxygen on oxygen is CRITICAL: CRITICAL: log files /var/log/udp2log/packet-loss.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [06:01:58] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:13:12] (03CR) 10TTO: [C: 031] "Well done on figuring out Git and Gerrit :)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154239 (https://bugzilla.wikimedia.org/69594) (owner: 10Calak) [06:20:58] PROBLEM - Puppet freshness on db1010 is CRITICAL: Last successful Puppet run was Sun 17 Aug 2014 04:20:28 UTC [06:25:59] RECOVERY - udp2log log age for oxygen on oxygen is OK: OK: all log files active [06:27:48] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: Epic puppet fail [06:28:49] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:08] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:38] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:38] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:48] PROBLEM - puppet last run on search1010 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:48] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:48] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:48] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:48] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:58] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:08] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:18] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:18] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:18] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:19] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:39] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:41] (03PS1) 10TTO: Set wgUploadNavigationUrl for eowiki to Commons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154448 (https://bugzilla.wikimedia.org/69055) [06:39:09] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: Puppet has 1 failures [06:44:48] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:44:48] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:45:18] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:45:18] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:45:39] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:46:08] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:46:09] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:46:38] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:46:38] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:46:48] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:46:49] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:46:49] RECOVERY - puppet last run on search1010 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:46:49] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:46:49] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:47:08] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [06:47:18] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:47:18] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:48:08] km-mpb:~ km$ ssh bast1001.wikimedia.org [06:48:08] ssh: connect to host bast1001.wikimedia.org port 22: No route to host [06:48:14] is it just me? [06:48:17] I can ping it fine... [06:49:08] well that was weird, I got in now [06:57:09] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:20:18] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Sun Aug 17 07:20:12 UTC 2014 [07:51:08] PROBLEM - udp2log log age for oxygen on oxygen is CRITICAL: CRITICAL: log files /var/log/udp2log/packet-loss.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [09:22:09] !log ongoing schema change wikidatawiki & testwiki wb_entity_per_page.epp_redirect_target. osc_host.sh processes on terbium ok to kill in emergency [09:22:15] Logged the message, Master [10:37:16] (03CR) 10Jforrester: [C: 04-1] Enable GlobalCssJs on all CentralAuth wikis minus loginwiki (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154432 (https://bugzilla.wikimedia.org/57891) (owner: 10Legoktm) [10:51:39] PROBLEM - puppet last run on analytics1013 is CRITICAL: CRITICAL: Puppet has 1 failures [11:09:39] RECOVERY - puppet last run on analytics1013 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [11:13:58] PROBLEM - Puppet freshness on db1011 is CRITICAL: Last successful Puppet run was Sun 17 Aug 2014 09:12:57 UTC [11:30:02] (03PS1) 10TTO: Clean up indents, comments, spacing in InitialiseSettings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154455 (https://bugzilla.wikimedia.org/29902) [11:33:28] RECOVERY - Puppet freshness on db1011 is OK: puppet ran at Sun Aug 17 11:33:21 UTC 2014 [11:53:15] (03PS1) 10QChris: Disable udp2log forward to University of Minnesota [operations/puppet] - 10https://gerrit.wikimedia.org/r/154457 (https://bugzilla.wikimedia.org/69661) [12:08:16] oxygen's root mount is full. https://gerrit.wikimedia.org/r/154457 would fix the cause. [12:08:28] Any ops around that could help me getting that change merged and [12:08:50] getting /var/log/udp2log/udp2log.log removed from oxygen? [12:09:13] (Corresponding bug https://bugzilla.wikimedia.org/show_bug.cgi?id=69661) [12:11:21] ottomata, jgage ^ [12:17:48] bblack ^ [12:26:41] springle: bah, am back [12:27:05] springle: it's active, but I was re-running all previous queries one after the other. I had changed my output format and had to have them re-run [12:27:32] springle: I babysat it for a while, and then went to sleep. sorry about that [12:27:58] springle: did it kill anything? [12:31:11] springle: so that's also how about 150 people running queries simulataneously would look like [12:31:28] springle: only 48 would ever actually run simultaneously, though [12:31:50] springle: unless I fucked up during one of the runner restarts and di dn't let them kill queries. ugh [12:34:30] springle: I'm redoing the query killer to be far more robust - it'll just do show processlist and kill things based on 'time' there instead of using my own time counter. should be deployed later today [13:13:29] (03CR) 10Ottomata: [C: 032 V: 032] Disable udp2log forward to University of Minnesota [operations/puppet] - 10https://gerrit.wikimedia.org/r/154457 (https://bugzilla.wikimedia.org/69661) (owner: 10QChris) [13:14:06] ottomata: Thanks! [13:15:28] RECOVERY - Disk space on oxygen is OK: DISK OK [13:15:46] thanks qchris! [13:17:08] RECOVERY - udp2log log age for oxygen on oxygen is OK: OK: all log files active [13:41:12] PROBLEM - Puppet freshness on nickel is CRITICAL: Last successful Puppet run was Sun 17 Aug 2014 13:37:55 UTC [13:43:12] PROBLEM - Puppet freshness on nickel is CRITICAL: Last successful Puppet run was Sun 17 Aug 2014 13:37:55 UTC [13:45:12] PROBLEM - Puppet freshness on nickel is CRITICAL: Last successful Puppet run was Sun 17 Aug 2014 13:37:55 UTC [13:47:12] PROBLEM - Puppet freshness on nickel is CRITICAL: Last successful Puppet run was Sun 17 Aug 2014 13:37:55 UTC [13:49:12] PROBLEM - Puppet freshness on nickel is CRITICAL: Last successful Puppet run was Sun 17 Aug 2014 13:37:55 UTC [13:51:12] PROBLEM - Puppet freshness on nickel is CRITICAL: Last successful Puppet run was Sun 17 Aug 2014 13:37:55 UTC [13:53:12] PROBLEM - Puppet freshness on nickel is CRITICAL: Last successful Puppet run was Sun 17 Aug 2014 13:37:55 UTC [13:55:12] PROBLEM - Puppet freshness on nickel is CRITICAL: Last successful Puppet run was Sun 17 Aug 2014 13:37:55 UTC [13:57:12] PROBLEM - Puppet freshness on nickel is CRITICAL: Last successful Puppet run was Sun 17 Aug 2014 13:37:55 UTC [13:58:52] RECOVERY - Puppet freshness on nickel is OK: puppet ran at Sun Aug 17 13:58:42 UTC 2014 [14:19:06] * YuviPanda chuchles at qchris_away being called qchrist [14:19:10] our lord and saviour? [14:26:21] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Epic puppet fail [14:36:12] * qchris_away slaps YuviPanda around a bit with a large trout [14:37:13] :-P [14:46:21] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [15:00:21] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Sun 17 Aug 2014 12:59:39 UTC [15:17:11] (03PS1) 10Yuvipanda: quarry: Increase max_connections on local db to 1024 [operations/puppet] - 10https://gerrit.wikimedia.org/r/154467 [15:17:13] (03PS1) 10Yuvipanda: quarry: Add a cron based query killer [operations/puppet] - 10https://gerrit.wikimedia.org/r/154468 [15:17:52] some_devil: :P [15:36:52] (03PS2) 10Yuvipanda: quarry: Add a cron based query killer [operations/puppet] - 10https://gerrit.wikimedia.org/r/154468 [15:39:21] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: Epic puppet fail [15:47:25] (03PS3) 10Yuvipanda: quarry: Add a cron based query killer [operations/puppet] - 10https://gerrit.wikimedia.org/r/154468 [15:53:21] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sun 17 Aug 2014 13:52:52 UTC [15:59:21] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [16:13:12] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sun Aug 17 16:13:08 UTC 2014 [16:48:20] (03PS1) 10Calak: Change FlaggedRevs configuration on fa.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154473 (https://bugzilla.wikimedia.org/69668) [16:54:21] PROBLEM - puppet last run on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:56:11] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 711 seconds ago with 0 failures [17:01:21] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Sun 17 Aug 2014 12:59:39 UTC [17:04:21] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Epic puppet fail [17:09:02] (03CR) 10Calak: [C: 031] Clean up indents, comments, spacing in InitialiseSettings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154455 (https://bugzilla.wikimedia.org/29902) (owner: 10TTO) [17:15:21] (03PS4) 10Yuvipanda: quarry: Add a cron based query killer [operations/puppet] - 10https://gerrit.wikimedia.org/r/154468 [17:15:23] (03PS2) 10Yuvipanda: quarry: Use mariadb 10 [operations/puppet] - 10https://gerrit.wikimedia.org/r/154467 [17:23:21] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [17:31:32] PROBLEM - puppetmaster backend https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:32:21] RECOVERY - puppetmaster backend https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.058 second response time [18:40:01] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Sun Aug 17 18:39:54 UTC 2014 [18:41:44] !log Upgrading Zuul to latest version (that is not a friday afterall) [18:41:52] Logged the message, Master [18:43:08] hashar: hehe :) [18:44:30] what can possibly goes wrong [18:45:09] assert( 'Sunday' !== 'Friday' ) [18:45:18] That passes for me [18:45:31] !log Zuul upgraded [18:45:37] Logged the message, Master [18:45:59] But I'm the guy who broke the crap out of MediaWiki-Vagrant for lots of people and then left on vacation for a week :( [18:48:02] bd808: shouldn't it be != ? ;) [18:55:39] bd808: we are all human! [18:55:46] how was your trip? [18:56:16] hashar: It was fun. We did a lot touristy things in London [18:56:31] I should go there one day [18:56:50] I have no excuse: cheap direct flight and an old close friend living there in a huge house [18:57:54] hashar: cheap flight and huge house - surprised you have decided to go yet :p [18:58:02] My advice: wear comfortable shoes. We walked and walked and walked. [18:58:03] *haven't [18:58:52] with kids, you can't really walk though :/ [18:59:56] but but it is hackable http://smg.photobucket.com/user/jalexandre/media/Weber/siege-bebe-remorque-velo-2.jpg.html [19:02:01] My wife's favorite trip was to see the Harry Potter studio tour. I think I liked the Tate Modern best [19:02:55] time for a blue chimay [19:17:46] (03PS1) 10Hashar: contint: website config on labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/154484 [19:20:37] working on Sunday is just working during european mornings: it is super quiet around there [19:22:24] (03CR) 10Hashar: [C: 031] "Cherry picked on contint puppetmaster in labs. That made the Jenkins and Zuul publicly available as they should." [operations/puppet] - 10https://gerrit.wikimedia.org/r/154484 (owner: 10Hashar) [19:33:20] (03PS1) 10Hashar: contint: proxy the colocated Gerrit instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/154488 [19:41:22] (03PS2) 10Hashar: contint: proxy the colocated Gerrit instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/154488 [19:42:20] (03CR) 10Hashar: [C: 04-1] "PS2 fix typo in file name." [operations/puppet] - 10https://gerrit.wikimedia.org/r/154488 (owner: 10Hashar) [19:49:06] (03PS3) 10Hashar: contint: proxy the colocated Gerrit instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/154488 [19:52:20] (03PS4) 10Hashar: contint: proxy the colocated Gerrit instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/154488 [19:57:22] (03PS5) 10Hashar: contint: proxy the colocated Gerrit instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/154488 [20:06:55] (03CR) 10Hashar: [C: 031 V: 032] "cherry picked on contint puppet master and works as expected. This is just for labs, can be merged in at anytime." [operations/puppet] - 10https://gerrit.wikimedia.org/r/154488 (owner: 10Hashar) [20:09:46] (03PS2) 10Calak: Change FlaggedRevs configuration on fa.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154473 (https://bugzilla.wikimedia.org/69668) [20:12:05] (03CR) 10Legoktm: Enable GlobalCssJs on all CentralAuth wikis minus loginwiki (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154432 (https://bugzilla.wikimedia.org/57891) (owner: 10Legoktm) [20:23:18] (03PS1) 10Hashar: gerrit: prettify Zuul build results [operations/puppet] - 10https://gerrit.wikimedia.org/r/154524 (https://bugzilla.wikimedia.org/66095) [20:24:34] (03PS3) 10Reza: Change FlaggedRevs configuration on fa.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154473 (https://bugzilla.wikimedia.org/69668) (owner: 10Calak) [20:44:15] (03PS2) 10Hashar: gerrit: prettify Zuul build results [operations/puppet] - 10https://gerrit.wikimedia.org/r/154524 (https://bugzilla.wikimedia.org/66095) [20:45:58] (03CR) 10Hashar: [C: 031] "Chad, Chris, could use a review of the Gerrit regex but should be fine since that is a copy paste of what is deployed on OpenStack." [operations/puppet] - 10https://gerrit.wikimedia.org/r/154524 (https://bugzilla.wikimedia.org/66095) (owner: 10Hashar) [20:58:41] (03CR) 10Jforrester: "Should be OK to go out on Tuesday." (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/154432 (https://bugzilla.wikimedia.org/57891) (owner: 10Legoktm) [21:07:57] !log running migrateAccount.php without --safe or --auto on terbium for bug 69291 [21:08:02] Logged the message, Master [21:09:42] legoktm: progress! :P [21:10:32] PROBLEM - puppetmaster backend https on palladium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:11:21] RECOVERY - puppetmaster backend https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.075 second response time [22:13:01] (03CR) 10QChris: [C: 04-1] gerrit: prettify Zuul build results (034 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/154524 (https://bugzilla.wikimedia.org/66095) (owner: 10Hashar) [22:24:49] (03PS1) 10Ori.livneh: mediawiki: use 'udplog' service alias instead of hard-coding fluorine [operations/puppet] - 10https://gerrit.wikimedia.org/r/154710 [22:29:27] (03PS2) 10Ori.livneh: mediawiki: use 'udplog' service alias instead of hard-coding fluorine [operations/puppet] - 10https://gerrit.wikimedia.org/r/154710 [22:42:09] (03PS3) 10Ori.livneh: mediawiki: use 'udplog' service alias instead of hard-coding fluorine [operations/puppet] - 10https://gerrit.wikimedia.org/r/154710 [23:50:37] ori: btw, I'm working on making txstatsd and graphite work in trusty now [23:53:30] (03PS1) 10Yuvipanda: graphite: Fix graphite-auth to work with trusty [operations/puppet] - 10https://gerrit.wikimedia.org/r/154712