[00:00:08] Logged the message, Master [00:00:15] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: [00:00:32] Logged the message, Master [00:02:56] !log aaron started scap [00:03:13] Logged the message, Master [00:13:22] !log aaron finished scap [00:13:39] Logged the message, Master [00:25:01] (03PS1) 10Aaron Schulz: Use full fanout for sync-wikiversions [operations/puppet] - 10https://gerrit.wikimedia.org/r/104171 [00:25:37] (03CR) 10Aaron Schulz: "Goes from 20->3 seconds" [operations/puppet] - 10https://gerrit.wikimedia.org/r/104171 (owner: 10Aaron Schulz) [00:26:01] (03Abandoned) 10Aaron Schulz: Sync out the wikiversions.cdb in scap-1 only [operations/puppet] - 10https://gerrit.wikimedia.org/r/103870 (owner: 10Aaron Schulz) [00:37:32] (03CR) 10Ori.livneh: [C: 032] Use full fanout for sync-wikiversions [operations/puppet] - 10https://gerrit.wikimedia.org/r/104171 (owner: 10Aaron Schulz) [00:40:56] (03PS1) 10BryanDavis: [WIP] Kibana puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/104172 [00:41:39] (03CR) 10jenkins-bot: [V: 04-1] [WIP] Kibana puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/104172 (owner: 10BryanDavis) [00:42:41] (03PS2) 10BryanDavis: [WIP] Kibana puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/104172 [01:02:52] (03CR) 10BryanDavis: [WIP] Kibana puppet class (034 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/104172 (owner: 10BryanDavis) [01:24:11] (03PS1) 10Aaron Schulz: Call refreshCdbJsonFiles in periodic l10update script [operations/puppet] - 10https://gerrit.wikimedia.org/r/104175 [01:55:31] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [02:07:43] !log LocalisationUpdate completed (1.23wmf7) at Sat Dec 28 02:07:43 UTC 2013 [02:08:01] Logged the message, Master [02:13:11] !log LocalisationUpdate completed (1.23wmf8) at Sat Dec 28 02:13:10 UTC 2013 [02:13:28] Logged the message, Master [02:27:24] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Dec 28 02:27:24 UTC 2013 [02:27:42] Logged the message, Master [03:22:11] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [03:22:51] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 35.43 ms [03:25:21] PROBLEM - Apache HTTP on mw31 is CRITICAL: Connection refused [03:26:21] RECOVERY - Apache HTTP on mw31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.418 second response time [04:56:31] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [07:46:31] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: Last successful Puppet run was Sat 28 Dec 2013 04:45:24 AM UTC [07:50:52] RECOVERY - Puppet freshness on searchidx1001 is OK: puppet ran at Sat Dec 28 07:50:45 UTC 2013 [07:57:31] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [08:26:53] (03PS1) 10Aaron Schulz: Added MW version argument to scap [operations/puppet] - 10https://gerrit.wikimedia.org/r/104189 [09:05:52] (03PS1) 10Aaron Schulz: Disable MessageBlobStore::clear() via hook [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104191 [09:06:22] (03CR) 10Ori.livneh: [C: 031] "Nice! That's a good idea. Have you tested it? Happy to merge if so." (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/104189 (owner: 10Aaron Schulz) [09:14:11] (03CR) 10Aaron Schulz: "Haven't tested much other than RSYNC_ARGS variable generation and some bits. If you want to test around this weekend be my guess though :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/104189 (owner: 10Aaron Schulz) [09:14:39] (03CR) 10Aaron Schulz: "Needs https://gerrit.wikimedia.org/r/104189 on all branches" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104191 (owner: 10Aaron Schulz) [09:58:41] PROBLEM - SSH on lvs4001 is CRITICAL: Server answer: [09:59:41] RECOVERY - SSH on lvs4001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [10:58:31] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [13:51:20] (03PS3) 10Tim Landscheidt: Tool Labs: package libfcgi-dev [operations/puppet] - 10https://gerrit.wikimedia.org/r/84288 (owner: 10DrTrigon) [13:59:31] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [17:00:31] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [18:45:05] (03PS1) 10Tim Landscheidt: Tools: Remove obsolete tyrant class/role [operations/puppet] - 10https://gerrit.wikimedia.org/r/104219 [18:49:27] (03CR) 10Tim Landscheidt: "The change removes the requirement of python-uwsgi as well as the introducing change (#74145, "Tool Labs: Exec environ needs uwsgi for vas" [operations/puppet] - 10https://gerrit.wikimedia.org/r/104219 (owner: 10Tim Landscheidt) [19:01:00] (03CR) 10coren: [C: 032] "It might be reasonable to reintroduce it eventually; in the meantime the tyrant setup is indeed obsolete." [operations/puppet] - 10https://gerrit.wikimedia.org/r/104219 (owner: 10Tim Landscheidt) [20:01:31] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [20:37:20] huh, why is https://en.wikipedia.org/wiki/MediaWiki:Redirect-page blank? [20:37:39] that message exists in the code [20:38:14] hm, or maybe not [20:39:20] (not deployed yet) [21:03:20] (03CR) 10Tim Landscheidt: "Petan, could you please "abandon" this change? Thanks." [operations/puppet] - 10https://gerrit.wikimedia.org/r/96478 (owner: 10Petrb) [21:03:53] (03PS1) 10Aaron Schulz: Removed fileJournal config; this has been used for some time [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104316 [21:17:51] (03PS1) 10Aaron Schulz: Setup and enabled redisLockManager for all file backends in use [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104317 [23:02:31] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Last successful Puppet run was Tue 24 Dec 2013 07:29:50 PM UTC [23:34:39] (03PS1) 10Ryan Lane: Add redis config for keystone in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/104322