[00:44:27] (03PS1) 10Ori.livneh: [HAT] text-frontend VCL: set Content-Type if not set [operations/puppet] - 10https://gerrit.wikimedia.org/r/141086 [00:45:38] (03PS2) 10Ori.livneh: [HAT] text-frontend VCL: set Content-Type if not set [operations/puppet] - 10https://gerrit.wikimedia.org/r/141086 [00:48:14] (03CR) 10Ori.livneh: [C: 031] rancid: also create system group for system user [operations/puppet] - 10https://gerrit.wikimedia.org/r/141078 (owner: 10Dzahn) [00:52:41] (03CR) 10Ori.livneh: "I cherry-picked this on the beta cluster and deployed it to the beta cluster text varnishes for testing." [operations/puppet] - 10https://gerrit.wikimedia.org/r/141086 (owner: 10Ori.livneh) [00:59:03] (03Abandoned) 10Ori.livneh: mw/apache 2.4 compat: remove DefaultType directive [operations/puppet] - 10https://gerrit.wikimedia.org/r/138891 (owner: 10Ori.livneh) [00:59:24] erm. [00:59:36] I suppose we'd still want to remove it once it's on the Varnishes. [00:59:53] I'll restore it once we get there. [01:51:04] (03PS1) 10TTO: Kill $wgEnableNewpagesUserFilter [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/141100 (https://bugzilla.wikimedia.org/58932) [02:07:47] Nemo_bis: yes, any sort of notification of schema changes is good [02:18:22] !log LocalisationUpdate completed (1.24wmf9) at 2014-06-21 02:17:18+00:00 [02:18:30] Logged the message, Master [02:30:02] !log LocalisationUpdate completed (1.24wmf10) at 2014-06-21 02:28:59+00:00 [02:30:07] Logged the message, Master [02:46:23] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 21 02:45:17 UTC 2014 (duration 45m 16s) [02:46:28] Logged the message, Master [03:17:01] PROBLEM - MySQL InnoDB on es1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:17:51] RECOVERY - MySQL InnoDB on es1001 is OK: OK longest blocking idle transaction sleeps for 0 seconds [04:06:21] PROBLEM - LighttpdHTTP on dataset1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:07:12] RECOVERY - LighttpdHTTP on dataset1001 is OK: HTTP OK: HTTP/1.1 200 OK - 5122 bytes in 0.001 second response time [07:24:49] (03CR) 10Legoktm: [C: 031] Kill $wgEnableNewpagesUserFilter [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/141100 (https://bugzilla.wikimedia.org/58932) (owner: 10TTO) [09:54:49] (03PS1) 10QChris: Parametrize redis' dbfilename [operations/puppet] - 10https://gerrit.wikimedia.org/r/141116 (https://bugzilla.wikimedia.org/66911) [10:40:03] (03PS8) 10QChris: Enable the new backup role in wikimetrics if set [operations/puppet] - 10https://gerrit.wikimedia.org/r/139558 (https://bugzilla.wikimedia.org/66119) (owner: 10Milimetric) [10:40:05] (03PS1) 10QChris: Have Wikimetrics use the redis module's configuration again [operations/puppet] - 10https://gerrit.wikimedia.org/r/141120 (https://bugzilla.wikimedia.org/66911) [10:41:31] (03CR) 10jenkins-bot: [V: 04-1] Enable the new backup role in wikimetrics if set [operations/puppet] - 10https://gerrit.wikimedia.org/r/139558 (https://bugzilla.wikimedia.org/66119) (owner: 10Milimetric) [10:41:48] (03CR) 10jenkins-bot: [V: 04-1] Have Wikimetrics use the redis module's configuration again [operations/puppet] - 10https://gerrit.wikimedia.org/r/141120 (https://bugzilla.wikimedia.org/66911) (owner: 10QChris) [10:47:52] (03CR) 10QChris: Enable the new backup role in wikimetrics if set (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/139558 (https://bugzilla.wikimedia.org/66119) (owner: 10Milimetric) [10:50:57] (03CR) 10QChris: "The Jenkins failure is expected, as the commit from the" [operations/puppet] - 10https://gerrit.wikimedia.org/r/141120 (https://bugzilla.wikimedia.org/66911) (owner: 10QChris) [11:33:12] (03PS4) 10Nikerabbit: Enable ContentTranslation extension on beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/140723 [15:14:18] (03CR) 10JanZerebecki: [C: 031] [HAT] Load mod_version on application servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/141059 (owner: 10Ori.livneh) [15:20:02] (03CR) 10JanZerebecki: "Not sure if that is important enough, but you could use instead as HHVM vs mod_php have no technical dependency on an a" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/141062 (owner: 10Ori.livneh) [16:08:01] PROBLEM - puppet disabled on ms-be1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:08:11] PROBLEM - SSH on ms-be1012 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:08:11] PROBLEM - RAID on ms-be1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:08:21] PROBLEM - swift-object-auditor on ms-be1012 is CRITICAL: Timeout while attempting connection [16:08:21] PROBLEM - swift-object-server on ms-be1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:08:21] PROBLEM - swift-account-replicator on ms-be1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:08:21] PROBLEM - DPKG on ms-be1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:08:21] PROBLEM - swift-account-reaper on ms-be1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:08:21] PROBLEM - swift-container-auditor on ms-be1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:08:21] PROBLEM - check if dhclient is running on ms-be1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:08:22] PROBLEM - swift-container-replicator on ms-be1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:08:22] PROBLEM - swift-object-updater on ms-be1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:08:23] PROBLEM - swift-container-updater on ms-be1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:08:23] PROBLEM - swift-container-server on ms-be1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:08:24] PROBLEM - swift-object-replicator on ms-be1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:08:24] PROBLEM - swift-account-auditor on ms-be1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:08:25] PROBLEM - swift-account-server on ms-be1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:08:39] <_joe_> we do have some checks [16:08:52] <_joe_> uff, going to console [16:10:31] PROBLEM - Disk space on ms-be1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:11:21] PROBLEM - check configured eth on ms-be1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:11:28] <_joe_> http://paste.debian.net/hidden/d3d9bdd7/ < -- ms-be1012 [16:12:57] <_joe_> !log restarted ms-be1012, see http://paste.debian.net/hidden/d3d9bdd7/ for console output (will expire in 3d) [16:13:02] Logged the message, Master [16:15:11] RECOVERY - check configured eth on ms-be1012 is OK: NRPE: Unable to read output [16:15:11] RECOVERY - DPKG on ms-be1012 is OK: All packages OK [16:15:11] RECOVERY - swift-account-replicator on ms-be1012 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-account-replicator [16:15:11] RECOVERY - swift-object-server on ms-be1012 is OK: PROCS OK: 101 processes with regex args ^/usr/bin/python /usr/bin/swift-object-server [16:15:11] RECOVERY - swift-container-server on ms-be1012 is OK: PROCS OK: 25 processes with regex args ^/usr/bin/python /usr/bin/swift-container-server [16:15:11] RECOVERY - swift-container-replicator on ms-be1012 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-replicator [16:15:11] RECOVERY - swift-object-auditor on ms-be1012 is OK: PROCS OK: 2 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [16:15:12] RECOVERY - swift-container-auditor on ms-be1012 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [16:15:12] RECOVERY - check if dhclient is running on ms-be1012 is OK: PROCS OK: 0 processes with command name dhclient [16:15:13] RECOVERY - swift-account-reaper on ms-be1012 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-account-reaper [16:15:13] RECOVERY - swift-container-updater on ms-be1012 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-updater [16:15:14] RECOVERY - swift-account-auditor on ms-be1012 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-account-auditor [16:15:14] RECOVERY - swift-account-server on ms-be1012 is OK: PROCS OK: 25 processes with regex args ^/usr/bin/python /usr/bin/swift-account-server [16:15:15] RECOVERY - swift-object-replicator on ms-be1012 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-object-replicator [16:15:15] RECOVERY - swift-object-updater on ms-be1012 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-object-updater [16:15:21] RECOVERY - Disk space on ms-be1012 is OK: DISK OK [16:15:51] RECOVERY - puppet disabled on ms-be1012 is OK: OK [16:16:01] RECOVERY - SSH on ms-be1012 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [16:16:01] RECOVERY - RAID on ms-be1012 is OK: OK: optimal, 14 logical, 14 physical [18:10:05] cmjohnson, hey chris [18:10:47] hi papaul [18:11:10] cmjohnson, how are you today [18:11:26] not bad...rainy saturday here [18:11:46] cmjohnson, very sunny here [18:12:17] cmjohnson, thank you for the respond on the RT question [18:12:46] yw...lmk if you have any other questions [18:12:59] cmjohnson, yes i do [18:13:17] ok...ask away...taking the dog for a walk and I will get back to you in a few [18:13:53] cmjohnson, ok let me check if Rob is available [18:14:31] RobH, hey Rob [18:27:46] papaul: back what's up? [18:28:00] Ok [18:28:36] cmjohnson, it sames Rob is not available [18:32:50] cmjohnson, here is the part i did not understand in Rt i have received 5 boxes and i have the package slip but a ticket was not enter for this shipment. Do i create a new ticket for that saying i have received the shipments? [18:33:11] no, you there should be a procurement ticket [18:33:27] rob is not going to available today. I just happened to be doing something [18:33:33] so I am around [18:33:47] on the packing slip do you have a RT #? [18:34:36] Let me check [18:35:32] cmjohnson, yes i do [18:36:41] ok...so look that up in RT...as long as everything arrives add the packing slip and attach. For the juniper gear just comment the items that arrived and add the packing slips. Don't resolve yet [18:39:34] cmjohnson, ok [18:41:46] cmjohnson, when do you resolved the ticket? [18:42:23] when you receive the item...what is the RT# of one of them? [18:44:14] cmjohnson, 7077 [18:45:40] bad example...that is the juniper order. Just comment and add what did arrive [18:46:02] any other ticket you would just add the packing slip comment that everythign arrived and resolve the ticket. [18:47:15] cmjohnson, ok thanks for the clarification s i about to go work out i talk to you on monday [18:47:28] cmjohnson, have a great week end [18:47:36] okay...Germany plays @2pm your time today [18:49:15] cmjohnson, yes in our gym we have a tv there i wilk be watching the game [18:50:16] cool..have fun! [18:51:48] cmjohnson, thanks [18:57:24] OH MY DEAR LORD [18:57:40] I just realised gwicke is Gabriel Wicke [18:57:50] The author of the beloved Monobook skin [18:58:18] (Hi, gwicke!) [18:59:33] * twkozlowski apparently had an eureka moment [19:02:41] He is [19:02:51] I KNOW RIGHT [19:03:02] Not only that, but his wiki still has some ~2006 docs which are still better than those on mediawiki.org [19:08:27] (03CR) 10Withoutaname: [C: 031] Kill $wgEnableNewpagesUserFilter [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/141100 (https://bugzilla.wikimedia.org/58932) (owner: 10TTO) [20:24:47] (03PS2) 10Matanya: torrus: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/139786 [20:44:50] (03PS3) 10Matanya: torrus: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/139786 [20:45:56] (03PS4) 10Matanya: torrus: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/139786 [22:09:31] PROBLEM - Disk space on ms-be3003 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=94%):