[00:00:32] (03PS1) 10Dereckson: Add Əlavə namespace to az.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307108 (https://phabricator.wikimedia.org/T143851) [00:04:26] RECOVERY - puppet last run on oxygen is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [01:30:53] PROBLEM - MariaDB Slave Lag: s3 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 1278.64 seconds [01:36:43] (03CR) 1020after4: "this may be causing more problems than it solved? I'm not at all sure about this one: T144108" [puppet] - 10https://gerrit.wikimedia.org/r/284852 (https://phabricator.wikimedia.org/T132689) (owner: 1020after4) [01:39:04] TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333ERR444A ALVARO MOLINA.. JEM CASATE CONMIGO TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333ERR444A ALVARO MOLINA.. JEM CASATE CONMIGO TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333ERR444A ALVARO MOLINA.. JEM CASATE CONMIGO TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333ERR444A ALVARO MOLINA.. JEM CASATE CONMIGO TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P3 [01:39:05] P3 nagios-plugin-files - https://phabricator.wikimedia.org/P3 [01:45:35] RECOVERY - MariaDB Slave Lag: s3 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 0.78 seconds [02:24:04] !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.16) (duration: 11m 19s) [02:24:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:30:02] !log l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 28 02:30:02 UTC 2016 (duration 5m 58s) [02:30:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:38:30] 06Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests: Renaming the Aramaic (arc) Wikipedia to the Syriac (syc) Wikipedia - https://phabricator.wikimedia.org/T28725#2588980 (10MF-Warburg) >>! In T28725#2552717, @Liuxinyu970226 wrote: > Not sure if this is still needed or not, one user created [... [04:20:10] PROBLEM - Postgres Replication Lag on maps-test2002 is CRITICAL: CRITICAL - Rep Delay is: 1823.819469 Seconds [04:22:40] RECOVERY - Postgres Replication Lag on maps-test2002 is OK: OK - Rep Delay is: 0.0 Seconds [05:02:26] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [50.0] [05:04:55] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [05:18:16] (03PS1) 10BryanDavis: toollabs::static: Prune and gc git clone [puppet] - 10https://gerrit.wikimedia.org/r/307111 [05:19:27] (03CR) 10jenkins-bot: [V: 04-1] toollabs::static: Prune and gc git clone [puppet] - 10https://gerrit.wikimedia.org/r/307111 (owner: 10BryanDavis) [05:20:29] (03CR) 10BryanDavis: "Added ostriches as a reviewer because he may know if there are better ways to repack a long lived clone of an active upstream." [puppet] - 10https://gerrit.wikimedia.org/r/307111 (owner: 10BryanDavis) [05:21:10] (03PS2) 10BryanDavis: toollabs::static: Prune and gc git clone [puppet] - 10https://gerrit.wikimedia.org/r/307111 [06:36:11] PROBLEM - puppet last run on ms-be1020 is CRITICAL: CRITICAL: Puppet has 1 failures [07:00:51] PROBLEM - tools homepage -admin tool- on tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 531 bytes in 0.059 second response time [07:01:04] RECOVERY - puppet last run on ms-be1020 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:23:19] RECOVERY - tools homepage -admin tool- on tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.082 second response time [07:48:26] 06Operations, 10Beta-Cluster-Infrastructure, 07HHVM: Move the MW Beta appservers to Debian - https://phabricator.wikimedia.org/T144006#2589035 (10hashar) For beta I guess we can stick to have API and Web to be served by the same boxes. T125976 is about having the various maintenance script / cron to run o... [07:49:21] 06Operations, 10Beta-Cluster-Infrastructure, 07HHVM: Move the MW Beta appservers to Debian - https://phabricator.wikimedia.org/T144006#2589037 (10hashar) [08:49:39] (03CR) 10Hashar: "> Okay. So this is blocked on ops uploading the new package to apt.wm.o?" [puppet] - 10https://gerrit.wikimedia.org/r/307028 (owner: 10Thcipriani) [09:46:50] Hi. On? [11:18:38] (03CR) 10Alex Monk: "Why was it reopened instead of a new task being created?" [puppet] - 10https://gerrit.wikimedia.org/r/307028 (owner: 10Thcipriani) [11:50:00] 06Operations, 10MediaWiki-JobQueue: Restore 30 minutes delayed list update to no waiting, to stop killing sandbox functionality - https://phabricator.wikimedia.org/T139893#2589193 (10ManosHacker) [15:15:37] (03CR) 10Thcipriani: "> Why was it reopened instead of a new task being created?" [puppet] - 10https://gerrit.wikimedia.org/r/307028 (owner: 10Thcipriani) [15:42:12] PROBLEM - Host eeden is DOWN: PING CRITICAL - Packet loss = 100% [15:42:42] PROBLEM - Host ns2-v4 is DOWN: PING CRITICAL - Packet loss = 100% [15:51:23] RECOVERY - Host eeden is UP: PING OK - Packet loss = 0%, RTA = 84.40 ms [15:53:03] RECOVERY - Host ns2-v4 is UP: PING OK - Packet loss = 0%, RTA = 84.04 ms [16:17:19] TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333ERR444A ALVARO MOLINA.. JEM CASATE CONMIGO TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333ERR444A ALVARO MOLINA.. JEM CASATE CONMIGO TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333ERR444A ALVARO MOLINA.. JEM CASATE CONMIGO TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333ERR444A ALVARO MOLINA.. JEM CASATE CONMIGO TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333E [16:18:03] TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333ERR444A ALVARO MOLINA.. JEM CASATE CONMIGO TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333ERR444A ALVARO MOLINA.. JEM CASATE CONMIGO TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333ERR444A ALVARO MOLINA.. JEM CASATE CONMIGO TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333ERR444A ALVARO MOLINA.. JEM CASATE CONMIGO TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333E [16:18:04] TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333ERR444A ALVARO MOLINA.. JEM CASATE CONMIGO TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333ERR444A ALVARO MOLINA.. JEM CASATE CONMIGO TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333ERR444A ALVARO MOLINA.. JEM CASATE CONMIGO TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333ERR444A ALVARO MOLINA.. JEM CASATE CONMIGO TE MATARRRRRREE333E CON MIS PROPIAS MANOS HIJO DE P333E [16:31:29] PROBLEM - puppet last run on elastic2014 is CRITICAL: CRITICAL: Puppet has 1 failures [16:51:23] !log Ran T132839-Workarounds.sh from my home in terbium (see T132839) [16:51:25] T132839: Property suggester suggests human properties for non-human items - https://phabricator.wikimedia.org/T132839 [16:51:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:56:07] RECOVERY - puppet last run on elastic2014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:46:01] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: Puppet has 1 failures [18:10:55] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:13:22] ALVARO TE MATARE [18:29:05] (03PS1) 10Hashar: contint: stop including arcanist on Precise [puppet] - 10https://gerrit.wikimedia.org/r/307143 [18:36:16] (03CR) 10Hashar: "Cherry picked on CI puppetmaster" [puppet] - 10https://gerrit.wikimedia.org/r/307143 (owner: 10Hashar) [18:52:46] (03CR) 10Andrew Bogott: "new patch coming up..." (0312 comments) [puppet] - 10https://gerrit.wikimedia.org/r/294342 (https://phabricator.wikimedia.org/T91990) (owner: 10Andrew Bogott) [18:52:59] (03PS20) 10Andrew Bogott: Horizon tab for modifying instance puppet config [puppet] - 10https://gerrit.wikimedia.org/r/294342 (https://phabricator.wikimedia.org/T91990) [19:34:18] (03CR) 1020after4: [C: 031] contint: stop including arcanist on Precise [puppet] - 10https://gerrit.wikimedia.org/r/307143 (owner: 10Hashar) [19:54:18] (03CR) 10Hashar: "Thx! I have added it to next Tuesday Puppet SWAT." [puppet] - 10https://gerrit.wikimedia.org/r/307143 (owner: 10Hashar) [19:56:14] (03Abandoned) 10Hashar: nodepool: bump # of instances [puppet] - 10https://gerrit.wikimedia.org/r/285957 (https://phabricator.wikimedia.org/T133911) (owner: 10Hashar) [20:04:53] (03PS2) 10Andrew Bogott: Set up a root password for Labs instances [puppet] - 10https://gerrit.wikimedia.org/r/307086 (https://phabricator.wikimedia.org/T142531) [20:11:56] (03PS9) 10Hashar: Modification of Rakefile spec entry point [puppet] - 10https://gerrit.wikimedia.org/r/282484 (https://phabricator.wikimedia.org/T78342) (owner: 10Nicko) [20:52:10] Platonides: he is already klined [20:58:46] ok… [20:59:06] ALVARO MOLINA TE MATARE [21:00:22] ALVARO MOLINA TE MATARE [21:00:22] ALVARO MOLINA TE MATARE [21:00:23] ALVARO MOLINA TE MATARE [21:00:26] ALVARO MOLINA TE MATARE [21:00:27] ALVARO MOLINA TE MATARE [21:00:28] ALVARO MOLINA TE MATARE [21:00:58] Platonides: :-) [21:13:36] (03PS1) 10Hashar: Use task to run modules spec [puppet] - 10https://gerrit.wikimedia.org/r/307223 [21:14:18] (03CR) 10Hashar: "My ruby is pretty much a crap. I have been reading a bit more about rake tonight and found a potentially promising approach based on name" [puppet] - 10https://gerrit.wikimedia.org/r/282484 (https://phabricator.wikimedia.org/T78342) (owner: 10Nicko) [21:38:59] PROBLEM - Ensure mysql credential creation for tools users is running on labstore1001 is CRITICAL: CRITICAL - Expecting active but unit create-dbusers is failed [21:48:48] RECOVERY - Ensure mysql credential creation for tools users is running on labstore1001 is OK: OK - create-dbusers is active [22:11:56] (03CR) 10Alex Monk: Horizon tab for modifying instance puppet config (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/294342 (https://phabricator.wikimedia.org/T91990) (owner: 10Andrew Bogott) [22:18:54] (03CR) 10Andrew Bogott: Horizon tab for modifying instance puppet config (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/294342 (https://phabricator.wikimedia.org/T91990) (owner: 10Andrew Bogott) [22:20:04] (03PS21) 10Andrew Bogott: Horizon tab for modifying instance puppet config [puppet] - 10https://gerrit.wikimedia.org/r/294342 (https://phabricator.wikimedia.org/T91990) [22:29:28] error? [22:59:47] PROBLEM - puppet last run on mw2141 is CRITICAL: CRITICAL: puppet fail [23:27:00] RECOVERY - puppet last run on mw2141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:56:17] TE MATARE ALVARO MOLINA [23:56:18] TE MATARE ALVARO MOLINA [23:56:19] TE MATARE ALVARO MOLINA