[00:00:15] PROBLEM - Kafka Broker Messages In Per Second on graphite1001 is CRITICAL Anomaly detected: 0 data above and 46 below the confidence bounds [00:01:46] (03PS2) 10Yuvipanda: mesos: Setup marathon properly [puppet] - 10https://gerrit.wikimedia.org/r/213189 [00:07:05] PROBLEM - Debian mirror in sync with upstream on carbon is CRITICAL: /srv/mirrors/debian is over 13 hours old. [00:08:45] RECOVERY - Debian mirror in sync with upstream on carbon is OK: /srv/mirrors/debian is over 0 hours old. [00:16:55] PROBLEM - Kafka Broker Messages In Per Second on graphite1001 is CRITICAL Anomaly detected: 0 data above and 45 below the confidence bounds [00:18:54] (03PS1) 10Yuvipanda: mesos: Setup a marathon event reciever on mesos proxy [puppet] - 10https://gerrit.wikimedia.org/r/213191 [00:20:06] (03PS2) 10Yuvipanda: mesos: Setup a marathon event reciever on mesos proxy [puppet] - 10https://gerrit.wikimedia.org/r/213191 [00:23:01] (03PS3) 10Yuvipanda: mesos: Setup a marathon event reciever on mesos proxy [puppet] - 10https://gerrit.wikimedia.org/r/213191 [00:25:19] (03PS4) 10Yuvipanda: mesos: Setup a marathon event reciever on mesos proxy [puppet] - 10https://gerrit.wikimedia.org/r/213191 [00:32:05] PROBLEM - Debian mirror in sync with upstream on carbon is CRITICAL: /srv/mirrors/debian is over 13 hours old. [00:32:10] (03PS5) 10Yuvipanda: mesos: Setup a marathon event reciever on mesos proxy [puppet] - 10https://gerrit.wikimedia.org/r/213191 [00:33:45] RECOVERY - Kafka Broker Messages In Per Second on graphite1001 is OK No anomaly detected [00:34:07] (03PS6) 10Yuvipanda: mesos: Setup a marathon event reciever on mesos proxy [puppet] - 10https://gerrit.wikimedia.org/r/213191 [00:35:25] RECOVERY - Debian mirror in sync with upstream on carbon is OK: /srv/mirrors/debian is over 0 hours old. [00:58:54] PROBLEM - Debian mirror in sync with upstream on carbon is CRITICAL: /srv/mirrors/debian is over 13 hours old. [01:02:24] RECOVERY - Debian mirror in sync with upstream on carbon is OK: /srv/mirrors/debian is over 0 hours old. [01:51:47] (03PS4) 10Yuvipanda: dynamicproxy: Add redundanturl dynamicproxy [puppet] - 10https://gerrit.wikimedia.org/r/212997 [01:51:49] (03PS7) 10Yuvipanda: mesos: Setup a marathon event reciever on mesos proxy [puppet] - 10https://gerrit.wikimedia.org/r/213191 [01:51:51] (03PS3) 10Yuvipanda: mesos: Add marathon master class [puppet] - 10https://gerrit.wikimedia.org/r/213189 [01:55:53] (03PS8) 10Yuvipanda: mesos: Setup a marathon event reciever on mesos proxy [puppet] - 10https://gerrit.wikimedia.org/r/213191 [02:04:25] (03PS9) 10Yuvipanda: mesos: Setup a marathon event reciever on mesos proxy [puppet] - 10https://gerrit.wikimedia.org/r/213191 [02:12:33] (03PS10) 10Yuvipanda: mesos: Setup a marathon event reciever on mesos proxy [puppet] - 10https://gerrit.wikimedia.org/r/213191 [02:12:35] (03PS4) 10Yuvipanda: mesos: Add marathon master class [puppet] - 10https://gerrit.wikimedia.org/r/213189 [02:16:31] (03PS5) 10Yuvipanda: dynamicproxy: Add redundanturl dynamicproxy [puppet] - 10https://gerrit.wikimedia.org/r/212997 (https://phabricator.wikimedia.org/T99923) [02:16:33] (03PS11) 10Yuvipanda: mesos: Setup a marathon event reciever on mesos proxy [puppet] - 10https://gerrit.wikimedia.org/r/213191 (https://phabricator.wikimedia.org/T99923) [02:16:35] (03PS5) 10Yuvipanda: mesos: Add marathon master class [puppet] - 10https://gerrit.wikimedia.org/r/213189 (https://phabricator.wikimedia.org/T99923) [02:29:29] !log l10nupdate Synchronized php-1.26wmf6/cache/l10n: (no message) (duration: 06m 34s) [02:29:40] Logged the message, Master [02:33:44] PROBLEM - are wikitech and wt-static in sync on silver is CRITICAL: wikitech-static CRIT - wikitech and wikitech-static out of sync (93353s 90000s) [02:34:27] !log LocalisationUpdate completed (1.26wmf6) at 2015-05-24 02:33:23+00:00 [02:34:33] Logged the message, Master [02:53:26] !log l10nupdate Synchronized php-1.26wmf7/cache/l10n: (no message) (duration: 06m 57s) [02:53:38] Logged the message, Master [02:58:20] !log LocalisationUpdate completed (1.26wmf7) at 2015-05-24 02:57:17+00:00 [02:58:26] Logged the message, Master [03:35:35] PROBLEM - puppet last run on mw1023 is CRITICAL Puppet has 1 failures [03:35:35] PROBLEM - puppet last run on es2007 is CRITICAL Puppet has 1 failures [03:50:45] PROBLEM - puppet last run on mw2015 is CRITICAL Puppet has 1 failures [03:52:25] RECOVERY - puppet last run on mw1023 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [03:52:25] RECOVERY - puppet last run on es2007 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [04:05:55] RECOVERY - puppet last run on mw2015 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:05] PROBLEM - puppet last run on labsdb1004 is CRITICAL Puppet has 3 failures [05:10:04] RECOVERY - are wikitech and wt-static in sync on silver is OK: wikitech-static OK - wikitech and wikitech-static in sync (10704 90000s) [05:20:34] PROBLEM - Router interfaces on cr2-codfw is CRITICAL host 208.80.153.193, interfaces up: 111, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-5/0/3: down - Core: pfw-codfw:xe-15/0/0 {#10901} [10Gbps DF]BR [05:22:14] RECOVERY - Router interfaces on cr2-codfw is OK host 208.80.153.193, interfaces up: 112, down: 0, dormant: 0, excluded: 0, unused: 0 [05:42:38] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun May 24 05:41:35 UTC 2015 (duration 41m 34s) [05:42:44] Logged the message, Master [06:26:45] PROBLEM - carbon-cache too many creates on graphite1001 is CRITICAL 1.69% of data above the critical threshold [1000.0] [06:31:45] PROBLEM - puppet last run on db1002 is CRITICAL Puppet has 1 failures [06:33:54] PROBLEM - puppet last run on mw1118 is CRITICAL Puppet has 1 failures [06:33:54] PROBLEM - puppet last run on elastic1027 is CRITICAL Puppet has 1 failures [06:33:55] PROBLEM - puppet last run on cp3042 is CRITICAL Puppet has 1 failures [06:35:05] PROBLEM - puppet last run on mw2093 is CRITICAL Puppet has 1 failures [06:40:16] PROBLEM - Disk space on graphite2001 is CRITICAL: DISK CRITICAL - free space: /var/lib/carbon 36780 MB (3% inode=99%) [06:45:15] RECOVERY - puppet last run on db1002 is OK Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:45:44] RECOVERY - puppet last run on elastic1027 is OK Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:46:56] RECOVERY - puppet last run on mw2093 is OK Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:47:25] RECOVERY - puppet last run on mw1118 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:35] RECOVERY - puppet last run on cp3042 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:52:24] PROBLEM - Debian mirror in sync with upstream on carbon is CRITICAL: /srv/mirrors/debian is over 19 hours old. [06:54:04] RECOVERY - Debian mirror in sync with upstream on carbon is OK: /srv/mirrors/debian is over 0 hours old. [07:17:35] PROBLEM - Debian mirror in sync with upstream on carbon is CRITICAL: /srv/mirrors/debian is over 20 hours old. [07:19:15] RECOVERY - Debian mirror in sync with upstream on carbon is OK: /srv/mirrors/debian is over 0 hours old. [08:00:14] RECOVERY - carbon-cache too many creates on graphite1001 is OK Less than 1.00% above the threshold [500.0] [08:09:15] RECOVERY - carbon-cache write error on graphite1001 is OK Less than 1.00% above the threshold [1.0] [08:50:15] PROBLEM - puppet last run on mw1003 is CRITICAL Puppet has 1 failures [08:55:34] !log resize existing whisper files with new retention on graphite2001 [08:55:42] Logged the message, Master [09:05:35] RECOVERY - puppet last run on mw1003 is OK Puppet is currently enabled, last run 41 seconds ago with 0 failures [09:15:26] 10Ops-Access-Requests, 6operations, 5Patch-For-Review: Shell and research access for Moushira Elamrawy - https://phabricator.wikimedia.org/T100091#1305120 (10Rdicerb) (I was told by @Ori that a token could count as approval) [09:24:45] 10Ops-Access-Requests, 6operations, 5Patch-For-Review: Shell and research access for Moushira Elamrawy - https://phabricator.wikimedia.org/T100091#1306351 (10Krenair) Who would need to sign off on such an exception to the process from ops? Mark? [09:25:06] 6operations, 10Wikimedia-Git-or-Gerrit: git.wikimedia.org replication from gerrit stopped or lags - https://phabricator.wikimedia.org/T99990#1306353 (10QChris) Beginning at 2015-05-21 15:47 Gerrit's replication logs are full of errors like ``` org.eclipse.jgit.errors.TransportException: [...]/gerrit/mediawiki... [09:31:01] (03CR) 10Chad: [C: 032] Unit test to verify the existence of the proper branch symlinks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/212753 (https://phabricator.wikimedia.org/T99886) (owner: 1020after4) [09:31:09] (03Merged) 10jenkins-bot: Unit test to verify the existence of the proper branch symlinks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/212753 (https://phabricator.wikimedia.org/T99886) (owner: 1020after4) [09:31:39] (03PS3) 10EBernhardson: Only return mostly fresh data for elasticsearch ganglia monitoring [puppet] - 10https://gerrit.wikimedia.org/r/212322 [09:41:42] (03PS1) 10QChris: Turn off sshd MAC and KEX hardening for gerrit replication targets [puppet] - 10https://gerrit.wikimedia.org/r/213216 (https://phabricator.wikimedia.org/T99990) [09:43:59] (03CR) 10QChris: Turn off sshd MAC and KEX hardening for gerrit replication targets (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/213216 (https://phabricator.wikimedia.org/T99990) (owner: 10QChris) [09:46:42] (03CR) 10Paladox: [C: 031] Turn off sshd MAC and KEX hardening for gerrit replication targets [puppet] - 10https://gerrit.wikimedia.org/r/213216 (https://phabricator.wikimedia.org/T99990) (owner: 10QChris) [09:47:14] 6operations, 6Commons, 6Multimedia, 7HHVM, and 4 others: Convert Imagescalers to HHVM, Trusty - https://phabricator.wikimedia.org/T84842#1306421 (10greg) ``` greg moved this task to This week: May 4-8 on the Roadmap workboard. greg moved this task to This week: May 11-15 on the Roadmap workboard. greg move... [09:48:15] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: There are 2 unmerged changes in mediawiki_config (dir /srv/mediawiki-staging/). [09:48:54] PROBLEM - puppet last run on carbon is CRITICAL Puppet last ran 2 days ago [09:49:08] _joe_: I was able to build php-luasandbox without this: https://gerrit.wikimedia.org/r/#/c/212789/3 [09:49:17] I suppose it wont hurt though [09:59:34] (03PS1) 10Ori.livneh: [UNTESTED] Use INotify to watch for configuration file changes [debs/pybal] - 10https://gerrit.wikimedia.org/r/213223 [10:00:44] <^d> !log gerrit: manually gc'd all repos to help with clone times [10:00:44] PROBLEM - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 804.408591024 [10:00:49] Logged the message, Master [10:02:14] (03CR) 10Ori.livneh: "The test suites pass, so nothing is obviously broken, but I am not sure how to test this." [debs/pybal] - 10https://gerrit.wikimedia.org/r/213223 (owner: 10Ori.livneh) [10:04:15] RECOVERY - puppet last run on carbon is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [10:06:48] 6operations, 6Commons, 6Multimedia, 7HHVM, and 4 others: Convert Imagescalers to HHVM, Trusty - https://phabricator.wikimedia.org/T84842#1306452 (10ori) >>! In T84842#1306421, @greg wrote: > What ya'll think now? :) We should probably not schedule this until we resolve the issue Giuseppe spotted in T84842... [10:11:08] (03PS20) 10Paladox: Adding task support instead of using Bug: which was for bugzilla [puppet] - 10https://gerrit.wikimedia.org/r/209741 [10:21:29] 6operations, 10Wikimedia-Git-or-Gerrit, 5Patch-For-Review: git.wikimedia.org replication from gerrit stopped or lags - https://phabricator.wikimedia.org/T99990#1306479 (10Paladox) Also branch on http://git.wikimedia.org/summary/operations%2Fpuppet for master needs to be changed to production or production ne... [10:25:24] (03PS1) 10Ori.livneh: mediawiki: Touch /etc/wikimedia-image-scaler on image scalers [puppet] - 10https://gerrit.wikimedia.org/r/213228 (https://phabricator.wikimedia.org/T84842) [10:25:48] (03PS2) 10Ori.livneh: mediawiki: Touch /etc/wikimedia-image-scaler on image scalers [puppet] - 10https://gerrit.wikimedia.org/r/213228 (https://phabricator.wikimedia.org/T84842) [10:26:22] (03PS3) 10Ori.livneh: mediawiki: Touch /etc/wikimedia-image-scaler on image scalers [puppet] - 10https://gerrit.wikimedia.org/r/213228 (https://phabricator.wikimedia.org/T84842) [10:26:40] (03CR) 10Ori.livneh: [C: 032 V: 032] mediawiki: Touch /etc/wikimedia-image-scaler on image scalers [puppet] - 10https://gerrit.wikimedia.org/r/213228 (https://phabricator.wikimedia.org/T84842) (owner: 10Ori.livneh) [10:42:28] (03PS2) 10Paladox: Turn off sshd MAC and KEX hardening for gerrit replication targets [puppet] - 10https://gerrit.wikimedia.org/r/213216 (https://phabricator.wikimedia.org/T99990) (owner: 10QChris) [11:28:23] (03PS12) 10Yuvipanda: mesos: Setup a marathon event reciever on mesos proxy [puppet] - 10https://gerrit.wikimedia.org/r/213191 (https://phabricator.wikimedia.org/T99923) [11:31:45] (03PS13) 10Yuvipanda: mesos: Setup a marathon event reciever on mesos proxy [puppet] - 10https://gerrit.wikimedia.org/r/213191 (https://phabricator.wikimedia.org/T99923) [11:32:30] (03PS2) 10Faidon Liambotis: Add moushira to bastion-only and researchers [puppet] - 10https://gerrit.wikimedia.org/r/212946 (https://phabricator.wikimedia.org/T100091) (owner: 10Ori.livneh) [11:32:37] (03PS3) 10Faidon Liambotis: Add moushira to bastion-only and researchers [puppet] - 10https://gerrit.wikimedia.org/r/212946 (https://phabricator.wikimedia.org/T100091) (owner: 10Ori.livneh) [11:39:56] (03PS4) 10Faidon Liambotis: Add moushira to bastion-only and researchers [puppet] - 10https://gerrit.wikimedia.org/r/212946 (https://phabricator.wikimedia.org/T100091) (owner: 10Ori.livneh) [11:40:02] ori: change was wrong [11:40:09] 6operations, 10Wikimedia-Git-or-Gerrit, 5Patch-For-Review: git.wikimedia.org replication from gerrit stopped or lags - https://phabricator.wikimedia.org/T99990#1306635 (10demon) >>! In T99990#1306479, @Paladox wrote: > Also branch on http://git.wikimedia.org/summary/operations%2Fpuppet for master needs to be... [11:40:28] paravoid: how do you determine the next available uid? [11:40:46] aren't they supposed to match labs uids? [11:40:46] (03CR) 10Faidon Liambotis: [C: 032] Add moushira to bastion-only and researchers [puppet] - 10https://gerrit.wikimedia.org/r/212946 (https://phabricator.wikimedia.org/T100091) (owner: 10Ori.livneh) [11:40:54] oh, is that so? [11:40:57] yeah [11:41:04] ldap [11:41:10] paravoid: sorry, didn't know that. or did, but forgot. [11:41:11] yes, Krenair is right [11:41:17] no worries [11:41:19] paravoid, so... we're just going ahead with that change without the 3 day thing? [11:41:27] yes [11:41:33] says... you? [11:41:36] yes [11:41:57] I'm making an exception because of the on-going hackathon [11:42:07] (03PS14) 10Yuvipanda: mesos: Setup a marathon event reciever on mesos proxy [puppet] - 10https://gerrit.wikimedia.org/r/213191 (https://phabricator.wikimedia.org/T99923) [11:42:23] YuviPanda: receiver [11:42:41] Krenair: why are you asking? do you find this problematic? [11:43:02] paravoid: haha! :) [11:43:14] That's one of the words I keep fucking up [11:43:18] I'm just surprised it's allowed [11:43:26] 'i' before 'e' except after 'c' [11:44:35] (03CR) 10JanZerebecki: [C: 031] "For now that is the easiest option." [puppet] - 10https://gerrit.wikimedia.org/r/213216 (https://phabricator.wikimedia.org/T99990) (owner: 10QChris) [11:44:49] paravoid: thanks! [11:45:41] (03PS15) 10Yuvipanda: mesos: Setup a marathon event receiver on mesos proxy [puppet] - 10https://gerrit.wikimedia.org/r/213191 (https://phabricator.wikimedia.org/T99923) [11:46:08] (03CR) 10Paladox: [C: 031] Turn off sshd MAC and KEX hardening for gerrit replication targets [puppet] - 10https://gerrit.wikimedia.org/r/213216 (https://phabricator.wikimedia.org/T99990) (owner: 10QChris) [11:46:26] (03PS3) 10Paladox: Turn off sshd MAC and KEX hardening for gerrit replication targets [puppet] - 10https://gerrit.wikimedia.org/r/213216 (https://phabricator.wikimedia.org/T99990) (owner: 10QChris) [11:54:16] RECOVERY - Disk space on graphite2001 is OK: DISK OK [12:14:45] RECOVERY - puppet last run on labsdb1004 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [12:16:49] (03PS1) 10Dereckson: National Heritage Day Santiago Editatón throttle rule [mediawiki-config] - 10https://gerrit.wikimedia.org/r/213257 (https://phabricator.wikimedia.org/T100051) [12:38:01] _joe_: https://gerrit.wikimedia.org/r/#/c/213216/ may be up your alley [12:42:56] Krinkle: there's a 5pm session in the main auditorium called 'everything wrong with toollabs'. you should come :) [12:42:56] <_joe_> ori: I think it's more moritz's [12:45:11] (03Abandoned) 10Glaisher: Enable "Form Refresh" as a BetaFeature [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175406 (https://phabricator.wikimedia.org/T73477) (owner: 10Glaisher) [12:47:18] ori, _joe_: I'll look into it tomorrow [12:47:27] \o/ [12:49:15] moritzm: I also disabled the new stuff for toollabs, mostly so we can deal with it on a non hackathon weekend :) [12:52:05] <_joe_> moritzm: thanks a lot [12:52:33] <_joe_> (translating from yuvi: we can deal with it when users are not physically complaining with him) [13:06:57] YuviKTM: OK [13:26:18] (03PS1) 10coren: Tool Labs: install calibre for wsexport [puppet] - 10https://gerrit.wikimedia.org/r/213272 (https://phabricator.wikimedia.org/T100165) [13:37:39] (03CR) 10coren: [C: 032] "Trivial package addition to tools" [puppet] - 10https://gerrit.wikimedia.org/r/213272 (https://phabricator.wikimedia.org/T100165) (owner: 10coren) [13:51:05] (03PS4) 10Paladox: Turn off sshd MAC and KEX hardening for gerrit replication targets [puppet] - 10https://gerrit.wikimedia.org/r/213216 (https://phabricator.wikimedia.org/T99990) (owner: 10QChris) [13:53:37] (03PS6) 10Andrew Bogott: Revive the old ceph module [puppet] - 10https://gerrit.wikimedia.org/r/212914 [13:53:39] (03PS2) 10Andrew Bogott: Revert "Remove role::ceph::*, unused now" [puppet] - 10https://gerrit.wikimedia.org/r/212938 [13:55:11] (03CR) 10Andrew Bogott: [C: 04-2] "this probably won't be merged since it only solves half the problem (communication between instances /with/ floating IPs) which is potenti" [puppet] - 10https://gerrit.wikimedia.org/r/210720 (owner: 10Andrew Bogott) [14:30:21] (03PS1) 10Ori.livneh: varnish: add Python library for iterating on log records [puppet] - 10https://gerrit.wikimedia.org/r/213293 [15:00:20] akosiaris, hi, we are running into a problem with the osmdb.eqiad.wmnet - it seems the gis database was created with ASCII encoding as oppose to UTF 8 [15:00:22] could you check? [15:00:58] akosiaris, also, is it possible to re-import that database? it would be great if postgis is updated to version 2+ [15:14:35] (03CR) 10JanZerebecki: "Please stop rebasing, that serves no pupose but notifies me everytime." [puppet] - 10https://gerrit.wikimedia.org/r/213216 (https://phabricator.wikimedia.org/T99990) (owner: 10QChris) [15:40:36] PROBLEM - puppet last run on sca1001 is CRITICAL puppet fail [15:52:18] (03PS4) 10Gergő Tisza: Basic role for Sentry [puppet] - 10https://gerrit.wikimedia.org/r/199598 (https://phabricator.wikimedia.org/T84956) (owner: 10Gilles) [15:56:55] (03CR) 10Paladox: "Ok Sorry I will let the author an reviewer do it from now on." [puppet] - 10https://gerrit.wikimedia.org/r/213216 (https://phabricator.wikimedia.org/T99990) (owner: 10QChris) [16:09:54] <_joe_> /go YuviKTM [16:09:58] <_joe_> meh [16:16:15] RECOVERY - puppet last run on sca1001 is OK Puppet is currently enabled, last run 55 seconds ago with 0 failures [16:21:28] (03PS5) 10Gergő Tisza: Basic role for Sentry [puppet] - 10https://gerrit.wikimedia.org/r/199598 (https://phabricator.wikimedia.org/T84956) (owner: 10Gilles) [16:30:18] jouncebot_: next [16:30:18] In 22 hour(s) and 29 minute(s): Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150525T1500) [16:57:35] PROBLEM - Disk space on db1005 is CRITICAL: DISK CRITICAL - free space: /a 36798 MB (3% inode=99%) [17:06:14] RECOVERY - Disk space on db1005 is OK: DISK OK [17:18:07] !log stop mysqld db1002 db1003 db1004 db1005 db1006 db1007 [17:18:11] Logged the message, Master [17:58:24] PROBLEM - puppet last run on mw2070 is CRITICAL puppet fail [18:06:45] PROBLEM - Debian mirror in sync with upstream on carbon is CRITICAL: /srv/mirrors/debian is over 31 hours old. [18:10:04] RECOVERY - Debian mirror in sync with upstream on carbon is OK: /srv/mirrors/debian is over 0 hours old. [18:16:56] RECOVERY - puppet last run on mw2070 is OK Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:18:35] I can't access stat1001/1002/1003 (is this the right channel for such issues?) [18:19:29] this is the right channel I think [18:19:40] Though I saw you posted in -tech as well [18:19:41] I can't access stat1001/2/3 (via Putty) anything wrong? [18:19:45] via putty? [18:19:52] you're using windows or something ezachte_? [18:19:53] yes, via putty [18:20:27] what version of putty> [18:20:28] ? [18:22:14] 0.62 (I do same procedure every day) [18:24:09] try the latest version [18:24:31] valhallasw, is <= 0.62 known to be broken or is it really < 0.62? [18:30:10] yeah, 0.64 works, thanks :-) [18:33:44] PROBLEM - Debian mirror in sync with upstream on carbon is CRITICAL: /srv/mirrors/debian is over 31 hours old. [18:35:16] RECOVERY - Debian mirror in sync with upstream on carbon is OK: /srv/mirrors/debian is over 0 hours old. [18:47:19] ezachte_: your client update was needed due to recent changes to accepted SSH ciphers (for improved security) [18:47:48] glad to hear 0.64 works [19:30:00] (03PS6) 10Gergő Tisza: Basic role for Sentry [puppet] - 10https://gerrit.wikimedia.org/r/199598 (https://phabricator.wikimedia.org/T84956) (owner: 10Gilles) [19:42:30] 6operations: Backport and include linux-tools-3.19 to our jessie repository - https://phabricator.wikimedia.org/T100216#1307939 (10faidon) 3NEW [20:00:45] PROBLEM - puppet last run on sca1001 is CRITICAL Puppet has 10 failures [20:12:45] (03PS6) 10Yuvipanda: dynamicproxy: Add redundanturl dynamicproxy [puppet] - 10https://gerrit.wikimedia.org/r/212997 (https://phabricator.wikimedia.org/T99923) [20:12:55] (03CR) 10Yuvipanda: [C: 032 V: 032] dynamicproxy: Add redundanturl dynamicproxy [puppet] - 10https://gerrit.wikimedia.org/r/212997 (https://phabricator.wikimedia.org/T99923) (owner: 10Yuvipanda) [20:13:05] (03PS6) 10Yuvipanda: mesos: Add marathon master class [puppet] - 10https://gerrit.wikimedia.org/r/213189 (https://phabricator.wikimedia.org/T99923) [20:13:12] (03CR) 10Yuvipanda: [C: 032 V: 032] mesos: Add marathon master class [puppet] - 10https://gerrit.wikimedia.org/r/213189 (https://phabricator.wikimedia.org/T99923) (owner: 10Yuvipanda) [20:13:23] (03PS16) 10Yuvipanda: mesos: Setup a marathon event receiver on mesos proxy [puppet] - 10https://gerrit.wikimedia.org/r/213191 (https://phabricator.wikimedia.org/T99923) [20:13:30] (03CR) 10Yuvipanda: [C: 032 V: 032] mesos: Setup a marathon event receiver on mesos proxy [puppet] - 10https://gerrit.wikimedia.org/r/213191 (https://phabricator.wikimedia.org/T99923) (owner: 10Yuvipanda) [20:30:45] (03PS1) 10Yuvipanda: ores: Initial module, with web class / role [puppet] - 10https://gerrit.wikimedia.org/r/213354 [20:36:15] RECOVERY - puppet last run on sca1001 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [23:59:04] PROBLEM - puppet last run on sca1001 is CRITICAL puppet fail