[01:07:59] hmm icinga-wm has left. [01:08:05] and rejoined. [02:17:22] PROBLEM - Check health of redis instance on 6379 on rdb2001 is CRITICAL: CRITICAL ERROR - Redis Library - can not ping 127.0.0.1 on port 6379 [02:18:21] RECOVERY - Check health of redis instance on 6379 on rdb2001 is OK: OK: REDIS 2.8.17 on 127.0.0.1:6379 has 1 databases (db0) with 7843417 keys, up 5 minutes 34 seconds - replication_delay is 0 [02:20:13] PROBLEM - Check health of redis instance on 6479 on rdb2005 is CRITICAL: CRITICAL ERROR - Redis Library - can not ping 127.0.0.1 on port 6479 [02:21:12] RECOVERY - Check health of redis instance on 6479 on rdb2005 is OK: OK: REDIS 2.8.17 on 127.0.0.1:6479 has 1 databases (db0) with 3142893 keys, up 5 minutes 35 seconds - replication_delay is 0 [03:28:52] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 876.91 seconds [03:33:31] PROBLEM - puppet last run on analytics1053 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIPCity.dat.gz] [03:58:01] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 238.87 seconds [04:03:31] RECOVERY - puppet last run on analytics1053 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [04:19:48] robh: are you able to reconnect wikibugs? [09:26:01] (03Draft2) 10Biplab Anand: Update the project namespace in Nepali Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/404148 [09:29:46] (03PS3) 10Biplab Anand: Update the project namespace in Nepali Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/404148 (https://phabricator.wikimedia.org/T184865) [12:30:00] (03PS1) 10Jcrespo: Add Proxysql creation debian package script [software] - 10https://gerrit.wikimedia.org/r/404153 [12:44:58] (03PS1) 10Jcrespo: proxysql: Changes added (simplifications) to the proxysql class [puppet] - 10https://gerrit.wikimedia.org/r/404154 (https://phabricator.wikimedia.org/T171071) [12:45:40] (03CR) 10jerkins-bot: [V: 04-1] proxysql: Changes added (simplifications) to the proxysql class [puppet] - 10https://gerrit.wikimedia.org/r/404154 (https://phabricator.wikimedia.org/T171071) (owner: 10Jcrespo) [12:46:38] 10Operations, 10DBA, 10Performance-Team, 10Availability (Multiple-active-datacenters), 10Patch-For-Review: Perform testing for TLS effect on connection rate - https://phabricator.wikimedia.org/T171071#3899341 (10jcrespo) I have uploaded the proxysql 2.0 (still in developement) version to debian stretch,... [13:44:04] (03PS2) 10Jcrespo: Add Proxysql creation debian package script [software] - 10https://gerrit.wikimedia.org/r/404153 [14:06:14] 10Operations, 10Parsoid, 10VisualEditor, 10HTTPS: Parsoid, VisualEditor not working with SSL / HTTPS - https://phabricator.wikimedia.org/T178778#3702250 (10Seb35) @PlanetKrypton: did you try @Arlolra’s config? My rule of thumb about that is: put a domain on both sides or nowhere. But it seems your domain n... [15:41:31] (03PS1) 10Mholloway: Stop rewriting m.wikipedia.org and zero.wikipedia.org [puppet] - 10https://gerrit.wikimedia.org/r/404158 (https://phabricator.wikimedia.org/T69015) [15:42:50] (03CR) 10Giuseppe Lavagetto: "https://puppet-compiler.wmflabs.org/compiler02/9717/ this change is a noop." [puppet] - 10https://gerrit.wikimedia.org/r/402345 (owner: 10Giuseppe Lavagetto) [18:28:42] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:58:42] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:57:53] PROBLEM - Check Varnish expiry mailbox lag on cp4026 is CRITICAL: CRITICAL: expiry mailbox lag is 2027398 [22:05:05] (03PS1) 10Thcipriani: Scap: bump version to 3.7.6-1 [puppet] - 10https://gerrit.wikimedia.org/r/404219 (https://phabricator.wikimedia.org/T127762) [22:14:56] (03Draft1) 10Paladox: Fix gerrit support for latest scap version [software/gerrit] - 10https://gerrit.wikimedia.org/r/404221 [22:14:58] (03PS2) 10Paladox: Fix gerrit support for latest scap version [software/gerrit] - 10https://gerrit.wikimedia.org/r/404221 [22:15:12] (03PS3) 10Paladox: Fix gerrit support for latest scap version [software/gerrit] - 10https://gerrit.wikimedia.org/r/404221 (https://phabricator.wikimedia.org/T184882) [22:18:06] (03CR) 10Thcipriani: [C: 031] "Thanks paladox" [software/gerrit] - 10https://gerrit.wikimedia.org/r/404221 (https://phabricator.wikimedia.org/T184882) (owner: 10Paladox) [22:18:18] (03CR) 10Paladox: "Your welcome :)." [software/gerrit] - 10https://gerrit.wikimedia.org/r/404221 (https://phabricator.wikimedia.org/T184882) (owner: 10Paladox) [23:03:52] (03PS1) 10Thcipriani: Scap: git_fat -> git_binary_manager [software/logstash/plugins] - 10https://gerrit.wikimedia.org/r/404222 (https://phabricator.wikimedia.org/T184882) [23:05:00] (03PS1) 10Thcipriani: Scap: git_fat -> git_binary_manager [software/prometheus_jmx_exporter] - 10https://gerrit.wikimedia.org/r/404224 (https://phabricator.wikimedia.org/T184882) [23:05:58] (03PS1) 10Thcipriani: Scap: git_fat -> git_binary_manager [software/cassandra-metrics-collector] - 10https://gerrit.wikimedia.org/r/404226 (https://phabricator.wikimedia.org/T184882) [23:06:29] (03PS1) 10Thcipriani: Scap: git_fat -> git_binary_manager [software/logstash-logback-encoder] - 10https://gerrit.wikimedia.org/r/404227 (https://phabricator.wikimedia.org/T184882) [23:07:08] (03PS1) 10Thcipriani: Scap: git_fat -> git_binary_manager [software/cassandra-twcs] - 10https://gerrit.wikimedia.org/r/404228 (https://phabricator.wikimedia.org/T184882)