[00:01:13] (03PS7) 10Dzahn: icinga/planet: add plugin to check planet content updates [puppet] - 10https://gerrit.wikimedia.org/r/472713 (https://phabricator.wikimedia.org/T203208) [00:06:12] (03CR) 10Dzahn: "https://puppet-compiler.wmflabs.org/compiler1002/13428/einsteinium.wikimedia.org/" [puppet] - 10https://gerrit.wikimedia.org/r/472713 (https://phabricator.wikimedia.org/T203208) (owner: 10Dzahn) [00:06:56] 10Operations: netbox won't allow me to upload photos of the rack - https://phabricator.wikimedia.org/T209182 (10Peachey88) [01:07:08] !log smalyshev@deploy1001 Started deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting [01:07:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:07:12] !log smalyshev@deploy1001 Finished deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting (duration: 00m 04s) [01:07:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:10:01] !log smalyshev@deploy1001 Started deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting [01:10:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:10:40] !log smalyshev@deploy1001 Finished deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting (duration: 00m 39s) [01:10:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:40:17] (03CR) 10Legoktm: icinga/planet: add plugin to check planet content updates (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/472713 (https://phabricator.wikimedia.org/T203208) (owner: 10Dzahn) [02:48:30] PROBLEM - puppet last run on labvirt1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:19:19] RECOVERY - puppet last run on labvirt1004 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [03:38:49] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 921.35 seconds [04:02:29] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 260.09 seconds [05:13:36] (03CR) 10Dzahn: "actually.. i will amend to it to turn it into a more generic "check_last_modified" that can be used for other pages with less "planet" har" [puppet] - 10https://gerrit.wikimedia.org/r/472713 (https://phabricator.wikimedia.org/T203208) (owner: 10Dzahn) [06:31:50] PROBLEM - puppet last run on analytics1061 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/R/biocLite.R] [06:32:49] PROBLEM - puppet last run on ms-be1027 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/vim/vimrc.local] [06:33:00] PROBLEM - puppet last run on mw1307 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/puppet-enabled] [06:57:29] RECOVERY - puppet last run on analytics1061 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:58:19] RECOVERY - puppet last run on ms-be1027 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:58:30] RECOVERY - puppet last run on mw1307 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:47:38] (03PS1) 10Elukey: profile::prometheus::alerts: avoid false positives for Druid [puppet] - 10https://gerrit.wikimedia.org/r/472762 [07:51:04] (03CR) 10Elukey: [C: 032] profile::prometheus::alerts: avoid false positives for Druid [puppet] - 10https://gerrit.wikimedia.org/r/472762 (owner: 10Elukey) [08:23:54] 10Operations, 10Gadgets, 10MediaWiki-Cache, 10MW-1.33-notes (1.33.0-wmf.4; 2018-11-13), and 4 others: Mcrouter periodically reports soft TKOs for mc[1,2]035 leading to MW Memcached exceptions - https://phabricator.wikimedia.org/T203786 (10elukey) Added two graphs to the memcached grafana dashboard that rep... [10:05:03] 10Operations, 10Domains, 10QRpedia-General, 10Traffic: qrpedia.org and qrwp.org are down - https://phabricator.wikimedia.org/T209019 (10Samwilson) p:05Triage>03High [12:04:13] ^^ those aren't us (WMF), are they? they use a different registrar than we do. [13:10:19] yeah they're not us, I updated the ticket [14:10:54] (03PS10) 10Lucas Werkmeister (WMDE): dump url shorteners for wiki projects [puppet] - 10https://gerrit.wikimedia.org/r/278400 (https://phabricator.wikimedia.org/T116986) (owner: 10ArielGlenn) [14:11:14] (03CR) 10Lucas Werkmeister (WMDE): "Rebased, just in case this ever gets off the ground again…" [puppet] - 10https://gerrit.wikimedia.org/r/278400 (https://phabricator.wikimedia.org/T116986) (owner: 10ArielGlenn) [14:12:59] bjornskald, your task is about to expire! [14:15:20] bjornskjald [15:09:16] PROBLEM - Router interfaces on cr2-eqiad is CRITICAL: CRITICAL: host 208.80.154.197, interfaces up: 229, down: 1, dormant: 0, excluded: 0, unused: 0: https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [16:49:38] (03PS1) 10Zoranzoki21: Upload HD logos for multiple projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/472791 (https://phabricator.wikimedia.org/T150618) [17:01:32] (03CR) 10Urbanecm: [C: 04-1] "ruwikivoyage-1.5x doesn't match https://en.wikipedia.org/static/images/project-logos/ruwikivoyage.png." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/472791 (https://phabricator.wikimedia.org/T150618) (owner: 10Zoranzoki21) [17:08:36] (03PS1) 10Zoranzoki21: Use HD logos in InitialiseSettings.php for multiple projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/472792 (https://phabricator.wikimedia.org/T150618) [17:19:34] (03PS2) 10Zoranzoki21: Upload HD logos for multiple projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/472791 (https://phabricator.wikimedia.org/T150618) [17:20:35] (03PS2) 10Zoranzoki21: Use HD logos in InitialiseSettings.php for multiple projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/472792 (https://phabricator.wikimedia.org/T150618) [17:26:46] (03CR) 10Urbanecm: [C: 031] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/472791 (https://phabricator.wikimedia.org/T150618) (owner: 10Zoranzoki21) [17:31:52] (03PS3) 10Zoranzoki21: Upload HD logos for multiple projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/472791 (https://phabricator.wikimedia.org/T150618) [17:33:02] (03PS3) 10Zoranzoki21: Use HD logos in InitialiseSettings.php for multiple projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/472792 (https://phabricator.wikimedia.org/T150618) [17:33:56] (03CR) 10jerkins-bot: [V: 04-1] Use HD logos in InitialiseSettings.php for multiple projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/472792 (https://phabricator.wikimedia.org/T150618) (owner: 10Zoranzoki21) [18:09:37] PROBLEM - puppet last run on labpuppetmaster1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:40:16] RECOVERY - puppet last run on labpuppetmaster1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [18:58:44] (03PS4) 10Zoranzoki21: Use HD logos in InitialiseSettings.php for multiple projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/472792 (https://phabricator.wikimedia.org/T150618) [19:04:59] (03CR) 10Urbanecm: [C: 031] Use HD logos in InitialiseSettings.php for multiple projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/472792 (https://phabricator.wikimedia.org/T150618) (owner: 10Zoranzoki21) [22:40:00] (03PS2) 10Zoranzoki21: Disable FlaggedRevs on srwikinews and enable RC patrol [mediawiki-config] - 10https://gerrit.wikimedia.org/r/472745 [22:48:26] (03CR) 10Peachey88: [C: 04-1] "Missing relevant task number in commit message" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/472745 (owner: 10Zoranzoki21)