[00:00:05] 06Operations, 10Traffic: 404 loading images from Virgin Media - https://phabricator.wikimedia.org/T161360#3129726 (10Peachey88) p:05Triage>03High [00:03:05] (03PS1) 10Dzahn: deployment::server: convert to profile/role (pt. 1) [puppet] - 10https://gerrit.wikimedia.org/r/344728 [00:04:34] (03CR) 10jerkins-bot: [V: 04-1] deployment::server: convert to profile/role (pt. 1) [puppet] - 10https://gerrit.wikimedia.org/r/344728 (owner: 10Dzahn) [00:07:36] (03PS2) 10Dzahn: deployment::server: convert to profile/role (pt. 1) [puppet] - 10https://gerrit.wikimedia.org/r/344728 [00:16:54] RECOVERY - puppet last run on mw1298 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [00:17:25] (03PS3) 10Dzahn: mediawiki::maintenance: convert to profile/role (WIP) [puppet] - 10https://gerrit.wikimedia.org/r/342777 [00:25:48] (03PS1) 10Dzahn: url_downloader: convert to profile/role [puppet] - 10https://gerrit.wikimedia.org/r/344729 [00:26:41] (03PS2) 10Kaldari: Enable cookie blocking on English Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/324672 (https://phabricator.wikimedia.org/T152076) [00:34:17] (03CR) 10Zppix: [C: 031] Enable cookie blocking on English Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/324672 (https://phabricator.wikimedia.org/T152076) (owner: 10Kaldari) [00:38:52] (03CR) 10Krinkle: [C: 04-1] "Per https://phabricator.wikimedia.org/T161261#3127173 - better use a role instead so that access is associated with xhgui instead of tungs" [puppet] - 10https://gerrit.wikimedia.org/r/344531 (https://phabricator.wikimedia.org/T161261) (owner: 10Krinkle) [00:47:05] (03PS2) 10Dzahn: url_downloader: convert to profile/role [puppet] - 10https://gerrit.wikimedia.org/r/344729 [00:47:43] (03CR) 10Dzahn: "that would be hieradata/role/common/xhgui.yaml" [puppet] - 10https://gerrit.wikimedia.org/r/344531 (https://phabricator.wikimedia.org/T161261) (owner: 10Krinkle) [01:00:25] (03PS3) 10Dzahn: url_downloader: convert to profile/role [puppet] - 10https://gerrit.wikimedia.org/r/344729 [01:01:16] (03CR) 10Dzahn: [C: 031] varnish: remove varnish::monitoring::ganglia [puppet] - 10https://gerrit.wikimedia.org/r/337002 (owner: 10Ema) [01:02:06] (03CR) 10jerkins-bot: [V: 04-1] url_downloader: convert to profile/role [puppet] - 10https://gerrit.wikimedia.org/r/344729 (owner: 10Dzahn) [01:02:09] 06Operations, 06Performance-Team, 06Reading-Web-Backlog, 10Traffic, and 3 others: Performance review #2 of Hovercards (Popups extension) - https://phabricator.wikimedia.org/T70861#3129780 (10Krinkle) >>! In T70861#3129177, @Nirzar wrote: >>We're not proposing no delay, because if you're mousing across thin... [01:04:44] PROBLEM - puppet last run on ms-be1013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [01:08:05] (03PS4) 10Dzahn: url_downloader: convert to profile/role [puppet] - 10https://gerrit.wikimedia.org/r/344729 [01:10:34] PROBLEM - puppet last run on mw1273 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [01:12:27] (03CR) 10Dzahn: "no-op in compiler: http://puppet-compiler.wmflabs.org/5912/" [puppet] - 10https://gerrit.wikimedia.org/r/344729 (owner: 10Dzahn) [01:29:34] PROBLEM - Postgres Replication Lag on maps1003 is CRITICAL: CRITICAL - Rep Delay is: 83719.706493 Seconds [01:29:34] PROBLEM - Postgres Replication Lag on maps1004 is CRITICAL: CRITICAL - Rep Delay is: 83719.707611 Seconds [01:29:44] PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: CRITICAL - Rep Delay is: 80194.272446 Seconds [01:29:45] (03PS1) 10Dzahn: admins: add musikanimal to deployers [puppet] - 10https://gerrit.wikimedia.org/r/344734 (https://phabricator.wikimedia.org/T161181) [01:30:04] PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 80210.336685 Seconds [01:30:04] PROBLEM - Postgres Replication Lag on maps1002 is CRITICAL: CRITICAL - Rep Delay is: 83751.587511 Seconds [01:30:14] PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 80223.065728 Seconds [01:31:23] 06Operations, 10Ops-Access-Requests: Requesting access to maps servers for pnorman - https://phabricator.wikimedia.org/T161274#3127143 (10Dzahn) @RStallman-legalteam Thank you! I see we also have "expiry_date" and "expiry_contact" (email address) in our code for access for contractors and you are a contact fo... [01:32:04] RECOVERY - puppet last run on ms-be1013 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [01:37:47] (03PS1) 10Dzahn: admins: create shell account for Goran S. Milovanovic [puppet] - 10https://gerrit.wikimedia.org/r/344735 (https://phabricator.wikimedia.org/T160980) [01:38:34] RECOVERY - puppet last run on mw1273 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [01:40:04] PROBLEM - check_puppetrun on boron is CRITICAL: CRITICAL: Puppet has 1 failures [01:41:34] (03PS1) 10Dzahn: admins: add goransm to researchers, analytics-wmde, analytics-users [puppet] - 10https://gerrit.wikimedia.org/r/344736 (https://phabricator.wikimedia.org/T160980) [01:43:18] 06Operations, 10Ops-Access-Requests: Requesting access to maps servers for pnorman - https://phabricator.wikimedia.org/T161274#3129882 (10Dzahn) > - need email address to tie to @Pnorman's shell account. There is an existing wikitech user "pnorman" with a @mac.com email address. UID 16082. That's the right... [01:45:04] RECOVERY - check_puppetrun on boron is OK: OK: Puppet is currently enabled, last run 89 seconds ago with 0 failures [01:50:34] RECOVERY - Postgres Replication Lag on maps1003 is OK: OK - Rep Delay is: 0.0 Seconds [01:50:34] RECOVERY - Postgres Replication Lag on maps1004 is OK: OK - Rep Delay is: 0.0 Seconds [01:53:05] RECOVERY - Postgres Replication Lag on maps1002 is OK: OK - Rep Delay is: 0.0 Seconds [01:53:34] PROBLEM - Postgres Replication Lag on maps1004 is CRITICAL: CRITICAL - Rep Delay is: 85159.849699 Seconds [01:53:34] PROBLEM - Postgres Replication Lag on maps1003 is CRITICAL: CRITICAL - Rep Delay is: 85159.852062 Seconds [01:54:34] RECOVERY - Postgres Replication Lag on maps1004 is OK: OK - Rep Delay is: 30.488254 Seconds [01:54:34] RECOVERY - Postgres Replication Lag on maps1003 is OK: OK - Rep Delay is: 30.492046 Seconds [01:54:44] RECOVERY - Postgres Replication Lag on maps2003 is OK: OK - Rep Delay is: 30.156595 Seconds [01:55:04] RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 46.111313 Seconds [01:55:14] RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 58.883601 Seconds [02:08:44] PROBLEM - puppet last run on db1075 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:31:19] !log l10nupdate@tin scap sync-l10n completed (1.29.0-wmf.17) (duration: 12m 22s) [02:31:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:33:04] PROBLEM - puppet last run on mw1286 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:36:46] !log l10nupdate@tin ResourceLoader cache refresh completed at Sat Mar 25 02:36:46 UTC 2017 (duration 5m 27s) [02:36:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:37:44] RECOVERY - puppet last run on db1075 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [03:00:04] RECOVERY - puppet last run on mw1286 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [03:03:52] 06Operations, 05Goal: reduce amount of remaining Ubuntu 12.04 (precise) systems in production - https://phabricator.wikimedia.org/T123525#3130018 (10Krinkle) [03:03:54] 06Operations, 06Release-Engineering-Team, 05DC-Switchover-Prep-Q3-2016-17: Understand the preparedness of misc services for datacenter switchover - https://phabricator.wikimedia.org/T156937#3130017 (10Krinkle) [03:03:59] 06Operations, 13Patch-For-Review: replace fluorine with mwlog servers (was: Upgrade fluorine to trusty/jessie) - https://phabricator.wikimedia.org/T123728#3130015 (10Krinkle) 05Resolved>03Open @fgiunchedi All performance data in coal data from before Feb 16 seems lost. See: * !log Re-create optimised indexes for xhgui in mongodb on tungsten per https://github.com/perftools/xhgui/tree/v0.7.0#installation (lost after T161196) [03:15:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:15:16] T161196: tungsten is out of space on /srv - https://phabricator.wikimedia.org/T161196 [03:33:32] 06Operations, 10Ops-Access-Requests: Requesting access to maps servers for pnorman - https://phabricator.wikimedia.org/T161274#3130107 (10Pnorman) Yes, my email is penorman@mac.com [04:20:14] PROBLEM - mailman I/O stats on fermium is CRITICAL: CRITICAL - I/O stats: Transfers/Sec=331.80 Read Requests/Sec=3474.70 Write Requests/Sec=9.30 KBytes Read/Sec=22635.60 KBytes_Written/Sec=3386.40 [04:32:14] RECOVERY - mailman I/O stats on fermium is OK: OK - I/O stats: Transfers/Sec=26.20 Read Requests/Sec=53.60 Write Requests/Sec=92.70 KBytes Read/Sec=811.60 KBytes_Written/Sec=994.80 [05:00:44] PROBLEM - puppet last run on francium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [05:29:44] RECOVERY - puppet last run on francium is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [05:32:54] PROBLEM - puppet last run on elastic1028 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [06:00:54] RECOVERY - puppet last run on elastic1028 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:01:24] PROBLEM - Router interfaces on cr1-eqiad is CRITICAL: CRITICAL: host 208.80.154.196, interfaces up: 232, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-4/2/0: down - Core: cr1-codfw:xe-5/2/1 (Telia, IC-307235, 34ms) {#2648} [10Gbps wave]BR [06:15:24] PROBLEM - puppet last run on cp3010 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [06:31:24] RECOVERY - Router interfaces on cr1-eqiad is OK: OK: host 208.80.154.196, interfaces up: 234, down: 0, dormant: 0, excluded: 0, unused: 0 [06:35:24] PROBLEM - Router interfaces on cr1-eqiad is CRITICAL: CRITICAL: host 208.80.154.196, interfaces up: 232, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-4/2/0: down - Core: cr1-codfw:xe-5/2/1 (Telia, IC-307235, 34ms) {#2648} [10Gbps wave]BR [06:44:24] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:52:24] RECOVERY - Router interfaces on cr1-eqiad is OK: OK: host 208.80.154.196, interfaces up: 234, down: 0, dormant: 0, excluded: 0, unused: 0 [07:18:27] (03PS2) 10Giuseppe Lavagetto: Add missing discovery entries [puppet] - 10https://gerrit.wikimedia.org/r/344088 [07:47:34] PROBLEM - IPv6 ping to ulsfo on ripe-atlas-ulsfo is CRITICAL: CRITICAL - failed 20 probes of 278 (alerts on 19) - https://atlas.ripe.net/measurements/1791309/#!map [07:47:50] (03PS3) 10Giuseppe Lavagetto: Add new discovery entries [dns] - 10https://gerrit.wikimedia.org/r/344093 [07:48:07] (03CR) 10jerkins-bot: [V: 04-1] Add new discovery entries [dns] - 10https://gerrit.wikimedia.org/r/344093 (owner: 10Giuseppe Lavagetto) [07:51:15] PROBLEM - Check HHVM threads for leakage on mw1169 is CRITICAL: CRITICAL: HHVM has more than double threads running or queued than apache has busy workers [07:52:09] (03PS4) 10Giuseppe Lavagetto: Add new discovery entries [dns] - 10https://gerrit.wikimedia.org/r/344093 [07:52:34] RECOVERY - IPv6 ping to ulsfo on ripe-atlas-ulsfo is OK: OK - failed 13 probes of 278 (alerts on 19) - https://atlas.ripe.net/measurements/1791309/#!map [08:06:15] RECOVERY - Check HHVM threads for leakage on mw1169 is OK: OK [08:08:54] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:36:54] RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [08:57:44] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:05:04] PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: CRITICAL - Rep Delay is: 24744.888241 Seconds [09:05:05] PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 24744.89135 Seconds [09:05:14] PROBLEM - Postgres Replication Lag on maps2004 is CRITICAL: CRITICAL - Rep Delay is: 24753.688423 Seconds [09:08:15] RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds [09:08:24] RECOVERY - Postgres Replication Lag on maps2003 is OK: OK - Rep Delay is: 0.0 Seconds [09:08:24] RECOVERY - Postgres Replication Lag on maps2004 is OK: OK - Rep Delay is: 0.0 Seconds [09:25:54] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [10:10:24] PROBLEM - puppet last run on labstore1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:17:45] 07Puppet, 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Beta-Cluster-reproducible: New instance have broken puppet configuration when using puppetmaster standalone - https://phabricator.wikimedia.org/T148929#2736876 (10Ladsgroup) I tried to work around this bug by doing the ex... [10:29:19] 07Puppet, 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Beta-Cluster-reproducible: New instance have broken puppet configuration when using puppetmaster standalone - https://phabricator.wikimedia.org/T148929#3130632 (10Ladsgroup) After fighting with this for hours and thanks t... [10:31:51] 07Puppet, 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Beta-Cluster-reproducible: New instance have broken puppet configuration when using puppetmaster standalone - https://phabricator.wikimedia.org/T148929#3130633 (10Ladsgroup) I might be wrong and I haven't tested this hypo... [10:37:14] PROBLEM - puppet last run on ganeti1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:38:24] RECOVERY - puppet last run on labstore1004 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [10:48:12] 07Puppet, 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Beta-Cluster-reproducible: New instance have broken puppet configuration when using puppetmaster standalone - https://phabricator.wikimedia.org/T148929#3130652 (10Tarrow) I also suffered with this for a very long time. Ev... [11:02:24] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [11:05:14] RECOVERY - puppet last run on ganeti1003 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [11:31:24] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [11:42:54] PROBLEM - puppet last run on labsdb1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [12:09:54] RECOVERY - puppet last run on labsdb1009 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [12:43:14] PROBLEM - puppet last run on analytics1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:13:14] RECOVERY - puppet last run on analytics1001 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [13:32:32] Hi. I've an emergency throtlle rule to deploy (an event where people are currently stuck to create new accounts). [13:36:15] (03PS1) 10Dereckson: Add throttle rule for BordeauxJS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344785 (https://phabricator.wikimedia.org/T161402) [13:37:20] (03CR) 10Dereckson: [C: 032] "Emergency deploy for Unbreak now! (ongoing event)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344785 (https://phabricator.wikimedia.org/T161402) (owner: 10Dereckson) [13:38:44] (03Merged) 10jenkins-bot: Add throttle rule for BordeauxJS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344785 (https://phabricator.wikimedia.org/T161402) (owner: 10Dereckson) [13:38:54] (03CR) 10jenkins-bot: Add throttle rule for BordeauxJS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344785 (https://phabricator.wikimedia.org/T161402) (owner: 10Dereckson) [13:39:44] (03PS1) 10Dereckson: Allow to create account on wikidata for BordeauxJS event [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344786 (https://phabricator.wikimedia.org/T161402) [13:40:09] (03CR) 10Dereckson: [C: 032] "Emergency deploy for Unbreak now! (ongoing event)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344786 (https://phabricator.wikimedia.org/T161402) (owner: 10Dereckson) [13:40:14] PROBLEM - puppet last run on conf1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:41:08] (03Merged) 10jenkins-bot: Allow to create account on wikidata for BordeauxJS event [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344786 (https://phabricator.wikimedia.org/T161402) (owner: 10Dereckson) [13:41:17] (03CR) 10jenkins-bot: Allow to create account on wikidata for BordeauxJS event [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344786 (https://phabricator.wikimedia.org/T161402) (owner: 10Dereckson) [13:43:11] !log dereckson@tin Synchronized wmf-config/throttle.php: Add throttle rule for BordeauxJS (T161402) (duration: 00m 50s) [13:43:16] Done. [13:43:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:43:19] T161402: Throttle rule for BordeauxJS - https://phabricator.wikimedia.org/T161402 [13:52:13] (03PS3) 10Hashar: (WIP) compile authdns::config [puppet] - 10https://gerrit.wikimedia.org/r/343747 [13:52:25] (03CR) 10jerkins-bot: [V: 04-1] (WIP) compile authdns::config [puppet] - 10https://gerrit.wikimedia.org/r/343747 (owner: 10Hashar) [13:53:07] (03CR) 10Hashar: "Made it to run gdnsd checkconf. Though it fails to find geo-maps, probably a missing package on y local machine." [puppet] - 10https://gerrit.wikimedia.org/r/343747 (owner: 10Hashar) [14:08:14] RECOVERY - puppet last run on conf1003 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [14:29:34] PROBLEM - mediawiki-installation DSH group on mw2256 is CRITICAL: Host mw2256 is not in mediawiki-installation dsh group [15:02:55] (03PS2) 10Dereckson: Test for throttle rules: parameters logic [mediawiki-config] - 10https://gerrit.wikimedia.org/r/318301 [15:06:54] PROBLEM - puppet last run on prometheus1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:18:54] PROBLEM - puppet last run on dbproxy1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:34:57] (03PS1) 10Dereckson: Deploy Timeless skin to labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344788 (https://phabricator.wikimedia.org/T160643) [15:35:54] RECOVERY - puppet last run on prometheus1002 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [15:40:25] (03CR) 10Paladox: "Do we have to branch timeless?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344788 (https://phabricator.wikimedia.org/T160643) (owner: 10Dereckson) [15:45:02] (03CR) 10Dereckson: "Yes we've: https://gerrit.wikimedia.org/r/344789" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344788 (https://phabricator.wikimedia.org/T160643) (owner: 10Dereckson) [15:46:54] RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [15:53:59] (03CR) 10Paladox: [C: 031] "thanks." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344788 (https://phabricator.wikimedia.org/T160643) (owner: 10Dereckson) [16:56:14] PROBLEM - puppet last run on mw1226 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [17:24:14] RECOVERY - puppet last run on mw1226 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:07:54] PROBLEM - Disk space on ruthenium is CRITICAL: DISK CRITICAL - free space: / 1771 MB (3% inode=90%) [18:49:04] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:05:34] PROBLEM - puppet last run on mw2104 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:15:15] PROBLEM - puppet last run on ms-be2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:15:15] PROBLEM - puppet last run on rdb2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:18:05] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:24:24] PROBLEM - puppet last run on elastic2005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:25:34] PROBLEM - puppet last run on heze is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:28:12] (03PS1) 10Dereckson: Fix wgLogoHD keys [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344796 (https://phabricator.wikimedia.org/T161416) [19:30:24] PROBLEM - Check health of redis instance on 6479 on rdb2005 is CRITICAL: CRITICAL ERROR - Redis Library - can not ping 127.0.0.1 on port 6479 [19:31:24] RECOVERY - Check health of redis instance on 6479 on rdb2005 is OK: OK: REDIS 2.8.17 on 127.0.0.1:6479 has 1 databases (db0) with 3642972 keys, up 2 days 3 hours - replication_delay is 0 [19:32:54] PROBLEM - puppet last run on mc1034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:34:34] RECOVERY - puppet last run on mw2104 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [19:34:37] (03CR) 10Luke081515: [C: 031] Deploy Timeless skin to labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344788 (https://phabricator.wikimedia.org/T160643) (owner: 10Dereckson) [19:35:15] PROBLEM - puppet last run on db2019 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:43:14] RECOVERY - puppet last run on rdb2006 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:52:24] RECOVERY - puppet last run on elastic2005 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [19:53:34] PROBLEM - puppet last run on mw2091 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:53:34] RECOVERY - puppet last run on heze is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [19:58:42] (03CR) 10Urbanecm: [C: 031] "LGTM. Sorry if this is my mistake! BTW I think there should be some jenkins jobs that will catch this kind of typo. Should I create a task" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344796 (https://phabricator.wikimedia.org/T161416) (owner: 10Dereckson) [20:00:54] RECOVERY - puppet last run on mc1034 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:03:04] (03CR) 10Dereckson: "I'm preparing a test, I'll attach it too to T161416." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344796 (https://phabricator.wikimedia.org/T161416) (owner: 10Dereckson) [20:03:17] (03PS1) 10Dereckson: Test wgLogoHD keys [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344798 (https://phabricator.wikimedia.org/T161416) [20:04:14] RECOVERY - puppet last run on db2019 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [20:04:45] (03CR) 10Dereckson: "When IS is incorrect" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344798 (https://phabricator.wikimedia.org/T161416) (owner: 10Dereckson) [20:06:02] (03CR) 10Dereckson: "We can't use a dataProvider here by the way if we want to let PHPUnit work in normal workflow: wgLoad is triggered by setUp, but data prov" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344798 (https://phabricator.wikimedia.org/T161416) (owner: 10Dereckson) [20:08:22] !log Ran mwscript deleteEqualMessages.php on public wikis (T45917) - deleted 5 pages across 5 wikis [20:08:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:08:30] T45917: Delete all redundant "MediaWiki" pages for system messages - https://phabricator.wikimedia.org/T45917 [20:09:29] (03CR) 10Urbanecm: "I really can't do a code review, I don't understand the code." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344798 (https://phabricator.wikimedia.org/T161416) (owner: 10Dereckson) [20:10:54] RECOVERY - Disk space on ruthenium is OK: DISK OK [20:12:34] PROBLEM - puppet last run on elastic2009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:14:14] RECOVERY - puppet last run on ms-be2006 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [20:23:34] RECOVERY - puppet last run on mw2091 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [20:23:55] (03CR) 10Dereckson: "Some inline comments about this code by the way." (034 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344798 (https://phabricator.wikimedia.org/T161416) (owner: 10Dereckson) [20:27:14] PROBLEM - puppet last run on db2058 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:27:24] PROBLEM - puppet last run on mw2158 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:40:34] RECOVERY - puppet last run on elastic2009 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [20:41:34] PROBLEM - puppet last run on ms-be2037 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:47:34] PROBLEM - puppet last run on achernar is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:48:24] RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 18 minutes ago with 0 failures [20:53:34] PROBLEM - puppet last run on mw2186 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:55:14] RECOVERY - puppet last run on db2058 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [20:55:24] RECOVERY - puppet last run on mw2158 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [20:59:40] (03CR) 10Krinkle: [C: 031] Fix wgLogoHD keys [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344796 (https://phabricator.wikimedia.org/T161416) (owner: 10Dereckson) [21:03:24] PROBLEM - puppet last run on mw2122 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:03:24] PROBLEM - puppet last run on mw2226 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:10:34] RECOVERY - puppet last run on ms-be2037 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [21:16:14] PROBLEM - puppet last run on maps2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:21:34] RECOVERY - puppet last run on mw2186 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [21:25:24] PROBLEM - puppet last run on db2067 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:26:34] PROBLEM - puppet last run on achernar is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [21:26:42] 06Operations, 06Performance-Team: Consolidate performance website and related software - https://phabricator.wikimedia.org/T158837#3131234 (10Krinkle) [21:27:24] RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 27 minutes ago with 0 failures [21:31:24] RECOVERY - puppet last run on mw2122 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [21:32:24] RECOVERY - puppet last run on mw2226 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [21:40:08] (03CR) 10Isarra: [C: 031] Deploy Timeless skin to labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344788 (https://phabricator.wikimedia.org/T160643) (owner: 10Dereckson) [21:43:15] PROBLEM - puppet last run on ms-be2007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:43:15] Hi, im rececving this error on wikipedia [21:43:16] Failed to set referrer policy: The value 'origin-when-cross-origin' is not one of 'no-referrer', 'origin', 'no-referrer-when-downgrade', or 'unsafe-url'. Defaulting to 'no-referrer'. [21:43:23] Krinkle ^^ [21:43:30] Using mac os 10.12.3 and safari [21:44:15] RECOVERY - puppet last run on maps2002 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [21:52:24] PROBLEM - puppet last run on maps-test2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:53:24] RECOVERY - puppet last run on db2067 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [22:01:15] PROBLEM - puppet last run on sca2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:03:24] PROBLEM - puppet last run on ms-be2024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:10:14] RECOVERY - puppet last run on ms-be2007 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [22:12:34] PROBLEM - puppet last run on pc2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:13:14] PROBLEM - puppet last run on labsdb1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:14:22] (03CR) 10Harej: [C: 031] Deploy Timeless skin to labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344788 (https://phabricator.wikimedia.org/T160643) (owner: 10Dereckson) [22:21:35] (03CR) 10Bartosz Dziewoński: [C: 031] Deploy Timeless skin to labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344788 (https://phabricator.wikimedia.org/T160643) (owner: 10Dereckson) [22:25:07] (03CR) 10Legoktm: [C: 032] Deploy Timeless skin to labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344788 (https://phabricator.wikimedia.org/T160643) (owner: 10Dereckson) [22:26:15] (03Merged) 10jenkins-bot: Deploy Timeless skin to labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344788 (https://phabricator.wikimedia.org/T160643) (owner: 10Dereckson) [22:27:35] (03CR) 10jenkins-bot: Deploy Timeless skin to labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/344788 (https://phabricator.wikimedia.org/T160643) (owner: 10Dereckson) [22:28:06] !log legoktm@tin Synchronized wmf-config/: No-op labs only changes https://gerrit.wikimedia.org/r/#/c/344788/ (duration: 00m 52s) [22:28:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:30:15] RECOVERY - puppet last run on sca2003 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [22:31:30] Thanks legoktm [22:32:09] is it already enabled? can't find it at the options [22:32:18] or is there still one step to do? [22:32:24] RECOVERY - puppet last run on ms-be2024 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [22:33:55] Sagan: it reached beta cluster: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/7095/console [22:34:18] Sagan: you tested it at https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page? [22:34:27] no, deploymentwiki [22:34:34] PROBLEM - puppet last run on db2012 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:34:35] still have only 4 skins at my preferences [22:34:45] ah indeed I don't see it at https://en.wikipedia.beta.wmflabs.org/wiki/Special:Version [22:34:50] en shows the same [22:35:27] on deployment-tin.deployment-prep.eqiad.wmflabs skins/Timeless/skin.json is well available [22:37:15] PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:37:15] PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:38:09] dereckson@deployment-tin:/srv/mediawiki-staging$ mwscript eval.php enwiki [22:38:14] > print_r($wmgUseTimeless) [22:38:14] 1 [22:38:16] that's good too [22:38:38] It shows for me [22:38:39] ⧼skinname-timeless⧽ 0.6 (2cbcbdf) 22:48, 24 March 2017 [22:38:51] Is that a bug? [22:38:54] Issara ^^ [22:38:58] yup [22:39:01] now available too [22:39:05] https://en.wikipedia.beta.wmflabs.org/?useskin=timeless here we go [22:39:19] but apparently without the l10n [22:39:25] yeah [22:39:26] Yeh [22:39:58] legoktm: any idea how to refresh l10n on beta cluster? [22:40:14] (something like l10nupdate) [22:41:03] l10n done now [22:41:14] RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [22:41:15] it's apparently automatically handled too [22:41:18] yeah, works nice [22:41:34] RECOVERY - puppet last run on pc2004 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [22:41:50] Dereckson: everything is automatic [22:43:01] it takes like 5-30 min for everything to be fully deployed [22:43:12] Is the figure shown in the search bar correct? [22:43:24] PROBLEM - puppet last run on mw2183 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:43:24] PROBLEM - puppet last run on mw2227 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:44:03] paladox: at least it's coherent with https://en.wikipedia.beta.wmflabs.org/wiki/Special:Statistics [22:44:30] ah thanks. [22:44:31] So it's correct :) [22:45:40] I like the footer but could do with smaller text. [22:46:03] And the colour of the text could be different expecially text that is hyperlinked. [22:47:24] PROBLEM - puppet last run on elastic2033 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:48:24] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:50:05] PROBLEM - Start a job and verify on Trusty on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/grid/start/trusty - 185 bytes in 0.466 second response time [22:50:25] RECOVERY - puppet last run on maps-test2001 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [22:55:04] RECOVERY - Start a job and verify on Trusty on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 2.827 second response time [22:57:25] PROBLEM - puppet last run on mw2120 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:58:24] PROBLEM - puppet last run on restbase2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:58:25] PROBLEM - puppet last run on db2056 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:59:24] PROBLEM - puppet last run on mw2138 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:02:24] PROBLEM - puppet last run on ms-be2031 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:03:34] RECOVERY - puppet last run on db2012 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [23:06:15] RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:07:14] RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [23:11:24] RECOVERY - puppet last run on mw2183 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:13:24] RECOVERY - puppet last run on mw2227 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:13:24] PROBLEM - puppet last run on mw2253 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:16:24] PROBLEM - puppet last run on mw2134 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:16:24] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:25:34] PROBLEM - puppet last run on mw2234 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:26:15] PROBLEM - puppet last run on elastic2006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:26:24] RECOVERY - puppet last run on db2056 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [23:26:24] RECOVERY - puppet last run on mw2120 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:26:34] PROBLEM - puppet last run on mw2126 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:27:24] RECOVERY - puppet last run on restbase2006 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [23:28:24] RECOVERY - puppet last run on mw2138 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [23:30:24] RECOVERY - puppet last run on ms-be2031 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [23:39:24] PROBLEM - puppet last run on db2042 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:41:24] RECOVERY - puppet last run on mw2253 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:45:24] RECOVERY - puppet last run on mw2134 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [23:45:24] RECOVERY - puppet last run on elastic2033 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:48:24] PROBLEM - puppet last run on mc2023 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:49:24] PROBLEM - puppet last run on mw2168 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:49:24] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:54:16] RECOVERY - puppet last run on elastic2006 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:54:34] RECOVERY - puppet last run on mw2234 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [23:56:34] RECOVERY - puppet last run on mw2126 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [23:58:15] PROBLEM - puppet last run on mc2026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues