[00:04:36] PROBLEM - jmxtrans on analytics1018 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, regex args -jar.+jmxtrans-all.jar [00:04:56] PROBLEM - jmxtrans on analytics1022 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, regex args -jar.+jmxtrans-all.jar [00:05:06] PROBLEM - jmxtrans on analytics1012 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, regex args -jar.+jmxtrans-all.jar [00:05:26] PROBLEM - jmxtrans on analytics1021 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, regex args -jar.+jmxtrans-all.jar [00:08:26] RECOVERY - jmxtrans on analytics1022 is OK: PROCS OK: 1 process with command name java, regex args -jar.+jmxtrans-all.jar [00:09:24] 6operations, 10SEO: GWT accounts - https://phabricator.wikimedia.org/T103567#1407748 (10Wwes) Can we fast track this please as we are seeing all sorts of movement on our rankings. I am unable to request this. I just need the https versions of the sites turned on please. [00:09:42] 6operations, 10SEO: GWT accounts - https://phabricator.wikimedia.org/T103567#1407750 (10Wwes) p:5Normal>3High [00:15:05] RECOVERY - jmxtrans on analytics1018 is OK: PROCS OK: 1 process with command name java, regex args -jar.+jmxtrans-all.jar [00:17:26] RECOVERY - jmxtrans on analytics1012 is OK: PROCS OK: 1 process with command name java, regex args -jar.+jmxtrans-all.jar [00:17:45] RECOVERY - jmxtrans on analytics1021 is OK: PROCS OK: 1 process with command name java, regex args -jar.+jmxtrans-all.jar [00:23:56] PROBLEM - puppet last run on mw1138 is CRITICAL Puppet has 1 failures [00:35:28] 6operations, 10SEO: GWT accounts - https://phabricator.wikimedia.org/T103567#1407780 (10Wwes) Please also double check @stu and @ori accounts to make sure they have same access. Thanks. [00:38:26] PROBLEM - Host mw2027 is DOWN: PING CRITICAL - Packet loss = 100% [00:40:05] RECOVERY - Host mw2027 is UPING OK - Packet loss = 0%, RTA = 44.70 ms [00:41:36] RECOVERY - puppet last run on mw1138 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [01:03:16] PROBLEM - nutcracker port on silver is CRITICAL - Socket timeout after 2 seconds [01:06:45] RECOVERY - nutcracker port on silver is OK: TCP OK - 0.000 second response time on port 11212 [02:17:48] !log l10nupdate Synchronized php-1.26wmf11/cache/l10n: (no message) (duration: 05m 56s) [02:17:57] Logged the message, Master [02:20:52] !log LocalisationUpdate completed (1.26wmf11) at 2015-06-28 02:20:52+00:00 [02:20:59] Logged the message, Master [02:27:40] 6operations, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, 6Multimedia, and 6 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1407929 (10Bawolff) I was recently playing around with amazon's cloud stuff. And I tried installing mediawiki. Much to my... [02:30:26] PROBLEM - Disk space on analytics1021 is CRITICAL: DISK CRITICAL - free space: / 1060 MB (3% inode=91%) [03:11:22] 6operations, 10Traffic, 7HTTPS, 5HTTPS-by-default, 5Patch-For-Review: Switch to ECDSA hybrid certificates - https://phabricator.wikimedia.org/T86654#1407938 (10BBlack) >>! In T86654#1407738, @JanZerebecki wrote: >>>! In T86654#1405484, @BBlack wrote: >> On security issues, I lean towards thinking that EC... [03:15:52] 6operations, 10Traffic, 7HTTPS, 5HTTPS-by-default, 5Patch-For-Review: Switch to ECDSA hybrid certificates - https://phabricator.wikimedia.org/T86654#1407941 (10BBlack) Also, after writing all of the above, I feel it's important to reference xkcd's wisdom on this topic as well as a counterpoint: https://x... [03:34:36] PROBLEM - puppet last run on wtp1002 is CRITICAL Puppet has 1 failures [03:35:26] PROBLEM - puppet last run on mw1146 is CRITICAL Puppet has 2 failures [03:35:36] PROBLEM - puppet last run on mw1057 is CRITICAL Puppet has 1 failures [03:36:36] PROBLEM - puppet last run on mw1210 is CRITICAL Puppet has 1 failures [03:38:56] PROBLEM - puppet last run on es2010 is CRITICAL puppet fail [03:50:06] PROBLEM - puppet last run on mw1039 is CRITICAL puppet fail [03:50:27] PROBLEM - puppet last run on mw2095 is CRITICAL puppet fail [03:50:46] PROBLEM - puppet last run on mw2192 is CRITICAL puppet fail [03:51:17] PROBLEM - puppet last run on cp4018 is CRITICAL puppet fail [03:52:16] PROBLEM - puppet last run on mw1247 is CRITICAL puppet fail [03:52:25] RECOVERY - puppet last run on wtp1002 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [03:52:36] RECOVERY - puppet last run on mw1210 is OK Puppet is currently enabled, last run 15 seconds ago with 0 failures [03:52:55] PROBLEM - puppet last run on cp2020 is CRITICAL puppet fail [03:53:05] PROBLEM - puppet last run on lvs3004 is CRITICAL puppet fail [03:53:07] RECOVERY - puppet last run on mw1146 is OK Puppet is currently enabled, last run 48 seconds ago with 0 failures [03:53:16] RECOVERY - puppet last run on mw1057 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [03:53:46] PROBLEM - puppet last run on mw1159 is CRITICAL puppet fail [03:56:45] RECOVERY - puppet last run on es2010 is OK Puppet is currently enabled, last run 54 seconds ago with 0 failures [04:07:46] RECOVERY - puppet last run on mw1039 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [04:08:16] RECOVERY - puppet last run on mw2095 is OK Puppet is currently enabled, last run 49 seconds ago with 0 failures [04:08:56] RECOVERY - puppet last run on cp2020 is OK Puppet is currently enabled, last run 2 seconds ago with 0 failures [04:09:05] RECOVERY - puppet last run on lvs3004 is OK Puppet is currently enabled, last run 45 seconds ago with 0 failures [04:09:06] RECOVERY - puppet last run on cp4018 is OK Puppet is currently enabled, last run 10 seconds ago with 0 failures [04:10:17] RECOVERY - puppet last run on mw2192 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [04:11:26] RECOVERY - puppet last run on mw1159 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [04:11:46] RECOVERY - puppet last run on mw1247 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [04:30:54] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 28 04:30:54 UTC 2015 (duration 30m 53s) [04:31:01] Logged the message, Master [05:23:26] PROBLEM - puppet last run on mw1015 is CRITICAL Puppet has 1 failures [05:39:16] RECOVERY - puppet last run on mw1015 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:23:48] 6operations, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1407989 (10Steinsplitter) [06:29:47] PROBLEM - puppet last run on cp2013 is CRITICAL Puppet has 1 failures [06:29:47] PROBLEM - puppet last run on mw2090 is CRITICAL puppet fail [06:30:56] PROBLEM - puppet last run on db1023 is CRITICAL Puppet has 1 failures [06:31:15] PROBLEM - puppet last run on tin is CRITICAL puppet fail [06:31:35] PROBLEM - puppet last run on wtp2019 is CRITICAL puppet fail [06:31:36] PROBLEM - puppet last run on mc2011 is CRITICAL Puppet has 1 failures [06:31:46] PROBLEM - puppet last run on mw1076 is CRITICAL puppet fail [06:31:56] PROBLEM - puppet last run on mw2192 is CRITICAL puppet fail [06:31:56] PROBLEM - puppet last run on db1021 is CRITICAL Puppet has 1 failures [06:32:16] PROBLEM - puppet last run on cp2001 is CRITICAL Puppet has 1 failures [06:32:46] PROBLEM - puppet last run on cp2014 is CRITICAL Puppet has 1 failures [06:32:56] PROBLEM - puppet last run on logstash1006 is CRITICAL Puppet has 1 failures [06:33:06] PROBLEM - puppet last run on cp4004 is CRITICAL Puppet has 1 failures [06:33:06] PROBLEM - puppet last run on cp4014 is CRITICAL Puppet has 1 failures [06:33:17] PROBLEM - puppet last run on cp3008 is CRITICAL Puppet has 1 failures [06:34:26] PROBLEM - puppet last run on db1034 is CRITICAL Puppet has 1 failures [06:34:26] PROBLEM - puppet last run on gallium is CRITICAL Puppet has 1 failures [06:34:36] PROBLEM - puppet last run on mw1065 is CRITICAL Puppet has 2 failures [06:34:46] PROBLEM - puppet last run on wtp2015 is CRITICAL Puppet has 2 failures [06:34:46] PROBLEM - puppet last run on db2040 is CRITICAL Puppet has 1 failures [06:34:46] PROBLEM - puppet last run on mw2093 is CRITICAL Puppet has 2 failures [06:34:46] PROBLEM - puppet last run on mw2113 is CRITICAL Puppet has 1 failures [06:34:46] PROBLEM - puppet last run on mw2079 is CRITICAL Puppet has 1 failures [06:35:26] PROBLEM - puppet last run on analytics1010 is CRITICAL Puppet has 1 failures [06:35:35] PROBLEM - puppet last run on mw2022 is CRITICAL Puppet has 2 failures [06:35:55] PROBLEM - puppet last run on mw1002 is CRITICAL Puppet has 1 failures [06:35:56] PROBLEM - puppet last run on mw2146 is CRITICAL Puppet has 1 failures [06:35:56] PROBLEM - puppet last run on mw2126 is CRITICAL Puppet has 1 failures [06:36:06] PROBLEM - puppet last run on mw2045 is CRITICAL Puppet has 1 failures [06:36:15] PROBLEM - puppet last run on mw2011 is CRITICAL Puppet has 1 failures [06:36:26] PROBLEM - puppet last run on mw1011 is CRITICAL Puppet has 1 failures [06:46:36] RECOVERY - puppet last run on cp2001 is OK Puppet is currently enabled, last run 14 seconds ago with 0 failures [06:46:47] RECOVERY - puppet last run on mw2126 is OK Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:46:56] RECOVERY - puppet last run on db1023 is OK Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:46:56] RECOVERY - puppet last run on db1034 is OK Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:46:56] RECOVERY - puppet last run on gallium is OK Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:47:05] RECOVERY - puppet last run on cp2014 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:06] RECOVERY - puppet last run on mw1065 is OK Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:47:15] RECOVERY - puppet last run on tin is OK Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:47:16] RECOVERY - puppet last run on logstash1006 is OK Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:47:16] RECOVERY - puppet last run on wtp2015 is OK Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:47:16] RECOVERY - puppet last run on db2040 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:17] RECOVERY - puppet last run on mw2113 is OK Puppet is currently enabled, last run 37 seconds ago with 0 failures [06:47:17] RECOVERY - puppet last run on mw2093 is OK Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:47:17] RECOVERY - puppet last run on mw2079 is OK Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:47:26] RECOVERY - puppet last run on cp4004 is OK Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:47:26] RECOVERY - puppet last run on cp4014 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:36] RECOVERY - puppet last run on cp3008 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:37] RECOVERY - puppet last run on wtp2019 is OK Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:47:37] RECOVERY - puppet last run on cp2013 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:45] RECOVERY - puppet last run on mc2011 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:55] RECOVERY - puppet last run on mw1076 is OK Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:48:06] RECOVERY - puppet last run on analytics1010 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:06] RECOVERY - puppet last run on db1021 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:06] RECOVERY - puppet last run on mw2192 is OK Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:48:06] RECOVERY - puppet last run on mw2022 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:35] RECOVERY - puppet last run on mw1002 is OK Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:48:36] RECOVERY - puppet last run on mw2146 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:46] RECOVERY - puppet last run on mw2045 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:56] RECOVERY - puppet last run on mw2011 is OK Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:49:15] RECOVERY - puppet last run on mw1011 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:49:36] RECOVERY - puppet last run on mw2090 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:56] PROBLEM - puppet last run on ganeti2004 is CRITICAL puppet fail [07:14:56] RECOVERY - puppet last run on ganeti2004 is OK Puppet is currently enabled, last run 20 seconds ago with 0 failures [08:11:55] PROBLEM - puppet last run on mw1002 is CRITICAL Puppet has 1 failures [08:14:29] bd808: sweet! that simplifies things a lot [08:27:15] PROBLEM - Host graphite1002 is DOWN: PING CRITICAL - Packet loss = 100% [08:27:25] RECOVERY - configured eth on graphite1002 is OK - interfaces up [08:27:25] RECOVERY - salt-minion processes on graphite1002 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [08:27:36] RECOVERY - Host graphite1002 is UPING OK - Packet loss = 0%, RTA = 1.63 ms [08:27:46] RECOVERY - SSH on graphite1002 is OK: SSH OK - OpenSSH_6.7p1 Debian-5 (protocol 2.0) [08:27:47] RECOVERY - RAID on graphite1002 is OK optimal, 2 logical, 4 physical [08:27:56] RECOVERY - Disk space on graphite1002 is OK: DISK OK [08:28:05] RECOVERY - dhclient process on graphite1002 is OK: PROCS OK: 0 processes with command name dhclient [08:28:05] RECOVERY - puppet last run on mw1002 is OK Puppet is currently enabled, last run 58 seconds ago with 0 failures [08:28:35] RECOVERY - DPKG on graphite1002 is OK: All packages OK [08:43:36] PROBLEM - puppet last run on mw2005 is CRITICAL puppet fail [09:01:36] RECOVERY - puppet last run on mw2005 is OK Puppet is currently enabled, last run 56 seconds ago with 0 failures [09:07:46] PROBLEM - puppet last run on wtp2005 is CRITICAL puppet fail [09:23:56] RECOVERY - puppet last run on wtp2005 is OK Puppet is currently enabled, last run 40 seconds ago with 0 failures [09:46:45] PROBLEM - puppet last run on db1065 is CRITICAL Puppet has 1 failures [09:52:17] (03CR) 10Glaisher: Set wikidata's logo specifically for wikidatawiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/221405 (https://phabricator.wikimedia.org/T54214) (owner: 10Alex Monk) [09:59:15] RECOVERY - puppet last run on db1065 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [10:04:34] 6operations, 10SEO: GWT accounts - https://phabricator.wikimedia.org/T103567#1408165 (10Stu) Thanks, @wwes. I don't currently have access to any of the https variants of wikipedia.org. Since we've now switched over to https in our canonical URLs, that's where Google is sending all the traffic now so the http... [11:12:02] 6operations, 6Research-and-Data, 7Database: Test and fix db1047 BBU - https://phabricator.wikimedia.org/T103345#1408240 (10jcrespo) I believe this server is also affected by the faulty TokuDB replication I mentioned on T100408#1323731, because restarting replication helps with lag. I want to reboot and upgr... [11:46:25] 6operations, 6Research-and-Data, 7Database: Test and fix db1047 BBU - https://phabricator.wikimedia.org/T103345#1408265 (10Peachey88) >>! In T103345#1403973, @jcrespo wrote: > Low priority because for now the hosts works as intended. I may be mistaken here, but lagging regularly is not "works as intended" [11:50:15] PROBLEM - puppet last run on db2040 is CRITICAL puppet fail [11:59:31] 6operations, 7Database: db1002-db1007 - decom or repurpose? - https://phabricator.wikimedia.org/T103005#1408273 (10Peachey88) [11:59:32] 6operations, 7Database: review eqiad database server quantities / warranties / service(s) - https://phabricator.wikimedia.org/T103936#1408272 (10Peachey88) [12:08:06] RECOVERY - puppet last run on db2040 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [12:09:49] 6operations, 6Research-and-Data, 7Database: Test and fix db1047 BBU - https://phabricator.wikimedia.org/T103345#1408281 (10jcrespo) @Peachey88, please expand about the problems that you are currently suffering due to db1047. [12:21:50] 6operations, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1408296 (10Tau) Hello to everyone! This is my first post here. I am an administrator of one wikiproject. I had set InstantCom... [12:40:22] 6operations, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1408309 (10Reedy) @bawolff was taking about this last night, see his comment above. Can you install the php5-curl package, r... [12:46:26] 6operations, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1408317 (10Tau) Okay, I will try and let you know. [13:18:26] PROBLEM - puppet last run on cp4012 is CRITICAL puppet fail [13:19:35] PROBLEM - puppet last run on mw1142 is CRITICAL Puppet has 1 failures [13:35:45] RECOVERY - puppet last run on mw1142 is OK Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:36:17] RECOVERY - puppet last run on cp4012 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [14:31:55] 6operations, 10Gather, 7Database, 7Schema-change: Update Gather DB schema for flagging backend - https://phabricator.wikimedia.org/T103611#1408364 (10jcrespo) > What's wrong with it? I thought it just resolves script names to the right subdirectory. Also, it is still recommended by all documentation (e.g.... [14:42:53] (03PS2) 10Alex Monk: Set wikidata's logo specifically for wikidatawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/221405 (https://phabricator.wikimedia.org/T54214) [14:49:28] 6operations, 10Gather, 7Database, 7Schema-change: Update Gather DB schema for flagging backend - https://phabricator.wikimedia.org/T103611#1408366 (10jcrespo) @Tgr: >>! In T103611#1406031, @Tgr wrote: > Even easier, just use `UPDATE gather_list SET gl_perm = 1, gl_perm_override = 1 WHERE gl_perm = 2` the... [14:57:02] (03CR) 10MZMcBride: Set $wgAjaxEditStash = true (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178553 (owner: 10Ori.livneh) [17:05:46] PROBLEM - puppet last run on cp2007 is CRITICAL puppet fail [17:08:10] (03PS4) 10Reedy: Remove wmgDualLicense, orphaned [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209840 (https://phabricator.wikimedia.org/T62023) [17:15:58] I'm about ready to merge that as a noop [17:17:48] (03CR) 10Reedy: Set $wgAjaxEditStash = true (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178553 (owner: 10Ori.livneh) [17:20:13] (03CR) 10Reedy: "Can we remove the -2 on this with the symlinks?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175007 (owner: 10Ori.livneh) [17:21:56] RECOVERY - puppet last run on cp2007 is OK Puppet is currently enabled, last run 56 seconds ago with 0 failures [17:35:16] PROBLEM - git.wikimedia.org on antimony is CRITICAL - Socket timeout after 10 seconds [17:40:46] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 60569 bytes in 0.214 second response time [18:04:36] 6operations, 10Traffic, 7HTTPS, 5HTTPS-by-default, 5Patch-For-Review: Switch to ECDSA hybrid certificates - https://phabricator.wikimedia.org/T86654#1408481 (10JanZerebecki) Thanks. Btw. cloudflare had again disabled ECDSA since quite some time. No comment from them in their blog. Any idea if there wher... [18:06:18] (03CR) 10JanZerebecki: [C: 031] ssl_ciphersuite: re-order ECDSA ahead of RSA [puppet] - 10https://gerrit.wikimedia.org/r/220377 (owner: 10BBlack) [18:33:00] (03PS9) 10Addshore: rsync wikidata json dumps to labs /public/dumps [puppet] - 10https://gerrit.wikimedia.org/r/215585 (https://phabricator.wikimedia.org/T100885) [19:35:37] (03PS1) 10Merlijn van Deen: [WIP] [tools/apt] Multi-distro packaging [puppet] - 10https://gerrit.wikimedia.org/r/221456 [19:35:39] (03PS1) 10Merlijn van Deen: [WIP] [tools/apt] Multiversioning for Python 2/3 [puppet] - 10https://gerrit.wikimedia.org/r/221457 [19:36:25] (03CR) 10jenkins-bot: [V: 04-1] [WIP] [tools/apt] Multiversioning for Python 2/3 [puppet] - 10https://gerrit.wikimedia.org/r/221457 (owner: 10Merlijn van Deen) [19:36:27] (03CR) 10jenkins-bot: [V: 04-1] [WIP] [tools/apt] Multi-distro packaging [puppet] - 10https://gerrit.wikimedia.org/r/221456 (owner: 10Merlijn van Deen) [19:37:54] 6operations, 10Traffic: Support ALPN + HTTP/2 - https://phabricator.wikimedia.org/T96848#1408516 (10BBlack) [19:37:57] 6operations, 10Traffic: Test then switch to openssl 1.0.2 + nginx 1.9.2 - https://phabricator.wikimedia.org/T96850#1408511 (10BBlack) 5Open>3Resolved a:3BBlack We've already switched to nginx 1.9.2 w/ SO_REUSEPORT enabled. We're still using debian's openssl 1.0.1 with it for now, but we'll address opens... [19:37:58] 6operations, 10Traffic, 7HTTPS, 5HTTPS-by-default, 5Patch-For-Review: Switch to ECDSA hybrid certificates - https://phabricator.wikimedia.org/T86654#1408518 (10BBlack) [19:38:19] 6operations, 10Traffic: check if services behind misc-web enforce http->https redirect or not - https://phabricator.wikimedia.org/T103773#1408521 (10BBlack) [19:38:24] 6operations, 6Performance-Team, 10Traffic, 5Patch-For-Review, 7Varnish: Ensure {text-domain}/w/load.php requests do not bypass cache for session cookies - https://phabricator.wikimedia.org/T101892#1408523 (10BBlack) [20:08:21] 6operations, 10Traffic: openssl 1.0.2 packaging for jessie - https://phabricator.wikimedia.org/T104143#1408533 (10BBlack) 3NEW [20:08:40] 6operations, 10Traffic: openssl 1.0.2 packaging for jessie - https://phabricator.wikimedia.org/T104143#1408541 (10BBlack) [20:08:43] 6operations, 10Traffic, 7HTTPS, 5HTTPS-by-default, 5Patch-For-Review: Switch to ECDSA hybrid certificates - https://phabricator.wikimedia.org/T86654#1408540 (10BBlack) [20:30:16] PROBLEM - puppet last run on cp3006 is CRITICAL puppet fail [20:32:34] 6operations, 10Traffic, 7HTTPS, 5HTTPS-by-default, 5Patch-For-Review: Switch to ECDSA hybrid certificates - https://phabricator.wikimedia.org/T86654#1408545 (10BBlack) >>! In T86654#1408481, @JanZerebecki wrote: > Thanks. > > Btw. cloudflare had again disabled ECDSA since quite some time. No comment fro... [20:48:06] RECOVERY - puppet last run on cp3006 is OK Puppet is currently enabled, last run 26 seconds ago with 0 failures [20:50:08] PROBLEM - git.wikimedia.org on antimony is CRITICAL - Socket timeout after 10 seconds [20:53:27] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 60557 bytes in 0.678 second response time [21:04:27] PROBLEM - git.wikimedia.org on antimony is CRITICAL - Socket timeout after 10 seconds [21:11:27] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 60579 bytes in 0.357 second response time [21:20:07] PROBLEM - puppet last run on cp3017 is CRITICAL puppet fail [21:29:26] PROBLEM - git.wikimedia.org on antimony is CRITICAL - Socket timeout after 10 seconds [21:31:05] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 60557 bytes in 0.299 second response time [21:36:11] (03CR) 10Ori.livneh: [C: 031] enable SPDY header compression [puppet] - 10https://gerrit.wikimedia.org/r/221376 (owner: 10BBlack) [21:37:46] RECOVERY - puppet last run on cp3017 is OK Puppet is currently enabled, last run 44 seconds ago with 0 failures [23:00:55] PROBLEM - puppet last run on mw1154 is CRITICAL Puppet has 1 failures [23:16:46] RECOVERY - puppet last run on mw1154 is OK Puppet is currently enabled, last run 39 seconds ago with 0 failures [23:17:38] PROBLEM - puppet last run on cp3047 is CRITICAL puppet fail [23:19:55] (03CR) 10Springle: [C: 031] tendril: sync changes from github repo [software/tendril] - 10https://gerrit.wikimedia.org/r/221184 (https://phabricator.wikimedia.org/T98816) (owner: 10Dzahn) [23:20:03] (03CR) 10Springle: [C: 031] tendril: git clone from wmf repo via puppet [puppet] - 10https://gerrit.wikimedia.org/r/221172 (https://phabricator.wikimedia.org/T98816) (owner: 10Dzahn) [23:23:26] PROBLEM - Host mw2027 is DOWN: PING CRITICAL - Packet loss = 100% [23:24:46] RECOVERY - Host mw2027 is UPING WARNING - Packet loss = 28%, RTA = 43.16 ms [23:35:17] RECOVERY - puppet last run on cp3047 is OK Puppet is currently enabled, last run 50 seconds ago with 0 failures [23:51:16] !log ori Synchronized php-1.26wmf11/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js: I6ffdc977e87: Parse older format of Geo cookies (duration: 00m 13s) [23:51:22] Logged the message, Master [23:54:28] 7Puppet, 6Community-Liaison, 10MediaWiki-extensions-NavigationTiming, 6Performance-Team, and 3 others: Track state (region) - https://phabricator.wikimedia.org/T101819#1408727 (10ori)