[00:22:28] PROBLEM - RAID on snapshot1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:24:26] RECOVERY - RAID on snapshot1002 is OK no RAID installed [00:49:37] PROBLEM - RAID on snapshot1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:53:26] RECOVERY - RAID on snapshot1002 is OK no RAID installed [01:00:17] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:01:27] PROBLEM - RAID on snapshot1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:02:16] RECOVERY - puppet last run on snapshot1002 is OK Puppet is currently enabled, last run 23 minutes ago with 0 failures [01:03:08] RECOVERY - RAID on snapshot1002 is OK no RAID installed [01:13:47] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:15:37] RECOVERY - puppet last run on snapshot1002 is OK Puppet is currently enabled, last run 9 minutes ago with 0 failures [01:32:07] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 7.14% of data above the critical threshold [500.0] [01:37:57] RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0] [02:00:16] RECOVERY - Last backup of the tools filesystem on labstore1002 is OK - Last run successful [02:04:18] PROBLEM - Disk space on labstore1002 is CRITICAL: DISK CRITICAL - /run/lock/storage-replicate-labstore-tools/snapshot is not accessible: Permission denied [02:23:37] (03PS1) 10Tim Landscheidt: Move role::labs::tools::* to module role [puppet] - 10https://gerrit.wikimedia.org/r/231866 [02:24:17] PROBLEM - RAID on snapshot1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:24:36] (03CR) 10jenkins-bot: [V: 04-1] Move role::labs::tools::* to module role [puppet] - 10https://gerrit.wikimedia.org/r/231866 (owner: 10Tim Landscheidt) [02:25:18] PROBLEM - puppet last run on snapshot1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:25:50] !log l10nupdate@tin Synchronized php-1.26wmf18/cache/l10n: l10nupdate for 1.26wmf18 (duration: 11m 52s) [02:25:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:26:07] RECOVERY - RAID on snapshot1002 is OK no RAID installed [02:29:16] RECOVERY - puppet last run on snapshot1002 is OK Puppet is currently enabled, last run 21 minutes ago with 0 failures [02:29:19] !log l10nupdate@tin LocalisationUpdate completed (1.26wmf18) at 2015-08-16 02:29:19+00:00 [02:29:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:52:38] PROBLEM - LVS HTTP IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection timed out [02:56:27] RECOVERY - LVS HTTP IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 301 TLS Redirect - 497 bytes in 0.999 second response time [03:01:16] RECOVERY - Last backup of the others filesystem on labstore1002 is OK - Last run successful [03:35:17] PROBLEM - puppet last run on elastic1011 is CRITICAL Puppet has 1 failures [03:35:47] PROBLEM - puppet last run on mw1029 is CRITICAL Puppet has 1 failures [03:36:06] PROBLEM - puppet last run on mw1109 is CRITICAL Puppet has 1 failures [03:51:38] PROBLEM - puppet last run on mw1104 is CRITICAL Puppet has 1 failures [03:55:16] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 14.29% of data above the critical threshold [500.0] [04:00:57] RECOVERY - Last backup of the maps filesystem on labstore1002 is OK - Last run successful [04:01:07] RECOVERY - puppet last run on mw1109 is OK Puppet is currently enabled, last run 1 second ago with 0 failures [04:02:17] RECOVERY - puppet last run on elastic1011 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [04:02:57] RECOVERY - puppet last run on mw1029 is OK Puppet is currently enabled, last run 38 seconds ago with 0 failures [04:08:57] RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0] [04:18:37] RECOVERY - puppet last run on mw1104 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [05:39:28] !log l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 16 05:39:28 UTC 2015 (duration 39m 27s) [05:39:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [06:32:38] PROBLEM - puppet last run on mw2018 is CRITICAL Puppet has 1 failures [06:32:57] PROBLEM - puppet last run on mw1170 is CRITICAL Puppet has 1 failures [06:33:17] PROBLEM - puppet last run on mw2073 is CRITICAL Puppet has 1 failures [06:52:12] 6operations, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia, 10Wikimedia-Video: Backport libtheora 1.2.0alpha package to Trusty - https://phabricator.wikimedia.org/T109207#1543030 (10brion) 3NEW [06:57:48] RECOVERY - puppet last run on mw2018 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:07] RECOVERY - puppet last run on mw1170 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:27] RECOVERY - puppet last run on mw2073 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [07:01:27] PROBLEM - RAID on snapshot1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:03:16] RECOVERY - RAID on snapshot1002 is OK no RAID installed [07:21:37] PROBLEM - OCG health on ocg1003 is CRITICAL ocg_job_status 627544 msg: ocg_render_job_queue 3458 msg (=3000 critical) [07:21:38] PROBLEM - OCG health on ocg1001 is CRITICAL ocg_job_status 627780 msg: ocg_render_job_queue 3637 msg (=3000 critical) [07:21:38] PROBLEM - OCG health on ocg1002 is CRITICAL ocg_job_status 627784 msg: ocg_render_job_queue 3638 msg (=3000 critical) [07:35:14] 6operations, 10MediaWiki-extensions-TimedMediaHandler, 6Multimedia, 10Wikimedia-Video: Backport libtheora 1.2.0alpha package to Trusty - https://phabricator.wikimedia.org/T109207#1543043 (10brion) {F1523209} {F1523212} {F1523221} [07:40:37] (03CR) 10Giuseppe Lavagetto: [C: 031] "I like both the concept and the implementation" (031 comment) [debs/pybal] - 10https://gerrit.wikimedia.org/r/230931 (owner: 10Ori.livneh) [09:15:22] (03PS1) 10Faidon Liambotis: xenon: redirect stdout to /dev/null, avoid cronspam [puppet] - 10https://gerrit.wikimedia.org/r/231873 [09:16:09] (03CR) 10Faidon Liambotis: [C: 032] xenon: redirect stdout to /dev/null, avoid cronspam [puppet] - 10https://gerrit.wikimedia.org/r/231873 (owner: 10Faidon Liambotis) [09:19:02] (03PS1) 10Faidon Liambotis: exim: remove unreferenced +relay_domains from mailman [puppet] - 10https://gerrit.wikimedia.org/r/231874 [09:20:03] (03PS2) 10Faidon Liambotis: exim: remove unreferenced +relay_domains from mailman [puppet] - 10https://gerrit.wikimedia.org/r/231874 [09:20:24] (03CR) 10Faidon Liambotis: [C: 032 V: 032] exim: remove unreferenced +relay_domains from mailman [puppet] - 10https://gerrit.wikimedia.org/r/231874 (owner: 10Faidon Liambotis) [09:27:13] 6operations, 10Wikimedia-Language-setup: Rename "be-x-old" to "be-tarask" - https://phabricator.wikimedia.org/T11823#1543111 (10Liuxinyu970226) [09:36:39] 6operations, 10Wikimedia-Site-Requests: Move the wiki of WMEE - https://phabricator.wikimedia.org/T31919#1543120 (10Liuxinyu970226) [09:37:23] 6operations, 10Wikimedia-Language-setup: Renaming the Aramaic (arc) Wikipedia to the Syriac (syc) Wikipedia - https://phabricator.wikimedia.org/T28725#1543121 (10Liuxinyu970226) [09:38:40] 6operations, 10Wikimedia-Language-setup: language code change for Samogitian: "bat-smg" to "sgs" - https://phabricator.wikimedia.org/T27522#1543126 (10Liuxinyu970226) [09:39:02] 6operations, 10Wikimedia-Language-setup: Rename emlwiki -> eglwiki - https://phabricator.wikimedia.org/T36217#1543128 (10Liuxinyu970226) [09:39:48] 6operations, 10Wikimedia-Site-Requests, 5Patch-For-Review: Rename "chapcomwiki" to "affcomwiki" - https://phabricator.wikimedia.org/T41482#1543132 (10Liuxinyu970226) [10:02:17] RECOVERY - OCG health on ocg1003 is OK ocg_job_status 738293 msg: ocg_render_job_queue 0 msg [10:02:36] RECOVERY - OCG health on ocg1001 is OK ocg_job_status 738335 msg: ocg_render_job_queue 0 msg [10:02:37] RECOVERY - OCG health on ocg1002 is OK ocg_job_status 738345 msg: ocg_render_job_queue 0 msg [10:16:37] PROBLEM - Hadoop NodeManager on analytics1041 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:18:27] RECOVERY - Hadoop NodeManager on analytics1041 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [10:20:47] PROBLEM - RAID on snapshot1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:22:38] RECOVERY - RAID on snapshot1002 is OK no RAID installed [11:16:45] (03PS4) 10Giuseppe Lavagetto: WIP: Add etcd configuration client [debs/pybal] - 10https://gerrit.wikimedia.org/r/225649 (owner: 10Ori.livneh) [12:07:39] 6operations, 10Wikimedia-Mailing-lists, 5Patch-For-Review: Education coop mailing list rename - https://phabricator.wikimedia.org/T107445#1543216 (10Selsharbaty-WMF) Thanks, Rob! [14:19:58] (03PS1) 10John F. Lewis: mirrors: add debian.wikimedia.org as a servername [puppet] - 10https://gerrit.wikimedia.org/r/231881 [14:20:05] (03PS1) 10John F. Lewis: cname debian.wikimedia.org to carbon (apt) [dns] - 10https://gerrit.wikimedia.org/r/231882 [14:56:27] (03CR) 10Faidon Liambotis: [C: 04-1] "Why?" [dns] - 10https://gerrit.wikimedia.org/r/231882 (owner: 10John F. Lewis) [14:58:22] (03CR) 10John F. Lewis: "Seemed logical seeing ubuntu.wm.o is a cname. Since it seems to be a mirror for Debian as well it just aligns with consistency. No reason " [dns] - 10https://gerrit.wikimedia.org/r/231882 (owner: 10John F. Lewis) [15:01:40] (03CR) 10Faidon Liambotis: [C: 04-2] "ubuntu.wm.org existed before mirrors. When I added Debian, I opted into a more generic "mirrors" hostname that could possibly host other p" [dns] - 10https://gerrit.wikimedia.org/r/231882 (owner: 10John F. Lewis) [15:09:08] (03Abandoned) 10John F. Lewis: cname debian.wikimedia.org to carbon (apt) [dns] - 10https://gerrit.wikimedia.org/r/231882 (owner: 10John F. Lewis) [15:09:21] (03Abandoned) 10John F. Lewis: mirrors: add debian.wikimedia.org as a servername [puppet] - 10https://gerrit.wikimedia.org/r/231881 (owner: 10John F. Lewis) [15:09:30] paravoid: okay :) [15:10:26] heh :) [15:25:06] 6operations, 10Math: Install texlive-extra-utils on mw appservers - https://phabricator.wikimedia.org/T109195#1543293 (10Reedy) [15:26:21] 6operations, 10Math: Install texlive-extra-utils on mw appservers - https://phabricator.wikimedia.org/T109195#1542819 (10Reedy) It seems the wanted package is https://launchpad.net/ubuntu/+source/texlive-extra/2013.20131112-1 (lists all the required binaries) so hence http://packages.ubuntu.com/trusty/texlive-... [15:29:35] (03PS1) 10Reedy: Install texlive-extra-utils on mw appservers [puppet] - 10https://gerrit.wikimedia.org/r/231889 (https://phabricator.wikimedia.org/T109195) [15:30:07] 6operations, 10Math, 5Patch-For-Review: Install texlive-extra-utils on mw appservers - https://phabricator.wikimedia.org/T109195#1543301 (10Reedy) Can you give a rough use case for this please @Mathmensch? Straight off, I can't see why it'd be a problem, but a justification is always helpful :) [15:30:50] * Reedy corrects himself [15:31:07] PROBLEM - RAID on snapshot1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:33:06] RECOVERY - RAID on snapshot1002 is OK no RAID installed [15:40:55] (03PS1) 10Alex Monk: Use meta.wikimedia.org/api/rest_v1 instead of parsoid-lb in the sysadmin table script [puppet] - 10https://gerrit.wikimedia.org/r/231890 [15:42:45] 6operations, 10Math, 5Patch-For-Review: Install texlive-extra-utils on mw appservers - https://phabricator.wikimedia.org/T109195#1543307 (10Mathmensch) @Reedy I need a symbol for an average (integral w/ horizontal stroke) and a really wide hat for the Fourier transform (the current one is too small, this ne... [15:42:46] PROBLEM - puppet last run on mw1095 is CRITICAL Puppet has 1 failures [15:45:43] 6operations, 10Math, 5Patch-For-Review: Install texlive-extra-utils on mw appservers - https://phabricator.wikimedia.org/T109195#1543312 (10Mathmensch) See also tex.stackexchange.com/questions/100574/really-wide-hat-symbol [15:47:19] (03CR) 10Reedy: [C: 04-1] "I'm not sure if this is the right package" [puppet] - 10https://gerrit.wikimedia.org/r/231889 (https://phabricator.wikimedia.org/T109195) (owner: 10Reedy) [15:48:41] 6operations, 10Math, 5Patch-For-Review: Install texlive-extra-utils on mw appservers - https://phabricator.wikimedia.org/T109195#1543321 (10Reedy) >>! In T109195#1543293, @Reedy wrote: > It seems the wanted package is https://launchpad.net/ubuntu/+source/texlive-extra/2013.20131112-1 (lists all the required... [16:04:09] https://wikiversity.org [16:04:14] help [16:05:31] 6operations, 10Wikimedia-Apache-configuration: wikiversity.org and wikinews.org redirects to /503.html - https://phabricator.wikimedia.org/T109226#1543344 (10Glaisher) 3NEW [16:09:36] RECOVERY - puppet last run on mw1095 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures [16:24:37] could someone look into that unbreak now task? ^ [16:24:54] paravoid: ^ [16:41:29] Glaisher: not for me [16:43:08] weird [16:43:30] ori: someone else reported it at #wikipedia [16:43:37] Krenair: also confirmed it. [16:43:43] i'll check the logs [16:44:00] thanks [17:05:41] Glaisher: nothing looks off. Krenair, Glaisher -- can you still reproduce this? [17:06:14] still redirects to 503.html [17:07:25] ori: https://wikiversity.org/ gives me a redirect to en.wv.o/503.html, but https://en.wikiversity.org works [17:07:38] ori: so it's something with the generic language selector/search page [17:08:24] valhallasw`cloud: can you 'curl -vvv -I -L https://wikiversity.org/' and pastebin the output? [17:09:15] yes, just a second. [17:09:48] it works OK from tools-bastion-01 [17:14:05] yeah i tried webpagetest.org from paris [17:14:09] i get the 503 [17:14:14] so probably some esams issue [17:14:14] ori: https://bpaste.net/show/239e3cfb2284 [17:16:03] yeah, http://www.webpagetest.org/result/150816_GT_KA6/ [17:17:57] one of the varnish frontends in esams cached the 503 [17:27:41] i'm going to simulate this for a second by stopping the hhvm service on mw1017 (testwiki) [17:46:58] 6operations, 10Wikimedia-Apache-configuration: wikiversity.org and wikinews.org redirects to /503.html - https://phabricator.wikimedia.org/T109226#1543440 (10Southparkfan) @Glaisher these pages show up normally for me. Do you experience this issue on more than one device? [17:48:05] In the future I need to read this channel before I comment on tasks. [17:48:11] 6operations, 10Wikimedia-Apache-configuration: wikiversity.org and wikinews.org redirects to /503.html - https://phabricator.wikimedia.org/T109226#1543441 (10Glaisher) It seems it's an esams related issue. @ori is investigating it currently. [17:48:27] 6operations, 10Wikimedia-Apache-configuration: wikiversity.org and wikinews.org redirects to /503.html - https://phabricator.wikimedia.org/T109226#1543443 (10ori) An error page was cached in a Varnish cache server in ESAMS (Amsterdam), so this error is only reproducible when connecting from Europe (or anywhere... [17:48:29] ;) [17:49:04] broken for me from sweden [17:49:13] Yeah normally 'my traffic' would be served by esams too, but since I'm not in the NL now it didn't [17:50:48] ori: need any help? [17:51:27] 6operations, 10Wikimedia-Apache-configuration: wikiversity.org and wikinews.org redirects to /503.html - https://phabricator.wikimedia.org/T109226#1543444 (10ori) The root cause rests with the interaction of these two configuration directives in our Apache configuration: ``` ErrorDocument 503 /503.html Rewrit... [17:51:33] Reedy: that's basically the reason ^ [17:51:44] the URL needs to be purged on that varnish [17:51:48] Fallout from last night? [17:52:30] yeah [17:52:41] well, the bug exists independetly of that [17:52:45] well, sure [17:52:52] but likely cached last night [17:52:58] yep [18:06:06] PROBLEM - SSH on snapshot1002 is CRITICAL - Socket timeout after 10 seconds [18:07:59] RECOVERY - SSH on snapshot1002 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2wmfprecise2 (protocol 2.0) [18:11:56] (03PS3) 10Reedy: Disable Extension:Oversight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/169612 (https://phabricator.wikimedia.org/T62373) [18:12:01] woo, it rebased [18:12:02] hahah [18:12:45] (03CR) 10Alex Monk: [C: 032] Disable Extension:Oversight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/169612 (https://phabricator.wikimedia.org/T62373) (owner: 10Reedy) [18:12:50] (03Merged) 10jenkins-bot: Disable Extension:Oversight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/169612 (https://phabricator.wikimedia.org/T62373) (owner: 10Reedy) [18:14:23] !log krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/169612/ - remove Extension:Oversight (duration: 00m 25s) [18:14:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:15:01] !log krenair@tin Synchronized wmf-config/extension-list: https://gerrit.wikimedia.org/r/#/c/169612/ - remove Extension:Oversight (duration: 00m 21s) [18:15:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:15:23] :) [18:16:03] Next target: DisableAccount? [18:33:46] 6operations, 10Wikimedia-Apache-configuration: wikiversity.org and wikinews.org redirects to /503.html - https://phabricator.wikimedia.org/T109226#1543485 (10BBlack) I've banned the bad redirects (all cases -- there were probably others besides this one wikiversity example) on the expression "obj.http.Location... [18:35:20] 6operations, 10Traffic, 10Wikimedia-Apache-configuration: wikiversity.org and wikinews.org redirects to /503.html - https://phabricator.wikimedia.org/T109226#1543487 (10BBlack) p:5Unbreak!>3High [18:36:07] thanks bblack [18:36:27] Krenair: Isn't that still used on some private wikis? [18:36:36] What? [18:36:39] Extension:Oversight? [18:36:50] No, Extension:DisableAccount [18:37:00] Oh, yes it is [18:37:34] https://phabricator.wikimedia.org/T106067 has details about this extension [18:38:39] It prevents an issue that wgBlockDisablesLogin has, fix is not yet deployed but will be next week [19:03:07] RECOVERY - Disk space on labstore1002 is OK: DISK OK [21:37:33] (03PS1) 10Ori.livneh: os_version: Add Ubuntu Vivid & Wily, Debian Stretch & Buster [puppet] - 10https://gerrit.wikimedia.org/r/231972 [21:37:35] (03PS1) 10Ori.livneh: Add simple haveged module [puppet] - 10https://gerrit.wikimedia.org/r/231973 (https://phabricator.wikimedia.org/T82576) [21:40:29] (03PS2) 10Ori.livneh: Add simple haveged module; apply on sodium [puppet] - 10https://gerrit.wikimedia.org/r/231973 (https://phabricator.wikimedia.org/T82576) [21:52:31] 6operations, 10Wikimedia-Mailing-lists: Mailman Upgrade (Jessie & Mailman 2.x) and migration to a VM - https://phabricator.wikimedia.org/T105756#1543718 (10ori) The setup should be resilient to entropy starvation, so we can enable STARTTLS. I am not sure if this requires any explicit action on our part, but on... [22:06:15] 6operations, 10Wikimedia-Mailing-lists: Ensure mailman VM setup has adequate entropy for STARTTLS - https://phabricator.wikimedia.org/T109239#1543725 (10ori) 3NEW a:3Dzahn [22:07:01] 6operations, 10Wikimedia-Mailing-lists: Ensure mailman VM setup has adequate entropy for STARTTLS - https://phabricator.wikimedia.org/T109239#1543725 (10ori) [22:08:26] PROBLEM - RAID on snapshot1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:10:16] RECOVERY - RAID on snapshot1002 is OK no RAID installed [22:10:16] (03CR) 10John F. Lewis: [C: 04-1] "Process -1. Unless there is really any urgency for this, STARTTLS should ideally come along with the move. Since during the process sodium" [puppet] - 10https://gerrit.wikimedia.org/r/231973 (https://phabricator.wikimedia.org/T82576) (owner: 10Ori.livneh) [22:10:31] ori: just social comments more than technical :) [22:26:47] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 13.33% of data above the critical threshold [500.0] [22:40:26] RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0] [22:53:14] (03CR) 10MZMcBride: "Thanks for this, I kept meaning to file a task." [puppet] - 10https://gerrit.wikimedia.org/r/231689 (owner: 10GWicke) [22:56:16] PROBLEM - RAID on snapshot1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:00:07] RECOVERY - RAID on snapshot1002 is OK no RAID installed [23:12:31] (03PS1) 10Alex Monk: Include /api/ rewrite on wikimediafoundation.org [puppet] - 10https://gerrit.wikimedia.org/r/231976 [23:13:12] (03CR) 10Alex Monk: "I597e0e96" [puppet] - 10https://gerrit.wikimedia.org/r/231689 (owner: 10GWicke) [23:28:06] PROBLEM - LVS HTTP IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection timed out [23:28:34] again? [23:29:56] RECOVERY - LVS HTTP IPv6 on mobile-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 301 TLS Redirect - 497 bytes in 1.004 second response time [23:39:01] (03CR) 10MZMcBride: "Thanks!" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/231976 (owner: 10Alex Monk)