[00:00:54] New patchset: Ryan Lane; "Only sync repo in sync_all if it should be sync'd" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37594 [00:01:39] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37594 [00:09:05] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:12:08] New patchset: Ryan Lane; "Fix parsoid keys" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37595 [00:12:32] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37595 [00:25:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.160 seconds [00:31:57] New patchset: Catrope; "Install git-core for Parsoid, needed for git-deploy" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37596 [00:32:09] Ryan_Lane: https://gerrit.wikimedia.org/r/37596 [00:32:31] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37596 [00:41:58] New patchset: Ryan Lane; "Fix grain lookup in salt module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37597 [00:42:15] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37597 [00:54:14] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [00:55:38] New patchset: Ryan Lane; "Wrap hash keys and values in single quotes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37599 [00:58:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:00:53] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37599 [01:13:44] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.824 seconds [01:28:49] PROBLEM - Puppet freshness on db1043 is CRITICAL: Puppet has not run in the last 10 hours [01:37:49] PROBLEM - Puppet freshness on ms-be3 is CRITICAL: Puppet has not run in the last 10 hours [01:48:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:08] New review: Krinkle; "I'll take this with a grain of salt if you don't mind." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37597 [02:01:58] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 9.424 seconds [02:26:46] !log LocalisationUpdate completed (1.21wmf5) at Sat Dec 8 02:26:46 UTC 2012 [02:26:58] Logged the message, Master [02:37:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:40:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.489 seconds [03:05:07] RECOVERY - Puppet freshness on db1043 is OK: puppet ran at Sat Dec 8 03:04:42 UTC 2012 [03:18:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:21:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.014 seconds [03:47:43] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [03:55:49] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [03:57:19] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:12:02] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.170 seconds [04:46:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:01:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.483 seconds [05:35:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:38:14] PROBLEM - Puppet freshness on ms-be3002 is CRITICAL: Puppet has not run in the last 10 hours [05:51:01] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.035 seconds [06:23:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:36:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.831 seconds [07:12:04] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:23:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 9.955 seconds [07:53:55] PROBLEM - Puppet freshness on db59 is CRITICAL: Puppet has not run in the last 10 hours [07:57:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:03:49] PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours [08:03:49] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [08:03:49] PROBLEM - Puppet freshness on sq48 is CRITICAL: Puppet has not run in the last 10 hours [08:10:43] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.420 seconds [08:36:03] PROBLEM - Puppet freshness on db9 is CRITICAL: Puppet has not run in the last 10 hours [08:45:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:56:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.641 seconds [09:20:00] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [09:20:00] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [09:32:07] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:04] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [09:37:04] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [09:45:10] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.034 seconds [10:18:01] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:33:10] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 9.030 seconds [10:55:50] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [11:05:53] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:20:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.039 seconds [11:38:53] PROBLEM - Puppet freshness on ms-be3 is CRITICAL: Puppet has not run in the last 10 hours [11:54:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.030 seconds [12:39:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:54:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.045 seconds [13:28:12] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:39:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.343 seconds [13:48:36] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [13:49:23] Can someone please delete /var/log/l10nupdate from fenari? It's over a year old and other logs are written to /var/log/l10nupdatelog [13:49:24] Thanks [13:50:26] !log LocalisationUpdate completed (1.21wmf5) at Sat Dec 8 13:50:26 UTC 2012 [13:50:38] Logged the message, Master [13:56:33] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [14:04:03] !log reedy synchronized php-1.21wmf5/cache/l10n [14:04:13] Logged the message, Master [14:13:48] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.052 seconds [14:26:44] Can someone else rm -rf /home/wikipedia/common/php-1.21wmf1 ? Thanks! [14:31:02] !log reedy synchronized wmf-config/ [14:31:11] Logged the message, Master [14:32:49] !log reedy synchronized live-1.5/ [14:32:58] Logged the message, Master [14:52:26] PROBLEM - Puppet freshness on sq59 is CRITICAL: Puppet has not run in the last 10 hours [14:58:26] PROBLEM - Puppet freshness on sq72 is CRITICAL: Puppet has not run in the last 10 hours [14:59:29] PROBLEM - Puppet freshness on mc12 is CRITICAL: Puppet has not run in the last 10 hours [14:59:29] PROBLEM - Puppet freshness on search1004 is CRITICAL: Puppet has not run in the last 10 hours [14:59:29] PROBLEM - Puppet freshness on ms-fe2 is CRITICAL: Puppet has not run in the last 10 hours [14:59:29] PROBLEM - Puppet freshness on kaulen is CRITICAL: Puppet has not run in the last 10 hours [14:59:29] PROBLEM - Puppet freshness on mc16 is CRITICAL: Puppet has not run in the last 10 hours [14:59:30] PROBLEM - Puppet freshness on snapshot1 is CRITICAL: Puppet has not run in the last 10 hours [14:59:30] PROBLEM - Puppet freshness on snapshot1003 is CRITICAL: Puppet has not run in the last 10 hours [14:59:31] PROBLEM - Puppet freshness on search1016 is CRITICAL: Puppet has not run in the last 10 hours [14:59:31] PROBLEM - Puppet freshness on sq73 is CRITICAL: Puppet has not run in the last 10 hours [14:59:32] PROBLEM - Puppet freshness on sq65 is CRITICAL: Puppet has not run in the last 10 hours [14:59:33] PROBLEM - Puppet freshness on sq77 is CRITICAL: Puppet has not run in the last 10 hours [14:59:33] PROBLEM - Puppet freshness on sq79 is CRITICAL: Puppet has not run in the last 10 hours [14:59:33] PROBLEM - Puppet freshness on sq80 is CRITICAL: Puppet has not run in the last 10 hours [14:59:34] PROBLEM - Puppet freshness on virt1007 is CRITICAL: Puppet has not run in the last 10 hours [14:59:34] PROBLEM - Puppet freshness on sq84 is CRITICAL: Puppet has not run in the last 10 hours [14:59:35] PROBLEM - Puppet freshness on sq86 is CRITICAL: Puppet has not run in the last 10 hours [14:59:35] PROBLEM - Puppet freshness on virt2 is CRITICAL: Puppet has not run in the last 10 hours [14:59:36] PROBLEM - Puppet freshness on virt1008 is CRITICAL: Puppet has not run in the last 10 hours [15:00:33] PROBLEM - Puppet freshness on mw3 is CRITICAL: Puppet has not run in the last 10 hours [15:00:33] PROBLEM - Puppet freshness on cadmium is CRITICAL: Puppet has not run in the last 10 hours [15:00:33] PROBLEM - Puppet freshness on sq70 is CRITICAL: Puppet has not run in the last 10 hours [15:00:33] PROBLEM - Puppet freshness on search1007 is CRITICAL: Puppet has not run in the last 10 hours [15:00:50] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:01:26] PROBLEM - Puppet freshness on pdf2 is CRITICAL: Puppet has not run in the last 10 hours [15:12:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.350 seconds [15:39:32] PROBLEM - Puppet freshness on ms-be3002 is CRITICAL: Puppet has not run in the last 10 hours [15:48:05] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.016 seconds [16:22:31] PROBLEM - Puppet freshness on constable is CRITICAL: Puppet has not run in the last 10 hours [16:28:31] PROBLEM - Puppet freshness on search1001 is CRITICAL: Puppet has not run in the last 10 hours [16:34:13] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:34] PROBLEM - Puppet freshness on dataset2 is CRITICAL: Puppet has not run in the last 10 hours [16:35:34] PROBLEM - Puppet freshness on sq44 is CRITICAL: Puppet has not run in the last 10 hours [16:35:34] PROBLEM - Puppet freshness on search18 is CRITICAL: Puppet has not run in the last 10 hours [16:35:34] PROBLEM - Puppet freshness on search14 is CRITICAL: Puppet has not run in the last 10 hours [16:35:34] PROBLEM - Puppet freshness on sq81 is CRITICAL: Puppet has not run in the last 10 hours [16:36:37] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [16:36:37] PROBLEM - Puppet freshness on search1024 is CRITICAL: Puppet has not run in the last 10 hours [16:36:37] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Puppet has not run in the last 10 hours [16:36:37] PROBLEM - Puppet freshness on search16 is CRITICAL: Puppet has not run in the last 10 hours [16:36:37] PROBLEM - Puppet freshness on search13 is CRITICAL: Puppet has not run in the last 10 hours [16:36:38] PROBLEM - Puppet freshness on search1018 is CRITICAL: Puppet has not run in the last 10 hours [16:36:38] PROBLEM - Puppet freshness on search17 is CRITICAL: Puppet has not run in the last 10 hours [16:36:39] PROBLEM - Puppet freshness on search20 is CRITICAL: Puppet has not run in the last 10 hours [16:36:39] PROBLEM - Puppet freshness on search21 is CRITICAL: Puppet has not run in the last 10 hours [16:36:40] PROBLEM - Puppet freshness on search19 is CRITICAL: Puppet has not run in the last 10 hours [16:36:40] PROBLEM - Puppet freshness on snapshot3 is CRITICAL: Puppet has not run in the last 10 hours [16:36:41] PROBLEM - Puppet freshness on search31 is CRITICAL: Puppet has not run in the last 10 hours [16:36:41] PROBLEM - Puppet freshness on sq36 is CRITICAL: Puppet has not run in the last 10 hours [16:36:42] PROBLEM - Puppet freshness on snapshot2 is CRITICAL: Puppet has not run in the last 10 hours [16:37:31] PROBLEM - Puppet freshness on mw10 is CRITICAL: Puppet has not run in the last 10 hours [16:37:31] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [16:47:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.717 seconds [17:21:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:31] PROBLEM - Puppet freshness on lardner is CRITICAL: Puppet has not run in the last 10 hours [17:22:31] PROBLEM - Puppet freshness on mc7 is CRITICAL: Puppet has not run in the last 10 hours [17:22:31] PROBLEM - Puppet freshness on search1005 is CRITICAL: Puppet has not run in the last 10 hours [17:22:31] PROBLEM - Puppet freshness on search1017 is CRITICAL: Puppet has not run in the last 10 hours [17:22:31] PROBLEM - Puppet freshness on search24 is CRITICAL: Puppet has not run in the last 10 hours [17:22:32] PROBLEM - Puppet freshness on search28 is CRITICAL: Puppet has not run in the last 10 hours [17:22:32] PROBLEM - Puppet freshness on search30 is CRITICAL: Puppet has not run in the last 10 hours [17:22:33] PROBLEM - Puppet freshness on search27 is CRITICAL: Puppet has not run in the last 10 hours [17:22:33] PROBLEM - Puppet freshness on search34 is CRITICAL: Puppet has not run in the last 10 hours [17:22:34] PROBLEM - Puppet freshness on search33 is CRITICAL: Puppet has not run in the last 10 hours [17:22:34] PROBLEM - Puppet freshness on sq76 is CRITICAL: Puppet has not run in the last 10 hours [17:22:35] PROBLEM - Puppet freshness on tola is CRITICAL: Puppet has not run in the last 10 hours [17:23:34] PROBLEM - Puppet freshness on mc10 is CRITICAL: Puppet has not run in the last 10 hours [17:23:34] PROBLEM - Puppet freshness on mc15 is CRITICAL: Puppet has not run in the last 10 hours [17:23:34] PROBLEM - Puppet freshness on analytics1004 is CRITICAL: Puppet has not run in the last 10 hours [17:23:34] PROBLEM - Puppet freshness on search1015 is CRITICAL: Puppet has not run in the last 10 hours [17:23:34] PROBLEM - Puppet freshness on sq74 is CRITICAL: Puppet has not run in the last 10 hours [17:24:37] PROBLEM - Puppet freshness on analytics1010 is CRITICAL: Puppet has not run in the last 10 hours [17:24:37] PROBLEM - Puppet freshness on analytics1014 is CRITICAL: Puppet has not run in the last 10 hours [17:24:37] PROBLEM - Puppet freshness on analytics1017 is CRITICAL: Puppet has not run in the last 10 hours [17:24:37] PROBLEM - Puppet freshness on analytics1018 is CRITICAL: Puppet has not run in the last 10 hours [17:24:37] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: Puppet has not run in the last 10 hours [17:24:38] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Puppet has not run in the last 10 hours [17:24:38] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: Puppet has not run in the last 10 hours [17:24:39] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Puppet has not run in the last 10 hours [17:34:49] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 2.187 seconds [17:54:38] PROBLEM - Puppet freshness on db59 is CRITICAL: Puppet has not run in the last 10 hours [18:04:31] PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours [18:04:31] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [18:04:31] PROBLEM - Puppet freshness on sq48 is CRITICAL: Puppet has not run in the last 10 hours [18:08:52] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.020 seconds [18:37:31] PROBLEM - Puppet freshness on db9 is CRITICAL: Puppet has not run in the last 10 hours [18:56:16] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:07:40] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.166 seconds [19:08:30] New patchset: Dereckson; "(bug 42853) Namespaces configuration for en.wikisource" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/37639 [19:10:03] New review: Dereckson; "shellpolicy" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/37639 [19:21:37] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [19:21:37] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [19:38:34] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [19:38:34] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [19:42:10] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.306 seconds [20:30:48] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:52] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.023 seconds [20:56:45] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [21:16:06] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:27:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.482 seconds [21:35:34] New patchset: Alex Monk; "(bug 41173) Don't hardcode enwiki URLs for MoodBar config" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/37649 [21:36:10] New patchset: Platonides; "(Bug 41173) Do not redirect Moodbar to enwiki from nlwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/37650 [21:37:57] Platonides, ^ [21:38:38] Only real difference that I can see is that your one specifies the Dutch special page name [21:38:55] Which imo isn't necessary [21:39:54] PROBLEM - Puppet freshness on ms-be3 is CRITICAL: Puppet has not run in the last 10 hours [21:41:27] redirecting to the English when it is localised is a bit ugly [21:41:34] I'm not sure if we can use the Linker/SpecialPage functions at this point in the config... [21:41:38] I don't know why the extension doesn't redirect to Special:FeedbackDashboard by itself [21:57:21] Platonides, well it's used in the JS. We could change it to use mw.util.wikiGetlink but then we'd still be pointing to the english specialpage name [22:03:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:34] doesn't that resolve the specialpage names? [22:06:07] we could pre-resolve it in php for the js, although I don't know how it is done [22:14:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.579 seconds [22:49:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:04:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.035 seconds [23:04:28] New patchset: Ryan Lane; "Remove denies from config - it's an incorrect rule" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37655 [23:09:53] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37655 [23:36:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:23] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [23:49:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.356 seconds [23:49:24] RoanKattouw_away: so, I'm pretty sure I know why things were looking flakey yesterday :D [23:49:33] Submodule 'cpp/contrib/pugixml' (git://github.com/zeux/pugixml.git) registered for path 'cpp/contrib/pugixml' [23:49:33] Cloning into 'cpp/contrib/double-conversion'... [23:49:35] ^^ [23:49:38] that hangs [23:49:39] forever [23:50:02] because those systems can't talk to github [23:50:07] and they shouldn't be anyway ;) [23:51:24] ah. seems it doesn't hang forever, but the timeout is incredibly long [23:51:48] this is also why the symlink isn't happening. it's failing out before then [23:52:10] the timeout for git submodule init is much longer than salt's timeout [23:57:20] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours