[00:06:56] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [00:06:56] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [00:06:56] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [00:06:56] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [00:06:56] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [00:06:57] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:56] PROBLEM - Puppet freshness on mchenry is CRITICAL: No successful Puppet run in the last 10 hours [01:06:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:07:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.142 second response time [01:46:47] PROBLEM - SSH on pdf2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:38] RECOVERY - SSH on pdf2 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [01:52:47] PROBLEM - SSH on pdf2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:54:38] RECOVERY - SSH on pdf2 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [02:12:01] !log LocalisationUpdate completed (1.22wmf12) at Sat Aug 10 02:12:01 UTC 2013 [02:12:13] Logged the message, Master [02:27:23] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Aug 10 02:27:23 UTC 2013 [02:27:34] Logged the message, Master [02:57:48] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:58:39] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.236 second response time [04:52:49] PROBLEM - Puppet freshness on db9 is CRITICAL: No successful Puppet run in the last 10 hours [05:02:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:03:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.142 second response time [05:06:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:07:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.138 second response time [05:08:04] (03PS1) 10Tim Landscheidt: Tools: Puppetize tips.sh [operations/puppet] - 10https://gerrit.wikimedia.org/r/78486 [05:10:34] (03CR) 10Tim Landscheidt: [C: 04-1] "Needs to be tested on toolsbeta first." [operations/puppet] - 10https://gerrit.wikimedia.org/r/78486 (owner: 10Tim Landscheidt) [05:19:39] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:22:29] (03PS2) 10Dzahn: add timezone for vi.wikivoyage (bug 52034) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78347 [05:22:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:23:29] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [05:24:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.138 second response time [05:26:36] (03PS2) 10Dzahn: wgSitename for vi.wikivoyage (bug 52034) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78355 [05:31:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:33:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [05:52:28] !log apt-get upgrading kaulen (bugzilla) [05:52:40] Logged the message, Master [05:54:39] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:55:30] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [05:57:43] !log deleting broken bzapi files from bugzilla server [05:57:54] Logged the message, Master [05:58:30] !log deleting codereview-proxy (pre-gerrit,svn) from bugzilla server [05:58:42] Logged the message, Master [06:02:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:03:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [06:22:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [06:31:29] PROBLEM - search indices - check lucene status page on search19 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 60051 bytes in 0.137 second response time [06:34:15] (03PS1) 10Dzahn: removing bzapi stuff, deactivated per RT #5582 [operations/puppet] - 10https://gerrit.wikimedia.org/r/78489 [06:35:44] (03CR) 10Aklapper: [C: 031] removing bzapi stuff, deactivated per RT #5582 [operations/puppet] - 10https://gerrit.wikimedia.org/r/78489 (owner: 10Dzahn) [06:36:00] (03CR) 10Dzahn: [C: 032] "bzapi be gone" [operations/puppet] - 10https://gerrit.wikimedia.org/r/78489 (owner: 10Dzahn) [06:43:01] can we delete this ? http://codereview-proxy.wikimedia.org/ [06:47:23] mutante: We still want to keep running E:CodeReview afaik [06:47:29] although you need to confirm with chad [06:48:56] mutante: yes, I think we need it :/ [06:50:20] alright, ehem, i had it removed for 5 minutes but restored it [06:50:37] that means we'll have to migrate it away from Tampa... [06:50:54] or it will disappear $someday [06:51:02] I think it can go [06:51:26] / If you can't directly access the remote SVN repo, you can set this [06:51:26] / to an offsite proxy running this fun little proxy tool: [06:51:48] As the repos are all now readonly... [06:52:07] I looked through the code and had a hard time verifying that it is used only for writes [06:52:32] but I think it'd be fair to just disable it and see what happens [06:53:00] because even read access is needed only for archival reasons; it's not like anybody's work would be massively disrupted if it went away for a little bit. [06:53:19] As long as all revisions have a diff in the code_rev table.. [06:53:45] Reedy: we could comment out $wgSubversionProxy in CommonSettings.php [06:53:53] that's easy to revert [06:54:23] and if things look alright then decom the service [06:55:04] creates a bug :) [06:57:03] thx all for input, pasted the above to 52694 [06:57:11] rand() apache would seem to be able to ping formey [06:58:05] but that != svn [07:01:38] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:03:29] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [07:05:23] mutante: mw1 in pmtpa would seemingly be able to wget files from svn.wikimedia.org/svnroot [07:05:34] ^directly wget [07:05:36] Which would suggest the proxy isn't needed [07:06:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:07:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [07:09:32] Reedy: confirmed. root@mw1:/tmp# wget http://svn.wikimedia.org/svnroot/mediawiki/GOODBYE [07:09:38] 2013-08-10 07:09:22 (10.2 MB/s) - `GOODBYE' saved [79/79] [07:09:45] I just did the exact same ;) [07:09:49] hehe [07:10:24] Aha [07:10:29] I think I've got some test criteria [07:10:39] Let me try ori-l's suggesting of disabling the proxy in MW config [07:10:43] cool [07:12:09] !log reedy synchronized wmf-config/CommonSettings.php 'Disable CR proxy' [07:12:19] Logged the message, Master [07:14:11] Oh [07:14:24] I wonder if it's there to prevent all the apaches having the svn client stuff installed [07:14:30] Disabling it does break it [07:15:52] !log reedy synchronized wmf-config/CommonSettings.php 'rv' [07:16:03] Logged the message, Master [07:16:43] 8725 rows with cr_diff being null [07:25:10] Reedy: could we just curl those URLs in a script? [07:25:28] I was just thinking of doing something similar [07:25:30] Directly, no [07:25:50] With the API enabled, in it's current state, MW will defer the diff loading to be done client side from the API using JS [07:27:16] example url? [07:27:55] http://www.mediawiki.org/wiki/Special:Code/Wikimedia/1052 [07:28:06] ^ No one else visit it!! [07:32:58] Reedy: another one [07:32:59] plz [07:36:36] https://www.mediawiki.org/wiki/Special:Code/pywikipedia/4360 through https://www.mediawiki.org/wiki/Special:Code/pywikipedia/4370 [07:36:46] All those have no diffs [07:37:26] I think the wikimedia repo is now down to only having refs that won't get ever have a diff due to folder adding and such [07:40:01] {"code":{"rev":{"repo":"pywikipedia","id":4370,"diff":"Diff too large."}}} [07:40:06] maybe that's another reason [07:41:40] Yeah [07:42:14] I think we cache them when it's too large [07:42:17] cache/write to db [07:44:08] Though, that revision would look to have a huge diff in the db table [07:44:35] length(cr_diff) = 534736 [07:46:02] ori-l: http://p.defau.lt/?7uU6kOx6VZ_JohopMl7RjA [07:49:51] * Reedy has a hacked up maintenance script [07:53:28] ori-l: http://p.defau.lt/?yXDO4WPII3X1Qzcslkfo7w [07:54:15] heading to a talk but looks cool [08:23:40] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:24:40] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [08:37:06] (03PS1) 10Dzahn: add RewriteRule for Bugzilla, page.cgi?id=bug-writing.html to http://www.mediawiki.org/wiki/How_to_report_a_bug (bug 45271) [operations/puppet] - 10https://gerrit.wikimedia.org/r/78499 [08:39:23] (03PS2) 10Dzahn: add RewriteRule for Bugzilla, page.cgi?id=bug-writing.html to http://www.mediawiki.org/wiki/How_to_report_a_bug (bug 45271) [operations/puppet] - 10https://gerrit.wikimedia.org/r/78499 [08:40:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:41:51] (03CR) 10Dzahn: [C: 032] "wfm" [operations/puppet] - 10https://gerrit.wikimedia.org/r/78499 (owner: 10Dzahn) [08:42:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.122 second response time [08:44:24] (03CR) 10Dzahn: "curl https://bugzilla.wikimedia.org/page.cgi?id=bug-writing.html" [operations/puppet] - 10https://gerrit.wikimedia.org/r/78499 (owner: 10Dzahn) [08:56:14] Just pulling MW diffs is generating quite a bit of load on formey [08:56:14] http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=cpu_report&s=by+name&c=Miscellaneous+pmtpa&h=formey.wikimedia.org&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 [09:00:53] (03PS1) 10Aklapper: Add RewriteRule for Bugzilla, page.cgi?id=fields.html to https://www.mediawiki.org/wiki/Bugzilla/Fields [operations/puppet] - 10https://gerrit.wikimedia.org/r/78501 [09:02:39] (03CR) 10Dzahn: [C: 032] "just like 78499" [operations/puppet] - 10https://gerrit.wikimedia.org/r/78501 (owner: 10Aklapper) [09:10:30] (03PS1) 10TTO: Deny reupload permission to users and autoconfirmed on ckbwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78502 [09:28:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:30:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 4.413 second response time [09:35:43] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [09:35:43] PROBLEM - Puppet freshness on holmium is CRITICAL: No successful Puppet run in the last 10 hours [09:35:43] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [09:35:43] PROBLEM - Puppet freshness on pdf3 is CRITICAL: No successful Puppet run in the last 10 hours [09:35:43] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: No successful Puppet run in the last 10 hours [09:35:44] PROBLEM - Puppet freshness on sq41 is CRITICAL: No successful Puppet run in the last 10 hours [09:35:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 6.291 second response time [10:07:14] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [10:07:14] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [10:07:14] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [10:07:14] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [10:07:14] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [10:07:15] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [10:23:14] PROBLEM - Puppet freshness on mchenry is CRITICAL: No successful Puppet run in the last 10 hours [10:49:39] (03PS1) 10TTO: Set up Babel categories for eswikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78505 [10:50:56] (03PS2) 10TTO: Set up Babel categories for eswikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78505 [11:02:12] !log reedy synchronized wmf-config/CommonSettings.php [11:02:23] Logged the message, Master [11:05:24] (03PS1) 10TTO: Set up babel categories for ruwikinews [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78506 [11:07:27] (03PS3) 10TTO: Set up Babel categories for eswikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78505 [11:21:07] (03PS1) 10TTO: Add missing HTTP error pages on bits.wm.o [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78507 [11:30:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:32:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.153 second response time [12:32:09] !log reedy synchronized wmf-config/CommonSettings.php [12:55:51] git.wikimedia.org seemed to be dead [13:46:25] PROBLEM - SSH on pdf2 is CRITICAL: Server answer: [13:55:25] RECOVERY - SSH on pdf2 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [14:02:14] (03CR) 10Reedy: [C: 04-1] "As we currently use /usr/local/apache/common in our Apache configs, these should really link there" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78507 (owner: 10TTO) [14:03:15] sigh, antimony, what now? https://ganglia.wikimedia.org/latest/?r=day&cs=&ce=&c=Miscellaneous+eqiad&h=antimony.wikimedia.org&tab=m&vn=&mc=2&z=medium&metric_group=ALLGROUPS [14:03:28] indeed Reedy, since midnight [14:04:14] Google seem like a good bet [14:07:48] I've emailed opsen [14:22:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [14:32:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [14:52:57] PROBLEM - Puppet freshness on db9 is CRITICAL: No successful Puppet run in the last 10 hours [15:31:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:32:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.135 second response time [15:45:31] (03PS1) 10Danny B.: cswikisource: Set AbuseFilter notifications [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78533 [16:27:32] paravoid: Thanks [16:27:43] Looks like it lasted maybe half an hour before doing it again :( [16:36:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:37:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [17:57:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:58:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [18:23:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:24:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [18:39:21] (03PS2) 10Tim Landscheidt: Tools: Puppetize tips.sh [operations/puppet] - 10https://gerrit.wikimedia.org/r/78486 [18:58:37] (03CR) 10Tim Landscheidt: [C: 04-1] Tools: Puppetize tips.sh [operations/puppet] - 10https://gerrit.wikimedia.org/r/78486 (owner: 10Tim Landscheidt) [19:02:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:03:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [19:08:09] (03PS3) 10Tim Landscheidt: Tools: Puppetize tips.sh [operations/puppet] - 10https://gerrit.wikimedia.org/r/78486 [19:36:54] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [19:36:54] PROBLEM - Puppet freshness on holmium is CRITICAL: No successful Puppet run in the last 10 hours [19:36:54] PROBLEM - Puppet freshness on pdf3 is CRITICAL: No successful Puppet run in the last 10 hours [19:36:54] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [19:36:54] PROBLEM - Puppet freshness on sq41 is CRITICAL: No successful Puppet run in the last 10 hours [19:36:55] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: No successful Puppet run in the last 10 hours [19:41:47] (03PS2) 10Dereckson: Set up babel categories for ruwikinews [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78506 (owner: 10TTO) [19:42:01] (03CR) 10Dereckson: [C: 031] Set up Babel categories for ru.wikinews [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78506 (owner: 10TTO) [19:47:49] (03PS1) 10Dereckson: Enable NewUserMessage extension on ckb.wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/78613 [20:07:55] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [20:07:55] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [20:07:55] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [20:07:55] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [20:07:55] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [20:07:56] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [20:23:54] PROBLEM - Puppet freshness on mchenry is CRITICAL: No successful Puppet run in the last 10 hours [20:57:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [21:08:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:10:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [21:18:32] (03CR) 10Tim Landscheidt: "Tested on toolsbeta. Bail out "just in case": "ssh $HOST rm -f /etc/profile.d/tips.sh"." [operations/puppet] - 10https://gerrit.wikimedia.org/r/78486 (owner: 10Tim Landscheidt) [21:29:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:30:46] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 6.689 second response time [22:29:16] RECOVERY - search indices - check lucene status page on search19 is OK: HTTP OK: HTTP/1.1 200 OK - 60075 bytes in 0.113 second response time [23:23:52] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:24:42] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [23:55:18] !log restarted gitblit on antimony [23:55:30] Logged the message, Mistress of the network gear. [23:59:03] Reedy: ^^