[00:00:42] <gerrit-wm>	 New patchset: CSteipp; "Add Ex:MergeUser and Ex:GeoCrumbs to Wikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30310
[00:01:39] <mutante>	 i can also login on wikidatawiki with central auth account:)
[00:01:47] <Reedy>	 yeah, works straight away :D
[00:02:05] <Reedy>	 mutante: Does nginx need restarting too after puppet has run?
[00:03:20] <mutante>	 i dont think so, we just changed apache config
[00:04:03] <Reedy>	 for the ssl cert update?
[00:04:09] <Reedy>	 or does it pick it up automatically?
[00:04:12] <mutante>	 oh, the other one
[00:04:29] <mutante>	 that it might, yea
[00:04:34] <mutante>	 need restart
[00:05:03] <mutante>	 unless puppet does it lets see
[00:06:11] <mutante>	 i see a notify for Service nginx in puppet when nginx.conf changes
[00:08:08] <mutante>	 watching a puppet run ..brb
[00:10:53] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.904 seconds
[00:11:36] <mutante>	 /Stage[main]/Protoproxy::Proxy_sites/Proxy_configuration[wikidata]/Nginx_site[wikidata]/File[/etc/nginx/sites-available/wikidata]/content) content changed
[00:12:45] <Reedy>	 sounds promising
[00:12:46] <mutante>	 hmm, not really the restart, because it is not nginx.conf
[00:14:36] <Reedy>	 Ryan_Lane: After updating protoproxy does anything need doing to nginx to make it take certificate changes etc?
[00:17:46] <mutante>	 !log reloading nginx on ssl1001
[00:17:54] <mutante>	 configtest and reload are fine
[00:17:59] <morebots>	 Logged the message, Master
[00:18:21] <mutante>	 not sure about reload vs. restart
[00:18:45] <mutante>	 for this specific change
[00:19:20] <mutante>	 my local connection just keeps dropping ..grrr
[00:24:51] <Danny_B|backup>	 ha, mutante awake!
[00:25:25] <mutante>	 !log restarting nginx on ssl hosts for changed server name for wikidata
[00:25:38] <morebots>	 Logged the message, Master
[00:26:28] <mutante>	 Reedy: the changed has been applied and i restarted them, after all it was a change to a server_name
[00:26:42] <mutante>	 them = ssl1-4 and 1001-1004
[00:27:15] <mutante>	 the ones that include protoproxy
[00:29:08] <Reedy>	 Still reporting as giving *.wikimedia.org
[00:29:09] <Reedy>	 Hmm
[00:47:04] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:58:44] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.900 seconds
[01:32:38] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:42:02] * aude  checks in 

[01:42:14] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 274 seconds
[01:44:15] <aude>	 mutante: if you didn't figure out, the ULS sets the language of the sidebar and everything and it should be sticky
[01:46:53] <nagios-wm>	 PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours
[01:46:53] <nagios-wm>	 PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours
[01:46:53] <nagios-wm>	 PROBLEM - Puppet freshness on storage3 is CRITICAL: Puppet has not run in the last 10 hours
[01:46:53] <nagios-wm>	 PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours
[01:47:11] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 16 seconds
[01:47:30] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.033 seconds
[01:51:35] <gerrit-wm>	 New patchset: CSteipp; "Add extensions for dewikivoyage into labs" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30319
[02:00:19] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 272 seconds
[02:20:34] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[02:33:27] <logmsgbot_>	 !log LocalisationUpdate completed (1.21wmf2) at Sat Oct 27 02:33:26 UTC 2012
[02:33:43] <morebots>	 Logged the message, Master
[02:35:25] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.190 seconds
[03:27:10] <nagios-wm>	 RECOVERY - Puppet freshness on sq76 is OK: puppet ran at Sat Oct 27 03:26:57 UTC 2012
[03:46:38] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds
[03:57:17] <nagios-wm>	 PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours
[04:43:10] <nagios-wm>	 PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours
[04:43:10] <nagios-wm>	 PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours
[05:00:07] <nagios-wm>	 PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours
[05:42:21] <nagios-wm>	 PROBLEM - MySQL Replication Heartbeat on db1042 is CRITICAL: CRIT replication delay 181 seconds
[05:42:48] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db1042 is CRITICAL: CRIT replication delay 203 seconds
[05:44:00] <nagios-wm>	 RECOVERY - MySQL Replication Heartbeat on db1042 is OK: OK replication delay 0 seconds
[05:44:27] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db1042 is OK: OK replication delay 0 seconds
[06:41:51] <nagios-wm>	 PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours
[07:16:20] <nagios-wm>	 PROBLEM - MySQL Slave Delay on db1042 is CRITICAL: CRIT replication delay 182 seconds
[07:17:34] <nagios-wm>	 PROBLEM - MySQL Replication Heartbeat on db1042 is CRITICAL: CRIT replication delay 224 seconds
[07:41:00] <apergos>	 !log depooled sq82 in pybal, squid3 was running on it again and it's still precise
[07:41:14] <morebots>	 Logged the message, Master
[07:42:36] <apergos>	 maybe that's wrong, I only saw one squid process over there
[07:42:39] <apergos>	 *sigh*
[08:00:28] <nagios-wm>	 PROBLEM - Backend Squid HTTP on sq82 is CRITICAL: Connection refused
[08:07:49] <nagios-wm>	 RECOVERY - MySQL Slave Delay on db1042 is OK: OK replication delay 5 seconds
[08:07:49] <nagios-wm>	 RECOVERY - MySQL Replication Heartbeat on db1042 is OK: OK replication delay 0 seconds
[08:45:40] <gerrit-wm>	 New patchset: Faidon; "dhcp: remove explicit selection of the precise-installer" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/30323
[08:49:00] <paravoid>	 !log reinstalling sq82 with lucid
[08:49:14] <morebots>	 Logged the message, Master
[08:51:37] <nagios-wm>	 PROBLEM - Host sq82 is DOWN: PING CRITICAL - Packet loss = 100%
[08:56:28] <gerrit-wm>	 Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/30323
[08:57:19] <nagios-wm>	 RECOVERY - Host sq82 is UP: PING OK - Packet loss = 0%, RTA = 0.48 ms
[09:01:22] <nagios-wm>	 PROBLEM - SSH on sq82 is CRITICAL: Connection refused
[09:04:40] <nagios-wm>	 RECOVERY - SSH on sq82 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0)
[09:28:39] <nagios-wm>	 PROBLEM - NTP on sq82 is CRITICAL: NTP CRITICAL: No response from NTP server
[09:37:22] <gerrit-wm>	 New patchset: Faidon; "autoinstall: unify squid/varnish partman with raid1" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/30324
[09:38:28] <gerrit-wm>	 Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/30324
[09:38:47] <paravoid>	 meh, who needs testing.
[09:46:20] <nagios-wm>	 RECOVERY - Frontend Squid HTTP on sq82 is OK: HTTP OK HTTP/1.0 200 OK - 604 bytes in 0.004 seconds
[09:49:55] <nagios-wm>	 RECOVERY - NTP on sq82 is OK: NTP OK: Offset 0.02965557575 secs
[09:58:55] <nagios-wm>	 RECOVERY - Backend Squid HTTP on sq82 is OK: HTTP OK HTTP/1.0 200 OK - 459 bytes in 0.016 seconds
[10:02:12] <paravoid>	 !log repooling sq82 frontend in pybal
[10:02:27] <morebots>	 Logged the message, Master
[10:04:42] <paravoid>	 seems to work
[11:48:15] <nagios-wm>	 PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours
[11:48:15] <nagios-wm>	 PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours
[11:48:15] <nagios-wm>	 PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours
[11:48:15] <nagios-wm>	 PROBLEM - Puppet freshness on storage3 is CRITICAL: Puppet has not run in the last 10 hours
[13:29:14] <nagios-wm>	 PROBLEM - HTTP on kaulen is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:42:00] <nagios-wm>	 PROBLEM - SSH on kaulen is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:46:48] <nagios-wm>	 RECOVERY - SSH on kaulen is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0)
[13:47:15] <nagios-wm>	 RECOVERY - HTTP on kaulen is OK: HTTP OK HTTP/1.1 200 OK - 461 bytes in 3.368 seconds
[13:58:48] <nagios-wm>	 PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours
[14:44:48] <nagios-wm>	 PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours
[14:44:48] <nagios-wm>	 PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours
[15:01:44] <nagios-wm>	 PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours
[16:43:03] <nagios-wm>	 PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours
[19:51:00] <ori-l>	 @search rss
[19:51:00] <wm-bot>	 No results were found, remember, the bot is searching through content of keys and their names
[20:51:05] <logmsgbot_>	 !log reedy synchronized wmf-config/CommonSettings.php  'Comment out wgULSGeoService'
[20:51:18] <morebots>	 Logged the message, Master
[21:19:45] <logmsgbot_>	 !log reedy synchronized wmf-config/CommonSettings.php  'Set geoservice to geoiplookup.wm.o'
[21:19:58] <morebots>	 Logged the message, Master
[21:49:13] <nagios-wm>	 PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours
[21:49:13] <nagios-wm>	 PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours
[21:49:13] <nagios-wm>	 PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours
[21:49:13] <nagios-wm>	 PROBLEM - Puppet freshness on storage3 is CRITICAL: Puppet has not run in the last 10 hours
[21:49:38] <gerrit-wm>	 New patchset: Aude; "redirect wikimania.wikimedia.org to wikimania2013 site" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/30421
[22:05:00] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:06:28] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.054 seconds
[23:15:00] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:18:09] <nagios-wm>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.103 seconds
[23:40:41] <gerrit-wm>	 New review: Jeremyb; "would be approved if I could ;)" [operations/apache-config] (master) C: 1;  - https://gerrit.wikimedia.org/r/30421
[23:41:32] * jeremyb  waves aude

[23:52:58] <aude>	 heh :)
[23:53:19] <Krenair>	 I would +1 but there's not much point, heh
[23:53:21] <jeremyb>	 oh, you're there. /me wasn't sure. 2am and all ;)
[23:53:57] * aude  not tired

[23:54:09] <nagios-wm>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:54:28] <aude>	 now if you have any ideas about fixing wikidata.org and making the language sticky :)
[23:54:41] <aude>	 squid config or whatever it is
[23:55:01] <jeremyb>	 nothing's in squid i think
[23:55:10] <jeremyb>	 sticky?
[23:55:20] <aude>	 http://www.wikidata.org/wiki/Special:RecentChanges
[23:55:27] <jeremyb>	 *click*
[23:55:35] <aude>	 appears in arabic in chrome; in norwegian in firefox
[23:55:47] <aude>	 it's random other language for other people sometimes
[23:55:57] * aude  seen dutch, icelandic, spanish :)

[23:56:46] <aude>	 no ssl yet but that's coming
[23:57:01] <aude>	 https://bugzilla.wikimedia.org/show_bug.cgi?id=41451
[23:57:43] <jeremyb>	 tell me more about these langs?
[23:57:45] <jeremyb>	 headers?
[23:58:09] <aude>	 let's see....
[23:58:20] <jeremyb>	 ahhh, i see that's the bug
[23:58:22] <aude>	 the special pages seem especially stuck
[23:58:28] <jeremyb>	 i figured the bug was about ssl not langs
[23:58:29] <aude>	 recent changes
[23:58:30] * jeremyb  reads

[23:58:55] <aude>	 ssl is simple to do but will be next week or whenever
[23:58:59] <jeremyb>	 are any other wikimedia wikis using ULS?
[23:59:11] <jeremyb>	 ssl probably needs a robh?
[23:59:15] <jeremyb>	 iirc
[23:59:33] <nagios-wm>	 PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours