[00:07:14] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [00:22:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [00:31:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:32:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [00:35:12] New review: Ottomata; "SO COOL! What happens when there is already a repo at wikimedia/puppet-cdh4? Do we need to do some..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71248 [01:07:55] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [01:10:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [01:22:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:23:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [01:31:46] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -0.00390458107 secs [01:33:36] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -0.001895427704 secs [01:39:14] PROBLEM - Puppet freshness on mc1005 is CRITICAL: No successful Puppet run in the last 10 hours [01:39:14] PROBLEM - Puppet freshness on mw41 is CRITICAL: No successful Puppet run in the last 10 hours [01:40:14] PROBLEM - Puppet freshness on amssq44 is CRITICAL: No successful Puppet run in the last 10 hours [01:40:15] PROBLEM - Puppet freshness on db1036 is CRITICAL: No successful Puppet run in the last 10 hours [01:41:14] PROBLEM - Puppet freshness on ms-be3 is CRITICAL: No successful Puppet run in the last 10 hours [01:42:14] PROBLEM - Puppet freshness on mw58 is CRITICAL: No successful Puppet run in the last 10 hours [01:42:14] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [01:42:14] PROBLEM - Puppet freshness on sq42 is CRITICAL: No successful Puppet run in the last 10 hours [01:46:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [01:52:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:53:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.141 second response time [02:05:46] !log LocalisationUpdate completed (1.22wmf8) at Sun Jun 30 02:05:45 UTC 2013 [02:09:14] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 10 hours [02:09:14] PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 10 hours [02:09:14] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: No successful Puppet run in the last 10 hours [02:09:14] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: No successful Puppet run in the last 10 hours [02:09:14] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: No successful Puppet run in the last 10 hours [02:09:59] !log LocalisationUpdate completed (1.22wmf9) at Sun Jun 30 02:09:59 UTC 2013 [02:10:28] PROBLEM - Puppet freshness on amssq34 is CRITICAL: No successful Puppet run in the last 10 hours [02:10:29] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [02:10:29] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [02:10:29] PROBLEM - Puppet freshness on analytics1025 is CRITICAL: No successful Puppet run in the last 10 hours [02:10:29] PROBLEM - Puppet freshness on analytics1010 is CRITICAL: No successful Puppet run in the last 10 hours [02:11:28] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [02:11:28] PROBLEM - Puppet freshness on db1029 is CRITICAL: No successful Puppet run in the last 10 hours [02:11:28] PROBLEM - Puppet freshness on analytics1023 is CRITICAL: No successful Puppet run in the last 10 hours [02:11:28] PROBLEM - Puppet freshness on es6 is CRITICAL: No successful Puppet run in the last 10 hours [02:11:28] PROBLEM - Puppet freshness on mw1004 is CRITICAL: No successful Puppet run in the last 10 hours [02:12:27] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [02:12:27] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [02:12:27] PROBLEM - Puppet freshness on cp3009 is CRITICAL: No successful Puppet run in the last 10 hours [02:12:27] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [02:12:27] PROBLEM - Puppet freshness on cp3012 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:28] PROBLEM - Puppet freshness on cp1002 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:29] PROBLEM - Puppet freshness on cp1049 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:29] PROBLEM - Puppet freshness on cp1005 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:29] PROBLEM - Puppet freshness on db39 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:29] PROBLEM - Puppet freshness on ekrem is CRITICAL: No successful Puppet run in the last 10 hours [02:13:29] PROBLEM - Puppet freshness on helium is CRITICAL: No successful Puppet run in the last 10 hours [02:13:30] PROBLEM - Puppet freshness on ms10 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:30] PROBLEM - Puppet freshness on mc1001 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:31] PROBLEM - Puppet freshness on mw1043 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:31] PROBLEM - Puppet freshness on mw1063 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:31] PROBLEM - Puppet freshness on mw1087 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:32] PROBLEM - Puppet freshness on mw1106 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:33] PROBLEM - Puppet freshness on mw124 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:33] PROBLEM - Puppet freshness on mw20 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:33] PROBLEM - Puppet freshness on mw43 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:34] PROBLEM - Puppet freshness on mw57 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:35] PROBLEM - Puppet freshness on srv255 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:35] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:35] PROBLEM - Puppet freshness on stat1 is CRITICAL: No successful Puppet run in the last 10 hours [02:13:36] PROBLEM - Puppet freshness on titanium is CRITICAL: No successful Puppet run in the last 10 hours [02:17:33] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 30 02:17:33 UTC 2013 [02:23:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:24:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [02:40:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:41:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.148 second response time [03:08:29] New patchset: Jforrester; "Enable VisualEditor for all logged-in users of the English Wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71261 [03:08:31] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [03:12:29] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [03:12:29] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [03:12:29] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [03:12:29] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [03:12:29] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [03:12:29] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [03:12:29] PROBLEM - Puppet freshness on sodium is CRITICAL: No successful Puppet run in the last 10 hours [03:12:30] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [03:12:30] New review: Catrope; "(1 comment)" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/71261 [03:12:31] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [03:17:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:18:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [03:32:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:33:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [03:35:46] New patchset: Krinkle; "Make 'visualeditor-enable' pref hidden on wikis where VE is default" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/70652 [03:35:55] New patchset: Krinkle; "Enable VisualEditor for all logged-in users of the English Wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71261 [03:40:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:41:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [03:51:59] New patchset: Krinkle; "Set wgVisualEditorDisableForAnons = true for enwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71261 [03:57:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:58:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 7.292 second response time [03:59:45] New patchset: Catrope; "Set wgVisualEditorDisableForAnons = true for enwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71261 [04:00:03] New review: Catrope; "Looks good now. Do not deploy until July 1" [operations/mediawiki-config] (master) C: -2; - https://gerrit.wikimedia.org/r/70652 [04:04:22] New patchset: Catrope; "Set wgVisualEditorDisableForAnons = true for enwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71261 [04:05:56] New patchset: Catrope; "Enable VisualEditor for all logged-in users (but not anons) on enwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71261 [04:06:31] New review: Catrope; "This is a good-to-go state now. Do not deploy until July 1." [operations/mediawiki-config] (master) C: -2; - https://gerrit.wikimedia.org/r/71261 [04:07:59] PROBLEM - Varnish HTTP mobile-frontend on cp3012 is CRITICAL: HTTP CRITICAL - No data received from host [04:08:30] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [04:16:29] RECOVERY - Varnish HTTP mobile-frontend on cp3012 is OK: HTTP OK: HTTP/1.1 200 OK - 705 bytes in 0.176 second response time [04:27:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:28:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [04:51:48] PROBLEM - RAID on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:52:48] RECOVERY - RAID on mc15 is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [04:57:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:58:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [05:10:01] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [05:59:51] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server [06:00:11] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [06:07:10] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [06:30:20] RECOVERY - Disk space on ms-be1002 is OK: DISK OK [06:31:21] RECOVERY - Disk space on ms-be1001 is OK: DISK OK [07:00:18] PROBLEM - SSH on cp1043 is CRITICAL: Server answer: [07:00:38] PROBLEM - SSH on cp1044 is CRITICAL: Server answer: [07:03:18] RECOVERY - SSH on cp1043 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [07:03:39] RECOVERY - SSH on cp1044 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [07:12:24] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [07:15:34] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [07:28:24] RECOVERY - LDAP on sanger is OK: TCP OK - 0.027 second response time on port 389 [07:31:34] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [07:31:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:32:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [08:02:32] RECOVERY - LDAP on sanger is OK: TCP OK - 3.028 second response time on port 389 [08:05:42] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [08:08:02] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [08:23:43] New patchset: MaxSem; "Switch GeoData to implicit commits" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71273 [08:24:49] Change merged: MaxSem; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71273 [08:28:37] !log maxsem synchronized wmf-config/CommonSettings.php 'https://gerrit.wikimedia.org/r/71273' [08:30:37] RECOVERY - LDAP on sanger is OK: TCP OK - 0.027 second response time on port 389 [08:30:54] !log maxsem synchronized php-1.22wmf8/extensions/GeoData/solrupdate.php [08:32:27] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -0.000781416893 secs [08:32:47] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -0.00251185894 secs [08:33:47] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [08:35:37] RECOVERY - LDAP on sanger is OK: TCP OK - 3.027 second response time on port 389 [08:37:49] !log maxsem synchronized php-1.22wmf8/extensions/GeoData [08:39:14] !log maxsem synchronized php-1.22wmf9/extensions/GeoData [08:39:47] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [08:52:23] PROBLEM - Puppet freshness on wtp1015 is CRITICAL: No successful Puppet run in the last 10 hours [08:59:23] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [09:04:44] RECOVERY - LDAP on sanger is OK: TCP OK - 1.023 second response time on port 389 [09:06:25] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: No successful Puppet run in the last 10 hours [09:07:46] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [09:08:37] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [09:09:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:10:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [09:22:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.135 second response time [09:34:46] RECOVERY - LDAP on sanger is OK: TCP OK - 1.027 second response time on port 389 [09:37:46] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [09:43:40] RECOVERY - LDAP on sanger is OK: TCP OK - 1.024 second response time on port 389 [09:46:50] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [09:52:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.122 second response time [09:53:41] RECOVERY - LDAP on sanger is OK: TCP OK - 1.026 second response time on port 389 [09:55:30] PROBLEM - DPKG on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:56:20] RECOVERY - DPKG on mc15 is OK: All packages OK [09:56:50] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [10:02:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:03:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [10:08:42] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [10:23:43] RECOVERY - LDAP on sanger is OK: TCP OK - 3.028 second response time on port 389 [10:26:43] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [10:31:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [10:32:42] RECOVERY - LDAP on sanger is OK: TCP OK - 0.030 second response time on port 389 [10:35:52] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [11:00:37] New review: JanZerebecki; "Maybe a missing [PT] flag https://httpd.apache.org/docs/2.4/rewrite/flags.html#flag_pt is the cause ..." [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65443 [11:02:44] RECOVERY - LDAP on sanger is OK: TCP OK - 1.028 second response time on port 389 [11:05:44] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [11:08:22] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [11:37:47] RECOVERY - LDAP on sanger is OK: TCP OK - 3.028 second response time on port 389 [11:39:17] PROBLEM - Puppet freshness on mc1005 is CRITICAL: No successful Puppet run in the last 10 hours [11:39:17] PROBLEM - Puppet freshness on mw41 is CRITICAL: No successful Puppet run in the last 10 hours [11:40:17] PROBLEM - Puppet freshness on amssq44 is CRITICAL: No successful Puppet run in the last 10 hours [11:40:17] PROBLEM - Puppet freshness on db1036 is CRITICAL: No successful Puppet run in the last 10 hours [11:40:49] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [11:41:17] PROBLEM - Puppet freshness on ms-be3 is CRITICAL: No successful Puppet run in the last 10 hours [11:42:18] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [11:42:18] PROBLEM - Puppet freshness on mw58 is CRITICAL: No successful Puppet run in the last 10 hours [11:42:18] PROBLEM - Puppet freshness on sq42 is CRITICAL: No successful Puppet run in the last 10 hours [11:43:47] RECOVERY - LDAP on sanger is OK: TCP OK - 3.028 second response time on port 389 [11:45:17] PROBLEM - Disk space on ms-be1001 is CRITICAL: DISK CRITICAL - free space: / 5690 MB (3% inode=98%): [11:46:48] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [12:01:47] RECOVERY - LDAP on sanger is OK: TCP OK - 3.030 second response time on port 389 [12:04:47] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [12:09:02] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [12:09:33] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:33] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:33] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:33] PROBLEM - Puppet freshness on amssq31 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:33] PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 10 hours [12:11:32] PROBLEM - Puppet freshness on db1029 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:32] PROBLEM - Puppet freshness on manganese is CRITICAL: No successful Puppet run in the last 10 hours [12:11:32] PROBLEM - Puppet freshness on db58 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:32] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:32] PROBLEM - Puppet freshness on ms-be2 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:42] RECOVERY - LDAP on sanger is OK: TCP OK - 1.024 second response time on port 389 [12:12:32] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:32] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:32] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:32] PROBLEM - Puppet freshness on cp3009 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:32] PROBLEM - Puppet freshness on cp3012 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:33] PROBLEM - Puppet freshness on cp1002 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:33] PROBLEM - Puppet freshness on cp1005 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:34] PROBLEM - Puppet freshness on cp1049 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:34] PROBLEM - Puppet freshness on db39 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:34] PROBLEM - Puppet freshness on helium is CRITICAL: No successful Puppet run in the last 10 hours [12:13:34] PROBLEM - Puppet freshness on ekrem is CRITICAL: No successful Puppet run in the last 10 hours [12:13:34] PROBLEM - Puppet freshness on mc1001 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:35] PROBLEM - Puppet freshness on ms10 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:35] PROBLEM - Puppet freshness on mw1043 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:35] PROBLEM - Puppet freshness on mw1087 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:36] PROBLEM - Puppet freshness on mw1106 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:37] PROBLEM - Puppet freshness on mw1063 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:37] PROBLEM - Puppet freshness on mw124 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:37] PROBLEM - Puppet freshness on mw20 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:38] PROBLEM - Puppet freshness on mw57 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:39] PROBLEM - Puppet freshness on mw43 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:39] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:39] PROBLEM - Puppet freshness on srv255 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:40] PROBLEM - Puppet freshness on stat1 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:41] PROBLEM - Puppet freshness on titanium is CRITICAL: No successful Puppet run in the last 10 hours [12:17:52] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [12:31:45] New patchset: Matthias Mullie; "Add cron to generate CSVs for ee-dashboard" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71282 [12:34:54] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:36:42] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [12:42:47] RECOVERY - LDAP on sanger is OK: TCP OK - 3.029 second response time on port 389 [12:45:56] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [12:51:48] PROBLEM - Disk space on ms-be1002 is CRITICAL: DISK CRITICAL - free space: / 5700 MB (3% inode=98%): [12:59:47] RECOVERY - LDAP on sanger is OK: TCP OK - 3.030 second response time on port 389 [13:02:57] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [13:06:40] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [13:13:19] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [13:13:19] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [13:13:20] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [13:13:20] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [13:13:20] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [13:13:20] PROBLEM - Puppet freshness on sodium is CRITICAL: No successful Puppet run in the last 10 hours [13:13:20] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [13:21:39] PROBLEM - Disk space on mc15 is CRITICAL: Timeout while attempting connection [13:22:30] RECOVERY - Disk space on mc15 is OK: DISK OK [13:30:50] RECOVERY - LDAP on sanger is OK: TCP OK - 0.027 second response time on port 389 [13:33:59] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [13:51:35] New patchset: coren; "Tool Labs: install libpoe-perl (user request)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71288 [13:52:29] New review: coren; "Trivial package add." [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/71288 [13:52:30] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71288 [14:02:54] RECOVERY - LDAP on sanger is OK: TCP OK - 3.029 second response time on port 389 [14:05:54] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [14:07:37] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [14:19:27] RECOVERY - LDAP on sanger is OK: TCP OK - 1.025 second response time on port 389 [14:22:37] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [14:30:07] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:30:57] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [14:31:38] PROBLEM - SSH on searchidx1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:38] RECOVERY - SSH on searchidx1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [14:35:57] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:37:20] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [14:40:31] RECOVERY - LDAP on sanger is OK: TCP OK - 1.027 second response time on port 389 [14:43:40] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [15:04:31] RECOVERY - LDAP on sanger is OK: TCP OK - 0.026 second response time on port 389 [15:07:09] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [15:08:38] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [15:10:08] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [15:16:38] RECOVERY - LDAP on sanger is OK: TCP OK - 3.030 second response time on port 389 [15:19:40] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [15:56:31] RECOVERY - LDAP on sanger is OK: TCP OK - 1.023 second response time on port 389 [15:59:42] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [16:03:31] RECOVERY - LDAP on sanger is OK: TCP OK - 0.027 second response time on port 389 [16:07:00] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:07:41] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [16:18:32] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:19:21] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [16:19:41] RECOVERY - LDAP on sanger is OK: TCP OK - 0.027 second response time on port 389 [16:22:50] PROBLEM - LDAP on sanger is CRITICAL: Connection timed out [16:23:23] did gerrit justgo down? [16:24:33] AzaToth: wfm [16:25:33] MatmaRex: prolly a bug then [16:25:45] MatmaRex: was adding a reviewer, and it never exited "Working..." [16:28:19] yes. ldap on sanger. grrr [16:28:28] has that been happening all day? [16:30:21] ugh [16:40:45] AzaToth: the 'working...' is a known bug. Hard refresh + cache clear. also supposedly works better on chrome [16:40:55] k [16:41:54] YuviPanda: what do you think about https://gerrit.wikimedia.org/r/#/c/71245/ [16:47:43] RECOVERY - LDAP on sanger is OK: TCP OK - 0.027 second response time on port 389 [16:50:49] !log ldap on sanger has been flapping all day. just restarted it, it seeeeems happier, time will tell [17:12:32] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [17:16:14] AzaToth: not sure if that's something I'd want to comment on. Haven't really used that at all [17:40:43] PROBLEM - RAID on ms-be1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [17:43:33] RECOVERY - RAID on ms-be1002 is OK: OK: State is Optimal, checked 1 logical device(s) [18:07:05] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [18:40:31] hi, I have a developer which is unable to create an account on https://wikitech.wikimedia.org/wiki/Special:UserLogin/signup [18:40:53] Each time he tried to create and account, he gets the error message "There was either an authentication database error or you are not allowed to update your external account." [18:41:32] Kelson: did he read the note at the bottom? [18:43:14] MatmaRex: In the meantim yes... I has forgotten to put a shell nickname.... [18:43:15] sorry [18:43:25] :) [18:47:19] MatmaRex: Do you know who is responsible to give git write access to a repository? [18:47:59] Kelson: probably ^demon or qchris_away, they're the go-to gerrit guys [18:48:24] MatmaRex: ok, perfect, then will try to catch one of them tomorrow :) [18:53:20] PROBLEM - Puppet freshness on wtp1015 is CRITICAL: No successful Puppet run in the last 10 hours [19:00:20] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [19:07:20] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: No successful Puppet run in the last 10 hours [19:10:10] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [19:18:49] RECOVERY - Disk space on ms-be1002 is OK: DISK OK [19:18:49] RECOVERY - Disk space on ms-be1001 is OK: DISK OK [19:56:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [19:59:07] Kelson_away: Hi. That depends a bit on what repository you want to have write access. [19:59:42] Kelson_away: In general the project owners sholud be able to give you access. [20:09:07] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [20:22:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:23:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [20:29:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:31:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 6.631 second response time [21:06:35] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [21:30:55] Reedy: are you free? [21:34:24] YuviPanda: are you able to import on beta? [21:34:55] betalabs? [21:34:58] yes [21:34:59] no I don't have persm [21:35:00] *perms [21:35:15] whom can have perms, or give me perms to do an import? [21:35:41] want to throw ower [[en:Barack Obama]] + all templates there [21:40:03] PROBLEM - Puppet freshness on mc1005 is CRITICAL: No successful Puppet run in the last 10 hours [21:40:03] PROBLEM - Puppet freshness on mw41 is CRITICAL: No successful Puppet run in the last 10 hours [21:40:09] YuviPanda: is Cmcmahon lurking on irc? [21:40:23] nope [21:42:06] PROBLEM - Puppet freshness on db1036 is CRITICAL: No successful Puppet run in the last 10 hours [21:42:06] PROBLEM - Puppet freshness on amssq44 is CRITICAL: No successful Puppet run in the last 10 hours [21:42:07] PROBLEM - Puppet freshness on ms-be3 is CRITICAL: No successful Puppet run in the last 10 hours [21:43:06] PROBLEM - Puppet freshness on mw58 is CRITICAL: No successful Puppet run in the last 10 hours [21:43:07] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [21:43:07] PROBLEM - Puppet freshness on sq42 is CRITICAL: No successful Puppet run in the last 10 hours [21:44:28] AzaToth: I dunno. Do I? [21:46:06] Reedy: I do want http://en.wikipedia.org/wiki/Special:Export of "Barack Obama" + templates http://en.wikipedia.beta.wmflabs.org/wiki/Special:Import [21:46:11] Yeah [21:46:19] I've no idea if I've got permissions [21:46:45] ReedyTalkMy settingsWatchlistMy changesLog out [21:46:48] My settings? [21:47:27] mwdeploy@deployment-bastion:/data/project/apache/common-local/php-master$ mwscript maintenance/createAndPromote.php --wiki=enwiki --bureaucrat Reedy --force [21:47:27] enwiki: Promoting User:Reedy into bureaucrat... [21:47:27] done. [21:47:29] Wheee [21:49:14] I've given you sysop [21:49:19] ty [21:49:54] ah, importupload isn't enabled ヾ [21:50:22] lol, same as enwiki [21:50:38] any reasons it's disabled? [21:51:24] https://meta.wikimedia.org/wiki/Importer ? [21:51:26] same config as enwiki [21:51:50] Nemo_bis: afaik, vanilla mw has import upload available [21:51:52] importer [21:52:14] Given you importer too [21:52:41] haha [21:52:43] NOTE: [21:52:43] Import has been disabled on the English Wikipedia. Due to their global sysop ability, Stewards can see this page and perform Imports. However, if you are a steward the preferred way to bring a page from another wiki to here is to do a cut and paste of the content, and link to the page on the other wiki in the edit summary. [21:52:43] Please do not attempt to import pages to the English Wikipedia. [21:52:56] now I understand ツ [21:53:36] IMPORT ALL THE PAGES [21:54:12] does it seriously say "cut and paste"? [21:54:35] yes [21:54:41] hell yes [21:54:55] http://i.imgur.com/HB8OlFQ.png [21:55:35] wtf? do they copyvio? [21:55:47] http://en.wikipedia.beta.wmflabs.org/wiki/Special:RecentChanges ツ [21:55:58] se4598: dun ask me [21:57:38] se4598: linking the other page in the summary is usually considered enough in terms of attribution [21:57:38] Nemo_bis: ^ [21:57:40] and importing causes all sorts of issues [21:57:59] (which will probably become irrelevant once SUL unification happens, though) [21:58:06] MatmaRex: I see [21:58:17] damn, it takes time [21:58:29] the xml was only around 800KB [21:58:36] it's still importing [21:58:39] i think de.wp is the only large one doing imports [21:58:46] okai [21:59:29] http://en.wikipedia.org/wiki/Special:Log/import [21:59:38] 2013-06-29T08:55:00 Graham87 (talk | contribs | block) imported AppleTalk by file upload (1 revision: import old edit from the Nostalgia Wikipedia, see User:Graham87/Import) [22:00:14] graham is the only person who cares about old lost revisions and so he's allowed to restore them :P [22:00:23] hehe [22:05:26] Reedy: thanks [22:07:08] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [22:10:09] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 10 hours [22:10:09] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: No successful Puppet run in the last 10 hours [22:10:09] PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 10 hours [22:10:09] PROBLEM - Puppet freshness on amssq31 is CRITICAL: No successful Puppet run in the last 10 hours [22:10:09] PROBLEM - Puppet freshness on amssq32 is CRITICAL: No successful Puppet run in the last 10 hours [22:13:08] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [22:13:08] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [22:13:08] PROBLEM - Puppet freshness on db1031 is CRITICAL: No successful Puppet run in the last 10 hours [22:13:08] PROBLEM - Puppet freshness on cp3012 is CRITICAL: No successful Puppet run in the last 10 hours [22:13:08] PROBLEM - Puppet freshness on db1001 is CRITICAL: No successful Puppet run in the last 10 hours [22:14:08] PROBLEM - Puppet freshness on cp1002 is CRITICAL: No successful Puppet run in the last 10 hours [22:14:08] PROBLEM - Puppet freshness on cp1005 is CRITICAL: No successful Puppet run in the last 10 hours [22:14:08] PROBLEM - Puppet freshness on ms10 is CRITICAL: No successful Puppet run in the last 10 hours [22:14:08] PROBLEM - Puppet freshness on helium is CRITICAL: No successful Puppet run in the last 10 hours [22:14:08] PROBLEM - Puppet freshness on db39 is CRITICAL: No successful Puppet run in the last 10 hours [22:14:09] PROBLEM - Puppet freshness on mc1001 is CRITICAL: No successful Puppet run in the last 10 hours [22:14:09] PROBLEM - Puppet freshness on mw1063 is CRITICAL: No successful Puppet run in the last 10 hours [22:14:10] PROBLEM - Puppet freshness on cp1049 is CRITICAL: No successful Puppet run in the last 10 hours [22:14:10] PROBLEM - Puppet freshness on mw1043 is CRITICAL: No successful Puppet run in the last 10 hours [22:14:11] PROBLEM - Puppet freshness on mw20 is CRITICAL: No successful Puppet run in the last 10 hours [22:14:11] PROBLEM - Puppet freshness on mw1106 is CRITICAL: No successful Puppet run in the last 10 hours [22:14:12] PROBLEM - Puppet freshness on ekrem is CRITICAL: No successful Puppet run in the last 10 hours [22:14:12] PROBLEM - Puppet freshness on mw43 is CRITICAL: No successful Puppet run in the last 10 hours [22:14:13] PROBLEM - Puppet freshness on mw57 is CRITICAL: No successful Puppet run in the last 10 hours [22:14:13] PROBLEM - Puppet freshness on titanium is CRITICAL: No successful Puppet run in the last 10 hours [22:14:14] PROBLEM - Puppet freshness on stat1 is CRITICAL: No successful Puppet run in the last 10 hours [22:14:14] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: No successful Puppet run in the last 10 hours [22:14:15] PROBLEM - Puppet freshness on srv255 is CRITICAL: No successful Puppet run in the last 10 hours [22:14:15] PROBLEM - Puppet freshness on mw1087 is CRITICAL: No successful Puppet run in the last 10 hours [22:14:16] PROBLEM - Puppet freshness on mw124 is CRITICAL: No successful Puppet run in the last 10 hours [22:59:46] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [23:08:04] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [23:14:04] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [23:14:04] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [23:14:04] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [23:14:04] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [23:14:04] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [23:14:05] PROBLEM - Puppet freshness on sodium is CRITICAL: No successful Puppet run in the last 10 hours [23:14:05] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [23:14:06] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [23:14:06] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [23:27:25] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server