[00:00:04] RECOVERY - search indices - check lucene status page on search20 is OK: HTTP OK: HTTP/1.1 200 OK - 60075 bytes in 0.112 second response time [00:00:33] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 22 seconds [00:14:53] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [00:24:33] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 201 seconds [00:25:33] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 15 seconds [00:28:34] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 195 seconds [00:30:33] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 21 seconds [01:09:14] Ryan_Lane: are you in a position to allow a bigdelete? see -tech. [01:45:07] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 231 seconds [01:49:07] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 235 seconds [01:50:07] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 3 seconds [02:09:04] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 237 seconds [02:12:10] !log LocalisationUpdate completed (1.22wmf2) at Sun Apr 28 02:12:10 UTC 2013 [02:12:19] Logged the message, Master [02:14:04] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 237 seconds [02:21:04] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 27 seconds [02:39:01] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replicat [02:48:51] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 28 02:48:51 UTC 2013 [02:48:59] Logged the message, Master [06:25:09] New patchset: Catrope; "Enable VisualEditor on test2wiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61250 [06:32:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:33:23] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 192 seconds [06:33:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.556 second response time [06:33:42] New review: Catrope; "Depends on Parsoid config change, do not merge before 3pm PDT on Monday" [operations/mediawiki-config] (master) C: -2; - https://gerrit.wikimedia.org/r/61250 [06:34:12] New review: Catrope; "I mean 2pm" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61250 [06:35:23] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 12 seconds [06:52:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:53:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [08:01:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:02:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [08:43:07] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [08:43:07] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [08:56:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:58:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.150 second response time [09:33:44] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [09:52:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.146 second response time [10:15:15] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [10:22:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [10:56:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:57:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.139 second response time [11:22:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [13:08:00] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [13:08:00] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [13:08:00] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [13:13:31] !log restarting Zuul, it has a finished build stuck in the `postmerge` pipeline. [13:13:39] Logged the message, Master [13:18:00] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [14:12:55] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [14:29:15] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 215 seconds [14:30:15] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 7 seconds [14:35:53] New patchset: Andrew Bogott; "Add robots.txt and a privacy policy to mediawiki_singlenode." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61069 [14:41:25] PROBLEM - Puppet freshness on vanadium is CRITICAL: No successful Puppet run in the last 10 hours [15:40:10] PROBLEM - Host mw1041 is DOWN: PING CRITICAL - Packet loss = 100% [16:02:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:03:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [16:06:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [16:26:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:27:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 2.390 second response time [16:31:57] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:57:43] PROBLEM - DPKG on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:58:34] RECOVERY - DPKG on mc15 is OK: All packages OK [17:08:13] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 184 seconds [17:08:43] PROBLEM - Disk space on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [17:09:13] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 5 seconds [17:09:34] RECOVERY - Disk space on mc15 is OK: DISK OK [17:32:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:33:13] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 184 seconds [17:33:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [17:34:13] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 2 seconds [18:18:06] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 184 seconds [18:19:06] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 15 seconds [18:23:07] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 184 seconds [18:25:06] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 3 seconds [18:43:53] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [18:43:54] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [19:28:38] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 208 seconds [19:30:38] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 27 seconds [19:34:18] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [19:34:38] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 200 seconds [19:35:33] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 25 seconds [19:54:33] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 240 seconds [19:55:33] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 28 seconds [19:58:34] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 208 seconds [19:59:33] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 18 seconds [20:13:38] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 208 seconds [20:14:38] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 13 seconds [20:15:18] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [20:53:37] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 208 seconds [20:54:37] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 3 seconds [21:39:05] New patchset: Ori.livneh; "Set EventLogging client encoding to utf8mb4 (was: utf8)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61278 [21:57:14] New patchset: Ori.livneh; "Set EventLogging client character encoding to utf8mb4 (was: utf8)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61278 [21:57:31] Hello, I would like to ask which is the preferred ide to open a portion of the wikimedia project with? (Im trying understand how dump works in backupTextPass.inc in maintenance). Eclipse has problems recognizing .inc files. [22:00:30] sib: this is a question for #mediawiki, but it looks like you can configure eclipse to recognize .inc files as PHP: http://stackoverflow.com/a/1959637 [22:14:34] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 184 seconds [22:15:34] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 28 seconds [22:18:35] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 208 seconds [22:20:35] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 27 seconds [22:28:34] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 208 seconds [22:29:04] Change abandoned: Ori.livneh; "MySQLdb version in Precise repos does not know about utf8mb4." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61278 [22:29:35] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 16 seconds [22:34:35] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 195 seconds [22:35:50] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 21 seconds [22:39:00] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 222 seconds [22:42:00] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 16 seconds [22:45:00] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 196 seconds [22:49:00] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 228 seconds [22:54:00] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 213 seconds [23:03:00] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 24 seconds [23:08:58] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [23:08:58] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [23:08:59] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [23:08:59] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 229 seconds [23:09:58] RECOVERY - MySQL [23:14:18] erm? [23:14:29] icinga-wm: something's missing there!