[01:54:46] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [01:54:46] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [02:11:45] !log reedy synchronized wmf-config/InitialiseSettings.php 'touch' [02:11:49] Logged the message, Master [02:21:56] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1610s [02:25:26] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1820s [02:25:51] !log LocalisationUpdate completed (1.18) at Sun Feb 5 02:25:51 UTC 2012 [02:25:53] Logged the message, Master [02:44:56] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 17s [03:00:16] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 5s [03:14:16] PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=64%): /var/lib/ureadahead/debugfs 0 MB (0% inode=64%): [03:25:56] RECOVERY - Disk space on srv221 is OK: DISK OK [04:01:40] zzz =_= [05:52:01] any techs here and awake? which is more server and resource efficient? to spam blacklist is it better to add domain names or regex, or in the end it doesn't really matter? [05:52:20] multiple domain names (that would be) [07:07:53] RECOVERY - mysqld processes on db35 is OK: PROCS OK: 1 process with command name mysqld [07:21:23] PROBLEM - MySQL Slave Delay on db35 is CRITICAL: CRIT replication delay 94001 seconds [07:27:53] PROBLEM - LVS Lucene on search-pool2.svc.pmtpa.wmnet is CRITICAL: Connection timed out [07:58:59] RECOVERY - LVS Lucene on search-pool2.svc.pmtpa.wmnet is OK: TCP OK - 0.001 second response time on port 8123 [08:17:39] PROBLEM - Lucene on search6 is CRITICAL: Connection timed out [08:35:39] PROBLEM - LVS Lucene on search-pool2.svc.pmtpa.wmnet is CRITICAL: Connection timed out [09:09:49] RECOVERY - MySQL Slave Delay on db35 is OK: OK replication delay 0 seconds [12:06:19] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [12:06:19] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [12:48:02] PROBLEM - LVS Lucene on search-pool2.svc.pmtpa.wmnet is CRITICAL: Connection timed out [12:51:58] !log restarted lsearchd on search6 [12:52:00] Logged the message, Master [12:55:22] RECOVERY - Lucene on search6 is OK: TCP OK - 0.002 second response time on port 8123 [13:13:02] RECOVERY - LVS Lucene on search-pool2.svc.pmtpa.wmnet is OK: TCP OK - 0.002 second response time on port 8123 [13:49:56] PROBLEM - LVS Lucene on search-pool2.svc.pmtpa.wmnet is CRITICAL: Connection timed out [14:01:46] RECOVERY - LVS Lucene on search-pool2.svc.pmtpa.wmnet is OK: TCP OK - 0.003 second response time on port 8123 [14:07:38] New review: Dzahn; "did not look at the details yet, but +1 for sudoers.d. " [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/2283 [14:14:08] Reedy, ping [14:18:50] ? [14:19:47] Reedy, I'd need you to update the interwiki cache [14:19:54] Reedy, https://meta.wikimedia.org/w/index.php?title=Interwiki_map&action=historysubmit&diff=3373446&oldid=3352951 [14:20:18] need it as a workaround to a bug which is blocking some translations on wikimania wiki [14:23:46] !log reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache' [14:23:48] Logged the message, Master [14:24:51] Reedy, ok, works thank you very much! [15:18:07] Hey, when did AntiSpamMeta's vhost (the cloak) change? [15:18:16] I coulda sworn it was different before [15:22:02] We don't run it [15:22:06] ask in #wikimedia-ops [15:22:22] I doubt a op did that [15:22:27] still [15:22:29] we don't run it [15:22:37] i've no idea who actually run is [15:23:10] or what the hell it actually does [15:30:42] LL2|JedIRC, November 16 [15:30:49] not that this says anything [15:31:00] Reedy: AfterDeath runs it, and it has nothing to do with wiki*dia :) (detects floods, op requests etc in some channels on freenode) [15:39:14] PROBLEM - LVS Lucene on search-pool2.svc.pmtpa.wmnet is CRITICAL: Connection timed out [15:56:07] RECOVERY - LVS Lucene on search-pool2.svc.pmtpa.wmnet is OK: TCP OK - 0.003 second response time on port 8123 [16:34:07] PROBLEM - LVS Lucene on search-pool2.svc.pmtpa.wmnet is CRITICAL: Connection timed out [16:34:51] !log asher synchronized wmf-config/db.php 'returning db35 to service' [16:34:53] Logged the message, Master [17:03:23] RECOVERY - LVS Lucene on search-pool2.svc.pmtpa.wmnet is OK: TCP OK - 0.002 second response time on port 8123 [17:08:34] PROBLEM - MySQL Slave Delay on db1017 is CRITICAL: CRIT replication delay 1856 seconds [17:21:48] problems with interwiki [17:22:12] de -wiki [17:22:39] nl-wiki seems fine [17:23:08] http://de.wikipedia.org/wiki/Wikipedia:WikiProjekt#footer [17:24:16] seems cache? [17:25:38] What problem? [17:26:58] Romaine, looks fine to me [17:27:32] those red links were back at the bottom of the pache [17:27:38] after nulledit they were gone again [17:27:44] page [17:27:46] Which redlinks? [17:27:48] Hmm [17:27:52] probably don't worry about it too much then [17:27:54] interwiki's [17:28:05] and not shown in the sidebar [17:28:24] if null edit fixed it, it'd not worth investigating [18:29:51] !log reedy synchronized php-1.18/includes/api [18:29:53] Logged the message, Master [19:43:19] zzz =_= [19:59:08] PROBLEM - Host db1001 is DOWN: PING CRITICAL - Packet loss = 100% [20:00:27] RECOVERY - MySQL Slave Delay on db1017 is OK: OK replication delay NULL seconds [20:51:45] PROBLEM - MySQL Replication Heartbeat on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [21:13:36] do the boxes that work through the job-queue also have memcached? [21:18:20] tr|nn|, I think we try and keep it to one or the other [21:18:27] doesn't make any difference as it's remotely accessible anyway [21:18:32] so they all use the same memcached pool [21:20:43] Reedy, that absolutely makes sense. thx :) [22:16:57] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [22:16:57] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [23:44:50] is it possible to disable Babel Auto Creation of categories= [23:49:30] yes [23:50:08] to disable for pt.wiktioanry, a bugzilla request is necessary? [23:51:37] PROBLEM - MySQL Replication Heartbeat on db42 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:54:57] done [23:57:09] gn8 folks