[00:15:18] (03CR) 10Jforrester: [C: 04-1] "Please don't merge until questions on the bug have been answered." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/109639 (owner: 10TTO) [00:45:02] (03PS1) 10Ori.livneh: gdash: add weekly graphs for frontend & ve dashboards [operations/puppet] - 10https://gerrit.wikimedia.org/r/110664 [00:45:06] (03CR) 10jenkins-bot: [V: 04-1] gdash: add weekly graphs for frontend & ve dashboards [operations/puppet] - 10https://gerrit.wikimedia.org/r/110664 (owner: 10Ori.livneh) [00:45:38] (03CR) 10Ori.livneh: [C: 032] Don't allow less than 1 thread in mw-update-l10n [operations/puppet] - 10https://gerrit.wikimedia.org/r/110609 (owner: 10Hoo man) [00:46:00] (03PS2) 10Ori.livneh: gdash: add weekly graphs for frontend & ve dashboards [operations/puppet] - 10https://gerrit.wikimedia.org/r/110664 [00:46:20] (03CR) 10Ori.livneh: [C: 032 V: 032] gdash: add weekly graphs for frontend & ve dashboards [operations/puppet] - 10https://gerrit.wikimedia.org/r/110664 (owner: 10Ori.livneh) [00:49:45] James_F|Away: http://gdash.wikimedia.org/dashboards/ve/ now with weekly views [01:11:07] recent changes size on the beta cluster is broken. clicking to show 100/250/500 changes doesn't do anything; it just stays on 50 [01:24:57] PROBLEM - SSH on sodium is CRITICAL: Server answer: [01:25:57] RECOVERY - SSH on sodium is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [01:25:57] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [01:26:57] PROBLEM - SSH on lvs1002 is CRITICAL: Server answer: [01:29:57] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [01:31:57] RECOVERY - SSH on lvs1002 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [01:39:57] PROBLEM - SSH on lvs1002 is CRITICAL: Server answer: [01:40:57] RECOVERY - SSH on lvs1002 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [01:45:55] is this totally obsolete? spence is gone and I don't see snmpd in use: https://wikitech.wikimedia.org/wiki/SNMP [02:02:04] !log LocalisationUpdate completed (1.23wmf11) at 2014-02-01 02:02:04+00:00 [02:02:13] Logged the message, Master [02:17:03] !log LocalisationUpdate completed (1.23wmf12) at 2014-02-01 02:17:03+00:00 [02:17:16] Logged the message, Master [02:25:35] !log LocalisationUpdate ResourceLoader cache refresh completed at 2014-02-01 02:25:34+00:00 [02:25:42] Logged the message, Master [02:44:00] jgage: yes [03:23:37] PROBLEM - MySQL Processlist on db1040 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 339 statistics [03:25:37] RECOVERY - MySQL Processlist on db1040 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 11 statistics [03:51:03] thanks ori, i'll move it to Obsolete: [04:33:07] (03PS1) 10Hoo man: Remove deprecated wgCopyrightIcon in favor of wgFooterIcons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110668 [05:16:07] PROBLEM - SSH on calcium is CRITICAL: Server answer: [05:16:17] PROBLEM - SSH on virt1000 is CRITICAL: Server answer: [05:17:07] RECOVERY - SSH on calcium is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [05:17:17] PROBLEM - SSH on ssl1006 is CRITICAL: Server answer: [05:17:57] PROBLEM - SSH on lvs1002 is CRITICAL: Server answer: [05:17:57] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [05:18:17] RECOVERY - SSH on virt1000 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [05:18:17] RECOVERY - SSH on ssl1006 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [05:19:57] RECOVERY - SSH on lvs1002 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [05:23:57] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [06:54:49] (03PS4) 10Yurik: Handle the change with netmapper and varnish 3.0.4 & later [operations/puppet] - 10https://gerrit.wikimedia.org/r/102887 [06:56:05] (03CR) 10BBlack: [C: 032 V: 032] "Either way we'd need this to smooth the transition, so it goes in for now." [operations/puppet] - 10https://gerrit.wikimedia.org/r/102887 (owner: 10Yurik) [07:46:40] (03PS1) 10ArielGlenn: getMWVersion.php prints its result, don't print it here too [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110672 [07:56:02] (03CR) 10ArielGlenn: "Need this or equivalent to fix r109061 which left getMWVersion printing out the version string twice for a given run. This broke dumps, se" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110672 (owner: 10ArielGlenn) [11:26:23] !log restarting gdnsd on ns0/1/2 to reload maxmind geoip databases (FOSDEM geolocated esams, among others ;)) [11:26:32] Logged the message, Master [12:19:20] (03PS1) 10Springle: db1052 increase LB [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110682 [12:19:51] (03CR) 10Springle: [C: 032] db1052 increase LB [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110682 (owner: 10Springle) [12:19:57] (03Merged) 10jenkins-bot: db1052 increase LB [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110682 (owner: 10Springle) [12:21:23] !log springle synchronized wmf-config/db-eqiad.php 'db1052 increase LB' [12:21:32] Logged the message, Master [13:13:45] happy fosdem [13:26:17] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [14:24:17] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [15:35:41] (03CR) 10Alexandros Kosiaris: [C: 04-1] Split exim stats to own class and add it to mchenry (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/110524 (owner: 10Nemo bis) [16:23:05] (03CR) 10Alexandros Kosiaris: [C: 032] Remove both temp dirs in scap-recompile [operations/puppet] - 10https://gerrit.wikimedia.org/r/110426 (owner: 10Reedy) [18:57:53] (03CR) 10Hoo man: "includes/Skin.php might still rely on this, needs to be re-checked" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110668 (owner: 10Hoo man) [20:16:20] going to update extensions/FeaturedFeeds on both branches; see https://bugzilla.wikimedia.org/show_bug.cgi?id=30148#c8 for context [20:18:56] !log ori synchronized php-1.23wmf12/extensions/FeaturedFeeds 'I8a600d37a: Reject FeedItem timestamps set too far in the future' [20:19:04] Logged the message, Master [20:19:28] !log sync-dir: mw1054: No route to host; mw1073: Connection timed out [20:19:36] Logged the message, Master [20:20:18] !log ori synchronized php-1.23wmf11/extensions/FeaturedFeeds 'I8a600d37a: Reject FeedItem timestamps set too far in the future' [20:20:26] Logged the message, Master [20:33:40] ori, was that exception so widespread? [20:34:28] MaxSem: no [20:34:57] but exceptions should be exceptional [20:55:55] tell that to invalid message parameter:P [20:57:52] btw ori, you might be interested in https://gerrit.wikimedia.org/r/38252 :) [22:04:16] (03PS1) 10Ebe123: Add el.wikivoyage to it.wikivoyage transwiki import [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110822 [22:07:31] (03CR) 10Odder: [C: 031] Add el.wikivoyage to it.wikivoyage transwiki import [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110822 (owner: 10Ebe123) [22:41:02] (03PS5) 10Ori.livneh: A short script for viewing exceptions [operations/puppet] - 10https://gerrit.wikimedia.org/r/38252 (owner: 10MaxSem) [22:41:12] (03CR) 10Ori.livneh: [C: 032 V: 032] A short script for viewing exceptions [operations/puppet] - 10https://gerrit.wikimedia.org/r/38252 (owner: 10MaxSem) [23:00:23] Got a person in -tech complaining of 403s from his host on download.wikimedia.org and dumps.wikimedia.org [23:00:31] 2014-02-01 - 14:56:11 my IP: 199.101.103.89 [23:00:41] 2014-02-01 - 14:57:16 used to be tor exit node on 80 and 443 [23:00:44] 2014-02-01 - 14:57:22 now middle relay [23:00:46] 2014-02-01 - 14:57:31 http://torstatus.blutmagie.de/router_detail.php?FP=B86780CB489EF7B6E996A9F5C42CA609BC6A2C07 [23:00:57] Do we do special things to tor relays? [23:02:35] Only exit nodes [23:02:54] but we use an external thingy to get a list of exit nodes and it's only updated by cron every X [23:02:59] (that's in puppet) [23:05:43] Hm. [23:20:14] marktraceur: no more than two simultaneous downloads from that host via same ip, we cap [23:20:35] we do nothing special for tor one way or another [23:21:25] (not really here, just happened to look in)