[00:09:43] (03PS1) 10Yuvipanda: eventlogging: Make ganglia usage optional [puppet] - 10https://gerrit.wikimedia.org/r/173634 (https://bugzilla.wikimedia.org/73479) [00:13:27] (03PS2) 10Yuvipanda: eventlogging: Make ganglia usage optional [puppet] - 10https://gerrit.wikimedia.org/r/173634 (https://bugzilla.wikimedia.org/73479) [00:17:09] (03PS3) 10Yuvipanda: eventlogging: Make ganglia usage optional [puppet] - 10https://gerrit.wikimedia.org/r/173634 (https://bugzilla.wikimedia.org/73479) [00:21:46] (03PS4) 10Yuvipanda: eventlogging: Make ganglia usage optional [puppet] - 10https://gerrit.wikimedia.org/r/173634 (https://bugzilla.wikimedia.org/73479) [00:35:43] (03PS5) 10Yuvipanda: eventlogging: Make ganglia usage optional [puppet] - 10https://gerrit.wikimedia.org/r/173634 (https://bugzilla.wikimedia.org/73479) [00:37:01] jesus christ, eventlogging [00:46:49] YuviPanda: ? [00:52:18] PROBLEM - MySQL Slave Delay on db1016 is CRITICAL: CRIT replication delay 353 seconds [00:52:20] PROBLEM - MySQL Replication Heartbeat on db1016 is CRITICAL: CRIT replication delay 355 seconds [00:53:36] RECOVERY - MySQL Slave Delay on db1016 is OK: OK replication delay 0 seconds [00:53:39] RECOVERY - MySQL Replication Heartbeat on db1016 is OK: OK replication delay -0 seconds [02:16:33] !log LocalisationUpdate completed (1.25wmf7) at 2014-11-16 02:16:32+00:00 [02:16:42] Logged the message, Master [02:22:50] !log LocalisationUpdate completed (1.25wmf8) at 2014-11-16 02:22:49+00:00 [02:22:53] Logged the message, Master [03:03:48] PROBLEM - puppet last run on uranium is CRITICAL: CRITICAL: Puppet has 1 failures [03:21:08] RECOVERY - puppet last run on uranium is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [03:33:17] PROBLEM - puppet last run on mw1074 is CRITICAL: CRITICAL: Puppet has 2 failures [03:33:57] PROBLEM - puppet last run on mw1198 is CRITICAL: CRITICAL: Puppet has 1 failures [03:34:07] PROBLEM - puppet last run on cp1038 is CRITICAL: CRITICAL: Puppet has 1 failures [03:39:17] PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: Puppet has 1 failures [03:41:47] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Puppet has 1 failures [03:44:38] PROBLEM - puppet last run on mw1130 is CRITICAL: CRITICAL: Puppet has 1 failures [03:51:17] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:51:28] RECOVERY - puppet last run on cp1038 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [03:51:49] RECOVERY - puppet last run on mw1074 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [03:52:08] PROBLEM - puppet last run on mw1202 is CRITICAL: CRITICAL: Puppet has 1 failures [03:54:27] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:58:08] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [04:02:12] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [04:09:18] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Nov 16 04:09:17 UTC 2014 (duration 9m 16s) [04:09:20] Logged the message, Master [04:09:28] RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:30:08] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:27] PROBLEM - puppet last run on labcontrol2001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:28] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 2 failures [06:40:48] PROBLEM - puppet last run on es1003 is CRITICAL: CRITICAL: Puppet has 2 failures [06:41:48] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:58] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:47:17] RECOVERY - puppet last run on labcontrol2001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:47:17] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:49:18] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: Puppet has 2 failures [06:50:08] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:17] RECOVERY - puppet last run on es1003 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [07:00:09] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [07:06:38] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:07:27] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:11:23] (03CR) 1020after4: [C: 031] memcached: Make ganglia inclusion optional [puppet] - 10https://gerrit.wikimedia.org/r/173510 (owner: 10Yuvipanda) [09:43:01] (03PS1) 10Ebrahim: Fix most annoying SVG bug, SVG path data number parsing issue [debs/librsvg] - 10https://gerrit.wikimedia.org/r/173639 [10:10:20] PROBLEM - check_puppetrun on db1008 is CRITICAL: CRITICAL: Puppet has 2 failures [10:15:17] RECOVERY - check_puppetrun on db1008 is OK: OK: Puppet is currently enabled, last run 178 seconds ago with 0 failures [10:58:20] (03CR) 10Reedy: [C: 04-1] "Just filed https://rt.wikimedia.org/Ticket/Display.html?id=8879 requesting review, build and deploy of this." [debs/librsvg] - 10https://gerrit.wikimedia.org/r/173639 (owner: 10Ebrahim) [11:22:03] (03PS2) 10Ebrahim: Fix most annoying SVG bug, SVG path data number parsing issue [debs/librsvg] - 10https://gerrit.wikimedia.org/r/173639 [11:23:02] (03CR) 10Ebrahim: "Lot of issues https://commons.wikimedia.org/wiki/Category:Pictures_showing_a_librsvg_bug belong can be fixed by this patch." [debs/librsvg] - 10https://gerrit.wikimedia.org/r/173639 (owner: 10Ebrahim) [13:45:01] (03PS1) 10Krinkle: Declare 'error' log group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/173649 [13:49:53] (03CR) 10Reedy: [C: 032] Declare 'error' log group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/173649 (owner: 10Krinkle) [13:50:00] (03Merged) 10jenkins-bot: Declare 'error' log group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/173649 (owner: 10Krinkle) [13:53:47] !log krinkle Synchronized wmf-config/InitialiseSettings.php: If9194b73c3256e0064ff (duration: 00m 07s) [13:53:53] Logged the message, Master [14:10:51] !log krinkle Synchronized php-1.25wmf7/includes: I8764cf5df87b226 (duration: 00m 10s) [14:10:56] Logged the message, Master [14:12:32] (03PS1) 10Reedy: MWExceptionHandler::handle -> MWExceptionHandler::handleException [mediawiki-config] - 10https://gerrit.wikimedia.org/r/173653 [14:13:03] !log reedy Synchronized rpc/RunJobs.php: (no message) (duration: 00m 16s) [14:13:03] (03CR) 10Reedy: [C: 032] MWExceptionHandler::handle -> MWExceptionHandler::handleException [mediawiki-config] - 10https://gerrit.wikimedia.org/r/173653 (owner: 10Reedy) [14:13:05] Logged the message, Master [14:13:10] (03Merged) 10jenkins-bot: MWExceptionHandler::handle -> MWExceptionHandler::handleException [mediawiki-config] - 10https://gerrit.wikimedia.org/r/173653 (owner: 10Reedy) [14:15:02] !log krinkle Synchronized php-1.25wmf8/includes: I8764cf5df87b (duration: 00m 10s) [14:15:04] Logged the message, Master [14:20:03] (03PS1) 10Reedy: Fix Undefined index: HTTPS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/173654 [14:20:38] !log reedy Synchronized wmf-config/CommonSettings.php: Fix Undefined index: HTTPS (duration: 00m 17s) [14:20:41] Logged the message, Master [14:20:48] PROBLEM - Apache HTTP on mw1218 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:46] (03CR) 10Reedy: [C: 032] Fix Undefined index: HTTPS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/173654 (owner: 10Reedy) [14:21:53] (03Merged) 10jenkins-bot: Fix Undefined index: HTTPS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/173654 (owner: 10Reedy) [14:23:22] !log reedy Synchronized wmf-config/InitialiseSettings.php: Disable error Loggroup (duration: 00m 18s) [14:23:24] Logged the message, Master [14:23:49] PROBLEM - Apache HTTP on mw1194 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:24:25] (03PS1) 10Reedy: Disable 'error' ErrorLogGroup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/173655 [14:24:38] PROBLEM - Apache HTTP on mw1187 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:47] (03CR) 10Reedy: [C: 032] Disable 'error' ErrorLogGroup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/173655 (owner: 10Reedy) [14:25:54] (03Merged) 10jenkins-bot: Disable 'error' ErrorLogGroup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/173655 (owner: 10Reedy) [15:31:50] (03PS1) 10Krinkle: logstash: Update regex in 'exception' grokker [puppet] - 10https://gerrit.wikimedia.org/r/173658 [15:31:55] bearND: ^^ [15:33:23] oops [15:33:25] bd808|BUFFER: ^^ [16:04:17] (03PS3) 1020after4: set up redirects for bugzilla urls to redirect to phabricator. [puppet] - 10https://gerrit.wikimedia.org/r/173483 [16:05:13] (03CR) 1020after4: [C: 031] "This should be ready to go but I'd appreciate if someone sanity checked my code briefly before merging." [puppet] - 10https://gerrit.wikimedia.org/r/173483 (owner: 1020after4) [16:35:02] o_O https://arbcom.en.wikipedia.org/wiki/Main_Page [16:35:04] Your connection is not private [16:35:05] Attackers might be trying to steal your information from arbcom.en.wikipedia.org (for example, passwords, messages, or credit cards). [16:35:07] (on chrome) [16:35:59] hmm. error on FF as well [16:37:41] Glaisher: SSL cert does not cover that iirc [16:37:55] seems so, yeah [16:38:25] https://arbcom-en.wikipedia.org/wiki/Main_Page is the correct URL [16:39:41] Huh, the non-registered users page is... odd. There's two login links XD [16:40:03] Just you? [16:40:27] Reedy: It redirects properly for me. [16:40:39] Do you have HTTPS Everywhere? [16:41:55] Reedy: I disabled it, I don't know if it caches it though, let me run a few tests. [16:42:16] I explicitly added redirects for this bad ssl cert sites to HTTPSE [16:43:28] Reedy: Huh, Sorry, I guess I'm a bad test. It appears to have cached the redirect... [16:44:23] Yeah [16:44:44] I'm sorry :/ [17:18:37] (03CR) 10BryanDavis: [C: 031] logstash: Update regex in 'exception' grokker [puppet] - 10https://gerrit.wikimedia.org/r/173658 (owner: 10Krinkle) [17:39:52] (03CR) 10Aklapper: "Disclaimer: I'm not a programmer so some of my comments about the URL matches here might be incompetent and hence ignorable." [puppet] - 10https://gerrit.wikimedia.org/r/173483 (owner: 1020after4) [18:03:34] (03CR) 10Ori.livneh: [C: 04-1] set up redirects for bugzilla urls to redirect to phabricator. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/173483 (owner: 1020after4) [18:47:22] (03PS1) 10Ori.livneh: Minor spring cleaning [debs/pybal] - 10https://gerrit.wikimedia.org/r/173666 [18:48:16] (03CR) 10Ori.livneh: [C: 032] "Trivial, no code changes" [debs/pybal] - 10https://gerrit.wikimedia.org/r/173666 (owner: 10Ori.livneh) [18:48:41] (03Merged) 10jenkins-bot: Minor spring cleaning [debs/pybal] - 10https://gerrit.wikimedia.org/r/173666 (owner: 10Ori.livneh) [19:43:01] (03PS1) 10Ori.livneh: make monitor constructor accept a custom reactor object [debs/pybal] - 10https://gerrit.wikimedia.org/r/173673 [20:24:48] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: puppet fail [20:43:08] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [21:15:44] a friend just got a fundraising banner which says %MINIMUM% instead of $3 [21:15:47] http://main-protagonist.eterna23.net/wiki-percent.png [21:15:51] usa, desktop [21:16:13] though for me it says $3 [21:17:55] also the formatting is different. A/B test? [21:19:09] now he says he's seeing %AVERAGE% as well [21:20:54] fwiw the browser is Konqueror [21:24:06] jgage: labs is out, if you are here already [21:25:33] sorry, what does labs is out mean? [21:25:51] jgage: Could not resolve hostname bastion.wmflabs.org [21:26:00] ack [21:26:04] lemme grab work laptop [21:26:18] sorry for ruining your weekend :/ [21:26:48] you'll have to try harder than that ;) [21:27:08] don't challange me :D [21:27:16] haha [21:28:58] hm dns looks ok to me [21:28:59] dig @labs-ns1.wikimedia.org bastion.wmflabs.org [21:29:52] jgage: try to ping [21:30:25] and new i responds [21:30:28] now [21:30:35] yeah, wfm [21:30:47] it knows you're talking about it so it decided to behave [21:30:51] what the heck [21:31:13] it is called the sysadmin syndrome [21:31:23] thanks for your help! :) [21:31:29] sure :) [21:31:33] please tell me if it recurs [21:31:39] reoccurs [21:32:00] hm they're both correct [21:32:12] I'm sure you might find intersting data in ganglia/shinken one if those [21:35:30] jgage: out of interest; what would ruin your weekend? :p [21:47:23] my sysadmin superstition does not allow me to utter such things, lest they come to pass