[00:10:13] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [00:50:16] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [01:05:16] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [01:20:16] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [01:35:16] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [01:41:33] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 228 seconds [01:42:09] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 209 seconds [01:49:03] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 628s [01:54:36] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 15 seconds [01:57:46] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 25s [01:57:54] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 0 seconds [03:04:13] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [03:04:13] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [04:10:13] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [05:39:46] PROBLEM - Host srv266 is DOWN: PING CRITICAL - Packet loss = 100% [07:21:58] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [07:32:01] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [07:35:55] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [07:35:55] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [07:35:55] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [08:12:57] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [09:47:54] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [10:11:01] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [10:50:55] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [11:05:54] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [11:20:54] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [11:35:55] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [13:05:01] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [13:05:01] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [13:11:53] I've found another obfuscated JS loading 3rd party content on a WMF wiki [13:12:02] I've disabled it, for nw [13:12:08] * for now [13:22:14] Where was it? [13:23:59] Krenair: piwiki [13:27:02] wow, it was there for a year: https://pi.wikipedia.org/w/index.php?title=%E0%A4%AE%E0%A5%80%E0%A4%A1%E0%A4%BF%E0%A4%AF%E0%A4%BE%E0%A4%B5%E0%A4%BF%E0%A4%95%E0%A4%BF:Vector.js&action=history [14:04:27] I'm not exactly suprised [14:04:52] Reedy: Why? [14:05:12] It happens all over [14:05:16] even more so on the smaller wikis [14:05:20] Reedy: Yes [14:06:26] I already told Chris Steipp how that situation could be improved dramatically [14:06:38] But I'm not sure, whether he got time to work on that project [14:09:11] How could it be improved dramatically? [14:09:34] Krenair: We could automatically scan the wikis for 3rd party inclusions [14:09:40] using real browsers [14:11:00] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [14:11:51] but I neither got the time nor do I want to spent the hardware for that, so Chris wanted to take care [17:23:00] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [17:32:55] PROBLEM - Puppet freshness on labstore1 is CRITICAL: Puppet has not run in the last 10 hours [17:36:58] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [17:36:58] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [17:36:58] PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: Puppet has not run in the last 10 hours [17:57:02] Reedy: There? [17:57:08] yes [17:57:46] About the piwiki JS stuff... as you know, I've already removed it and I'll contact the user [17:58:07] Is it needed to make a village pump post or smth. like that, to let the community know? [17:59:07] No idea [17:59:12] or to tell legal? [17:59:16] When I fix stuff I tend to just do it and move on [17:59:25] this is the MS thing right? [17:59:29] jeremyb: Yes [18:00:07] hoo: how did you do it anyway? are you a steward? [18:00:13] Reedy: Well, the siwiki issues some months back lead to desysops in several projects... [18:00:18] jeremyb: Global interface editor [18:00:41] huh, didn't know such a thing existed [18:00:43] http://meta.wikimedia.org/wiki/Interface_editors [18:00:51] Added for krinkle among others ;) [18:00:59] hoo: Emailing Phillipe is probably the best thing to do [18:01:06] yeah, i see [18:01:30] Reedy: I'll do, I have to contact him anyway [18:01:44] it's fine if I let the user know on his talk page, no? [18:02:54] sure [18:03:26] Ok, going to mail Phillipe this week, then ;) [18:14:01] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [19:23:58] hoo: siwiki? [19:25:01] Nemo_bis: That thing https://si.wikipedia.org/wiki/%E0%B6%B4%E0%B6%BB%E0%B7%92%E0%B7%81%E0%B7%93%E0%B6%BD%E0%B6%9A_%E0%B7%83%E0%B7%8F%E0%B6%9A%E0%B6%A0%E0%B7%8A%E0%B6%A1%E0%B7%8F%E0%B7%80:%E0%B6%B6%E0%B7%92%E0%B6%9F%E0%B7%94%E0%B7%80%E0%B7%8F [19:25:30] ouch, [[Interface editors]] has been expanded a lot recently [19:27:36] hoo: it's not the first time this happens [19:28:01] Nemo_bis: I know, we had another case only today [19:28:06] last time, Roan added an exception to the upload rules, uploaded the fonts locally, disabled the script [19:28:27] then we only had to ask the l10n team to enable WebFonts [19:28:32] I know... told him in person, in Berlin ;) [19:29:23] "him"? [19:29:42] Roan ;) [19:30:05] hm, wrong person [19:30:57] have users been pointed to https://www.mediawiki.org/wiki/Extension:WebFonts#Adding_fonts ? [19:31:27] well, not that section specifically maybe [19:32:58] Nemo_bis: Well, webfonts are only the issue in few cases [19:35:05] hoo: are you speaking in general or of si projects? [19:35:33] in general, of all WMF projects [19:36:29] I've already talked a lot about to Chris Steipp [19:48:58] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [20:11:54] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [20:51:58] PROBLEM - Puppet freshness on ms-be1003 is CRITICAL: Puppet has not run in the last 10 hours [21:03:20] hoo|away: when you find local JS/CSS hacks for fonts, inoput methods and so on, can you please notify me so that I ensure they're informed about proper solutions? [21:04:21] (btw it's wonderful that you're going on with this cleanup) [21:06:58] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [21:11:23] Nemo_bis: Now that you mention it: https://ka.wikipedia.org/wiki/%E1%83%9B%E1%83%94%E1%83%93%E1%83%98%E1%83%90%E1%83%95%E1%83%98%E1%83%99%E1%83%98:ConvertInput.js I wrote that a year ago [21:14:48] hoo|away: Georgian is not supported yet, I suppose you should file a bug against Narayam to get the script added to the extension [21:15:24] Nemo_bis: Not today... I'm awake for about 35 hrs already :P [21:15:29] Good night ;) [21:15:35] good night :) [21:15:44] gute nacht! [21:15:55] I'll file the bug and let you add more info later [21:16:35] ok, cc me or send it as memo to me [21:21:58] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [21:36:57] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [22:28:52] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [22:29:56] hmmmmmm [22:30:22] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [22:30:37] ok then... [23:05:54] PROBLEM - Puppet freshness on ms-be1009 is CRITICAL: Puppet has not run in the last 10 hours [23:05:54] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [23:24:03] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [23:28:33] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [23:36:21] PROBLEM - SSH on nescio is CRITICAL: Server answer: [23:42:04] RECOVERY - SSH on nescio is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [23:46:25] PROBLEM - SSH on nescio is CRITICAL: Server answer: [23:47:19] PROBLEM - SSH on ssl3001 is CRITICAL: Server answer: [23:53:46] RECOVERY - SSH on nescio is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [23:59:28] PROBLEM - SSH on nescio is CRITICAL: Server answer: