[00:00:32] (03CR) 10TTO: "Note, needs a manual rebase now :( Sorry, this is kind of my fault" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104355 (owner: 10Odder) [00:01:19] LD time! I'm frist! [00:05:03] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [00:05:03] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [00:06:52] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 00:06:46 UTC 2013 [00:07:02] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 00:06:56 UTC 2013 [00:07:03] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 12:06:46 AM UTC [00:07:03] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [00:07:51] !log maxsem synchronized php-1.23wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/104682' [00:08:06] Logged the message, Master [00:08:44] (03CR) 10MaxSem: [C: 032] Updating schema ID for EchoInteraction [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103611 (owner: 10Kaldari) [00:08:55] (03Merged) 10jenkins-bot: Updating schema ID for EchoInteraction [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/103611 (owner: 10Kaldari) [00:11:52] !log maxsem synchronized wmf-config/CommonSettings.php 'https://gerrit.wikimedia.org/r/103611' [00:12:07] Logged the message, Master [00:22:48] I'm done [00:24:33] !log Running a swift eqiad->tampa sync script (no-deletes) to catch any inconsistencies that built up [00:24:50] Logged the message, Master [00:49:19] PROBLEM - Disk space on terbium is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=79%): [00:53:19] RECOVERY - Disk space on terbium is OK: DISK OK [00:54:02] (03PS1) 10Aaron Schulz: Avoid md5sum calls in MergeCdbFileUpdates [operations/puppet] - 10https://gerrit.wikimedia.org/r/104699 [00:55:41] who just cleared off the space on terbium ? [00:55:44] and what'd' you delete ? [00:55:50] (03PS2) 10Aaron Schulz: Disable MessageBlobStore::clear() via hook [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104191 [00:56:28] Reedy: that should avoid any notices [00:56:34] I guess that can be merged anytime then [01:04:24] (03CR) 10Aaron Schulz: [C: 032] Disable MessageBlobStore::clear() via hook [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104191 (owner: 10Aaron Schulz) [01:04:48] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [01:04:48] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [01:05:22] (03Merged) 10jenkins-bot: Disable MessageBlobStore::clear() via hook [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104191 (owner: 10Aaron Schulz) [01:06:31] !log aaron synchronized wmf-config/CommonSettings.php 'Disable MessageBlobStore::clear() via hook' [01:06:49] Logged the message, Master [01:06:59] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 01:06:55 UTC 2013 [01:07:18] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 01:07:10 UTC 2013 [01:07:48] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 01:07:10 AM UTC [01:07:48] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 01:06:55 AM UTC [01:08:50] (03PS2) 10Aaron Schulz: Setup and enabled redisLockManager for all file backends in use [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104317 [01:26:59] PROBLEM - RAID on terbium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:27:48] RECOVERY - RAID on terbium is OK: OK: optimal, 1 logical, 2 physical [01:33:58] PROBLEM - RAID on terbium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:34:51] RECOVERY - RAID on terbium is OK: OK: optimal, 1 logical, 2 physical [01:38:01] PROBLEM - RAID on terbium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:38:02] PROBLEM - DPKG on terbium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:39:01] PROBLEM - SSH on terbium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:39:52] PROBLEM - puppet disabled on terbium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:40:31] PROBLEM - twemproxy process on terbium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:41:02] RECOVERY - DPKG on terbium is OK: All packages OK [01:41:31] RECOVERY - twemproxy process on terbium is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [01:41:51] RECOVERY - puppet disabled on terbium is OK: OK [01:43:52] RECOVERY - SSH on terbium is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [01:45:52] RECOVERY - RAID on terbium is OK: OK: optimal, 1 logical, 2 physical [01:50:01] PROBLEM - RAID on terbium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:51:51] RECOVERY - RAID on terbium is OK: OK: optimal, 1 logical, 2 physical [01:55:01] PROBLEM - RAID on terbium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:55:52] RECOVERY - RAID on terbium is OK: OK: optimal, 1 logical, 2 physical [01:59:01] PROBLEM - RAID on terbium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:04:46] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [02:04:46] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [02:06:56] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 02:06:47 UTC 2013 [02:07:36] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 02:07:33 UTC 2013 [02:07:46] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 02:07:33 AM UTC [02:07:46] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 02:06:47 AM UTC [02:11:19] (03PS1) 10Aaron Schulz: Cleaned up RSYNC_ARGS generation in scap-2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/104704 [02:11:54] ori: ^ in my local faffing around with rsync, it seems like the include is needed first [02:14:00] !log aaron started scap: php-1.23wmf6 testing scap timing [02:17:31] !log aaron started scap: php-1.23wmf6 testing scap timing [02:20:06] PROBLEM - puppetmaster https on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:24:56] RECOVERY - RAID on terbium is OK: OK: optimal, 1 logical, 2 physical [02:25:57] PROBLEM - HTTP on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:27:56] PROBLEM - puppet disabled on terbium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:28:06] PROBLEM - SSH on terbium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:28:06] PROBLEM - RAID on terbium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:28:06] PROBLEM - DPKG on terbium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:28:46] RECOVERY - puppet disabled on terbium is OK: OK [02:28:56] RECOVERY - RAID on terbium is OK: OK: optimal, 1 logical, 2 physical [02:28:56] RECOVERY - SSH on terbium is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [02:28:56] RECOVERY - DPKG on terbium is OK: All packages OK [02:30:52] !log LocalisationUpdate completed (1.23wmf8) at Tue Dec 31 02:30:52 UTC 2013 [02:33:11] * Aaron|home cancels...probably hanging on terbium [02:39:47] looks looks Cirrus jobs eating cpu on terbium [02:54:25] !log LocalisationUpdate completed (1.23wmf7) at Tue Dec 31 02:54:25 UTC 2013 [02:59:31] ori: so that run definitely was hitting all active versions, so the MW_VERSIONS_SYNC must not be getting set [03:01:56] RECOVERY - HTTP on virt0 is OK: HTTP OK: HTTP/1.1 302 Found - 457 bytes in 0.071 second response time [03:01:57] RECOVERY - puppetmaster https on virt0 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.165 second response time [03:02:53] !log aaron started scap: php-1.23wmf6 Timing test [03:05:28] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [03:05:41] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [03:06:38] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 03:06:36 UTC 2013 [03:06:49] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 03:06:46 UTC 2013 [03:07:28] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 03:06:36 AM UTC [03:07:38] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 03:06:46 AM UTC [03:20:57] !log aaron finished scap: php-1.23wmf6 Timing test [03:21:13] Logged the message, Master [03:23:18] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Dec 31 03:23:18 UTC 2013 [03:23:35] Logged the message, Master [03:24:51] (03CR) 10Ori.livneh: [C: 032] Cleaned up RSYNC_ARGS generation in scap-2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/104704 (owner: 10Aaron Schulz) [03:26:22] TimStarling: odd, the rsync command for sync-common stalls on terbium (even just running it there directly) [03:26:28] no other server has that problem [03:29:00] you can see the command in <>, if I add -v to it I can see that it receives a few .git files for the rsync listing and then stalls [03:29:08] PROBLEM - RAID on terbium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:29:29] ori: ;) [03:29:45] it sucks that one server hangs all of scap forever though [03:29:59] RECOVERY - RAID on terbium is OK: OK: optimal, 1 logical, 2 physical [03:30:35] * Aaron|home wonders if it would be too evil to add timeout to the remote dsh commands... [03:52:00] Aaron|home: apparently terbium is slow because of copyFileBackend or forceSearchIndex or both [03:52:03] apparently it was swapping [03:53:23] copyFileBackend is writing very heavily to the local drive [03:55:04] domas was complaining about it in #mediawiki_security but apparently you are only in that channel as AaronSchulz [04:04:41] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [04:04:51] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [04:06:41] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 04:06:37 UTC 2013 [04:06:49] TimStarling: it only writes directly if there are things to copy, which there are very little...maybe from virtual memory on disk though [04:06:51] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 04:06:37 AM UTC [04:06:51] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 04:06:42 UTC 2013 [04:07:29] well, it seems pretty busy in top [04:07:41] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 04:06:42 AM UTC [04:09:42] seems very busy in iotop, which I just managed to install [04:10:58] yeah, the output just shows it comparing json file listings and finding very little to copy, so I assume the desk i/o is swap (of course there is good network i/o though) [04:11:06] *disk [04:12:09] check vmstat [04:12:30] procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- [04:12:30] r b swpd free buff cache si so bi bo in cs us sy id wa [04:12:34] 3 6 597540 469320 71652 1394104 0 0 11443 25329 31851 24294 48 4 28 20 [04:12:41] no swap, much disk activity [04:14:49] if I strace one of them at random (6565) it shows a lot of files being created and deleted [04:21:20] * Aaron|home restarted with less concurrency [04:21:40] TimStarling: any pattern to the file names? [04:21:56] there was lots of swapping before [04:22:04] but few more processes got killed by oom killer [04:22:24] Aaron|home: do those processes have to take 1.5G ? [04:22:38] http://paste.tstarling.com/p/UqzmQl.html [04:22:52] let me see if I can get the actual file names in swift [04:23:01] Aaron|home: technically, if one puts the files on shm, then it is much faster! [04:25:31] but for that one needs not to waste RAM in memory leaks [04:25:32] :) [04:29:25] oh, it's stopped now [04:33:23] I think it's the --missingonly used on the deleted files containers [04:34:54] oh, wait, no, I just misread the code [04:35:06] * Aaron|home is still confused then [04:50:16] Aaron|home: it's happening again... [04:50:33] yeah, not it actually is copying files [04:50:35] *now [04:54:12] what would have happened when it ran out of disk space earlier? would any files have been corrupted? [05:03:54] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [05:04:04] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [05:06:55] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 05:06:52 UTC 2013 [05:07:04] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 05:06:57 UTC 2013 [05:07:05] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [05:07:54] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 05:06:52 AM UTC [05:08:19] TimStarling: cloudfiles seems to ignore the result of fwrite()...not sure [05:09:04] fclose() is the most common place for disk exhaustion to be reported [05:11:25] or fflush() too I suppose [05:16:00] meh, I'll run this later...I think the mtimes are higher in eqiad slightly for lots of files due to the previous write order, so more copying happens than needed [05:16:58] !log package upgrade sanitarium hosts db1053 db1054 db1057 [05:17:15] Logged the message, Master [05:17:21] using the md5 value would avoid that...I added that in passing to a patch still in gerrit [05:20:33] !log aaron started scap: php-1.23wmf6 Trying timing test again [05:20:52] Logged the message, Master [05:25:52] !log aaron finished scap: php-1.23wmf6 Trying timing test again [05:26:13] Logged the message, Master [05:26:35] 1/2 that time was searchidx1001 [05:30:10] TimStarling: ok, any idea what's wrong with searchidx1001? [05:30:54] * Aaron|home waits an age for << time sync-common >> to finish [05:31:51] looks like disk activity again [05:32:04] this reminds me, you know how you said dsh is slow? [05:32:59] it shouldn't be that slow, it may be your agent, or the network connection to it, that's at fault [05:34:13] TimStarling: the scap took 5 min, and really would be far less without searchidx1001 (and with some more perf fixes too) [05:34:36] I'm running it on a kde install on a USB on my gaming desktop [05:34:59] since using windows clients (git-bash/putty) gives slow garbage results [05:38:13] does searchidx even use MW? [05:38:50] it's java [05:38:55] says iotop [05:40:52] as you might expect [05:41:57] * Aaron|home will be glad when elastic replaces all this garbage [05:42:20] indexing should really run at low priority [05:43:03] I don't think it makes sense to stall web processes for scap, but it probably makes sense to stall search indexing and cron jobs [05:44:44] "The priority within the best-effort class will be dynamically derived from the CPU nice level of the process: io_priority = (cpu_nice + 20) / 5. " [05:45:32] so it should be enough to just run it with nice -n10 or something [05:46:22] ah, right [05:47:19] lol, my pipe broke before sync-common finished ;) [05:48:13] appservers run sshd at nice level -10, so scap will get an IO priority boost there since it runs from sshd [05:48:33] it'd be funny if salt turned out to be slower for that reason, and nobody could work out why ;) [05:48:48] on searchidx1001, sshd is running at nice 0 [05:49:09] so is java [05:52:36] * Aaron|home sees rsync in top now [06:04:43] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [06:04:43] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [06:05:25] TimStarling: so there is a reasonable fix for this? [06:06:33] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 06:06:31 UTC 2013 [06:06:43] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 06:06:31 AM UTC [06:07:03] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 06:06:57 UTC 2013 [06:07:27] well, we could extend the sshd nice level hack to all mediawiki-installation servers, instead of just the ones with applicationserver::service [06:07:44] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 06:06:57 AM UTC [06:20:46] TimStarling: any reason not to? [06:22:36] well, I'm not sure what else runs via ssh [06:22:41] backups etc. [06:23:45] but if there are low-priority things running via ssh, we can fix them individually [06:24:27] it's probably the right approach [06:25:17] maybe we should renice sshd everywhere, in the base module [07:04:19] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [07:04:39] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [07:06:50] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 07:06:45 UTC 2013 [07:07:19] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 07:06:45 AM UTC [07:07:29] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 07:07:21 UTC 2013 [07:07:39] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 07:07:21 AM UTC [07:11:54] ori: what do you think of that? [07:12:09] renicing sshd in the base module? [07:12:21] yes [07:13:18] that said searchidx1001 is really having a hard time (all commands are slow to start there), so it's a bit of an outlier, but it should help [07:14:02] have you tried renicing sshd on searchidx1001? what was the effect on scap runtime? [07:14:25] can I even do that? [07:14:39] i think so [07:16:07] doing it everywhere might be worth doing, but it will still be the case that scap will hang if a host is saturated, right? [07:16:23] without indicating to the deployer that this is what is happening [07:16:27] I don't have permission [07:17:01] ori: yeah, that server was like 90% of the wait [07:17:32] it was terbium earlier due to some overactive script there [07:18:28] i reniced it [07:18:29] old priority 0, new priority -10 [07:19:48] so there's a logging problem that wouldn't be resolved by renicing [07:20:20] ori: btw, do you have any clue why $DSH_EXPORTS might not get applied to scap-2 on the remote hosts? [07:22:00] !log aaron started scap: php-1.23wmf6 Timing test [07:22:18] Logged the message, Master [07:26:33] !log aaron finished scap: php-1.23wmf6 Timing test [07:26:48] 4 minutes [07:26:55] 4min41 sec, with that search service taking like 1min [07:27:03] much better though [07:27:10] Logged the message, Master [07:27:41] though the list of servers is shuffled, so if it alternates between being near the first or last the long-tail will show differently [07:28:00] the earlier that box is synced, the faster scap will randomly seem [07:30:47] i sometimes do careless things, like running zgrep with a regexp on several large rotated log files [07:30:53] with the default priority [07:31:50] if i were to do that on fluorine, and sshd was reniced -10 [07:32:13] that would put the demuxer in danger, right? [07:33:14] i'm not suggesting i should be careless with impunity, just pointing out that people do resource-intensive things in ssh sessions sometimes without realizing it [07:33:43] what if the scap commands just have nice in them? [07:33:54] yeah, that seems better [07:35:30] so<> ? [07:37:49] well nice -n 10 I mean [07:38:11] -10 that is [07:38:57] i think so, but let's wait and see what paravoid thinks [07:39:47] !log resume schema changes for gerrit 51675 externallinks.el_id [07:39:51] that will give perm errors though [07:40:05] Logged the message, Master [07:40:07] if renicing sshd everywhere allows us to debug overloads it could be useful but i figure ops use the console in such cases [07:43:09] Aaron|home: maybe in limits.conf [08:03:10] (03CR) 10Ori.livneh: [C: 04-1] Avoid md5sum calls in MergeCdbFileUpdates (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/104699 (owner: 10Aaron Schulz) [08:04:21] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [08:04:31] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [08:05:34] (03PS2) 10Aaron Schulz: Avoid md5sum calls in MergeCdbFileUpdates [operations/puppet] - 10https://gerrit.wikimedia.org/r/104699 [08:07:01] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 08:06:56 UTC 2013 [08:07:21] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 08:07:12 UTC 2013 [08:07:21] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 08:06:56 AM UTC [08:07:31] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 08:07:12 AM UTC [09:04:32] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [09:04:32] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [09:07:02] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 09:06:54 UTC 2013 [09:07:12] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 09:07:11 UTC 2013 [09:07:32] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 09:07:11 AM UTC [09:07:32] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 09:06:54 AM UTC [09:56:36] (03PS2) 10Hashar: Central OAuth wiki for Labs (metawiki) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104666 (owner: 10CSteipp) [10:01:35] (03CR) 10Hashar: [C: 04-1] "Copy pasted bug first comment as a commit summary to get some context attached to the code." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104666 (owner: 10CSteipp) [10:04:30] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [10:04:31] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [10:05:56] akosiaris: https://gerrit.wikimedia.org/r/#/c/104504/ i hope this is what you meant :) [10:06:50] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 10:06:46 UTC 2013 [10:07:20] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 10:07:11 UTC 2013 [10:07:31] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 10:06:46 AM UTC [10:07:31] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 10:07:11 AM UTC [10:14:12] (03CR) 10Alexandros Kosiaris: [C: 04-1] "The approach is sound. Mostly minor fixes plus one big fix (scopes thingy)" (036 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/104504 (owner: 10Matanya) [10:26:38] (03PS5) 10Matanya: ircecho: move to a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/104504 [10:28:04] thanks akosiaris. comments done [10:45:19] (03PS2) 10Odder: Create Chinese Wikivoyage (zhwikivoyage) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104355 [10:45:28] (03CR) 10jenkins-bot: [V: 04-1] Create Chinese Wikivoyage (zhwikivoyage) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104355 (owner: 10Odder) [10:50:07] (03PS3) 10Odder: Create Chinese Wikivoyage (zhwikivoyage) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104355 [10:57:19] (03PS1) 10Odder: Close wikimania2013 wiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104726 [11:04:51] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [11:05:01] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [11:06:41] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 11:06:34 UTC 2013 [11:07:01] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 11:06:34 AM UTC [11:07:32] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 11:07:29 UTC 2013 [11:07:51] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 11:07:29 AM UTC [11:22:33] (03CR) 10Alexandros Kosiaris: [C: 04-1] "There is one breaking change. Other than that LGTM. Adding a" (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/104504 (owner: 10Matanya) [11:25:08] (03CR) 10Matanya: "yeah, will fix those and add the description, good idea." (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/104504 (owner: 10Matanya) [11:28:25] (03PS6) 10Matanya: ircecho: move to a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/104504 [11:55:15] (03PS7) 10Alexandros Kosiaris: ircecho: move to a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/104504 (owner: 10Matanya) [11:55:47] (03PS2) 10Hashar: adding '*.raa.se' to the wgCopyUploadsDomains array. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104683 (owner: 10Dan-nl) [11:57:04] (03CR) 10Hashar: [C: 032] "deploying. We might want to make that list configurable by some user group via a new special page :-)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104683 (owner: 10Dan-nl) [11:57:13] (03Merged) 10jenkins-bot: adding '*.raa.se' to the wgCopyUploadsDomains array. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104683 (owner: 10Dan-nl) [11:58:45] !log hashar synchronized wmf-config/InitialiseSettings.php 'adding *.raa.se to the wgCopyUploadsDomains array {{gerrit|104683}}' [11:58:56] (03CR) 10Hashar: "deployed in production." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104683 (owner: 10Dan-nl) [11:59:26] (03CR) 10Alexandros Kosiaris: [C: 032] ircecho: move to a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/104504 (owner: 10Matanya) [11:59:36] Logged the message, Master [12:01:36] hello [12:04:54] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [12:05:04] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [12:05:30] buona sera signore [12:06:32] akosiaris: mind merging in a change for ryan deployment system please ? : -D https://gerrit.wikimedia.org/r/#/c/103095 [12:06:39] adds in the integration/kss.git repo in git-deploy [12:06:42] rgist ? [12:06:49] oh rgist [12:06:55] we need a wiki page to list all the possible names [12:07:04] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 12:06:54 UTC 2013 [12:07:04] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 12:06:59 UTC 2013 [12:07:05] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [12:07:28] i firmly believe rgist is the best [12:07:31] icinga-wm: make your mind up [12:07:41] it has all the previous names deeply embedded :-) [12:07:54] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 12:06:59 PM UTC [12:08:06] ryan will disagree obviously :-D [12:08:12] hashar: Do not +2 until the repo integration/kss has been created on gerrit. [12:08:33] akosiaris: yeah it has been created and I forged a commit for it let me clarify [12:08:52] forging ????? isn't that illegal ? :P [12:08:56] ok merging [12:09:11] (03CR) 10Alexandros Kosiaris: [C: 032] Add integration/kss to deployment repo config [operations/puppet] - 10https://gerrit.wikimedia.org/r/103095 (owner: 10Spage) [12:09:13] (03CR) 10Hashar: "The repository has been created and I added a dummy commit that introduce the .gitreview file with https://gerrit.wikimedia.org/r/104730 :" [operations/puppet] - 10https://gerrit.wikimedia.org/r/103095 (owner: 10Spage) [12:09:34] there you go [12:09:43] akosiaris: thanks, no urgency to have it deployed on tin since the repository is empty for now. S will follow up later I guess [12:12:57] (03PS2) 10Hashar: contint: package curl on slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/97526 [12:17:23] (03PS1) 10Alexandros Kosiaris: Migrate ircecho module's nrpe checks to nrpe module [operations/puppet] - 10https://gerrit.wikimedia.org/r/104733 [12:29:34] PROBLEM - Auth DNS on labs-ns0.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:30:24] RECOVERY - Auth DNS on labs-ns0.wikimedia.org is OK: DNS OK: 0.078 seconds response time. nagiostest.beta.wmflabs.org returns 208.80.153.219 [12:37:49] (03CR) 10Hashar: "bunch of nitpicking, haven't looked at apache/kibana configuration." (034 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/104172 (owner: 10BryanDavis) [12:47:48] (03CR) 10Hashar: [C: 031] "I am not a big fan of adding new extension-list but the change is fine and temporary. So I guess it is not a big deal." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95996 (owner: 10Aude) [12:48:48] (03CR) 10Alexandros Kosiaris: [C: 032] Migrate ircecho module's nrpe checks to nrpe module [operations/puppet] - 10https://gerrit.wikimedia.org/r/104733 (owner: 10Alexandros Kosiaris) [12:56:37] PROBLEM - Auth DNS on labs-ns0.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:57:27] RECOVERY - Auth DNS on labs-ns0.wikimedia.org is OK: DNS OK: 0.149 seconds response time. nagiostest.beta.wmflabs.org returns 208.80.153.219 [12:59:56] (03CR) 10Aude: "how about Thursday afternoon? or someone choose when :)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95996 (owner: 10Aude) [13:23:41] (03CR) 10Hashar: "Thursday afternoon is fine to me. Want to do that before elastic search upgrade which is at 5pm UTC. Adding an entry on https://wikitech." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95996 (owner: 10Aude) [13:29:50] akosiaris: got a dns entry for you if you got time https://gerrit.wikimedia.org/r/104376 for kr.wikimedia.org [13:48:36] PROBLEM - Auth DNS on labs-ns0.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:49:26] RECOVERY - Auth DNS on labs-ns0.wikimedia.org is OK: DNS OK: 5.333 seconds response time. nagiostest.beta.wmflabs.org returns [13:57:36] PROBLEM - Auth DNS on labs-ns0.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:58:26] RECOVERY - Auth DNS on labs-ns0.wikimedia.org is OK: DNS OK: 0.149 seconds response time. nagiostest.beta.wmflabs.org returns 208.80.153.219 [14:05:39] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [14:05:49] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [14:06:59] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 14:06:50 UTC 2013 [14:07:09] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 14:07:00 UTC 2013 [14:07:39] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 02:06:50 PM UTC [14:07:49] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 02:07:00 PM UTC [14:18:39] PROBLEM - Auth DNS on labs-ns0.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call [14:20:29] RECOVERY - Auth DNS on labs-ns0.wikimedia.org is OK: DNS OK: 0.320 seconds response time. nagiostest.beta.wmflabs.org returns 208.80.153.219 [14:47:47] (03PS1) 10Dan-nl: beta: gwtoolset filebackend [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104739 [14:52:57] (03CR) 10Alexandros Kosiaris: [C: 032] Add kr.wikimedia DNS entry [operations/dns] - 10https://gerrit.wikimedia.org/r/104376 (owner: 10John F. Lewis) [14:56:03] (03PS1) 10Dan-nl: production: add gwtoolset to extension-list [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104741 [14:56:29] (03CR) 10Hashar: [C: 032] beta: gwtoolset filebackend [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104739 (owner: 10Dan-nl) [14:56:50] (03Merged) 10jenkins-bot: beta: gwtoolset filebackend [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104739 (owner: 10Dan-nl) [15:04:33] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [15:05:33] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [15:07:12] >>> new Date [15:07:15] bah [15:07:33] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 15:07:24 UTC 2013 [15:07:33] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 15:07:24 UTC 2013 [15:07:33] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [15:08:33] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 03:07:24 PM UTC [15:48:11] (03PS1) 10Hashar: retab certs.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/104742 [15:48:12] (03PS1) 10Hashar: certs.pp puppet lint fixes [operations/puppet] - 10https://gerrit.wikimedia.org/r/104743 [15:49:32] (03CR) 10Hashar: "I am not sure of the impacts in production if I have made any mistake in there :(" [operations/puppet] - 10https://gerrit.wikimedia.org/r/104743 (owner: 10Hashar) [15:53:41] whoops, missed hashar [16:04:59] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [16:05:18] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [16:07:28] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 16:07:18 UTC 2013 [16:07:38] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 16:07:28 UTC 2013 [16:07:59] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 04:07:28 PM UTC [16:08:18] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 04:07:18 PM UTC [17:04:09] something seems unhappy with centralnotice [17:04:50] mwalkerz: are you centralnotice? [17:05:04] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [17:05:05] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [17:05:30] No banner exists where tmp_name = B13_123115_yer_enUS [17:07:14] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 17:07:06 UTC 2013 [17:07:14] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 17:07:11 UTC 2013 [17:08:04] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 05:07:11 PM UTC [17:08:05] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 05:07:06 PM UTC [17:09:30] manybubbles: does it need to be disabled? any meta-wiki admin or staffer can do so until it's fixed [17:09:58] Nemo_bis: I dunno if it needs to be disabled, jus that it is logging more errors then normal [17:10:08] I can't tell if those errors are catastrophic from here [17:10:48] it might just need some configuration fixed [17:10:53] or a banner uploaded, or something [17:29:33] (03CR) 10Chad: "Code's live, so this can probably go live whenever." [operations/puppet] - 10https://gerrit.wikimedia.org/r/103768 (owner: 10BryanDavis) [18:04:01] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [18:04:11] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [18:06:47] ^ Epoch Fail [18:07:01] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 18:06:57 UTC 2013 [18:07:11] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 06:06:57 PM UTC [18:07:21] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 18:07:18 UTC 2013 [18:08:01] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 06:07:18 PM UTC [18:12:31] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:13:21] RECOVERY - RAID on searchidx1001 is OK: OK: optimal, 1 logical, 4 physical [18:27:04] please revert https://git.wikimedia.org/commitdiff/mediawiki%2Fextensions%2FParserFunctions/164a6469b6d95c447869a1a427a66150b76a2c58 and deploy it as hotfix to hewikisource [18:27:30] this change breaks magic words and many templates are now broken [18:38:42] *HOTFIX* please revert https://git.wikimedia.org/commitdiff/mediawiki%2Fextensions%2FParserFunctions/164a6469b6d95c447869a1a427a66150b76a2c58 - Hebrew Wikisource it currently broken [18:39:24] eranroz: is there a bug for this? [18:39:44] https://bugzilla.wikimedia.org/46613 [18:39:49] siebrand: ^^ [18:40:08] ah, I see [18:40:11] he's commented [18:42:08] eranroz: have you commented on the bug (are you kipod)? [18:42:20] no [18:42:38] it seems there is disagreement about the next step here [18:42:40] but this bug happend on march and then there was revert for the specific change [18:42:51] the site is currently broken [18:42:59] we can take everything back and then disscuss it [18:44:32] Reedy: thoughts ^^ [18:46:54] https://gerrit.wikimedia.org/r/#/c/56196/ [18:47:03] Reedy everted such change feww months ago [18:47:07] when it first happend [18:47:26] there is probably a bug in Transalte extension that automaticly breaks the hebrew translations [18:47:36] whenever someone change other messages there [18:51:17] eranroz: can you show me a broken page? [18:51:22] also, interesting http://i.imgur.com/ChiXrgS.jpg [18:51:54] You have a really unreadable screen, greg-g [18:52:04] well this is because of RTL issuses [18:52:13] i'm not sure it is a bug :) [18:52:21] twkozlowski: sorry, dumb default of jpg there [18:52:22] http://imgur.com/LuL2wwh [18:52:33] well, my username is incorrectly displayed :) [18:53:18] for exahttps://he.wikisource.org/wiki/%D7%95%D7%99%D7%A7%D7%99%D7%98%D7%A7%D7%A1%D7%98:%D7%9E%D7%96%D7%A0%D7%95%D7%9F [18:53:22] https://he.wikisource.org/wiki/%D7%95%D7%99%D7%A7%D7%99%D7%98%D7%A7%D7%A1%D7%98:%D7%9E%D7%96%D7%A0%D7%95%D7%9F [18:53:40] (the village pump message at the top) [18:53:54] ah, I see [18:54:03] we can fix the specific template there, but it occurs in all templates [18:54:04] Reedy: around? [18:54:09] eranroz: yeah [18:58:40] <^d> Force running l10n update now. [19:00:43] thanks (https://gerrit.wikimedia.org/r/#/c/104752/) [19:00:57] !log LocalisationUpdate completed (1.23wmf8) at Tue Dec 31 19:00:56 UTC 2013 [19:02:16] !log LocalisationUpdate completed (1.23wmf7) at Tue Dec 31 19:02:16 UTC 2013 [19:02:22] <^d> I think that's a lie though. [19:04:02] <^d> So we're changing i18n file format but nobody decided to update the LocalisationUpdate extension? [19:04:22] ? [19:05:15] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [19:05:16] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [19:05:32] ^d: ugggggggggggggggggggg [19:06:45] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 19:06:35 UTC 2013 [19:06:54] ori: does https://gerrit.wikimedia.org/r/#/c/104699/ look ok now? [19:07:16] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 07:06:36 PM UTC [19:07:35] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 19:07:26 UTC 2013 [19:08:11] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Dec 31 19:08:11 UTC 2013 [19:08:15] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 07:07:26 PM UTC [19:08:16] <^d> greg-g: Nevermind. I was misreading the exception. [19:08:23] <^d> In any case, it's sorta broke [19:08:44] just a guess, because of AaronSchulz's changes? [19:08:59] <^d> What changes? [19:10:08] <^d> I'm getting stuff like http://p.defau.lt/?HkROozC11wjb2kUyQdGWhw [19:10:32] huh [19:12:38] !log LocalisationUpdate completed (1.23wmf8) at Tue Dec 31 19:12:38 UTC 2013 [19:14:02] !log LocalisationUpdate completed (1.23wmf7) at Tue Dec 31 19:14:02 UTC 2013 [19:14:42] ^d: errors reading the i18n files? I never touched that stuff [19:15:04] I thought that's what the cdb file stuff was, sorry [19:16:17] <^d> AaronSchulz: I know, that's why I said what changes :p [19:19:10] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Dec 31 19:19:10 UTC 2013 [19:21:54] ^d: so, still broken, what's next? :( [19:22:01] <^d> I'm trying something else. [19:22:04] k [19:23:06] !log demon synchronized php-1.23wmf7/extensions/ParserFunctions 'Updating PFuncs to master' [19:23:33] !log demon synchronized php-1.23wmf8/extensions/ParserFunctions 'Updating PFuncs to master' [19:23:52] <^d> Well, that's not enough either :\ [19:25:17] ^d: what are you trying to do? [19:25:28] <^d> Playing whack a mole. [19:25:37] well, besides that [19:25:57] <^d> hebrew wikis are broken with parser functions. [19:26:05] * AaronSchulz volunteers Reedy to make that make-wmf-branch change for extensions [19:26:09] <^d> A patch was committed, which I merged to master. [19:27:54] <^d> Tried to run l10nupdate, but that broke [19:33:09] AaronSchulz: reviewing [19:35:26] ^d: what is an example page? [19:35:38] <^d> https://he.wikisource.org/wiki/%D7%95%D7%99%D7%A7%D7%99%D7%98%D7%A7%D7%A1%D7%98:%D7%9E%D7%96%D7%A0%D7%95%D7%9F [19:35:53] <^d> https://gerrit.wikimedia.org/r/#/c/104752/ was the revert [19:39:01] i just noticed the revert have change something in the last line in the diff [19:39:35] (dont know what) [19:39:53] no new line at the EOF [19:40:08] can it cause problems to parsers [19:40:12] ? [19:40:12] !log aaron started scap: active [19:42:43] AaronSchulz: change looks ok if you commit to adding detailed usage notes (explaining, for example, when you would want to specify --no-checksum, and when not) in a follow-up commit [19:44:30] (03CR) 10Ori.livneh: "Looks ok if you commit to adding detailed usage notes (explaining, for example, when you would want to specify --no-checksum, and when not" (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/104699 (owner: 10Aaron Schulz) [19:51:36] (03PS1) 10Aaron Schulz: Made RefreshCdbJsonFiles include newlines in the JSON [operations/puppet] - 10https://gerrit.wikimedia.org/r/104762 [20:01:27] (03PS3) 10Aaron Schulz: Avoid md5sum calls in MergeCdbFileUpdates [operations/puppet] - 10https://gerrit.wikimedia.org/r/104699 [20:01:28] ori: there we go ^ [20:02:43] ottomata: hey, you around? can we talk about elasticsearch plugins. I want them so bad [20:03:21] ... [20:03:42] manybubbles: hey ja but not working right now [20:03:52] i'm helping my little cousin install linux, wooo! [20:04:07] manybubbles: but i need to get that stuff worked out soon too [20:04:11] so let's talk about it next week [20:04:29] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [20:04:29] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [20:07:09] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 20:07:04 UTC 2013 [20:07:29] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 08:07:04 PM UTC [20:07:39] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 20:07:29 UTC 2013 [20:08:29] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 08:07:29 PM UTC [20:20:16] (03PS1) 10Chad: Prioritize priority CirrusSearch jobs [operations/puppet] - 10https://gerrit.wikimedia.org/r/104763 [20:20:53] ^d: are those jobs really fast? [20:21:07] <^d> Should be, they're single page updates. [20:21:11] !log aaron finished scap: active [20:21:14] <^d> Triggered by page edits [20:21:22] <^d> (Rather than bulk updates) [20:21:23] I'd imagine just the fact that it's a separate queue will prioritize it better that before [20:21:42] ^d: so they take like 100s of ms? [20:21:49] or less [20:22:29] <^d> Yeah, at most a few 100ms each [20:23:16] <^d> http://p.defau.lt/?fF9p9rtdKx_vnVVRhfCrWQ [20:23:56] Hey folks. I need a package installed on stat1.wikimedia.org. Can someone tell me the right way to request this? [20:24:45] ^d: https://he.wikisource.org/wiki/%D7%95%D7%99%D7%A7%D7%99%D7%98%D7%A7%D7%A1%D7%98:%D7%9E%D7%96%D7%A0%D7%95%D7%9F looks fine now [20:25:05] <^d> Thanks. [20:25:41] they .122 sec/job [20:25:45] *they are [20:26:00] according to getJobProfileTimes on fluorine [20:26:28] (03CR) 10Aaron Schulz: [C: 031] "Average runtime is .122sec/job, which is fine" [operations/puppet] - 10https://gerrit.wikimedia.org/r/104763 (owner: 10Chad) [20:26:37] * halfak will just file a ticket with ops-requests and hope for the best.  [20:28:06] halfak: so thats the right first step [20:28:24] if its already an ubuntu package, thats much much easier. [20:28:31] RobH: Thanks. It is. [20:28:42] someone in ops reviews the package and ensures its ok to run, then we can add into system via puppet manifests [20:29:04] if you are comfortable doing the puppet work, you can check it into gerrit and attach a reference to it in the rt request [20:29:16] but everything after 'ops-request' is optional [20:29:23] more folks can do though, the faster stuff goes [20:29:43] then you can follow up with whoever is on RT triage duty that week normally [20:30:36] What's "RT triage duty" and how do I find out who is on it? [20:30:54] RT: name in topic [20:30:57] this week it is alex [20:31:14] https://wikitech.wikimedia.org/wiki/RT_Triage_Duty [20:31:34] Basically operations department now has a volunteer from our team every week as a point person for all operations requests [20:31:43] they triage the ops-request queue and other queues as well [20:32:01] so if you email and don't get a response for something, you have a go to person [20:32:19] Cool. So if this is a blocker for me (it is) I should ping the triager after filing the ticket? [20:32:35] its not a bad idea =] [20:33:05] though rt triage doesnt mean he has to personally do every task, but he will either do it or find someone to do it [20:33:19] RobH: Makes sense. Thanks for the help. [20:33:27] or if no one is available he can ping ken and see if we have to free up another project in progress [20:34:24] halfak: There's also the shortcut [20:34:33] You just need to bribe someone with a whisky [20:34:47] s/bribe/encourage/ [20:35:04] I would totally do that. How's the promise of whiskey? Also what type is preferred? [20:35:08] harder to do with ops though [20:35:17] since right now alex is triage in greece [20:35:21] (03CR) 10Manybubbles: [C: 031] Prioritize priority CirrusSearch jobs [operations/puppet] - 10https://gerrit.wikimedia.org/r/104763 (owner: 10Chad) [20:35:30] next week could be sean in AU, or me in SF, etc... [20:35:37] thats a LOT of whiskey to ship about [20:36:45] (03PS4) 10Ori.livneh: Avoid md5sum calls in MergeCdbFileUpdates [operations/puppet] - 10https://gerrit.wikimedia.org/r/104699 (owner: 10Aaron Schulz) [20:36:50] (03CR) 10Ori.livneh: [C: 032 V: 032] Avoid md5sum calls in MergeCdbFileUpdates [operations/puppet] - 10https://gerrit.wikimedia.org/r/104699 (owner: 10Aaron Schulz) [20:36:59] Hmmm. For SF, I can always just recruit a collaborator to obtain and deliver whiskey. [20:37:05] * halfak is in MN.  [20:37:51] halfak: what's the package? [20:38:01] python3-dev [20:38:09] oh, that's hardly objectionable [20:38:09] (03PS1) 10Chad: Move all other small misc. wikis over to Cirrus [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104765 [20:38:10] I just need the headers to compile oursql for python3 [20:38:32] ? [20:38:33] RobH: I'm going to add it to puppet. stat1 is explicitly cordoned off for analyst tooling [20:38:42] hrmm [20:38:51] running a db on it wha? [20:38:59] stat1 is insanity [20:39:01] (03PS2) 10Chad: Move all other small misc. wikis over to Cirrus [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104765 [20:39:08] i think someone will want to review that ;] [20:39:17] RobH: analysts typically load datasets from $datastore and process them [20:39:21] I just created the ticket. See https://rt.wikimedia.org/Ticket/Display.html?id=6561 [20:39:32] well, whatever reasoning you guys need it for please list it [20:39:48] so good enough, not saying no or nothin [20:39:55] just saying more info the better, makes it faster [20:40:14] * ori nods [20:56:22] ori: https://gerrit.wikimedia.org/r/#/c/104762/ is fairly simple [20:56:41] and the files don't crash vim with that ;) [21:05:29] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [21:05:39] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [21:07:09] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 21:07:07 UTC 2013 [21:07:19] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 21:07:17 UTC 2013 [21:07:29] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 09:07:07 PM UTC [21:07:39] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 09:07:17 PM UTC [21:10:27] hey Reedy, GWToolset on Commons no longer has i18n available to it. are you able to take a look at https://gerrit.wikimedia.org/r/#/c/104741/ and merge/deploy it if you're okay with it? [21:10:35] AaronSchulz: reviewing [21:10:51] ori: oh, and you can look at https://gerrit.wikimedia.org/r/#/c/103619/ (not related) :) [21:10:57] also, re: --trustmtime, i forgot to actually merge it earlier [21:11:00] so i just merged it now [21:11:06] in case you're wondering why it's not showing up on tin [21:11:11] AaronSchulz: kk [21:12:40] (03PS2) 10Reedy: production: add gwtoolset to extension-list [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104741 (owner: 10Dan-nl) [21:12:56] dan-nl: That means it needs a scap run... [21:13:19] thanks reedy … what's a scap run? [21:13:30] rebuilds the message caches etc [21:13:41] ah, okay ... [21:13:43] Or... I merge it now and wait for localistion update to fix it in a few hours [21:14:37] I wonder if extension-list-1.23wmf7 should just be deleted rather than a blank line [21:14:41] whichever you feel is best is fine [21:14:47] probably [21:14:56] AaronSchulz: Does scap work atm? [21:15:20] Reedy: yeah, I ran it earlier today, I can run again [21:15:29] * AaronSchulz is always curious what the timing is [21:15:50] Running it with no changes? [21:16:56] ? [21:17:19] that would be post merge of course [21:17:28] re: extension-list-1.23wmf7, i'm not sure why we needed it … with nothing in it i imagine you're right and the file can be deleted now. [21:18:39] Reedy: if you'd like i can deleted it and upload another patch set. just let me know [21:19:07] (03CR) 10Ori.livneh: [C: 032] Made RefreshCdbJsonFiles include newlines in the JSON [operations/puppet] - 10https://gerrit.wikimedia.org/r/104762 (owner: 10Aaron Schulz) [21:25:36] (03PS3) 10Reedy: production: add gwtoolset to extension-list [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104741 (owner: 10Dan-nl) [21:27:45] Reedy: should wmf-config/extension-list-labs stay in the config for future extensions that want to transition from labs to production or is there no consequence to placing new extensions in the extension-list immediately? [21:27:50] (03Merged) 10jenkins-bot: production: add gwtoolset to extension-list [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104741 (owner: 10Dan-nl) [21:28:19] dan-nl: It's wrapped in a file existence check [21:28:40] dan-nl: Putting it in extension-list is a problem if the extension is not in all of the 2-3 currently active branches [21:29:08] Reedy: when did that become the case? It used to not be a long time ago afaik [21:29:23] ~I [21:29:29] It's been a problem for a while [21:30:19] Which the extension-list-1.XXwmfY was the workaround [21:30:46] it used to give a notice and move on [21:31:20] It did [21:31:23] reedy@tin:/a/common$ time scap "Rebuild 1.23wmf8 l10n cache with gwtoolset in it" [21:31:23] Invalid MediaWiki version "Rebuild 1.23wmf8 l10n cache with gwtoolset in it" [21:31:27] Say wut? [21:31:44] reedy@tin:/a/common$ scap --help [21:31:44] Invalid MediaWiki version "--help" [21:31:47] Helpful. [21:31:53] scap never had --help [21:31:59] hmm … so how can i find out if gwtoolset is in all current active branches? [21:32:00] just do "scap active " [21:32:03] Reedy: lol [21:32:08] (03CR) 10Ori.livneh: "Looks good to me. What impact do you expect this to have? Should we notify someone in ops?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/104763 (owner: 10Chad) [21:32:12] 59169 [21:32:52] dan-nl: It's only usually a problem around first deploying an extension [21:33:05] It then goes into a config to make sure it's branched for every version going forward (until removed) [21:34:28] AaronSchulz: mind sending out a note about how scap has changed to engineering? [21:35:06] greg-g: it might be good to let things settle a bit first [21:35:20] !log reedy started scap: active Rebuild 1.23wmf8 l10n cache with gwtoolset in it [21:35:39] ori: when will that be? [21:35:45] we merged a lot of stuff over the past couple of days, and we don't have a good grasp of the impact quite yet [21:36:09] I'm not sure; at least a few days, I think. AaronSchulz, what do you think? [21:36:15] a few days?! [21:36:31] so, there's people planning to push stuff out on Thursday [21:36:44] just saying :) [21:36:46] That would've been fun waiting for SF to wake up [21:36:49] they don't need to do anything differently [21:36:54] Y U NO LET ME PREP DEPLOYMENT [21:36:56] ori: active? [21:37:02] [21:31:19] reedy@tin:/a/common$ time scap "Rebuild 1.23wmf8 l10n cache with gwtoolset in it" [21:37:02] [21:31:19] Invalid MediaWiki version "Rebuild 1.23wmf8 l10n cache with gwtoolset in it" [21:37:07] yeah, Reedy wasn't aware of how to deploy now [21:37:07] [21:31:57] just do "scap active " [21:37:10] That's something different [21:37:22] so, ie: something needs to either be communicated, or done differently [21:37:25] ah, hah, right. That's a good point [21:37:48] would have been better to test with a different command [21:38:06] "git deploy" was taken [21:38:13] leave scap alone, create scaptest or something and only switch when ready [21:39:02] meh, I don't really agree [21:39:26] why? [21:39:44] things were broken on Monday for manybubbles, now they are different for Reedy without notice... [21:39:49] seems things were done incorrectly to me [21:40:33] ok, i'm going to be slightly mean for a moment, because i feel strongly about this [21:40:39] cool [21:41:44] the official deployment sprint was done "correctly" from that perspective because things didn't break or change without notice, but the reason for that is that we didn't really make substantial changes [21:41:58] * greg-g nods [21:42:16] something very clearly is being done correctly at the moment because aaron managed to make substantial improvements to the deployment process [21:42:33] in some sense, my argument is a false dichotomy, because you can make substantial changes *and* communicate them [21:42:53] right [21:43:00] I was going to say those things are not related [21:43:03] and i was wrong above to suggest we should wait; i did forget about the subargument change [21:43:26] but people often fail to appreciate the cost of running a parallel setup, and the cost of simulating production conditions, and the cost of architecting on dry land [21:43:28] so why not make big major changes in a way that won't break other people's work? [21:43:37] I mean, we talk about feature flags all the time, why not here? [21:44:09] because bash is not a nice programming language and even trivial logic breaks so the overhead of adding that would not be worth it [21:44:16] (obviously not "scap --new blah", but that'd be neat, you know what I mean ;) ) [21:44:26] it's a question of standards and priorities and organizational culture [21:44:30] I mean what I said before, scaptest or something [21:44:50] we are not launching rockets into space; wikimedia is an experiment and we are where we are because were bold and tried things out [21:44:57] * greg-g sighs [21:45:13] sure, everything is an experiment so it doesn't matter when anyone breaks anything [21:45:17] arugment not valid [21:45:38] that is not license to break things with abandon, but man, i get tired of this protocol [21:45:42] sorry, I should have said the same thing you did about being mean [21:45:48] it is exaggerated and it stiffles work and creativity [21:46:01] kind of, so, tell me why doing this in scaptest wouldn't work? [21:46:06] (03CR) 10Manybubbles: "Andrew Otto is who we've been working with mostly in ops but I believe he is out this week. I've added him as a reviewer." [operations/puppet] - 10https://gerrit.wikimedia.org/r/104763 (owner: 10Chad) [21:46:43] maybe it would, but that's hindsight; the scope of the work didn't start out being substantial, it's just that aaron kept discovering more opportunities for optimizing [21:47:04] sure, I understand and appreciate that momentum aspect [21:47:25] just sometimes there's a line, is all [21:47:50] did the site go down or something? [21:48:10] on the bright side, we had already disabled pt.wiki emergency captcha earlier :D [21:48:18] Nemo_bis: :) :) [21:48:48] ori: no, just complaining that if you two are offline for any amount of time when someone else is trying to do something, they're stuck to no fault of their own [21:49:20] that's it really. [21:49:37] that is regrettable and in hindsight it is almost always the case that any breakage, even minor, could have been avoided [21:49:50] just maybe a heads up to the list like "hey, so, we started making some changes to scap, got bigger than we planned, not done, but we'll let you know when it's all clear" would be really nice [21:49:54] :) [21:50:04] I mean, scap is kind of important ;) [21:50:09] but harping on that reflects lopsided priorities, imo [21:50:21] the big news story is that things are finally moving with it [21:50:22] informed deployers vs progress? [21:50:28] yay! to that (honestly) [21:50:47] (also, my vs up there was just tryign to clarify, not continue an attack) [21:51:14] Nemo_bis: yay! \o/ [21:51:42] IIRC facebook senior engs issued tshirts for the staff that say something like "failures happen" [21:51:53] * greg-g nods [21:52:01] Reedy: just an fyi, the i18n for gwtoolset is now back on production … thanks! [21:52:19] out of recognition that creativity and momentum are rare and fleeting and easily suffocated by an exaggerated emphasis on safety and control [21:52:28] fair [21:52:31] dan-nl: Yeah, not completely finished though ;) [21:53:01] Reedy: on an unrelated note [21:53:11] twkozlowski: No [21:53:13] No you can't. [21:53:22] anyways , thanks for hearing me out. [21:53:30] ori: so, I apologize for over reacting, I just see worst-case scenarios and people asking me the "shoulda coulda" things [21:53:39] ori: ditto [21:53:42] Reedy: OK [21:54:01] :D [21:54:38] greg-g: yeah, me too (overreacting). i understand your perspective. [21:55:21] ori: hifive [21:55:50] * greg-g was gonna type "hi5" but thought that was too lolspeek [21:56:41] !log reedy finished scap: active Rebuild 1.23wmf8 l10n cache with gwtoolset in it [21:57:38] twkozlowski: Wassup? [21:57:42] dan-nl: Should be all fixed now [21:57:49] thanks again :) [21:59:13] AaronSchulz: OK, so I think that if -n "$1" but "$1" "!= "active" and not -d "$MW_COMMON_SOURCE/$1", we should assume $1 is a log message rather than a version [21:59:20] I noticed https://wikitech.wikimedia.org/wiki/Add_a_wiki is seriously outdated or plainly wrong [21:59:33] when I went through the instructions for zhwikivoyage [21:59:40] ori: I was actually making a regex [21:59:55] it's taking forever to get working in my test script though [22:00:06] even though it's trivial [22:00:52] maybe require --flag / --option=value [22:04:02] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [22:04:03] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [22:05:22] (03PS3) 10Manybubbles: Move all other small misc. wikis over to Cirrus [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104765 (owner: 10Chad) [22:06:52] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 22:06:48 UTC 2013 [22:07:03] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 10:06:48 PM UTC [22:07:04] (03PS1) 10Aaron Schulz: Allow "scap " calls as they previously worked [operations/puppet] - 10https://gerrit.wikimedia.org/r/104772 [22:07:13] ori: ^ [22:07:22] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 22:07:18 UTC 2013 [22:07:38] AaronSchulz: hmmm [22:08:03] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 10:07:18 PM UTC [22:08:05] AaronSchulz: better, IMO: https://dpaste.de/mQA2 [22:08:47] actually, after VERSIONS="${1#--versions=}", there should be a call to "shift" [22:09:41] (03PS4) 10Chad: Move all other small misc. wikis over to Cirrus [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104765 [22:10:07] the way you have it, if i mean to type active but instead type "acitve", scap proceeds without complaining and 'acitve' is logged [22:10:43] this way if the first condition is true ($1 starts with '--version=.*') you can fail if the parameter value is invalid [22:10:45] ori: can you commit that then? [22:10:54] sure, i'll update your patch [22:11:05] most of my reading on options parsing in bash involved piles of messy code [22:12:17] though I guess with this the order is pretty well known [22:18:30] (03PS5) 10Manybubbles: Move all other small misc. wikis over to Cirrus [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/104765 (owner: 10Chad) [22:26:20] !log aaron started scap: active timing test [22:29:47] hey, Flow has some code that only fails in production. We need to run some debugging code on test2wiki (mw1063). Is that OK? Do we need an LD slot to mess with it? [22:29:52] !log aaron finished scap: active timing test [22:30:18] 3m45sec, 63 seconds was on two slow boxes [22:31:41] spagewmf: That sounds pretty wrong [22:31:47] Testwiki runs on any apache [22:31:51] testwiki runs on one apache [22:31:58] and that apache isn't mw1063 [22:31:59] it's mw1017 [22:32:16] yeah 1063 does not look unique [22:32:28] (03PS2) 10Ori.livneh: Allow "scap " calls as they previously worked [operations/puppet] - 10https://gerrit.wikimedia.org/r/104772 (owner: 10Aaron Schulz) [22:32:49] ^ AaronSchulz [22:33:12] Reedy: fine, we can run on testwiki rather than test2wiki. It's a crazy "This shouldn't happen" PHP thing, so we're stumped. [22:33:26] spagewmf: should be fine [22:33:38] Just didn't want you to be looking in the wrong place and wtfing more [22:34:13] spagewmf: might be worth investigating out loud on #wikimedia-dev to increase the chance that someone overhears the analysis and has an idea [22:34:49] Reedy: Thanks. ori: good point [22:35:36] AaronSchulz: I'll wait for you to +1 (or -1, if you see issues) before merging [22:36:02] seems to work [22:36:10] (03CR) 10Aaron Schulz: [C: 031] Allow "scap " calls as they previously worked [operations/puppet] - 10https://gerrit.wikimedia.org/r/104772 (owner: 10Aaron Schulz) [22:36:39] (03CR) 10Ori.livneh: [C: 032] Allow "scap " calls as they previously worked [operations/puppet] - 10https://gerrit.wikimedia.org/r/104772 (owner: 10Aaron Schulz) [22:36:52] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 22:36:48 UTC 2013 [22:37:12] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 10:36:48 PM UTC [22:37:15] running puppet on tin [22:37:32] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 22:37:23 UTC 2013 [22:38:12] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 10:37:23 PM UTC [22:39:48] AaronSchulz: want to try it? [22:40:15] ok [22:41:33] !log aaron started scap: testing scap with no --versions flag [22:42:43] where is morebots? [22:43:02] leaving early for the New Years? [22:43:45] celebrating another undeserved year of life [22:43:55] ori: heh, it will be nice when the tampa boxes are gone from the dsh groups... >:) [22:44:16] !log aaron finished scap: testing scap with no --versions flag [22:44:30] 2m55sec [22:44:50] who decides what bots are worthy of live? oh...wait... [22:45:35] labs-morebots is alive and well on #wikimedia-labs, so it must be a failure of this specific instance [22:45:40] * ori investigates [22:46:10] Reedy: do you want to make that extension branch change or should I stab at it? [22:46:46] AaronSchulz: You can just cut the extension list and move it down to fix the problem ;) [22:46:56] Meanwhile, I'm doing just fine, because my implementation doesn't suck. [22:47:22] logmsgbot passes Turing test [22:48:50] Reedy or anyone, how can we put a modified PHP file just on mw1017? [22:49:16] change it on tin, sync-common on mw1017 [22:49:17] modified how? [22:49:24] but yes, that [22:49:28] or sudo -u mwdeploy EDITOR /path/to/file [22:53:49] ori, wfDebugLog("huh") [22:59:18] !log morebots went missing. job was active on toollabs but stuck. nothing useful in logs. restarted. [23:00:05] Logged the message, Master [23:01:29] AaronSchulz: i quibbled with the tone, but greg-g is probably right to ask for an e-mail [23:07:02] RECOVERY - Puppet freshness on virt1003 is OK: puppet ran at Tue Dec 31 23:06:51 UTC 2013 [23:07:03] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Tue Dec 31 23:06:56 UTC 2013 [23:07:12] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 11:06:56 PM UTC [23:07:12] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Last successful Puppet run was Tue 31 Dec 2013 11:06:51 PM UTC [23:27:21] glory be, problem is a condition someone's adding to watchlist query [23:27:24] AND (page_namespace != '90') [23:27:35] null propogates everywhere in sql :( [23:27:45] should be AND (page_namespace IS NULL OR page_namespace != '90')