[00:14:46] PROBLEM - Puppet freshness on neon is CRITICAL: No successful Puppet run in the last 3 hours [00:17:46] PROBLEM - Puppet freshness on tin is CRITICAL: No successful Puppet run in the last 3 hours [01:02:27] PROBLEM - MySQL Processlist on db1007 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:02:27] PROBLEM - MySQL InnoDB on db1007 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:02:36] PROBLEM - MySQL Slave Delay on db1007 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:02:36] PROBLEM - MySQL Replication Heartbeat on db1007 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:02:36] PROBLEM - MySQL Recent Restart on db1007 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:03:17] RECOVERY - MySQL InnoDB on db1007 is OK: OK longest blocking idle transaction sleeps for 0 seconds [01:03:17] RECOVERY - MySQL Processlist on db1007 is OK: OK 0 unauthenticated, 0 locked, 5 copy to table, 0 statistics [01:03:26] RECOVERY - MySQL Slave Delay on db1007 is OK: OK replication delay 0 seconds [01:03:26] RECOVERY - MySQL Replication Heartbeat on db1007 is OK: OK replication delay -1 seconds [01:03:27] RECOVERY - MySQL Recent Restart on db1007 is OK: OK 2244372 seconds since restart [01:04:46] PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 3 hours [01:08:25] !log db1007 is having tough times due to special page updates [01:08:42] Logged the message, Master [01:13:27] PROBLEM - MySQL InnoDB on db1007 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:13:27] PROBLEM - MySQL Processlist on db1007 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:14:26] RECOVERY - MySQL Processlist on db1007 is OK: OK 0 unauthenticated, 0 locked, 4 copy to table, 0 statistics [01:14:26] RECOVERY - MySQL InnoDB on db1007 is OK: OK longest blocking idle transaction sleeps for 0 seconds [01:16:36] PROBLEM - MySQL Slave Delay on db1007 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:17:36] RECOVERY - MySQL Slave Delay on db1007 is OK: OK replication delay 19 seconds [01:18:57] !log Killed a few long queries on db1007 [01:19:12] Logged the message, Master [01:20:27] PROBLEM - MySQL InnoDB on db1007 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:21:16] RECOVERY - MySQL InnoDB on db1007 is OK: OK longest blocking idle transaction sleeps for 0 seconds [01:21:27] fuck [01:22:12] MaxSem: did you save what the queries were? [01:22:19] yup [01:22:26] I might have to run in a second [01:22:28] good deal :) [01:22:48] Was about to paste them into email when noticed that there's another wikis that are slamming it now:P [01:23:10] :/ [01:23:20] need any help? [01:23:57] yer any good with linux? how do I look up a process' args? [01:24:04] (assuming I can sudo as it) [01:24:45] ps aux | grep someprocess [01:25:09] I have to run, ping people if needed :) [01:26:02] howly fuck [01:26:24] even if I go on rampage killing those processes it'll just spawn moar [01:32:32] Is it the cronjob/loop? [01:33:15] MaxSem: ^ [01:33:20] yup [01:33:40] aka mwdeploy 29076 0.0 0.0 4400 616 ? Ss 01:00 0:00 /bin/sh -c /usr/local/bin/mwscriptwikiset updateSpecialPages.php s7.dblist --override --only=Wantedpages > /home/mwdeploy/updateSpecialPages/s7@17-WantedPages.log 2>&1 [01:33:57] Killed on terbium [01:34:16] Oh, there's moar [01:35:18] Now they're gone [01:35:27] reedy@terbium:~$ ps aux | grep -i special [01:35:27] reedy 4536 0.0 0.0 9384 928 pts/4 S+ 01:35 0:00 grep --color=auto -i special [01:35:27] reedy@terbium:~$ [01:35:34] yup [01:35:43] !log Killed updateSpecialPages and related processes on terbium [01:35:46] They need disabling [01:35:53] We can't keep this happening [01:36:00] Logged the message, Master [01:36:04] But they were fine using the "idle" tampa slaves [01:37:37] and I killed the remaining queries [01:38:00] Wait for icinga-wm to catch up then [01:38:12] * Reedy blames Nemo_bis [01:38:47] also, like 5-6 queries were filesorting the page table on large wikis at the same time - probably they would've been faster if they weren't run in parallel [01:40:50] hmm - lag is zero but still there are no threads as if LB considered it deadly lagged [01:41:06] Why were there so many processes running simultaneously? [01:41:33] to DOS it properly?XD [01:41:36] Which I guess is the real issue [01:41:48] mwdeploy 29092 0.0 0.0 12308 1492 ? S 01:00 0:00 /bin/bash /usr/local/bin/mwscriptwikiset updateSpecialPages.php s7.dblist --override --only=Mostlinked [01:41:48] mwdeploy 29094 0.0 0.0 12308 1496 ? S 01:00 0:00 /bin/bash /usr/local/bin/mwscriptwikiset updateSpecialPages.php s7.dblist --override --only=Mostrevisions [01:41:48] mwdeploy 29096 0.0 0.0 12308 1492 ? S 01:00 0:00 /bin/bash /usr/local/bin/mwscriptwikiset updateSpecialPages.php s7.dblist --override --only=Wantedpages [01:41:48] mwdeploy 29104 0.0 0.0 12308 1496 ? S 01:00 0:00 /bin/bash /usr/local/bin/mwscriptwikiset updateSpecialPages.php s7.dblist --override --only=Fewestrevisions [01:41:48] mwdeploy 29105 0.0 0.0 12308 1496 ? S 01:00 0:00 /bin/bash /usr/local/bin/mwscriptwikiset updateSpecialPages.php s7.dblist --override --only=Deadendpages [01:41:50] mwdeploy 29132 0.0 0.0 12308 1492 ? S 01:00 0:00 /bin/bash /usr/local/bin/mwscriptwikiset updateSpecialPages.php s7.dblist --override --only=Ancientpages [01:41:52] etc [01:41:56] yup [01:42:07] Need staggering [01:42:39] Noting that paste above was around 25-30% of the processes killed [01:42:42] we can easily run one of these per DB cluster [01:44:11] Now, when are the next runs actually scheduled.... [01:45:46] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 3 hours [01:46:05] Hmm [01:46:16] Those s7 ones won't run till 17th December [01:46:27] *bzzt* Previous update run was unsuccessful, re-running after a 15 minutes wait. [01:46:31] :P [01:46:46] PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 3 hours [01:47:41] icinga-wm, I hate you too [01:48:17] don't shoot the messenger ;) [01:48:47] yeah? [01:48:53] what about my sleep? [01:49:16] sspeaking of sleep... [01:49:18] * MaxSem is going to bed [01:49:28] The Australians should be awake ;) [01:50:54] not everyone is a fucking loser like me and you who's not having a week end in addition to not having night sleep:P [01:51:03] good night;) [01:52:34] MaxSem: We can pipe icinga-wm to your mobile via SMS if you prefer [02:02:02] !log LocalisationUpdate completed (1.23wmf3) at Sun Nov 17 02:02:01 UTC 2013 [02:02:15] Logged the message, Master [02:02:55] !log LocalisationUpdate completed (1.23wmf4) at Sun Nov 17 02:02:54 UTC 2013 [02:03:14] Logged the message, Master [02:08:53] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Nov 17 02:08:53 UTC 2013 [02:09:08] Logged the message, Master [03:07:54] (03CR) 10TTO: "Please see I028589438e502bc1ca30f0148e71b706656331c4, of which this change is partly duplicative." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95796 (owner: 10Odder) [03:15:46] PROBLEM - Puppet freshness on neon is CRITICAL: No successful Puppet run in the last 3 hours [03:18:46] PROBLEM - Puppet freshness on tin is CRITICAL: No successful Puppet run in the last 3 hours [04:05:46] PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 3 hours [04:46:47] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 3 hours [04:47:46] PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 3 hours [06:16:46] PROBLEM - Puppet freshness on neon is CRITICAL: No successful Puppet run in the last 3 hours [06:19:46] PROBLEM - Puppet freshness on tin is CRITICAL: No successful Puppet run in the last 3 hours [06:27:16] PROBLEM - udp2log log age for lucene on oxygen is CRITICAL: CRITICAL: log files /a/log/lucene/lucene.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [06:29:16] RECOVERY - udp2log log age for lucene on oxygen is OK: OK: all log files active [07:06:46] PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 3 hours [07:47:46] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 3 hours [07:48:46] PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 3 hours [07:56:42] Reedy: they just need to be resorted [08:01:55] (commented on the bug) [08:51:35] (03PS1) 10Odder: (bug 56334) Namespace l20n for angwiki and angwiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95846 [08:54:31] twkozlowski: 20? [09:09:54] (03CR) 10TTO: "Nice work." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95846 (owner: 10Odder) [09:11:22] (03CR) 10Odder: "My apologies; I didn't realise you were working on $wgSitename, too." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95796 (owner: 10Odder) [09:12:39] (03CR) 10Odder: "Sorry? l20 stands for 'localization' and I had to use an abbreviation to fit in 62 characters :-)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95846 (owner: 10Odder) [09:17:46] PROBLEM - Puppet freshness on neon is CRITICAL: No successful Puppet run in the last 3 hours [09:18:04] (03PS5) 10TTO: Clean up wgSiteName in InitialiseSettings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86418 [09:20:00] (03CR) 10TTO: "That would be l10n: I thought you had made a typo and was making a silly joke." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95846 (owner: 10Odder) [09:20:46] PROBLEM - Puppet freshness on tin is CRITICAL: No successful Puppet run in the last 3 hours [09:26:01] (03PS2) 10Odder: (bug 56334) Namespace l20n for angwiki and angwiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95846 [09:38:12] (03PS2) 10Odder: (bug 44629) Clean up $wgMetaNamespace, $wgMetaNamespaceTalk [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95796 [10:07:46] PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 3 hours [10:48:46] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 3 hours [10:49:46] PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 3 hours [12:18:46] PROBLEM - Puppet freshness on neon is CRITICAL: No successful Puppet run in the last 3 hours [12:21:46] PROBLEM - Puppet freshness on tin is CRITICAL: No successful Puppet run in the last 3 hours [12:40:59] (03PS1) 10Mark Bergsma: Take mark off the SMS list [operations/puppet] - 10https://gerrit.wikimedia.org/r/95859 [12:42:35] (03CR) 10Mark Bergsma: [C: 032] Take mark off the SMS list [operations/puppet] - 10https://gerrit.wikimedia.org/r/95859 (owner: 10Mark Bergsma) [13:08:06] PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 3 hours [13:49:06] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 3 hours [13:50:06] PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 3 hours [15:19:06] PROBLEM - Puppet freshness on neon is CRITICAL: No successful Puppet run in the last 3 hours [15:22:06] PROBLEM - Puppet freshness on tin is CRITICAL: No successful Puppet run in the last 3 hours [16:09:06] PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 3 hours [16:50:06] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 3 hours [16:51:06] PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 3 hours [17:11:56] I've been getting quite a few HTTP 503 errors for some time while trying to add some categories with pywikipediabot [17:17:29] twkozlowski: https://bugzilla.wikimedia.org/show_bug.cgi?id=55219? [17:18:26] hm... 503 Service unavailable :/ [17:28:01] (03PS1) 10MaxSem: Serialize special page updates [operations/puppet] - 10https://gerrit.wikimedia.org/r/95876 [17:36:25] paravoid: are you available? [18:20:06] PROBLEM - Puppet freshness on neon is CRITICAL: No successful Puppet run in the last 3 hours [18:23:06] PROBLEM - Puppet freshness on tin is CRITICAL: No successful Puppet run in the last 3 hours [18:53:03] Hm, Wikisource is awfully slow for me right now. [18:59:16] twkozlowski: how about Commons? I've been uploading a 5 MB file for what looks like several minutes now [19:00:01] Nemo_bis: Same. [19:04:53] Also, I'm seeing blank pages after page loads. [19:10:06] PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 3 hours [19:10:38] you're right [19:11:00] !log The 5xx errors mega-spikes every ten minutes are back [19:11:15] Logged the message, Master [19:12:56] !log 20 % packet loss from Toolserver to bits-lb.esams.wikimedia.org [19:13:09] (that's usually not a good sign) [19:13:10] Logged the message, Master [19:14:44] * MatmaRex np U2 - Sunday Bloody Sunday [19:14:54] heh [19:16:37] icinga-wm is on the spot as always [19:32:51] Nemo_bis: i see similar issues from my local connection and from pmtpa. only the last hop though. so i guess broken internally @ esams [19:32:54] :( [19:33:38] i'm getting ~98% loss from here and ~15% from pmtpa [19:34:02] but it's taking different routes [19:35:37] PROBLEM - check_job_queue on arsenic is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , enwiki (259877), Total (278383) [19:35:37] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , enwiki (259877), Total (278383) [19:35:45] to be clear, to the same hostname Nemo_bis was trying. bits-lb.esams.wikimedia.org [19:36:15] (03PS1) 10Faidon Liambotis: Switch mobile-lb to eqiad [operations/dns] - 10https://gerrit.wikimedia.org/r/95884 [19:36:36] (03CR) 10jenkins-bot: [V: 04-1] Switch mobile-lb to eqiad [operations/dns] - 10https://gerrit.wikimedia.org/r/95884 (owner: 10Faidon Liambotis) [19:37:27] (03PS2) 10Faidon Liambotis: Switch mobile-lb to eqiad [operations/dns] - 10https://gerrit.wikimedia.org/r/95884 [19:37:55] (03CR) 10Faidon Liambotis: [C: 032] Switch mobile-lb to eqiad [operations/dns] - 10https://gerrit.wikimedia.org/r/95884 (owner: 10Faidon Liambotis) [19:39:00] jeremyb: yes, mtr gives me 14 % packet loss at last hop to bits-lb.esams.wikimedia.org after 1000 attempts [19:39:19] (from Milan) [19:39:42] (03PS1) 10Faidon Liambotis: Fix TTL for gerrit's AAAA to match A [operations/dns] - 10https://gerrit.wikimedia.org/r/95885 [19:40:00] (03CR) 10Faidon Liambotis: [C: 032] Fix TTL for gerrit's AAAA to match A [operations/dns] - 10https://gerrit.wikimedia.org/r/95885 (owner: 10Faidon Liambotis) [19:40:40] paravoid: please ping me when you are available. I'd like to talk about some design changes i need you wizdom about [19:40:48] I am here but dealing with the outage [19:41:08] partial outage, but still [19:41:15] yes we noticed :) thanks [19:41:26] another outage? [19:41:27] when ever you have time. not urgent. [19:41:33] Aaron|home: see above [19:41:34] Aaron|home: yeah... [19:41:39] danke paravoid [19:45:48] aude: so exactly do end up trying to get output for the same rev ID and content but where the output needs to be different than the first parse that was just done? [19:46:51] Aaron|home: i wonder if there's a word missing from that sentence? [19:47:46] *how exactly [19:51:06] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 3 hours [19:52:06] PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 3 hours [20:01:40] (03PS1) 10Nemo bis: Make the monthly querypages updates not hit each cluster on the same day [operations/puppet] - 10https://gerrit.wikimedia.org/r/95889 [20:02:26] (03CR) 10Nemo bis: "As said on the bug this morning, I think Ib4c84101c7f04e8b0a96f4c05891f4d1b40154be will be more effective." [operations/puppet] - 10https://gerrit.wikimedia.org/r/95876 (owner: 10MaxSem) [20:03:00] (03CR) 10Reedy: [C: 031] "LGTM as an improvement over spawning them all at the same time!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/95876 (owner: 10MaxSem) [20:04:14] Nemo_bis, as a result you're running the same page update on the whole cluster at the same time [20:04:24] (03CR) 10Dereckson: "What about the $wmf variable in docroot/noc/db.php ($wmf = wmfClusters())?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94598 (owner: 10Arav93) [20:04:34] might not be the best idea [20:04:58] MaxSem: why not? [20:05:09] => rand9) [20:05:12] => rand() [20:05:53] rather, I suspect I may be adding duplicate cronjobs which make puppet cry [20:06:45] You mean rather than them being updated? [20:06:52] We should make a game of site outage or special page updates [20:06:59] ^^ [20:08:35] (03PS1) 10Faidon Liambotis: Switch bits-lb to eqiad (away from esams) [operations/dns] - 10https://gerrit.wikimedia.org/r/95890 [20:08:54] gotta shoot them all [20:09:14] (03CR) 10jenkins-bot: [V: 04-1] Switch bits-lb to eqiad (away from esams) [operations/dns] - 10https://gerrit.wikimedia.org/r/95890 (owner: 10Faidon Liambotis) [20:09:25] Reedy: whey did you need unzip on tin? [20:09:37] because someone gave me a zip file [20:09:46] (03PS2) 10Faidon Liambotis: Switch bits-lb to eqiad (away from esams) [operations/dns] - 10https://gerrit.wikimedia.org/r/95890 [20:10:20] tin also felt left out as fenari already had it [20:10:27] (03CR) 10Faidon Liambotis: [C: 032] Switch bits-lb to eqiad (away from esams) [operations/dns] - 10https://gerrit.wikimedia.org/r/95890 (owner: 10Faidon Liambotis) [20:11:06] *sigh* [20:11:13] that seems to do it [20:11:34] back to just 98% CPU instead of pegged to 100% [20:11:42] Reedy: I'd like to add it to a role, so it won't be just thrown there, any idea which? [20:11:45] my ping looks better [20:11:57] yup [20:12:19] funny how that happened the day mark leaves :) [20:13:03] leaves what :O [20:13:07] did bits cache rate go down? [20:13:08] (for vacation) [20:13:13] ah [20:13:16] * Nemo_bis phews [20:13:17] MaxSem: I switched bits to eqiad [20:13:39] just wondering what happened [20:14:09] the traffic increased organically, the esams LVS box's CPU had trouble keeping up with the load [20:14:31] packets were delayed and some of them lost [20:15:26] mhm. having only the topmost architect as a DC technician is bound to cause problems:P [20:15:38] we have smarthands [20:16:54] looks good from here too [20:16:56] can we count on them to install a bunch of new servers?;) [20:17:11] it's a little more complicated than that [20:17:36] the CPU load is unbalanced between the CPUs, I experimented a bit with tuning RPS settings [20:17:41] with not much luck [20:17:53] maybe using the second ethernet card would make a difference [20:18:01] I can also use some of the other lvs boxes [20:19:56] it's messy with puppet though, since either I'd need to rework on it a lot, or shuffle traffic around for eqiad too (which it's not a very good idea) [20:26:39] http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&h=amslvs1.esams.wikimedia.org&m=cpu_report&s=by+name&mc=2&g=network_report&c=LVS+loadbalancers+esams [20:26:42] bits [20:30:27] (03CR) 10Nemo bis: "Quick math to avoid duplicate cronjobs panic (Ib4c84101): we run one page per cluster in each cron, 6*7=42; we call updatequerypages::cron" [operations/puppet] - 10https://gerrit.wikimedia.org/r/95889 (owner: 10Nemo bis) [20:31:08] (03CR) 10Nemo bis: "* I0a5d8603" [operations/puppet] - 10https://gerrit.wikimedia.org/r/95889 (owner: 10Nemo bis) [20:32:22] Nemo_bis: btw, thanks for pointing out both the 5xx spike and the packet loss [20:32:43] you did half the investigation, when I jumped in I found the root cause very quickly because of that [20:32:46] you're welcome; it was twkozlowski making me check it though :) [20:33:14] and I remembered what to check because that's what you had done last time, so [20:33:41] * Nemo_bis is just shell/script monkey as usual [20:34:58] paravoid: you may want to !log something if the crisis is over :) [21:21:06] PROBLEM - Puppet freshness on neon is CRITICAL: No successful Puppet run in the last 3 hours [21:24:06] PROBLEM - Puppet freshness on tin is CRITICAL: No successful Puppet run in the last 3 hours [21:46:30] (03PS1) 10Faidon Liambotis: Switch European bits-lb & mobile-lb back to esams [operations/dns] - 10https://gerrit.wikimedia.org/r/95950 [21:46:56] (03CR) 10Faidon Liambotis: [C: 032] Switch European bits-lb & mobile-lb back to esams [operations/dns] - 10https://gerrit.wikimedia.org/r/95950 (owner: 10Faidon Liambotis) [21:55:43] damn [21:55:46] peak hours are over [21:55:51] I can't test my change :) [22:11:06] PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 3 hours [22:52:06] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 3 hours [22:53:06] PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 3 hours [23:11:18] (03CR) 10Twotwotwo: "Tried and failed to submit this as a comment on gerrit, so here it is over e-mail:" [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/63139 (owner: 10Twotwotwo) [23:11:44] (03CR) 10Twotwotwo: "Except obviously the "Tried and failed to submit this in Gerrit" tag on the comment is no longer accurate. :)" [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/63139 (owner: 10Twotwotwo) [23:26:11] (03CR) 10MZMcBride: Make the monthly querypages updates not hit each cluster on the same day (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/95889 (owner: 10Nemo bis) [23:28:25] (03CR) 10Nemo bis: Make the monthly querypages updates not hit each cluster on the same day (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/95889 (owner: 10Nemo bis) [23:36:20] (03PS1) 10Tim Starling: Set zero load on snapshot hosts [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95957 [23:40:23] (03CR) 10Springle: Make the monthly querypages updates not hit each cluster on the same day (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/95889 (owner: 10Nemo bis) [23:48:04] (03PS1) 10Cmjohnson: fixing netboot.cfg for elastic [operations/puppet] - 10https://gerrit.wikimedia.org/r/95960 [23:49:32] (03CR) 10Cmjohnson: [C: 032] fixing netboot.cfg for elastic [operations/puppet] - 10https://gerrit.wikimedia.org/r/95960 (owner: 10Cmjohnson) [23:51:56] (03PS2) 10Nemo bis: Make the monthly querypages updates not hit each cluster on the same day [operations/puppet] - 10https://gerrit.wikimedia.org/r/95889 [23:56:23] (03PS1) 10Faidon Liambotis: Replace Linux RPS setting with a smart mechanism [operations/puppet] - 10https://gerrit.wikimedia.org/r/95963 [23:56:25] ...my sunday evening... [23:57:06] :) [23:58:16] (03CR) 10jenkins-bot: [V: 04-1] Replace Linux RPS setting with a smart mechanism [operations/puppet] - 10https://gerrit.wikimedia.org/r/95963 (owner: 10Faidon Liambotis) [23:58:25] bleh