[00:05:35] (03CR) 10Ori.livneh: [C: 031] "There certainly are resources on this domain that are expensive to generate (LTR-flipped CSS, compiled LESS, dynamically-generated JavaScr" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/154234 (owner: 10Tim Starling) [00:10:47] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 01:50:18 UTC [00:28:27] PROBLEM - Disk space on stat1002 is CRITICAL: DISK CRITICAL - free space: /tmp 0 MB (0% inode=99%): [00:31:27] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Epic puppet fail [00:36:27] RECOVERY - Disk space on stat1002 is OK: DISK OK [00:41:47] PROBLEM - Puppet freshness on elastic1007 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 06:23:11 UTC [00:50:27] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [00:54:27] PROBLEM - Disk space on stat1002 is CRITICAL: DISK CRITICAL - free space: /tmp 530 MB (0% inode=99%): [00:56:27] RECOVERY - Disk space on stat1002 is OK: DISK OK [01:41:33] ori: Re User:Caothu9669, what type of spam? 'cause he created another account on Friday. [01:42:36] scfc_de: it was a weird gerrit change, let me find the link [01:42:52] scfc_de: https://gerrit.wikimedia.org/r/157663 [01:43:55] legoktm: k, thanks. [02:11:47] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 01:50:18 UTC [02:12:07] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3616 MB (3% inode=99%): [02:15:33] !log LocalisationUpdate completed (1.24wmf18) at 2014-09-02 02:14:30+00:00 [02:15:48] Logged the message, Master [02:26:39] !log LocalisationUpdate completed (1.24wmf19) at 2014-09-02 02:25:36+00:00 [02:26:47] Logged the message, Master [02:42:47] PROBLEM - Puppet freshness on elastic1007 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 06:23:11 UTC [02:53:37] !log restarted dbstore1002 mysqld for upgrade [02:53:44] Logged the message, Master [03:01:07] RECOVERY - Disk space on virt0 is OK: DISK OK [03:16:28] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Sep 2 03:15:22 UTC 2014 (duration 15m 21s) [03:16:34] Logged the message, Master [03:33:52] (03PS1) 10Springle: depool db1035 for upgrade, move s3 vslow/dump to db1019 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/157779 [03:34:31] (03CR) 10Springle: [C: 032] depool db1035 for upgrade, move s3 vslow/dump to db1019 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/157779 (owner: 10Springle) [03:34:35] (03Merged) 10jenkins-bot: depool db1035 for upgrade, move s3 vslow/dump to db1019 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/157779 (owner: 10Springle) [03:35:34] !log springle Synchronized wmf-config/db-eqiad.php: depool db1035 (duration: 00m 07s) [03:35:40] Logged the message, Master [03:41:47] PROBLEM - mysqld processes on db1035 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [03:41:59] (03PS1) 10Springle: prepare db1035 for upgrade [puppet] - 10https://gerrit.wikimedia.org/r/157780 [03:42:56] (03CR) 10Springle: [C: 032] prepare db1035 for upgrade [puppet] - 10https://gerrit.wikimedia.org/r/157780 (owner: 10Springle) [04:12:16] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 01:50:18 UTC [04:43:16] PROBLEM - Puppet freshness on elastic1007 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 06:23:11 UTC [04:54:03] (03PS1) 10Springle: prepare db1067 for upgrade [puppet] - 10https://gerrit.wikimedia.org/r/157783 [04:55:17] (03CR) 10Springle: [C: 032] prepare db1067 for upgrade [puppet] - 10https://gerrit.wikimedia.org/r/157783 (owner: 10Springle) [04:59:14] !log springle Synchronized wmf-config/db-eqiad.php: depool db1067 (duration: 00m 07s) [04:59:25] Logged the message, Master [05:09:22] (03CR) 10Chmarkine: [C: 031] puppetmaster - use ssl_ciphersuite [puppet] - 10https://gerrit.wikimedia.org/r/153986 (owner: 10Dzahn) [05:37:26] !log springle Synchronized wmf-config/db-eqiad.php: repool db1067, warm up (duration: 00m 08s) [05:37:32] Logged the message, Master [05:39:52] (03PS1) 10KartikMistry: WIP: Update cxserver (beta) config [puppet] - 10https://gerrit.wikimedia.org/r/157787 [05:42:07] (03CR) 10KartikMistry: [C: 04-1] "Dictionaries packages need to update." [puppet] - 10https://gerrit.wikimedia.org/r/157787 (owner: 10KartikMistry) [05:55:13] (03PS1) 10Chmarkine: gerrit: Enable StrictTransportSecurity max-age=7days [puppet] - 10https://gerrit.wikimedia.org/r/157789 (https://bugzilla.wikimedia.org/38516) [05:58:35] !log dump s3 db1035 to db1069:3313 [05:58:41] Logged the message, Master [06:11:06] PROBLEM - Host mw1192 is DOWN: PING CRITICAL - Packet loss = 100% [06:13:06] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 01:50:18 UTC [06:17:56] RECOVERY - Disk space on db1047 is OK: DISK OK [06:21:25] PROBLEM - Puppet freshness on cp4013 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 06:18:34 UTC [06:22:52] !log removed txt files filling up db1047 /tmp, looked like analytics SELECT INTO OUTFILE, dated mid-August [06:22:57] Logged the message, Master [06:23:25] PROBLEM - Puppet freshness on cp4013 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 06:18:34 UTC [06:25:25] PROBLEM - Puppet freshness on cp4013 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 06:18:34 UTC [06:26:58] (03PS2) 10KartikMistry: WIP: Update cxserver (beta) config [puppet] - 10https://gerrit.wikimedia.org/r/157787 [06:27:25] PROBLEM - Puppet freshness on cp4013 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 06:18:34 UTC [06:29:25] PROBLEM - Puppet freshness on cp4013 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 06:18:34 UTC [06:29:35] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 3 failures [06:29:45] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:45] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:46] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:55] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:26] PROBLEM - Disk space on elastic1014 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=96%): [06:30:26] PROBLEM - Disk space on elastic1004 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=96%): [06:31:25] PROBLEM - Puppet freshness on cp4013 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 06:18:34 UTC [06:32:16] hrmmmmmmmmmm [06:32:22] ^d, ping? [06:33:25] PROBLEM - Puppet freshness on cp4013 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 06:18:34 UTC [06:35:25] PROBLEM - Puppet freshness on cp4013 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 06:18:34 UTC [06:35:46] PROBLEM - puppet last run on db1011 is CRITICAL: CRITICAL: Puppet has 2 failures [06:37:25] PROBLEM - Puppet freshness on cp4013 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 06:18:34 UTC [06:38:35] RECOVERY - Puppet freshness on cp4013 is OK: puppet ran at Tue Sep 2 06:38:26 UTC 2014 [06:43:59] PROBLEM - Puppet freshness on elastic1007 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 06:23:11 UTC [06:45:39] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:45:49] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:45:50] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:46:39] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:46:40] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:53:49] RECOVERY - puppet last run on db1011 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:01:29] RECOVERY - Disk space on elastic1004 is OK: DISK OK [07:02:12] !log removed all-but-latest large slow logs on elastic1004 and elastic1014 [07:02:18] Logged the message, Master [07:02:29] RECOVERY - Disk space on elastic1014 is OK: DISK OK [07:05:28] <_joe_> someone already on mw1192? [07:11:03] nope [07:11:25] springle: ouch, are they filling their disks regularly with log files? [07:20:42] <_joe_> !log powercycling mw1192, blank console, unresponsive [07:20:48] Logged the message, Master [07:23:19] RECOVERY - Host mw1192 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms [07:24:06] (03PS3) 10KartikMistry: WIP: Update cxserver (beta) config [puppet] - 10https://gerrit.wikimedia.org/r/157787 [07:24:42] so, backend fail, huh [07:24:47] unrelated to the xfs lockup? [07:24:57] (and load avg) [07:25:22] jeremyb: no I don't think so [07:25:44] and much less widespread than what we've seen previously it seems [07:26:18] well last time we had logs mentioning memcache? i wonder about this time [07:26:38] (I also wonder if my memory is accurate :P) [07:26:54] (03CR) 10Nikerabbit: "Is apertium going to be puppetized in separate patch?" [puppet] - 10https://gerrit.wikimedia.org/r/157787 (owner: 10KartikMistry) [07:28:17] (03PS4) 10KartikMistry: WIP: Update cxserver (beta) config [puppet] - 10https://gerrit.wikimedia.org/r/157787 [07:29:22] (03CR) 10KartikMistry: "No need of apertium on beta (as of now) as we've http://apertium.wmflabs.org" [puppet] - 10https://gerrit.wikimedia.org/r/157787 (owner: 10KartikMistry) [07:30:15] jeremyb: yep that was the case originally [07:30:56] <_joe_> mw1192 has a load average of 64 [07:31:35] <_joe_> I second godog's idea of reducing the weight of those api servers to 15 [07:33:35] _joe_, post reboot? [07:34:14] <_joe_> jeremyb: yes, it's at full CPU [07:34:30] <_joe_> (now gone down to 27, still....) [07:34:30] huh [07:34:43] _joe_: yep we could try, memory usage doesn't seem so bad on the rest [07:34:44] <_joe_> well post-reboot you need to rebuild the apc cache [07:34:54] <_joe_> godog: that was exactly my thought [07:35:48] <_joe_> well, lemme repackage hhvm with another patch we need to backport [07:36:19] <_joe_> godog: if no-one answers, I think we can move on later in the day [07:36:35] <_joe_> (uniforming the weight of the appservers a bit) [07:36:45] _joe_: sounds good, worst case rollback is easy [07:36:52] <_joe_> yep [07:37:06] <_joe_> worst case we cause our first outage :) [07:37:26] <_joe_> you know people have placed bets on the date for that? [07:38:08] hahah no I wasn't aware, we should be allowed to bet *g* [07:38:47] (03CR) 10Nikerabbit: "All dependencies should sooner or later be puppetized and run on beta labs. Preferable sooner, as that allows ops to highlight possible bl" [puppet] - 10https://gerrit.wikimedia.org/r/157787 (owner: 10KartikMistry) [07:40:36] what do you get if you have an outage before you get shell/root? [07:41:24] a bigger prize possibly, that's slight harder to do [07:46:31] (03CR) 10Giuseppe Lavagetto: puppet: hiera backend for the WMF (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/151869 (owner: 10Giuseppe Lavagetto) [07:47:14] https://gerrit.wikimedia.org/r/106109 "broke the world", https://gerrit.wikimedia.org/r/133205 was the fix. :-) [07:47:42] (03CR) 10Giuseppe Lavagetto: puppet: hiera backend for the WMF (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/151869 (owner: 10Giuseppe Lavagetto) [07:48:53] <_joe_> jeremyb: eheh apache is quite scary [07:49:13] heh [07:49:22] <_joe_> it's very very easy to fuck things up [07:49:38] <_joe_> I'd like to find the time to write unit tests for our apache config [08:06:44] (03PS1) 10Giuseppe Lavagetto: Add pcre overflow patch that should prevent beta from crashing regularly [debs/hhvm] - 10https://gerrit.wikimedia.org/r/157790 [08:13:59] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 01:50:18 UTC [08:15:55] godog: looks like. ottomata logged the same apparent full disk problem for elastic* yesterday [08:17:54] <_joe_> jeremyb: btw, thanks a lot for the help on RT last week [08:18:11] which? [08:18:39] * jeremyb also gets confused with all the new italian/greek names :P [08:19:43] (i have a funny interpretation of "new") [08:19:55] indeed [08:20:26] <_joe_> jeremyb: Giuseppe [08:20:37] right, /whois told me that :) [08:20:41] unless new >= 1.2 years (at least for the greek guys) [08:20:48] <_joe_> Giuseppe == Joseph [08:21:48] <_joe_> I was kinda convinced that was widely known and thus my nick was not confusing anyone [08:22:12] <_joe_> that turned out to be false both for Italian and English native speakers [08:24:19] _joe_, oh, was last week your week? [08:24:30] calendar on https://wikitech.wikimedia.org/wiki/RT_Triage_Duty has lapsed [08:25:36] <_joe_> uhh too many places to put that in [08:25:44] <_joe_> will amend that sorry [08:26:02] <_joe_> as an excuse, this was my second time on duty only [08:26:07] we need to update that too ? [08:26:16] here's something I have never done when on RT [08:26:24] idk. it should say that it's not updated if it's not updated [08:26:34] what other places are there? [08:27:02] (originally that was the page where people would sign up for slots weeks ahead) [08:27:07] this channel's topic [08:27:13] right [08:27:33] that's it? [08:27:34] and the techops meeting's minutes/notes (which are not public...) [08:27:45] <_joe_> uh no, I was sure I put that in that wikitech page [08:28:00] <_joe_> I may have not saved it [08:28:04] <_joe_> :/ [08:28:23] <_joe_> I thought I edited a page on another wiki [08:29:24] there's officewiki... [08:35:09] PROBLEM - check_mysql on lutetium is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 774 [08:40:09] RECOVERY - check_mysql on lutetium is OK: Uptime: 1171608 Threads: 2 Questions: 3401470 Slow queries: 2028 Opens: 5187 Flush tables: 2 Open tables: 64 Queries per second avg: 2.903 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [08:41:09] PROBLEM - puppet last run on neon is CRITICAL: CRITICAL: Epic puppet fail [08:41:40] springle: gah, okay thanks [08:44:59] PROBLEM - Puppet freshness on elastic1007 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 06:23:11 UTC [09:01:09] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [09:11:22] (03CR) 10Alexandros Kosiaris: [C: 04-1] WIP: Update cxserver (beta) config (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/157787 (owner: 10KartikMistry) [09:15:49] RECOVERY - OCG health on ocg1003 is OK: OK: /mnt/tmpfs 0B: /srv/deployment/ocg/output 4279847341B: /srv/deployment/ocg/postmortem 1402390B: ocg_job_status 9746 msg: ocg_render_job_queue 0 msg [09:23:51] (03PS1) 10Filippo Giunchedi: releases: do not include hostname in sudoers.d [puppet] - 10https://gerrit.wikimedia.org/r/157797 [09:25:22] (03PS2) 10Filippo Giunchedi: releases: do not include hostname in sudoers.d [puppet] - 10https://gerrit.wikimedia.org/r/157797 [09:40:24] (03PS1) 10Giuseppe Lavagetto: Fix startup of hhvm [debs/hhvm] - 10https://gerrit.wikimedia.org/r/157798 [09:45:59] (03CR) 10Giuseppe Lavagetto: "This is in no way important for our installation, but is part of the needed packaging work." [debs/hhvm] - 10https://gerrit.wikimedia.org/r/157798 (owner: 10Giuseppe Lavagetto) [09:46:30] PROBLEM - ElasticSearch health check for shards on logstash1001 is CRITICAL: CRITICAL - elasticsearch http://10.64.32.138:9200/_cluster/health error while fetching: Request timed out. [09:47:29] RECOVERY - ElasticSearch health check for shards on logstash1001 is OK: OK - elasticsearch status production-logstash-eqiad: {ustatus: uyellow, unumber_of_nodes: 3, uunassigned_shards: 1, utimed_out: False, uactive_primary_shards: 36, ucluster_name: uproduction-logstash-eqiad, urelocating_shards: 0, uactive_shards: 102, uinitializing_shards: 0, unumber_of_data_nodes: 3} [10:00:27] <_joe_> !log depooling mw1192, high CPU temperatures; we may need to check fan status [10:00:32] Logged the message, Master [10:01:15] (03PS5) 10KartikMistry: WIP: Update cxserver (beta) config [puppet] - 10https://gerrit.wikimedia.org/r/157787 [10:03:39] PROBLEM - Disk space on stat1002 is CRITICAL: DISK CRITICAL - free space: /tmp 3744 MB (3% inode=99%): [10:04:50] (03CR) 10KartikMistry: WIP: Update cxserver (beta) config (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/157787 (owner: 10KartikMistry) [10:06:39] RECOVERY - Disk space on stat1002 is OK: DISK OK [10:12:19] (03CR) 10Alexandros Kosiaris: [C: 032] pdns: qualify vars [puppet] - 10https://gerrit.wikimedia.org/r/157681 (owner: 10Matanya) [10:14:59] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 01:50:18 UTC [10:21:23] PROBLEM - Puppet freshness on mw1083 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 10:18:12 UTC [10:23:23] PROBLEM - Puppet freshness on mw1083 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 10:18:12 UTC [10:25:23] PROBLEM - Puppet freshness on mw1083 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 10:18:12 UTC [10:27:23] PROBLEM - Puppet freshness on mw1083 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 10:18:12 UTC [10:29:23] PROBLEM - Puppet freshness on mw1083 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 10:18:12 UTC [10:29:39] godog: question: isn't jenkins uploading to caesium ? [10:31:23] PROBLEM - Puppet freshness on mw1083 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 10:18:12 UTC [10:31:39] matanya: for debian packages? no, not sure about the rest tho [10:31:43] PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: Puppet has 1 failures [10:32:11] godog: it might be an answer to your question in #8251 [10:32:24] if it is desired, not sure about the setup [10:33:23] PROBLEM - Puppet freshness on mw1083 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 10:18:12 UTC [10:33:26] matanya: indeed! [10:35:23] PROBLEM - Puppet freshness on mw1083 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 10:18:12 UTC [10:37:23] PROBLEM - Puppet freshness on mw1083 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 10:18:12 UTC [10:38:24] RECOVERY - Puppet freshness on mw1083 is OK: puppet ran at Tue Sep 2 10:38:13 UTC 2014 [10:46:00] PROBLEM - Puppet freshness on elastic1007 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 06:23:11 UTC [10:49:55] RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [10:50:10] PROBLEM - puppet last run on mc1014 is CRITICAL: CRITICAL: Puppet has 1 failures [10:57:44] (03PS1) 10Filippo Giunchedi: elasticsearch: handle request timeout and increase timeout [puppet] - 10https://gerrit.wikimedia.org/r/157805 [10:58:24] (03CR) 10jenkins-bot: [V: 04-1] elasticsearch: handle request timeout and increase timeout [puppet] - 10https://gerrit.wikimedia.org/r/157805 (owner: 10Filippo Giunchedi) [10:59:53] (03PS2) 10Filippo Giunchedi: elasticsearch: handle request timeout and increase timeout [puppet] - 10https://gerrit.wikimedia.org/r/157805 [11:02:42] looks like we have consensus on weighting api servers to 15, I can do that in an hour or so if nobody wants to do it earlier [11:09:10] RECOVERY - puppet last run on mc1014 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [11:24:20] PROBLEM - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 994.007393736 [12:01:50] ^d: andre__: hi, can you share an estimate of the number of subscribers of wikitech-l with me? (it says you're the list admins) [12:02:40] (or is it super duper private data?) [12:03:04] I wonder if this is displayed anywhere in Mailman [12:16:00] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 01:50:18 UTC [12:36:27] !log increase weight to 15 for mw1132 -> mw1148 [12:36:33] Logged the message, Master [12:40:58] <_joe_> godog: looking at the load average [12:41:07] <_joe_> it doesn't seem to explode [12:42:43] (03PS3) 10Filippo Giunchedi: elasticsearch: handle request timeout and increase timeout [puppet] - 10https://gerrit.wikimedia.org/r/157805 [12:43:02] yep seems "fine", in which they are fairly loaded anyways [12:43:30] PROBLEM - puppet last run on pc1001 is CRITICAL: CRITICAL: Puppet has 1 failures [12:45:47] if it seems okay we can bring everything to 15 in the next one/two hours [12:47:00] PROBLEM - Puppet freshness on elastic1007 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 06:23:11 UTC [12:48:36] <_joe_> godog: +1 [12:51:20] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: Puppet has 1 failures [13:01:30] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [13:09:20] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:12:50] PROBLEM - puppet last run on cp4018 is CRITICAL: CRITICAL: Puppet has 1 failures [13:19:12] (03PS5) 10Giuseppe Lavagetto: varnish: add a php engine token [puppet] - 10https://gerrit.wikimedia.org/r/156793 [13:29:31] (03CR) 10Giuseppe Lavagetto: "cherry-picked on beta, seems to work fine." [puppet] - 10https://gerrit.wikimedia.org/r/156793 (owner: 10Giuseppe Lavagetto) [13:29:50] RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [13:37:15] (03CR) 10Alexandros Kosiaris: [C: 031] elasticsearch: handle request timeout and increase timeout (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/157805 (owner: 10Filippo Giunchedi) [13:38:40] PROBLEM - Disk space on elastic1009 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 20129 MB (3% inode=99%): [13:44:15] (03PS6) 10Giuseppe Lavagetto: varnish: add a php engine token [puppet] - 10https://gerrit.wikimedia.org/r/156793 [13:52:47] (03CR) 10Giuseppe Lavagetto: [C: 032] varnish: add a php engine token [puppet] - 10https://gerrit.wikimedia.org/r/156793 (owner: 10Giuseppe Lavagetto) [13:57:58] (03PS1) 10coren: Labs: provide saner nscd defaults [puppet] - 10https://gerrit.wikimedia.org/r/157816 (https://bugzilla.wikimedia.org/70076) [13:59:42] _joe_: ^^ look at this? First attempt to lighten the load on dnsmasq by caching instead. [14:00:30] PROBLEM - Host nickel is DOWN: PING CRITICAL - Packet loss = 100% [14:00:41] <_joe_> nickel down? [14:01:16] _joe_: Looks like. [14:02:31] <_joe_> it is in fact [14:02:40] * Coren takes alook. [14:02:45] <_joe_> Coren: thanks [14:03:01] <_joe_> godog: your change on the balancing of the api servers seem to work [14:03:13] Eeew. Looks like the kernel panicked. [14:03:40] PROBLEM - Disk space on elastic1009 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 20176 MB (3% inode=99%): [14:04:05] _joe_: yeah but I was expecting more of a drop in the ones at weight 20 :( will balance the rest to 15 now [14:04:25] <_joe_> godog: but there was [14:10:19] (03PS5) 10Giuseppe Lavagetto: mediawiki: funnel all php requests through HHVM [puppet] - 10https://gerrit.wikimedia.org/r/157691 [14:10:21] <_joe_> Coren: I guess you may powercycle nickel [14:10:40] PROBLEM - Disk space on elastic1009 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 20073 MB (3% inode=99%): [14:10:50] RECOVERY - Host nickel is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [14:12:22] _joe_: Dying disk(s) [14:13:09] * Coren files RT [14:13:31] <_joe_> oh shit [14:13:32] Do we normally shut a box with dying disks down to avoid making things worse or do we keep it up? [14:13:52] <_joe_> it's the ganglia frontend [14:13:57] Poop. [14:14:16] sda is dying. lots of media errors and out of remappable cylinders. [14:14:48] <_joe_> so... we need to replace it somehow [14:14:56] isn't it raid? [14:15:10] mark: It is. Right now, it's doing the right thing. [14:15:14] Sep 2 14:14:38 nickel kernel: [ 309.649807] raid1: sda1: redirecting sector 51478016 to another mirror [14:15:30] doesn't sound like the right thing to me [14:15:32] mark: But it just returned from a kernel panic, so I expect that sbd is also ill. [14:15:35] remove sda from the mirror [14:16:14] (03CR) 10Andrew Bogott: Labs: provide saner nscd defaults (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/157816 (https://bugzilla.wikimedia.org/70076) (owner: 10coren) [14:16:14] it's trying to resync at the moment [14:16:39] mark: we crossed wires; I've set it to failed. [14:16:40] PROBLEM - Disk space on elastic1009 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 20134 MB (3% inode=99%): [14:16:45] ok [14:17:00] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 01:50:18 UTC [14:17:26] (03PS6) 10Giuseppe Lavagetto: mediawiki: funnel all php requests through HHVM [puppet] - 10https://gerrit.wikimedia.org/r/157691 [14:17:31] mark: So I open RT for disk swap. We keep it up on one disk until then? [14:17:41] yes [14:17:45] kk [14:17:50] no point trying to keep using a disk that's known faulty [14:17:50] PROBLEM - Disk space on nickel is CRITICAL: DISK CRITICAL - free space: /mnt/ganglia_tmp 0 MB (0% inode=80%): [14:17:59] <_joe_> mh [14:18:12] <_joe_> that ^^ probably needs fixing [14:18:25] Hm. Disk full is unrelated, but probably while the last few bits of bad disk were used. [14:18:33] * Coren fixes that too. [14:18:51] PROBLEM - RAID on nickel is CRITICAL: CRITICAL: Active: 1, Working: 1, Failed: 1, Spare: 0 [14:19:40] PROBLEM - Disk space on elastic1009 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 20109 MB (3% inode=99%): [14:20:24] (03CR) 10Andrew Bogott: [C: 031] Labs: provide saner nscd defaults (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/157816 (https://bugzilla.wikimedia.org/70076) (owner: 10coren) [14:20:50] PROBLEM - puppet last run on nickel is CRITICAL: CRITICAL: Puppet has 1 failures [14:21:01] /mnt/ganglia_tmp is... tmpfs. That's _bad_. [14:21:15] Well, it's not bad that it's tmpfs, it's bad that it's full. [14:21:46] (03PS7) 10Giuseppe Lavagetto: mediawiki: funnel all php requests through HHVM [puppet] - 10https://gerrit.wikimedia.org/r/157691 [14:28:43] Reedy: if you look here, you can see JohnLewis implying that those require_once statements are unneeded because of the the extension list. Which is it? https://gerrit.wikimedia.org/r/#/c/155789/16/wmf-config/wikitech.php [14:29:22] Coren: might be due to codfw being added [14:29:53] andrewbogottL while https://gerrit.wikimedia.org/r/#/c/155789/20/wmf-config/extension-list-wikitech exists; they are useless. That's the base of it. [14:30:12] mark: I think it is; we'll need to tweak the RRAs to reduce the resolution. [14:30:57] JohnLewis: I don't understand, what do you mean by 'they are useless'? [14:31:12] mark: The kernel won't let tmpfs use over half of physical mem IIRC, and we're at that limit. [14:31:14] JohnLewis: andrewbogott extension-list is only used for localisation update [14:31:19] what is wikitech.php used for? [14:31:37] ok. So, JohnLewis, you now agree with Reedy that we need those include statements? [14:31:42] eh? well I lost the whole point of extension lists there. [14:31:55] no, let's not tweak the RRDs [14:31:59] let's solve the problem [14:32:01] andrewbogott: from what aude said; yeah. [14:32:05] JohnLewis: for example, localisation cache might be run on aawikibooks [14:32:06] aude: right now wikitech is maintained by hand, I'm trying to get it on the deployment train. [14:32:15] extension list installs all of wikibase on wikibooks [14:32:27] for purpose of building i18n cache [14:32:34] (and all the things) [14:32:36] <_joe_> Coren: it's strange it's full now, and it wasn't before [14:32:47] <_joe_> maybe some hiccup with cleanup in that dir? [14:33:31] codfw boxes? [14:33:34] those create new RRDs [14:33:52] _joe_: At first glance, I see nothing in there that's insane. There's a lot of pmtpa stuff that might no longer be relevant. [14:34:21] mark: yeah, as new boxen come alive in codfw, the usage will increase (obviously) and we might have just hit the limit. [14:34:31] it's not the first time ;) [14:35:04] marktraceur, ^d: So which of us wants to SWAT this morning? [14:35:10] <_joe_> my advice would be; see which files there are not opened by a program, and maybe delete those? [14:35:30] anomie: I wouldn't mind taking my first swing at it, if it's a smallish one [14:35:45] James_F|Away (also marktraceur and ^d): 149597 doesn't seem appropriate for SWAT; the guidelines say "No new features/extensions" [14:36:01] And I see no exceptions for "But it's only beta labs" [14:36:26] Shouldn't beta labs get automatically synced anyway? [14:36:32] marktraceur: Except for the one I just mentioned, it seems small at first glance. [14:36:41] We'd just need to merge the patch and sync the no-op change. [14:36:52] anomie: it's not new, it was renamed/split [14:37:20] MatmaRex: Still, it's adding a new extension to the deployment. [14:37:32] right. just saying. [14:38:49] isn't there an extension-list-labs ? [14:38:50] RECOVERY - puppet last run on nickel is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [14:40:04] (03CR) 10Aude: SpecialCite is now CiteThisPage (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/149597 (owner: 10Jforrester) [14:40:20] Anyway, yeah, I'll do it [14:40:28] _joe_: Not helpful; basically none of it is opened at any one time (ganglia clearly only opens as needed). Ima check for files untouched in a long time. [14:40:44] <_joe_> on a tmpfs? [14:40:45] <_joe_> :) [14:40:59] ctime should be okay. [14:41:01] the tmpfs is mirrored to on-disk [14:41:12] and from on-disk [14:41:12] so YMMV [14:41:19] depending on what rsync options used etc :) [14:41:36] (03CR) 10Aude: SpecialCite is now CiteThisPage (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/149597 (owner: 10Jforrester) [14:42:24] (03CR) 10Andrew Bogott: [C: 031] "ok, I'm now convinced that this is the right way to handle extensions." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/155789 (owner: 10Andrew Bogott) [14:42:40] PROBLEM - Disk space on elastic1009 is CRITICAL: DISK CRITICAL - free space: /var/lib/elasticsearch 20138 MB (3% inode=99%): [14:42:51] mark: think it's okay if I blow away pmtpa data untouched in 2 years? [14:43:32] (03CR) 10Andrew Bogott: [C: 031] Add temporary virt1000.wikimedia.org.erb for wikitech migration to multiversion [puppet] - 10https://gerrit.wikimedia.org/r/157704 (owner: 10Reedy) [14:43:39] (03PS3) 10Andrew Bogott: Add temporary virt1000.wikimedia.org.erb for wikitech migration to multiversion [puppet] - 10https://gerrit.wikimedia.org/r/157704 (owner: 10Reedy) [14:44:33] Coren: yes [14:44:34] mark: That should recover some 200M of space. Not a /lot/, but should help in the short term. [14:46:45] anomie: Protocol question, should I make a point of doing a separate sync for each change or is it acceptable to scap all four? [14:47:31] (03CR) 10Andrew Bogott: [C: 031] Add temporary virt1000.wikimedia.org.erb for wikitech migration to multiversion [puppet] - 10https://gerrit.wikimedia.org/r/157704 (owner: 10Reedy) [14:47:57] " depending on what rsync options used etc :)" That may actually be why the space blew up. There may be more on-disk that fits in ram and while someone else might have cleaned up the tmpfs before rebooting copied it all back. [14:48:00] PROBLEM - Puppet freshness on elastic1007 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 06:23:11 UTC [14:48:14] yes [14:48:15] be careful [14:48:19] it's a sucky setup really [14:48:26] perhaps we can do without tmpfs on ssd [14:48:26] marktraceur: I always scap (actually sync-file or sync-dir) each individual change, and then make the responsible person test to make sure it didn't break anything. In some cases where one person has a bunch of related patches doing all-at-once could be ok, but that's the exception rather than the rule. [14:48:34] KK. [14:48:35] although rrd on ssd is still not great [14:49:01] mark: I should go clean up the on-disk copy then. [14:49:07] anomie: So like, MatmaRex's patches that are a revert on both branches could be one scap. [14:49:11] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [14:49:23] But James_F's patches for different config changes, not so much [14:49:50] Er, I guess I could just sync-file the two MatmaRex patches, but in theory. [14:50:08] marktraceur: Could be, yes, but especially on Monday and Tuesday I prefer to do the one for phase0 wikis first to avoid breaking the sites end users care about, Just In Case�. [14:50:33] Sure [14:51:54] I don't know that I've ever had one where someone said "oh crap, this didn't actually work, revert!" when testing it after the patch was merged, but I don't want to get complacent. ;) [14:52:24] (03CR) 10Alexandros Kosiaris: [C: 04-1] mediawiki: funnel all php requests through HHVM (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/157691 (owner: 10Giuseppe Lavagetto) [14:52:26] Indeed [14:52:29] my swat only affects wikisource, by the way [14:52:33] mark: Other possible watermarks (mtime) for cleanup: unwritten to in 120 days cleans ~1.3G; 200 days cleans ~900M, 300 days cleans ~600M [14:52:36] so one of the branches might not be necessary [14:52:42] but i never know which wikis are running what when [14:53:03] mark: those are all files whose mtime are more than the specified number of days away so in theory no new data came in in that interval. [14:53:08] MatmaRex: https://www.mediawiki.org/wiki/MediaWiki_1.24/Roadmap. Wikisources will be on wmf19 until this afternoon. [14:53:20] And then 20, so both are necessary [14:53:24] <^d> wikiversions.json is the only way to know for sure :p [14:53:32] Oh, it's an 18 patch. Are we running 18 anywhere? [14:53:48] ... I was wrong. s/wmf19/wmf18/ [14:53:51] Coren: yeah decommissioned boxes and such [14:54:00] (and then 19 this afternoon, not 20) [14:54:05] marktraceur: ^ [14:54:05] yeah, somone needs to make a timeline/calendar [14:54:13] Oh, yeah, so both need to happen [14:54:35] MatmaRex: You can also just hit Special:Version on whatever wiki you're wondering about, of course [14:54:36] MatmaRex: That would be [[Deployments]] :) [14:54:44] mark: that's what I assumed. 300 days seems like a nice point to cut off. 365 days only cleans up 100M or so - I expect much of pmtpa was decom between 300 and 365 days ago? [14:55:11] (That sounds about right no me) [14:55:29] marktraceur: as i proved yesterday, [[deployments]] is also confusing [14:55:39] * marktraceur fetches coffee before swatting [14:57:12] Coren: yeah, for instance the squid cleanups I did when I came back from vacation in december [14:57:29] or perhaps that falls just outside of it [14:57:35] mark: Allright, blowing up data older than 300 days. [14:57:41] k [14:58:14] MatmaRex: I'll do yours first, James_F|Away is still |Away, that tardy sumbitch [14:58:45] Step 1. Sign into gerrit [14:58:50] RECOVERY - Disk space on nickel is OK: DISK OK [15:00:02] ^^ is okay now; but we'll run into the same problem eventually as codfw ramps up. An easy (if ugly) fix would be to add moar ramz to nickel. [15:00:05] manybubbles, anomie, ^d, marktraceur: Respected human, time to deploy SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140902T1500). Please do the needful. [15:00:17] Coren: akosiaris has a request for a new box in [15:00:21] OK, I'm obviously taking it, MatmaRex is first. Waiting for Jenkins to merge. [15:00:25] so that should be handled [15:00:30] * Coren nods. [15:00:38] Indeed. [15:01:09] Coren: https://rt.wikimedia.org/Ticket/Display.html?id=7921 for reference [15:05:21] Syncing the wmf18 change [15:05:22] !log marktraceur Synchronized php-1.24wmf18/includes/EditPage.php: [SCAP] Revert "Toolbar: Only show on WikiText pages" (duration: 00m 08s) [15:05:28] Logged the message, Master [15:05:29] MatmaRex: Testy test? :) [15:05:44] James_F|Away: You're up next, better get to the office [15:06:03] marktraceur: testy testy, verified fixed [15:06:04] thanks [15:06:09] Sweet, syncing 19 [15:06:11] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [15:06:18] !log marktraceur Synchronized php-1.24wmf19/includes/EditPage.php: [SCAP] Revert "Toolbar: Only show on WikiText pages" (duration: 00m 07s) [15:06:24] Logged the message, Master [15:07:21] I'd ask you to test again, but I don't think it would help [15:07:30] * marktraceur waits for James_F [15:08:58] (03CR) 10coren: Labs: provide saner nscd defaults (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/157816 (https://bugzilla.wikimedia.org/70076) (owner: 10coren) [15:09:03] (03CR) 10BryanDavis: "Tried to answer question about extension-list-wikitech inline." (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/155789 (owner: 10Andrew Bogott) [15:09:29] (03CR) 10coren: [C: 032] Labs: provide saner nscd defaults [puppet] - 10https://gerrit.wikimedia.org/r/157816 (https://bugzilla.wikimedia.org/70076) (owner: 10coren) [15:10:04] Oh, is James gracing us with his presence now? :D [15:10:20] marktraceur: hey, sorry. [15:10:34] No problem, we still have plenty of time [15:10:38] * marktraceur starts merging wantonly [15:11:23] (03CR) 10Andrew Bogott: "Yep, Aude explained on IRC as well. Thanks! Is this patch now deserving of a +1?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/155789 (owner: 10Andrew Bogott) [15:11:25] (03CR) 10MarkTraceur: [C: 032] Enable TemplateData GUI on Norwegian Bokmål (nowiki) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/157476 (https://bugzilla.wikimedia.org/70216) (owner: 10Jforrester) [15:11:41] (03Merged) 10jenkins-bot: Enable TemplateData GUI on Norwegian Bokmål (nowiki) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/157476 (https://bugzilla.wikimedia.org/70216) (owner: 10Jforrester) [15:11:59] (03CR) 10John F. Lewis: [C: 031] "Restore my looks good as my stupidity was correctly :)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/155789 (owner: 10Andrew Bogott) [15:12:26] <_joe_> akosiaris: thanks for the review, but I was about to abort that patch alltoghether. I realized things are more simple and straight forward if I just define two puppet functions there [15:14:01] _joe_: cool [15:14:34] !log marktraceur updated /a/common to {{Gerrit|Ia1758b21e}}: depool db1035 for upgrade, move s3 vslow/dump to db1019 [15:14:42] Logged the message, Master [15:14:56] Hm [15:14:59] <_joe_> btw, yesterday I made some pretty neat benchmarking of various apache proxy methods; I should post those somewhere [15:15:00] That created a merge commit. [15:15:09] Oops. [15:15:17] Not ff only? [15:15:28] I guess it was [15:15:36] I should have rebased first [15:15:39] No big deal [15:15:56] marktraceur: Forgot to rebase! ;) [15:16:41] I have one more with which to redeem myself [15:16:54] Hah [15:16:59] :-) [15:16:59] !log marktraceur Synchronized wmf-config/InitialiseSettings.php: [SCAP] Enable the TemplateData GUI editor on Norwegian Wikipedia (duration: 00m 07s) [15:17:03] Encoding error because my message included non-ASCiI [15:17:06] Logged the message, Master [15:17:07] bd808: ^^ [15:17:21] James_F: Test? [15:17:30] marktraceur: Works great. [15:17:38] Sweet. [15:17:45] Yay for PHP config which is instant. [15:17:53] marktraceur: https://wikitech.wikimedia.org/w/index.php?title=Module:Deployment_schedule&diff=125469&oldid=118682 [15:18:03] No waiting five minutes for bits. [15:18:10] <^d> James_F: We could've gone with hphpc :p [15:18:15] marktraceur: That weird logging error is weird. I've never been able to reproduce on a test scap setup. [15:18:16] <^d> Then we would've had to wait an hour or more! [15:18:42] MatmaRex: You're my favourite [15:18:49] ^d: :-) [15:19:03] that should make it harder for me to miss which day the deployment is scheduled on [15:19:22] (03CR) 10BryanDavis: "One small change needed to move $wgExtensionEntryPointListFiles addition into the body of CommonSettings.php. Other than that I think it l" (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/155789 (owner: 10Andrew Bogott) [15:19:29] (03CR) 10MarkTraceur: [C: 032] SpecialCite is now CiteThisPage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/149597 (owner: 10Jforrester) [15:19:35] (03Merged) 10jenkins-bot: SpecialCite is now CiteThisPage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/149597 (owner: 10Jforrester) [15:19:45] Argh. [15:20:06]