[00:02:45] wikipedia-commons-local-temp: 5350932.5477619 MiB [00:02:49] paravoid: it's mocking me :) [00:04:22] (03PS1) 10Aude: Move Wikibase settings to own file [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94069 [00:05:12] (03PS2) 10Aude: Move Wikibase settings to own file [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94069 [00:08:01] paravoid: so getFileStat() is giving them an mtime of false if I use the lastmod from the listings... [00:08:57] wfTimestamp( TS_UNIX, false ) of course gives the current time so it thinks almost nothing is expired [00:11:49] work fine locally with ceph [00:20:05] (03PS1) 10Dzahn: bugzilla module - WIP [operations/puppet] - 10https://gerrit.wikimedia.org/r/94075 [00:21:26] (03CR) 10jenkins-bot: [V: 04-1] bugzilla module - WIP [operations/puppet] - 10https://gerrit.wikimedia.org/r/94075 (owner: 10Dzahn) [00:24:20] (03PS2) 10Dzahn: bugzilla module - WIP [operations/puppet] - 10https://gerrit.wikimedia.org/r/94075 [00:26:01] paravoid: ahh, swift changed the date format [00:26:17] that also explains why the copy script was wasting so much time I bet [00:47:40] ugh, that's almost but not quite TS_ISO_8601 [00:49:24] apergos (when you are around again): ganglia works great again, thanks! [01:06:55] PROBLEM - Puppet freshness on analytics1010 is CRITICAL: No successful Puppet run in the last 10 hours [01:08:20] ergh i have been getting intermittent timeouts from gerrit for the last hour or so [01:10:04] MaxSem: fyi unit tests fail for me in https://gerrit.wikimedia.org/r/#/c/88903/3 [01:10:32] 1) CoordTest::testEquals with data set #7 (Coord, NULL, false, 'Comparison with null') [01:10:32] urgh [01:10:32] Argument 2 passed to CoordTest::testEquals() must be an instance of Coord, null given [01:11:55] PROBLEM - Puppet freshness on analytics1014 is CRITICAL: No successful Puppet run in the last 10 hours [01:16:55] PROBLEM - Puppet freshness on analytics1009 is CRITICAL: No successful Puppet run in the last 10 hours [01:19:55] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: No successful Puppet run in the last 10 hours [01:20:55] PROBLEM - Puppet freshness on testsearch1001 is CRITICAL: No successful Puppet run in the last 10 hours [01:22:55] PROBLEM - Puppet freshness on analytics1017 is CRITICAL: No successful Puppet run in the last 10 hours [01:23:55] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: No successful Puppet run in the last 10 hours [01:27:55] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: No successful Puppet run in the last 10 hours [01:28:55] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: No successful Puppet run in the last 10 hours [01:28:55] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: No successful Puppet run in the last 10 hours [01:33:08] (03PS1) 10Ori.livneh: Enable ResourceLoader module storage in beta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94084 [01:33:55] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: No successful Puppet run in the last 10 hours [01:33:57] (03CR) 10Ori.livneh: [C: 032] Enable ResourceLoader module storage in beta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94084 (owner: 10Ori.livneh) [01:36:55] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: No successful Puppet run in the last 10 hours [01:38:28] !log ori synchronized wmf-config/InitialiseSettings.php 'I064c1b202: enable ResourceLoader module storage on mediawikiwiki' [01:38:49] Logged the message, Master [01:40:55] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: No successful Puppet run in the last 10 hours [01:42:25] !log ori synchronized php-1.23wmf2/resources/startup.js 'touch' [01:42:41] Logged the message, Master [01:42:55] PROBLEM - Puppet freshness on testsearch1003 is CRITICAL: No successful Puppet run in the last 10 hours [01:49:55] PROBLEM - Puppet freshness on stat1002 is CRITICAL: No successful Puppet run in the last 10 hours [01:51:55] PROBLEM - Puppet freshness on analytics1011 is CRITICAL: No successful Puppet run in the last 10 hours [01:51:55] PROBLEM - Puppet freshness on testsearch1002 is CRITICAL: No successful Puppet run in the last 10 hours [01:54:55] PROBLEM - Puppet freshness on analytics1018 is CRITICAL: No successful Puppet run in the last 10 hours [02:15:28] !log LocalisationUpdate completed (1.23wmf2) at Thu Nov 7 02:15:27 UTC 2013 [02:15:43] Logged the message, Master [02:17:49] (03CR) 10Krinkle: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94084 (owner: 10Ori.livneh) [02:23:35] !log LocalisationUpdate completed (1.23wmf1) at Thu Nov 7 02:23:35 UTC 2013 [02:23:51] Logged the message, Master [02:57:02] The authenticity of host 'tin (10.64.0.196)' can't be established. [02:57:04] * Aaron|home hmms [03:00:38] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Nov 7 03:00:37 UTC 2013 [03:00:59] Logged the message, Master [03:07:00] !log aaron synchronized php-1.23wmf1/includes 'eaa917adad163c50eaa90378db99ed1d0cb13048' [03:07:15] Logged the message, Master [03:12:45] !log aaron synchronized php-1.23wmf2/includes 'eaa917adad163c50eaa90378db99ed1d0cb13048' [03:13:02] Logged the message, Master [05:11:50] (03CR) 10Ori.livneh: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94084 (owner: 10Ori.livneh) [05:55:08] (03PS1) 10ArielGlenn: hadoop install needs java [operations/puppet] - 10https://gerrit.wikimedia.org/r/94104 [05:57:07] (03CR) 10ArielGlenn: [C: 032] hadoop install needs java [operations/puppet] - 10https://gerrit.wikimedia.org/r/94104 (owner: 10ArielGlenn) [05:57:12] hey apergos [05:57:18] morning [05:57:37] something might be broken with ULSFO bits [05:57:56] oh? [05:58:04] some requests are taking an unreasonably long time [05:58:16] that might be very much a mark or paravoid thing [05:58:18] time to first byte is always quick, but some requests are bursty [05:58:30] with gaps of no data [05:58:34] taking a very long time to complete [05:58:37] ugh [05:58:48] RECOVERY - Puppet freshness on analytics1009 is OK: puppet ran at Thu Nov 7 05:58:38 UTC 2013 [05:59:14] a number of users are reporting it on VPT on enwiki, so far all are hitting ULSFO [05:59:40] scrollback in #wikimedia-tech probably useful [06:03:33] i think we might want to revert mark's changes [06:03:57] ori-l: you mean config-geo changes? [06:05:15] PROBLEM - RAID on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:05:24] 2f8b05af658f4cc858ece3a6b32f7a924f47ea58 this is the last thing to add to ulsfo [06:05:31] (dns repo) [06:06:05] RECOVERY - Puppet freshness on analytics1010 is OK: puppet ran at Thu Nov 7 06:05:57 UTC 2013 [06:06:10] canada, might not be so much, immediately before is california, oregon and washington with [06:06:12] 177113bf7bf2174c592c7d25e1df12168a7d686d [06:06:15] RECOVERY - RAID on snapshot3 is OK: OK: no RAID installed [06:06:28] (ignore snapshot3 whines please) [06:07:15] PROBLEM - DPKG on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:07:34] one of the reports on #-tech was oregon [06:07:42] what about the others? [06:08:03] i'm in australia [06:08:15] RECOVERY - DPKG on snapshot3 is OK: All packages OK [06:09:44] how does your traffic even go there [06:10:04] it should be going via esams according to the file [06:10:48] beats me! [06:10:57] i must be special [06:11:04] apergos: OC? [06:11:05] RECOVERY - Puppet freshness on analytics1014 is OK: puppet ran at Thu Nov 7 06:10:55 UTC 2013 [06:11:10] let me look again [06:11:20] yeah oc, ok [06:11:55] i can reproduce it pretty reliably now with this URL: 'https://bits.wikimedia.org/en.wikipedia.org/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki%2CSpinner%7Cjquery.triggerQueueCallback%2CloadingSpinner%2CmwEmbedUtil%7Cmw.MwEmbedSupport&only=scripts&skin=vector&version=20131106T025802Z' [06:12:43] ori-l: here's one localization dude :) [06:12:59] hey [06:13:00] morning [06:13:02] what's up [06:13:36] bits is always fast to first byte but some response bodies come staggered, with very long pauses in between [06:14:09] reports from oregon, australia, california (me) [06:14:30] some trace routes and resp headers in #-tech, all point to ULSFO [06:14:31] looking [06:15:20] just bits? [06:15:26] yes [06:15:39] ori-l: uls was enwiki (api) [06:15:44] not bits [06:16:03] but who knows if that's a red herring [06:16:16] yes, true [06:16:33] also, good morning :) [06:16:59] yes, good morning! [06:19:05] RECOVERY - Puppet freshness on analytics1019 is OK: puppet ran at Thu Nov 7 06:18:58 UTC 2013 [06:22:25] RECOVERY - Puppet freshness on analytics1017 is OK: puppet ran at Thu Nov 7 06:22:21 UTC 2013 [06:23:05] RECOVERY - Puppet freshness on analytics1020 is OK: puppet ran at Thu Nov 7 06:23:03 UTC 2013 [06:25:16] both http and https [06:26:28] (03PS1) 10ArielGlenn: fix up java package for statistics [operations/puppet] - 10https://gerrit.wikimedia.org/r/94105 [06:27:00] (03CR) 10jenkins-bot: [V: 04-1] fix up java package for statistics [operations/puppet] - 10https://gerrit.wikimedia.org/r/94105 (owner: 10ArielGlenn) [06:27:35] RECOVERY - Puppet freshness on analytics1016 is OK: puppet ran at Thu Nov 7 06:27:25 UTC 2013 [06:28:05] RECOVERY - Puppet freshness on analytics1026 is OK: puppet ran at Thu Nov 7 06:27:56 UTC 2013 [06:28:15] RECOVERY - Puppet freshness on analytics1013 is OK: puppet ran at Thu Nov 7 06:28:06 UTC 2013 [06:28:21] (03PS2) 10ArielGlenn: fix up java package for statistics [operations/puppet] - 10https://gerrit.wikimedia.org/r/94105 [06:29:04] ori-l: v6 and v4? [06:29:40] (03CR) 10ArielGlenn: [C: 032] fix up java package for statistics [operations/puppet] - 10https://gerrit.wikimedia.org/r/94105 (owner: 10ArielGlenn) [06:31:35] RECOVERY - Puppet freshness on stat1002 is OK: puppet ran at Thu Nov 7 06:31:25 UTC 2013 [06:31:44] ffs, I see nothing wrong [06:32:30] (03PS1) 10Spage: Enable Flow discussions on a few test wiki pages [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94106 [06:33:05] RECOVERY - Puppet freshness on analytics1027 is OK: puppet ran at Thu Nov 7 06:32:57 UTC 2013 [06:35:49] (03CR) 10Spage: [C: 04-1] "Do not +2 until we have an approved deployment window. We hope to deploy Flow in November." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94106 (owner: 10Spage) [06:35:55] RECOVERY - Puppet freshness on analytics1012 is OK: puppet ran at Thu Nov 7 06:35:53 UTC 2013 [06:36:01] ori-l: 07 06:29:04 < jeremyb> ori-l: v6 and v4? [06:38:10] (03PS1) 10ArielGlenn: fix up java package for elastic search [operations/puppet] - 10https://gerrit.wikimedia.org/r/94107 [06:39:11] (03CR) 10ArielGlenn: [C: 032] fix up java package for elastic search [operations/puppet] - 10https://gerrit.wikimedia.org/r/94107 (owner: 10ArielGlenn) [06:39:30] jeremyb: erm ,sec [06:40:05] RECOVERY - Puppet freshness on analytics1015 is OK: puppet ran at Thu Nov 7 06:39:59 UTC 2013 [06:42:15] (03PS1) 10ArielGlenn: and fix up the java dependency for elastic search too, woops [operations/puppet] - 10https://gerrit.wikimedia.org/r/94108 [06:43:17] (03CR) 10ArielGlenn: [C: 032] and fix up the java dependency for elastic search too, woops [operations/puppet] - 10https://gerrit.wikimedia.org/r/94108 (owner: 10ArielGlenn) [06:44:01] (03PS2) 10Spage: Enable Flow discussions on a few test wiki pages [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94106 [06:45:05] RECOVERY - Puppet freshness on testsearch1001 is OK: puppet ran at Thu Nov 7 06:44:56 UTC 2013 [06:46:03] (03CR) 10Spage: [C: 04-1] "PS2 I think has the right $wg - $wmg logic." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94106 (owner: 10Spage) [06:46:39] (03PS1) 10Faidon Liambotis: Switch North America back to eqiad [operations/dns] - 10https://gerrit.wikimedia.org/r/94109 [06:47:00] (03CR) 10Faidon Liambotis: [C: 032] Switch North America back to eqiad [operations/dns] - 10https://gerrit.wikimedia.org/r/94109 (owner: 10Faidon Liambotis) [06:51:05] RECOVERY - Puppet freshness on analytics1011 is OK: puppet ran at Thu Nov 7 06:50:57 UTC 2013 [06:51:55] RECOVERY - Puppet freshness on testsearch1002 is OK: puppet ran at Thu Nov 7 06:51:52 UTC 2013 [06:54:15] RECOVERY - Puppet freshness on analytics1018 is OK: puppet ran at Thu Nov 7 06:54:13 UTC 2013 [07:01:18] back in a little while (errand) [07:05:37] PROBLEM - MySQL Processlist on db1047 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 65 copy to table, 0 statistics [07:32:47] RECOVERY - Puppet freshness on testsearch1003 is OK: puppet ran at Thu Nov 7 07:32:43 UTC 2013 [07:38:58] hrmmm [07:39:12] mchenry is still in pmtpa i guess [07:39:30] RT 1804 [07:41:26] yes, it is [07:47:09] i don't suppose someone has a clue how ipv6 mapped addresses work? [07:47:42] 1) are any in rdns? should they be? [07:47:49] no, no [07:48:04] 2) i read modules/interface/manifests/add_ip6_mapped.pp but can't seem to replicate the pairing i see in the wild [07:48:43] paravoid: any special reason? context is 3645. i now see iodine has the same issue williams once had [07:48:48] RT 3645 that is [07:50:53] apparently 2620:0:861:2:92b1:1cff:fe00:a9d5 is iodine [08:05:44] PROBLEM - MySQL Processlist on db1047 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 65 copy to table, 0 statistics [08:12:55] PROBLEM - RAID on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:13:34] PROBLEM - Disk space on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:14:04] jeremyb: IPv6-mapped IPv4's? [08:14:13] that address is certainly not of that kind [08:15:24] PROBLEM - DPKG on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:16:24] RECOVERY - DPKG on snapshot3 is OK: All packages OK [08:17:54] RECOVERY - RAID on snapshot3 is OK: OK: no RAID installed [08:18:34] RECOVERY - Disk space on snapshot3 is OK: DISK OK [08:19:03] Jasper_Deng: this one fits the pattern: [08:19:04] mchenry.wikimedia.org has address 208.80.152.186 [08:19:04] mchenry.wikimedia.org has IPv6 address 2620:0:860:2:208:80:152:186 [08:19:24] PROBLEM - DPKG on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:19:27] that's not formally an IPv6-mapped address [08:19:41] Jasper_Deng: did you read the puppet manifest i mentioned/ [08:19:43] ? [08:19:56] * Jasper_Deng thinks jeremyb fogot that those digits are not equal to the IPv4 address in numerical value [08:19:57] so maybe for some reason that manifest isn't applied on iodine. or williams [08:20:06] Jasper_Deng: huh? [08:20:13] 07 08:19:41 < jeremyb> Jasper_Deng: did you read the puppet manifest i mentioned/ [08:20:26] * Jasper_Deng doesn't have OTRS access [08:20:42] 208:80:152:86!=208.80.152.186 [08:20:48] if you expand their binary digits [08:20:56] this is more Wikimedia's thing, I think [08:21:21] Jasper_Deng: well sure... but if you have all the parts of the infra follow the same convention... [08:21:35] PROBLEM - Disk space on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:21:40] bits does not [08:21:52] ugh [08:22:34] RECOVERY - Disk space on snapshot3 is OK: DISK OK [08:23:49] bonjour [08:23:54] PROBLEM - RAID on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:24:04] PROBLEM - SSH on snapshot3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:25:34] PROBLEM - Disk space on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:25:54] RECOVERY - RAID on snapshot3 is OK: OK: no RAID installed [08:27:05] RECOVERY - SSH on snapshot3 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [08:27:24] RECOVERY - DPKG on snapshot3 is OK: All packages OK [08:27:24] RECOVERY - Disk space on snapshot3 is OK: DISK OK [08:28:54] PROBLEM - RAID on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:29:54] RECOVERY - RAID on snapshot3 is OK: OK: no RAID installed [08:30:24] PROBLEM - DPKG on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:31:24] RECOVERY - DPKG on snapshot3 is OK: All packages OK [08:34:34] PROBLEM - Disk space on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:36:24] RECOVERY - Disk space on snapshot3 is OK: DISK OK [08:37:53] (03PS1) 10Jeremyb: RT 3645 - give iodine a predictable IPv6 address [operations/puppet] - 10https://gerrit.wikimedia.org/r/94111 [08:38:49] (03CR) 10Jeremyb: "I have no idea if this is right, just copied from sodium's site.pp block." [operations/puppet] - 10https://gerrit.wikimedia.org/r/94111 (owner: 10Jeremyb) [08:40:44] PROBLEM - MySQL Processlist on db1047 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 65 copy to table, 0 statistics [08:41:13] gah, dns mixes tabs and spaces!! [08:43:24] PROBLEM - DPKG on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:44:50] RD: shall we kill otrs-test as a domain name? [08:45:15] RECOVERY - DPKG on snapshot3 is OK: All packages OK [08:45:54] i wonder if rd has a labs acct :P [08:54:54] PROBLEM - RAID on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:55:55] RECOVERY - RAID on snapshot3 is OK: OK: no RAID installed [08:56:37] huh, so the subnet for an ip6 mapped IP is based on subnet [08:56:38] err [08:56:42] how recursive [08:56:51] (03PS1) 10Nikerabbit: Use correct bits for uls fonts on labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94112 [08:56:53] huh, so the subnet for an ip6 mapped IP is based on row [08:57:11] just guessing from reading templates/1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa [09:01:24] PROBLEM - DPKG on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:03:15] RECOVERY - DPKG on snapshot3 is OK: All packages OK [09:04:31] (03CR) 10Petrb: [C: 031] tool labs exec_environ: add python-twitter package [operations/puppet] - 10https://gerrit.wikimedia.org/r/93426 (owner: 10Jeremyb) [09:04:34] PROBLEM - Disk space on snapshot3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:05:24] RECOVERY - Disk space on snapshot3 is OK: DISK OK [09:06:37] PROBLEM - MySQL Processlist on db1047 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 65 copy to table, 0 statistics [09:12:44] (03PS1) 10Jeremyb: rm otrs-test.wm.o (testing is over) [operations/dns] - 10https://gerrit.wikimedia.org/r/94114 [09:14:58] (03PS1) 10Jeremyb: rm tesla (old virt box) [operations/dns] - 10https://gerrit.wikimedia.org/r/94115 [09:16:17] (03CR) 10Jeremyb: "is tesla dead yet?" [operations/dns] - 10https://gerrit.wikimedia.org/r/94115 (owner: 10Jeremyb) [09:25:32] (03CR) 10Hashar: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94112 (owner: 10Nikerabbit) [09:26:15] (03PS2) 10Nikerabbit: Use correct bits for uls fonts on labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94112 [09:32:36] (03PS2) 10Jeremyb: rm tesla (old virt box) [operations/dns] - 10https://gerrit.wikimedia.org/r/94115 [09:37:02] (03CR) 10Hashar: [C: 04-1] "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94112 (owner: 10Nikerabbit) [09:37:17] !log uploaded snappy version 1.0.4-1build1wmf1 to apt.wikimedia.org [09:37:32] Logged the message, Master [09:38:13] (03CR) 10Hashar: "vary static using $wmfVersionNumber . There is a few example in the config file, for example:" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94112 (owner: 10Nikerabbit) [09:39:52] is it really called snappy? [09:40:07] huh [09:40:10] packages.ubuntu.com says saucy [09:40:20] oh, it's a package anme [09:40:21] name [09:40:35] never heard of that before :) [09:40:37] PROBLEM - MySQL Processlist on db1047 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 65 copy to table, 0 statistics [09:40:45] a compression library [09:40:53] really fast one too [09:41:11] akosiaris: same as http://packages.ubuntu.com/precise/snappy ? [09:41:16] i guess not [09:41:25] nope [09:41:43] http://packages.ubuntu.com/precise/libsnappy1 [09:41:58] kind of confusing huh ? [09:42:14] source package called snappy, binary package called libsnappy1 ... meh [09:42:52] (03CR) 10Matthias Mullie: "I assume Parsoid is working for these wikis? If so, we should save content in HTML (to not have to re-parse wikitext to html for every req" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94106 (owner: 10Spage) [09:42:58] it was missing JNI bindings and recompiled it with them added and enabled so that kafka could happily compress/uncompress stuff [09:43:25] oh, really? you can have a binary package with the same name as a source package if they're different sources packages? [09:43:33] seems like a recipe for disaster [09:44:03] (03CR) 10Matthias Mullie: "And not sure how this one ($wgFlowDefaultWikiDb) works exactly, but the comments in Flow.php suggest that we'll want all wiki's data in 1 " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94106 (owner: 10Spage) [09:44:14] source package namespace is different the binary package namespace [09:44:32] each package needs to be unique in its namespace [09:44:50] so no ... not a recipe for disaster [09:45:28] akosiaris: source package being named differently than binaries is very common [09:45:36] in libraries, it's the rule [09:45:39] exactly [09:45:57] still... it always confuses me for a second [09:46:19] i made the error of going to snappy player's page at least 2-3 times [09:46:21] paravoid: and how often do you then have the reverse too? another source package with a binary that's the same name as the first source? [09:46:38] akosiaris: right, *that* confusion is why it's such a recipe :) [09:46:55] i got a source package named 'voluptuous' providing the binary package 'python-voluptuous' [09:46:56] :/ [09:47:51] meh... not really. You cant really err that bad there. So no destruction. Only going huh??? for a second [09:48:29] you are going to recompile the package and not test for example ? [09:49:22] Reedy: https://bugzilla.wikimedia.org/show_bug.cgi?id=54985 [09:51:38] reedy is sleeping :D [09:52:15] he will see it when he get up :) [09:52:20] *s [09:52:36] when i'll be sleeping most likely [09:53:05] it's only noon now. why would you be asleep? [09:53:14] :P [09:55:15] (03PS3) 10Nikerabbit: Use correct bits for uls fonts on labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94112 [09:58:59] (03CR) 10Hashar: [C: 032] Use correct bits for uls fonts on labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94112 (owner: 10Nikerabbit) [09:59:22] (03Merged) 10jenkins-bot: Use correct bits for uls fonts on labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94112 (owner: 10Nikerabbit) [10:00:37] !log hashar synchronized wmf-config/CommonSettings.php 'Use correct bits for uls fonts on labs {{gerrit|94112}}' [10:00:52] Logged the message, Master [10:01:01] !log hashar synchronized wmf-config/InitialiseSettings.php 'touch' [10:01:16] Logged the message, Master [10:03:37] PROBLEM - MySQL Processlist on db1047 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 65 copy to table, 0 statistics [10:12:06] PROBLEM - Host mw27 is DOWN: PING CRITICAL - Packet loss = 100% [10:12:36] RECOVERY - Host mw27 is UP: PING OK - Packet loss = 0%, RTA = 35.49 ms [10:14:39] (03PS1) 10Jeremyb: rdns for bits-lb, mobile-lb @ ulsfo [operations/dns] - 10https://gerrit.wikimedia.org/r/94120 [10:16:26] (03CR) 10Mark Bergsma: [C: 04-2] "IPs are being changed at the moment. reverse DNS already reflects the new IPs." [operations/dns] - 10https://gerrit.wikimedia.org/r/94120 (owner: 10Jeremyb) [10:16:50] ok :) [10:17:18] (03Abandoned) 10Jeremyb: rdns for bits-lb, mobile-lb @ ulsfo [operations/dns] - 10https://gerrit.wikimedia.org/r/94120 (owner: 10Jeremyb) [10:19:38] mark: any comment on https://gerrit.wikimedia.org/r/94111 ? [10:20:41] * aude wonders what timezone jeremyb is in now :) [10:20:55] aude: uga :) [10:21:02] heh [10:34:08] bunnyland TZ? [10:40:46] PROBLEM - MySQL Processlist on db1047 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 65 copy to table, 0 statistics [10:42:14] !log gallium / jenkins : running git gc --agressive on Zuul git repositories under /srv/ssd/zuul/git [10:42:25] Logged the message, Master [10:42:46] PROBLEM - MySQL Processlist on db1047 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 65 copy to table, 0 statistics [10:43:01] (03PS1) 10Mark Bergsma: Send non-wiki[pm]edia esams text traffic to Varnish [operations/dns] - 10https://gerrit.wikimedia.org/r/94122 [10:47:46] PROBLEM - MySQL Processlist on db1047 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 65 copy to table, 0 statistics [10:50:18] (03CR) 10Mark Bergsma: [C: 032] Send non-wiki[pm]edia esams text traffic to Varnish [operations/dns] - 10https://gerrit.wikimedia.org/r/94122 (owner: 10Mark Bergsma) [10:54:24] (03CR) 10Akosiaris: [C: 032] Check for administratively disabled puppet [operations/puppet] - 10https://gerrit.wikimedia.org/r/93965 (owner: 10Akosiaris) [11:03:46] PROBLEM - MySQL Processlist on db1047 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 65 copy to table, 0 statistics [11:07:57] PROBLEM - puppet disabled on db9 is CRITICAL: NRPE: Command check_puppet_disabled not defined [11:08:33] :-D [11:08:58] grrr why not defined ? [11:10:58] RECOVERY - puppet disabled on db9 is OK: OK [11:11:05] (03PS1) 10Mark Bergsma: Send esams wikimedia-lb traffic to Varnish [operations/dns] - 10https://gerrit.wikimedia.org/r/94123 [11:12:03] (03CR) 10Mark Bergsma: [C: 032] Send esams wikimedia-lb traffic to Varnish [operations/dns] - 10https://gerrit.wikimedia.org/r/94123 (owner: 10Mark Bergsma) [11:18:13] !log gallium on swap death thanks to git eating all memory :/ [11:18:28] Logged the message, Master [11:24:19] git pack-objects is the hog [11:25:46] yup [11:25:47] killed it [11:25:52] waiting for the job to complete [11:26:07] the max git window size is up to 8GB but gallium only has 4GB of spare mem :/ [11:26:24] that's gonna be an issue [11:26:36] killed it again [11:26:50] yeah would have to look at giving out a smaller window size [11:27:19] !log gallium : killed git process [11:27:38] Logged the message, Master [11:28:08] core.packedGitLimit [11:28:08] :( [11:30:27] logged https://bugzilla.wikimedia.org/show_bug.cgi?id=56717 [11:30:30] will fix that later on [11:30:36] ok [11:30:47] it is out of swap now. [11:31:23] yep see that [11:31:58] I am out again, focusing on the dojo [11:40:47] PROBLEM - MySQL Processlist on db1047 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 66 copy to table, 0 statistics [12:03:47] PROBLEM - MySQL Processlist on db1047 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 65 copy to table, 0 statistics [12:44:44] PROBLEM - Host palladium is DOWN: PING CRITICAL - Packet loss = 100% [12:45:21] (03PS1) 10Mark Bergsma: Send esams wikipedia traffic to Varnish [operations/dns] - 10https://gerrit.wikimedia.org/r/94126 [12:45:35] palladium being down is me [12:45:37] no worries [12:47:15] ok [12:47:24] RECOVERY - Host palladium is UP: PING OK - Packet loss = 0%, RTA = 0.20 ms [12:47:46] do you see anyone worrying? [12:47:59] nope [12:48:05] hint: it's not just for palladium :) [12:48:17] top 5 site down [12:48:18] ts ts ts... [12:48:19] who cares :) [12:48:53] we ? [12:48:59] (03CR) 10Mark Bergsma: [C: 032] Send esams wikipedia traffic to Varnish [operations/dns] - 10https://gerrit.wikimedia.org/r/94126 (owner: 10Mark Bergsma) [12:50:04] PROBLEM - Host palladium is DOWN: PING CRITICAL - Packet loss = 100% [12:51:40] * apergos whistles [12:56:14] RECOVERY - Host palladium is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [12:57:23] so... palladium. [12:57:27] OK: no RAID installed [12:57:48] but it seems to have two SEAGATE ST91000640SS in RAID1 configuration [12:57:55] md? [12:58:26] SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08) [12:58:47] is it hardware raid or software raid? [12:58:54] just rebooted an went into bios [12:59:28] that is were i saw the two disk and the LSI Corp Config Utility For Dell SAS 6 and the Dell VIRTUAL DISK (lol) [12:59:33] so hardware [12:59:44] lsictl ? [13:00:43] mptsas, fun [13:00:45] didn't know we had these [13:00:55] lots [13:02:03] root@palladium:~# mpt-status [13:02:03] ioc0 vol_id 0 type IM, 2 phy, 930 GB, state OPTIMAL, flags ENABLED [13:02:08] mpt-status [13:02:08] ioc0 vol_id 0 type IM, 2 phy, 930 GB, state OPTIMAL, flags ENABLED [13:02:08] ioc0 phy 0 scsi_id 9 SEAGATE ST91000640SS AS02, 931 GB, state ONLINE, flags NONE [13:02:08] ioc0 phy 1 scsi_id 1 SEAGATE ST91000640SS AS03, 931 GB, state ONLINE, flags NONE [13:02:16] a yes [13:02:21] dude, how do you think mpt-status got there just know? :) [13:02:24] now [13:02:28] ahahaha [13:02:35] I also modprobe'd mptctl fwiw [13:02:43] and i said... hey puppet had it already installed yey!!! [13:02:48] so, are you gonna fix check-raid.py? [13:02:50] :-( [13:02:54] seems like it [13:02:55] I can, sounds trivial [13:03:21] it has a weird check at the beginning to find which utility to call.... [13:03:44] plus the Package resource, plus find a way to put mptctl to /etc/modules, I don't think we have a definition for that [13:04:15] seems like a plan [13:05:15] hah [13:05:16] mpt-status --autoload -s [13:05:22] autoload loads the kernel module [13:05:23] even easier [13:05:42] make sure to add a Service def to disable the stupid service too [13:06:49] that is the one that sends spam all the time the moment a drive breaks right ? [13:07:00] man i hated that thing [13:42:48] PROBLEM - MySQL Processlist on db1045 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 70 copy to table, 2 statistics [13:57:48] RECOVERY - MySQL Processlist on db1045 is OK: OK 0 unauthenticated, 0 locked, 1 copy to table, 19 statistics [14:10:33] PROBLEM - MySQL Slave Running on db74 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: Error Deadlock found when trying to get lock: try restarting transac [14:15:33] RECOVERY - MySQL Slave Running on db74 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [14:16:16] (03CR) 10Daniel Kinzler: "@ariel: i replied to your comment. In short:" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/92925 (owner: 10Dzahn) [14:16:26] (03CR) 10Daniel Kinzler: [C: 031] make Special:EntityData redirects for wikidata 303 ..instead of 302 [operations/apache-config] - 10https://gerrit.wikimedia.org/r/92925 (owner: 10Dzahn) [14:18:02] (03CR) 10Thehelpfulone: [C: 031] (bug 56398) Update logo for wikimania2014wiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92882 (owner: 10Odder) [14:18:20] Reedy, ^ can you +2 that please? [14:18:23] PROBLEM - MySQL Processlist on db1045 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 72 copy to table, 3 statistics [14:29:23] PROBLEM - MySQL Processlist on db1045 is CRITICAL: CRIT 1 unauthenticated, 0 locked, 66 copy to table, 1 statistics [14:36:23] RECOVERY - MySQL Processlist on db1045 is OK: OK 1 unauthenticated, 0 locked, 1 copy to table, 2 statistics [14:44:40] (03PS2) 10Hashar: zuul: dependencies for Gearman based version [operations/puppet] - 10https://gerrit.wikimedia.org/r/93454 [14:45:48] (03Abandoned) 10Hashar: sartorize contint scripts for jenkins slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/91493 (owner: 10Hashar) [14:46:23] (03CR) 10Hashar: ""Whatever the name of our deployment system is" does not let us sync minions in both prod and labs." [operations/puppet] - 10https://gerrit.wikimedia.org/r/91493 (owner: 10Hashar) [14:48:26] (03Abandoned) 10Hashar: gbp: point upstream to testing/3.0.3plus-rc1 branch [operations/debs/varnish] - 10https://gerrit.wikimedia.org/r/90718 (owner: 10Hashar) [15:15:58] (03PS1) 10Akosiaris: Adding mptsas support in check-raids [operations/puppet] - 10https://gerrit.wikimedia.org/r/94134 [15:17:22] (03CR) 10jenkins-bot: [V: 04-1] Adding mptsas support in check-raids [operations/puppet] - 10https://gerrit.wikimedia.org/r/94134 (owner: 10Akosiaris) [15:17:24] (03PS3) 10Cmjohnson: decommision db3[29] db4[2-6] db5[1235689] [operations/puppet] - 10https://gerrit.wikimedia.org/r/93052 (owner: 10Springle) [15:17:31]