[00:00:05] !log reedy synchronized php-1.21wmf7/maintenance/mcc.php [00:00:18] Logged the message, Master [00:02:17] Reedy should rename himself to Synchronizer [00:04:36] So, wait, why'd the Apaches go ape-shit? [00:05:06] post-mortem forthcoming [00:05:29] Susan: [00:05:30] The last time that search was failed over from eqiad to pmtpa, the search_prefix pool was not failed over, so some small amount of traffic was still going to eqiad. I started java upgrading in the "inactive" datacenter, eqiad. Apparently, this needs some more attention (haven't done this bit yet), and all queries to the search_prefix vip hung, which caused search opensearch queries to hang, which caused the api to go down. Ain't no f [00:07:58] RobH: I'm sat in a room at 11.7C/53F.. And you're cold? :p [00:08:09] ? [00:08:09] RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds [00:08:13] its cold outside. [00:08:18] Reedy: cold aisle? [00:08:24] notpeter: "Ain't no f ..." # You got cut off. [00:08:34] Ain't no failure like a cascading failure, because a cascading failure don't stop... [00:08:37] "Office" [00:08:41] Reedy, no heating? [00:08:42] Ain't no failure like a cascading failure, because a cascading failure don't stop...\ [00:08:42] Susan: ^ [00:08:47] :D [00:08:59] it was 8.4C earlier [00:09:04] inside [00:09:42] Interesting. I didn't think search load was high enough to take anything down. [00:10:06] Susan, it's a prefix search [00:10:12] Ah. [00:10:21] normal search was fine, prefix was misrouted to eqiad. [00:10:22] I do love how the biggest threat to Wikimedia is almost always Wikimedia, though. [00:10:28] when you type something in the search bar, prefix search gets called several times [00:10:32] Right, right. [00:11:22] Susan: I know some fool gave me root on this thing ;) [00:11:27] PROBLEM - Puppet freshness on mw40 is CRITICAL: Puppet has not run in the last 10 hours [00:11:59] Reedy: your review queue is ludicrous :) [00:12:27] Most are unread [00:15:00] hehe [00:15:31] well, there's both the us messing with ourselves and then the "this would have worked if there were 0 external users using the site" [00:15:44] so to be safe … we must revoke all of our root access and unplug the servers from the outside world [00:15:54] hahaha [00:16:07] progress is a lie. we must protect the data at all costs [00:16:37] exactly! [00:19:42] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 289 seconds [00:21:30] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 17 seconds [00:22:14] We should get people to send back all the images they've borrowed [00:22:17] We need them [00:37:31] !log reloading unresponsive ms1002 [00:37:42] Logged the message, Mistress of the network gear. [00:38:41] !log powercycling ms1002 [00:38:50] Logged the message, Master [00:41:13] PROBLEM - Host ms1002 is DOWN: PING CRITICAL - Packet loss = 100% [00:41:50] udevd-work[443]: open /dev/null failed: No such file or directory [00:42:07] just a timing thing.i suppose [00:42:20] /dev/null not existing yet when udevd wants it [00:42:27] ms1002 back at login: [00:42:36] yay [00:42:40] let's see if puppet works [00:42:52] it's actually starting, which is better than it's ever done [00:43:01] RECOVERY - Host ms1002 is UP: PING OK - Packet loss = 0%, RTA = 26.51 ms [00:43:04] we can also install package upgrades while on it [00:43:07] or? [00:43:19] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Thu Jan 17 00:43:05 UTC 2013 [00:43:22] not because we want to not change it to find out the issue.. shrug [00:43:23] yay [00:43:27] sounds good [00:43:38] you want to get that ? [00:44:08] !log installing package upgrades on ms1002 [00:44:17] Logged the message, Master [00:44:21] hey, among them are also salt packages [00:45:29] LeslieCarr: hey [00:45:34] you were looking for me? [00:45:43] she _just_ logged off [00:45:51] but i can still relay a message [00:46:09] she pinged me a few hours ago [00:46:10] I was out [00:46:22] she cant remember:) [00:46:37] so probably not that urgent [00:46:43] thanks :) [00:46:47] sure [00:47:16] paravoid: do you know about ms1002 ? [00:47:37] it uses misc::images::rsyncd and misc::images::rsync [00:47:43] ariel would know [00:47:45] I don't for sure [00:48:16] alright, because i was wondering if we can remove that whole class [00:48:23] let me add Ariel to a gerrit change then [00:48:31] heh, already on it:)ok [00:49:53] Ryan_Lane: when installing package upgrades and getting newer salt packages, i am being asked if i want to keep current or install maintainer version for /etc/salt/minion [00:50:11] mutante: always use the default [00:50:30] New patchset: Reedy; "Make mctest wikiless" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44368 [00:50:32] ok, that is keeping the current one [00:50:37] yep [00:50:51] sockpuppet in it. looks good [00:51:08] * Ryan_Lane nods [00:51:15] New patchset: Reedy; "Make mctest and mcc wikiless" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44368 [00:51:42] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44368 [01:01:31] New patchset: Asher; "ruby + string concat doesn't work in erb without to_s casting, so nevermind that" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44369 [01:02:13] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44369 [01:10:42] !log DNS update to remove some useless stuff from esams zone (RT-4331) [01:12:21] Logged the message, Master [01:15:43] RECOVERY - mysqld processes on db1001 is OK: PROCS OK: 1 process with command name mysqld [01:15:52] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [01:15:52] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [01:20:31] PROBLEM - MySQL Slave Running on db1001 is CRITICAL: CRIT replication Slave_IO_Running: No Slave_SQL_Running: No Last_Error: Rollback done for prepared transaction because its XID was not in the [01:21:25] PROBLEM - MySQL Replication Heartbeat on db1001 is CRITICAL: CRIT replication delay 68609 seconds [01:26:58] RECOVERY - MySQL Replication Heartbeat on db1001 is OK: OK replication delay seconds [01:27:34] RECOVERY - MySQL Slave Running on db1001 is OK: OK replication [01:32:22] PROBLEM - MySQL Replication Heartbeat on db1001 is CRITICAL: CRIT replication delay 68785 seconds [01:32:55] PROBLEM - MySQL Slave Delay on db1001 is CRITICAL: CRIT replication delay 68789 seconds [01:52:52] New patchset: Asher; "issues with remote secondary master expansion and account pws" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44372 [01:53:20] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44372 [02:05:01] New patchset: Asher; "adding mha config for the datacenter failover case make configs 0400" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44374 [02:11:11] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44374 [02:26:58] !log LocalisationUpdate completed (1.21wmf7) at Thu Jan 17 02:26:58 UTC 2013 [02:27:10] Logged the message, Master [02:38:37] PROBLEM - Puppet freshness on db1047 is CRITICAL: Puppet has not run in the last 10 hours [02:38:37] PROBLEM - Puppet freshness on ms-fe1004 is CRITICAL: Puppet has not run in the last 10 hours [02:38:37] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [02:38:37] PROBLEM - Puppet freshness on ms-fe1003 is CRITICAL: Puppet has not run in the last 10 hours [02:45:28] !log LocalisationUpdate completed (1.21wmf8) at Thu Jan 17 02:45:26 UTC 2013 [02:45:39] Logged the message, Master [02:54:39] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: Puppet has not run in the last 10 hours [03:21:45] New patchset: Lwelling; "Add PHP Pear Mail_mime class to Apache manifest" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44377 [03:49:19] !log stopped replication on db67.pmtpa (enwiki research slave), re-replicating off the pmtpa master instead of cross datacenter. original config isn't well supported by mha. [03:49:29] Logged the message, Master [03:51:16] PROBLEM - Varnish HTCP daemon on cp1043 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 1001 (varnishhtcpd), args varnishhtcpd worker [03:52:42] PROBLEM - MySQL Replication Heartbeat on db67 is CRITICAL: CRIT replication delay 341 seconds [03:56:36] !log resumed replication on db67 [03:56:46] Logged the message, Master [03:58:06] RECOVERY - MySQL Replication Heartbeat on db67 is OK: OK replication delay 0 seconds [04:04:51] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:08:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.066 seconds [04:08:39] On what PHP version runs our wikis? [04:09:13] -our wikis +our wikis infrastructure [04:09:16] Dereckson: 5.3.10; we support 5.3+ [04:09:49] Dereckson: isn't that on [[special:version]] ? [04:09:55] * jeremyb runs off to sleep [04:10:09] Thank you. [04:11:33] np! and yes, i think it's also on special:version [04:16:06] RECOVERY - Varnish HTCP daemon on cp1043 is OK: PROCS OK: 1 process with UID = 1001 (varnishhtcpd), args varnishhtcpd worker [04:17:19] New patchset: Asher; "reducing pt-heartbeat interval to 500ms" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44378 [04:18:13] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44378 [04:52:59] New patchset: Lwelling; "Add PHP Pear Mail_mime class to Apache manifest" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44377 [05:06:35] fyi: Thu Jan 17 5:05:01 UTC 2013 hume foundationwiki Error connecting to db1013.eqiad.wmnet: Can't connect to MySQL server on 'db1013.eqiad.wmnet' (111) [05:06:42] something fundraising related [05:16:42] PROBLEM - Puppet freshness on bast1001 is CRITICAL: Puppet has not run in the last 10 hours [05:27:39] PROBLEM - Puppet freshness on fenari is CRITICAL: Puppet has not run in the last 10 hours [05:29:09] New patchset: Asher; "stashing $::mw_primary under /etc/mha/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44380 [05:29:55] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44380 [05:45:48] RECOVERY - MySQL Slave Delay on db1001 is OK: OK replication delay 0 seconds [05:46:06] RECOVERY - MySQL Replication Heartbeat on db1001 is OK: OK replication delay 0 seconds [06:02:21] New review: ArielGlenn; "Pretty sure this can go. Feel free to: merge it in yourself, or nag me when you are around and I'll ..." [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/36146 [07:26:01] PROBLEM - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:27:40] RECOVERY - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 54939 bytes in 0.616 seconds [07:27:49] PROBLEM - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:30:13] PROBLEM - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:31:25] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 9.32704181159 (gt 8.0) [07:31:52] RECOVERY - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is OK: HTTP OK HTTP/1.1 200 OK - 51838 bytes in 0.747 seconds [07:31:52] PROBLEM - LVS HTTPS IPv4 on wiktionary-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:33:04] RECOVERY - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK HTTP/1.1 200 OK - 47607 bytes in 0.763 seconds [07:33:31] RECOVERY - LVS HTTPS IPv4 on wiktionary-lb.esams.wikimedia.org is OK: HTTP OK HTTP/1.1 200 OK - 66326 bytes in 0.881 seconds [07:35:39] er? [07:37:10] apergos: http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&s=by+name&c=LVS%2520loadbalancers%2520esams&tab=m&vn= hrm [07:37:48] network eh? [07:38:15] looks like it to me [07:58:10] RECOVERY - Packetloss_Average on locke is OK: OK: packet_loss_average is 3.44014175182 [08:36:16] PROBLEM - Puppet freshness on vanadium is CRITICAL: Puppet has not run in the last 10 hours [09:13:46] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 196 seconds [09:14:40] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 212 seconds [09:20:58] RECOVERY - MySQL Replication Heartbeat on db1035 is OK: OK replication delay 0 seconds [09:22:01] RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 0 seconds [09:24:16] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [09:33:06] RECOVERY - Puppet freshness on mw40 is OK: puppet ran at Thu Jan 17 09:32:43 UTC 2013 [09:48:33] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [09:53:31] PROBLEM - Puppet freshness on virt1008 is CRITICAL: Puppet has not run in the last 10 hours [09:55:12] New review: Hashar; "As I said on IRC, wmf-config/InitialiseSettings-labs.php is loaded on labs AFTER InitialiseSettings...." [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/44278 [09:56:30] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [09:58:50] New review: Hashar; "For mobile-labs.php, maybe consider including the mobile.php file, then override settings for labs." [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/44278 [10:38:14] PROBLEM - Puppet freshness on cp1026 is CRITICAL: Puppet has not run in the last 10 hours [10:47:05] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 184 seconds [10:48:26] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 232 seconds [10:52:11] RECOVERY - MySQL Replication Heartbeat on db1035 is OK: OK replication delay 0 seconds [10:52:38] RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 0 seconds [10:55:11] New patchset: Ori.livneh; "(Bug 43879) Update eowikiquote logo" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44390 [11:00:27] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44390 [11:17:14] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [11:17:14] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [11:18:53] PROBLEM - MySQL Replication Heartbeat on db1007 is CRITICAL: CRIT replication delay 182 seconds [11:20:32] PROBLEM - MySQL Slave Delay on db1007 is CRITICAL: CRIT replication delay 228 seconds [11:25:56] RECOVERY - MySQL Slave Delay on db1007 is OK: OK replication delay 0 seconds [11:26:05] RECOVERY - MySQL Replication Heartbeat on db1007 is OK: OK replication delay 0 seconds [11:43:30] !log "mwscriptwikiset cleanupUploadStash.php all.dblist" on hume (kill it if swift overloads) [11:43:40] Logged the message, Master [11:45:49] New patchset: Denny Vrandecic; "Remove wikidata portal (not used anymore)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44400 [11:49:41] paravoid: https://gerrit.wikimedia.org/r/#/c/37968/ [11:49:56] foreachwiki cleanupUploadStash.php works and is shorter too ;) [11:50:12] oh [11:50:48] ok, I'll leave it run for now [11:51:09] and when that first run completes, I'll merge that commit anyway [11:51:18] thanks [11:51:25] I've been doing it manually now and again when I remember [12:03:36] New patchset: Reedy; "RT #2295: Run cleanupUploadStash across all wikis daily" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37968 [12:39:25] PROBLEM - Puppet freshness on db1047 is CRITICAL: Puppet has not run in the last 10 hours [12:39:25] PROBLEM - Puppet freshness on ms-fe1003 is CRITICAL: Puppet has not run in the last 10 hours [12:39:25] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [12:39:25] PROBLEM - Puppet freshness on ms-fe1004 is CRITICAL: Puppet has not run in the last 10 hours [12:45:36] paravoid: good afternoon :-] Are you aware of any tool to convert a python pip package to a Debian package? [12:46:00] paravoid: I thought about copy pasting an existing one and just change the names around :-] [12:46:32] the tool is called Hashar! [12:47:30] yeah I noticed that :-D [12:47:46] ahh I need to learn about uupdate [12:55:28] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: Puppet has not run in the last 10 hours [12:56:43] !log Gerrit: created repository operations/debs/python-voluptuous to package http://pypi.python.org/pypi/voluptuous/ [12:56:47] Logged the message, Master [12:56:51] wich me luck [12:57:13] wish [12:57:15] whatever [13:13:47] hmm [13:13:53] less than 15 minutes to build it [13:14:04] I found out a script that does everything for us [13:14:05] https://wikitech.wikimedia.org/view/Debianize_python_package [13:14:12] (based on fpm ) [13:14:23] but then I don't have debian directory to push in Gerit [13:42:16] New patchset: Denny Vrandecic; "(bug 41847) Drop the www. from the domain wikidata.org" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44406 [13:42:39] fpm? ewww [13:42:49] New patchset: Denny Vrandecic; "(bug 41847) Drop the www. on wikidata.org" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/44407 [13:42:51] %: [13:42:56] dh $@ --with python2 [13:42:59] should be enough :) [13:43:52] hashar: http://wiki.debian.org/Python/Packaging [13:44:33] ahh [13:44:38] weird [13:44:50] I see varnish not honouring If-Range: at all [13:45:20] New patchset: Denny Vrandecic; "(bug 41847) Drop the www. on wikidata.org" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/44407 [13:47:06] where? [13:47:08] everywhere? [13:47:25] as far as I can tell it doesn't inspect the header anywhere in the source [13:47:35] hm that reminds me [13:47:44] which matches with my testing, I did If-Range: requests and it ignored the header [13:48:53] paravoid: yeah doing that right now [13:49:21] is that the opera bugzilla bug? [13:49:24] yes [13:49:32] i can't reproduce the results in the bug at all [13:51:26] and of course upstream does not properly tag its versions grmblblbl [13:54:39] New patchset: Hashar; "initialize release" [operations/debs/python-voluptuous] (master) - https://gerrit.wikimedia.org/r/44408 [13:55:24] mark: paravoid: here is my packaging :-] [14:08:15] New patchset: Hashar; "initialize release" [operations/debs/python-voluptuous] (master) - https://gerrit.wikimedia.org/r/44411 [14:08:42] bah [14:10:07] New patchset: Hashar; "(bug 44061) initialize release" [operations/debs/python-voluptuous] (master) - https://gerrit.wikimedia.org/r/44408 [14:10:19] Change abandoned: Hashar; "dupe of https://gerrit.wikimedia.org/r/#/c/44408/" [operations/debs/python-voluptuous] (master) - https://gerrit.wikimedia.org/r/44411 [14:11:55] New review: Hashar; "our bug is https://bugzilla.wikimedia.org/show_bug.cgi?id=44061" [operations/debs/python-voluptuous] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/44408 [14:17:14] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:18:58] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.038 seconds [14:45:06] New patchset: Mark Bergsma; "Add missing create_container param dstconn" [operations/software] (master) - https://gerrit.wikimedia.org/r/44418 [14:45:06] New patchset: Mark Bergsma; "Remove gets from the stats, not very useful" [operations/software] (master) - https://gerrit.wikimedia.org/r/44419 [14:45:43] Change merged: Mark Bergsma; [operations/software] (master) - https://gerrit.wikimedia.org/r/44418 [14:45:58] Change merged: Mark Bergsma; [operations/software] (master) - https://gerrit.wikimedia.org/r/44419 [14:48:40] paravoid: filling an ITP :-D [14:53:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.481 seconds [15:02:52] New patchset: Hashar; "(bug 44061) initialize release" [operations/debs/python-voluptuous] (master) - https://gerrit.wikimedia.org/r/44408 [15:03:25] !log demon synchronized php-1.21wmf8/extensions/Wikibase/repo/resources/Resources.php 'Deploying Ie22a7213' [15:03:34] Logged the message, Master [15:03:45] New review: Hashar; "Fixed various lintian errors from PS2." [operations/debs/python-voluptuous] (master); V: 0 C: -1; - https://gerrit.wikimedia.org/r/44408 [15:03:49] !log demon synchronized php-1.21wmf8/extensions/Wikibase/repo/resources/wikibase.ui.entityViewInit.js 'Deploying Ie22a7213' [15:03:59] Logged the message, Master [15:06:53] New patchset: Mark Bergsma; "Sync deletes on the source to the destination" [operations/software] (master) - https://gerrit.wikimedia.org/r/44422 [15:13:08] !log reedy synchronized docroot [15:13:18] Logged the message, Master [15:16:22] !log reedy synchronized live-1.5/ [15:16:32] Logged the message, Master [15:17:01] New patchset: Reedy; "Update php symlink" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44423 [15:17:17] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44400 [15:17:50] !log reedy synchronized extract2.php [15:18:00] Logged the message, Master [15:18:07] PROBLEM - Puppet freshness on bast1001 is CRITICAL: Puppet has not run in the last 10 hours [15:28:21] New review: Hashar; "You should be able to listen on port 80 as well :-D" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/44164 [15:29:04] PROBLEM - Puppet freshness on fenari is CRITICAL: Puppet has not run in the last 10 hours [15:30:44] New patchset: Mark Bergsma; "Sync deletes on the source to the destination" [operations/software] (master) - https://gerrit.wikimedia.org/r/44422 [15:32:12] New review: Dereckson; "Shellpolicy issue mitigated." [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/42774 [15:33:19] New patchset: Hashar; "(bug 44061) initialize release" [operations/debs/python-voluptuous] (master) - https://gerrit.wikimedia.org/r/44408 [15:33:26] PROBLEM - Host virt1008 is DOWN: PING CRITICAL - Packet loss = 100% [15:33:49] New review: Hashar; "IPT #698354" [operations/debs/python-voluptuous] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/44408 [15:36:17] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:55] New patchset: Mark Bergsma; "Sync deletes on the source to the destination" [operations/software] (master) - https://gerrit.wikimedia.org/r/44422 [15:42:49] New patchset: Reedy; "(bug 43760) Enable Collection on is.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/42976 [15:43:06] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/42976 [15:44:09] New patchset: Reedy; "(bug 43834) Enable WebFonts on en.wiktionary" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/43351 [15:44:23] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/43351 [15:45:17] RECOVERY - Host virt1008 is UP: PING OK - Packet loss = 0%, RTA = 26.59 ms [15:45:24] New patchset: Reedy; "(bug 29692) Per-wiki namespace aliases shouldn't override (remove) global ones" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/25737 [15:45:26] New patchset: Mark Bergsma; "Sync deletes on the source to the destination" [operations/software] (master) - https://gerrit.wikimedia.org/r/44422 [15:45:37] New patchset: Reedy; "Remove wgUseTagFilter. Same as default and no longer needed" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/43004 [15:45:49] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/43004 [15:46:19] New patchset: Reedy; "(bug 43830) Namespace configuration for de.wiktionary" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/43359 [15:46:24] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/43359 [15:46:43] New patchset: Reedy; "(bug 43851) Allow enwikivoyage bureaucrats to remove sysops" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44276 [15:46:53] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44276 [15:47:27] New review: Reedy; "Apache change: https://gerrit.wikimedia.org/r/#/c/44407/" [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/44406 [15:48:15] New patchset: Reedy; "Update php symlink" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44423 [15:48:21] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44423 [15:49:32] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44091 [15:49:47] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.031 seconds [15:49:55] New patchset: Reedy; "(bug 43769) Close ik.wiktionary and zh-min-nan.wikiquote" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/42978 [15:50:00] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/42978 [15:50:39] New review: Reedy; "Needs rebasing" [operations/mediawiki-config] (master); V: 0 C: -1; - https://gerrit.wikimedia.org/r/42995 [15:50:48] New patchset: Reedy; "(bug 43760) Enable WikiLove on is.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/42915 [15:51:46] !log Created wikilove tables on iswiki [15:51:56] Logged the message, Master [15:52:05] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/42915 [15:52:22] New patchset: Reedy; "Ensure confirmed has the same rights as autoconfirmed." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/43703 [15:52:57] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/43703 [15:53:29] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44085 [15:53:43] New patchset: Reedy; "(bug 43659) Import sources for eo.wikisource" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/42774 [15:53:58] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/42774 [15:55:08] !log reedy synchronized wmf-config/ [15:55:18] Logged the message, Master [15:59:14] PROBLEM - Host virt1008 is DOWN: PING CRITICAL - Packet loss = 100% [15:59:50] RECOVERY - SSH on virt1008 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [15:59:59] RECOVERY - Host virt1008 is UP: PING OK - Packet loss = 0%, RTA = 26.52 ms [16:17:28] New patchset: Reedy; "Bug 44039 - Add a category to $wgArticleFeedbackBlacklistCategories for Portuguese Wikibooks" [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/44428 [16:18:00] Change merged: Reedy; [operations/mediawiki-config] (newdeploy) - https://gerrit.wikimedia.org/r/44428 [16:18:27] Damn it. [16:19:15] New patchset: Reedy; "Bug 44039 - Add a category to $wgArticleFeedbackBlacklistCategories for Portuguese Wikibooks" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44429 [16:19:38] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44429 [16:20:07] !log reedy synchronized wmf-config/InitialiseSettings.php [16:20:18] Logged the message, Master [16:22:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:26:14] RECOVERY - NTP on virt1008 is OK: NTP OK: Offset -0.03484725952 secs [16:33:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.680 seconds [16:35:18] New patchset: Cmjohnson; "adding virt1008 to decom.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44431 [16:35:55] New review: Cmjohnson; "Looks good to me" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/44431 [16:35:55] Change merged: Cmjohnson; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44431 [16:38:47] apergos, around? [16:38:57] yep, what's going on? [16:39:16] apergos, remember you merged https://gerrit.wikimedia.org/r/#/c/43218/ - it doesn't seem to be running [16:39:46] what hosst is that again? [16:40:15] hume. currently running these scripts manually there - see a lot of unindexed data [16:40:17] ah hume [16:40:29] well there was some problem with it that I fixed up [16:40:38] mm or was it another hume cron [16:40:41] lemme look [16:41:40] I see no emails, so that's good [16:42:32] I see em in the crontab over there [16:42:44] weird [16:42:56] yep [16:43:24] oh you know what [16:43:29] PROBLEM - MySQL Replication Heartbeat on db1007 is CRITICAL: CRIT replication delay 250 seconds [16:43:30] I bet .... [16:43:36] foreachwikiindblist /home/wikipedia/common/wikipedia.dblist extensions/GeoData/solrupdate.php [16:43:45] that it doesn't know whre foreachwikiindblist is [16:43:49] and we never hear about it cause [16:44:05] I'm running it right now [16:44:05] PROBLEM - MySQL Replication Heartbeat on db58 is CRITICAL: CRIT replication delay 231 seconds [16:44:07] command => "/usr/local/bin/update-geodata >/dev/null 2>&1" [16:44:14] PROBLEM - MySQL Replication Heartbeat on db56 is CRITICAL: CRIT replication delay 295 seconds [16:44:15] yeah but you're running from cmd line [16:44:18] not out of cron [16:44:24] owcie, path? [16:44:41] PROBLEM - MySQL Replication Heartbeat on db1024 is CRITICAL: CRIT replication delay 322 seconds [16:44:42] try taking the redirs to dev null off the puppet change [16:44:52] and let's see what it complains about [16:44:59] PROBLEM - MySQL Slave Delay on db56 is CRITICAL: CRIT replication delay 341 seconds [16:45:26] PROBLEM - MySQL Replication Heartbeat on db1028 is CRITICAL: CRIT replication delay 367 seconds [16:45:59] mark: i am racking 10 db's today and want to put them in b5 where the netapp is located. Do you have any future plans for that rack or okay to put db's there? [16:46:02] RECOVERY - MySQL Replication Heartbeat on db56 is OK: OK replication delay 0 seconds [16:46:02] RECOVERY - MySQL Replication Heartbeat on db58 is OK: OK replication delay 0 seconds [16:46:27] on it [16:46:29] RECOVERY - MySQL Replication Heartbeat on db1024 is OK: OK replication delay 0 seconds [16:46:47] RECOVERY - MySQL Slave Delay on db56 is OK: OK replication delay 0 seconds [16:47:04] cmjohnson1: no don't do that [16:47:14] RECOVERY - MySQL Replication Heartbeat on db1007 is OK: OK replication delay 0 seconds [16:47:15] that rack is reserved for the netapp [16:47:22] why not put it in row C, isn't there plenty of space there? [16:47:24] k [16:48:22] if there isn't one already, designate one rack to be the DB rack [16:49:02] RECOVERY - MySQL Replication Heartbeat on db1028 is OK: OK replication delay 21 seconds [16:49:40] New patchset: MaxSem; "GeoData scripts debugging" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44432 [16:49:47] apergos, ^^^ [16:50:05] mark: okay...was thinking of filling in all the space but...i will use row c [16:50:08] thx [16:50:34] looking [16:51:28] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44432 [16:51:48] Reedy: was https://gerrit.wikimedia.org/r/#/c/25737/ supposed to be merged? [16:51:57] No [16:52:02] oki [16:52:38] trying to stop it becoming too stale (less manual rebasing) [16:52:42] till it gets reviewed again [16:54:12] updated, now we wait for next cron run [16:58:29] New patchset: Demon; "Enable micro design for itwikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44434 [17:01:33] New patchset: Demon; "Also match "bug: 123"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44436 [17:02:55] New patchset: Demon; "Also match "bug: 123"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44436 [17:03:26] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/44434 [17:06:15] !log demon synchronized wmf-config/InitialiseSettings.php 'micro design for itwikivoyage' [17:06:25] Logged the message, Master [17:09:35] MaxSem: [17:09:37] /usr/local/bin/update-geodata: line 8: foreachwikiindblist: command not found [17:09:38] /usr/local/bin/update-geodata: line 9: foreachwikiindblist: command not found [17:09:43] just sayin... :-) [17:10:07] apergos, thanks - fixing [17:10:25] yw [17:17:47] New patchset: MaxSem; "Use full path in GeoData scripts to fix cron" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44439 [17:18:44] apergos, ^^ [17:19:24] 2>&1 I suppose you want? [17:19:52] or yu want stderr to be emailed to us? [17:19:52] MaxSem: [17:21:20] apergos, yes - if there are errors, they'd better be known. so if possible, I'd like to keep it this way unless it creates noise. well, if it creates noise you can always kick me to fix it:) [17:22:22]