[00:02:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 6.249 second response time [00:02:56] binasher: wouldn't have it any other way \m/ \m/ [00:02:57] package_name => 'mariadb-server-5.5', [00:03:24] pgehres: RoanKattouw: http://pastebin.mozilla.org/2232511 [00:03:41] binasher: I captured an MW backtrace BTW [00:03:55] waitforslaves.log on fluorine [00:04:20] New review: Pyoungmeister; "need to flesh out role::db::labsdb." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53907 [00:06:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:07:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 6.565 second response time [00:07:56] RECOVERY - Puppet freshness on mw1008 is OK: puppet ran at Thu Mar 21 00:07:47 UTC 2013 [00:10:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:40] pgehres: BOOM, I got it to work [00:40:49] .p..vjg......I....def...3MASTER_POS_WAIT('db1041-bin.000603', 604351573, 10)..?.................................. [00:46:32] pgehres: https://gerrit.wikimedia.org/r/55008 [00:48:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [00:52:10] New patchset: Tim Starling; "Add libjpeg-turbo-progs to imagescalers for image rotation" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52707 [00:52:39] New review: Tim Starling; "It's OK to put libjpeg-turbo-progs to imagescalers, just not on API apaches." [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/52707 [00:52:53] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52707 [00:53:05] yes, we had that discussion on the wrong changeset [00:53:11] I mentioned that this changeset isn't it [00:54:23] TimStarling: what about the screwing with your plans regarding memory usage? [00:57:13] paravoid: T-2 days, HURRY! [00:57:24] heh [00:59:36] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [01:04:02] !log catrope synchronized php-1.21wmf11/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c' [01:04:19] !log catrope synchronized php-1.21wmf12/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c' [01:19:26] RECOVERY - Varnish traffic logger on cp1028 is OK: PROCS OK: 3 processes with command name varnishncsa [01:34:45] PROBLEM - MySQL Recent Restart on db1051 is CRITICAL: NRPE: Command check_mysql_recent_restart not defined [01:34:45] PROBLEM - MySQL disk space on db1051 is CRITICAL: NRPE: Command check_disk_6_3 not defined [01:35:06] PROBLEM - MySQL Replication Heartbeat on db1051 is CRITICAL: NRPE: Command check_mysql_slave_heartbeat not defined [01:35:14] PROBLEM - MySQL Slave Delay on db1051 is CRITICAL: NRPE: Command check_mysql_slave_delay not defined [01:35:14] PROBLEM - Full LVS Snapshot on db1051 is CRITICAL: NRPE: Command check_lvs not defined [01:35:33] PROBLEM - mysqld processes on db1051 is CRITICAL: NRPE: Command check_mysqld not defined [01:35:33] PROBLEM - MySQL Slave Running on db1051 is CRITICAL: NRPE: Command check_mysql_slave_running not defined [01:35:33] PROBLEM - MySQL Idle Transactions on db1051 is CRITICAL: NRPE: Command check_mysql_idle_transactions not defined [01:40:33] PROBLEM - Varnish traffic logger on cp1028 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [01:48:34] RECOVERY - Varnish traffic logger on cp1028 is OK: PROCS OK: 3 processes with command name varnishncsa [01:51:17] !log catrope synchronized php-1.21wmf12/includes/LinksUpdate.php 'Debugging hack for move page bug on nlwikimedia' [01:52:42] * Aaron|home eyes RoanKattouw [01:52:52] * RoanKattouw reverts live hack [01:52:59] There appears to be a bug in wmf12 when moving a page [01:53:14] See the backtrace in exception.log for exception 458e1031 [01:53:28] (the title in the exception message is the target title of the move) [01:55:50] OK yeah this is a bug in wmf12 [01:57:12] Reporting, then heading home [01:57:13] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 189 seconds [01:57:43] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 198 seconds [01:58:50] !log catrope synchronized php-1.21wmf12/includes/LinksUpdate.php 'Revert debugging hack' [02:00:14] RoanKattouw: morebots? [02:00:41] sigh [02:03:12] PROBLEM - Puppet freshness on colby is CRITICAL: Puppet has not run in the last 10 hours [02:07:10] PROBLEM - Puppet freshness on cp1026 is CRITICAL: Puppet has not run in the last 10 hours [02:16:58] !log LocalisationUpdate completed (1.21wmf12) at Thu Mar 21 02:16:57 UTC 2013 [02:18:50] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 0 seconds [02:19:20] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [02:19:40] PROBLEM - MySQL Slave Delay on db66 is CRITICAL: CRIT replication delay 221 seconds [02:19:51] PROBLEM - MySQL Replication Heartbeat on db66 is CRITICAL: CRIT replication delay 231 seconds [02:33:49] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 185 seconds [02:37:49] RECOVERY - MySQL Slave Delay on db1047 is OK: OK replication delay 0 seconds [02:38:09] PROBLEM - Puppet freshness on db66 is CRITICAL: Puppet has not run in the last 10 hours [02:41:31] !log LocalisationUpdate completed (1.21wmf11) at Thu Mar 21 02:41:30 UTC 2013 [03:27:26] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [03:27:28] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [03:27:28] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [03:27:28] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [03:41:41] RECOVERY - MySQL Slave Delay on db66 is OK: OK replication delay 0 seconds [03:41:52] RECOVERY - MySQL Replication Heartbeat on db66 is OK: OK replication delay 0 seconds [04:05:32] New patchset: Tim Starling; "Removed favicon.ico files obsoleted by I35d3af43" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55014 [04:07:41] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [04:09:44] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [04:52:13] New patchset: Ryan Lane; "Use mysqldump --single-transaction where possible" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55015 [04:54:36] New patchset: Ryan Lane; "Use mysqldump --single-transaction where possible" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55015 [04:57:51] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55015 [06:03:02] PROBLEM - Puppet freshness on constable is CRITICAL: Puppet has not run in the last 10 hours [06:31:07] New patchset: Tim Starling; "Roll back all wikis to php-1.21wmf11 due to bug 46397" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55016 [06:31:56] !log tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: [06:32:04] Logged the message, Master [06:32:44] Change merged: Tim Starling; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55016 [06:33:57] !log tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: [06:34:03] Logged the message, Master [06:38:10] New review: Mattflaschen; "-1'ing myself. Do not submit until the error (see previous comments) is fixed." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/54970 [06:38:56] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [06:55:07] New review: Ryan Lane; "Cleaning it up manually is fine, since this is a single system. It removes clutter from the repo." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52742 [06:55:29] New patchset: Tim Starling; "Remove search::apple-dictionary-bridge from ekrem" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52742 [06:55:48] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52742 [06:58:28] RECOVERY - MySQL Slave Delay on db67 is OK: OK replication delay NULL seconds [07:00:56] PROBLEM - MySQL Slave Running on db67 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: No Last_Error: Error You have an error in your SQL syntax: check the manual that co [07:07:32] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 183 seconds [07:08:12] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 188 seconds [07:17:13] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 19 seconds [07:17:32] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [08:39:04] PROBLEM - Puppet freshness on arsenic is CRITICAL: Puppet has not run in the last 10 hours [08:39:26] New patchset: Aude; "allow CORS on both www.wikidata.org and wikidata.org" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55024 [08:40:04] PROBLEM - Puppet freshness on xenon is CRITICAL: Puppet has not run in the last 10 hours [08:42:03] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55024 [08:43:36] New review: Aude; "I don't know if this would mean their edits would be treated as logged out. that would be bad and w..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55024 [08:44:12] New review: Aude; "let's see if this helps and it might :)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55024 [09:14:41] !log olivneh synchronized wmf-config/CommonSettings.php 'Add 'wikidata.org' to to allow CORS' [09:14:47] Logged the message, Master [09:15:26] second time this week i let bash variable expansion eat part of my sync message [09:15:29] damn it [09:15:54] anyways aude ^ synced [09:16:11] ori-l: thanks [09:16:14] * aude hopes it helps [09:29:37] !log created git repo operations/debs/python-statsd [09:29:43] Logged the message, Master [09:32:51] New patchset: Hashar; ".gitreview file" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55028 [09:40:48] New patchset: Hashar; "logmsbot is no more in #wikimedia-tech" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55031 [09:41:39] New review: Hashar; "The previous changes are:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55031 [09:45:10] is Nagios officially dead and buried? [09:46:35] apergos: if you are around, would you be kind enough to kick logmsgbot out of #wikimedia-tech ? The change is https://gerrit.wikimedia.org/r/#/c/55031/ (on fenari ) [09:46:43] ori-l: ^^^ [09:48:43] lookng [09:51:24] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55031 [09:59:46] New review: Nemo bis; "fu I75abf6144d231f7335d025a59e448500d253fe13" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8344 [10:04:31] so [10:04:45] I browse github and open an issue about some file lacking [10:04:51] I then clone the repo and the file is there [10:04:53] WTF? [10:05:13] answer: upstream managed to commit the file before I cloned the repo!!! lightning speed! [10:05:44] I thought it was going to be a bug in github interface [10:12:45] PROBLEM - Puppet freshness on aluminium is CRITICAL: Puppet has not run in the last 10 hours [10:12:46] PROBLEM - Puppet freshness on amssq31 is CRITICAL: Puppet has not run in the last 10 hours [10:12:46] PROBLEM - Puppet freshness on amssq34 is CRITICAL: Puppet has not run in the last 10 hours [10:12:46] PROBLEM - Puppet freshness on amssq35 is CRITICAL: Puppet has not run in the last 10 hours [10:12:46] PROBLEM - Puppet freshness on amssq37 is CRITICAL: Puppet has not run in the last 10 hours [10:12:46] PROBLEM - Puppet freshness on amssq40 is CRITICAL: Puppet has not run in the last 10 hours [10:12:46] PROBLEM - Puppet freshness on amssq44 is CRITICAL: Puppet has not run in the last 10 hours [10:13:03] building a debian package [10:13:10] wishhh me luck [10:14:45] PROBLEM - Puppet freshness on sodium is CRITICAL: Puppet has not run in the last 10 hours [10:16:25] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 190 seconds [10:16:45] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 193 seconds [10:18:19] dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/python-statsd_1.5.8-1.diff.1kbj2k [10:18:20] almost there [10:22:39] unexpected upstream changes? [10:22:47] is than an eufemism for hash mismatch? [10:25:18] na debian oddity [10:28:45] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [10:28:52] aude: what will "[10:14:40] !log olivneh synchronized wmf-config/CommonSettings.php 'Add 'wikidata.org' to to allow CORS'" hopefully fix? [10:29:25] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 0 seconds [10:48:33] PROBLEM - Puppet freshness on amssq43 is CRITICAL: Puppet has not run in the last 10 hours [10:56:46] yeahhh [10:56:51] only two errors :-] [11:00:33] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [11:06:32] PROBLEM - Puppet freshness on carbon is CRITICAL: Puppet has not run in the last 10 hours [11:10:34] New patchset: MaxSem; "Relax checks for translation memory Solr" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55036 [11:25:52] PROBLEM - Host niobium is DOWN: PING CRITICAL - Packet loss = 100% [11:27:23] !log Rebooting niobium with hyperthreading disabled [11:27:30] Logged the message, Master [11:30:52] RECOVERY - Host niobium is UP: PING OK - Packet loss = 0%, RTA = 0.48 ms [11:39:29] New patchset: Hashar; "Initial upstream branch." [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55038 [11:39:29] New patchset: Hashar; "Imported Upstream version 1.5.8" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55039 [11:39:30] New patchset: Hashar; "Merge tag 'upstream/1.5.8'" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55040 [11:39:30] New patchset: Hashar; "Inital deb packaging" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55041 [11:39:35] oops [11:39:57] stupid me [11:40:01] Change abandoned: Hashar; "(no reason)" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55038 [11:40:10] Change abandoned: Hashar; "(no reason)" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55039 [11:40:17] Change abandoned: Hashar; "(no reason)" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55040 [11:40:23] Change abandoned: Hashar; "(no reason)" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55041 [11:41:40] mark: hi, when using git build package should we push the upstream code too or just our debian/ directory? [11:41:59] yes too [11:42:09] but probably not through gerrit review [11:44:17] PROBLEM - Host strontium is DOWN: CRITICAL - Plugin timed out after 15 seconds [11:44:25] mark: I guess non ops will not be able to push to gerrit :-D [11:44:29] yeah [11:44:32] not sure how to deal with that [11:44:44] it's just the initial push that is the problem really [11:45:19] that is what I just did :( [11:45:27] git-review pushed the upstream branch under master branch [11:46:40] there is also a pristine-tar branch :D [11:47:17] would you be kind enough to create the upstream / pristine-branch for the python-statsd repo ? https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/python-statsd,branches [11:47:57] then I can submit my various local branches, get them merged and I can get ottomatta to review my debian/* files :-) [11:49:27] <^demon> We could probably grant Create Reference a bit wider on operations/debs/* [11:49:31] <^demon> Then people could make their own branches. [11:50:10] <^demon> Anyway, breakfast. [11:50:13] you are the bosses https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs,access :D [11:50:17] RECOVERY - Host strontium is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [11:50:54] ottomata: good morning :-] You have missed the discussion about upstream / pristine-tar branches under operations/debs :( [11:51:31] haha, awesome, i am not yet awake [11:51:38] hopefully someone will summarize, maybe in that email thread? [11:51:42] take your time and enjoy the breakfast [12:03:16] PROBLEM - Puppet freshness on colby is CRITICAL: Puppet has not run in the last 10 hours [12:07:14] PROBLEM - Puppet freshness on cp1026 is CRITICAL: Puppet has not run in the last 10 hours [12:38:56] PROBLEM - Puppet freshness on db66 is CRITICAL: Puppet has not run in the last 10 hours [12:40:46] PROBLEM - Varnish HTTP bits on palladium is CRITICAL: Connection refused [12:48:46] RECOVERY - Varnish HTTP bits on palladium is OK: HTTP OK: HTTP/1.1 200 OK - 637 bytes in 0.001 second response time [12:49:33] !log Rebooting palladium with hyperthreading disabled [12:49:40] Logged the message, Master [12:51:27] PROBLEM - Host palladium is DOWN: PING CRITICAL - Packet loss = 100% [12:55:36] RECOVERY - Host palladium is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [13:00:51] ori-l: hey :) i am missing background. what will "[10:14:40] !log olivneh synchronized wmf-config/CommonSettings.php 'Add 'wikidata.org' to to allow CORS'" hopefully fix? [13:10:50] PROBLEM - Host cp3021 is DOWN: CRITICAL - Plugin timed out after 15 seconds [13:14:52] RECOVERY - Host cp3021 is UP: PING OK - Packet loss = 0%, RTA = 82.90 ms [13:15:07] New patchset: QChris; "Make hooks-bugzilla comment on abandoning/restoring a change" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55048 [13:25:56] New patchset: Matthias Mullie; "Re-enable AFTv5 on enwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55049 [13:27:40] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [13:27:40] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [13:27:40] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [13:27:40] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [13:33:09] ^demon: https://gerrit.wikimedia.org/r/54692 [13:33:14] moin moin [13:33:32] <^demon> I don't know ruby either :p [13:34:38] PROBLEM - Host cp3022 is DOWN: PING CRITICAL - Packet loss = 100% [13:34:46] jeremyb_: ask zeljkof :-] [13:35:02] ^demon: i tested it! [13:35:17] ^demon: and it was based on consultation with #ruby :) [13:35:35] hashar, jeremyb_: what's up? ruby superhero needed? :) [13:35:40] * jeremyb_ spies zeljkof [13:35:42] jeremyb_: you might want to add that information to the commit message too :-D [13:35:52] zeljkof: jeremyb is asking for review on a erb template in puppet [13:35:54] zeljkof: https://gerrit.wikimedia.org/r/54692 [13:36:14] zeljkof: erb is not really my game, but I can take a look [13:36:33] it's not even erb. it's just ruby [13:36:34] <^demon> jeremyb_: Then it's probably ok :) [13:36:35] I just don't understand why we have an ERB template to generate the default [13:36:43] uh oh: Code Review - Error, Server Unavailable, 0 [13:36:49] when we could put the settings verbatim in the file and have ircecho handle them :-] [13:36:50] heh [13:37:10] hashar: default? [13:37:21] templates/ircecho/default.erb [13:37:35] that's the only template though [13:37:35] that expand some puppet variables to shell env variables [13:37:45] no shell env [13:37:52] we could just pass them as is and migrate all the ruby stuff to ircecho itself [13:38:07] errr, i guess? [13:38:08] jeremyb_: I have no idea what is going on in the file :) [13:38:16] i see no problem with leaving it as is [13:38:17] not a lot of help from me here [13:38:40] zeljkof: did you see the change was just adding one line? [13:38:57] before change it works in 1.8, not in 1.9.x [13:39:03] after change it works everywhere [13:39:03] jeremyb_: yes [13:39:09] output is the same in both places [13:39:28] jeremyb_: the change looks good to me, but I lack context to see if it could break anything later on [13:39:49] zeljkof: hint, i wrote that file to begin with. :) [13:40:02] i think hopefully i might have sufficient context [13:40:24] jeremyb_: how come you picked ruby? :) [13:40:47] New patchset: Hashar; "futureproof for ruby1.9.x (not in use yet)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54692 [13:40:52] zeljkof: i didn't. there's not really any other choices [13:41:11] New review: Hashar; "please amend the commit message to explain why casting is needed with ruby 1.9 :-]" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/54692 [13:41:51] jeremyb_: puppet is written in ruby? [13:41:55] yes [13:42:03] hashar is a fan of extra long commit msgs [13:42:10] jeremyb_: I like it already :) did not know that [13:42:13] ok, have to run for a few mins, bbiab [13:42:20] zeljkof: heh [13:42:57] jeremyb_: you can't just throw away a code without any context nor explaining what it is fixing. That kind of force us to play Sherlock Holmes to find it out. [13:43:44] hashar: it fixes an exception...; i think the inline comment i made indicates that it was related to bare strings that were not arrays [13:44:03] New patchset: Demon; "Block misbehaving bot" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55054 [13:44:05] the method i was using in 1.8 to force it to be an array didn't work in 1.9.x [13:44:35] anyway, really have to run for a bit [13:44:47] <^demon> You're shitting me. [13:45:16] ^demon: err? [13:45:41] <^demon> Well, somebody wrote a stupid bot that keeps requesting a tar.gz of DPLForum. [13:45:45] <^demon> That's what 55054 is for. [13:45:59] <^demon> But now I'm also getting bingbot indexing us? It's not listening to robots.txt [13:46:05] right, i read it :) [13:46:13] heh [13:46:21] you can block on useragent too if you like [13:46:41] <^demon> Well the user agent for the DPLForum one is just a generic Safari-looking UA. [13:46:58] <^demon> But it's requesting way too regularly of a stupid file to be a real person. [13:47:18] New review: Hashar; "Shouldn't you update the rule in the Proxy part?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55054 [13:47:39] ^demon: the gerrit apache conf also have a , you might want to block there [13:48:16] ^demon: i meant for bing [13:48:23] <^demon> Ah yes [13:48:24] <^demon> That. [13:48:38] New patchset: Demon; "Block misbehaving bot" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55054 [13:48:41] New patchset: Mark Bergsma; "Cache 4xx responses on the mobile backend servers for 5m" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55055 [13:48:52] <^demon> "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" [13:50:11] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55055 [13:50:46] <^demon> mark: Could you look at 55054 for me? [13:51:05] in a bit [13:51:16] babysitting my own change now [13:51:21] <^demon> Ok, thanks. [13:55:06] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55054 [13:55:18] it's live now [13:55:55] <^demon> Ok, thanks. [13:58:29] RECOVERY - Host cp3022 is UP: PING OK - Packet loss = 0%, RTA = 82.76 ms [14:04:03] <^demon> jeremyb_: Guy is still being an idiot and making requests, but at least he's not eating resources now :p [14:07:24] poke ^demon [14:07:30] * aude needs someone to review https://gerrit.wikimedia.org/r/#/c/49069/ [14:07:53] some people are still ending up at wikidata.org instead of www.wikidata.org [14:08:35] PROBLEM - Puppet freshness on hume is CRITICAL: Puppet has not run in the last 10 hours [14:10:04] New review: Aude; "if you request http://wikidata.org that *does* redirect." [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49069 [14:10:29] <^demon> aude: lgtm, but I can't deploy that. [14:10:34] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [14:10:54] ^demon: not good to me [14:11:04] <^demon> hrm? [14:11:09] * jeremyb_ comments [14:11:13] ok [14:11:38] New patchset: Demon; "Send analytics repos to #wikimedia-analytics" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55057 [14:11:41] what i did was make it like how mediawiki.org is handled [14:11:47] or that's what we want [14:12:55] aude: i don't think that's what you want... [14:13:06] ? [14:13:27] we don't want anyone to end up on wikidata.org without the www [14:13:37] it's been only problems to have both [14:14:18] huh... it helps if i do the git pull in the right repo [14:14:23] New review: Yuvipanda; "Wheeee!" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/55057 [14:14:24] yeah :) [14:15:16] The Hebrew Wikivoyage was recently approved by the board of trustees and the language committee (https://bugzilla.wikimedia.org/show_bug.cgi?id=46416). Would anyone around be able to help launch it? [14:15:18] New patchset: Hashar; "expose realm in mw-deployment-vars.sh" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55058 [14:15:18] New patchset: Hashar; "(bug 41285) adapt `foreachwiki` for labs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55059 [14:16:05] WikiJunkie_: you probably want to coordinate with relrod [14:16:07] errr [14:16:09] reedy [14:16:35] when will he be around ? [14:16:55] idk. he's here now but maybe not really here [14:17:02] just hang out here for a while :) [14:17:15] oh ok [14:19:04] * jeremyb_ waits for the git pull [14:19:08] ok [14:23:22] ottomata.meal( 'breakfast' ).isComplete? [14:25:19] yes! [14:25:22] true! [14:25:29] yeah, no likey pristine? [14:25:42] i just did what the tutorial recommended :/ [14:25:58] If you want to be able to exactly recreate the original tarball (orig.tar.gz) from Git you should also specify the --pristine-tar option. This is recommended. [14:26:19] New review: Anomie; "Why not use the getRealmSpecificFilename function from $MW_COMMON_SOURCE/multiversion/MWRealm.sh?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55059 [14:26:24] New review: Jeremyb; "incomplete" [operations/apache-config] (master) C: -1; - https://gerrit.wikimedia.org/r/49069 [14:26:37] ^demon: aude: ^ [14:27:16] ok, thanks [14:27:53] ottomata: so I got the package ready in a labs instance with all the changes thanks to your tutorial :-] But I would need the creation of 2 branches in the repository`upstream` and `pristine-tar` [14:28:07] ottomata: should be possible at https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/python-statsd,branches if you are part of the `ops` group [14:28:22] New review: Anomie; "In that case, why not throw in WMF_DATACENTER too?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55058 [14:29:00] ah so, what i did before, was give me push direct push rights [14:29:02] can I do that for you? [14:29:36] are you in the ldap ops group? [14:29:43] ottomata: nop :-] [14:29:56] ottomata: so I need branch to submit my changes and get them merged for me [14:31:27] ok [14:31:34] the integration group now has push rights [14:31:34] so [14:31:43] you should be able to directly push the upstream and pristine tar branches now [14:31:51] trying [14:32:09] still need the branches :-] [14:32:33] git push gerrit upstream:upstream [14:32:39] mm maybe I should use refs/something [14:33:00] New patchset: Aude; "(bug 45005) Redirect wikidata.org to www.wikidata.org" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49069 [14:33:43] still need to something about the language subdomains, but not sure exaclty what [14:33:58] ottomata: I can't create the branch (tried: git push gerrit upstream:refs/heads/upstream ) [14:34:13] ottomata: seems it need a Push Branch permission [14:34:52] Change abandoned: Hashar; "??" [operations/debs/python-statsd] (refs/meta/config) - https://gerrit.wikimedia.org/r/55065 [14:35:58] this doesn't woork? [14:35:58] git push gerrit upstream [14:36:13] nop because the branch does not exist in Gerrit :( [14:36:30] hm. [14:36:34] i never had that problem [14:36:41] so we need to either create the branches manually or allow "Create Reference" right [14:37:18] the ldap/ops group is allowed to Create Reference in operations/debs/* https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs,access [14:37:21] must be that :-] [14:37:42] we probably don't want to grant that to the integration group [14:37:49] i'm editing this [14:37:50] https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/python-statsd,access [14:38:05] New review: Jeremyb; "(1 comment)" [operations/apache-config] (master) C: -1; - https://gerrit.wikimedia.org/r/49069 [14:38:20] ok, i gave create reference perms to integeration grouop [14:38:24] try now? [14:38:36] works! [14:38:40] ah, cool! [14:39:03] https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/python-statsd,branches :-] [14:39:19] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 182 seconds [14:40:09] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 202 seconds [14:40:36] !g 55041 [14:40:36] https://gerrit.wikimedia.org/r/#q,55041,n,z [14:40:45] Change restored: Hashar; "(no reason)" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55041 [14:40:54] New patchset: Hashar; "Inital deb packaging" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55041 [14:41:10] stupid gerrit dependencies [14:41:24] Change abandoned: Hashar; "(no reason)" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55041 [14:41:41] New patchset: Hashar; "Inital deb packaging" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55066 [14:41:51] New review: Hashar; "resent with https://gerrit.wikimedia.org/r/55066" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55041 [14:42:14] ottomata: that worked thanks! You can remove `integration` from https://gerrit.wikimedia.org/r/#/admin/projects/operations/debs/python-statsd,access :-] [14:43:39] yeehaw [14:44:08] mk [14:44:30] arharharhhhhh [14:44:38] the change I pushed are in Gerrit [14:44:41] and abandonned [14:44:51] and Gerrit created a dependency on an abandoned change [14:44:54] :-/ [14:44:55] wa why? haha [14:45:01] hehe [14:45:21] Change restored: Hashar; "(no reason)" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55040 [14:45:28] Change restored: Hashar; "(no reason)" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55039 [14:45:38] Change abandoned: Hashar; "(no reason)" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55039 [14:45:45] oh yeah, it gets weird if you abandon a change that's not on a topic branch, right? [14:45:48] Change abandoned: Hashar; "(no reason)" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55040 [14:45:52] because then your local is no longer in sync wiiht gerrit [14:45:55] local master [14:45:58] so I sent the changes by mistakes to gerrit [14:46:10] then git pushed them by passing gerrit entirely [14:46:12] git reset --hard origin/master [14:46:17] now the Gerrit DB is no more in sync :/ [14:46:31] Change abandoned: Hashar; "(no reason)" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55066 [14:46:54] going to change the Change-Id values and resubmit [14:46:59] PROBLEM - Puppet freshness on sanger is CRITICAL: Puppet has not run in the last 10 hours [14:47:16] errr, what? [14:47:18] hashar: huh? [14:47:23] oh [14:47:44] changing the change-id is a bad solutin [14:47:47] solution* [14:47:55] I dont care [14:48:06] :( [14:49:38] hashar [14:49:45] maybe just copy your debian directory [14:49:50] re clone [14:49:54] copy it back in [14:49:58] and make a new changeset [14:50:12] yeah need to do that [14:50:17] New review: Jeremyb; "realm==production yields a labs dblist?" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/55059 [14:50:31] New patchset: Aude; "(bug 45005) Redirect wikidata.org to www.wikidata.org" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49069 [14:50:44] jeremyb_: how is it now? [14:51:01] * aude tested it and it works with en.*** [14:51:01] New patchset: Hashar; "Inital deb packaging" [operations/debs/python-statsd] (master) - https://gerrit.wikimedia.org/r/55068 [14:51:20] en.**.org or **.org [14:51:36] ottomata: na it still detect the parent Change-Id and bind the parent commit to the abandoned change. I need to change all the change-id in the master branch [14:52:12] New patchset: Aude; "(bug 45005) Redirect wikidata.org to www.wikidata.org" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/49069 [14:52:22] hm? [14:52:29] if you recloned it wouldn't detect any change id [14:52:29] eek, tab [14:52:40] i'm saying, start over [14:52:57] git clone ssh://otto@gerrit.wikimedia.org:29418/operations/debs/python-statsd [14:53:11] cp -r /tmp/debian python-statsd [14:53:11] cd python-statsd [14:53:11] git add debian [14:53:12]