[01:03:39] (03PS1) 10Reedy: Fix undefined $errstr usage [operations/software] - 10https://gerrit.wikimedia.org/r/125682 [01:04:42] lovely variable name [01:04:59] (03PS3) 10Reedy: Remove prettify, seems to be unused [operations/software] - 10https://gerrit.wikimedia.org/r/118952 [01:47:47] PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:48:07] PROBLEM - MySQL InnoDB on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:48:38] RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [01:48:57] RECOVERY - MySQL InnoDB on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [02:12:38] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 2889 MB (3% inode=99%): [02:12:52] (03PS2) 10Chad: Use proper php.ini comment format [operations/puppet] - 10https://gerrit.wikimedia.org/r/121695 [02:13:00] (03Abandoned) 10Chad: Use proper php.ini comment format [operations/puppet] - 10https://gerrit.wikimedia.org/r/121695 (owner: 10Chad) [02:18:37] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3811 MB (3% inode=99%): [02:20:27] PROBLEM - Disk space on lvs3001 is CRITICAL: DISK CRITICAL - free space: / 1777 MB (3% inode=97%): [02:23:01] !log LocalisationUpdate completed (1.23wmf21) at 2014-04-14 02:22:58+00:00 [02:23:07] Logged the message, Master [02:42:07] !log LocalisationUpdate completed (1.23wmf22) at 2014-04-14 02:42:05+00:00 [02:42:12] Logged the message, Master [02:47:17] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [02:47:17] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [02:47:17] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [02:47:17] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [03:00:27] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.000277777777778 [03:00:37] RECOVERY - Disk space on virt0 is OK: DISK OK [03:04:53] (03CR) 10Ori.livneh: [C: 032] Fix undefined $errstr usage [operations/software] - 10https://gerrit.wikimedia.org/r/125682 (owner: 10Reedy) [03:23:03] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 14 03:22:58 UTC 2014 (duration 22m 57s) [03:23:07] Logged the message, Master [03:48:07] PROBLEM - MySQL InnoDB on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:49:47] PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:49:57] RECOVERY - MySQL InnoDB on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [03:50:38] RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [04:00:27] RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0 [04:25:04] ugh [04:25:18] Gloria: I ugh'd. [04:25:24] It's the beginning of the end. [04:28:01] Ugh. [04:36:05] grrrit-wm is slow [04:36:10] (03PS1) 10Ori.livneh: Graphite/Carbon: set max open files via Upstart 'limit' directive [operations/puppet] - 10https://gerrit.wikimedia.org/r/125686 [04:36:13] (03PS1) 10Ori.livneh: Graphite/Carbon: less logspam [operations/puppet] - 10https://gerrit.wikimedia.org/r/125687 [04:37:00] (03CR) 10Ori.livneh: [C: 032 V: 032] Graphite/Carbon: set max open files via Upstart 'limit' directive [operations/puppet] - 10https://gerrit.wikimedia.org/r/125686 (owner: 10Ori.livneh) [04:37:38] (03CR) 10Ori.livneh: [C: 032] Graphite/Carbon: less logspam [operations/puppet] - 10https://gerrit.wikimedia.org/r/125687 (owner: 10Ori.livneh) [05:02:24] hi _joe_ [05:03:15] <_joe_> hi matanya [05:03:28] <_joe_> just reconnected as TOR seems to be dead [05:03:45] <_joe_> well, TOR on freenode [05:33:38] PROBLEM - Graphite Carbon on tungsten is CRITICAL: CRITICAL: Not all configured Carbon instances are running. [05:34:38] RECOVERY - Graphite Carbon on tungsten is OK: OK: All defined Carbon jobs are runnning. [05:48:17] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [05:48:17] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [05:48:17] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [05:48:17] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [05:55:26] <_joe_> off to school - BB in ~ 1 hour [06:00:27] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.000277700638711 [07:00:27] RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0 [07:08:28] (03CR) 10Dzahn: "the ticket says there is a remaining question what to do with /root and is not 100% clear to me if it can be shutdown" [operations/puppet] - 10https://gerrit.wikimedia.org/r/123626 (owner: 10Matanya) [07:15:15] (03CR) 10Alexandros Kosiaris: [C: 032] Create and import shapelines from a pre-existing dump [operations/puppet] - 10https://gerrit.wikimedia.org/r/123767 (owner: 10Alexandros Kosiaris) [07:42:49] (03CR) 10Dzahn: [C: 04-1] "yes, i was going to merge this because it is indeed what upstream changed and the point is to reduce the diff between default and custom f" (034 comments) [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/119726 (owner: 1001tonythomas) [07:45:47] (03PS3) 10Dzahn: decom : brewster [operations/puppet] - 10https://gerrit.wikimedia.org/r/123626 (owner: 10Matanya) [07:50:54] (03CR) 10Dzahn: [C: 032] "the only remaining question was what to do with old package files in /root. please speak up on Alex' mail thread "apt.wikimedia.org migrat" [operations/puppet] - 10https://gerrit.wikimedia.org/r/123626 (owner: 10Matanya) [07:54:16] !log brewster - disabling puppet agent, removed from site.pp, revoke puppet cert [07:54:22] Logged the message, Master [07:59:04] forgot to do this earlier [07:59:38] waves, hi apergos [08:06:45] apergos: fyi, /root/atg on brewster has "utfnormal" packaging stuff, and "mwbzutils", if you think that should be saved / is not on the new apt.wm, now would be a good time [08:07:06] because of brewster decom and the files in /root there [08:07:07] toss please [08:07:14] great, thx [08:07:22] yw [08:30:00] (03PS1) 10Dzahn: decom: remove brewster incl. mgmt [operations/dns] - 10https://gerrit.wikimedia.org/r/125695 [08:31:40] search looks to be broken on wikitech [08:32:48] <_joe_> apergos: yes I can confirm this. [08:33:03] <_joe_> I'm getting a white page [08:33:07] I"ll look at it in a little bit then [08:33:15] <_joe_> which usually is a php error [08:33:16] (encountered this in doing something else) [08:33:30] yep I expect there is an exception of some sort [08:33:40] confirmed [08:33:52] maybe a "virt0" somewhere hardcoded left [08:33:59] where it is now virt1000 ? [08:36:56] not in error.log hrmm [08:39:13] config/Local.php:$wgCirrusSearchServers = array( '10.2.2.30' ); [08:40:44] <_joe_> eh. [08:41:04] it can talk to that, it's lvs1003 [08:41:23] from virt1000 [08:41:27] which is now wikitech [08:41:53] but i did not look for any recent changes in that config yet [08:44:12] (03Abandoned) 10Alexandros Kosiaris: Remove DNS entries for brewster [operations/dns] - 10https://gerrit.wikimedia.org/r/123206 (owner: 10Alexandros Kosiaris) [08:44:23] (03CR) 10Alexandros Kosiaris: [C: 032] decom: remove brewster incl. mgmt [operations/dns] - 10https://gerrit.wikimedia.org/r/125695 (owner: 10Dzahn) [08:48:38] PROBLEM - Puppet freshness on lvs3001 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [08:48:38] PROBLEM - Puppet freshness on lvs3003 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [08:48:38] PROBLEM - Puppet freshness on lvs3002 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [08:48:38] PROBLEM - Puppet freshness on lvs3004 is CRITICAL: Last successful Puppet run was Thu 01 Jan 1970 12:00:00 AM UTC [08:50:15] maybe broken by something similar to " Merge "beta: drop pmtpa reference for CirrusSearch"" [09:01:17] !log brewster - stop lighttpd,bacula-fd,haproxy,dhcp3-server,rsync,nrpe,salt [09:01:22] Logged the message, Master [09:05:03] ACKNOWLEDGEMENT - check configured eth on virt1001 is CRITICAL: virbr0 reporting no carrier. daniel_zahn see RT #7251 [09:05:03] ACKNOWLEDGEMENT - check configured eth on virt1002 is CRITICAL: virbr0 reporting no carrier. daniel_zahn see RT #7251 [09:05:03] ACKNOWLEDGEMENT - check configured eth on virt1003 is CRITICAL: virbr0 reporting no carrier. daniel_zahn see RT #7251 [09:05:03] ACKNOWLEDGEMENT - check configured eth on virt1004 is CRITICAL: virbr0 reporting no carrier. daniel_zahn see RT #7251 [09:05:03] ACKNOWLEDGEMENT - check configured eth on virt1005 is CRITICAL: virbr0 reporting no carrier. daniel_zahn see RT #7251 [09:05:03] ACKNOWLEDGEMENT - check configured eth on virt1006 is CRITICAL: virbr0 reporting no carrier. daniel_zahn see RT #7251 [09:05:03] ACKNOWLEDGEMENT - check configured eth on virt1007 is CRITICAL: virbr0 reporting no carrier. daniel_zahn see RT #7251 [09:10:34] CUSTOM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , enwiki (336047), Total (455012) [09:10:47] should the limit be raised again? ^ [09:11:04] 199k limit vs. 336k actual [09:11:58] PROBLEM - SSH on lvs1002 is CRITICAL: Server answer: [09:14:58] RECOVERY - SSH on lvs1002 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.3 (protocol 2.0) [09:24:31] https://bugzilla.wikimedia.org/show_bug.cgi?id=63839 meh [09:26:05] !log deleting huge pybal log on lvs3001 [09:26:11] Logged the message, Master [09:26:28] RECOVERY - Disk space on lvs3001 is OK: DISK OK [09:32:57] probably 3002-3004 are the same? [09:36:58] after restarting apache, logs to the error log went there instead of to a deleted error.log.1; I updated the bug reoprt with the php fatal [09:37:06] for search on wikitech [09:37:52] the disk space issue was only on that 1 host [09:38:01] puppet disabled on all though [09:38:24] apergos: ah [09:50:00] ACKNOWLEDGEMENT - Disk space on virt1000 is CRITICAL: DISK CRITICAL - free space: / 2304 MB (3% inode=85%): daniel_zahn due to backups from migration. need to remind Andrew though to move to new backup location [09:54:51] apergos: Can you check extensions were recursively checked out for wikitech please? [10:05:28] !log had to toss extensions/Elastica on virt1000 and run git submodule update --init --recursive seems to be working now [10:05:34] Logged the message, Master [10:05:39] thanks ree dy [10:11:38] PROBLEM - Puppet freshness on labsdb1004 is CRITICAL: Last successful Puppet run was Mon 14 Apr 2014 07:10:31 AM UTC [10:14:32] "Slow CirrusSearch query rate" [10:23:12] (03CR) 10Dzahn: [C: 04-1] "./check_apt_mirror: 14: ./check_apt_mirror: let: not found" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/112738 (owner: 10Matanya) [10:30:48] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [10:33:55] (03PS2) 10Dzahn: apt mirror: add monitoring script to verify mirror is up to date [operations/puppet] - 10https://gerrit.wikimedia.org/r/112738 (owner: 10Matanya) [10:35:44] (03CR) 10Dzahn: [C: 031] "- "-lt 2 -o -eq 2" is the same as just "-le 2"" [operations/puppet] - 10https://gerrit.wikimedia.org/r/112738 (owner: 10Matanya) [10:37:50] wth was that [10:38:50] <_joe_> paravoid: a spike in 5xx responses for a minute, it seems [10:39:47] seems like mobile esams [10:40:55] http://ganglia.wikimedia.org/latest/graph_all_periods.php?h=cp3012.esams.wikimedia.org&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2&st=1397471984&g=mem_report&z=large&c=Mobile%20caches%20esams [10:40:59] *sigh* [10:41:07] still the same shit [10:41:17] compact_memory not having the right effect, still [10:49:07] are you saying domas was wrong? ;p [10:49:24] !log reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache' [10:49:30] Logged the message, Master [10:49:44] (03PS1) 10Reedy: Update interwiki cache [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/125703 [10:50:13] (03CR) 10Reedy: [C: 032] Update interwiki cache [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/125703 (owner: 10Reedy) [10:50:20] (03Merged) 10jenkins-bot: Update interwiki cache [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/125703 (owner: 10Reedy) [10:50:38] (03CR) 10Dzahn: [C: 032] "it does the job like this. of course needs to be added to NRPE check" [operations/puppet] - 10https://gerrit.wikimedia.org/r/112738 (owner: 10Matanya) [10:53:23] (03CR) 10TTO: "While you're in an interwiki mood, Reedy, could you please take a look at Iec1ee0ab80a16ae6f5d89528c060331881f72c4a and maybe even Ie327c7" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/125703 (owner: 10Reedy) [10:54:17] (03CR) 10Dzahn: "and please take this as well when syncing mw config anyways, changes a comment line only" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123853 (owner: 10Dzahn) [10:58:08] (03PS1) 10Faidon Liambotis: ssl: fix whitespace-damaged files [operations/puppet] - 10https://gerrit.wikimedia.org/r/125704 [10:58:31] (03PS2) 10Faidon Liambotis: ssl: fix whitespace-damaged certificates [operations/puppet] - 10https://gerrit.wikimedia.org/r/125704 [10:59:13] (03CR) 10Faidon Liambotis: [C: 032 V: 032] ssl: fix whitespace-damaged certificates [operations/puppet] - 10https://gerrit.wikimedia.org/r/125704 (owner: 10Faidon Liambotis) [11:06:50] (03PS1) 10Dzahn: remove ms6 ipmi, decom [operations/dns] - 10https://gerrit.wikimedia.org/r/125705 [11:08:41] !log brewster - shut down [11:08:45] Logged the message, Master [11:10:16] (03CR) 10ArielGlenn: [C: 032] remove ms6 ipmi, decom [operations/dns] - 10https://gerrit.wikimedia.org/r/125705 (owner: 10Dzahn) [11:12:56] (03CR) 10Dzahn: "apergos, it was also this" [operations/dns] - 10https://gerrit.wikimedia.org/r/124901 (owner: 10Matanya) [11:13:35] (03CR) 10Dzahn: "also removing mgmt in Change-Id: Idf410b3e02a3" [operations/dns] - 10https://gerrit.wikimedia.org/r/124901 (owner: 10Matanya) [11:15:23] (03CR) 10Dzahn: [C: 032] "changes to the script itself in a separate change on purpose" [operations/puppet] - 10https://gerrit.wikimedia.org/r/123852 (owner: 10Dzahn) [11:18:14] (03CR) 10Dzahn: "matanya, let's make a new patch that handles the non-db67 db's in here" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122406 (owner: 10Dzahn) [11:18:49] (03CR) 10Dzahn: [C: 031] decom: remove brewster incl. mgmt [operations/dns] - 10https://gerrit.wikimedia.org/r/125695 (owner: 10Dzahn) [11:20:28] (03PS4) 10Dzahn: include 'bastionhost' on bastion hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/122399 [11:20:36] (03CR) 10Dzahn: [C: 031] include 'bastionhost' on bastion hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/122399 (owner: 10Dzahn) [11:25:28] (03PS3) 10Dzahn: lint role/deployment [operations/puppet] - 10https://gerrit.wikimedia.org/r/122338 [11:25:34] (03CR) 10Dzahn: lint role/deployment (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122338 (owner: 10Dzahn) [11:28:48] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [11:29:47] (03PS2) 10Dzahn: lint labsproxy.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/122335 [11:29:58] (03CR) 10Dzahn: lint labsproxy.pp (034 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122335 (owner: 10Dzahn) [11:31:14] (03PS2) 10Reedy: replace hume with terbium in a comment [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123853 (owner: 10Dzahn) [11:31:20] (03CR) 10Reedy: [C: 032] replace hume with terbium in a comment [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123853 (owner: 10Dzahn) [11:31:28] (03Merged) 10jenkins-bot: replace hume with terbium in a comment [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/123853 (owner: 10Dzahn) [11:36:17] apergos: you responded "yes we should" for dataset1001's network upgrade; RT ticket? :) [11:36:32] yes. I did. [11:37:35] you did file one or you did respond? [11:37:52] (03CR) 10Giuseppe Lavagetto: [C: 031] "All the lint changes are harmless, including the change of type in the deployer_groups definition" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122338 (owner: 10Dzahn) [11:39:48] I responded only, it should not be done just yet unless it can be done without interruption of the network connection [11:39:55] in about a day it can happen [11:40:36] RT ticket so we won't forget? [11:43:57] (03CR) 10Dzahn: lint role/keystone (labs) (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122334 (owner: 10Dzahn) [11:44:18] (03PS3) 10Andrew Bogott: Remove deprecated roles, and lint remaining labsproxy.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/122335 (owner: 10Dzahn) [11:44:27] I'm on rt so tomorrow I'll be looking at it again (and perhaps doing it) [11:44:58] at least that was my plan, if you want me to pass it to you that's ok too [11:45:08] (03CR) 10Andrew Bogott: "We're not using the pmtpa/eqiad-proxy bits anymore, so I greatly simplified this by ripping out old code." [operations/puppet] - 10https://gerrit.wikimedia.org/r/122335 (owner: 10Dzahn) [11:47:00] no, that's fine [11:47:03] (03PS2) 10Dzahn: lint role/keystone (labs) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122334 [11:47:16] (03CR) 10ArielGlenn: [C: 031] "I think this is fine, just a note that the line" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122338 (owner: 10Dzahn) [11:48:04] (03CR) 10Dzahn: [C: 031] "nice :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122335 (owner: 10Dzahn) [11:52:02] (03CR) 10Dzahn: "eh, actually it's in here twice, once with true and once with false ?!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122338 (owner: 10Dzahn) [11:53:03] (03PS4) 10Dzahn: lint role/deployment [operations/puppet] - 10https://gerrit.wikimedia.org/r/122338 [11:53:12] (03CR) 10Dzahn: [C: 032] "the disabled one is just labs" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122338 (owner: 10Dzahn) [12:00:28] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.00138888888889 [12:01:45] (03CR) 10Dzahn: "no changes on tin after puppet run" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122338 (owner: 10Dzahn) [12:03:15] (03CR) 10Dzahn: [C: 032] Remove deprecated roles, and lint remaining labsproxy.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/122335 (owner: 10Dzahn) [12:06:25] (03PS3) 10Dzahn: lint role/keystone (labs) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122334 [12:09:15] (03PS2) 10Alexandros Kosiaris: Import coastlines and land polygons into OSM db [operations/puppet] - 10https://gerrit.wikimedia.org/r/123792 [12:10:28] (03CR) 10Andrew Bogott: [C: 032] lint role/keystone (labs) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122334 (owner: 10Dzahn) [12:12:23] :) [12:16:25] (03CR) 10Dzahn: [C: 031] "re: importing coastlines. does this get slower while processing Norway?:) Slartibartfast may have copyright http://en.wikipedia.org/wiki/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/123792 (owner: 10Alexandros Kosiaris) [12:18:34] mutante: ahahahaha. Only if the shapefile also contains a stavromula beta [12:18:51] *gg* [12:23:02] (03CR) 10Dzahn: "we should make a new bz test instance in eqiad, can do soon" [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/124140 (owner: 1001tonythomas) [12:27:11] (03CR) 10Dzahn: "https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=svc" [operations/dns] - 10https://gerrit.wikimedia.org/r/120063 (owner: 10Dzahn) [12:30:47] (03CR) 10Alexandros Kosiaris: [C: 032] include 'bastionhost' on bastion hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/122399 (owner: 10Dzahn) [12:33:10] woot, merge day for me, thanks [12:33:48] I wish that would ever be the case for me... [12:34:13] https://gerrit.wikimedia.org/r/116019 still rotting around... [12:34:33] hoo: I can press abandon for you [12:34:40] Oh wait, no I can't ;) [12:35:13] That looks pretty trivial if you just poke someone from Ops... [12:35:17] :P Less rights, less abuse [12:35:37] I poked more or less everyone yet... mh [12:35:46] oh apergos is on rt duty, so I can annoy him this time [12:35:49] :P [12:36:26] (03PS8) 10Hoo man: Introduce an admins::release user group [operations/puppet] - 10https://gerrit.wikimedia.org/r/116019 [12:36:32] mh, it even cleanly rebased [12:38:44] (03CR) 10Dzahn: [C: 031] "to reply to questions above: the problem this wants to solve is that admins::restricted gives more access than actually needed, it is not " [operations/puppet] - 10https://gerrit.wikimedia.org/r/116019 (owner: 10Hoo man) [12:39:36] don't abandon it, it's good that it points out that restricted does not just mean bastion [12:40:09] even uploaded a PS [12:40:14] I wish to do more such changes, but if review is this slow, this will take ages [12:40:42] hoo: we just talked about entire refactoring of admins.pp with half the ops team in Athens [12:40:46] something will happen [12:40:59] hoo: You should add some more reviewers [12:41:02] * Reedy grins [12:41:05] it is probably just that everybody wanted to wait for that before introducing that [12:41:30] and point out again why you added it in the first place (gives extra access etc) [12:42:00] I explained it to various people on IRC :P [12:42:14] should have probably noted that in the bug [12:43:46] (03CR) 10ArielGlenn: "I would prefer that users in this group are added to all bastions, not just bast1001 which happens to be the bation for the primary dc rig" [operations/puppet] - 10https://gerrit.wikimedia.org/r/116019 (owner: 10Hoo man) [12:43:55] gah typos. anyways [12:44:04] (03CR) 10Dzahn: "admins::restricted gives terbium access. terbium access is not needed for release uploaders." [operations/puppet] - 10https://gerrit.wikimedia.org/r/116019 (owner: 10Hoo man) [12:44:58] apergos: They only need the bastion for the release server... they don't need to ssh into pmtpa or even esams, right? [12:46:21] paravoid: Do you want https://gerrit.wikimedia.org/r/#/c/121912/ deploying? [12:46:22] there should be one class that gives an account on all (non-ops) bastions but nothing else [12:46:31] but also more than just one [12:46:43] Reedy: ? [12:46:51] I want it merged & deployed, yes :) [12:46:51] atom recent changes [12:46:53] right [12:47:06] mutante: ahhh, that's what I wanted to do first but then someone told me that it's a stupid idea ... *sigh* [12:47:07] code is sane, just going to fix the code style and add a RELEASE-NOTES entry [12:47:14] awesome [12:47:17] thank you very much [12:47:21] mutante: yep that's what I was getting at, a bastions only class [12:47:36] then if service A moves to $otherdc we don't have to shuffle access around [12:48:27] agrees, it should add one bastion in each dc, but also not add more than bastions [12:48:43] and the rest is about naming [12:49:01] not changing the existing one called "restricted" is less work [12:49:07] mutante: Ok, but fenari isn't good as a bastion as hit has a f*cking lot of private data on it [12:49:17] s/hit/it/ [12:49:31] yea, fenari, i'd say exclude it already [12:49:47] hopefully it won't be alive that much longer [12:49:48] If we wait long enough mutante will murder fenari [12:50:39] heh [12:51:16] So you people now want the bastions only group which you didn't want first? [12:51:24] apergos: i agree to both :p fenari is not a good bastion but we have no other pmtpa bastion [12:51:55] just prepare the class as bastions and if it has just one member for now, it's ok? [12:52:48] So basically: Back to PS1 and rebase? [12:52:53] https://gerrit.wikimedia.org/r/#/c/116019/1/manifests/admins.pp [12:53:08] (ignore the white space changes) [12:54:18] if you ask me,ehm..yes [12:55:09] (03CR) 10Alexandros Kosiaris: [C: 032] Import coastlines and land polygons into OSM db [operations/puppet] - 10https://gerrit.wikimedia.org/r/123792 (owner: 10Alexandros Kosiaris) [12:56:09] apergos: i think he's right, don't even encourage them to use fenari anymore, while calling it bastion class to be flexible later [12:56:38] RECOVERY - Puppet freshness on labsdb1004 is OK: puppet ran at Mon Apr 14 12:56:31 UTC 2014 [12:59:55] (03PS1) 10Dzahn: remove pmtpa payments LVS monitoring [operations/puppet] - 10https://gerrit.wikimedia.org/r/125715 [13:00:28] RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0 [13:01:03] Ok, will put that thing back to the PS1 state later today [13:02:16] +1 [13:05:28] PROBLEM - DPKG on mw1149 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:05:37] (03PS1) 10Alexandros Kosiaris: Correct land_polygons file names [operations/puppet] - 10https://gerrit.wikimedia.org/r/125717 [13:05:38] PROBLEM - DPKG on mw1153 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:05:48] PROBLEM - DPKG on mw1150 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:06:28] RECOVERY - DPKG on mw1149 is OK: All packages OK [13:06:39] PROBLEM - DPKG on mw1152 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:06:48] RECOVERY - DPKG on mw1150 is OK: All packages OK [13:07:18] PROBLEM - Apache HTTP on mw1149 is CRITICAL: Connection refused [13:07:38] RECOVERY - DPKG on mw1152 is OK: All packages OK [13:07:38] RECOVERY - DPKG on mw1153 is OK: All packages OK [13:07:38] PROBLEM - Apache HTTP on mw1150 is CRITICAL: Connection refused [13:07:44] that would be me [13:07:48] PROBLEM - Apache HTTP on mw1153 is CRITICAL: Connection refused [13:08:08] PROBLEM - Apache HTTP on mw1152 is CRITICAL: Connection refused [13:08:18] RECOVERY - Apache HTTP on mw1149 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.090 second response time [13:08:38] RECOVERY - Apache HTTP on mw1150 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.086 second response time [13:08:48] RECOVERY - Apache HTTP on mw1153 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.108 second response time [13:08:58] PROBLEM - DPKG on mw1151 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:09:08] RECOVERY - Apache HTTP on mw1152 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.092 second response time [13:09:58] RECOVERY - DPKG on mw1151 is OK: All packages OK [13:11:49] (03CR) 10Alexandros Kosiaris: [C: 032] Correct land_polygons file names [operations/puppet] - 10https://gerrit.wikimedia.org/r/125717 (owner: 10Alexandros Kosiaris) [13:13:08] PROBLEM - DPKG on mw1154 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:13:38] PROBLEM - DPKG on mw1155 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:13:58] PROBLEM - DPKG on mw1156 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:14:38] RECOVERY - DPKG on mw1155 is OK: All packages OK [13:14:38] PROBLEM - Apache HTTP on mw1156 is CRITICAL: Connection refused [13:14:38] PROBLEM - Apache HTTP on mw1155 is CRITICAL: Connection refused [13:14:58] RECOVERY - DPKG on mw1156 is OK: All packages OK [13:14:58] PROBLEM - Apache HTTP on mw1154 is CRITICAL: Connection refused [13:15:07] !log staggered upgrades for all pending updates on all mw* boxes & restarting apaches/other core services [13:15:08] RECOVERY - DPKG on mw1154 is OK: All packages OK [13:15:13] Logged the message, Master [13:15:58] PROBLEM - DPKG on terbium is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:15:58] RECOVERY - Apache HTTP on mw1154 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.180 second response time [13:16:38] RECOVERY - Apache HTTP on mw1156 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.177 second response time [13:16:39] RECOVERY - Apache HTTP on mw1155 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.177 second response time [13:16:39] PROBLEM - DPKG on tin is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:17:39] RECOVERY - DPKG on tin is OK: All packages OK [13:17:58] RECOVERY - DPKG on terbium is OK: All packages OK [13:20:18] PROBLEM - DPKG on mw1159 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:20:18] PROBLEM - DPKG on mw1160 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:20:38] PROBLEM - DPKG on mw1158 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:20:38] PROBLEM - DPKG on mw1157 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:21:38] PROBLEM - DPKG on mw1004 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:21:39] RECOVERY - DPKG on mw1158 is OK: All packages OK [13:21:39] PROBLEM - DPKG on mw1002 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:21:39] PROBLEM - Apache HTTP on mw1159 is CRITICAL: Connection refused [13:21:39] RECOVERY - DPKG on mw1157 is OK: All packages OK [13:21:48] PROBLEM - Apache HTTP on mw1157 is CRITICAL: Connection refused [13:21:58] PROBLEM - DPKG on mw1003 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:22:09] !log reedy synchronized php-1.23wmf21/includes/api/ApiFeedRecentChanges.php 'I268d0a53067738ba96bee74c593358b0b28cc083' [13:22:13] Logged the message, Master [13:22:14] <_joe_> should I worry? [13:22:18] RECOVERY - DPKG on mw1160 is OK: All packages OK [13:22:18] RECOVERY - DPKG on mw1159 is OK: All packages OK [13:22:28] no [13:22:28] PROBLEM - DPKG on mw1001 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:22:31] it's just me [13:22:36] upgrading the universe :) [13:22:38] RECOVERY - DPKG on mw1004 is OK: All packages OK [13:22:38] RECOVERY - DPKG on mw1002 is OK: All packages OK [13:22:38] RECOVERY - Apache HTTP on mw1159 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.104 second response time [13:22:38] PROBLEM - DPKG on mw1006 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:22:44] (I !logged it) [13:22:48] RECOVERY - Apache HTTP on mw1157 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 809 bytes in 0.101 second response time [13:22:54] !log reedy synchronized php-1.23wmf22/includes/api/ApiFeedRecentChanges.php 'I268d0a53067738ba96bee74c593358b0b28cc083' [13:22:58] RECOVERY - DPKG on mw1003 is OK: All packages OK [13:22:58] PROBLEM - DPKG on mw1005 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:22:58] Logged the message, Master [13:22:59] paravoid: done and done [13:23:02] <_joe_> paravoid: sorry lost it [13:23:26] Reedy: thanks a bunch [13:23:29] RECOVERY - DPKG on mw1001 is OK: All packages OK [13:23:38] RECOVERY - DPKG on mw1006 is OK: All packages OK [13:23:47] _joe_: can't blame you ;) [13:23:56] oh boy how I wish this was more automated [13:23:58] RECOVERY - DPKG on mw1005 is OK: All packages OK [13:24:28] PROBLEM - DPKG on tmh1001 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:25:58] PROBLEM - DPKG on mw1011 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:26:18] PROBLEM - DPKG on mw1008 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:26:18] PROBLEM - DPKG on mw1012 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:26:28] RECOVERY - DPKG on tmh1001 is OK: All packages OK [13:26:38] PROBLEM - DPKG on mw1007 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:26:38] PROBLEM - DPKG on mw1009 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:26:48] PROBLEM - DPKG on mw1010 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:26:58] RECOVERY - DPKG on mw1011 is OK: All packages OK [13:27:18] RECOVERY - DPKG on mw1008 is OK: All packages OK [13:27:18] RECOVERY - DPKG on mw1012 is OK: All packages OK [13:27:38] RECOVERY - DPKG on mw1007 is OK: All packages OK [13:27:38] RECOVERY - DPKG on mw1009 is OK: All packages OK [13:27:48] RECOVERY - DPKG on mw1010 is OK: All packages OK [13:29:48] PROBLEM - DPKG on tmh1002 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:29:58] PROBLEM - DPKG on mw1014 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:30:18] PROBLEM - DPKG on mw1013 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:30:28] PROBLEM - DPKG on mw1016 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:30:28] PROBLEM - DPKG on mw1015 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:30:51] (03PS2) 10Manybubbles: Make Elasticsearch more reliable in beta [operations/puppet] - 10https://gerrit.wikimedia.org/r/125331 [13:30:58] RECOVERY - DPKG on mw1014 is OK: All packages OK [13:31:09] ottomata: morning [13:31:18] RECOVERY - DPKG on mw1013 is OK: All packages OK [13:31:28] RECOVERY - DPKG on mw1016 is OK: All packages OK [13:31:28] RECOVERY - DPKG on mw1015 is OK: All packages OK [13:31:48] RECOVERY - DPKG on tmh1002 is OK: All packages OK [13:34:51] (03PS1) 10Dzahn: move LDAP admin permissions,tools out of site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/125721 [13:35:52] (03CR) 10jenkins-bot: [V: 04-1] move LDAP admin permissions,tools out of site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/125721 (owner: 10Dzahn) [13:35:58] (03PS2) 10Dzahn: move LDAP admin permissions,tools out of site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/125721 [13:37:01] (03CR) 10jenkins-bot: [V: 04-1] move LDAP admin permissions,tools out of site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/125721 (owner: 10Dzahn) [13:38:14] (03PS3) 10Dzahn: move LDAP admin permissions,tools out of site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/125721 [13:39:16] (03CR) 10jenkins-bot: [V: 04-1] move LDAP admin permissions,tools out of site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/125721 (owner: 10Dzahn) [13:40:07] duh [13:41:58] PROBLEM - DPKG on mw1017 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:42:15] manybubbles: morning! [13:42:29] (03PS4) 10Dzahn: move LDAP admin permissions,tools out of site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/125721 [13:42:58] RECOVERY - DPKG on mw1017 is OK: All packages OK [13:43:25] ahh CirrusSearch query rate [13:43:31] gonna work on that as soon as I get through my emails [13:43:44] i made a tweak to it friday hoping it would be ok, but apparently not [13:43:51] akosiaris, do you still object to https://gerrit.wikimedia.org/r/#/c/125183/ ? [13:44:10] ottomata: I don't see it complaining here. [13:45:47] (03Abandoned) 10Andrew Bogott: Added instancetype fact to get labs instance flavor. [operations/puppet] - 10https://gerrit.wikimedia.org/r/117823 (owner: 10Andrew Bogott) [13:45:48] ah ha, oh [13:45:53] i was just looking at scroll back [13:46:01] hah, must have been looking at scrollback from friday [13:46:23] all I see this morning is dkpg [13:46:40] hey, if you end up doing another repartition today let me know [13:46:48] PROBLEM - DPKG on mw1018 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:46:51] (03PS3) 10Andrew Bogott: openstack: qualify var [operations/puppet] - 10https://gerrit.wikimedia.org/r/119488 (owner: 10Matanya) [13:47:20] I'd like to sneak in a jvm config change on the restart to see if that speeds things up a tad. I'll do it manually to whichever one you are restarting and it if works puppet it. [13:47:25] if it doesn't I'll revert it [13:47:59] manybubbles: yeah plan on more reparitioning [13:48:00] ok cool [13:48:48] RECOVERY - DPKG on mw1018 is OK: All packages OK [13:49:51] matanya: just to reconfirm my understanding of how this works… if $openstack_version is set in site.pp, that's the same scope as $::openstack_version in other manifests? [13:49:53] (I'm looking at https://gerrit.wikimedia.org/r/#/c/119488/ ) [13:52:08] PROBLEM - DPKG on mw1034 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:52:08] PROBLEM - DPKG on mw1049 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:52:13] andrewbogott: not really. My only question is if this interface is needed at all. [13:52:18] PROBLEM - DPKG on mw1027 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:52:18] PROBLEM - DPKG on mw1020 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:52:18] PROBLEM - DPKG on mw1035 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:52:18] PROBLEM - DPKG on mw1032 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:52:18] PROBLEM - DPKG on mw1045 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:52:26] (still me) [13:52:28] PROBLEM - DPKG on mw1023 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:52:28] PROBLEM - DPKG on mw1021 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:52:28] PROBLEM - DPKG on mw1041 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:52:28] PROBLEM - DPKG on mw1028 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:52:34] andrewbogott: no. $openstack_version in site.pp != $::openstack_version [13:52:38] PROBLEM - DPKG on mw1031 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:52:38] PROBLEM - DPKG on mw1043 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:52:38] PROBLEM - DPKG on mw1048 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:52:38] PROBLEM - DPKG on mw1030 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [13:52:38]