[04:03:02] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=609321 edit summary: /* User:Ngiamba */ done [04:14:49] !log bots [bots-1] wm-bot was gone after netsplit. booted bouncer && wmib [04:15:00] Logged the message, Master [04:15:14] !log bots [bots-labs] labs-morebots was gone after netsplit. booted [04:15:14] Logged the message, Master [04:15:14] jeremyb: [09:34:57 PM] Any chance someone can figure out why http://bots.wmflabs.org/~richs/flickr2commons.php is giving a Forbidden error? [04:15:22] legoktm: i assume it's related to why ~petrb sometimes does [04:15:43] is there a way it can be fixed since the ts is being really flake-y right now? [04:15:44] legoktm: petan knows about it. it's intermittent [04:16:01] * jeremyb doesn't really want to look at it... [04:16:08] maybe if it's still broken in a couple days. ;P [04:16:28] (i also don't know anything about how that apache is set up) [04:21:00] ok... [04:21:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:27:52] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Critical: 3% free memory [04:41:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:51:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:11:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:22:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:41:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:52:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:11:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:22:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:28:33] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:29:33] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 152 processes [06:33:33] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 146 processes [06:39:32] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 147 processes [06:41:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:52:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:01:43] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c.pmtpa.wmflabs output: Warning: 19% free memory [07:11:42] RECOVERY Free ram is now: OK on bots-2 i-0000009c.pmtpa.wmflabs output: OK: 20% free memory [07:12:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:22:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:29:42] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c.pmtpa.wmflabs output: Warning: 19% free memory [07:43:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:52:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:13:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:23:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:43:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:53:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:13:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:23:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:26:50] jeremyb did wm-bot die? [09:31:04] freenode server disconnected for some reason [09:31:05] I see [09:31:07] in logs [09:31:16] connectivity issue :/ [09:31:31] for some reason the freenode is rejecting the connection when it's made too early [09:43:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:53:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:13:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:24:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:28:52] PROBLEM Current Load is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:29:22] PROBLEM Disk Space is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:30:32] PROBLEM Current Users is now: UNKNOWN on dumps-bot3 i-00000503.pmtpa.wmflabs output: NRPE: Call to fork() failed [10:33:42] PROBLEM dpkg-check is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:33:52] PROBLEM SSH is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Server answer: [10:35:33] PROBLEM Current Users is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:43:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:54:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:57:52] PROBLEM Free ram is now: WARNING on dumps-bot3 i-00000503.pmtpa.wmflabs output: Warning: 7% free memory [10:58:42] RECOVERY dpkg-check is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: All packages OK [10:58:52] RECOVERY Current Load is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: OK - load average: 0.25, 0.33, 0.32 [10:58:52] RECOVERY SSH is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [10:59:22] RECOVERY Disk Space is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: DISK OK [11:00:33] RECOVERY Current Users is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [11:02:53] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Critical: 5% free memory [11:07:52] PROBLEM Free ram is now: WARNING on dumps-bot3 i-00000503.pmtpa.wmflabs output: Warning: 6% free memory [11:13:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:24:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:43:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:54:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:02:03] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-6 i-000004e1.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [12:07:02] RECOVERY dpkg-check is now: OK on wikidata-dev-6 i-000004e1.pmtpa.wmflabs output: All packages OK [12:14:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:16:30] Hola! Any of the puppet people here? I've got a question: Which items from the labs puppet stuff are displayed in the web interface of the labs console and can be selected there? There is much lesser to choose than in the actual git repo (operations/puppet). [12:17:22] PROBLEM Disk Space is now: WARNING on kubo i-000003dd.pmtpa.wmflabs output: DISK WARNING - free space: / 320 MB (3% inode=66%): [12:21:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [12:22:13] PROBLEM Disk Space is now: CRITICAL on kubo i-000003dd.pmtpa.wmflabs output: DISK CRITICAL - free space: / 286 MB (2% inode=66%): [12:24:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:36:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [12:44:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:51:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [12:54:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:12:53] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Critical: 3% free memory [13:14:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:25:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:44:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:55:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:14:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:25:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:44:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:53:46] hello , we need a project to develope and test farsi fonts, can someone help me ? [14:54:34] we need to run a mediawiki (accesible from outside) to test and deploy webfonts [14:55:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:56:21] mardetanha: What is your labs username, and what would you like the project called? [14:56:52] andrewbogott: mardetanha, and name:fawikitest [14:57:51] may I suggest 'farsi-wikitest' so that we can tell from the name what it's for? [14:58:41] andrewbogott: yes, sure [15:00:08] 11/26/2012 - 15:00:08 - Creating a project directory for farsi-wikitest [15:04:37] mardetanha: OK, you have a project with a quota for one public IP address. [15:04:45] There's documentation about mediawiki setup here: https://labsconsole.wikimedia.org/wiki/Help:Single_Node_MediaWiki [15:04:56] Be sure you read & understand the bit about security groups before forging ahead. [15:05:40] andrewbogott: sure :) [15:05:42] and thanks [15:05:56] np [15:06:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 4% free memory [15:14:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:16:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [15:25:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:30:57] PROBLEM Current Load is now: CRITICAL on fawikitest i-00000519.pmtpa.wmflabs output: Connection refused by host [15:31:10] PROBLEM Free ram is now: CRITICAL on fawikitest i-00000519.pmtpa.wmflabs output: Connection refused by host [15:31:37] PROBLEM Current Users is now: CRITICAL on fawikitest i-00000519.pmtpa.wmflabs output: Connection refused by host [15:32:13] PROBLEM Disk Space is now: CRITICAL on fawikitest i-00000519.pmtpa.wmflabs output: Connection refused by host [15:32:23] PROBLEM Total processes is now: CRITICAL on fawikitest i-00000519.pmtpa.wmflabs output: Connection refused by host [15:32:53] PROBLEM dpkg-check is now: CRITICAL on fawikitest i-00000519.pmtpa.wmflabs output: Connection refused by host [15:36:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [15:40:52] RECOVERY Current Load is now: OK on fawikitest i-00000519.pmtpa.wmflabs output: OK - load average: 0.42, 0.97, 0.68 [15:41:02] RECOVERY Free ram is now: OK on fawikitest i-00000519.pmtpa.wmflabs output: OK: 1056% free memory [15:41:32] RECOVERY Current Users is now: OK on fawikitest i-00000519.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [15:42:12] RECOVERY Disk Space is now: OK on fawikitest i-00000519.pmtpa.wmflabs output: DISK OK [15:42:22] RECOVERY Total processes is now: OK on fawikitest i-00000519.pmtpa.wmflabs output: PROCS OK: 84 processes [15:42:52] RECOVERY dpkg-check is now: OK on fawikitest i-00000519.pmtpa.wmflabs output: All packages OK [15:44:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:50:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:51:26] paravoid: in the mood for some LDAP surgery? ;) [15:51:44] LDAP user wants a rename and then also wants to have a gerrit account [15:51:54] he already has SVN [15:52:07] i've no idea how hard the rename part is. forget it if it's a pain [15:52:46] https://www.mediawiki.org/w/index.php?title=Developer_access&oldid=609321#User:SHL [15:53:11] (at any rate, I can't make the account myself) [15:53:17] but ^demon can! [15:53:57] oh, he's still 3+ days idle [15:56:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:56:32] ^demon: see the end of http://bots.wmflabs.org/~petrb/logs/%23wikimedia-labs/20121126.txt if you want to make me an account ;) [15:56:42] 26 15:52:46 < jeremyb> https://www.mediawiki.org/w/index.php?title=Developer_access&oldid=609321#User:SHL [15:57:24] Mard|awat: did you see the recent webfonts deployment? [15:57:30] (to prod) [15:57:55] jeremyb: yes, taxy is waiting for me, we will talk, when i back [15:58:14] https://gerrit.wikimedia.org/r/#q,I642e411b70da15efe769758a20e95211672bf902,n,z [15:58:25] Mard|awat: bye! be sure to ask sumanah for taxi stories [16:14:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:21:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:26:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:33:09] ^demon|sick: ugh. has chicken soup? [16:33:22] or at least something with turkey in it? ;) [16:34:00] * ^demon|sick needs soup. [16:34:02] <^demon|sick> And sleep. [16:34:08] <^demon|sick> Sleep will be good. [16:34:10] slap wel [16:34:41] * jeremyb redirects his account creation req @ robla ;) [16:34:51] https://www.mediawiki.org/w/index.php?title=Developer_access&oldid=609321#User:SHL [16:40:52] Hi andrewbogott! I think I just figured out howto get the extensions stuff shorter, like git::extension ("Bla":) [16:41:25] Silke_WMDE_: You mean by making all the include_once() stuff boilerplate? [16:41:38] huh? [16:41:47] no [16:42:05] I made a changes to generic-definitions.pp [16:42:42] and in the wikidata.pp file I can then skip half of the lines to clone extensions [16:43:27] Does that mean that there's mediawiki-specific stuff in generic-definitions? [16:45:01] no, but the git::clone stuf is defined there [16:45:44] sounds good then… go ahead and submit the change to gerrit so I can see what you mean :) [16:45:49] similar to what is already there, I added a definition for git::extension with defaults that are always the same [16:45:59] ok [16:46:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:49:33] PROBLEM Current Users is now: CRITICAL on mwreview-andrewdevel i-0000051b.pmtpa.wmflabs output: Connection refused by host [16:50:13] PROBLEM Disk Space is now: CRITICAL on mwreview-andrewdevel i-0000051b.pmtpa.wmflabs output: Connection refused by host [16:50:53] PROBLEM Current Load is now: CRITICAL on mwreview-andrewdevel i-0000051b.pmtpa.wmflabs output: Connection refused by host [16:50:53] PROBLEM Free ram is now: CRITICAL on mwreview-andrewdevel i-0000051b.pmtpa.wmflabs output: Connection refused by host [16:51:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:52:23] PROBLEM Total processes is now: CRITICAL on mwreview-andrewdevel i-0000051b.pmtpa.wmflabs output: Connection refused by host [16:54:33] PROBLEM dpkg-check is now: CRITICAL on mwreview-andrewdevel i-0000051b.pmtpa.wmflabs output: Connection refused by host [16:56:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:57:23] RECOVERY Total processes is now: OK on mwreview-andrewdevel i-0000051b.pmtpa.wmflabs output: PROCS OK: 84 processes [16:59:32] RECOVERY Current Users is now: OK on mwreview-andrewdevel i-0000051b.pmtpa.wmflabs output: USERS OK - 2 users currently logged in [16:59:32] RECOVERY dpkg-check is now: OK on mwreview-andrewdevel i-0000051b.pmtpa.wmflabs output: All packages OK [17:00:12] RECOVERY Disk Space is now: OK on mwreview-andrewdevel i-0000051b.pmtpa.wmflabs output: DISK OK [17:00:52] RECOVERY Current Load is now: OK on mwreview-andrewdevel i-0000051b.pmtpa.wmflabs output: OK - load average: 0.09, 0.62, 0.59 [17:00:53] RECOVERY Free ram is now: OK on mwreview-andrewdevel i-0000051b.pmtpa.wmflabs output: OK: 952% free memory [17:16:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:21:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:26:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:46:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:51:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:56:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:16:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:17:39] paravoid: so, how hard / trivial is an LDAP rename? i.e. should I just have sumanah do the creation as is or do you want to rename first? [18:21:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:22:22] PROBLEM Total processes is now: UNKNOWN on dumps-bot3 i-00000503.pmtpa.wmflabs output: NRPE: Call to fork() failed [18:24:32] PROBLEM Current Users is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:26:43] PROBLEM dpkg-check is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:26:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:26:53] PROBLEM Current Load is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:26:53] PROBLEM SSH is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Server answer: [18:27:23] PROBLEM Disk Space is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:27:23] PROBLEM Total processes is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:29:53] PROBLEM host: i-0000051c.pmtpa.wmflabs is DOWN address: i-0000051c.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000051c.pmtpa.wmflabs) [18:37:52] PROBLEM Free ram is now: WARNING on dumps-bot3 i-00000503.pmtpa.wmflabs output: Warning: 6% free memory [18:39:32] RECOVERY Current Users is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [18:40:12] RECOVERY host: i-0000051c.pmtpa.wmflabs is UP address: i-0000051c.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 0.66 ms [18:40:12] PROBLEM dpkg-check is now: CRITICAL on mwreview-andrewdevel2 i-0000051c.pmtpa.wmflabs output: Connection refused by host [18:40:52] PROBLEM Current Load is now: CRITICAL on mwreview-andrewdevel2 i-0000051c.pmtpa.wmflabs output: Connection refused by host [18:41:32] PROBLEM Current Users is now: CRITICAL on mwreview-andrewdevel2 i-0000051c.pmtpa.wmflabs output: Connection refused by host [18:41:42] RECOVERY dpkg-check is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: All packages OK [18:41:52] RECOVERY Current Load is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: OK - load average: 0.29, 0.63, 0.70 [18:41:52] RECOVERY SSH is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [18:42:12] PROBLEM Disk Space is now: CRITICAL on mwreview-andrewdevel2 i-0000051c.pmtpa.wmflabs output: Connection refused by host [18:42:22] RECOVERY Disk Space is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: DISK OK [18:43:02] PROBLEM Free ram is now: CRITICAL on mwreview-andrewdevel2 i-0000051c.pmtpa.wmflabs output: Connection refused by host [18:43:04] welp no more sumanah ;( [18:43:14] it's noisy in here [18:44:22] PROBLEM Total processes is now: CRITICAL on mwreview-andrewdevel2 i-0000051c.pmtpa.wmflabs output: Connection refused by host [18:48:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:51:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:54:22] RECOVERY Total processes is now: OK on mwreview-andrewdevel2 i-0000051c.pmtpa.wmflabs output: PROCS OK: 84 processes [18:55:53] RECOVERY Current Load is now: OK on mwreview-andrewdevel2 i-0000051c.pmtpa.wmflabs output: OK - load average: 0.18, 0.80, 0.62 [18:55:53] RECOVERY dpkg-check is now: OK on mwreview-andrewdevel2 i-0000051c.pmtpa.wmflabs output: All packages OK [18:56:33] RECOVERY Current Users is now: OK on mwreview-andrewdevel2 i-0000051c.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [18:56:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:57:13] RECOVERY Disk Space is now: OK on mwreview-andrewdevel2 i-0000051c.pmtpa.wmflabs output: DISK OK [18:57:53] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Critical: 5% free memory [18:58:03] RECOVERY Free ram is now: OK on mwreview-andrewdevel2 i-0000051c.pmtpa.wmflabs output: OK: 794% free memory [19:18:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:21:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:27:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:48:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:51:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:57:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:11:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 9% free memory [20:18:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:21:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:25:36] 11/26/2012 - 20:25:35 - Updating keys for ant at /export/keys/ant [20:28:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:33:16] Hi. [20:48:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:51:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:59:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:05:28] 11/26/2012 - 21:05:27 - Updating keys for ant at /export/keys/ant [21:18:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:21:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:29:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:49:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:51:44] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:59:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:07:15] anyone know why beta labs has been crippled for a while? I just went to /home/wikipedia/logs and nothing has been updated there since Nov 22 [22:19:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:22:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:30:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:42:43] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c.pmtpa.wmflabs output: Warning: 19% free memory [22:49:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:52:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:55:59] chrismcmahon: probably the udp2log daemon has an issue, it is running on deployment-dbdump [22:56:01] restarted it [22:58:20] and memcached does not seems to work [22:58:39] might be fixed with https://gerrit.wikimedia.org/r/35318 [23:00:23] bed time for me, have a good day [23:00:36] chrismcmahon: logs are working again [23:00:38] cya! [23:00:44] night hashar [23:00:52] thanks hashar [23:00:59] * chrismcmahon wonders if that fixed the other beta issues [23:01:05] chrismcmahon: still have to get mediawiki-config updated though [23:01:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:01:23] might be doing that tomorrow morning [23:01:30] it just to late to do it right now :( [23:01:38] chris might be able to update the conf on beta though [23:01:44] and merge the above change ( 35318 ) [23:01:49] bed for real [23:01:49] !! [23:01:49] petan needs a new hobby :P [23:04:29] oh hey, 35318 memcache is broken again I guess [23:07:33] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 155 processes [23:07:43] RECOVERY Free ram is now: OK on bots-2 i-0000009c.pmtpa.wmflabs output: OK: 20% free memory [23:08:06] *BOOM* [23:19:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:22:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:30:42] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c.pmtpa.wmflabs output: Warning: 19% free memory [23:31:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:49:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:52:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:59:22] PROBLEM Free ram is now: CRITICAL on bots-cb i-0000009e.pmtpa.wmflabs output: Critical: 4% free memory