[00:01:27] Change on 12mediawiki a page Developer access was modified, changed by Mono link https://www.mediawiki.org/w/index.php?diff=596903 edit summary: [00:19:17] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:26:08] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:27:36] meh [00:27:49] hem [00:49:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:56:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:00:25] PROBLEM Disk Space is now: UNKNOWN on wikidata-dev-5 i-000004d3.pmtpa.wmflabs output: Invalid host name i-000004d3.pmtpa.wmflabs [01:01:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 6.95, 6.48, 5.56 [01:05:13] PROBLEM Disk Space is now: CRITICAL on wikidata-dev-5 i-000004d3.pmtpa.wmflabs output: Connection refused by host [01:12:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: WARNING - load average: 6.00, 5.52, 5.21 [01:19:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:26:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:36:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 8.55, 7.45, 5.85 [01:46:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.98, 4.34, 4.87 [01:49:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:56:55] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:57:35] RECOVERY Current Users is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [02:09:03] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4High - 6normal) [Bug 40526] new security rule not applied - https://bugzilla.wikimedia.org/show_bug.cgi?id=40526 [02:19:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:27:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:41:43] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory [02:44:55] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.11, 5.90, 5.27 [02:49:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:54:42] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 19% free memory [02:57:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:19:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:28:26] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:37:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 4.38, 4.89, 5.22 [03:37:54] RECOVERY Free ram is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: OK: 32% free memory [03:45:36] PROBLEM Current Users is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in [03:49:26] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:59:05] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:04:52] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Warning: 15% free memory [04:19:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:26:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.67, 4.15, 4.69 [04:29:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:29:52] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Critical: 4% free memory [04:34:52] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f.pmtpa.wmflabs output: OK: 95% free memory [04:35:37] "(Remove host name)" function in Special:NovaAddress is broken? [04:35:53] it says "The requested host does not exist. " on removal [04:49:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:54:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 4.91, 5.37, 5.05 [04:59:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:15:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 5.76, 5.82, 5.39 [05:19:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:29:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:44:55] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.86, 4.49, 4.96 [05:45:32] RECOVERY Current Users is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [05:49:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:00:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:12:55] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 4.92, 5.49, 5.41 [06:19:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:30:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:31:33] PROBLEM Total processes is now: WARNING on aggregator-test1 i-000002bf.pmtpa.wmflabs output: PROCS WARNING: 154 processes [06:35:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 4.06, 4.32, 4.88 [06:48:46] RECOVERY Disk Space is now: OK on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: DISK OK [06:49:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:51:32] RECOVERY Total processes is now: OK on aggregator-test1 i-000002bf.pmtpa.wmflabs output: PROCS OK: 150 processes [07:01:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:02:57] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.78, 4.72, 4.99 [07:02:57] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: OK - load average: 3.95, 4.19, 4.80 [07:19:36] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:21:15] hello [07:25:45] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: WARNING - load average: 4.97, 5.34, 5.14 [07:31:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:31:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 5.08, 4.93, 5.04 [07:36:42] PROBLEM Disk Space is now: WARNING on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: DISK WARNING - free space: / 78 MB (5% inode=51%): [07:36:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 5.13, 4.87, 4.97 [07:49:46] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:55:54] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: OK - load average: 5.17, 4.80, 4.96 [08:01:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:19:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:31:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:49:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:56:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: WARNING - load average: 6.21, 5.59, 5.21 [09:01:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:10:38] !log wikidata-dev wikidata-dev3 Adjusted cronjob for http://wikidata-docs.wikimedia.de to cover PHP and JS documentation. [09:10:40] Logged the message, Master [09:12:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 5.67, 5.57, 5.67 [09:19:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:26:53] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: OK - load average: 4.10, 4.18, 4.74 [09:31:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:32:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.37, 4.52, 4.88 [09:49:55] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:55:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 4.00, 5.00, 5.07 [10:02:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:20:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.78, 4.52, 4.95 [10:21:07] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:32:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:41:42] RECOVERY Disk Space is now: OK on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: DISK OK [10:49:42] PROBLEM Disk Space is now: WARNING on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: DISK WARNING - free space: / 78 MB (5% inode=51%): [10:51:53] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:03:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:04:47] !ping [11:04:48] pong [11:05:40] !htmllogs [11:05:40] experimental: http://bots.wmflabs.org/~wm-bot/html/%23wikimedia-labs [11:08:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 4.31, 4.87, 5.09 [11:09:45] RECOVERY Disk Space is now: OK on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: DISK OK [11:21:54] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:33:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:52:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:56:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: WARNING - load average: 10.88, 7.91, 5.88 [12:01:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 7.44, 7.66, 6.21 [12:03:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:03:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.38, 6.38, 5.33 [12:03:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 19% free memory [12:10:21] hashar: found the reason that some extensions where not updated https://gerrit.wikimedia.org/r/#/c/29615/ [12:10:36] git submodule foreach stops if one subcommand fails [12:11:59] Change on 12mediawiki a page Developer access was modified, changed by Bean49 link https://www.mediawiki.org/w/index.php?diff=597096 edit summary: /* User:Bean49 */ new section [12:12:43] Change on 12mediawiki a page Developer access was modified, changed by Bean49 link https://www.mediawiki.org/w/index.php?diff=597097 edit summary: [12:22:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:33:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:42:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: WARNING - load average: 5.62, 5.72, 5.19 [12:52:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: OK - load average: 3.46, 4.58, 4.91 [12:54:14] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:03:56] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:05:35] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 195 processes [13:10:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 102 processes [13:24:15] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:34:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:54:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:04:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:24:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:34:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:34:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: WARNING - load average: 6.46, 6.27, 5.58 [14:39:09] o`_´o Hm. hi! [14:39:38] I made my puppet playground a puppetmaster::self via the labsconsole/webinterface. [14:40:14] Now that somebody has merged changes how can I trigger my dear server to get the new stuff? [14:41:17] I think the incantation was: sudo puppetd -atv [14:42:42] PROBLEM Disk Space is now: WARNING on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: DISK WARNING - free space: / 78 MB (5% inode=51%): [14:42:43] (I mean changes from gerrit) [14:47:21] git pull [14:47:26] Platonides: That applies my local puppetmaster to my machine. But I would like to get news from gerrit. I tried to change the fetch URL to my login, then su from root to myself. [14:47:52] ...and then git pull complains about publickey denial stuff. [14:48:55] you seem to have an odd configuration [14:49:00] how do you have that setup? [14:50:10] Silke_WMDE: https://labsconsole.wikimedia.org/wiki/Help:Self-hosted_puppetmaster you are looking for that? [14:50:25] cd /var/lib/git/operations/puppet [14:50:26] sudo GIT_SSH=/var/lib/git/ssh git pull --rebase [14:52:45] j^: ah -> That's probably what I'm looking for. Only: it also says "Permission denied (publickey)." [14:54:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:55:10] On my laptop git pull works and as I understand it, I should take my ssh key with me on the labs server. [14:55:12] if you are running git as root, your agent won't allow access to your public key [14:55:30] but if the git folder is owned by root a pull as yourself won't work [14:55:43] ok, let me check [14:55:43] the easiest way is probably to pull from the https url [14:56:15] ok [14:57:05] hm. i made me the owner of the folder and tried with my own login -> still the same error [14:58:25] Silke_WMDE: you have to login with ssh key forwarded and run sudo git as you [14:59:04] j^: i thought that was what I did ... *doubt* [15:00:59] * Silke_WMDE is double checking [15:04:36] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 195 processes [15:04:36] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:16:51] !ping [15:16:51] pong [15:16:58] @infobot-detail ping [15:16:58] Info for ping: this key was created at N/A by N/A, this key was displayed 48 time(s), last time at 10/24/2012 3:16:51 PM (00:00:07.0580430 ago) [15:24:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:29:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 102 processes [15:29:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: OK - load average: 3.22, 4.29, 4.87 [15:34:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:54:37] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:54:51] Silke_WMDE: Did my changes to role::mediawiki::labs ruin your work in progress? [15:55:05] :) [15:55:34] I take that as a yes :( [15:55:40] Well.. indirectly. I can't get your changes pulled to my puppetmaster in labs. [15:56:16] I could only look at them on my laptop. [15:56:21] Oh… yeah, I've had that problem as well. Lemme see if I can figure out how to do it and document. [15:56:38] I mean, not w/regard to that change in particular, just with rebasing in general. [15:56:48] it is documented [15:57:13] https://labsconsole.wikimedia.org/wiki/Help:Self-hosted_puppetmaster (bottom) [15:57:27] But, generally: you won't be able to develop properly in /etc/puppet because of file permissions. So I tend to keep my own copy of puppet in ~ or /data/project and then copy files over as I edit them. [15:58:02] Hm, it is documented! That must be newish [16:01:03] andrewbogott: you mean ~ on your labs instance? [16:01:09] right. [16:01:18] ok [16:01:36] That has the advantage of being project-wide, which means you can always burn down your test instance and start a new one without losing ~/puppet [16:01:51] If /etc/puppet becomes hopelessly mangled :) [16:01:56] o_O Yeah, cool! [16:02:30] So you also merge directly from your labs instance [16:03:02] Right, but from a repo that's separate from the one the puppetmaster is using. [16:04:28] (btw, I'm also trying to run through Hashar's instructions on that page to see what the deal is.) [16:04:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:06:10] I liked the idea to tell a mediawiki via puppet if it should get updates from git or not! On our Wikidata instances we have both cases. [16:07:08] I haven't tested the auto-update class much, but it seems to work OK. [16:08:13] I need to change it to rebase instead of pull, as it is I think bad things will happen to people who want to test a patch in an auto-updated repo [16:08:47] So far the biggest issue for me is this timeout problem (if it really is a timeout problem) when puppet tries to get mediawiki core. [16:08:56] PROBLEM Current Load is now: CRITICAL on testlabs-abogott i-000004df.pmtpa.wmflabs output: Connection refused by host [16:09:19] Silke_WMDE: Is that still happening? We upped the timeout and I changed the checkout to do --depth=1 which speeds things up. [16:09:33] PROBLEM Current Users is now: CRITICAL on testlabs-abogott i-000004df.pmtpa.wmflabs output: Connection refused by host [16:10:14] PROBLEM Disk Space is now: CRITICAL on testlabs-abogott i-000004df.pmtpa.wmflabs output: Connection refused by host [16:10:52] PROBLEM Free ram is now: CRITICAL on testlabs-abogott i-000004df.pmtpa.wmflabs output: Connection refused by host [16:11:21] I didn't have you latest version. I just played around with the timeout a bit but had no reliability. [16:11:41] What does depth 1 do? depth in the sense of folders?? [16:12:51] depth=1 means it doesn't clone the whole repo, only the most recent patchset. [16:13:00] ah, cool [16:13:25] That may have unintended consequences… but since mostly people aren't using that repo to develop in anyway I think it won't matter…. [16:13:29] I /hope/ it won't matter [16:13:40] :) [16:13:52] RECOVERY Current Load is now: OK on testlabs-abogott i-000004df.pmtpa.wmflabs output: OK - load average: 0.18, 0.76, 0.53 [16:13:57] how did you do your setup in ssh/config on a labs instance to talk to gerrit? [16:14:32] RECOVERY Current Users is now: OK on testlabs-abogott i-000004df.pmtpa.wmflabs output: USERS OK - 1 users currently logged in [16:14:39] For me, the ssh agent forwarding doesn't work. [16:15:15] RECOVERY Disk Space is now: OK on testlabs-abogott i-000004df.pmtpa.wmflabs output: DISK OK [16:15:57] RECOVERY Free ram is now: OK on testlabs-abogott i-000004df.pmtpa.wmflabs output: OK: 1052% free memory [16:16:30] It 'just works' for me... [16:16:45] If you are able to get from bastion to your instance then your key-forwarding is working, at least the first time. [16:16:56] I presume you are doing 'ssh -A ' from bastion? [16:17:24] yes [16:17:51] OK. Give me a minute, I'm almost set up with a fresh puppetmaster:self instance, then I'll test. [16:24:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [16:34:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:35:08] Um… bad news, Silke_WMDE, the instructions on that page for updating puppet worked just fine for me. [16:35:19] Are you able to sudo on your instance? [16:35:33] Yes. And on labs "ssh smeyer@gerrit.wikimedia.org -p 29418" is successful [16:36:18] But the "sudo GIT_SSH=/var/lib/git/ssh git pull --rebase" command fails with a public key error. [16:39:00] hashar, can you help us out with this? (I'm assuming you set up this stuff) [16:39:26] sorry just checking mails and about to go cooking the dinner [16:39:34] is the ssh key in gerrit ? [16:40:01] ohh that is puppet master self [16:40:03] Got it [16:40:16] Got it? You mean it works now? [16:40:22] maybe try to export GIT_SSH? export GIT_SSH=/var/lib/git/ssh [16:40:26] then git pull [16:40:37] yes, it works [16:40:41] Oh! What changed? [16:41:10] puppetmaster::self is a nice trick but it could some helper scripts to easily update and or apply some patchsets [16:41:25] off for dinner :-] [16:41:28] Erm... Before, when it didn't work, I played with my ssh config and with the git config in the folder [16:41:34] hashar: Thanks for checking in! [16:41:54] I forgot to revert one of those changes. [16:41:59] m) [16:42:07] ah shit [16:42:43] Sorry for the confusion [16:42:47] cool! OK, so that means you're off and running? [16:42:49] Yes [16:43:02] Thanks for your help! [16:43:16] yep! [16:46:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: WARNING - load average: 5.81, 5.71, 5.23 [16:57:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [17:04:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:07:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 195 processes [17:12:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 102 processes [17:29:12] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [17:31:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: OK - load average: 3.34, 3.83, 4.61 [17:33:33] PROBLEM Current Users is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: USERS WARNING - 6 users currently logged in [17:34:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:38:01] <^demon> Ryan_Lane: You're off the hook tomorrow for precise upgrades :) [17:41:09] ^demon: I saw that in the scrollback :) [17:41:26] <^demon> Faidon's gonna help me and Antoine out. [17:47:44] * Damianz gives ^demon a c00k1e [17:49:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4 i-000004d7.pmtpa.wmflabs output: WARNING - load average: 6.32, 6.17, 5.40 [17:56:08] 10/24/2012 - 17:56:02 - User j may have been modified in LDAP or locally, updating key in project(s): mediahandler,deployment-prep,swift [17:56:22] 10/24/2012 - 17:56:19 - Updating keys for j at /export/keys/j [17:59:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [18:03:12] !log deployment-prep csteipp: updated MapSources extension to latest for wikivoyage [18:05:26] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:19:17] [bz] (8NEW - created by: 2Arthur Richards, priority: 4Unprioritized - 6normal) [Bug 40605] Supporting MobileFrontend on beta labs - https://bugzilla.wikimedia.org/show_bug.cgi?id=40605 [18:20:02] So... is Wikidata and the Wikibase extension actually the same project in terms of code repo? [18:20:20] aude: ^ :) [18:21:00] qgil: Wikidata is an old repo [18:21:12] Only care about Wikibase and WikibaseSolr [18:21:50] Reedy aude we have https://www.ohloh.net/p/wikibase/enlistments and https://www.ohloh.net/p/wikidata/enlistments [18:22:12] Can they be merged? Old repos are ok for history and stats [18:22:37] The code was migrated [18:22:56] <^demon> qgil: Also, for extra fun, "Wikidata" in SVN is an entirely different project. [18:23:00] :) [18:23:13] Yeah, they're the same [18:23:25] StickToThatLanguage is also deprecated/abandoned now too [18:23:25] Is https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/extensions/Wikibase.git the right place? [18:23:43] <^demon> Yes. [18:23:55] oook, thanks [18:24:44] And I guess "Wikibase" is the project name that should prevail, Reedy ? [18:25:05] qgil: just so you know, we're in the middle of a deployment window for the next 90 min [18:25:27] ah ok, sorry for the noise ! [18:25:51] qgil: not sure [18:26:02] it's still called the wikidata project... [18:26:17] you guys are funny ;) [18:26:51] <^demon> qgil: We're bad at naming things. Exhibit A: http://www.mediawiki.org/wiki/Wikipmediawiki [18:27:04] :D [18:28:52] RECOVERY Disk Space is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: DISK OK [18:29:13] <^demon> Ryan_Lane: https://gerrit.wikimedia.org/r/#/c/28351/ is cool. It removes one more usage of the gerrit2 ldap account. [18:29:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [18:29:32] RECOVERY Current Users is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [18:29:42] PROBLEM Free ram is now: WARNING on aggregator2 i-000002c0.pmtpa.wmflabs output: Warning: 8% free memory [18:30:24] RECOVERY Current Load is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: OK - load average: 0.13, 0.43, 0.57 [18:30:53] RECOVERY SSH is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [18:31:52] RECOVERY dpkg-check is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: All packages OK [18:32:19] ^demon: are the jenkins tests already going for puppet lint? [18:32:23] I don't want to merge this until then :) [18:32:34] RECOVERY Total processes is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: PROCS OK: 229 processes [18:33:06] PS: https://www.ohloh.net/p/wikidata won - thank you [18:33:38] <^demon> Ryan_Lane: I copied over the shell script for linting to jenkins and tested it out. [18:33:50] <^demon> See the first comment on that change. [18:34:48] did you test it with a failure? :) [18:35:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:36:34] <^demon> Nobody writes bad code :p [18:36:58] heh [18:37:01] ok. I'll merge this in [18:37:39] I'm more than happy to get rid of this code in the hooks [18:37:47] the hooks are too damn complicated as is [18:37:53] Ryan_Lane, I read your email about not giving out editinterface [18:38:07] yep [18:38:07] Has the WMF ever considered restricting it in production for similar reasons with steward/OS/CU accounts? [18:38:11] no clue [18:38:23] I think there's enough people looking for malicious changes there [18:38:52] it's more difficult on labsconsole and it's *way* more dangerous [18:38:55] ^demon: change from 2.4.2.1-2 to 2.4.2-1 failed ? [18:38:56] eh [18:38:58] wtf? [18:39:22] do we have a specific version listed in puppet? [18:39:23] heh [18:39:25] we shouldn't [18:39:34] we do that manually anyway [18:39:41] <^demon> We have 2.4.2-1 listed in puppet. [18:39:52] <^demon> 2.4.2.1-2 must've been the custom one you deployed. [18:40:03] yeah [18:40:08] Im going to change that to ensure => "present" [18:40:21] <^demon> Ok, sounds good. [18:40:21] well, ensure => present [18:50:51] <^demon> Ryan_Lane: Ok, enabled the job on jenkins so you'll get your feedback from there now. [18:51:29] ^demon: can you send that info to the ops list? [18:51:38] Having your cloudadmin cookies stolen via js would suck [18:51:39] <^demon> Yeah [18:52:04] Though we really have general lack of ssl potential stealing happening on wikipedia anyway :P [18:59:24] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [19:04:43] PROBLEM Free ram is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:05:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 195 processes [19:05:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:05:32] PROBLEM Total processes is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:06:52] PROBLEM Disk Space is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:07:32] PROBLEM Current Users is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:08:22] PROBLEM Current Load is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:08:52] PROBLEM SSH is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: Server answer: [19:09:52] PROBLEM dpkg-check is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:29:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [19:36:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:59:30] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [20:00:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 288 processes [20:06:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:14:42] PROBLEM Free ram is now: WARNING on aggregator2 i-000002c0.pmtpa.wmflabs output: Warning: 8% free memory [20:14:53] RECOVERY dpkg-check is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: All packages OK [20:15:33] RECOVERY Total processes is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: PROCS OK: 228 processes [20:16:53] RECOVERY Disk Space is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: DISK OK [20:17:33] RECOVERY Current Users is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [20:18:23] RECOVERY Current Load is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: OK - load average: 0.10, 0.36, 0.56 [20:18:57] RECOVERY SSH is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [20:29:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [20:36:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:48:47] Change on 12mediawiki a page Developer access was modified, changed by Theopolisme link https://www.mediawiki.org/w/index.php?diff=597242 edit summary: [20:54:42] PROBLEM Free ram is now: UNKNOWN on aggregator2 i-000002c0.pmtpa.wmflabs output: NRPE: Call to fork() failed [20:55:32] PROBLEM Current Users is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:56:22] PROBLEM Current Load is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:57:52] PROBLEM dpkg-check is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:58:32] PROBLEM Total processes is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:59:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [20:59:42] PROBLEM Free ram is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:01:53] PROBLEM SSH is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: Server answer: [21:01:53] PROBLEM Disk Space is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [21:06:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:09:49] * Damianz waves at Sharihareswara [21:13:18] hi Damianz! [21:16:42] PROBLEM Disk Space is now: WARNING on testing-arky i-0000033b.pmtpa.wmflabs output: DISK WARNING - free space: / 78 MB (5% inode=51%): [21:28:13] hm [21:28:21] I can't ssh into catsort-pub [21:28:33] it's returning via salt, though [21:28:37] no sorting the cats then [21:28:38] and its security rules look fine [21:29:06] and I can ping it [21:29:32] sshd is running right [21:29:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [21:29:36] checking [21:29:37] and/or installed [21:29:55] it isn't [21:30:08] that would be your issue then [21:30:16] i-000001cc.pmtpa.wmflabs: /bin/sh: 1: /etc/init.d/sshd: not found [21:30:18] wtf [21:30:23] * Damianz points to the debugging chart of checking simple shit first :) [21:30:30] ah [21:30:30] ssh [21:30:59] because ubuntu has such an awesome naming system [21:31:01] i-000001cc.pmtpa.wmflabs: initctl:dbus_error.c:69: Unhandled error from nih_dbus_error_raise: Method "Get" with signature "ss" on interface "org.freedesktop.DBus.Properties" doesn't exist [21:31:13] wtf? [21:31:32] probably network manager being screwy [21:31:42] I wonder... [21:31:43] ah [21:31:45] Did someone dist upgrade this box [21:31:48] yep [21:31:52] and it failed [21:31:57] kill them with fire [21:32:35] Sadly bastion drops connections too often to do crazy stuff like distupgrades sensibly :( [21:32:58] it's not just that [21:33:10] there's a million things that could go wrong [21:33:33] well... the solution is to make puppet more accessible and useable within labs so people are happy to re-install boxes [21:33:56] yep [21:33:58] so [21:34:08] the upgrade will start ssh on another port [21:34:14] which you can't access [21:34:16] but that port isn't open [21:34:23] exactly [21:35:07] I'm still a bit meh towards security rules esp when it comes to public ips. [21:36:38] I think they are needed [21:36:52] Oh I think they're needed [21:37:27] Personally for public ips I'd either use a seperate security group (probably not possible) or terminate them on locked down machines and nat back to vms.. I think exposing the likes of ssh on the public addresses is stupid [21:37:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:37:53] It opens up a potential for external users to gain access without aproval and cause issues... though we don't block outgoing ports so people can drop shells anyway [21:40:42] hm [21:40:45] I may be able to fix this instance [21:40:49] salt to the rescue! :) [21:41:12] Damianz: ah. I see what you mean [21:44:16] well, really, it doesn't matter much [21:44:27] nearly all wikimedia servers are accessible via ssh [21:44:36] we require keys [21:44:42] and we keep the instances up to date automatically [21:44:51] we have a right that's required to have shell access [21:44:52]