[00:13:38] !fingerprints [00:13:38] ssh keys for bastion hosts: https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints [00:35:18] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#1125310 (10bd808) [00:36:43] 10Tool-Labs-tools-Other: Fix tool kmlexport - https://phabricator.wikimedia.org/T92963#1125310 (10bd808) 05Open>03stalled Marking as stalled pending a response from @para on OSI-approved license selection. [02:31:52] 10Tool-Labs-tools-Other, 10Phabricator: Stashbot shouldn't subscribe itself to tasks - https://phabricator.wikimedia.org/T135790#2311403 (10bd808) p:05Triage>03Low [02:52:48] PROBLEM - Free space - all mounts on tools-worker-1004 is CRITICAL: CRITICAL: tools.tools-worker-1004.diskspace.root.byte_percentfree (<20.00%) [04:27:27] 06Labs, 10Tool-Labs: categorylinks tables on Tool Labs are missing rows - https://phabricator.wikimedia.org/T135801#2311533 (10MZMcBride) [04:33:29] RECOVERY - Puppet run on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:34:23] 06Labs, 10Tool-Labs: categorylinks tables on Tool Labs are missing rows - https://phabricator.wikimedia.org/T135801#2311547 (10MZMcBride) ``` MariaDB [enwikisource_p]> SELECT -> cl_to -> FROM page -> JOIN categorylinks -> ON cl_from = page_id -> WHERE page_id = cl_... [04:43:09] 06Labs, 10Tool-Labs: categorylinks tables on Tool Labs are missing rows - https://phabricator.wikimedia.org/T135801#2311549 (10MZMcBride) Poking at this a bit further, returns: ``` { "batchcomp... [04:44:46] 06Labs, 10Tool-Labs, 10DBA: categorylinks tables on Tool Labs are missing rows - https://phabricator.wikimedia.org/T135801#2311555 (10MZMcBride) [06:06:15] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:35:11] 06Labs, 10Beta-Cluster-Infrastructure, 06Operations, 10Traffic: deployment-cache-upload04 (m1.medium) / is almost full - https://phabricator.wikimedia.org/T135700#2311660 (10Joe) @hashar that was exactly my plan [08:10:57] 06Labs, 10Labs-Infrastructure, 06Operations: investigate slapd memory leak - https://phabricator.wikimedia.org/T130593#2311787 (10MoritzMuehlenhoff) [08:16:47] mh puppet failures [08:34:52] PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228) [09:19:07] !log Wikidata-dev disabled nfs for the project, if you want it you can enable it per host again [09:19:07] Wikidata-dev is not a valid project. [09:19:15] !log wikidata-dev disabled nfs for the project, if you want it you can enable it per host again [09:19:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL, Master [09:54:13] jzerebecki: you got puppet failures too? at least I know it's not something I screwed up [09:55:44] Does anyone know if we should worry about this or not? The message seems kinda scary lol [09:59:09] hello i got an email about puppet failing on the maps-warper instance. looking at graphite it appears to not have worked also yesterday, with what looks like every 30 mins spikes of load to 1. I'm running "sudo puppet agent -tv" currently. Is this a known issue across many instances? [09:59:31] also, a "df -h" seemed to hang [10:00:13] 10PAWS, 10Jupyter-Hub: I can't login my bot in JUPYTER - https://phabricator.wikimedia.org/T135306#2311942 (10Maathavan) @yuvipanda It doesn't works. same reply. [10:00:32] chippy both I and jzerebecki are having issues with puppet, but no one is awake yet considering it's only 5:59 eastern, and 2:59 pacific where the foundation is [10:00:55] AmandaNP, okay thanks :-) I will scale down my poking around in the server [10:01:22] I tried looking for my puppet logs, apparently they don't exist [10:01:32] or i'm looking in the wrong place [10:01:44] I can see some in /var/log/puppet [10:02:18] ya that's completely empty for me [10:05:00] hmm, I dunno [10:06:10] i tried running it manually, but it hang, and in the log it said it was skipping as it was running on a pid, however, no proceess was running on that pid [10:06:11] 2016-05-19 10:15 hashar: puppet broken on deployment-tin : ?[1;31mError: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter trusted_group on node deployment-tin.deployment-prep.eqiad.wmflabs?[0m [10:06:28] 2016-05-20 05:58 bblack: gdnsd stopped on eeden.esams, puppet disabled [10:06:48] chippy, AmandaNP: an nfs server is down, that makes the puppet run hang [10:07:04] ahh [10:07:21] is it known to be down or do we need to report it? [10:07:23] jzerebecki, yeah, I suspected so. Do you think this is would be why "df -h" is also hanging? because its trying to connect to nfs mounts? [10:07:30] its known, in channel topic [10:07:41] " Normal but /public/dumps is currently offline for maint" [10:08:15] yeah df -hl is quick [10:08:15] yes df -h hangs because of that [10:08:50] well this saved me from the heart attack I almost had waking up :P [10:10:36] I noticed the regular pattern in the load via http://graphite.wmflabs.org/dashboard/#maps_warper1 yesterday but the email got me more worried [10:19:53] 10Tool-Labs-tools-Other, 07I18n: [[Wikimedia:Pageviews-api-incomplete-data/en]] i18n issue - https://phabricator.wikimedia.org/T135817#2311990 (10Liuxinyu970226) [10:20:19] 10Tool-Labs-tools-Other, 07I18n: [[Wikimedia:Pageviews-api-incomplete-data/en]] i18n issue - https://phabricator.wikimedia.org/T135817#2311923 (10Liuxinyu970226) [11:05:23] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:21:16] something is eating up labsdb1003's free disk space [11:28:20] catscan2? [11:29:29] it is back to okish levels, but a cleanup is needed, starting by u3532__ and s51187__xtools_tmp [11:40:17] RECOVERY - Puppet run on tools-docker-builder-03 is OK: OK: Less than 1.00% above the threshold [0.0] [13:34:14] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:14:11] I'm trying to serve my tool's webservice from a different directory, and I tried symlinks as bd808 suggested, but they don't work correctly, and I tried to set the document root in the .lighttpd.conf config file, but my tool's webservice exits with 'Duplicate config variable in conditional 0 global: server.document-root'. I think this means that it's conflicting with the default configuration. How can I fix this? [14:23:26] 06Labs, 10DBA: categorylinks tables on Tool Labs are missing rows - https://phabricator.wikimedia.org/T135801#2312698 (10Krenair) [14:23:41] 06Labs, 10DBA: categorylinks tables on Labs replicas are missing rows - https://phabricator.wikimedia.org/T135801#2311533 (10Krenair) [14:26:37] 06Labs, 10DBA: categorylinks tables on Labs replicas are missing rows - https://phabricator.wikimedia.org/T135801#2311533 (10Krenair) That query returns the same thing against the production master server [14:57:12] tom29739: what went wrong when you tried to use a symlink? [14:58:40] bd808, it made the symlink, but it was in the folder. [14:59:39] I'm not sure I understand what that means [14:59:44] So I wanted to symlink the public folder to the public_html folder. But it didn't show the files in the public folder when I looked inside the public_html folder, like it should have done. [15:00:12] It created a folder inside the public_html folder, so ~/public_html/public/index.php [15:00:23] Rather than ~/public_html/index.php [15:00:41] The second is what I wanted it to do, the first is what it did. [15:00:55] Ah. Did you move the public_html directory out of the way before making the symlink? [15:01:02] No. [15:01:10] that's probably what bit you [15:01:37] ln -s ~/foo ~/public_html would make a ~/public_html/foo if public_html is a directory [15:02:06] that's ln trying to be "helpful" [15:02:52] 06Labs, 10Labs-Infrastructure: Puppet is hanging on labvirt1003 - https://phabricator.wikimedia.org/T135850#2312847 (10Andrew) [15:03:00] so try `mv ~/public_html ~/public_html.bak; ln -s ~/whatever/public ~/public_html` [15:03:26] you can rmdir ~/public_html too if you know that there is nothing worth having inside it [15:04:18] It seems to have worked. [15:04:46] * tom29739 tries to make lighttpd to work with it... [15:05:48] tom29739: can you write up a short explanation on wikitech? This seems like something that others will want to do at some point [15:06:22] Sure. Just a paragraph in Help:Tool Labs/Web ? [15:07:11] yeah. that would be great start [15:08:00] I'm working up the courage to send an email labs-l asking for help writing/editing docs [15:08:20] We really need more docs that don't assume that the reader is a unix guru [15:09:34] Some of the docs need you to be familiar with the toollabs way of working too. [15:10:04] I come from a windows background, so some of the stuff is just alien. [15:11:31] Wow, the headings are messed up on that page... [15:28:41] bd808, this any good: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web [15:28:53] I don't pride myself of documentation writing. [15:29:00] *on [15:30:30] LGTM tom29739. Thanks for that [15:30:42] :) [15:30:45] and everyone thinks their own writing sucks :) [15:31:07] the main point is to get things on-wiki where they can be found and improved on [15:32:22] Someone should make a glossary or something like that for it. Because newbies may not know what terms (e.g. grid) mean. [15:33:09] PROBLEM - Host tools-mail is DOWN: CRITICAL - Host Unreachable (10.68.16.27) [15:33:36] Agreed. there are lot of places to start making things better. [15:33:46] This might be a place to add terms -- https://wikitech.wikimedia.org/wiki/Help:Terminology [15:34:14] that's a Labs wide help page, but having N glossaries seems less useful [15:35:35] Often, it's not the docs not being there, it's finding them that's the trouble. [15:35:45] yes [15:36:29] I'm not a huge fan of the way that some of the doc pages are transcluded as sections of the main help page either [15:36:41] it leads to some editing difficulties [15:38:03] RECOVERY - Host tools-mail is UP: PING OK - Packet loss = 0%, RTA = 2.24 ms [15:45:21] valhallasw`cloud, legoktm two wikibugs? [15:49:02] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Temporarily Unavailable - string 'Magnus' not found on 'http://tools.wmflabs.org:80/' - 383 bytes in 0.003 second response time [15:49:44] hmm [15:49:54] this is just admin page moving i bet [15:57:10] 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Puppet is hanging on labvirt1003 - https://phabricator.wikimedia.org/T135850#2313009 (10Andrew) 9:22 AM it's again stuck related to lldp, it hangs when trying to interact with lldpctl 9:23 AM might be a broken cable or NIC, something Chri... [16:04:03] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3669 bytes in 0.034 second response time [16:16:26] PROBLEM - Host tools-webgrid-generic-1401 is DOWN: CRITICAL - Host Unreachable (10.68.18.51) [16:17:05] ^ is us moving things around [16:19:43] RECOVERY - Host tools-webgrid-generic-1401 is UP: PING OK - Packet loss = 0%, RTA = 0.60 ms [16:23:51] PROBLEM - Host tools-webgrid-lighttpd-1202 is DOWN: CRITICAL - Host Unreachable (10.68.18.46) [16:28:54] RECOVERY - Host tools-webgrid-lighttpd-1202 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [16:34:05] 10PAWS, 10Jupyter-Hub: I can't login my bot in JUPYTER - https://phabricator.wikimedia.org/T135306#2313086 (10yuvipanda) @Maathavan now? [16:35:15] PROBLEM - Host tools-webgrid-lighttpd-1207 is DOWN: CRITICAL - Host Unreachable (10.68.16.215) [16:39:14] RECOVERY - Puppet run on tools-docker-builder-03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:54:37] I noticed http://quarry.wmflabs.org is down, is that a known issue? /ping YuviPanda [16:59:36] Nettrom: some VMs currently need to be migrated, it's likely affected by it [16:59:50] Nettrom: there is some shuffling going on but I don't think it's purposely down [16:59:52] ah, yes, I saw the email, that explains it, thanks [16:59:52] that^ [17:01:48] checked phabricator and it’s in the list of the VMs affected, should’ve checked that first, sorry :) [17:20:32] Nettrom: am in the process of fixing it btw [17:21:57] Nettrom: fixed [17:25:47] RECOVERY - Host tools-webgrid-lighttpd-1207 is UP: PING OK - Packet loss = 0%, RTA = 0.94 ms [17:29:57] YuviPanda: that’s awesome, thanks so much! :) [17:31:01] Nettrom: np. [17:32:38] PROBLEM - Host tools-webgrid-lighttpd-1210 is DOWN: CRITICAL - Host Unreachable (10.68.17.163) [17:46:05] RECOVERY - Host tools-webgrid-lighttpd-1210 is UP: PING OK - Packet loss = 0%, RTA = 0.84 ms [17:47:47] PROBLEM - Host tools-webgrid-lighttpd-1402 is DOWN: CRITICAL - Host Unreachable (10.68.16.35) [17:59:50] 06Labs, 10Tool-Labs: PHP 5.5 sessionclean cron job hanging on tool labs bastions - https://phabricator.wikimedia.org/T135861#2313341 (10bd808) [18:03:00] RECOVERY - Host tools-webgrid-lighttpd-1402 is UP: PING OK - Packet loss = 0%, RTA = 0.93 ms [18:03:58] PROBLEM - Host tools-webgrid-lighttpd-1407 is DOWN: CRITICAL - Host Unreachable (10.68.17.251) [18:04:30] hiyaa [18:04:39] got some really weird dns something happening in labs [18:04:45] dig vem.maps-team.eqiad.wmflabs [18:04:50] vem.maps-team.eqiad.wmflabs. 58 IN A 10.68.17.12 [18:04:50] vem.maps-team.eqiad.wmflabs. 58 IN A 10.68.22.84 [18:05:13] druid102.analytics.eqiad.wmflabs. 3 IN A 10.68.17.12 [18:05:33] andrewbogott: any idea what's going on there? [18:05:49] reverse lookup for 10.68.17.12 shows [18:05:50] both [18:05:55] 12.17.68.10.in-addr.arpa. 41 IN PTR druid102.analytics.eqiad.wmflabs. [18:05:55] 12.17.68.10.in-addr.arpa. 41 IN PTR vem.maps-team.eqiad.wmflabs. [18:06:10] ottomata: we have an occassional dns leak issue that either is still happening or this is been like this for a bit [18:06:12] not sure which [18:06:14] ottomata: do those instances actually still exist? [18:06:18] druid102 does [18:06:23] no idea what vem.maps-team is [18:06:50] i'm working with druid102 now and its causing things to timeout on requests i guess because it is resolving to vem.maps-team or something [18:08:20] ottomata: if druid102 is new, you can just rebuild it. Otherwise make me a phab task and I'll try to figure out what's happening [18:09:58] ah its not new, but ic oudl rebuild, would prefer not to [18:10:06] i might just edit /etc/hosts to override for now maybe [18:10:23] am in the middle of troubleshooting cluster setup, etc. [18:10:29] ottomata: open a phab task too etc [18:12:03] is labstore2004 just installed or something? [18:12:22] no [18:12:25] and i see icinga is changing hostgroups around [18:12:30] a month or so? [18:12:37] afaik no one has touched it either [18:12:38] removes labs_eqiad, adds mysql_eqiad [18:12:54] for a couple hosts lately [18:13:32] mutante: ah, no idea that's probably from a more recent hiera thing possibly labtest related I know andrewbogott is in the middle of a fwe things but I imagine he'll have to poke at it [18:13:48] Is it https://gerrit.wikimedia.org/r/#/c/289315/ maybe? [18:13:57] Is that the same kind of host group? [18:14:15] i think there are 2 separate things going on [18:14:19] I think that some hosts are in labs_services and labstore groups both now, by mistake [18:14:27] a) it seems since that change that you link, and that i saw .. [18:14:35] 06Labs: Weird extra DNS entries in labs - https://phabricator.wikimedia.org/T135864#2313395 (10Ottomata) [18:14:37] icinga keeps changing hostgroups from labs to mysql something [18:14:42] thar yall go ^^ :) [18:14:47] like it changes it back on each run maybe [18:14:55] mutante: last time that happend to me [18:14:55] b) unrelated. somehow this: [18:15:00] it was strontium being behind / wrong [18:15:06] and so it was a diff value depending on what master [18:15:06] Error: Could not find any host matching 'labstore2004 [18:15:17] that last one is the actual problem for icinga ,so it cant restart [18:15:32] the other thing is not breaking it ..but keeps showing up [18:15:49] that weird I'm not sure as nothing there has changed to my knowledge specific to labstore2004 [18:15:52] ottomata: heh, I read it as 'extra weird DNS entries' and was like 'so there are normal weird ones?!?' [18:16:13] chasemp: that seems likely , yes [18:16:38] because: [18:16:39] 11:21 < icinga-wm> RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [18:17:06] I about pulled my hair out on puppet continually switching values and that's all it was [18:19:57] RECOVERY - Host tools-webgrid-lighttpd-1407 is UP: PING OK - Packet loss = 0%, RTA = 0.78 ms [18:20:34] on the next run, now it removed service checks for labstore2003 [18:20:50] it's like it's in the middle of just taking them out of icinga [18:21:18] and needs a couple runs for that. so if a host is already gone but the service is still there it gets that error [18:21:38] -define host { [18:21:38] - address 10.192.21.6 [18:22:01] keeps running puppet on neon [18:22:14] PROBLEM - Host tools-worker-1010 is DOWN: CRITICAL - Host Unreachable (10.68.19.138) [18:22:25] heheh [18:22:35] maybe a change that includes "standard" and now doesnt or something? [18:23:31] that may be it [18:23:41] and that may be me give me a sec to look [18:24:09] ok,thanks, be right back [18:24:44] andrewbogott: ^ 1010, did labvirt1003 just go down? [18:25:01] no, I'm migrating it... [18:25:04] was I not supposed to? [18:25:05] andrewbogott: ah, ok [18:25:23] andrewbogott: didn't matter, since k8s has enough capacity and it'll route around if/when it goes down [18:25:27] andrewbogott: but is ok to migrate too :) [18:36:47] mutante: https://gerrit.wikimedia.org/r/#/c/289892/1 [18:39:55] andrewbogott: aha! :) [18:40:57] much quick fixes, cool [18:55:05] RECOVERY - Host tools-worker-1010 is UP: PING OK - Packet loss = 0%, RTA = 1.10 ms [19:00:56] (03PS1) 10Merlijn van Deen: add hostname to userinfo [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/289899 [19:01:09] legoktm: ^ [19:03:40] (03CR) 10jenkins-bot: [V: 04-1] add hostname to userinfo [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/289899 (owner: 10Merlijn van Deen) [19:03:45] baaah [19:04:07] oh, the comma. right [19:04:28] (03PS2) 10Merlijn van Deen: add hostname to userinfo [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/289899 [19:04:28] thank you, jenkins <3 [19:09:08] PROBLEM - Host tools-exec-1216 is DOWN: CRITICAL - Host Unreachable (10.68.17.255) [19:19:10] ^ that's me migrating things [19:32:47] PROBLEM - Free space - all mounts on tools-worker-1004 is CRITICAL: CRITICAL: tools.tools-worker-1004.diskspace.root.byte_percentfree (<66.67%) [19:44:38] !log tools tools-exec-1406 having issues rebooting [19:47:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [20:15:33] 06Labs: Weird extra DNS entries in labs - https://phabricator.wikimedia.org/T135864#2313731 (10scfc) [20:15:36] 06Labs, 10Labs-Infrastructure, 06Operations: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#2313733 (10scfc) [20:28:32] 06Labs, 10Tool-Labs: PHP 5.5 sessionclean cron job hanging on tool labs bastions - https://phabricator.wikimedia.org/T135861#2313341 (10scfc) Have the new packages Nish Aravamudan referred to in https://bugs.launchpad.net/ubuntu/+source/php5/+bug/1356113/comments/42 been tested in Labs? It would be nice not t... [20:40:22] RECOVERY - Host tools-exec-1216 is UP: PING OK - Packet loss = 0%, RTA = 0.89 ms [20:42:36] PROBLEM - Host tools-exec-1407 is DOWN: CRITICAL - Host Unreachable (10.68.18.16) [20:52:08] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 10Diffusion: create conduit method for the creation of phabricator policy objects - https://phabricator.wikimedia.org/T135249#2313790 (10mmodell) [21:00:42] RECOVERY - Host tools-exec-1407 is UP: PING OK - Packet loss = 0%, RTA = 2.38 ms [21:03:10] PROBLEM - Host tools-exec-1219 is DOWN: CRITICAL - Host Unreachable (10.68.18.40) [21:14:42] odd, I know its being worked on as it's on 1003 but my instance (maps-warper) is seeing a nice flatish load minimum of 1.0 for about 11 hours, never goes below it [21:21:03] RECOVERY - Host tools-exec-1219 is UP: PING OK - Packet loss = 0%, RTA = 0.89 ms [21:41:31] PROBLEM - Host tools-exec-1216 is DOWN: CRITICAL - Host Unreachable (10.68.17.255) [21:43:45] andrewbogott: is the volatility still expected for tools things^? [21:44:37] chasemp: you mean the host alerts? That's me evacuating labvirt1003 [21:44:43] ok just checking :) [21:48:14] bd808: When I try to ssh to commtech-1.commtech.eqiad.wmflabs from labs bastion, it says No route to host. [21:48:56] kaldari: same issue for me [21:49:36] bd808: who's in charge of DNS/routing issues for labs? [21:49:46] andrewbogott and chasemp [21:50:37] * andrewbogott looks [21:50:59] andrewbogott: says it's on labvirt1010 on wikitech [21:51:06] do we hvae other instances there that work? [21:51:13] kaldari: when did it last work for you? [21:51:45] andrewbogott: I haven't used it in months [21:51:55] It's in an 'error' state, not running at all [21:52:22] It was up and running ... a couple of weeks ago? [21:52:50] will a reboot help if it's already in the error state? [21:54:25] I'm trying to start it [21:55:05] you have to 'reset-state' before you can reboot an instance in state 'error'. I'm not sure if horizon does that [21:59:30] kaldari: try it now? [21:59:55] andrewbogott: I can log in now [22:00:05] great [22:00:13] I don't know what happened — I've never seen this particular failure before [22:01:52] restarting vagrant now [22:02:17] Yay, http://commtech.wmflabs.org/wiki/Main_Page is back up and running [22:02:49] Such and exciting default page! :) [22:02:53] andrewbogott: Thanks for your help! [22:03:02] very [22:12:50] 06Labs, 10Tool-Labs: PHP 5.5 sessionclean cron job hanging on tool labs bastions - https://phabricator.wikimedia.org/T135861#2314261 (10bd808) To my knowledge, no that PPA build has not been tested. Rather than maintain a patch to the deb, I would suggest a small bit of puppet code to replace the /etc/cron.d/... [22:27:50] RECOVERY - Host tools-exec-1216 is UP: PING OK - Packet loss = 0%, RTA = 1.14 ms [22:48:04] RECOVERY - Puppet run on tools-exec-1215 is OK: OK: Less than 1.00% above the threshold [0.0] [23:39:09] 06Labs, 10Tool-Labs, 13Patch-For-Review, 15User-bd808: PHP 5.5 sessionclean cron job hanging on tool labs bastions - https://phabricator.wikimedia.org/T135861#2314438 (10bd808) a:03bd808 [23:39:15] !log tools Forced puppet run on bastion-02 & bastion-05 to apply fix for T135861 [23:39:16] T135861: PHP 5.5 sessionclean cron job hanging on tool labs bastions - https://phabricator.wikimedia.org/T135861 [23:39:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master