[04:52:27] 10Phabricator, 6Operations, 10ops-eqiad: iridium (Phabricator host) went down, Possible cpu heat issue - https://phabricator.wikimedia.org/T131742#2176955 (10Peachey88) [12:43:31] 10Phabricator, 6Operations, 10ops-eqiad: iridium (Phabricator host) went down, Possible cpu heat issue - https://phabricator.wikimedia.org/T131742#2176955 (10Cmjohnson) There has been several severs with cpu heat issues over the last few months. Re-applying thermal paste has been an effective fix. Iridium i... [14:12:50] 10Phabricator, 6Labs: Create custom Phab form for requesting a new Labs project - https://phabricator.wikimedia.org/T128300#2177571 (10chasemp) p:5Triage>3Normal [14:32:03] 10Phabricator, 6Labs: Git broken on phabricator labs machines - https://phabricator.wikimedia.org/T127139#2177600 (10chasemp) p:5Triage>3Normal [15:18:41] 10Phabricator, 6Operations, 6Release-Engineering-Team, 10ops-eqiad: iridium (Phabricator host) went down, Possible cpu heat issue - https://phabricator.wikimedia.org/T131742#2177789 (10chasemp) p:5Triage>3High [15:21:49] 10Phabricator, 6Operations, 6Release-Engineering-Team, 10ops-eqiad: iridium (Phabricator host) went down, Possible cpu heat issue - https://phabricator.wikimedia.org/T131742#2177807 (10chasemp) Some details from an email @tstarling sent out as a notice ```...So I powered it up, and it came up, but /var/lo... [15:23:24] 10Phabricator, 6Operations, 6Release-Engineering-Team, 10ops-eqiad, 15User-greg: iridium (Phabricator host) went down, Possible cpu heat issue - https://phabricator.wikimedia.org/T131742#2177828 (10chasemp) a:3greg tossing your way because of the nature of the issue and need for immediate feedback for... [15:24:53] 10Phabricator, 6Operations, 6Release-Engineering-Team, 10ops-eqiad, 15User-greg: iridium (Phabricator host) went down, Possible cpu heat issue - https://phabricator.wikimedia.org/T131742#2177840 (10greg) yeah, stab in the dark guess is it might happen again tonight after the dumps (I assume?) run. When... [15:25:49] 10Phabricator, 6Operations, 6Release-Engineering-Team, 10ops-eqiad: iridium (Phabricator host) went down, Possible cpu heat issue - https://phabricator.wikimedia.org/T131742#2177842 (10greg) a:5greg>3None [15:40:54] 10Phabricator, 6Operations, 6Release-Engineering-Team, 10ops-eqiad: iridium (Phabricator host) went down, Possible cpu heat issue - https://phabricator.wikimedia.org/T131742#2177971 (10Cmjohnson) @greg Downtime Max 10 minutes but not even that long. Can do whenever you're ready [16:16:24] 5Gerrit-Migration, 10Phabricator, 10Continuous-Integration-Infrastructure, 6Operations, and 4 others: Make sure phab can talk to gearman and nodepool instances can talk to phabricator - https://phabricator.wikimedia.org/T131375#2178070 (10mmodell) Thanks @dzahn for setting this up so quickly. I tested that... [16:16:55] 5Gerrit-Migration, 10Phabricator, 10Continuous-Integration-Infrastructure, 6Operations, and 4 others: Make sure phab can talk to gearman and nodepool instances can talk to phabricator - https://phabricator.wikimedia.org/T131375#2178072 (10mmodell) [16:39:14] 10Phabricator, 6Project-Admins, 6Stewards-and-global-tools: Create acl*stewards - https://phabricator.wikimedia.org/T131766#2178114 (10MarcoAurelio) a:5MarcoAurelio>3None >>! In T131766#2178095, @Luke081515 wrote: > This needs to be done by an admin, because "normal" project creators can't adjust policys... [16:46:03] 10Phabricator, 6Project-Admins, 6Stewards-and-global-tools: Create acl*stewards - https://phabricator.wikimedia.org/T131766#2178040 (10Krenair) > If I understood the process correctly, if we add the project to the subscribers field, all members of that group will get access to the task containing it. That d... [17:06:15] 10Phabricator, 6Operations, 6Release-Engineering-Team, 10ops-eqiad: iridium (Phabricator host) went down, Possible cpu heat issue - https://phabricator.wikimedia.org/T131742#2178278 (10greg) >>! In T131742#2177971, @Cmjohnson wrote: > @greg Downtime Max 10 minutes but not even that long. Can do whenever yo... [17:06:27] 10Phabricator, 6Operations, 6Release-Engineering-Team, 10ops-eqiad: iridium (Phabricator host) went down, Possible cpu heat issue - https://phabricator.wikimedia.org/T131742#2178279 (10greg) a:3Cmjohnson [17:12:23] 10Phabricator, 6Project-Admins, 6Stewards-and-global-tools: Create acl*stewards - https://phabricator.wikimedia.org/T131766#2178296 (10MarcoAurelio) >>! In T131766#2178153, @Krenair wrote: >> If I understood the process correctly, if we add the project to the subscribers field, all members of that group will... [17:12:57] 10Phabricator, 6Operations, 10hardware-requests: We need a backup phabricator front-end node - https://phabricator.wikimedia.org/T131775#2178301 (10mmodell) [17:14:42] 10Phabricator, 6Operations, 10hardware-requests: We need a backup phabricator front-end node - https://phabricator.wikimedia.org/T131775#2178316 (10jeremyb-phone) [17:15:53] 10Phabricator, 6Operations, 10hardware-requests: We need a backup phabricator front-end node - https://phabricator.wikimedia.org/T131775#2178325 (10greg) See also: {T131742} :) [17:27:19] 10Phabricator, 6Project-Admins, 6Stewards-and-global-tools: Create acl*stewards - https://phabricator.wikimedia.org/T131766#2178405 (10Krenair) >>! In T131766#2178296, @MarcoAurelio wrote: >>>! In T131766#2178153, @Krenair wrote: >>> If I understood the process correctly, if we add the project to the subscri... [17:29:08] hi everyone: I need to take phabricator down for up to 10 mins...i would like to do this in about 10 mins from now to fix the cpu heat issue....does anyone object? [17:31:21] cmjohnson1: I think you are cool man [17:31:29] setting the irc topic in -ops is probably a good idea [17:31:31] thanks [17:31:43] will do [17:31:48] cool..thx [17:53:07] 10Phabricator, 6Operations, 6Release-Engineering-Team, 10ops-eqiad: iridium (Phabricator host) went down, Possible cpu heat issue - https://phabricator.wikimedia.org/T131742#2178470 (10Cmjohnson) a:5Cmjohnson>3None Clean off the old thermal paste and reapplied. Let's monitor for the next few days. Lea... [17:55:33] 10Phabricator, 6Project-Admins, 6Stewards-and-global-tools: Create acl*stewards - https://phabricator.wikimedia.org/T131766#2178492 (10MarcoAurelio) I think my language was not clear, sorry. When we encounter a bug that affects our work as a team, we inform the rest of the team with a short summary of what's... [18:44:27] 10Phabricator, 6Operations, 6Release-Engineering-Team, 10ops-eqiad: iridium (Phabricator host) went down, Possible cpu heat issue - https://phabricator.wikimedia.org/T131742#2176955 (10hashar) @cmjohnson do we have a system to monitor temperature? lm_sensors comes to mind, also found out Diamond has a coll... [18:47:38] 10Phabricator, 6Operations, 10hardware-requests: We need a backup phabricator front-end node - https://phabricator.wikimedia.org/T131775#2178647 (10mmodell) >>! In T131775#2178325, @greg wrote: > See also: {T131742} :) Plus, every deployment involves significant downtime because phabricator services must al... [19:31:36] 5Gerrit-Migration, 10Phabricator, 10Continuous-Integration-Infrastructure, 6Operations, and 4 others: Make sure phab can talk to gearman and nodepool instances can talk to phabricator - https://phabricator.wikimedia.org/T131375#2178776 (10chasemp) Afaik the 'talk to phabricator' portion here is relevant fo... [21:55:54] 10Phabricator: Turn off or delete the Collaboration team Herald rule - https://phabricator.wikimedia.org/T131813#2179246 (10Mattflaschen) [22:08:16] 10Phabricator: Turn off or delete the Collaboration team Herald rule - https://phabricator.wikimedia.org/T131813#2179246 (10greg) Pretty sure that's H6, right @Mattflaschen ? @Aklapper is "archiving" the right action there? [22:08:24] 10Phabricator: Turn off or delete the Collaboration team Herald rule - https://phabricator.wikimedia.org/T131813#2179289 (10Krenair) 5Open>3Resolved a:3Krenair Archived {H26} [22:33:13] 10Phabricator, 10MediaWiki-Authentication-and-authorization, 7Accessibility, 7Security-General: Phabricator asks to log in unnecessarily, complicately, with traps, and madly - https://phabricator.wikimedia.org/T131697#2175269 (10Krenair) > First, there is a big Login form. But this is a trap. If I try this...