[00:01:45] (03Merged) 10jenkins-bot: PhpAutoPrepend: Remove PhpAutoPrepend-labs.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486177 (owner: 10Krinkle) [00:03:07] (03CR) 10jenkins-bot: PhpAutoPrepend: Remove PhpAutoPrepend-labs.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486177 (owner: 10Krinkle) [00:22:41] RECOVERY - puppet last run on install1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:26:10] (03CR) 10BryanDavis: "I think this should be either 'clouddb1001.clouddb-services.eqiad.wmflabs' or maybe better 'tools.db.svc.eqiad.wmflabs'." [puppet] - 10https://gerrit.wikimedia.org/r/491013 (https://phabricator.wikimedia.org/T193264) (owner: 10Bstorm) [00:29:25] (03CR) 10BryanDavis: "Inline note about private vs public ip usage" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/491005 (https://phabricator.wikimedia.org/T193264) (owner: 10Arturo Borrero Gonzalez) [00:36:26] (03CR) 10Bstorm: "> Patch Set 1:" [puppet] - 10https://gerrit.wikimedia.org/r/491013 (https://phabricator.wikimedia.org/T193264) (owner: 10Bstorm) [00:58:02] (03CR) 10BryanDavis: "> I agree. As of now, clouddb1001.clouddb-services.eqiad.wmflabs" [puppet] - 10https://gerrit.wikimedia.org/r/491013 (https://phabricator.wikimedia.org/T193264) (owner: 10Bstorm) [01:12:12] PROBLEM - toolschecker: All k8s worker nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 185 bytes in 0.178 second response time [01:39:44] (03PS1) 10Zoranzoki21: Add namespace Додатак on srwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/491024 (https://phabricator.wikimedia.org/T216343) [01:42:16] (03PS7) 10Zoranzoki21: Add new throttle rule for T215839 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489819 (https://phabricator.wikimedia.org/T215839) [01:45:29] (03PS1) 10Zoranzoki21: Set $wgArticleCountMethod = 'any' on fiwikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/491025 (https://phabricator.wikimedia.org/T216333) [02:15:58] 10Operations, 10DNS, 10Domains, 10Traffic, 10HTTPS: Merge Wikipedia subdomains into one, to discourage censorship - https://phabricator.wikimedia.org/T215071 (10Liuxinyu970226) @Platonides > However note that in the event that your browser can't support TLS v1.3, you won't be able to view the page at all... [02:40:14] (03CR) 10Bstorm: "> Patch Set 1:" [puppet] - 10https://gerrit.wikimedia.org/r/491013 (https://phabricator.wikimedia.org/T193264) (owner: 10Bstorm) [03:01:40] (03CR) 10Bstorm: "> Patch Set 2:" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/491005 (https://phabricator.wikimedia.org/T193264) (owner: 10Arturo Borrero Gonzalez) [03:37:35] PROBLEM - puppet last run on analytics1064 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIP2-City.mmdb.gz],File[/usr/share/GeoIP/GeoIP2-City.mmdb.test] [04:03:37] RECOVERY - puppet last run on analytics1064 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:37:02] 10Operations, 10Office-IT, 10Wikimedia-Mailing-lists: Mailing list migration for Arbitration Committee to Google Group - https://phabricator.wikimedia.org/T215940 (10Aklapper) [07:02:35] 10Operations, 10DNS, 10Domains, 10Traffic, and 2 others: Point wikipedia.in to 205.147.101.160 instead of URL forward - https://phabricator.wikimedia.org/T144508 (10Aklapper) @Naveenpf : ping? ^ [07:04:54] 10Operations, 10Wikimedia-Mailing-lists, 10Patch-For-Review, 10User-herron: Ban recurrent spam to Wikimedia mailing lists (January 2019) - https://phabricator.wikimedia.org/T215251 (10Aklapper) lgtm too; could probably be resolved? (And big thanks!) [08:30:06] RECOVERY - toolschecker: All k8s worker nodes are healthy on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.688 second response time [09:23:04] RECOVERY - High load average on labstore1007 is OK: OK: Less than 85.00% above the threshold [16.0] https://grafana.wikimedia.org/dashboard/db/labs-monitoring [12:29:54] 10Operations, 10Cloud-VPS, 10netops, 10cloud-services-team (Kanban): toolsdb: firewalling changes for new setup (temporal mysql replication) - https://phabricator.wikimedia.org/T216353 (10aborrero) [12:30:31] 10Operations, 10Cloud-VPS, 10DBA, 10netops, 10cloud-services-team (Kanban): toolsdb: firewalling changes for new setup (temporal mysql replication) - https://phabricator.wikimedia.org/T216353 (10aborrero) [12:31:32] 10Operations, 10Cloud-VPS, 10DBA, 10netops, 10cloud-services-team (Kanban): toolsdb: firewalling changes for new setup (temporal mysql replication) - https://phabricator.wikimedia.org/T216353 (10aborrero) [12:38:51] 10Operations, 10Cloud-VPS, 10DBA, 10netops, 10cloud-services-team (Kanban): toolsdb: firewalling changes for new setup (temporal mysql replication) - https://phabricator.wikimedia.org/T216353 (10aborrero) [12:40:35] 10Operations, 10Cloud-VPS, 10DBA, 10netops, 10cloud-services-team (Kanban): toolsdb: firewalling changes for new setup (temporal mysql replication) - https://phabricator.wikimedia.org/T216353 (10aborrero) [12:59:29] (03PS3) 10Arturo Borrero Gonzalez: cloudvps: refresh FQDN A record for tools.db.svc.eqiad.wmflabs [puppet] - 10https://gerrit.wikimedia.org/r/491005 (https://phabricator.wikimedia.org/T193264) [13:13:55] 10Operations, 10Cloud-VPS, 10DBA, 10netops, 10cloud-services-team (Kanban): toolsdb: firewalling changes for new setup (temporal mysql replication) - https://phabricator.wikimedia.org/T216353 (10ayounsi) Thanks! `lang=diff [edit firewall family inet filter cloud-in4] term labsdb { ... } + /*... [13:14:08] !log add term labsdb_return to cloud-in4 - T216353 [13:14:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:14:11] T216353: toolsdb: firewalling changes for new setup (temporal mysql replication) - https://phabricator.wikimedia.org/T216353 [13:16:21] 10Operations, 10Cloud-VPS, 10DBA, 10netops, 10cloud-services-team (Kanban): toolsdb: firewalling changes for new setup (temporal mysql replication) - https://phabricator.wikimedia.org/T216353 (10aborrero) 05Open→03Resolved a:03aborrero It works! ` aborrero@labsdb1005:~$ telnet 172.16.7.153 3306 Tr... [16:52:42] 10Operations, 10Cloud-VPS, 10DBA, 10netops, 10cloud-services-team (Kanban): toolsdb: firewalling changes for new setup (temporal mysql replication) - https://phabricator.wikimedia.org/T216353 (10Bstorm) We need labstore1004 and 5 more than these. 10.64.37.18, 10.64.37.19 and 10.64.37.20 maintain-dbuser... [16:56:12] 10Operations, 10Cloud-VPS, 10DBA, 10netops, 10cloud-services-team (Kanban): toolsdb: firewalling changes for new setup (temporal mysql replication) - https://phabricator.wikimedia.org/T216353 (10Bstorm) 05Resolved→03Open [16:57:51] 10Operations, 10Cloud-VPS, 10DBA, 10netops, 10cloud-services-team (Kanban): toolsdb: firewalling changes for new setup (temporal mysql replication) - https://phabricator.wikimedia.org/T216353 (10Bstorm) Without labstores1004 and 1005 being able to hit the database, new user accounts on toolsdb will not b... [17:28:14] (03PS1) 10Ammarpad: Add new namespaces for th.wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/491054 (https://phabricator.wikimedia.org/T216322) [19:18:20] 10Operations, 10Cloud-VPS, 10DBA, 10netops, 10cloud-services-team (Kanban): toolsdb: firewalling changes for new setup (temporal mysql replication) - https://phabricator.wikimedia.org/T216353 (10Bstorm) We've found a way around the criticality of this. We'll still need those components, and we should up... [19:18:34] 10Operations, 10Cloud-VPS, 10DBA, 10netops, 10cloud-services-team (Kanban): toolsdb: firewalling changes for new setup (temporal mysql replication) - https://phabricator.wikimedia.org/T216353 (10Bstorm) p:05Unbreak!→03Low [19:20:25] 10Operations, 10Cloud-VPS, 10DBA, 10netops, 10cloud-services-team (Kanban): toolsdb: firewalling changes for new setup (temporal mysql replication) - https://phabricator.wikimedia.org/T216353 (10Bstorm) [19:26:34] 10Operations, 10Cloud-VPS, 10netops, 10User-Marostegui, 10cloud-services-team (Kanban): toolsdb: firewalling changes for new setup (temporal mysql replication) - https://phabricator.wikimedia.org/T216353 (10Marostegui) [20:24:11] (03PS1) 10Arturo Borrero Gonzalez: toolforge: introduce profile for maintain_dbusers in services nodes [puppet] - 10https://gerrit.wikimedia.org/r/491064 (https://phabricator.wikimedia.org/T216373) [20:25:09] (03CR) 10jerkins-bot: [V: 04-1] toolforge: introduce profile for maintain_dbusers in services nodes [puppet] - 10https://gerrit.wikimedia.org/r/491064 (https://phabricator.wikimedia.org/T216373) (owner: 10Arturo Borrero Gonzalez) [20:27:39] (03PS2) 10Arturo Borrero Gonzalez: toolforge: introduce profile for maintain_dbusers in services nodes [puppet] - 10https://gerrit.wikimedia.org/r/491064 (https://phabricator.wikimedia.org/T216373) [21:20:57] !log The slave of labsdb1005.eqiad.wmnet is now clouddb1001.clouddb-services.eqiad.wmflabs [21:20:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:23:18] (03CR) 10Zoranzoki21: [C: 03+1] Add new namespaces for th.wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/491054 (https://phabricator.wikimedia.org/T216322) (owner: 10Ammarpad) [21:23:33] (03CR) 10Zoranzoki21: [C: 03+1] Set wgArticleCountMethod='any' for zhwikiversity [mediawiki-config] - 10https://gerrit.wikimedia.org/r/487115 (owner: 10Ammarpad) [21:40:09] (03PS1) 10Andrew Bogott: NFS: allow the clouddb-services project to mount tools home and project dirs [puppet] - 10https://gerrit.wikimedia.org/r/491184 [21:53:54] (03PS2) 10Andrew Bogott: NFS: allow the clouddb-services project to mount tools home and project dirs [puppet] - 10https://gerrit.wikimedia.org/r/491184 [21:55:12] (03CR) 10Andrew Bogott: [C: 03+2] NFS: allow the clouddb-services project to mount tools home and project dirs [puppet] - 10https://gerrit.wikimedia.org/r/491184 (owner: 10Andrew Bogott) [22:24:34] (03PS1) 10Andrew Bogott: nfs-exportd: add the 'exportdir' config value [puppet] - 10https://gerrit.wikimedia.org/r/491186 [22:30:21] (03CR) 10Andrew Bogott: [C: 03+2] nfs-exportd: add the 'exportdir' config value [puppet] - 10https://gerrit.wikimedia.org/r/491186 (owner: 10Andrew Bogott) [22:37:03] (03PS3) 10Andrew Bogott: toolforge: introduce profile for maintain_dbusers in services nodes [puppet] - 10https://gerrit.wikimedia.org/r/491064 (https://phabricator.wikimedia.org/T216373) (owner: 10Arturo Borrero Gonzalez) [22:53:48] (03PS4) 10Andrew Bogott: wmcs services: introduce profile for maintain_dbusers in services nodes [puppet] - 10https://gerrit.wikimedia.org/r/491064 (https://phabricator.wikimedia.org/T216373) (owner: 10Arturo Borrero Gonzalez) [23:18:42] (03CR) 10Andrew Bogott: [C: 03+2] wmcs services: introduce profile for maintain_dbusers in services nodes [puppet] - 10https://gerrit.wikimedia.org/r/491064 (https://phabricator.wikimedia.org/T216373) (owner: 10Arturo Borrero Gonzalez) [23:22:02] (03PS1) 10Andrew Bogott: wmcs services: fixed a few resource paths for maintain_dbusers [puppet] - 10https://gerrit.wikimedia.org/r/491188 [23:23:08] (03CR) 10Andrew Bogott: [C: 03+2] wmcs services: fixed a few resource paths for maintain_dbusers [puppet] - 10https://gerrit.wikimedia.org/r/491188 (owner: 10Andrew Bogott)