[00:32:42] (03CR) 10Alex Monk: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/498641 (owner: 10Alex Monk) [00:33:33] (03CR) 10jerkins-bot: [V: 04-1] uwsgi::app: Handle ensure absent [puppet] - 10https://gerrit.wikimedia.org/r/498641 (owner: 10Alex Monk) [00:34:52] so wait what exactly is jenkins' problem? [00:35:05] https://integration.wikimedia.org/ci/job/operations-puppet-tests-stretch-docker/8597/console is looking okay? [00:35:53] except the bit about moving tox files around? [00:35:58] tox logs [00:45:12] Could it be [00:45:14] 00:33:27 ---> wmf_style lint [00:45:14] 00:33:27 wmf-style: total violations delta 1 [00:45:18] Krenair: ^^ [00:45:56] paladox, but no new and no resolved violations..? [00:46:16] Yup [00:46:33] I carnt see anything else in that log that would explain it [00:52:02] Krenair: you could test by creating a new change and adding something the description in the file. [01:13:37] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [01:27:55] (03PS2) 10Alex Monk: ferm::service: Allow ensure absent without proto/port [puppet] - 10https://gerrit.wikimedia.org/r/498645 [01:34:51] (03PS1) 10Paladox: Update its-base [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/498691 [01:35:10] (03CR) 10Paladox: [V: 03+2 C: 03+2] Update its-base [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/498691 (owner: 10Paladox) [02:06:25] Is there an SRE member online? (not high priority) [02:07:35] probably not [02:07:44] does it need to be SRE? maybe someone else could help? [02:36:34] (03CR) 10Alex Monk: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/498655 (owner: 10Alex Monk) [02:37:48] (03CR) 10jerkins-bot: [V: 04-1] nrpe::monitor_service and monitoring::service: Allow ensure absent without description/nrpe_command [puppet] - 10https://gerrit.wikimedia.org/r/498655 (owner: 10Alex Monk) [02:38:25] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [02:38:54] (03PS4) 10Alex Monk: Allow ensure absent in monitoring classes without description/nrpe_command [puppet] - 10https://gerrit.wikimedia.org/r/498655 [02:40:05] (03CR) 10jerkins-bot: [V: 04-1] Allow ensure absent in monitoring classes without description/nrpe_command [puppet] - 10https://gerrit.wikimedia.org/r/498655 (owner: 10Alex Monk) [02:40:56] (03PS5) 10Alex Monk: Allow ensure absent in monitoring classes without description/nrpe_command [puppet] - 10https://gerrit.wikimedia.org/r/498655 [02:48:18] (03CR) 10Alex Monk: "Opened T219085 for the Jenkins failures" [puppet] - 10https://gerrit.wikimedia.org/r/498641 (owner: 10Alex Monk) [03:08:15] 10Operations, 10Beta-Cluster-Infrastructure, 10Mathoid, 10Core Platform Team Backlog (Watching / External), and 2 others: remove mathoid from scb - https://phabricator.wikimedia.org/T200832 (10Krenair) deployment-mathoid still exists and has been failing puppet runs since December 3rd when profile::mathoid... [03:11:11] PROBLEM - puppet last run on analytics1054 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:25:25] 10Operations, 10Gerrit, 10Release-Engineering-Team: Create Gerrit Administrator right policy - https://phabricator.wikimedia.org/T218686 (10Legoktm) >>! In T218686#5039340, @hashar wrote: > We need a dedicated policy. Granting Gerrit administrative rights is a lot more responsibilities and require a lot of t... [03:34:41] PROBLEM - puppet last run on mw2172 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIPCity.dat.gz] [03:38:59] PROBLEM - puppet last run on argon is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:42:53] RECOVERY - puppet last run on analytics1054 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [04:00:57] PROBLEM - puppet last run on es1017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [04:01:03] RECOVERY - puppet last run on mw2172 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [04:05:21] RECOVERY - puppet last run on argon is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:21:47] PROBLEM - puppet last run on wtp1033 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [04:27:21] RECOVERY - puppet last run on es1017 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [04:48:11] RECOVERY - puppet last run on wtp1033 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [05:08:35] PROBLEM - puppet last run on cumin1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [05:19:51] (03PS1) 10Alex Monk: Allow customisation of cumin list on a per-project basis [puppet] - 10https://gerrit.wikimedia.org/r/498697 [05:20:51] (03PS2) 10Alex Monk: network: Allow customisation of cumin list on a per-project basis [puppet] - 10https://gerrit.wikimedia.org/r/498697 [05:24:24] (03PS1) 10Alex Monk: network: labs: remove outdated entries [puppet] - 10https://gerrit.wikimedia.org/r/498698 [05:31:41] PROBLEM - puppet last run on cp1080 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [05:40:15] RECOVERY - puppet last run on cumin1001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [05:43:19] PROBLEM - puppet last run on wdqs1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [05:51:15] (03PS1) 10Alex Monk: openstack: Add an instance spread check for deployment-prep [puppet] - 10https://gerrit.wikimedia.org/r/498699 (https://phabricator.wikimedia.org/T219088) [05:58:03] RECOVERY - puppet last run on cp1080 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:09:41] RECOVERY - puppet last run on wdqs1009 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:32:47] PROBLEM - puppet last run on phab1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/prometheus-puppet-agent-stats] [06:59:05] RECOVERY - puppet last run on phab1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [09:20:41] Is ruslik/rr0 here? [10:02:55] (03CR) 10GTirloni: [C: 03+1] network: labs: remove outdated entries [puppet] - 10https://gerrit.wikimedia.org/r/498698 (owner: 10Alex Monk) [10:11:44] 10Operations, 10Cloud-VPS, 10Discovery-Search (Current work), 10cloud-services-team (Kanban): rack/setup/install cloudelastic100[1-4].eqiad.wmnet systems - https://phabricator.wikimedia.org/T194186 (10GTirloni) [10:15:20] 10Operations, 10Cloud-VPS, 10DNS, 10Traffic, 10cloud-services-team (Kanban): Inconsistent lists of labs-ns* nameservers - https://phabricator.wikimedia.org/T205344 (10GTirloni) [10:16:58] 10Operations, 10Cloud-VPS, 10Discovery-Search (Current work), 10Patch-For-Review, 10cloud-services-team (Kanban): Setup elasticsearch on cloudelastic100[1-4] - https://phabricator.wikimedia.org/T214921 (10GTirloni) [10:17:14] 10Puppet, 10Cloud-VPS, 10serviceops, 10Patch-For-Review, and 2 others: upgrade simplelamp class (apache -> httpd and mysql -> mariadb) or deprecate it - https://phabricator.wikimedia.org/T215662 (10GTirloni) [10:28:24] 10Operations, 10Cloud-VPS, 10Shinken, 10Graphite, 10cloud-services-team (Kanban): Clean up labs graphite datapoints - https://phabricator.wikimedia.org/T111540 (10GTirloni) [10:28:38] 10Operations, 10Cloud-VPS, 10Patch-For-Review, 10cloud-services-team (Kanban): Set up LVS for labs dns recursors - https://phabricator.wikimedia.org/T119660 (10GTirloni) [10:28:59] 10Operations, 10Cloud-VPS, 10Patch-For-Review, 10cloud-services-team (Kanban): labmon1001 graphite instance archiver keeps archiving the same instances - https://phabricator.wikimedia.org/T120377 (10GTirloni) [10:29:20] 10Operations, 10Cloud-VPS, 10Patch-For-Review, 10cloud-services-team (Kanban): labmon1001 graphite instance archiver keeps archiving the same instances - https://phabricator.wikimedia.org/T120377 (10GTirloni) [10:29:32] 10Operations, 10Cloud-VPS, 10Traffic, 10HTTPS, 10cloud-services-team (Kanban): add a https-only option to dynamicproxy - https://phabricator.wikimedia.org/T120486 (10GTirloni) [10:31:06] 10Operations, 10Cloud-VPS, 10cloud-services-team (Kanban): investigate slapd memory leak - https://phabricator.wikimedia.org/T130593 (10GTirloni) [10:31:17] 10Operations, 10Cloud-VPS, 10LDAP, 10cloud-services-team (Kanban): investigate slapd memory leak - https://phabricator.wikimedia.org/T130593 (10GTirloni) [10:32:09] 10Operations, 10Cloud-VPS, 10DNS, 10Mail, and 3 others: Set SPF (... -all) for toolserver.org - https://phabricator.wikimedia.org/T131930 (10GTirloni) [10:32:35] 10Operations, 10Cloud-VPS, 10monitoring, 10cloud-services-team (Kanban): Have a paging check for Nova API accessible - https://phabricator.wikimedia.org/T133656 (10GTirloni) [10:35:19] 10Operations, 10Cloud-Services, 10cloud-services-team (Kanban): Designate seems very slow to delete records? - https://phabricator.wikimedia.org/T149057 (10GTirloni) [10:38:57] 10Operations, 10Cloud-VPS, 10DNS, 10Maps, and 2 others: multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert - https://phabricator.wikimedia.org/T161256 (10GTirloni) [10:41:15] 10Puppet, 10Cloud-VPS, 10cloud-services-team (Kanban): Invesitgate use of Puppet "environments" for per-project Puppet manifests - https://phabricator.wikimedia.org/T170370 (10GTirloni) [12:10:24] 10Operations, 10Striker, 10LDAP, 10cloud-services-team (Kanban): Store Wikimedia unified account name (SUL) in LDAP directory - https://phabricator.wikimedia.org/T148048 (10GTirloni) [12:11:55] PROBLEM - Host ms-be1041 is DOWN: PING CRITICAL - Packet loss = 100% [12:13:11] RECOVERY - Host ms-be1041 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [12:33:34] 10Operations, 10Operations-Software-Development, 10Continuous-Integration-Config, 10Patch-For-Review: Puppet tox: properly lint both Py2 and Py3 files - https://phabricator.wikimedia.org/T184435 (10Volans) Given that py2 EOL is at the end of 2019, I'm not sure it's worth to spend our energies to make this... [12:34:34] (03CR) 10Volans: "> Patch Set 4:" [puppet] - 10https://gerrit.wikimedia.org/r/498379 (https://phabricator.wikimedia.org/T144169) (owner: 10GTirloni) [12:41:12] (03CR) 10Volans: check_icinga: don't page on secondary host alerts (031 comment) [software/external-monitoring] - 10https://gerrit.wikimedia.org/r/498425 (owner: 10Volans) [12:49:20] 10Operations, 10Beta-Cluster-Infrastructure, 10Traffic, 10HTTPS: https://sv.wikipedia.beta.wmflabs.org/ has invalid certificate - https://phabricator.wikimedia.org/T202564 (10Zoranzoki21) 05Open→03Resolved [12:49:23] PROBLEM - puppet last run on labtestnet2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:14:41] PROBLEM - puppet last run on analytics1054 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:15:52] RECOVERY - puppet last run on labtestnet2002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [13:41:05] RECOVERY - puppet last run on analytics1054 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:13:08] 10Operations, 10Beta-Cluster-Infrastructure, 10Traffic, 10HTTPS: https://sv.wikipedia.beta.wmflabs.org/ has invalid certificate - https://phabricator.wikimedia.org/T202564 (10Krenair) 05Resolved→03Open @Zoranzoki21: they're just cherry-picks... specifically https://gerrit.wikimedia.org/r/#/c/497670/ an... [14:22:32] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://phabricator.wikimedia.org/T174916 [14:26:13] (03PS3) 10Daimona Eaytoy: Remove $wgAbuseFilterProfile [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486280 (https://phabricator.wikimedia.org/T191039) [14:26:17] (03PS8) 10Daimona Eaytoy: Remove $wgAbuseFilterRuntimeProfile [mediawiki-config] - 10https://gerrit.wikimedia.org/r/486470 (https://phabricator.wikimedia.org/T191039) [14:32:19] PROBLEM - proton endpoints health on proton1002 is CRITICAL: /{domain}/v1/pdf/{title}/{format}/{type} (Print the Foo page from en.wp.org in letter format) timed out before a response was received https://wikitech.wikimedia.org/wiki/Services/Monitoring/proton [14:33:35] RECOVERY - proton endpoints health on proton1002 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/proton [14:42:21] PROBLEM - pdfrender on scb1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://phabricator.wikimedia.org/T174916 [14:43:29] RECOVERY - pdfrender on scb1002 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.148 second response time https://phabricator.wikimedia.org/T174916 [14:51:23] PROBLEM - pdfrender on scb1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://phabricator.wikimedia.org/T174916 [14:53:59] RECOVERY - pdfrender on scb1002 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 9.616 second response time https://phabricator.wikimedia.org/T174916 [14:54:29] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 9.636 second response time https://phabricator.wikimedia.org/T174916 [14:57:47] PROBLEM - pdfrender on scb1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://phabricator.wikimedia.org/T174916 [14:58:19] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://phabricator.wikimedia.org/T174916 [15:00:17] RECOVERY - pdfrender on scb1002 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 6.186 second response time https://phabricator.wikimedia.org/T174916 [15:00:22] !log Restart pdfrender on scb1002 and scb1004 [15:00:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:02:01] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.002 second response time https://phabricator.wikimedia.org/T174916 [15:40:37] PROBLEM - puppet last run on etcd1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:59:30] (03PS9) 10Zoranzoki21: Enable VisualEditor at fiwikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/446583 (https://phabricator.wikimedia.org/T192135) [15:59:39] (03PS6) 10Zoranzoki21: IS.php: Add wgProofreadPagePageJoiner, set it per default on '-' and at zhwikisource on __PAGEJOIN__ [mediawiki-config] - 10https://gerrit.wikimedia.org/r/482502 (https://phabricator.wikimedia.org/T205826) [15:59:51] (03PS7) 10Zoranzoki21: Add category at wgGettingStartedExcludedCategories for srwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/482534 [16:00:00] (03PS8) 10Zoranzoki21: Add categories for all Croatian projects at wmgBabelMainCategory [mediawiki-config] - 10https://gerrit.wikimedia.org/r/482548 [16:12:13] RECOVERY - puppet last run on etcd1006 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures [16:17:31] PROBLEM - puppet last run on mw1268 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [16:43:51] RECOVERY - puppet last run on mw1268 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:23:36] (03PS1) 10Alex Monk: db-labs: add new slave deployment-db06 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/498729 (https://phabricator.wikimedia.org/T219087) [19:24:56] (03CR) 10CRusnov: [C: 03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/498645 (owner: 10Alex Monk) [19:57:55] (03PS4) 10Alex Monk: uwsgi::app: Handle ensure absent [puppet] - 10https://gerrit.wikimedia.org/r/498641 [19:58:37] (03CR) 10jerkins-bot: [V: 04-1] uwsgi::app: Handle ensure absent [puppet] - 10https://gerrit.wikimedia.org/r/498641 (owner: 10Alex Monk) [20:01:57] 10Operations, 10Puppet, 10Operations-Software-Development: wmf_style check in puppet silently fails when it finds the addition of an error that was also already occurring in the same file - https://phabricator.wikimedia.org/T219085 (10Krenair) [20:04:49] (03PS5) 10Alex Monk: uwsgi::app: Handle ensure absent [puppet] - 10https://gerrit.wikimedia.org/r/498641 [20:05:31] (03CR) 10jerkins-bot: [V: 04-1] uwsgi::app: Handle ensure absent [puppet] - 10https://gerrit.wikimedia.org/r/498641 (owner: 10Alex Monk) [20:06:48] 10Operations, 10Puppet, 10Operations-Software-Development: wmf_style check in puppet silently fails when it finds the addition of an error that was also already occurring in the same file - https://phabricator.wikimedia.org/T219085 (10Krenair) I made https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/498... [20:08:10] (03PS6) 10Alex Monk: uwsgi::app: Handle ensure absent [puppet] - 10https://gerrit.wikimedia.org/r/498641 [20:10:21] 10Operations, 10Puppet, 10Operations-Software-Development: wmf_style check in puppet silently fails when it finds the addition of an error that was also already occurring in the same file - https://phabricator.wikimedia.org/T219085 (10Krenair) a:03Krenair [20:10:29] (03PS1) 10Alex Monk: wmf_style lint: Don't silently fail on new errors that already appear on other lines [puppet] - 10https://gerrit.wikimedia.org/r/498731 (https://phabricator.wikimedia.org/T219085) [20:11:27] (03CR) 10jerkins-bot: [V: 04-1] wmf_style lint: Don't silently fail on new errors that already appear on other lines [puppet] - 10https://gerrit.wikimedia.org/r/498731 (https://phabricator.wikimedia.org/T219085) (owner: 10Alex Monk) [20:13:07] (03PS2) 10Alex Monk: wmf_style lint: Show new errors that already appear on other lines [puppet] - 10https://gerrit.wikimedia.org/r/498731 (https://phabricator.wikimedia.org/T219085) [20:23:11] (03PS1) 10Paladox: Fix groups to prevent javascript errors with url [software/gerrit] (wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/498733 [20:23:39] (03CR) 10Paladox: [V: 03+2 C: 03+2] Fix groups to prevent javascript errors with url [software/gerrit] (wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/498733 (owner: 10Paladox) [20:24:41] that's the second part of the ldap fix. [21:20:43] RECOVERY - Long running screen/tmux on notebook1003 is OK: OK: SCREEN detected but not long running. [21:44:11] (03PS2) 10Urbanecm: Initial configuration for hiwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/498317 (https://phabricator.wikimedia.org/T218155) [21:44:59] (03CR) 10jerkins-bot: [V: 04-1] Initial configuration for hiwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/498317 (https://phabricator.wikimedia.org/T218155) (owner: 10Urbanecm) [21:49:05] (03PS3) 10Urbanecm: Initial configuration for hiwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/498317 (https://phabricator.wikimedia.org/T218155) [21:49:51] (03CR) 10jerkins-bot: [V: 04-1] Initial configuration for hiwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/498317 (https://phabricator.wikimedia.org/T218155) (owner: 10Urbanecm) [21:51:33] (03PS4) 10Urbanecm: Initial configuration for hiwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/498317 (https://phabricator.wikimedia.org/T218155) [21:52:07] (03PS5) 10Urbanecm: Initial configuration for hiwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/498317 (https://phabricator.wikimedia.org/T218155) [21:52:53] (03CR) 10jerkins-bot: [V: 04-1] Initial configuration for hiwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/498317 (https://phabricator.wikimedia.org/T218155) (owner: 10Urbanecm) [21:58:29] (03PS6) 10Urbanecm: Initial configuration for hiwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/498317 (https://phabricator.wikimedia.org/T218155) [21:59:14] (03CR) 10jerkins-bot: [V: 04-1] Initial configuration for hiwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/498317 (https://phabricator.wikimedia.org/T218155) (owner: 10Urbanecm) [22:04:39] (03PS7) 10Urbanecm: Initial configuration for hiwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/498317 (https://phabricator.wikimedia.org/T218155) [22:05:38] (03CR) 10jerkins-bot: [V: 04-1] Initial configuration for hiwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/498317 (https://phabricator.wikimedia.org/T218155) (owner: 10Urbanecm) [22:06:45] (03PS8) 10Urbanecm: Initial configuration for hiwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/498317 (https://phabricator.wikimedia.org/T218155) [22:07:21] (03PS9) 10Urbanecm: Initial configuration for hiwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/498317 (https://phabricator.wikimedia.org/T218155) [22:08:11] (03CR) 10jerkins-bot: [V: 04-1] Initial configuration for hiwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/498317 (https://phabricator.wikimedia.org/T218155) (owner: 10Urbanecm) [22:12:19] (03PS10) 10Urbanecm: Initial configuration for hiwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/498317 (https://phabricator.wikimedia.org/T218155) [22:13:12] (03CR) 10jerkins-bot: [V: 04-1] Initial configuration for hiwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/498317 (https://phabricator.wikimedia.org/T218155) (owner: 10Urbanecm) [22:45:13] (03PS1) 10Jgreen: switch payments.wikimedia.org to codfw datacenter [dns] - 10https://gerrit.wikimedia.org/r/498745 [23:10:41] (03CR) 10Jgreen: [C: 03+2] switch payments.wikimedia.org to codfw datacenter [dns] - 10https://gerrit.wikimedia.org/r/498745 (owner: 10Jgreen) [23:18:01] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1004 is OK: OK: Less than 70.00% above the threshold [25.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=2&fullscreen