[00:37:52] PROBLEM - Puppet errors on tools-exec-1403 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:07:53] RECOVERY - Puppet errors on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [01:36:32] Cyberpower678: I haven't added in at-rest encryption yet, but https://tools.wmflabs.org/mysql-php-session-test/ shows database backed sessions working. [01:36:55] The library I wrote to do this is at https://github.com/bd808/toolforge-php-mysql [01:37:17] bd808: I'm 90% certain it has something to do with the encryptions [01:37:24] it's not published to packagist yet, but it is usable [01:37:51] Because when it was trying to encrypt the data with a non-existent protocal, it was writing blank data into the DB. [01:38:28] Naturally that means the sessions died immediately, and prevented logins, because it died immediately afterwards, but it was able to load. [01:38:47] Once I changed the protocal, it was able to load the first page, and then it 500'd on me. [01:38:59] So it could be an issue with actual decryption. [01:39:08] sounds like it [01:39:26] I would expect something in the error.log [01:39:45] But it works as it should on my machine, and the error.log just throws an unexpected end of file error regarding the webservice [01:40:04] Nothing PHP related. [01:40:25] Which leads me to believe maybe openssl needs reinstalling? [01:40:27] that makes me think that PHP is crashing hard [01:41:02] It's using PHP's OpenSSL extension [01:42:15] PROBLEM - Puppet errors on tools-exec-1438 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:43:09] Cyberpower678: maybe you could rig up a cli test for your encryption/decryption and see if you can get a better error message. [01:43:39] I'm going to add at-rest encryption to my library too, but I know how you distrust other people's code. :) [01:44:31] * bd808 -> dinner [01:44:35] bd808: It's not that I distrust, but rather favor my own because I know how it works, and can more easily fix it. :) [01:45:28] bd808: But I am starting to change my views on that. Slowly [01:48:00] PROBLEM - Puppet errors on tools-exec-1418 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [02:17:15] RECOVERY - Puppet errors on tools-exec-1438 is OK: OK: Less than 1.00% above the threshold [0.0] [02:23:00] RECOVERY - Puppet errors on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [02:43:13] PROBLEM - Puppet errors on tools-exec-1438 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:18:14] RECOVERY - Puppet errors on tools-exec-1438 is OK: OK: Less than 1.00% above the threshold [0.0] [03:18:23] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:53:22] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [03:54:14] PROBLEM - Puppet errors on tools-exec-1438 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [03:56:45] 10Toolforge, 10InternetArchiveBot (v1.5), 10User-Zppix, 10User-bd808: IABot Management interface: Make the login sessions last longer or add an option for "remember me" - https://phabricator.wikimedia.org/T170849#3473644 (10Cyberpower678) [04:24:14] RECOVERY - Puppet errors on tools-exec-1438 is OK: OK: Less than 1.00% above the threshold [0.0] [04:45:16] PROBLEM - Puppet errors on tools-exec-1438 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [04:49:56] 10Cloud-Services, 10Operations, 10ops-eqiad: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3473675 (10madhuvishy) @Cmjohnson These two need to be in the public vlan. [05:05:02] 10Data-Services, 10cloud-services-team (Kanban), 10Operations, 10ops-eqiad, 10Patch-For-Review: Degraded RAID on labsdb1001 - https://phabricator.wikimedia.org/T171538#3473678 (10Marostegui) 05Open>03Resolved ``` [root@labsdb1001 05:04 /root] # megacli -pdrbld -showprog -physdrv\[16:9\] -aALL Device... [05:50:13] RECOVERY - Puppet errors on tools-exec-1438 is OK: OK: Less than 1.00% above the threshold [0.0] [05:57:59] Cyberpower678: the library at https://github.com/bd808/toolforge-php-mysql handles session encryption now too. [06:11:13] PROBLEM - Puppet errors on tools-exec-1438 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:25:13] How can I set up uWSGI in Toolforge ? [06:29:18] r96340: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Python_.28uWSGI.29 [06:32:36] I tried "webservice start --backend=kubernetes python" and other something else and I have almost no basics so I don't know what exactly need to type in to start it. [06:33:27] I'm using PuTTY. If that will make diffrent. [06:34:44] PROBLEM - Puppet errors on tools-exec-1441 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:42:27] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1419 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:43:18] How can I set up uWSGI on Toolforge? I tried "webservice start --backend=kubernetes python" and other something else and I have almost no basics so I don't know what exactly need to type in to start it. I'm using PuTTY. If that will make diffrence. [06:45:45] PROBLEM - Puppet errors on tools-exec-1416 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:46:13] RECOVERY - Puppet errors on tools-exec-1438 is OK: OK: Less than 1.00% above the threshold [0.0] [06:47:44] r96340: if you got an error, please post the error message [06:48:35] or you may want to look into https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool a tutorial [06:50:24] It seems solved now. I think. Thx [06:58:26] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:00:51] np [07:09:46] RECOVERY - Puppet errors on tools-exec-1441 is OK: OK: Less than 1.00% above the threshold [0.0] [07:17:23] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1419 is OK: OK: Less than 1.00% above the threshold [0.0] [07:25:46] RECOVERY - Puppet errors on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [07:33:29] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [07:47:53] So now I setup-ed the webservice for python, how can I use only browser to run my code...? [08:09:36] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1401 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:10:12] PROBLEM - Puppet errors on tools-k8s-etcd-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:11:56] PROBLEM - Puppet errors on tools-worker-1026 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:12:44] PROBLEM - Puppet errors on tools-exec-1412 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:12:53] PROBLEM - Puppet errors on tools-worker-1007 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:13:03] PROBLEM - Puppet errors on tools-services-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:13:27] PROBLEM - Puppet errors on tools-exec-1419 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:13:55] PROBLEM - Puppet errors on tools-cron-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:14:09] PROBLEM - Puppet errors on tools-exec-1415 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:14:13] PROBLEM - Puppet errors on tools-exec-1432 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:14:29] PROBLEM - Puppet errors on tools-exec-1426 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:14:50] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1428 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [08:15:38] PROBLEM - Puppet errors on tools-docker-builder-05 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:16:08] PROBLEM - Puppet errors on tools-static-10 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:16:48] PROBLEM - Puppet errors on tools-exec-1416 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [08:17:09] PROBLEM - Puppet errors on tools-worker-1005 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:17:27] PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [08:17:47] PROBLEM - Puppet errors on tools-worker-1003 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:18:07] PROBLEM - Puppet errors on tools-redis-1002 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:18:22] PROBLEM - Puppet errors on tools-worker-1002 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:18:50] PROBLEM - Puppet errors on tools-exec-1425 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:19:00] PROBLEM - Puppet errors on tools-exec-1430 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:19:10] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1417 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:19:10] PROBLEM - Puppet errors on tools-worker-1004 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:19:15] PROBLEM - Puppet errors on tools-worker-1027 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:19:23] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:19:25] PROBLEM - Puppet errors on tools-exec-1433 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:20:21] PROBLEM - Puppet errors on tools-exec-1402 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:20:33] PROBLEM - Puppet errors on tools-docker-registry-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:20:45] PROBLEM - Puppet errors on tools-worker-1008 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:21:03] PROBLEM - Puppet errors on tools-elastic-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:22:20] PROBLEM - Puppet errors on tools-worker-1021 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:22:32] PROBLEM - Puppet errors on tools-proxy-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:22:34] PROBLEM - Puppet errors on tools-elastic-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:22:36] PROBLEM - Puppet errors on tools-prometheus-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:22:46] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:23:06] PROBLEM - Puppet errors on tools-worker-1001 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:23:52] PROBLEM - Puppet errors on tools-worker-1023 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:24:18] PROBLEM - Puppet errors on tools-exec-1407 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:24:38] PROBLEM - Puppet errors on tools-exec-1413 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:24:56] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1404 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:25:03] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1410 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:25:41] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1424 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [08:25:47] PROBLEM - Puppet errors on tools-proxy-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:25:49] PROBLEM - Puppet errors on tools-worker-1006 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [08:25:55] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:26:15] PROBLEM - Puppet errors on tools-exec-1408 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:27:34] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:27:36] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:27:36] PROBLEM - Puppet errors on tools-services-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [08:27:52] PROBLEM - Puppet errors on tools-mail is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:27:54] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1407 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [08:27:58] PROBLEM - Puppet errors on tools-bastion-05 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:28:10] <_joe_> I'm looking into those puppet failures [08:28:23] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:28:37] PROBLEM - Puppet errors on tools-exec-1406 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:28:45] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:28:53] PROBLEM - Puppet errors on tools-exec-1403 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:29:03] <_joe_> i'm looking into those failures [08:29:11] PROBLEM - Puppet errors on tools-exec-1431 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:29:13] PROBLEM - Puppet errors on tools-package-builder-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:29:47] PROBLEM - Puppet errors on tools-elastic-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:29:49] PROBLEM - Puppet errors on tools-exec-1422 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [08:29:57] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1408 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:30:46] PROBLEM - Puppet errors on tools-exec-1440 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:32:04] PROBLEM - Puppet errors on tools-webgrid-generic-1403 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:32:22] PROBLEM - Puppet errors on tools-worker-1017 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:32:34] PROBLEM - Puppet errors on tools-worker-1010 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:32:54] PROBLEM - Puppet errors on tools-k8s-etcd-03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:32:58] PROBLEM - Puppet errors on tools-flannel-etcd-03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:34:04] PROBLEM - Puppet errors on tools-checker-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:34:10] PROBLEM - Puppet errors on tools-k8s-etcd-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:36:00] PROBLEM - Puppet errors on tools-prometheus-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:37:14] PROBLEM - Puppet errors on tools-exec-1438 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:37:20] PROBLEM - Puppet errors on tools-exec-1414 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:37:22] PROBLEM - Puppet errors on tools-exec-1401 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:37:33] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:37:35] PROBLEM - Puppet errors on tools-checker-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:37:43] PROBLEM - Puppet errors on tools-exec-1405 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:37:55] PROBLEM - Puppet errors on tools-exec-1424 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:38:09] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1420 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:38:09] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:38:18] PROBLEM - Puppet errors on tools-grid-shadow is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:38:23] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1419 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:38:27] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:38:52] PROBLEM - Puppet errors on tools-worker-1025 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [08:39:03] <_joe_> !log restarted puppetmaster on tools [08:39:03] _joe_: Unknown project "restarted" [08:39:04] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1427 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:39:50] PROBLEM - Puppet errors on tools-puppetmaster-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:39:52] PROBLEM - Puppet errors on tools-exec-1421 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:40:24] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:42:10] PROBLEM - Puppet errors on tools-exec-1435 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:42:20] PROBLEM - Puppet errors on tools-paws-master-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:42:34] PROBLEM - Puppet errors on tools-logs-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:43:01] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1414 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:43:07] PROBLEM - Puppet errors on tools-grid-master is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:44:13] PROBLEM - Puppet errors on tools-puppetmaster-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:44:13] PROBLEM - Puppet errors on tools-exec-gift-trusty-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:44:15] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1405 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:47:32] PROBLEM - Puppet errors on tools-k8s-master-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:47:40] I setup-ed the webservice for python, so how can I use only browser to run my code now...? [08:48:25] Change to cellphone. [08:48:44] PROBLEM - Puppet errors on tools-flannel-etcd-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:49:54] PROBLEM - Puppet errors on tools-exec-1429 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:50:24] PROBLEM - Puppet errors on tools-worker-1022 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [08:51:58] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [08:54:27] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:54:53] PROBLEM - Puppet errors on tools-worker-1013 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:56:13] PROBLEM - Puppet errors on tools-worker-1018 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:56:15] PROBLEM - Puppet errors on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:57:22] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1425 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [08:57:28] PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:57:28] PROBLEM - Puppet errors on tools-flannel-etcd-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:58:08] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1418 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:58:16] PROBLEM - Puppet errors on tools-worker-1014 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:58:54] PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:59:04] PROBLEM - Puppet errors on tools-exec-1428 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:59:32] PROBLEM - Puppet errors on tools-worker-1016 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:59:36] PROBLEM - Puppet errors on tools-exec-1404 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [09:00:07] PROBLEM - Puppet errors on tools-exec-1409 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:00:31] PROBLEM - Puppet errors on tools-worker-1015 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [09:00:45] PROBLEM - Puppet errors on tools-exec-1441 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:02:09] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1409 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:02:31] PROBLEM - Puppet errors on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:02:34] PROBLEM - Puppet errors on tools-exec-1417 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:02:42] PROBLEM - Puppet errors on tools-redis-1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:02:54] PROBLEM - Puppet errors on tools-worker-1019 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:02:56] PROBLEM - Puppet errors on tools-worker-1009 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:04:12] PROBLEM - Puppet errors on tools-static-11 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:04:57] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1422 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:06:44] on CI self puppet master I got: Data retrieved from Integration/host/integration-puppetmaster01 is String, not Hash or nil at /etc/puppet/manifests/realm.pp:51 [09:06:49] which is: 51 $app_routes = hiera('discovery::app_routes') [09:07:27] PROBLEM - Puppet errors on tools-docker-registry-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:07:29] PROBLEM - Puppet errors on tools-exec-1442 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:08:03] PROBLEM - Puppet errors on tools-exec-1423 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:08:11] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1426 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:09:04] PROBLEM - Puppet errors on tools-exec-1418 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:09:08] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1415 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:09:10] PROBLEM - Puppet errors on tools-bastion-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:09:50] PROBLEM - Puppet errors on tools-worker-1012 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:09:56] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1421 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:10:12] PROBLEM - Puppet errors on tools-worker-1011 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [09:10:22] PROBLEM - Puppet errors on tools-paws-worker-1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:11:10] PROBLEM - Puppet errors on tools-exec-1427 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:12:37] filled as T171712 and apparently Giuseppe is looking into it [09:12:39] T171712: integration puppetmaster yield String, not Hash or nil at /etc/puppet/manifests/realm.pp:51 - https://phabricator.wikimedia.org/T171712 [09:13:09] PROBLEM - Puppet errors on tools-exec-1410 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [09:21:24] 10Tools, 10Wikimedia-General-or-Unknown: "Page Watchers" from article tool links is 404 - https://phabricator.wikimedia.org/T120927#1864966 (10Samtar) {F8854084} No longer appears in the Tools sidebar on the English Wikipedia [09:42:10] hi there [09:42:57] puppet should be fixed now [09:43:07] well it will be on tools whenever the puppet master rebases [09:45:37] 10Tools, 10Wikimedia-General-or-Unknown: "Page Watchers" from article tool links is 404 - https://phabricator.wikimedia.org/T120927#1864966 (10jcrespo) This points already to the internal link: https://en.wikipedia.org/wiki/MediaWiki:Histlegend Maybe https://en.wikipedia.org/w/index.php?title=English_Wikipedia... [09:47:17] hi [09:49:15] i approved for toolforge access. should i request for shell too? [09:52:40] hi i approved for toolforge access. should i request for shell too? [09:53:16] !help [09:53:17] assas: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [10:02:25] RECOVERY - Puppet errors on tools-flannel-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:02:37] RECOVERY - Puppet errors on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:04:05] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1427 is OK: OK: Less than 1.00% above the threshold [0.0] [10:04:13] RECOVERY - Puppet errors on tools-package-builder-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:04:47] RECOVERY - Puppet errors on tools-exec-1422 is OK: OK: Less than 1.00% above the threshold [0.0] [10:04:49] RECOVERY - Puppet errors on tools-puppetmaster-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:04:53] RECOVERY - Puppet errors on tools-worker-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [10:05:07] RECOVERY - Puppet errors on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [10:05:46] RECOVERY - Puppet errors on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:05:56] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [10:06:14] RECOVERY - Puppet errors on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [10:06:15] RECOVERY - Puppet errors on tools-worker-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [10:07:04] chasemp [10:07:05] RECOVERY - Puppet errors on tools-webgrid-generic-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [10:07:21] RECOVERY - Puppet errors on tools-worker-1017 is OK: OK: Less than 1.00% above the threshold [0.0] [10:07:31] RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0] [10:07:33] awwwa: he is most probably not connected yet [10:07:33] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [10:07:33] RECOVERY - Puppet errors on tools-worker-1010 is OK: OK: Less than 1.00% above the threshold [0.0] [10:07:36] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [10:07:54] RECOVERY - Puppet errors on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [10:07:54] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [10:07:54] RECOVERY - Puppet errors on tools-worker-1019 is OK: OK: Less than 1.00% above the threshold [0.0] [10:07:56] RECOVERY - Puppet errors on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [10:08:07] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [10:08:13] RECOVERY - Puppet errors on tools-worker-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [10:08:23] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [10:08:39] RECOVERY - Puppet errors on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [10:08:46] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [10:08:54] RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0] [10:08:56] RECOVERY - Puppet errors on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [10:09:04] RECOVERY - Puppet errors on tools-exec-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [10:09:10] RECOVERY - Puppet errors on tools-exec-1431 is OK: OK: Less than 1.00% above the threshold [0.0] [10:09:12] RECOVERY - Puppet errors on tools-static-11 is OK: OK: Less than 1.00% above the threshold [0.0] [10:09:16] !help [10:09:16] uuut: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [10:09:33] RECOVERY - Puppet errors on tools-worker-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [10:09:37] RECOVERY - Puppet errors on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [10:09:47] RECOVERY - Puppet errors on tools-elastic-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:09:57] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [10:10:31] RECOVERY - Puppet errors on tools-worker-1015 is OK: OK: Less than 1.00% above the threshold [0.0] [10:10:44] RECOVERY - Puppet errors on tools-exec-1440 is OK: OK: Less than 1.00% above the threshold [0.0] [10:10:44] RECOVERY - Puppet errors on tools-exec-1441 is OK: OK: Less than 1.00% above the threshold [0.0] [10:12:09] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [10:12:25] RECOVERY - Puppet errors on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [10:12:29] RECOVERY - Puppet errors on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [10:12:33] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [10:12:33] RECOVERY - Puppet errors on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [10:12:35] RECOVERY - Puppet errors on tools-checker-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:12:41] RECOVERY - Puppet errors on tools-redis-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [10:12:53] RECOVERY - Puppet errors on tools-k8s-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:12:55] RECOVERY - Puppet errors on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0] [10:12:58] RECOVERY - Puppet errors on tools-flannel-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:13:08] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [10:13:50] RECOVERY - Puppet errors on tools-worker-1025 is OK: OK: Less than 1.00% above the threshold [0.0] [10:14:06] RECOVERY - Puppet errors on tools-checker-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:14:10] RECOVERY - Puppet errors on tools-k8s-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:14:52] RECOVERY - Puppet errors on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [10:14:56] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1422 is OK: OK: Less than 1.00% above the threshold [0.0] [10:15:13] RECOVERY - Puppet errors on tools-worker-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [10:15:23] RECOVERY - Puppet errors on tools-paws-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [10:15:59] RECOVERY - Puppet errors on tools-prometheus-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:16:11] RECOVERY - Puppet errors on tools-exec-1427 is OK: OK: Less than 1.00% above the threshold [0.0] [10:17:12] RECOVERY - Puppet errors on tools-exec-1435 is OK: OK: Less than 1.00% above the threshold [0.0] [10:17:15] RECOVERY - Puppet errors on tools-exec-1438 is OK: OK: Less than 1.00% above the threshold [0.0] [10:17:23] RECOVERY - Puppet errors on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [10:17:27] RECOVERY - Puppet errors on tools-docker-registry-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:17:32] RECOVERY - Puppet errors on tools-exec-1442 is OK: OK: Less than 1.00% above the threshold [0.0] [10:17:34] RECOVERY - Puppet errors on tools-logs-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:17:44] RECOVERY - Puppet errors on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [10:17:56] RECOVERY - Puppet errors on tools-exec-1424 is OK: OK: Less than 1.00% above the threshold [0.0] [10:18:02] RECOVERY - Puppet errors on tools-exec-1423 is OK: OK: Less than 1.00% above the threshold [0.0] [10:18:06] RECOVERY - Puppet errors on tools-grid-master is OK: OK: Less than 1.00% above the threshold [0.0] [10:18:08] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [10:18:11] RECOVERY - Puppet errors on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [10:18:13] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1426 is OK: OK: Less than 1.00% above the threshold [0.0] [10:18:17] RECOVERY - Puppet errors on tools-grid-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [10:18:26] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1419 is OK: OK: Less than 1.00% above the threshold [0.0] [10:18:28] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [10:19:02] RECOVERY - Puppet errors on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [10:19:08] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [10:19:10] RECOVERY - Puppet errors on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:19:12] RECOVERY - Puppet errors on tools-puppetmaster-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:19:14] RECOVERY - Puppet errors on tools-exec-1432 is OK: OK: Less than 1.00% above the threshold [0.0] [10:19:37] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [10:19:45] !help [10:19:45] uyuuhyuy: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [10:19:51] RECOVERY - Puppet errors on tools-exec-1421 is OK: OK: Less than 1.00% above the threshold [0.0] [10:19:56] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1421 is OK: OK: Less than 1.00% above the threshold [0.0] [10:20:14] RECOVERY - Puppet errors on tools-k8s-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:20:22] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [10:21:07] !help [10:21:07] ttrryy: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [10:21:08] RECOVERY - Puppet errors on tools-static-10 is OK: OK: Less than 1.00% above the threshold [0.0] [10:21:56] RECOVERY - Puppet errors on tools-worker-1026 is OK: OK: Less than 1.00% above the threshold [0.0] [10:22:20] RECOVERY - Puppet errors on tools-paws-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:22:44] RECOVERY - Puppet errors on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [10:22:46] RECOVERY - Puppet errors on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [10:22:55] RECOVERY - Puppet errors on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [10:23:03] RECOVERY - Puppet errors on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:23:05] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [10:23:05] RECOVERY - Puppet errors on tools-redis-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [10:23:25] RECOVERY - Puppet errors on tools-exec-1419 is OK: OK: Less than 1.00% above the threshold [0.0] [10:23:52] RECOVERY - Puppet errors on tools-cron-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:24:08] RECOVERY - Puppet errors on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [10:24:12] RECOVERY - Puppet errors on tools-worker-1027 is OK: OK: Less than 1.00% above the threshold [0.0] [10:24:12] RECOVERY - Puppet errors on tools-exec-gift-trusty-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:24:14] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [10:24:24] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [10:24:29] RECOVERY - Puppet errors on tools-exec-1426 is OK: OK: Less than 1.00% above the threshold [0.0] [10:24:51] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [10:25:17] RECOVERY - Puppet errors on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [10:25:39] RECOVERY - Puppet errors on tools-docker-builder-05 is OK: OK: Less than 1.00% above the threshold [0.0] [10:26:05] RECOVERY - Puppet errors on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:26:47] RECOVERY - Puppet errors on tools-exec-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [10:27:10] RECOVERY - Puppet errors on tools-worker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [10:27:25] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:27:27] RECOVERY - Puppet errors on tools-proxy-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:27:31] RECOVERY - Puppet errors on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:27:34] RECOVERY - Puppet errors on tools-elastic-03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:27:38] RECOVERY - Puppet errors on tools-prometheus-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:28:07] RECOVERY - Puppet errors on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [10:28:21] RECOVERY - Puppet errors on tools-worker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [10:28:45] RECOVERY - Puppet errors on tools-flannel-etcd-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:28:51] RECOVERY - Puppet errors on tools-exec-1425 is OK: OK: Less than 1.00% above the threshold [0.0] [10:28:59] RECOVERY - Puppet errors on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0] [10:29:09] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [10:29:09] RECOVERY - Puppet errors on tools-worker-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [10:29:23] RECOVERY - Puppet errors on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [10:29:39] RECOVERY - Puppet errors on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [10:29:56] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [10:29:56] RECOVERY - Puppet errors on tools-exec-1429 is OK: OK: Less than 1.00% above the threshold [0.0] [10:30:28] RECOVERY - Puppet errors on tools-worker-1022 is OK: OK: Less than 1.00% above the threshold [0.0] [10:30:32] RECOVERY - Puppet errors on tools-docker-registry-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:30:42] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1424 is OK: OK: Less than 1.00% above the threshold [0.0] [10:30:44] RECOVERY - Puppet errors on tools-worker-1008 is OK: OK: Less than 1.00% above the threshold [0.0] [10:30:50] RECOVERY - Puppet errors on tools-worker-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [10:31:19] RECOVERY - Puppet errors on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [10:32:04] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [10:32:21] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [10:32:21] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1425 is OK: OK: Less than 1.00% above the threshold [0.0] [10:32:45] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [10:33:52] RECOVERY - Puppet errors on tools-worker-1023 is OK: OK: Less than 1.00% above the threshold [0.0] [10:34:20] RECOVERY - Puppet errors on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [10:35:04] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [10:48:05] !help [10:48:05] rrrreee: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [10:50:30] !help [10:50:30] rrrreee: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [10:55:17] rrrreee: hi, can we help? [10:56:00] i have a question [10:56:42] i approved for toolsforge. should i requst for shell access too? [10:57:57] rrrreee: What's your LDAP/wikitech username? Have you followed the guide at https://wikitech.wikimedia.org/wiki/Help:Getting_Started ? [10:58:41] yeah. how can request for shell access? [11:00:00] Well if you've done that and been approved for toolsforge, you already have SSH access to the Tools VPS project [11:02:28] yes i have SSH key, made my tools and can login with putty. but seems im not in shell users group [11:03:15] and shell user request page already is a draft page [11:04:28] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [11:06:24] rrrreee: Are you expecting production shell access? :/ [11:08:31] yeah [11:09:00] And why do you require production shell access? (https://wikitech.wikimedia.org/wiki/Production_shell_access) [11:09:23] runing bot [11:09:32] flagged bot [11:10:56] You can do that on toolforge :) have a look at https://wikitech.wikimedia.org/wiki/Help:Toolforge#What_is_a_Tool_account.3F - you have access to everything you need to run a bot [11:11:46] ok, its mean no need to shell accsess for runing bots? [11:13:28] Well when you log into toolforge using PUTTY, that is a shell you can use to run tools such as bots. https://wikitech.wikimedia.org/wiki/Help:Toolforge/Developing may have some more useful information [11:13:41] You don't need to request production shell access to run a bot [11:14:35] thanks alot! [11:15:00] No worries, feel free to stick around here - are you new to developing Wikipedia bots? :) [11:16:24] yes :) [11:16:47] Awesome, well we're glad to have you here! :) [12:33:29] (03Draft2) 10MarcoAurelio: Update composer.json [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/367879 [12:33:36] (03CR) 10jerkins-bot: [V: 04-1] Update composer.json [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/367879 (owner: 10MarcoAurelio) [12:35:32] (03PS3) 10MarcoAurelio: Update composer.json [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/367879 [12:44:56] !log wikidata-dev soft restarting federated-wikis2 [12:44:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [12:52:07] (03Draft2) 10MarcoAurelio: Adding code of conduct [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/367884 [12:52:32] bd808: cool, so does that mean you figured out what went wrong with my decryption system? [12:58:56] (03CR) 10MarcoAurelio: [C: 032] Adding code of conduct [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/367884 (owner: 10MarcoAurelio) [12:59:22] (03Merged) 10jenkins-bot: Adding code of conduct [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/367884 (owner: 10MarcoAurelio) [13:01:48] !log tools.stewardbots [[gerrit:367884|Added code of conduct]] [13:01:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [13:43:11] 10Tools, 10Wikimedia-General-or-Unknown: "Page Watchers" from article tool links is 404 - https://phabricator.wikimedia.org/T120927#3474736 (10MZMcBride) There are user scripts that still reference this old tool: PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:18:10] PROBLEM - Puppet errors on tools-worker-1005 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:25:21] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [14:53:09] RECOVERY - Puppet errors on tools-worker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [15:06:25] Cyberpower678: nope. I just found and used a well documented and reviewed openssl crypto wrapper. [15:07:19] Grrr [15:07:36] So what is breaking mine. :/ [15:10:19] bd808: I still feel like something is wrong with PHP on toolforge though, and should be investigated. [15:11:19] Because my very sensitive webserver on my machine handled it just fine, while toolforge's well configured webserver fails miserably [15:11:43] Cyberpower678: we need a targeted testcase that reproduces the problem. [15:12:01] I'll see if I can create that CLI script to replicate the issue. [15:13:02] the main thing I know about crypto is that it is easy to mess up the implementation [15:13:04] Because my encryption routines uses randomized keys, coupled with my OAuth consumer key, which should make it impossible to crack, to the outside. [15:13:33] *consumer secret [15:13:41] s/impossible/harder/ [15:14:37] 10Tools, 10Wikimedia-General-or-Unknown: "Page Watchers" from article tool links is 404 - https://phabricator.wikimedia.org/T120927#3475007 (10Samtar) Quite possibly! Are there any plans to fix the tool? If not, redirecting from the tool to `index.php?title=[title]&action=info#mw-pageinfo-watchers` could be a... [15:14:53] My maths degree makes me wince at "impossible" and similar absolute statements without a peer-reviewed formal proof ;) [15:15:04] bd808: You'd need to get your hands on the secret key. Or you could just try to brute force the entire thing. You'll have someone's session data in about a million years. :p [15:15:46] bd808: so yes, not impossible, but painfully difficutly [15:15:54] *difficult [15:16:05] agreed :) [15:17:53] bd808: the beautiful thing about my setup is, that if the OAuth consumer ever needed to be refreshed, or changed, that change automatically wipes out existing sessions since it can no longer decrypt the older sessions. :D [15:18:23] Otherwise users will see the interface being weird when they use an old session with a no longer functional consumer. :p [15:18:58] but that happens on the other end too. If the consumer is revoked then all grants tied to it are invalid [15:19:11] oh, you mean your app would be aware [15:19:35] Essentially. If it can't decrypt the session handler would get blank. Should anyways [15:20:05] That forces the user to log back in and start a new sessions which have them using the new consumer. [15:20:37] *If it can't decrypt the data, the session handler would get blank data. [15:21:07] I don't get it. what's the point of session encryption when it's on server-side with valid access restrictions? if someone can actually break access control they can essentially hack into the memory / read the encryption key / etc. right? [15:21:36] zhuyifei1999_: well yes and no [15:21:43] DB access is different from tool access. [15:22:25] * zhuyifei1999_ thinks it's easier to break tool access than db access [15:22:28] Part of the key is on the tool, so they would need both access credentials to fully decrypt the data [15:22:39] tool access requires SSH [15:23:27] or a php exploit [15:23:36] or nfs exploit [15:24:04] bd808: I don't think my tool can be exploited to produce that information though. [15:24:20] Feel free to report any security vulnerabilities though. [15:24:38] Cyberpower678: will try :) [15:24:41] which tool? [15:24:44] zhuyifei1999_: I welcome it. [15:24:52] https://tools.wmflabs.org [15:24:55] https://tools.wmflabs.org/iabot [15:24:58] k [15:25:17] There's also an API, https://tools.wmflabs.org/iabot/api.php [15:26:10] zhuyifei1999_: Most of my efforts in development were focused towards security. [15:27:00] k [15:30:54] Cyberpower678: is it this repo? https://github.com/cyberpower678/Cyberbot_II [15:31:28] zhuyifei1999_: https://github.com/cyberpower678/Cyberbot_II/tree/master/IABot/www [15:31:36] k [15:37:06] Cyberpower678: this bug report in libressl may be related to your problems -- https://github.com/libressl-portable/portable/issues/79 -- segfaults when using aes-256-cbc-hmac-sha1. [15:37:42] bd808: That exactly the encryption protocol I'm using. [15:37:52] I know. I was googling [15:38:08] Grunt. Then let's try a different one. [15:38:23] Preferably with an IV length > 0 [15:39:57] also reported at openssl directly -- https://rt.openssl.org/Ticket/Display.html?id=3024&user=guest&pass=guest -- with a resolution of "The current releases warn about those modes not being supported." [15:40:17] I saw no such warning. :/ [15:41:00] it probably is only in a readme or maybe when using the openssl cli [15:42:40] Well let's try it out then. [15:42:48] * Cyberpower678 clears the old sessions table [15:42:52] the things I'm reading seem to indicate that the segfaults started when Lucky 13 () was patched [15:47:36] zhuyifei1999_: find something yet? :) [15:47:51] * zhuyifei1999_ is watching TV right now [15:47:57] will check later [15:48:00] Ok [15:51:37] Cyberpower678: ok, I'm starting to read the code [15:58:42] gah I still feel like php is a trap. globals... [16:00:39] bd808: I changed the encryption and it seems to have fixed the problem. Thanks for your help. :D [16:00:56] The interface is now using the DB to handle sessions. [16:01:21] Let's if sessions remain persistent now, of if I fucked up somewhere. :p [16:02:34] bd808: Funny that of all of the encryptions, I have to pick the one that segfaults PHP. :p [16:10:54] 10Toolforge, 10InternetArchiveBot (v1.5), 10User-Zppix, 10User-bd808: IABot Management interface: Make the login sessions last longer or add an option for "remember me" - https://phabricator.wikimedia.org/T170849#3475200 (10Cyberpower678) The problem was that AES-256-CBC-HMAC-SHA1 was causing segmentation... [16:26:56] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:45:10] PROBLEM - Puppet errors on tools-worker-1027 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:48:11] Cyberpower678: nice. glad I could help point you in the right direction on that [16:48:19] :) [16:49:28] bd808: wow. There are already 2823 session entries in there. :p [16:50:12] really? that's kind of crazy, but hopefully just means that the tool is getting some love and use :) [16:50:41] someday™ we will get metrics for toolforge usage... someday [16:51:48] Cyberpower678: unrelated to my attempt to hack the tool, but... why are you loading from fonts.gstatic.com? [16:52:13] oh man. I need to test and deploy that patch zhuyifei1999_ [16:52:23] bd808: lol :P [17:01:01] bd808: oh when it's done, please ping me [17:06:56] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [17:08:00] 10Toolforge, 10InternetArchiveBot (v1.5), 10User-Zppix, 10User-bd808: IABot Management interface: Make the login sessions last longer or add an option for "remember me" - https://phabricator.wikimedia.org/T170849#3475412 (10Cyberpower678) 05Open>03Resolved [17:17:24] Cyberpower678: in case you didn't see, I sent you a few PMs [17:25:12] RECOVERY - Puppet errors on tools-worker-1027 is OK: OK: Less than 1.00% above the threshold [0.0] [17:55:12] 10Cloud-VPS, 10Operations: rack/setup/install labtestmetal2001.codfw.wmnet - https://phabricator.wikimedia.org/T168891#3475586 (10RobH) [17:55:34] 10Cloud-VPS, 10Operations, 10Patch-For-Review: rack/setup/install labtestpuppetmaster2001 - https://phabricator.wikimedia.org/T167157#3475589 (10RobH) [18:44:45] what's current practice regarding puppetizing a VPS instance? [18:45:52] MaxSem: "puppetize all the things", but I'm not sure if we solved the issue of 'all puppet manifests are in an ops-only repository" [18:46:36] I see only quary as an example of user-facing puppetized service. anything else? [18:46:45] *quarry [18:47:39] * zhuyifei1999_ wants per-project puppet in non-self-hosted-puppetmaster-way badly [18:48:34] MaxSem: I don't have a good overview of all projects, but toolforge is obviously user-facing ;-) Then we have PAWS, various monitoring stuff, I think most ORES stuff is also on VPS [18:49:55] Personally I would love to put my system configuration for my new data store in Puppet but I don't know how to Puppetz [18:51:08] I know how to puppet, but would like to not have to go to ops for review and have to adhere to production standards [18:52:16] So basically what we need is a cloud puppet repository [18:52:26] Right. If I remember correctly, newer puppet masters allow to have multiple repositories of modules, which would allow for non-self-hosted-puppetmaster situations. I think the only way to do puppet as non-ops currently is to essentially fork and regularly merge the repository. [18:55:37] Or, is there anything stopping me from having my own puppet repository that I use for my purposes? [18:55:44] I'm not sure where the limitation is [18:57:07] harej: the main thing to keep in mind is that the instance needs to track the ops/puppet repo because otherwise it may not get essential configuration updates [18:57:34] one thing you probably can do, though, is write manifests locally and apply them with 'puppet apply manifest.pp' manually [18:58:16] So unless I go out of my way there's no getting around the ops/puppet repo since it has some fundamental settings that should be universal among VPS instances? [18:58:29] then you can't depend on existing manifests in ops/puppet, but that's maybe not necessarily a bad thing, as that means your manifests break less quickly :-) [18:59:09] right. [19:01:09] Meaning I would have my own splinter repository that I would need to manually keep in sync with ops/puppet [19:03:14] that is one option, the other option is to have something completely independent, and apply that in a different way (i.e. using puppet apply rather than puppet agent) [19:03:50] same syntax, but puppet agent gets the manifests from the puppetmaster, while puppet apply reads from a file [19:05:00] that second puppet idea can get really messy though in practice. That's how labs-vagrant used to work and it ended up with the 2 Puppet repos fighting over some files [19:06:05] Goodness, don't you hate when the machines are fighting each other? [19:06:19] today the best/easiest thing to do is run a project local puppetmaster that tracks ops/puppet and add in your own local commits for roles that you know won't be landed upstream [19:06:55] there is an idea at T170370 for a nicer system [19:06:56] T170370: Invesitgate use of Puppet "environments" for per-project Puppet manifests - https://phabricator.wikimedia.org/T170370 [19:07:07] bd808: the problem with that is that major breakage does not get solved for you (e.g. the LDAP thing last week) [19:07:09] but it needs investigation and proff-of-concept work [19:08:02] today if you detach a VPS from ops/puppet entirely I would give it 3 weeks tops before it is unreachable [19:08:33] without LDAP you are going to be stuck and other things could change too [19:08:58] Yeah, so somehow you still want the repo to automatically track, but with changes applied [19:09:10] that's what the existing role does [19:09:16] that's worked for 3+ years [19:09:34] its what we built out for deployment-prep in the eqiad migraiton [19:09:34] yes, unless you have local modifications [19:09:43] no you can have them [19:09:47] the pull rebases [19:09:52] and stashes dirty files too [19:10:17] it breaks occasionally, but mostly when you are modifying an upstream module [19:10:21] hm, that sounds sane enough. Only risk is if the rebase fails, but then all bets are off anyway [19:10:35] if you keep you things separate it works pretty solidly [19:10:40] *nod* [19:12:55] this is the magic updater I started for that -- https://github.com/wikimedia/puppet/blob/production/modules/puppetmaster/files/git-sync-upstream [19:16:23] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:21:08] !log mediawiki-vagrant Fixed broken puppet on mwv-builder-02.mediawiki-vagrant.eqiad.wmflabs; exim package had busted apt-get. (T168581) [19:21:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mediawiki-vagrant/SAL [19:21:10] T168581: Skill Share: Setup MediaWiki development environment - https://phabricator.wikimedia.org/T168581 [19:22:06] !log mediawiki-vagrant Started mw-v container on mwv-builder-02.mediawiki-vagrant.eqiad.wmflabs (T168581) [19:22:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mediawiki-vagrant/SAL [19:45:51] bd808: ooooh! *bookmarks* [19:51:09] it didn't used to abort on dirty diff and instead did a stash and pop. I keep forgetting that got removed. [20:01:21] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [20:48:23] 10Cloud-VPS, 10cloud-services-team (Kanban), 10Operations: Switch to new labs puppetmasters - https://phabricator.wikimedia.org/T171786#3476152 (10Andrew) [21:07:16] 10Cloud-Services, 10wikitech.wikimedia.org, 10BetaFeatures, 10Edit-Review-Improvements, and 3 others: ERI requesting opt-in on wikitech but not available - https://phabricator.wikimedia.org/T165822#3476251 (10Etonkovidova) Checked in wikitech.wikimedia.org - the guided RC page filter invite is not displaye... [22:33:24] PROBLEM - Puppet errors on tools-bastion-03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [22:33:48] !log tools hotpatching an hiera value on tools master to see effects [22:33:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:53:26] RECOVERY - Puppet errors on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [22:57:21] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:59:03] I made a thing -- https://packagist.org/packages/bd808/toolforge-mysql -- right now its major claim to fame is easy setup and use of mysql session storage for Toolforge tools.