[02:44:28] <bd808>	 zhuyifei1999_: none of the new grid hosts will have data in graphite (I don't think). I know that at least the tracking of grid job state has moved from graphite to prometheus. We may still have some diamond system state collectors, but they are being removed too.
[02:46:02] <zhuyifei1999_>	 bd808: I was looking at nagf and the data were simply missing for a few hours
[02:46:15] <zhuyifei1999_>	 I don't think it could be a rebuild or anything
[02:46:39] <zhuyifei1999_>	 how do you check prometheus data again?
[02:46:54] <bd808>	 Probably the NFS service related lockup we have seen a lot in the last month or so.
[02:47:03] <zhuyifei1999_>	 yeah, likely
[02:47:21] * bd808 tried to remember himself how to check prometheus stuff 
[02:50:07] <bd808>	 zhuyifei1999_: https://tools-prometheus.wmflabs.org/tools/graph is the place you can go to poke at the data directly. Making nice dashbards is something we need to work on at dashboard/db/labs-project-board?orgId=1
[02:50:25] <bd808>	 worst paste ever -- https://grafana-labs.wikimedia.org/
[02:50:27] <zhuyifei1999_>	 ok
[02:52:19] <zhuyifei1999_>	 is it expected that most of the graphs on https://grafana-labs.wikimedia.org/dashboard/db/labs-project-board?orgId=1&var-project=tools&var-server=All don't seem to load?
[02:53:03] <zhuyifei1999_>	 (firefox 66 here)
[02:53:55] <bd808>	 I see some data for most everything there except the "reqstats" section
[02:56:02] <zhuyifei1999_>	 checking console, I think it's just it's sending too many requests and causing 502s
[02:56:04] <zhuyifei1999_>	 thanks
[14:17:44] <andrewbogott>	 !log cloudinfra moved cloudinfra-internal-puppetmaster01 to cloudvirt1026
[14:17:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL
[18:55:57] <sjoerddebruin>	 https://tools.wmflabs.org seems down?
[18:56:27] <pintoch>	 ah, it's not just me!
[18:58:00] <pintoch>	 (it started at 18:55 according to my monitoring)
[18:58:18] <sjoerddebruin>	 and it's back
[18:58:38] <pintoch>	 \o/
[20:03:54] <andrewbogott>	 !log tools moving tools-worker-1003 and tools-worker-1004 to eqiad1-r
[20:03:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:36:11] <andrewbogott>	 !log tools moving tools-worker-1006 and tools-worker-1007 to eqiad1-r
[22:36:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL