July 23, 2020
Today we managed to get some insight into a developer using nerd.vision on his product and services. The developer works exclusively on Fusion Reactor Cloud, an application performance monitoring solution from Intergral which provides this product to thousands of customers worldwide. nerd.vision is a new line product from the same company and being able to see a developer use nerd.vision in a development environment provided valuable feedback and real world situations to improve what we deliver to customers.
I felt the need to use nerd.vision when I received a customer support case for an issue affecting one of our production systems. This customer reported that they were unable to see a very small subset of their data in our cloud platform.
We use a multi-tier pipeline of data ingest processors to ingest customer data into our system. This particular support case indicated that one of those systems has an issue present that was preventing a certain type of data from being visible to the end user.
Given the complex nature of our data ingest system, I had to investigate several avenues in order to discover in which service the issue resided.
Once the culprit service was identified, we were then able to resolve the issue and restore service to our end user.
nerd.vision empowered me to explore this customer’s data in real time, allowing me to observe the problem while it was happening, thereby providing me with a wealth of information I could then use to find the root cause and resolve the problem.
This was enough for me to see that valid ids had been obtained during the processing of the customer data, indicating that it had all been ingested into our data stores. As nerd.vision exposed these ids to me, I was able to perform a manual lookup in our data store leading me to discover the data had simply been tagged incorrectly thereby causing it to not be visible to the customer.
I simply adjusted this tag manually and the support case was resolved.
The Conditional Tracepoints were the most valuable tool available to me. Considering the 100’s of clients we have sending data to our systems every minute, I can only imagine how painful it would have been to have to filter through so many snapshots to find the one in particular I was after.
I also liked the real time nature of the nerd.vision platform as it meant I wouldn’t have to go down traditional routes of redeploying services with debug output or additional logging installed. I was able to simply apply a tracepoint with 1 click and immediately see the information I was after.