Splunk - troubleshooting remote agents with the phonehome logs

An issue popped up the other day that was pretty interesting (from a Splunk admin perspective) so figured I would share. This will likely be pretty long but hopefully someone will benefit. We had a number of servers with Splunk universal forwarders stop sending logs but in doing a spot check on their firewalls the server owner noticed traffic still going to our Splunk infrastructure backend. What had happened? The answer to the question lies in the UF phone home logs - do you know where they are and how to read them?

A change in log format for Splunk UF 6.x relative to tracking apps using the Deployment Server

I realized two things yesterday as I was troubleshooting various Splunk things. The first relates to having multiple input configs sent to a centralized syslog server. The second relates to changes to the internal 6.x UF logs as it relates to tracking apps that have been installed or removed.