Prefered Way to Fetch Logs from Azure?

Hi,

We are obviously beholden to events in the platform getting logged in a timely manner - due to the Eventhub fetcher being a fetcher, there is a slight delay caused by Logpoint due to the fetch interval, but Logpoint should be able to handle that (especially with the new “Delay Alert” option). But if the events aren’t there, there isn’t much that can be done about that - I haven’t heard that about EventHubs, and I don’t know what the Microsoft SLAs are for that, although I have heard that events through the Office 365 Management API can sometimes be delayed quite significantly on the Microsoft side.

Using the Log Analytics Workspace fetcher would be an option, but again Azure Monitor would be configured to send the Entra ID events to a Log Analytics Workspace instead of an EventHub - so whether that would be any more reliable is an unanswered question. The approach so far has been via EventHubs, but if the data is sent to a Log Analytics Workspace and an appropriate KQL query can be constructed to retrieve them with the Log Analytics Workspace fetcher it would be interesting to see whether events arrive there earlier and more consistently.

Most promising are probably our very latest changes to the Universal REST Fetcher that enables us to use the Graph API to query the endpoints directly - now again, there is the risk that they wouldn’t return the events to us in time either when we ask for them, but there is probably less potential of something introducing a delay on the Microsoft side. The Graph API will become an option within Q1 at a guess.

The intention of the “delay alert” mechanism is to set it as high as the highest delay we would ever expect if it is variable - but we are really talking about something like perhaps 15 minutes or so, so 24 hours is of course pretty much unusable.

I have not heard from anyone else that they have had THAT much of a delay through their EventHub. What do Microsoft say? I remember seeing an SLA for their Office 365 Management API and it was frightingly long, but weirdly anything I can find on EventHubs is from Chinese Microsoft pages.

There does seem to be something called “Throughput Units” and overall limits on a Namespace - is there anything else going on in that tenant that could somehow impact this, or perhaps any other EventHubs that might be eating up Throughput?

Here is a test python script (remove the .txt from the file name) you can use with your test eventhub, to see the timestamps of the incoming logs.

You just need to adjust the values in CLIENT_CREDENTIALS and install the azure-eventhub package like this:

pip install azure-eventhub

 It may take a while, but you will see it.