In a previous blog post, Rashmi showed us how to synchronise logs from CloudHub to an external logging system. This follows an increasing demand to utilise MuleSoft Anypoint CloudHub logs, events, and dashboard statistics as part of a broader monitoring strategy which aims to:
In this post we show how to access and process Anypoint API Gateway logs using AWS Lambda and the ELK stack.
API Gateway in CloudHub offers the ability to pull data using Logs, Analytics, and Dashboard Statistics API’s. I’ll call these the LAD API’s. An application invoking them needs to:
Logstash input http_poller plugin or elastic httpbeat does not meet these requirements, not yet.
So that brings us to Amazon Lambda—a serverless compute service that runs your code in response to the events and automatically manages the underlying compute resources for you. Lambda instances run briefly (for a period of seconds), complete their tasks and shutdown. We decided to use Lambda as a transport service for pulling API Gateway application logs, events, and dashboard statistics from CloudHub and transmitting them to Elasticsearch. It is important to mention that Lambda service is limited to Amazon cloud services platform. Other cloud services platform providers such as Google, IBM, and Microsoft have recently announced their own event triggered serverless applications. However, for the sake of this discussion, we will stick with Lambda. Figure below shows the high level solution with Lambda collectors.
The solution consists of:
Lambda Collector Type | CloudHub API |
Logs | https://anypoint.mulesoft.com/cloudhub/api/v2/applications/{domain}/instances/{instanceId}/logs?head=true&offset=<pointer_to_last_read_logline>&limit=<number_of_loglines> |
Events | https://anypoint.mulesoft.com/analytics/1.0/{organisationId}/events?format=json&apiIds=&apiVersionIds=&startDate=<start_date>&endDate=<end_date>&fields=Application.Application Name.Browser.City.Client IP.Continent.Country.Hardware Platform.Message ID.OS Family.OS Major Version.OS Minor Version.OS Version.Postal Code.Request Outcome.Request Size.Resource Path.Response Size.Response Time.Status Code.Timezone.User Agent Name.User Agent Version.Verb.Violated Policy Name |
Stats | https://anypoint.mulesoft.com/cloudhub/api/v2/applications/{domain}/dashboardStats?startDate=<start_date_iso8601_format>&endDate=<end_date_iso8601_format>&interval=<time_between_samples_in_ms> |
Note: Amazon offers managed Elasticsearch service. At the time of writing this blog, the managed service comes with Elasticsearch version 1.5.3. We decided to use the latest available version of Elasticsearch 2.3.3 for this demonstration as it includes fixes to bugs raised since 1.5.3 and it offers the ability to customize colors for visualisations and dashboards.
Below, we share some example dashboards we have built from the information captured (using Lambda collectors) from Anypoint CloudHub APIs. For client privacy reasons, we have blanked out some of the information captured in visualization legends and titles.
Statistics dashboard screenshot below shows cpu and memory usage information for an application hosted in Anypoint CloudHub.
Figure below shows custom dashboard for an application hosted in Anypoint CloudHub. Information presented on the dashboard includes:
Figure below shows events dashboard for all applications hosted in Anypoint CloudHub. Information presented on the dashboard includes:
The cost of bulding and sustaining such a solution is influenced by, number of Lambda collectors, the ELK stack topology and the amount of data transferred between API Gateway and Elasticsearch. Factors that can add to the cost include:
We discussed the use of Lambda collectors to retrieve API Gateway application logs, events, and dashboard statistics from Anypoint CloudHub and to publish them to Elasticsearch. We also shared examples of Kibana dashboards that can be produced using the information captured from Anypoint CloudHub. While information captured in the application logs can be reviewed and revised over time, it would be advantageous for the development team to collaborate with business teams and operational teams early in the development cycle, to put together basic guidelines on what should and should not be retained in these logs.