48
loading...
How to get mobile app logs to Elasticsearch using Bugfender
When we speak to our customers, they tell us that Bugfender is great for capturing application logs and then pinpointing the logs of a specific user. By pulling all the logs from a developer’s mobile and web apps and centralizing them in a single dashboard, Bugfender gives that developer a 360-degree sweep of their entire user base.
But what if you want to use this data somewhere else? What if you could provide this data within an Elasticsearch database?
By doing that, you could open up heaps of new possibilities. You could go far beyond fixing bugs: you could perform analytics, for example, or correlate security events. With Elasticsearch, you can perform a free-text search to make aggregations, histograms, and statistics, and you can hook up to Kibana, which allows you to view security information and event management (SIEM).
Well, if this prospect has whetted your appetite, then great news: there is a way to use the Bugfender API to download your logs and push them to an Elasticsearch database, in real time.
(Elasticsearch is merely an example. You could use the same technique to push Bugfender to your favorite database, like MySQL, Postgres, BigQuery, HBase, Solr, or anything else).
Bugfender offers you an API that can you can use to gather logs in real time. Logs come in pages, and cursors for the next and previous logs come with each page, like this:
{
"data": [
{
"uuid": "a774e285-78d4-54bb-9dba-dca684f11400",
"device.udid": "6e306bed-4d7b-46f3-acee-fe6339b3c6fb",
"device.name": "PLK-AL10",
"device.type": "PLK-AL10",
"version.version": "1.4.6",
"version.build": "2020110306",
"language": "en",
"os_version": "6.0",
"timezone": "CAT",
"text": "This is a test log.",
"method": "test",
"file": "SourceFile",
"line": 1,
"log_level": 0,
"tag": "SynchronizationJob",
"time": "2021-04-20T09:34:24.157Z",
"thread_id": "21960",
"thread_name": "AndroidJob-2",
…
},
…
],
"previous": "https://dashboard.bugfender.com/api/…",
"next": "https://dashboard.bugfender.com/api/…",
}Then you can push each page to your database. In fact, scrolling through pages is as easy as calling the “previous” and “next” URLs.
To make things easier, we created a sample project: https://github.com/bugfender/bugfender-integration-elasticsearch
This project essentially performs two steps, ad infinitum:
This project also takes care of other details, like error handling and retrying, task cancellation and resuming, and authentication.
We encourage you to download it and adapt it to your needs. It might be useful already as-is, but it might also serve as a starting point for your own project. For example, you might want to do some transformations of the data downloaded from Bugfender before pushing it to your own database.
The core of the integration lies here:
// syncOnePage synchronizes one page of logs, returns error if something failed
func (i *Integration) syncOnePage(ctx context.Context) error {
// get a page from Bugfender
logs, err := i.bugfenderClient.GetNextPage(ctx)
if err != nil {
return err
}
if ctx.Err() != nil {
return ctx.Err()
}
// put it in the destination
err = i.destination.WriteLogs(ctx, logs)
if err != nil {
return err
}
if i.verbose {
log.Printf("Wrote %d logs", len(logs))
}
return ctx.Err()
}You might also find these parts interesting:
If you do not have an API key yet, contact us to get one. You can follow the instructions here: https://support.bugfender.com/en/articles/4667823-api-access.
Then run like this (newlines added for readability):
./bugfender-integration-elasticsearch
-app-id=1234
-client-id=your_client_id
-client-secret=your_client_secret
-state-file state.json
-es-index logs
-es-nodes http://127.0.0.1:9200The state-file parameter is useful to remember the synchronization progress. You can stop the synchronization with Ctrl+C and running the command again will pick up where it left off.
As we have seen, the Bugfender API can be handy to extract logs for further processing. You can run analytics, merge logs from multiple sources or archive them in your preferred long-term storage solution. But the API can do more than what we covered in this article.
If you would like to learn more, you can see the full documentation here: https://dashboard.bugfender.com/api/.
We can’t wait to see what you build with it!