Time-graph Implementation with Golang and Redis TimeSeries

A few days ago, at my workplace, we suddenly needed to present a graph of the user's earnings across multiple date ranges. And we had to do it fast. A previous implementation had been done using our database which worked fine but was quite slow.

Then came Redis TimeSeries.

Before we proceed, we have to remember that Redis TimeSeries is a module on the Redis Stack. The Redis Stack is an extension of the core features of the Redis OSS. It provides other features like debugging, queryable JSON documents, and probabilistic data structures. Let's back to the topic at hand.

Redis TimeSeries, which is a module on the Redis Stack. provides a structure to store time-based data. It is a linked list of memory chunks. Each chunk has a predefined size of samples and each sample is a 128-bit tuple: 64 bits for the timestamp and 64 bits for the value.

Redis TimeSeries only gives the user the ability to store a timestamp and the value, which was perfect for us. Also, it offers a wide range of aggregation, which can also be of help if you need to add values over a particular range of time or find the average of those values over time. I will be detailing the steps I took and my experience in migrating our graph data from our database to Redis TimeSeries.

Installation and Setup

Installation

I had some issues getting the Redis TimeSeries module onto my local machine. After some struggle, I found out the best way was to get RedisTimeSeries via Redis-stack image on Docker. To do this first, you have to install docker(steps can be found here) on your machine. You might need to add Docker to your path, so you can easily run the commands needed to install the Redis-stack, instructions can be found at this link.

Next, you want to pull the latest Redis-stack image, get a container and start working. The steps to do all of that can be found here.

Good luck.

Setup

Similar to the way we use the client of Redis to store data, we need a client for the RedisTimeSeries module to get started. The Golang package/helper can be found here. It can be installed with the following command.

$ go get github.com/RedisTimeSeries/redistimeseries-go

More instructions can be found in the link to the package above. However, I had some issues using those instructions though due to my local Redis server not needing a password. So I utilized the following piece of code to get a working Redis Time Series client.

import (
    redis_timeseries_go "github.com/RedisTimeSeries/redistimeseries-go"
    "github.com/go-redis/redis"

    redigo "github.com/gomodule/redigo/redis"
)

opt, err := redis.ParseURL(redisURL)
if err != nil {
    continue
}

pool := redigo.Pool{
    Dial: func() (redigo.Conn, error) {
        return redigo.Dial("tcp", opt.Addr, redigo.DialPassword(opt.Password))
    },
}

timeSeriesClient := redis_timeseries_go.NewClientFromPool(&pool, "pug")

In our project at my workplace, we already utilize go-redis which can be found here to parse the URL to our Redis servers on various environments. The trick here was to use redigo to create a pool, so I could use redis_timeseries_go.NewClientFromPool(&pool, "pug") to create a working Redis Time Series client. This pool will use the address and password that is gotten from parsing with go-redis

We now have the client timeSeriesClient with which we can call commands on.

Storing Data

As I wrote earlier, the Redis Timeseries allows us to store timestamps and an associated numeric value. There are many variations on the commands that are used to add data to a time-series. A full list of the commands that can be executed in Redis Time Series can be found here.

The code snippet below shows how it was written in our case:

import (
    redis_timeseries_go "github.com/RedisTimeSeries/redistimeseries-go"
)

//The interval between a quarter of a year in milliseconds
//Keep in mind, you'll be work in milliseconds with redis timeseries
QuarterlyInterval := 7776000000000

_, err := timeSeriesClient.AddAutoTsWithOptions("graph_series", earningAtCertainPoint, redis_timeseries_go.CreateOptions{
    Labels: map[string]string{
        "user_id":   user_id,
    },
    RetentionMSecs: QuarterlyInterval,
})

if err != nil {
    return errors.Wrap(err, "timeSeriesClient.AddWithOptions")
}

Now let's break this code down. The command, or in this case; method, that we're using to insert our data is theAddAutoTsWithOptions method. This method allows us to just store a value with its corresponding timestamp automatically generated and stored. AddAutoTsWithOptions also allows us to create a new key dynamically, if there was no key with the same name before, saving us a few lines of code.

The key"graph_series" is the name of this time series allows us to access our data when we eventually need to use it

Moving on, we're adding Labels to our time-series, in a bit to differentiate it from other time-series. Labels can also help to filter data from multiple time-series based on the values of their labels. We used the struct redis_timeseries_go.CreateOptions{} to add our label, which is a map[string]string. Labels differ from keys as they are used to attach specific values or characteristics to a time-series.

We don't want the data on this time-series to be stored for too long and want it to be pruned after a specific period. That duration of time is passed into RetentionMSecs: QuarterlyInterval. This implies that data in this time-series will only be kept around for a quarter of a year before being deleted.

Once again, you can check for other commands that can be used in a Redis time series here. If you don't have any need for a label or retention time, you can just use redis_timeseries_go.DefaultCreateOptions.

Retrieving Data

We've been able to store our data. It's time to read it. For our use case, we wanted to also aggregate our data and be able to retrieve it on an hourly, weekly, monthly, and quarterly basis. Let's take a look at this snippet;

import (
    redis_timeseries_go "github.com/RedisTimeSeries/redistimeseries-go"
)

startDate := time.Now().AddDate(0, 0, -2).UnixMilli() //2 days back
endDate := time.Now().UnixMilli()

timeBucket := 864000000 //Time duration for a day in milliseconds

rangeOptions := redis_timeseries_go.RangeOptions{
    AggType:    redis_timeseries_go.LastAggregation,
    TimeBucket: timeBucket,
    Count:      -1,
}

points, err := timeSeriesClient.RangeWithOptions(redisKey, startDate, endDate, rangeOptions)
if err != nil {
    return errors.Wrap(err, "timeSeriesClient.RangeWithOptions")
}

Let's go line by line now. Firstly; we're defining the time range within which we'll get our data with the variables startDate and endDate. Note how the times are in UnixMilli().

Moving on, the timeBucket determines how the data in the time-series will be grouped. In this case, we used the duration for a day so the data in our time series will be grouped by days. For example, you could collect a million data points per second in a day, but all these data points have to be combined into one, i.e. a representative of the day's data.

So you might be wondering "How will this grouping work?"

Here's where the AggType comes in, and with this, we can choose how this data will be grouped. This data can be added together, the average of the data can be used instead, the maximum value can be used, or the minimum value can be used. However, in our use case, we chose the last value recorded in the day. We felt it was very representative of the balance in the user's wallet at the end of the day. This type of aggregation can be accessed by using the constant redis_timeseries_go.LastAggregation. Other aggregations like redis_timeseries_go.AvgAggregation, redis_timeseries_go.MaxAggregation, redis_timeseries_go.MinAggregation can also be used.

After we had come to terms with the parameters of our query, we chose the query method that helped us work with our needs; timeSeriesClient.RangeWithOptions. This query let us get a list of data points, within startDate to endDate that fits the criteria we added to our rangeOptions.

This method returns points which, as you expect, is a slice of data that contains our timestamp and our stored numeric value. These points can be further formatted according to your discretion and sent to your client applications for consumption.

Improvements

We were able to achieve a sizable improvement in the speed at which we produced graph data for our front-end app. We cut down the speed to a speedy 92 milliseconds, which is hundreds of milliseconds faster than our initial implementation.

What next?

You might need to store the aggregated versions of a time-series in another time-series. This can be done by the use of Rules, which we won't touch on for now. You might also wish to filter across multiple time series and even adjust how your data points fill in a time bucket.

Instructions on how to do all that was mentioned above can be found in the docs here. Good luck!

Conclusion

I hope my experience with RedisTimeSeries made for a good read. I certainly enjoyed my time researching and writing up the code and I hope you do too. If you have any questions, please feel free to reach out to me on Twitter or via my email.

Have a great day!