redis + python: web cache management | as a usage (pattern

I needed a simple cache solution using with flask and therefore used SimpleCache of werkzeug. I used to cache some json data returning from inner modules such as list of things, list of that, list of this; using a pattern similar to this one:

[Customer1.UserList.Page1] = {1:'bla', 2:'foo', 3:'bar'}
[Customer1.UserList.Page2] = {4:'bla', 5:'foo', 6:'bar'}
...
[Customer2.UserList.Page2] = {34:'bla', 35:'foo', 36:'bar'}

With a managable and smart expiry timeout (ie. 20 secs), this worked nice.

There were 3 problems in front of me, [1] on each action, I needed check if result is on cache, if not evaluate the action then push to cache, then code comes messy; so it should have had simpler usage [2] I needed to extend the timeout so that application is less busier overall, and [3] whenever a user makes a data manipulation, I needed to remove the keys with some specific pattern for the user.

Looking around, I found nearly nothing in terms of w/pattern key deleting.

At least, I found the LUA script sheerun shared on stackoverflow. Then wrote a method wrapper which would check the cache for me and if nothing there, evaluate the result and push it into the cache. So this section would be “more” automized.

Pushing to the cache…

This is the wrapper:

def getFromCacheOrSet(callable, *params, **opts):
    key = opts.get('key', callable.__name__) + '.' + '.'.join(map(unicode, params)).encode('utf-8')
    to = opts.get('timeout', 30)
    extendTime = opts.get('extendTime', False)
    try:
        v = __cached.get(key)
        if v is None:
            v = callable(*params)
            __cached.setex(key, to, v)
            return v
        else:
            if extendTime: __cached.expire(key, to)
            return v    #fromCache
    except Exception as e:
        print('Error: get from cache fai

What it makes is that, if you would like to run a multiplication method such as multiply(4,16) you pass the method, its parameters to the wrapper so that it takes care of getting from the cache or evaluating. The point is wierdest line is obviously the “key” section which makes itself a key for the cache. Key is builded from the name of the method and its argument values.

For multiply(4, 16) key becomes “multiply.4.16″, if you rather would like to choose another string for this operation, you can pass it as key=”MultiplicationOp” in opts as in:

getFromCacheOrSet(multiply, 4, 16, key="MultiplicationOp", timeout=20)    # 20 secs

then here we get this key on the cache:

MultiplicationOp.4.16 = 64

It evaluates and pushes this information to the cache.

About pattern deleting…

This is not the best way by far. However, since I could not find a better way, it does the job that I have to settle. As I mentioned earlier, thanks to sheerun@stackoverflow I have something.

This is the removing part:

def removeFromCache(callable, *params, **opts):
    try:
        key = opts.get('key', callable.__name__) + '.' + '.'.join(map(unicode, params)).encode('utf-8')
        __cached_del_keys(keys=[], args=[key + '*'])
    except Exception as e:
        print('Error: Couldnot delete from cache. Hopefully will expire itself soon.', e)
        return False

What happens here is, deleting the matching keys fits the call with as we previously made for multiplication. Such as:

removeFromCache(multiply, 4, key="MultiplicationOp")

key value is mandatory only if you used it while setting the value (obviously). Warning should be this, in here there is no 2nd argument which get to be 16 here. This is because we would like to delete all “multiply.4.*” keys.

The LUA script which I call before the methods above:

__cached = redis.StrictRedis(host='localhost', port=6379, db=0) ##### db1 !
__DEL_LUA = ("""local keys = redis.call('keys', ARGV[1])
 for i=1,#keys,5000 do
 redis.call('del', unpack(keys, i, math.min(i+4999, #keys)))
 end
 return keys""")
__cached_del_keys = __cached.register_script(__DEL_LUA)

Overall usage, eg:

getFromCacheOrSet(getCustomerList, "customer_1", "arg1", "arg2", 3, timeout=1200)  # 1200sec = 20 mins expire
getFromCacheOrSet(getCustomerList, "customer_1", "arg1", "arg2", 2, timeout=1200)
getFromCacheOrSet(getCustomerList, "customer_2", "arg5", "arg6", 1, timeout=1200)
## customer_1 does something to list, have to refresh
removeFromCache(getCustomerList, "customer_1", "arg1")  # remove all getCustomerList.customer_1.arg1.***

I hope anyone would be happy to use.

As a note; it’s better serializing the objects before pushing into the cache and deserializing respective text into object after getting from cache server. These might help:

def serialize(obj):
    return base64.b64encode(pickle.dumps(obj))
def deserialize(s):
    return pickle.loads(base64.b64decode(s))

References:
1. redis-py
2. redis
3. werkzeug SimpleCache and RedisCache
4. flask
5. issue on stackoverflow

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s