Patch to make Cache Machine play nicely with multiple Databases #29
Conversation
|
Thanks Ben! |
|
Thanks Jeff! I've been working with Cache Machine a ton, it's been incredibly useful. One potentially useful thing I've implemented is an audit trail inside the CacheMachine iter magic method. It tracks, on a per model basis, cache hits/misses/total_time_saved (how long it takes to iterate through the MySQL queryset). It samples the data, so only 5% of calls are logged. It's implemented in Redis, though I could generalize it to other backends if you wanted to pull it upstream. The main goal was to be able to tweak cache timeouts on a per model basis and then analyze the impact on hit/misses/time saved. We justed started running it in production, so we'll see how useful it is. |
|
That sounds awesome! I don't know how useful it would be upstream since there's not a standard place to send and view stats like that, but I'd love to see the code and hear about the results. |
|
This seems like it would be a valuable addition, even if it just output the stats to a text file somewhere. |
I was having an issue where some cached queries were being pulled and stored from a slave database (e.g. Profile.objects.using('slave').all()), whereas others were being pulled from the master.
That led to unpredictable behavior during attempts to add/update ManyToMany and FK relations on a model instance pulled from master. If the M2M queryset to be added is pulled from the slave, Django raises a ValueError. You can't save relations across different DBs.
I went ahead and added db info to the query key to namespace the different databases. It's working for me and it doesn't add much overhead.
I am more than willing to live with fewer cache hits to avoid the headache of erratic, error prone code.
Can you see any issues with this code as far as invalidation goes?