Skip to content

Fix TypeError caused by incorrect CachedQuerySet unpickling. #85

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

Fix TypeError caused by incorrect CachedQuerySet unpickling. #85

wants to merge 1 commit into from
+5 −0

Conversation

emorozov
Copy link

This problem is present in both PyPI version and latest github version of django-cache-machine. It manifests itself like this:
Traceback:
File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/core/handlers/base.py" in get_response

  1.                 response = wrapped_callback(request, _callback_args, *_callback_kwargs)
    File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/views/generic/base.py" in view
  2.         return self.dispatch(request, _args, *_kwargs)
    File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/views/decorators/csrf.py" in wrapped_view
  3.     return view_func(_args, *_kwargs)
    File "/data/cache/buildout/eggs/djangorestframework-2.3.6-py2.7.egg/rest_framework/views.py" in dispatch
  4.         response = self.handle_exception(exc)
    File "/data/cache/buildout/eggs/djangorestframework-2.3.6-py2.7.egg/rest_framework/views.py" in dispatch
  5.         response = handler(request, _args, *_kwargs)
    File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/utils/decorators.py" in _wrapper
  6.         return bound_func(_args, *_kwargs)
    File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/utils/decorators.py" in _wrapped_view
  7.                 response = view_func(request, _args, *_kwargs)
    File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/utils/decorators.py" in bound_func
  8.             return func(self, _args2, *_kwargs2)
    File "/home/jmv/linguist/staging/src/lingq_api/views.py" in wrapper
  9.     language = get_object_or_404(profile.languages(), code=language)
    File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/shortcuts/init.py" in get_object_or_404
  10.     return queryset.get(_args, *_kwargs)
    File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/db/models/query.py" in get
  11.     num = len(clone)
    File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/db/models/query.py" in len
  12.     self._fetch_all()
    File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/db/models/query.py" in _fetch_all
  13.         self._result_cache = list(self.iterator())
    File "/home/jmv/linguist/staging/parts/django-cache-machine/caching/base.py" in iter
  14.             self.cache_objects(to_cache)
    File "/home/jmv/linguist/staging/parts/django-cache-machine/caching/base.py" in cache_objects
  15.     cache.add(query_key, objects, timeout=self.timeout)
    File "/data/cache/buildout/eggs/django_devserver-0.8.0-py2.7.egg/devserver/utils/stats.py" in wrapped
  16.     return stats.run(func, key, logger, _args, *_kwargs)
    File "/data/cache/buildout/eggs/django_devserver-0.8.0-py2.7.egg/devserver/utils/stats.py" in run
  17.     value = func(_args, *_kwargs)
    File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/core/cache/backends/memcached.py" in add
  18.     return self._cache.add(key, value, self._get_memcache_timeout(timeout))
    File "/data/cache/buildout/eggs/Django-1.6.8-py2.7.egg/django/core/cache/backends/memcached.py" in _get_memcache_timeout
  19.     elif int(timeout) == 0:

Exception Type: TypeError at /api/languages/es/progress/
Exception Value: int() argument must be a string or a number, not 'object'

It was very difficult to find the culprit, but finally I've found out that CachedQuerySet stores Django DEFAULT_TIMEOUT in ints instances, which is a marker object. Django caching code compares this value with DEFAULT_TIMEOUT and if they are not identical, assumes that timeout is an integer value. But in unpickled instances, timeout value is always some different object.

My fix is not the most elegant, but I've spent 5+ hours trying to debug this issue and it makes any development using d-c-m impossible (bug appears and disappears at random).

@tobiasmcnulty
Copy link
Member

Closing and re-opening this PR to trigger a rebuild on Travis.

@tobiasmcnulty tobiasmcnulty reopened this Jul 3, 2015
@tobiasmcnulty
Copy link
Member

Hi @emorozov - could you add a regression test for this bug to the PR? Thanks for the report and fix!

@pbassut
Copy link

pbassut commented Oct 11, 2015

I'm also having this issue. What could be a possible fix for this? Only waiting for the PR to come out?

@pbassut
Copy link

pbassut commented Oct 11, 2015

My workaround was to subclass the manager and set the timeout manually, like this:

class MasterQuerySet(CachingQuerySet):
    pass

class MasterQuerySetManager(CachingManager):
    def get_queryset(self):
        qs = MasterQuerySet(self.model)

        qs.timeout = 60
        return qs

    def __getattr__(self, name):
        return getattr(self.get_queryset(), name)

And use that on my model like this:
objects = MasterQuerySetManager()

Waiting for this to be fixed to drop this ugly workaround

@tobiasmcnulty
Copy link
Member

@pbassut I'll take a look at this today. Do you have a way to reproduce the issue?

@tobiasmcnulty
Copy link
Member

Okay, I think I've discovered what the issue is:

  • cache-machine is caching a set of objects, which you're then attempting to re-cache manually
  • this doesn't break in the test environment because everything is run under the same process, hence DEFAULT_TIMEOUT can be safely pickled/unpickled
  • you're using a multi-process WSGI server (who isn't?), each with its own DEFAULT_TIMEOUT, so if objects are cached by cache-machine in a different process than they're re-cached by your code, you get this exception (might not fail all the time)

@tobiasmcnulty tobiasmcnulty mentioned this pull request Oct 11, 2015
Merged
@tobiasmcnulty
Copy link
Member

Fixed by PR #103

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@emorozov @tobiasmcnulty @pbassut