Do we need to call the returned function here?

I am not so familiar with defining decorators. But don't we still need this to return the retry version of the inner function?

The decorator should return a function, that's why I tend to think the final () are not needed, but I haven't tested.

If we omit the trailing (), the decorator would return the wrapped function object instead of the function’s result.

Did you manage to test it in async mode? I'm worried we'll have some weird behaviour with the concurrent mode if retries are using await asyncio.sleep(..), that would allow other calls to be "enqueued" and almost all of the following calls will have to wait, which will still fail pretty quickly since they are all waiting and retrying at the same time.

I havn't thought about that but we should definitely avoid this. One solution could be adding randomisation to delay retries. something like the below:

  def handle_async(self, func: AF) -> AF:
        @retry(
            retry=retry_if_exception_type(RateLimitError),
            stop=stop_after_attempt(self.max_attempts),
            wait=wait_exponential(
                multiplier=self.multiplier,
                min=self.min_wait,
                max=self.max_wait,
            ) + (wait_random(0, 1) if self.jitter else wait_fixed(0)),  # add jitter
            before_sleep=before_sleep_log(logger, logging.WARNING),
        )

wdyt?

To be tested ;)

looks like Tenacity already has a function that does exponential backoff + jitter, so I ended up using it 8cfacb4

Which parameters are these methods getting as input?

func representing sync/async functions that need rate limit handling applied to them as it is the case for current RetryRateLimitHandler (the default one)

Do you think this interface could be used to limit the number of requests per second for instance?

If we would like to extend our strategy for rate limit handling and do throttling, we can define a new subclass:

class ThrottlingRateLimitHandler(RateLimitHandler):
    
   def __init__(self, requests_per_window: int = 10, window_size: float = 1.0):
        self.requests_per_window = requests_per_window
        self.window_size = window_size
        ....

    
    def handle_sync(self, func: F) -> F:
        @functools.wraps(func)
        def wrapper(*args: Any, **kwargs: Any) -> Any:
            # handle throttling here
            return func(*args, **kwargs)
        return wrapper
        
     .....

then we can basically use the same decorators with the invoke functions of the LLM classes, we just need to ensure we pass the right rate limit handler in their constructors.