google.appengine.ext.remote_api.throttle module

Summary

Client-side transfer throttling for use with remote_api_stub.

This module is used to configure rate limiting for programs accessing AppEngine services through remote_api.

See the Throttle class for more information.

An example with throttling: — from google.appengine.ext import db from google.appengine.ext.remote_api import remote_api_stub from google.appengine.ext.remote_api import throttle from myapp import models import getpass import threading

def auth_func():

return (raw_input(‘Username:’), getpass.getpass(‘Password:’))

remote_api_stub.ConfigureRemoteDatastore(‘my-app’, ‘/remote_api’, auth_func) full_throttle = throttle.DefaultThrottle(multiplier=1.0) throttle.ThrottleRemoteDatastore(full_throttle)

# Register any threads that will be using the datastore with the throttler full_throttle.Register(threading.currentThread())

# Now you can access the remote datastore just as if your code was running on # App Engine, and you don’t need to worry about exceeding quota limits!

houses = models.House.all().fetch(100) for a_house in houses:

a_house.doors += 1

db.put(houses) —

This example limits usage to the default free quota levels. The multiplier kwarg to throttle.DefaultThrottle can be used to scale the throttle levels higher or lower.

Throttles can also be constructed directly for more control over the limits for different operations. See the Throttle class and the constants following it for details.

Contents

class google.appengine.ext.remote_api.throttle.DatastoreThrottler(throttle)source

Bases: google.appengine.ext.remote_api.throttle.Throttler

AddCost(cost_proto)source

Add costs from the Cost protobuf.

google.appengine.ext.remote_api.throttle.DefaultThrottle(multiplier=1.0)source

Return a Throttle instance with multiplier * the quota limits.

exception google.appengine.ext.remote_api.throttle.Errorsource

Bases: exceptions.Exception

Base class for errors in this module.

google.appengine.ext.remote_api.throttle.InterruptibleSleep(sleep_time)source

Puts thread to sleep, checking this threads exit_flag four times a second.

Parameters

sleep_time – Time to sleep.

google.appengine.ext.remote_api.throttle.SleepHandler(*throttle_names)source
exception google.appengine.ext.remote_api.throttle.ThreadNotRegisteredErrorsource

Bases: google.appengine.ext.remote_api.throttle.Error

An unregistered thread has accessed the throttled datastore stub.

class google.appengine.ext.remote_api.throttle.Throttle(get_time=function, thread_sleep=InterruptibleSleep, layout=None)source

Bases: object

A base class for upload rate throttling.

Transferring large number of entities, too quickly, could trigger quota limits and cause the transfer process to halt. In order to stay within the application’s quota, we throttle the data transfer to a specified limit (across all transfer threads).

This class tracks a moving average of some aspect of the transfer rate (bandwidth, records per second, http connections per second). It keeps two windows of counts of bytes transferred, on a per-thread basis. One block is the “current” block, and the other is the “prior” block. It will rotate the counts from current to prior when ROTATE_PERIOD has passed. Thus, the current block will represent from 0 seconds to ROTATE_PERIOD seconds of activity (determined by: time.time() - self.last_rotate). The prior block will always represent a full ROTATE_PERIOD.

Sleeping is performed just before a transfer of another block, and is based on the counts transferred before the next transfer. It really does not matter how much will be transferred, but only that for all the data transferred SO FAR that we have interspersed enough pauses to ensure the aggregate transfer rate is within the specified limit.

These counts are maintained on a per-thread basis, so we do not require any interlocks around incrementing the counts. There IS an interlock on the rotation of the counts because we do not want multiple threads to multiply-rotate the counts.

There are various race conditions in the computation and collection of these counts. We do not require precise values, but simply to keep the overall transfer within the bandwidth limits. If a given pause is a little short, or a little long, then the aggregate delays will be correct.

AddThrottle(name, limit)source
AddThrottles(layout)source
AddTransfer(throttle_name, token_count)source

Add a count to the amount this thread has transferred.

Each time a thread transfers some data, it should call this method to note the amount sent. The counts may be rotated if sufficient time has passed since the last rotation.

Parameters

  • throttle_name – The name of the throttle to add to.

  • token_count – The number to add to the throttle counter.

ROTATE_PERIOD = 600
Register(thread)source

Register this thread with the throttler.

Sleep(throttle_name=None)source

Possibly sleep in order to limit the transfer rate.

Note that we sleep based on prior transfers rather than what we may be about to transfer. The next transfer could put us under/over and that will be rectified after that transfer. Net result is that the average transfer rate will remain within bounds. Spiky behavior or uneven rates among the threads could possibly bring the transfer rate above the requested limit for short durations.

Parameters

throttle_name – The name of the throttle to sleep on. If None or omitted, then sleep on all throttles.

TotalTransferred(throttle_name)source

Return the total transferred, and over what period.

Parameters

throttle_name – The name of the throttle to total.

Returns

A tuple of the total count and running time for the given throttle name.

VerifyThrottleName(throttle_name)source
class google.appengine.ext.remote_api.throttle.ThrottleHandler(throttle)source

Bases: urllib2.BaseHandler

A urllib2 handler for http and https requests that adds to a throttle.

AddRequest(throttle_name, req)source

Add to bandwidth throttle for given request.

Parameters

  • throttle_name – The name of the bandwidth throttle to add to.

  • req – The request whose size will be added to the throttle.

AddResponse(throttle_name, res)source

Add to bandwidth throttle for given response.

Parameters

  • throttle_name – The name of the bandwidth throttle to add to.

  • res – The response whose size will be added to the throttle.

http_request(req)source

Process an HTTP request.

If the throttle is over quota, sleep first. Then add request size to throttle before returning it to be sent.

Parameters

req – A urllib2.Request object.

Returns

The request passed in.

http_response(unused_req, res)source

Process an HTTP response.

The size of the response is added to the bandwidth throttle and the request throttle is incremented by one.

Parameters

  • unused_req – The urllib2 request for this response.

  • res – A urllib2 response object.

Returns

The response passed in.

https_request(req)source

Process an HTTPS request.

If the throttle is over quota, sleep first. Then add request size to throttle before returning it to be sent.

Parameters

req – A urllib2.Request object.

Returns

The request passed in.

https_response(unused_req, res)source

Process an HTTPS response.

The size of the response is added to the bandwidth throttle and the request throttle is incremented by one.

Parameters

  • unused_req – The urllib2 request for this response.

  • res – A urllib2 response object.

Returns

The response passed in.

google.appengine.ext.remote_api.throttle.ThrottleRemoteDatastore(throttle, remote_datastore_stub=None)source

Install the given throttle for the remote datastore stub.

Parameters

  • throttle – A Throttle instance to limit datastore access rates

  • remote_datastore_stub – The datstore stub instance to throttle, for testing purposes.

class google.appengine.ext.remote_api.throttle.ThrottledHttpRpcServer(throttle, *args, **kwargs)source

Bases: google.appengine.tools.appengine_rpc.HttpRpcServer

Provides a simplified RPC-style interface for HTTP requests.

This RPC server uses a Throttle to prevent exceeding quotas.

class google.appengine.ext.remote_api.throttle.ThrottledHttpRpcServerFactory(throttle, throttle_class=None)source

Bases: object

A factory to produce ThrottledHttpRpcServer for a given throttle.

class google.appengine.ext.remote_api.throttle.ThrottledHttpRpcServerOAuth2(throttle, *args, **kwargs)source

Bases: google.appengine.tools.appengine_rpc_httplib2.HttpRpcServerOAuth2

class google.appengine.ext.remote_api.throttle.Throttlersource

Bases: object

PosthookHandler(service, call, request, response)source
PrehookHandler(service, call, request, response)source
exception google.appengine.ext.remote_api.throttle.UnknownThrottleNameErrorsource

Bases: google.appengine.ext.remote_api.throttle.Error

A transfer was added for an unknown throttle name.