Python Classes¶
-
class
apicrud.
AESEncrypt
(secret)¶ AES encryption for strings
Provides easier-to-use AES CBC encrypt/decrypt operations for strings
- Parameters
secret (str) – passphrase (suggest at least 16 characters)
-
decrypt
(enc)¶ decrypt an object
- Parameters
enc (bytes) – encrypted object
- Returns
decrypted string
- Return type
str
-
encrypt
(raw)¶ encrypt a string
- Parameters
raw (str) – object to be encrypted
- Returns
encrypted object
- Return type
bytes
-
class
apicrud.
AESEncryptBin
(secret)¶ AES encryption for binaries
Provides easier-to-use AES CBC encrypt/decrypt operations for byte objects
- Parameters
secret (str) – passphrase (suggest at least 16 characters)
-
decrypt
(enc)¶ decrypt an object
- Parameters
enc (bytes) – encrypted object
- Returns
decrypted bytes
- Return type
bytes
-
encrypt
(raw)¶ encrypt a byte object
- Parameters
raw (bytes) – object to be encrypted
- Returns
encrypted object
- Return type
bytes
-
class
apicrud.
AccessControl
(policy_file=None, model=None)¶ Role-based access control
Definitions:
principal: a user or role
membership: parent resource type for privacy sharing
model: database model name (e.g. Person)
resource: resource type (e.g. person)
rbac: role-based access control (defined in rbac.yaml)
role: a group name (e.g. admin or list-<id>-<level>)
privacy: sharing options as defined in rbac.yaml (e.g. secret [default], public, invitee, member, manager)
actions: crudlghij (create, read, update, del, list, guest/member, host/manager, invitee, join)
In rbac.yaml, define the RBAC policies for each principal/resource combination. That file will be parsed into a singleton variable upon initial startup. This implementation implements RBAC similar to that of kubernetes or AWS IAM, with the added capability of a simple privacy permission within each object (database record) which creates an implied ACL for read-only access by members of the object’s group.
Group names currently used are:
admin
user
pending (new-account confirmation)
pendingtotp
person
<resource>-<id>-<privacy>
These are defined in session_auth.py’s account_login() method.
- Parameters
policy_file (str) – name of the yaml definitions file
model (obj) – a model to be validated for permissions
-
apikey_create
()¶ Generate an API key - a 41-byte string. First 8 characters (48 bits) are an access key ID prefix; last 32 characters (192 bits) are the secret key.
- Returns: tuple
key ID (str) - public portion of key secret (str) - secret portion hashvalue (str) - hash value for database
-
static
apikey_hash
(secret)¶ Generate a hash value from the secret :param secret: secret key :type secret: str
-
apikey_verify
(key_id, secret)¶ Verify an API key
- Parameters
key_id (str) – the public key_id at time of generation
secret (str) – the unhashed secret
- Returns: tuple
uid (str): User ID if valid scopes (list): list of scope IDs
-
load_rbac
(filename)¶ Read RBAC default policies from rbac.yaml, process any string substitutions, and convert * for re.match()
- Parameters
filename (str) – filename containing RBAC definitions
-
rbac_permissions
(query=None, owner_uid=None, membership=None, id=None, privacy=None)¶ Evaluate an access request for self.auth roles of self.uid in self.resource against defined policies
- Parameters
query (obj) – an existing record (takes precedence over owner_uid)
owner_uid (str) – owner-uid of a record
membership (str) – resource type which defines membership privacy
id (str) – the resource ID if membership is set
- Returns
actions available to principal
- Return type
set
-
with_filter
(query, access='r')¶ Apply RBAC and privacy to a query
- Parameters
query (obj) – a resource query in SQLalchemy
access (str) – one of lrwcd (list, read, write, create, delete)
- Returns
updated SQLalchemy query with filter applied
- Return type
obj
TODO restrictions on contact-read by list-id
-
with_permission
(access, query=None, new_uid=None, membership=None, id=None)¶ Evaluate permission to access an object identified by an open query or new uid. Pass in at least one of the query/uid/eid params
- Parameters
access (str) – one of lrwcd (list, read, write, create, delete)
query (obj) – a resource query by id in SQLalchemy
new_uid (str) – user id of a new record
membership (str) – resource type which defines membership privacy
id (str) – resource ID
- Returns
True if access allowed
- Return type
bool
-
class
apicrud.
AccountSettings
(account_id, db_session=None, uid=None)¶ Access class for account settings, with cache - converts db record to object attributes
Each account is associated with an entry in the Settings model; this class provides access to these key-value pairs as read-only attributes. Because these functions are called frequently, the db entry is loaded into memory for a configurable expiration period (config.REDIS_TTL).
- Parameters
account_id (str) – ID in database of a user’s account
db_session (obj) – a session connected to database
uid (str) – User ID
-
property
locale
¶ Returns the language for the uid if specified in the user’s profile
-
uncache
()¶ Clear the cached settings for account_id
-
class
apicrud.
BasicCRUD
(resource=None, model=None)¶ Controller base class
Create/Read/Update/Delete/find controller operations.
This class provides permission-based, paginated access to database models behind your application’s endpoints. Most endpoints need no boilerplate code, and can inherit these functions directly. Some endpoints only need a few lines of code before or after inheriting these functions. You can always write your own custom function for special-case endpoints.
- Parameters
resource (str) – a resource name (endpoint prefix)
model (obj) – the model corresponding to the resource
-
static
create
(body, id_prefix='x-', limit_related={})¶ Controller for POST endpoints. This method assigns a new object ID, sets the _created_ timestamp, evaluates user’s permissions, adds a default category_id if the model has this attribute, and inserts a row to the back-end database.
- Parameters
body (dict) – resource fields as defined by openapi.yaml schema
id_prefix (str) – generated objects will be assigned a random 10- to 16-character ID; you can set a unique prefix if desired
limit_related (dict) – limits on number of related records, keyed by relationship name
- Returns
first element is a dict with the id, second element is response code (201 on success)
- Return type
tuple
-
db_get
(id)¶ Activate a SQLalchemy query object for the specified ID in the current model
- Parameters
id (str) – object ID
- Returns
query object
- Return type
obj
-
static
delete
(ids, force=False)¶ Controller for DELETE endpoints. This method looks for existing records, evaluates user’s permissions, and updates or removes rows in the back-end database.
- Parameters
ids (list of str) – record IDs to be flagged for removal
force (bool) – flag for removal if false; remove data if true
- Returns
first element is a dict with the id, second element is response code (200 on success)
- Return type
tuple
-
static
find
(**kwargs)¶ Find records which match query parameters passed from connexion by name, in a dictionary that also includes user and token info
- Parameters
cursor_next (str) – pagination token to fetch subsequent records
filter (dict) – field/value pairs to query (simple queries only, with string or list matching; or * for any)
limit (int) – max records to fetch
offset (int) – old-style pagination starting offset
sort (str) – <field>[:{asc|desc}]
status (str) – value is added to filter
- Returns
items (list), count(int), cursor_next (str)
- Return type
dict
-
static
get
(id)¶ Controller for GET endpoints. This method evaluates privacy settings against the user’s permissions, looks up category, owner and geocode values, and fetches the object from back-end database.
- Parameters
id (str) – ID of the desired resource
- Returns
first element is a dict with the object or error message, second element is response code (200 on success)
- Return type
tuple
-
static
update
(id, body, access='u', limit_related={})¶ Controller for PUT endpoints. This method looks for an existing record, evaluates user’s permissions, and updates the row in the back-end database.
- Parameters
body (dict) – fields to be updated
access (str) – access-level required for RBAC evaluation
limit_related (dict) – limits on number of related records, indexed by relationship name
- Returns
first element is a dict with the id, second element is response code (200 on success)
- Return type
dict
-
static
update_contact
(id, body)¶ This is a special-case function for the contact-update resource
validate sms carrier
keep person identity in sync with primary contact
- Parameters
id (str) – resource ID
body (dict) – as defined in openapi.yaml
-
class
apicrud.
Grants
(db_session=None, ttl=None)¶ Account usage limits
An account’s usage limits are specified here in the grants table; the free-service tier is defined and passed in via load_defaults(). Records in grants table are owned by administrator-level user. If a record matches a user uid, the default grant name=value is overridden.
-
db_session
¶ existing db session
- Type
obj
-
ttl
¶ how long to cache a grant in memory
- Type
int
-
crud_get
(crud_results, id)¶ Process results from BasicCRUD.get() for grants endpoint. If the id is found in database, perform the standard CRUD get(). Otherwise, look for a hybrid id in form uid:grant and return the cached Grant value. Grant values are serialized as strings even if they are integers (decimal, octal, hex).
- Parameters
crud_results (tuple) – preliminary response
name (str) – name filter, if specified
-
find
(crud_results, **kwargs)¶ Process results from BasicCRUD.find() for grants endpoint
- Parameters
crud_results (tuple) – preliminary response
name (str) – name filter, if specified
-
get
(name, uid=None)¶ Get the cached value of a named grant, if it hasn’t expired Note that if any grant assigned to a uid expires before others, the earliest expiration applies to all the uid’s grants
- Parameters
name (str) – name of a grant, as defined in service config
uid (str) – user ID
- Returns
granted limit or None if undefined
- Return type
int or str
-
load_defaults
(defaults)¶ Load default values from a dict of keyword: value pairs
- Parameters
defaults (dict) – new defaults
-
uncache
(uid)¶ Remove grants from cache, any time a user’s status changes
- Parameters
uid (str) – user ID
-
-
class
apicrud.
Metrics
(uid=None, redis_conn=None, db_session=None, func_send=None)¶ This implementation supports standard system metrics and two types of usage-billing: granted limits per period, or metered usage.
See this article for a description of how redis makes the implementation straightforward: https://www.infoworld.com/article/3230455/how-to-use-redis-for-real-time-metering-applications.html The data store is in-memory, with snapshot/append persistence to disk.
All metrics are defined in the metrics section of service_config.yaml. These follow a de facto standard described in best-practices documentation section of prometheus.io website. At time of implementation, flask prometheus client is not mature/user-friendly enough or suitable for usage-billing so to keep this consistent with service_config.yaml, there is no intent now or in the future to use it.
Thanks to the expiring keys feature of redis, grant limits can be implemented without any periodic cleanup task. Upon first use, a decrementing key is created with expiration set to the configured period. If it counts down to zero before expiration (for example, 50 video uploads per day), the user is denied access to the resource until the key expires. If a user never returns, there will be no redis keys consuming system resources unless and until the user comes back, at which point a new grant period starts. Scaling to tens of millions of users is practical within a small footprint, and this is why user-tracking metrics are stored in redis rather than a traditional database.
-
uid
¶ ID of a user, for usage tracking
- Type
str
-
redis_conn
¶ existing redis connection
- Type
obj
-
db_session
¶ existing db connection
- Type
obj
-
func_send
¶ function name for sending message via celery
- Type
obj
-
check
(name)¶ Check remaining credit against grant-style metric
- Params:
name (str): a metric name
- Returns
amount of credit remaining, or None
- Return type
float
-
collect
(**kwargs)¶ Prometheus-compatible metrics exporter. Collects current metrics from redis, evaluates metric-type settings in service_config.yaml, and returns a plain-text result in this form:
# TYPE process_resident_memory_bytes gauge process_resident_memory_bytes{instance=”k8s-01.ci.net”} 20576.0 # TYPE process_start_time_seconds gauge process_start_time_seconds{instance=”k8s-01.ci.net”} 1614051156.14 # TYPE api_calls_total counter api_calls_total{resource=”person”} 4
-
find
(**kwargs)¶ Look up metrics defined by filter
- Returns
tuple (results dict, status)
-
store
(name, labels=[], value=None)¶ Store a metric: redis keys are in form mtr:<name>:<labels> To avoid possible ambiguity and multiple redis keys for the same label set, this function sorts labels before storing.
- Params:
name (str): a metric name labels (list): labels, usually in form <label>=<value> value (int or float): value to store
- Returns
true in all cases except for grant exceeded
- Return type
bool
-
-
class
apicrud.
Mutex
(lockname, redis_host=None, maxwait=20, ttl=0, redis_conn=None)¶ Simple mutex implementation for non-clustered Redis
- Parameters
lockname (str) – a unique name for the lock
redis_host (str) – IP or DNS name of redis service
maxwait (int) – seconds to wait for a lock
ttl (int) – seconds to hold lock
redis_conn (obj) – existing redis connection
-
acquire
()¶ Acquire a mutex lock
- Raises
TimeoutError – if the resource is unavailable
-
release
()¶ Release a lock
-
class
apicrud.
RateLimit
(redis_conn=None)¶ Rate Limiting
-
call
(service='*', uid=None)¶ Apply the granted rate limit for uid to a given service Two redis keys are used, for even and odd intervals as measured modulo the Unix epoch time. The current interval’s key is incremented if we haven’t reached the limit yet, then we delete that key at first call seen next interval and increment the other key. The keys expire within interval-1 seconds to keep the cache clear after the user stops sending calls.
- Parameters
service (str) – name of a service
uid (str) – a user ID
- Returns
true if limit exceeded
- Return type
bool
-
-
class
apicrud.
ServiceConfig
(file=None, models=None, reset=False, **kwargs)¶ Service configration - have it your way!
Flask and application configuration global default values are defined in service_config.yaml here in the source directory. Overrides of these values can be specified in a few ways and are evaluated in this order:
Environment variables set by parent process
Values defined in a file in yaml format
Values passed as a keyword arg at first class invocation
For runtime security, the config singleton is stored as an immutable namedtuple: to change values, update settings and restart the container running the service.
An endpoint /config/v1/config provides read-only access to these values except those of type password. Always override those secret values before deploying your service.
Attribute keys specified as env vars are UPPERCASE, and attribute keys stored in the config object are also uppercase. Use lowercase to specify attribute keys in kwargs or the yaml input file.
- Parameters
file (str) – path of a YAML file defining override values
models (obj) – sqlalchemy db models
reset (boolean) – reset cached values (for unit tests)
**kwargs – key=value pair arguments to override values
- Raises
AttributeError if invalid specification –
-
set
(key, value)¶ Set a single value
- Parameters
key (str) –
- new value (value) –
-
class
apicrud.
ServiceRegistry
(aes_secret=None, redis_conn=None, public_url=None, reload_params=False, ttl=None)¶ Service Registry
Services or the UI discover one another through this service registry. Each microservice instance submits its identity and capabilities to this central registry, implemented as expiring redis keys which are updated at a fixed frequency. Encryption provides modest protection against injection attacks.
- Parameters
aes_secret (str) – an AES secret [default: config.REDIS_AES_SECRET]
public_url (str) – URL to serve [default: config.PUBLIC_URL]
redis_conn (obj) – connection to redis
reload_params (bool) – force param reload, for unit-testing
ttl (int) – how long to cache instance’s registration
-
find
(service_name=None)¶ Finds one or all services
- Parameters
service_name (str) – a service, or None for all
- Returns
- dict - instances (list of registered services)
url_map (public url for each top-level resource)
-
get
()¶ return service registration for local instance
- Returns: dict(name, id, info)
Key info is dict(endpoints, ipv4, port, public_url, created)
-
register
(resource_endpoints, service_name=None, instance_id='build-13530454-project-613164-apicrud', tcp_port=None)¶ register an instance serving a list of endpoints
- Parameters
resource_endpoints (list of str) – controller endpoints served
service_name (str) – microservice name
instance_id (str) – unique ID of instance
tcp_port (int) – port number of service
-
static
update
()¶ background function to update registration at the defined interval from local memory cache, until the instance terminates.
-
class
apicrud.
SessionAuth
(func_send=None, roles_from=None, redis_conn=None)¶ Session Authorization
Functions for login, password and role authorization
- Parameters
func_send (function) – name of function for sending message
roles_from (obj) – model for which to look up authorizations
-
account_add
(username, uid)¶ Add an account with the given username
- Parameters
username (str) – new / unique username
uid (str) – existing user
-
account_login
(username, password, method='local', otp=None)¶ Log in using local or OAuth2 credentials
- Parameters
username (str) – account name or email
password (str) – credential
method (str) – local, or google / facebook / twitter etc
otp (str) – one-time or backup password
- Returns
Fields include jwt_token (contains uid / account ID), ID of entry in settings database, and a sub-dictionary with mapping of endpoints registered to microservices
- Return type
dict
-
api_access
(apikey, totp_cookie=None)¶ Access using API key
- Parameters
apikey (str) – the API key
totp_cookie (str) – a TOTP bypass cookie
- Returns
uid, scopes (None if not authorized)
- Return type
dict
-
auth_params
()¶ Get authorization info
-
get_roles
(uid, member_model, resource=None, id=None)¶ Get roles that match uid / id for a resource Each is in the form <resource>-<id>-<privacy level>
- Parameters
uid (str) – User ID
member_model (obj) – the DB model that defines membership in resource
resource (str) – the resource that defines privacy (e.g. list)
id (str) – ID of the resource (omit if all are desired)
- Returns
authorized roles
- Return type
list of str
-
methods
()¶ Return list of available auth methods
-
update_auth
(member_model, id, resource=None, force=False)¶ Check current access, update if recently changed
- Parameters
member_model (obj) – model (e.g. Guest) which defines membership in resource
id (str) – resource id of parent resource
resource (str) – parent resource for which membership should be checked
force (bool) – perform update regardless of logged-in permissions
-
class
apicrud.
SessionManager
(ttl=None, redis_conn=None)¶ Session Manager - for active user sessions
Each login session is stored as an encrypted JSON dict in redis, indexed by sub:token
- Parameters
ttl (int) – seconds until a session expires
redis_conn (obj) – connection to redis service
-
create
(uid, roles, key_id=None, **kwargs)¶ Create a session, which is an encrypted JSON object with the values defined in https://tools.ietf.org/html/rfc7519 for JWT claim names:
exp - expiration time, as integer Unix epoch time
iss - a constant JWT_ISSUER
jti - JWT ID, the randomly-generated token
sub - the uid of a user
We add these:
auth - authorized roles
any other key=value pairs the caller passes as kwargs
The session automatically expires based on object’s ttl. Part of the jti token is used in redis key, to allow a user to log into multiple sessions. The rest of the token is encrypted, to secure it from replay attack in the event redis traffic is compromised.
- Parameters
uid – User ID
roles – Authorized roles
key_id – session key ID for redis (defaults to uid)
nonce – a unique identifier for the token (random if not specified)
ttl – duration of session (defaulted from class init)
- Returns
Keys include auth (authorized roles), exp / iss / jti / sub (as above), along with parameters passed into this function
- Return type
dict
-
delete
(uid, token, key_id=None)¶ Cancel a session
- Parameters
uid – User ID
token (str) – The token value passed from create as ‘jti’
key_id (str) – session key ID for redis
-
get
(uid, token, arg=None, key_id=None)¶ Get one or all key-value pairs stored by session create
- Parameters
uid (str) – User ID
token (str) – The token value passed from create as ‘jti’
arg (str) – key of desired value (None to fetch all)
key_id (str) – session key ID for redis (defaults to uid)
- Returns
single value or dictionary of all session keys
- Return type
dict or str
-
update
(uid, token, arg, value, key_id=None)¶ Update a specified session key
- Parameters
uid – User ID
token (str) – The token value passed from create as ‘jti’
arg (str) – key to update
value (str) – new value for key
key_id (str) – session key ID for redis (defaults to uid)
|
Role-based access control |
|
Access class for account settings, with cache - converts db record to object attributes |
|
AES encryption for strings |
|
Controller base class |
const.py |
|
database.py |
|
exceptions.py |
|
geocode.py |
|
|
Account usage limits |
health.py |
|
initialize.py |
|
|
This implementation supports standard system metrics and two types of usage-billing: granted limits per period, or metered usage. |
|
Simple mutex implementation for non-clustered Redis |
|
Rate Limiting |
|
Service configration - have it your way! |
|
Service Registry |
|
Session Authorization |
|
Session Manager - for active user sessions |
utils.py |