DynamoDB OneTable API Overview

DynamoDB OneTable (OneTable) is an access library for DynamoDB applications that use one-table design patterns with NodeJS. OneTable makes dealing with DynamoDB and one-table design patterns dramatically easier while still providing easy access to the full DynamoDB API.

We've worked very hard to make the OneTable API expressive, terse and elegant to use. I hope you will find that it makes your DynamoDB developments proceed faster and more easily than every before.

This article takes a quick tour through the OneTable API and demonstrates basic calls and patterns when using the API.

OneTable Overview

A good way to read this article is to checkout the OneTable Overview working sample and follow along in VS Code. If you are using TypeScript, use the TypeScript sample instead.

OneTable Overview Sample — A quick tour through OneTable.

OneTable TypeScript Sample — A TypeScript quick tour.

Installation

npm i dynamodb-onetable

Quick Tour

Import the OneTable library. If you are not using ES modules or Typescript, use require to import the libraries.

import {Table} from 'dynamodb-onetable'

If you are using the AWS SDK V2, import the AWS DynamoDB class and create a DocumentClient instance.

import DynamoDB from 'aws-sdk/clients/dynamodb'
const client = new DynamoDB.DocumentClient(params)

This version includes prototype support for the AWS SDK v3.

If you are using the AWS SDK v3, import the AWS v3 DynamoDBClient class and the OneTable Dynamo helper. Then create a DynamoDBClient instance and Dynamo wrapper instance.

import {DynamoDBClient} from '@aws-sdk/client-dynamodb'
import Dynamo from 'dynamodb-onetable/Dynamo'
const client = new Dynamo({client: new DynamoDBClient(params)})

Initialize your your OneTable Table instance and define your models via a schema. The schema defines your single-table entities, attributes and indexes.

import MySchema from './Schema'

const table = new Table({
    client: client,
    name: 'MyTable',
    schema: MySchema,
})

Schemas

Schemas define your models (entities), keys, indexes and attributes. Schemas look like this:

export default {
    indexes: {
        primary: { hash: 'pk', sort: 'sk' },
        gs1:     { hash: 'gs1pk', sort: 'gs1sk', project: ['gs1pk', 'gs1sk', 'data'] },
    },
    models: {

        Account: {
            pk:         { type: String, value: 'account#${id}' },
            sk:         { type: String, value: 'account#' },
            id:         { type: String, uuid: true, validate: Match.ulid },
            name:       { type: String, required: true, unique: true, validate: Match.name },
            balance:    { type: Number, default: 0 },

            //  Search by account name or by type
            gs1pk:      { type: String, value: 'account#' },
            gs1sk:      { type: String, value: 'account#${name}${id}' },
        },

        User: {
            pk:         { type: String, value: 'account#${accountId}' },
            sk:         { type: String, value: 'user#${email}' },
            accountId:  { type: String, required: true },
            id:         { type: String, uuid: true, validate: Match.ulid },
            name:       { type: String, required: true, validate: Match.name },
            email:      { type: String, required: true, validate: Match.email, crypt: true },

            address:    { type: Object, default: {}, schema: {
                street: { type: String, /* map: 'data.street' */ },
                city:   { type: String, /* map: 'data.city' */ },
                zip:    { type: String, /* map: 'data.zip' */ },
            } },

            status:     { type: String, required: true, default: 'active', enum: ['active', 'inactive'] },
            balance:    { type: Number, default: 0 },

            //  Search by user name or by type
            gs1pk:      { type: String, value: 'user#' },
            gs1sk:      { type: String, value: 'user#${name}#${id}' },
        },

        Product: {
            pk:         { type: String, value: 'product#${id}' },
            sk:         { type: String, value: 'product#' },
            id:         { type: String, uuid: true, validate: Match.ulid },
            name:       { type: String, required: true },
            price:      { type: Number, required: true },

            //  Search by product name or by type
            gs1pk:      { type: String, value: 'product#' },
            gs1sk:      { type: String, value: 'product#${name}#${id}' },
        },

        Invoice: {
            pk:         { type: String, value: 'account#${accountId}' },
            sk:         { type: String, value: 'invoice#${id}' },

            accountId:  { type: String, required: true },
            date:       { type: Date, default: () => new Date() },
            id:         { type: String, uuid: true },
            product:    { type: String },
            count:      { type: Number },
            total:      { type: Number },

            //  Search by invoice date or by type
            gs1pk:      { type: String, value: 'invoice#' },
            gs1sk:      { type: String, value: 'invoice#${date}#${id}' },
        }
    }

This schema has models for: Account, User, Product and Invoice.

The schema also defines your indexes. The index key values are defined for each model and can derive their values from other attributes via value templating.

Most interactions with the OneTable API are via model objects that correspond to each schema model. For example,

const Account = table.getModel('Account')

For typescript, we use a slightly different formulation that provides type safety for your model references at compile time.

type AccountType = Entity<typeof Schema.models.Account>
const Account = table.getModel<AccountType>('Account')

Create a Table

Normally you would create your tables separately, hopefully via infrastructure-as-code. But for development, you can create a table directly from the schema via:

await table.createTable()

Creating an Item

Once the table is prepared, you can create an item:

let account = await Account.create({name: 'Acme Airplanes'})

This will write the following to DynamoDB:

{
    pk:         'account:8e7bbe6a-4afc-4117-9218-67081afc935b',
    sk:         'account:98034',
    name:       'Acme Airplanes',
    id:         '8e7bbe6a-4afc-4117-9218-67081afc935b',
    balance:    0,
    created:    '2021-07-05T04:58:55.136Z',
    updated:    '2021-07-05T04:58:55.136Z',
}

This API will return the following, and omit the underlying hidden properties: pk, sk, gs1pk, gs1sk.

{
    name:       'Acme Airplanes',
    id:         '8e7bbe6a-4afc-4117-9218-67081afc935b',
    balance:    0,
    created:    '2021-07-05T04:58:55.136Z',
    updated:    '2021-07-05T04:58:55.136Z',
}

To return the hidden properties, set {hidden: true} in the params:

let account = await Account.create({name: 'Acme Airplanes'}, {hidden: true})

Context Properties

If creating a multi-tenant application, it us helpful to have the account ID be part of the primary key. Rather than having each API specify this account ID, it is best to provide this via a context property that ensures it is never omitted. Context properties are blended with the API properties and take precedence.

table.setContext({accountId: account.id})

Now when we create a user, it will automatically set the user account ID to the context property value.

let user = await User.create({name: 'Road Runner', email: '[email protected]'})

Getting Items

To get an item, we provide the properties require to construct the key. In the schema, the Account keys were specified as:

{
    pk: { type: String, value: 'account#${id}' },
    sk: { type: String, value: 'account#' },
}

So we need to provide the account id and then OneTable will construct the keys for you.

This will retrieve an account, using the primary key.

account = await Account.get({id: '8e7bbe6a-4afc-4117-9218-67081afc935b'})

Using a secondary index

We can also fetch items using a secondary index which allows us to retrieve items by other attributes such as the Account name or User name. An overloaded secondary index can be used to retrieve different items using alternative properties. The schema defines a Global Secondary Index (GSI) and then each model entity uses different values from their value templates into the GSI key fields.

For example, the User type constructs the GSI sort key using the user name and user ID.

{
    gs1pk: { type: String, value: 'user#' },
    gs1sk: { type: String, value: 'user#${name}#${id}' },
}

So we can retrieve a user by name from the GSI 1 using:

user = await User.get({name: 'Road Runner'}, {index: 'gs1', follow: true, log: true})

This will use the secondary index and construct a query to search for users that have the gs1sk set to user#Road Runner. Because the user ID portion is not provided, OneTable searches for a user that begins with the provided string.

To save storage costs, the GSI in this example only projects the gs1pk, gs1sk and data attributes to the GSI. That means the other user attributes are not returned when querying only from the GSI. OneTable addresses this by providing the follow: true parameter which follows the GSI read with a read from the primary index to fetch the full user record.

The {log: true} is useful to see the actual DynamoDB request logged to the console.

Finding items

To find all the users for an account:

let users = await User.find()

Remember, the account ID is provided by the context, so we don't need to explicitly provide it.

If we want to filter items, we can add non-key properties to use as filters. Here we find the users that have a role of 'admin'.

let adminUsers = await User.find({accountId: account.id, role: 'admin'})

Alternatively, we can use a where clause. This will find users with an account balance over $100. Where clauses are useful for more complex filtering expressions. A where clause supplies property references by enclosing the name in ${}. Similarly, values are enclosed in {} without the dollar.

users = await User.find({}}, {
    where: '${balance} > {100}'
})

Note: these are not Javascript string templates, so do not use back-ticks when specifying.

Updating Items

To update items, you can provide the keys and new values via the properties. Remember that when updating you cannot change an item's keys in DynamoDB.

user = await User.update({email: '[email protected]', balance: 0, role: 'admin'})

To help prevent accidental creation when updating, OneTable adds a constraint to ensure your items already exist and you don't accidentally create new items by providing incorrect key ingredient properties.

To update nested properties, provide them via the params.set. This will update just the property you specify.

user = await User.update({email: '[email protected]'}, {set: {'address.zip': '{"98034"}'}})

You can also use expressions to atomically update your attributes. For example:

user = await User.update({email: '[email protected]'}, {add: {balance: 10}})
user = await User.update({email: '[email protected]'}, {set: {balance: '${balance} - {20}'}})

Item Collections

A key power of DynamoDB is to fetch multiple related items in a single efficient request.

To fetch an account, the account users and invoices in one request, use table.fetch.

let collection = await table.fetch(['Account', 'User', 'Invoice'], {pk: `account#${account.id}`})

This returns an object map with lists for Account, User and Invoice.

In this request, we need to provide the actual primary key value as each model type may have different value templates to construct their pk value.

Batch Adding

To efficiently add many items, DynamoDB has batch operations. OneTable makes these easy. This code snippet creates 200 users in batches of 25.

let batch = {}
let i = 0, count = 0
while (i++ < 200) {
    User.create({name: `user${i}`, email: `user${i}@acme.com`}, {batch})
    if (++count >= 25) {
        await table.batchWrite(batch)
        batch = {}
        count = 0
    }
}

Transactions

OneTable transactions are expressed similarly to batches.

let transaction = {}
await Account.update({id: account.id, status: 'active'}, {transaction})
await User.update({id: user.id, role: 'user'}, {transaction})
await table.transact('write', transaction)

Pagination

The DynamoDB API has limits on the maximum size of request response data. So you often need to use pagination to step through large result sets.

OneTable uses the start param on request APIs and in result sets to mark the current reading position.

let start
do {
    users = await User.find({}, {start, limit: 25})
    //  Operate on this batch of 25 users ...
    start = users.start
} while (users.start)

Summary

Checkout the full overview sample at:

OneTable Overview Sample — A quick tour through OneTable.

OneTable TypeScript Sample — A TypeScript quick tour.

All OneTable Samples

You can read more in the detailed documentation at:

GitHub OneTable or

NPM OneTable.

Why OneTable?

DynamoDB is a great NoSQL database that comes with a steep learning curve. Folks migrating from SQL often have a hard time adjusting to the NoSQL paradigm and especially to DynamoDB which offers exceptional scalability but with a fairly low-level API.

The standard DynamoDB API requires a lot of boiler-plate syntax and expressions. This is tedious to use and can unfortunately can be error prone at times. I doubt that creating complex attribute type expressions, key, filter, condition and update expressions are anyone's idea of a good time.

Net/Net: it is not easy to write terse, clear, robust Dynamo code for one-table patterns.

Our goal with OneTable for DynamoDB was to keep all the good parts of DynamoDB and to remove the tedium and provide a more natural, "JavaScripty / TypeScripty" way to interact with DynamoDB without obscuring any of the power of DynamoDB itself.

Other Working Samples

OneTable Overview Sample — A quick tour through OneTable.

OneTable CRUD Sample — Basic CRUD.

OneTable TypeScript Sample — Basic TypeScript use.

OneTable Migrate Sample — how to use OneTable Migrate and the Migrate CLI.

OneTable Packed Attributes Sample — How to use packed attributes.

OneTable SenseDeep Sample — How to access SenseDeep log data.

All OneTable Samples

SenseDeep with OneTable

At SenseDeep, we've used the OneTable module extensively with our SenseDeep serverless developer studio. All data is stored in a single DynamoDB table and we extensively use one-table design patterns. We could not be more satisfied with DynamoDB implementation. Our storage and database access costs are insanely low and access/response times are excellent.

Contact

You can contact me (Michael O'Brien) on Twitter at: @mobstream, or email and read myBlog.

To learn more about SenseDeep and how to use our serverless developer studio, please visit https://www.sensedeep.com/.

Links