Evaluating AWS EventBridge Schema Discovery

This article is going to experiment with the AWS Schema Registry with Event Discovery and evaluate EventBridge Atlas which is a 3rd party tool that generates documentation pages for it.

This is part 1 of 2. Part 2 will include the code repo.

Getting Started (for me, not you)

Architecture-wise I don't need much for evaluating things. I'll need an event bus with discovery turned on... and really thats it. I ❤️ CDK so I use that to generate my CloudFormation and to make an event bus and turn on schema discovery it's as simple as:

const bus = new EventBus(this, 'inquisitorBus', {
      eventBusName: 'inquisitorBus',
    });
    new CfnDiscoverer(this, 'inquisitorDisco', {
      sourceArn: bus.eventBusArn,
      description: 'Inquisitor Discoverer',
    });

From there we need to send some events to see if they're discovered...

aws events put-events --entries '[{"Source": "mystore","DetailType": "Review Created","EventBusName":"inquisitorBus","Detail": "{\"star_rating\": 5,  \"description\": \"The size and length fit me well and the design is fun. I felt very secure wearing this tshirt. \",  \"helpful_count\": 34,  \"unhelpful_count\": 1,  \"pros\": [\"lightweight\",\"fits well\"  ],  \"cons\": [],  \"customer\": {\"name\": \"Julian Wood\",\"email\": \"[email protected]\",\"phone\": \"+1 604 123 1234\"  },  \"product\": {\"product_id\": 788032119674292922,\"title\": \"Encrypt Everything Tshirt\",\"sku\": \"encrypt-everything-tshirt\",\"inventory_id\": 23190823132,\"size\": \"medium\",\"taxable\": true,\"image_url\": \"https://img.mystore.test/encrypt-tshirt.jpg\",\"weight\": 200.0}}"}]'

After sending the first event I checked the Discovered Schema Registry in AWS Console to find.... nothing.

It does say "It could take several minutes for schemas to appear once discovery has been enabled."

Ultimately it took around 5 minutes for the schema to be discovered...

I'd like to see if the schema discovery is smart enough to infer optional properties of payloads, so let's send some variations of the event:

aws events put-events --entries '[{"Source": "mystore","DetailType": "Review Updated","EventBusName":"inquisitorBus","Detail": "{\"star_rating\": 5,  \"description\": \"The size and length fit me well and the design is fun. I felt very secure wearing this tshirt. \",  \"helpful_count\": 34,  \"unhelpful_count\": 1,  \"pros\": [\"lightweight\",\"fits well\"  ],  \"cons\": [],  \"customer\": {\"name\": \"Julian Wood\",\"email\": \"[email protected]\",\"phone\": \"+1 604 123 1234\"  },  \"product\": {\"product_id\": 788032119674292922,\"title\": \"Encrypt Everything Tshirt\",\"sku\": \"encrypt-everything-tshirt\",\"inventory_id\": 23190823132,\"size\": \"medium\",\"taxable\": true}}"}]'

aws events put-events --entries '[{"Source": "myOtherStore","DetailType": "Review Created","EventBusName":"inquisitorBus","Detail": "{\"star_rating\": 5,  \"description\": \"The size and length fit me well and the design is fun. I felt very secure wearing this tshirt. \",  \"helpful_count\": 34,  \"unhelpful_count\": 1,  \"pros\": [\"lightweight\",\"fits well\"  ],  \"cons\": [],  \"customer\": {\"name\": \"Julian Wood\",\"email\": \"[email protected]\",\"phone\": \"+1 604 123 1234\"  },  \"product\": {\"product_id\": 788032119674292922,\"title\": \"Encrypt Everything Tshirt\",\"sku\": \"encrypt-everything-tshirt\",\"inventory_id\": 23190823132,\"size\": \"medium\",\"taxable\": true,\"image_url\": \"https://img.mystore.test/encrypt-tshirt.jpg\",\"weight\": 200.0}}"}]'

aws events put-events --entries '[{"Source": "mystore","DetailType": "Review Created","EventBusName":"inquisitorBus","Detail": "{\"star_rating\": 5,  \"helpful_count\": 34,  \"unhelpful_count\": 1,  \"pros\": [\"lightweight\",\"fits well\"  ],  \"cons\": [],  \"customer\": {\"name\": \"Julian Wood\",\"email\": \"[email protected]\",\"phone\": \"+1 604 123 1234\"  },  \"product\": {\"product_id\": 788032119674292922,\"title\": \"Encrypt Everything Tshirt\",\"sku\": \"encrypt-everything-tshirt\",\"inventory_id\": 23190823132,\"size\": \"medium\",\"taxable\": true,\"image_url\": \"https://img.mystore.test/encrypt-tshirt.jpg\"}}"}]'

aws events put-events --entries '[{"Source": "mystore","DetailType": "Review Created","EventBusName":"inquisitorBus","Detail": "{\"helpful_count\": 34,  \"unhelpful_count\": 1, \"customer\": {\"name\": \"Julian Wood\",\"email\": \"[email protected]\",\"phone\": \"+1 604 123 1234\"  },  \"product\": {\"product_id\": 788032119674292922,\"sku\": \"encrypt-everything-tshirt\",\"inventory_id\": 23190823132,\"taxable\": true,\"image_url\": \"https://img.mystore.test/encrypt-tshirt.jpg\"}}"}]'

aws events put-events --entries '[{"Source": "mystore","DetailType": "Review Created","EventBusName":"inquisitorBus","Detail": "{\"helpful_count\": 34}"}]'

There is a fairly consistent (several minute) lag between events and discovery.

It does NOT infer optional properties and only uses the most recent payload type for the schema, which is a little disappointing. It DOES store multiple versions of the schemas though... so you could re-combine them. I suppose the best practice in this case would be to have a consistent event structure and if you need optional parameters, have different event detail types?

Is there a schema registry change event?

AWS has two provided events aws.schemas@SchemaCreated and aws.schemas@SchemaVersionCreated. The source of these events is aws.schemas and the detail-types are SchemaCreated and SchemaVersionCreated... so if I subscribe to just the source using the default event bus you can get these notifications. The event payload looks like this:

{
    "version": "0",
    "id": "f480b317-a12f-5ccc-89ac-2446862c329b",
    "detail-type": "Schema Version Created",
    "source": "aws.schemas",
    "account": "359317520455",
    "time": "2021-07-24T15:01:29Z",
    "region": "us-east-1",
    "resources": [
        "arn:aws:schemas:us-east-1:359317520455:schema/discovered-schemas/mystore@ReviewCreated"
    ],
    "detail": {
        "SchemaName": "mystore@ReviewCreated",
        "SchemaType": "OpenApi3",
        "RegistryName": "discovered-schemas",
        "CreationDate": "2021-07-24T15:01:19Z",
        "Version": "5"
    }
}

This will be useful in part 2 😈

At this point I think we're ready to kick the tires on EventBridge Atlas.

EventBridge Atlas

EventBridge Atlas isn't an npm library. Setup for it wants you to clone the repo, install and then locally run / configure everything. It doesn't auto-update. It would be pretty easy to create a pipeline to re-generate the docs on a Schema Event. Part 2 will deal with improving that, but for now let's just use the default.

It's a pretty simple project that downloads the schema and then processes it through some parsers to support 4 different documentation engines.

Slate - pretty bare bones...

asyncapi - I like how this one split out the messages separate from the schemas

docuowl - similar to slate

flow (node diagram) - nice for visualization and would be great combined with x-ray or something, but for dev purposes I don't think it's that useful

asyncapi looks to be the most useful.

So what does the EventBridge Atlas project actually do?

First it has you run npm run generate-metadata-templates command, but this step is optional. This uses the aws sdk to list all the schemas, retrieves them all as JSON, and extends them a bit. Nothing too special here... could be done as part of a lambda execution and saving to s3 and/or dynamodb. This step is so that you can take these templates and add your own notes manually.

Next you have your choice of documentation engines... Using your choice you run npm run build:<engine> which in turn runs npm run generate... which does similar stuff to generating the metadata templates. It also retrieves the schemas and extends them with some OpenAPI functionality. After that it runs everything through a parser to get it in the preferred engine's format. The project in general requires docker, but it looks like it's only for the slate and docuowl engines... which I don't want to use anyways.

Next Steps

This concludes part 1. I played with Schema Discovery and kicked the tires on EventBridge Atlas and learned a lot in the process. But I can make this better.

Part 2 will involve:

Having a lambda subscribe to the Schema Change Events

Use the lambda to generate and store metadata for the events (and infer optional properties)

Generate the TypeScript Interfaces to be published via an npm package

Store the automated documentation in DynamoDB (for merging / extending)

Host the documentation

Stay tuned...