In part 1 of my Concepts to Integration blog series, I explained the three main concepts you should understand as you’re preparing to scope or start an integration project: abstraction, metadata, and standardization. These concepts will help you build more robust integrations, and lower your total cost of ownership once the project is out the door. I explained abstraction in my first post, and now I’ll move on to metadata.
What is Metadata
Metadata is data about data. Metadata is important to integration, especially in the Logical-level of your abstracted integration. Metadata is the universal layer between the logical operations and field mappings (e.g., Upsert a HubSpot Contact), and the underlying API method(s) (e.g., an HTTP POST of a JSON body). More specifically, Metadata is information about the entities/objects, their supported actions, and fields available via API.
In order to create any logical mapping from one application to another, you need to have structural and descriptive information about the applications. Think of metadata as a card catalog to an API. Don’t remember card catalogs? How about a Swagger specification/definition?.
What to define
When you’re building an integration, it’s helpful to define metadata for each of the following:
- Entities – this should be a dictionary of the available endpoints (or objects) you plan to use. Be sure to describe any special properties (e.g., such as supports Bulk/Batch operations).
- Actions – these are the operations you wish to support on your entities. Examples of actions are your common HTTP Verbs, CRUD operations, or special/virtual operations like Merge or Add-to-List. It’s a good idea to use universal terms (e.g., instead of “PUT” or “Merge,” use “Update”).
- Properties – these describe the fields and their data types. Remember that properties may be different depending upon requests and responses. Also, make sure to describe special properties like primary/foreign keys, indexes, required/not null, etc.
- Relationships – Be sure to define any relationships that an entity has to another entity, and how that relationship is described (this could be a foreign key or a virtual property). Be sure to distinguish between parent, child, and complex relationships.
After you define metadata, you should be able to look at an entity and determine the types of operations you can perform on it (e.g., Query, Update, Create), its input and output properties (e.g., firstName, lastName, email, and accountId), its declared relationships with foreign keys (e.g., my Contact relates 1:1 to an Account via an accountId property), and its special properties (e.g. supports Bulk).
Discovering Metadata
Enterprise software and many SaaS applications are often easily customizable by its users. This makes defining your metadata definition more difficult. Fortunately, there are design patterns for defining instance-specific Metadata – or what I like to call, Discovering Metadata!
One route to describing your metadata dynamically is using a metadata service, such as a Swagger definition, WSDL, or OData $metadata web service. If there’s no service available, you can always use Reflection on source quer(ies). I plan to address dynamic metadata later in a future post and I’ll be sure to discuss the pros and cons to discovering vs. static metadata (Spoiler: sometimes you’ll want to use both).
Robust metadata provides the means to understand how to map your collections, and where functions and formulas are needed for transformations.
