Not everyone is an integration specialist. Developers who focus on enterprise architecture or custom application development may sometimes be called upon to address some of their company’s most pressing integration needs because of their high skill level. Regardless of whether you are an integration specialist or an integration newcomer we recommend you take a methodical approach to your project to ensure that you make the best technology decisions for your company. In that light, there are three main concepts you should understand as you’re preparing to scope or start an integration project: abstraction, metadata, and standardization. Understanding these concepts will help you build and implement more robust integrations, and lower the total cost of ownership once the project is out the door.
Concept 1: Abstraction
When you first start an integration project, you are likely wondering how you can bring together data from two applications that have very different structures. Furthermore, you probably are considering how you can setup your integration to meet the current business requirements, while still leaving yourself the flexibility to adapt the integration to changes that are likely to occur in those business requirements. A concept that can help is abstraction, which will allow you to breakdown your integration challenge into several key dimensions, or layers, so you can better understand your overall integration requirements before you start looking at possible technology solutions. The layers of your abstraction will allow you scale your integration efforts as new endpoints and instances are added or requirements to the scope of work change.
Similar to the layers of abstraction for a database system, you can consider an integration as having three layers: the View, Logical, and Physical layers (see the Figure). Each is described below.
- Physical Layer – this is the lowest layer of abstraction and it defines how you actually transfer the data. Most people approaching an integration will first think about this layer because applications readily reveal the communication protocols and APIs that they support. Answer these questions about the applications you are integrating to get a solid understanding of your requirements at the Physical layer:
- Do the actual data transfer mechanisms or payloads differ between API’s?
- Are the API payloads representing flat, relational, hierarchical, or binary data?
- Do you need to handle pagination or batch processing, and what do those calls look like?
- Do you have transactional operations you need to cache the results for?
- Logical Layer – the next higher level of abstraction describes what relationship the data in one system has with the data in another system. The field mapping, relationships, transformations, and metadata about a system’s configuration is considered part of the logical layer. To understand your requirements at the Logical layer, answer these questions:
- What API endpoints or tables are you going to be working with? These are your entities.
- What are the properties, fields, and data types included in each entity?
- What are the available actions to get or set data on an entity?
- Do the inputs and outputs match when you are performing operations?
- Are you asking for data or receiving webhook messages?
- View Layer – the highest level of abstraction is information on your overall integration solution, such as a description of what it does and how it does it. This information will get exposed to other developers and consumers of the integration to explain the key characteristics of the integration so that they can effectively deploy, run, manage, and adapt the integration over time. For example, at the view layer a description will help you distinguish two connectors; one that is a Marketo connector that passes data using the SOAP mechanism and one that uses the RESTful mechanism. To understand your requirements at the View layer, consider these questions:
- Are you integrating a database, an FTP drive, or a web service?
- For each instantiated integration, how will you handle authorization?
- Do you need to call an Authentication endpoint, provide user credentials for Basic Auth, or do you need to programmatically work through 3-legged OAuth 2.0 for access?
- Is there a specific port or protocol to use?
- Do you need to be whitelisted?
Abstraction is more than arranging HTTP verbs and authentication methods into buckets or classes. And it’s more methodical than jumping right to the Physical abstraction layer. It really does help you scope out the challenge of integrating two very different systems and gives you the requirements and a roadmap for how to approach the project.
Figure – Determining how you will integrate two very different applications starts with breaking the problem into three abstraction layers.
Let’s consider two examples so you can see how using abstraction layers can help you approach planning an integration between two SaaS applications:
We want to setup a bi-directional integration between the Microsoft Dynamics 365 sales module and the HubSpot marketing automation system. We want the integration to offer recurring updates and make those updates in batch operations. We start by evaluating each application on the three abstraction layers, resulting in:
|Abstraction Layer||Application 1||Application 2|
|View||Dynamics 365 Sales/CRM (SOAP SDK), ADFS Auth||HubSpot (RESTful HTTP API), OAuth 2.0|
|Logical||Leads, Contacts, Accounts, Opportunities||Contacts, Companies, Deals|
|Physical||Flat/Relational data, XML payloads||Hierarchical data, JSON payloads|
This tells us that our integration will need to make the two applications compatible on the logical layer by translating between the CRM’s Leads and Contacts, with HubSpot’s Contact entity. This also gives us an idea of how compatible these applications are on the physical layer by quickly presenting how the data is stored (flat/relational vs. hierarchical) and how the data is accessed and transferred (SOAP and XML vs. RESTful and JSON). These become the key requirements for our integration project.
In any integration project, the abstraction method will reveal the unique requirements of that specific project. So if we consider another example, where we want to create a bi-directional integration between the Dynamics 365 Operations module (or Dynamics AX) and Salesforce, we get different results. In this case we want to use event-driven messaging and transfer information on transactional operations between the applications. Using the abstraction method, we would get:
|Abstraction Layer||Application 1||Application 2|
|View||Salesforce.com (SOAP), Basic or OAuth 2.0||Dynamics 365 Operations/AX (OData), Basic Auth|
|Logical||Accounts, Products, Opportunities, Opportunities Line Items||Customers, Items, Sales Order Headers, Sales Order Line Items|
|Physical||Flat/Relational data, XML payloads, has fields for Last Modified (when/by who)||Flat/Relational data, JSON payloads, does not have fields for Last Modified|
This second example shows that our logical layer aligns nicely between these systems – however, data about when records were changed and by whom was not included in #2; that will pose problems for tracking net-change later on.
The abstraction method is a powerful way to start looking at your integration projects. It provides you with a structure to start analyzing the task of integrating very different applications. When considering tackling this task yourself, use this method to develop a scope and scale for your project. When looking at using an integration platform for your project, this will give you parameters to use to evaluate whether that platform can solve your specific integration challenge.
By breaking down integration concepts into smaller chunks, you can pair the most complex and robust monolithic API with the latest-and-greatest microservices. In my next post, I’ll discuss the concept around Metadata in more depth.