Earlier this year we opened our latest data center in Europe to provide our customers with two key features when it comes to our Cloud Agents:
- First is data sovereignty – making sure data doesn’t leave the region
- Second involves improved performance – removing latency caused by the distance between datastores
We have already written about the security, so I wanted to delve into the performance aspect of the new European data center and explain how easy it is to “Make the Switch” and take advantage of this new offering by Scribe.
I’m a facts and figures type of product guy, so I wanted to prove to myself that there is a performance improvement. Logically, if you move two end-points closer together, the amount of time it takes for them to communicate should be reduced. With small batches of records this might not make a huge difference, but in the integration space where you might be moving thousands of records, each millisecond is compounded.
I started off setting up an Azure SQL Standard instance in Europe and one in the US. I then installed a new Cloud Agent in Europe. My test was simply, select accounts from a SQL database and insert them back into the same database to keep this as neutral as possible. One thing I needed to take into account is what I like to call “the performance sweet spot”, the total number of records you need to process to see peak performance.
There is some overhead needed to start any job and begin processing records. If you test with too few records the integration time is affected noticeably by the startup process when compared to larger record sets. At some point the number of records processed per second averages out. For my example, I found this point right around 30K records. This will vary based on the specific scenario and other factors.
I found that when I used an Agent in Europe to transfer data to a location in Europe Scribe Online processed around 55 records per second. When the data was transferred to the US (to a similar Azure instance), performance was reduced to 9 records per second, so by reducing the distance between the Agent and the datastore, I saw a significant improvement.
In an effort to be thorough in my testing, I did the same test in reverse, using a US based agent. Similar to the European Agent test, as soon as the processing left the region, performance dropped to around 9 records per second.
A few disclaimers here, your “mileage” may vary because my test was validating the basic principle using small servers. My test does demonstrate that when processing large amounts of data performance improves by moving your Cloud Agent to the data center nearest your datastore.
How to make the Switch…
You can find out which data center you are associated with by going into Scribe Online and selecting More – > Edit Organization. At the bottom of the Edit dialog you see either Virginia United States or Frankfurt Germany. This cannot be updated through the User Interface. However, if you want to change data centers we have a few options available.
- Do it Yourself (DYI)
The first option is a DYI project – you are welcome to create a new Organization in Scribe Online, which prompts you to choose the data center associated with the new Organization. Once you select a data center, the Organization automatically starts with a trial subscription. You can then add a Cloud Agent, and configure your Connections and Solutions in this Organization. Contact us at email@example.com to have us migrate your current subscription to the new Organization. Please note, with this option execution history is kept under your original Organization, and new history is under the new Organization. If you are moving Maps from your original Solutions to your new Solutions using our export/import, be sure to set the Net Change value on the Maps in the new Solutions.This method provides a great way to validate the performance yourself since every implementation is slightly different.
- Scribe Service Offering
If you are not comfortable with the whole DYI process, we have another option. We can make the change for you for a small service fee, but make it seamless. If this is something you are interested in pursuing, please contact firstname.lastname@example.org. With this option your history will remain (based on our data retention policy), however access to source query data is not possible.
We are considering expanding the offering of data centers in other regions. Please feel free to reach out to me or anyone at Scribe with suggestions for the next location.