Understanding the concept of capturing changed data from your source endpoint is an important piece to building a successful on-going data integration. As a best practice, capturing the records that have changed – the delta – your integrations will process data sequentially, closer to real-time, and more efficiently.
Source data changes can come from an application’s users, workflows, or even other integrations. These applications and databases might audit or add version numbers to the records that are changed, however, most modern applications will keep a Modified On field to track changes.
As an out-of-the-box feature, you’ll be able to choose a DateTime field from your table/entity and use for capturing modified records.
In this podcast, Nate and Mark will talk about using your application or database’s modified on DateTime field as one type of detection pattern for tracking net-change using Scribe Online Integration Service (IS). In addition to describing how the “High Water Mark” works, they’ll also discuss considerations for choosing the right DateTime field, handling row errors, and the use of an “Ignore User.”