Synchronize data from external sources
Prerequisite: Create a partitioned table
By using the new keyword PARTITIONED BY
, you will be able to synchronize your external database with Indexima at minimum cost.
You can create a partitioned external table like this :
CREATE EXTERNAL TABLE nyc_yellow FROM my_source_table IN 'jdbc:...'() PARTITIONED BY (year, month);
Synchronize an external table
By using the SYNCHRONIZE keyword on the desired partition, Indexima will apply all retrieve and all updates from the external source.
You can use it like this :
SYNCHRONIZE nyc_yellow PARTITION (year = 2016, month = 11);
Automatically synchronize external tables
Instead of running the SYNCHRONIZE
command, Indexima will periodically ask Snowflake what are the last changes. By enabling this feature, Indexima will always be up-to-date with your external database.
For this purpose, you have 3 parameters to play with :
external.synchronize.check.rate
: Number of seconds between external table synchronization check.external.synchronize.check.cron
: Cron expression for table synchronization check, more precise thanexternal.synchronize.check.rate
.external.synchronize.check.user
: The user who will run the SYNCHRONIZE during the automatic update.