Supported Modes

Depending on the plugin, Splitgraph can perform data integration in one or more of the three modes:

Full reload

Load the data from a source into a Splitgraph repository. This creates a new version ("tag") of the data with the source dataset reloaded from scratch.

All data sources support a full reload.

Incremental load (change data capture)

Some data sources also support incremental replication: only copying over new or changed data since the last time the ingestion ran. In some cases, you might need to specify a monotonically increasing column to be used as a "replication cursor".

Live querying

Query the data live (at source) without ingestion. This is also sometimes called "data federation". Splitgraph will create a special tag in a repository called :live that you can reference in order to run the query against the original data source:

SELECT * FROM "some/repo:live".some_table