Network tunnels

You can add data from data sources accessible only from within a private network or behind network address translation (NAT) by creating a network tunnel. Each tunnel connects a machine with access to a private data source with Splitgraph Cloud. The following plugins support network tunneling:

• postgres_fdw
• mongo_fdw
• mysql_fdw
• elasticsearch
• airbyte-clickhouse
• airbyte-cockroachdb
• airbyte-db2
• airbyte-mssql
• airbyte-mysql
• airbyte-oracle
• airbyte-postgres
• airbyte-redshift
• airbyte-tidb

Temporary access

The sgr cloud tunnel command can be used to create a tunnel which can be used to create a repository suitable for live querying or loading data. For example, to query a local PostgreSQL instance, the tunnel can be established by running:

sgr cloud tunnel localhost:5432

sgr displays the network address of the tunneled service, e.g.:

To connect to localhost:5432 from Splitgraph, use the following connection parameters:
Host: fd71:5f64:7628:51cb:7884:f261:0ab5:7b52
Port: 31337
launching rathole client

Use this host and port to fill out the external repository creation form.

Once the repository exists, it can be used to:

• Execute ad-hoc queries against the private data source, potentially joining it with tables from other repositories.
• Load the data source's contents using the "Sync Now" button, or during the repository creation process.
• Use a CREATE TABLE statement to import data from the private data source applying transformations.

Data loaded from the private data source will be available even after the tunnel disconnects, but live querying will no longer be possible. Once disconnected, a temporary tunnel cannot be revived, but it is possible to create repositories with persistent tunnels which can be reconnected.

Persistent tunnels

It is also possible to create a tunneled external repository which allows the re-establishing of the tunnel. This type of repository can be created using a splitgraph.yml file with the .external.tunnel field set to true.

Once the repository is created with sgr cloud load based on the splitgraph.yml definition, a tunnel can be started with:

sgr cloud tunnel namespace/repository host:port

This command can be used to re-establish the tunnel from any host with access to the private data source. The tunnel can only be used by a single host at any time.

For a more detailed walkthrough, see our tunnel jumpstart guide.