Fauna
This page guides you through setting up a Fauna source.
Overview
The Fauna source supports the following sync modes:
- Full Sync - exports all the data from a Fauna collection.
- Incremental Sync - exports data incrementally from a Fauna collection.
You need to create a separate source per collection that you want to export.
Preliminary setup
Enter the domain of the collection's database that you are exporting. The URL can be found in the docs.
Full sync
Follow these steps if you want this connection to perform a full sync.
- Create a role that can read the collection that you are exporting. You can create the role in the Dashboard or the fauna shell with the following query:
CreateRole({
name: "airbyte-readonly",
privileges: [
{
resource: Collections(),
actions: { read: true },
},
{
resource: Indexes(),
actions: { read: true },
},
{
resource: Collection("COLLECTION_NAME"),
actions: { read: true },
},
],
});
Replace COLLECTION_NAME with the name of the collection configured for this connector. If you'd like to sync
multiple collections, add an entry for each additional collection you'd like to sync. For example, to sync
users and products, run this query instead:
CreateRole({
name: "airbyte-readonly",
privileges: [
{
resource: Collections(),
actions: { read: true },
},
{
resource: Indexes(),
actions: { read: true },
},
{
resource: Collection("users"),
actions: { read: true },
},
{
resource: Collection("products"),
actions: { read: true },
},
],
});
- Create a key with that role. You can create a key using this query:
CreateKey({
name: "airbyte-readonly",
role: Role("airbyte-readonly"),
});
- Copy the
secretoutput by theCreateKeycommand and enter that as the "Fauna Secret" on the left. Important: The secret is only ever displayed once. If you lose it, you would have to create a new key.
Incremental sync
Follow these steps if you want this connection to perform incremental syncs.
- Create the "Incremental Sync Index". This allows the connector to perform incremental syncs. You can create the index with the fauna shell or in the Dashboard with the following query:
CreateIndex({
name: "INDEX_NAME",
source: Collection("COLLECTION_NAME"),
terms: [],
values: [{ field: "ts" }, { field: "ref" }],
});
Replace COLLECTION_NAME with the name of the collection configured for this connector.
Replace INDEX_NAME with the name that you configured for the Incremental Sync Index.
Repeat this step for every collection you'd like to sync.
- Create a role that can read the collection, the index, and the metadata of all indexes. It needs access to index metadata in order to validate the index settings. You can create the role with this query:
CreateRole({
name: "airbyte-readonly",
privileges: [
{
resource: Collections(),
actions: { read: true },
},
{
resource: Indexes(),
actions: { read: true },
},
{
resource: Collection("COLLECTION_NAME"),
actions: { read: true },
},
{
resource: Index("INDEX_NAME"),
actions: { read: true },
},
],
});
Replace COLLECTION_NAME with the name of the collection configured for this connector.
Replace INDEX_NAME with the name that you configured for the Incremental Sync Index.
If you'd like to sync multiple collections, add an entry for every collection and index
you'd like to sync. For example, to sync users and products with Incremental Sync, run
the following query:
CreateRole({
name: "airbyte-readonly",
privileges: [
{
resource: Collections(),
actions: { read: true },
},
{
resource: Indexes(),
actions: { read: true },
},
{
resource: Collection("users"),
actions: { read: true },
},
{
resource: Index("users-ts"),
actions: { read: true },
},
{
resource: Collection("products"),
actions: { read: true },
},
{
resource: Index("products-ts"),
actions: { read: true },
},
],
});
- Create a key with that role. You can create a key using this query:
CreateKey({
name: "airbyte-readonly",
role: Role("airbyte-readonly"),
});
- Copy the
secretoutput by theCreateKeycommand and enter that as the "Fauna Secret" on the left. Important: The secret is only ever displayed once. If you lose it, you would have to create a new key.
Export formats
This section captures export formats for all special case data stored in Fauna. This list is exhaustive.
Note that the ref column in the exported database contains only the document ID from each document's
reference (or "ref"). Since only one collection is involved in each connector configuration, it is
inferred that the document ID refers to a document within the synced collection.
| Fauna Type | Format | Note |
|---|---|---|
| Document Ref | { id: "id", "collection": "collection-name", "type": "document" } | |
| Other Ref | { id: "id", "type": "ref-type" } | This includes all other refs, listed below. |
| Byte Array | base64 url formatting | |
| Timestamp | date-time, or an iso-format timestamp | |
| Query, SetRef | a string containing the wire protocol of this value | The wire protocol is not documented. |
Ref types
Every ref is serialized as a JSON object with 2 or 3 fields, as listed above. The type field must be
one of these strings:
| Reference Type | type string |
|---|---|
| Document | "document" |
| Collection | "collection" |
| Database | "database" |
| Index | "index" |
| Function | "function" |
| Role | "role" |
| AccessProvider | "access_provider" |
| Key | "key" |
| Token | "token" |
| Credential | "credential" |
For all other refs (for example if you stored the result of Collections()), the type must be "unknown".
There is a difference between a specific collection ref (retrieved with Collection("name")), and all the reference
to all collections (retrieved with Collections()). This is why the type is "unknown" for Collections(),
but not for Collection("name")
To select the document ID from a ref, add "id" to the "Path" of the additional column. For example, if "Path"
is ["data", "parent"], change the "Path" to ["data", "parent", "id"].
To select the collection name, add "collection", "id" to the "Path" of the additional column. For example, if
"Path" is ["data", "parent"], change the "Path" to ["data", "parent", "collection", "id"]. Internally, the
FQL Select is used.