The Sync Engine

Plain-language guide to the Sync Engine: declared vs observed, the four states, when it acts, and the blind spot behind install failures.

The mental model

The Sync Engine is a thermostat, not a script. It continuously compares two pictures and closes the gap: DECLARED (what should exist, computed from IC's database) and OBSERVED (what actually exists in Snowflake and Sigma). Any single run is disposable; if it crashes, the next run picks up from wherever reality actually is. So "did the sync job run?" is usually the wrong question. The right one is: does anything currently show as OBSERVED or FAILED?

What it manages

Ten kinds of objects: Snowflake warehouses, databases, schemas, roles, users, and two kinds of grants; plus Sigma connections, users, and workbook grants.

The four states

State	Meaning
SYNCED (green)	Declared and observed match. Nothing to do. Where most things live.
TRANSIENT (amber)	The engine knows there is a gap and is working on it. Normal and short-lived. New things start here.
OBSERVED (blue)	Reality drifted (for example, someone changed Snowflake by hand). The next run pulls it back.
FAILED (red)	Tried three times, gave up. Terminal. Carries the actual error messages. Un-parks only when the goal changes or someone resets it.

When it acts

There is no polling schedule. It is event driven. Any app change that could affect Snowflake or Sigma drops a flag in the same database transaction. After each request and background task, the system checks for flags and starts a run. A monitor runs every minute as a backstop. Changes usually propagate within seconds.

Only one run happens at a time system-wide. That is today's main limitation: one org's slow reconcile can delay everyone else's. That is the queue delay seen in #ic-app-run-failures.

The four kinds of runs

Reconcile: the normal, continuous, flag-driven run. 99% of activity.
Snapshot refresh: throw away everything the engine believes and re-discover it from Snowflake and Sigma. Recovery operation, expensive.
Reset failed: move FAILED objects back to TRANSIENT and try again. The "we fixed the underlying problem" button.
Ad hoc sync: a manually scoped run for specific orgs.

The blind spot to know about

The engine verifies its own changes and re-checks an org when that org is flagged. But a SYNCED object in a quiet org is not re-observed on a schedule. So if a proxy role breaks externally (for example, a pipeline rebuild drops a grant), the engine can keep showing SYNCED until something triggers a re-observation. This means an externally broken permission can look healthy until something triggers a re-observation. When in doubt, trigger a re-observation.

The first-open timing race

Because sync triggers when access changes, brand new access plus an immediate open can beat the engine while the queue is still working. This is the "errors on first open" pattern, and is likely related to the embed-path 409. Setting up access ahead of time avoids it.

App Install and Run Troubleshooting (D&A)
How Insight Cloud Apps Fit Together

Source: "Sync Engine" guide on the engineering hub (systems/insight-cloud/sync-engine).