Troubleshooting App Installs & Runs

A failed variant run almost always comes down to one of a dozen known causes, and most of them you can identify (and often fix) in a few minutes. This guide walks through it in order: get the error, check the things your team owns, check what the sync engine thinks, check what Snowflake actually says, then report with everything an engineer needs.

The one rule: capture the exact error text before anything else. A failed status with no error message can't be triaged by anyone.

The second rule, learned the hard way: the sync engine's state is what the platform believes. Snowflake is what is true. When a run fails on a permission error and the sync engine says everything is fine, the sync engine is the thing that's wrong. Step 4 exists for exactly this case.

Step 1: Get the error message

There's no screen in Insight Cloud that shows a variant's error yet, but it's two clicks away:

1. Open #ic-app-run-failures in Slack.

2. Find your failed run and click the email for details.

An alternative way, from within the platform:

1. Open the variant's page in Insight Cloud.

2. Open your browser's developer tools (F12 or right-click, then Inspect) and go to the Network tab. Click the Fetch/XHR filter.

3. The page refreshes an item called settings every 30 seconds. Click the newest one.

4. In the response, find the run history and scroll to the most recent run. The error message is at the end of that line; skip past the stack trace.

Copy the whole error message. You'll use it in every step below. While you're in that response, note two more things:

• The app version the org is running (the procedure name carries it, e.g. ..._DAILY_10). Different versions of the same app read different tables, so two orgs on different versions need different grants and can fail differently.

• Which table the error names. Everything below depends on it.

Step 2: Check the things D&A owns

Do these checks before reporting anything. They resolve the majority of failures.

Check A: Permissions on the source table

If the error mentions a table that "does not exist or is not authorized," open the source table in the Snowflake database explorer.

There are three separate layers here, and all three have to be right. Most dead-end diagnosis comes from checking one and assuming the others.

Layer 1 lets the platform re-grant things. Layers 2 and 3 are what actually let a run read the table, and layer 2 is invisible to every platform tool. A run fails if any one of the three is missing, and the error text is identical in all three cases.

Proxy roles are named after the table: <DATABASE>__<SCHEMA>__<TABLE>__ROLE.

Remember: when a pipeline rebuilds a table, Snowflake silently drops the permissions on it. If your pipeline didn't re-apply them, this is your failure. Re-grant, then see Step 5 to retry.

About SEEK_INSIGHTS_ADMIN: it is not a role to ignore. It owns some shared tables and proxy roles, and grants made by hand are often made through it. If you see it in a privileges list, read it rather than skipping past it; it may be the only thing holding a permission together.

Check B: Timing

Did the run start before your pipeline finished loading the source data? If scheduled runs fail overnight but manual reruns work in the morning, the fix is the schedule, not the permissions. Adjust the variant schedule to start after the pipeline completes.

Check C: Is anything about this app custom?

This is a real branch, not a footnote. If the app touches a hand-built setup, the standard flow below will not find your answer and the sync engine cannot help you.

Signs you are in this case:

• The failing table lives in a database the sync engine does not manage. Today that includes DOORDASH_SEEK.SHARED_MODEL_RUNS, a shared table several orgs both read and write.

• The app both reads and writes the same table.

• Roles or grants for it were applied by hand.

• There is a side pipeline or Prefect flow involved.

If any of those apply, go straight to Step 4 and check Snowflake directly. There will be no grant record to retry, or the record will exist and be wrong. Then include "this app uses a custom setup" in your report, and name the person who maintains it.

Check D: Was a role recreated recently?

⚠️ CREATE OR REPLACE ROLE drops the role and destroys every grant to it and from it. It reads like "make sure this exists." It is not that.

If any proxy role was rebuilt with CREATE OR REPLACE ROLE, then every app runner that held it silently lost access, and the sync engine still reports those grants as healthy. This has caused multi-day outages.

Use CREATE ROLE IF NOT EXISTS instead. It leaves an existing role and its grants alone.

Step 3: Check what the sync engine thinks

Open the SF Grant to Role State table through the Sigma prod replica connection:

1. Filter to your org ID first (grab it from the global Organizations list); the table covers every org.

2. Filter the state column to failed.

3. Look at who is being granted what, and read the error text (it repeats once per retry; the first one is enough).

How to read what you find:

• App runner role failures (..._F_APPS_RUNNER): this is why your run failed. Usually the app runner is missing usage on a source table's proxy role, often with an error like "role didn't exist at the time the grant was attempted." These often fix with a retry (Step 5).

• Reader role failures (names containing a version number before READER, like ..._APP__9__READER): your run is fine; this breaks viewing the dashboard. Report it, but as a viewing issue, not a run failure.

• A failure that names a table or role that no longer exists: usually a stale record from an app version that is gone. Retry will never clear it. Report it for cleanup rather than retrying repeatedly.

• Nothing failed at all: do not stop here. This does not mean the grants are healthy. It means the engine believes they are healthy. Go to Step 4.

Step 4: Check what Snowflake actually says

Do this whenever the run fails on a permission error and Step 3 showed nothing wrong. The sync engine records a grant as SYNCED at the moment it succeeds and does not re-check it afterwards. If the grant is later destroyed (most commonly by a role being recreated), the record stays green forever and no platform tool will tell you.

Three read-only commands in a Snowflake worksheet. SHOW commands change nothing and need no warehouse.

4a. Does the proxy role have SELECT on the table, and who owns the table?

SHOW GRANTS ON TABLE <database>.<schema>.<table>; 

Look for a row where privilege = SELECT and grantee_name is the table's proxy role. Also read the OWNERSHIP row.

4b. When was the proxy role created, and how many roles hold it?

SHOW ROLES LIKE '<DATABASE>__<SCHEMA>__%'; 

created_on tells you if the role was rebuilt recently. granted_to_roles is a count; if one role in a set has a lower count than its siblings, an org is missing a grant.

4c. Which orgs actually hold the proxy role?

SHOW GRANTS OF ROLE <DATABASE>__<SCHEMA>__<TABLE>__ROLE; 

One row per app runner that holds it. Compare against the orgs that have the app installed. A missing org is your failure, regardless of what Step 3 said.

If you need to know who changed something, this needs a warehouse (pick a small one, not an app warehouse):

SELECT START_TIME, USER_NAME, ROLE_NAME, QUERY_TYPE, EXECUTION_STATUS, LEFT(QUERY_TEXT, 300) AS qry FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY WHERE QUERY_TEXT ILIKE '%<role or table name>%' AND START_TIME >= DATEADD(day, -14, CURRENT_TIMESTAMP()) ORDER BY START_TIME; 

This is the full history of every create, drop, and grant touching that object, with the user who ran it. It lags about 45 minutes and keeps 365 days.

Note: app runner roles are assigned to org service users, not to people, so you cannot USE ROLE one to test a read yourself. Confirming the grants exist plus an actual rerun is the test.

Step 5: Retry, the right way

Read the red before you act on it. An org's red badge counts every failed resource in that org, for every app. It does not mean your app is affected, and it does not mean your app will fail.

Before retrying:

1. Check Step 3 for what is actually failing in that org. If every failed row is a READER role, or targets a different app's tables, your run failure is somewhere else and retrying will not help.

2. If the failures are stale records naming things that no longer exist, retrying will send them straight back to red. That is expected, not a new problem.

3. If the failing item is your app's app-runner grant, retry is the right move.

The sequence that works:

1. Click the org node, then Retry failed resources in the side panel.

2. It stays red until the engine gets to the retry, then turns yellow (TRANSIENT). If the page doesn't seem to update, click Refresh at the top.

3. Yellow is your cue to rerun the variant. The run itself completes the remaining work; if it succeeds, the item turns green (SYNCED). Don't wait for green before running; green comes after a successful run.

4. Confirm in Step 3 that your specific grant flipped to SYNCED. Do not use the badge colour as your confirmation. An org can stay red because of unrelated items while your grant is now fine.

Two more rules:

1. One retry. If it goes red again with the same error, stop; the underlying cause isn't fixed. Go back to Step 2, or report it (Step 6).

2. Don't retry into a backlog. If the visualizer shows a large queue of pending items, wait for it to drain first.

And the case retry cannot fix: if the sync engine says SYNCED and Snowflake says the grant is missing, there is nothing to retry. Nothing is red. The grant has to be re-made by hand, and it should be reported, because the drift itself is the bug.

Full button-by-button guidance is in Using the Sync Engine Visualizer.

Step 6: Report it

Use Contact Us in Insight Cloud for every failure, even ones you fixed yourself, and even if you think someone already reported it. Duplicates are welcome: frequency is how fixes get prioritized.

Include these six things:

1. The exact error text (from Step 1)

2. Org, app, variant name, and the app version

3. What you found in Steps 2 through 4 (which of the three permission layers was wrong? anything custom? any role recreated?)

4. Whether a rerun/retry worked

5. Rough time it happened, and whether it was a scheduled or manual run

6. Whether the sync engine and Snowflake agreed. If the engine said SYNCED and the grant was missing, say so explicitly. That is a platform bug and it needs to be counted separately from a normal permission failure.

Quick reference: error to most likely cause

Related articles

• Using the Sync Engine Visualizer: reading the graph, every button explained, and what the tool cannot tell you

• The Sync Engine: what SYNCED / FAILED actually mean

• App & Data Lifecycle: the full journey from raw data to dashboard

• Known Issues & How to Report a Problem: current open issues

• Model Reuse: Freshness and Shared Models: the proper way to share data across apps