What happened when I started treating Power BI reports like code

Summary

I started building Power BI reports with AI agents working directly on the project files. It made me much faster. It also forced me to answer a question I didn't have before:

How much autonomy can I give an AI agent before I need stronger engineering controls?

This case is about how my way of working changed: what development gained by going through the agent, what new kind of risk showed up, and which controls I had to add as the agent made bigger changes.

The shift: from a GUI to a project

For years a Power BI report was a .pbix file: a binary edited by clicking in Power BI Desktop. With the project format (PBIP), the model becomes TMDL and the report becomes PBIR: folders of plain-text files you can read, diff and version.

That changed more than the format. An AI agent that works with files can now add a visual, change a filter, reorganize a page or refactor measures. The unit of work stopped being a GUI and became a project.

There are tools that give an agent ways to interact with Power BI, and they're useful. For the way I work, something else mattered more: a versioned artifact the agent edits like code. The two ideas complement each other.

The problem: faster at everything, including breaking things

If an agent can change a whole report in seconds, it can break it in seconds too. And the dangerous mistakes aren't the ones that stop the project from opening: they're the ones that leave a number that looks fine and is wrong.

A concrete example. The agent changes a card's filter so it covers two funnel stages:

  "filter": { "In": {
    "Expressions": [ Stages[Order] ],
-   "Values": [ [ 3 ] ]
+   "Values": [ [ 2 ], [ 3 ] ]
  } }

And the measure that card shows, in another folder of the project, resolves the row with MAX:

Deals at stage =
    VAR _order = MAX ( Stages[Order] )
    RETURN
        CALCULATE ( COUNTROWS ( Deals ), Stages[Order] = _order )

Each piece is valid on its own. Together, the filter admits stages 2 and 3, but the measure uses MAX and ends up resolving stage 3 only: the value shown may not represent the context the visual appears to have. Nothing is malformed, so no structural validator has any reason to flag it.

This isn't a lab case. In a real report, a visual produced an impossible conversion rate above 100%. The underlying measure was valid; the problem was the filter context passed by the visual. The example above is its synthetic equivalent, and anyone can reproduce it from the repo.

The filter lives in the report and the measure in the model. Nothing was checking the relationship between the two.

Valid is not the same as correct

What sorted it out for me was separating three different questions, which need three different kinds of check. The first one already had tools. For the second I couldn't find any, and that's where pbip-guardrails ended up coming from.

The guardrails around the agent

With that clear, the workflow ended up like this. It's not an end-to-end automated pipeline: some steps are done by a tool and others are decided by a person, and the diagram tells them apart.

1. A known Git state. I didn't start with this process: it emerged as the agent made bigger changes. Now, before letting an agent make substantial changes, I want the project in a known Git state. That gives me a readable diff and a cheap rollback path.
2. pbip lint. Runs Microsoft's validators and the coherence checks. If something fails, the change goes back to the agent with the finding, or gets rolled back.
3. safe-copy. The report is first published as a copy under another name (ZZ-…). Since publishing matches items by name, the copy is created as a new item and never overwrites the real one.
4. Look at it. There's no Power BI renderer for Linux: the only proof a visual looks right is opening it. Only then is the real one published.

What each tool catches

I built a synthetic report with defects planted on purpose and ran it through every layer. The point isn't to show that one tool "wins": different tools answer different questions.

The last two rows show how they complement each other. The PBIR validator flags the visual outside the canvas; the TMDL parser catches any syntax error, while my check only recognizes one specific pattern that had once broken a whole project. This is what the finding from the example above looks like:

$ pbip lint examples/broken --family 'Segment (\d+)'
  'Pipeline' [filters] cardVisual b0000000000000000001: Order IN ['2L', '3L']
      -> ['Deals at stage'] depend(s) on the row's Order: the value shown may not match the filter

I didn't trust the AI

An agent wrote a good part of pbip-guardrails's code, quickly. That leaves the question any engineer would ask: how do I know it didn't invent a solution that only looks like it works? That's why nothing was accepted on its own: every step was checked against something independent.

1. AI-generated code
2. ↓→ Known fixture
3. ↓→ Tests
4. ↓→ Official tools
5. ↓→ Parity with the previous version
6. ↓→ Real-world read-only validation

1. A case with known answers. A script generates a healthy report and a broken one, with a defect planted for every kind of check.
2. Automated tests. The healthy one must give zero findings; the broken one, exactly the ones planted.
3. The official tools as a reference. Microsoft's parser and CLI run on the same case: the healthy one passes, the broken TMDL fails where it should.
4. Parity with the previous version. The checks started as an internal script. The generalized version had to return exactly the same findings on several real PBIP projects, read-only.
5. Fixes that came from looking, not assuming. The first "healthy" report the agent generated had a real TMDL error, and the linter itself caught it. I kept that as a lesson: the fixture gets validated too.

Limits

• Passing the linter doesn't guarantee the report renders. A query can fail under some filter, or a permission can be missing. That's why the ZZ- copy exists.
• Level 3 is still human. No tool knows what "active customer" means in your business.
• The checks are heuristic and tuned not to be noisy. Each rule discounts the known legitimate cases; if you get hundreds of findings, the threshold is wrong, not the report.

What I learned

The code

pbip-guardrails is open source (MIT): the coherence checks, the integration with Microsoft's validators, the safe copy for publishing, a synthetic sample to try it on, and a CI example.

View pbip-guardrails on GitHub →