Azure DevOps to Helpjuice: automating release note production

Problem

Release notes sit at an awkward intersection: structured enough that the process is always the same, but just complex enough that automating it never quite makes it onto the backlog. So they stay manual. Someone opens the board, reads through what shipped, writes it up, publishes it. Every release, every product, every cycle.

The work isn’t difficult. That’s part of the problem. It’s the kind of task where the time cost is real but the individual steps feel too small to justify a proper fix — pull the same fields, apply the same format, publish to the same place. It accumulates quietly until it’s taking up a meaningful chunk of someone’s week.

There’s plenty of reasons to automate the process.

I designed the solution in three parts:

1. Data extraction

The starting point was a saved query in Azure DevOps, filtered to the previous iteration, scoped to the work item types that produce release notes, and tagged RN APPLIES to exclude items that shipped but didn’t need documenting.

SELECT
    [System.Id],
    [System.WorkItemType],
    [System.Title],
    [System.AssignedTo],
    [System.State],
    [System.Tags]
FROM WorkItems
WHERE
    [System.IterationPath] = @CurrentIteration - 1
    AND [System.WorkItemType] IN ('Product Backlog Item', 'Bug')
    AND [System.Tags] CONTAINS 'RN APPLIES'
ORDER BY [System.Id]

2. Publishing trigger (Power Automate)

I use Power Automate to automatically extract the result set of the Azure query, write it to an Excel file, and send it via email.

For products with a fixed release cadence, I know when the board “closes” and at what time I can start producing the release notes. Take a product whose sprint review happens biweekly on Fridays and ends at 13:00. The Power Automate flow looks like this:

[1] Recurrence         — triggers biweekly, Fridays at 13:00
     │
[2] Get query results  — runs the saved Azure DevOps WIQL query
     │
[3] Run script         — Office Script writes query results to Excel
     │
[4] Apply to each      — iterates over each work item
     │
[5] Send an email      — delivers the Excel file to the team

3. Helpjuice API integration

A Python script is the core of the pipeline. It handles three things in sequence: reading the Azure DevOps export, working out where the new article should land in the knowledge base, and publishing it.

The input is an Excel file with the Azure DevOps query results exported by Power Automate. I used Pandas to process it: filter out work items where RN Description is empty (items that shipped but don’t need documenting), and pull the columns that matter: ID, work item type, title, and the description field itself. RN Description is a custom field the product team populates directly in Azure DevOps. It contains the user story context and a plain-language description of what changed. That’s the content the release note is built from.

Versioning is handled through the Helpjuice API rather than tracked externally. Before publishing, the script queries the release notes category, reads the existing article titles, extracts the latest release number by pattern, and increments it. The new article inherits the next version in sequence without anyone maintaining a separate reference.

The article itself is built from an embedded HTML template. The script iterates over the processed work items and injects each one (type label, title, and RN Description content) into an accordion block. The full set gets wrapped in the article shell and published via the Helpjuice REST API, either as a new article or as an update if one already exists for that release.

The output at this point is a complete, structured article. In practice it still gets a manual pass before going live, because RN Description quality varies depending on who wrote it and when.

The script removes the mechanical work; the editorial step at the end stays human.

Outcome

The pipeline is live for several products, publishing directly to the internal Helpjuice knowledge base. It isn’t perfect. RN Description quality depends on the engineers writing it, and the manual review step before publication means it hasn’t fully removed human time from the process, just repositioned it. The time now goes on judgment, not mechanics.

The broader effect was organisational. Having a working pipeline made it easier to argue for the process changes that make automation sustainable: requiring RN Description on relevant work items, defining what “release-ready” means before the sprint closes, establishing who reviews before publish. The automation created leverage for governance conversations that hadn’t been possible before.

What I’d do differently

The current pipeline is reactive. It runs when Power Automate fires, pulls whatever was in the query, and publishes. It has no awareness of content quality.

The version I’d build now would add an LLM-based validation layer between extraction and publication. Before the article goes to Helpjuice, each RN Description would be evaluated against a small rubric: is it written for an end user or an engineer, does it describe the change or just reference a ticket number, is the scope clear. Not to rewrite the content automatically (that introduces a different class of errors) but to flag items that are likely to need rework and surface them to the reviewer before they’re embedded in a published article.

The rubric itself would be versioned and stored alongside the pipeline code. That matters because “good release note” means different things across product lines and audiences, and the evaluation criteria should reflect that explicitly rather than encoding it as reviewer intuition.

The other change worth making: replacing the Power Automate trigger with an Azure DevOps webhook and a lightweight API endpoint. The current flow is difficult to test in isolation because the trigger logic lives in a GUI tool with no version history. A webhook handler in Python is a function. It can be tested, logged, and deployed the same way as the rest of the pipeline.