IBM i Project Builds

A project orchestrator runs many child tasks. Each child compiles into its own task library, so when children are accepted their source comes together on the project branch, but their compiled objects stay in separate libraries. A project build brings them together: one library holding the project's accepted work, compiled from one exact commit of the project branch, with the normal libraries below it on the library list. People test the combined work in it, and a phase isn't approved without one. Optionally, later children and reviewers build over it.

Project builds are off until you turn them on for an environment's IBM i Build connection. While they are off, orchestrators, their test links and their approvals work as before, and agents aren't offered the project build tools.

Turning project builds on

Edit the environment's IBM i connection with the Build feature, available for Tasks, and turn on Project Builds in its Build Settings. See Project builds settings for Code Libraries, Storage Limit (%), Project Build Check and Build Children and Reviewers Over It.

Project builds need:

• A library list, either the connection's Default list or one chosen when the orchestrator is launched. The build's own library goes in front of it.
• codermake 4.4.0 or later in the environment's image, whose SQL check stops a script from creating its object outside the build's library, and an image with the project build script. Rebuild the base image and the environment's image to get both.
• An orchestrator whose project branch exists, which happens once the first child is approved into it, and whose container hasn't been removed.

If the environment's settings can't be read (an environment.json that can't be read or parsed), CoderFlow doesn't take that for off: approvals that may need a build are refused with project_build_unverified, and sessions on an opened build are refused with the reason, never started on the task library instead.

Turning Project Builds off again closes any opened build: the orchestrator's test link sends its own library again, as before, and pages loaded for a project build's library are still refused. Approvals no longer need a build. Existing builds' libraries are dropped as described in Clean-up.

The IBM i user profile the connection uses builds every project build. Give it no write authority to the normal or production libraries: codermake checks SQL qualified names, but a build should not be able to change those libraries in any other way either.

Building the project

A project build starts when:

• the orchestrator calls request_project_build, usually after accepted children land and before an approval checkpoint; or
• a person chooses Build project in the Project builds section of the orchestrator task's Task Details.

Without a commit, the project branch's tip is built. Only one build runs per project at a time. A request made while one runs waits, and a newer request replaces an older waiting one, so an older request never builds after a newer one. A build CoderFlow lost track of (cut off by a server restart, or past its time limit) counts as running until CoderFlow has confirmed it has ended in the orchestrator's container.

The build runs in the orchestrator's container, which is started if it is stopped and isn't stopped for being idle while the build runs. It works in a separate checkout, so the orchestrator's own work is never touched:

1. CoderFlow checks the IBM i system ASP against the Storage Limit (%), and reads a fingerprint of the normal code libraries.
2. The script checks out the baseline, the commit the project branch started from on its base branch, and marks every target there as built.
3. It checks out the commit being built and creates a new library, registered before it is created, like a task library, so it is dropped once the orchestrator's container is removed at the latest (or, for one given to a child or reviewer, once the last of their containers is).
4. codermake compiles everything that changed between the baseline and the commit, and what depends on it, into the new library. Everything else is found in the normal libraries below it.
5. It creates a data area, CFPRJBLD, holding the commit, and sets the library's text to CoderFlow project build <number> @ <commit>.
6. It runs the Project Build Check, when one is set.

A build is published only if it compiles, the check passes, and the normal libraries did not change while it ran. If they changed, the build fails and says which library changed; request a build again once the libraries are settled. Once published, a build never changes. A failed build is dropped and never replaces the last good one.

A build that runs longer than four hours is stopped and fails, so a hung compile or connection doesn't keep the orchestrator's container from its idle stop. Stopping a build also stops everything it started, such as a compiler, before its library is dropped, including a process that left the build's process group: each one carries the build's run marker.

Its log, /task-output/project-builds/<number>/build.log, and its compile listings, under logs/ beside it, stay in the orchestrator's task output; Log shows the end of the log.

Programs a project deletes

When the commit deletes a program or file that the baseline built and a normal library still holds, a library in front of the normal libraries can't hide it: the old object would still be found. With Code Libraries set, the build is then a full build: every target, with the code libraries left off the library list. Without them, the build fails and names the objects.

Knowing what you are running

Each build shows its number, commit, baseline, library, the objects it built and its status: Building, Published, Failed, Superseded, or Out of date. In a session on the IBM i, DSPDTAARA DTAARA(<library>/CFPRJBLD) shows the commit, and the library's text names it.

Testing a project build

Choose Open on a published build. CoderFlow checks it is still current, then sends the build's library in the orchestrator task's test link (the X-Agentic-Task-Lib header Profound UI reads) for the sessions started from then on. A session keeps the library it started with until it ends, so people can keep testing while a newer build is opened. Use task library sends the orchestrator's own library again.

While a build is open, a session starts only on it, and only once CoderFlow has checked it again:

• Each session start is checked: a Genie or Rich Display start page (/profoundui/genie, /profoundui/start and their /auth/ forms) or one of the environment's launch URLs, whatever its query string, and Genie's session initialization (PUI0002110.pgm with init=1) for the library its start page was loaded for.
• A stale build refuses sessions. If the build is out of date, or can't be checked because the IBM i doesn't answer, the session is refused with a message saying why, rather than started on another library, and no other request carries its library either. The test link stays set to that build, and keeps refusing sessions, until you open a current build or choose Use task library.
• A page loaded for another library is refused: one loaded before you opened the build, for a build you have since moved off, or for a build whose library was dropped. Reload the test link to start a session on what it is set to now.

When a newer build is published, the opened one is Superseded, but the test link stays on it: new sessions still start on it while it matches the IBM i, and the Project builds section says so. Open the newer build to move to it. A superseded build can't be opened again once you have moved off it.

A full build can't be opened. It is correct only with the code libraries off the library list, and the test link can only put one library in front of a session's list, not take libraries off it, so a session would still find the programs the project deleted. Its Project Build Check runs with the full build's list, and it serves approvals, and children and reviewers, which compile with its list.

Approvals

While project builds are on, approving a phase needs a published, current project build of the exact commit being approved:

• create_pull_request checks the tip of the branch the pull requests are opened from: the project branch, or the branch it's given;
• stage_project_landing checks the exact commit it stages, as fetched, before it changes the orchestrator's workspace;
• approving a staged orchestrator task checks the staged commit again.

A check that passes only while the build is being checked doesn't count: if a newer build supersedes it, or it is released, before the check ends, the approval is refused. Each is refused with project_build_required (or project_build_running, project_build_out_of_date, or project_build_unverified when the IBM i or the branch can't be read) until there is one. Repositories other than the Build connection's build repository need nothing.

When the normal libraries change

A deployment or a manual compile can change the normal libraries under a project build. Before a build is opened, before each session on it, before it is given to a child or a reviewer, and before each approval, CoderFlow compares the normal code libraries, and the build's own library, with the fingerprint taken when it was published: each object's name, type, attribute, creation time and size. The size of a file, data area, data queue or user space isn't compared, since it follows the data it holds. In the build's own library, the programs, files and other code objects and its CFPRJBLD data area are compared; its SQL journal and journal receivers, which grow and roll over as people test, are not. A build that no longer matches is Out of date for good: it isn't used again, and a new build of the project branch's tip is requested.

Children and reviewers

By default, a child's task library builds as described in What a new library builds: its first build compiles all the project's accepted work as well as its own change, so its sessions run the combined work.

With Build Children and Reviewers Over It on in the connection's Project builds settings, children and reviewers build over the project's newest usable build instead: the newest published one, checked against the IBM i as for any new use when the task's container is created. Turning it on affects tasks created afterwards. When there is no usable build, or the settings can't be read, the task builds as by default.

• A child launched on the project branch has the build's library next on its library list, after its own task library and before the normal libraries, and its first build compiles only what changed since the build's commit. The child's task log names the project build. The build stays on the child's list for its container's life: after a restart, and in a new library the clean-up moves it to. The clean-up only ever touches the child's own library.
• A child isn't built over the build when the build's commit isn't in its checkout's history (a shallow checkout is first deepened by up to 300 commits to find it), or when its checkout deletes files under the build directory since that commit, since their objects would still be found in the build's library. It then builds as by default, and its task log says why.
• A reviewer of a project's child gets a task library of its own, always with the same build next on its list. When its own checkout has the build's commit and deleted nothing since, as a reviewer's checkout of the project branch usually does, that checkout is marked from the build's commit too, as a child's is; otherwise it keeps its usual baseline. To compile the reviewed change itself instead of trusting the child's objects, the reviewer runs coderflow-ibmi-layer <checkout> with a checkout from /task-output/review-checkouts.json, then codermake in the build directory it prints. The helper empties the reviewer's library first, so the library holds one checkout's build at a time, then marks the checkout built as of the build's commit. It refuses a checkout without the build's commit, one that deletes files since it, and any library that isn't the reviewer's own, and it ends everything it started before it reports success. Reviewers are told how in their instructions.

A child or reviewer whose library can't be registered with CoderFlow gets none, rather than one nothing would drop; its task log says why, and it builds without a library of its own. A build given to a child or reviewer is kept until none of their containers is left, even after the orchestrator's container is removed, as described in Clean-up.

What that costs at run time. Profound UI takes one library for a session, so the agent's Genie sessions (genie_start.sh) and a person's test link get only the child's or reviewer's own library in front of the normal libraries, not the project build. Programs the task didn't rebuild then come from the normal libraries: the older copies, not the project's accepted work. Turn it on for projects whose children compile and check but don't test at run time, or test the combined work through the orchestrator's project builds. That's why it is off by default. CoderFlow has asked the Profound UI team for a session header that takes several libraries.

Clean-up

A superseded or out-of-date build's library is dropped once nothing can use it:

• the test link isn't set to it;
• no session was started on it in the last five minutes, until the session's job holds a lock on it;
• no child or reviewer whose container still exists, stopped or not, was given it (see Children and reviewers);
• no IBM i job holds a lock on the library or an object in it. Locks are checked on the IBM i the library was created on, over the connection it was registered with, even if the Build connection has since changed. While that can't be checked, the library waits.

CoderFlow checks after each build, when the test link changes, and every 10 minutes. A failed build never served a session, so its library is dropped at once. A build cut off by a server restart, or past its time limit, is stopped in the orchestrator's container first, then whatever library it registered is dropped; if the stop can't be confirmed, CoderFlow tries again every 10 minutes, and the next build waits until it is. Every remaining project build is dropped once the orchestrator's container is removed, except one given to a child or reviewer whose container still exists: that one is kept until the last such container is removed, then dropped within 10 minutes. Project build libraries appear with the kind ibmi-project-build on the environment's External Resources tab, for anyone with environments:manage there, and in Server Administration → External Resources.

Limits

• A child's or reviewer's sessions get one library. Over a project build, their test sessions still run the normal libraries' older copies of programs they didn't rebuild, as described in Children and reviewers.
• Data comes from the normal data libraries. A file or table the project changed is built empty into the project build. Programs that find files through the library list then open that empty copy.
• One library per test link. A full build can't be opened, as described above.
• The baseline is where the project started. CoderFlow doesn't yet know which commit the normal libraries hold, so it builds everything the project changed since it left its base branch. Base-branch changes the project never synced are left to the normal libraries.
• Nothing here has run on an IBM i yet. Treat the first pilot as a test of the feature.