AIStore 5.1 is a short-cycle reliability and scalability release following the major 5.0 transition. It closes the two-release security rollout opened by 5.0: the bridge gate is removed, per-node Ed25519 request signing and startup node-join authentication become active when configured, and pre-5.0 nodes are refused.

The main themes are:

1. Intra-cluster security. The cryptographic mechanisms staged in 5.0 can now be enabled. Nodes publish verifying keys, protected requests and transport-stream setup are signed, configured startup join credentials are enforced, and the 5.0 version boundary is checked in both join directions.

2. List-objects scale-out. Listing very large buckets in large clusters no longer falls on the primary gateway. Listings are distributed across all gateways - consistent with the way the AIS datapath has operated since inception.

3. Shard index v2 and on-demand cache. A new ordered, packed representation removes the decoded per-entry map, substantially reducing allocation and load cost. A new memory-aware cache is aimed primarily at Get-Batch over large TAR datasets, where many member reads repeatedly revisit the same shards. End-to-end cache performance characterization is still pending.

4. Blob download and prefetch. Large remote-object downloads now have explicit size and memory admission rules, parent-to-child cancellation, and cleanup of buffers, manifests, and object locks on failure.

5. Reliability under failure. Erasure coding completes every destination when one of them fails and rebuilds peer streams after node restarts; global rebalance continues after per-object failures.

6. kTLS transmit offload (experimental). An opt-in path, for controlled evaluation, that lets HTTPS GET transfers (i.e., AIS transmit) use sendfile(2) while the kernel - or a capable NIC - encrypts TLS records. Large-scale benchmarks are planned.

7. Python SDK 2.0, released alongside 5.1. Interrupted ObjectFile reads now resume without gaps or duplicate bytes even when the next response starts at byte zero rather than the requested offset. The SDK also improves parallel reads, batch extraction, ETL, and PyTorch integration. The new major version reflects its compatibility changes: the minimum Python version is now 3.10, and several previously announced API removals take effect.

Separately, 5.1 sets the objective for chunked storage - a third storage layout that, like n-way mirroring and erasure coding, is governed by bucket configuration - and deprecates the per-job overrides that contradict it. Implementation is planned for v5.2.

Upgrade boundary: v5.0 remains the mandatory bridge from every 4.x release. Upgrade your 4.x cluster to 5.0 before starting 5.1; direct 4.x-to-5.1 upgrades are rejected. See Upgrade Notes below for other compatibility changes.

Table of Contents

1. Security and Upgrade Completion
2. List Objects
3. S3 Compatibility
4. Object Metadata and Response Headers
5. Shard Index v2 and On-Demand Cache
6. Erasure Coding
7. Global Rebalance
8. Chunked Storage
9. Blob Downloader and Prefetch
10. Internet Downloader
11. Jobs
12. Memory Pressure
13. kTLS Transmit Offload (experimental)
14. Socket Buffers
15. Native Go API
16. Python SDK, ETL, and ML
17. Miscellaneous Improvements and Fixes
18. Documentation
19. Removed and Deprecated APIs
20. Upgrade Notes
21. Acknowledgments

Security and Upgrade Completion

v5.0 established the new network and metadata boundaries but deliberately suppressed the cryptographic mechanisms while 4.x nodes could still be present in the cluster. v5.1 removes that bridge behavior and completes the transition.

Per-node request authentication

Every 5.1 node generates an Ed25519 keypair at startup and publishes its verifying key through the cluster map. When auth.intra_cluster.request_auth is enabled, AIS signs and verifies protected proxy-to-target redirects, node-to-node control requests, and transport-stream connection setup.

The signing key remains in memory only. Restarting a node rotates its key; cluster-map propagation distributes the replacement. Transport bundles treat a changed verifying key, network endpoint, or membership state as a new peer incarnation and reopen the affected streams.

The setting remains opt-in. With request_auth disabled, the arrival-network checks and proxy-mediation rules introduced in 5.0 still apply: when auth.client_auth_required is enabled, targets reject unmarked requests sent directly to their public endpoints. An unsigned redirect marker can, however, be supplied by a client and is not authenticated at the receiving target. auth.client_auth_required protects requests at proxies; it does not independently authenticate redirected requests at targets.

Network path and identity: AIS records each request's arrival network and enforces the corresponding routing checks even when signing is disabled. This establishes where a request entered, but cannot prove that a public redirect marker came from a gateway. To keep client authentication effective across redirects, enable both auth.client_auth_required and auth.intra_cluster.request_auth after completing the upgrade.

Enabling or disabling request_auth requires no cluster or node restart, and no downtime of any kind. Nodes sign immediately when it is enabled and accept unsigned traffic for a bounded convergence window before becoming strict. The same mechanism provides a short startup window while verifying keys converge after a full-cluster restart; it is not intended to make a long-running 5.0/5.1 mixture secure.

Node-join authentication

When auth.intra_cluster.node_join_secret_path is configured, 5.1 reads the locally provisioned secret and uses a timestamped HMAC proof to authenticate startup self-join in both directions:

• the primary verifies the joining node before admitting its identity and public key;
• the joining node verifies the primary before accepting the returned cluster metadata.

The secret itself is never transmitted or persisted in AIS metadata. The configured pathname is cluster-wide, while the file must be provisioned locally on every proxy and target with restrictive permissions. Slow keepalive paths that would admit an absent or restarted node use the same proof.

The secret is read once during node startup and is not reloaded by a cluster-configuration update. Setting or changing the path persists the new configuration but does not affect running processes until they restart.

Administrative-join and force-join authentication remain outside the implemented boundary. See Node-Join Authentication for the exact status, provisioning order, credential requirements, and recovery procedures.

Version boundary

Joining nodes and primaries exchange their respective AIS software versions. A 5.1 node refuses a pre-5.0 primary, and a 5.1 primary refuses a pre-5.0 node through both self-join and administrative join. Missing version headers are treated as pre-5.0.

This closes the two-phase sequence introduced by 5.0: the bridge release accepted 4.x peers long enough to cross the metadata boundary; 5.1 depends on that boundary and fails a skipped-bridge upgrade with a specific compatibility error.

Additional enforcement

Proxy mediation now covers node-control queries as well. Previously, a target would answer requests for its configuration, status, and runtime information sent directly to its public endpoint - even on clusters that require client authentication, which only gateways enforce. Such requests are now rejected, the same way as direct bucket and object requests. Node control goes through a gateway, which authorizes the request before relaying it to the target over the intra-cluster network.

Two related changes concern the control plane itself. A control operation now uses one cluster-map snapshot from sender validation through downstream dispatch, so identity decisions within one request cannot straddle map versions. And relayed requests clear system-controlled intra-cluster headers before AIS constructs the next hop.

For the complete request matrix and the distinction between proxy mediation and data-plane authentication, see AuthN: Authentication Boundaries.

Object names and append handles

Both fixes below originated in reports from outside the project (see Acknowledgments).

Object names are now validated wherever AIS turns them into on-disk paths, not only at the gateway: copy and transform destinations (single- and multi-object), rename, promote, archive destination shards, list- and range-based operations such as prefetch, evict, and delete, names returned by a remote listing, and blob download. Names containing path-traversal components are rejected before they can reach the filesystem.

Append handles are bound to the object that created them. A handle carries only a validated tie-breaker for the object's work file; the target derives the work file...

AIStore 5.0 is a major release. It changes deployment requirements and persisted cluster metadata, replaces the intra-cluster trust model, restructures cluster configuration, and is a mandatory bridge for upgrades - a 4.x cluster must pass through 5.0 on the way to 5.1 and later.

Versions 4.8 and 4.9 were maintenance releases on the 4.7 line. They carried selected fixes from ongoing development rather than successive snapshots of the main branch. Version 5.0 is the next full main-trunk release and contains all development since 4.7, including fixes delivered early in 4.8 and 4.9.

The main themes are:

1. Intra-cluster security. Through v4.x, the optional intra-cluster authentication path used a shared cluster key and HMAC: one secret held identically by every node, with no way to tell which node signed what. In parallel, AIS permitted a collapsed network configuration in which the public listener and the intra-cluster listeners were the same listener - so a node's only evidence about where a request came from was the request itself. v5.0 requires a distinct public listener, removes the shared-key mechanism, and stages its per-node Ed25519 replacement. Signing remains disabled for the 5.0 bridge and is enabled after the cluster crosses it.

2. Cluster configuration. Many sections are now optional in the persisted representation, and sections that contain only defaults are no longer written out in full. This is a metadata-format change, not merely a smaller encoding: it makes operator overrides easier to distinguish, corrects several long-standing zero-value behaviors, and makes persisted 5.0 configuration incompatible with 4.x.

3. Node lifecycle. Maintenance entry and exit, shutdown, and decommission gain a batch form; administrative membership admission is serialized; rebalance conflicts consistently return ErrBusy; and the state a cluster lands in when a maintenance transition does not complete is now named, documented, and retryable.

4. Control-plane robustness. Heavy job churn, stream cleanup, keepalive work, job startup, and shutdown could interfere with one another in ways that delayed cluster coordination. Those paths are now separated so that activity in one does not stall the rest of the control plane.

5. Security, reliability, and correctness. This release closes a presigned-S3 host-hijacking path, a use-after-free in the memory subsystem, a nil-deref that turned sustained backend throttling into a cluster-wide crash loop, and a concurrency defect that could return one chunk's bytes in place of another's - alongside a systematic pass over object-name, archive, and header validation.

Other highlights include ranged get-batch reads, opt-in ETL builds with a destination-less inspect dry-run, and updates across the CLI, Python SDK, shard index, and Kubernetes deployments.

This release preserves backward compatibility for upgrades from v4.x, including rolling upgrades.
Version 5.0 is, however, the mandatory bridge to v5.1 and later: clusters must upgrade to 5.0 first rather than jump directly from 4.x to 5.1 and later versions.
As always, downgrades are not supported. Optionally, back up cluster metadata and see section Upgrade Notes (below) for details.

Table of Contents

1. Intra-Cluster Security
2. Cluster Configuration
3. Node Lifecycle and Membership
4. Control-Plane Robustness
5. Security, Reliability, and Correctness
6. Get-Batch
7. AuthN and Access Control
8. TLS
9. S3 Compatibility
10. Remote Backends and Rate Limiting
11. ETL
12. Storage, Metadata, and APIs
13. CLI, Python SDK, Shard Index, and Kubernetes
14. Documentation
15. Upgrade Notes
16. Acknowledgments

Intra-Cluster Security

For intra-cluster security, 5.0 provides network separation, per-node Ed25519 identity, and node-join authentication. A request is checked first by arrival network and then by sender identity; the second check depends on the first.

The network boundary and proxy-mediation policy take effect in 5.0. The cryptographic parts - Ed25519 request signing and node-join proofs - are staged behind the bridge gate for 5.1.

The three logical networks

AIS separates traffic across three logical roles: the public network for user-facing API, CLI, SDK, metrics, and external tooling; the intra-cluster control network for health, keepalives, cluster metadata, membership, and coordination; and the intra-cluster data network for object movement between nodes.

These are roles, not necessarily separate physical networks. The data role carries both ordinary target-to-target requests and long-lived data streams; both follow the same network boundary and, after the bridge release, both are authenticated.

The separation rule

Through v4.x, all three roles could collapse onto a single HTTP listener. Starting with 5.0, at least one intra-cluster endpoint is required, and it must be separate from the public endpoint. Intra-control and intra-data may share a listener, with a warning. Public multi-homing remains supported, but each intra-cluster endpoint must use a single IP address or host. Partial configuration - a hostname without its corresponding port - is rejected.

"host_net": {
    "hostname": "ais-pub.example.net",
    "hostname_intra_control": "ais-intra.example.net",
    "hostname_intra_data": "ais-intra.example.net",
    "port": "51081",
    "port_intra_control": "51081",
    "port_intra_data": "51081"
}

The rule is about HTTP endpoint separation, not physical NIC separation. A deployment may use fewer physical networks than logical roles as long as the public listener is distinct. Physical isolation remains strongly recommended for production - bulk data movement fills NIC, TCP, and application queues and causes head-of-line blocking for latency-sensitive control messages - but that is a performance argument, not what this check enforces.

A v4.x deployment with a collapsed host_net will not start on 5.0.

When the data network is not separately configured, it now defaults to the control network. In v4.x it defaulted to public, so deployments that omitted intra-data configuration moved object data over the public listener.

Arrival network

Every accepted connection is now identified by the listener that received it: public, intra-control, or intra-data. That identity cannot be supplied or changed by request headers or by a caller-selected address.

AIS checks the arrival network before it considers claimed node identity. A public request therefore cannot become an intra-cluster request merely by presenting internal headers. When control and data share a listener, both roles remain available there; the public listener never participates in that fallback.

Internal control operations are no longer exposed on the public listener. Daemon and health requests received there expose only user-facing behavior. Public requests that claim an existing node identity are rejected, while self-join requests have any caller-supplied identity removed before processing. Starting with 5.0, these checks are always enforced, regardless of whether request signing or node-join authentication is configured.

Network separation does not authenticate the sender. It establishes where a request arrived so that user ACLs and per-node signatures can be applied without trusting caller-controlled headers.

Blog: One Cluster, Two Perimeters, Three Networks.

Per-node identity

Under CSK/HMAC every node held the same secret. Any node could produce a signature any other node would accept; a leaked secret compromised the whole cluster with no partial recovery. Verification told a node that someone in the cluster had signed, never which node - so authorization could not depend on caller identity.

The post-bridge Ed25519 model changes both properties. Each node generates its own keypair, and its public key is distributed through the cluster map. A receiver verifies each protected request against the claimed sender's public key, establishing which node sent it.

The cluster map therefore gains a per-node verifying-key field. This metadata-schema change is introduced in 5.0, but the bridge leaves the field empty while 4.x nodes may still be present. Starting with 5.1, every admitted node publishes its verifying key and the cluster can rely on it.

That implementation is present but not active in v5.0. A 5.0 node generates its keypair locally, strips the verifying key from join metadata, and neither signs nor verifies intra-cluster requests. This keeps a rolling 4.x-to-5.0 upgrade possible. Once the bridge gate is removed, pre-5.0 peers are refused.

Keys are held in memory only. Every node generates a fresh keypair at startup and, after the bridge, publishes its new public key through the cluster map. There is no signing-key file to provision, protect, back up, or rotate; a restart performs the rotation, and the cluster distributes the replacement key automatically.

rotation_grace controls transitions when request authentication is changed: signing starts immediately when protection is enabled, strict rejection of unsigned requests begins after the grace window, and nodes continue signing briefly after protection is disabled so the cluster can converge.

AIStore 4.9 is a small patch release focused on space cleanup.
It follows 4.8 by about five weeks and is 100% backward compatible.

Space cleanup visits EC slices and metafiles in a given bucket, retaining each one only when its counterpart is present; both checks read the bucket's EC properties. The bucket is now properly initialized before the traversal.

• 7b2fe2d16: space/cleanup: fix nil-deref on EC (cherry-picked from f45b936)

AIStore 4.8 is a small release focused on authorization correctness, rebalance stability, and ETL initialization.
It follows 4.7 by about four weeks and is 100% backward compatible.

The primary changes authorize every input bucket referenced by a get-batch request, harden global-rebalance preemption, and keep the 2PC transaction GC from aborting slow ETL initialization.

A get-batch (multi-object streaming) request can reference multiple input buckets.
This release adds per-bucket authorization for get-batch: each bucket in apc.MossReq is validated and authorized before the batch is served, with proxy-side parsing that mirrors the target's unmarshaling of apc.MossIn entries.

get-batch (aka Get-Batch) serves multiple objects and//or archived files in a single request, across one or more buckets, whether or not those buckets hold .tar, .tgz, or other serialized shards.

• 1b86b3601: get-batch: authorize all input buckets
• 4d0a8a6b8: get-batch: authorize all input buckets (part two)
• cf0b9e1e8: follow-up
• 417b04f03: get-batch: initialize input buckets before access check (fix nil-props panic)
• 8f7f1bb19: get-batch: skip the path bucket by bucket equality not pointer identity

Preemption of currently executing global rebalance now waits for the previous local instance to clear from its singleton manager, and stage/abort notifications are made defensive against async cleanup and sent with the originating rebalance ID.
Fixes #317.

• 4721ba634: global rebalance: preemption and stage notification

The 2PC transaction GC reclaimed begin-phase transactions after a flat 2 * max_host_busy timeout, which could abort a slow ETL init (pod creation plus readiness) mid-flight.
ETL init now honors init_timeout plus slack, and the GC warning threshold derives from the per-txn deadline.

• 524a2193d: etl: respect init_timeout in 2PC transaction GC for ETL initialization
• ee96000e3: etl: refine begin-phase GC timeout (follow-up)

Fixes a race in CPU-utilization sampling by serializing concurrent samplers under lock.

• d35d3e007: sys/cpu: fix CPU sampling race

golang.org/x dependencies with known CVEs are updated to recent, patched releases: crypto v0.53.0, net v0.56.0, sys v0.46.0.

• eb59a3829: build: bump golang.org/x/{crypto,net,sys}

AIStore 4.7 is a small patch release for improved long-term stability and observability.

The primary changes fix unbounded growth of xaction notification metadata passed between IC members, extend Prometheus with Go runtime metrics and a per-node primary-proxy gauge, and tighten outbound network policy for the external downloader.

IC Notification Listener Cleanup

Finished xaction notification listeners were not being reclaimed reliably in long-lived clusters, causing IC ownership-table metadata to grow without bound.
Cleanup timing and IC synchronization of listener state are corrected.

Commits

• 47628799e: core: fix notification listener time-based cleanup
• d23009750: core: use consistent nl unmarshal, test atomic marshal round trips

Observability

Prometheus can now include a low-cardinality subset of Go runtime metrics (goroutines, GC, heap) alongside the existing AIS counters.
These are off by default and gated by the cluster Enable-Go-Runtime-Metrics feature flag.
Each node also exports ais_node_primary_info, a gauge labeled with primary_id for the primary proxy reflected in its cluster map.

Commits

• 7e08a8461: stats: register subset of go runtime metrics with prometheus
• da930ae0c: observability: conditionally publish Go-runtime Prometheus metrics
• 0c395682c: stats: add primary info metric

Downloader egress controls

The downloader now applies egress policy to its outbound HTTP connections.
By default, destinations in non-public address space are not reachable.
Clusters that need to fetch from private networks can enable the Dload-Allow-Private-Egress feature flag; a subset of sensitive ranges stays blocked regardless of configuration.

Reported by tonghuaroot <tonghuaroot@gmail.com>. Thanks for the detailed write-up.

Commits

• 06f8bfc83: ext/dload: restrict downloader egress to mitigate SSRF

Other Changes

• Sendfile transmit — the GET sendfile path held back in 4.6 is now enabled for eligible plain-HTTP, file-backed objects, including range reads.
• Shard index summary — new target-side summary-shard xaction aggregates indexed vs unindexed TAR coverage for a bucket.
• Space cleanup — completed chunk manifests are validated by load, not LOM flag alone.
• S3 compatibility — response XML root element names match the AWS spec for strict clients.

Commits

• da541a1be: sendfile transmit path: enable; add range-read (part two)
• fe855df60: shard index: add bucket shard summary xaction
• 1739b1c59: space/cleanup: validate completed chunk manifests
• 9a5d91c0d: ais/s3: emit AWS-spec XML root element names for S3 responses

AIStore 4.6 is a short-cycle reliability and scalability release following 4.5.

The drivers are practical: stabilize get-batch under heavy ML-style stress, replace the list-objects startup grace period with an explicit two-phase commit (2PC) protocol, avoid cgroup-v2 broadcast throttling in large clusters, harden primary-restart bootstrap, guard against prefetch misconfiguration, and make prefetch/blob-downloader behavior easier to observe and control.

Get-Batch is the largest area of work. Under heavy ML-style load - many concurrent batch requests, large objects, slow or disconnecting clients - the prior implementation could leak work items, take too long to abort, or mishandle late arrivals after cleanup. The 4.6 changes restructure the work-item lifecycle around explicit ownership, receive-side pinning, progress-based cleanup timers, bounded abort latency, and stricter late-arrival rejection.

Remote list-objects, also referred to as R-flow, gets a structural fix. The temporary startup-grace workaround from 4.5 is replaced with an explicit begin/commit/abort sequence: targets that are about to receive pages prepare first, the randomly selected designated target starts paging only after the receive side is ready, and abort has a dedicated cleanup path.

Broadcast parallelism in containerized deployments is also improved. In deployments using cgroup-v2 CPU quotas, a proxy with a small visible CPU count could collapse one-to-many broadcasts to single-digit parallelism even when targeting dozens of targets. A dedicated cluster broadcast wait group now covers the mostly I/O-bound intra-cluster fanout paths: keepalive, rebalance, generic broadcasts, and xaction target-to-target control.

The intra-cluster control plane is faster on the hot path. Keepalives and broadcasts - invoked behind essentially every cluster operation - no longer register a timer, allocate a context, and defer a cancel on every call. A separate fix covers a former primary restarting without a local cluster map: rather than assuming the primary role from local configuration alone, the proxy probes alternative URLs and validates returned node-state-info first.

Prefetch and blob-downloader observability is another release driver. A single prefetch job can spawn many blob-download child jobs. The 4.6 release adds clearer parent/child reporting, per-xaction stats, bucket-keyed Prometheus metrics, and rejection counters so operators can understand progress and throttling behavior without correlating logs manually. Get-Batch and list-objects extend their control-message output in the same direction. The cross-cutting picture is summarized in Observability.

No data-format migration is required. Operational and API behavior changes are summarized in Upgrade Notes.

Table of Contents

1. Get-Batch
2. Remote List-Objects (R-flow)
3. Large-Cluster Broadcast Scalability
4. Intra-Cluster Control Plane
5. Primary Restart (bootstrap)
6. Prefetch and Blob Downloader
7. Space Cleanup
8. Shard-Index Cleanup
9. Xaction Go API
10. ETL
11. Python SDK and aisloader
12. Other Fixes
13. Documentation and Tools
14. Observability
15. Upgrade Notes

Get-Batch

Work-item lifecycle

Get-Batch is highly concurrent by design: a single x-moss xaction multiplexes many batch requests, each driven by a remote client. A slow or disconnected client can leave its work item parked on the target side - receivers still pinned, SGLs still held - while making no forward progress. We call these abandoned work items; detecting them reliably without falsely killing legitimately slow batches is one of the problems 4.6 addresses.

Work-item ownership is now an explicit three-state model rather than a single demand flag. Assembly claims a work item before consuming it; abandoned-work GC may clean only unowned work; and receive paths use pin/unpin discipline, so data and control receives cannot accidentally keep touching a work item that is already cleaning up or abandoned.

Abandoned work items are marked before cleanup, and new receives are rejected once cleanup begins. Cleanup waits for a stable receive/assembly quiet window, using last-receive and last-assembly progress timestamps rather than only the original start time. This matters for large batches that can legitimately run for a long time while still making forward progress.

The work-item memory pool (sync.Pool) is removed in favor of explicit receive-side pin/unpin. The housekeeper gets a dedicated x-moss maintenance callback, so abandoned-work cleanup no longer depends on the generic DemandBase idle/fini path, and each cleanup pass is capped so the housekeeper remains responsive.

The shared data-mover (SDM) receive-side lock ordering is tightened as well: the receiver-demux mutex is no longer held across Close(), and Rx drop now distinguishes SDM close from xaction abort.

Abort and late-arrival handling

Abort latency is now bounded. An unbounded wait-group.Wait() is replaced with a bounded poll, and drainRx is extracted with a brief mode for fast cleanup paths. Stuck receive-side work items are forced through cleanup via the abandoned path. Late assembly is rejected after x-moss is aborted or done; late Assemble calls after abort cleanup are tolerated; and internal stopped sentinels are treated as quiet assembly termination.

Stop/Abort terminal semantics are tightened. DemandBase.Stop is now idempotent and protected by its own atomic, and the common SetStopping returns its CAS result so get-batch can use it to eliminate a race against its own concurrent Abort. DemandBase.Stop is now called before waiting and quiescing.

Get-from-neighbor (GFN) is the internal target-to-target object-fetch path, used both during global rebalance, when objects migrate between targets, and by get-batch when a target needs an object it does not have locally. GFN error handling is corrected as well. When get-batch engages GFN, not-found remains a soft error while other GFN failures are treated as hard errors. Membership changes are detected and enforced during hole-timeout and failed-GFN paths, so a target-set change no longer looks like an ordinary object-level miss.

Observability and docs

Control messages now include target IDs and use clearer formatting. Documentation is updated to clarify supported scope and known limitations.

Commits

• 127f69be6: get-batch: bound abort latency; amend gcAbandoned for stuck Rx
• 564efd619: get-batch: exit gracefully via housekeeper (fix)
• 926bdbb42: get-batch: amend abandoned work-item cleanup (add 2nd housekeep)
• cd07a4ec4: get-batch: amend abandoned work-item cleanup (part two)
• 9ad57d35f: get-batch: amend abandoned work-item cleanup (part three)
• f3756a125: get-batch: abort vs active assembly (part four)
• 3f247c528: get-batch: remove work-items mem-pool; Rx pin/unpin discipline
• b87ea0cf0: get-batch: add cleanup stability window; track last assembly progress
• edc4d7338: get-batch: gate pin-wi before touching last-Rx; recycled() to use last-asm
• f6fe8d279: get-batch: escalate GFN error (fix)
• 62937942f: get-batch: update CtlMsg to include target ID (observability; minor)
• 198732215: docs/get_batch: clarify supported scope and known limitations
• b55803190: core/xactions: Stop/Abort terminal semantics (fix)
• 3924f9e81: shared data-mover: demux/transport lock-order

Remote List-Objects (R-flow)

Multi-target remote-bucket list-objects - the R-flow mechanism, one of several distinct list-objects paths in AIS - gains the explicit prepare phase anticipated in the 4.5 release notes.

In a multi-target R-flow run, one target is designated to perform the backend listing and distribute pages to the others. In 4.5, the designated target waited a brief grace period before issuing the backend call and beginning page distribution - a temporary workaround until the flow gained an explicit prepare phase.

The workaround is replaced with an explicit two-phase protocol. The begin phase broadcasts startup, renews the x-lso, opens streams, and registers receive paths. The commit phase performs strict x-lso lookup and starts paging. The abort phase performs best-effort lookup and aborts the xaction if present. The previously reserved feature flag used by the workaround is removed as part of the same cleanup.

The same area gains better runtime observability, extending the dedicated ais show job rendering introduced in 4.5. List-objects now reports lightweight per-reque...

AIStore 4.5 is a focused release on three major areas: global rebalance scalability, indexed archive access via the new shard index, and a get-batch flow reordering that substantially reduces memory pressure under load.

The work on global rebalance is one of the headlines. AIS removes the legacy per-object ACK machinery that had become a scalability ceiling for very large rebalances (counting millions of migrated objects), replaces the cleanup behavior that depended on that machinery with a new explicit cleanup mode, and optimizes the lifecycle around data movers, transport endpoints, stage transitions, and peer status queries.

Remote list-objects (R-flow) is another practical driver for this release. Multi-target remote-bucket listing now starts the (N - 1) targets before the designated target (DT) begins backend listing and page distribution. The same area also gets flow-control cleanup, corrected page accounting, and a dedicated CLI job view for remote-list xactions.

The shard index is a new experimental subsystem for indexed extraction from TAR shards. It lets GET and get-batch read files from TAR shards directly using a persisted index instead of scanning the full archive. The 4.5 implementation includes the index format and binary pack/unpack support, persistence in a system bucket, a bucket-scoped indexing xaction, CLI support, read-path integration, tests, and a micro-benchmark.

Streaming get-batch responses now use explicit write deadlines while sending data to the client. This lets AIS detect terminated, stalled, or unreachable clients promptly and abort the request instead of continuing to assemble and transmit a large batch that no client is still reading. The flow was reordered and optimized, work-item cancellation now propagates across senders, and admission control is stricter under load.

Authentication and access control adds support for externally provisioned RSA signing keys, JWKS refresh on cache miss for rotated key pairs, configurable maximum token age, and a more general signing-key configuration. Intra-cluster request validation is also tightened: spoofed caller headers are rejected on the public network, and internal-network checks require a validated Smap entry.

CLI and observability gain several user-visible improvements: dynamic ais show cluster cpu and ais show cluster memory views, a new ais performance intra-data view, shard-index commands, better rebalance rendering including cleanup mode, and improved help for force-join / split-brain recovery workflows.

This release preserves backward compatibility.
The few additive API fields, configurable behavior changes, and operational migration notes are summarized in Upgrade Notes.

Table of Contents

1. Global Rebalance
2. Shard Index
3. Get-Batch
4. Intra-Cluster Control Plane
5. AuthN
6. Stats and Observability
7. Blob Downloader and Prefetch
8. CLI
9. Core, Config, and xactions
10. Python SDK, ETL, and aisloader
11. Documentation and Website
12. Build, CI, and Tools
13. Upgrade Notes

Global Rebalance

AIStore 4.5 delivers the largest rebalance update in several release cycles. The main theme is replacing per-object state with stage-level coordination and explicit cleanup semantics.

Per-object ACKs removed

Rebalance no longer tracks individual object acknowledgments. The previous mechanism - ACK messages back to the sender, sender-side maps of unacknowledged objects, retransmit and wait loops, and an ACK-driven lazy-delete path - has been removed in favor of stage-level coordination.

In the new model, traversal sends objects without keeping per-object state on the sender, and a post-traverse barrier takes the place of the old wait-for-ACK drain. Intra-cluster transport headers carry a compact opaque payload with the rebalance generation, so receivers can recognize and reject objects that arrive late from a previous run. Object and byte totals reported by ais show rebalance and ais show job now come directly from the transmitted counters rather than from ACK accounting.

The practical effect is lower memory pressure and simpler lifecycle behavior during large rebalances, where per-object ACK state had become the limiting factor.

Cleanup mode

Removing per-object ACKs also removed the old incidental mechanism that trimmed misplaced source copies after migration. AIStore 4.5 adds an explicit replacement: rebalance cleanup mode.

Cleanup mode walks local mountpaths, identifies misplaced object copies using Smap HRW ownership, and removes a misplaced copy only after verifying that the canonical copy exists in the expected location with matching object identity. Identity checks include size and checksum, and use version / ETag when available.

CLI:

$ ais start rebalance --cleanup
$ ais start rebalance --cleanup --force

By default, cleanup mode keeps diverged copies. With --force, it can also remove copies that differ from the canonical peer version. This is an advanced operator option.

Cleanup mode is intentionally distinct from regular rebalance:

• it has its own preflight checks;
• it refuses to start while rebalance or resilver is active;
• it requires at least two active targets;
• it bypasses config.Rebalance.Enabled;
• it uses no data mover, no streams, and no GFN;
• it skips EC-enabled buckets and busy objects.

ais show job and ais show rebalance render cleanup-mode runs with a dedicated view that reports removed objects and bytes rather than migration TX/RX counters.

Lifecycle and transport

A series of lifecycle changes makes rebalance more robust through abort, preempt, renew, and finalization paths: fresh data mover construction per run, safer handling of duplicate transport endpoints after abort, narrower mutex scope in the finalization path, a consistent same-targets predicate across preempt and renew, and corrected stage-reached detection.

One change is operator-visible: rebalance CtlMsg now carries per-stage timing and final status, and ais show job --all continues to surface this information after the run completes.

CtlMsg is the control-message interface supported by AIS xactions (jobs). Each xaction uses it to report job-specific runtime state - counters, stage information, start parameters, and error summaries - so that ais show job and related commands can render a current view without requiring separate CLI plumbing for every xaction.

See also

• docs/rebalance.md - user-facing guide: concepts, configuration, and CLI workflows including cleanup mode.
• reb/README.md - internal design notes: execution flow, stages, and lifecycle.

Commits

• 05eddc1e1: global rebalance: remove per-object ACKs (major upd)
• d8d4f9931: global rebalance: introduce safe cleanup mode
• 28d55f6c9: global rebalance: cleanup-mode observability
• 0cab3b877: global rebalance: consolidate primary's logic to run cleanup mode
• b6037a3da: cli: ais show job to correctly show rebalance in cleanup mode
• 785a1b769: cli: 'start rebalance [--cleanup]' to show ID; add formatted action helpers
• 0df4c7549: global-rebalance/cleanup-mode: add CLI-based scripted test
• 3c5ead654: space cleanup: detect cluster-wide misplaced objects via Smap HRW
• ad5256b96: docs: update rebalance.md (add cleanup mode; examples)
• a10928632: apply rebalance conflict/abort policy consistently across jobs
• 913f4bd89: lazy-delete: reduce noisy logs and bump work channel cap

Shard Index

Experimental in 4.5. The on-disk format, xaction semantics, and CLI surface may change in subsequent releases.

The shard index is a new subsystem for indexed archive extraction, primarily intended for ML and data-lake workloads that repeatedly fetch named samples from large TAR shards. When a client requests a file inside a TAR shard via archpath, AIS can now resolve the lookup through a persisted index rather than scanning the archive. The fast path is integrated into both GET and get-batch.

A shard index is a compact binary structure built in a single pass over a source TAR. It supports USTAR, GNU, and PAX variants, encodes entries with Pack/Unpack, and embeds the source object's checksum and size so the index can detect re-uploads and treat itself as stale when the underlying shard has changed. Indexes are persisted as objects in a new system bucket (ais://.sys-shardidx); shards that have an index carry a dedicated flag in their on-disk metadata.

Indexing is performed by a bucket-scoped xaction (job). It walks the bucket, skips non-TAR objects, rebuilds stale or corrupted indexes, and treats per-object failures as non-fatal - a single bad shard does not abort the run.

Index presence, lookup, va...

AIStore 4.4 is a short release cycle focused on runtime correctness, container awareness, and consolidation.

This release is the first to deliver full cgroup v2 support, improving AIS behavior in constrained and containerized deployments. CPU and memory accounting are now cgroup-aware, container detection and startup initialization are more robust, and CPU utilization is reported as a smoothed moving average rather than an instantaneous sample.

4.4 also promotes native bucket inventory (NBI) into the default AIS path for S3-compatible inventory workflows. The release adds non-recursive inventory listing, removes the legacy S3-specific inventory implementation, and updates the S3 compatibility layer to route inventory-backed requests through NBI. NBI should now be regarded as stable.

Additional changes improve S3 bucket region discovery and error reporting, add per-bucket OCI regions and instance principal authentication, introduce per-bucket user-defined custom metadata, and update the CLI, documentation, tracing, and build tooling.

This release includes roughly 50 commits since 4.3 and is fully backward compatible.

Table of Contents

1. Runtime: cgroup v2, CPU, and Memory
2. Native Bucket Inventory (NBI)
3. S3 Compatibility and Remote Bucket Discovery
4. OCI Backend
5. Bucket Metadata and BMD Changes
6. CLI
7. Python SDK and ETL
8. Tracing
9. Website
10. Documentation
11. Build, CI, and Tools
12. Upgrade Notes

Runtime: cgroup v2, CPU, and Memory

The main feature of 4.4 is full cgroup v2 support, with a broader runtime refactor to make AIS behave correctly in standard containers and restricted Kubernetes-style environments.

See Also

• AIS in Containerized Environments

The work starts with startup and environment detection. AIS now initializes CPU and memory accounting earlier and more explicitly via sys.Init(), revises container auto-detection with additional heuristics, and adds the ForceContainerCPUMem feature flag to override failed detection when needed. The initialization path also applies container-aware CPU counting and GOMAXPROCS, and avoids retrying failed cgroup parsing later at runtime.

On the memory side, AIS now supports cgroup-v2 memory reporting based on memory.max, memory.current, and memory.stat, while keeping memory.stat best-effort so that missing or unreadable auxiliary details do not turn into fatal runtime failures. Host and container memory readers have been refactored into stateless helpers, and related error reporting has been cleaned up and prefixed consistently.

On the CPU side, bare-metal /proc/stat parsing is corrected to use specific enumerated fields and avoid double-counting. The runtime now maintains global cached CPU state and reports CPU utilization as a smoothed, time-aware moving average rather than an instantaneous sample. The new model is better suited for operator-facing reporting and long-running process decisions, and it also supports CPU throttling (cgroup v2).

The CLI reflects these changes directly. ais show cluster now shows SYS CPU(%) instead of load average by default, moves load averages behind --verbose, and adds THROTTLED(%) when the displayed proxy or target section contains at least one non-zero cgroup-v2 throttling value.

The last part of the work addresses deployment realism. On Kubernetes, cgroup-v2 files may live under per-pod subtrees rather than fixed global paths, so AIS now resolves /sys/fs/cgroup paths at init time instead of hardcoding them. Startup logging also returns a more descriptive container tag, and the implementation includes explicit constrained-container testing and documentation.

Commits

• dd03cae2c: sys: bare-metal CPU usage parsing (fix)
• 1b1397ee3: sys: cpu and memory (part two; major)
• 3fb3aeb7e: sys: cpu and memory (support cgroup-v2 memory stats)
• ac6c9461a: sys: cpu and memory (cgroup v2 in a constrained container)
• 074554a72: sys: CPU utilization is now a moving average; [API change]
• 30e7f430a: sys: cpu and memory (resolve /sys/fs/cgroup/ paths at init time)
• 366a8906e: cli: show CPU utilization (EMA); load-averages only when verbose

Native Bucket Inventory (NBI)

Native bucket inventory was introduced in 4.3. In 4.4, it becomes the clear and supported path for AIS-managed inventory-backed listing.

The most visible functional addition is support for non-recursive inventory listing via --nr. That closes an important gap for users who want inventory-backed listing while preserving delimiter-style behavior rather than flattening the namespace unconditionally.

Just as importantly, 4.4 removes the older S3-specific inventory implementation and routes S3-compatible inventory-backed requests to NBI instead. This includes removal of deprecated S3-only backend hooks, old R-flow plumbing, and deprecated CLI/API pieces such as --s3-inventory. At the S3 compatibility layer, AIS keeps the existing Ais-Bucket-Inventory and Ais-Inv-Name headers, but now interprets them through NBI with validation.

With 4.4, NBI is no longer experimental. The docs are updated accordingly; 4.4 adds benchmark tooling and documentation around large-bucket listing to support that shift in status.

See Also

• Native Bucket Inventory: Up to 17x Faster Remote Bucket Listing

Commits

• ddbca5e78: native bucket inventory: support non-recursive mode (--nr)
• 95b023365: remove legacy S3 inventory; route S3 compat to NBI; update docs
• 8cab32fcf: perf: add NBI benchmark python script; update docs
• b06d42815: blog: native bucket inventory blog

S3 Compatibility and Remote Bucket Discovery

4.4 also improves general S3 usability outside NBI.

First, AIS now uses HeadBucket for region discovery. When the bucket region is initially unknown, the discovered region is propagated back into the session and used to construct a new client with the now-complete connection tuple. This also removes an older special workaround and includes a ListBuckets() special case when credentials do not provide a region. The overall effect is simpler and more reliable region handling for S3 buckets.

Second, 4.4 improves the behavior and messaging when users try to add a inaccessible S3 bucket, including clearer guidance for cases such as explicitly specifying extra.aws.cloud_region.

Third, AIS now returns more standard S3 error codes. The S3 layer introduces structured error information via s3.ErrInfo and consistently reports codes such as NoSuchKey, NoSuchBucket, and NoSuchUpload, including NoSuchKey for GET, HEAD, DELETE, and COPY of non-existent objects.

Commits

• 829559e73: s3: use HeadBucket for region discovery
• db3a5a4a5: s3 compat: return standard S3 error codes (NoSuchKey, et al.)

OCI Backend

The OCI backend gains two notable capabilities in 4.4.

First, AIS now supports per-bucket OCI regions via extra.oci.region. Backend operations are routed through region-keyed clients, making it possible to use different OCI regions on a bucket-by-bucket basis rather than relying on a single process-wide assumption.

Second, AIS adds native instance principal authentication via OCI_INSTANCE_PRINCIPAL_AUTH. The implementation also serializes OCI client creation per region, adds regression tests, and extends CLI autocompletion for extra.oci.*. Because this is marked as a BMD change, it belongs in the upgrade notes as well.

Commits

• ac35234f9: [BMD change] oci: support per-bucket regions and instance principal auth

Bucket Metadata and BMD Changes

4.4 introduces extra.custom, a new per-bucket user-defined metadata field in ExtraProps.

The new field is an opaque string, up to 128 characters, settable via bucket properties APIs. Validation rejects control characters and enforces a maximum length consistently across providers. CLI help and usage examples are updated accordingly, and bucket-property error messages now include unmarshal details to make bad inputs easier to diagnose. This change is also marked as a BMD change.

Commits

• fed41c850: [BMD change] add extra.custom: per-bucket user-defined metadata; cli help

CLI

Most visible CLI work in 4.4 is tied to the runtime and NBI changes already de...

AIStore 4.3 introduces native bucket inventory (NBI) for remote buckets, backed by a new system-bucket namespace. Buckets with millions or tens of millions of objects can now be listed from AIS-managed inventory snapshots instead of traversing the backend on every call.

This release also adds full IPv6 networking support across all three logical networks and the transport streaming layer, with cluster-wide address-family configuration and automatic IPv4 fallback.

4.3 expands authentication and key-management capabilities with JWKS persistence, manual key rotation, standard JWT claims, and a new key-provider abstraction. Client-side multipart download is significantly improved across the Python SDK, Go APIs, and aisloader, with a new multiprocessing-based parallel downloader backed by a shared-memory ring buffer. The Python SDK and ETL toolchain add streaming transform support across multiple server styles and improve retry handling.

Improvements include media-aware worker parallelism, global rebalance hardening, per-bucket GCP credentials, remote-bucket support for rechunk, multipart whole-object checksum, and various backend and reliability fixes.

Object HEAD v2, introduced as experimental in 4.2, is now stable.

This release includes over 300 commits since 4.2 and is fully backward compatible.

Table of Contents

1. Native Bucket Inventory (NBI)
2. System Buckets
3. IPv6 Networking
4. Multi-Part Download (MPD)
5. Multi-Part Upload: Whole-Object Checksum
6. Global Rebalance
7. Multi-Credential GCP Backend
8. Authentication and Key Management
9. Media-Aware Worker Parallelism
10. Object HEAD v2
11. ETL
12. Rechunk: Remote Bucket Support and --sync-remote
13. Python SDK
14. CLI
15. Runtime, Backend, and Reliability Fixes
16. Documentation
17. Build, CI, and Tools
18. Upgrade Notes

Native Bucket Inventory (NBI)

Native bucket inventory is experimental in 4.3. It gives AIS a built-in way to create and reuse inventory snapshots of remote buckets.

NBI currently applies to remote buckets only: cloud buckets (including S3, GCS, Azure, and OCI) and remote AIS buckets.

The goal is straightforward: make listing large remote buckets - those with millions or tens of millions of objects - substantially faster to list. Direct backend listings at that scale can take minutes or longer and repeat the same remote roundtrips on every next-page call. With NBI, AIS builds a cluster-managed inventory snapshot once, then serves subsequent listings from local storage, supporting pagination and prefix filtering without touching the backend.

The implementation includes:

• new CLI commands to create, inspect, and remove inventories
• inventory-backed list-objects via the same distributed flow used for in-cluster ais:// buckets
• paginated reads and continuation tokens
• prefix filtering over inventory snapshots
• inventory metadata including object count, chunk count, participating targets, and cluster-map version
• support for empty inventory names (lookup by bucket name)
• handling of small buckets where some targets may have nothing to list
• improved cache behavior and page-size handling during listing
• deprecation of the older S3-specific inventory path in favor of a native AIS implementation (see Upgrade Notes)

NBI is primarily a latency-reduction tool. Since ais:// data and metadata are already in the cluster, there is no latency tax to justify a separate inventory layer.
Still, it remains a valid option for point-in-time consistency (snapshotting) - to be possibly considered in the future.

Commits

• c4ee603ed: Initial NBI support; create ais://.sys-inventory on first use
• 8daff7a67: Create-inventory 2PC; target validation and execution
• 416fc041b: Add top-level ais nbi {create|rm|show}
• 4be7dcd90: Initial list-objects integration for NBI
• f3472105d: Switch NBI listing from R-flow to A-flow
• ffefd96db: Replace page-based knobs with NamesPerChunk
• ee7514cde: Best-effort page sizing for distributed NBI listing
• 846ee81af: Add chunk cache; cover tiny-page and empty-name cases
• 51a1b3889: Extend metadata; improve default and verbose show nbi
• 364e67361: Handle small buckets and empty inventories

System Buckets

System buckets establish a reserved namespace for AIS-managed internal data.

The first system bucket is:

• ais://.sys-inventory - stores native bucket inventory snapshots as AIS-managed internal objects

System buckets are created on demand and can be listed and inspected with regular CLI commands. At the same time, they are infrastructure: users are not expected to write into them directly or depend on their internal object layout.

This namespace lays the groundwork for more AIS-managed content in future releases.

Commits

• 347e27e12: Introduce system buckets; reserve leading-dot names
• c4ee603ed: First system bucket: ais://.sys-inventory
• 2813f2f1d: Add system-bucket docs; clarify NBI and bucket naming

IPv6 Networking

IPv6 support is now available across all three logical HTTP networks and the transport streaming layer.

It is enabled cluster-wide via the net.use_ipv6 configuration knob. In production, nodes are typically configured with explicit addresses. When addresses are not configured, each node attempts to discover usable IPv6 addresses and, if none are available, falls back to IPv4. Either way, the effective IP family is applied consistently for the lifetime of the process.

Setting net.use_ipv6 in the cluster configuration is a cluster-wide address-family preference, not a dual-stack mode. Once the effective family is determined at startup, AIS avoids dual-stack dialing entirely, with no fallback probing on each connection. This keeps connection setup predictable and eliminates the latency overhead that dual-stack resolution can introduce at scale.

Under the hood, this required moving address-family resolution earlier in startup, before listeners and clients are fully brought up. That keeps listeners, intra-cluster HTTP clients, and related transport paths aligned with the same runtime networking choice, and avoids mixed-family failures such as binding on IPv4 while still attempting IPv6 connections elsewhere. This work also extends local playground and CI coverage, and is documented in docs/networking.md.

A reminder that AIS separates traffic across three logical networks: a user-facing public network, and two intra-cluster networks for control and data. Each can be placed on a dedicated NIC or VLAN. The most impactful use is isolating the intra-cluster data network - which carries global rebalance, get-batch, and other bulk data movement - from the public network serving foreground I/O. In fact, on loaded clusters, there is also strong motivation to isolate the latency-sensitive (small-bandwidth) control plane from both of the above. Logical network separation is a designed-in capability that pays off most at high drive counts and high utilization, and is worth configuring explicitly rather than relying on shared bandwidth of 100GbE links.

See Also

• AIStore Networking Model

Commits

• 3a4e3ae49: Initial IPv6 support; config and primary networking path
• 1b77779aa: IPv6 fallback logic; Host2IP dialability filtering
• f3630bc68: Local playground IPv6 support; add AIS_USE_IPv6
• fa5cb4ca3: Apply effective IP family consistently across listeners and clients
• dd8f6c5e7: Avoid dual-stack dialing when effective family is IPv4
• 3f7f1418f: IPv4 fallback for configured IPv6 preference

Multi-Part Download (MPD)

Multi-part download (MPD), also referred to as parallel download, sees substantial improvements in 4.3 across the Python SDK, Go client APIs, and aisloader.

For a detailed walkthrough of the architecture, benchmarks, and PyTorch integration,
see the companion blog post: [Parallel Download: 9x Lower Latency for Large-Object Reads](https://aistore.nvidia.com/blog/2026/03/25...

AIStore 4.2 focuses on correctness and reliability, authentication and observability, API modernization, and operational fixes across backends and tooling.

The resilver subsystem was substantially rewritten, introducing explicit preemption on mountpath events and full support for chunked object relocation.

Security enhancements include Prometheus metrics for AuthN, persistent RSA key pairs, OIDC-compatible discovery and JWKS endpoints.

New APIs replace legacy polling with explicit condition-based waiting for batch jobs and introduce a chunk-aware HEAD(object), enabling more scalable job monitoring and efficient large-object access.

List-objects implementation was corrected for non-recursive walks on AIS buckets, while the Python SDK and CLI add faster, chunk-aware download paths with parallelism and progress reporting.

Additional improvements and fixes span: cloud bucket namespaces, multipart retry behavior, backend interoperability, and premature global-rebalance completion reporting.

In particular, 4.2 adds support for namespace-scoped cloud buckets (e.g., s3://#prod/data, s3://#dev/data, etc.). This enables multi-tenant scenarios where different users/accounts access same-named buckets in different cloud accounts via respective (different) profiles and/or endpoints.

AIStore 4.2 maintains full backward compatibility with v4.1 and earlier releases. Overall, this release improves the system's availability in presence of disk faults, observability and correctness under load, and modernizes long-standing APIs.

Table of Contents

1. Resilver
2. Authentication and Observability
3. New APIs
   • 3.1 Xaction v2
   • 3.2 Object HEAD v2
4. List Objects: Non-Recursive Walks
5. Multipart transfers (downloads, uploads, and backend interoperability)
6. Global Rebalance
7. Filesystem Health Checker (FSHC)
8. ETag and Last-Modified Normalization
9. Python SDK
10. CLI
11. Documentation
12. Build and CI
13. Miscellaneous fixes across subsystems

Resilver

Resilver is AIStore’s node-local counterpart to global rebalance: it redistributes objects across a target’s mountpaths to restore correct placement and redundancy after volume changes (attach, detach, enable, disable).

Version 4.2 introduces a major rewrite of the resilver xaction for correctness and reliability. The previous implementation relied on retry-based copying loops; the new implementation uses deterministic copy (object replica) selection and explicit lifecycle management.

Resilver is a single-target operation: cluster-wide execution is now disallowed to prevent cross-target interference; attempts to start resilver without a target ID are rejected by both the CLI and by AIStore itself (i.e., calling the API directly without a target ID will also fail).

Mountpath event-triggered resilvers remain internal and continue to register with IC automatically (the events are: enable, disable, attach, and detach).

Improvements include:

• Preemption on mountpath events - disable/detach/attach/enable operations now abort any running resilver and restart appropriately; handles back-to-back mountpath events without data loss
• Chunked object relocation - step-wise relocation with rollback and cleanup on error; validates chunk placement
• Mountpath jogger lifecycle - filesystem walks terminate when the parent xaction aborts
• Concurrent access fixes across shared resilver state
• Runtime progress visibility - live counters (visited objects, active workers) via 'ais show job' CLI
• Deterministic primary copy selection - to eliminate contention between concurrent goroutines

New documentation: docs/resilver.md

Commit Highlights

• 805f9ab93: Rewrite copy-recovering path; deterministic primary copy selection
• 48153ca6d: Preempt running xaction upon mountpath events; add tests
• 55c890fdd: Relocate chunked objects with rollback
• 140c6e432: Stop joggers; revise concurrent access to shared state
• d2a56a220: Add stress tests; consolidate resilver tests
• 1a53f10e9: Revise mountpath jogger; add walk-stopped sentinel
• 96213840f: Wire walks to parent xaction abort

Authentication and Observability

Building on v4.1 authentication improvements, this release adds OIDC-compatible discovery and JWKS endpoints, persistent RSA key pairs, and production-grade observability for AuthN.

AuthN now supports both HMAC-SHA256 and RSA (RS256) token signing. When no HMAC secret is provided via config or AIS_AUTH_SECRET_KEY, AuthN will initialize and persist an RSA keypair on disk and use it to issue RS256-signed JWTs.

From the operational perspective, the important changes include:

• OIDC discovery + JWKS endpoints: AuthN now serves /.well-known/openid-configuration and a public JWKS endpoint for RS256 verification; JWKS responses include cache control based on token expiry.
• Cluster validation handshake: either the HMAC secret checksum or the RSA public key, depending on the configured signing method.

• Persistent RSA key pairs for AuthN - RSA private key is loaded from disk if present; otherwise generated once and persisted (key rotation not yet implemented)
• Prometheus metrics for OIDC/JWKS: counters for invalid iss/kid, plus latency histograms for issuer discovery and key fetches.
• Improved logging and validation - clearer token and ACL failure diagnostics; stricter permission checks (including admin and bucket-admin)

And separately, CLI:

• ais authn command, to inspect OIDC configuration and display RSA public keys
• ais authn show oidc and ais authn show jwks - to display discovery and JWKS output (JSON or table)

Commit Highlights

• b930247cc: Add metrics for total counts and JWKS caching
• 49733a2d6: Refactor access check; add initial Prometheus metrics
• 4253d7bde: Persistent RSA key pair for AuthN
• 2fb333675: Add RSA signing and validation to AuthN service
• 26e4d5098: Improve logging on token/ACL failures
• 48b830470: CLI: view OIDC config and show RSA public key
• df8f9e764: Fix CheckPermissions to validate all permission types

New APIs

3.1 Xaction v2

Xaction (eXtended action) is AIStore’s abstraction for asynchronous batch jobs. All xactions expose a uniform API and CLI for starting, stopping, waiting, and reporting both generic and job-specific statistics.

Version 4.2 introduces explicit separation between IC-notifying and non-IC xactions, replacing legacy polling with explicit condition-based waiting and formalizing two observation modes:

• IC-based status observation for xactions that actively report progress via IC
• Snapshot-based observation for xactions that do not

For IC-notifying xactions, this avoids polling every target and makes waiting scale predictably even in large clusters. Snapshot-based xactions continue to use explicit snapshot inspection.

Background: IC (Information Center) runs on three AIS gateways (one primary and
two random). Targets asynchronously notify IC of xaction progress, eliminating
per-target polling.

Observation APIs

Built-in Conditions

Conditions are expressed as xact.ArgsMsg methods and apply to waiting APIs:

Conditions that req...