Aarkam.AM Interactive CLI & Operator Fabric Architecture
Architectural blueprint and operational specification of the single-binary cluster orchestration tool (am) across 27 management domains.
Aarkam.AM (am / am.exe) is the sixth core module of the platform. Engineered as a high-performance, single-binary CLI, it serves as the primary distributed control-plane interface for Site Reliability Engineers (SREs), storage administrators, and automated deployment pipelines across sovereign datacenters.
1. Architectural Role & Execution Model
In distributed sovereign architectures, web-based graphical management is insufficient for automated Day-2 operations, emergency disaster recovery, or air-gapped automation. am operates independently of the web portal, communicating directly with Rokka.Coordinator and physical Rokka.StorageNodes over mutual TLS (mTLS):
Self-contained AOT-compiled binary (Linux ELF / Windows PE). Requires no external runtime, Python environment, or DLL packages.
Built-in CancellationToken-driven live refresh engine (--watch <TimeSpan>) replacing brittle shell sleep loops.
Native structured output serialization via --format table|json|yaml|ndjson|csv for programmatic consumption with jq.
2. The 27 Operational Command Domains
The am architecture structures cluster administration into 27 discrete functional command domains:
| Domain | Scope & Responsibility | Primary Operational Commands |
|---|---|---|
cluster |
Global Fabric Health & Topology | am cluster status, am cluster doctor, am cluster config |
mon |
Raft Consensus & Election Quorum | am mon status, am mon leader, am mon peers, am mon election |
node |
Storage Fleet Lifecycle & Drainage | am node list, am node add, am node drain, am node evacuate |
volume |
Physical Mounts & NVMe Storage Pools | am volume list, am volume mount, am volume scrub |
object |
Chunk Verification & Inspection | am object inspect, am object verify, am object locate |
ec |
Cauchy RS Matrix Codec & Benchmarks | am ec profile list, am ec benchmark, am ec test-repair |
repair |
Autonomous Scrubbing & Healing | am repair status, am repair run, am repair pause |
abodi |
Hardware S.M.A.R.T. & ML Telemetry | am abodi score, am abodi smart, am abodi forecast |
kdouja |
Direct I/O LSM Internals & Compaction | am kdouja stats, am kdouja compact, am kdouja segments |
s3 |
S3 Gateway Rate Limits & Routing | am s3 gateway status, am s3 reload-routes |
maintenance |
Zero-Downtime Host Servicing | am maintenance start <node>, am maintenance complete |
rebalance |
Cluster Capacity Equalization | am rebalance plan, am rebalance execute, am rebalance throttle |
placement |
Failure Domain Simulation (Rack/Row) | am placement simulate, am placement map |
diagnose |
Incident Triage & Root Cause Analysis | am diagnose cluster, am diagnose bundle -o report.tar.gz |
tenant |
Cryptographic Multi-Tenancy | am tenant list, am tenant create, am tenant rotate-keys |
quota |
Capacity & Rate Limit Governance | am quota set <tenant> --max-gb 50000, am quota inspect |
snapshot |
Point-in-Time Volume Snapshots | am snapshot create, am snapshot clone, am snapshot rollback |
backup |
Off-Cluster Archival & Cold Storage | am backup plan, am backup run, am backup verify |
restore |
Disaster Recovery Restorations | am restore volume, am restore fabric |
cert |
mTLS PKI & Zero-Downtime Rotation | am cert status, am cert rotate, am cert verify-chain |
network |
Fabric Transport, MTU & Latency Mesh | am network ping-mesh, am network mtu-test |
upgrade |
Safe Rolling Software Updates | am upgrade preflight, am upgrade apply --rolling |
auth |
Zero-Trust Operator Authentication | am auth login, am auth session, am auth revoke |
3. Deterministic Exit Codes & Automation
To enable safe, unattended automation in CI/CD runners, Ansible playbooks, and Kubernetes operators, am emits strictly deterministic POSIX exit codes:
| Exit Code | Constant | Condition Meaning | Automation Response |
|---|---|---|---|
0 |
Success |
Operation completed normally with verified consistency. | Continue pipeline. |
1 |
GeneralFailure |
Unhandled internal exception or daemon fault. | Halt & alert operator. |
2 |
InvalidArguments |
Malformed CLI syntax or missing mandatory parameters. | Review command syntax. |
3 |
AuthenticationFailure |
Invalid operator certificate or rejected session key. | Trigger key re-authentication. |
5 |
NetworkFailure |
Target node unreachable or socket handshake timeout. | Retry with backoff. |
7 |
QuorumUnavailable |
Raft MON consensus group has lost quorum. | Trigger urgent cluster alert. |
8 |
ClusterUnhealthy |
Cluster state is degraded or under critical disk pressure. | Run diagnostic triage. |
9 |
ResourceNotFound |
Requested node, volume, pool, or object does not exist. | Verify resource list. |
10 |
Conflict |
State conflict (e.g. node already undergoing maintenance). | Inspect active locks. |
11 |
SafetyCheckFailed |
Preflight check blocked operation to prevent data loss. | Evaluate safety criteria. |
13 |
DataIntegrityFailure |
Checksum or bitrot corruption detected during scrub. | Initiate Cauchy RS repair. |