Skip to main content

Ozone Shell (ozone sh)

The Ozone Shell, invoked via the ozone sh command, is a primary command-line interface (CLI) for interacting with Apache Ozone. It allows administrators and users to manage resources and perform basic data operations directly against the Ozone Manager (OM) using Ozone's native RPC protocol.

Overview​

The Ozone Shell provides a structured way to manage the Ozone namespace and related entities. It's essential for administrative tasks and useful for basic data interaction and scripting.

Key Capabilities:

  • Volume Management: Create, delete, list volumes; get volume information; set/clear owner and quotas (space and namespace).
  • Bucket Management: Create, delete, list buckets; get bucket information; set/clear properties like versioning, replication, layout, encryption key, owner, and quotas; link buckets.
  • Key (Object) Management: Put (upload), get (download), delete, list keys; get key information; manage multipart uploads (though less common via shell).
  • ACL Management: Set, get, add, and remove Access Control Lists (ACLs) for volumes, buckets, keys, and prefixes.
  • Snapshot Management: Create, delete, list, rename snapshots; get snapshot info; manage snapshot diffs.
  • S3 Credentials Management: Manage S3 access keys and secrets (primarily for multi-tenancy).
  • Administrative Actions: Perform cluster-level administrative tasks (e.g., decommissioning, safemode management - though often under ozone admin).

Syntax​

The general syntax follows a consistent pattern:

ozone sh <resource> <action> [options] <path_or_arguments>
  • <resource>: The type of Ozone entity you are interacting with (e.g., volume, bucket, key, snapshot, s3, prefix).
  • <action>: The specific operation to perform on the resource (e.g., create, list, info, delete, put, get, setacl, setquota).
  • [options]: Command-specific flags that modify the action's behavior (e.g., --quota, --layout, --acl, --type, --replication, -k, -r).
  • <path_or_arguments>: The Ozone URI path to the target resource (e.g., /volumeName, /volumeName/bucketName, /volumeName/bucketName/keyName) or other required arguments depending on the command.

Getting Help:

You can get help for any command level:

# Help for all shell commands
ozone sh --help

# Help for volume commands
ozone sh volume --help

# Help for creating a volume
ozone sh volume create --help

Common Commands and Examples​

Here are examples grouped by resource type, drawing from common use cases and test scenarios.

Volume Operations​

# Create a volume
ozone sh volume create /data-vol

# Create a volume with quotas
ozone sh volume create --space-quota 5GB --namespace-quota 10000 /quota-vol

# List volumes
ozone sh volume list /
# Example Output:
# [ {
#   "metadata" : { },
#   "name" : "data-vol",
#   "admin" : "hadoop",
#   "owner" : "hadoop",
#   "creationTime" : "2024-04-11T18:00:00.000Z",
#   "modificationTime" : "2024-04-11T18:00:00.000Z",
#   "quotaInBytes" : -1,
#   "quotaInNamespace" : -1,
#   "usedNamespace" : 0
# }, ... ]

# Get volume information
ozone sh volume info /data-vol

# Set volume quota
ozone sh volume setquota --space-quota 10GB /data-vol

# Clear volume namespace quota
ozone sh volume clrquota --namespace-quota /data-vol

# Delete an empty volume
ozone sh volume delete /data-vol

# Recursively delete a volume and its contents (USE WITH CAUTION!)
ozone sh volume delete -r /quota-vol

Bucket Operations​

# Create a bucket (defaults: OBS layout, RATIS/THREE replication)
ozone sh bucket create /data-vol/raw-data

# Create an FSO bucket
ozone sh bucket create --layout FILE_SYSTEM_OPTIMIZED /data-vol/fso-data

# Create an EC bucket (requires pre-defined EC policy 'rs-6-3-1024k')
ozone sh bucket create --replication rs-6-3-1024k --type EC /data-vol/ec-data

# Create an encrypted bucket (requires KMS setup and key 'my-bek')
ozone sh bucket create -k my-bek /data-vol/secure-data

# List buckets in a volume
ozone sh bucket list /data-vol

# Get bucket information
ozone sh bucket info /data-vol/raw-data

# Set bucket quota
ozone sh bucket setquota --namespace-quota 50000 /data-vol/raw-data

# Set bucket replication config (change existing bucket to EC)
ozone sh bucket set-replication-config /data-vol/raw-data --type EC --replication rs-3-2-1024k

# Link an existing bucket into another path (e.g., S3 volume)
ozone sh bucket link /data-vol/raw-data /s3v/linked-raw-data

# Delete an empty bucket
ozone sh bucket delete /data-vol/raw-data

# Recursively delete a bucket (USE WITH CAUTION!)
ozone sh bucket delete -r /data-vol/fso-data

Key (Object) Operations​

# Upload (put) a local file to Ozone
# Note the argument order: <destination_ozone_path> <source_local_path>
ozone sh key put /data-vol/raw-data/logs/app.log /local/path/app.log

# Upload a file with specific EC replication
ozone sh key put /data-vol/ec-data/archive.zip /local/path/archive.zip --type EC --replication rs-6-3-1024k

# Download (get) a key from Ozone to a local directory
ozone sh key get /data-vol/raw-data/logs/app.log /tmp/

# Get key information (metadata)
ozone sh key info /data-vol/raw-data/logs/app.log
# Example Output:
# {
#   "volumeName" : "data-vol",
#   "bucketName" : "raw-data",
#   "name" : "logs/app.log",
#   "dataSize" : 10240,
#   "creationTime" : "2024-04-11T18:15:00.000Z",
#   "modificationTime" : "2024-04-11T18:15:00.000Z",
#   "replicationConfig" : {
#     "replicationType" : "RATIS",
#     "requiredNodes" : 3,
#     "replication" : "THREE"
#   },
#   "ozoneKeyLocations" : [ ... ],
#   "metadata" : { },
#   "fileEncryptionInfo" : null,
#   "updateID" : 12345,
#   "file" : false
# }

# List keys within a bucket (or prefix)
ozone sh key list /data-vol/raw-data/logs/

# Delete a key
ozone sh key delete /data-vol/raw-data/logs/app.log

ACL Operations​

# Set ACLs (replaces existing ACLs) on a bucket
ozone sh bucket setacl /data-vol/secure-data --acl user:dev_user:rw,group:dev_group:r

# Get ACLs of a bucket
ozone sh bucket getacl /data-vol/secure-data

# Add an ACL (appends to existing ACLs) to a volume
ozone sh volume addacl /data-vol --acl user:test_user:rl

# Remove a specific ACL permission from a key
ozone sh key removeacl /data-vol/raw-data/logs/app.log --acl user:test_user:w

# Add a default ACL to a bucket (applied to new keys created within)
ozone sh bucket addacl /data-vol/fso-data --acl default:group:analytics:r

Snapshot Operations​

# Create a snapshot of a bucket
ozone sh snapshot create /data-vol/raw-data my-snapshot-`date +%Y%m%d`

# List snapshots for a bucket
ozone sh snapshot list /data-vol/raw-data

# Get info about a specific snapshot
ozone sh snapshot info /data-vol/raw-data my-snapshot-20240411

# Delete a snapshot
ozone sh snapshot delete /data-vol/raw-data my-snapshot-20240411

# Get differences between two snapshots
ozone sh snapshot diff /data-vol/raw-data my-snapshot-20240410 my-snapshot-20240411

When to Use​

The Ozone Shell (ozone sh) is the recommended tool for:

  • Cluster Administrators: Performing setup, configuration changes, managing volumes/buckets, setting quotas and ACLs, managing snapshots.
  • Scripting: Automating administrative or basic data management tasks.
  • Basic Data Interaction: Simple uploads, downloads, listings, and deletions for exploration or troubleshooting.
  • Accessing Ozone-Specific Features: Managing features not exposed via other interfaces (e.g., certain snapshot operations, detailed ACL management).

It is generally not the ideal interface for:

  • High-performance, parallel data access within applications (use the Java API or ofs/s3a).
  • Complex filesystem operations requiring strong atomicity guarantees (use ofs with FSO buckets).
  • Interfacing with applications designed for the S3 API (use the S3 Gateway or s3a).