IBM Code Engine: A Practical Guide to Bash Script Logs

echo "INFO: Starting the data processing job..."

FILENAME="dataset_2025-01-15.csv"
printf "INFO: Processing file: %s\n" "$FILENAME"

In Bash, you can redirect the output of a command to a specific stream. File descriptor 1 represents stdout, and 2 represents stderr. To send a message to stderr, you redirect it using >&2.

# This goes to stdout (normal log)
echo "INFO: Checking for source file..."

# This goes to stderr (error log)
echo "ERROR: Source file not found!" >&2

Why is this so important? Most logging tools, including the Code Engine UI and IBM Log Analysis, can filter or color-code messages based on their stream. By sending errors to stderr, you make it trivial to spot problems without having to read every single line.

A Practical Example: A Simple Backup Script

Let's put this together in a script that could be run as a Code Engine job. This script will copy a file, adding a timestamp, and it will log its progress and any potential errors correctly.

#!/bin/bash

# Exit immediately if a command exits with a non-zero status.
set -e

SOURCE_FILE="/app/data.txt"
BACKUP_DIR="/app/backups"

echo "INFO: Starting backup process for $SOURCE_FILE..."

mkdir -p "$BACKUP_DIR"

if [ ! -f "$SOURCE_FILE" ]; then
  echo "ERROR: Source file not found at $SOURCE_FILE. Cannot create backup." >&2
  exit 1
fi

TIMESTAMP=$(date +%Y%m%d-%H%M%S)
BACKUP_FILE="$BACKUP_DIR/data-$TIMESTAMP.bak"

echo "INFO: Creating backup: $BACKUP_FILE"
cp "$SOURCE_FILE" "$BACKUP_FILE"

# Check if the copy was successful
if [ $? -eq 0 ]; then
  echo "SUCCESS: Backup completed successfully."
else
  echo "ERROR: Failed to copy file to $BACKUP_FILE." >&2
  exit 1
fi

In this example:

• Informational messages about the process starting and succeeding are sent to stdout via echo.
• Critical error messages (file not found, copy failed) are explicitly redirected to stderr with >&2.
• The script exits with a non-zero status code (exit 1) on error, which clearly marks the Code Engine job run as "Failed."

Structuring Your Logs for Clarity

As your scripts become more complex, you'll want more than just simple text messages. Structured logging is the practice of formatting your logs in a consistent, machine-readable way, often using JSON. This makes your logs incredibly easy to search, filter, and analyze in a proper logging tool.

Even without full JSON, you can add structure. A simple improvement is to add a log level and a timestamp to every message.

echo "INFO: $(date -u +%Y-%m-%dT%H:%M:%SZ) - Backup process starting..."

For the ultimate in parsability, you can format your logs as JSON objects. This is more verbose but pays huge dividends when you need to query your logs.

FILENAME="data.txt"
echo "{\"level\": \"INFO\", \"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\", \"message\": \"Processing file\", \"file\": \"$FILENAME\"}"

Here’s a quick comparison:

Log Style	Pros	Cons
Plain Text	Simple to write, easy for humans to read.	Difficult to parse automatically, hard to filter on specific data points.
Structured (JSON)	Machine-readable, powerful querying and filtering, easy integration with log analysis platforms.	More verbose, can be harder for humans to read without tools, requires careful escaping in Bash.

Advanced Techniques: Logging Functions

To avoid repeating logging logic and ensure consistency, it's best practice to create dedicated logging functions in your script. This centralizes your log formatting and makes the main body of your script much cleaner.

#!/bin/bash

# Central logging function
log() {
  local level=$1
  shift
  local message=$@
  echo "[${level}] [$(date -u +%Y-%m-%dT%H:%M:%SZ)] - ${message}"
}

# Helper functions for different levels
log_info() {
  log "INFO" "$@"
}

log_error() {
  log "ERROR" "$@" >&2
}

# --- Main script logic ---
log_info "Starting the application."

FILENAME="/app/data.txt"
if [ ! -f "$FILENAME" ]; then
  log_error "File not found: $FILENAME"
  exit 1
fi

log_info "Successfully validated file presence."

With this approach, you can easily change your log format in one place (the log function) and have it apply to your entire script. The intent of each log message is also clearer (log_info vs. log_error).

How to Access Your Logs in Code Engine

Now that your script is generating beautiful, informative logs, how do you see them?

Via the IBM Cloud CLI

The CLI is the quickest way to get logs for a specific job run. First, you might need to list the job runs to find the name of the one you're interested in, then fetch its logs.

# Ensure you're targeting your Code Engine project
ibmcloud ce project select --name my-project

# List runs for a job named 'my-backup-job'
ibmcloud ce jobrun list --job my-backup-job

# Get logs for a specific job run
ibmcloud ce jobrun logs --name my-backup-job-run-abcde

Via the IBM Cloud Console (UI)

For a more visual approach:

1. Navigate to your Code Engine project in the IBM Cloud Console.
2. In the left-hand menu, click Jobs.
3. Select the job you're interested in (e.g., my-backup-job).
4. Go to the Job runs tab.
5. Click on the name of the specific run you want to inspect.
6. On the job run's details page, you'll find a Logs tab. This will show you the combined stdout and stderr output from your script.

Going Further: Integrating with IBM Log Analysis

The logs you see in the Code Engine UI are transient and have limited search capabilities. For production workloads, you need a centralized, long-term logging solution. This is where IBM Log Analysis comes in.

You can configure your Code Engine project to automatically forward all logs to an IBM Log Analysis instance. This is done in your project's settings under "Logging." Once configured, you gain several powerful advantages:

• Long-Term Storage: Retain logs for days, weeks, or months.
• Powerful Searching: Use a robust query language to search across thousands of logs and filter by level, timestamp, or any field in your structured JSON logs.
• Alerting: Set up alerts to be notified automatically when a specific error (like your `ERROR: File not found` message) appears in the logs.

Conclusion: From Silent Scripts to Insightful Logs

Effective logging is not an afterthought; it's a core component of building reliable, maintainable applications in a serverless environment like IBM Code Engine. By understanding and using the standard output and error streams, you can transform your Bash scripts from opaque processes into transparent, debuggable workflows.

Start simple with echo, distinguish between information and errors with >&2, and graduate to structured logging with functions and JSON as your needs grow. By mastering these techniques, you'll spend less time guessing and more time building, confident that you have the visibility you need to keep your jobs running smoothly.