Session 2: Essential Command Line Tools

Master the core command-line utilities that form the backbone of IoT development workflows. These tools are essential for file manipulation, text processing, system navigation, and automation tasks that you'll use daily as an IoT professional.

Duration 2 hours
Module Linux Basics
Session 2 of 4
Difficulty Beginner

Session Learning Objectives

By the end of this session, you will be able to:

Master File Operations

Use essential commands like ls, cp, mv, rm, mkdir, and rmdir for professional file and directory management in IoT projects.

Process Text Efficiently

Utilize text processing tools like cat, less, head, tail, and grep for analyzing configuration files, logs, and sensor data.

Search and Locate Files

Find files and programs using find, locate, which, and whereis commands to navigate complex IoT system structures.

Handle Archives Professionally

Create and extract compressed archives using tar, gzip, and zip for deployment, backup, and distribution workflows.

1. File Operations - Your Digital Toolkit

Listing Files and Directories (ls) - Your System Explorer

The ls command is your primary tool for exploring and understanding file systems. In IoT development, you'll constantly need to examine directory contents, check file permissions, and understand system organization. Think of ls as your digital file explorer with superpowers.

Why This Matters for IoT: IoT systems often have complex directory structures with configuration files, device drivers, log files, and application code scattered across the system. Mastering ls helps you navigate these systems efficiently and troubleshoot issues quickly.

# Basic listing - your starting point
ls    # List files in current directory
ls /home/user    # List files in specific directory
ls -l    # Long format (detailed information) - most important option
ls -la    # Long format including hidden files (files starting with .)
ls -lh    # Human-readable file sizes (shows KB, MB, GB instead of bytes)
ls -lt    # Sort by modification time (newest first) - great for finding recent changes
ls -lS    # Sort by file size (largest first) - useful for finding space hogs
ls -lr    # Reverse order - oldest first or smallest first
ls -R    # Recursive listing (shows subdirectories) - be careful with large directories

# IoT-specific examples that you'll use regularly
ls -la /dev/tty*    # List serial devices (Arduino, sensors, GPS modules)
ls -lh /var/log/    # Check log file sizes (important for disk space management)
ls -lt /etc/systemd/system/    # Recent system service changes
ls -la /sys/class/gpio/    # GPIO pin interfaces (Raspberry Pi, embedded systems)
ls -lh /opt/    # Third-party IoT software installations

Understanding ls -l Output - Reading the Matrix

The long format output contains crucial information for system administration and security. Let's decode what each part means:

# Example output breakdown:
# drwxr-xr-x 2 user group 4096 Jan 15 10:30 documents
# |    |    |    |    |    |    |    |    |    |
# |    |    |    |    |    |    |    |    |    └── File/directory name
# |    |    |    |    |    |    |    |    └── Last modification date/time
# |    |    |    |    |    |    |    └── File size in bytes
# |    |    |    |    |    |    └── Group owner
# |    |    |    |    |    └── User owner  
# |    |    |    |    └── Number of hard links
# |    |    |    └── Other permissions (everyone else)
# |    |    └── Group permissions
# |    └── Owner permissions
# └── File type indicator

# File type indicators (first character):
# d = directory (you can cd into it)
# - = regular file (text, binary, executable)
# l = symbolic link (shortcut to another file)
# b = block device (hard drives, USB storage)
# c = character device (serial ports, terminals)
# p = named pipe (inter-process communication)
# s = socket (network communication)

# Permission format: rwx (read, write, execute)
# r = read permission (4 in numeric)
# w = write permission (2 in numeric)  
# x = execute permission (1 in numeric)
# - = no permission (0 in numeric)

Creating and Removing Directories - Building Your Workspace

Directory management is fundamental to organizing IoT projects, system configurations, and data storage. Professional IoT developers maintain clean, logical directory structures that make projects maintainable and scalable.

# Creating directories - building your project structure
mkdir project    # Create single directory
mkdir -p project/src/sensors    # Create nested directories (parents created automatically)
mkdir dir1 dir2 dir3    # Create multiple directories at once
mkdir -m 755 public_dir    # Create with specific permissions (755 = rwxr-xr-x)
mkdir -p ~/iot-projects/{hardware,software,documentation}  # Brace expansion

# Professional IoT project structure example
mkdir -p iot-temperature-monitor/{
    src/{sensors,networking,data-processing},
    config/{development,production,testing},
    docs/{api,hardware,deployment},
    tests/{unit,integration,hardware},
    logs/{application,system,sensors},
    scripts/{deployment,maintenance,backup}
}

# Removing directories - cleaning up safely
rmdir empty_directory    # Remove empty directory only (safe)
rm -r directory_with_files    # Remove directory and all contents (be careful!)
rm -rf dangerous_removal    # Force remove without prompts (very dangerous!)
rm -ri directory_name    # Interactive removal (asks before each file)

# Safe removal practices for IoT projects
ls -la target_directory/    # Always check contents first
rm -ri old_project/    # Use interactive mode for important directories
find old_logs/ -name "*.log" -mtime +30 -delete  # Remove old log files safely

Best Practices for IoT Directory Organization

Source code organized by function (sensors/, networking/, ui/)
Configuration files separated by environment (dev/, prod/, test/)
Documentation including API docs, hardware specs, deployment guides
Test files organized by type (unit/, integration/, hardware/)
Log files separated by component and rotated regularly
Automation scripts for deployment, backup, and maintenance

Copying and Moving Files - Data Management Essentials

File manipulation is critical for managing code, configurations, firmware updates, and data files in IoT systems. Understanding these commands prevents data loss and enables efficient workflow automation.

# Copying files - preserving and duplicating data
cp file.txt backup.txt    # Copy file to new name
cp file.txt /backup/    # Copy to directory (keeps original name)
cp -r project/ project_backup/  # Copy directory recursively (includes subdirectories)
cp -p file.txt backup.txt    # Preserve permissions, ownership, and timestamps
cp -u source.txt dest.txt    # Copy only if source is newer (update mode)
cp *.txt /backup/    # Copy all .txt files using wildcards
cp -v source dest    # Verbose mode (shows what's being copied)

# Advanced copying for IoT workflows
cp -a /etc/iot/ /backup/iot-config-$(date +%Y%m%d)/  # Archive mode with timestamp
cp --parents src/sensors/temp.py /backup/  # Preserve directory structure
rsync -av --progress large_dataset/ /backup/  # Better for large files

# Moving/renaming files - reorganizing and updating
mv old_name.txt new_name.txt    # Rename file (same directory)
mv file.txt /new/location/    # Move file to different directory
mv *.log /var/log/iot/    # Move all log files
mv project/ /opt/iot-applications/    # Move entire directory
mv temp_sensor_{old,new}.py    # Rename using brace expansion

# IoT configuration management examples
cp /etc/network/interfaces /etc/network/interfaces.backup.$(date +%Y%m%d)
mv sensor_config.json /etc/iot/sensors/
cp -r firmware_v1.2/ firmware_v1.3/  # Create new firmware version
mv *.hex /opt/iot/firmware/latest/    # Deploy firmware files

# Batch operations for IoT data management
for file in sensor_data_*.csv; do
    mv "$file" "processed_$file"
done

Removing Files - Safe Deletion Practices

File deletion in IoT systems requires extra caution because you might be removing critical configuration files, firmware, or irreplaceable sensor data. Always follow safe practices to prevent system failures.

# Safe file removal practices
rm file.txt    # Remove single file
rm file1.txt file2.txt    # Remove multiple files
rm *.tmp    # Remove all temporary files
rm -i important.txt    # Interactive removal (asks for confirmation)
rm -f stubborn_file.txt    # Force removal (ignores write protection)
rm -rf directory/    # Remove directory and contents (DANGEROUS!)

# Professional safety practices
ls -la *.log    # Always list files before removing
rm -i *.log    # Use interactive mode for important files
find /tmp -name "*.tmp" -mtime +1 -ls    # List files before deletion
find /tmp -name "*.tmp" -mtime +1 -delete # Delete after verification

# IoT-specific cleanup examples
# Clean old sensor data (keep last 30 days)
find /var/log/sensors/ -name "*.log" -mtime +30 -ls
find /var/log/sensors/ -name "*.log" -mtime +30 -delete

# Remove temporary firmware files
rm -f /tmp/firmware_*.tmp
rm -rf /tmp/iot_build_*/

# Clean cache files safely
find /var/cache/iot -type f -mtime +7 -name "*.cache" -delete

# Emergency recovery - if you accidentally delete something
# Immediately stop using the system and use recovery tools
# testdisk, photorec, or extundelete might help recover files

Pro Tips for File Operations

# Use aliases for safety and efficiency
alias ll='ls -la'
alias la='ls -la'
alias rm='rm -i'    # Always ask before removing
alias cp='cp -i'    # Ask before overwriting
alias mv='mv -i'    # Ask before overwriting

# Useful file operation combinations
ls -la && pwd    # Show files and current location
cp important.conf{,.backup}    # Quick backup (creates important.conf.backup)
mkdir -p ~/backups/$(date +%Y/%m/%d)  # Create dated backup directory

# File operation with error handling
cp source.txt dest.txt && echo "Copy successful" || echo "Copy failed"
rm -f temp_* && echo "Cleanup complete"

# Working with files containing spaces or special characters
ls -la "file with spaces.txt"
cp "sensor data.csv" sensor_data.csv
rm -f "temp file (1).txt"

2. Text Processing Tools - Your Data Analysis Arsenal

Viewing File Contents - Reading Your System

Text processing is fundamental to IoT development because most configuration files, log files, and data files are text-based. These tools help you examine sensor data, debug configuration issues, and monitor system behavior.

# Display entire file contents
cat file.txt    # Show complete file content (good for small files)
cat file1.txt file2.txt    # Concatenate multiple files
cat -n file.txt    # Show with line numbers (helpful for debugging)
cat -b file.txt    # Number only non-blank lines
cat -A file.txt    # Show all characters including hidden ones

# View large files efficiently (essential for log analysis)
less file.txt    # Paginated viewing with search capabilities
    # less commands: q=quit, /=search, n=next match, G=end, g=beginning
more file.txt    # Simple pager (less powerful than less)
head file.txt    # First 10 lines (quick file preview)
head -n 20 file.txt    # First 20 lines
tail file.txt    # Last 10 lines (recent log entries)
tail -n 5 file.txt    # Last 5 lines
tail -f /var/log/syslog    # Follow file changes in real-time (live monitoring)

# IoT monitoring and debugging examples
tail -f /var/log/iot/sensor.log    # Monitor sensor logs in real-time
head -n 50 /etc/iot/device_config.json    # Check configuration file start
cat /proc/cpuinfo | head -n 20    # System information for IoT device specs
tail -n 100 /var/log/mqtt.log | grep ERROR    # Recent MQTT errors

# Advanced text viewing for IoT data analysis
zcat compressed_sensor_data.gz | head -20    # View compressed files
tail -f /var/log/syslog | grep --line-buffered "temperature"  # Live temperature monitoring
watch "tail -n 10 /var/log/sensors.log"    # Refresh view every 2 seconds

Searching Text with grep - Your Pattern Detective

grep is one of the most powerful tools for IoT developers. You'll use it constantly to find specific information in log files, search for configuration parameters, debug network issues, and analyze sensor data patterns.

# Basic searching - finding needles in haystacks
grep "error" logfile.txt    # Find lines containing "error"
grep -i "ERROR" logfile.txt    # Case-insensitive search (matches error, Error, ERROR)
grep -n "config" file.txt    # Show line numbers with matches
grep -v "debug" logfile.txt    # Show lines NOT containing "debug" (inverse match)
grep -c "warning" logfile.txt    # Count matching lines (useful for statistics)
grep -r "TODO" project/    # Recursive search in directory (find all TODOs)
grep -l "import" *.py    # List files containing pattern (not the lines)

# Advanced pattern matching for IoT analysis
grep "^Error" logfile.txt    # Lines starting with "Error"
grep "\.json$" filelist.txt    # Lines ending with ".json"
grep -E "(error|warning|critical)" log.txt  # Multiple patterns (extended regex)
grep -A 3 -B 3 "error" log.txt    # Show 3 lines before and after match (context)
grep -w "temp" sensor.log    # Match whole words only (not "temperature")
grep -o "temperature:[0-9]*" data.log    # Show only the matching part

# IoT-specific search examples that you'll use daily
grep "temperature" /var/log/sensors.log    # Find temperature readings
grep -i "wifi.*connected" /var/log/network.log    # WiFi connection events
grep -r "192.168" /etc/network/    # Find IP configurations
ps aux | grep "iot_daemon"    # Find IoT processes
grep -E "sensor_[0-9]+" config.json    # Find sensor configurations
journalctl | grep -i "mqtt" | tail -20    # Recent MQTT system messages

# Complex IoT data analysis patterns
grep -E "temperature:[0-9]{2,3}" sensor.log    # Temperatures with 2-3 digits
grep -P "(?<=temp:)\d+\.\d+" data.csv    # Perl regex for decimal temperatures
grep -A 5 "ERROR" /var/log/iot.log | grep -v "^--$"  # Errors with context, clean output

Text Processing Pipeline - Combining Tools for Power

The real power of Linux text processing comes from combining commands using pipes (|). This allows you to create sophisticated data processing workflows that are essential for IoT data analysis and system monitoring.

# Basic pipeline concepts
cat /var/log/syslog | grep "error" | head -10    # Find recent errors
ls -la | grep "\.txt$"    # List only text files
ps aux | grep "python" | wc -l    # Count Python processes
cat sensor_data.csv | cut -d',' -f2 | sort | uniq    # Extract and analyze CSV column

# IoT data processing pipelines
# Analyze sensor data patterns
cat sensor_readings.csv | grep "temperature" | cut -d',' -f3 | sort -n | tail -10
# Find top 10 highest temperatures

# Monitor network connections for IoT devices
netstat -an | grep ":1883" | wc -l    # Count MQTT connections
ss -tuln | grep ":80" | awk '{print $5}' | cut -d':' -f1 | sort | uniq -c
# Analyze HTTP connections by IP

# System monitoring for IoT devices
journalctl --since "1 hour ago" | grep -i error | wc -l    # Error count last hour
ps aux | grep -E "(sensor|iot|mqtt)" | awk '{print $3}' | paste -sd+ | bc
# Total CPU usage of IoT processes

# Log analysis for troubleshooting
tail -1000 /var/log/iot.log | grep -E "(error|warning)" | cut -d' ' -f1-3 | sort | uniq -c
# Count errors by timestamp pattern

# Data extraction and transformation
cat device_status.json | jq '.sensors[] | select(.type=="temperature") | .value'
# Extract temperature values from JSON (requires jq)

# Advanced IoT monitoring pipeline
tail -f /var/log/sensors.log | grep --line-buffered "temperature" | \
while read line; do
    temp=$(echo $line | grep -o "[0-9]*\.[0-9]*")
    if (( $(echo "$temp > 30.0" | bc -l) )); then
        echo "HIGH TEMP ALERT: $temp°C at $(date)"
    fi
done

Additional Text Processing Tools

These additional tools expand your text processing capabilities for complex IoT data analysis tasks.

# cut - extract columns from structured data
cut -d',' -f1,3 sensor_data.csv    # Extract columns 1 and 3 from CSV
cut -c1-10 file.txt    # Extract characters 1-10 from each line
ps aux | cut -d' ' -f1,11-    # Extract username and command

# sort - organize data for analysis
sort file.txt    # Sort lines alphabetically
sort -n numbers.txt    # Sort numerically
sort -r file.txt    # Reverse sort
sort -k2 -n data.csv    # Sort by second column numerically
sort -u file.txt    # Sort and remove duplicates

# uniq - find unique or duplicate entries
uniq file.txt    # Remove consecutive duplicate lines
uniq -c file.txt    # Count occurrences of each line
uniq -d file.txt    # Show only duplicate lines
sort file.txt | uniq -c | sort -nr    # Count and sort by frequency

# wc - count lines, words, characters
wc file.txt    # Count lines, words, characters
wc -l file.txt    # Count lines only
wc -w file.txt    # Count words only
ls | wc -l    # Count files in directory

# IoT data analysis examples
cut -d',' -f2 sensor_data.csv | sort -n | uniq -c    # Frequency of sensor values
grep "temperature" logs/*.log | wc -l    # Total temperature readings
sort -k3 -n device_list.txt | tail -5    # Top 5 devices by third column

3. File Searching and Location - Your System GPS

Finding Files with find - The Ultimate Search Tool

The find command is incredibly powerful for locating files in complex IoT systems. You'll use it to find configuration files, locate log files, search for specific file types, and perform system maintenance tasks.

# Basic find syntax and patterns
find /path -name "filename"    # Find by exact name
find /home -name "*.txt"    # Find by pattern (all .txt files)
find . -name "config*"    # Find in current directory (files starting with "config")
find /etc -iname "*.CONF"    # Case-insensitive search
find / -name "sensor*" 2>/dev/null    # Search entire system, ignore permission errors

# Find by file properties (essential for system administration)
find /var -size +100M    # Files larger than 100MB (find space hogs)
find /var/log -size +50M -name "*.log"    # Large log files
find /tmp -mtime -7    # Files modified in last 7 days
find /home -mtime +30    # Files older than 30 days
find /etc -atime -1    # Files accessed in last 24 hours
find /home -type f    # Files only (not directories)
find /home -type d    # Directories only
find /dev -type c    # Character devices (serial ports, etc.)
find /dev -type b    # Block devices (hard drives, etc.)

# Advanced find operations for IoT management
find /var/log -name "*.log" -exec ls -lh {} \;    # List details of found files
find /tmp -name "*.tmp" -delete    # Delete temporary files
find . -name "*.py" -exec grep -l "import serial" {} \;    # Find Python files using serial
find /etc -name "*network*" -exec cp {} /backup/ \;    # Backup network configs

# IoT-specific searches you'll use regularly
find /etc -name "*iot*" -o -name "*sensor*"    # Find IoT-related configs
find /var/log -name "*sensor*" -mtime -1    # Recent sensor logs
find /home -name "*.ino" -o -name "*.cpp"    # Arduino source files
find /sys -name "*gpio*" -type d    # GPIO device directories
find /dev -name "ttyUSB*" -o -name "ttyACM*"    # USB serial devices
find /opt -name "*mqtt*" -type d    # MQTT software installations

# Complex find expressions for system maintenance
find /var/log -name "*.log" -size +10M -mtime +7 -exec gzip {} \;
# Compress old large log files

find /home -name "*.tmp" -o -name "*.cache" -o -name "*.bak" | xargs rm -f
# Clean up temporary files

find /etc -name "*.conf" -exec grep -l "192.168" {} \; 2>/dev/null
# Find config files with local IP addresses

Quick File Location Tools - Speed and Efficiency

These tools provide faster alternatives for common file location tasks, especially useful when you know roughly what you're looking for.

# locate command - lightning-fast database search
locate filename.txt    # Quick search in pre-built database
locate "*.conf" | head -10    # Find config files (first 10 results)
locate -i arduino    # Case-insensitive search
locate -c "*.py"    # Count matching files
sudo updatedb    # Update locate database (run as root, usually automated)

# Limitations of locate:
# - Database updated daily (may miss recent files)
# - Shows files that may have been deleted since last update
# - Very fast but not always current

# which and whereis - finding executables and documentation
which python3    # Find executable location in PATH
which -a python    # Show all instances in PATH
whereis python3    # Show binary, source, and manual page locations
type python3    # Show command type and location (shell builtin)
command -v python3    # POSIX-compliant way to find commands

# IoT development tool location examples
which gcc    # Find C compiler
which pip3    # Find Python package manager
which node    # Find Node.js interpreter
whereis systemctl    # Find systemd control command
locate arduino | grep bin    # Find Arduino IDE installation
which mosquitto_pub    # Find MQTT publishing tool
type docker    # Check if Docker is available

# Practical workflow examples
# Check if required tools are installed
for tool in gcc python3 pip3 git; do
    if which $tool > /dev/null; then
        echo "$tool: $(which $tool)"
    else
        echo "$tool: NOT FOUND"
    fi
done

# Find all Python installations
whereis python | tr ' ' '\n' | grep bin

Advanced Search Techniques

# Combining search tools for complex queries
find /etc -name "*.conf" | xargs grep -l "wifi"    # Config files mentioning wifi
locate "*.py" | xargs grep -l "import gpio"    # Python files using GPIO

# Search with regular expressions
find /var/log -regex ".*sensor[0-9]+\.log"    # Sensor logs with numbers
grep -r --include="*.json" "temperature" /etc/    # JSON files with temperature

# Performance considerations
find /large/directory -name "*.txt" -print0 | xargs -0 ls -la
# Handle filenames with spaces safely

# Search and execute complex operations
find /var/log -name "*.log" -mtime +30 -exec sh -c 'echo "Archiving $1"; gzip "$1"' _ {} \;
# Archive old log files with status messages

4. Archive and Compression - Your Data Management System

Working with tar Archives - The Universal Container

tar (Tape ARchive) is essential for IoT development workflows. You'll use it for backing up projects, distributing firmware, creating deployment packages, and managing system files. Understanding tar is crucial for professional IoT development.

# Creating archives - packaging your work
tar -cf archive.tar files/    # Create tar archive (uncompressed)
tar -czf archive.tar.gz files/    # Create gzip compressed archive (most common)
tar -cjf archive.tar.bz2 files/    # Create bzip2 compressed archive (better compression)
tar -cJf archive.tar.xz files/    # Create xz compressed archive (best compression)

# Professional archive naming with timestamps
tar -czf backup-$(date +%Y%m%d).tar.gz project/
tar -czf iot-firmware-v$(cat version.txt)-$(date +%Y%m%d_%H%M%S).tar.gz firmware/

# Extracting archives - unpacking received files
tar -tf archive.tar    # List contents without extracting (preview)
tar -xf archive.tar    # Extract archive to current directory
tar -xzf archive.tar.gz    # Extract gzip compressed archive
tar -xjf archive.tar.bz2    # Extract bzip2 compressed archive
tar -xzf archive.tar.gz -C /opt/    # Extract to specific directory

# Advanced tar options for professional use
tar -czf project.tar.gz --exclude="*.tmp" --exclude="node_modules" project/
# Exclude temporary files and dependencies

tar -czf backup.tar.gz --exclude-vcs project/
# Exclude version control files (.git, .svn, etc.)

tar -xzf archive.tar.gz --strip-components=1
# Remove top-level directory when extracting

tar -czf - project/ | ssh user@remote "cat > remote_backup.tar.gz"
# Create archive and send directly to remote server

# IoT project archiving examples
tar -czf iot_temperature_monitor_v1.2.tar.gz \
    --exclude="*.log" \
    --exclude="__pycache__" \
    --exclude="*.pyc" \
    iot_temperature_monitor/

tar -czf sensor_configs_backup_$(date +%Y%m%d).tar.gz /etc/sensors/
tar -xzf esp32_libraries.tar.gz -C ~/Arduino/libraries/
tar -czf firmware_release.tar.gz --exclude="*.debug" firmware/

Compression Tools - Optimizing Storage and Transfer

Different compression tools serve different purposes in IoT development. Understanding when to use each tool helps optimize storage space and transfer times.

# gzip compression - fast and widely supported
gzip large_file.txt    # Compress file (replaces original with .gz)
gzip -k large_file.txt    # Keep original file (-k = keep)
gzip -9 file.txt    # Maximum compression (slower)
gzip -1 file.txt    # Fastest compression (larger file)
gunzip file.txt.gz    # Decompress file
zcat file.txt.gz    # View compressed file without extracting
zgrep "pattern" file.txt.gz    # Search in compressed file

# zip archives - cross-platform compatibility
zip archive.zip file1.txt file2.txt    # Create zip archive
zip -r project.zip project/    # Recursive zip (include subdirectories)
zip -9 -r high_compression.zip project/    # Maximum compression
unzip archive.zip    # Extract zip file
unzip -l archive.zip    # List zip contents without extracting
unzip archive.zip -d /target/directory/    # Extract to specific directory
unzip -q archive.zip    # Quiet extraction (no output)

# Advanced compression for IoT workflows
# Compress log files to save space
find /var/log -name "*.log" -mtime +7 -exec gzip {} \;

# Create deployment packages
zip -r iot_dashboard_$(date +%Y%m%d).zip \
    dashboard/ \
    -x "dashboard/node_modules/*" \
    -x "dashboard/*.log"

# Compress sensor data for archival
gzip -9 sensor_data_$(date +%Y%m%d).csv

# IoT data management examples
gzip /var/log/sensor_data_old.csv    # Compress old sensor data
zip -r iot_project_$(date +%Y%m%d).zip ~/iot_project/ \
    -x "*/.*" -x "*/__pycache__/*" -x "*/node_modules/*"
unzip arduino_libraries.zip -d ~/Arduino/libraries/
tar -czf daily_backup.tar.gz /etc/iot/ /var/log/iot/ /opt/iot/

# Automated backup script example
#!/bin/bash
BACKUP_DIR="/backup/iot"
DATE=$(date +%Y%m%d_%H%M%S)
mkdir -p "$BACKUP_DIR"
tar -czf "$BACKUP_DIR/iot_backup_$DATE.tar.gz" \
    --exclude="*.log" \
    --exclude="*.tmp" \
    /etc/iot/ /opt/iot/ /var/lib/iot/

Archive Management Best Practices

Use descriptive names with dates: project_v1.2_20240115.tar.gz
gzip for speed, bzip2 for size, xz for maximum compression
Always exclude logs, temp files, and build artifacts
Test archives with -t option before relying on them
Store backups on different devices/locations for safety
Use scripts and cron jobs for regular backup tasks

5. Command History and Shortcuts - Efficiency Mastery

Command History Management - Your Command Memory

Efficient command-line usage is crucial for IoT development productivity. Mastering history and shortcuts can save hours of typing and reduce errors in complex command sequences.

# History commands - accessing your command memory
history    # Show complete command history
history | grep "git"    # Search command history for specific commands
history | tail -20    # Show last 20 commands
history -c    # Clear current session history
history -w    # Write current session to history file

# History expansion - reusing previous commands
!!    # Repeat last command (very useful)
!n    # Repeat command number n (from history output)
!string    # Repeat last command starting with string
!?pattern?    # Repeat last command containing pattern
^old^new    # Replace 'old' with 'new' in last command
!!:p    # Print last command without executing (preview)

# Advanced history techniques
sudo !!    # Run last command with sudo (extremely common)
!ssh:p    # Preview last ssh command
!grep -2    # Run second-to-last grep command
echo !$    # Use last argument from previous command
echo !^    # Use first argument from previous command
echo !*    # Use all arguments from previous command

# IoT development workflow examples
history | grep "mosquitto"    # Find MQTT broker commands
!ssh    # Repeat last SSH connection to IoT device
sudo !!    # Run last command with root privileges
!scp    # Repeat last file transfer command
history | grep "systemctl" | tail -5    # Recent service management commands

Keyboard Shortcuts - Speed of Light Navigation

These shortcuts are essential for professional command-line efficiency. Practice them until they become muscle memory.

# Essential keyboard shortcuts (memorize these!)
Ctrl+R    # Reverse search through history (most important shortcut)
Ctrl+P / Up Arrow    # Previous command in history
Ctrl+N / Down Arrow    # Next command in history
Ctrl+A    # Move cursor to beginning of line
Ctrl+E    # Move cursor to end of line
Ctrl+U    # Clear line before cursor
Ctrl+K    # Clear line after cursor
Ctrl+W    # Delete word before cursor
Ctrl+L    # Clear screen (same as 'clear' command)
Ctrl+C    # Interrupt current command
Ctrl+Z    # Suspend current command
Ctrl+D    # Exit shell or end input

# Advanced cursor movement
Alt+F    # Move forward one word
Alt+B    # Move backward one word
Alt+D    # Delete word after cursor
Alt+Backspace    # Delete word before cursor
Ctrl+Y    # Paste last deleted text
Ctrl+_    # Undo last edit

# Professional workflow shortcuts
Ctrl+R, type "ssh", Enter    # Quick search and execute SSH command
Ctrl+A, "sudo ", Ctrl+E    # Add sudo to beginning of current command
Ctrl+U, type new command    # Clear and start new command
Ctrl+L    # Clear screen while keeping current command

Tab Completion and Wildcards - Smart Typing

Tab completion and wildcards reduce typing errors and speed up command entry significantly. They're essential for working with complex IoT file structures.

# Tab completion - let the system help you type
ls /etc/net[TAB]    # Complete to /etc/network/
systemctl status iot[TAB]    # Complete service names
cd ~/proj[TAB]    # Complete directory names
vim /var/log/[TAB][TAB]    # Show all options in directory
python3 sensor_[TAB]    # Complete script names

# Double-tab for multiple options
ls /usr/bin/py[TAB][TAB]    # Show all commands starting with "py"

# Wildcards - pattern matching for efficiency
ls *.txt    # All files ending with .txt
ls sensor_*.log    # Files matching pattern
cp config_?.json backup/    # Single character wildcard (config_1.json, config_a.json)
rm temp[0-9].txt    # Files with numbers 0-9 (temp1.txt, temp2.txt, etc.)
ls {*.py,*.cpp,*.h}    # Multiple patterns (Python, C++, header files)
ls file[!0-9].txt    # Files NOT containing numbers

# Brace expansion - generate multiple arguments
mkdir project/{src,docs,tests,config}   # Create multiple directories
cp file.txt{,.backup}    # Create backup copy (file.txt.backup)
echo {1..10}    # Generate sequence: 1 2 3 4 5 6 7 8 9 10
echo {a..z}    # Generate alphabet
echo {01..12}    # Generate: 01 02 03 ... 12

# IoT-specific wildcard examples
ls /dev/ttyUSB*    # List all USB serial devices
cp sensor_config_{dev,prod,test}.json /etc/iot/
find . -name "*.{ino,cpp,h}"    # Arduino source files
ls /sys/class/gpio/gpio{18,19,20}/    # Specific GPIO pins
rm /tmp/iot_temp_*    # Remove all IoT temporary files

# Advanced pattern matching
ls sensor_data_[0-9][0-9][0-9][0-9]*.csv    # Files with 4-digit years
cp *.{jpg,png,gif} images/    # Copy all image files
ls config_{production,staging,development}.json    # Environment configs

Customizing Your Command Line Environment

# Useful aliases for IoT development (add to ~/.bashrc)
alias ll='ls -la'
alias la='ls -la'
alias l='ls -CF'
alias ..='cd ..'
alias ...='cd ../..'
alias grep='grep --color=auto'
alias iot-logs='tail -f /var/log/iot/*.log'
alias iot-status='systemctl status iot-*'
alias sensor-data='cd /var/lib/iot/sensors'

# Environment variables for IoT development
export IOT_HOME="/opt/iot"
export SENSOR_CONFIG="/etc/iot/sensors"
export IOT_LOGS="/var/log/iot"

# Custom functions for common IoT tasks
iot-backup() {
    tar -czf "iot-backup-$(date +%Y%m%d_%H%M%S).tar.gz" \
        /etc/iot /opt/iot /var/lib/iot
}

sensor-restart() {
    sudo systemctl restart iot-sensor-$1
    sudo systemctl status iot-sensor-$1
}

# History configuration
export HISTSIZE=10000    # Commands in memory
export HISTFILESIZE=20000    # Commands in history file
export HISTCONTROL=ignoredups:erasedups    # Avoid duplicates
shopt -s histappend    # Append to history file

6. Hands-on Practice Exercises - Real-World Scenarios

Professional IoT Development Scenarios

These exercises simulate real IoT development tasks that you'll encounter in professional environments. Work through them to build practical skills and confidence.

Exercise 1
Create a professional IoT project structure with proper organization and backup system
Exercise 2
Find and analyze system configuration files for network and IoT services
Exercise 3
Monitor and analyze IoT system logs for troubleshooting and performance analysis
Exercise 4
Create automated backup and deployment scripts using archives and compression
# Exercise 1: Professional IoT Project Setup
# Create a comprehensive project structure
mkdir -p iot-smart-home/{
    src/{sensors,controllers,networking,ui},
    config/{development,production,testing},
    docs/{api,hardware,deployment,troubleshooting},
    tests/{unit,integration,hardware,performance},
    logs/{application,system,sensors,network},
    scripts/{deployment,maintenance,backup,monitoring},
    data/{raw,processed,archived},
    firmware/{current,previous,development}
}

# Navigate and explore the structure
cd iot-smart-home
tree . || find . -type d | sort    # Show directory structure
ls -la    # Verify creation

# Create sample files to practice with
touch src/sensors/{temperature.py,humidity.py,motion.py}
touch config/development/{database.json,network.yaml,sensors.conf}
touch logs/sensors/{temp_sensor.log,humidity_sensor.log}
echo "# Smart Home IoT Project" > README.md

# Exercise 2: System Configuration Analysis
# Find all network-related configuration files
find /etc -name "*network*" -type f 2>/dev/null | head -10
find /etc -name "*wifi*" -o -name "*wlan*" 2>/dev/null

# Analyze network configuration
grep -r "192.168" /etc/network* 2>/dev/null | head -5
cat /etc/hostname
cat /etc/hosts | grep -v "^#"

# Find IoT and sensor related configurations
find /etc -name "*iot*" -o -name "*sensor*" 2>/dev/null
locate "*.conf" | grep -E "(iot|sensor|mqtt)" | head -10

# Exercise 3: Log Analysis and Monitoring
# Create sample log data for practice
sudo mkdir -p /var/log/iot
echo "$(date): Temperature sensor initialized" | sudo tee -a /var/log/iot/sensors.log
echo "$(date): ERROR: Humidity sensor connection failed" | sudo tee -a /var/log/iot/sensors.log
echo "$(date): MQTT broker connected successfully" | sudo tee -a /var/log/iot/network.log

# Analyze system logs for IoT-related entries
journalctl --since "1 hour ago" | grep -i "iot\|sensor\|mqtt" | head -10
grep -i "error\|warning" /var/log/syslog | tail -20
tail -f /var/log/syslog | grep --line-buffered "sensor" &    # Background monitoring

# Count different types of log entries
grep -c "ERROR" /var/log/iot/sensors.log 2>/dev/null || echo "No errors found"
grep -c "WARNING" /var/log/iot/sensors.log 2>/dev/null || echo "No warnings found"

# Exercise 4: Automated Backup and Deployment
# Create a comprehensive backup script
cat > backup_iot_project.sh << 'EOF'
#!/bin/bash
# IoT Project Backup Script
BACKUP_DIR="/backup/iot"
PROJECT_DIR="$HOME/iot-smart-home"
DATE=$(date +%Y%m%d_%H%M%S)

# Create backup directory
mkdir -p "$BACKUP_DIR"

# Create project backup
echo "Creating project backup..."
tar -czf "$BACKUP_DIR/iot_project_$DATE.tar.gz" \
    --exclude="*.log" \
    --exclude="*.tmp" \
    --exclude="__pycache__" \
    --exclude="node_modules" \
    "$PROJECT_DIR"

# Create configuration backup
echo "Backing up configurations..."
sudo tar -czf "$BACKUP_DIR/iot_configs_$DATE.tar.gz" \
    /etc/iot/ \
    /etc/systemd/system/iot-* \
    2>/dev/null

# List created backups
echo "Backup completed:"
ls -lh "$BACKUP_DIR"/*$DATE*
EOF

chmod +x backup_iot_project.sh
./backup_iot_project.sh

# Create deployment package
tar -czf iot-smart-home-deployment.tar.gz \
    --exclude="logs/*" \
    --exclude="data/raw/*" \
    --exclude="*.tmp" \
    iot-smart-home/

# Verify archive contents
tar -tzf iot-smart-home-deployment.tar.gz | head -20

Advanced Practice Challenges

# Challenge 1: Complex log analysis pipeline
# Find all temperature readings above 30°C in the last hour
grep "temperature" /var/log/sensors.log 2>/dev/null | \
grep "$(date +%Y-%m-%d)" | \
grep -o "[0-9]*\.[0-9]*°C" | \
sed 's/°C//' | \
awk '$1 > 30 {print $1"°C"}' | \
wc -l

# Challenge 2: System health monitoring
# Create a one-liner to check IoT system status
ps aux | grep -E "(iot|sensor|mqtt)" | grep -v grep | wc -l && \
df -h | grep -E "(var|opt|home)" && \
free -h | grep Mem

# Challenge 3: Automated file organization
# Organize log files by date
for logfile in /var/log/iot/*.log; do
    if [ -f "$logfile" ]; then
        date_dir="/var/log/iot/archive/$(date +%Y/%m)"
        sudo mkdir -p "$date_dir"
        sudo cp "$logfile" "$date_dir/"
    fi
done 2>/dev/null

# Challenge 4: Configuration file validation
# Check all JSON configuration files for syntax errors
find /etc/iot -name "*.json" 2>/dev/null | while read jsonfile; do
    if python3 -m json.tool "$jsonfile" > /dev/null 2>&1; then
        echo "✓ $jsonfile: Valid JSON"
    else
        echo "✗ $jsonfile: Invalid JSON"
    fi
done

Session Summary & Next Steps

What You've Accomplished

Congratulations! You now have a comprehensive toolkit of command-line skills essential for IoT development. Here's what you can do:

  • File Management Mastery: You can efficiently organize, copy, move, and manage files in complex IoT project structures
  • Text Processing Power: You can analyze log files, search for patterns, and process sensor data using powerful command combinations
  • System Navigation: You can quickly locate files, executables, and configurations across the entire system
  • Archive Management: You can create professional backup and deployment packages for IoT projects
  • Efficiency Optimization: You can work at professional speed using history, shortcuts, and automation techniques

Real-World IoT Applications

These skills directly translate to professional IoT development tasks:

  • System Administration: Managing IoT device configurations, logs, and system files
  • Debugging and Troubleshooting: Analyzing log files and system behavior to identify issues
  • Data Analysis: Processing sensor data and system metrics for insights
  • Deployment and Maintenance: Creating deployment packages and managing system updates
  • Automation: Building scripts and workflows for repetitive tasks
  • Security: Managing file permissions and system access for IoT devices

Key Commands Mastery Summary

ls -la, cp -r, mv, rm -i, mkdir -p, rmdir
cat, less, head, tail -f, grep -r, cut, sort, uniq
find -name -type -exec, locate, which, whereis
tar -czf, tar -xzf, gzip, zip -r, unzip
history, !!, Ctrl+R, Tab completion, aliases
|, >, >>, <, tee, xargs, command chaining

Preparation for Session 3: Permissions & Process Management

In the next session, we'll explore Linux security model and process management. To prepare and reinforce today's learning:

  • Daily Practice: Use these commands in your regular development work - they become natural with repetition
  • Create Projects: Set up real IoT project structures and practice organizing files professionally
  • Explore Your System: Use find and grep to explore system configurations and understand how Linux organizes files
  • Build Workflows: Create simple scripts that combine multiple commands for common tasks
  • Join Communities: Participate in Linux and IoT forums where these skills are discussed and used
  • Read Documentation: Try "man command" for any command to see all available options and examples
← Session 1 Session 3 →