Bash scripts can be surprisingly slow if you’re not careful, but a few tweaks can make a huge difference.
Here’s a simple script that processes a list of files:
#!/bin/bash
for file in /path/to/my/files/*; do
if [[ -f "$file" ]]; then
echo "Processing: $file"
grep "pattern" "$file" > /dev/null
wc -l "$file"
fi
done
Now, let’s make it faster.
1. Avoid Unnecessary Subshells
Every time you use $(command) or command within backticks, Bash starts a new subshell. This has overhead. If you can, use built-in commands or shell expansions instead. For example, instead of count=$(ls | wc -l), use shopt -s nullglob; files=(/path/to/my/files/*); count=${#files[@]}. The latter avoids spawning ls and wc.
2. Use source or . Instead of bash for Dotfiles
If your script needs to load variables or functions from another script (like a configuration file), using bash config.sh runs config.sh in a subshell, and its changes are lost when it exits. Use source config.sh or . config.sh to execute it in the current shell, making its definitions available to your main script.
3. Minimize External Command Calls
Many operations can be done with Bash built-ins. For instance, string manipulation is much faster with ${variable/pattern/replacement} than calling sed. Checking if a file exists is [[ -f "$file" ]] instead of test -f "$file" or [ -f "$file" ] (though the latter two are often optimized). Even arithmetic can be done with (( a = b + c )) instead of expr $b + $c.
4. Optimize Loops
In the example script, for file in /path/to/my/files/* might expand a very large glob, which can be slow. If you’re processing a predictable set of files or a list from standard input, be more specific. If you must glob, consider shopt -s globstar for recursive globbing (**/*.txt) if that’s what you need, but be aware it can be slower than targeted commands if the directory structure is deep or wide.
5. Process Files in Parallel
If your script performs independent operations on many files, you can use tools like xargs -P or parallel to run them concurrently.
find /path/to/my/files -type f -print0 | xargs -0 -P 4 -I {} bash -c 'echo "Processing: {}"; grep "pattern" "{}" > /dev/null; wc -l "{}"'
This example runs four bash -c processes in parallel. find ... -print0 | xargs -0 is crucial for handling filenames with spaces or special characters.
6. Efficiently Read Files Line by Line
Reading a file line by line using while read -r line; do ... done < file.txt is generally efficient. Avoid reading the whole file into memory with $(cat file.txt) unless the file is small. The -r option prevents backslash interpretation.
7. Use set -e and set -o pipefail
While not strictly a speed hack, these options prevent silent failures that can lead to wasted cycles. set -e exits immediately if a command exits with a non-zero status. set -o pipefail makes a pipeline’s exit status the status of the last command to exit with a non-zero status, or zero if all commands exit successfully. This catches errors early.
8. Avoid eval
eval is a security risk and is often slow because it requires Bash to parse and execute a string as a command. If you find yourself using eval, there’s almost always a safer and faster alternative, often involving arrays or more direct variable expansion.
9. Use printf Instead of echo
For more complex output formatting or when dealing with escape sequences, printf is generally more robust and sometimes faster than echo, which has historically had inconsistent behavior across different shells and versions.
printf "Processing: %s\n" "$file"
10. Profile Your Script
Use set -x to trace your script’s execution and identify slow parts. For more detailed profiling, you can use time on individual commands or perf for deeper system-level analysis.
time grep "pattern" "$file"
The next thing you’ll likely encounter is dealing with very large numbers of files or extremely large individual files, which will push you towards more specialized tools or languages.