Wednesday, July 23, 2014

Embedding python in bash scripts

As a software development consultant, I do a lot of bash scripting.  I can do a lot of really creative things using nothing but bash and the binutils at my disposal, but sometimes I'll come across something that's just easier to do in a higher level scripting language.  Enter python.

Conversely, there are a lot of things that are just easier and more straight forward to do in bash, so writing everything in straight python may be more work than it's worth.

Here's a quick posting to describe how you can embed some python code into your bash scripts and get the best of both worlds.  Note: just as a heads, up, the examples in this posting are quite contrived.

Calling python from bash is easy.  You simply use python's '-' argument and pipe in your python code.  I typically wrap my python code in a bash function.


function current_datetime {
python - <<END
import datetime

# Call it

# Call it and capture the output
echo Current date and time: $DT

You can also pass data into your embedded python script.  I do that using environment variables:


function line {
PYTHON_ARG="$1" python - <<END
import os
line_len = int(os.environ['PYTHON_ARG'])
print '-' * line_len

# Do it one way
line 80

echo 'Handy'

# Do it another way
echo $(line 80)

My usual use-case for doing this is if I'm extending someone else's bash scripts and have to 'go off the reservation' a bit.  Sometimes I'm updating an existing 'legacy' script and need to look up some data... maybe do a REST call or something.  Here's an example bash script that uses curl to call a REST service to get some weather data.  Then is passes the raw JSON response to an embedded python script to interpret and format the results:


function format_weather_data() {
PYTHON_ARG="$1" python - <<END
import os
import json

json_data = os.environ['PYTHON_ARG']
data =json.loads(json_data)
lookup = {
    '200': 'thunderstorm with light rain',
    '201': 'thunderstorm with rain',
    '202': 'thunderstorm with heavy rain',
    '210': 'light thunderstorm',
    '211': 'thunderstorm',
    '212': 'heavy thunderstorm',
    '221': 'ragged thunderstorm',
    '230': 'thunderstorm with light drizzle',
    '231': 'thunderstorm with drizzle',
    '232': 'thunderstorm with heavy drizzle',
    '300': 'light intensity drizzle',
    '301': 'drizzle',
    '302': 'heavy intensity drizzle',
    '310': 'light intensity drizzle rain',
    '311': 'drizzle rain',
    '312': 'heavy intensity drizzle rain',
    '313': 'shower rain and drizzle',
    '314': 'heavy shower rain and drizzle',
    '321': 'shower drizzle',
    '500': 'light rain',
    '501': 'moderate rain',
    '502': 'heavy intensity rain',
    '503': 'very heavy rain',
    '504': 'extreme rain',
    '511': 'freezing rain',
    '520': 'light intensity shower rain',
    '521': 'shower rain',
    '522': 'heavy intensity shower rain',
    '531': 'ragged shower rain',
    '600': 'light snow',
    '601': 'snow',
    '602': 'heavy snow',
    '611': 'sleet',
    '612': 'shower sleet',
    '615': 'light rain and snow',
    '616': 'rain and snow',
    '620': 'light shower snow',
    '621': 'shower snow',
    '622': 'heavy shower snow',
    '701': 'mist',
    '711': 'smoke',
    '721': 'haze',
    '731': 'sand, dust whirls',
    '741': 'fog',
    '751': 'sand',
    '761': 'dust',
    '762': 'volcanic ash',
    '771': 'squalls',
    '781': 'tornado',
    '800': 'clear sky',
    '801': 'few clouds',
    '802': 'scattered clouds',
    '803': 'broken clouds',
    '804': 'overcast clouds',
    '900': 'tornado',
    '901': 'tropical storm',
    '902': 'hurricane', 
    '903': 'cold',
    '904': 'hot',
    '905': 'windy',
    '906': 'hail',
    '950': 'setting',
    '951': 'calm',
    '952': 'light breeze',
    '953': 'gentle breeze',
    '954': 'moderate breeze',
    '955': 'fresh breeze',
    '956': 'strong breeze',
    '957': 'high wind, near gale',
    '958': 'gale',
    '959': 'severe gale',
    '960': 'storm',
    '961': 'violent storm',
    '962': 'hurricane',

print "Current temperature: %g F" % data['main']['temp']
print "Today's high: %g F" % data['main']['temp_max']
print "Today's low: %g F" % data['main']['temp_min']
print "Wind speed: %g mi/hr" % data['wind']['speed']
weather_descs = [lookup.get(str(i['id']), '*error*') for i in data['weather']]
print "Weather: %s" % ', '.join(weather_descs)



format_weather_data "$(curl -s $WEATHER_URL)"

Hope you find this information useful. 

Wednesday, January 22, 2014

Formatting GUIDs with sed

At my current client, I'm often given GUIDs for one reason or another.  Often they are not in the format that I need them to be.  Sometimes they have the dashes and I need them to be dash-free.  Other times, the dashes have been removed and I need them back.

Here's some quick, down and dirty sed commands to swap them back and forth:

To remove the dashes:

$ echo 8EC60070-685F-41DB-C881-EACF9E74E4BD | sed 's/-//g'

To put them back:

$ echo 8EC60070685F41DBC881EACF9E74E4BD | sed -rn 's/([0-9A-F]{8})([0-9A-F]{4})([0-9A-F]{4})([0-9A-F]{4})([0-9A-F]{12})/\1-\2-\3-\4-\5/p')

It goes without saying you can embed this very sed command in a bash script and run it over many rows in a file.