/devices/pseudo/bitbucket@0,0:pseudo (Posts about Python)

Five letter words in the English language

jmcp — Thu, 21 Jul 2022 02:37:00 GMT

Like many people I quite enjoy playing Wordle, and I quite enjoy playing Worldle, too. I like both of these games so much that I've made completing them my Start Of Day (SOD) procedure.

Yesterday's Worldle was Vatican City, a place that J and I visited when we had an amazing five week long trip to Europe in 2005. Worldle gives you a black-on-white image of the geographic area to guess, and each day's image is approximately the same size in your window.

I thought this was pretty easy to guess - to me it just looks like a fortified place like a castle or ... a city state. There aren't too many of those on this planet so it was a 1 from 1 situation. I forgot to save a copy of the clue version, so imagine this picture below but with everything inside the walls filled in, in black:

This morning I got to thinking - if you've only got the edges of a word (start and end letters), how many unique combinations are there? [Yes, I'm only thinking about the English language].

A quick check of /usr/share/dict/words on my workstation (standard Linux dictionary installed) shows that out of 104334 words there are 7044 with five letters. Clarifying that just a little, if you remove those that are capitalised (proper nouns and acronyms) you're left with 4667 five letter words.

>>> import re
>>> words = open("words", "r").readlines()
>>> len(words)
104334
>>> fivers = [j.strip() for j in words if len(j) == 6]
>>> len(fivers)
7044
>>> fivers[0:20]
["ABC's", "ABM's", 'AFAIK', "AFC's", "AMD's", 'ANSIs', 'ANZUS', "AOL's", 'ASCII', "ASL's", 'ASPCA', "ATM's", "ATP's", 'AWACS', "AWS's", "AZT's",'Aaron', 'Abbas', 'Abdul', "Abe's"]

>>> fivers = []
>>> for w in words:
...     m = re.match("^[a-z]{5}$", w.strip())
...     if m:
...         fivers.append(m.group(0))
...
>>> len(fivers)
4667
>>> fivers[0:20]
['abaci', 'aback', 'abaft', 'abase', 'abash', 'abate', 'abbey', 'abbot', 'abeam', 'abets', 'abhor', 'abide', 'abler', 'abode', 'abort', 'about', 'above', 'abuse', 'abuts', 'abuzz']

That's good enough to look at the initial and final letter combinations. To do that we'll use a set:

>>> enders = [(i[0], i[-1]) for i in fivers]
>>> len(enders)
4667
>>> snend = set(enders)
>>> len(snend)
411
>>> snend
{('a', 'z'), ('t', 's'), ('p', 'x'), ('d', 'f'), ('s', 'i'), ('u', 'e'), ('z', 'i'), ('v', 'a'), ('g', 'a'), ('y', 'h'), ('j', 't'), ('t', 'x'), ('m', 'i'), ('l', 'e'), ('j', 'n'), ('k', 't'), ('q', 'k'), ('o', 'l'), ('h', 'x'), ('l', 'h'), ('u', 'a'), ('k', 'n'), ('b', 'l'), ('d', 'i'), ('y', 'a'), ('v', 'd'), ('g', 'd'), ('v', 'g'), ('k', 'u'), ('o', 't'), ('r', 'y'), ('c', 'r'), ('c', 'l'), ('s', 'p'), ('j', 'i'), ('l', 'a'), ('b', 't'), ('m', 'm'), ('k', 'i'), ('u', 'd'), ('c', 'f'), ('n', 'r'), ('u', 'g'), ('n', 'l'), ('p', 't'), ('y', 'd'), ('f', 's'), ('y', 'g'), ('t', 'l'), ('p', 'n'), ('d', 'm'), ('u', 'r'), ('a', 's'), ('l', 'd'), ('q', 'y'), ('p', 'u'), ('x', 'v'), ('l', 'g'), ('m', 'c'), ('b', 'i'), ('d', 'p'), ('t', 't'), ('k', 'z'), ('r', 'x'), ('h', 't'), ('h', 'n'), ('s', 'e'), ('s', 's'), ('s', 'h'), ('o', 'm'), ('e', 'y'), ('h', 'u'), ('x', 'n'), ('z', 'h'), ('w', 'n'), ('t', 'i'), ('g', 'n'), ('b', 'm'), ('m', 'e'), ('m', 'h'), ('c', 'm'), ('b', 'p'), ('z', 'a'), ('d', 'e'), ('d', 'h'), ('c', 'o'), ('p', 'p'), ('a', 'l'), ('s', 'w'), ('b', 'c'), ('m', 'a'), ('j', 'e'), ('t', 'm'), ('c', 'k'), ('f', 't'), ('v', 'o'), ('h', 'z'), ('j', 's'), ('c', 'c'), ('r', 't'), ('a', 'f'), ('k', 'e'), ('n', 'o'), ('z', 'd'), ('a', 't'), ('r', 'n'), ('k', 's'), ('t', 'p'), ('d', 'a'), ('m', 'w'), ('w', 'z'), ('f', 'u'), ('g', 'z'), ('s', 'r'), ('o', 'e'), ('s', 'l'), ('m', 'd'), ('y', 'o'), ('u', 'k'), ('b', 'e'), ('f', 'i'), ('s', 'f'), ('b', 'h'), ('d', 'd'), ('a', 'i'), ('m', 'r'), ('q', 't'), ('l', 'o'), ('d', 'g'), ('p', 'e'), ('q', 'n'), ('p', 's'), ('o', 'a'), ('p', 'h'), ('c', 'y'), ('d', 'r'), ('b', 'a'), ('t', 'e'), ('o', 'w'), ('f', 'm'), ('t', 'h'), ('c', 'a'), ('j', 'r'), ('n', 'y'), ('e', 't'), ('j', 'l'), ('h', 'e'), ('a', 'm'), ('b', 'w'), ('o', 'd'), ('e', 'n'), ('h', 's'), ('k', 'l'), ('f', 'p'), ('o', 'g'), ('u', 'y'), ('n', 'a'), ('b', 'd'), ('x', 's'), ('b', 'g'), ('y', 'y'), ('t', 'a'), ('w', 's'), ('o', 'r'), ('i', 's'), ('c', 'd'), ('c', 'g'), ('a', 'c'), ('b', 'r'), ('l', 'y'), ('s', 'm'), ('t', 'w'), ('x', 'x'), ('n', 'd'), ('i', 'x'), ('p', 'r'), ('b', 'f'), ('p', 'l'), ('t', 'd'), ('s', 'o'), ('t', 'g'), ('s', 'k'), ('p', 'f'), ('f', 'e'), ('s', 'c'), ('t', 'r'), ('f', 'h'), ('a', 'e'), ('m', 'o'), ('r', 's'), ('h', 'r'), ('a', 'h'), ('c', 'v'), ('h', 'l'), ('d', 'b'), ('t', 'f'), ('d', 'o'), ('l', 'x'), ('f', 'a'), ('w', 'l'), ('i', 'l'), ('d', 'k'), ('a', 'a'), ('j', 'p'), ('j', 'o'), ('k', 'o'), ('w', 't'), ('q', 's'), ('v', 't'), ('i', 't'), ('g', 't'), ('s', 'y'), ('a', 'w'), ('v', 'n'), ('k', 'k'), ('z', 'y'), ('f', 'd'), ('f', 'g'), ('o', 'o'), ('a', 'd'), ('b', 'b'), ('u', 't'), ('a', 'g'), ('s', 'a'), ('p', 'm'), ('b', 'o'), ('m', 'y'), ('f', 'r'), ('u', 'n'), ('o', 'c'), ('x', 'i'), ('f', 'l'), ('r', 'r'), ('c', 'b'), ('e', 's'), ('r', 'l'), ('y', 'n'), ('a', 'r'), ('b', 'k'), ('g', 'i'), ('p', 'o'), ('d', 'y'), ('f', 'f'), ('l', 't'), ('n', 'b'), ('p', 'k'), ('l', 'n'), ('s', 'd'), ('p', 'c'), ('s', 'g'), ('h', 'm'), ('k', 'h'), ('t', 'o'), ('x', 'm'), ('t', 'k'), ('h', 'o'), ('i', 'm'), ('g', 'm'), ('q', 'l'), ('j', 'a'), ('t', 'c'), ('k', 'a'), ('o', 'y'), ('h', 'c'), ('r', 'i'), ('w', 'p'), ('q', 'f'), ('g', 'p'), ('b', 'y'), ('d', 'x'), ('j', 'd'), ('p', 'y'), ('e', 'l'), ('k', 'd'), ('r', 'm'), ('p', 'a'), ('q', 'i'), ('t', 'y'), ('h', 'h'), ('f', 'o'), ('r', 'p'), ('w', 'e'), ('p', 'w'), ('a', 'o'), ('i', 'e'), ('g', 'e'), ('f', 'k'), ('w', 'h'), ('v', 's'), ('f', 'c'), ('g', 's'), ('g', 'h'), ('m', 't'), ('r', 'c'), ('a', 'k'), ('p', 'd'), ('h', 'a'), ('m', 'n'), ('p', 'g'), ('c', 'x'), ('e', 'i'), ('q', 'm'), ('d', 't'), ('u', 's'), ('s', 'b'), ('w', 'a'), ('d', 'n'), ('y', 's'), ('h', 'd'), ('w', 'w'), ('h', 'g'), ('l', 's'), ('r', 'e'), ('r', 'h'), ('w', 'd'), ('f', 'y'), ('i', 'd'), ('w', 'g'), ('i', 'g'), ('g', 'g'), ('a', 'y'), ('e', 'p'), ('o', 'n'), ('w', 'r'), ('v', 'r'), ('i', 'r'), ('g', 'r'), ('v', 'l'), ('r', 'a'), ('b', 'n'), ('g', 'l'), ('e', 'c'), ('c', 't'), ('k', 'b'), ('q', 'e'), ('w', 'f'), ('c', 'n'), ('b', 'u'), ('g', 'f'), ('q', 'h'), ('r', 'w'), ('u', 'l'), ('n', 't'), ('y', 'l'), ('n', 'n'), ('r', 'd'), ('r', 'g'), ('t', 'n'), ('c', 'i'), ('q', 'a'), ('l', 'r'), ('a', 'x'), ('e', 'e'), ('l', 'l'), ('y', 't'), ('e', 'h'), ('p', 'b'), ('z', 's'), ('n', 'i'), ('b', 'z'), ('t', 'b'), ('u', 'i'), ('e', 'a'), ('j', 'y'), ('m', 's'), ('k', 'y'), ('q', 'r'), ('v', 'm'), ('d', 's'), ('e', 'w'), ('c', 'p'), ('t', 'z'), ('l', 'i'), ('w', 'o'), ('e', 'd'), ('i', 'o'), ('g', 'o'), ('e', 'g'), ('f', 'n'), ('w', 'k'), ('a', 'n'), ('e', 'r'), ('u', 'p'), ('a', 'u'), ('o', 's'), ('z', 'l'), ('b', 's'), ('s', 't'), ('c', 'e'), ('l', 'p'), ('r', 'b'), ('m', 'l'), ('s', 'n'), ('c', 's'), ('c', 'h'), ('r', 'o'), ('h', 'y'), ('v', 'e'), ('l', 'c'), ('m', 'f'), ('s', 'u'), ('b', 'x'), ('v', 'h'), ('d', 'l'), ('n', 'e'), ('f', 'z'), ('w', 'y'), ('n', 's'), ('n', 'h'), ('i', 'y'), ('g', 'y')}

That's quite a few pairs! I'm easily amused by things like this, so let's see how many words are in the list which start with 'g' and end with 'y':

>>> gy = [j for j in fivers if j.startswith("g") and j.endswith("y")]
>>> gy
['gabby', 'gaily', 'gamey', 'gassy', 'gaudy', 'gauzy', 'gawky', 'gayly', 'geeky', 'giddy', 'gimpy', 'gipsy', 'glory', 'gluey', 'godly', 'golly', 'goody', 'gooey', 'goofy', 'gouty', 'gravy', 'grimy', 'gully', 'gummy', 'gunny', 'guppy', 'gushy', 'gusty', 'gutsy', 'gypsy']
>>> len(gy)
30

Let's check the distribution (aren't buckets fun?) amongst all the start/end combinations:

>>> buckets = {}
>>> for b in snend:
...     buckets[b] = len([j for j in fivers if j.startswith(b[0]) and endswith(b[1])])
...

I admit some surprise at seeing that there are 92 start/end combinations which only have one word in the list

>>> unobuckets = [b for b in buckets if buckets[b] == 1]
>>> len(unobuckets)
92
>>> unobuckets
[('a', 'z'), ('p', 'x'), ('z', 'i'), ('y', 'h'), ('t', 'x'), ('j', 'n'), ('h', 'x'), ('k', 'n'), ('d', 'i'), ('y', 'a'), ('v', 'g'), ('k', 'u'), ('j', 'i'), ('k', 'i'), ('u', 'g'), ('y', 'g'), ('q', 'y'), ('p', 'u'), ('x', 'v'), ('l', 'g'), ('b', 'i'), ('d', 'p'), ('k', 'z'), ('r', 'x'), ('h', 'u'), ('x', 'n'), ('z', 'a'), ('b', 'c'), ('h', 'z'), ('a', 'f'), ('n', 'o'), ('z', 'd'), ('f', 'u'), ('g', 'z'), ('y', 'o'), ('u', 'k'), ('f', 'i'), ('q', 'n'), ('o', 'w'), ('f', 'm'), ('j', 'l'), ('f', 'p'), ('o', 'g'), ('x', 's'), ('x', 'x'), ('c', 'v'), ('d', 'b'), ('t', 'f'), ('l', 'x'), ('j', 'p'), ('z', 'y'), ('b', 'b'), ('o', 'c'), ('g', 'i'), ('f', 'f'), ('n', 'b'), ('k', 'h'), ('i', 'm'), ('j', 'a'), ('h', 'c'), ('q', 'f'), ('d', 'x'), ('q', 'i'), ('r', 'p'), ('f', 'c'), ('q', 'm'), ('w', 'a'), ('w', 'w'), ('h', 'g'), ('i', 'g'), ('e', 'p'), ('e', 'c'), ('b', 'u'), ('r', 'w'), ('n', 't'), ('n', 'n'), ('q', 'a'), ('p', 'b'), ('n', 'i'), ('b', 'z'), ('u', 'i'), ('q', 'r'), ('v', 'm'), ('t', 'z'), ('l', 'i'), ('a', 'u'), ('z', 'l'), ('l', 'p'), ('r', 'b'), ('m', 'f'), ('s', 'u'), ('f', 'z')]

Let's choose five:

>>> for b in ("a", "z"), ("f", "z"), ("q", "n"), ("r", "x"), ("d", "x"):
...     q = [j for j in fivers if j.startswith(b[0]) and j.endswith(b[1])]
...     print(q)
...
['abuzz']
['frizz']
['queen']
['relax']
['detox']

I didn't really have a point to make here, I just wanted to share my amusement at how many five letter words there are to guess compared to the approximately 300 geographic entities on the planet that you'll shown the edges of in Worldle. Also that knowing Python means you can make short work of asking and answering these questions.

Some Pythonic Kafka stuff

jmcp — Mon, 10 May 2021 03:00:00 GMT

I've been actively interviewing over the last few months, and was recently talking with a cloud streaming provider about a Staff Engineer SRE role specializing in Apache Kafka. I've been doing a lot of work in that space for $employer and quite like it as a data streaming solution.

The interviews went well, and they sent me a do-at-home challenge with a one week timeout.

The gist of it was I had to create a Flask app which allowed the user to enter a URL to monitor for status on a given frequency, use Python to write a Kafka Producer to publish this data to a topic, and write a Kafka Consumer to read from the topic and insert into a PostgreSQL database.

Fun!

I did a brief investigation into using threads within a Flask app for the monitoring code, but quickly decided that a better architecture would be to do the monitoring via a separate daemon. Separation of concerns to allow for easier maintenance. Suddenly I'm all about ongoing maintenance rather then how quickly I can deliver a new feature... Hmmm.

The next step was to sketch out the table schema I wanted in the database:

CREATE SEQUENCE public.urltargets_seq
            INCREMENT BY 1
            MINVALUE 1
            MAXVALUE 9223372036854775807
            START 1
            CACHE 1
            NO CYCLE;

    CREATE SEQUENCE public.monitor_results_seq
            INCREMENT BY 1
            MINVALUE 1
            MAXVALUE 9223372036854775807
            START 1
            CACHE 1
            NO CYCLE;

    CREATE TABLE IF NOT EXISTS urltargets (
            urltargets_pk           int4 NOT NULL DEFAULT nextval('urltargets_seq'::regclass),
            urltarget                       varchar(1024) NOT NULL,
            monitor_frequency       int NOT NULL CHECK (monitor_frequency in (1, 2, 5, 10, 20, 30)),
            CONSTRAINT urltargets_pkey PRIMARY KEY (urltargets_pk)
    );

    CREATE TABLE IF NOT EXISTS monitor_results (
            monitor_results_pk      int4 NOT NULL DEFAULT nextval('monitor_results_seq'::regclass),
            http_status                     int NOT NULL,
            start_time                      timestamp with time zone NOT NULL,
            duration            int4 NOT NULL,
            urltarget_fk            int4 NOT NULL,
            CONSTRAINT monitor_results_fk_fkey FOREIGN KEY (urltarget_fk) REFERENCES urltargets(urltargets_pk)
    );

Having decided that I would offer monitoring frequencies of 1, 2, 5, 10, 20 and 30 minutes, I created views for the Flask app to use as well, rather than direct queries. They all look like this, with other values substituted in as you would expect.

CREATE OR REPLACE VIEW view_1m as
            SELECT mr.monitor_results_pk
                    ,  ut.urltarget
                    ,  mr.http_status
                    ,  mr.start_time
                    ,  mr.duration
            FROM monitor_results mr
            JOIN urltargets ut on ut.urltargets_pk = mr.urltarget_fk
            WHERE ut.monitor_frequency = 1
    ;

Since I really like seeing schemata visualised, I created a nice(ish) ERD as well:

Well that was straight forward, how about the application and daemon?

I split out the setup functionality into a separate file importable by the Flask app, the monitor daemon and the consumer. This contained the database connection, Kafka Producer and Kafka Consumer code. There's an interesting little niggle in the Kafka Producer setup which is not immediately obvious and required a bit of digging in StackOverflow as well as enabling debug output with librdkafka:

	`def _get_kafka_configuration():`
	`"""`
	`Common function to retrieve the Kafka configuration.`
	`"""`
	`global appconfig`

	`configuration = {`
	`"bootstrap.servers": appconfig["kafka"]["broker"],`
	`"group.id": "website-monitor",`
	`"ssl.key.location": appconfig["kafka"]["keyfile"],`
	`"ssl.certificate.location": appconfig["kafka"]["certfile"],`
	`"ssl.ca.location": appconfig["kafka"]["cafile"],`
	`"security.protocol": "SSL",`
	`# "debug": "eos, broker, admin", # re-enable if needed`
	`'transaction.timeout.ms': 60000,`
	`'enable.idempotence': True`
	`}`
	`return configuration`


	`def setup_kafka_producer(view):`
	`"""`
	`Creates the connection to our Kafka brokers so we can publish`
	`messages to the topics we want. We take the {view} argument so`
	`that we can avoid blatting multiple producers together and getting`
	`errors from the broker about zombies and fencing. See`
	`https://stackoverflow.com/questions/50335227/how-to-pick-a-kafka-transaction-id`
	`for more details`

	`Return: a Producer`
	`"""`

	`configuration = _get_kafka_configuration()`
	`configuration["transactional.id"] = "website-monitor" + str(view)`
	`kafkaProducer = Producer(configuration)`
	`try:`
	`kafkaProducer.init_transactions()`
	`except KafkaError as ke:`
	`# If we can't do this, then we have to quit`
	`print(f"""Producer failed to init_transactions(), throwing {ke}""")`
	`raise`

	`return kafkaProducer`

When I was working with the daemon, my first iteration tried opening the DB connection and Producer in each thread's (one for each frequency) __init__() function, and .... that didn't work.

The DB connection is not picklable, so does _not_ survive the call to os.fork(). Once I had rewritten the setup and run methods to get the DB connection, that part was groovy.

The Kafka Producer still required a bit of work. After reading through stackoverflow and the upstream for librdkafka, I saw that I needed to similarly delay initialising the producer until the thread's run() method was called. I also observed that each Producer should also initialise the transaction feature, but leave the begin... end of the transaction to when it was called to publish a message.

I still had a problem, though - some transactions would get through, but then the Producer would be fenced. This was the niggle, and where the StackOverflow comments helped me out:

Finally, in distributed environments, applications will crash or —worse!— temporarily lose connectivity to the rest of the system. Typically, new instances are automatically started to replace the ones which were deemed lost. Through this process, we may have multiple instances processing the same input topics and writing to the same output topics, causing duplicate outputs and violating the exactly once processing semantics.

We call this the problem of “zombie instances.” [emphasis added]

I realised that I was giving the same transactional id to each of the six producer instances (setting the 'transactional.id' in the configuration dict generated by _get_kafka_configuration(), so I needed to uniqify them somehow. I decided to pass the monitoring frequency of the thread to the setup function, and ... booyah, I had messages being published.

That was a really nice feeling.

There is one other aspect of the monitoring daemon that I need to mention. Since each thread reads its list of URLs to monitor each time it wakes, I wanted to parallelize this effort. Monitoring each of the URLs in series could easily take too long from a sleep(...) point of view, and I really did not want to just fork a whole heap of processes and thread either - avoiding the potential for a fork-bomb.

To work around this I used the Python standard library concurrent.futures with a ThreadPoolExecutor for each target URL. Adding attributes to the future object enabled me to use an add_done_callback so that when the future crystallized it would then publish the message.

	`def run(self):`
	`"""`
	`Runs the monitor, updates account-keeping and kicks off`
	`notifications if required. Then back to sleep.`
	`"""`
	`self._producer = setup_kafka_producer(self._view)`
	`self._conn = setup_db()`
	`self._cursor = self._conn.cursor()`
	`while True:`
	`alltargets = self._get_targets()`
	`if alltargets:`
	`# We use a 'with' statement to ensure threads in the pool`
	`# are cleaned up promptly`
	`with cf.ThreadPoolExecutor(max_workers=50) as executor:`
	`for tgt in alltargets:`
	`future = executor.submit(check_url,`
	`tgt[0], tgt[1])`
	`future.args = (tgt[0], tgt[1])`
	`future.producer = self._producer`
	`future.add_done_callback(construct_and_publish)`
	`self._futures.append(future)`
	`for future in cf.as_completed(self._futures):`
	`if future.done():`
	`self._futures.remove(future)`

	`sleep(self._sleeptime)`

The check and publish methods are outside of the thread definition:

	`def construct_and_publish(input):`
	`"""`
	`Callback function for the concurrent.future that each thread`
	`makes use of to query a website. Turns the future's attributes`
	`into a message for 'url-to-monitor' topic, then publishes that`
	`message to the topic.`
	`"""`
	`if input.cancelled() or input.exception():`
	`errmsg = """Monitor attempt for {args} failed"""`
	`print(errmsg.format(args=input.args,`
	`file=stderr, flush=True))`
	`else:`
	`message = json.dumps(dict(zip(msgFields, input.result())))`
	`input.producer.begin_transaction()`
	`publish_message(producer=input.producer,`
	`topic="url-monitor-results",`
	`message=message.encode('utf-8'))`
	`input.producer.commit_transaction()`


	`def check_url(url, fk):`
	`"""`
	`Performs an 'HTTP GET' of the supplied url and returns a tuple containing`
	`(fk, start_time, duration, http_status).`
	`The start_time is expressed in milliseconds since the UNIX Epoch.`
	`"""`
	`start_time = datetime.timestamp(datetime.now())`
	`result = requests.get(url)`
	`duration = datetime.timestamp(datetime.now()) - start_time`
	`return (fk, start_time, duration, result.status_code)`

With the monitoring daemon written, I now needed the other end of the pipe - writing the Kafka Consumer to read from the topic and insert into the database. This was straightforward: we're polling for messages on both configured topics, when we read one we write it to the appropriate DB table using a prepared statement, commit and then do it all again with a while loop.

	`urlToMonitorStmt = "INSERT INTO urltargets (urltarget, monitor_frequency "`
	`urlToMonitorStmt += "VALUES (%(urltarget)s, %(monitor_frequency)s)"`

	`urlMonitorResultsStmt = "INSERT INTO monitor_results (http_status, "`
	`urlMonitorResultsStmt += "urltarget_fk, start_time, duration) "`
	`urlMonitorResultsStmt += "VALUES (%(http_status)s, %(targetId)s, "`
	`urlMonitorResultsStmt += "to_timestamp(%(start_time)s), %(duration)s)"`

	`lookups = {`
	`"url-to-monitor": urlToMonitorStmt,`
	`"url-monitor-results": urlMonitorResultsStmt`
	`}`

	`if __name__ == "__main__":`

	`consumer = setup_kafka_consumer()`

	`connection = setup_db()`

	`while True:`
	`with connection.cursor() as curs:`
	`msglist = consumer.consume(200)`
	`for msg in msglist:`
	`if not msg:`
	`continue`
	`elif msg.error():`
	`print("Received error during poll: {error}".format(`
	`error=msg.error()))`
	`else:`
	`stmt = lookups[msg.topic()]`
	`values = json.loads(msg.value().decode('utf-8'))`
	`curs.execute(stmt, values)`
	`curs.execute("COMMIT")`

	`connection.close()`

Of course there should be error handling for the execute(). There should also be packaging and tests and templates for the report. Do not @ me, etc etc.

The reason why all these pieces are missing is because the day before I was due to hand this assignment in to my interviewer, I received a very, very nice offer from another company that I'd also been interviewing with - and I accepted it.

An unexpected live coding challenge

jmcp — Sat, 08 May 2021 16:00:00 GMT

A few weeks ago I was in a technical interview, and was asked to do a live coding challenge. I was surprised, because this is the sort of thing that I expect a recruiter and hiring manager to mention ahead of time. Preparation is very important, and while I know that there are many people for whom live coding is a thrill, there are plenty of other people for whom it can be a terrifying experience.

I'm glad to say that I'm not terrified by it, but it's definitely not an ideal environment for me.

So after a few minutes of me describing what I've done in my career (it seemed pretty clear that the interviewer hadn't actually read my resume), and a few technical questions, we got into the challenge.

"""
For a given string composed of parenthesis ("(", "{", "["), check if the string is valid parenthesis.
1. "()" -- valid
2. "({})" -- valid
3. "(}{)" -- invalid
4. "{()}[{}]" -- valid
5. "({(}))" -- invalid
"""

I noted immediately that this is an issue which requires the processing function to track state, because you not only need to determine open and closed pairings, but also what type it is.

It took a minute to sketch out the classifications that I needed, talking through my decision process all the while:

OPENS = ["(", "{", "["]
CLOSES = [")", "}", "]"]

braces = [ "{", "}"]
parens = [ "(", ")"]
brackets = [ "[", "]"]

classes = { "braces": braces,
            "parens": parens,
            "brackets": brackets
}

I was able to stub out a check function pretty quickly, but got stuck when I went from the stub to implementation, because I realised that I needed to keep track of what the previous element in the string was.

Oh no! How do I do that? (A stack, btw)

Mental blank :(

I needed time to jog my memory, so I asked the interviewer to tell me about himself, what he does on the team and a few other questions.

This, I think, was a very good decision - with the focus of the interview not on me, I could step back and think about what basic data types in Python I could use to implement a stack.

The data type I needed is indeed pretty basic: a list().

A Python list() lets you push (the append() operation) and pop so with the addition of another data structure

counts = { "braces": 0,
           "parens": 0,
           "brackets": 0
}

and a short function to return the class of the element

def __classof(c):
    """ returns whether 'c' is in brackets, braces or parens """
    if c in braces:
        return "braces"
    elif c in brackets:
        return "brackets"
    else:
        return "parens"

we're now in a much better position for the algorithm.

By this time I had also calmed myself down, because everything came together pretty easily for me.

With the above code blocks already noted, here is the body of the function:

def check_valid(input):
    """ For the given examples at top, determine validity.
        Assumption: the input is _only_ braces, parens and brackets
    """

    # start
    c = input[0]

    stack = list()
    stack.append(c)

    counts[__classof(c)] += 1

    for c in input[1:]:
        if c in OPENS:
            ## increment count & add to stack
            stack.append(c)
            counts[__classof(c)] += 1
        else:
            ## closing checks
            if __classof(c) !=  __classof(stack[-1]):
                return "invalid"
            else:
                # decrement count_ (__classof(c))
                counts[__classof(c)] -= 1
                stack.pop()

    return "valid"

We're playing fast and loose here with input validity checking - there's no "is this an empty string?" and we're not handling a single-character string, let alone validating that our input only contains braces, parens and brackets.

With this main() function, though, we're doing pretty well:

## main
strings = ["""()""",
           """({})""",
           """(}{)""",
           """{()}[{}]""",
           """({(}))""",
           """](){}"""
           ]

for element in strings:
    print("""{element:20}""".format(element=element),
          check_valid(element))

which gives us the following output:

()                   valid
({})                 valid
(}{)                 invalid
{()}[{}]             valid
({(}))               invalid
](){}                valid

Using the criteria specified, the final case is invalid, given that it starts with a terminating rather than initiating/opening element - there's nothing to balance the element with. However at that point my time was up and I didn't worry about it.

My interviewer then asked whether I had considered using recursion to solve the problem.

I hadn't considered recursion because I generally don't have to for the use-cases I need to write - and in this particular problem space it didn't seem to me to be an effective use of resources.

Consider the longest case, {()}[{}]. If you're recursing on the check function, then you'll wind up calling the function four times, so that's four new stack frames to be created and destroyed. That doesn't strike me as particularly efficient in time or space. Iterating over the input, however, avoids all of the setup + teardown overhead.

Anyway, it was a relatively fun exercise, and I'm glad I did it. I was able to keep a cool head and buy myself enough time to jog my memory and finish the problem, and it worked the first time (I know, that _never_ happens!).

For future encounters like this, I think it's handy to remember these points:

Breathe
Talk through what you are doing, and why
If you hit a problem, see point 1, and then say that you're stuck and need to think through a particular part of the issue.
If you need to stop talking so you can think, say that that's what you need to do.

It is my impression that if your interviewer is a decent person, they will help you work through your point of stuckness so that you can complete as much as possible of the task.

Access token retrieval in Python and Java

jmcp — Sun, 27 Dec 2020 17:00:00 GMT

At $work I'm part of our API team, enabling access to the rather large datasets that we have acquired (and generated) over the years. We make heavy use of Postman for testing, but every now and again I find that I want to do something on the commandline.

All of our APIs require authorization, for which we use OAuth, and specifically the Bearer Token.

Rather than having to fire up Postman to extract a Bearer Token, I decided to write a Python script to do it for me, so I could then set an environment variable and pass that to curl as a command-line argument. I was a little lazy and hard-coded my clientid and secret - I'm not going to be requesting anybody else's token!

#!/usr/bin/python3
import requests

url="""https://$AUTHSERVER/access/oauth/token?grant_type=client_credentials&client_id={client_id}&client_secret={client_secret}"""
client_id = "nope"
client_secret = "still_nope"

resp = requests.get(url.format(client_id=client_id,
                               client_secret=client_secret))
print("export BEARER=\"Authorization: Bearer " +
      resp.json()["access_token"] + "\"")

I'm taking advantage of the fact that I know through many years of use that the requests package does a lot of heavy lifting for me, particularly the JSON decoding.

Since $work has a shutdown between Christmas and New Year, I figured that I would spent some time implmenting this in Java. Not because I have a need to, but because I need to get more Java under my belt since $work is a Java shop and cross-pollination / polyglotting is useful.

The first step was to determine how to send an HTTP GET for a specific URI. A few searches later and I'd arrived at

import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.http.HttpResponse.BodyHandlers;
import java.net.URI;

and since it seems cleaner to import the Exceptions that these throw, I added

import java.io.IOException;
import java.net.URISyntaxException;

A bit more hard-coding laziness for the clientid and secret and I had the beginnings of a solution. (Note that I do not claim that any of this is using Best Practices, it's just a building block).

class bearerTokenCLI {

    private static URI authServer;
    private static HttpResponse<String> response;

    public static void main(String... args) {

        try {
            authServer = new URI("https", null,
                    "$AUTHSERVER", 443,
                    "/access/oauth/token",
                    "grant_type=client_credentials" +
                            "&client_id=nope" +
                            "&client_secret=still_node", null);
        } catch (URISyntaxException exc) {
            System.out.println("Received URISyntaxException");
            System.out.println(exc.getReason());
            System.out.println(exc.getMessage());
            System.out.println(exc.getInput());
        }

        System.out.println("Requesting " + authServer.toString());

Ok so far - we've created a new URI object, caught the specific exception that it could throw, and (because I'm learning) printing the stringified version of the URI to stdout.

Now we need an HttpRequest to send via an HttpClient:

HttpRequest request = HttpRequest.newBuilder(authServer).build();
HttpClient client = HttpClient.newHttpClient();

try {
    response = client.send(request, BodyHandlers.ofString());
} catch (java.io.IOException | java.lang.InterruptedException jiie) {
    /*
     * Note that this catch() uses Java7++ syntax for handling
     * multiple exceptions in the same block
     */
    System.out.println("Received java.io.IOException");
    System.out.println(jiie.getCause());
    System.out.println(jiie.getMessage());
    System.out.println(jiie.getStackTrace());
}

Assuming we didn't get an exception, we need to check that the HTTP Status Code of the response is OK, or 200:

if (response.statusCode() != 200) {
    /*
     * Something went wrong so print the url we requested, status code,
     * an error message and the response body as text.
     */
    System.out.println("Request was unsuccessful. " +
            "Received status code " + response.statusCode());
    System.out.println("URL requested was\n" + authServer.toString());
    System.out.println("Response body text:\n" + response.body());
    System.exit(1);
}

If it isn't ok, we bail out, and otherwise we check for the Content-Type header being set to 'application/json'. Why that specific value? If you refer to the RFCs for OAuth (RFC6749 and RFC6750) specifically section 5 of the latter, you see that

The parameters are included in the entity-body of the HTTP response using the "application/json" media type as defined by [RFC4627]. The parameters are serialized into a JavaScript Object Notation (JSON) structure by adding each parameter at the highest structure level. Parameter names and string values are included as JSON strings. Numerical values are included as JSON numbers. The order of parameters does not matter and can vary.

Let's check for that type then.

/*
 * Check that we've got 'application/json' as the Content-Type.
 * Per https://tools.ietf.org/html/rfc7231#section-3.1 we know that
 * Content-Type is a semicolon-delimited string of
 *     type/subtype;charset;...
 * More importantly, we know from https://tools.ietf.org/html/rfc6749#section-5.1
 * that the response type MUST be JSON.
 */
List<String> contentTypeHeader = response.headers().map().get("Content-Type");
if (contentTypeHeader.isEmpty()) {
    System.out.println("ERROR: Content-Type header is empty!");
    System.out.println(response.headers());
    System.exit(1);
}

Since contentTypeHeader is a List<T> we can either iterate over it, or, since we know that it can only occur once in an HTTP response we can grab element 0 directly. Here's iteration (and yes, I know we should be confirming that we've actually got 'application/json', do not @ me, etc etc):

for (String el: contentTypeHeader) {
    String contentType = el.split(";")[0];
    System.out.println("Actual Content-Type bit:   " + contentType);
}

On the other hand, making use of our knowledge that there's only one Content-Type header in the response, and per RFC7231 we know the format of the header, we can take a shortcut and grab the value directly:

String contentType = contentTypeHeader.get(0).split(";")[0];
if (!contentType.equalsIgnoreCase("application/json")) {
    /* Not JSON! */
    System.out.println("Content-Type is " + contentType +
            " not application/json. Exiting");
    System.exit(1);
}

So far, so good. It took me several hours to get to this point because I not only had to refresh my memory of those RFCs, but also realising that a short-circuit was possible.

Now we can move onto the response body text. By way of printing out response.getClass() I know that the response is an instance of class jdk.internal.net.http.HttpResponseImpl, and visual inspection of it shows that it's JSON. But how do I turn that into an array that I can pull the access_token information from?

At first I tried using Google's GSON but I just couldn't get my head around it. I need to find and understand more code examples. Until I do that, however, I turned to Jackson JR, which I found a lot more straightforward.

We need another import, this time

import com.fasterxml.jackson.jr.ob.JSON;

And then we construct a Map<String, Object> from the response body:

try {
    Map<String, Object> containerJSON = JSON.std.mapFrom(response.body());
    String accessToken = containerJSON.get("access_token").toString();
    System.out.println("export BEARER=\"BEARER " + accessToken + "\"\n");
} catch (IOException exc) {
    System.out.println("Caught exception " + exc.toString());
    System.out.println("Message:\n" + exc.getMessage());
}

You'll observe that I'm again being a bit lazy here by wrapping this block in the one try {...} catch (..) {..} block. Whyso? Because by this point we should be certain that we've actually got an access_token element in the response, and if we don't then there's something going wrong upstream.

Finally, how do we build this thing? As much as I'd like to just run javac over the source and create a solitary jar, I've found that including external dependencies is made immensely easier by using a build system like Maven, Ant or Gradle in the Java ecosystem. For C, of course, there's no place like make(1s) (ok, or GNU Make).

I started with using a Maven *archetype*, added this dependency to pom.xml:

<dependencies>
    <dependency>
        <groupId>com.fasterxml.jackson.jr</groupId>
        <artifactId>jackson-jr-objects</artifactId>
        <version>2.12.0</version>
    </dependency>
</dependencies>

and added the Maven Assembly plugin to the <build> lifecycle. Then building was a matter of

$ mvn clean install package assembly:single

and then I could run the package with

$ java -jar target/bearer_token_cli-1.0-SNAPSHOT-jar-with-dependencies.jar

All up, I estimate that researching and writing this in Java took me about 12 hours. Most of which was ecosystem research and exploration. There was only one syntax issue which tripped me up - I needed an Array and for about 10 minutes was searching through Javadocs for an appropriate class before I remembered I could use String[] arrName. Sigh.

Learning the ecosystem is the thing I'm finding most difficult with Java - it's huge and there are so many different classes to solve overlapping problems. I haven't even begun to work with @Annotations or dependency injection for my own code yet. Truth be told, after a decade+ of working in Solaris, the idea that anything could be injected into the code I've written puts a chill down my spine. I'm sure I'll get past it one day.

A brief introduction to the Python Database API

jmcp — Tue, 02 Jun 2020 23:37:00 GMT

As popular as NoSQL databases currently are, there is still an immense amount of data in the world for which the relational database is still the most appropriate way to store, access and manipulate it. For those of us who like to use Python rather than JDBC, we are fortunate to have bindings for the major databases which implement the Python Database API, aka PEP249.

This standard is important because it allows you to write your application in a way that promotes cross-platform (cross-database engine) portability. While you do still need to be aware of differences between the databases (for example, Oracle's VARCHAR2 vs PostgreSQL's VARCHAR), the essential tasks that you need to accomplish when making use of a database are abstracted from you.

In this brief introduction I will show you how to install two popular Python database binding packages, connect to a database and run queries.

Installing the bits you need

I assume that you've got Python installed on your system already, and that the database you want to connect to is set up and running. What you need now are the bindings for that particular database. For the Oracle database, this is a package known as cx_Oracle. For PostgreSQL, I suggest psycopg2 though there are other bindings available.

You will definitely need to install the pre-packaged database client libraries on your host (or in your container) as well. For Oracle look for your platform in the Oracle Instant Client download pages, for PostgreSQL on linux you will need libpq-dev as supplied by your package manager. On MacOS, please find the appropriate option from the PostgreSQL MacOSX download page.

After you've installed these pieces, you can then pip install --user cx_Oracle or pip install --user psycopg2. My personal preference is to install bindings in a venv. For Solaris 11 you can get the cx_Oracle bits as well as the Oracle Instant Client by uttering pkg install cx_Oracle. [On a personal note, I made that possible - see the History tab on the Solaris Userland cx_Oracle github].

Getting a connection to the database

Now that you have the correct packages installed, you can start investigating. The first thing to do is start an interactive interpreter and import the modules you need:

$ python3.8
>> import cx_Oracle

(Note that unless you're using Solaris' pre-packaged version, you must point LD_LIBRARY_PATH to where the module can locate the Instant Client libraries, specifically libclntsh.so)

$ python 3.8
>>> import psycopg2
>>> from psycopg2.extras import execute_batch

We need a connection to the database, which in Oracle terms is a DSN, or Data Source Name. This is made up of the username, password, host (either address or IP), port and database instance name. Put together, the host, port and database instance are what Oracle calls the "service name".

The common example you will see in Oracle documentation (and on stackoverflow!) is

scott/tiger@orcl:1521/orcl

I'm a bit over seeing that, so I've created a different username and DBname:

DEMOUSER/DemoDbUser1@dbhost:1521/demodb

Since Oracle defaults to using port 1521 on the host, you only need to specify the port number if your database is listening on a different port.

An Oracle connection is therefore

user = "DEMOUSER"
passwd = "DemoDbUser1"
servicename = "dbhost:1521/demodb"

try:
    connection = cx_Oracle.connect(user, passwd, servicename)
except cx_Oracle.DatabaseError as dbe:
    print("""Unable to obtain a connection to {servicename}""".format(servicename=servicename))
    raise

With PostgreSQL we can also specify whether we want SSL enabled.

try:
    connection = psycopg2.connect(dbname=dbname,
                                  user=dbuser,
                                  password=dbpassword,
                                  host=dbhost,
                                  port=dbport,
                                  sslmode=dbsslmode)
except psycopg2.OperationalError as dboe:
    print("""Unable to obtain a connection to the database.\n
          Please check your database connection details\n
          Notify dbadmin@{dbhost}""".format(dbhost=dbhost),
          file=sys.stderr, flush=True)
    raise

Once we have the connection, we need a cursor so we can execute statements:

cursor = connection.cursor()

Now that we have our connection and a cursor to use, what can we do with it? SQL baby!

We need a handy dataset to muck around with, so I've got just the thing courtesy of my previous post about determining your electorate. My JSON files have a fairly basic structure reflecting my needs for that project, but that is still sufficient our purposes here.

>>> import json
>>> actf = json.load(open("find-my-electorate/json/ACT.json", "r"))
>>> actf.keys()
dict_keys(['Brindabella', 'Ginninderra', 'Kurrajong', 'Murrumbidgee', 'Yerrabi'])
>>> actf["Yerrabi"].keys()
dict_keys(['jurisdiction', 'locality', 'blocks', 'coords'])

The 'jurisdiction' key is the state or territory name (ACT, Australian Capital Territory in this case), the 'locality' is the electorate name, 'blocks' is a list of the Australian Bureau of Statistics Mesh Block which the electorate contains, and 'coords' is a list of the latitude, longitude pairs which are the boundaries of those blocks.

Now we need a schema to operate within. This is reasonably simple - if you've done any work with SQL databases. To start with, we need two SEQUENCE s, which are database objects from which any database user may generate a unique integer. This is the easiest and cheapest way that I know of to generate values for primary key columns. (For a good explanation on them, see the Oracle 19c CREATE SEQUENCE page). After that, we'll create two tables: electorate and geopoints. (We're going to ignore the Mesh Block data, it's not useful for this example).

-- This is the Oracle DB form
DROP TABLE GEOPOINTS CASCADE CONSTRAINTS;
DROP TABLE ELECTORATES CASCADE CONSTRAINTS;

DROP SEQUENCE ELECTORATE_SEQ CASCADE;
DROP SEQUENCE LATLONG_SEQ CASCADE;

CREATE SEQUENCE ELECTORATE_SEQ INCREMENT BY 1 MINVALUE 1 NOMAXVALUE;
CREATE SEQUENCE LATLONG_SEQ INCREMENT BY 1 MINVALUE 1 NOMAXVALUE;

CREATE TABLE ELECTORATES (
    ELECTORATE_PK   NUMBER DEFAULT DEMOUSER.ELECTORATE_SEQ.NEXTVAL NOT NULL PRIMARY KEY,
    ELECTORATE      VARCHAR2(64) NOT NULL,
    STATENAME       VARCHAR2(3) NOT NULL -- Using the abbreviated form
);

CREATE TABLE GEOPOINTS (
    LATLONG_PK      NUMBER DEFAULT DEMOUSER.LATLONG_SEQ.NEXTVAL NOT NULL PRIMARY KEY,
    LATITUDE        NUMBER NOT NULL,
    LONGITUDE       NUMBER NOT NULL,
    ELECTORATE_FK   NUMBER REFERENCES ELECTORATES(ELECTORATE_PK)
);

-- Now for the PostgreSQL form
DROP TABLE IF EXISTS GEOPOINTS CASCADE;
DROP TABLE IF EXISTS ELECTORATES CASCADE;

DROP SEQUENCE IF EXISTS ELECTORATE_SEQ CASCADE;
DROP SEQUENCE IF EXISTS LATLONG_SEQ CASCADE;

CREATE SEQUENCE ELECTORATE_SEQ INCREMENT BY 1 MINVALUE 1 NO MAXVALUE;
CREATE SEQUENCE LATLONG_SEQ INCREMENT BY 1 MINVALUE 1 NO MAXVALUE;

CREATE TABLE ELECTORATES (
    ELECTORATE_PK   INTEGER DEFAULT NEXTVAL('ELECTORATE_SEQ') NOT NULL PRIMARY KEY,
    ELECTORATE      VARCHAR(64) NOT NULL,
    STATENAME       VARCHAR(3) NOT NULL -- Using the abbreviated form
);

CREATE TABLE GEOPOINTS (
    LATLONG_PK      INTEGER DEFAULT NEXTVAL('LATLONG_SEQ') NOT NULL PRIMARY KEY,
    LATITUDE        NUMERIC NOT NULL,
    LONGITUDE       NUMERIC NOT NULL,
    ELECTORATE_FK   INTEGER REFERENCES ELECTORATES(ELECTORATE_PK) NOT NULL
);

We need to execute these statements one by one, using cursor.execute(). Come back once you've done that and we've got our schema set up.

[I see that a short time has passed - welcome back]

Now we need to populate our database. For both connection types, we'll make use of prepared statements, which allow us to insert, update, delete or select many rows at a time. For the Oracle connection we'll use the executemany() function, but for PostgreSQL's psycopg2 bindings we'll use execute_batch() instead. (See the psycopg2 website note about it).

For the Oracle examples, we'll use a bind variable so that when we INSERT into the ELECTORATES table we get the ELECTORATE_PK to use in the subsequent INSERT to the GEOPOINTS table. That saves us a query to get that information. The psycopg2 module does not have this support, unfortunately. Since we have many records to process, I've created some small functions to enable DRY

# Oracle version
>>> epk = cursor.var(int)
>>> electStmt = """INSERT INTO ELECTORATES (ELECTORATE, STATENAME)
...                VALUES (:electorate, :statename)
...                RETURNING ELECTORATE_PK INTO :epk"""
>>> geoStmt = """INSERT INTO GEOPOINTS(LATITUDE, LONGITUDE, ELECTORATE_FK)
...              VALUES (:lat, :long, :fk)"""

>>> def addstate(statename):
...     for electorate in statename:
...         edict = {"electorate": statename[electorate]["locality"],
...                  "statename": statename[electorate]["jurisdiction"],
...                  "epk": epk}
...         cursor.execute(electStmt, edict)
...         points = list()
...         for latlong in statename[electorate]["coords"]:
...             points.append({"latitude": float(latlong[1]),
...                            "longitude": float(latlong[0]),
...                            "fk": int(epk.getvalue(0)[0])})
...         cursor.executemany(geoStmt, points)
...         connection.commit()

>>> allpoints = dict()
>>> states = ["act", "nsw", "nt", "qld", "sa", "tas", "wa", "vic", "federal"]

>>> for st in states:
...     allpoints[st] = json.load(open("json/{stu}.json".format(stu=st.upper()), "r"))
...     addstate(allpoints[st])

For the PostgreSQL version, we need to subtly change the INSERT statements:

>>> electStmt = """INSERT INTO ELECTORATES (ELECTORATE, STATENAME)
...                VALUES (%(electorate)s, %(statename)s)
...                RETURNING ELECTORATE_PK"""
>>> geoStmt = """INSERT INTO GEOPOINTS(LATITUDE, LONGITUDE, ELECTORATE_FK)
...                VALUES (%(latitude)s, %(longitude)s, %(fk)s)"""

Another thing we need to change is the first execute, because bind variables are an Oracle extension, and we're also going to change from executemany() to execute_batch():

>>> def addstate(statename):
...     for electorate in statename:
...         edict = {"electorate": statename[electorate]["locality"],
...                  "statename": statename[electorate]["jurisdiction"]}
...         cursor.execute(electStmt, edict)
...         epk = cursor.fetchone()[0]
...         points = list()
...         for latlong in statename[electorate]["coords"]:
...             points.append({"latitude": float(latlong[1]),
...                            "longitude": float(latlong[0]), "fk": epk})
...         execute_batch(cursor, geoStmt, points)
...         connection.commit()

The rest of the data loading is the same as with Oracle. Since I'm curious about efficiencies, I did a rough benchmark of the data load with executemany() as well, and it was around half the speed of using execute_batch(). YMMV, of course, so always test your code with your data and as close to real-world conditions as possible.

Now that we have the data loaded we can do some investigations. I'm going to show the PostgreSQL output for this and call out differences with Oracle where necessary.

While some parts of our state and territory borders follow rivers and mountain ranges, quite a lot of them follow specific latitudes and longitudes. The border between Queensland and New South Wales, for example is mostly along the 29th parallel. Between the Northern Territory and South Australia it's the 26th parallel, and that between South Australia and Western Australia is along meridian 129 East.

If you go to a mapping service and plug in 29S,141E (the line between Queensland and New South Wales):

you'll see that the exact point is not where the boundary is actually drawn. That means we need to use some fuzziness in our matching.

>>> cursor.execute("""SELECT STATENAME, ELECTORATE FROM ELECTORATES E WHERE ELECTORATE_PK IN
...                   (SELECT DISTINCT ELECTORATE_FK FROM GEOPOINTS WHERE LATITUDE BETWEEN -29.001 AND -28.995)
...                   ORDER BY STATENAME, ELECTORATE""")
>>> results = cursor.fetchall()
>>> for s, e in results:
...     print("{s:6} {e}".format(s=s, e=e))
...
NSW    Ballina
NSW    Barwon
NSW    Clarence
NSW    Lismore
NSW    New England
NSW    Northern Tablelands
NSW    Page
NSW    Parkes
QLD    Maranoa
QLD    Southern Downs
QLD    Warrego
SA     Giles
SA     Grey
SA     Stuart
SA     Stuart
WA     Durack
WA     Geraldton
WA     Kalgoorlie
WA     Moore
WA     North West Central

Hmm. Not only did I not want SA or WA electorates returned, the database has also given me the federal electorates as well. Let's update our electorates table to reflect that jurisdictional issue:

>>> cursor.execute("""ALTER TABLE ELECTORATES ADD FEDERAL BOOLEAN""")
>>> connection.commit()

Popping into a psql session for a moment, let's see what we have:

demodb=> \d electorates
                                    Table "public.electorates"
    Column     |         Type          | Collation | Nullable |               Default
---------------+-----------------------+-----------+----------+-------------------------------------
 electorate_pk | integer               |           | not null | nextval('electorate_seq'::regclass)
 electorate    | character varying(64) |           | not null |
 statename     | character varying(3)  |           | not null |
 federal       | boolean               |           |          |
Indexes:
    "electorates_pkey" PRIMARY KEY, btree (electorate_pk)
Referenced by:
    TABLE "geopoints" CONSTRAINT "geopoints_electorate_fk_fkey" FOREIGN KEY (electorate_fk) REFERENCES electorates(electorate_pk)

Now let's populate that column. Unfortunately, though, some state electorates have the same name as federal electorates - and some electorate names exist in more than one state, too! (I'm looking at you, Bass!). Tempting as it is to zorch our db and start from scratch, I'm going to take advantage of this information:

We added the federal electorate list after all the states,
the federal list was constructed starting with the ACT, and therefore
Federal electorates will have a higher primary key value than all the states and territories.

With that in mind here's the first federal electorate entry:

demodb=> SELECT ELECTORATE_PK, ELECTORATE, STATENAME FROM ELECTORATES WHERE STATENAME = 'ACT' ORDER BY ELECTORATE_PK, STATENAME;
 electorate_pk |  electorate  | statename
---------------+--------------+-----------
             1 | Brindabella  | ACT
             2 | Ginninderra  | ACT
             3 | Kurrajong    | ACT
             4 | Murrumbidgee | ACT
             5 | Yerrabi      | ACT
           416 | Bean         | ACT
           417 | Canberra     | ACT
           418 | Fenner       | ACT
(8 rows)

Let's check how many electorates have a primary key higher than Bean:

>>> cursor.execute("""SELECT COUNT(ELECTORATE_PK), MAX(ELECTORATE_PK) FROM ELECTORATES""")
>>> cursor.fetchall()
[(566, 566)]

And a quick check to see that we do in fact have 151 electorates with that condition:

>>> 566 - 415
151

We do. Onwards.

>>> cursor.execute("""UPDATE ELECTORATES SET FEDERAL = TRUE WHERE ELECTORATE_PK > 415""")
>>> connection.commit()
>>> cursor.execute("""SELECT COUNT(*) FROM ELECTORATES WHERE FEDERAL IS TRUE""")
>>> cursor.fetchall()
[(151,)]

Likewise, we'll set the others to federal=False:

>>> cursor.execute("""UPDATE ELECTORATES SET FEDERAL = FALSE WHERE ELECTORATE_PK < 416""")
>>> connection.commit()

Back to our queries. I want to see both sorts of electorates, but grouped by whether they are federal or state electorates:

>>> cursor.execute("""SELECT E.STATENAME, E.ELECTORATE, E.FEDERAL FROM ELECTORATES E
...                   WHERE E.ELECTORATE_PK IN
...                       (SELECT DISTINCT ELECTORATE_FK FROM GEOPOINTS WHERE LATITUDE BETWEEN -29.001 AND -28.995)
...                   GROUP BY E.STATENAME, E.ELECTORATE, E.FEDERAL
...                   ORDER BY STATENAME, ELECTORATE""")
>>> fedstate = cursor.fetchall()
>>> fedstate
[('NSW', 'Ballina', False), ('NSW', 'Barwon', False), ('NSW', 'Clarence', False), ('NSW', 'Lismore', False), ('NSW', 'New England', True), ('NSW', 'Northern Tablelands', False), ('NSW', 'Page', True), ('NSW', 'Parkes', True), ('QLD', 'Maranoa', True), ('QLD', 'Southern Downs', False), ('QLD', 'Warrego', False), ('SA', 'Giles', False), ('SA', 'Grey', True), ('SA', 'Stuart', False), ('WA', 'Durack', True), ('WA', 'Geraldton', False), ('WA', 'Kalgoorlie', False), ('WA', 'Moore', False), ('WA', 'North West Central', False)]

>>> def tfy(input):
...     if input:
...         return "yes"
...     else:
...         return ""

>>> for res in fedstate:
...     fmtstr = """{statename:6} {electorate:30} {federal}"""
...     print(fmtstr.format(statename=res[0], electorate=res[1], federal=tfy(res[2])))
...
NSW    Ballina
NSW    Barwon
NSW    Clarence
NSW    Lismore
NSW    New England                    yes
NSW    Northern Tablelands
NSW    Page                           yes
NSW    Parkes                         yes
QLD    Maranoa                        yes
QLD    Southern Downs
QLD    Warrego
SA     Giles
SA     Grey                           yes
SA     Stuart
WA     Durack                         yes
WA     Geraldton
WA     Kalgoorlie
WA     Moore
WA     North West Central

I could have added a WHERE E.FEDERAL = TRUE to the query, too.

Finally, let's see what state electorates in WA and SA are on the border:

>>> cursor.execute("""SELECT STATENAME, ELECTORATE FROM ELECTORATES E
...                   WHERE ELECTORATE_PK IN
...                       (SELECT DISTINCT ELECTORATE_FK FROM GEOPOINTS WHERE LATITUDE BETWEEN -60.00 AND -25.995
                           AND LONGITUDE BETWEEN 128.995 AND 129.1)
...                   AND FEDERAL = FALSE ORDER BY STATENAME, ELECTORATE""")
>>> results = cursor.fetchall()
>>> for _res in results:
...     print("""{statename:6} {electorate}""".format(statename=_res[0], electorate=_res[1]))
...
NT     Namatjira
SA     Giles
WA     Kalgoorlie
WA     North West Central

Why do we have that electorate from the Northern Territory? It's because the coordinates are a bit fuzzy and we had to use a range (the BETWEEN) in our query.

Finally

I apologise, because while this was supposed to be a brief introduction I did get side-tracked with data investigation. I suppose that's an occupational hazard since I'm a Data Engineer working for a company which provides GIS-related services. Anyway, in terms of depth, this was definitely only scratching the surface of what is possible with Python and databases. I encourage you to go and read the Python Database API as well as the SQL reference manual(s) for your chosen database and its binding documentation.

init(Apache Spark)

jmcp — Thu, 10 Oct 2019 16:00:00 GMT

In a previous post I wrote about how I've started down the Data Science path, kicking off with an exploration of sentiment analysis for political tweets. This is a topic which I will come back to in the future, not least because the nltk corpus I've made use of for au-pol-sentiment is based on British political tweets. While Australia and Britain share a common political heritage, I'm not completely confident that our political discourse is quite covered by that corpus.

In the meantime, another aspect of Data Science in practice is the use of an ecosystem called Apache Spark. Leaving aside my 20+ years of muscle memory spelling it as SPARC, this is a Machine Learning engine, described on the homepage as a unified analytics engine for large-scale data processing.

My experience is that when I want or need to learn a new toolkit or utility, the best way to do so is by trying to directly solve a specific problem with it. One problem (ok, not really a problem, more a set of questions) I have is that with all of the data I've gathered since 2013 from my solar inverter I'm dependent on pvoutput.org for finding per-year and per-month averages, maxima and minima. So with a data science-focused job interview (with Oracle Labs) approaching, I decided to get stuck in and get started with Apache Spark.

The first issue I faced was implementing the ETL pipeline. I have two types of files to load - the first contains the output from solarmonj, the second has the output from my solar inverter script.

Here's the first schema form:

Field name	Units
Timestamp	seconds-since-epoch
Temperature	float (degrees C)
energyNow	float (Watts)
energyToday	float (Watt-hours)
powerGenerated	float (Hertz)
voltageDC	float (Volts)
current	float (Amps)
energyTotal	float (Watt-hours)
voltageAC	float (Volts)

The second schema is from jfy-monitor, and has this schema:

Field name	Units
Timestamp	ISO8601-like ("yyyy-MM-dd'T'HH:mm:ss")
Temperature	float (degrees C)
PowerGenerated	float (Watts)
VoltageDC	float (Volts)
Current	float (Amps)
EnergyGenerated	float (Watts-Hours)
VoltageAC	float (Volts)

There are two other salient pieces of information about these files. The first is that the energyTotal and EnergyGenerated fields are running totals of the amount of energy generated on that particular day. The second is that in the first version of the schema, energyTotal needs to be multiplied by 1000 to get the actual KW/h value.

With that knowledge ready, let's dive into the code.

The first step is to start up a Spark session:

from pyspark.sql.functions import date_format
from pyspark import SparkContext
from pyspark.sql import SparkSession

# Basic Spark session configuration
sc = SparkContext("local", "PV Inverter Analysis")
spark = SparkSession(sc)

# We don't need most of this output
log4j = sc._jvm.org.apache.log4j
log4j.LogManager.getRootLogger().setLevel(log4j.Level.ERROR)

I observed while prototyping this in the pyspark REPL environment that if I didn't turn the logging output right down, then I'd see squillions of INFO messages.

The second step is to generate a list of files. As you might have guessed, I've got a year/month/day hierarchy - but the older files have a csv extension. To get those files (and since I want to be able to process any given year or year+month combination), I need to use some globbing:

import glob

def generateFiles(topdir, year, month):
    """Construct per-year dicts of lists of files"""
    allfiles = {}
    kkey = ""
    patterns = []
    months = []
    # Since some of our data dirs have months as bare numbers and
    # others have a prepended 0, let's match them correctly.
    if month:
        if month < 10:
            months = [month, "0" + str(month)]
        else:
            months = [month]
    if year and month:
        patterns = ["{year}/{monthp}/**".format(year=year, monthp=monthp)
                    for monthp in months]
        kkey = year
    elif year:
        patterns = ["{year}/*/**".format(year=year)]
        kkey = year

    if patterns:
        globs = []
        for pat in patterns:
            globs.extend(glob.glob(os.path.join(topdir, pat)))
        allfiles[kkey] = globs
    else:
        for yy in allYears:
            allfiles[yy] = glob.glob(os.path.join(topdir,
                                                  "{yy}/*/*".format(yy=yy)))
    return allfiles

To load in each file, I turned to the tried-and-true Python standard module csv, and - rather than having a v1 and v2 processing function, I model DRY and use an input argument to determine which set of elements to match:

import csv
from datetime import datetime

def importCSV(fname, isOld):
    output = []
    if isOld:
        multiplier = 1000.0
    else:
        multiplier = 1.0

    csvreader = csv.reader(open(fname).readlines())
    for row in csvreader:
        try:
            if isOld:
                (tstamp, temp, _enow, _etoday, powergen, vdc,
                 current, energen, vac) = row
            else:
                (tstamp, temp, powergen, vdc, current, energen, vac) = row
        except ValueError as _ve:
            # print("failed at {row} of {fname}".format(row=row, fname=fname))
            continue

        if "e" in temp:
            # invalid line, skip it
            continue

        if isOld:
            isostamp = datetime.fromtimestamp(int(tstamp))
        else:
            isostamp = datetime.fromisoformat(tstamp)

        output.append({
            "timestamp": isostamp,
            "Temperature": float(temp),
            "PowerGenerated": float(powergen),
            "VoltageDC": float(vdc),
            "Current": float(current),
            "EnergyGenerated": float(energen) * multiplier,
            "VoltageAC": float(vac)})
    return output

Now we get to the Apache Spark part. Having got a dictionary of anonymous dicts I can turn them into an Resilient Distributed Dataset (RDD) and thence a DataFrame. I chose the DataFrame model rather than a Row because that matches up nicely with my existing data format. For other applications (such as when I extend my Twitter Sentiment Analysis project with the streaming API) I'll use the Row datatype instead.

def now():
    """ Returns an ISO8601-formatted (without microseconds) timestamp"""
    return datetime.now().strftime("%Y-%M-%dT%H:%m:%S")

allFiles = generateFiles("data", qyear, qmonth)

print(now(), "Importing data files")

for k in allFiles:
    rddyear = []
    for fn in allFiles[k]:
        if fn.endswith(".csv"):
            rddyear.extend(importCSV(fn, True))
        else:
            rddyear.extend(importCSV(fn, False))
    rdds[k] = rddyear

print(now(), "All data files imported")

for year in allYears:
    rdd = sc.parallelize(rdds[year])
    allFrames[year] = rdd.toDF()
    newFrame = "new" + str(year)
    # Extend the schema for our convenience
    allFrames[newFrame] = allFrames[year].withColumn(
        "DateOnly", date_format('timestamp', "yyyyMMdd")
    ).withColumn("TimeOnly", date_format('timestamp', "HHmmss"))
    allFrames[newFrame].createOrReplaceTempView("view{year}".format(
        year=year))

print(now(), "Data transformed from RDDs into DataFrames")

The reason I chose to extend the frames with two extra columns is because when I search for the record dates (minimum and maximum), I want to have a quick SELECT which I can aggregate on.

ymdquery = "SELECT DISTINCT DateOnly from {view} WHERE DateOnly "
ymdquery += "LIKE '{yyyymm}%' ORDER BY DateOnly ASC"

for year in allYears:
    for mon in allMonths:
        if mon < 10:
            yyyymm = str(year) + "0" + str(mon)
        else:
            yyyymm = str(year) + str(mon)

        _dates = spark.sql(ymdquery.format(
            view=view, yyyymm=yyyymm)).collect()
        days = [n.asDict()["DateOnly"] for n in _dates]

        _monthMax = frame.filter(
            frame.DateOnly.isin(days)).agg(
                {"EnergyGenerated": "max"}).collect()[0]
        monthMax = _monthMax.asDict()["max(EnergyGenerated)"]

I keep track of each day's maximum, and update my minval and minDay as required. All this information is then stored in a per-month dict, and then in a per-year dict.

The last stage is to print out the record dates, monthly and yearly totals, averages and other values.

Running this utility on my 4-core Ubuntu system at home, I get what I believe are ok timings for whole-year investigations, and reasonable timings if I check a specific month.

When I run the utility for January 2018, the output looks like this:

(v-3.7-linux) flerken:solar-spark $ SPARK_LOCAL_IP=0.0.0.0 time -f "%E"  spark-submit --executor-memory 2G --driver-memory 2G solar-spark.py  -y 2018 -m 1
19/10/15 12:23:57 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark default log4j profile: org/apache/spark/log4j-defaults.properties

[Most INFO-level output elided]

19/10/15 12:23:58 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/10/15 12:23:58 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://0.0.0.0:4040

2019-23-15T12:10:59 Importing data files
2019-23-15T12:10:59 All data files imported
/space/jmcp/web/v-3.7-linux/lib/python3.7/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/session.py:366: UserWarning: Using RDD of dict to inferSchema is deprecated. Use pyspark.sql.Row instead
2019-24-15T12:10:01 Data transformed from RDDs into DataFrames
2019-24-15T12:10:01 Analysing 2018
2019-24-15T12:10:01          January
2019-24-15T12:10:15 All data analysed
2019-24-15T12:10:15 2018 total generation: 436130.00 KW/h
2019-24-15T12:10:15         January total:               436130.00 KW/h
2019-24-15T12:10:15         Record dates for January:    Max on 20180131 (16780.00 KW/h), Min on 20180102 (10560.00 KW/h)
2019-24-15T12:10:15         Average daily generation  14068.71 KW/h
2019-24-15T12:10:15 ----------------

0:18.76

While that processing is going on, you can see a dashboard with useful information about the application at http://localhost:4040:

You can find the code for this project in my GitHub repo solar-spark.

Microservices (part 2)

jmcp — Thu, 03 Oct 2019 16:00:00 GMT

One principle that I work on is that I should always extend the fix (learnt via the Kepner-Tregoe Analytical Troubleshooting training many years ago). Following my investigation of how to provide a more accessible method of determining your electorate, I came back to the political polling ideas and got to thinking about how we can track the temperature of a conversation in, for example #auspol.

The term for this is sentiment analysis and while the major cloud providers have their own implementations of this (Microsoft Azure Text Analytics, AWS Comprehend, Google Cloud Natural Language API) you can also use Python's nltk in the comfort of your own venv. It's cheaper, too!

A bit of searching lead me to @Chapagain's post which was very useful and got me started - thankyou

I decided that I really want to do something more real-time, and while I could have done more scraping with Beautiful Soup, a quick look at the html that's returned with you run

import requests

url = "https://twitter.com/search?q=%23auspol&src=typed_query&f=live"
result = requests.get(url)
print(result.text)

is eye-wateringly complex. (Go on, try it!) I just couldn't be bothered with that so I signed up for a Twitter developer account and started looking at the APIs available for searching. These are easily used with the Twython library:

from twython import Twython

twitter = Tython(consumer_key, consumer_secret,
                 access_token, access_token_secret)
hashtag = "#auspol"
results = twitter.search(q=hashtag, result_type="recent")
for tweet in results["statuses"]:
    sentiment = classifier_func(tweet["text"])
    print(sentiment.prob("pos"))

I hit the rate limit a few times until I realiased that there was a while true going on inside Twython when using the cursor method. In my print-to-shell proof of concept, I got past that by using the search function inside a while loop with a 30second sleep call. I knew that that wasn't good enough for a web app, and would actually be a road block for doing a properly updated graph-focused page.

For that I would need a charting library, and some JavaScript. I started out using Chart.js, but quickly realised that it didn't have any sort of flow, so then I retooled to use C3js instead.

The initial render of the template provides the first set of data, which is a JavaScript array ([]), and checks for a saved hashtag and the id of the most recently found tweet using Window.sessionStorage(). Then we set up a function to get new data when called:

function getNewData() {
    var xhr = new XMLHttpRequest();
    xhr.open("GET", "/sentiment?hashtag="+hashtag+"&lastid="+lastid, true);
    xhr.onload = function (e) {
        if ((xhr.readyState === 4) && (xhr.status === 200)) {
            parsed = JSON.parse(xhr.responseText);
            sessionStorage.setItem("lastid", parsed["lastid"]);
            lastid = parsed["lastid"];
            labels = parsed["labels"];
            // Did we get new data points?
            if (parsed["chartdata"].length > 1) {
                // yes
                newDataCol = [ parsed["chartdata"] ];
                curidx += parsed["chartdata"].length - 1;
            } else {
                newDataCol = [];
            }
        }
    };
    xhr.onerror = function (e) {
        console.error(xhr.statusText);
    };
    xhr.send(null);
};

Finally, we need to tell the window to call our updateChart() function every 30 seconds, and define that function:

function updateChart() {
    getNewData();
    if (newDataCol !== []) {
        chart.flow({
            columns: prevDataCol,
            done: function () {
               chart.flow({
                   columns: newDataCol,
                   line: { connectNull: true }
               })
            }
        });
        prevDataCol = newDataCol;
    }
};

/* Update the chart every 30 seconds */
window.setInterval(updateChart, 30000);

So that you can change the hashtag to watch, I added a small <form> element which POSTs the new hashtag to the /sentiment method on submit and then re-renders the template.

What I'm particularly happy with is that the JavaScript took me only a few hours last Saturday morning and was pretty straightforward to write.

You can find the code for this project in my GitHub repo au-pol-sentiment.

Microservices (part 1)

jmcp — Thu, 26 Sep 2019 16:00:00 GMT

Since I departed from my comfortable niche in Solaris engineering earlier this year, I've spent a considerable amount of time and energy in re-training and upskilling to assist my employment prospects. Apart from acquainting myself with a lot of terminology, I've written code. A lot of code. Mostly, as it turns out, has been related to microservices.

This post is about a microservice I wrote to assist with accessibility in a specific part of the Australian electoral process (finding out which electorate you live in) and some supporting digressions.

You can find all the code for this microservice and its associated data preparation in my GitHub repos grabbag and find-my-electorate.

On 18 May 2019, Australia had a federal election, and in the lead up to that event I became very interested in political polling. While I have a few ideas on the subject which are on my back burner, mind-mapping the various components of political polling got to wondering how do the various state, territory and federal electoral commissions map a voter's address to an electorate?

My first point of call was the Australian Electoral Commission and their Find my electorate site. This is nicely laid out and lets you find out which electorate you are in - by postcode. This is all well and good if you're in a densely populated area, like the electorate of Brisbane - three suburbs. If, however, you choose somewhere else, like 2620 which covers a lot of Canberra and surrounding districts, you wind up with several electorates covering 2620.

The AEC's website is written in asp.net, which is up to the task, but when you have more than one page of results the authors of the page make use of some (to my mind) squirrelly features and callbacks which make scraping the site difficult. As best I can determine, the AEC doesn't provide an API to access this information, so Another Method was required.

At this point, I turned to the standard libraries for this sort of thing in the Python worldL Beautiful Soup and requests. I started by setting up a quick venv to keep the dependencies contained

$ python3.7 -m venv scraping-venv
$ . scraping/bin/activate
(scraping-venv) $ pip install requests bs4 json csv

Now since we know the url to GET, we can get the first page of responses very easily:

import requests
from bs4 import BeautifulSoup

url =  "https://electorate.aec.gov.au/LocalitySearchResults.aspx?"
url += "filter={0}&filterby=Postcode"

result = requests.post(url.format(2620))
resh = BeautifulSoup(result.text, "html.parser")

Beautiful Soup parses the response text, and gives us a tree-like structure to work with. Making use of the Chrome devtools (or the Firefox devtools ) I could see that I need to find a <table> with an attribute of *ContentPlaceHolderBody_gridViewLocalities* - what a mouthful! - and then process all the table rows (<tr>) within that table

tblAttr = "ContentPlaceHolderBody_gridViewLocalities"
restbl = resh.find_all(name="table", attrs={"id": tblAttr})
rows = restbl[0].find_all("tr")

Using a for loop we can construct a dict of the data that we actually need. Simple!

What do we do, though, when we want to get the second or later pages of result? This is where the squirrelly features and callbacks come in. The page makes use of an __EVENTARGUMENT element which is POST ed as payload back to the same url. The way that we determine this is to look for a row with the class pagingLink, then for each table data (<td>) element check for its contents matching this regex

".*__doPostBack.'(.*?gridViewLocalities)','(Page.[0-9]+)'.*")

And after that we can recursively call our query with the extra payload data in the argument list:

def queryAEC(postcode, extrapage):
    """
    Queries the AEC url and returns soup. If extrapage is empty
    then we pass the soup to findFollowups before returning.
    """
    url = "https://electorate.aec.gov.au/LocalitySearchResults.aspx?"
    url += "filter={0}&filterby=Postcode"

    if not extrapage:
        res = requests.post(url.format(postcode))
    else:
        payload["__EVENTARGUMENT"] = extrapage
        res = requests.post(url.format(postcode), data=payload)

I now had a script to run which extracted this info and pretty-printed it (as well as the same info in JSON):

$ ./postcode.py 2620
State    Postcode   Locality                         Electorate
ACT      2620       BEARD                            Canberra
ACT      2620       BOOTH DISTRICT                   Bean
NSW      2620       BURRA                            Eden-Monaro
NSW      2620       CARWOOLA                         Eden-Monaro
NSW      2620       CLEAR RANGE                      Eden-Monaro
ACT      2620       CORIN DAM                        Bean
NSW      2620       CRESTWOOD                        Eden-Monaro
NSW      2620       ENVIRONA                         Eden-Monaro
NSW      2620       FERNLEIGH PARK                   Eden-Monaro
NSW      2620       GOOGONG                          Eden-Monaro
NSW      2620       GREENLEIGH                       Eden-Monaro
NSW      2620       GUNDAROO                         Eden-Monaro
ACT      2620       HUME                             Bean
NSW      2620       KARABAR                          Eden-Monaro
ACT      2620       KOWEN DISTRICT                   Canberra
ACT      2620       KOWEN FOREST                     Canberra
NSW      2620       MICHELAGO                        Eden-Monaro
ACT      2620       OAKS ESTATE                      Canberra
ACT      2620       PADDYS RIVER DISTRICT            Bean
NSW      2620       QUEANBEYAN                       Eden-Monaro
NSW      2620       YARROW                           Eden-Monaro
NSW      2620       QUEANBEYAN EAST                  Eden-Monaro
NSW      2620       QUEANBEYAN WEST                  Eden-Monaro
NSW      2620       RADCLIFFE                        Eden-Monaro
ACT      2620       RENDEZVOUS CREEK DISTRICT        Bean
ACT      2620       ROYALLA                          Bean
NSW      2620       ROYALLA                          Eden-Monaro
NSW      2620       SUTTON                           Eden-Monaro
ACT      2620       TENNENT DISTRICT                 Bean
ACT      2620       THARWA                           Bean
NSW      2620       THARWA                           Eden-Monaro
NSW      2620       THARWA                           Eden-Monaro
NSW      2620       THE ANGLE                        Eden-Monaro
NSW      2620       THE RIDGEWAY                     Eden-Monaro
NSW      2620       TINDERRY                         Eden-Monaro
NSW      2620       TRALEE                           Eden-Monaro
ACT      2620       TUGGERANONG DISTRICT             Bean
NSW      2620       URILA                            Eden-Monaro
NSW      2620       WAMBOIN                          Eden-Monaro
ACT      2620       WILLIAMSDALE                     Bean
NSW      2620       WILLIAMSDALE                     Eden-Monaro

That really is quite a few suburbs.

So now that we've got a way to extract that information, how do we make it available and useful for everybody? With a microservice! I hear you cry.

The very first microservice I wrote (in 2011-12, the subject of a future post) used CherryPy, because we'd embedded it within Solaris IPS (image packaging system) and didn't need any further corporate approvals. The path of least resistance. This time, however, I was unconstrained regarding approvals, so had to choose between Django and flask. For no particular reason, I chose flask.

It was pretty easy to cons up the requisite templates, and write the /results method. It was at this point that my extend the fix habit (learnt via the Kepner-Tregoe Analytical Troubleshooting training many years ago) kicked in, and I started exploring the Electoral Commission of Queensland website for the same sort of information. To my surprise, the relatively straight-forward interface of the AEC was not available, and the closest analogue was an interactive map.

After a brief phone conversation with ECQ and more digging, I discovered that the 2017 boundaries were available from QLD Spatial in shapefile, MapInfo and Google Maps KML formats. This was very useful, because KML can be mucked about with directly using Beautiful Soup. After not too much effort I had the latitude+longitude pairs for the boundaries extracted and stored as JSON . My phone conversation with ECQ also took me down the path of wanting to translate a street address into GeoJSON - and that took me to the Google Maps API. I did investigate OpenStreetMap's api, but testing a few specific locations (addresses where we've lived over the years) gave me significantly different latitude+longitude results. I bit the bullet and got a Google Maps API key .

The next step was to research how to find out if a specific point is located within a polygon, and to my delight the Even-odd rule has example code in Python, which needed only a small change to work with my data arrangement.

With that knowledge in hand, it was time to turn the handle on the Google Maps API :

keyarg = "&key={gmapkey}"
queryurl = "https://maps.googleapis.com/maps/api/geocode/json?address="
queryurl += "{addr} Australia"
queryurl += keyarg

...
# Helper functions
def get_geoJson(addr):
    """
    Queries the Google Maps API for specified address, returns
    a dict of the formatted address, the state/territory name, and
    a float-ified version of the latitude and longitude.
    """
    res = requests.get(queryurl.format(addr=addr, gmapkey=gmapkey))
    dictr = {}
    if res.json()["status"] == "ZERO_RESULTS" or not res.ok:
        dictr["res"] = res
    else:
        rresj = res.json()["results"][0]
        dictr["formatted_address"] = rresj["formatted_address"]
        dictr["latlong"] = rresj["geometry"]["location"]
        for el in rresj["address_components"]:
            if el["types"][0] == "administrative_area_level_1":
                dictr["state"] = el["short_name"]
    return dictr

When you provide an address, we send that to Google which does a best-effort match on the text address then returns GeoJSON for that match. For example, if you enter

42 Wallaby Way, Sydney

the best-effort match will give you

42 Rock Wallaby Way, Blaxland NSW 2774, Australia

I now had a way to translate a street address into a federal electorate, but with incomplete per-State data my app wasn't finished. I managed to get Federal, Queensland, New South Wales, Victoria and Tasmania data fairly easily (see the links below) and South Australia's data came via personal email after an enquiry through their contact page. I didn't get any response to several contact attempts with either Western Australia or the Northern Territory, and the best I could get for the ACT was their electorate to suburb associations.

I remembered that the Australian Bureau of Statistics has a standard called Statistical Geography, and the smallest unit of that is called a Mesh Block:

Mesh Blocks (MBs) are the smallest geographical area defined by the ABS. They are designed as geographic building blocks rather than as areas for the release of statistics themselves. All statistical areas in the ASGS, both ABS and Non ABS Structures, are built up from Mesh Blocks. As a result the design of Mesh Blocks takes into account many factors including administrative boundaries such as Cadastre, Suburbs and Localities and LGAs as well as land uses and dwelling distribution. (emphasis added)

Mesh Blocks are then aggregated into SA1s:

Statistical Areas Level 1 (SA1s) are designed to maximise the spatial detail available for Census data. Most SA1s have a population of between 200 to 800 persons with an average population of approximately 400 persons. This is to optimise the balance between spatial detail and the ability to cross classify Census variables without the resulting counts becoming too small for use. SA1s aim to separate out areas with different geographic characteristics within Suburb and Locality boundaries. In rural areas they often combine related Locality boundaries. SA1s are aggregations of Mesh Blocks. (emphasis added)

With this knowledge, and a handy SA1-to-postcode map in CSV format

$ head australia-whole/SED_2018_AUST.csv
SA1_MAINCODE_2016,SED_CODE_2018,SED_NAME_2018,STATE_CODE_2016,STATE_NAME_2016,AREA_ALBERS_SQKM
10102100701,10031,Goulburn,1,New South Wales,362.8727
10102100702,10053,Monaro,1,New South Wales,229.7459
10102100703,10053,Monaro,1,New South Wales,2.3910
10102100704,10053,Monaro,1,New South Wales,1.2816
10102100705,10053,Monaro,1,New South Wales,1.1978
....

I went looking into the SA1 information from the ABS shapefile covering the whole of the country. Transforming the shapefile into kml is done with ogr2ogr and provides us with an XML schema definition. From the CSV header line above we can see that we want the SA1_MAINCODE_2016 and (for validation) the STATE_NAME_2016 fields. Having made a per-state list of the SA1s, we go back to the kml and process each member of the document:

  <gml:featureMember>
    <ogr:SED_2018_AUST fid="SED_2018_AUST.0">
      <ogr:geometryProperty>
        <gml:Polygon srsName="EPSG:4283">
          <gml:outerBoundaryIs>
            <gml:LinearRing>
              <gml:coordinates>
  ....
              </gml:coordinates>
            </gml:LinearRing>
          </gml:outerBoundaryIs>
        </gml:Polygon>
      </gml:polygonMember>
    </ogr:geometryProperty>
    <ogr:SED_CODE18>30028</ogr:SED_CODE18>
    <ogr:SED_NAME18>Gladstone</ogr:SED_NAME18>
    <ogr:AREASQKM18>2799.9552</ogr:AREASQKM18>
  </ogr:SED_2018_AUST>
</gml:featureMember>

The gml:coordinates are what we really need, they're space-separated lat,long pairs.

for feature in sakml.findAll("gml:featureMember"):
sa1 = feature.find("ogr:SA1_MAIN16").text
mb_coord[sa1] = mb_to_points(feature)

for block in mb_to_sed:
    electorate = mb_to_sed[block]
    sed_to_mb[electorate]["coords"].extend(mb_coord[block])

After which we can write each jurisdiction's dict of localities and lat/long coordinates out as JSON using json.dump(localitydict, outfile).

To confirm that I had the correct data, I wrote another simple quick-n-dirty script jsoncheck.py to diff the SA1-acquired JSON against my other extractions. There was one difference of importance found - Queensland has a new electorate McConnel, which was created after the most recent ABS SA1 allocation.

So that's the data preparation done, back to the flask app! The app listens at the root (/), and is a simple text form. Hitting enter after typing in an address routes the POST request to the results function where we call out to the Google Maps API, load the relevant state JSONified electorate list, and then locate the Federal division. There are 151 Federal divisions, so it's not necessarily a bad thing to search through each on an alphabetic basis and break when we get a match. I haven't figured out a way to (time and space)-efficiently hash the coordinates vs divisions. After determining the Federal division we then use the same method to check against the identified state's electorate list.

The first version of the app just returned the two electorate names, but I didn't think that was very friendly, so I added another call to the Google Maps API to retrieve a 400x400 image showing the supplied address on the map; clicking on that map takes you to the larger Google-hosted map. I also added links to the Wikipedia entries for the Federal and state electorates. To render the image's binary data we use b64encode:

from base64 import b64encode

keyarg = "&key={gmapkey}"
imgurl = "https://maps.googleapis.com/maps/api/staticmap?size=400x400"
imgurl += "&center={lati},{longi}&scale=1&maptype=roadmap&zoom=13"
imgurl += "&markers=X|{lati},{longi}"
imgurl += keyarg

# Let's provide a Google Maps static picture of the location
# Adapted from
# https://stackoverflow.com/questions/25140826/generate-image-embed-in-flask-with-a-data-uri/25141268#25141268
#
def get_image(latlong):
    """
    latlong -- a dict of the x and y coodinates of the location
    Returns a base64-encoded image
    """
    turl = imgurl.format(longi=latlong["lng"],
                         lati=latlong["lat"],
                         gmapkey=gmapkey)
    res = requests.get(turl)
    return b64encode(res.content)


....

# and in the results function
    img_data = get_image(dictr["latlong"])

    return render_template("results.html",
                           ...,
                           img_data=format(quote(img_data))
                           ...)

Putting that all together gives us a rendered page that looks like this:

To finalise the project, I ran it through flake8 again (I do this every few saves), and then git commit followed by git push.

Reference data locations

Jurisdiction	URL
Federal	aec.gov.au/Electorates/gis/gis_datadownload.htm
Queensland	qldspatial.information.qld.gov.au/catalogue/custom/detail.page?fid={079E7EF8-30C5-4C1D-9ABF-3D196713694F}
New South Wales	elections.nsw.gov.au/Elections/How-voting-works/Electoral-boundaries
Victoria	ebc.vic.gov.au/ElectoralBoundaries/FinalElectoralBoundariesDownload.html
Tasmania	www.tec.tas.gov.au/House_of_Assembly_Elections/index.html

Tasmania's state parliament has multi-member electorates, which have the same boundaries as their 5 Federal divisions.

South Australia data was provided via direct personal email.

Australian Capital Territory, Western Australia and Northern Territory data was extracted from the ABS shapefile after ogr2ogr-converting from MapInfo Interchange Format.

Where did the old posts go?

jmcp — Tue, 15 May 2018 00:00:00 GMT

When I set up my blog in 2006, I chose to use roller, which was the same engine that the now-defunct blogs.sun.com used at the time. Later, having gotten tired of the interface and being more impressed with Wordpress' facility for image galleries, I started running my own instance of that.

After a while, however, the frequent CVEs in both PHP and the Wordpress base+plugin systems got me sufficiently motivated to change that I did.

My friends Shawn and Liane suggested going for a static site generator, so after having a brief look at Pelican and Nikola, I instead chose Hugo.

Hugo's chief attraction was the relative ease with which I could create image galleries, along with the ability to import Wordpress sites. Yay, thought I, I can have a fairly seamless transition, and away we went.

I didn't actually check the site backup that I made before turning off the Wordpress instance, however, and when I went looking for the pkgrepo procedure that I used for building darktable and didn't find the content that I wanted, I was a bit annoyed.

Given that Solaris' packaged version of Go is somewhat behind the community version, and that Hugo depends on a much newer version, this was also the trigger for me to re-explore Pelican and Nikola, both of which are written in Python. After a brief flirtation with Pelican I settled instead on Nikola and did the initial Hugo to Nikola migration fairly easily. Chris pointed me to the gallery directive plugin and I was able to make a start with some of my more recent gallery collections. A quick implementation of a PR for captioned and ordered images got me the rest of the way and then I could get back to the real problem: the missing content.

Fortunately for me, the wayback machine had a copy of the old site entries, and with a quick installation of the wayback machine downloader I was able to grab the whole site as it was up to 2016.

Phew!

Except that all the post files were chock full of Wordpress and roller's javascript and expanded css, which were a real mess to look at, let alone extract the posts from.

So I did what anybody else would do, and wrote some Python using BeautifulSoup to provide a best effort extraction which would translate the html+js+css into the plain-text (and therefore portable) ReStructured Text format. Now since this is a best effort attempt, I'm not too concerned about getting the output as perfect rst which matches my original post, and I went through about 20 different entries to tidy up the input so that the script would produce something close to what I wanted. I knew that I'd have to go and post-process quite a few entries as well, I just wanted to not have to do too much to get that going.

I've converted about 340 posts extracted from the wayback machine archive using this script wp-to-rest.py which took about 2 days to write and finesse, and another day or so to muck around with several posts to get them into better shape (ie, running nikola build doesn't yell at me). There are still a bunch of broken links in there, but at least now I've got all my content back, and can very easily fix things up as I get the inclination.

Monitoring my PV Inverter

jmcp — Mon, 02 Apr 2018 18:00:00 GMT

I'd like to share with you a way to build on the Solaris Analytics components sstored and WebUI that I use with our PV inverter.

In April 2013 we had 15 solar panels installed on our roof, providing 3.9KW. This came with a JFY SunTwins 5000TL inverter, which has the useful feature of data monitoring via an RS232 serial port. The installers provided a cd with a 32bit Windows app which, while useful to start with, did not allow me to push the generation data up to a service like pvoutput.org.

Not being interested in leaving a laptop running Windows on during daylight hours, I searched for open source monitoring software, finding Jonathan Croucher's solarmonj via whirlpool. Importantly, I also found a programmer reference manual for the inverter, although it is poorly translated. I also discovered that there are a number of gaps between what it describes and what packet analysis and the windows app actually do.

I'm not too keen on C++, but it built fine on the raspberry pi that I had available to use, and ran well enough. With a bit of shell scripting around it I was able to upload to pvoutput.org and see how things were going.

Solarmonj has some attributes which I dislike: it's not (wasn't, at that time) a daemon, logfile generation isn't enterprise-y, it doesn't take command line arguments (so the device path is compiled in), doesn't handle multiple inverters from the same daemon, and doesn't let you send any arbitrary command listed in the spec document. It's single purpose - but performs that purpose well enough.

(Checking out Jonathan's github repo, I see that the version I was using did in fact get an update about 3 years ago, but I had the utility in "set and forget" mode, so never noticed).

I've written a new monitor which builds on Croucher's work, with these features: - written in Python, daemonises, is configurable, handles multiple inverters, updates pvoutput.org and sstored as well as writing to a local file.

With this project I'm also providing sstored configuration files and a WebUI sheet:

Zerothly, you can get all the code for this project from my Github repo. It's still a work in progress but it's at the point of being sufficient for my needs so I'm happy to share it.

Since this post is about how to plug your app into Solaris Analytics, I won't delve too much into the SMF and IPS components of the code.

At the end of my previous post on the work blog I mentioned that I would discuss using the C and Python bindings for the Stats Store. This just post covers Python bindings, leaving detailed coverage of the C interface for another day.

First of all, let's have a look at two basic architecture diagrams:

The C and Python bindings enable read-write access to sstored, so that you can write your own provider. We call this a "userspace provider" because it operates outside of the kernel of sstored.

For both bindings, we have three methods of putting data into sstored:

per-point synchronous (using a door_call())
per-point asynchronous (using a shared memory region)
bulk synchronous (using a door_call())

The code that I've written for this utility is using the asynchronous method which (at the bottom of the stack) depends on an mmap region which is shared between the daemon and the client process. Since we do not have a real speed constraint for updating sstored, I could have used the synchronous method. I'll discuss the bulk synchronous method later.

To start with, my daemon needs a connection to sstored. Assuming that the service is online, this is very simple:

from libsstore import SStore, SSException

sst = SStore()

(I've written the daemon so that each attached inverter has its own connection to sstored, so sst is a thread instance variable).

The user that you run this code as must have these authorizations:

solaris.sstore.update.res
solaris.sstore.write

Add these to the user by uttering

# usermod -A +solaris.sstore.update.res,solaris.sstore.write $USER

Once you've got those authorizations sorted, you can add the appropriate resource to the class. I've chosen to name the resources with each inverter's serial number. My device's serial number is 1522130110183:

RESOURCE_SSID_PREFIX = "//:class.app/solar/jfy//:res.inverter/"
STATS = [
    "temperature",
    "power-generated",
    "voltage-dc",
    "current",
    "energy-generated",
    "voltage-ac"
    ]

# hr is Human-Readable, after we've processed the binary response
# from the inverter
hr_serial = 1522130110183
resname = RESOURCE_SSID_PREFIX + hr_serial

try:
    sst.resource_add(resname)
    self.print_warnings()
except SSException as exc:
    print("Unable to add resource {0} to sstored: {1}".format(
        resname, SSException.__str__))
    usesstore = False
    raise

stats = []
for sname in STATS:
        stats.append("{0}{1}//:stat.{2}".format(
            RESOURCE_SSID_PREFIX, hr_serial, sname))
try:
    stats_array = self.sst.data_attach(stats)
    print_warnings()
except SSException as exc:
    print("Unable to attach stats to sstored\n{0} / {1}".format(
        exc.message, exc.errno), file=sys.stderr)
    usesstore = False
    sst.free()
    sst = None

Each time we query the inverter, we get back binary data which needs decoding and extracting. This is the "feature" of the documentation which annoys me most: it doesn't match the data packet returned, so I had to go and click through the inverter's front panel while watching retrieved values so I could determine the field names and units. Ugh. Anyway, inside the thread's run() method:

stats = query_normal_info()
if not stats:
    return
if usesstore:
    sstore_update(stats)

Now comes the magic:

def sstore_update(vals):
    """
    Updates the stats in sstored after stripping out the ignore[12]
    fields in JFYData. We're using the shared memory region method
    provided by data_attach(), so this is a very simple function.
    """

    values = {}
    for idx, fname in enumerate(JFYData):
        values[self.stats[idx]] = vals[fname] / JFYDivisors[idx]
    sst.data_update(values)

Then we go back to sleep for 30 seconds, and repeat.

An essential part of this project are the JSON metadata files that we provide to the Stats Store. Without these, the daemon does not know where the class, resources and statistics fit inside the namespace, nor does it know what units or description to provide when we run sstore info for any of these statistics.

All resources underneath a class must have the same statistics, and we need to decide on the resource namespace prior to adding the class to sstored. Here is the file class.app.solar.jfy.json, which in the service/jfy package I deliver to /usr/lib/sstore/metadata/json/site:

{
    "$schema": "//:class",
    "copyright": "Copyright (c) 2018, James C. McPherson. All rights reserved.",
    "description": "JFY Solar Inverter monitor",
    "id": "app/solar/jfy",
    "namespaces": [
        {
            "name-type": "string",
            "resource-name": "inverter"
        }
    ],
    "stability": "stable",
    "stat-names": [
        "//:stat.temperature",
        "//:stat.power-generated",
        "//:stat.voltage-dc",
        "//:stat.current",
        "//:stat.voltage-ac",
        "//:stat.energy-generated"
    ]
}

This is validated by the daemon using the schemas shipped in /usr/lib/sstore/metadata/json-schema, and comes with a companion file stat.app.solar.jfy.json. On your Solaris 11.4 system you can check these using the soljsonvalidate utility. (Note that it's not currently possible to get sstored to dynamically re-read metadata definitions, so a svcadm restart sstore is required.

The last component that we have is the WebUI sheet, which I have packaged so that it is delivered to /usr/lib/webui/analytics/sheets/site.

This is accessible to you once you have logged in to your BUI instance and selected the Solaris Analytics app from the menu.

Prior to accessing that sheet, however, you need to configure the service. On my system, the attached RS232 port is /dev/term/0, and I have a PVOutput.org API Key and System ID:

# svccfg -s jfy
svc:/application/jfy> listprop config
config           application
config/debug     boolean     false
config/logpath   astring     /var/jfy/log/
config/usesstore boolean     true
svc:/application/jfy> listprop devterm0
devterm0                 inverter
devterm0/devname         astring     /dev/term/0
devterm0/pvoutput_apikey astring     elided
devterm0/pvoutput_sysid  count       elided

To create your system's configuration, simply add in the appropriate definitions below. I suggest naming your inverter property group is a way that you find useful; the constraint is that it be of the type inverter:

svc:/application/jfy> addpg devterm0 inverter
svc:/application/jfy> setprop devterm0/devname = astring: "/dev/term/0"
svc:/application/jfy> setprop devterm0/pvoutput_apikey = astring: "your api key goes here"
svc:/application/jfy> setprop devterm0/pvoutput_sysid = count: yourSysIDgoesHere
svc:/application/jfy> refresh
svc:/application/jfy> quit
# svcadm enable jfy

Since I'm running the service with debugging enabled, I can see copious details in the output from

$ tail -f `svcs -L jfy`

[ 2018 Apr  4 06:46:22 Executing start method ("/lib/svc/method/svc-jfy start"). ]
args: ['/usr/lib/jfy/jfymonitor.py', '-F', '/var/jfy/cfg', '-l', '/var/jfy/log', '-d']

response b'\xa5\xa5\x00\x000\xbf\x101522130110183   \xfa\xcb\n\r'
response b'\xa5\xa5\x02\x010\xbe\x01\x06\xfd\xbe\n\r'
Registration succeeded for device with serial number 1522130110183 on /dev/term/0
Inverter map:
id   1: application
id   2: 1522130110183
{u'devterm0': {u'devname': u'/dev/term/0', u'pvoutput_sysid': u'elided', u'pvoutput_apikey': u'elided'}}
[ 2018 Apr  4 06:46:44 Method "start" exited with status 0. ]

And there you have it - a brief example of how to use the Python bindings for the Solaris Analytics feature.

If you have questions or comments about this post, please send me a message on Freenode, where I'm jmcp. Alternatively, add a comment to the github repo.