I had a weird conundrum come up. It comes at a weird time at work, where projects are shifting around and features are growing, but as I was battle-testing my code, I had a realization my code was simply not good enough. For that, I wanted to add a classical Monad from Haskell called Maybe.
The Maybe monad is implemented as follows:
data Maybe a = Just a | Nothing
That's it. It's two different states. One has a value, while the other has nothing. In Rust, this is Option<T> with done with the following:
pub enum Option<T> {
None,
Some(T),
}
Both have similar concepts, Haskell enforces a bit more mathematical-ness in how they use monads, whereas Rust wants to add safety and nullable data without all that monad nonsense. This doesn't exist in Python, and I frankly feel like it probably should. I'll explain my thoughts.
The Existence of None
The project I'm on involves parsing data from a source, but the data from JSON can sometimes end up being None. A usage of None would imply that there's nothing there, but something being None can still be a valid state (at least in our application; I'm not fond of it). While parsing objects, I sometimes run into a None value, and it's hard to wrap a program around None as if it were a "good" value.
>>> {}.get("key") # None
>>> {'key': {}}.get("key", {}).get("subkey") # None
This is the ordinary using dictionary's get method. You can add additional default returns when a key is not found. Otherwise, you will get None which implies a failure. Or, it might mean a success?
>>> {'key': None}.get("key") # None; a failure?
Is it dumb? A little bit. I wish my data source did not include a key with None, and I wouldn't have this issue at all, but I'm realizing the problem isn't that the data source uses None, it's more like... None alone doesn't go far enough to include reasoning.
While scanning deeply-nested dictionary objects, I need to evaluate key paths and see what they end up at, whether it's some nested value, or realize the possibility that the key path simply does not exist at all.
def get_nested_attr(full_key, obj, default=None):
keys = full_key.split('.')
curr_obj = obj
for key in keys:
if key in curr_obj:
curr_obj = curr_obj.get(key)
else:
return default
return curr_obj
This is an example function I use to scan deep-nested objects. It works for dictionaries, but it can be extended to work on plain Python objects as well with getattr() and hasattr(), should you want to go that route. It works by taking a string that looks like "key1.key2", break it apart, and iterate an object until we reach some final value. If we don't find a value, it returns None, implying that the key does not exist. It has a keyword parameter to mutate what the default value should be in case no existence is found, so you can change it how you want.
>>> get_nested_attr("key1.key2", {'key1':{'key2':500}})
500
>>> get_nested_attr("key1.key2", {}) # None
Logically sound, right? There's nothing really wrong with this, at least until you realize your source data happens to use None as a valid data type.
>>> get_nested_attr("key1.key2", {'key1':{'key2':None}})
# None... Is this right though?
After doing all that deep iteration, we end up with None, but it's not clear if it implies a success or failure. And frankly I don't want to start throwing exceptions, because it wouldn't even make sense to do so.
The obvious fix would be to then add a way to provide clarity on whether or not the thing you found was actually what you wanted.
def get_nested_attr(full_key, obj, default=None):
keys = full_key.split('.')
curr_obj = obj
for key in keys:
if key in curr_obj:
curr_obj = curr_obj.get(key)
else:
return (False, default)
return (True, curr_obj)
This just ends up adding further burden on the end user (me) to solidify code even more with blandness. Say we're checking for two keys in two different JSON blobs, we want to check for two key paths to be equal or not.
data1, data2, = {...}, {...}
found1, value1 = get_nested_attr("key1.key2", data1)
found2, value2 = get_nested_attr("key1.key2", data2)
if found1 and found2:
if value1 == value2:
print("Both are equal")
else:
print("They are not equal")
else:
print("Data was not found in both")
I am not a very big fan of this pattern, simply because user burden is now placed on the end user (me, again) to remember that get_nested_attr() returns a tuple of values and not a single thing. It doesn't feel very smart, and honestly that if is really boring.
My head got started down the ol' Haskell brain train yet again and I wondered if encapsulation of data was needed here. Granted, encapsulating data with other data is pretty atypical, and we just did that with Python here, by returning a pair of values, one describing the other's situation. However, the code surrounding the unpacking of the pair is boring, tedious, and would just end up in a ton of boilerplate that I would have to deal with.
The Maybe Class
My frustration got the best of me. I put together a Maybe class, here's how silly it is.
class Maybe(object):
__slots__ = ['value', 'has_value']
def __init__(self, value):
self.value = value
self.has_value = False
def is_some(self):
return self.has_value
def is_nothing(self):
return not self.has_value
A class that takes in a value and wraps it into a new object with two methods to dictate if it is something, or if it's just nothing. Is it a weird class? Sure. Monads are not exactly first-class features Python has, and forcing this into a code base would no doubt get some resistance.
The thing to note is that has_value isn't even remotely mutated here. I left that out, because it's not the job of the object itself to mutate that value. I opted to leave that outside, because that's not everything. Let's introduce two helper functions now.
def Some(x):
m = Maybe(x)
m.has_value = True
return m
def Nothing():
return Maybe(None)
Meet Some(x) and Nothing(); remember these names? It's like I mashed Haskell and Rust together and got these names. I couldn't use None() because that's a reserved keyword in Python, for obvious reasons. If this were Ruby (science project!), we wouldn't need the parentheses for the Nothing() function. Now instead of returning a value from a dictionary search or a pair of values, we can use this instead.
(These are not class methods or static methods because I simply do not want to type Maybe.Some(x) or Maybe.Nothing() each time I invoke it. Too much extra typing.)
def get_nested_attr(full_key, obj):
keys = full_key.split('.')
curr_obj = obj
for key in keys:
if key in curr_obj:
curr_obj = curr_obj.get(key)
else:
return Nothing()
return Some(curr_obj)
When we find something, we can wrap it into a Maybe object that has it's values filled out correctly, whereas if we find nothing, we get a Maybe object with it's fields pretty much left as blank as possible. How does that change our previous code using this function?
data1, data2, = {...}, {...}
lhs = get_nested_attr("key1.key2", data1)
rhs = get_nested_attr("key1.key2", data2)
if lhs.is_some() and rhs.is_some():
if lhs.value == rhs.value:
print("Both are equal")
else:
print("They are not equal")
else:
print("Data was not found in both")
Admittedly, not much has changed, but burden has been slightly lifted off our shoulders for having to unpack a pair of values from a return tuple. Rust and Haskell can leverage the Maybe type to better effect with their core functionality, while our syntax does not necessarily allow us to do that nearly as much.
Helper Methods
Since it's Python, we can add some overrides to some methods that might prevent us from having to do so much boilerplate. First one is to add an override for __bool__() to reduce the weight-lifting of if statements comparing Maybe types.
class Maybe(object):
# ...
def __bool__(self):
return self.has_value
The is_some() method and a __bool__() function were effectively the same, so why not add this in? Next is the __eq__() method to compare two Maybe types together and inspect the has_value attribute.
class Maybe(object):
# ...
def __eq__(self, o):
if isinstance(o, Maybe):
return (self.has_value == o.has_value) and (self.value == o.value)
return self.has_value and (self.value == o)
# Testing it
>>> Some(5) == Some(5)
True
>>> Some(5) == Nothing()
False
>>> Nothing() == Nothing()
True
data1, data2, = {...}, {...}
lhs = get_nested_attr("key1.key2", data1)
rhs = get_nested_attr("key1.key2", data2)
if lhs and rhs:
if lhs == rhs:
print("Both are equal")
else:
print("They are not equal")
else:
print("Data was not found in both")
The genius of short-ciruiting here means that if I'm comparing a Some(x) and a Nothing(), then it will not even proceed to the second argument of the and operation and will bail out. So if you happened to have something very complicated and requires time to compare, it won't execute if it's compared to a Nothing() value. Well, obviously, I guess. It also works on non-Maybe types by checking if the input is a Maybe instance since Python won't exactly do type enforcement for us.
Unpacking the values from a Some(x) value are trivial by simply doing Some(x).value, but this is technically a violation of the safety that Maybe is giving you. Granted it's Python, you can do whatever the hell you want, but it'd be nice if we were able to glue Maybe objects together somehow in a nice and friendly fashion. We would need a syntax like if let Some(x) like Rust, but how can we get close to emulating that?
The sad reality is we can't use something like with Some(x) as x to unpack the data, as that is not how with was designed in mind. Should you want to break out of with, you'd have to use an Exception, which I am not doing. However, nothing's stopping you from using... an Iterator!
class Maybe(object):
# ...
def __iter__(self):
if self.has_value:
if hasattr(self.value, '__iter__'):
return self.value
return iter([self.value])
return iter([])
Now you can sequence Maybe code together using for as the means to group it all up.
data1, data2, = {...}, {...}
for lhs in get_nested_attr("key1.key2", data1):
for rhs in get_nested_attr("key1.key2", data2):
# lhs and rhs are now both raw types here
if lhs == rhs:
print("Both are equal")
The difference here is that any code under the first for loop will not be run unless we have a valid Some(x) type being used as if it were the iterator. Meaning any further unpacking is never executed unless the first condition is a success. The unpacked variable is bound for us to use throughout the expression without us having to check if the context is within a Some(x) first either. This is pretty cool.
# unpacking two Some() values
for x in Some(5):
for y in Some(6):
print(x + y) # => prints 11
# unpacking with a Nothing()
for x in Some(5):
for y in Nothing():
print(x + y) # => never executes!
The tradeoff is that for does not have a meaningful else construct that we could use in the bad case of if the iterator is a Nothing() type. For two-way branching in this case, you're probably better off using an if still, but there could be circumstances where this unpacking and executing abstraction is handy to have. I'm pretty fond of it, even if it abuses normal Python constructs and everyone might call me a sinner.
(There is an else construct for for blocks, but it's terrible and acts more like a goto block; you would have to use break to leave a loop to avoid executing the else statement, which is silly.)
Closing Thoughts
The more I work on my current project, the more I'm frustrated by how lackluster Python is in the world of abstract data types. I might need even more encapsulations like with a State monad just to do some crucial work that involves finer granularity and the ability to pass through certain data to record how a process went. Or hell, I wish I had an Either monad to do railway-oriented programming. Perhaps this is a starting step with the Maybe monad, as the others could be slightly derived from that alone.
I would think I'm not alone in thinking that Python needs to take more inspiration from other languages to help it grow up. I'm probably not the first one to try and implement a baby-sized monad like Maybe and see how it would interact with Python's syntax. If lambdas weren't so shunned by the Python community, I'd use them more here.
The scope which Maybe currently sits in my project is small; I don't know how much it will grow, at least it probably won't grow much out beyond my control, but I think I'm okay with that for right now. I'll eventually flesh out some kind of tooling to help me work towards collecting state and logging data. I do believe railway programming is a good design, even if Python does not immediately agree with that sentiment. Either way, I don't write about Python enough, but I thought this was worth a write.
Cheers and thanks for reading.