An(other) Inflection Point

Spend enough time in technology and you see recurring patterns ebb and flow. The democratization of technology - areas of specialization evolving into commodities - has largely been a net positive. But if you get caught up in the swirl of it you sometimes miss the bigger picture. I did. In my nearly 10-year pivot to consulting in and around the analytics domain I hadn't realized my roles were essentially solving the wrong problem over and again. The pandemic gave me a chance to take a step back from that to assess in a way I hadn't up to this point, and so this post is a bit on the next step in my journey.

SPOILER ALERT: My response above is the 280 character version of this post - so if you're running short of time that's the tl;dr.

A Warehouse Full of Cannibals

I've written about my first software project outside of school building a warehouse management system for a small company that build physical products. We were building electronic musical instruments "from a cold start" as well as doing refurbishments of vintage electronic music gear. But we were staffing up and had grown beyond the ability for Bob (and myself, ostensibly) to go around with all of the parts and component information as a function of collective memory. The challenge was that with so many parts sources - including stripping operational parts out of otherwise-broken systems - meant tracking how much of something we could build was a major concern. Most of our work was standard off-the-shelf components but there was just enough of the boutique part tracing that it really required its own solution.

We not only had to track the parts we ordered from suppliers, but we also had to calibrate for raw materials and lead time required to fabricate some of our own parts built in-house. And all of this was before we had to start cannibalizing out of a smorgasbord of defunct instruments sitting on the back shelves of the warehouse. In many ways it was a logistics nightmare, but the silver lining is that it was built to eventually include some predictive capabilities. Little did I know that it would be the kind of problem I'd be solving over and again for the next few decades.

The Big Event & The Bigger Picture

Inventory management seems pretty mundane - and if you're lucky, it is. But almost no one is so lucky. There's the many-to-many relationships that emerge: multiple suppliers that can provide the same part, cost versus lead time considerations, it all factors in. And at that time there was no easy online search to find supplier part counts and delivery lead times. You had to get folks on the phone, which takes time and patience. And this system was as much about providing visibility into what's "out there" as what we had on hand. The good news is that once the information was "in the system" then a generalist - an office manager or other line worker - can do a lookup and have a meaningful conversation with nearly anyone in the supply chain. Up to that point, it was either Bob or myself that had to spend time "on the horn" and creating a fan-out for that group of tasks was a pretty sizable force multiplier.

It started with an accounting template, part of what Microsoft's Access plus Visual Basic offering. As I mentioned in my early years with Moog, his son worked at Microsoft and he sent us a beta of the product, and it shaped really well to our problem space. I expanded it in two directions - as mentioned we kept notes on sources for off-the-shelf parts, and I also built tables for various custom circuit boards and parts we assembled in the shop with the full bill of materials (BOM) that each would consume. The "fun" part was coding an optimizer that could give us a view into how many boards we could build before running out of certain key components. So instead of a surprise when looking into a nearly empty bin, we could run checks before setting up for an assembly run to see what we'd need to resupply ahead of time. And beyond the "0 checks" we also used it to show the "shape" that the bins we would be afterward, and provide a view into what we'd need to order to replenish stock before the next run. We had been burned enough times that the investment in a bespoke-ish system was certainly worth the effort. And once we had gotten the primitives in place, the portions that provided the higher level view was an easy add-on.

It struck me that the folks who build event based systems and the folks who are consumed by byzantine analytics re-platforming never talk to each other because they're both busy solving similar problems, albeit for different reasons.

My next job was as a developer for an accounting company that provided electronic tax filing software for CPAs. So that wasn't merely dancing around an accounting mindset - this was accounting with a twist. We also had to deal with a variety of filing regulations with each state, plus US federal tax codes. And there were specific guidelines on how to note a preparer's overrides and all of the amendments and indemnifications that came with it. As it turns out, this was an unexpected early lesson in the winding path my career has taken since then. Here's a 2016 keynote from Greg Young - the person who originally coined the term 'CQRS' and founder of EventStore DB, who sums up the event sourced landscape in this brief excerpt.

The 'Wrong' Kind of DDD

But the road to this realization was full of detours. While event sourcing and event-driven systems are often associated with domain driven design (DDD) the next decade or so of my career was subsumed with another DDD - database driven development. I spent some time with IBM Global Services and managing their partner e-commerce DBs around the globe. And that actually had some event sourcing functions as we were correcting and re-loading data in preparation for Y2K. But when I went to Siebel Systems we built out a lab with more than a dozen application and database based systems to simulate various workloads. That was when DBAs - database architects ruled the world. Because databases were often the greatest concentration of compute and storage (and therefore expense) it tended to also be the place where business logic would reside. And as anyone who's led a "digital transformation" project will tell you the industry is still living with that legacy imbalance in enterprise n-tier systems to this day.

You're Projecting

Fast forward to my first Microsoft Gold Partner assignment, a state level court system data warehousing project. They wanted to have a complete history for all court documents, but the old source system didn't record deletes. Well - to be specific they actually removed records instead of simply marking them with a 'deleted' indicator in the table. This was a vestige of the precious mentality of limited storage and compute available in databases from a bygone era. There had been plenty of opportunities to fix that omission by subtraction, but no one went to the trouble to do so in the intervening years. So they assumed that it could be magically fixed in the new data warehouse as though the information could be generated out of thin air. I eventually found a hybrid solution which could create a differential from monthly backups, loaded as snapshots for comparison. The re-built record delete indicators were only accurate to the end of the month in which they were deleted, but it was better than nothing. The process was tremendously time-consuming and very expensive, but as I've found over the years most analytics systems are the "later" choice of most "pay now or pay later" decision-making processes.

What I found since that time is that most "digital transformation" analytics projects are in-effect forensically reconstructing event sourcing from legacy operational systems, so that it could then be re-projected into an analytic work product. I make a passing reference to the inherent fallacy of "digital transformation" in my post about agile patterns and practices and this is another facet of the same argument. Here again Greg Young articulates the problem space succinctly.

I don't want this article to preach the gospel of all things event sourced. There are plenty of those around the web. But it struck me that the folks who build event based systems and the folks who are consumed by byzantine analytics re-platforming never talk to each other because they're both busy solving similar problems, albeit for different reasons. So while Greg's keynote presentation was certainly educational on its own merits - it's also a message in a bottle to folks drowning in technical debt created by others (as I was) when trying to extract analytic value out of legacy operational systems that are fundamentally ill-suited to their task.

Toward the end of my tenure as a Microsoft Gold Partner consultant I had lunch with an exec that was trying to convince me to stay on a project. My argument was that "digital transformation" was a double lie, because it wasn't transforming anything - and was also a lie of omission because we were failing a duty to inform that client of the costs of maintaining the burden of that technical debt. I referred to it as "affirmative corporate gaslighting", a form of positive toxicity. His answer? It's not so much a lie as a fact that needs ample opportunity to become true. It was said with a wry smile and I laughed at the joke, but it definitely was a signal that I was in the wrong part of the technology business.

A More Functional Approach

My path to embracing F# was not without its detours and diversions. It's not my first or even my second functional-friendly language. With my time at Siebel I picked up java and spent nearly a decade in that "family". While I had done quite a bit of java proper I also wrote quite a bit of production code with Groovy. Much of that time I felt like I was wrestling as much with the JVM as solving business problems. And between that and Spark/Scala later in the analytics phase of my consulting work I really grew frustrated with the java runtime and shifted focus to C#. Like most folks in analytics I dabbled a bit with Python, but found that the inherent functional programming (FP) style of R along with its unique ability to efficiently process large observation sets made it a clear choice above Python. So from that time forward I was primarily working in C# or R, and only stepped out to Python when it was already present in Jupyter notebooks or other mixed code environments.

When I took a pause during the pandemic to reconsider my options as a systems architect, I kept stumbling across facets of event sourcing that resonated with functional programming. I thought it might be a form of survivor bias, since I had used a handful of FP languages and really like working with R. But then this last footnote from Greg really brought the point home.

So that put a floor under what my personal experiments were hinting at - I wanted to find a general-purpose functional language that had avenues for real-time operational and analytic workloads. I had taken a brief look at F# a few years ago, but it didn't stick. The story wasn't really cohesive, but now with an impressive portfolio of ports to or interop with .NET and the growing meta-programming libraries that extend F#, I thought it was time to consider it the one language to rule them all, at least for my own designs.

To The Point

Unlike my prose, F# is succinct. So aside from being able to take advantage of "life in the .NET ecosystem" it also has the ability to pack a great deal of context into a relatively small block of code. I remember being resistant to that idea when going from java "proper" to Groovy, having grown acclimated to the former's more verbose syntax. And I hear similar comments from C# developers being perplexed with the terse structure of F# code. But with my exposure to Python and R, F# felt like the best of all worlds. You get the clarity of a composed structure using static typing with robust type inference and a compiler that really keeps you from going far astray. F# had a high quality REPL long before it was cool, and there is hot module reload in the SAFE Stack with a more general HMR implementation coming to .NET 6 later this year. From the perspective of supporting rapid development, it has the goods.

For an example of F#'s succinct syntax consider this sample from my Low Code is Dead. Long Live Low Code! blog post. The idea is to illustrate how an F# function extrapolates to C# and then the Intermediate Language that the .NET compiler uses to produce machine code. This isn't to say F# is better or worse than any other language, just that it has certain traits that make it advantageous for a direct form of expression.

Meta Programming

The thing that really turned my head was Fable. I found out about it after I had just rolled off of a project to build a streaming platform. In that project I had run into a brick wall with Blazor WebAssembly because of the inability to stream DRM protected content through the WASM sandbox. It was a road block that sent me scurrying back to Node and JavaScript, and while the polyglot life is the new normal is today's tech scene it was a point of frustration. Fable offers all of the type safety of TypeScript while rendering JavaScript that can run on any browser.

The real genius of it emerges with the SAFE Stack, where sharing allows a single code file to transpile to multiple targets. So you can create one definition that informs both the server interface and the client applications. I was pretty excited about the SAFE Stack because of my work in the R ecosystem - with R markdown, Shiny server and Blogdown's transpiling to HTML. So while I recognized the "meta" pattern I had seen in R, the SAFE Stack demonstrated clearly that F# could span domains in a way I hadn't seen before.

Machine Learning

And it's worthwhile to point out the growth in .NET machine learning projects, both to facilitate interop with other frameworks as well as purpose-built .NET workloads. The Apache Spark library is particularly notable that it works with either C# or F# and code can be deployed to Azure HDI, Amazon's Elastic MapReduce or Databricks. While .NET gets credit for "playing well with others" by being able to run on Mac and Linux, the transpile and interop story of .NET and particularly F# goes much further than that. While I discuss some F# specific tooling, these links provide a quick view into open source initiatives in the machine learning community.

An Overnight Success Twenty Years in the Making

I really enjoyed watching these interviews with Don Syme, for the context as well as candor and humor he brings to the conversation. But I also liked the structure of the interview, where background info was displayed to "season" the screen as Don responded to questions.

It shows that F# may seem like a newcomer to certain fields, but that has more to do with where it can be deployed and the underpinnings supported by the .NET ecosystem more than F# itself. With .NET 6 Long Term Support and new tooling like .NET interactive, F# will compare favorably with a wide variety of general purpose and domain specific languages, and with that I expect adoption will increase across the board.

Kinesthetic Learning

There's no substitute for learning while doing. But I'm also a reader. So I set up a large music stand in my home office with several books perched on it which I've started to refer as my "F# bookshelf". I'm currently working through Isaac Abraham's "Get Programming with F#" but I will thumb through others for reference and to occasionally provide a view into what's down the road. But the real work is sitting with the lessons and samples side-by-side in a working environment.

• Get Programming with F# by Issac Abraham

• Machine Learning Projects for .NET Developers by Matthais Brandewinder

• Expert F# 4.0 by Don Syme, et al

• Stylish F# by Kit Eason

• Domain Modeling Made Functional by Scott Wlaschin

Online Resources

Books often come with downloadable companion code - but there are also many free and paid online resources that facilitate the curious programmer's journey.

• F# From the Ground Up (Kit Eason [Udemy paid courseware - highly recommended])
• F# for Fun and Profit (Scott Wlaschin's site)
• Compositional IT News & Blog (company founded by Isaac Abraham)
• Learning F# (F# Foundation's compendium)
• Functional koans in F# (Chris Marinos on GitHub)

There are many standard-bearers for F#, and they offer long-standing and substantial bodies of work to help software engineers approach regardless of their technical background.

Akka.NET, Actor Model and the Reactive Manifesto

This is where I'll catch some flack from both mathematicians and computer scientists. But my personal opinion is that serverless apps - both stateless and durable functions - are really a rehash of the actor model, which has been around for decades. I had some early brushes with the concept while in school, but it really sunk in when I started looking for ways to make operational systems more elastic and reactive. It turns out that I'm not alone in this, and actor model in Akka.NET (the name being an approved homage to the original Akka implementation) brings event-driven systems shoulder-to-shoulder with event sourced architectures. To many software engineers event driven and event sourced are distinct from one another, so it's worthwhile to consider each on their own before you find where they might overlap in a given domain.

Here is a clip from a presentation by Aaron Stannard, co-creator of Akka.NET and CTO of Petabridge. They cover a few highlights of the Akka.NET actor system.

Actor systems are a form of orchestrated ephemeral compute that can very efficiently create a cohesive reactive application surface. There are many of these systems showing up across tech stacks well beyond public cloud serverless planes or Akka and Akka.NET. The reactive manifesto was authored in 2014 and serves as a common point of departure for many, and from my perspective is at least partly responsible for the renewed interest in the actor pattern. I count myself as both early and late to this party. Given the dilution that has undercut the agile manifesto, I'll be curious to see how this manifesto ages. While I think that in some ways it doesn't go far enough, I added my signature to show solidarity with its most high-minded goals.

The part of Akka.NET that really has my attention is that Petabridge has provided an entire site on learning how to implement Akka.NET step-by-step, and it's free. I'm definitely adding that site to my list of online learning resources. While it's written in and around C#, I'm looking forward to embracing and bring over Scott Wlaschin's "railway" pattern in F#. I had a sense I would need to maintain and even sharpen my C# chops if I was going dive into F# with increased momentum. From early on I had a sense that leaning on R would only get me so far down the road, but this new ground is exciting.

Beyond Reactive with Wilderness Labs & Xamarin

I lump mobile apps and IoT with automotive telematics because they're all compute at the edge receiving and producing data in exchange with a remote backplane. I also mention them here because the same folks who created Wilderness Labs, a .NET embedded IoT solution, also created the original Xamarin framework, a cross-platform stack targeting apps for Android, iOS and beyond. I'm working on a Meadow F7 custom sensor driver that's part of a stealth project. But Wilderness Labs just released an update that included an F# project template and direct .NET Standard 2.1 support. So expect more on this, as it will be a showcase on how F# can support a variety of applications, even microcontroller workloads.

Other Learning Projects

Aside from the exercises, kata, koan and other assorted bits and bytes, I'm also building a few of my own little projects to help sharpen my F# chops. My first foray was an Azure Function app that takes a URL and retrieves openGraph data from the targeted web page. There's also the Fauxlemetry CLI project, which is a small applet I use to generate large sets of production-scaled event data.

I'm also learning how to use Snowflaqe, a meta-programming project from Zaid Ajaj. It's a fascinating microcosm into one of F#'s real super-powers - that it can be used to generate new, canonical code - so meta! I used it to generate a client that scraped GitHub's public GraphQL schema which then formed the structured types for a client interface library. The Types.fs file generated more than 3500 lines of error-free code that I would never have to look at other than to satisfy my curiosity. It's like magic. My goal was relatively mundane by comparison - to collect repository data, chart the relative growth of functional-friendly languages relative to other programming styles over the past 10 years. It's still in early stages but I'm hoping to complete the project end-to-end in F#/.NET and resist falling back to R.

But that's just the start. Right now the challenge is to maintain the core lessons (and the hardware project above) while not getting too distracted by the "new shiny" idea that pops into my head. It's the kind of problem I love to have.

The Future is Functional

My vision of a path forward with F# is taking shape. Fable, Xamarin, Meadow, C#/.NET interop face out while Akka.NET, Kubernetes and Event Store provide the base for a reactive platform that's fast, flexible, resilient and maintainable. The clarity of this sits in stark contrast to those lessons in my career where poor early choices and tech fragmentation were a project's undoing.

And the objective should not only be improved user experience or reduced burn rate but also a more sustainable operational framework. We as an industry must look beyond cost considerations to sustainability and reduction of carbon footprint. Bringing these intelligent tools together gives me a renewed sense that it's not only possible to create a solution greater than the sum of its parts, but that its inherent efficiency is also a more responsible approach. And so, the work continues...

Coda: Microsoft, the Enterprise & The Open Source Balancing Act

I'm sure that Microsoft takes many figurative arrows to the chest and back while navigating between the demands of their enterprise customers and the broader open source community. That's what happens when you make a large target of yourself, such as acquiring GitHub. But Microsoft understood that it's not just ideas from Redmond that are worth supporting. Seeing what Aaron Stannard has done with Petabridge and Akka.NET, and likewise Greg Young's work with Event Store, and Isaac Abraham with Compositional IT and Farmer, as well as Bryan Costanich's leading edge work with Wilderness Labs each show that a well-managed open source business model yields innovation that the enterprise can rely on.

And with GitHub Sponsors there's more ways to support open-source while fitting within the budget constraints of a start-up or growing company. I'm sponsoring Zaid Ajaj for his incredible work with Snowflaqe, Feliz and myriad other projects he supports. And Maxime Mangel has a Patreon page for supporting his Fable Compiler. If it's worth making these tools part of your business, it's worth finding a business model to support them. And GitHub Sponsors should be applauded for providing a platform which expands the number of professional contributers to the world of software engineering. I'm hopeful that other companies that make their money by strip-mining open source projects - or - by building moats around rebranded scrapes of others' technologies will take note and see that the better way is to create a rising tide that lifts all ships.

And Microsoft isn't new to this. Don Syme and Microsoft Research deserve credit for making the long-term investments to bring cloistered concepts into the real world. And FSharp.org continues to create fertile ground for ideas to grow in the .NET ecosystem. Like so many that have read this far (bless you!) I've watched Microsoft find its way forward and have been frustrated at times. But on balance I find that the investments they're making, not just with navel-gazing concerns of high-performance compute, but also their sustainability initiatives have the makings of a more responsive and more responsible future. It can be difficult to see a silver lining as a global pandemic becomes endemic. But as a life-long technologist I can say with confidence that this landscape makes me more hopeful than ever.