Why I’m a #Technology #Evangelist: Technology #Empowers You

I’ve been working with computers and writing code since I was a little boy. My first computer was the Commodore 64, and now I carry around a device in my pocket that literally has 33 million times the amount of memory and is also millions of times faster. My first programs were written in Basic and generally not longer than a few hundred lines. In my prime as a development lead, I could crank out thousands of lines of Java or Perl or C++ every day.

My goal, however is really to write less. Writing more code takes more time, and the more time I spend on one thing, the less time I have to spend on something else. For me, that is what technology is actually about. Some people worry that technology is breaking down some of the core values that hold our society together. Some worry that technology will keep eliminating jobs and making it harder for people to make a good living. I certainly acknowledge that there are challenges that technology brings, but there is so much value as well. The biggest value is empowering people to do other things better and faster. Computer technology has only been around for 70 years or so, but the pace of change is accelerating.

When I arrived at college in 1993, I had never heard of the internet. I didn’t have an email address or a social media account. I arrived at MIT from Montana and dived into a world on the cutting edge of technology. A lot of the things that we take for granted today like email and instant messaging were being actively used while the rest of the world barely knew about them (we used a tool called Zephyr that allowed us to IM long before AOL instant messenger brought IM to the masses). The world wide web was in its infancy and there were only a couple hundred websites. Most people were still using older networks like Gopher (look that one up) and FTP for sharing things.

By the time I graduated 5 years later, the web had exploded. Companies and individuals were creating web content at a furious pace. You started seeing URLs on billboards and TV. You could get software from the web instead of the store. You looked up the weather online. Even getting your news was just a click away. Life had completely changed. Now my children are growing up in a world where they can literally talk to an artificial intelligence that was pure science fiction only a decade ago.

My dad was a lawyer, and when he started his job, it was a lot of work to find the information he needed to do his job. The laws were published in large heavy books that were updated every year, and they took up a lot of space on shelves (and this was really just for Montana State Law – there was another set of books for Federal Law). There was also a large number of books that captured past legal cases that were used as references for building new cases. Being a lawyer involves a lot of research as you write briefs and other documents that are filed as part of the legal process. There are legal assistants to help out, but back then it still involved a lot of pulling books off shelves and pouring through pages to find the needed information. Typing up the documents was done on a typewriter. It was a tedious manual process to format the text, and if you made mistakes, it was a manual process to first erase the mistake using liquid paper and retyping. My dad was one of the first people I knew as a kid to own a computer, simply because it made writing those briefs easier. Around the time I got to college, an electronic service called WestLaw was introduced that allowed you to lookup law information on your computer. Initially they sent you CDs that you would have to insert for the lookups, but now it is all online. The books quickly became office decoration, and suddenly my dad could do his research and write his documents in a fraction of the time. He didn’t end up working less (my dad loved the law and it was his passion), but he was empowered to do his job better and faster.

I did use books for research in the early days of my career, but now I do most of my research online, either through Google or Safari Online or online tutorials. It used to be that a task of writing some code involving something I was unfamiliar with meant that it would be days or weeks before I could complete it, but not I can search and quickly find answers. I don’t even really read tech books cover to cover any more, mostly because I learn as I go and I learn through much more interactive means.

In the last year or so, I’ve stopped writing code and started spending more time guiding development teams who are building software. My goal is to empower them to get the job done fast, cheap, and with good quality. There a lot of tools that we lean on to do this. We use Jenkins to continuously build, test, and quality scan our code. We use Git to enable our teams to focus on parallel work-streams and not worry so much about trying to merge it all together. We use Docker and containers to enable flexible environments that can scale up and down as needed. We use Ansible to build out virtual environments in a matter of minutes. We use the cloud to innovate, prototype, and design.

This is why I have become a technology evangelist. I see a new technology or webinar or eBook pop up on my Twitter feed or on Facebook or on Flipboard, and my first question is how can this technology enable me to do my job or to help my team do their jobs better. Containers, cloud and Machine Learning are really exciting things to work with, but their real value is how they enable your business to deliver better, faster, and with higher quality. I’m here find and champion those enablers.

 

Advertisements

The #Agile Journey

I would like to think that I have been doing Agile my whole life. Not scrum and stories and all that, just the spirit of Agile. Breaking big tasks down into little tiny chunks and designing each piece so that it fits into the big picture. When I started my career after college, I started in small companies with folks that were extremely talented and didn’t need a lot of management for the day to day tasks, so what meetings we did have were mostly about communicating what it was that needed to be done.

When I moved to larger companies, a lot of that changed. I no longer had a small team that was independent and could easily divide and conquer work. Instead, we had projects that were sophisticated but were themselves just small pieces in the larger picture, and the integrations between my projects and other projects really made it much harder to effectively work out the plan for delivering results.

Companies have relied on Waterfall for a long time to help manage this because sometimes you do need a decent amount of planning up front when you are trying to integrate these projects together to deliver enterprise solutions. Even with Agile, that doesn’t entirely go away. What does go away is the need to specify every last detail of every aspect of your application before even starting work. Instead, you identify some pieces of work that can be started while still figuring out the harder stuff.

This process is a journey, and it might not happen smoothly. You have to train folks to pick up new routines, change how they think about requirement definitions, and really let go of that desire to have it all spelled out ahead of time. It can be hard for folks that are used to building out requirements in template documents that have predefined sections that they fill in. Those documents help folks think about all the different things that will be required to build the application.

At the same time, however, those documents are a huge drain on time and often get put on a shelf as a reference while the project is underway and then forgotten afterwards. Agile stories can sometimes be like that as well, but you can write an Agile story in minutes. A business requirements document for a waterfall project can take days, weeks, or even months to put together. They also contain a lot of redundancies, unnecessary specifications, and empty sections that just are distractions from the actual content.

One thing that can help is to build templates for your Agile documents (features, stories, etc) that folks can fill in for every story. Teams will have to decide how strict they want to be about the contents (I sometimes have trouble formulating a story in the “As the user of the system I need XYZ” format), but having those templates can help folks transition from the business requirement documents to writing stories and features.

Another thing that can help is to understand the fact that part of being Agile is figuring out how you want the process to work. You will take training, read books, watch webinars, etc., but ultimately teams will have to try some different approaches and figure out what helps them be more effective. Depending on what kind of solutions you are trying to build, you might find different approaches work better. Kanban might make more sense than Scrum. You might find it better to have an actual board in your team room and folks move index cards around. Try different things and get feedback from the retrospectives on what is working and what isn’t.

Most of all, don’t spend time debating about whether Agile is more effective than Waterfall or not. Start small with a few teams if you aren’t sure it will work. It might not. Sometimes projects are so dependent on each other that you might need that business requirement document up front to coordinate a larger design. In general, however, Agile works with most projects, and sooner or later you will likely end up making the move.

Start the journey today. The sooner you get sprinting, the sooner you will be shipping results to customers and delivering value.

Why Agile Works

Software is written to perform some business function, and lately it has largely been about automating tasks that have been typically done manually by people. Teams of business analysts, managers, and engineers are brought together to design and build the software. In the early days of software engineering, software would get released in weeks, months, or years, but now companies will release software every single day, if not multiple times a day. This rapid development is helping companies deliver value to their customers faster, and that is generally a good thing.

Customers aren’t always getting value though. Facebook continues to upgrade and improve their user interface, but sometimes their users don’t want the changes that have been made. In fact, it can often alienate customers to have to continually adapt to changing interfaces. Some people love trying to do things in new ways and provide feedback on what they like and don’t like. Other people need time to learn how to use software, and when these changes roll out, they have to start over again.

As an engineer, I can tell you that we often feel like our jobs are never done. We may have delivered a solution that works, but we also feel like there are better, more efficient ways of doing things. We try to do things in different ways to differentiate ourselves from our competition. We try to add more and more features to keep customers excited about what we are offering. A lot of companies will push products that don’t even have a customer base yet, but if they push hard enough and market it as if the product is something that everyone needs, some customers will buy into that illusion.

All of this comes at a cost, however. A lot of projects are eagerly launched that eventually are delayed, shelved, or cancelled entirely. When you spend dollars on development efforts that don’t result in a product that goes out the door, you are wasting more than time – you are losing potential revenue because you could have been spending that time on other efforts that could have been generating revenue.

There are two really good books about this topic. The first is called The Goal, and is about a man who is given a short amount of time to make a plant become profitable or the plant will be closed. What he realizes is that it isn’t about cutting costs – which unfortunately is how a lot of companies address issues with their bottom line. Instead, the real problem is the complicated processes that prevent finished products from getting out the door, often leaving heavy inventories sitting around idle. As he streamlines the processes and gets the products moving out the door, the increased sales more than compensate for the increased costs, and the plant becomes one of the most profitable in the entire business.

The other book to read is called The Phoenix Project. It is similar to The Goal, but is about software development. The process of software development can often be like a factory that is trying to build things. When parts of the solution get bogged down in the process, it can have effects down the line. There are many things companies can do to improve these processes, but the most important one is to adopt DevOps strategies so that code is continually being built, tested, and made available for release as soon as possible. You can only realize the value of your work when it goes out the door, and you may have to make quick adjustments if you deliver something that doesn’t actually give your customers value.

The real challenge is correctly identifying what it is that will give your customers value. If you aren’t doing this correctly, you can have the most efficient delivery process in the world and still not be profitable. A lot of companies fail simply because they spend time building products that really don’t bring value to customers. Large companies like Apple, Microsoft, and Google can afford to experiment and take these risks, but a lot of companies really can’t afford to do so.

Getting feedback from customers early can be vital to successful projects, and this is largely why Agile has been so successful. Companies are able to produce small pieces of functionality and make quick adjustments if they determine that they are going in the wrong direction. They can cancel projects sooner if they realize that the product they were developing would not actually succeed with customers. Users get to see small incremental changes and give feedback rather than being blindsided with large changes that may make them go to other solutions. Companies can also quickly change direction if other needs are prioritized (and this happens a lot), so it makes it much easier to address issues in a timely fashion.

You don’t have to do Agile the way other companies do it, and in fact there are many different types of processes that are often labeled as Agile but are actually different. You may have to try a few different ways to find the right one for your teams. The important thing is that you keep product flowing out the door rapidly so that your customers can quickly tell you if you are going in the right direction or not. If you don’t know where you are going, how do you ever expect to get there?

Modern App Development

When I first started writing web apps, it was with mod_perl. A lot of folks still use mod_perl, but there world has clearly moved on to Python, ruby, node, etc. Now you don’t have to start every project from scratch, and often you are really just focused on building business logic. Often you don’t even have to worry about data modeling because frameworks are becoming so smart that they can build the physical models from your logical ones. This is enabling some developers to launch really powerful applications in days instead of weeks or months. This also makes it a whole lot more fun to just code.

Building really solid platforms, however, is not going away. It is one thing to build a cheap fast prototype, it is another to build a robust system that can handle millions of transactions a day. This is where you generally start breaking up your app into services that are separate but connected. This is also powerful because now you can start leveraging technologies that better suit each part of the process. Things like Python are great for web UIs, but for transactional processes that require complex business logic, you are better off using something stronger like Java.

You also need to make sure you have testing in place. A lot of frameworks now making testing a natural part of the build process. You don’t have to set up a build pipeline, it happens automatically. You just put the tests in a certain spot or mark them so that the compilers know they are there.

Don’t ignore code quality. You remember everything now but in two years will you still remember why some logic exists? How will new developers adopt what you have done? I have always been a little hesitant to use frameworks that work by conventions or custom syntax. Make sure that you comment these things so that new developers immediately know what each file is for. Use tools like sonar to make sure your quality is high and to help you identify potential problems.

Rapid development is great, but remember that you will be supporting this code for a while, and you would rather spend your time writing new code than maintaining old code.

Why Big Projects Fail

Recently some coworkers of mine mentioned that they were reading The Mythical Man Month in a reading group. It really should be required reading for anyone who wants to lead a project of any size. It’s a relatively short read, but the lesson in it is extremely important. It was one of the first books I ever read when I was at MIT, and was actually part of one of the first classes on software development I took. I still take the lessons from that book very seriously, and it is one of the reasons that I believe working Agile will in the long run be better for all of us. We are all aware that there is a point when it doesn’t matter how many people you throw onto a project or even how well you document something that is really complex, it just hits a point of diminishing returns or even ends up making the process take longer than it would with less people.

One of the classes that I took at MIT was called Computer Systems Engineering. Oddly enough, it was a class that involved a lot of reading and writing. We would read about all sorts of technologies like Kerberos, Operating Systems, Threading, etc., and then we would be tasked with writing essays on these various topics. One of the lectures was all about big engineering projects that failed. They started out small and talked about projects that companies had put in place to upgrade or replace dated technologies. As the lecture went on, the amount of money and resources that went into each project continued to increase. Near the end of the lecture, they talked about how British Airways started a project to replace their entire reservation system. The company worked on the project for nearly two years, spent almost a billion dollars (yes, with a b) and in the end, had to scrap the entire project.

The moral of the lecture was pretty simple: complexity kills projects. Imagine you were trying to build the Space Shuttle from scratch. That is the kind of thing where you have to design every detail very carefully and rigorously test it over and over because if you are wrong, people die. If you have ever seen the move The Right Stuff, you know that America’s Space Program was only successful after years and years of failed tests. Once they even pushed the button to launch a rocket and the only thing that happened was the top part of the capsule that holds the parachute popped off. After years and years of failures, they finally had systems that worked, and eventually they worked so well that the number of casualties has been extremely small considering how dangerous Space really is. The complexity of it, however, made it take years and years. As engineers, we don’t have years and years. We can’t constantly build things, test them, and watch them blow up.

Instead, we need to break those really complex things down into smaller pieces. We can move forward if we define the problems as smaller tangible problems that can be easily defined, easily implemented, easily tested, and easily delivered. The goal isn’t to deliver the whole system at once, but rather to build a piece at a time (or many pieces in parallel – think of how the ISS space station was built, with many countries building small pieces that fit together to make the whole). This is why the Microservices approach is really taking the world by storm, because quite simply, it is easier to deliver a thousand small things that are independently testable than to try to deliver one big thing that requires thousands of tests.

One problem that I often witness is that companies often struggle just defining what it is that they want to build. Someone defines a grand vision of the direction the company wants to take. “We want to better connect with our customers and deliver what they want, not what we think they want.” The vision is sound, but how does it translate into actual business requirements? Unfortunately, a lot of us are stuck with some of the architectures we have, and so we define it in terms of how to get where we are to where we want to be. Therefore, we start drawing a big architecture that we dream is the future. As we begin working on it, however, we realize that the requirements for the big architecture are complex and not well defined, and the project stalls while we work out what those actual requirements are and what approach will be the best to meet those requirements.

To put it simply, the problem you are trying to solve is actually many problems. It is just as if you were trying to figure out how to clean your whole house at once. There are so many different things that need to be done that you could get lost in trying to figure out what you should do first or how to best accomplish the task. So how do you tackle this big problem? You break it down into smaller tasks. Clean the kitchen. Clean the bathrooms. Clean the bedrooms.

Each of these tasks have their own subtasks. Clean the fridge. Clean the toilet. Pick up clothes on the floor. Now you have simple, tangible tasks where the requirement is clear and the task can be completed in a short measurable period of time. Not only that, you can pick and choose which tasks as you go – you might get tired of cleaning bathrooms and decide to work on the kitchen instead. You might realize you have friends coming over later, so you switch to cleaning the living room first. You can change direction at almost any time and come back to other tasks later.

This is what Agile really is about. If you don’t break your project down into small tangible pieces that your engineering team can clearly understand, how can you really expect them to deliver what you actually want? How can you expect that you even know what you really want? When are you actually ready to start? It may give some people comfort believing that everything has been planned or designed ahead of time, but how many times have you ever seen a project that went exactly according to plan? Never. Not once. Requirements change. People leave. Budgets get cut. Timelines change. If you keep trying to deliver software in a way that can’t adapt to change, you will fail.

 

Describing Machine Learning With Algebra

I volunteered for a brown bag session recently, and of course I chose Machine Learning to be my topic. Machine Learning, however, is not an easy topic. Sure, it is just machines learning, right?

Yes, but there is more to it than that. You understand that it is about learning, but how exactly do you make a machine learn? I talked previously about neural networks and that gave some insight into it, but I didn’t really go into what is actually happening during the steps of learning. Some of it is complicated math, and I might go into that at some point, but I think there is some simple math that can help clear this up a little. To do this, we need to go back to algebra.

algebra

As we all recall, algebra is all about solving for xMany of us took our first look at an equation and said, “what the hell is this?”. As an aside, if you are looking to teach algebra to your little ones, you should definitely check out the DragonBox Algebra game. It does an incredible job of taking a hard concept like algebra and turning it into a fun game that helps kids do algebra without even realizing they are doing math!

Let’s take a step back though, and imagine that you are trying to connect two points on a plane:

points

The idea is that there is some relationship between these data points that you want to capture so that you can predict other possible data points that are similar. In the case of two points, the best fit is a line:

line

Mathematically, there is an equation that defines this line that fits these points:

CodeCogsEqn

Now imagine we have three points:

threepoints

It is pretty clear that our line isn’t going to properly fit these points. But a second-order equation (a parabola) like this might:

CodeCogsEqn (1)

This might look something like this (forgive my rough sketch):

parabola

More points might require even higher-order equations. So the first step of the problem is to identify the kind of equation that can represent the solution. The second problem is to solve for the values of A, B and C in our above equation that give rise to the parabola that we see above. For supervised machine learning, where we are essentially trying to fit some data in order to make accurate predictions, this is essentially trying to solve these two problems – find a model that has the ability to fit the data, and then doing the work to find the parameters that actually allow it to do so.

The details on how we find the values of A, B, and C are a little more complicated than what I want to go into today, but it essentially comes down to making some initial guesses at the values, calculating how far off our guess was, and then slowly adjusting their values in a way that brings the error level down. The most common method of doing this is called Gradient Descent, and I described that in another article, but there are plenty of materials online that describe the method. The short answer is that there is a way to calculate the direction to change each of the variables you are trying to calculate, and you do it in small steps so you don’t overshoot the optimal values. That is what machine learning training is all about.

I wouldn’t call it rocket science, but it definitely involves some math. Fortunately there are frameworks out there that are doing most of that math for you so that you can just focus on picking the right model and feeding it the data. Definitely take a look at my article on TensorFlow and Machine Learning Basics for an example of how to do this.

 

All Around the World

Just a quick note that I’ve been watching who lands on my blog, and although the majority is from the United States, I’ve also had visitors from Columbia, Chile, France, England, Germany, Switzerland, and Italy, in no particular order. It is kind of awesome to see folks from all around the world stopping by, even if just for a moment. It gives me hope for this world knowing that there are people all over the place who are curious to learn. Welcome visitors, wherever you come from. Feel free to leave a quick comment telling me where you come from and what brought you here.