Ten years in software

Last updated

I have been working professionally as a Software Engineer for the past 10 years. In that time, I've learned a huge amount, gained a bit of confidence, and largely ignored the social nature of our field. I haven't given back to the community and now feel like it's a good time to change that. I've been very lucky in my career thus far and want to share the broad lessons that I've learned along the way.

This is part one of a series of pieces written reflecting on my career:

tl;dr

To solve practically any problem, use the scientific method: first gather data in order to develop a hypothesis, and then create a test to prove or disprove it. It's surprising how well this works.

A first taste of real work

My first job as a programmer was when I was a student at UCI working at a small voice-over-IP company my friend was founding. None of us really knew what we were doing, and as a result I ended up sort of aimlessly tinkering with Asterisk and building a configuration language to handle our routing rules. The job earned me a paycheck, but not much else. Due to the lack of direction and not being personally invested, I ended up parting ways with the business in order to focus on school. I slowly lost contact with the founders, who stuck it out and eventually ended up forming the company FlowRoute.

In some ways I felt like I missed out on an opportunity to be part of a "real startup" from the get go, but I just didn't feel personally invested. I took away the feeling of wanting to build something I was passionate about as well as able to work with people who I could really learn from.

As I approached the end of my degree, I found many of my peers bitterly working at jobs they were thankful for only due to their paycheck. I didn't want to share that fate.

In my senior year of college, I took a course where a small group of students would work with a local business to build software for them: identify the requirements, scope and plan the work, and regularly demonstrate progress made on the project to the clients. This class paired me with Sendio, a small company which made an anti-spam email appliance (sold as a physical 1U or 2U rackable host) that would plug into your network and act as your SMTP gateway/proxy/"smarthost". It used a challenge-response system to defend against SPAM. It worked very, very well. I ended up working with a few classmates to build a terminal configuration UI to allow non-technical users to plug a monitor and keyboard into the host to configure the server's network and perform other maintenance tasks. The project itself was well defined, well managed, and my future boss always made himself available to help. After enjoying working with the project and the people, I accepted their offer and started working once the class finished.

Drinking from the fire hose

Looking back, I was incredibly lucky to have this as my first job out of college. For the first year or so, the engineering staff was two people: myself and my wonderful boss, Cameron Brown. He took the time to show me the ropes by educating me on how the entire system worked, showing me how to approached diagnosing problems, and giving me challenging tasks that forced me to deeply understand the system and its protocols. While my other college friends were working on small features at large banks or building mind-numbing network driver installation UIs, I was lucky enough to dive headfirst into the process of comprehending, building, maintaining, and growing an entire system.

During my time there, I was exposed to and ultimately felt responsible for the development and uptime of a system which ran on hundreds of hosts in extremely varied network environments. There was a lot to do. I needed to learn about the internals of RHEL flavored Linux; the architecture of qmail; how the various protocols which power the Internet (IP, TCP, DNS, SMTP, HTTP) and how the host OS can be configured and misconfigured; how to configure SELinux correctly; how to build RPMs; how to perform remote software and security updates to minimize losing hosts in the process; how PostgreSQL replication works and how to use it in a master/slave configuration; how cryptography and public data published in DNS is used to make mail delivery more reliable and trustworthy via DKIM and SPF; how to recover from data loss/corruption in our software stacks, etc etc. Exposure to these tasks was both exciting and daunting. There was a never-ending supply of things to do, which was incredibly useful as my first job out of college. Today, I only vaguely remember the (now likely out-of-date) details of each of these systems and tasks, but thankfully I still hold on to one invaluable skill: how to read and verify documentation.

The best (and most challenging) part of working at a small company is that it forces you to find things out for yourself. If you have a problem and the only other engineer you work with doesn't know the solution, you're on your own to figure it out. This is the job. If you're lucky, there is documentation that describes how a tool is used, what all of its internal components do, and lays out the pre- and post- conditions of its interface. If you're unlucky, you've got to rely on outdated blog posts with misinformation or worse, rely on your own observations to develop hypothesis for how the tools should operate. And if you're really unlucky, you've got only the source code or binaries. At the end of the day, regardless of how much (or how little) documentation you have, it's critical to verify the documented behavior and prove that the hypotheses you develop are correct.

The scientific method works

Sometimes documentation is wrong, sometimes "experts" (strangers on the Internet) share misinformation, and often generalizations made from your own observations are incorrect. The only thing to do is to just do the fact checking work yourself. And the process to do this is simple and proven: use the scientific method.

  1. Gather data: read documentation, ask questions, and make observations
  2. Make a hypothesis: when X is true and Y happens, Z should become true
  3. Verify with a test: prove or disprove the hypothesis by running a manual or automated test

No matter how big and complex software is, its behavior can be observed, its dependencies researched, its unknowns broken down, and every assumption made can be verified. If you keep at it, you can wrap your head around anything.