tl;dr - it's unclear that very high test coverage is better than "ok" test coverage.
Interesting perspective on the people side of the software engineering business:
Like most developers, I always get excited about new, up and coming open source technologies. In the spirit of Thoughtworks Radar, albeit much more casually, I'd like to list some technologies that I'm think are quite interesting and worth monitoring.
Reputable technologies - Significant use in production and established communities
- Node.js + Express
Rising star - Worth experimenting / prototyping with these technologies
- Relay + GraphQL
- Flow Type (by Facebook)
- Deis - Heroku-like open-source PaaS
- Angular 2
- React Native
- AWS Lambda
- Babel.js - transcompiles future JS into ES5
- ES7 - async, await
- Redux - new, but popular flux-like architecture and tiny footprint
Moonshots - These are new, unproven technologies but have a potentially large benefit.
- Black screen - terminal emulator built in Electron
- Otto - created by the makers of Vagrant, provides a simpler workflow through more "magic"
I've heard a bit about Deis for a while but I never quite understood how it compared to other open source technologies like Kubernetes. This Stack Overflow answer was quite helpful: http://stackoverflow.com/questions/27242980/whats-the-difference-between-kubernetes-flynn-deis
Especially insightful was the comment by the founder of Deis, with a link to this technology stack diagram:
- People overestimate the impact of key individuals on an outcome (e.g. entrepreneurs, CEOs, politicians) by failing to account for other factors such as organizational culture, industry trends, economic climate, and so on.
- External factors (to an individual) can be considered luck - and they are very important.
- Linear regression when used properly is a simple but powerful tool for isolating the effect of something - however it only measures correlation not causality.
- Causation is much harder to prove than correlation. Experiments (e.g. a/b testing) are ideal but not always practical, particularly for macro decisions (e.g. new product launch, M&A)
- Complex outcomes (e.g. 5-year company growth, civil war in the Middle East) oftentimes have multiple causes and defy simple explanations.
- Plans are too optimistic because people do not account for potential unknown unknowns.
- Improve estimates by using reference points from similar cases. Modify your estimate based on these references by acknowledging differences from your current case with previous cases.
- Consider reverse causality. Does good company culture lead to good financial performance? Or does good financial performance help foster a good company culture?
- Consider confounding factors. Is there a third variable that affects both the independent and dependent variables you are looking at?
- Look for probabilistic distributions. Things are rarely surefire and once in a while failure is inevitable.
- Is there a worst case scenario that you haven't thought of yet? How undesirable is it? Is it so bad (e.g. the Great Recession) that once is one too many times?
- Does the organization as a whole not take enough risks due to incentives for individuals to avoid taking risks?
- Check for survivor bias. Are there instances that should be considered but are not because they are not well-known / out of business / etc?
- Is the halo effect causing you to subconsciously use the company's good financial performance to attribute positive beliefs on other aspects of the company (e.g. strategy, culture, people, etc)?
- Am I considering a point in time (e.g. cross-sectional) vs over time (e.g. longitudinal)? Studies over long periods of time are ideal. Asking individuals to recall events after the fact is likely to be affected with hindsight bias (i.e. risky decisions are seen as foolish / prescient given how things unfolded).
- People overestimate their ability to assess other's skills. Quantitatively assessing interview scores with post-hiring job reviews allows you to improve the hiring process in a systematic way.
- When analyzing the root cause of a failure, consider whether it was the failing of an individual (e.g. that person was irresponsible and should be fired) or the failing of a system (e.g. that mistake could have happened to any of us).
- Self-evaluate what lens / filters affects your perspective.
- Make decisions based on expected value when possible - it will slowly add up
- Use multiple disciplines when possible - think quantitatively and qualitatively.
Thoughts from reading the books Thinking Fast and Slow by Daniel Kahneman and The Halo Effect by Phil Rosenzweig.
- Humans are prone to cognitive biases which can lead to suboptimal decision making. Many of these biases are driven by our desire to make sense of what is happening around us and to create stories that logically explain major events.
Halo effect is the tendency to attribute traits (independent variables) based on outcome (dependent variable). For example, if a company has several years of growth, its CEO will be praised for being pioneering and aggressively entering an adjacent market. In the counterfactual situation where the company struggled, the same CEO might be criticized for taking wantonly taking risks and neglecting the core business.
- People, particularly experts and optimists, greatly overestimate their impact or understanding. Leaders of organizations, such as the President, CEOs, etc., are seen as responsible for the overall outcome of their organizations. Studies have shown that a better CEO is only likely to outperform his or her counterparts 6 out of 10 times (i.e. a 10% improvement from flipping the coin). CFOs of major companies were asked to predict the movement of the S&P 500 markets. Not only were they significantly off, they were overtly confident about their predictions.
- People go for simplistic explanations because it's easier to comprehend and follow. Complex explanations such as "growing a business requires a careful analysis of the competitive landscape, assessment of potential disruptive innovations, and managing risks through a balanced portfolio" is harder to follow than a simple explanation such as "building a great company is about focusing on customers and empowering employees". The reason why the first is more complex is not that there's more steps - it's that it focuses not on concrete actions (e.g. promote employees within your organization to groom leaders), but on a mindset (e.g. assess various scenarios in how 3D printing can affect this industry).
Survivor bias can lead us to idolizing risk-taking companies because we are not as aware about that failures.Rather than taking into account the risk inherent in big movies such as acquisitions or entering a new market, people focus on the successful stories. Michael Raynor wrote a book The Strategy Paradox: Why Committing to Success Leads to Failure that covers this issue.
- System 1 is the lizard brain - it feeds off emotions and generates our gut reactions. System 2 is the higher order thinking brain. It can carefully consider tradeoffs, but it takes significant energy to use it, so we conserve energy by not using it whenever possible.
- The framing of a choice makes a big impact in how people respond -- even professionals (such as doctors about medical decisions). People consistently are risk-averse when given a choice of a guaranteed win (100% to win $500) over a very likely win (80% chance to win $650). People are risk-taking when given two bad choices and have a chance to entirely avoid the bad outcome (although with a smaller expected value).The prospect theory helps explain these decisions by focusing on the gains and losses vs. the final state, which is what expected utility theory focused on. Prospect theory also emphasizes the importance of the reference point, which is typically the previous states.
- The two selves are the remembering self which is how you think about your memories and what you use to make future decisions and the experiencing self which is how you feel at a given moment.
Excellent talk by Rich Hickey. Even if you never use Clojure, it's an enlightening talk about the benefits of immutable data that's applicable to everyday programming:
This post is a brainstorm of ideas on how to focus on the core technical problems and address solving supporting technical issues as efficiently as possible. Of course this makes some assumption of what is your "core problem", but I will assume this is a "typical consumer web / mobile app". If you're a devtool company such as Docker, for example, then your core domain and supporting domains may be completely different.
- Follow the mantra "borrow, buy, or build" (I may have re-ordered it)
- Ideally, there is a well-supported open source project that solves the problem that you can use. I think examples of this are Docker, Angular, Meteor, and so on. These are all very different technologies that but they all have in common that: 1) large, active communities around them (which means you will have support when you run into issues), 2) many examples of them used significantly in production, and 3) strong core teams behind the projects (in this case, each of them have a corporation backing them)
- Secondarily, if there's a commercial product or service that addresses your need, then strongly consider using it. This is a case where you may end up using Docker the open source project first, and then realize it's very useful to use something like Docker's registry for private repos.
- Lastly, there is building your own solution. Sometimes, this means using an open source project like Docker registry but hosting it on your own physical servers or even putting it on AWS. This last one is typically about either 1) saving money (since you don't have to pay the premium pricing for a ready-to-go solution by the official provider) or 2) maximizing flexibility (e.g. add additional features, being able to switch vendors, etc.)
- Use a managed service for storing persistent data
- Use one of the big 3 cloud service providers' (AWS, Google Cloud Platform, Microsoft Azure) proprietary solutions (e.g. Aurora for AWS). You will probably pick one of the 3 providers for the majority of your cloud services anyway:
- Potentially may be the most powerful solution since AWS can optimize the hardware, networking, specifically for that DB product. The biggest downside is vendor lock-in. If AWS becomes more expensive relative to others in the future, it may be difficult to switch.
- Use a smaller cloud service focused on DBaaS such as Compose.io (although it's now part of IBM) or Heroku's PostgresSQL service.
- Databases can take quite a bit of work to setup, maintain, backup, etc. It's also the least forgiving part of your tech stack - mistakes can sink a business here. What you're really paying for, IMO, is 1) convenience - so you don't need to hire a dedicated sysadmin, 2) maintenance & backups - doing routine patches, nightly backups, essentially all the maintenance tasks that ought to do but might not know or remember to do, and 3) someone to call if something goes deeply awry (this will usually cost the most in the form of an add-on enterprise support plan).
- Use a widely-used UI framework
- Wide usage is very important because that ensures most of those nasty browser incompatibility bugs gets caught by the community and not on your product.
- While many people worry about having a cookie-cutter theme (especially with Bootstrap), in practice if you customized the variables, e.g. font-faces, color scheme, base font size, etc., you can still have a unique UI look for your product.
- The most interesting UI library out there is Semantic UI which uses a natural language approach and focuses on making it very easy to customize your site by swapping / combining style guides / themes.
- Use a container management service
- Containers have moved from the point of hype and have seen widespread production usages in many companies. Since the early days of containers, there's been many approaches to managing them, from using basic shell scripts to using traditional configuration management tools (e.g. Chef, Puppet, etc) to powerful resource management frameworks like Mesos.
- The most popular open source project seems to be Kubernetes (backed by Google which provides Google Container Engine) which was based on a decade plus of experience using containers by Google (from their legacy system Borg).
- In theory, Kubernetes can be run on any cloud service, but I think Google Container Engine will provide the easiest setup (when I tried it a while ago, it was not trivial to setup in Azure, even with all the instructions).
- Amazon has their own Elastic Container Services, which I don't know too much about, but it's a proprietary solution so there's concerns about vendor lock-in. It seems like Kubernetes has support from virtually every significant cloud player except for Amazon.
- Use a container registry service
- If you do go the container route, you will need to store all the Docker images in a Docker registry. (The size of all the images can add up quite quickly if you want to keep old versions).
- Docker and Google offer paid services.
Use an event tracking service
- Services like Mixpanel are incredibly useful and provide better reporting than most homegrown solutions.
- Use a monitoring / logging service
- New Relic has a proprietary solution that seems quite advanced (focuses on monitoring availability + performance) - it's pricey but there's a free tier.
- Many companies like to use ELK (Elasticsearch, Logstash, and Kibana) for monitoring.
Use a CI / CD service
- The most established services in this space are Codeship, Circle CI, and Travis CI (which has mostly focused on open source projects and recently offered private project services).
- Shippable is a relatively new comer and focuses extensively on Docker / container-related capabilities. Their pricing also seems like the cheapest at $10/container.
Use pager duty
- Not sure what's the next alternative to PagerDuty, but they do a great job of creating a straightforward application that does what it sounds like.
Use a code repository service (e.g. GitHub)
- Translation management service
- Visual diff - not aware of any services / well-supported open source project for this area
- Task management (Asana, Hansoft X)
- Cross browser / device testing (Sauce Labs)