We've all been there. We have to choose a design or development (or both!) shop for a big project. We have to invest time and money and then put it all on the line and in the hands of a development shop. Success depends on picking the right team, but businesses often choose poorly.
Why am I qualified to speak to this subject? I am a co-founder of a custom software design and development shop, Anthroware, based in Asheville, NC. We deal with this subject as product/brand strategy consultants, designers, developers, and software architects on a daily basis. We watch clients make decisions about whom to hire, some good and some very bad. We see clients ask the wrong questions and use bad data to make expensive decisions about development.
This article is a brief summary of what I wish my clients (past, present, and future) knew about this process. If you take the time to read this, it will give you some insight and help you to be well-armed to make a great decision about your next design or development project. This information is useful for many industries, not just software design and development or technology consulting services.
1. It's not about the ballpark estimate. Period.
If you ask for a 'free ballpark estimate' for your project it will be inaccurate. Without going through some sort of requirements gathering phase it is impossible for a design / development shop to quickly judge how much time and money your project will take. Ballpark estimates are subjective and are either really low (only estimate the bare minimum specified features which is rarely enough) or high enough (read: padded) to account for the unknowns. I spent quite a bit of time ranting on about requirements gathering in another blog post, here.
“Without going through some sort of requirements gathering phase it is impossible for a design / development shop to quickly judge how much time and money your project will take.”
Another mission critical component of this first ‘discovery’ or ‘requirements gathering’ phase is to consider some key aspects of your project or product.
- Why does the world need this project? Can you articulate this with clarity? We call this the product or project mission. Defining this early helps maintain focus as your project comes into being.
- What are the performance criteria to measure success of the project. Is it adoption? Revenue? Efficiency?
- How can I test critical assumptions about this idea with my users to learn as much as I can before we start an expensive build?
- Does my market or audience support this product?
This kind of strategic ideation along with a solid requirements gathering phase provides a holistic picture of many of the struggles or critical elements to any project. This is a big part of the the information you need to go forth and make a good business decision about your technology investment.
Other than that, you need to know how much it'll cost to build and how to pick a good development partner.
To get a real idea of how much time and money a project will take you must undergo a requirements gathering phase and expect to spend a few dollars. Best case scenario is that you are able to pick your development shop first, and then go through a requirements gathering process with them- at which point they are not only providing a real assessment of time and cost but also working through decisions about the technology direction and high-level architecture. After this process you are armed with great information about what it will actually take to complete the project, and use that information to make a good business decision about the project. Lets figure out how to choose a development partner early on, without asking for a ballpark estimate.
Why is a ballpark estimate bad?
The bottom line is that ballpark estimates do not compare companies in an objective manner. You can't compare apples to apples without using some new metrics.
Anthroware was recently bidding for a website design and development job. There were very few questions answered out of the gate, but the company insisted on getting a ballpark as the first round of interview. Knowing what we know about software, we knew that we didn't have the whole story, and submitted an estimate for almost $50k- fully expecting to be able to explain this number to the client.
The competing estimates came in much lower because they estimated the three features the client asked for, but I don't believe they didn't know it would take more. We never got a chance to explain our bid before the guy that submitted the $15k bid got the work.
Guess what, when the rubber hit the road the client ended up spending more than $50k for the project because the 3 'requirements' that were estimated for the low bid did not describe the actual amount of time, effort, and expertise needed to complete the project.
This real-life example stinks! The client realized quickly the project cost much more than the original 'estimate' and is probably already sick of hearing "that's not in the spec.” It's expensive to switch companies at this point because they are contractually obligated to spend $15k with the guy that won the bid.
What's even worse is that there is very little chance of the client feeling like they got a good deal out of this.
Here at Anthroware, we talk a lot about whether or not we are giving our clients value. Being honest about the true cost from beginning helps us set appropriate expectations and goals with our clients and staff, which is better for everyone involved.
The big question is if ballpark estimates are so bad, then why do companies use them as one of the primary data points to make a decision about which firm to hire? You don't know? Me neither.
2. How to choose a company without a ballpark estimate.
It's actually not too tough to make a good decision about a development and/or design firm. You need to understand what characteristics are important to a healthy product development lifecycle. It's also important to understand which items are subjective (like if you feel that you can trust the company) and which items are objective (key metrics and true or false statements).
Let's start by naming some characteristics that could make a large impact on your development:
- Do you trust that the company has a culture of personal accountability? “Do I trust the people?” is too subjective. As humans we generally WANT to trust people. But if a company has a culture of personal accountability, it's apparent in how they deal with their employees and clients. If you trust that you are hiring folks that will work to solve problems and not shift blame then you will be able to imagine how working with them might be a pleasure!
- Capacity and Schedule. Is this a small company with 'a developer' that you are considering trusting your $500k project to? Are they tied up for the next six months? You want to ask who will be working on your project, and if it's a large project, then you want a team of a few developers/designers to do the work. Will you be doing any of the work in-house, like quality assurance testing? If not, then you want to make sure the company has the capacity to take care of those needs as well. Small teams can effectively and efficiently execute large projects, and at some level adding more bodies to the problem does not equate to faster or better development. At some level this is a subjective indicator and at some level it's objective. Use your head.
- Technologies. Your project is unique. Not all software development platforms are good and certainly some are better suited to certain tasks than others. Technology decisions should be made based on things like: what software already exists that this will need to interact with? How many users will be utilizing this product simultaneously? Is there existing code that could potentially be used or updated? Development platforms are like tools in a toolbox. You wouldn't want a company that only knows Wordpress to build your special education goals tracking application because it needs to hold protected student records and Wordpress is not secure enough. That's just one example of thousands. Make sure you ask the company you are interviewing what technologies would work for this type of problem and why they would pick one vs. another to architect a solution. Beware of companies that only work on one technology framework. They will try to fit your project into their technology vs. trying to make the best technology decision about development. Does the company I'm interviewing work in multiple technologies and are they talking to me about why to choose one vs. another? Yes or no.
- Safe Development and Build Practices. Technology and design companies alike should be taking steps to nurture communication and to protect the assets (code, design work, documents, etc.) for your project. A safe development cycle means that there is good communication and no chance of losing work. At Anthroware, like many shops, we organize our project development cycle using Agile development methodologies. This provides us a lightweight framework for communication with our clients and staff. It also provides a fast feedback loop because we can post progress for review quickly. From a safe "build” standpoint, we use measures to ensure that the changes/updates/additions we make to any project never get published until they are tested and ready. For software development, this means having a series of servers that mimic your production environment so that the quality and function of your product can be tested before 'going live'. I could probably write an entire article on this. For now just make sure you are asking about safe development practices- it's amazing to hear the stories about how some companies post un-tested changes to live and worse! Does this company use safe development/build practices? Yes or no.
- Can this shop do all of my project, or just part of it? This one is a 'yes or no' type question, but it's relevance or importance to your project is subjective. The idea is that if you hire one company to deliver a turn key solution, then if a problem arises, they work on fixing it instead of blame-shifting to a different company when a road bump happens. Think about building a house, if you go to an architect to design the home and then go to a separate builder to construct it, when something comes up about the design of the house the builder blames the architect. Blame-shifting; the productivity killer. Conversely, if you instead go to a firm that has in-house architects and builders under the same roof, when that same problem arises you, the customer, have a much better experience because the same company is vested with the responsibility of building your house from blueprints to kitchen cabinets. It's not your problem, it's theirs. There are reasons to split this up, but for many projects your customer experience will be better if you go with a company that does all of your project. If you hire a design firm to design the software, and a separate development firm to build it prepare yourself for each company to be protective over their budget.
- Ask for references. It's totally OK to ask a company for some references. Companies young and old still have to be honest about their skill level, if they've done the type of work before, and how it has gone in the past. As a brand new company, we had to explain over and over again that 'corporately we haven't done this type of job together, but individually we have combined 60 years of experience with this'. It got old and I thought we'd never sell anything, but we were honest and eventually word got out we did indeed know how to make software 'go'.
- Value. Normally, clients try to get a feeling for value based on the ballpark estimate (boo, hiss). As we suggested earlier, this is a bad idea. The reason it's last on this list is because the question of "value” isn't just monetary. Look at all the items in this list; scoring well in these items (or not scoring well) directly affects value. However, budget and schedule is really important on it's own, so lets talk about how to get a sense of value without a ballpark estimate. If two companies that have been in business the same amount of years, both quote $20k for a job, how would you pick? Maybe the answers to some of our other questions are different, and that will help us make a decision, but lets go ahead and ask two more questions: (1) What is the average years of experience for the individuals (architects, developers and design only, not management, and cap the number at 10) who will be working on my project? (2) What is the average hourly rate for projects of this size? These questions help you answer several questions.
- Is this a team of fresh out of college coders and
designers or a team of highly advanced and experienced professionals who have done this many times before? - Knowing the average hourly rate, I can estimate (roughly) how many hours each company has estimated to complete the project. If the average hourly rate is $116 for Company A, then they are estimating $20k/$116 = 173 hours to complete the project. If Company B's average hourly rate is $150, then they are estimating $20k/$150 = 134 hours to complete the project. This may be a significant scheduling benefit if Company B can get the job done faster.
- This is cool, lets call it "value ratio". Ultimately you want to know how far your dollar is going. Here's a kicker, Company B has an average team years of experience of 8 years while Company A's average is 5 years. Value Ratio = (Ave. Hourly Cost)/(Ave. Years Exp.). Company A's Value = 116/5 = 23.2, that's pretty high. Company B's Value = 150/8 = 18.75. Even thought Company B is much more expensive hourly, the cost to experience ratio is better. Occasionally, if a company is able to keep overhead really low (like Anthroware), you will see ratios below 15. At Anthroware, our average hourly rate for large projects is currently ~$135/hr, and we hire mostly senior engineers because we want to get the project done right the first time… Anthroware's Value = 135/9 = 15.0. This is a great way to compare companies that gives you a sense of value. It's not perfect, but its much better than using a ballpark.
So there you have it. You're all set. You know everything you need to know, so just plug in some data and let the metrics do the decision making. Well, almost. This is an exercise to help make decisions about your development shop. The most cost effective solution may not be the most valuable- that should be apparent now. But also these other questions to ask are all important. When we take the ballpark estimate off the table, because it's a worthless fantasy, we are able to focus on the needs of our project.
3. The other factors you need to be aware of.
Whether this is your first dive into building a technical product or your 100th, it's important to understand that no matter how carefully you choose your development/design/marketing partner there will be problems, additional costs, and unforeseen issues. Ask yourself how resolving differences in opinion or differences in interpretation of the specification (what's in the contract) would be resolved with the various companies.
Are you trying to hit a moving target based on customer feedback? If so, it might make sense to weed out any companies that aren't practicing Agile development methodologies. On a related note, specifically ask how often you will be able to see progress. If an agile shop is making internal pushes every few days you could expect to see updates to your demo or 'staging' server once a week or less.
Some projects have specific concerns that can be used to help your decision making process. For example, if the product uses protected data (HIPPA, IDEA act, FERPA, etc.), not all agencies are able to handle that correctly. Make sure you ask if their team has experience with these kind of things if your project has specific legal requirements.
A lot of the success of the project still rests on the clients' shoulders. Good communication, being responsive to questions and emails, patience, and just being available go a long way. Just because you have done an excellent job hiring a great team to work on your project doesn't mean it's not going to be a lot of work for you or your product manager. Pick a team you can work alongside.
4. Contracts.
Contracts, contracts, contracts. I could write for pages about contracts, but I'm not a lawyer so take this for free advice and leave it at that. The essence of a GOOD contract is to clearly communicate the deliverables of the project and limit liability. Also, at some point (usually upon final payment) the contract helps legally transfer intellectual property from the development shop to the product owner. In fact, when done correctly, contracts are meant to be a helpful, fair, communication tool for both sides.
5. Wrap it up.
Whew - that's a lot to talk about at your next development partner interview. And these are the cliff notes! The key principles are similar to if you were hiring an employee.
Honesty, personal accountability, expertise, value. Try starting with the value ratio. If a company has a good (low) value ratio, and they hit the marks on the other items we discuss here the chances of hiring well are very high.
Hopefully, you are now convinced that asking agencies for ballpark estimates doesn't actually give you much (if any) usable information about choosing a company and if you give this method a try please email the shop and let me know how it went.
