There has got to be a better way of doing Software Engineering than what we’re currently slogging through at work. I heard a guy on NPR talking about Software Engineering broadly and mentioning that one of the problems is that there really aren’t any established standards. Why is software so much harder to build than “real stuff” like bridges and buildings and highways. Granted, there are plenty of examples of poorly constructed bridges, buildings, and highways but it seems like there are more standards and better practices applied in those disciplines. Maybe it’s because people’s lives depend on quality design and construction. But then again, much software out there can be “life saving” in similar ways. The software than runs medical systems or military systems is life altering. Maybe that’s why so many of those systems are out-dated technologically simply because no quality product (no matter how much newer) has been successfully developed.
So, what’s the problem and why hasn’t it been solved? Maybe it’s because software is too easy to build (hey, it’s just rearranging rust on a metal plate). It’s easy to release something, hope for the best and just fix problems when they come. A lot easier than constantly taking a bridge apart to fix engineering faults. So maybe the issue is that because software is “so easy” to fix, few people take the time to get it right first when they can almost just as easily get it wrong the first 20 or so times. Why debug a program when the public does this for free by submitting trouble tickets? Of course, some products actually work remarkably well and seem to have been tested and released in an environment that really was focused on quality. But even these products DO have major problems and DO have major releases that fix problems and create new ones.
In building bridges, we have physics to contend with and although it’s difficult to beat and it’s amazing what humans can accomplish to “beat physics”, it’s a relatively simple set of rules that will determine whether the design will work or not. Civil engineers don’t get involved in string theory to build a bridge, they assume a pretty normal set of conditions — the same sort of conditions that allowed Greek builders 2,000+ years ago to build beautiful and functional buildings. In software, we don’t really have a set of rules. We’re trying to make the customer happy and so we have to modify it to meet their needs. Their needs (and desires) change daily so it’s an always moving target. A bridge might be built a certain way but will only be rebuilt when the desire is so great that it warrants the amazingly high cost to redo the work. Software suffers from the problem that every change is just so darn easy — or at least that’s the belief.
Or maybe it has to do with what we’re building. We’re not trying to build something that exists, we’re trying to optimize (automate and streamline) the flow of information. When you’re building a bridge, you’re concerned (usually) with making a platform that things can move across to get to the other side of the bridge. When you build a software program you’re concerned with making stuff move quicker and quicker and better and better to more and more places. It has no solid end-goal. “Better” is pretty much what Version 2.0 of all software products advertise. Better because it moves information faster, or more efficiently to more places or presents it differently (in some abstractly “better” way). It’s all so subjective. If you have an existing software program that runs “slowly” you could rewrite it specifying that it should run “twice as fast”. But this is a silly requirement if current technology allows it to run 200 times as fast. Things change so often and so quickly that what you want at the time of the writing of the software is “the best”. But the best will have changed between the time that the project requirements were written and the time that the software product is released. It seems just as silly to say “200 times as fast” at the beginning of the project (because the engineers assure you it can be done) and meet that goal only to find out that your system is now out of date because it can now be done 400 times as quickly with only minimally more effort.
A bridge has subjective elements of design. It should be “beautiful” or “classic” or something like that. These details are worked through with the architect and the customer who nail down what that means exactly. Functionality is essential though. With regards to these subjective aspects, the customer has to decide how much beauty can be got for how much money and come to a decision on that. There’s going to be a trade-off on the beauty part. In software, although I rarely would call one of my own creations “beautiful”, most of the design ideas are subjective. “We want it better” means little. “We want it faster” means more but doesn’t tell me as an engineer what the final product should look like. “We want it 100 times faster” means a lot but either it’s not feasible to do this at this time, or if it IS feasible, it’s unlikely that the customer really wanted it that much faster (i.e. it was probably the engineers idea to put the number “100 times” into the requirement since they knew that was what WAS feasible). If this is the case, the customer might end up with a product 100 times faster but won’t be happy to find that this puts them behind the curve when the product is done. When you’re talking about a highly integrated product or a “system” of multiple software packages, hardware, and processes, it’s almost impossible to ever deliver a “modern” system.
What’s the answer? I have no idea. It seems like there’s got to be some sort of way structuring a design and development process to focus on the unique aspects of software products. Check this space for more ramblings.
#1 by Peter on January 29, 2007 - 5:04 pm
“In software, we don’t really have a set of rules.”
Very shades-of-The-Matrix …
It seems to me that people fail to realize that software is function and performance, not structure.
An application is not a bridge. It is a way, a method, a process, but old school managers treat them like buildings.
The key – it would seem – is to recognize the nature of the beast.
Software engineering is more like exploring than like laying brick and mortar. It is engineering, not construction. It is invention, not manufacture.
Most of all, it is creative, not rote.
That is to say, it is those things at its core.
The rest is coding. That part is essentially building a bridge. You can plan that part roughly the same way. Can you hit a snag in the coding? Of course … same as you can hit a snag driving piles for your bridge. Can there be defects in the final product? Of course – people make mistakes.
That’s only part of the answer, though.
Software “works” by display – by revelation – by illustration. If information comes out, software is working. If the right information comes out, the software is working properly. In that sense, sofware implementation is more like medicine – only medicine performed on something unintelligently (by comparison) designed.
Ah well – enough ramblings for a reply, eh.
Keep up the good work!
#2 by Andrew Flanagan on January 31, 2007 - 4:51 pm
I understood your problem! Sorry, I don’t think there’s any solution because they will always find ways to make things smaller and faster or bigger and better. If they weren’t able to, you’d probably be out of a job. So keep up the good work, sweetie!
~SARAH~
#3 by Bill on March 5, 2007 - 6:46 pm
This is a nice looking blog Andrew!
Your compare/contrast with building a bridge is a productive one. Designing software seems one of the toughest engineering problems, yet the science is so new (the most experienced are practically self trained), and there is so much we are still figuring out.
In the past we thought of software as I/O (inputs and outputs); essentially software engineering was considered “finding the best way to go from point A (input) to point B (output)” – i.e. the simple act of coding that Peter mentions.
Yet changing business requirements (shifting the I’s and O’s daily in respond to user requests, environmental changes) bring such systems to their knees; the underlying design is not robust enough to sustain those continual changes in an organized way.
For example I have a SQL programmer who makes 5 changes a week to our software and we have to go back-and-forth a half dozen times on every little tiny change he makes. The whole thing is just a growing rat’s nest of knarly scripting code with raw SQL statements embedded everywhere. Each change is taking progressively longer as his code becomes more complicated, and as things get worse, each change is more of a disaster that takes literally weeks and months to debug for production.
Enterprise business process software like SAP use gigantic frameworks to apply some method to the madness, but the method IS madness. These huge systems are not so useful to mid-size companies and agencies (say less than 200M revenue) which rapidly changing environments but cannot justify armies of consultants deploying huge systems which they will underutilize dramatically.
One issue here is that designers seem to forget that needs will change. Software systems delivery is an ongoing partnership. These days there are few packaged products that do anything nontrivial without the need for constant updating and modification.
The focus needs to be on building a system that can sustain a queue of changes without exponentially growing the complexity (i.e. complexity is ke^n where n is the number of features at any given time). Maybe some of this may be done through a component model, which to day is still fraught with problems. The thing is, automobile designers are getting it more or less right. Cars seem to break down less than computers, and the same ideas apply (cars these days are filled with embedded that work together in a network).