Wednesday, July 27, 2011

Practical Software Architecture

Why with all of today’s technology are so many development shops still creating projects in an antiquated fashion?  It’s not the simple question of thin or thick client, client/ server, browser based.  It’s not even the question about development language, C#, VB, Java, PHP, Python, Ruby or even COBOL.   Forget about the extensions and libraries, all the various languages has them and sure some are well designed, of course don’t reinvent the wheel.  You have your design patterns, SOA, and different methodologies agile, waterfall and even wow extreme programming.  Hold on, I’ll go scour and grab all the buzz words and give the big sell. Oh forget that mess you get my meaning.

Where is the architecture?  Oh forget all that other mess, sure you can modularize, minimize dependencies, utilize design patterns up the wazzoo and get wonderful but where is your maintainability, expandability.  How do you go about fixing a defect? Who is responsible for the fix?  The person who developed the module or can it be anyone?  Now you have version control issues, and your QA issues, source merging issues, is there rollback plan?  What about the legacy vs. new data after adding a new feature?  At the end of it all how does what’s the maintenance staff? Well you have the picture, and even better the dream of developers out there.  They have their job security. 

This is a crazy discussion; yes every IT shops have varying requirements.  Yet, many IT shops have the same nightmare.  They all have various needs and business requirements for their systems operations.  Yet, how do they expand as requirements change?   This is not a discussion about the some system specific applications, such as operating systems, word processors, the spreadsheets, some hardware embedded solution or purely utility based.   Those are specific requirements out of this scope, and to be discussed in an upcoming episode.   In my belief most application are business database type application dealing with customer support or financial based.

How does an IT shop proceed?  There could be a change in process and focus. There are solutions for all the unique and various requirements.  And there would be many architects and experts that would provide various solutions for the varying requirements.  They are not wrong! I just believe they are not thinking about the big picture, this is where focus is lost, “The Big Picture”

The most perfect example and opportunity is a brand new IT shop all starting from scratch.  Where do we go from here?  And there are many options.  I think most will purchase a system which will meet most of the needs.  They may get lucky and be in a business where there is a package that completely meets their requirements.  The plan is set.  Now look into the future as they grow and expand, and may even change business plan and move in various directions.  Now, they say lets develop our own application and so we have the ability to customize as needed to our new requirements.  The several years later what occurs as direction changes again? 

Some shops are in this situation having a new legacy system that needs to be upgraded.   What occurs now?  Technology has changed and maybe written in an outdated language.  Do they continually upgrade the legacy system or perform yet another rewrite on a new system patching in the old?  New you have new technologies, new patterns, and new libraries.   What happens now?  Are they using the same experts or new experts?   There are a few frames of thought in this circumstance.  You can just maintain the existing code base, perhaps rewrite modules as needed and continually build on and place duck tape along the way.  There are many considerations; planning, budget and even talent.  Many existing is working exactly for what it was intended.  Does the system meet the business requirements?   Will there be expandability as the business change?  And given if the IT shop went the route of purchasing an off the shelf solution, has it been able to grow as the business as grown?  

Where does this madness end?  It is the big picture and it is in your architecture.  Through my experiences and many various opportunities and have seen all sides of the inner working of an organization, the good, bad and the ugly.  Yet, one prevailing thought in many organizations is the barely manageable legacy systems.   Definitely not all of them were terrible, it was the architecture.  I say this now just because of experience and reflection on past projects.  I have to admit some of my projects were terrible and non-maintainable and even a mess, but today and going forward, I want to have a new outlook.

My challenge to all you architects, developers and including project managers is to create nice systems.  Give forethought to what is being created and really ask yourself “How the team can make it better?”   Can you make a better system which is the holy grail of being maintainable, expandable and yes satisfy your customer’s requirements?  And now I hope all you are asking the question “How can we do that?”

My simple answer is to “Simplify”.  I hope you all have heard of the “KISS” expression “Keep It Simple Stupid”.   That philosophy has been rotating around for the longest time, yet nobody really follows through.
   
One of my recent projects required me to create hundreds of form letters to be email or faxed and incorporating various data elements from the system, needing to be recalled and resent as the original.  The business requirements, minimal as they were, asked me to create individual screens for each letter.  To be fair each letter needed different information from various areas from the system.  Perhaps, my management just wanted me to keep me busy for weeks on end, creating all these reports or letters, but I decided to keep it simple.  So, I was lazy gave it some thought and generated these forms within 4 days.  And not only created the hundred but created additional limitless numbers within minutes.

Allowing you to reflect and consider possible solutions, many could give lots of feedback.  What was created was an engine with an interface.  The engine was able to collect information about the form letter, fill in any additional information from the system, allows the user to fill in additional information as necessary, send the email or faxed and saved the original as it was sent for historical purposes.   Letters were stored in RTF format and a table was able to select the proper letter and version to be sent.  An XML of all the fields as entered was also stored to be recalled at any time.  Once this engine was created, the source code wasn’t touch again for years.  It handled any additional form, and even the various versions, yet still was able to recall any old version.
 
What is this Engine?  Well, allow me to digress.  I am always looking at new technologies, and various ways to solve a problem.  Years ago, being an avid gamer, I started noticing these huge games way pass the Pac Man or the Tetris era; I started looking at the Halo, GTA and various other games and researching the technology on how they were created.  Yes, they all had graphics engines to help with the 3D applications, but I also concluded there must be some scripting to follow its particular story line.  And discovered why not apply this to applications? And realized they can be applied to many real world situations.

Now back to architecture, yes this requires a little more thought and I say generalize the problem down to the common functionality.  It is not over designing the problem, and adding functionality not necessary.    It is about designing a solution that is elegant and will working in many applications.  I would rather code a solution once, than must be stuck into recoding a solution many times.   In real world examples most all manufacturing and mechanical type of jobs are accomplished using a pattern or engine of some fashion.  Why not apply this to software architecture and development?

Monday, July 25, 2011

About me and My Technology History


~ 1982 Started on an Atari 800 in School. Learned BASIC and Logo programming.
1983 Got first computer Mattel Aquarius Computer and started my Home Computing. Just played a bunch of games.
1984 Traded up to an Atari 800XL. Played around with Atari Basic. Typed in source code from various magazines at the time.
- Got first 110 baud acoustic coupler modem. Hello world! First experience with BBS.
1985 Traded up to a 300 baud modem. Wow!! Now that was fast. Couldn’t want to get to 1200 baud. Outrageous!!
- Upgraded to the Atari 130XE. Power without the price!
- Started running BBS as a SysOp. Got real interested in communications, yet still a Game Machine. Had hundreds of old 5 1/4 Single sided floppy discs holding 130 kb (per side)
- Anyone remember War Dialers?
1986 Friend gave me an old IBM PC Jr. Hello Intel running old MS DOS 2.1. Wow. Hello Command Line. This included a Monochrome monitor and Hercules Graphics card.
- Got my first Hayes 1200 baud modem. Now lighting fast. ZModem/Xmodem what’s that?
- Hello FidoNet, hello the entire world. Communicate across the world. Remarkable messages can be sent across the world to everyone. No long distance calls just allow the hopping and transport messages within calling zones.
- Developed in COBOL on a Burroughs Mainframe, programs were submitted using Punch cards. Programs were Compilers needing hours to need CPU time. One error, you have to reprocess the next day.
- Did finally upgrade to a real 8088 machine. Had to swap out the Intel Process for a NEC Z20 chip. Now finally 4-bit color.
~ 1987 The punch cards were replaced with dumb terminals now you could actually type in a program. Did JCL and RPG development. Talk about painful! Think about the Hard Drive was a whooping 10 megabytes. Five platters about 8 inches high and three foot wide.
- Developed first commercial application using GWBasic to run a Retail Video rental store. Remember line numbers? I even did cursor control and having fields allowing for customer tracking and VCR tape rental history.
1988 First real IT position. Introduced to Windows 2.0 and PageMaker. Learn dBase for database development. My first introduction to automation. Used dBase to produce forms produced to PageMaker.
- Got my introduction to FoxBASE and Clipper. Now I could develop and compile database applications into EXE files for distribution.
- Introduction to the 3.5 floppy discs. 720k or 1.44 megs. This is way better than the 360k of the 5.25 floppy.
1989 Built my first 80286SX machine from scratch. Remember setting IRQ manually and understand IRQ sharing vs. stacking setting COM ports and video manually? And the old ISA bus? Welcome VGA now 16 bit color. And can we say 9600 baud amazingly fast, but don’t forget about the 16550A uart chip, and the old 40 mb MFM drive.
- A new IT position developing on a PDP-8 talking about memory optimization you only had 4k to play with. Then picked-up R:BASE. Relational database huh?? What is this SQL non-sense??
1990 Thought I was an IT genius. Actually started teaching "Introduction to Computers" and "PageMaker".
- Learned networking the hard way through BNC 10Base2. A fantastic 4megabits network. Great for home use now I can network my machines together. And hello Novell "What is a file server?"
1991 Introduction to Token Ring. Just listen for the clicks of the relay as it passed around the room. Or was it the keyboard clicks of the Form Virus on 18th? Don’t forget to protect them floppy discs.
~ 1992 Now you have Visual Basic for DOS. Wow this is great. Are you saying I processed all them keyboard strokes manually for no reason??
- Developed my first Data Warehouse, well not really since never heard the term before. However, I transformed and processed all the companies’ data into a single database to allow fast and easy access.
1993 Turned in my FoxPro days for some real development. Hello Turbo Pascal. And wow even some Object Oriented Programming. Started to pick up some architectural skills. Redesigned a system which allowed Field Services Techs to dial up and receive updated Technical Specifications. Just the shell, but provided email, navigation and searching capabilities. Once it was released was in operation without change for 5 years.
- Hello CompuServe, seriously now thousands are hooked up across the world at the same time?
- Well now, then you have the real internet. Forget all those BBS dialup sites. Now I connect to one place and get everything? Started scouring all the RFC and researched how this could happen.
- How do I dial in to my local ISP? TCPIP, PPP huh? I need a daemon what?
1994 Hey what's this Windows for Workgroups 3.1? Yuck, I don’t need an interface. Give me my DOS prompt. Slow and all them clicks. I can type faster than that.
- Got my Turbo Pascal, OOP and some nice interfaces. I can write up an application easy enough and even my FoxPro Database is faster in DOS.
- Now comes along you have Ethernet so your saying 10 megabits? And no closed loops, no BNC terminators?? Now this is fun.
1996 Windows 95 well I guess this is better. At least it is faster, and does not crash as much. Well guess I will have to adjust. Still want my DOS prompt. Oh wait; there is a FoxPro and Delphi for Windows now. Not too bad. Guess it’s time to convert everything to Windows. Hey what's this multi-tasking??
- At this time, much of my now useless knowledge of how the hardware interfaces with the operation system became obsolete. Who cares about interrupts and IRQ anymore?
1998 Then about this time I started consulting leaving my position of 6 years. My intent was to experience all the different possibilities and opportunities, I was able to move around the country working on different applications, and view first hand various IT shops and their operation.
- Slowly started to realize the various development styles. My first opportunity I saw how systems could be non-maintainable and visualizing how to this be changed. And realized firsthand the struggle to balance change vs. maintenance.
1999 I took another position to design from scratch the application the architecture through implementation. This time doing purely browser based interfaces. Thin client with the SQL server backend. Very interesting experience visualizing how to organize database and applications interfaces. And strategies to incorporate external third party systems into a seamless implementation. This gave me pause to experience how can various systems be reconciled
2000 After completing that project took another opportunity to architect and develop another system from the ground up. Leading a team of 5, we developed the solution to have a pure dependency free and pure client server system. Utilizing a database layer, a business unit layer and because of user requirements and client based GUI. All the various modules were built on an underlying framework handling display and communication between layers. I thought it was a nice framework, centralized logging and expandable user interface.
2001 My next contract was to take an old FoxPro system and made a simple conversion into windows platform. From the onset, I believed I knew both languages well enough to architect a solution incorporating new business requirements and existing functionality. It did not turn into a mess, but is where really started to realize keeping things simple, yet do not over architect a system. No need for fancy frameworks, just keep it simple and follow the requirements.
With the release of Windows XP, hardware just because a second layer. I started to focus my concentration on protocols and fine tune internet development.
2002 After the internet bubble busted and a lowering economy I took a position as a system developer. For now no longer the wild spree of developing new corporations. Trim the fat shall we say?
- After a very short time, many opportunities developed to utilize many of my talents. Yes, this was a legacy system build and expanded to no end without any consideration architecture. And so I enjoyed architecture, and learning from some triumphs, mistakes or just the wrong direction of the past, I was able to focus implementation of some new ideas.
2003 Being promoted to a middle management position and able to assemble a team of analysis, developers and QA there was a great team. I was on a mission to design, at least the subsystem, I was taking into consideration all of the various aspects of my previous experience.
- Taking on the role of project manager, I started implementing changes in different short phases. Oh no... Thinking Agile. Wait I haven’t heard of that yet, but it was the right way to thinking.
- All in little step and phases we begin modularizing and releasing dependencies between modules of several systems till we were able to begin development.
2004 We completed the conversion into an absolutely independent sub-system yet still utilizing backend interfaces. However we were now able to design a new system, completely new data model, and architecture which are infinitely expandable. Good success, able to transform legacy data into new model, and provide a better user experience. Fantastic.
2005 Given the success of the new system, I was then assigned a new role to develop a data warehouse with a team of my choosing. Whoa, what is a data warehouse? We started having no clue, and through plenty of research later discovered Star vs. Snowflake and Kimball vs. Inmon. We had to discover SSAS and SSIS and evaluated 20 various BI packages. Now, I am changing my direction from software architecture to BI/DW architecture, learning a completely new technology?
2006 After a while with SSRS experience generating pure reports. We actually went a different direction. One that provided a better user experience but more work on the back end. Yes customer experience comes first and foremost. Yet still pondering the question how does SSRS differ from a Crystal Reports or other reporting engines I still wonder.
2007 Now just confident in my architecture ability and not the platform the team of now 4 decided on a direction. We developed the various now defined Agile phases and milestones and proceeded with implementation.
2010 Phases were developed we all learned many new technologies. So far an awesome time…
2011 Starting this Blog.  Wait for many exciting and controversial subjects