TheGarage Tech Blog

(Editors note: This piece was originally written in Sept 2007 but was never published; it evidently moved off the front page as a draft and was forgotten. Since the material is  still relevant, I decided not to let a bit of content go to waste. )

*  *  *  *  *

Cory Doctorow in the Guardian on free data sharing:

Since the 1970s, pundits have predicted a transition to an “information economy”. The vision of an economy based on information seized the imaginations of the world’s governments. For decades now, they have been creating policies to “protect” information: stronger copyright laws, international treaties on patents and trademarks, treaties to protect anti-copying technology.

The thinking is simple: an information economy must be based on buying and selling information. Therefore, we need policies to make it harder to get access to information unless you’ve paid for it.

…

The world’s governments might have bought into the old myth of the information economy, but not so much that they’re willing to ban the PC and the internet.

Cory Doctorow is an activist, science fiction author and co-editor of the blog Boing Boing.

It’s a good article explaining why the Internet was no bubble for a lot of people but I think his premise is off. It’s not so much that the government bought into the “old myth” of how information needs to be secured instead of shared, it’s that they need it to be that way.

The old myth is not a myth. In a spice economy, he who controls the spice controls the universe. In an information economy, he who controls the information controls the universe.

Although abstraction is crucial in any effort to build a class library of re-usable business objects it is important to not get carried away and become what is known in my book as a “purist”. Physical patterns don’t always match logical patterns. Something beautiful on paper may be a clusterduck to implement in the real world. Pardon my French but if you are reading this kind of dry material, you probably already speak French. Our goal here is flexibility and efficiency, not adherence to rigid dogma for the sake of purity.

As an example of the point I am trying to make above lets revisit the Data Access class for a moment. There are some purists who will say no business logic should be embedded in the data access layer and no data access functions should be found in the business layer. On paper that is correct and a proper design should reflect that. In the real world where some guy is trying to piece everything together one can either go through great contortions to satisfy the separation of concerns aspect of encapsulation as laid out by the purist, or one can have a boat load of stored procedures in the database to return pre-configured datasets.

The reason why the stored procedure scenario breaks the rules of encapsulation is because the business layer is required to know something about how to return certain sets of data from the back-end data stores. I have concluded that breaking this rule is preferable to maintaining a SQL command object in every business object. The end result is the same.

I don’t know if this is even technically breaking the encapsulation rule but I have read a couple of articles that suggest it is. In a logical diagram I can draw the stored procedures and views as business rules, which are clearly in the business layer but in a physical diagram the procedures are stored in the back end database and thus are retrieved using the connection object in the DAL. The best of both worlds.

So what I am saying here is that stored procedures will be used extensively in this project including the current Employee business object. As mentioned at the end of the previous article we need to figure out what all is needed in the employee object. One thing the Employee object will definitely need is a database connection so it will need to know how to ask the Data Access Layer (DAL) for a connection. This is likely a static property but probably should nonetheless be acquired from a system initialization mechanism, like an .ini file. The value of this property should never be the connection string as the connection string is the business of the Data Access Layer.

Also, the Employee object must know how to get data to satisfy every method and property that is to be made available according to the contract made at instantiation. If the employee object says it can produce a punch history list, the object has to know the secret words to get a punch history list back from the DAL.

The calling program says the secret words to get a new employee object with ‘1234′ as the employee id –> set myEmp = new Employee('1234')

Everything goes well and Bob Jones’ employee record is returned encapsulated in an employee object. To get Bob’s punch history, I just need to request it from the Employee object. Depending on implementation the punch history maybe stored inside the object or returned as a record set. Either way, when me.getPunchHistory() is called, the object has to know how to request it from the DAL –>

myDBConn.RunProc(’stored_proc_Punch_History’, ‘1234′)

That’s it in a nutshell. Every bit of data I may want to retrieve, such as schedule, absence, vacation, is made available through the employee object utilizing stored procedures. The basic employee stuff is populated when the employee object is instantiated, the rest is provided as needed. The good news is that from now on, whenever a new application is needed that will interact with employee data, all the data access functions are predefined and the programmer can focus on writing the needed application instead of worrying about database connections, security, SQL queries, and all the rest of the administrative stuff.

In the next article we will take a look a how to display our employee information to the user, which is in the presentation layer of an n-tier application. Decisions made here should not in any way affect what the Data Access Layer will ultimately look like. But this is one of those cases where the physical architecture of an actual enterprise environment dictates as to how the logical models are implemented. VB.net is not the best tool for implementing design patterns because of the code-behind-the-form method used in the product. Such a construct physically binds the presentation and business layers, but not necessarily logically.

As I mentioned above, this contradiction can be avoided if the developer eschews all the nice functionality provided by VB.net, namely the typed datasets available in the connection object which can bind data directly to the components on the web forms. The code is generated automatically and the SQL query results are shoved directly into the presentation layer…

Considering that we are mandated to use VB.Net as the development platform, what should we do? Use VB.Net datasets, or write our own SQL Command objects to maintain order with the model-view-controller design pattern evident in the n-tier model?

So I’ve pitched building a library of business objects as the solution to transform about a million lines of VB code sitting in a hundred or so 7- to 10-year-old Access database applications into a valuable corporate asset of reusable code objects built using current technology.

The director of the IT department is not an IT guy but he has a great interest in sorting out the huge tangle of code he inherited with his job: it is expensive to maintain, difficult to secure, and is not easy to enhance. In short, it is a strategic liability.

Prior to me coming along, the solution has always been assumed that every application would simply be re-written in something else, that something else likely being Visual Basic dot Net. Since day 1 I have been lobbying against that approach in favor of analyzing the applications and consolidating the code under a set of centrally managed business objects as defined by the companies existing back-end data structures and their application needs.

In other words, I’ve proposed… get ready for the buzzwords… to build the middleware in a multi-tiered distributed database architecture. The task is daunting and to the casual observer may seem impossible but it is doable nonetheless. Though my boss is in charge of the IT department, I would consider him a casual observer so his response, as I had hoped and expected, was “show me.” We picked one of the more sophisticated MS Access database applications in the inventory and I am in the process of converting it to an n-tiered application using object oriented design and implementation.

Because I am having to write the code as well as do the design work, as well as support the current state of the beast, getting the design laid out has been a slower process than I am accustomed. The area I support is HR/Payroll along with all the supporting applications for time and attendance, manpower scheduling, and other personnel related activities so it was a no-brainer to start off by building the employee object first since every single application will at some point need to manipulate an employee’s data in some way.

As I’ve touched on before, these objects in the business tier are built separately from both the interfaces that utilize them and the objects that actually access the SQL data stores. Ideally the business objects are built to work independently of the consuming applications and the source of the data being manipulated. This is accomplished on the application side by defining a contract between the business object and the application that consumes it: if an application requests an employee from me under certain terms (the parameter list in the constructor call) I will return an employee object with certain information (the public properties and methods). No mention is made of what the application will do with the data nor does the business object disclose how the data is made available.

On the data access side,  the business object becomes the consumer and has a contract with an object in the data access layer (an ADO.net connection object most likely.) If I know how to ask for something, say an employee’s punch history, the data access object will return the data without disclosing where it came from and without regard to what I am doing with it. We’ll get to security later.

Lets take an inventory of what we have so far towards this goal.  We have a meta-data dictionary of all the VB code base that was created with the Access documenter utility we built. We have a simple DBConnection class to serve as a basic data access layer. And now we are starting on our first business business object: the employee.

Next time we will take a closer look at the employee object and decide what we need in there and what will be the best way to set up the design. We will also be tweaking our DBConnection class as we go along to take advantage of some of the very nice features of ADO, namely the typed dataset object which looks very unwieldy but one simply cannot ignore the power of the tools provided in the class.

6:00 AM, The Garage — I don’t have much time to write this morning but I have been busy in the tech world so It’s time to start a new series. I am still working on sortin gout the VB6 code base for the guys I am currently working for. Remember, they are the ones with over a hundred MS Access databases interfacing their SQL data. I just about have them sold on developing a library of reusable business objects so I am going to bring everyone up to date on the progress and do a few follow-up articles on the process as it continues.

I support the majority of the Access applications (along with the people who use them) as well as several IIS web applications built using Microsoft’s Web Classes architecture (along with the people who use them) and the Ceridian HR/Payroll application and SQL database (along with the people who use them.) The guy who created the mess wrote all VB code in all these applications made some attempts to encapsulate the common objects but really didn’t understand the basic concept of Object Oriented Design. For example, he built a class for database connectivity but all of the connection parameters were hard coded in modules running the main application code, which of course defeats the whole purpose of reusable, low-maintenance code.

12:00 PM, on-site, Tx City — One of the main purposes of OOD and OOP is encapsulation, meaning all the components of an object, both methods and properties are encapsulated withing the object. As a programmer, if I want a new database connection and I have my database connectivity defined in a reusable object, dbConn for example, I want to be able to request a database connection and not have to worry about how the connection is made.

So if I need to write a little utility that lists all employees and their job codes I want to be able to get a database connection by simply requesting it. I don’t want to have to set connection strings and server names and passwords and such. In a large enterprise there is likely not only many databases, but many database (SQL/Oracle/Domino/etc) servers as well. Changes made to the back-end infrastructure will break any code that has the connection parameters hard wired.

For example, here is how the apps are all written now using ODBC DSN connections:

Set dbADC = New DBConn
dbADC.DBToUse = "DSN=TimeAtt"
Set dbBuildSec = New DBConn
dbBuildSec.DBToUse = "DSN=BuildSec"

Using the above method, if a database is moved or a server crashes all the DSN configurations have to be changed on every computer running the application, which is not as bad as this scenario:

conn = New OleDb.OleDbConnection("PROVIDER=SQLOLEDB;" _
& "server=Server_002;" _
& "Initial Catalog=BuildSec;" _
& "User Id=userid;" _
& "Password=pwd;")

in which case every instance of the code in every instance of every application that uses the code would have to be changed. In short, a nightmare — in which I have been living for about five months now.

So what if one chooses to do things my way? Well the concept of building a class to manage database connectivity is sound but heck what’s the use if robust code is not the end result. Where the previous coders failed is in not understanding encapsulation or how to build overloaded constructors (polymorphism) in their class so the class can handle disparate requests when a new object is instantiated from the class.

Set myTandA = new DBConnection("TimeAtt")
Set myBuildSec = new DBConnection("BuildSec")
Set myEmpRpts = new DBConnection("HRRpts", "Access")

Then the constructors in my DBConnection class might look something like this:

Public Class DBConnection
Private mConnectionString As String
Private mConn As ADODB.Connection

Sub New()

    'This is the default constructor and will return an error

End Sub

Sub New(ByVal dbname As String)
        'Connection strings defined in Constant declarations

        Select Case UCase(dbname)

            Case "TimeATT"
                mConnectionString = TIME_&_ATTENDANCE_CONNECTION_STRING

            Case "VHR_DATASQL"
                mConnectionString = HR_CONNECTION_STRING

            Case "EARNHIST"
                mConnectionString = EARNHIST_CONNECTION_STRING

            Case "BUILDSEC"
                mConnectionString = BUILDSEC_CONNECTION_STRING

         End Select

         mConn = New ADODB.Connection
         mConn.ConnectionString = mConnectionString
         mConn.Open()
End Sub

Sub New(ByVal dbname As String, ByVal dbtype As String)

    'this contructor will be used to open non-sql data sources
    'valid types will be Access, ODBC, etc)

End Sub

Pretty sweet huh. Now if something happens in the server room or even if database vendors are changed, the code only has to be updated in the class definition to reflect the changes and the programs utilizing the class go on like nothing ever happened.

To be continued…

Where I work they have over a hundred Microsoft Access databases that act as interfaces into their corporate SQL data. These databases range from Access 97 to Access 2003. My client would like to get away from this code base but so much of their business rules are embedded in the access code (VB6) that it has proven difficult to even contemplate ditching MS Access and VB6. Lax version control has made the planning much more difficult and tedious. Live and learn, right.

To get a handle on the task at hand I wanted to document all the databases by recording all the objects into another database. A meta-database, as it were. Didn’t seem like a big deal until I started trying to access different properties of some of the different objects I was interested in; specifically the record source for reports and forms and the content of the code modules. I came to the conclusion in the process that the Microsoft object model is so unwieldy that I now categorize it as crap.

So the Internet search for knowledge began and the insight that I needed to finish my little utility came from John Barnett who wrote a similar utility called mdbDoc that builds a nice html document of all the db objects contained in an Access application. His app was set up as an MS Access add-on (.mda) that could be executed from within any Access application. It is a very nice piece of work except that the html formatting is embedded with the code and that it has to be run one database at a time. I needed to document a couple hundred databases all at once and with an eye toward consolidation I needed to be able to view all the queries from all the applications sorted together in one place. Same with the code modules and tables. Though his utility wasn’t suitable for what I needed I was able to configure it as if it were a function in my app and attached the resulting HTML doc to a rich text field in the database structure I built for each .mdb file.

There were a couple of subroutines in John’s code that I was going to have to write or do without so with much gratitude I borrowed the ListCodeBlocks and dependent subs from John Barnett’s mdbDoc. I have a nice Logger class that I use when scripting in Domino but since I was doing this all in VB6 and I am so rusty with VB, I borrowed John’s mdbdclsFileHandle too. Another nice piece of work that takes away all the mundane tasks of reading and writing to disk.

So here it is. The following code builds a Domino database with a record for each MS Access object. The back end could just as easily by SQL Server or MySQL. I just chose Domino because it was easily accessible and convenient. Now I can sort by object type regardless of what database actually contains the object.

To use the app you fill an array with all the target directories you want to scan. You can rig it to crawl servers but in my case that was extra features that weren’t needed.

When you start the application you check off the objects you want to document and click ‘Do It!’ When the app finishes you open up your target database and look at the results. In Domino it looks something like this:

The main code follows. If you want the modDocumenter code or the file handle class in John Barnett’s mdbd utility, you can get it from his site, linked above. (If you have a free editor like like PSPad or ConText, paste the code over there. To me it’s easier to read code that way.)

Read the rest of this entry »