Computer Analyst: Litton Systems Canada, 1980-1992

Shortly after moving to Toronto I took a job at Litton Systems Canada (LSL) — a division of Litton Industries. This division was primarily tasked with producing guidance systems, and at one point most of the commercial airline guidance systems in the US and Canada came from LSL.

My position was in a section called Test Equipment Engineering. They were responsible for making sure that all pieces of test equipment used in manufacturing were precisely in spec and accounted for. There were literally thousands of items that were considered “test equipment” — anything from a sophisticated electronic device down to a screwdriver, and also the software that ran any automated tests. Our task was to track every item, recall it according to its schedule, have it tested in a lab, and then update the records and return it to use. Our customers (many contracts were from the US pentagon and other countries) required such records be kept accurately.

When I started there, they had a rudimentary program based on old hierarchical database technology, but they wanted to modernize and improve the system. I was in charge of the migration to Unix-based computers (Hewlett-Packard HP-UX) and a relational database (Informix). I was instrumental in selecting the computer model and other related hardware, doing most of the database programming, and managing a few other workers.

The system pumped out printed forms for each item that was due for calibration, then these forms went to the calibration labs where engineers would retrieve the physical devices (each had a serial number tag). After they were calibrated (tested against standards), the forms were filled in and the devices were returned to use. Some devices required calibration only every few months, while others called for weekly or bi-weekly testing.

I was also responsible for our department’s computer terminal installations and re-locations. In those days, workers had a terminal on their desks, not a computer. The terminals were connected using a COAX cable, and communicated directly to the UNIX computer. One of the attached photos shows the COAX cable tap that was used to connect an individual terminal to the coaxial network line.

Another project that LSL worked on was the first “heads-up” display for the Apache helicopter. Although I wasn’t an engineer, someone offered me a tiny role just for fun: The system had some control buttons with a novel LCD backing that could change icons depending on its function. I was given the opportunity to design one of the LCD icons! I don’t recall what it was, nor do I know if it was ever actually used in production, but it was something of a thrill.

The Litton job (which I kept for 12 years until we left Toronto), was my introduction to UNIX, Hewlett-Packard, and relational databases.

Litton System Canada keychain souvenir
Coax tap connector

Computer Analyst: Abbott Labs, 1992-1994

In 1992 we decided to move from Toronto to Chicago. After 12 years at Litton, this was a major change in my career. After taking a few months to settle in Chicago, I turned to job searching. I found an ad for a consulting position at Abbott Labs headquarters in north Chicago, and easily landed that temporary position.

Turned out that one Abbott division was trying to move from legacy mainframe computing to HP Unix-based systems to save money. After a few months working at Abbott, they offered to make me full-time and I accepted.

The division I worked for was called Clinical Services, and was responsible for gathering, analyzing and reporting on all the clinical trials data for new pharmaceutical products. Of course, my role was as an HP Unix expert, figuring out the hardware, software and design best suited to meet the division processing requirements. The work was interesting, and looked to be a 3-year project to get all the users and applications moved to Unix.

The Abbot Park location was beautiful, very much like a university campus, with multiple buildings in a large park-like setting. Many of the people I interacted with also reminded me of a university environment, since they were PhD medical researchers.

Working at Abbott gave me a deep insight into the trials and tribulations of pharmaceutical research. Although these companies charge what seems like crazy prices for new drugs, and they do make huge profits, then also take huge risks. A small anecdote that made me more sympathetic to this: There was a research scientist that I became friendly with. I’d pass his office a couple of times a week and often stop for a brief chat. He was always at his computer with all sorts of molecular models displayed. One day as I passed I saw him sitting in front of a blank screen. I asked why and he told me that after 5 years researching a promising new drug, he had just proven it to be ineffective! His last 5 years was for nothing, as he said “I’m back to the drawing board.”

I was also blown away by the requirement to print literally 3-5 CARTONS of paper reports about each drug trial — and the FDA required 3 copies, so that could be up to 15 cartons! I had to set up a room with multiple high-speed printers and design a method to organize the print queues to optimize print times while keeping reports organized.

I was two years into this 3-year project when I was spirited away by a local headhunter that I knew. We had already showed solid success by moving more than half of the users and applications to Unix, with the costs being less that a single year of support costs on the mainframe. Moreover, turn around times were faster, so everyone was happy. This headhunter described a new venture within a large company that desperately needed HP Unix expertise, and I had the perfect reputation. The company was Mercer, described in a separate post.

Computer Analyst: William Mercer, 1994-1995

A headhunter I knew called me while I was at Abbott Labs, encouraging me to check out a new job. Although I was happy at Abbott, the challenge sounded interesting, so I went on the interview.

William Mercer, a large New York based insurance company was launching a new venture which they called the Mercer Administration Center (MAC). This was to be a service to outsource employee medical insurance management for large corporations. They had been at it for a year, but were facing some major challenges, many of them, it seemed, HP computer related.

The idea of being involved from the ground up in a new venture sounded good, and they wanted me to take over total ownership of the computing requirements, including showing them how to properly separate the phases of the software development lifecycle (Development, Test, Production.) In addition, they were offering a significant increase over what I was getting from Abbott, and the drive from home was less than half the distance. So I accepted.

After a few weeks I started to understand what had happened there. Seems that the “subject experts” — the ones responsible for figuring our all the ins and outs of a medical insurance management system — didn’t really understand the business. As a result, the company failed to deliver as promised to their first big customer, Pepsi. They had quoted a one year development time, but then failed to tell the customer until after 11 months that they were not going to meet their own deadline. When Pepsi asked how much longer they needed, they said “another year!” Remarkably, Pepsi agreed…

I did my job and established a proper software life cycle computing environment, and trained the programmers on how to properly use it. Over the months that followed, I became very uneasy because it was apparent to me when attending business meetings that they still seemed to have the same problems. Soon, it became clear that one upper level manager based in Louisville was actually lying about the project to the Mercer Insurance parent company! He was (again) re-assuring management that they were on track, when in fact everyone knew they were a mess.

I could clearly see that they were going to miss delivery, probably by another year! So I jumped in (because I’m not the type to silently watch something like that), and at a company meeting where that manager and others were re-assuring all the staff that everything was on schedule, I stood up and asked how they expected us to believe that when we could see that things were still a mess. There was a bit of a tense exchange, and I stated that Pepsi was highly unlikely to grant us another year to complete the project. The meeting, needless to say, became quite noisy, and was quickly ended. Afterwards I was approached by many employees, thanking me for having the guts to state publicly what we all knew was true.

However, I could see the writing on the wall, so I called up my friendly headhunter and said I was ready to move on. Luckily, he found me a great position — because a few months later, William Mercer found out the truth and pulled the plug on the MAC, shutting it down permanently.

I.T. Manager: VP Unix Systems, Information Resources Inc, 1996-2001

When I wanted out of my job at Mercer, my headhunter quickly found me a position at Information Resources in downtown Chicago. I already had a reputation as being a very good problem solver, and IRI was having some problems. But what convinced me even more was that this position carried a vice president title, and paid a lot more that I was making.

My first interview was with a consultant who worked long-term at IRI. He was older, and quite technical. In the interview I got the idea that he wasn’t telling me the whole story, so I pressed him about the problems. He then pulled out a paper from his desk and explained what was going on. Even during the interview, I had some suggestions, which he got excited by. A few days later I was called in for a second interview with one of the “Group Presidents”. The one pressing question that was concerning him was “What if we have a major failure on a Friday night or Saturday, and you can’t be reached?” — To which I looked him in the eye and said “If I am the only person in my organization that can fix the problem, then I’m not doing my job.” He loved that answer, and I got the job offer.

By the way, although I was impressed by the “vice president” title, this was a very large company, and they had about a dozen VPs!

When I came on board, I found out that the Unix servers were repeatedly crashing for all sorts of reasons, disrupting work for clients, and severely affecting data delivery times. (IRI processed scan data from retailer sales, both providing analysis and raw data back to the product manufacturers and distributors. Getting the data in a very timely manner was critical to keeping up to or ahead of competition in the retail space.)

In my third week, while I was still getting to know people and evaluating the situation technically, I was asked to participate in a video conference with Proctor and Gamble. In those days a video conference was a big deal — required a special room with thousands of dollars of equipment and a dedicated high speed line. What I discovered was that P&G was IRI’s biggest customer (millions per year), and they had almost 1000 people whose job it was to analyze the data we supplied. If we were late by even a day, that was 1000 people being paid to sit on their hands!

Although they understood that I was new, they pressed for a fix to their problems. Without really knowing yet what was going on, I promised them a short-term fix within two weeks, and a longer-term fix within 3 months. I basically went out on a limb, just trusting my technical instincts, knowing that these systems should be stable.

Long story short, I found out that the people managing the servers had not applied system software patches for over two years — because, in their words, they were “too busy fighting fires”. Well, those fires were mostly due to the missing software patches! I brought in HP themselves to quickly bring the systems up to the latest patch levels (dozens of software patches were missing), and in a week we suddenly had almost no outages!

I made quite an impression in the first year, solving many similar issues. Some problems were much more complex, but I was not only technically proficient, I was very good at bridging the gap between the technical staff and the business staff, so I was able to get both sides to understand each other better and work out difficulties.

I’ve posted a few interesting emails from the first year. One of the major products at the time was called “Timeshare”, and was basically a portal into our servers that our clients could use to access our analysis programs. The actual packages of analyzed data were called “InfoViews”. Some of these terms may appear in the posted emails.

I will create a couple of separate posts covering specific episodes of my tenure at IRI.

The above letter was written immediately after I returned to my office after a company meeting. It was to my staff but of course I copied senior management.

The snippets below are some of the replies I got to that letter, which management forwarded to many departments, and to the events of the meeting in general.

I.T. Manager: IRI, the EDT project, circa 1997-2001

One of the difficulties of IRI’s data delivery to its customers was that many of the monthly files were very large. In those days most electronic transfers were slow and often unreliable, especially with very large files. So most of our deliveries were on tape cartridges, send to our customers.

Although this had been working for some years, it was fraught with problems. The tape creation process was finicky, and often had to be restarted. Sometimes delivered tapes couldn’t be read by the customer and we had to create a new set, resulting in an additional 2-day delivery delay.

In particular, we have very large monthly deliveries to our offices overseas — the U.K., Italy, France, etc. These had become especially problematic. No one understood why, but most of the time one or more tapes in a delivery set couldn’t be read. They suspected airport x-ray screenings, so they actually started having someone hand-carry them to Europe! However this didn’t solve the problem.

Much effort had been spent on the mainframe side (where the tapes were generated), yet they could not determine the cause of the problems. So they next tried using FTP to electronically move the files. However, in those days, FTP had file size restrictions, and it was also unreliable and non-recoverable — meaning that if the transfer failed 2 hours into a 3 hour transfer, it had to be started over from the beginning.

I was approached to see if I could help. After some thought, I proposed a software solution that I would engineer and program with the help of one of my staff. There was some skepticism (especially from the mainframe folks), but I was given the green light to try.

I had three main objectives: (1) Transfer any size file 100% reliably, (2) maintain security (data lines were not very secure, and much of the data was considered proprietary), and (3) engineer a solution that our client IT departments would be willing to install on their systems.

The third of those requirements steered me towards developing the entire system using only Unix scripting rather than any compiled language, and in a way that did not require special system privileges. As an IT manager, I knew that would be the only way to overcome any client objections.

I designed a system based loosely on package deliveries by the likes of FedEx. I defined each delivery as a package, with an accompanying manifest. Much like when you ship a lot of goods, you divide your shipment into smaller packages, my large files would be split into “chunks”, each of which would be sent separately, re-assembled at the receiving end based on the manifest. For reliability, if any “chunk” failed to deliver (verified by a mathematical checksum), the system would proceed with the other chunks and only later re-deliver those chunks that got missed. In that way, an unreliable line would never force a restart, simply require one or two small chunks of the delivery to be resent.

Obviously there was a lot more to this, but that’s the nutshell version. The EDT system (as I called it) became quite a success story. Clients were thrilled because it cut a day or more off their delivery times; the European offices saw a 4-5 day reduction. Even more importantly, once in place there were zero errors!

I’ve attached a number of email images, and a full slide show that I presented live to Frito-Lay in Plano, Texas.

A partial client list of companies that I got onto the EDT platform

The following are clips from emails ( bragging rights 🙂 )