I.T Manager Archives -

I.T. Manager: VP Unix Systems, Information Resources Inc, 1996-2001

When I wanted out of my job at Mercer, my headhunter quickly found me a position at Information Resources in downtown Chicago. I already had a reputation as being a very good problem solver, and IRI was having some problems. But what convinced me even more was that this position carried a vice president title, and paid a lot more that I was making.

My first interview was with a consultant who worked long-term at IRI. He was older, and quite technical. In the interview I got the idea that he wasn’t telling me the whole story, so I pressed him about the problems. He then pulled out a paper from his desk and explained what was going on. Even during the interview, I had some suggestions, which he got excited by. A few days later I was called in for a second interview with one of the “Group Presidents”. The one pressing question that was concerning him was “What if we have a major failure on a Friday night or Saturday, and you can’t be reached?” — To which I looked him in the eye and said “If I am the only person in my organization that can fix the problem, then I’m not doing my job.” He loved that answer, and I got the job offer.

By the way, although I was impressed by the “vice president” title, this was a very large company, and they had about a dozen VPs!

When I came on board, I found out that the Unix servers were repeatedly crashing for all sorts of reasons, disrupting work for clients, and severely affecting data delivery times. (IRI processed scan data from retailer sales, both providing analysis and raw data back to the product manufacturers and distributors. Getting the data in a very timely manner was critical to keeping up to or ahead of competition in the retail space.)

In my third week, while I was still getting to know people and evaluating the situation technically, I was asked to participate in a video conference with Proctor and Gamble. In those days a video conference was a big deal — required a special room with thousands of dollars of equipment and a dedicated high speed line. What I discovered was that P&G was IRI’s biggest customer (millions per year), and they had almost 1000 people whose job it was to analyze the data we supplied. If we were late by even a day, that was 1000 people being paid to sit on their hands!

Although they understood that I was new, they pressed for a fix to their problems. Without really knowing yet what was going on, I promised them a short-term fix within two weeks, and a longer-term fix within 3 months. I basically went out on a limb, just trusting my technical instincts, knowing that these systems should be stable.

Long story short, I found out that the people managing the servers had not applied system software patches for over two years — because, in their words, they were “too busy fighting fires”. Well, those fires were mostly due to the missing software patches! I brought in HP themselves to quickly bring the systems up to the latest patch levels (dozens of software patches were missing), and in a week we suddenly had almost no outages!

I made quite an impression in the first year, solving many similar issues. Some problems were much more complex, but I was not only technically proficient, I was very good at bridging the gap between the technical staff and the business staff, so I was able to get both sides to understand each other better and work out difficulties.

I’ve posted a few interesting emails from the first year. One of the major products at the time was called “Timeshare”, and was basically a portal into our servers that our clients could use to access our analysis programs. The actual packages of analyzed data were called “InfoViews”. Some of these terms may appear in the posted emails.

I will create a couple of separate posts covering specific episodes of my tenure at IRI.

My Timeshare Letter Download

The above letter was written immediately after I returned to my office after a company meeting. It was to my staff but of course I copied senior management.

The snippets below are some of the replies I got to that letter, which management forwarded to many departments, and to the events of the meeting in general.

My shlichos: IRI Team Building retreat, 1996

In 1996 IRI decided that to help improve morale they would hire a team building company to help with interpersonal and interdepartmental relationships. Anyone with any management responsibility had to attend their training sessions, which I found somewhat common sense and, at least for me, a waste of time.

After these training sessions, the company (Malandro Consulting) would schedule a 4-day retreat with all the managers. Working with IRI, they identified a nice lodge in southern Wisconsin, about an hour drive from Chicago. Of course, being one of the company VPs, I was expected to attend.

The problem was the date: It was during chol-ha’moed Sukkos, a time when I would only eat or drink in a sukkah! So I politely told management that I would not be able to attend. They were upset because I was such a key player. They asked me all sorts of questions to try to understand. I explained the concept of the sukkah. They asked how I was managing to eat while at work, and I explained that there was a kosher restaurant less than a mile away that had a sukkah, and I went there for lunch.

A few days later, they came by to say that they had moved the entire event to a small hotel in the area — so that I could go to that restaurant for my meals! I was astounded that they would change the plans for about 50 people, and from a beautiful resort to a cheap hotel, just for me!

One of their “rules” was that everyone stay for the full 4 days. This, I explained was also a problem because the restaurant was closed at night and early morning, and I did not even drink outside of a sukkah. I told them that if I was going to attend, I had to go home at night and return the next morning. Certain that this would be the final straw, I was surprised when they agreed — I would be the only one allowed to go home each night!

This turned into an interesting lesson in Judaism for a lot of people in that group, many of whom had never heard of Sukkos.

On the second afternoon, in a group session, we went around the room with the instruction to share something of a personal nature — I guess a way to make everyone seem less of just a business colleague. I have no recollection what I shared, but there was a young woman there who shared that she had a major falling out with her mother and they hadn’t even talked in over five years. Different people then responded in turn, some trying to offer suggestions or just emotional support. When my turn came, I explained the Jewish concept of honoring one’s parents, and that my advice was that she should push past whatever negative feelings she had and call her mother! This was a radical answer to most in the room — that she should be the one to break the barrier.

The next morning there was another group session. This same woman asked if she could say something. She said that she thought hard about what I said, and she actually called her mother! In that single lengthy phone call, they managed to break down a lot of barriers and begin to mend fences. She thanked me profusely for giving her my advice. The room was stunned. After that, I felt like this was my entire purpose for being there — why the retreat was moved to accommodate my Sukkos observance.

I.T. Manager: First in Chicago to adopt Blackberry pagers, 1997

As a technology manager, one of the most critical needs is for easy and accurate communications. Cellphones were not yet in general use, and until the late 1990’s, every IT group carried pagers. Those pagers simply beeped to tell you that someone needed a call back, and displayed the callback number; then you had to get to a wired phone to call in.

In 1997 a Canadian company called Research In Motion (RIM), and also an American company, Motorola, each came out with an interactive pager with a keyboard. Being always interested in new technologies, and in a position that allowed me flexibility within my own budget, I immediately requested samples of each one to test them.

What I found, was that the RIM pager was far and away the better device. It was clearly designed by people who actually needed such devices and used them personally. Just as one example: When the Motorola pager received a message, you took it off its holster, flipped open a cover, and pressed a button to open your message. When the RIM pager received a message, you took it off its holster and looked at the screen — a tiny magnet in the holster signaled to the pager that it had been removed, and triggered the message to be displayed. This small difference was huge in situations like board meetings, where it was possible to discreetly slip out the pager and glance down at the message.

There were many other advantages, which I detailed in a comparison chart, feature by feature. When I showed my little chart to the RIM salesperson, he practically begged me for a copy, saying it was better than the sales materials the company had provided!

In any case, having done my evaluation, I ordered these RIM pagers for everyone on my team. Although the company was RIM, these devices were known by what I presume had been a development code name and then became identified as this device — the Blackberry pager. [Many years later, RIM actually re-branded their company as Blackberry.]

These devices could run numerous “apps”, including email, note-taking, instant messaging, etc. The instant messaging was quite unique, and was the first instance of BBM (BlackBerry Messenger). It pioneered many of the features that we today associate with Whatsapp and the like — a notification that you message was sent, a further notification that your message arrived on the recipient’s pager, and yet a further notification that the recipient had read the message. Seeing that little “R” show up was very comforting, knowing that your message had been seen!

A few months after we started using these Blackberry pagers, I was on vacation in Florida, on a beach with the family. My pager alerted me, and it was a critical situation that I had to quickly mobilize several team members and tell them exactly how to proceed. A few months prior and I would have had to leave the beach and find a phone. But I was able to use BBM to quickly reach the needed people and explain what they had to do. The “R” notification gave me the assurance that they got the messages (no possibility of someone saying they never got the message!) and they were able to chat with me in real time with any questions. The situation was resolved quickly without me having to disrupt the family beach time! I was very impressed; it was clear to be that Blackberry was going to be a corporate fixture!

One of my original Blackberry pagers, turned on in about 2015, with some old 2003 email messages still in the inbox!

I.T. Manager: IRI Y2K project, 1999-2000

Anyone who had anything to do with computers towards the end of 1999 will remember the industry panic over the “Y2K problem.”

For those who may not, here’s the situation in a nutshell: Since commercial computer programs were created from the 1960’s on, programmers needed a way to signal to the program that processing was complete; that there was no additional data to be processed. Back then, the assumption was that those programs would certainly be replaced within a decade, so it became a convention to use the year “00” to signify “the end”. No one ever thought about more than a 2-digit year until the 90’s.

Suddenly, in the 1990’s it became clear that there were tens of thousands old computer programs still running everything from payrolls, to elevators, to air traffic control… and many of them were programmed to stop working if they encountered “00” as the 2-digit year!

Lots of experts sounded the alarm that every company worldwide would have to embark on a huge “Y2K (year 2000) project” — to test, certify, and possibly re-program systems to ensure that they would keep working when the date flipped to 01/01/2000.

Like companies everywhere we at IRI also had to re-certify every piece of software that we used. This was a bigger effort than it sounds like. We had to find times off-hours when systems were less busy, to simulate the change of date and test each program’s operation. Code that failed had to be examined re-coded, and re-tested. I had a team devoted to this testing. Luckily in our case, most of our software was not so old as to have used that old “end processing” convention, so there were few failures. Still, everyone held their breath when 01/01/2000 actually arrived. I had team members out at the data center at midnight just in case something slipped though testing.

After the successful date transition, I presented a small gift to each of my Y2K team members. It was the latest gizmo: a digital music player. No, the iPod didn’t exist yet for another 10 years! This was a unit called the Diamond by Rio. Although the appearance inspired the Apple iPod, the “wheel” was just a series of buttons.

This device came out in 1998, sold for about $200 with 32 MB memory for about 30 minutes of music! But wait! For another $200 you could pop in a whopping 64MB card and then hold a full 90 minutes of music. The team loved them!

I.T. Manager: IRI, the EDT project, circa 1997-2001

One of the difficulties of IRI’s data delivery to its customers was that many of the monthly files were very large. In those days most electronic transfers were slow and often unreliable, especially with very large files. So most of our deliveries were on tape cartridges, send to our customers.

Although this had been working for some years, it was fraught with problems. The tape creation process was finicky, and often had to be restarted. Sometimes delivered tapes couldn’t be read by the customer and we had to create a new set, resulting in an additional 2-day delivery delay.

In particular, we have very large monthly deliveries to our offices overseas — the U.K., Italy, France, etc. These had become especially problematic. No one understood why, but most of the time one or more tapes in a delivery set couldn’t be read. They suspected airport x-ray screenings, so they actually started having someone hand-carry them to Europe! However this didn’t solve the problem.

Much effort had been spent on the mainframe side (where the tapes were generated), yet they could not determine the cause of the problems. So they next tried using FTP to electronically move the files. However, in those days, FTP had file size restrictions, and it was also unreliable and non-recoverable — meaning that if the transfer failed 2 hours into a 3 hour transfer, it had to be started over from the beginning.

I was approached to see if I could help. After some thought, I proposed a software solution that I would engineer and program with the help of one of my staff. There was some skepticism (especially from the mainframe folks), but I was given the green light to try.

I had three main objectives: (1) Transfer any size file 100% reliably, (2) maintain security (data lines were not very secure, and much of the data was considered proprietary), and (3) engineer a solution that our client IT departments would be willing to install on their systems.

The third of those requirements steered me towards developing the entire system using only Unix scripting rather than any compiled language, and in a way that did not require special system privileges. As an IT manager, I knew that would be the only way to overcome any client objections.

I designed a system based loosely on package deliveries by the likes of FedEx. I defined each delivery as a package, with an accompanying manifest. Much like when you ship a lot of goods, you divide your shipment into smaller packages, my large files would be split into “chunks”, each of which would be sent separately, re-assembled at the receiving end based on the manifest. For reliability, if any “chunk” failed to deliver (verified by a mathematical checksum), the system would proceed with the other chunks and only later re-deliver those chunks that got missed. In that way, an unreliable line would never force a restart, simply require one or two small chunks of the delivery to be resent.

Obviously there was a lot more to this, but that’s the nutshell version. The EDT system (as I called it) became quite a success story. Clients were thrilled because it cut a day or more off their delivery times; the European offices saw a 4-5 day reduction. Even more importantly, once in place there were zero errors!

I’ve attached a number of email images, and a full slide show that I presented live to Frito-Lay in Plano, Texas.

EDTSlideShow CLICK OR SCROLL TO SEE ALL PAGES

A partial client list of companies that I got onto the EDT platform

The following are clips from emails ( bragging rights 🙂 )

Consultant: IRI lead-up to exit, 2001-2002

After 911 (Sept 2001) many businesses suffered as the economy sank. IRI was already in some business trouble versus its competitors before that, but this made it much worse. Everyone could see that we were in for some significant layoff, although we didn’t know exactly when or how bad it might be.

In mid-2001 I was “offered” a different role in IRI — still a VP, but as an “Internal technical consultant.” There were two reasons for this — 1) The poor business results caused the board of directors to toss out some senior management (including the brilliant founder, Gian Fulgoni) and replace them with their old cronies. To some of these stodgy managers, I was a “threat” because I was outspoken and known to be very innovative. 2) Although never confirmed, I believe that they were preparing for the layoffs that they knew would be coming, and it would be easier to retain a lower-paid younger worker in charge of the Unix groups, and would be easier to “justify” laying off an older manager in a “consultant” role.

Be that as it may, I accepted the change, because it actually allowed me to do more of the sorts of things I was interested in. I spent more time expanding and enhancing the EDT product, and also took a lead role in charting a course for replacing IRI’s “Load balancer” — software to move batch jobs around to various servers to make best use of the total computing power. We had a legacy load balancer that was no longer supported and did a poor job. I was charged with finding a new product and working out the migration path. After sourcing the best product, I worked with the software vendor to make a number of changes that I was able to demonstrate would vastly improve their product, changes that they actually implemented.

Small anecdote: When we were still about two months away from making the load balancer move, we suddenly had a serious failure of the old existing product, and it was significantly hurting the business. Since a young guy was now in charge of the Unix systems, I left it to him to solve the problem. He poked around a bit and declared that it was not able to be fixed, and that we needed to move to the new system. Senior management asked me if we could switch, and I told them we were not ready to make that move yet, still waiting on some of the software vendor’s changes and full testing. So there was a meeting with the group president (the same one who did my second interview and hired me). At the meeting, when it was strongly stated that the current system couldn’t be fixed, he looked at me and asked me directly if I agreed with that opinion. I said I did not; it seemed to me that whatever the problem was could be discovered and corrected, allowing us the time to get the new system ready. My answer infuriated the new guys, but the president simply asked me “How long do you think it would take to get it fixed?” Not knowing anything about the problems, I replied “48 hours.” I remember there were chuckles and a few gasps — they had already been dealing with it for over a week. I simply trusted my diagnostic skills and felt that I could do it. They gave me the green light to take over, and I locked myself away in my office and started examining the system, logs, files… anything that might give me a clue. I got to a point where my intuition told me that there must be corruption in a certain key file, and further, that if I removed it, the software should re-build it. I had no proof of this, just a strong technical intuition. So I removed the file and fired up the system — and we were back in business. Total time: about 12 hours, much to the astonishment of many (and the chagrin of those who said it couldn’t be done.)

By January 2022 the company did its first massive round of layoffs — between 150-200 people, many middle managers. I was one of them. Over the next year, they laid off many more, and outsourced all the computer work to India. They then took the company private (had been traded), and within another year they were a shadow of what they had been. When I joined IRI they had about 55% of their market; that steadily dwindled staring in 2001, and today they are less that 15%.

I’ve attached a couple of documents just as an example of the complexity of the projects I was leading, and maybe to show off my communications skills (I was known at IRI to be excellent in both written and verbal presentation skills. Several times I addressed groups of 20-50 people to present and explain a new technology project… the teacher in me I guess.)

Load Balancer Replacement – Product Selection CLICK OR SCROLL TO SEE ALL PAGES

LSF Discrete job flow steps slideshow CLICK OR SCROLL TO SEE ALL PAGES

I.T. Manager: Jewish Educational Toys, Chicago, circa 2004-2017

A friend of mine in Chicago owned a small manufacturing and wholesale business that was one of the first Jewish Toy companies. It was a very small business: He ran the Chicago office (2-3 people) responsible for product creation, art, sourcing manufacturing from China, and sales. His twin brother ran the warehouse in Pittsburgh (1-2 people) responsible for receiving and warehousing product, and packing and shipping all orders.

It was actually quite remarkable that they manages to build this business basically just the two of them, with a few other mainly part-time or seasonal employees.

Knowing that I had been out of work for a while, they asked if I’d be interested in helping them in the office during the upcoming busy Chanukah season. Figuring that I had nothing better to do at the time, I agreed.

As a former IT manager, and someone who had been involved with computer systems for many years, I was appalled at what I found — they were using an inventory and sales system that looked to me like it was from 1980! Let me note a few examples:

** Each person’s desktop computer had a COPY of the company database. They were not interconnected or synchronized in any way. Ever morning they would get a copy of the database MAILED to them from Pittsburgh on a CD, and they would each have to update their local database. At best that put them a day or two out-of-date. If one person made a sale, none of the others could see it until the next synchronization!

** Order fulfillment in Pittsburgh (remember these were wholesale orders only) was done by printing out the orders, the using them to pick the products and package them. Once the order was printed, the only way to check status or progress was to find that piece of paper in a huge stack. If a wholesale customer called in for status, or to make a change, we would have to call Pittsburgh, and they would have to rummage through the pile to find the order. Something as simple as getting a delivery date would typically take two days!

In order to run the business, Chicago and Pittsburgh had to be constantly on the phone!I was amazed that they were actually able to keep the business going in such an inefficient manner.

So I quickly embarked on a search for a good inventory and sales fulfillment software package. This wasn’t something that they had even considered — having been convinced (duped) by a sales person at a trade show some years earlier that their system was the best out there for wholesale.

After some investigation I settled on a package called Acctivate (by Alterity). Not only was it very comprehensive, it seamlessly integrated with Quickbooks, which is what they were already using for accounting.

It did take some convincing to get them to agree to purchase both a new server, and this not-cheap software. They were afraid of the change, and neither brother was very technical (to say the least). But I did manage to convince them, and worked quickly to adapt Acctivate to their particular manufacturing and wholesale business needs.

After the slight learning curve, they began to see huge advantages. Tasks that used to take 2-3 days were now 2-3 minutes. Everyone had instant access to all the data in real time, and became much more productive. It was funny — after some months using the new system, they were afraid that their business was failing because they used to have 4 phone lines ringing off the hook and now it was mostly quiet. But the reality was sales and inquiries were being handled so quickly that they were doing more business with much less “busy” work.

So… what began as a simple “come help us do some phone sales during the busy season” progressed into a role as IT manager, as well as sales associate. Normally this would have been very boring to me, but I found ways to make it interesting — like creating valuable reports in Acctivate to help them understand and grow their business, building them a functional wholesales web site and creating the processes to keep it synchronized with their back-end inventory system, and eventually convincing them to abandon their error-prone manual catalog creation in favor of a catalog that I programmed as a report. The catalog creation time went from a couple of months to a few minutes, with all data, images, etc. coming directly from the inventory system, ensuring 100% accuracy. For the first time they had a catalog, website, and back end inventor4y system that were all precisely in sync.

I stayed at that job probably way too long, being that I was getting paid much less than ever before, but I combined it with many of the other activities that are listed in this blog — my photography, teaching in the girls’ high school, and various online business attempts. I take pride in the fact that rather than letting this simple job get me down, I constantly found ways to create interesting and useful projects.

I actually found on the Acctivate web site, an excerpt from a product testimonial that I gave them at some point (although the wording doesn’t sound like what I would have said — someone wrote it after phone interview)

A couple of sample pages from the automatically-generated catalog…

A few snapshots of the JET web site: