May Issue of the Taos Newsletter: Linux

Interview with Ted Sergott,
CIO and Vice President of Engineering, PaymentOne

Taos: Tell us a little about your company:

Ted: PaymentOne is a payment services provider. We provide payment processing for internet goods and are uniquely able to bring broadband providers and content partner providers together. Rather than having to integrate into numerous different broadband providers, content providers can just integrate into one company - Payment One. Conversely, we offer the same thing to the broadband providers. So, instead of having to integrate into 10, 20, or more content providers, they just integrate to us and we handle the payment processing for them across all the content providers. We are not the same as Pay Pal where a person is buying physical goods such as a car or computer from someone over the internet. We do the payment processing primarily in cases where the “good” is the internet itself. For example, music downloads, movie downloads, publications, web hosting, etc.

Taos: How have you deployed Linux in your environment so far?

Ted: We use Linux in three different areas - on the web servers, the application servers, and the utility servers, like DNS and email.

Taos: So what requirements drove the adoption of Linux in your environment and why is that so important to you?

Ted: There are several factors. First, we are a small company, so definitely cost and value. Other factors included scalability and the need for a good, solid product that was secure.

Taos: Do you have future plans for further deployment of Linux in your environment?

Ted: Yes. In contrast to the web server level, where we primarily use Linux, at the apps server level, we also use Solaris and Microsoft 2000. Over time, we will be increasing the use of Linux in the application server tier and it will be an ongoing process.

Taos: Is there anything preventing you from doing that now or is it just a matter of evolution?

Ted: At the application server tier, a lot of current packages don’t run on Linux. Also, there is a fair amount of technology that we acquired and now own that wasn’t built to use Linux. So, in those cases, we would have to migrate off those apps and technologies to make the move to Linux and there is really no reason to do that, at least for the short term. A second reason is that, for some applications, Linux may not be the safest bet and we feel more comfortable with Sun Solaris rather than Linux. It comes down to what provides the best value when making trade-offs.

Taos: So, in other words, in some mission-critical applications, Linux may not be proven well enough to adopt right now?

Ted: Correct. It’s mostly at the Database server tier where I would absolutely avoid Linux if I had to build a high-quality, mission-critical system – one that dealt with financial data, and that had thousands of reps and millions of consumers logging onto it. It’s just not yet well-proven under those circumstances. Of course, as more companies use it running in production, and uncover bugs and issues, it’s getting there. But right now, in that environment, I would not do it.

On the other hand, there are other situations where I would use it. For example, if I were at a start-up company building something from scratch, I absolutely would use it at the database server tier. In fact, we are using it at that level, but only in the areas of data reporting and data warehousing - not in hardcore mission-critical production systems.

Taos: Are there other aspects of the market where you feel Linux is or is not ready yet?

Ted: At the desktop, it is used extensively in engineering environments. But in business environments - where business people need to send and receive documents from Microsoft desktop applications to and from Linux applications - it’s probably not the best choice.

However, I do see it being experimented with everywhere. For example, in a lot of huge database system environments where most people are still using Sun Solaris. Every single one of those companies is running Linux in a test environment. And, at some point, it will be in production. If you had talked to people two years ago, there were very few, if any of them trying Linux. Now, there are hundreds and that will grow to thousands. Then, it will reach critical mass and it will be ready for prime time.

Taos: And with your crystal ball, what do you think the time horizon is for that?

Ted: I would say that it’s probably down to 18 months to two years for the server side of things. It’s getting that close.

Taos: And what about on the desktop side?

Ted: From my perspective, it appears that Microsoft has a strangle hold on that market and they can make it really difficult for their products to work with Linux if they choose to do that, so the desktop side may take longer. On the other hand, a lot of tech people at a lot of companies are getting pressure from their CFO’s to reduce costs and that can be done with Linux. Of course, there will be some pain associated with that so it’s really hard to predict. My crystal ball is less clear to me on the desktop side.

Taos: Are you using Linux on your desktops in the engineer environment here?

Ted: Yes, and my software engineers have to have both. However, no one outside of engineering has Linux on the desktop.

Taos: What feedback are you getting from them? Do they like it?

Ted: Yes, of course. Every engineer is a big advocate of Linux and I have yet to find one who’s not! Everyone knows it, is learning it, will know it… It’s definitely hitting critical mass with engineering. And, if you’re a software engineer, you have to know Linux at this point.

Taos: When you began deploying Linux in your environment, did you start from scratch or did you migrate from something else?

Ted: We migrated from Microsoft. I have been here two years and pretty much inherited a 98% to 100% Microsoft O/S shop. However, in engineering and in QA, there were a lot of applications or “sort-of-built” applications in Linux , but nothing was deployed. When I got here, we quickly deployed those with very successfully results - more secure and with less patches than Microsoft. We saw immediate performance improvements in those applications. Now, to be fair, we upgraded the hardware as well. We went from old hardware running the Microsoft O/S to new hardware running Linux - Dell Linux. The results were phenomenal. We had some cases of 80% reduction in bugs.

Taos: What about the skills of your system administrators - did they needing retraining?

Ted: Initially, we had software engineers with that skill, but we didn’t really have anyone on the Sys Admin side with experience or who was trained on Linux. So, yes there was a learning curve. But, we rolled it out slowly and sent them to training classes. We also hired a couple of senior administrators in the last six months who have Linux experience. Because Linux is widely deployed at the web server level and at the app server level, there are a lot of experienced people out there - maybe ten years of Unix and two years of Linux. Sys Admins with those skills are much easier to find now than two or three years ago.

Taos: How did you approach the cost/benefit analysis of deploying Linux - taking your total environment into consideration and all the various associated costs to manage it and run it, including skills sets?

Ted: The skill set consideration is very important. If the team doesn’t know a technology, then it could be very disruptive and dangerous to move very quickly off or onto something. Also, we are a small company, so cost is always a factor. We look at everything very, very closely. However, more than all of that, the most important questions are “Will the system we build work? Will it be high quality, reliable, and easy to maintain? Will it meet the business need?” With that in mind, for the size of our company, Microsoft is a good solution; Solaris, frankly, is a little bit expensive; and Linux is the ideal choice.

Taos: When you went through the process of asking “Will it work? Will it meet our business needs?”, did you test it in a QA environment first and then move it to production?

Ted: Yes, that’s correct. We have a very good QA department and did a thorough test in QA. Then, we ran some Linux web servers in parallel with Microsoft web servers at the same time and, when Linux was fully proved in production, we just removed the Microsoft ones.

Taos: When you ran the two environments, did you find performance gains when you compared Linux versus the others?

Ted: Yes - a lot of performance gains. I would say at least 50%. Now, again, that was on newer hardware too, to be fair.

Taos: What Linux distributions are you using in your company?

Ted: We’re a Red Hat shop and we have about 130 servers with roughly 30% of those on Linux.

Taos: Earlier you mentioned an 80% reduction in bugs, and just now a 50% gain in performance. Were there other gains as well?

Ted: Yes, in the area of security & software patches. Regarding worms, bugs, and security issues, we see far fewer with Linux. Also, we install far fewer patches for Linux than Microsoft. Again, we don’t use Linux on a desktop, just at the server level.

Taos: Have you had any support issues with the distributions you’re using?

Ted: No. But, it’s different than if you have production problems or production issues. We have a pretty solid engineering team and, yes, we’ve had production issues but we have resolved those on our own. I couldn’t point to any as being Linux specific.

Taos: When you presented at our Experts in the Industry series, you had a slide that talked about the total cost of ownership of Linux versus your Microsoft and Solaris equipment. What were those numbers again?

Ted: Linux is free – that one’s easy to remember! (laughs) Microsoft is roughly $800 - 900 per server and Solaris is more like $1100 or more. Actually with Microsoft, you have to pay every time you call them.

Taos: Do you foresee sometime in the future when you will pay for support or licensing for your Linux environment?

Ted: I would say yes, but I don’t think that’s in the near future because the size of our company and where we are. We haven’t paid for it yet and we are fine. Maybe if we grew five times the size, brought in a lot of new people, and if I and my managers felt we were losing our internal expertise, total control, over a lot of Linux deployments, then it may be a good thing from a Quality Assurance perspective to pay for it. I can see why a CIO in a large shop would do that, but the issue is if it comes anywhere close to the cost of Solaris, I would still use Solaris.

Taos: Why is that?

Ted: Just because Solaris has been around for so long and it’s proven. I personally built mission critical financial systems on Solaris and I have yet to see anyone to do that with Linux at the database server level. And, of course, I have done that on mainframes which are far larger than Unix machines. I don’t know anyone doing Linux on the mainframe yet either. So to me, if cost was not an issue, if proprietary hardware was not an issue, then I would use Solaris. It’s still a faster, higher performing operating system than Linux right now. I firmly believe that.

Taos: To close, what advice would you give others regarding Linux?

Ted: Today, staff at most companies are looking into Linux, whether they’re in the Microsoft camp or the Java camp - either side - everyone is looking at Linux. It’s definitely here - it’s accepted, it’s a good product , and it’s only going to be more widely used. I would say you should be using Linux, and if you’re not, you really need to ask yourself and your engineers why not. At a minimum, you need to be experimenting with it because of the value factor alone – examining cost and reliability verses quality, etc. It’s really becoming “best of breed” in the web server area and the utility server area. In my opinion, it works better than Microsoft, so you should ask yourself “Why pay a $1000 extra per server?” It just makes no sense.

Ted Sergott has more than 20 years of technical and management experience including extensive experience in the design, development and support of high-volume, web-enabled, mission-critical financial systems. As the CIO of PaymentOne, Ted manages computer operations, network engineering, software engineering, quality assurance, and project management for the company. PaymentOne offers complete outsourcing solutions for internet commerce, payment processing, and risk management.

Prior to PaymentOne, Ted was a Managing Director at Charles Schwab, where he was responsible for the development and operation of several successful financial systems; he managed the entire bond trading platform, built its IPO system, and was instrumental in developing their CRM system. Prior to Charles Schwab, Ted held technical leadership positions at Wells Fargo, Bank of America NT&SA, the Clorox Company, and Long's Drug Stores. Significant contributions at these companies included the development and support of ACH, credit card, retail banking, EDI, and point-of-sale systems.

Ted holds a Bachelor of Science in Computer Sciences from Madonna University in Livonia, Michigan, and an MBA from St. Mary's College in Moraga, California. Ted and his wife Sharon live in Moraga with their four children.