Clint Sharp’s Blog an’ Vlog

I just got back from spending three days at Cisco’s corporate headerquarters.  I’ve spent a good portion of my career dealing with vendors.  I’ve spent days in meetings, evaluating products, specing out hardware, leased lines, and other ancillary pieces for building out networks and datacenters, but rarely have I left a set of meetings so jazzed about the future.

The first day was spent in design meetings and demo sessions.  We opened the day by outlining where we’re currently at with our network.  Currently, it’s a fairly well designed network with quite a few well known warts, but for the most part it functions very well.  We’re not really hitting any capacity issues anywhere, but for the first time we’re really trying to plan out 2 to 5 years in advance.  We know our current backbone, which is OC12 at the core nationwide, won’t last us for probably more than the next 12 to 18 months.  We know this, because every time we improve our network, we find new ways to utilize it and we blow away all our traffic projections.  However, we know that while our current design is working really well, we don’t want to upgrade 10 core locations to OC48 since half the sites sitting on that core won’t need that kind of connectivity.  So we spent the first part of the morning drawing up a new core architecture with fewer sites and ideas for making it more highly available, including using DPR and/or MPLS fast reroute to provide sub-50 millisecond or sub-100 millisecond failover time in the event of a network failure.  I think we have a promising set of ideas to go into our meetings next month with Level 3, who we purchase the majority of our core connectivity from.

The second half of the day was spent in demos.  We saw some interesting new products on the Unified Communications front, as well a demo of Telepresence.  Cisco makes this point often, but it can’t be stressed enough.  Telepresence cannot be compared to video conferencing.  We have an existing video conference system, which Telepresence has inspired me to improve, but it’s clunky and awkward by comparison.  Telepresence really makes you feel like you’re sitting across the room from the other participants.  Our current video conference system is used maybe once a month nationwide, but I guarantee Telepresence would be booked 100% in our company.  I’d like to try to make our existing video conference system more like Telepresence, because a lot of the ideas Telepresence puts forward in terms of controlling the environment, using high quality video, and directional audio for the partipants can be adapted to our existing system.  It won’t be Telepresence, but it certainly would improve what we currently have.

We finished up the day with a demo of products Cisco is pushing for retail and a discussion about their Service Control Engine product.  The retail demo inspired a discussion about using EVDO as a backhaul for our retail stores, which is so obvious as to be a real slap in the face that we’re not already testing it out.  We already own that last mile connectivity, why are we continuing to pay our competitors for connectivity to our retail outlets?  Also, the Service Control Engine discussion was fascinating.  We went from discussing a product we didn’t even know we owned to talking about all the things it could do for us.  If you hear about providers shaping traffic down for Peer to Peer traffic and dunning customers for high bandwidth utilization, this is the product that’s doing it.  It can shape traffic at wire speeds including multiple gigabit connections.  Really really impressive.

The next two days were spent in Cisco’s Customer Proof of Concept lab.  We went there to prove that the 7600 platform which we’re currently basing our core network architecture on will scale to meet our demands over the next 2 to 5 years.  As I expected it did.  I also had an excellent discussion with a Cisco executive over in the Unified Communications area about how we can leverage our investment in our IP Telephony infrastructure to really improve communications and productivity.

Cisco really went out of their way to get us access to people who would could answer our questions, give us good ideas for the future and give us access to equipment to test our their claims for our own assurance.  They’re really the gold standard that I’m holding everyone else up to in terms of how they should treat their customers.  We could have spent the same, dollar for dollar, with many other vendors and barely gotten a sales engineer to give us a call back, but Cisco seems to understand from the top down that people sell products, the products don’t sell themselves.  They understand that we’re looking for a vendor to partner with us to help us get to our next level.  Most often in this business there are two or more products that will do any given job, and while technically one might be superior, in the end both would work for a given task.  The decision comes down to more than what exactly the product does, and while having a superior product helps, having a superior organization that stands behind that product and offers it at competitive prices is in my book the best way to get your product into customers hands.  Cisco has a product for nearly anything you need in the networking world, and while we won’t ever be exclusively a Cisco shop, they’re much more likely to sell us a product on a given day due to the fact that they spend so much time focusing on helping their customers implement their solutions and spend so much time focusing on helping their customers support their existing installations.  At the end of the day, the most important question I have to ask when making a purchasing decision usually isn’t what is the best product, it’s which is the best company.  Right now, Cisco’s going a long way to convincing me they are.

Over on Bram Cohen’s blog (of BitTorrent fame), made a comment that reminded me of a pretty common misconception about the ISP business. Mark said, in reference to Bram and attempting to make money from BitTorrent:

Unfortunately for you,ISPs crack down on heavy bandwidth users, particularly uploaders and enforce their TOS.

By definition, seeders create upstream bandwidth. The ISPs dont want to see more upstream usage Bram, i know its a tough concept for you, but in the mind of the ISP, upstream use = bad. MOre upstream b/w use = more bad. Which in turn pushes them not to increase the bandwidth available to end users, but to evaluate where the upstream use is coming from and look at shutting it off and throttling it. Call me crazy, but that equates to a challenge for the BT universe.

This couldn’t be further from the truth. The upstream bandwidth is not a concern for most ISPs, especially your standard Cable or DSL provider. Broadband ISPs have to order bandwidth synchronously, in various speed increments, because all high capacity circuits come in synchronous form. This means if you order a Gigabit Ethernet connection to an upstream provider, you’re actually buying 2 Gigabits per second worth of bandwidth, one from your provider and another towards your provider. Broadband ISPs sell bandwidth asynchoronously (the A in ADSL), which means they’re selling you something like 6 megabits down and 784k or less upstream. Obviously, it’s well known that ISPs oversubscribe their upstream links which is why peak times can see serious difficulties achieving the full downstream. However, no matter how you do the math of an ISP which serves mostly bandwidth consumers, this still leaves large amounts of bandwidth on the upstream on the ISPs egress connections. In fact, most Broadband ISPs don’t provision large WANs to haul traffic back to central egress points, because it makes more sense to dump the traffic off at a peering point in the market, assuming you’re in a large metro where you can cheap bandwidth. Because ISPs providing broadband to homes and businesses also have such a one way usage pattern, it means that they cannot negotiate peering arrangements with other ISPs for an even trade of bandwidth, meaning they’re always going to be paying for that bandwidth.

In the business, this is generally called Tier 2 bandwidth. Broadband ISPs will sell large hicap circuits to businesses largely interested in serving content much cheaper than you can buy Tier 1 bandwidth from a major internet provider (something like UUNet, etc). This also becomes very evident if you attempt to buy bandwidth from a Tier 1 provider and a Tier 2 or Tier 3 provider, because it becomes very difficult to load balance your traffic over to the Tier 2 providers because of the way BGP routing works. There are methods for overcoming this (AS padding), but it still never works out quite the way you want.

So, in summary, ISPs which are selling bandwidth to users which are using BitTorrent don’t give a rats ass about the upstream. They’ve got loads to spare. What they’re concerned with is the users who are using large amounts of upstream bandwidth are generally also heavy consumers of downstream, and those people are causing oversubscription issues.

Apparantly I haven’t updated this blog in 3 months. My how things change. So, since my last post, I’ve accepted a new job in Denver and my daughter is three months older (crawling, teething, sitting up on her own, etc).

I’m not exactly sure what causes me to not update the blog, but I suppose it’s just life in general. It’s hard to want to sit down and write a blog post when I’m so busy with work and the home life. Katie keeps me busy when I’m at home (because when I’m at home, usually I’ve been gone for some time and it’s my turn to make up for time lost in childcare). When I’m not at home, I’m on the road and I’m busy working.

Lets give the work update because I’m sure that’s mostly what I’ll be writing about in the near future. Almost two months ago it became apparant it was time for a change. I needed to be making more money, and it became obvious that my goals and plans for doing that in Fort Smith, Arkansas were for not. It’s unfortunate, and I’m sure there were many execution failures on my part that could have resulted in a more positive outcome for me continuing to live in Fort Smith, but the reality of the situation is that, in the end, it didn’t work out for the second time. It’s unfortunate, and I’m very sad about it. However, what’s done is done, and while I can reflect upon it and determine what I’ve done wrong, in the end it’s just not going to work out. So, near the end of August I started contacting old friends in the Wireless world, people I used to work at back when I worked at AT&T Wireless/Cingular. Obviously, the most influential of these people was Dave Truzinski, who is the current CIO of Leap Wireless, which you might know as Cricket Communications. A couple of days later Dave came back to me wondering if I’d be interested in a contracting position, and things have progressed from there.

Firstly, let me tell you a bit about Cricket. Cricket is a unique play in the wireless world. In terms of wireless companies out there, Cricket/Leap is very small. Annual revenues were around $1 billion for 2005. Cingular, for example, when I left was around a $60 billion unit of SBC/BellSouth (soon to be the combined AT&T). So in terms of POPs covered and subscribers, Leap is barely on the radar of most industry analysts (I’ve rarely seen Om Malik writing about them, but I think they probably deserve more coverage than they get. They just spent nearly a billion dollars in the AWS spectrum auction for example). So, as I was saying, Cricket is a unique play in the wireless world. Cricket focuses mainly on lower income subscribers, competing with MVNOs like Virgin Mobile, TracFone, etc. However, Cricket is unique in this market being that it owns its own network. Secondly, Cricket is very unique is that it offers all you can eat plans, of which I believe only Metro PCS also offers aside from Cricket.

So, I accepted a contract position with Cricket working on their new nationwide network rollout. We’ve rolled out a killer new nationwide WAN which should provide a good amount of growth room. We’re current doing and will be doing a lot of very interesting things over this grand new network. My job for this contract was basically to help get the ball rolling in finishing the rollout. I flew to various cities including San Diego, Houston, San Antonio, Nashville, Phoenix, Spokane, Tulsa, and probably some others I’ve forgotten over 4 weeks to help them get the facilities provisioned and ready to complete the rollout.

After the contract was nearing an end, I became involved with the IPTel project. I’m now currently working on rolling out Cisco IP Telephony system. I’ve done this before with AT&T Wireless, when we rolled out this same solution to the Caribbean. I came onto a project with numerous issues and a long way to go to completion, so it should be a lot of fun. There’s nothing I like more than to be a fireman, so a project in trouble is my idea of a good time. There’s lots of work to be done and lots of room to roll out some really cool telephony features, so I’m pretty happy at the moment. I’ve written in the past and continue to feel that there’s a lot of issues with larger companies (and while Cricket may be small for the Wireless world, a $1 billion annual company is still a large company in my book), there are certain aspects that make it to where I can fit in quite well. Unfortunately, smaller companies can’t offer the range of opportunities to work on cool projects and cool equipment that larger companies can. Hopefully Cricket is the right size and I can be happy there. Who knows, that remains to be seen, but I’m very excited about the prospects now.

I leave on Monday for San Diego again to work on another project in trouble for a week, then hopefully back to IPTel. I’m going to try to keep this blog updated on the progress of this project, what I’ve seen and experienced with Cisco IP Telephony implementations (especially my experience with it compared to things I really like about Asterisk), and what’s going in general with my business life. Hopefully if I can get some free time at home, I’ll also update with a video of what’s going on with my kid, because she’s really the light of my life, and she’s really at the perfect video age.

Best of luck to the few readers I have left, and I hope I can find time to keep you all updated in the near future.

The one thing I like about writing, and blogging in particular, is that it gives one a chance to air out their thoughts, in public, with the potential for feedback. However, one of the most important aspects that is easily overlooked, and this obviously applies to all journaling and not just blogging, is that it gives one the chance to take partially formed ideas and simply by the act of writing them down, flesh them out into fully formed thoughts. This is what I am going to attempt today. Buckle your seatbelts.

In case you’ve fallen behind in following my life (wouldn’t be hard, I don’t think I’ve talked about it much here in the last few months), I’m currently working for Kirkham Systems and Fireball Media. Kirkham Systems is an established IT services company servicing the Fort Smith, Arkansas metro area offering IT services to small businesses. It’s been very successful, although obviously no business is as successful as it’s proprietors would hope. Fireball Media is an advertising agency, brand spanking new, less than a couple months old, so we really have no expectations or data to suggest whether that one is performing well or not. Time will tell.

Thus far, Kirkham Systems is excelling in its sweet spot of providing services to companies with about 5 to 25 PCs. More than that, most people have decided to hire full time IT help, less than that people generally don’t have the budget to afford our services or prefer to let their kids best friend come in and help them out with those ‘puters. Kirkham Systems also does a significant amount of web design and custom software. It’s a very well rounded company, distinguishing itself not only with its customer service but also in its ability to offer a full suite of services from networking, to custom software, to web design.

However, we haven’t as yet been able to crack the nut that I’d really like to crack. Unfortunately with a company like Kirkham Systems, companies who hire a full time IT staff have usually done so because they feel they’re going to get the best service from having those services in house. If I owned my own business, I can’t see wanting to outsource a full time staff member to waiting in queue at some other company. I know from experience you’d probably get better service than the average person staffing a one person IT shop, but it’s a hard sell. We don’t sell hardware or software solutions, so we really don’t have a lot to offer companies with an IT staff. Like I said, it’s a tough nut to crack.

It’s very difficult to make a service company grow exponentially like a product company can. Take any recent product phenomena like the iPod and you can see what I mean. Rarely can you come up with a service based idea that will catch on like a product can. You can’t hold a service in your hand, you can’t start up its motor, you can’t look at after the salesman has come and gone. You grow a service company one customer at a time, and while you can, and we are, building other existing channels outside of direct sales and standard advertising based marketing, it’s very difficult to find a niche that is wide, has limited competition, and has the potential for growth that a product based company could have.

I’m still looking for the magic solution, but I’m afraid there isn’t one. I think this is largely why service based businesses tend to not be chains, or at least not large ones. Chains require profit margins and scalability that isn’t present in service based businesses. Larger prodcut companies, chains (restraunts, retail, etc) included, require being able to gain effeciencies with the size of the company such that the potential revenue for a given employee grows with the success of the product (or in the case of retail, the mix of products I suppose). Service based businesses, since their revenues are largely based on labor, are limited to revenue per employee based on some multiple of billable hours. Administrative overhead merely cuts into the profit generated by a revenue generating employee, thusly, to grow to the size required to maintain a large administrative staff and bureaucracy as is typical of larger companies becomes nearly inpossible. I believe Brad Feld has written about this. I don’t know what the exact size would be, I think it would depend on the company, but I think every service based business would hit a point where it becomes unprofitable to add more billable employees because the overhead required to manage those employees becomes too expensive.

I think perhaps I’m pondering a problem that I would have learned about in Business School. Perhaps I should go back to college :). I may be writing about a problem that’s been written about so extensively as to make my thoughts seem like a rehash of some famous business writer’s thesis. Maybe someone reading this can point me to it. Am I right or wrong? Am I missing the examples of people who have found the magic bullet and turned service based businesses into large enterprise?

I live this, every day:

“Design consists of creating things for clients who may not know what they want, until they see what you’ve done, then they know exactly what they want, but it’s not what you did.” - Brian Sooy

Mark Cuban has a great post about finding the right partner for your business and accepting your faults. While I don’t think you should ever stop trying to improve yourself, which in my case is attempting to be more organized, I do think he’s got a very valid point in knowing when to accept your faults and make sure you’re backing yourself up with a good partner. Thankfully, I think I’ve found a good one with Tom, so we’re covered there. The sky is the limit!

This came across an email list I’m on recently:

Ning co-founder Marc Andreessen recently said…

Ideally we’ll never meet any of our customers. We actually had to take the sign down from our front door because one of our customers actually stopped in, uninvited, and said, “Hi, I love your service.� And we’re like, “why are you here?� And so down came the sign.

Drop-bys like that should only happen in sitcoms as far as I’m concerned… The consumer internet businesses in a sense are ideal businesses from the standpoint of never meeting your customers.

Only in the technology business would anyone be caught dead uttering such an utterly stupid statement, and even then it doesn’t make it any less of a moronic comment. Your customers are your bread and butter. You should jump up and down if someone takes the time to stop by your office just to tell you how much they like you, and you should be just as excited if someone takes the time to tell you what you’re doing poorly, because it’s a chance to save a customer and make an advocate. This is something I’d expect to see on Rick Segal’s blog, in one of his infamous (at least to me) overheard dumb business conversations. I can’t honestly see anyone with this kind of attitude being successful in any business in the long run, technology or not.

A while back, I wrote about what I’m calling the Local Web. The Local Web, in my mind, is a group (an infinite number of groups are possible) which arrange their interconnectedness by sharing a geographical point of reference, traditionally Metropolitican Statistical Areas, or MSAs. The Local Web is already built in many of the larger cities, with directories and vertical search engines to allow you to search for stuff in major metropolitan areas, but a good percentage if not the majority of Americans live outside of a major metropolitan area. The connected netizens from those areas are being largely overlooked by current major initiatives to create localized web experiences.

I’m starting an experiment in a town that should be the perfect size. My hometown is Fort Smith, Arkansas, a town of about 80,000 with about a quarter million in the MSA. There are billions of dollars of business done every year here, and many companies here ship worldwide. However, for doing business in town, most people still reach for the phone book. The reason for this, of course, is because you can spend days Googling around for information about Fort Smith businesses without finding much but spam sites. No one in this town has made a concerted effort to make sure things are easily found on the web about businesses they’d like to do business with.

So, I’m starting an experiment. I’m going to organize a blogger meetup to start. I’ve already found several local bloggers and I’m going to find or create more. I’m going to organize them and attempt to get them to write about business and other activities (softball, church, whatever) they that they do locally and where they do them at. I’m going to try to incent people to create links from site to site across town and try to make information more easily indexable by the search engines so that when you search for something in the area you don’t end up at a spam site. We will be holding the meetings at Kirkham Systems of Fort Smith.

Once this is going strongly, I, along with the staff of Kirkham Systems are going to start showing the results to local businesses and convince them they should have a website with a blog and incent them to link to the people they’re doing business with and write about their experiences with it. The goal is to create an interconnected web of links focused on this geographical area, so that if you end up at Kirkham Systems website you’ll find annotated links about the people we do business with, and when you end up there you can find the people they do business with.

If I’m right, by the time I’m done, Google will be a far more interesting resource to find information about businesses, things and places in Fort Smith, Arkansas than any other resource, anywhere. This may seem boring to people who live on the coasts and can find a well designed and well organized website for even local businesses, but for the large portions of the country that have been ignored by businesses attempting to organize information for them on the web, I think this will be a large step forward. No one understands or cares about this because they haven’t been educated as to what it can mean for both their businesses, themselves and their community. My goal is to educate everyone here.

The Local Web is long overdue.

I’m totally going Rick Segal on your asses tonight.  If you’re not reading his blog and you’re in any way, shape, or form involved in business, you should be.

So, we have a name for name for FedEx Kinkos, Head Injuries R’Us.   Honestly, I’m not a big Kinkos customer, since I don’t really deal much in paper, but this week while we were down in the Bay Area, we had need for paper versions of several documents, so naturally we ended up at Kinkos.

I don’t know that I’ve ever felt like I needed to invest so much effort to pay someone.  We had need for some quick and dirty business cards for me which required printing color on cardstock.  This seems like a simple request.  I could do it on my inkjet printer at home with little to no effort, so thusly Kinkos should be able to accomplish it in a fraction of the time.  Well, long story short, not this Kinkos.  They had one color printer down, and the other one was completely backed up.  They told us "Come back tomorrow."  Who knows how many sales they lost.

To me it seems like a simple queueing issue.  I needed 1 sheet of cardstock printed.  Whatever else was in the queue then probably consisted of much larger batches with customers who were not in the store at the time.  The revenue from my purchase would have been insignificant, but what they’ve done by not training their employees with how to deal with outage situations is to guarantee they will not receive my business when it comes time to do a larger printing.  They could have simply printed my page and delayed the other jobs by a minute and I would have been none the wiser and neither, most likely, would the other customers they were printing for that day.

Later in the day, we are at yet another Kinkos, to print up color copies of a document and purchase two binders to place those documents in.  We waited for over 10 minutes to pay $6 for the binders.  We contemplated laying the cash on the counter and walking out, but we didn’t have the change.

The thing to learn from this experience if you’re running a business, is that every interaction with a customer is valuable.  Kinkos is a business which markets to people whose time is valuable.  I understand outages and downtime, but your employees must be trained to deal with them.  I could have walked out of the store a happy customer had a manager or employee been thinking on their feet about what to do with a 50% capacity loss.  Telling a customer to come back tomorrow isn’t good business and it’s likely that I won’t come back again, period.  Secondly, if you run a retail/service business, make sure the retail side always has at least one register dedicated to it, rather than tieing up 100% of your capacity to answer service related questions.  A customer shouldn’t have to wait 10 minutes to pay for something, otherwise she’s likely to not pay at all.

Finally catching up on weeks of RSS news I haven’t read, and I came across this on Seth Godin’s blog:

Appoint a CNO—chief no officer. No longer can someone say no to an idea and leave it at that. If you want to turn something down, you’ve got to pass it on to your boss. Then either he says yes or gives it to his boss. For a “no” to be official, it’s got to be approved by the chief no officer and countersigned by every manager along the way.

As he says, it’s not that simple, but it could be! This kind of an idea, this kind of attitude, could fix most problems I have with large companies. The culture of no, that I’ve spoken about to collegues multiple times, is a good part of what I have a problem with working for large companies. It’s far easier to do nothing than to do something, that after a while attempting to do the right thing becomes far more work than it’s worth, fighting against all the no answers between you and the top rungs. Businesses that maintain the status quo die, and the entire bureaucracy of the business is setup for people to protect their fiefdom’s and maintain the status quo. This is why you will find me working for small companies for the foreseeable future after this job is over.

I’m in Miami and searching for DAT72 (DDS-5) tapes. When Wayne bought this tape drive he picked up the wrong kind of tapes. So far, I’ve been unsuccessful in finding anyone who carries DAT72/DDS-5 tapes, but TigerDirect does have an outlet store down here. I’m doubtful they even carry the things, but I’ve now went through two phone calls of 20 minutes where they ended up hanging up on me and now I’m on my third call that I’ve been on hold for 30 minutes.

I don’t care how big of a retail outlet they are, there is absolutely no excuse for this kind of performance. Rick Segal writes about this kind of stuff all the time, and I thoroughly enjoy his insights. I’ve got a few of my own, and I’m going to start posting them here more regularly. If you’re a company that’s the size of TigerDirect (who does a booming catalog business), there’s no excuse to have outlet stores that are poor performers that will tarnish your brand. Their $7/hr an hour retail employees which obviously couldn’t care less about their jobs have ruined the TigerDirect brand for me, and I’m very unlikely to order from their catalog now. All the controls they’ve setup to make sure their catalog phone centers are for not if the retail stores can’t handle their phones properly.