Subscribe to the RSS feed.
The Ashley Madison hack: lessons beyond moralising or hysteria
The Ashley Madison hack has inflamed media attention to a frenzy. And of course it would. The site offered aspiring philanderers the opportunity to encounter like-minded people. The cheaters’ paradise turned out to be less of a niche-proposition than one might imagine, with an estimated 37 million users. A group of crackers took it upon themselves to break in and reveal the names and details of the privacy-coveting members of the service.
The fallout of the leak has been, of course, costly and embarrassing. The tale’s clash of ancient-morality-tale and 21st-century digital-paranoia makes it a tsunami of a story. On the one hand, it’s a veritable Elizabethan plot of conniving philanderers and cuckolds, and the devious but peculiarly puritanical blackmailers who would unmask them. Its themes could have flown from Christopher Marlowe’s quill. On the other hand, the story encapsulates a shiny Gibson-esque dystopian nightmare. It paints a picture of a world where no secret nor confidence can finally be protected from the marauding swarm of digital predators.
Much media attention has focussed on the dark depths of the human condition the story plumbs. Plenty of smart pieces have been written about Ashley Madison, but there have been disappointingly few attempts to shine some rational light on what systematic steps we can take to prevent such tragedies from happening. Whatever else the story reveals, it confirms that “tragedy” is not too strong a word to use. Online security has now become a matter of life-and-death. We always knew that about military secrets and the like. Now we are forced to acknowledge that there are no “trivial” exceptions. As in the aerospace industry, though, we need respond to disasters not with hand-wringing emotionalism, but with calm reflection on what we can put into place to limit the severity of such attacks, if not prevent them. So let us see what lessons might actually be learned from the Ashley Madison furore.
The primary lesson involves no specific techniques nor one-stop-solutions to thwart Black Hats. Instead, it involves ethos. One might think this a little namby-pamby compared with getting the latest shiny IDS (Intrusion Detection System) installed, or in trundling through a PCI Compliance (the credit-card industry’s gold-standard of audited security policy) document; however, rooting the culture of your project deeply in security’s soil makes a difference between digital Security Theatre, and actually having some chance of thwarting the bad guys. The distinction between performative “showbiz” security and the actual culture one needs to have in place to keep everyone safe is one that we at Positive have felt keenly from our very beginning. Security is not a “layer”. It’s not some protective faerie dust you sprinkle on your system. It’s not an amulet you hang at the doorway of your data-centre. Crucially, security cannot be treated as that annoying dweeb who’s there to buzzkill your startup’s launch party. If you have “prioritized public relations over robust security” then it should be no surprise when such priorities eventually destroy your business.
Sadly, treating security as an annoying adjunct to the fun is the biggest mistake entrepreneurs have always made, and continue to make. They beguile themselves with the spit and polish of a well-marketed lobby, and pay little time or money to securing the foundations that hold up the building. Splurging on logos, PR campaigns, apps and even SEO is sexy. It’s new media. It’s Silicon Valley glitz. Apportioning funds from the start to a pallid team of geeks who give a damn that you’ve used Pass1234 as the password that gives full access to all your servers, is not sexy. But it’s annoyingly vital. Getting an in-house security team, or a managed service like we provide at Positive, or engaging serious consultants who wag their fingers at you when you try to ignore that one little thing that’ll end up destroying your venture, is not an optional extra. It’s the sine qua non of establishing a service on the public Internet.
Another lesson involves continuity. It’s at this stage that one would usually engage cliches about eternally repainting the Forth Bridge in Scotland. Consider them duly engaged! Even those who’ve paid a modicum of attention and money for the initial security of a site-build then distressingly often chuck the thing onto some cheap slag-heap of an unmanaged hosting package, or perhaps float it up upon a lazily-drifting public cloud, pop open the champagne and then never bother themselves with the underpinnings of their venture again. Of course, just because the ship’s crew never peek under the hull doesn’t mean that the black-hatted buccaneers don’t. As every week passes, new holes pop open. With nobody there perspicaciously to patch them, your stout ship becomes quietly keelhauled. Eventually, it sinks.
But before it actually takes on water, why spend ongoing money on a bunch of stalwart geeks to maintain the vessel when it’s so much more fun to splash out on parties in its ball-room? A lesson from the Ashley Madison attack is not that the hull-breach was “very sophisticated”, as some have claimed. Quite the contrary, as the attackers made clear, the vessel was trivially open to plunder and, unforgivably, “nobody was watching”. Make sure you always have someone to watch over you, as the song goes, and keep that crow’s nest staffed.
This focus on systemic continuity is why we at Positive make such a distinction between a managed service and mere “hosting”. A team of conscientious geeks who darn and patch security holes as they inevitably pop open provide no guarantee of incorrigible security. Such attention does, however, shut “water-tight doors” against the severity of any attack that might succeed. If you discover a hole in your web application, make sure you have engaged a team empowered to patch that application without breaking it. Where any hole remains, limit those who have administrative access to the bowels of the application to the minimum trusted pool.
If that hole nevertheless does allow nefarious external access, make sure that the server’s operating system is sufficiently buttressed so that no “root” compromise can be achieved. If a root compromise somehow still succeeds, make sure that sensitive data is properly encrypted and that logs have already been shunted elsewhere, secure from tampering. Again, effective security is not installing a branded VPN in front of a leaky hulk and partying on, but is instead a systemic imagining of best and worst case scenarios, and putting a panoply in place against them.
A final lesson to be learned is that verifiable trust is crucial. You need a team whose provenance is proven, using software, systems and even procedures which allow full and transparent auditing. This is one of the reasons why we give tours to prospective clients of our datacentres, where the prospective client can go through full due-diligence prodding of every aspect of our operation. It’s why we encourage face-to-face meetings with our team. It’s why we not only take the time to learn how the applications entrusted into our care work, but also why we make a priority of getting to know the people and business behind the application. This is why we at Positive Internet are trusted by many of the world’s largest banks and insurance companies to look after some of their most precious data assets.
Crucially, verifiable trust is why we have focussed, for nearly two decades, on Free and Open Source software. As with peer-reviewed science or democracy, the transparency that Open Source promises is no panacea for security. But whilst Linus’s Law is not infallible, it throws into relief the least-worst method of software production we have. And, unlike security-through-obscurity proprietary systems, Free and Open Source platforms at least guarantee that the winner of a “game” between the black-hats and the white-hats should depend on skill and wisdom rather than a hopelessly skewed playing field. Whilst Ashley Madison used these technologies, it is notable that they used them in conjunction with proprietary plugins and services that seem to have been the gateway to the crack.
To suggest any magic-bullet response to the Ashley Madison hack would be dangerous hubris. No single piece of technology, no individual vulnerability scan, and no wizard with a compliance manual will guarantee against such break-ins. However, we should take very seriously the mortal differences that a project’s culture, ethos and priorities can make to its eventual vulnerability to such calamities. We should neither fall victim to a counsel of despair, nor to any silicon snake-oil vendor who assures us blithely that their special tonic will let you party on without a care. That might sound portentious, but it can be heeded simply by making the best use of open, transparent solutions and deep, trusted relationships with your technology providers. It is perhaps ironic that in its exhortation of blithe affairs, this is the one trusted relationship which Ashley Madison should have taken care to nurture.
The Debian Project: how freedom works
Yesterday, the Debian Project turned 19 years old. Positive Internet raises a glass to this miracle of human cooperation. More than an unassuming technical project, it has profound implications on what humans can achieve with unencumbered networks.
Operating systems don't lead to much heated enthusiasm in the saloons and hostelries of the nation; less so the different ways of distributing and putting together those operating systems. Talking about operating-system distributions seem a very definition of arcane. But bear with me: by examining one operating system's distribution in particular, we can find fascinating sociological, economical and philosophical insights.
The distribution I'm talking about is 'Debian'. But let's first backtrack a bit: what is a 'distribution'? When you buy Windows, or MacOS, in its shiny box or from a glinting app-store, you understand that you're getting something that makes your computer run, provides the windows and menus, and includes some basic software, like notepads and wordprocessors. In a sense, then, you could call those proprietary operating systems "distributions", I suppose. But because what gets put on the DVD or in the download is strictly regulated by the company who produces the operating system, there is usually just one distribution of such operating systems. Well, to be precise, there are some small variations: Microsoft, for example, allows you to purchase cheaper, more restricted versions of Windows for low-powered netbooks, and throws in an extra bucket or two of bells and whistles for their enterprise versions. But, because of their tight, proscriptive control of what goes into these sub-versions, it seems more appropriate to call them "editions" than completely different distributions.
Operating systems based on the Linux kernel tend to have less onerous licencing restrictions; however, the organisations that put together the distributions of the free software packages that run on top of the Linux kernel - the web servers, the databases, the desktop clients and so on - are often competent but otherwise uninspiring. RedHat, for example, for all the good work it does, is "just another" company, that employs technicians to put together and test these packages under its corporate roof, and sells support and management services. Certainly, it receives help from its community via the Fedora project, but it still acts as the central, commercial clearinghouse that is well understood and oft-immitated.
Debian is different: it has no centre, no hub, no commercial core, no large team of employees. Debian is an agreement, a coherent social structure, an arrangement between free agents - but it is not a corporation in the traditional understanding of that term. It is a group of individuals and organisations who, for manifold reasons, tend their own particular corner of a garden that matters to them. The interesting bit happens through an alchemy of the network effect, free software and combined, intermeshing self-interest. So long as the individual maintainer follows certain rules and conventions, their specific package, documentation or meta-infrastructure is included into the project as a whole. The benefit to the maintainer is that their work is given a supported place within a larger, useful infrastructure. And the benefit to the project is that it gains an additional useful component. The networked world, and the universe of Free and Open Source software, allows startlingly novel structures of production that nevertheless somehow speak right to the interest-matrices of our hunter-gatherer roots. Debian is the prime, working example of such novelty based on ancient cooperative instincts. No centre, no core, and yet somehow out of the chaos emerges a stable distribution of thousands of individual software packages produced by hundreds of different individuals in dozens of countries. And it all works meticulously. In a sense, Debian has had a universal, networked, liberated "app store" for more than a decade before the proprietary people caught on.
More recently, producers of proprietary operating systems have include "app stores", where people can distribute their software, and the operating-system owner, as gatekeeper to the walled garden, takes a cut for each application sold. People are treating the coherent, easily-updated app store as a novel development; in fact, projects like Debian have provided huge repositories of coherently distributed and updated applications from the beginning, but with one key difference. Debian provides these applications not as part of a "walled garden", but though a cooperative agreement on how applications should be installed and updated in its environment. Because the great majority of Debian's applications are under Free Software licences, the coherence and scope of the "app store" goes well beyond those offered in the proprietary realm.
Because Debian focuses itself profoundly on Free and Open software, the social contract of those who produce its thousands of packages reflects the contract's principles both in philosophy and engineering. There are guidelines for creating and maintaining individual software components, but no compulsion, no specific carrots and no hefty sticks. One would assume, then, that this peculiarly libertarian but cooperative environment would lead to chaos and instability. In fact, Debian is renowned as one of the most stable - perhaps even conservatively so - distributions available, and has been so for longer than companies like Google, Facebook and even Apple (in its revitalised form) have existed.
We at Positive Internet have used Debian almost since our inception, and have systems with many years of uptime on it. We take for granted the abilities it gives us to keep systems secure and reliable, and yet replete with a rich collection of important applications for running the latest web-services for our clients. The huge variety of tasks we have demanded from the distribution is testament to its flexibility: from small firewalls to massive bank-grade clusters, we don't give a second thought that Debian will be up to the job. And where it needs tweaking, or some special package demands particular tending, we know that a universe of maintainers and developers will be there with us in that task.
So, for 19 years, the Debian project has modestly, unassumingly, engineered myriad human minds, with myriad separate requirements and motivations, to deliver, e pluribus, unum. This project of massively heterogeneous conception, emerges as one unified, elegant body of work. It is a pity that so many political theorists, economists and philosophers are technophobes: would that they weren't, they would have a case-study of of two decades of cooperative productivity in the networked age that is uniquely captures our species' capabilities in a liberated networked world.
Debian has spent nearly 20 years proving conclusively that a well-tended garden needn't be walled. And for that, they deserve a toast or two.
Positive involved in another World Record event
Anyone visiting London in the last few months will probably have noticed that large eggs have appeared throughout the city. These intricately decorated pieces of public art played their part in promoting an auction - both online and at Sotheby's - in aid of the magnificent charities Action for Children and The Elephant Family.
We at Positive Internet were asked to supply the hosting infrastructure for the live online auction, for which we provisioned a special managed failover cluster in our flagship Positive Park datacentre. We took the responsibility seriously: the thought of the final moments of such an important live event's ruination by a server crash or network glitch is the stuff of bitter nightmares. We prefer sweet dreams - so we put our best team on it, who kept a close eye on these systems throughout the Easter Monday bidding.
In the end, the hard work paid off, because the online and Sotheby's auctions netted over a million pounds for the worthy causes, with Positive's managed cluster never missing a beat. Alan Newman of Sensible Development coordinated the whole project between Positive, the charity and creative agencies. Alan was kind enough to offer the following comment:
"We have worked with Positive Internet for many years: they understood exactly what was required for this high-profile project, and tailored the cluster so it'd cope with bursts of extremely popularity.
As part of the VIP managed service, Positive engineers offered live support throughout the auctions. They made sure everything went without a hitch. Positive is a partner rather than a simply a 'provider': everyone from the directors through to the oncall engineers is a pleasure to work with. It's clear they care about our projects as much as we do".
Such kind validation of our team really helps our egg-os. Of course, we work egg-stremely hard to ensure our managed service always entails this level of attention to detail; however, when we muse on the work that Action for Children egg-sel at, it's egg-straordinarily gratifying to know that our service has played a part in allowing them continue doing it so egg-selently. This evening, we're egg-suberant to find out that this is the second project in which we've been involved that's won a Guinness World Record! Indeed, it's won two. Frankly, we're oeuf-er the moon. (sorry - we'll egg-ress now)
Podcasting is fast becoming the default way people broadcast and listen
Positive Internet and our Jellycast venture are veterans at podcast hosting and distribution. Indeed, since we won the Guinness World Record with Ricky Gervais for his podcast, we have hosted a gratifying number of top-rated global media extravaganzas for people like Stephen Fry, and for organisations like the British Council. The most recent in our stable to gain the coveted iTunes top podcast status is Iain Lee's Pocket Radio Show. From an idea Iain and we discussed to a global sensation in mere weeks, this is just another of those unremarkably remarkable success stories that hosted services like we provide have made possible.
Iain, a well-known television and radio personality, has decided to devote his proven talents to podcasting. This is podcasting 2.0: professional, interactive, with the sophistication of a radio programme but with the immediacy of new media. All the finesse one would expect from a professionally produced programme is there: phone-ins, interviews, jingles - the signifiers of "proper" speech broadcasting that just a few years ago would have demanded the backing of a huge established media-organisation to marshal, but today require little more than a microphone, a laptop and an affordable mediacasting account with Positive Internet or Jellycast.
Iain's podcast represents an increasing trend in online audio. In the early days, podcasting was considered either an amateur's game, or a gimmicky adjunct to a "proper" broadcaster's output. As the medium has evolved, an increasing number of professional people are focusing on podcasting as their primary or even sole means of communicating with the public. The freedom that podcasting gives creative people has been an obvious lure from the start; as audiences have proven themselves loyal downloaders, an increasing number of popular podcasters are finding that they are able to sustain themselves financially through cultivating this audience, by obtaining sponsors and through democratic patronage from their followers. Podcasters control their own media destiny in ways unimaginable under the old broadcasting hierarchies. This freedom is intoxicating. Listeners enjoy the closer relationship they have with the broadcaster, without the alienation of intrusive middle-men.
In a sense, then, what is remarkable about podcasting is how unremarkable it has become. We take for granted the ability to select and download audio programmes at will. Commutes, flights, jogs and even hospital-stays now seem unimaginable without our favourite stash of comforting audio-accompaniment. And popular podcasts, like Iain's, are garnering tens of thousands of loyal listeners. Just a decade or two ago, the notion of getting radio-sized audiences by recording something in one's front-room and uploading it to a service like Positive Internet's would have seemed absurd science fiction. Now becoming a radio star is simply another in the range of services we offer. We work hard to ensure that our mediacasting bandwidth, reliability, support and speed is second to none, so that we can fade into the background and allow our hundreds of media champions to shine, and some to propel themselves all the way to the top of the charts. Just like Iain has done over the last few weeks.
We're such dab-handed veterans at this now that it's sometimes easy to forget how subtly revolutionary the services we provide can be. It's no longer grandiose theory to proclaim that anyone with an idea and with the talent to communicate it can do so without hindrance - to the whole planet. And that's surely something worth remarking on.
Rest easy: we have it covered
Excuses, excuses! We have been rather busy this last year establishing our new PosiCloud product, gliding more and more clients over the PCI compliance mountain and, as ever, helping some of the most popular sites and services on the web tick along merrily.
Almost inevitably, updating the PosiBlog took rather a backseat to this, but our New Year's resolution is to remedy the paucity of posts here! We hope to provide interesting tidbits on this blog more frequently, and look forward to hearing from everyone about topics you think we should cover. Feel free to mail us at email@example.com with feedback and ideas, or throw us a tweet on Twitter - we're @posipeople
In the meantime, we wish all our clients, customers, suppliers and staff a very restful Christmas and a wonderfully fruitful New Year. Naturally, we remain on duty throughout the festive period, so should any of our managed clients need help at midnight on Christmas Eve, or a quick chat at 3am on New Year's day, we'll be around, bushy-eyed and bright-tailed. Our flagship Positive Park datacentre will remain securely staffed at all times. With perhaps just a little more tinsel than usual.
How GoDaddy lets our whole industry down
The only thing more amazing than the Internet is how we take it for granted. In fact, we can actually slip into holding it in slight contempt. It remains the butt of many a radio panel-show's tedious jokes ("haha, Wikipedia is not always reliable. haha, the Internet contains pornography" and so forth). You'd have thought that nearly 20 years of the Internet's absorption into public consciousness would have dampened such fatuous punchlines. But no, the audience titters still, as if in eternal sympathetic vibration with the 90s collective memory of "it's just a geeky CB radio fad".
Recently, the director Steven Soderbergh discussed his latest film on a radio review programme. The movie makes fun of blogging, which he terms as no better than "graffiti on a bathroom wall". His leading man even wore prosthetic buck-teeth so he could be the "typical blogger". Quelle dérision! Soderbergh mused, happily, that in his test-screenings, the audience gained almost orgasmic pleasure every time the Internet was attacked. They couldn't get enough of Net bashing, which they enjoyed more than the rest of the film put together. The film-reviewers interviewers chuckled along, in amiable agreement. That they then immediately promoted the Internet podcast in which this interview was being promulgated didn't seem to phase them.
Such cognitive dissonance abounds. Even while they giggle nervously, most of the technophobic titterers would rather have a limb extricated than their smartphone. They would sooner cancel the whole postal service than one Email account. However they sneer at wikipedia, they would whimper if ordered to visit the public library every time they needed to look up a footling fact or all-encompassing concept instead of using this astonishing online resource.
In this season of Goodwill to all Men, we should pause to consider what an astonishingly lucky species we are to have managed to coalesce about us something as miraculous as the Internet. A miracle can be defined in a number of ways. One definition that captures its essence without having to rely on playing supernatural games is to term a miracle any happenstance that was somewhat unlikely, and did not necessarily follow the predicted contextual logic of the times. Such a miracle emerges as a delicate butterfly from a cocoon of capricious chaos. Such a miracle might be seen as something delicate, something as fragile as a growing crystal, however ubiquitous and sturdy its eventual shape. Its formation was mercurial and unlikely and, were we to reverse time and reset the experiment, so to speak, would likely not follow the same path again.
And when you start taking such a miracle for granted, you can put it on a neglectful path to destruction. You forget how lucky you are to enjoy its benefits. You forget how delicately inchoate was its formation. You assume that its features are part of a solidly inevitable march of historical progress. And your hubris bloats to the level where you even enjoy a little contempt for its stale ubiquity. Haha, that wikipedia. Hoho, those blogs. And you forget that the whole complex collection of services and social conventions that make up the Internet is predicated on a fundamental notion of liberty. And you forget that all liberty comes at the price of an eternal vigilance. And you allow the desperate, malign or merely opportunistic to take advantage of this amnesia and eviscerate the miracle. And then you walk despondently amidst the shards of the shattered crystal, wondered what went wrong. But no cyber-equivalent of humming "Where Have all the Flowers Gone" will bring it back.
Early in the New Year, the American government aims to help Big Media to eviscerate the Internet, to destroy the miracle and shatter its crystalline delicacy. As ever, Big Media is blaming "piracy" as its excuse to stamp its jackboot repeatedly on the face of our online lives, and is bundling this repression into a horrible proposal entitled SOPA (Stop Online Piracy Act). That there are plenty of more engaging and fruitful ways to fund creative endeavours is ignored. It seems we must all be coerced into accepting that a failing business-model from the last millennium deserves no less than full might of the State to bolster its clumsy missteps. Suffice to say, it is sadly not hyperbole to claim that the provisions of this act would help to destroy the Internet. It is technically inevitable. And it would give to the American government the same arbitrary rights of censorship now enjoyed by such places as North Korea and Saudi Arabia.
Fortunately, a large constituency of the online world refuses to join Soderbergh's titterers. It is taking action, writing to Congress and fighting against the dying of the enlightened Internet. Happily, this panoply against the disembowling of the Internet includes a wide range of organisations and individuals, many of whom can be found bickering in other circumstances, including Google and Microsoft. You might assume that the fightback would also include the Internet hosting providers of the world. And you would be correct. With one glaring and shameful exception: GoDaddy. We do not make a habit of dissing or even merely discussing our competitors. It is unprofessional and usually seems peevish at best; however, in this case, when the whole industry is standing up against the forces of darkness, we wish to join our ethically-sound competitors in drawing to attention such obnoxious behaviour. This is not the first time that GoDaddy has courted controversy. But this time, it doesn't seem merely happy to exterminate the odd pachyderm: it is eager to help to destroy the Internet that gives it profit. Such behaviour brings the whole hosting industry into disrepute, and we join our honourable competitors in countering this crawlingly tawdry display.
A large campaign has formed to urge people move their domains from GoDaddy in protest at their behaviour. It might be self-serving of us to support such a campaign. After all, capitalism enjoys nothing more than the opportunity to kick a competitor when it is down. This really does go beyond that for us, though. We have always revelled in the miracle of the open Internet, and feel an existential shudder in our core whenever someone attacks it. Thus, if you have domains with GoDaddy, our hostmistress will be happy to advise how to transfer them to someone more ethical. Whilst we would, of course, be delighted to take the domains under our own wing, we will also be happy to advise on how to transfer your domains to any competitor of ours more ethically sound than GoDaddy. This means so much to us that we would prefer to take the hit in time and effort to give business to a reputable competitor than to see GoDaddy remain unscathed by their wilful defilement of the miracle that sustains us all.
Update: After a flurry of bad publicity, GoDaddy have just announced that they will no longer support SOPA after all, although apparently they still support its principles.
The Art of GETTING Excellent Tech Support
There are a many tales of horrible technical support experiences. Listless automatons who seem to be reading from a laminated script. Long call queues leading nowhere. An infuriating refusal to allow you to speak with someone who can actually solve your problem. Being stuck as a modern-day Sisyphus in an eternal "support escalation" hell.
Of course, the primary response to such tales is to use a more competent, humane organisation. One like ours (hey, this is the Positive Internet blog, after all :-) We do try really hard not to be jobsworths. We get our hands dirty. We try to act as if those seeking support pay our salaries. Because, well, they do.
We like Email as a primary support medium. It means that you can paste us copies of logs, we can type the exact URL you need to visit without having to spell each letter out on the phone. It means that you have a full written record of the ongoing issue, and a later reference for how we solved it. In all, Email is perfect for this. Of course, too many organisations have tainted the form as an effective support channel. They take too long to respond, or don't respond at all. And when they do respond, it's illiterate, haughty or confused. We try very hard not to be that sort of company. Our 3-working-hour support promise and related efforts underlie this.
All this said, even with the best support team, getting help is a two-way street. There are things you can do when seeking support that can help you massively in getting good answers quickly. I'll present you here with some tips, all based on real-world examples. Some might seem obvious. Even then, you'd be surprised how often the obvious is overlooked:
Set your subject
The subject-line of your mailed support request is what the team sees when they scan a list of tickets. As such, they can be useful in quickly identifying which ticket is yours and in offering a quick summary of the problem. As such, it's puzzling how many people set their subject line to something generic like "urgent" or "help please". Or nothing at all. Instead, use a short, descriptive, identifying subject which will help the right people in the team get to your ticket more quickly. For example, let's say that one of your database queries is timing out, and you'd like help in identifying why. You might have the subject as something like:
Subject: Simple DB query on www.example.com times out
I think you'll agree that this will get the DB experts' attention more rapidly than something like:
This is especially true if the ticketing system is full of "urgents". Again, it may or may not surprise you to realise that many "urgent" subjects are not urgent in any reasonable interpretation of the word. Indeed, the word has become so devalued in support circles that it's seen as crying wolf. So, ironically, if your problem really is urgent, try not to use the word "urgent" if you want to get good support.
Say who you are
We certainly try to get to know our clients, and get quite good at identifying Email addresses and the like; however, you'd be surprised how often we receive a mail from an unrecognised address, asking for help on an unidentified site. What would you do if you received a message like this?:
Subject: site down
From: BR <firstname.lastname@example.org>
Hi. My site is down. Can you tell me when it is going to come back up?
Who are you? Are you even a client of ours? What site are you talking about? Of course, our first response will have to ask all these questions, which will just mean that the actual help will be delayed. Mention your name, the organisation for whom you work and any usernames/domains/server names in question. Yes, it'll take an extra few keystrokes, but I promise you that the disambiguation will save time and get your ticket answered more quickly.
Say what you see
Sometimes, we can all be a bit solipsistic. We forget that the people we talk to don't have a bird's eye view of our thoughts. So, when we say "the site's down", we forget that this comes with a lot of contextual baggage that we just take for granted. Which site? How do you mean "down"? How did you check? How can someone else confirm this?
A group of philosophers in the 20th Century, called the Logical Positivists, deployed what they termed the "Verificationists' Creed" when they tried to work out what things meant. It went like this: "The meaning of a statement is the method of its verification". Whilst the Logical Positivists are no more, this creed that was such an important part of their toolkit is astonishingly useful in giving and getting good technical support.
Use the creed as a rule of thumb when writing to support. Instead of saying "the site is down", think "To get the meaning of the statement 'the site is down', I need to describe the method of its verification. Otherwise, it's ambiguous at best, or meaningless at worst". Then, type that verificatory method out and send the ticket!
So, for example, instead of:
Hi. My site is down.
You might say:
Hi. I am trying to visit www.example.com in my Firefox 3.5 web browser, and it immediately returns an "Error 404" message.
Notice that this doesn't describe or summarise a "metaphysical" situation: it describes what you're doing - something we can try to replicate. So, always describe the steps you take rather than your opinion or summary of the results of those steps. Then we can follow those steps too, and can either confirm the problem or point out which step was in fact a misstep.
Say what you want to see
Sometimes, saying what you want to see isn't enough - you need to let us know what you'd want to see in normal operation. For example, if you complain that something looks odd, we need to have some yardstick of normality. So, let's say you send in a ticket, saying:
When I view my web page, it has a blue background. Can you fix it please?
Well, you've followed the verificationists' creed - you've explained what you see, rather than just saying "there's a problem with my web page"; however, we don't know what you expect to see instead of a blue background:
So, you might add something like:
When I tried to view the same site last night. it displayed with the correct yellow background. I haven't made any changes that I know of.
That way, not only can we validate what's wrong, but we can validate what state we need to help you to attain to make it right.
Say where you are
The Internet's a big and complicated place. Sometimes, problems that you might see at your end of it could be caused by any of a dozen of possible points between you and your website at our end. To help us verify where a problem might lie, you should make a habit of providing your remote IP address. You can find this by visiting a site like http://itempeter.com
We can then find out whether your ISP is having network problems, whether some local firewall is blocking you or whether there's a problem somewhere else between you and us.
Say what you're using
Sometimes, different software on different operating systems can produce different results when accessing Internet services. Let us know what you're running and how you're connecting to the Internet and we can analyse whether that might have any bearing on the issue.
If something seems to be awry, try to replicate the result using a different context or system. Try a different brand of web browser, try using a mobile phone on 3G to view the same issue, call a friend and ask whether they notice the same issue. This can help you and us to determine early on whether the problem's related to some local issue at your end, or whether there's a more general problem.
Tidy the thread
As replies on support tickets get batted back and forth, the quoted previous reply can obfuscate the fresh content. We've all struggled through a thicket of triple-quoted, confusing old text to find the craftily-inserted new update. Instead of solving your problem, we spend the time parsing it. The best thing to do is to delete most of the old baggage, and leave in just the sentence (or perhaps paragraph) to which you're responding. If you're adding responses to a number of previous question or comments, then delete the stuff in between with, perhaps [snip] in its place to show that you've done so. This sort of pruning can be really useful in getting directly to the evolving heart of the issue.
Keep things separated
If you have a number of issues, and they're all related then, by all means, mention them in the same ticket; however, if you have a number of unrelated issues you wish to discuss, it's probably best to put them in separate tickets with separate informative subjects. This means that not only will the successful solving of one problem not lead to the mistaken ignoring of the others - it also means that different members of the team can focus on the different issues in the different tickets in parallel.
In the end, our team's probably seen your issue before and, once it understands what's going on, will be able to solve it. And in the meantime, there are always pictures of kitten to look at!
Modern Gatekeepers to Speech: Careful with the Keys!
You just need to hear the beginning of some sentences to know things aren't going to end happily:
"Some of my best friends are ... "
"I can take a joke as much as the next man ... "
"Is it just me, or ... "
Then there's this one:
"I believe in the freedom of speech, but ... "
You know what's going to happen next. They're going to reveal exactly how much they don't believe in the freedom of speech. "I believe that people should be allowed to say things that don't offend me" is how it usually boils down.
What does this have to do with managed-services/hosting site? Quite a bit. You see, much speech these days gets transmitted across the Internet, and people like us provide the platforms that allow this transmission.So, it may not surprise you to learn (or perhaps it may) that, when someone uses our platform to say something that someone else doesn't like, then the offendee often gets in touch with us directly to complain about the offender. Invariably, they'll word their complaint one of three ways:
1) They'll simply demand that we stop hosting the person or organisation who's offended them, as if the right not to be offended is now the paramount requirement of our age.
2) They'll suggest that the person has committed libel or some related offence, with scant evidence and no legal backup.
3) They'll suggest that the person has somehow breached some sort of copyright or related "intellectual property", and so must be shut down and shut up. This is an increasingly common way to try to gag people, thanks to horrible legislation like the DMCA in the US and the EUCD here.
Occasionally the complaint will be backed up with hints at legal recourse or illegal denial-of-service attack. As it happens, with many hosting companies, any of these tactics is all it takes to get someone's services pulled. The hosting company can't be bothered to investigate the full facts, and it's simpler to be judge, jury and executioner than to demand due process. Benefit of the doubt? Not so much.
This is unfortunate. After all, we supposedly live within the rule of law, and we also live in a society that's supposed to protect robust heterogeneous discourse. Surely turning up with pitch-forks isn't the way to answer back? Instead, why not just.. answer back. With words! And when someone does turn up with their pitchforks, is simply retreating not giving in to mob justice?
However deplorable it was, there used to be a reason why people felt eventually driven to pitch-forks. When only a few privileged media held all the cards, and were the gate-keepers of opinion and public discourse, the pitch-forks were an understandable, if unfortunate, response. With the Internet, the playing-field is levelled, dramatically. A silly newspaper article or an unsavoury opinion can be countered with counterblasts in seconds, which can be heard and seen with at least the same vehemence as the original article that sparked it off. If someone offends, you set up your own blog posting. You start a twitter campaign. You enter into conversation. You neutralise that which offends. That's the way freedom of expression is supposed to work. Or perhaps you simply realise that being offended is the price you pay for living in a free society, and go and kick a tree. So long as you don't offend any passing arborphile.
Sadly, too many people cling to the old, "appeal to the authoritarian" approach as first instinct. It's like we revert to our childhood and go running to daddy, and ask him to "make the nasty boy go away". Organisations like ours are caught in the middle of this infantile appeal. Of course, should a court of law demand that we remove something, we do so. We respect the rule of law. But what if a large corporation or a disgruntled malcontent makes threats to us directly, with little basis, in the hope they'll scare us into silencing one of our customers? We try not to be cowed. We have rescued quite a number of sites from the timorous grasp of those who have timidly given in to such threats, and we're proud of that; but it worries us that too often, we seem to be a lone voice sticking up for this sort of principle. Maybe it's time that hosting companies agreed to give their clients the benefit of the doubt. Perhaps we need to agree to a kind of bill of rights, where we guarantee that we will not act tyrannously and arbitrarily kick our clients off at the first whiff of trouble. Of course, for this to be effective, we'll need laws that give us the benefit of the doubt in such circumstances too, and a legal system that can operate quickly and efficiently if it deems that something is genuinely illegal without punishing us for acting in good faith in the meantime. This might seem somewhat grandiose - after all, hosting is a private commercial venture; however, the Internet has become such an important platform of cultural and political expression that we gatekeepers are in a privileged position beyond what we might often realise. Perhaps as an industry we need to codify how we should live up to the responsibilities behind this privilege.
Netflix tries to pull focus but spoils its image
The US movie-streaming company NetFlix posted an article the other day proclaiming the brilliance of Free and Open Source software and how, without it, they'd be pretty much scuppered. All the Free Software they use to get their services running is indeed just the software that we at Positive have supported for all our existence. We try to put our money where our mouth is - sponsoring community conferences and projects, hosting mirrors and generally trying to be good corporate citizens to show at least a modicum of gratitude for the amazing technical bounty from which we and our clients (and Netflix, and Google, and Facebook, and Amazon and pretty much everyone else) benefit.
When we at Positive Internet started all those years ago, we had constantly to explain why Free Software wasn't simply the dreams of some hippies in a commune, but a major technical and philosophical revolution upon which the next generation of web successes would be based. And, indeed, they were.
So, when NetFlix gets press coverage about its use of Open Source projects, the coverage they've received (as if this should be news to anyone) is a little puzzling. What's more puzzling is the audacity of their touting their Open Source credentials when there's one glaring gap in their otherwise heartwarming tale: they provide no software for their services for those who run GNU/Linux on their desktops. So NetFlix are happy to take and patch GNU/Linux services from which they can profit, but they can't be bothered to ensure their software runs on the same platforms they're happy to "leverage" to their own ends. Take, take, take. The story repeats itself with so many other companies who exploit the riches that Free Software provides but give little in return: the satnav people TomTom are another example. Those little boxes you plug into your car's cigarette lighter and have liberated you from your map in the glove compartment run GNU/Linux and a plethora of other Free Software. TomTom would not be where they are today without this galaxy of freedom. And what have they provided in return? Nothing. Although their Windows and Mac clients are directly based upon the Free Software Firefox browser, they wilfully refuse to provide a GNU/Linux client which will allow people to upload maps and the like to their GPS services. It would have cost them veritable pennies to provide their software, and their GNU/Linux users have been begging for it. They can't be bothered to listen.
Until now, these selfish so-and-sos have had the excuse that, whilst they are happy to exploit the riches of the Free Software infrastructure, those who use GNU/Linux and the like on their desktops were currently so limited in their numbers that it hasn't been worth the company's while to port their software.
Until now, one would have simply retorted "forget the numbers and just do the decent thing ". Decency is not a very persuasive argument when it comes to business, so a more pragmatic response has recently made itself felt: with Android, which runs atop Linux, as well as any number of slates, netbooks and other new devices which don't simply run Windows coming down the line, the days of dismissing the end-users of these alternative platforms as a few lonely cranks is rapidly coming to an end. Those organisations who have gotten into the swing of producing GNU/Linux clients (like Spotify, for example) will not suddenly have to shock themselves into playing catchup for these new, vital platforms. Others, like NetFlix and TomTom might find that their early selfishness gains them a rude awakening. Sometimes, being decent earlier bears fruit later. It would be good if a few more organisations realised this.
Superconducting software requires freedom
Apps. Chances are even your maiden aunt Phylicia has heard of them: those little programs one can install with barely a breath, that one can update enmasse with nary a glance. Such elegance. Such innovation. This sort of revolutionary tech could only come out of the shiny, well-funded proprietary labs at Cupertino, surely? Until a couple of years ago, if you wanted to install an application, didn't you need to download it and run through an annoying set of dialogue-boxes? Then, didn't you often as not find that you also had to download the library or support program without which the application would refuse to run? And what if you wished to update all your programs at a stroke? Out of luck, surely? You had to re-download each update for each application and keep your fingers crossed. Or rely on a very limited "system update" function that seemed to ignore most of the important applications on your hard drive, but became obsessed with "updating" the one or two you never used. And that was on "friendly" systems. Heaven help one in a "messy" open source environment, you might think. Hooray for well funded, proprietary innovation! The days of having to tend one's system as if it were a complicated and cussed steam locomotive are over. Huzzah to Mr Jobs et al for surveying the chaos and razing it! So the narrative goes. The narrative, however, has it backwards.
Where, then, before Mr Jobs even had it as a glint in his eye, was any simple-to-install universe of applications? Where was this panacea which seemlessly brought in all the libraries as needed, didn't interfere with previously installed programs and allowed in-step updating at the click of a mouse or the press of a key? And where, today, does such a system exist with a scope and efficiency only dreamed of in any proprietary "app store"? In the world of the Free, that's where. Such systems have been taken for granted in the Free and Open Source world for well over a decade. As ever, when we talk about "Free", we're not simply talking about gratis software, but software released under Free licences that allow one to modify, install and distribute such software liberally. Projects released under these licences include some of the most popular applications in the world: the GNU/Linux operating system, the Apache web server, the Firefox web-browser and so forth. We at Positive have focused exclusively on these technologies since our start. So we know that the Free software ecosystem has offered for well over a decade the equivalent of the "app stores" that now abound. We also know that these Free repositories offer a flexibility and efficiency that the proprietary imitations will never be allowed to have. Paradoxically, the universe with the most "chaotic" development model has produced the most coherent distributions of software. Perhaps even more paradoxically, the least centralised, least proprietary distributions of this Free software have produced the most stable, most coherent ways of installing and updating swathes of software at a stroke, and in sync.
Now this might seem strange to you. Certainly, you may know that Open Source software has produced interesting and popular projects. Undoubtedly, it has some technocratic efficiencies. But isn't the "Freedom" associated with Free Software just a happy coincidence, a patchouli-fragranced frill exciting only to hippies and hopeless romantics? If we've learned anything in our 12 years at Positive, it's that the Freedom of Open Source software is what allows it to achieve such revolutionary innovation. And that's real, profound innovation, not the simple, glitzy-surface innovation so beloved by PR agencies.
If you think about it, this shouldn't be a surprise to you: history demonstrates that free societies, however messy and unruly, usually end up more prosperous, more interesting, more vivid. Unfree societies might pretend, for short periods of time, to make the trains run well, but we all know how that ends. Why should it be any different for software? Dictatorship and oligarchy fail through the inefficiencies of their repression, in as much as anything else. A case study has existed for nearly two decades now: we at Positive use the Debian GNU/Linux distribution. This contains not just the operating system, but a whole galaxy of all the software our clients would require, each package therein maintained by independent individuals and groups. The Debian project, effectively a loose collective with strong rules, has no overarching corporation at its helm. It has no imperative other than to produce excellent, coherently packaged, stable software. Each package maintainer understands his or her duties within the project, but otherwise follows his or her self interest in packaging software that he or she finds productive to use. And then adds it to the greater stew, so to speak. This infrastructure has, perhaps surprisingly, produced a software ecosystem that is not radical and brittle, but conservative and stable. Indeed, people's frustrations with it are that it sometimes seems too conservative, too willing to err on the side of caution rather than introduce sparkly new features quickly. The loosest, most chaotic, least centralised system of packaging up the most libertine of software has produced what is generally renowned as one of the most stable, productive, coherent and easily upgraded platforms.
Certainly, proprietary funding and fiddling allows for shallow surface coherency. But for a deeply productive, intrinsically sustainable software ecosystem, only freedom reduces the irrelevant friction to the degree demanded in today's networked world. Indeed, Free software might be termed "superconducting" software: where all unrelated impediments to smooth and efficient production are removed, be they legalistic, marketing-driven, shareholder-demanded or otherwise. If you still find this a paradox, then perhaps you should reconsider Churchill's statement: "It has been said that democracy is the worst form of government except all the others that have been tried". Similar could be said about software. In a future posting, we shall discuss more the role of Open Software and its Enemies: those who envy the unencumbered competition that such software offers, and how they repeatedly attempt to throw a spanner in its works.
Looking for something older? Visit the archives.