Tom’s Graphic Design Journal

So work has this help desk / ticket system. Very basic, but taking this conversation a step further...It could be any ticket system script you have on your server. I won't go into trac though -- but I know how it could be integrated just the same. I'm going to stick to PHP for now because this is the implementation used in my situation.

The help desk form has some specific fields related to the web site and then stores the ticket into our database and then e-mails it off to people on the development team. Basecamp doesn't have a ticketing system (STUPID - this is merely one reason why trac is infinitely better than basecamp) so our requests and e-mails are all separate from Basecamp. The project manager has to transcribe all the e-mails and store them in Basecamp for record keeping. Note now a time stamp discrepancy aside from it being very very time consuming.

So what to do? Well, there's a 3rd party site that has a ticketing system for Basecamp at http://bctix.com. It's ok and completely non-invasive...but it's also limited and non-intuitive. Plus not integrated with your personal site or Basecamp in terms of filling out a ticket support form.

No one besides the signals there are going to make a perfect integration into their system. They need to do that. Although we can copy their css/html and layouts to make our system "look" like it's part of Basecamp...minus the URL. In my case we have our own ticket system that looks like some tools on the client's site and not like Basecamp at all. No big deal. However...it is familiar looking to the client and the form is completely customized for the client's needs.

Now what the bctix site does is puts in the title of each message, numbers that essentially equate to different things like categories and status. Ugly and more important very easy to screw up. The other downside is that you have to use the bctix site's forms.

My super slick solution (which was hours of pain) is to add on to our already existing form ... note for others out there, this means add on to your already existing ticket system ... and make it send off to Basecamp via the API at the same time it saved the ticket to the db and alerted the staff by e-mail. Cool. Easy. So then what? You have to go into basecamp to update the ticket or reply to it? No no no, here's the cool part (where the headache came into play)...

Along with e-mailing the proper people, there was another e-mail address tacked on to the CC which contained helpdesk-{basecamp ticket number}@domain.com. This wildcard e-mail address was then handled by a script on the server (Qmail was being used). This now leaves an address to reply to that is our "basecamp bridge" and what happens now is the server reads any e-mails coming back at it and takes the number past the hyphen ... which came from the response we got from the basecamp API when the initial ticket was made ... and adds a comment instead of a message using the Basecamp API. Now we have all replies from our e-mail programs automatically saved as comments (replies) in Basecamp under the appropriate message! It's by id number too so we can change the title of the message to file it or do whatever we want. We won't get messed up. 

Cool. So if you haven't figured out where I got a headache...it was in the server setup to receive e-mails to a wildcard address and process the mail through a PHP script. The other part that was difficult was all the regex matching to removed nested quotes from e-mails, privacy policy statements our company had at the bottom of our e-mails by default, etc. So in the end, it was just our replies that made it to the comments and not the same ticket repeated over and over. Very clean. 

So how'd we get the server to handle the e-mail? I partially followed this tutorial on evolt. I also looked up the Qmail manual/documentation which came in handy. Also note we are using Media Temple's DV server...So you need root access sadly. Unless -- your host is super cool and in your control panel you can forward e-mails to a PHP script...and also setup the catch-allo or wildcard aliases. I think that's possible somewhere under some shared host...it sounds possible at least, though rare...but think about people who can e-mail in their blog posts, right??

So in the .qmail file I basically made the setting to run a PHP script that analyzed the headers. At first I thought I'd be REAL sneaky and read the message-ID and references headers to see the e-mail thread and keep using that to distinguish between comments or replies. I also set a uid in the initial e-mail that was sent off from the help desk (which is still sent off). That didn't work out so well because different e-mail programs destroy headers. Almost all (if not all) e-mail programs remove any custom headers you set. So that's actually good -- note the uid I set now acts as the distinction between a brand new e-mail or a reply...because obviously replies will never have that uid, right?

So that's it. Our script basically reads the e-mail address in the header and grabs the id and uses that to call the Basecamp API to leave a comment on the message id. Voila. A very slick and transparent integration. I'll try to clean up my experiment and post something later.

Lots! Aside from my full-time job, I just wanted to plug a shameless link. Cloud-9ine uses my media blog for a face so I can keep notes about my discoveries with Amazon's web services EC2 and S3. Course it's my blog so whatevs At any rate...and more importantly, I wanted to let anyone looking at my blog (and anyone interested) know that I'm working on making my media blog / cms infinitely scalable.

It has some refinements that need to be made. Some bits that could work a little faster and basically more dynamic...for example, adding one or adding ten images at once...Things like that. This is pushing the development really into the next phase of beta which is just polishing things off. I already have working demos, but I want to polish what I have before adding more features. I also want to of course figure out how it'll all work in terms of being deployed.

So how will that deployment work? Well, I still want to offer some sort of version to the world because I've been a leech long enough and I want to give back to the community. We all really need better blogging software anyway. WordPress (used right here folks) and Blogger, etc. are great...but...It's 2008. We have all this new wonderful technology that simply didn't exist when all these blogs and CMS' were built. No fault of anyone...and certainly WordPress is quite good. I won't go into my theories, there's enough of that here and where I think the future is going to take us with web development.

Back to the point. One of the things I envision happening in the whole web industry is a very acute change in how our web sites are hosted. Of course Amazon EC2. That's not the only provider either, there's others like Joyent and many "regular" web hosts that have been offering VPS solutions for a long while. That unique thing about EC2 (and Joyent to some extent, but that's Solaris and I believe more geared for Ruby - bleh) is that you can set the instances up in such a way that you can simply turn on and off more as you need. This is way cool and here's why:

Minervablog will be more of what they are coining the phrase "appliance." Where you will simply be able to turn on as many instances as you need to cover the traffic...But probably automatic. Meaning you will run one instance and as the traffic increases more will automatically turn on. As the traffic spike subsides, the instances will turn off. There will be backups, etc. Also, some nice little features.

Nice features? If instances automatically turning on isn't enough...another feature slated is the ability to "phone in" more support. Meaning, Minervablog will e-mail out to your phone an SMS (e-mail to sms via phone carrier) and alert you to errors. It will also give you the ability to e-mail back (if you can from your phone, cool, else, where ever) to make certain changes. Changes like starting up or turning off an instance, and more...Oh yes, there's more. This is just a taste.

I've been working with sites, and quite specifically, some fairly unstable sites. It's extremely handy to have e-mail alerts. However when you get one, you really can't do anything about it if you're out on the road or somewhere away from a computer. Well you still can't fix major issues, but with some of the features I have figured out, you'll at least be able to "reset", or add/remove power to, your site.

Note: I do quite a bit of research and let me tell you what I'm working on (whether you see it or not) is very big and doesn't exist elsewhere. I know it doesn't because the technology is all too new. Similar solutions (that are probably just as good and some better) do exist, but the cost and availability is the catch. Thanks to new technology, that's changing. Make no mistake...If it's a site that needs a CMS/blog and has a ton of traffic - this IS the solution it needs. Didn't I say it was 2008?

Of course it may be 2009 by the time this all comes together...But I hope not! I hope someone out there is interested, I just may work faster and stay up later each night.

All I can say is WOW. These services are great. Since I was a moron and went back in my browser before saving...I lost this entire article and I'm retyping it. So I'd like to summarize which is good anyway. Why I'm posting is because I want to shoot down some inconsistent information across blogs out there. I didn't realize what was going on until I started to pay close attention to the dates the articles were posted.

So what is Amazon EC2 and S3? EC2 is like pay as you go virtual web hosting...similar to VPS, but better. It's scalable, flexible, and more powerful for the money you spend. Though you can get cheaper VPS....if you get huge traffic spikes, you'll be in trouble without EC2. So EC2 is great for startups that don't want to pay for a huge empty room before using it. You pay as you go. I think all hosting should be this way. As many people talk about, it is changing the landscape of web hosting and I truely think everything will be like this. Which brings up my question, why isn't Google doing this?

Anyway. S3 now is a file storage / content delivery network. VERY affordable at that...also pay as you go. The reason I talk about EC2 and S3 together is because you kinda have to. S3 is your backup because EC2 won't save your data if you turn it off. That's the biggest draw back. However I bet they'll come up with some sort of solution eventually...like they did (very quickly) for many other issues people had with EC2.

So to kill one myth (that's actually not hard to find out, most blogs point this out)...RESTARTING your instance won't kill your data...shutting it down and turning it back on WILL. I haven't tested this, but people confirmed it...I don't know why I'd restart my instance...I see this is as more of a fail safe.

The next misconception is the IP addresses. Amazon does give you IP addresses now...rather you request them. The "elastic" IPs work great. I pointed my A record from godaddy to my server intance after assigning an IP and voila. The thing is that Amazon will charge you (or penalize) for keeping an IP and not associating it with an instance....rightfully so and they should charge more and more over time. It's very dick to hold on to an IP and not use it. Right?

The next biggest complaint I see is going back to the persistent storage. When your site is large enough....and depending on what you're doing...you start to think about scalability issues like file serving. You want to use CDNs...of course YSlow gives you a better grade for it right? Ok well you don't HAVE to use them but if you're hosting up tons of media, you better. What better service than S3?? Transfer between EC2 and S3 is free. Backup away. What does cost though of course is to serve that data to visitors...but it's cheap.

What about non media file? What about databases, scripts, and other files? The killer of this whole issue/debate is ElasticDrive (I don't know of any competitors). With ElasticDrive, you can mount your S3 (or any NAS) like it was another drive. It's transparent and your web application will simply think it's writing to another directory on your system. This is WICKED easy to setup on Amazon EC2 using S3. From my limited experience with their FREE 5gb version, I'm VERY happy. In fact, for a small guy, it's perfect...If I wante to host my personal site for example. I'll NEVER go over 5gb. Plus I'm not high trafficked nor do I make a profit...so thank you ElasticDrive people for being good people.

This crushes a lot of fears people have. It would be super wise to host your site's files on S3 and simply keep the instance of EC2 for the server. Now...turn on more instances if you need to that are mirror copies and you know for sure that each instance is the same. Wow. You're already half way there to one great scalability solution.

So the cost is the next and final thing I'll get into. I won't get into any numbers...you can read about some examples on Smug Mug's blog and Digital Web Magazine's article about Blue Origin. However, what I can say is that yes - you can get a $30 VPS...but it won't be able to offer anything near what EC2 offers. Amazon gives you the best price for the power I've seen. I've looked into other solutions and the last thing I'll do here is list some competitors. I do my homework people =)

So the cost of running a small site let's say (most people's personal sites and hell their friends on the same instance for that matter) is going to be about $80 a month. It's 10 cents an hour to run. The data transfer will vary...but if you're not hosting up a ton of media, then you'll be ok. For companies that do host a ton of media, the cost is generally (from everything I can read) better than anywhere else you'll find. Also as a company, a couple hundred a month is not bad at all. As an individual developer playing around...I REALLY don't want to go over that $80 a month. =) ...and yes I do have 3 shared web hosts that I have to consolidate so I'm being silly.

So in conclusion...EC2 and S3 rock and Amazon is setting a trend here. No, they aren't the first, but they sure are the most talked about. Now here's some others that aren't talked about as much which you should look into:

http://www.joyent.com - Joyent (partnered with facebook)
They looked good, but I'm not sure about Solaris and they are very anti Amazon...they also were responsible for writing a false blog/article about Amazon's services. So they are dirty in my mind...though they could've just had bad information and not done as much research as they should've (completely possible, there is bad info out there if you're not careful). The other gripe I have with them is the pricing and while you can get a refund, they really want you to buy into the whole 2+ years thing which is ABSURD for technology.

http://www.slicehost.com (A good VPS great for developers)

http://dev.aol.com/api/xdrive - Aol's Xdrive (for storage)

Perhaps a real good link would be Real Metrics, which shows you some information on variou web hosts. You have to pay a small fee to get all the listings, but check out the free ones and what kinda info you can get.

Note: The reasons why I chose EC2 + S3 were many but primarily because I wanted to gain the experience of setting that type of soution up so I can be prepared for a need for something like that in my career. 

So I've been looking at the Western Digital Passport external self powered hard drives for a while. Very nice looking. Great price. So I thought! Today I found at Best Buy a Simpletech self powered hard drive. 320GB for about $150. Holy crap! It also looks real sexy too in red.

So I was skeptical. I figured it's a cheap drive or something's wrong. I figured it was a coin toss and just got it anyway. Glad I did! It's actually a western digital drive inside! I thought who the heck is Simpletech? Never heard of them before...but looking in the device manager I saw a familiar looking name "WD..." I know Western Digital drives are named like that and so I looked up the numbers...Sure enough it's a WD Scorpio 320GB SATA Drive...wow!

This is much better than paying the same price for 120GB or maybe 200 some...plus I think the case looks nicer. We'll see if I run into any issues, but so far it's great. I find it really interesting how all these hard drives you see have wacky prices. You aren't really paying for the size anymore...I think it's just a matter of how many were made and what was left over. Who bought how many, etc.

So first off I finally found, cleaned up, and posted the source code to the little de