Images Belong in the Database
Posted by Sam
Logical Reasons
The more web programming and system administration I do the more I'm convinced that images and other forms of uploaded content belong in the database. I've pretty strongly suspected this for many a years but it's becoming more and more clear that this is just how things need to be. There are many reasons. Some are technical and some are logical, but all signs point to loading images in the database. Below are just a couple.
I'll start with the logical reason first and then get to the technical reasons. Logically all other forms of dynamic content will reside in the database yet programmer after programmer insists on putting images and other uploaded content on the filesystem. Now you've got some content in the database and it's tied to content on the filesystem. Suddenly the nice clean line is broken. You've blurred the lines on who does what and where it goes. What usually happens is programmers forget to clean up after themselves so files references are deleted in the database but still exist on the file system. You know why this happens? Because you shouldn't have put it there in the first place. Repeat after me - dynamic content in the database and application files in the filesystem. Got it?
Technical Reasons
So know that we've got the logical reasons behind us what are some of the technical reasons? For starters your database(s) will always be in sync. That's what they are meant to do. That's what they have to do. Your database(s) have to be synchronized. Your site depends on it. It doesn't matter if it's a simple site with a single database server or a huge site with a dozen database servers. You have to keep your database servers in sync. And databases are good at it. It's much much easier to scale out database servers than file servers. Having all your dynamic content in the database keeps everything in sync.
Some people will wrongly argue that having files in the database will cause slowdowns. To that I say....um maybe, but only if you're doing it wrong. Databases are very fast and have excellent caching but it's highly likely that multiple web servers can server images straight from disk faster than a database. Well that's great, but images still belong in the database. And in this case you can have your cake and eat it too. Check out this article about caching images on the file system with Rails. Hrm, one extra line of code lets you cache your image on the file system. Doesn't seem to bad to me. Elegant, simple and lightning fast.
The Reason for the Rant
Time after time I'm responsible for deploying other people's crappy software. They invariably come up with some stupid solution that lets them click the checkbox that says their product will work on a load balancer, but they neglect to tell you upfront what ridiculous hoops you have to jump through to get it work. The most recent product is called Ektron. I can't say what it's like to work with from a programming stand point but from an admin stand point it's a nightmare. Instead of loading the images in the database like they should they instead force you to share out files and do this dumb little virtual directory linking to the other servers. It's just annoying and not even close to elegant, but then I have yet to see a CMS system that is so I wasn't surprised. Thankfully the Ruby on Rails team understands what it takes to scale apps and they provide you with a nice foundation. If only the rest of the world would catch up.
Tags: rubyonrails web rant
The 5 Minute Guide to Securing Solaris
Posted by Sam
Disclaimer
This is a super quick guide to securing a Solaris server. It's not in depth and I highly recommend taking a deeper look at the technologies that are used.
Install Solaris Security Toolkit (formerly called JASS)
- Download Solaris Security Toolkit
- install and run
# pkgadd -d . SUNWjass
# /opt/SUNWjass/bin
# ./jass-execute secure.driver
A ton of stuff will scroll by. If you need remote access be sure to edit the tcp wrappers allow file otherwise you'll be locked out of the box. Also, if you remotely log in to SSH as root you'll need to allow root access and restart SSH. I always double check that I can remotely log in before closing the current console.
Links for Solaris Security Toolkit
Activate BSM Auditing
Auditing will let you watch as little or as much as you want on your box. It's also zone aware so you can see exactly what's going on in a specific zone. Here's a quick run through on how to turn it on and what I'm monitoring on my boxes. Keep in mind if you have a busy box these settings can produce some very large logs and slow down your system. Take a look at the links to learn exactly what these settings do and then decide how much logging you need.
- Edit /etc/security/audit_control so that it looks like this:
dir:/var/audit
flags:lo,ex,ad,pc,fm,fw,-fc,-fd,-fr
minfree:10
naflags:lo,ex,ad - Edit /etc/security/audit_startup so it looks like this:
/usr/bin/echo "Starting BSM services."
/usr/sbin/auditconfig -setpolicy +cnt
/usr/sbin/auditconfig -setpolicy +argv,arge
/usr/sbin/auditconfig -setpolicy +zonename
/usr/sbin/auditconfig -conf
/usr/sbin/auditconfig -aconf
- Run the bsmconv script
/etc/security/bsmconf - Add the following line to the crontab
0 0 * * * /usr/sbin/audit -n - Reboot and check out your newly created logs in /var/audit
Links for BSM Auditing
Showing Blastwave Packages with Upgrades Available
Posted by Sam
I love love love Blastwave. I couldn't imagine using Solaris without it. One of the annoying things though is if you want to see if newer packages are available you have to wade through every package that they have. And they have a lot. So here is a super simple one liner for showing just the packages that need to be upgraded.
pkg-get -c | grep -v "Not installed" | grep -v SAME | \
mailx -s "Blastwave Updates" yourname@example.com
And since you are probably using Solaris Zones here is a simple script that will look in all non-global zones and email out a report. Currently this sends one email per zone but that could easily be changed to send one email per physical box.
#!/usr/bin/sh
for zone in `/usr/sbin/zoneadm list | /usr/bin/grep -v global`; do
zlogin $zone "/opt/csw/bin/pkg-get -c | \
/usr/bin/grep -v \"Not installed\" | \
/usr/bin/grep -v SAME | \
/usr/bin/mailx -s \"Blastwave Updates for $zone\" yourname@example.com"
done
The Initial release of ZoneMan
Posted by Sam
I've been wanting to switch from BIND to PowerDNS for a while now but I haven't been able to find any admin tools that worked the way I wanted them to. This is the problem when you can write software. It seems that hardly anything is just right. Since PowerDNS can use a SQL back-end it seems that everybody wants to create a web admin which is fine for lots of things. One of the things I'm working on now is something I like to call Agile Systems Administration. If nobody has coined the term to date I'm coining it now. Basically I'm looking for ways to automate whatever I can. Since I deploy a LOT of sites DNS was an obvious choice to start automating.
This is where ZoneMan comes into play. It's a command line tool that works with MySQL, PostgreSQL or any other database that Rails works with. The usage is pretty simple. Here's an example of adding a new record.
$ ./zoneman add_record example.com www.example.com A 10.1.1.1
That's it. To break out what we just did. add_record is the command. Example.com is the domain that you are adding the record to. www.example.com is the record itself. 'A' is the record type. And lastly '10.1.1.1' is where the record is pointing to. Using that example you can see how easy it is to script new site setups from the DNS perspective. You can wrap that command in a bash script and make it even shorter.
example.com has been created
The initial release is already hosted at RubyForge. It's very rough and untested. I've also set up a project page on this site. If you have suggestions feel free to leave them in the comments. I hope to make this a very useful tool for managing PowerDNS. Enjoy.
My Love Hate Relationship with MySQL
Posted by Sam
The Hate
It seems everybody is using MySQL these days. It's getting plenty of press, especially lately with them getting bought by Sun and all. I've been using MySQL for at least 10 years. Only recently did I throw out an old O'Reilly book that cover MySQL version 3. But lately I've been pretty dissatisfied with MySQL and it's annoying limitations. For instance, why do we need so many database engines that all do different things? MyISAM, HEAP, InnoDB, NDB, Falcon and NitroEDB to name but six and I've probably left out a couple. But with all these options there isn't a single option if you want something as simple as say, transactions AND Full Text Indexing. You can get Full Text Indexing with MyISAM and transactions with InnoDB but if you want/need both then you are out of luck or relying on a third party hack. No offense meant to the Sphinx folks with that 3rd party hack comment, it's just that I want to tightly integrated into my database and I'd prefer not to have compile, install and configure another application for FTI. Then there's also the well documented performance limitations concerning the use of lots of cores.
A while back I was working on a project that I needed a modern database engine and Full Text Indexing. The searching was going to be a critical part of the site so that's what I focused on. I spent hours researching FTI on MySQL only to walk away realizing that it just wasn't going to work without going the MyISAM route or some third party hack. And the more research I did on MyISAM the scarier it got because it apparently has some serious scaling issues regarding FTI. All this research led me to one thing. Postgres. Postgres has one database engine that does everything. Oh and these days it happens to be really fast. Often faster than MySQL. Now that it's fast and they have the vacuuming stuff more automatic Postgres is a great database and people really should be looking into it for replacing MySQL IMHO. But Postgres aside, the more I research MySQL the more convinced I was that the company just doesn't get it. They seem to be more interested in creating more engines than in making a single engine that does 90 - 95% of what most people want.
The Love
So what do I love about MySQL? Well I'm glad you asked. The other day I was talking to a co-worker and I was telling him how excited I was about our new MySQL cluster. We have a master/slave configuration set up so recovering from a database disaster should be just a quick reconfigure away. And if we need the capacity we can point sites at the slaves. And we can do backups on the slaves without taking down the site because the database is locked. Needless to say I was excited. So many possibilities. I like everything to be redundant. Everything. So the database redundancy was a big deal. And MySQL makes it easy to set up. It really is nice. So after I finished extolling the virtues of our set up he asked me if this means that I no longer hate MySQL. This is where I realized that I had a love/hate relationship with it. From the admin side I love it. MySQL is easy to work with, runs anywhere and makes some cool stuff (like clustering) uber easy. From the development side I hate it for the reasons I stated above. If I'm programming I'm going to lean towards Postgres, but if I'm wearing my sys admin hat I'm going to lean towards MySQL. It gets really confusing when I'm doing both roles as if often the case. Hrm, maybe I need to look into Postgres' clustering and then it will be an easy choice.
Where to now?
I'm really hoping that Sun can do something to turn around MySQL because for a lot of projects it's a really good database but for just as many it's an awful choice. They really need to fix the scaling issues and stop with the database engine madness. A couple of database engines are OK, but come on. They are just out of control. Pick a couple of engines and just pour all your time into supporting them. And the scaling issue is a no brainer. Lots of cores are the norm and MySQL needs to get with the times or risk being left behind.
Tags: mysql