Luke Sheldrick's Blog » linux

KVM + KSM = Big Bag of Win

Luke Sheldrick — Mon, 26 Sep 2011 20:45:09 +0000

I bought a HP Microserver a few months ago, to expand my media storage at home. HP had a promotion where you’d buy the server, and they’d give you £100 cash back. Not bad considering you can pick them up online for around £220. £29 for 8GB of DDR3 for it, and it’s quite an ample home server. I bought four 3TB disks, and have them in Raid-5 for the main storage, and I also have an E-Sata disk shelf with four more 2TB disks, also in Raid-5. The main storage array gives me around 8TB of usable storage, and the other around 5.5TB, so I have quite a bit at home to be getting on with.

A colleague at work today pointed out that HP had extended the offer until the end of September, so I thought I would pick up another one. My reasoning is they’re bloody cheap for what they are, and it’ll make HA testing in my lab a lot easier, with two identical machines.

However I decided to see actually how loaded the server was with what I run on it. The tasks that usually run on it backups of my various colos, streaming to the AppleTVs at home, transcoding up to 8 channels via the DVB cards, and downloading various media online. Well, basically, it isn’t breaking a sweat.

It’s currently running Ubuntu Server 11.04, which has pretty good support for KVM, so decided to spin up a few VMs to see how it handled it. I wasn’t overly optimistic, as AMD Turion™ II processor, which is basically designed for netbooks. However after spinning up a another Ubuntu Server instance, the server seemed to have no issue with this at all.

At that point I decided to clone the VM a couple of times, and then spin them all up to load test. 4 clones later, they all booted fine, and the box carried on with no real issues. I showed another colleague at work, and he’d asked how it would cope with Windows guests. The reason he was interested is he has just completed his MSITP qualification, and had quite a bit of difficulty running all the required server instances using his laptop and VMWare Workstation.

This evening I installed a base install of XP, and cloned that 4 times. Again, no real issues, so I decided to give every host 1G of memory (the server only has 8GB total), and see how KSM got on with the VMs. At the moment, it’s still scanning through memory pages, however it’s been running for around 20 minutes, and it’s already reclaimed 2.5GB of memory. No doubt given it some more time, it’s going to reclaim even more.

To say I am impressed by the Microserver, and KVM + KSM would be an understatement.Think I’ll setup a HA cluster when I get the other server, and see how that compares the likes of ESX.

Below just the quick setup I was playing with this evening.

MythTV – Stuck Queue

Luke Sheldrick — Thu, 03 Feb 2011 22:06:07 +0000

I use DHCP at home, as most do, for convenience. Last week I changed the router at home to one compatible with DD-WRT as I was setting IPv6 up at home (something I should blog about later). When I set it up, until I migrated the old DHCP config over, everything on the LAN that had static DHCP allocations went a bit crazy. Once that was resolved everything went back to normal… or so I thought.

However I’d noticed that MythTV wasn’t recording anything. When looking through it had listed all of it’s encoders as remote and “currently not connected”, interesting I thought. It looked like somewhere along the line that box had got 192.168.5.6 instead of 192.168.5.5. So it didn’t think it was, well it’s self. Changing the DHCP assignments quickly resolved this.

A week later I noticed whilst programs had been recording, they were all sitting queued in Job Queue. Restarting the backend process nor did restarting the box. So I had a quick look on it’s DB and noticed the recordings were showing as remote, not local as they should be. Delving a bit further I noticed that it was listing the recordings as being recorded on “sidekick.oob.an0key.net” instead of “sidekick”. I’d set the domain on the router, and it was happily distributing via DHCP.

So to resolve my issue, I changed the hostname manually, back to sidekick.

To do this on ubuntu is pretty simple, edit
/etc/hostname
with the desired hostname and then
/etc/init.d/hostname.sh restart
However I was now stuck, with all the recordings showing as being recorded on the ‘remote’ host. The only way to fix this would be to edit the database manually. This is where it the less adventurous may want to stop.

So log on to your MySQL instance, it’ll prompt for MySQL’s root password.
sidekick:~# mysql -p -u root
Use mythtv’s database named mythconverg in my setup.
use mythconverg;
Then print the structure and contents of the jobqueue table.
select * from jobqueue;
This is what I was presented with.
mysql> select * from jobqueue; +-----+--------+---------------------+---------------------+------+------+-------+--------+---------------------+-------------------------+------+-------------------------+---------------------+ | id | chanid | starttime | inserttime | type | cmds | flags | status | statustime | hostname | args | comment | schedruntime | +-----+--------+---------------------+---------------------+------+------+-------+--------+---------------------+-------------------------+------+-------------------------+---------------------+ | 284 | 1001 | 2011-02-03 17:11:00 | 2011-02-03 17:15:01 | 256 | 0 | 0 | 272 | 2011-02-03 17:15:53 | sidekick | | Successfully Completed. | 2011-02-03 17:15:01 | | 283 | 1004 | 2011-02-02 22:00:00 | 2011-02-02 23:05:00 | 256 | 0 | 0 | 1 | 2011-02-03 17:25:24 | sidekick.oob.an0key.net | | | 2011-02-02 23:05:00 | | 282 | 1001 | 2011-02-02 21:00:00 | 2011-02-02 22:00:01 | 256 | 0 | 0 | 1 | 2011-02-03 17:25:24 | sidekick.oob.an0key.net | | | 2011-02-02 22:00:01 | | 279 | 1007 | 2011-02-01 21:00:00 | 2011-02-01 22:00:01 | 256 | 0 | 0 | 1 | 2011-02-03 17:25:24 | sidekick.oob.an0key.net | | | 2011-02-01 22:00:01 |
So I can see the that the job ran fine that came in after I changed the hostname. So what I decided to try is to update the hostname, and also the scheduledtime too as it was in the past.
mysql> update jobqueue set -> status=1,hostname="sidekick",comment="",schedruntime="2011-02-03 17:30:01" where status=1;
So this will update the hostname and schedruntime for anything with status 1 (which means queued). I ran that and restarted mythtv-backend, and all the queue was then worked though.

Job done. Hopefully this may help someone else in a sticky position.

My ‘cloud’ IM setup

Luke Sheldrick — Wed, 18 Aug 2010 23:00:19 +0000

For a long time now, I’ve had a number of different applications I use to connect to various different IM (instant messaging) mediums, such as Irssi for IRC, Pidgin for Jabber/MSN/etc on my desktops/laptops and usually BeeJive on my iPhone when I’m out and about.

This has all ‘kinda’ worked, it hasn’t been the most elegant of solutions, but it did do what I need it to. Until you add something like the iPad to the mix.

The problem I had, was I’d be signed on to say MSN on my MacBook, and then decide I’ve worked enough for the day, so sign off. Then whilst watching TV in the evening, want to send a message to someone, who only uses MSN, so grab my iPad and sign on via BeeJive. Well that kind of works, except, if I forget to sign off BeeJive, it keeps me online on their ‘cloud’, so I can connect back at any time via the iPad, only the iPad though.

If someone was to send me a message when the client is closed on the iPad, sure it sends a push message to it, but no other device. If I go out, and I know someone has sent me a message, I can’t connect to it via BeeJive on the iPhone, as it doesn’t keep them in sync. Same as if I wanted to log in with the MacBook, I can’t see what that message was.

So I decided this doesn’t work how I want and need it to. So I set about designing my own solution. The aim was to set something up running on my infrastructure, at little or no cost (not always easy when you add any iOS devices in the mix), and reliable.

What I came up with, works really well for me, so thought I’d document what I put together.

The server components run on a machine that is always on, the is the core of the solution. The underlying OS I used is a Fedora 14 box (so Fedora rawhide – the development branch). The packages I’ve used are widely available on pretty much every Linux distribution, ports (OS X, *BSD), so the OS here, really doesn’t make much of a difference.

Server side:
Irssi – a cross platform IRC client.
ZNC – a cross platform IRC bouncer.
BitlBee – a cross platform IM gateway for IRC.

Client side:
Colloquy Mobile – iOS IRC client – use this on the iPad and iPhone.
Adium (Beta) – OS X IM client – use the beta version as it supports IRC.
Pidgin – a cross platform IM client – use this on any Windows or Linux machines i happen to be on.

Again the client side really doesn’t matter, just as long as you have something you can connect to an IRC server with, it should be all good.

The setup uses BitlBee to connect to MSN, G-Talk, Jabber, Facebook chat, and pretty much any other IM network you’d want to connect into. BitlBee presents it’s self as a IRC server, and you contact list is all shown as a room. When you chat to someone you do so as you would traditionally. That part worked a treat.

At first I just had Irssi running with the proxy module enabled, this allows you to reconnect into your Irssi session, with a local client, and ‘pose’ as remote session. This worked well, so I just left Irssi running in a remote screen session, then when I wanted to connect in and chat, I would just open say Pidgin locally, and it would connect to all the rooms the Irssi session had.

This was fine, however, Irssi on it’s own doesn’t support playback. What I mean here is, if someone had sent me a DM, Irssi would have it, but when I logged in with Pidgin, Irssi wouldn’t send me a copy of that message, it only forwards new messages. For these kind of features, I’d need to employ a bouncer, not the thuggish type that stand outside nightclubs, but an IRC bouncer.

I tried a few, but settled with ZNC. I should point out here, that when using ZNC, there isn’t really any need to use Irssi in the equation any more, I just kept it as have a few custom scripts, and all my historic logs are there, so decided to keep it.

ZNC will connect to all your favourite IRC networks, keep you online, and when you connect to it with your client, it will replay all the conversations and DMs you missed since you last connected in. this was exactly the functionality I was looking for.

There are also a host of other cool things you can do with ZNC, so I have mine configured to set me away everywhere 5 minutes after my client disconnects. Also if you team colloquy mobile up with ZNC, you can have it push message your iPhone or iPad if you’re mentioned in a chat, or if someone sends you a DM, I have mine set to only do this if there are no other clients connected, else when I’m at computer having a conversation, both my iPhone and iPad have a bit of a push message spasm. This push message function was exactly what I was looking for to replace BeeJive, except this pushes it to all my devices, not just the one that has my account singed in.. neat I thought.

This my no means is the simplest way to set up your MSN, but for if you want all the prerequisites I did, it really works.

The server topology may be a bit complex so have (for my sins) put together a diagram of how it’s set out, along with a few screenshots.

Any questions, please feel free to ask.

Synergy

Luke Sheldrick — Tue, 17 Mar 2009 02:41:49 +0000

A friend told me of a magical cross platform application, which I’ve been using for a while, thought I’d share. Synergy;

“Synergy lets you easily share a single mouse and keyboard between multiple computers with different operating systems, each with its own display, without special hardware. It’s intended for users with multiple computers on their desk since each system uses its own monitor(s). ”

The picture above shows my very small desk, sporting 2x 22 Samsungs, and my Dell D420. Under the desk is an old HP desktop powering the two monitors (and soon to be 9′ touch screen above them, when I get the drivers setup). I rarely use the D420 these days, as I use my MacBook or Advent 4213 notebook when out and about, so didn’t mind giving it up to the desk.

I can use the keyboard and mouse on the laptop, to control both it, and the bigger screens. The neat thing is, with the config setup correctly, I can just move the mouse upwards and the pointer magically moves to the other screen, and the keyboard then controls that computer too. A few other neat options are to have a shared clipboard, and to activate the screensaver/lock screen across all computers with each other.

So on my tiny desk, I can fit in quite a lot of screen real estate. As for using the laptop as a primary keyboard, I started to do this at work when the hotdesk keyboards started to freak me out, and using the laptop keyboard is actually quite nice to use.

Have tested Synergy with os x, and all worked pretty flawlessly there too. Not had a chance to test under windows just yet, as rarely use it these days, but from what I’ve read it all works fine.

New home

Luke Sheldrick — Sun, 07 Sep 2008 12:03:46 +0000

In addition to moving my dns, to my self hosted infrastructure, I am also moving quite a few things away from hosting on Windows.

I eventually want to get rid of my Exchange 2007 server, as to be honest it is so much of a resource hog, and for a small user implementation, it’s far too OTT. I need to find a suitable alternative to sync all my devices with however (Laptop(s), iPhone, various Windows Mobiles) I am sure there is something out there that will do the job.

As for this blog, it was hosted on a Windows 2003 Server, the same one as the Exchange server, where performance isn’t great. So now it’s hosted on one of my Linux (Fedora) servers. I am quite happy with the transition, other than a lot of the asp back ends, now have to be re-written in perl or something similar… effort.

I am not too sure if it’s performance has been improved… let me know

Netgear SC101

Luke Sheldrick — Sun, 11 May 2008 15:25:33 +0000

Finally, some clever geek type person has writen a set of tools to be able to mount these horrid things under *nix.

I bought mine a couple of years back, bud didn’t realise you had to use a horrid sluggish windows app to be able to mount the thing.

Netgear use a tool called “Z-SAN” under windows.

However you can now mount the drives under *nix, there are a few draw backs however, you first have to partition te drives under windows, not a huge problem, as my desktop is a dual boot.

Secondly you can’t use passwords on the device. Again not a huge problem, I just stuck it on it’s own vlan, and restricted access to it, so only my linux NAS server could see it.

Then on my NAS server (FC9 + NFS + SMB) I used the utils the toolset provides, mounted the two drives, formatted, and mounted. Bob is your mothers brother it all works.

I just share the drives out now using my existing NAS (NFS + SMB) and supprisingly works really well.

Time will see how stable they are.

If you want to do this too, the code can be found at google code:

http://code.google.com/p/sc101-nbd/

Goodbye windows, hello linux

Luke Sheldrick — Sat, 26 Jan 2008 03:37:15 +0000

Well tonight decided that both the XP / Vista (dual boot) installed on my main desktop at home were too annoying and memory hogging.

Decided to give a Linux distro a go, as have become quite accustomed recently to the unix way of doing things (well have a few cent servers running, and have a macbook).

Went for Ubuntu Gutsy (7.10), and after fighting with it to play nicely with my nVidia graphics card, and Dual HP monitors, it seems pretty cool.

Still gotta get it to play with my web cam (shouldn’t be too difficult) and my Asus TV Tuner card, which so far doesn’t look too promising on the t’internet.

Oh and… can’t sleep again. Grr!

Mail Issues, it’s all a matter of time.

Luke Sheldrick — Sun, 09 Dec 2007 00:24:03 +0000

Last week, I started working on a new mail platform. I had in place, an Exchange 2003 box at home, and a backup mail server running Exim, based in Global Switch London.

I utilised Greylisting on both, however found the linux version ten times better. So set myself a task to setup a new front end, in front of my main Exchange box. Ruled out getting rid of it totally, as I personally prefer the features of Exchange (Push mail, calanders…etc).

So I setup a virtual machine (CentOS 4.4) on one of my virtual hosts, configured exim, with greylisting, and to relay all mail to my domains, to the internal IP of my Exchange box. I also wanted to setup a test Exchange 2008 box at the same time, so anything to my development domain, would go to a different internal IP.

After a few tweaks, magically it all worked… until I started to get NDR’s stating that mail was being delayed to my domain. A few telnet tests later, discovered that if you connected from a host with no rDNS, my mail server took 30 seconds to time out doing a reverse DNS check. Many mail servers would have timed out by this time. A quick tweak of my exim config to lower the DNS time out, from 30 seconds to 5, cured the problem.

For anyone wanting to do this, edit the exim.conf lines

rfc1413_hosts = *
rfc1413_query_timeout = 30s

rfc1413_hosts = *
rfc1413_query_timeout = 5s

Luke Sheldrick's Blog » linux

KVM + KSM = Big Bag of Win

Related Posts (auto generated)

MythTV – Stuck Queue

Related Posts (auto generated)

My ‘cloud’ IM setup

Related Posts (auto generated)

Synergy

Related Posts (auto generated)

New home

Related Posts (auto generated)

Netgear SC101

Related Posts (auto generated)

Goodbye windows, hello linux

Related Posts (auto generated)

Mail Issues, it’s all a matter of time.

Related Posts (auto generated)