/devices/pseudo/bitbucket@0,0:pseudo (Posts about Infrastructure)

Long Fat Networks

jmcp — Fri, 14 Jul 2017 19:00:00 GMT

Living in Australia generally means that you're on the end of a Long Fat Network (LFN), internet-wise. That's a serious technical term which is important to the networking stack when determining optimal data transfer sizes.

Two of my colleagues down in Melbourne are also with Aussie Broadband and using the top (100Mbit down, 40Mbit up) NBN speed tier. We also have company-issued hardware vpn units because we work from home fulltime. I was delighted at the bandwidth available from Aussie for our connections to work systems in the SF Bay Area, and when I had cause to update my systems to a new build I observed that it now took about 55 minutes on our media server, rather than the 80-90 minutes it took with the SkyMesh connection.

There was a fly in the ointment, however, because my colleagues and I calculated that while we should be getting 1Mb/s or more as a sustained transfer rate from the internal pkg server, we'd often get around 400kb/s. Since networking is supposed to be something Solaris is good at, we started digging.

The first thing we looked at was the receive buffer size, which defaults to 1Mb. Greg found https://fasterdata.es.net/host-tuning/other/ so we changed that for tcp, udp and sctp. While fasterdata document talked about using /usr/sbin/ndd, the Proper Way(tm) to do this in Solaris 11.x is with /usr/sbin/ipadm:

 # for pp in tcp udp sctp; do ipadm show-prop -p max-buf $pp; done

PROTO PROPERTY              PERM CURRENT      PERSISTENT   DEFAULT      POSSIBLE
tcp   max-buf               rw   1048576      --           1048576      1048576-1073741824

PROTO PROPERTY              PERM CURRENT      PERSISTENT   DEFAULT      POSSIBLE
udp   max-buf               rw   2097152      --           2097152      65536-1073741824

PROTO PROPERTY              PERM CURRENT      PERSISTENT   DEFAULT      POSSIBLE
sctp  max-buf               rw   1048576      --           1048576      102400-1073741824

To effect a quick and persistent change, we uttered:

# for pp in tcp udp sctp; do ipadm set-prop -p max-buf=1073741824 $pp; done

While that did seem to make a positive difference, transferring a sample large file from across the Pacific still cycled up and down in the transfer rate. The cycling was really annoying. We kept digging.

The next thing we investigated was the congestion window, which is where the afore-mentioned LFN comes in to play. That property is cwnd-max:

 # for pp in tcp sctp; do ipadm show-prop -p cwnd-max $pp; done

PROTO PROPERTY              PERM CURRENT      PERSISTENT   DEFAULT      POSSIBLE
tcp   cwnd-max              rw   1048576      --           1048576      128-1073741824

PROTO PROPERTY              PERM CURRENT      PERSISTENT   DEFAULT      POSSIBLE
sctp  cwnd-max              rw   1048576      --           1048576      128-1073741824

Figuring that if it was worth doing, it was worth overdoing, we bumped that parameter up too:

# for pp in tcp sctp; do ipadm set-prop -p cwnd-max=1073741824 $pp; done

$ curl -o moz.bz2 http://ftp.mozilla.org/pub/mozilla/VMs/CentOS5-ReferencePlatform.tar.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time Current
                                 Dload  Upload   Total   Spent    Left  Speed
  3 3091M    3  102M    0     0  4542k      0  0:11:36  0:00:23  0:11:13 5747k^C

While that speed cycled around a lot, it mostly remained above 5MB/s.

Another large improvement. Yay!

However... we still saw the cycling. Intriguingly, the period was about 20 seconds, so there was still something else to twiddle.

In the meantime, however, I decided to update our media server.

I was blown away.

23 minutes 1 second

Not bad at all, even considering that when pkg(1) is transferring lots of small files it's difficult to keep the pipes filled.

Now that both Greg and I had several interesting data points to consider, I asked some of our network gurus for advice on what else we could look at. N suggested looking at the actual congestion algorithm in use, and pointed me to this article on High speed TCP.

High-speed TCP (HS-TCP ). HS-TCP is an update of TCP that reacts better when using large congestion windows on high-bandwidth, high-latency networks.

The Solaris default is the newreno algorithm:

 # ipadm show-prop -p cong-default,cong-enabled tcp
PROTO PROPERTY              PERM CURRENT      PERSISTENT   DEFAULT      POSSIBLE
tcp   cong-default          rw   newreno      --           newreno      newreno,cubic,
                                                                        dctcp,
                                                                        highspeed,
                                                                        vegas
tcp   cong-enabled          rw   newreno,     newreno,     newreno      newreno,cubic,
                                 cubic,dctcp, cubic,dctcp,              dctcp,
                                 highspeed,   highspeed,                highspeed,
                                 vegas        vegas                     vegas

Changing that was easy:

# for pp in tcp sctp ; do ipadm set-prop -p cong-default=highspeed $pp; done

Off to pull down that bz2 from mozilla.org again:

 $ curl -o blah.tar.bz2 http://ftp.mozilla.org/pub/mozilla/VMs/CentOS5-ReferencePlatform.tar.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 3091M  100 3091M    0     0  5866k      0  0:08:59  0:08:59 --:--:-- 8684k

For a more local test (within Australia) I made use of Internode's facility:

$ curl -o t.test http://mirror.internode.on.net/pub/test/1000meg.test
  % Total    % Received % Xferd  Average Speed   Time    Time     Time Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  953M  100  953M    0     0  10.0M      0  0:01:35  0:01:35 --:--:-- 11.0M

And finally, updating my global zone.

 # time pkg update --be-name $NEWBE core-os@$version *incorporation@$version
            Packages to update: 291
       Create boot environment: Yes
Create backup boot environment:  No

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                            291/291     2025/2025  116.9/116.9  317k/s

PHASE                                          ITEMS
Removing old actions                       1544/1544
Installing new actions                     1552/1552
Updating modified actions                  2358/2358
Updating package state database                 Done
Updating package cache                       291/291
Updating image state                            Done
Creating fast lookup database                   Done
Reading search index                            Done
Building new search index                  1932/1932

A clone of $oldbe exists and has been updated and activated.
On the next boot the Boot Environment be://rpool/$newbe will be
mounted on '/'.  Reboot when ready to switch to this updated BE.


real    12m30.391s
user    4m4.173s
sys     0m21.496s

I think that's sufficient.

Weeeee – nbnco HFC and a new ISP!

jmcp — Thu, 22 Dec 2016 16:33:07 GMT

A few weeks ago we received a nice little letter from nbnco, advising that we would soon be able to get an HFC connection installed.

I was disappointed to discover that Internode was not going to offer either static IPv4 or an IPv6 service, let alone their Professional pack with NBN connections, and cast around for another ISP. Word of mouth got me to Skymesh and guineapig status (since they haven't formally started accepting HFC connections).

Since we're quite happy with the functionality of the Mikrotik CRS109-8G-1S-2HnD-IN, I declined purchasing a new modem; the nbnco installer provided the Blessed Arris cablemodem and all I needed then was for the connection to be activated.

Once it was, I very quickly reconfigured the mtik to do what I needed. Since Skymesh is using IPoE, I disabled the pppoe client activity. I also needed to change my firewall rules a little, to reflect that actions should be occurring on a different physical port (which I had to remember to take out of the bridge). Finally, a bunch of mucking around with IPv6 address assignment got me to the point where traceroute6.net was able to ping my server's designated public address. Yay!

While Skymesh doesn't offer PTR records, I'm still able to get A, AAAA, CNAME and MX records setup, courtesy of the free service at https://www.dynu.com.

Now that we're back up and running with a shiny new connection, I'm really pleased to see that http://beta.speedtest.net/result/5894030060 is pretty much what we get for both uploads and downloads on wired connections. Last week I celebrated that by having an hour-long video chat with a colleague down in Melbourne. With the 720p camera in my mbp, over the wireless connection. No dropouts, no blockiness :-)

I need to set up Smokeping, and see about a few other analytics things I can get going, but in general it's been a seamless transition.

We’ve got a new pool

jmcp — Thu, 26 Nov 2015 15:01:25 GMT

One of the features we looked for when we bought our house was an in-ground pool. As the kids have grown up and their confidence has increased, we've spent more and more time using the pool - last swimming season went from September 2014 to April 2015, with almost daily swims that stretched to being 2-3 hours long over the weekends in summer.

One aspect of our pool that we really did not like was the slate tiles around the side. While I'm sure they looked beautiful when installed, by the time we moved in (2007) they were looking tired and starting to shed. By the middle of this year we'd actually lost most of them, leaving some rather ugly concrete. The underwater surface was also starting to loosen, so the Barracuda would frequently pull bits up and leave them in the skimmer box.

We had to get it fixed.

It's been surprisingly difficult to get people to come and quote on a renovation job, and of those who did quote, some of them were more than the cost of a new pool. That didn't seem quite right. Eventually, J found Mark and his team at Sunseeker Pools, who not only quoted within our range, but helped us pick tiles and stone as well. He was also able to start within the next 3 weeks and estimated that it would take about 3 weeks to complete the whole job.

We had been umming and aahing about whether to get the pavers removed and replaced with tiles, thinking that would significantly add to the cost, however Mark's quote included that from the start so we were very happy. During construction we talked with him about our retaining wall, and he offered to build up the edge of the pool with a reinforced besser brick layer. This added about another 10% to the cost, but was very well worth it.

After a week's delay caused by the Brisbane Flu work started in earnest, with emptying the pool. That took about 24 hours, and showed just how bad a pool can get when you leave it unloved for a few months:

The next stage was to remove the pavers (we're planning on using them for some followup work in the rest of our outdoor area), jackhammer off the slate and the existing pool surface.

A day's worth of putting in formwork was quickly followed by the concrete truck (and pump), which delivered 3 cubic metres in about 45 minutes. That volume needed 3 days to cure, and then the tiles started going on. Mark and his team were very careful and precise in laying and cutting them all out, and then grouted them all at once. We went away to Ballandean (in the heart of Queensland's wine country) for a few days with friends, and when we came back the new pebblecrete had been laid, the waterline tiles and white river stone splashback had been installed and we just needed the acid wash.

The day after we got back, the acid wash subcontractor arrived and got to work - it took about 30 minutes to complete his task, and then we filled the pool. We've got about 40kL, and using town water from the hosepipe it took almost 24 hours to get up to the right level and it was time to put in chemicals. The chemical balance of Brisbane's town water is generally pretty close to what we need for a pool, so the only major concern that our pool shop had was that we add a lot of calcium. Apparently new pools can leach calcium, and it's nigh-on impossible to get back.

And with that, we have a new pool which we've been enjoying almost every day. It's nice to look at from the kitchen bench, too!

Thankyou very, very much to Mark and his team from Sunseeker Pools, and to Wayne and the gang at pool shop.

If you've got a suitable media-enabled browser, here's a video of the pool now - we really love the shimmery sparkly effect.

A recipe for running your pkg.depotd(1) server with SSL and Apache 2.4

jmcp — Thu, 26 Nov 2015 00:00:00 GMT

As part of my contribution to the darktable community, I provide the Solaris packages needed to run the application via a locally-hosted pkg repo. You can

# pkg set-publisher -g https://www.jmcpdotcom.com/packages/packages JMCP

and then install the bits very easily.

What was a little non-obvious (to me at least) was how to get the pkg.depotd process to only listen on a secured port.

If you look at the SMF properties for svc:/application/pkg/server, you will observe these two likely-looking candidates:

pkg/ssl_key_file
pkg/ssl_cert_file

They are not, however, what you need. Running pkg/server outside of svc:/application/pkg/depot is actually restricted to plain http because the backing framework here is CherryPy – and the version which pkg.depotd uses apparently has some issues with https.

Hmmph.

So I asked a few colleagues who have worked on our packaging system for assistance. Liane pointed me at https://docs.oracle.com/cd/E23824_01/html/E21803/apache-config.html, which was an excellent place to start. I did, however, need some handholding from Tim and eventually wound up with the following configuration which works with Apache v2.4.

Firstly, /etc/apache2/2.4/httpd.conf:

Follow the Apache docs for enabling ssl

Once you’ve settled on the port to run your pkg.depotd on, add a ReWriteRule like this:

RewriteEngine On
RewriteRule ^/packages$ https://%{SERVER_NAME}:83 [R,L]

Secondly, svc:/application/pkg/depot:default:

# svccfg -s application/pkg/depot:default
svc:/application/pkg/depot:default> setprop config/port = 83
svc:/application/pkg/depot:default> setprop config/ssl_ca_cert_file = "/path/to/your/SSL CA cert bundle"
svc:/application/pkg/depot:default> setprop config/ssl_cert_file = "/path/to/your/SSL cert file"
svc:/application/pkg/depot:default> setprop config/ssl_key_file = "/path/to/your/SSL key file"
svc:/application/pkg/depot:default> refresh
svc:/application/pkg/depot:default> quit

Thirdly, svc:/application/pkg/server:

# svccfg -s application/pkg/server add packages
# svccfg -s application/pkg/server:packages addpg pkg application
# svccfg -s application/pkg/server:packages
svc:/application/pkg/server:packages>addprop pkg/proxy_base = astring: "https://your.ssl.url.here/packages"
svc:/application/pkg/server:packages>addprop pkg/inst_root = astring: "/path/to/your/REPO/on/disk"
svc:/application/pkg/server:packages>addprop pkg/readonly = boolean: true
svc:/application/pkg/server:packages>addprop pkg/log_access = astring: "/path/to/access/logfile"
svc:/application/pkg/server:packages>addprop pkg/log_errors = astring: "/path/to/error/logfile"
svc:/application/pkg/server:packages>addprop pkg/standalone = boolean: false
svc:/application/pkg/server:packages>refresh
svc:/application/pkg/server:packages>quit

Once you’ve got those steps completed, it’s time to enable the services and add the publisher:

# svcadm enable pkg/server:packages pkg/depot:default
# pkg set-publisher -g https://your.ssl.url.here/packages/packages yourpublishername

Pretty simple (now that you know how).

Building a new system to run Solaris 12

jmcp — Sat, 31 May 2014 03:03:06 GMT

Several years ago I wrote about migrating my personal webserver from work's Ultra20 to my own Ultra20 M2. Sadly, that Ultra20 M2 (dual-core opterons, 2Gb ram) has just about given up being able to cope with what I need it to do.

Externally, I run the www.jmcpdotcom.com web and mail (client and server) services on it. Internally it's our media server, sharing out our collection of music, photos, movies and TV via NFS to the rpi. I just don't have enough ram in the U20M2 to do the needful, to the extent that when I've needed to transcode media files I do it from my workstation (Lenovo quad-core, 32Gb ram) across the gigabit backbone we're running. Ok, so that's a physical distance of about 1.5m, but it has still meant that I manually schedule intensive tasks for later at night when the family's asleep. That includes pkg update to whatever the latest available build is, since it regularly takes about 3 hours on the U20M2 rather than 15 minutes on my kernel zone or the laptop.

So... I had to built a new box.

I did a reasonable amount of research into what I could get, with the general provision that I should be able to bump the cpu and ram up with minimal effort. A high priority (but not mandatory) desire was for an onboard Intel gigabit nic ' they appear to me to be the most robust solution on Solaris today. I also wanted to migrate from the mpt(7d) SAS HBA to an mpt_sas if I could find one cheap enough. What I ended up with was this:

ASRock Fatal1ty H87-Performance motherboard
Intel Core I3-4150 (Haswell refresh, BX80646I34150)
16Gb DDR3 1600MHz ram
IBM 6 Gb SAS Host Bus Adapter for System x [46M0907] (an mpt_sas card)
Cooler Master Centurion 6' case
Corsair HX650 psu (80PLUS Gold)
Pioneer Blu-Ray combo drive

I'll comment on the specific features of the motherboard and case a little later.

This motherboard is UEFI, so I couldn't just take the disks out of the U20M2 and boot up the new system straight away. I pulled down the most recent Solaris 12 text installer usb image, booted and installed a basic bootable image. The cpio stage (the final stage) took a shade over 2 minutes to blat the image onto the disk. While it would have been really nice to then zfs send | zfs recv a snapshot of the U20M2 current BE across to the new box and try to boot it, I figured I had too many tweaks to make. I do so like customising my systems, y'see.

Still, it was very easy to copy across my cyrus-imap, mysql, exim, squid, privoxy and apache configurations. I also performed the rather simplistic operation of grabbing the list of installed packages from the U20M2 then uttering pkg install $LIST in the new system. Since most of them were in my local cache, that was a fairly quick operation to complete.

The service which caused me most annoyance was my dhcp server. I run this as a convenience for the hardware we have (and which visitors might bring) and while I do manual address assignment for certain systems, it's mostly completely automatic. After copying the old config files into place and creating /var/db/dhcpd.leases I thought I'd remembered everything. However, I'd missed one thing:

May 30 13:14:57 orrery dhcpd: [ID 702911 local7.error] No subnet declaration for ext0 (192.168.2.30).
May 30 13:14:57 orrery dhcpd: [ID 702911 local7.error] ** Ignoring requests on ext0.  If this is not what
May 30 13:14:57 orrery dhcpd: [ID 702911 local7.error]    you want, please write a subnet declaration
May 30 13:14:57 orrery dhcpd: [ID 702911 local7.error]    in your dhcpd.conf file for the network segment
May 30 13:14:57 orrery dhcpd: [ID 702911 local7.error]    to which interface ext0 is attached. **
May 30 13:14:57 orrery dhcpd: [ID 702911 local7.error]
May 30 13:15:04 orrery dhcpd: [ID 702911 local7.error] setsockopt: IP_PKTINFO: Bad file number

I've got two interfaces on this box, one physical, one vnic:

$ dladm show-phys; dladm show-vnic ; dladm show-ether
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             up         1000   full      e1000g0

LINK              OVER              SPEED  MACADDRESS        MACADDRTYPE VIDS
ext0              net0              1000   2:8:20:6e:f5:f5   fixed       0

LINK              PTYPE    STATE    AUTO  SPEED-DUPLEX                    PAUSE
net0              current  up       yes   1G-f                            bi

(The U20M2 has two physical nge interfaces; I'd like to get another physical gig-E nic for this new system at some point).

The thing I'd forgotten to do was tell the dhcp server which interfaces to listen on. I don't want it to listen on ext0, just net0.

$ sudo svccfg -s dhcp/server:ipv4
Password:
svc:/network/dhcp/server:ipv4> listprop config/listen_ifnames
config/listen_ifnames astring
svc:/network/dhcp/server:ipv4>

This, fortunately, is a very easy thing to fix:

svc:/network/dhcp/server:ipv4> setprop config/listen_ifnames = net0
svc:/network/dhcp/server:ipv4> refresh
svc:/network/dhcp/server:ipv4> end
svc:/network/dhcp/server:ipv4> quit
# svcadm refresh dhcp/server:ipv4 ; svcadm restart dhcp/server:ipv4

And off we went.

The next phase was to power down both old and new systems, and transfer disks. The IBM mpt_sas card only has one external SFF-8088 interface, but has four internal SATA connectors. I took four disks out of the 8disk Sans Digital TowerRAID TR8X+ (an 8 Bay SAS/SATA JBOD) and installed them inside the Cooler Master case, then booted up again.

All the ZFS pools imported without any hassles, apache/squid/exim/dhcp/cyrus-imap all came up immediately.. ALL GOOD!

Since one aspect of the zpool import was automatic enabling of the various nfs and smb shares, the new system was then declared to Be Sufficient(tm) and have WAF > 0.

Phew!

A note on the case

Having worked in on and around Sun (now Oracle)-designed hardware for quite a few years now, I'm delighted to see that concepts like disk sled rails have trickled down to the consumer end of the market. That said, why would you advertise a case as accepting 7 3.5" internal disks, but only supply rails for 4 disks? Also, I found it a bit difficult to get the SATA power and data cables connected to the internal disks without taking off the back side of the case as well. This is an area where a SATA disk backplane

comes in handy. I'd pay another AUD30 on top of the case price for a feature like that. Just sayin'.

A note on the motherboard

While I'd really like to have two onboard Intel gigE nics, I'm fine with just one. There's plenty of headroom for adding a PCI-E x1 (or even a PCI) card to handle that. There are the standard 4 DDR3 slots, currently populated with 4Gb dimms ' seems like plenty, at least for the moment. There's a serial port header if I want to get that set up (meh.... maybe), and the only niggle I have is that the Intel HD Audio doesn't seem to be supported by audiohd at the moment. I'll log a bug for that but really, who needs audio output capabilities on a server?

For El Goog: here's the output of prtconf -v for the board: ASRock_Fatal1ty_H87-prtconf-v.txt. I don't have lspci, but I do have scanpci output: ASRock_Fatal1ty_H87-scanpci.txt and ASRock_Fatal1ty_H87-scanpci-v.txt

I hope you find it useful.

A collection of laziness

jmcp — Sat, 16 Feb 2013 00:00:00 GMT

In preparation for my trip to Wellington next week, to present at the Multicore World 2013 conference, I’ve been building up a new vbox instance on my laptop.

To my great annoyance, I have to run Microsoft Windows 7 (x64) on my laptop, for three reasons: $EMPLOYER uses Cisco IP Communicator as a VoIP solution (no Solaris version), when I travel OS I call J and the kids every day using skype video calling, and the Intel HD graphics support in Xorg on Solaris is somewhat flaky.

So I run Solaris inside vbox, in seamless mode, and do all my coding and sysadminning tasks using Proper Editors(tm) and other interfaces.

Here’s the first bit of laziness: installations.

Two weeks ago I took delivery of a new workstation (a Lenovo M82, which promptly received a memory, disk and graphics upgrade) to replace the aging, hot and noisy Ultra40 M2 I’ve had since 2007. I downloaded the latest (at that point) USB live image from the internal site and happily booted + installed what I needed to… before remembering that the installer blats over any existing disk partitioning you’ve got. That was rather annoying, since I like to use raw slices for swap and dump devices (an old habit) rather than zvols. In order to keep my existing configurations, I cheated. Just a little (ok, rather a lot).

Technically, I should have setup an AI server and gone through my list of installed packages and made sure that I had them listed, blah blah blah. However, what I chose to do was zfs snap my current BE, boot the new box using the liveusb image (with livessh mode enabled), create my desired rpool config, then zfs send|zfs recv the snapshot… and mount the snapshot, make appropriate edits, install the bootloader and then reboot.

Simple!

It took about 25 minutes to transfer the snapshot from the Ultra 40 M2 to the Lenovo M82 (I’ve got a 16 port cheap-o gigE switch to connect the home systems), and about another 5 to go through the specific changes I needed in /etc (apart from those for svc:/system/identity:node and svc:/system/identity:domain. Installing the bootloader (grub2) was easy, but I had a weird problem getting the new system’s boot menu figured out. I ended up needing to remove /etc/zfs/zpool.cache and then recreating it (by running zpool status; zpool list would have done just as well). The SMF changes were done while I had the liveusb stick booted. To do this, after importing the new rpool I uttered

# svccfg
svc:>  repository /mnt/etc/svc/repository.db

and then selected the services I needed and edited them.

Prior to getting this Lenovo, I’d been using a non-global zone in the Ultra 40 M2 to serve out my webserver and handle mail. Part of this reconfiguration involved getting a miniSAS-connected jbod (from Other World Computing, shipped via Borderlinx) and physically moving my media and scratch pool disks into it. Then I attached the jbod to my Ultra 20 M2, and I just had to bring that up to date. So I snap’d the zone BE on the Ultra 40 M2, and send|recv’d it to the rpool on the Ultra 20 M2 – same liveusb+livessh sneakiness, same svccfg activities … and all done in under 30 minutes.

The longest part of the physical downtime was swapping the screws on the disk caddies when I removed them from the Ultra 40 M2 and inserted them into the jbod.

The second bit of laziness is not really laziness. I had a filesystem on the old box which I wanted to transfer to the new, and tried using rsync to move it. I’m a big fan of rsync, it does its job pretty well… mostly. However, this time it just wasn’t performing at all. I had about 8Gb to transfer, and after 30 minutes not only had I only seen 1.5Gb go across the wire, but trying to do any interactive work on the console of the other box was impossible. And I do mean, impossible – I got no response to keystrokes unless I paused the rsync. Then, having kicked myself because I really did just want the filesystem which happened to be on its own dataset, I snap’d it and kicked off a zfs send | zfs recv. All data then completely transferred in about 12 minutes. And with interactive performance going very nicely as well.

Now for the final bit of laziness:

Cut to yesterday, when I remembered that I really should get a new Vbox instance cons’d up on the laptop in preparation for my trips (I’ve got WLG next week, and SCA in mid-March). I still had the build 13 liveusb stick image, and I just could not be bothered downloading the equivalent ISO image so that I could boot the vbox with it. Running strings on an older Solaris 11 update 1 ISO showed the mkisofs command line that Release Engineering used to create it. So, I figured I would copy their example.

Firstly, I needed to mount the USB image in the filesystem:

# lofiadm -a /path/to/liveusb.img
/dev/lofi/1
# fstyp /dev/lofi/1
ufs
# mount -F ufs -o rw /dev/lofi/1 /mnt

Great… UFS. Ugh. Now to create a bootable ISO from it:

# mkisofs -v -R -J -o /tmp/bootable.iso -c .catalog -b boot/bios.img -no-emul-boot -boot-load-size 4 \
    -boot-info-table -eltorito-platform efi -eltorito-alt-boot -b boot/uefi.img -no-emul-boot \
    -N -l -R -U -allow-multidot -no-iso-translate -cache-inodes -d -D  /mnt

This commandline gave me exactly what I needed, so I moved the resulting ISO image across to the fileserver, fired up VirtualBox on the laptop and attached the image to my new vbox instance, and started installing.

Easy (and rather quick, too).

Notes to self: getting a ripped DVD image to play on the Beyonwiz

jmcp — Tue, 26 Jun 2012 00:00:00 GMT

Over the last few months I’ve been ripping our dvd collection[1] to files, so we can play them over the network on our PVR (Beyonwiz DP-P2) in a much more convenient fashion for us than finding the dvd and mucking around with the dvd player. C has been particularly enamoured of our Pixar collection: Toy Story; Toy Story 2; Monsters, Inc; Finding Nemo; (we have others, but she’s too young to watch them yet).

One aspect of this which has made things painful for J and I is that when we’re playing these Matroska files, the audio is out of sync with the video. The problem starts at the start of playback, and over the course of a movie gets to the state where the difference is 3 to 4 seconds. Most definitely headache-inducing stuff for us, but C doesn’t appear to notice or care – for now!

I did a bit of digging over the last day or so (you can see its relative importance in my stack of tasks!) and discovered that there are two components to the problem. Firstly, the Beyonwiz doesn’t seem to cope with Matroska files that have embedded chapter entities. Secondly, the Beyonwiz doesn’t seem to cope with more than one audio track (aac and ac3, for these) embedded in a Matroska file.

To fix this, I made a copy of my original Matroska file because when we upgrade our playback capabilities (to something with more grunt, and using either xbmc or Plex), this won’t be a problem. I used mkvpropedit to remove the chapter markers from the copy, and then used ffmpeg to transcode the copy to an mp4 and generally fix everything else.

Command-line wise, it went like this:

$ cp Toy_Story.mkv TS_1.mkv
$ mkvpropedit -c "" TS_1.mkv
$ ffmpeg -i TS_1.mkv -vcodec libx264 -acodec copy -ab 160 Toy_Story.mp4

Both ffmpeg and mkvpropedit came from the OpenIndiana SFE repos (http://pkg.openindiana.org/sfe and http://pkg.openindiana.org/sfe-encumbered)

Now the only problem I have to work around on the Beyonwiz is that it seems to have a memory leak with playing mkv or mp4 files, so if I try to play one and get the ‘Unsupported format’ error a reboot of the appliance is required. Tedious, but simple. Wish I had the source…..

[1] NOTE: this is our own, we-paid-hard-cash-for-the-disks DVD collection.

Some shuffling of deckchairs

jmcp — Sat, 16 Jun 2012 00:00:00 GMT

I’ve moved off the homeunix.com subdomain I’ve been using for the past few years. It seems that not only did I not read the email from DynDNS.org in June 2010, but that requesting an AAAA record for jmcp.homeunix.com was sufficient of a change to take that subdomain out of the grandfathered list and also turn off domain wildcarding.

Boo.

I figured that I might as well go for a proper vanity domain, so here it is: jmcpdotcom.com.

I think I’ve fixed up most of the links in various posts, but if you find one that’s still pointed incorrectly I’d appreciate a comment or email to advise.

Xpra 0.3.0 Solaris 11 package (x86) now available

jmcp — Wed, 23 May 2012 00:00:00 GMT

Antoine made the announcement about Xpra getting the bump to 0.3.0 overnight. I’ve now built an x86 package of it, and it’s available from my usual package location.

I built it with CFLAGS="-xlibmil -xlibmopt -xO5" using the Oracle Solaris Studio 12.3 compiler, but again, without vpx or x264.

Solaris 11 package for Xpra (x86)

jmcp — Tue, 08 May 2012 00:00:00 GMT

A while ago I mentioned that I had stumbled upon Xpra.

I’ve started building a Solaris 11 x86 package of Xpra, purely because it’s useful to me and I hope others might find it useful too. I’ll try to get the manifest and build script included in the Xpra svn repo, but in the meantime you may grab a copy Xpra_ips.tar at your leisure.

I haven’t included the vpx or x264 support options since they require codecs which I haven’t built and I’m not interested in having to worry about their legalities.

Apart from that, it’s a vanilla build.

The gzipped p5p is available here.

Please let me know if you have problems with it – especially any missing dependencies.