Perl glob performance

I was working on a project and found that my file server that I was pulling data off of seemed a little slow when getting the list of filenames. I was using 'glob' to get a list of files from the server and then kept the 10 most recent (the file names were sequential so a reverse sort gave that simply). But I found that the glob was taking a long time (there were about 4000 files in the directory and I could not remove them due to business rules). So I asked on the perl irc channel and found that they recommended using opendir/readdir over the glob call since glob does a stat on everyfile, making it run slower. So replacing (1) with (2) resulted in a 10 times speed up of my code for just this part of the code.

(1)

@files = glob "d:/path/to/directory/*";
$i = 1;
foreach $file (reverse(sort @files)) {
print $file . "\n";
$i++;
if ($i > 10) {
last;
}
}


(2)
opendir DH, "d:/path/to/directory/";
@files = readdir DH;
$i = 1;
foreach $file (reverse(sort @files)) {
print $file . "\n";
$i++;
if ($i > 10) {
last;
}
}
closedir DH;


I found this change useful so I thought I would share it.

Recover a failing disk using a Linux Live CD

I had a friends disk fail the other day and this is how I attempted to recover it.

First the situation, a Windows XP machine that was having trouble booting. I eventually got it to boot but shortly after that the disk in the machine started giving random read/write errors. So here is how I tried to recover it.

Tools:
Fedora Live CD (I used Fedora 8 cause that is what I had)
A Fedora Linux based file server with lots of diskspace or another hard drive.

My file server is setup as a samba server, so a Windows Server could be used in place if you have one.

• Boot the machine using the Live CD
• Switch to running as root
• Install dd_rescue and samba-client (on fedora the command is 'yum install dd_rescue samba-client')

Note: if you have to reboot, you will have to install these tools every time, unless you are using a Live CD on a USB Flash drive.

• Mount the file server using cifs (this is the Samba/Windows Server protocol)

Create the local mount point
mkdir /media/public

Mount the fileserver
mount -t cifs //[hostname or ip]/[share] [local path] -o username=[id]
e.g. mount -t cifs //192.168.0.210/public /media/public -o username=anyone
It will ask for a password, and you should give the password for the anyone account.

If you are using another disk you can just mount it. The device name will probably be /dev/sdb1, but check dmesg to be sure.

mount /dev/sdb1 /media/public

• Use dd_rescue to copy the partition to a file on the file server

dd_rescue -A /dev/sda1 /media/public/disk.img
This may take several hours or even days and will take up large amounts of diskspace. In my case it was 2 days and 40GB.

• Once the rescue is complete you can mount the disk on the server with a loop mount or copy it back to a new disk

Loop method (requires Linux)
Create the local mount point
mkdir /mnt/disk

Mount the image using a loop device
mount -o loop -t ntfs-3g [path to disk.img] [mount point]
e.g. mount -o loop -t ntfs-3g /data/public/disk.img /mnt/disk

Copy the needed files from the /mnt/disk directory.

If the disk image is not mounted, you might be able to boot the image using qemu with the following command:
qemu -m 512 -hda /data/public/disk.img

Copy method
(reboot back into live CD with the new disk installed and partitioned, install the tools,mount the server/disk again)
dd_rescue -A /media/public/disk.img /dev/sda1

If the new disk is larger than the old disk (it cannot be smaller) then you may need to use parted or Partition Magic to resize the partition. Also, you may need to run 'fixboot' and 'fixmbr' using the Windows Install CD in rescue mode to get the new disk to boot. I would recommend doing a clean install and just copying the data files back since the OS and applications may have been damaged by the disk errors.

Hopefully this is useful, but remember that depending on the level of disk failure it may not work.

Gnome MPlayer 0.9.2 is out

I released gnome-mplayer 0.9.2 today. Normally I wait about a month between releases, but since 0.9.1 had so many annoying little issues. I decided to try and push 0.9.2 out a little quicker. Turns out even with this short release I still got 90+ changes in. Seems like an awful lot. Anyway 0.9.2 seems ok, although I'm already moving forward and making a few more patches. Sometimes I think the mplayer team has the right idea with the no release policy, but I do like being able to refer to a release when fixing bugs.

Gnome MPlayer Subtitle Updates

I did some more work on the subtitle and audio language pickers today. The problem I had is that some media types don't have a label for the subtitle or audio ids. So for some cases media that had subtitles or different audio tracks this tracks where not getting picked up. So I got to work on fixing that today and what happens is that for ids that have a label, the label is placed in the menu, otherwise the raw id value is put in the menu. This should at least allow the user to try some of the different options and see if they get what they want.


As you can see in this example of an mkv file with many subtitles, all of the possible subtitles are in the menu with the active title marked. You can see on the video how Spanish is the subtitle set being used.

In the above example the media (dvdnav://) does not have a label for the audio languages and so we get the code. Since this is all the information that mplayer is providing at this point, I can't present a label but it is better than no option. In this case 128 is English and 129 is Spanish.

gnome-mplayer UI

I've gotten cover art working in gnome-mplayer, and I found that when I run the application as basically a music player (which I do alot) I found that seeing the cover art and the playlist can be handy. However, I don't care how this display looks when it is running in horizontal mode, but in vertical mode it looks pretty nice. Here is a screen shot of my current SVN code when I run in vertical mode.

gnome-mplayer 0.9.0..err 0.9.1 out

Well I released gnome-mplayer 0.9.0 on Friday and by late Saturday a bunch of compile bugs started coming in. Turns out I had done two things wrong.

1. I used some features only in glib 2.16 and higher
2. I didn't handle the gio check properly in autoconf

I've fixed it now, but that will remind me to check that everything compiles properly in the future.

Another rebuild

I've been working on my development machine to get it the way I wanted. When I started working on the GIO functions of gnome-mplayer I needed to upgrade to Fedora 9 so that I could get the libraries. It was working ok, but what I really wanted was to run in 64bit mode on my machine, since I do have 4GB of RAM and a 64bit CPU.

So I reinstalled the x86_64 version of Fedora 9. However, I still wanted to be able to run a few 32bit apps. One is my scanner driver and the other is mplayer. The 32bit version of mplayer has a few more codecs than the 64bit version so I wanted to have that. My first test was to get everything I needed from livna and setup the basics. I decided that I would first install mplayer.i386 from livna to make sure I had all the right basics. To do this I had to edit my fedora, fedora-updates and livna repo files. I copied the base setup to a new [fedora-i386] for example and then made sure I was getting the 32bit files. I then installed mplayer.i386 and mencoder.i386. I then rebuild gnome-mplayer in 64bit mode and made sure that all that was working.

So far my 64/32 setup is working ok. And it already helped me fix a minor install bug in gecko-mediaplayer.

Trying dual display

When I got my last video card I wanted to make sure I got one with two DVI ports on it, so if I ever wanted two work with two displays, I would be ready to go. Well my wife got a new display for her laptop and I so I stole the LCD off of her desk and stuck it on my machine.

Having the two displays is nice, but it is taking some getting used to. So far I think I like it and am going to stick with it.

gnome-mplayer 0.8.0 and gecko-mediaplayer 0.8.0

Both gnome-mplayer and gecko-mediaplayer 0.8.0 have been released. There are several great improvements to gnome-mplayer. The changes to gecko-mediaplayer were minor but it was released to keep version sync.

Enjoy

Nailer aka MPlayer Video Thumbnailer at Google Code

I have created a new Google Code project called mplayer-video-thumbnailer. This project holds the code I developed called nailer.

You can find this code at: http://code.google.com/p/mplayer-video-thumbnailer/

Getting ready for 0.7.0

I'm about ready for the next release of gnome-mplayer and gecko-mediaplayer. Both are looking pretty solid at this point. Again gnome-mplayer has had the larger amount of changes and it getting to be a very solid media player. Pretty happy with how it is turning out.

So hopefully the new one will be out on Aug 15th.

New home pages for gnome-mplayer and gecko-mediaplayer

Due to my home server having to be shutdown. I've setup home pages for my projects at

http://kdekorte.googlepages.com

So there you should find the pages for gnome-player, gecko-mediaplayer, nailer, sunflower-applet and grandr-applet

T60p battery life

I was able to get over 4.5hrs on my T60p the other day. That is fantastic for the laptop. The other part is the laptop appears to be running much cooler under Fedora 9 than my previous setup. About 5-10C cooler which for a laptop is a lot.

Fedora 9 on my T60p

My T60p is a machine provided by my employer and it had been running Fedora 7 and then I yum upgraded to Fedora 8. Some of the things on it were not working as best as they could, like the screen saver and such. So I decided to do a clean install of Fedora 9.

I got the DVD and reinstalled, formatting the / and /boot partitions and leaving the /home partition mounted by not formatted (don't want to lose my data). Prior to doing all this I got all the packages I needed to get on the VPN so that I could connect to my office and get the additional tools I needed as well as a few config files.

The reinstall went quickly and then I had to install about 400MB of post Fedora 9 patches and another 400MB of tools for work. The only gripe I have about the install is that I did have my network activated during install, why could of the updated packages not have been installed at that point, anyway, minor nit.

All in all about 3hrs later I had my laptop working with VPN and connection to my office and I was able to get on our email system and get my mail. I then grabbed a couple of other apps and then got VMWare working (needed the vmware-any-any 117 patch). After that I added my acpi scripts and everything was working fine. 3d is not working on the machine since fglrx does not work with Fedora 9 yet and the r500 drivers have not been pushed to updates yet. I understand that they are in queue, just waiting for a bit more stabilization. Everything else on the machine is working except for the dialup modem, but I think I would have to be really desperate to have to resort to that.

I was able to use my laptop on battery for about 2.5hrs the other night which is not bad and the meter said I still had about an hour of battery left. The other thing is while I was on battery the machine was VERY cool. And in fact the machine seems cool even when plugged in. Which is nice as a hot laptop is a pain. The machine is a dual core (2.16Ghz) with 3GB of RAM in a < 1" slab so to even be slightly warm is something short of a miracle.

So to completely reload a machine from DVD and have it all updating and working in about 4-6hrs is pretty great. I've very sure I could not have done that with a Windows box. And remember with Windows you still have to install Office and all that other stuff, where with Fedora it comes as part of the install.

Thinkpad T60p acpi script

This is the script I use on my T60p (2007-AD1) to get the most out of my battery life. With this script in place, I can get about 375-4hrs off of my battery.

/etc/acpi/actions/battery.sh

#!/bin/bash
state=`grep on-line /proc/acpi/ac_adapter/AC/state | wc -l`
if [ $state -gt 0 ]; then
hal-disable-polling --device /dev/scd0 --enable-polling
echo 0 > /proc/sys/vm/laptop_mode
echo 0 > /sys/devices/system/cpu/sched_mc_power_savings
hdparm -B 128 -s 0 /dev/sda
echo 499 > /proc/sys/vm/dirty_writeback_centisecs
for i in /sys/bus/usb/devices/*/power/autosuspend; do echo 0 > $i; done
ethtool -s eth0 wol g
echo 0 > /sys/module/snd_hda_intel/parameters/power_save
echo enable > /proc/acpi/ibm/bluetooth
echo max_performance > /sys/class/scsi_host/host0/link_power_management_
policy
else
hal-disable-polling --device /dev/scd0
echo 5 > /proc/sys/vm/laptop_mode
echo 1 > /sys/devices/system/cpu/sched_mc_power_savings
hdparm -B 1 -s 12 /dev/sda
echo 1500 > /proc/sys/vm/dirty_writeback_centisecs
killall beagled
for i in /sys/bus/usb/devices/*/power/autosuspend; do echo 1 > $i; done
ethtool -s eth0 wol d
echo 1 > /sys/module/snd_hda_intel/parameters/power_save
echo disable > /proc/acpi/ibm/bluetooth
echo min_power > /sys/class/scsi_host/host0/link_power_management_policy
fi


And of course the config file in /etc/acpi/events/battery.conf

# Configuration to turn on DPMS again on video activity, needed for some
# laptops. Disabled by default, uncomment if your laptop display stays blank
# after you close and open the lid.

event=battery.*
action=/etc/acpi/actions/battery.sh

Album Art

I was interesting in putting some album art for music files into gnome-mplayer but I have not found any good C libraries to do that. So if anyone has any pointers I'll look into them. I think Amazon my have a developer program for searching images, but I'm not sure I could distribute that with gnome-mplayer. So I'm looking for a GPL way of doing this.

I've thought about doing a screen scrape of google images, but a little concerned about that.

Catalyst 8.5 on Linux aka fglrx

Well I just upgraded to the Catalyst 8.5 drivers on Linux and they say that 2d should be fast. I ran a gtkperf test and the change was negligible.

So if you are expecting a big speed bump, I would lower your expectations.

Sprint Unlimited Plans

Well I read a rumor today that Sprint was eliminating the Unlimited part of its broadband access cards Terms of Service. I REALLY hope this is not true since we don't have a lot of choice for internet. And the Sprint broadband card is the best way to get on the net.

Application idea

I have an idea for some software that would be cool eye candy I think, but I don't have time to learn how to do it. But I think it would be something cool to implement.

Anyway, the idea is to take the Mac dock and the Nautilus desktop and merge the eye candy. As you mouse over the icons on the desktop, they get bigger and the other icons move out of the way, like the icons on the Mac dock.

I think that clutter could be used to complete this pretty easily.

Any takers?

Gnome MPlayer - Next

I added a feature to the DVD w/ menus playback yesterday that adds a menu button to the panel buttons that if you click this it takes you back to the main DVD Menu.

I also found that the latest svn of libdvdnav and mplayer don't get along. I've found that libdvdnav r1031 is the last one that works properly with mplayer if there is an error in the dvd audio decoding.

I think I'm getting close to wrapping up this next version of Gnome MPlayer.

Picked a camera

Since my Kodak camera broke I did some research on cameras out there. I liked the Canon SD890 (as seen on the web) but nobody at the big box stores had one. So we looked at a few and found the Olympus Stylus 1010. The price was ok, the zoom lens was huge at 7x and 10mp was right where I wanted it. The size of the camera is great as well, with it being about the size of my cell phone. So far the few pictures I have taken with it have come out well. I'll take it out to the house later today and take some outdoor pictures with it.

Sticking with the Asus EAH3650 Silent Magic

So after a few days of working with the card and the drivers I think I got a good solution here. Driver installation was painless and the quiet of the card is great. The heat from the card raised the temp of the box about 10 degrees F, but I was able to mitigate that by installing a small case fan that came with the case (Antec NSK1380). I still would like to avoid putting in the case fan. The machine with the fan runs at 49C/120F and 55C/130C without the fan. The fan is quiet enough that I can ignore it especially with a radio playing. I hear the Thermaltake Slot Fan only runs at 21db, so maybe next time I order something from NewEgg, I'll get it.

gtkperf -c 500 -a on the 3650

gtkperf -c 500 -a
GtkPerf 0.40 - Starting testing: Tue Apr 29 10:56:16 2008

GtkEntry - time: 0.09
GtkComboBox - time: 4.44
GtkComboBoxEntry - time: 3.42
GtkSpinButton - time: 0.86
GtkProgressBar - time: 1.01
GtkToggleButton - time: 1.44
GtkCheckButton - time: 1.21
GtkRadioButton - time: 1.43
GtkTextView - Add text - time: 5.38
GtkTextView - Scroll - time: 3.75
GtkDrawingArea - Lines - time: 1.45
GtkDrawingArea - Circles - time: 318.11
GtkDrawingArea - Text - time: 45.02
GtkDrawingArea - Pixbufs - time: 4.13
---
Total time: 391.73

Other than the Circles and Text results, this is one of the fastest results I've seen on my machine.

Test machine: 2.4 Ghz Q6600, 4GB RAM, Asus P5e-VM motherboard, Asus 3650 Silent Magic, fglrx 8.476, Fedora 8

Video Cards and Fedora 8

So as you know I've been having some issues with the Intel G35 video chip and Linux. So after thinking about it, I figured I should just get a real video card anyway. So I first ordered a Sapphire 2600xt card from NewEgg. Got the fglrx drivers from Livna and things were going well. Card is fast and responds well and the drivers even work ok. One problem however, the fan on the card is super loud. I have a small and quiet machine and the fan on the video card could be heard in the next room. Totally unacceptable for the goal of a quiet PC. So I did something I hardly ever do and I returned the card.

Next, I decided to stick with an ATI chip since I got burned on my last nVidia card (GeForce 3 Ti200). nVidia dropped support for that chip and basically turned a decent video card into junk. So I decided on the Asus 3650 Silent Magic card. Now specwise this card is a little slower, however in my testing with Extreme TuxRacer the card actually gives a better FPS than the 2600xt. SecondLife, the other big 3d app I test with still gives a great picture. The added benefit of the card... 0 noise, sure it takes up a second slot in my machine, but the board I have has just about everything I need on it and there are only 4 slots anyway. The card takes the PCIx16 and PCIx1 slots. I have a PCIx1 and a PCI slot free on the machine. About the only thing I could think of using the slots for would be a video capture card. And I think an external USB one, might be a better choice. My only concern at this point is the extra heat in the card from the video board. I might be able to move around some cables inside to help the airflow. I'm hoping that ATI will enable powerplay in the drivers for the board.

glxgears: ~6600fps
quake3: 90fps (it never goes above or below and I'm running 1280x1024 with triliniar filtering)
etracer: ~150fps
secondlife: recommended settings (bar is on level 3 of 4)

I am disappointed in the Intel video support since the motherboard I picked up, I specifically got for the G35 chip on it. So I could have saved a couple of bucks and got a motherboard with fewer features. I will admit that I am pleased with the Asus product's I've gotten. The motherboard is an Asus P5E-VM HDMI.

So I'll do some more testing with this video card and let you know how it goes.

Frustrations with the Intel G35 chip

Well, I think I've had about enough with the Intel G35 chip under Linux. I've installed drivers from git and used the prepackaged drivers with Fedora are full of problems as well. So I've ordered a ATI 2600XT card and am gonna try it with the fglrx driver. I'm currently running fglrx on my Thinkpad and the 3d experience is pretty good. So hopefully the free drivers will get there. But after 2 months of problems with video issues, I've had enough and gonna try another chipset.

Intel G35 chipset on Linux

Holy cow I've been having some issues with the Intel G35 on linux. Especially with 3d code and screen corruption and crashing. After a lot of git pulls I finally have a setup that actually works to my expectations.

As of today I did the following

pulled drm from git, compiled and installed and thing compiled and installed the linux-core subdirectory as well

pulled mesa from git master compiled and installed

pulled xserver from git master and compiled and installed with these options
./configure --prefix=/usr --with-mesa-source=/home/kdekorte/cvs/mesa/ --with-dri -driver-path=/usr/lib/dri --enable-builtin-fonts

pulled xf86-drv-intel from the 'origin/intel-batchbuffer' branch and then compiled and installed with these options

./configure --prefix=/usr --with-xserver-source=/home/kdekorte/cvs/xserver

After all that googleearth, etracer and quake3 all work as expected. And reasonably fast. Quake gives me 90fps and for some reason I think it could give more. etracer is in the 20-25 fps range.

history meme

Not that anyone cares...

$ history|awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head
187 cd
90 src/gnome-mplayer
87 ls
84 sudo
57 make
40 rpm
34 vi
34 exit
27 more
26 git

Gnome MPlayer Subtitles

I've been working alot on the Gnome Mplayer issues with subtitles. Now I have to admit that most of what I know about subtitles and mplayer, I've learned in the last couple of weeks. I do know that a current version of mplayer is needed for proper subtitle support. So if subtitles are not working right, then get a newer version of mplayer. I've been testing with mplayer SVN (Mar 25,2008) and everything is working pretty good. You can even switch subtitles while the video is playing back.

Now code locations

Due to internet connectivity issues I had to move the SVN repos for gnome-mplayer and gecko-mediaplayer to new sites.

Gnome MPlayer: http://code.google.com/p/gnome-mplayer/

Gecko MediaPlayer: http://code.google.com/p/gecko-mediaplayer/

This completes the move of these project off of my website. RPMs are now part of Livna, so you can get them there.

Firefox 3 beta 3 on Linux

I just installed Firefox 3 beta 3 for Linix on my machine and I am very impressed. Memory usage is down, performance is up and the native GTK theme looks great. I can hardly wait for the final version to come out.

gnome-mplayer and gecko-mediaplayer 0.6.0 released

The latest versions of gnome-mplayer and gecko-mediaplayer were released today. gnome-mplayer had the bulk of the changes. The playlist and GUI have been greatly enhanced.

When just playing audio files the video window disappears. The option to hide the playlist is still available, making the player use a minimal amount of screenspace.

How Unix people make children

who && gawk && uname && talk && date && wine && touch && unzip && strip && touch && finger && mount && fsck && more && yes; yes; more; yes; umount && make clean && sleep

New Development Machine

I finally got a new machine for personal use and this one should last me awhile.

Intel Q6600 (quad core) @ 2.4Ghz
4GB RAM
250GB HD
Antek NSK1380 case (small formfactor, about the size of a large 4 slice toaster)
Asus P5E-VM motherboard
Fedora 8 x86_64 as the OS

Some initial testing shows that I can compile mplayer from a make clean in about 2 mins when I use the -j 4 option to make. On my loaner laptop it was about 10-15 mins.

Hopefully now, I can get back to coding.

2008

Well a new year is a upon us once again. Amazing how many things got accomplished this year and all the things that are still on the table to complete... Someday I'll get the Star Destroyer Lego done...

I can be thankful that I have a caring family and a good job and I am in good health. After that what more can you ask for. I hope everyone has a great 2008 and I know personally I'm looking forward to the upcoming events... now if I can only sell my house... ;)