btm.geek

I’ve been trying to deal with a linux appliance’s memory problems for a while, here, and here. Because Netgear/Infrant’s build system removes binaries post-dpkg, it’s not really a full system and I sort of gave up debugging when I kept running into missing binaries (like strace). Some good people helped out (Thanks Mike Fedyk) but I went and opened a trouble ticket with netgear hoping to get to talk to an actual developer on the thing. They must exist somewhere, I can’t imagine netgear let them all go when they bought infrant or anything.

1) Netgear’s support site is terrible. There is not a ’support.netgear.com, go to the knowledge base. Support is achieved through product registration of all places under online support submissions (6).

2) The Readynas people have a nice forum, and it’s product specific. There’s a blog and everything, which is cool. But my thread stopped getting responses from them last week. No “I don’t know” or anything, just stopped responding to me.

3) So I opened the ticket with Netgear, and they respond with:

The Hardware Compatibility List Memory list/page http://www.readynas.com/?page_id=83

It’s the only guideline we have and if it’s not on the list its not supported nor with the scope of support we provide.

You question is already in the best place for an answer. The moderators are will pass all applicable data to the engineering staff as needed.

Totally in response to like, my first post of the thread, somehow ignoring the rest of it. In a hurry, fine.

4) I reply saying there’s a problem with the product and I need escalation. Escalation closes my ticket and responds with:

The forum where are posting is run by our Engineering Team. For your reference, the members of our team use Star Wars (TM) type names. Considering the kind of issue that you are having, you will have to correspond with them, as we at NETGEAR Level 1 and Level 2 Support cannot assist you with this type of issue.

We appreciate your patience and understanding.

The implication that I still have patience at this point is nice of them, however totally wrong.

I downgraded to an earlier version of raidiator on friday and saw no improvement in the memory black hole over the weekend. The frustrating part is being unable to tell where it is going, rather than trying to fix the problem with a particular daemon that I may not have the customized source for. My earlier blog entry about this is here. There’s more data from today in the netgear forum thread.

I did find this LKML thread by Mike Fedyk who did most of the upgrades to the munin memory script for 2.6. I can see in the thread that he decided to use the Total-Free-everythingelse=AppsUsed calculation, and I don’t see any big light bulbs in that thread to help solve my problem. I see on the net that someone that used to idle in #swn on irc is connected to a Mike Fedyk, so I’ve emailed him asking for an introduction before I try to harass him directly with the problem. I’m going to assume this is his LJ with a post about performance tuning.

My munin-users thread can be found here, for the record. I’m going to look around for more utilities to track down memory usage, although the lkml thread makes me feel like that may not be happening. I posted in the netgear thread asking for a kernel upgrade but the best advice I’ve gotten there so far is “our perl may be broken. stop running munin” so I’m not sure anyone technical is listening.

Man this is annoying. A file tree ended up with a .svn folder which contains files marked read-only. When copied with Vista all is fine until you try to delete the folder, when you’re told “you need permission to perform this action” with “try again” and “cancel” with options, trying again many times didn’t do as much as I would have hoped. Eventually we found the files with the read-only attributes. These files are stored on a samba server so I suppose I’ll see if I can get get samba or a cron script to strip those attributes. Removing the read only attribute allows you to delete the file, but I can’t find any way to enable the old XP style dialog that tells you it is marked read only but allows you to delete it anyways if you have permissions. UAC is off, by the way.

update:

Raidiator, the debian based distro that runs on infrant (i always say infarant) / netgear readynas products has ’store dos attributes = 1′ in the global section of /etc/samba/smb.conf. This stores the read-only / hidden / archive / system attributes in an extended attribute called user.DOSATTRIB:

getfattr -d entries
# file: entries
user.DOSATTRIB=”0×21″

Normally this is off and newer versions of samba use ‘map read only’ to determine what read only should be set to, based on the user write bit (default) (yes), the effective permissions of the user (permissions), or ignoring permissions and only using ’store dos attributes’ (no).

I put ’store dos attributes = 0′ in the share definition to override the global (/etc/frontview/samba/Shares.conf in raidiator) and reloaded samba (/etc/init.d/samba reload) and then the files properties showed that the file was not read only any longer, thus working around the problem of Vista not letting me delete read-only files.

Excel crashed the other day and the document from the sharepoint server could not be opened read/write due to it being ‘locked for editing’. Choosing ‘Check out’ in Sharepoint, then editing the document resolved the issue. Lots of Sharepoint 2003 discussion here.

We use git with a single bare repository for our puppet configuration, and each systems administrator has a local git repository clone which they push back to the origin. I wanted to set up email notification on this main repository which lives on a debian etch server.

I found post-receive-email in the git gitweb repository and assumed that it was not included in the debian package because it has a copyright with no OSS license included. It pulls its configuration from the git config, which is repository specific and kind of neat, but I had to modify it to call ‘git-repo-config’ instead of ‘git config’ because that’s all etch had. Again, assuming some weird debian problem, but I didn’t bother looking.

Then when I had trouble with it not working I noticed my ubuntu hardy box had a newer major revision of git-core than the debian etch box. That is 1.5.4.3-1ubuntu2 and 1.4.4.4-2 respectively. I poked around the git documentation a little bit and found that the post-receive hooks weren’t added until 1.5.1. But there is a 1.5.4 git-core deb in etch-backports.

If you want to upgrade multiple boxes with a local repository, you’ll need a copy more than git-core to meet the dependences. otherwise you can just use apt-get install after adding the backports repo.

add ‘deb http://www.backports.org/debian etch-backports main’ to /etc/apt/sources.list

sudo apt-get update
sudo apt-get install debian-backports-keyring
sudo apt-get update
sudo apt-get install apt-move
sudo rm /var/cache/apt/archives/git*
for package in gitk gitweb `apt-cache search '^git-*' --names-only | awk '{ print $1 }'` ; do sudo /usr/lib/apt-move/fetch $package ; done

latest debs are in /var/cache/apt/archives, for copying to a local repository.

git-core 1.5.4.2-1~bpo40+2 includes git-config and ‘post-receive-email’.

cd /path-to-bare-git-repo/.git/hooks
ln -sf /usr/share/doc/git-core/contrib/hooks/post-receive-email post-receive
sudo chmod a+x /usr/share/doc/git-core/contrib/hooks/post-receive-email
git-config hooks.mailinglist "to@example.org"

git-config --global user.name "Your Name"
git-config --global user.email "Your Email"

ruby-activeldap requires ruby-ldap and ruby-log4r (hah @ log4r). On Activeldap 0.7.4 via debian etch packages:

Remember that AD doesn’t like anonymous binds:

require 'activeldap'

ActiveLDAP::Base.connect(
  :host => "ad.example.org",
  :base => "dc=ad,dc=example,dc=org",
  :bind_dn => "cn=ldapbind,ou=service,dc=ad,dc=example,dc=org",
  :password => "password",
)

/usr/lib/ruby/1.8/activeldap/base.rb:312:in `connection': Unable to retrieve schema from server (plain) (ActiveLDAP::ConnectionError)

This error is deceiving though. I noticed via wireshark that it was trying to bind as ‘cn=username,dc=localdomain’, failing, and trying an anonymous bind, at which point AD was letting it search that weird referral land that typically breaks other ldap searches. After adding:

  :allow_anonymous => false

I got:

/usr/lib/ruby/1.8/activeldap/base.rb:1225:in `do_bind': Invalid credentials (LDAP::InvalidCredentials)

Using this worked:

ActiveLDAP::Base.connect(
  :host => "ad.example.org",
  :base => "dc=ad,dc=example,dc=org",
  :bind_format => "cn=%s,ou=service,dc=ad,dc=example,dc=org",
  :user => "ldapbind",
  :password => "password",
  :allow_anonymous => false
)

I’ve lost the class block using ldap_mapping I was using, but you could do things like:

class User < ActiveLdap::Base
  ldap_mapping :dn_attribute => ‘uid’, :prefix => “”
end

user = User.new(”myusername”)
puts user.mail

Awesomely enough you have to pay strict attention to what version of Activeldap you’re using. in Later versions ActiveLDAP becomes ActiveLdap and the Base.connect method becomes Base.establish_connection and works a little differently (using Activeldap 0.10.0 via gem). dnattr used with ldap_mapping becomes dn_attribute. ri is your friend here. Something like this works:

#!/usr/bin/ruby
# requires ruby-activeldap (libactiveldap-ruby1.8)
#     ruby-ldap (libldap-ruby1.8) ruby-log4r (liblog4r-ruby1.8)
# this particular syntax requires ruby-activeldap 0.10.0
# rubygems is required because I installed via gem. I don't know why.
# Bryan McLellan 

require ‘rubygems’
require ‘active_ldap’

ActiveLdap::Base.establish_connection(
  :host => “ad.example.org”,
  :base => “dc=ad,dc=example,dc=org”,
  :bind_dn => “cn=ldapbind,ou=service,dc=ad,dc=example,dc=org”,
  :password => “password”,
)

class User < ActiveLdap::Base
  ldap_mapping :dn_attribute => ‘uid’, :prefix => ‘ou=MyUsers, :classes => [”user”]
end

user = User.find(”myusername”)
puts user.mail

You need classes to tell activeldap what schema to load. Standard classes are things like [’top’, ‘account’, ‘posixAccount’]. You can list multiple schema’s in an array like I just did. I found user by ‘puts user.attribute_names’ and looking for the attribute I wanted. Note also that we’re using User.find instead of User.new. Previously User.find didn’t contain any attributes, now it does, whereas User.new will have empty attributes because it is in fact creating a new user class as one would expect (albeit in memory).

I’m going to post this as WP like to destroy my PRE blocks, and I haven’t looked for a solution yet.

ERROR 1044 (42000) at line 2: Access denied for user 'root'@'%' to database 'irm'

When I create the ‘root’@'%’ user via mysql, I forgot the grant option:

grant all on *.* to 'root'@'%' identified by 'password' with grant option;

I dug this out of the LVM HOWTO. I had an Ubuntu linux install on an IDE disk and I was moving this install to a newer SATA only box. I got both the disks running in the old computer and booted up on System Rescue CD. I copied my boot partition using gparted, then ran:

pvcreate /dev/newdiskpartition
vgextend oldvolumegroup /dev/newdiskpartition
pvmove /dev/olddiskpartition /dev/newdiskpartition
vgreduce oldvolumegroup /dev/olddiskpartition


I’d recommend thinking about all of this carefully before hitting enter. It took an hour or two to move 80GB of physical extents from IDE to SATA. Since I’m running ubuntu, I also mounted the new partition as /mnt, and ran ‘chroot /mnt /bin/bash’ then mounted the boot partition in /boot. I ran grub-install, updated /boot/menu.lst, and updated the UUID’s in /etc/fstab.

I downloaded Pidgin (formerly gaim) on a new machine, like I normally do. I quickly noticed that I could no longer change the size of the text input area. I subscribed to ticket #4986 and watched the arguments roll until eventually the developers simply closed the ticket as wontfix. I’ve heard rumors there is some turmoil within development, but really only the developer to user turmoil is externally visible. I’ve just been using pidgin 2.3 while this was all being discussed but I’m switching to the funpidgin fork now that the developers have expressed that pidgin will not have an option to manually resize the text input area.

While it seems like a lame fork, it’s up to the pidgin developers I suppose as to see where things go from here. Hopefully if the pidgin developers keep contributing new code that doesn’t suck, the funpidgin developers will keep integrating it and keep up with releases. Of course, what would just be best is a damn option in pidgin to enable manual resizing again. Looks like that’s not happening with the current developer hierarchy though.

Getting manual input sizing back is a matter of  Tools -> Plugins, then Enable Entry Area Manual Size. You will likely need to close the conversation window and re-open it.

We recently switched to subversion from cvs and after patching together a Bugzilla 3.0.3 install since the debian buzgilla package is currently orphaned, the dev lead stepped into the IT office and informed me that we needed svn + bugzilla integration for checkins. Meh. There’s a nice long howto here that covers everything, almost step by step, but most of it’s manual. If you ignore that it explains how to install everything, the configuration is somewhat short but still involved hacks with email due to the lack of an API in bugzilla that’s widely used.

SCMBug releases however, have debs, the latest being 0.23.4. you can download these and run:

dpkg -i scmbug-server_0.23.4_all.deb scmbug-common_0.23.4_all.deb ; apt-get install -f

I’m sure there’s a cleaner way to do that, but I haven’t stumbled across it yet and that works.

Installation documentation is nested deep in here.

Upon scmbug_daemon starting I saw:

** Scmbug error 77: The userlist mappings are enabled, but no mappings are configured.

My bugzilla install is currently in /usr/local/bugzilla due to the lack of a package, so I went in there and grabbed the corresponding information from localconfig to update /etc/scmbug/daemon.conf including database information (I keep my mysql databases consolidated in production), and made a point to update installation_directory to ‘/usr/local/bugzilla’.

I also enabled the mapping_regexes section (enabled =>1) and modified the “unix user mapping” to email addresses, since that’s what bugzilla uses.

I then installed scmbug-common and scmbug-tools on the subversion server and configured it like:

scmbug_install_glue --scm=Subversion --product=myproduct --repository=file:///srv/code/svn --daemon=10.0.0.19 --binary-paths=/bin,/usr/bin --bug 845

I made up the bug number, used the first one that didn’t exist in bugzilla yet. It requires all of those options. the ‘file://’ part of the svn url is required or you get the error “** Scmbug error 25: file:// prefix not specified for Subversion repository path.”

I was a little iffy about the product, because we separate out our repository by product but it’s all in one svn repository. There is chat about it all matching up here and having product be required in the scmbug_install_glue script was a little disconcerting in the way that I expect things to not work.

I used TortoiseSVN on a windows box to quickly make a new directory and tag it with a bug I made (845, after the fact of running the install script). I hit a couple default policy problems like that the bug wasn’t open yet, then that my commit message wasn’t over 50 characters. All this can be tuned in ‘/srv/code/svn/hooks/etc/scmbug/glue.conf’ after you’ve installed the glue.

Low and behold though, the install worked. Props to the scmbug folks, that was much cleaner than the alternatives.

update: checkin linkification

I modified some older diffs against bugzilla to linkify the file list on checkin. The were on bug #266 in bugzilla for scmbug, but I can’t create a login right now for whatever reason. hopefully people find it here, since I’m using this on 3.0.3 and viewsvn, which is different than what’s on the bug right now.

WP doesn’t like me pasting the diff, wrapping in pre or code tags, so it is here in my git repo.

earlier posts here and here.

A review:
1) We removed ntp from the linux guests and left it running on the vmware hosts.
2) We installed open-vm-tools on the guest and live enabled timesync using vmware-guestd

Notes revealed we were gaining about 40s a day.

3) set clock=pit (use clocksource=pit now) in the grub config as a kernel option and restarted a guest

That looks like about 40s over three weeks.

4) today I noticed a lot of “/dev/vmmon[3685]: host clock rate change request 500 -> 998″ messages on the vmware hosts (linux) and I set up the recommendations here which is ‘host.cpukHz = cpuspeedinkhz’, ‘host.noTSC = TRUE’, and ‘ptsc.noTSC = TRUE’ to work around possible speed step issues.

I accidentally used khz = mhz * 100 instead of khz = mhz * 1000 which made the time get way off when I stopped and then started the vm I testing was on. This was interesting though because I was afraid I’d have to stop vmware-server, not just an individual vmware-vmx process to get it to re-read /etc/vmware/config.

Looping ntpdate shows about 8/10th of a second gain over 20 minutes. Still more gain than I’d like to see. Will watch the graph and then try again in a week or two.

A local Vista computer started having intermittent login failures when a domain user tried to log in about a trust problem with the account database.

Since Vista disables the local administrator account even though it had a password. I used Nordahl’s ntpasswd linux boot cd to enable the local administrator account (if I hadn’t known the password I could have changed it as well). Of course the CD requires access to the syskey as the SAM is encrypted, but it always finds it automatically since nobody puts the syskey on floppy.

Then I logged in and removed the computer to the domain, changed it’s name, and rejoined it and things were fine.

Domain profiles were kept intact by the way.