Gajim spins up my hard drive in laptop mode

[osch0001@…]

When using laptop mode with Linux kernel version 2.6.15, Gajim spins up my hard drive every time I receive or send a message. Not surprisingly, it appears that this is happening when it writes to the log.db file (see below). What is surprising is that Gajim is forcing a write rather than letting the kernel cache it for later writing.

For those that don't know, laptop mode is a setting in the Linux kernel the delays flushing of dirty blocks until next time the disk spins up for some other reason (e.g., a read), an application forces a sync, or until a predefined time limit is hit. See  http://www.xs4all.nl/~bsamwel/laptop_mode/ for more information.

My Python is a bit rusty, so I haven't looked at exactly what Gajim is doing. However, I would love to see this fixed, as the constant spinning up and down caused by Gajim's syncs is hard on the drive.

dmesg output:

[17293612.468000] python(14059): dirtied inode 6262818 (logs.db-journal) on dm-5 [17293612.468000] python(14059): WRITE block 101406664 on dm-5 [17293612.468000] python(14059): dirtied inode 6264898 (logs.db) on dm-5 [17293612.468000] python(14059): WRITE block 100556184 on dm-5 [17293612.468000] python(14059): WRITE block 100556192 on dm-5 [17293612.468000] python(14059): dirtied inode 6262818 (?) on dm-5

[osch0001@…]

There is no reason the database "has to sync," especially when dealing

with data that is hardly critical such as chat logs. Even MySQL has an option to delay writing of data, so I would assume SQLite, which Gajim seems to use, would have a similar option given that it is designed for end-user applications.

Gajim spins up the disk more often then all of my other applications combined. I can use Gnome, Firefox (with history and caching enabled), or Thunderbird for five or more minutes without them spinning up my disk, but if I get one instant message, Gajim immediately spins up this disk.

Although I really do like Gajim, this is a deal breaker for me as I would like my Jabber client to do logging but I cannot have it spin up the disk every few seconds.

[asterix]

the pb if we don't commit instantly (let's say every 5 minutes) is that DB

is locked during 5 minutes. So we can't run 2 instances of gajim (second instance can't commit cause DB is locked during 5 minutes)

[osch0001@…]

Is running 2 instances something that is supported? I think most Gnome

apps use DBUS to check if there is already an instance open when the app is invoked and then open another window of that same app if it is already running.

I tried to add self.cur.execute("PRAGMA default_synchronous = OFF") after the database connection is open, but it still fsyncs when the connection closes, which seems to be immediately after writing.

[asterix]

yes it is supported with -p option (--profile)

Gajim doesn't use DBUS to see if it is already running, but the existance of a file.

and history manager also try to access the DB, which is imposible if DB is always locked.

What I did is to remove self.con.commit() and only commit every 5 minutes with a timeout_add() (if there is something to commit of course) but that lock the DB

[osch0001@…]

Wouldn't it make sense for the different profiles to have different DBs as

they have different config files? I don't know what the use case on the profile support is, but this would make sense to me.

If the above was changed, then the code could probably be refactored to only use one database connection per instance (e.g., have the history feature use the same connection).

However, your suggest seems like a big improvement over the current situation.

[jim++]

I agree that for another profile, logs should be in another file.

[ccarlin@…]

I get seconds of disc churn for every message I log. It sounds related to

this, and it makes gajim nearly unusable.

It's a huge shame too because I just found gajim and it puts other Jabber clients to shame. I really would like to see this solved; it is a terrible blight on an otherwise amazingly great client.

[patrys@…]

I only have a laptop and this does not seem to be a problem at all.

[asterix]

we can't use the same connection for gajim and history manager as it's 2

separate program / process

moreover commiting every 5 minutes means we loose data if there is a crash

[ccarlin@…]

People attempting to use laptop mode (or any system where data is cached

such as Linux or Windows) are accepting the risk that in the event of a crash data will be lost. It's not an unreasonable risk in this case, as the cache will be written out occasionally, and not too much information is likely to be spooled up.

As you can see the alternative for some people is not being able to use gajim at all, and frustratingly enough for a reason that isn't fundamental to the program's operation. The pretty log manager with its snazzy sql backend (frankly, I find sqlite far overhyped) is really secondary to the task of communicating with people.

Perhaps a better route to go would be using JEP-0136, message archiving, where messages would be stored on the server and not have to hit the hard drive at all.

[asterix]

for the moment we don't implement JEP136, and not many servers do.

I can add an advanced option to have this "laptop mode" wichi commit every 5 minutes, without the ability to run history manager while gajim is running.

[ccarlin@…]

I personally would be happy with that solution, asterix, though I hope

some of the other posters to this bug give feedback as well.

[knuckles@…]

I also see this when my system is doing very heavy I/O -- because I use XFS it tends to join writes so sometimes it takes a few seconds to start writing/reading and so gajim gets very irresponsive during these times.

I have an UPS, so laptop mode would be nice for me too, if it increases responsiveness =)

[patrys@…]

Another suggestion to avoid using DB in blocking mode:

At each graceful program exit, backup the logs.db file. Or even do it in the background.

[junglecow]

I suggest to the original reporter of this bug to create a RAM disk, move the log file there, and make a symbolic link from the gajim directory.

[anonymous]

That's certainly quite a lot of trouble and a huge workaround for this effect. Yes, it would work, but... seriously:

You could also suggest that the reporter (and everyone else having issues related to this behavior) invest in one of those new hybrid solid state drives or network mount a directory for holding the log files, but really this behavior should just be fixed in the single small program causing the problem.

There is NO need to access the disk every time a message is received. That SQLite insists on flusing the drive with every commit is a weakness in that library, one that comes into play through its use by Gajim. Gajim, if it insists on using this mediocre library, should work to make up for the weakness as it affects basic usability of the software.

[junglecow]

<sandos> hehe, ok, I have to agree with some other reporter.. 
         sqlute seems very disk-intensive
<sandos> heh, when logging into a conference, I can hear the disk
         work, and the display halts during that short time, for 
         each and every line
<Asterix> sandos, it should be easy to write a logger.py for 
          postgres or another DB system
<sandos> (weird, I thought sqlite was very fast)
<junglecow> sqlite is fast as long as you don't keep opening and 
            closing it
<Asterix> the pb if we keep it open: we can't runb gajim and 
          history manager in the same time because of locks
<sandos> right, sqlite doesnt have multi-user goodness..
<Asterix> but we can't require a user to install mysql tu run 
          gajim ...
<junglecow> maybe it's possible to do a delayed close of the 
            database
<junglecow> close the database after a configurable period of 
            inactivity (default 1 second)
<junglecow> that should help with disk thrashing
<junglecow> and if someone cares about laptop mode they can set it 
            to 24 hours
<sandos> a logging-cache or buffer could help alot
<sandos> buffer X seconds before writing to log
<sandos> with the risk of loosing stuff
<Asterix> hmmm yes could be a good situation ... but there is the 
          risk to loose data
<Asterix> when you crash because of an incoming message it's lost 
          ... you can't read it anymore
<junglecow> maybe best to use my original suggestion
<junglecow> keep the lock open for one second
<Asterix> could you add this suggestion in the ticket ?

Edited for brevity.

[ccarlin@…]

What other embedded databases exist out there? I know the obvious Berkeley (and its nifty native XML sibling), but I suppose you guys really want to stick with the SQL.

I'd like to point out that the worry over buffering would be handed off to the OS if not for the calls to flush the buffers. This is something that the OS should be dealing with anyway. It irks me that you guys even have to think about it.

[B.Steinbrink@…]

Replying to osch0001@umn.edu:

Is running 2 instances something that is supported? I think most Gnome apps use DBUS to check if there is already an instance open when the app is invoked and then open another window of that same app if it is already running. I tried to add self.cur.execute("PRAGMA default_synchronous = OFF") after the database connection is open, but it still fsyncs when the connection closes, which seems to be immediately after writing.

default_synchronous isn't supported anymore in SQLite 3.0, it was only available in 2.8. Using the new "PRAGMA synchronous = OFF" instead gets rid of the fsync() calls (checked with strace). In case of a system crash you can loose data that way, but you should not loose data when only gajim crashes. As the log data isn't really that important in case of a system crash I'd suggest to make the non-synchronous mode the default. It's a lot faster and makes it acceptable to run gajim on a laptop.

See also:  http://www.sqlite.org/pragma.html

[ccarlin@…]

Replying to B.Steinbrink@gmx.de:

As the log data isn't really that important in case of a system crash I'd suggest to make the non-synchronous mode the default. It's a lot faster and makes it acceptable to run gajim on a laptop.

Sounds like this would close the bug.

I don't view this behavior as dangerous. It doesn't remove all attempts to save critical data; it just passes responsibility for writing the data to the OS. It is entirely appropriate to let the OS decide how much and how quickly it should to write, and the OS will be able to do this more effectively since it knows more about hardware, contending applications, and settings like laptop mode.

I honestly can't imagine why synchronous isn't off by default.

[junglecow]

Replying to B.Steinbrink@gmx.de:

Using the new "PRAGMA synchronous = OFF" instead gets rid of the fsync() calls (checked with strace). In case of a system crash you can loose data that way, but you should not loose data when only gajim crashes. See also:  http://www.sqlite.org/pragma.html

Thanks, this is helpful information. I've been looking through the sqlite API but hadn't run across this. Attaching patch to resolve this issue.

[junglecow]

Disable sqlite's synchronous mode for logging (oops, forgot to clear con)

[asterix]

(In [f24d929db2a86873b01e20f11c655757c754be73]) use PRAGMA synchronous = OFF in sqlite to avoid hard drive spin up. fixes #2183