[ gphoto-Bugs-1098166 ] Large downloads make computer unusable due to memory leak

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[ gphoto-Bugs-1098166 ] Large downloads make computer unusable due to memory leak

SourceForge.net
Bugs item #1098166, was opened at 2005-01-07 21:29
Message generated for change (Comment added) made by kot
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=108874&aid=1098166&group_id=8874

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: do not use this gphoto2/core
Status: Closed
Resolution: Fixed
Priority: 9
Submitted By: Dean Brettle (brettle)
Assigned to: Hubert Figuiere (hfiguiere)
Summary: Large downloads make computer unusable due to memory leak

Initial Comment:
To reproduce:
1.  Run a gphoto2-based app.  I tried both gthumb and
gtkam.
2.  Download many pictures/videos from the camera (in
one shot), where "many" is enough to occupy roughly the
amount of RAM you have.  I was trying to download 207M
on a machine with 256M of RAM.

Result:

App consumes ever-increasing amounts of memory, causing
more and more swapping.  The system gets slower and
slower until it eventually freezes.

This occurs with both version 2.1.4 and with the
current CVS.

I'm running Fedora Core 3 with kernel 2.6.9-1.681_FC3
and I'm using a Canon PowerShot SD200.

Analysis:

gphoto2-filesys.c uses an in-memory cache.  It chooses
a cache size which takes up virtually all free memory
and swap.  As a result, large downloads cause massive
swapping.

Proposed solution:

This problem has been discussed for several years and
the current consensus seems to be that gphoto2 should
really cache files to disk[1].  While I agree that
would be ideal (assuming a reasonable cache size limit
was used), an intermediate and simpler solution is to
use a smaller memory cache.

I have a working patch (which I will attach) that
implements a smaller memory cache the size of which can
be configured by both users and apps.  The default
cache size is determined by the same algorithm used by
Mozilla Firefox[2] and results in the following:

System RAM   Cache    % RAM
<=16 Mb          0 Mb         0%
   32 Mb          2 Mb         6%
   64 Mb          4 Mb         6%
  128 Mb          8 Mb         6%
  256 Mb         14 Mb         5%
  512 Mb         22 Mb         4%
 1024 Mb         32 Mb         3%
 2048 Mb         44 Mb         2%
 4096 Mb         58 Mb         1%

If a user desires a different cache size, they can set

libgphoto=cache-capacity-default-kb=<whatever>

in ~/.gphoto/settings.

If an application desires a different cache size, it
can call gp_filesystem_set_cache_capacity().  The
application's cache capacity trumps the user's
libgphoto=cache-capacity-default-kb setting, but such
an application could allow the user to set its cache
size via an application-specific setting.

References:

[1] Previous discussion threads:
http://marc.theaimsgroup.com/?l=gphoto-devel&m=103778525621015&w=2
http://marc.theaimsgroup.com/?l=gphoto-devel&m=102918882213658&w=2
http://marc.theaimsgroup.com/?l=gphoto-devel&m=102935238904360&w=2
http://marc.theaimsgroup.com/?l=gphoto-devel&m=97169213319489&w=2
http://marc.theaimsgroup.com/?l=gphoto-devel&m=95576532725887&w=2

[2] Discussion of algorithm used by Mozilla Firefox:
https://bugzilla.mozilla.org/show_bug.cgi?id=105344


----------------------------------------------------------------------

Comment By: Mikhail Teterin (kot)
Date: 2005-09-16 03:58

Message:
Logged In: YES
user_id=173641

Can you, please, post the patch you added? I'd like to add
it to the FreeBSD port for the time being -- until the 2.1.7
release. Or is it the same as the
libgphoto2-finite-cache.patch attached to this bug report?
Thanks!

That said, the `gphoto2 -P' should not be caching anything
at all -- does not the library allow the client program to
specify, that there is no need to cache?

----------------------------------------------------------------------

Comment By: Marcus Meissner (marcusmeissner)
Date: 2005-08-25 20:59

Message:
Logged In: YES
user_id=48092

i now applied a patch to 2.1.99 that does:
 
- store the last x full images.
- x is read from settings ("libgphoto2" "cached-images")
- default x is 2
 
this should cover all cases and fix that specific problem for most
people. (except people with small memory footprints, or people
with very large movies)

----------------------------------------------------------------------

Comment By: Mikhail Teterin (kot)
Date: 2005-06-08 12:34

Message:
Logged In: YES
user_id=173641

GPhoto maintainers! Please, offer a workaround for this
problem.
 
Is the patch offered by Deane acceptable? Is there a
better/simpler fix/workaround? The problem has been
around very long now -- some (if imperfect) stop-gap
solution is urgently required.
 
Thanks!

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=108874&aid=1098166&group_id=8874


-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server.
Download it for free - -and be entered to win a 42" plasma tv or your very
own Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
gphoto-bugs mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gphoto-bugs