Slow execution when many files

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Slow execution when many files

Vincent Vande Vyvre
Hi,

I discover a strange behaviour with a Nikon D5000.

When the memory card has several pictures the communication between gphoto2 and the camera became very slowly.

Exemple with 40 files:

$ time gphoto2 --auto-detect
Modèle                        Port                                             
----------------------------------------------------------
Nikon DSC D5000 (PTP mode)     usb:001,032

$ time gphoto2 --list-files
Le dossier « / » ne contient aucun fichier.                                    
Le dossier « /store_00010001 » ne contient aucun fichier.
Le dossier « /store_00010001/DCIM » ne contient aucun fichier.
Le dossier « /store_00010001/DCIM/100VINCE » contient 40 fichiers.
#1     _DSC1184.JPG               rd  3375 KB 4288x2848 image/jpeg
#2     _DSC1184.NEF               rd  9277 KB application/x-unknown
...
[Skip for brevity]
...
#39    _DSC1203.JPG               rd  3931 KB 4288x2848 image/jpeg
#40    _DSC1203.NEF               rd 10403 KB application/x-unknown

real    0m2.395s
user    0m0.030s
sys        0m0.030s
================================================================================

$ time gphoto2 --show-exif 1-10
Downloading '_DSC1184.JPG' from folder '/store_00010001/DCIM/100VINCE'...      
Balises EXIF :
--------------------+-----------------------------------------------------------
Balise              |Valeur                                                     
--------------------+-----------------------------------------------------------
Make                |NIKON CORPORATION                                          
Model               |NIKON D5000                                                
...
[Skip]
...                                         
FlashPixVersion     |FlashPix version 1.0                                       
ColorSpace          |Erreur interne (valeur inconnue 65535)                     
--------------------+-----------------------------------------------------------

real    0m2.884s
user    0m0.043s
sys        0m0.030s
================================================================================

$ time gphoto2 --get-thumbnail 1-10
Downloading '_DSC1184.JPG' from folder '/store_00010001/DCIM/100VINCE'...      
Enregistrement du fichier en thumb__DSC1184.jpg
Downloading '_DSC1184.NEF' from folder '/store_00010001/DCIM/100VINCE'...
...
[Skip]
...
Downloading '_DSC1188.NEF' from folder '/store_00010001/DCIM/100VINCE'...
Enregistrement du fichier en thumb__DSC1188.NEF

real    0m2.827s
user    0m0.036s
sys        0m0.030s
================================================================================

These execution times seems reasonable for me. Now with 938 files into the card:

$ time gphoto2 --list-files > nikon.log
Le dossier « / » ne contient aucun fichier.
Le dossier « /store_00010001 » ne contient aucun fichier.
Le dossier « /store_00010001/DCIM » ne contient aucun fichier.
Le dossier « /store_00010001/DCIM/100VINCE » contient 938 fichiers.
#1     _DSC0715.JPG               rd  5779 KB 4288x2848 image/jpeg
#2     _DSC0715.NEF               rd 10739 KB application/x-unknown
...
[Skip]
...
#937   _DSC1183.JPG               rd  3151 KB 4288x2848 image/jpeg
#938   _DSC1183.NEF               rd 10232 KB application/x-unknown

real    0m51.957s
user    0m0.188s
sys        0m0.058s
================================================================================

$ time gphoto2 --show-exif 1-10
Balises EXIF :
--------------------+-----------------------------------------------------------
Balise              |Valeur                                                     
--------------------+-----------------------------------------------------------
Make                |NIKON CORPORATION                                          
Model               |NIKON D5000
...
[Skip]
...
FlashPixVersion     |FlashPix version 1.0                                       
ColorSpace          |Erreur interne (valeur inconnue 65535)                     
--------------------+-----------------------------------------------------------

real    0m52.381s
user    0m0.145s
sys        0m0.065s
================================================================================

$ time gphoto2 --get-thumbnail 1-10
Downloading '_DSC0715.JPG' from folder '/store_00010001/DCIM/100VINCE'...
...
[Skip]
...
Enregistrement du fichier en thumb__DSC0719.NEF

real    0m52.337s
user    0m0.073s
sys        0m0.108s
================================================================================

Note the +- 50 seconds of difference seems the "handshake" (if there's one), when the first result appears, all others results follows immediately.


Is this a well-know behaviour of the PTP or just with Nikon cameras ?

Is other users having the same experience ?

Regards,

Vincent
------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Gphoto-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gphoto-devel
Reply | Threaded
Open this post in threaded view
|

Re: Slow execution when many files

Vincent Vande Vyvre
Le 26/05/2016 06:30, Vincent Vande Vyvre a écrit :
Hi,

I discover a strange behaviour with a Nikon D5000.

When the memory card has several pictures the communication between gphoto2 and the camera became very slowly.

Exemple with 40 files:

$ time gphoto2 --auto-detect
Modèle                        Port                                             
----------------------------------------------------------
Nikon DSC D5000 (PTP mode)     usb:001,032

$ time gphoto2 --list-files
Le dossier « / » ne contient aucun fichier.                                    
Le dossier « /store_00010001 » ne contient aucun fichier.
Le dossier « /store_00010001/DCIM » ne contient aucun fichier.
Le dossier « /store_00010001/DCIM/100VINCE » contient 40 fichiers.
#1     _DSC1184.JPG               rd  3375 KB 4288x2848 image/jpeg
#2     _DSC1184.NEF               rd  9277 KB application/x-unknown
...
[Skip for brevity]
...
#39    _DSC1203.JPG               rd  3931 KB 4288x2848 image/jpeg
#40    _DSC1203.NEF               rd 10403 KB application/x-unknown

real    0m2.395s
user    0m0.030s
sys        0m0.030s
================================================================================

$ time gphoto2 --show-exif 1-10
Downloading '_DSC1184.JPG' from folder '/store_00010001/DCIM/100VINCE'...      
Balises EXIF :
--------------------+-----------------------------------------------------------
Balise              |Valeur                                                     
--------------------+-----------------------------------------------------------
Make                |NIKON CORPORATION                                          
Model               |NIKON D5000                                                
...
[Skip]
...                                         
FlashPixVersion     |FlashPix version 1.0                                       
ColorSpace          |Erreur interne (valeur inconnue 65535)                     
--------------------+-----------------------------------------------------------

real    0m2.884s
user    0m0.043s
sys        0m0.030s
================================================================================

$ time gphoto2 --get-thumbnail 1-10
Downloading '_DSC1184.JPG' from folder '/store_00010001/DCIM/100VINCE'...      
Enregistrement du fichier en thumb__DSC1184.jpg
Downloading '_DSC1184.NEF' from folder '/store_00010001/DCIM/100VINCE'...
...
[Skip]
...
Downloading '_DSC1188.NEF' from folder '/store_00010001/DCIM/100VINCE'...
Enregistrement du fichier en thumb__DSC1188.NEF

real    0m2.827s
user    0m0.036s
sys        0m0.030s
================================================================================

These execution times seems reasonable for me. Now with 938 files into the card:

$ time gphoto2 --list-files > nikon.log
Le dossier « / » ne contient aucun fichier.
Le dossier « /store_00010001 » ne contient aucun fichier.
Le dossier « /store_00010001/DCIM » ne contient aucun fichier.
Le dossier « /store_00010001/DCIM/100VINCE » contient 938 fichiers.
#1     _DSC0715.JPG               rd  5779 KB 4288x2848 image/jpeg
#2     _DSC0715.NEF               rd 10739 KB application/x-unknown
...
[Skip]
...
#937   _DSC1183.JPG               rd  3151 KB 4288x2848 image/jpeg
#938   _DSC1183.NEF               rd 10232 KB application/x-unknown

real    0m51.957s
user    0m0.188s
sys        0m0.058s
================================================================================

$ time gphoto2 --show-exif 1-10
Balises EXIF :
--------------------+-----------------------------------------------------------
Balise              |Valeur                                                     
--------------------+-----------------------------------------------------------
Make                |NIKON CORPORATION                                          
Model               |NIKON D5000
...
[Skip]
...
FlashPixVersion     |FlashPix version 1.0                                       
ColorSpace          |Erreur interne (valeur inconnue 65535)                     
--------------------+-----------------------------------------------------------

real    0m52.381s
user    0m0.145s
sys        0m0.065s
================================================================================

$ time gphoto2 --get-thumbnail 1-10
Downloading '_DSC0715.JPG' from folder '/store_00010001/DCIM/100VINCE'...
...
[Skip]
...
Enregistrement du fichier en thumb__DSC0719.NEF

real    0m52.337s
user    0m0.073s
sys        0m0.108s
================================================================================

Note the +- 50 seconds of difference seems the "handshake" (if there's one), when the first result appears, all others results follows immediately.


Is this a well-know behaviour of the PTP or just with Nikon cameras ?

Is other users having the same experience ?

Regards,

Vincent

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j


_______________________________________________
Gphoto-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gphoto-devel

I've found this report on digikam:
https://bugs.kde.org/show_bug.cgi?id=336022

It is not exactly the same problem but I think the difference is in the way that digikam import all the content of the memory card (i.e. not with '--get-all-files' nor with '--get-file 1-last' but with '--get-file x' in a loop wich enumerate the indexes of files).

So, if I download all the content of the camera with '--get-all-files' it take 55 seconds before the first import and all the 938 files are imported in 9 min. 5 sec. (+- 0.58 sec/picture) and that's all right for me.

This is clearly the "handshake" the problem.

Vincent

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Gphoto-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gphoto-devel
Reply | Threaded
Open this post in threaded view
|

Re: Slow execution when many files

Marcus Meissner-4
On Thu, May 26, 2016 at 07:42:17PM +0200, Vincent Vande Vyvre wrote:

> Le 26/05/2016 06:30, Vincent Vande Vyvre a écrit :
> > Hi,
> >
> > I discover a strange behaviour with a Nikon D5000.
> >
> > When the memory card has several pictures the communication between
> > gphoto2 and the camera became very slowly.
> >
> > Exemple with 40 files:
> >
> > $ time gphoto2 --auto-detect
> > Modèle Port
> > ----------------------------------------------------------
> > Nikon DSC D5000 (PTP mode)     usb:001,032
> >
> > $ time gphoto2 --list-files
> > Le dossier « / » ne contient aucun fichier.
> > Le dossier « /store_00010001 » ne contient aucun fichier.
> > Le dossier « /store_00010001/DCIM » ne contient aucun fichier.
> > Le dossier « /store_00010001/DCIM/100VINCE » contient 40 fichiers.
> > #1     _DSC1184.JPG               rd  3375 KB 4288x2848 image/jpeg
> > #2     _DSC1184.NEF               rd  9277 KB application/x-unknown
> > ...
> > [Skip for brevity]
> > ...
> > #39    _DSC1203.JPG               rd  3931 KB 4288x2848 image/jpeg
> > #40    _DSC1203.NEF               rd 10403 KB application/x-unknown
> >
> > real *0m2.395s*
> > user    0m0.030s
> > sys        0m0.030s
> > ================================================================================
> >
> > $ time gphoto2 --show-exif 1-10
> > Downloading '_DSC1184.JPG' from folder '/store_00010001/DCIM/100VINCE'...
> > Balises EXIF :
> > --------------------+-----------------------------------------------------------
> > Balise |Valeur
> > --------------------+-----------------------------------------------------------
> > Make                |NIKON CORPORATION
> > Model               |NIKON D5000
> > ...
> > [Skip]
> > ...
> > FlashPixVersion     |FlashPix version 1.0
> > ColorSpace          |Erreur interne (valeur inconnue 65535)
> > --------------------+-----------------------------------------------------------
> >
> > real *0m2.884s*
> > user    0m0.043s
> > sys        0m0.030s
> > ================================================================================
> >
> > $ time gphoto2 --get-thumbnail 1-10
> > Downloading '_DSC1184.JPG' from folder '/store_00010001/DCIM/100VINCE'...
> > Enregistrement du fichier en thumb__DSC1184.jpg
> > Downloading '_DSC1184.NEF' from folder '/store_00010001/DCIM/100VINCE'...
> > ...
> > [Skip]
> > ...
> > Downloading '_DSC1188.NEF' from folder '/store_00010001/DCIM/100VINCE'...
> > Enregistrement du fichier en thumb__DSC1188.NEF
> >
> > real *0m2.827s*
> > user    0m0.036s
> > sys        0m0.030s
> > ================================================================================
> >
> > These execution times seems reasonable for me. Now with 938 files into
> > the card:
> >
> > $ time gphoto2 --list-files > nikon.log
> > Le dossier « / » ne contient aucun fichier.
> > Le dossier « /store_00010001 » ne contient aucun fichier.
> > Le dossier « /store_00010001/DCIM » ne contient aucun fichier.
> > Le dossier « /store_00010001/DCIM/100VINCE » contient 938 fichiers.
> > #1     _DSC0715.JPG               rd  5779 KB 4288x2848 image/jpeg
> > #2     _DSC0715.NEF               rd 10739 KB application/x-unknown
> > ...
> > [Skip]
> > ...
> > #937   _DSC1183.JPG               rd  3151 KB 4288x2848 image/jpeg
> > #938   _DSC1183.NEF               rd 10232 KB application/x-unknown
> >
> > real *0m51.957s*
> > user    0m0.188s
> > sys        0m0.058s
> > ================================================================================
> >
> > $ time gphoto2 --show-exif 1-10
> > Balises EXIF :
> > --------------------+-----------------------------------------------------------
> > Balise |Valeur
> > --------------------+-----------------------------------------------------------
> > Make                |NIKON CORPORATION
> > Model               |NIKON D5000
> > ...
> > [Skip]
> > ...
> > FlashPixVersion     |FlashPix version 1.0
> > ColorSpace          |Erreur interne (valeur inconnue 65535)
> > --------------------+-----------------------------------------------------------
> >
> > real *0m52.381s*
> > user    0m0.145s
> > sys        0m0.065s
> > ================================================================================
> >
> > $ time gphoto2 --get-thumbnail 1-10
> > Downloading '_DSC0715.JPG' from folder '/store_00010001/DCIM/100VINCE'...
> > ...
> > [Skip]
> > ...
> > Enregistrement du fichier en thumb__DSC0719.NEF
> >
> > real *0m52.337s*
> > user    0m0.073s
> > sys        0m0.108s
> > ================================================================================
> >
> > Note the +- 50 seconds of difference seems the "handshake" (if there's
> > one), when the first result appears, all others results follows
> > immediately.
> >
> >
> > Is this a well-know behaviour of the PTP or just with Nikon cameras ?
> >
> > Is other users having the same experience ?
> >
> > Regards,
> >
> > Vincent
> >
> >
> > ------------------------------------------------------------------------------
> > Mobile security can be enabling, not merely restricting. Employees who
> > bring their own devices (BYOD) to work are irked by the imposition of MDM
> > restrictions. Mobile Device Manager Plus allows you to control only the
> > apps on BYO-devices by containerizing them, leaving personal data untouched!
> > https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
> >
> >
> > _______________________________________________
> > Gphoto-devel mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/gphoto-devel
>
> I've found this report on digikam:
> https://bugs.kde.org/show_bug.cgi?id=336022
>
> It is not exactly the same problem but I think the difference is in the way
> that digikam import all the content of the memory card (i.e. not with
> '--get-all-files' nor with '--get-file 1-last' but with '--get-file x' in a
> loop wich enumerate the indexes of files).
>
> So, if I download all the content of the camera with '--get-all-files' it
> take 55 seconds before the first import and all the 938 files are imported
> in 9 min. 5 sec. (+- 0.58 sec/picture) and that's all right for me.
>
> This is clearly the "handshake" the problem.

The core issue that we need to read the full directory information if we need
just 1 file from it. As all 968 files are probably in one directory, we need
to read all their meta information.

After we read it once it is cached and there are no further delays.

This is mostly due to PTP object handling which is backwards (linking
from files to directories upwards to the top instead of having
collections/directory listing).

Ciao, Marcus

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Gphoto-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gphoto-devel
Reply | Threaded
Open this post in threaded view
|

Re: Slow execution when many files

Vincent Vande Vyvre
Le 26/05/2016 23:01, Marcus Meissner a écrit :

> On Thu, May 26, 2016 at 07:42:17PM +0200, Vincent Vande Vyvre wrote:
>> Le 26/05/2016 06:30, Vincent Vande Vyvre a écrit :
>> ...
> The core issue that we need to read the full directory information if we need
> just 1 file from it. As all 968 files are probably in one directory, we need
> to read all their meta information.
>
> After we read it once it is cached and there are no further delays.
>
> This is mostly due to PTP object handling which is backwards (linking
> from files to directories upwards to the top instead of having
> collections/directory listing).
>
> Ciao, Marcus
>

Ok, clear.

Thanks for the explanation, Marcus.

Best,

Vincent

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Gphoto-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gphoto-devel