Speeding up USB PTP transfer

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Speeding up USB PTP transfer

Paulo Neves
Hello there,

I have gphoto2 running in an embedded controller and I would like to know if there is any way that the PTP downloading of the pictures can be sped up. If so what part of the code would it involve? With another machine I get much faster transfer rates. The embedded computer is not limited in the USB part because it works well transferring files over USB
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up USB PTP transfer

Marcus Meissner
On Sat, May 14, 2016 at 06:46:00AM -0700, Paulo Neves wrote:
> Hello there,
>
> I have gphoto2 running in an embedded controller and I would like to know if
> there is any way that the PTP downloading of the pictures can be sped up. If
> so what part of the code would it involve? With another machine I get much
> faster transfer rates. The embedded computer is not limited in the USB part
> because it works well transferring files over USB

Is this also with libgphoto2? Then its probably not libgphoto2 itself.

USB 1 vs USB 2 host controller perhaps?

We queue 4k BULK transfers, this should still saturate the bus sufficiently.

Ciao, Marcus

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Gphoto-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gphoto-devel
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up USB PTP transfer

Paulo Neves
Hey Marcus

This is with both libphoto2 and with gphoto2. In my program its seems to be a bit faster but even so it more or less doubles with different machines. Both the machines have integrated USB 2.0 host so the bottleneck should not be that.

What part of the code is responsible for the transfer size. Maybe I could try to tweak it and see what happens. The computer I am running is an Arietta G25. http://www.acmesystems.it/arietta.
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up USB PTP transfer

Marcus Meissner-4
On Sat, May 14, 2016 at 07:37:49AM -0700, Paulo Neves wrote:

> Hey Marcus
>
> This is with both libphoto2 and with gphoto2. In my program its seems to be
> a bit faster but even so it more or less doubles with different machines.
> Both the machines have integrated USB 2.0 host so the bottleneck should not
> be that.
>
> What part of the code is responsible for the transfer size. Maybe I could
> try to tweak it and see what happens. The computer I am running is an
> Arietta G25. http://www.acmesystems.it/arietta.

camlibs/ptp2/usb.c in ptp_usb_getdata  is the bulk transfer.

The buffer size is READLEN   ... which actually seems to be 64kb already, only the write
buffer is 4kb.

Ciao, Marcus

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Gphoto-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gphoto-devel
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up USB PTP transfer

Fazekas László
In reply to this post by Marcus Meissner
2016-05-14 17:26 keltezéssel, Marcus Meissner írta:

> On Sat, May 14, 2016 at 06:46:00AM -0700, Paulo Neves wrote:
>> Hello there,
>>
>> I have gphoto2 running in an embedded controller and I would like to know if
>> there is any way that the PTP downloading of the pictures can be sped up. If
>> so what part of the code would it involve? With another machine I get much
>> faster transfer rates. The embedded computer is not limited in the USB part
>> because it works well transferring files over USB
> Is this also with libgphoto2? Then its probably not libgphoto2 itself.
>
> USB 1 vs USB 2 host controller perhaps?
>
> We queue 4k BULK transfers, this should still saturate the bus sufficiently.
>
> Ciao, Marcus
Are you sure the transfer speed is the real problem? Maybe it's waiting
for something with a timeout.
What happens with lower resolution pictures?

Fazek

> ------------------------------------------------------------------------------
> Mobile security can be enabling, not merely restricting. Employees who
> bring their own devices (BYOD) to work are irked by the imposition of MDM
> restrictions. Mobile Device Manager Plus allows you to control only the
> apps on BYO-devices by containerizing them, leaving personal data untouched!
> https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
> _______________________________________________
> Gphoto-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/gphoto-devel
>
>


------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Gphoto-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gphoto-devel
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up USB PTP transfer

Paulo Neves
Thank you for your replies. Changing resolution had the expected outcome of making the transfer happen faster. I am not really sure how I can debug such an issue.

I still have not made the change to the bulk buffers length but I have run gphoto2 with debug and indeed I have some timeouts ocurring. I dont know what they mean exactly. But here are relevant logs

0.819162 gp_port_set_timeout         (2): Setting port timeout to 20000 milliseconds.
0.819273 ptp                         (2): event: nparams=0x1, code=0xC203, trans_id=0xFFFFFFFF, p1=0x5004
, p2=0x0, p3=0x0
0.819412 camera_sony_capture         (2): during event.code=c203 Param1=00005004
0.819511 gp_port_get_timeout         (2): Current port timeout is 20000 milliseconds.
0.819591 gp_port_set_timeout         (2): Setting port timeout to 150 milliseconds.
0.819676 gp_port_check_int           (3): Reading 24 = 0x18 bytes from interrupt endpoint...
0.974826 gp_port_check_int           (3): Reading 24 = 0x18 bytes from interrupt endpoint...
1.131070 gp_port_set_timeout         (2): Setting port timeout to 20000 milliseconds.
1.131317 ptp_usb_event [usb.c:492]   (0): Reading PTP event failed: Timeout reading from or writing to the port (-10)
1.131436 gp_port_get_timeout         (2): Current port timeout is 20000 milliseconds.
1.131519 gp_port_set_timeout         (2): Setting port timeout to 150 milliseconds.
1.131602 gp_port_check_int           (3): Reading 24 = 0x18 bytes from interrupt endpoint...
1.287334 gp_port_check_int           (3): Reading 24 = 0x18 bytes from interrupt endpoint...
1.374032 _cb_irq                     (2): 0x41e88 with status 0
1.374205 _cb_irq                     (3): interrupt (hexdump of 16 bytes)
0000  10 00 00 00 04 00 03 c2-ff ff ff ff 1d d2 00 00  ................

7.039238 gp_port_set_timeout         (2): Setting port timeout to 20000 milliseconds.
7.039347 ptp                         (2): event: nparams=0x1, code=0xC203, trans_id=0xFFFFFFFF, p1=0xD21D
, p2=0x0, p3=0x0
7.039449 camera_wait_for_event       (2): code=0xc203, param1 0x0000d21d
7.039556 camera_wait_for_event       (2): waiting for events timeout 1 ms
7.039648 gp_port_get_timeout         (2): Current port timeout is 20000 milliseconds.
7.039726 gp_port_set_timeout         (2): Setting port timeout to 150 milliseconds.
7.039807 gp_port_check_int           (3): Reading 24 = 0x18 bytes from interrupt endpoint...
7.193558 gp_port_check_int           (3): Reading 24 = 0x18 bytes from interrupt endpoint...
7.349797 gp_port_set_timeout         (2): Setting port timeout to 20000 milliseconds.
7.349998 ptp_usb_event [usb.c:492]   (0): Reading PTP event failed: Timeout reading from or writing to the port (-10)
7.350097 camera_wait_for_event       (2): no events received.

7.598700 gp_port_set_timeout         (2): Setting port timeout to 20000 milliseconds.
7.598814 ptp                         (2): event: nparams=0x1, code=0xC203, trans_id=0xFFFFFFFF, p1=0xD21D
, p2=0x0, p3=0x0
7.598915 camera_sony_capture         (2): during event.code=c203 Param1=0000d21d
7.599004 gp_port_get_timeout         (2): Current port timeout is 20000 milliseconds.
7.599080 gp_port_set_timeout         (2): Setting port timeout to 150 milliseconds.
7.599158 gp_port_check_int           (3): Reading 24 = 0x18 bytes from interrupt endpoint...
7.756056 gp_port_check_int           (3): Reading 24 = 0x18 bytes from interrupt endpoint...
7.912295 gp_port_set_timeout         (2): Setting port timeout to 20000 milliseconds.
7.912494 ptp_usb_event [usb.c:492]   (0): Reading PTP event failed: Timeout reading from or writing to th
e port (-10)
7.912612 gp_port_get_timeout         (2): Current port timeout is 20000 milliseconds.
7.912694 gp_port_set_timeout         (2): Setting port timeout to 150 milliseconds.
7.912777 gp_port_check_int           (3): Reading 24 = 0x18 bytes from interrupt endpoint...
8.068552 gp_port_check_int           (3): Reading 24 = 0x18 bytes from interrupt endpoint...
8.086027 _cb_irq                     (2): 0x41948 with status 0
8.086206 _cb_irq                     (3): interrupt (hexdump of 16 bytes)
0000  10 00 00 00 04 00 01 c2-ff ff ff ff 01 c0 ff ff  ................

8.086318 _cb_irq                     (2): requeuing complete transfer 0x41948

Reply | Threaded
Open this post in threaded view
|

Re: Speeding up USB PTP transfer

Marcus Meissner-4
On Mon, May 16, 2016 at 02:58:20AM -0700, Paulo Neves wrote:
> Thank you for your replies. Changing resolution had the expected outcome of
> making the transfer happen faster. I am not really sure how I can debug such
> an issue.
>
> I still have not made the change to the bulk buffers length but I have run
> gphoto2 with debug and indeed I have some timeouts ocurring. I dont know
> what they mean exactly. But here are relevant logs


These are interrupt timeouts. Usually libgphoto2 is waiting for something
from the camera during this time, so they are probably unavoidable.


> 1.374032 _cb_irq                     (2): 0x41e88 with status 0
> 1.374205 _cb_irq                     (3): interrupt (hexdump of 16 bytes)
> 0000  10 00 00 00 04 00 03 c2-ff ff ff ff 1d d2 00 00  ................
>
> 7.039238 gp_port_set_timeout         (2): Setting port timeout to 20000
> milliseconds.
> 7.039347 ptp                         (2): event: nparams=0x1, code=0xC203,
> trans_id=0xFFFFFFFF, p1=0xD21D
> , p2=0x0, p3=0x0

Is there really a 6 second pause here or did you truncate the logfile in between?

Ciao, Marcus

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Gphoto-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gphoto-devel
Reply | Threaded
Open this post in threaded view
|

Re: Speeding up USB PTP transfer

Paulo Neves
Hello.

Sorry that I didnt add the [...]. I copy pasted segments of the debug
log to make it shorter.

I solved the problem. It was the READLEN which was saturating the bulk
read. I got this information through the number of bytes read by each
operation. It matched the READLEN value. I kept increasing the value and
the performance increased dramatically, it halved the time. I settled
for a READLEN of 1000KByte.

I think my embedded computer is just too slow when compared to a DMA USB
host, or at least its my theory.

Best regards
Paulo Neves

On 16-05-2016 15:17, Marcus Meissner wrote:

> On Mon, May 16, 2016 at 02:58:20AM -0700, Paulo Neves wrote:
>> Thank you for your replies. Changing resolution had the expected outcome of
>> making the transfer happen faster. I am not really sure how I can debug such
>> an issue.
>>
>> I still have not made the change to the bulk buffers length but I have run
>> gphoto2 with debug and indeed I have some timeouts ocurring. I dont know
>> what they mean exactly. But here are relevant logs
>
> These are interrupt timeouts. Usually libgphoto2 is waiting for something
> from the camera during this time, so they are probably unavoidable.
>
>
>> 1.374032 _cb_irq                     (2): 0x41e88 with status 0
>> 1.374205 _cb_irq                     (3): interrupt (hexdump of 16 bytes)
>> 0000  10 00 00 00 04 00 03 c2-ff ff ff ff 1d d2 00 00  ................
>>
>> 7.039238 gp_port_set_timeout         (2): Setting port timeout to 20000
>> milliseconds.
>> 7.039347 ptp                         (2): event: nparams=0x1, code=0xC203,
>> trans_id=0xFFFFFFFF, p1=0xD21D
>> , p2=0x0, p3=0x0
> Is there really a 6 second pause here or did you truncate the logfile in between?
>
> Ciao, Marcus


------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Gphoto-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/gphoto-devel