Datasheet downloads timeout on Panasonic site due to fake Chrome User-Agent

[klausman]

Describe the bug
Datasheet downloads timeout for unclear reasons.

To Reproduce

Run php bin/console partdb:attachments:download -n -vvv to fetch missing attachments.

Expected behavior
Have the attachments downloaded

Screenshots

$ php bin/console partdb:attachments:download -n -vvv

 ! [NOTE] Found 6 attachments with external URL, that will be downloaded.

0/6 [░░░░░░░░░░░░░░░░░░░░░░░░░░░░]   0%           < 1 ms/< 1 ms
Starting download...09:42:00 INFO      [http_client] Request: "GET https://industrial.panasonic.com/cdbs/www-data/pdf/RDF0000/ABA0000C1259.pdf"
[
  "command" => [
    "name" => "partdb:attachments:download",
    "arguments" => [
      "command" => "partdb:attachments:download"
    ]
  ],
  "token" => null
2/6 [▓▓▓▓▓▓▓▓▓░░░░░░░░░░░░░░░░░░░]  33%            1 min/3 min
EEU-FR1A101B View All Specifications (ID: 190) from industrial.panasonic.com09:43:00 INFO      [http_client] Request: "GET https://industrial.panasonic.com/ww/products/pt/aluminum-cap-lead/models/EEUFR1A101B"
[
  "command" => [
    "name" => "partdb:attachments:download",
    "arguments" => [
      "command" => "partdb:attachments:download"
    ]
  ],
  "token" => null
3/6 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░░░░░░░░░]  50%            2 min/4 min
FR-A Series, Radial Lead Type (ID: 213) from industrial.panasonic.com09:44:00 INFO      [http_client] Request: "GET https://industrial.panasonic.com/cdbs/www-data/pdf/RDF0000/ABA0000C1259.pdf"
[
  "command" => [
    "name" => "partdb:attachments:download",
    "arguments" => [
      "command" => "partdb:attachments:download"
    ]
  ],
  "token" => null
4/6 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░░░░░]  66%            3 min/4 min, 30 s
FR-A Series, Radial Lead Type (ID: 218) from industrial.panasonic.com09:45:00 INFO      [http_client] Request: "GET https://industrial.panasonic.com/cdbs/www-data/pdf/RDF0000/ABA0000C1259.pdf"
[
  "command" => [
    "name" => "partdb:attachments:download",
    "arguments" => [
      "command" => "partdb:attachments:download"
    ]
  ],
  "token" => null
5/6 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░]  83%            4 min/4 min, 48 s
FR-A Series, Radial Lead Type (ID: 228) from industrial.panasonic.com09:46:00 INFO      [http_client] Request: "GET https://industrial.panasonic.com/cdbs/www-data/pdf/RDF0000/ABA0000C1259.pdf"
[
  "command" => [
    "name" => "partdb:attachments:download",
    "arguments" => [
      "command" => "partdb:attachments:download"
    ]
  ],
  "token" => null
6/6 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%            5 min/5 min
FR-A Series, Radial Lead Type (ID: 262) from industrial.panasonic.com09:47:00 INFO      [http_client] Request: "GET https://industrial.panasonic.com/cdbs/www-data/pdf/RDF0000/ABA0000C1259.pdf"
[
  "command" => [
    "name" => "partdb:attachments:download",
    "arguments" => [
      "command" => "partdb:attachments:download"
    ]
  ],
  "token" => null
6/6 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%            6 min/6 min
FR-A Series, Radial Lead Type (ID: 262) from industrial.panasonic.com


 [WARNING] Some attachments could not be downloaded:
 [WARNING] Attachment FR-A Series, Radial Lead Type (ID 189) could not be downloaded from https://industrial.panasonic.com/cdbs/www-data/pdf/RDF0000/ABA0000C1259.pdf:
           Transport error: Idle timeout reached for "https://industrial.panasonic.com/cdbs/www-data/pdf/RDF0000/ABA0000C1259.pdf".
 [WARNING] Attachment EEU-FR1A101B View All Specifications (ID 190) could not be downloaded from https://industrial.panasonic.com/ww/products/pt/aluminum-cap-lead/models/EEUFR1A101B:
           Transport error: Idle timeout reached for "https://industrial.panasonic.com/ww/products/pt/aluminum-cap-lead/models/EEUFR1A101B".
 [WARNING] Attachment FR-A Series, Radial Lead Type (ID 213) could not be downloaded from https://industrial.panasonic.com/cdbs/www-data/pdf/RDF0000/ABA0000C1259.pdf:
           Transport error: Idle timeout reached for "https://industrial.panasonic.com/cdbs/www-data/pdf/RDF0000/ABA0000C1259.pdf".
 [WARNING] Attachment FR-A Series, Radial Lead Type (ID 218) could not be downloaded from https://industrial.panasonic.com/cdbs/www-data/pdf/RDF0000/ABA0000C1259.pdf:
           Transport error: Idle timeout reached for "https://industrial.panasonic.com/cdbs/www-data/pdf/RDF0000/ABA0000C1259.pdf".
 [WARNING] Attachment FR-A Series, Radial Lead Type (ID 228) could not be downloaded from https://industrial.panasonic.com/cdbs/www-data/pdf/RDF0000/ABA0000C1259.pdf:
           Transport error: Idle timeout reached for "https://industrial.panasonic.com/cdbs/www-data/pdf/RDF0000/ABA0000C1259.pdf".
 [WARNING] Attachment FR-A Series, Radial Lead Type (ID 262) could not be downloaded from https://industrial.panasonic.com/cdbs/www-data/pdf/RDF0000/ABA0000C1259.pdf:
           Transport error: Idle timeout reached for "https://industrial.panasonic.com/cdbs/www-data/pdf/RDF0000/ABA0000C1259.pdf".

Server Side

• Part-DB Version: 2.12.2 (/98df91d7)
• PHP Version: 8.4.22-1
• Database Server: Postgres 18+292

Additional context

Curl'ing the same URLs as listed in the log form the very same server works fine and is fast. I am not sure what an "Idle timeout" is (as opposed to, say, a request timeout). It would also be nice to have a simple way of knowing which part a datasheet was for, so I could work around this by uploading it manually.

Edited to add:

I did a quick check with tshark, and it doesn't seem like there are any outgoing packets to industrial.panasonic.com while the script is running, so something internal is amiss.

[klausman]

This is caused by the fake Chrome headers. As I understand it, those were added for DigiKey's bot tarpit/protection, but it seems Panasonic doesn't like those. As a test, I removed all the special browser headers and used curl's user agent, and downloads worked again.

I dunno what the right fix here is. Maybe make the UA headers user-configurable or something? Alternatively, an auto-fallback/retry on timeout might be an approach? Finally, just using a fixed but "honest" UA might work, but that would probably regress the DK path.