1

How might I find out the maximum number of threads I can use? I'm new to programming, so I'm trying to figure out what I need to know. I have 950 GB of files to download from Amazon S3. I was thinking I could use two threads that each download half of that amount, running in parallel.

I'm hoping to speed up that bulk data download from Amazon S3 by multithreading my application, but it would be good to first know if my computer even supports multithreading.

Computer specs:

  • Model: MacBook (13-inch, Mid 2010, A1342)
  • Processor: 2.4 GHz Intel Core Duo
  • Memory: 8 GB 1067 MHz DDR3
  • Software: MacOS Sierra 10.12.6

The processor is very old and has been discontinued, so it doesn't support Intel hyperthreading. I'm not sure if it's any good with multithreading. But if it can support multithreading, how many threads can it support, and would there be an impact on performance if I use too many threads?

Please correct me if I am wrong, but I read that a non-hyperthreading core can support only 1 thread, so I'm thinking maybe my processor can support up to 2 threads, then any more threads after that would have negligible effect.

Giacomo1968
  • 53,069
  • 19
  • 162
  • 212
brienna
  • 135
  • 1
  • 5
  • 1
    Sounds like you need to read up on [Multitasking](https://en.wikipedia.org/wiki/Computer_multitasking). It’s very important to tell I/O-bound work from CPU-bound work. – Daniel B Jun 10 '18 at 18:10
  • What exactly do you mean when you say, “I'm hoping to speed up some bulk data download from Amazon S3…” What is this? Are you downloading files? Or something else? Amazon S3 is just a file storage setup. I’m not too sure how multithreading would help a bandwidth-bound task like this. – Giacomo1968 Jun 10 '18 at 18:17
  • I'm new to programming, so I'm trying to figure out what I need to know. I have 950 GB of files to download from Amazon S3. I was thinking I could use two threads that each download half of that amount, running in parallel. @JakeGould – brienna Jun 10 '18 at 18:19
  • 1
    @briennakh As [stated in this answer](https://superuser.com/a/1330145/167207), downloading files is not a CPU intensive task by any measure. You can launch multiple processes that can attempt to download more than one file in parallel, but it won’t be a CPU bound task. It’s all dependent on bandwidth and your network connection. – Giacomo1968 Jun 10 '18 at 18:23
  • Ahhh, ok. Thank you, I appreciate your time to answer my noob question. The reason why I had thought this would work is because of this recommendation: https://boto3.readthedocs.io/en/latest/guide/resources.html#multithreading-multiprocessing – brienna Jun 10 '18 at 18:23
  • @briennakh Cool! Happy to help. Just so you know, the main benefit of running two processes for file downloading is you are spreading your bets that you will have no downtime. Meaning, if you have one download going but then it slows down… Nothing can pick up the slack. But if you have—let’s say—two downloads going and one slows down, the second process can take advantage of the slow down of the other process to use more network bandwidth. A single process can never juggle tasks like that. – Giacomo1968 Jun 10 '18 at 18:47

2 Answers2

3

There are several false assumptions in your question so let me clarify here:

  1. Multithreading is possible even on single-core CPUs. The execution just splits in time so you have 1 thread active in any given moment, however the overall threads execution is parallel.
  2. Multithreading is not equivalent to speed-up. If naively designed, multithreading app is always slower than single-threaded app by tens or even hundreds of times.
  3. Downloading from outer networks in 99% of cases is not depending to multithreading. It may be worth to take a look to multi-range HTTP downloading like it was implemented in apps like NetAnts, ReGet and others from dial-up modem era. However, there is no warranty that multi-range downloading could be faster that regular one, due various network policies and involved server(s).

So, all after all. You definitely have to collect metrics in your app: What is slowest operation? And then re-design it to get more speed.

Yury Schkatula
  • 190
  • 1
  • 7
2

How might I find out the maximum number of threads I can use?

Intel Core Duo has 2 threads due to it having a 2 core single thread processor. Each core has the capability of a single thread.

But if it can support multithreading, how many threads can it support, and would there be an impact on performance if I use too many threads?

It supports 2 threads.

so I'm thinking maybe my processor can support up to 2 threads, then any more threads after that would have negligible effect.

It’s incapable of handling more than 2 threads. Your multi-threaded application would only have two threads since you can only generate 2 threads due to the hardware.

Ramhound
  • 41,734
  • 35
  • 103
  • 130