How can I only allow say 10 concurrent torrents?

simur612

Member
May 25, 2018
879
0
16
Hi all I'm a bit new to rtorrent & rutorrent, but I'm looking for a way to set a number of concurrent active torrents, like so a portion of my torrent list are active and a portion queued, because my server is having trouble with the 36 I am currently seeding/downloading and I'm getting a lot of "Storage error: [File chunk read error: Cannot allocate memory]" errors, which I'm guessing is down to this.

If anyone can help I'd be very grateful.
 

randac56

Member
May 25, 2018
915
0
16
you can't.

This is not how rtorrent works. Rtorrent works on global settings which happen to be much more efficient.

The entire idea of limiting "per torrent" is for programs which have to spawn a new process per torrent. With rtorrent, you create a "pool" of total connections and total speed, and the program splits them up among all running torrents.

What this means is you can let a use run 4 torrents or 300 torrents and it will have the same affect on the system.
 

peshua19

Member
May 25, 2018
897
0
16
What "server" do you have?
Can you provide all peer/connection/memory settings from your .rtorrent.rc?
 

randac56

Member
May 25, 2018
915
0
16
What "server" do you have?
Can you provide all peer/connection/memory settings from your .rtorrent.rc?
my current main seedbox is a 1.2 ghz single core celeron and i have no problem with 100-200 torrents...i've even had up to 400 torrents going.

my rtorrent.rc is simple.
Code:
min_peers = 10
max_peers = 200
min_peers_seed =1
max_peers_seed = 100
max_uploads = 30
#max_uploads_global = 400
#max_downloads_global = 300

download_rate = 0
upload_rate = 0

directory = /usr/home/rtuser1/rtorrent/download/
session = /usr/home/rtuser1/.session
schedule = low_diskspace,5,60,close_low_diskspace=512M

#ip =
#bind =
check_hash = no
use_udp_trackers = yes
encryption = allow_incoming,enable_retry,prefer_plaintext
dht = auto
peer_exchange = no
hash_read_ahead = 10
hash_interval = 10
hash_max_tries = 5
#max_open_sockets = 16424
#send_buffer_size = 78840
#receive_buffer_size = 78840
max_open_files = 512
execute = {sh,-c,/usr/local/bin/php /usr/local/www/rutorrent/php/initplugins.php wonslung &}
umask = 0002
log.execute=/tmp/rtorrent.log
port_range = 51001-51001
scgi_port = 127.0.0.1:5001
dht_port = 10001
 

randac56

Member
May 25, 2018
915
0
16
also, it should be noted:

the file chunk read error is normally caused by memory or filesystem issues.

I've seen this error a LOT more on linux based machines (over BSD based machines)
 

simur612

Member
May 25, 2018
879
0
16
Hmm. Thanks for the replies first of all... Second, do you know of any way to correct this "Storage error: [File chunk write error: Cannot allocate memory.]" issue? I've read its to do with max_memory_usage and upload slots, but I'm not sure how to fix it, and don't want to go tweaking things willy-nilly because I've only just got it (sorta) working.

Here are my settings:
Code:
# This is an example resource file for rTorrent. Copy to
# ~/.rtorrent.rc and enable/modify the options as needed. Remember to
# uncomment the options you wish to enable.

#RSS
schedule = rss,0,00:30:00,"execute={sh,-c,php /var/www/rtorrent/plugins/rss/update.php& exit 0}"

# Maximum and minimum number of peers to connect to per torrent.
#min_peers = 40
#max_peers = 100

# Same as above but for seeding completed torrents (-1 = same as downloading)
#min_peers_seed = 10
#max_peers_seed = 50

# Maximum number of simultanious uploads per torrent.
#max_uploads = 15

# Global upload and download rate in KiB. "0" for unlimited.
#download_rate = 0
upload_rate = 50

# Default directory to save the downloaded torrents.
directory = /media/store2/torrents/incomplete

# Default session directory. Make sure you don't run multiple instance
# of rtorrent using the same session directory. Perhaps using a
# relative path?
session = /media/store2/torrents/session

# Watch a directory for new torrents, and stop those that have been
# deleted.
#schedule = watch_directory,5,5,load_start=/media/store2/torrents/addfiles/*.torrent
#system.method.set_key = event.download.finished,remove_torrent,"execute=rm,$d.get_tied_to_file="
#system.method.set_key = event.download.finished,move_complete,"execute=mv,$d.get_base_path=,/media/store2/torrents/complete"
#schedule = untied_directory,5,5,stop_untied=

# Close torrents when diskspace is low.
schedule = low_diskspace,5,60,close_low_diskspace=100M

# Stop torrents when reaching upload ratio in percent,
# when also reaching total upload in bytes, or when
# reaching final upload ratio in percent.
# example: stop at ratio 2.0 with at least 200 MB uploaded, or else ratio 20.0
#schedule = ratio,60,60,"stop_on_ratio=200,200M,2000"

# The ip address reported to the tracker.
#ip = 127.0.0.1
#ip = rakshasa.no

# The ip address the listening socket and outgoing connections is
# bound to.
#bind = 127.0.0.1
#bind = rakshasa.no

# Port range to use for listening.
#port_range = 6890-6999

# Start opening ports at a random position within the port range.
#port_random = no

# Check hash for finished torrents. Might be usefull until the bug is
# fixed that causes lack of diskspace not to be properly reported.
#check_hash = no

# Set whetever the client should try to connect to UDP trackers.
#use_udp_trackers = yes

# Alternative calls to bind and ip that should handle dynamic ip's.
#schedule = ip_tick,0,1800,ip=rakshasa
#schedule = bind_tick,0,1800,bind=rakshasa

# Encryption options, set to none (default) or any combination of the following:
# allow_incoming, try_outgoing, require, require_RC4, enable_retry, prefer_plaintext
#
# The example value allows incoming encrypted connections, starts unencrypted
# outgoing connections but retries with encryption if they fail, preferring
# plaintext to RC4 encryption after the encrypted handshake
#
# encryption = allow_incoming,enable_retry,prefer_plaintext

# Enable DHT support for trackerless torrents or when all trackers are down.
# May be set to "disable" (completely disable DHT), "off" (do not start DHT),
# "auto" (start and stop DHT as needed), or "on" (start DHT immediately).
# The default is "off". For DHT to work, a session directory must be defined.
#
# dht = auto

# UDP port to use for DHT.
#
# dht_port = 6881

# Enable peer exchange (for torrents not marked private)
#
# peer_exchange = yes

#
# Do not modify the following parameters unless you know what you're doing.
#

# Hash read-ahead controls how many MB to request the kernel to read
# ahead. If the value is too low the disk may not be fully utilized,
# while if too high the kernel might not be able to keep the read
# pages in memory thus end up trashing.
#hash_read_ahead = 10

# Interval between attempts to check the hash, in milliseconds.
#hash_interval = 100

# Number of attempts to check the hash while using the mincore status,
# before forcing. Overworked systems might need lower values to get a
# decent hash checking rate.
#hash_max_tries = 10


#The Following Are VERY Important
encoding_list = UTF-8
scgi_local = /tmp/rpc.socket
schedule = chmod,0,0,"execute=chmod,777,/tmp/rpc.socket"

Thanks again in advance, and incase I don't get to say it elsewhere to you kudos for thinking about implementing a bounty system - best idea I've heard of to use in a a free open-source project in a while. Once everything's dandy and I find things that I'd like improved/added I'm deffo going to put some up!
 

randac56

Member
May 25, 2018
915
0
16
I've seen that error mainly when running out of ram on a linux based system. I don't think i've ever seen it happen on my bsd systems....you could always change to FreeBSD =)

did you try limiting the ram available? how much ram do you have?
 

simur612

Member
May 25, 2018
879
0
16
The RAM on the system is 512MB. I'm not sure how to limit available RAM, care to explain? And I don't really want to change to freeBSD at present, I've only just got everything almost perfectly configured with SAMBA etc. under ubuntu server, I really don't want to make a jump just yet. Maybe in the future when I have a secondary server and the chance to try it out without totally disabling my primary source of data on the network.

Regatds
 

lisas4567

Member
May 25, 2018
773
0
16
to set RAM usage , use something like :

max_memory_usage : 524288

This means it will use max 512MB
 

randac56

Member
May 25, 2018
915
0
16
to set RAM usage , use something like :

max_memory_usage : 524288

This means it will use max 512MB
i feel you.

512 MB ram is quite small. you might want to also limit the number of open files and the send/recieve speed

it's all in the rtorrent wiki
 

simur612

Member
May 25, 2018
879
0
16
Bleh, I know, I must try and get some more DDR ram!

I've tried updating with the following according to a few random google search results, hopefully it will help.

send_buffer_size = 4096
receive_buffer_size = 4096

I'll reply if problem is resolved in case any others are facing these issues on low-spec PCs.
 

randac56

Member
May 25, 2018
915
0
16
might also look at this setting:

Code:
max_open_files =
I think it defaults to 128 or something, maybe set it to 64 if you keep having trouble


and:

Code:
hash_read_ahead = 5
hash_interval = 5
hash_max_tries = 5
max_open_sockets = 8212
might help
 

simur612

Member
May 25, 2018
879
0
16
Thanks for that. Just as an update, I let the 58 torrents run over night and woke up to the same problem, but with some hash issues too.

I've added what you suggested, I'll update if it makes a difference! smiley.gif

Also, I saw what I think was a 'power' button in a screenshot of your rutorrent - is a plugin available? It would save me having to ssh into my server to restart / restart rtorrent.

Regards
 

simur612

Member
May 25, 2018
879
0
16
Hello,

I'm looking at this feature too. I have a few doubts.

1. Since you said it's from a pool, how far and how do we limit this pool? What if this pool just gets too big and consumes like 50% of the CPU.
2. I noticed that my .rtorrent.rc has something like this,

Quote

#This s is an example resource file for rTorrent. Copy to
# ~/.rtorrent.rc and enable/modify the options as needed. Remember to
# uncomment the options you wish to enable.

# Global
max_downloads_global = 10
max_uploads_global = 100​


So doesn't the above options limit the concurrent downloads and uploads?

3. Also, I noticed ruTorrent have the above lines as well. So once again, what exactly is the above line for?
 

randac56

Member
May 25, 2018
915
0
16
Hello,

I'm looking at this feature too. I have a few doubts.

1. Since you said it's from a pool, how far and how do we limit this pool? What if this pool just gets too big and consumes like 50% of the CPU.
2. I noticed that my .rtorrent.rc has something like this,

Quote
#This s is an example resource file for rTorrent. Copy to​
# ~/.rtorrent.rc and enable/modify the options as needed. Remember to​
# uncomment the options you wish to enable.​
# Global​
max_downloads_global = 10​
max_uploads_global = 100​


So doesn't the above options limit the concurrent downloads and uploads?

3. Also, I noticed ruTorrent have the above lines as well. So once again, what exactly is the above line for?
you can doubt all you want.
It won't change the facts.

what you need to understand is this:

many other torrent programs for linux:

*spawn a new process for each torrent
*have settings on a per torrent basis (each torrent may be able to make x connections)
*have no way for each process to "see" the other processes


rtorrent doesn't work like this. The problem with the other method is:

inactive torrents end up hurting you in many ways
all torrents are treated the same
hugely inefficient


this is the REASON x number of torrent limits exist in the first place. a perfect example is torrentflux, each torrent in torrentflux is it's own stand alone program
it doesn;t know what the other is doing
it adds up quick

rtorrent handles this problem by:


using POOLS

you can set a total bandwidth for the entire pool
you can set a total max number of connections for the entire pool


let's say your globals are:

1 MB/s up/down
100 total connections

with rtorrent you could run 1 really fast torrent

which might make 40 -50 connections but hit the peak of 1 MB/s

or you might run 100 torrents which will split this up among themselves. It will never go over these globals, it will effectively use very similar amounts of cpu and bandwidth all other things being equal. you need to adjust your thought process when it comes to rtorrent.

I had a box with 15 torrentflux users, we had to limit them all to 4-5 total torrents to keep it stable.

Switching to rutorrent allowed me to put no limit on number of torrents and just put limits on total number of connections each client can make (and i limited thier total upspeed)

by doing this, the job of management is shifted to the user. They decide if they want to run 3-4 torrents fast, or whatever.

Another thing this allows you to do is seed for MUCH LONGER because often, seeding takes up much less bandwidth and memory. rtorrent will intelligently poll the trackers and switch torrents from active to inactive depending on settings and availablility. It's amazing.
 

peshua19

Member
May 25, 2018
897
0
16
I am with you wonslung. rtorrent gives the best performance by length.

Just had a little laugh at you first reply to me. My question on "server" and config were meant to be for the OP and his stuff to see the settings and if i could identify a possible problem or to give an educated guess besides just spending time on giving general recommendations.

btw - currently running 198 torrents in my pool. When i check my status bar i do have
128/128 open files
15/66 used upload slots
1/610 download slots
0/32 http requests
1/35/768 handshake/open/max sockets

rtorrent manages pretty well which files to keep open based on necessity. The slots are handled depending on your bandwidth limits. Never have seen a memory problem (besides a bad RAM causing hash-failures) in 2 years with the machine running nearly 24/7 all year long. Max torrents in the pool were around 300 once but i had to clean out a little since the 2 TB drive for sharing has been full and i couldn't download anymore. But i also do have more RAM inside. On the other hand i just checked and only about 120MB are being used by rtorrent currently.
 

simur612

Member
May 25, 2018
879
0
16
Thank you for the explaination wonslug. That makes thing clearer for me.

Now I've another question which still makes me doesn't understand. I've changed ruTorrent's Max Download slots to 1 but reading from rTorrent's screen, I still see [D 0/10]. Why doesn't it change? Is this supposed to be like this?

I think this is a bug.... Once I changed the 'Speed Limit', depending whether it is Upload or Download, then only [D 0/1] or [U 0/50] takes efffect. Are you facing this problem wonslug?
 

peshua19

Member
May 25, 2018
897
0
16
When you change the speed limit, rtorrent will calculate how many slots it can use.

I am not sure how the rutorrent setting translates to rtorrent.
Anyhow the rtorrent settings for throttles are:

upload_rate = KB, download_rate = KB (Set the maximum global upload and download rates)
min_peers = value, max_peers = value (Set the minimum and maximum number of peers to allow in each download)
min_peers_seed = value, max_peers_seed = value (Set the minimum and maximum number of peers to allow while seeding, or -1 (default) to use max_peers)
max_uploads = value (Set the maximum number of simultaneous uploads per download)
max_uploads_div = value, max_downloads_div = value (Change the divider used to calculate the max upload and download slots to use when the throttle is changed. Disable by setting 0)
max_uploads_global = value, max_downloads_global = value (Max upload and download slots allowed. Disable by setting 0)​
 

simur612

Member
May 25, 2018
879
0
16
The speed calculation + update on the max_download_slot part well, not too sure, but some people reported it as bug at libtorrent.rhakasa.no.