Okay, it’s time for me to run a real BitTorrent system, not the mickey-mouse stuff that’s good for one or two torrents. I want to host hundreds, and then thousands of files that are reliably accessible over BitTorrent. There must be a way to do it, what is it?
The files are all MP3s, they’re the library of Morning Coffee Notes podcasts going back to June 2004. The process of creating a Torrent file has to be completely automatic. The seeding must take place on the server, also automatically, and there are hundreds of files, so it’s not possible to open each file in a BitTorrent client. They must use almost no resources unless someone is actually accessing it, and then it must be fully seeded by the server.
January 10, 2006 at 8:18 am
i wish i had time to help
all i can say is go dave go!
happy new year from Vancouver!
…Roland
January 10, 2006 at 8:43 am
What you need to do is host a tracker. A tracker allows refers downloaders to each other. Download the bittorrent source. The README file contains specific instructions on setting up a tracker. The documentation also has some info.
http://www.bittorrent.com/documentation.html
January 10, 2006 at 8:47 am
I wonder if there’s a sort of headless mode in Azureus. I’ve successfully seeded hundreds of files at a time (never tried thousands), and what Azureus seems to get bogged down on is updating its UI. I say this because turning down the update frequency on the UI (there’s a setting for it) speeds it up immensely.
I believe Azureus will even create an RSS feed for you.
January 10, 2006 at 8:52 am
Seems like a simple Perl script can be put together that does what you need.
See,
http://search.cpan.org/~orclev/Net-BitTorrent-File-1.02-fix/lib/Net/BitTorrent/File.pm
January 10, 2006 at 8:56 am
all the best for your future plannes and all the best for your wonderful movements in your aim ……..
January 10, 2006 at 9:06 am
You might also want to take a look at Broadcast Machine. You can see an example over at TWiTorrents.
January 10, 2006 at 9:10 am
Before are any more tutorials posted here, I know how BitTorrent works, I understand what trackers are, and what seeding is. I asked the questions carefully. Some people must be doing this seriously, if not, there’s some missing software, and we should figure out how to make it.
January 10, 2006 at 9:12 am
Michael, sounds like you have an answer.
I use Azureus, but I wasn’t aware that it would perform reasonably with hundreds of files. That’s a good place to start.
But I was kind of hoping for something written in C.
January 10, 2006 at 9:15 am
Broadcast Machine is not what I’m looking for. It requires that the user seed the files. It’s just a tracker. That’s the problem with BitTorrent, if the file is not always being downloaded it eventually falls off the net. I tried to make it clear by saying “reliably accessible.” Anything that requires the author to be the seed is not reliable.
January 10, 2006 at 9:16 am
It’s time for Apple to bake in support for bittorrent (or a clone) into iTunes. Until that happens, media casters of all kinds will have to sell out to some extent when they get popular. Libsyn and my service are nice low-cost hosting options for now, but they aren’t sustainable in the long term (I’m banking on the bittorrent thing becoming real sometime in the next 2 yrs). I’m guessing Podshow will buy Libsyn before then, but that is just speculation.
That is great that you will offer your MCN archives and other stuff this way–a great fit for your techie audience. I think you’ll probably end up seeding all the time because there is less demand for older shows. But then again I don’t know your numbers
January 10, 2006 at 9:27 am
Roger, my current project is integrating a podcatcher with the NewsRiver aggregator. Obviously, I don’t feel we need to have Apple do anything to make it real. I didn’t need their help to make podcasting real. ;->
About seeding, I want to seed all the time. That’s the point. I’m seeding right now. All the MP3s are available on my servers right now. They still will be after they’re converted to BT.
January 10, 2006 at 9:28 am
Check out Prodigem! It has everything you need!!
January 10, 2006 at 9:33 am
Dave, quick, check out http://www.macrumorslive.com/web/ they’re doing a chat style live-blogging of Steve’s MacWorld address.
January 10, 2006 at 11:06 am
Dave,
You want Osprey from ibiblio. You can see it in action at torrent.ibiblio.org. We’re handling a lot of torrents and the seeding is persistant, the metadata is easily searchable and distributable.
Enjoy,
P
Free and Open of course
January 10, 2006 at 11:06 am
Osprey http://osprey.ibiblio.org could be your answer. It has a web application for search/browse, a Bittorrent tracker, and Permaseed for seeding 100’s or 1000’s of torrents. Next on the project to-do list is a middleware layer to automate file upload->torrent file generation->seeding.
January 10, 2006 at 12:32 pm
Roger Strickland said,
January 10, 2006 at 9:16 am
“It’s time for Apple to bake in support for bittorrent (or a clone) into iTunes. ”
Apple Computer has had a bittorrent clone since the beginning. It is called Akamai.
Apple is Akamai’s biggest customer.
January 10, 2006 at 1:11 pm
Dave, I think I’ve emailed you a couple times over the last year about Prodigem. Check it out, its exactly what you need. It even has an API, but otherwise will happily create, seed and manage your torrents for you.
http://www.prodigem.com/torrents/
January 10, 2006 at 4:09 pm
Gary, is Prodigem something you run on your server, like Apache or is it a service that they provide that you use, like AOL?
January 10, 2006 at 6:11 pm
It’s a web service, written in PHP. The API is REST based (docs here – http://torrentocracy.com/mediawiki/index.php/Prodigem_API). Feel free to sign up for a free preview account to try it out. It comes with 100megs of storage capacity.
January 10, 2006 at 10:57 pm
Maybe this is similar to ideas that Carl Franklin discussed on DNR. I believe PWOP’s plan is to send out free boxes they will remotely manage to be seeds to people who will host them on their networks.
I think he talks about it on this episode:
http://www.dotnetrocks.com/default.aspx?showID=160
January 11, 2006 at 6:36 am
>Apple Computer has had a bittorrent clone since the beginning. It is called Akamai.
>Apple is Akamai’s biggest customer.
Jim – yes, but the Akamai mojo doesn’t kick in if you use iTunes to subscribe to a podcast using Advanced -> Subscribe to Podcast. So the “indie” podcaster ends up paying for all the bandwidth themselves.
And do you really think the free lunch will continue when there are over 100 million people listening to podcasts? I know what companies like Apple pay for premium bandwidth like Akamai’s, and I can tell you it isn’t sustainable forever. Apple will cave and put ads in the shows or charge for them, unless they a)bake in a p2p bittorrent clone or b)make so much money off their ITMS store they do the podcasting thing as a marketing gimmick.
I’d love to keep up with this thread, but it appears wordpress.com won’t let me subscribe to a comment thread tied to an individual post. That would be a nice feature.
January 11, 2006 at 7:52 am
“I’d love to keep up with this thread, but it appears wordpress.com won’t let me subscribe to a comment thread tied to an individual post. That would be a nice feature.”
Sure it will.
http://scripting.wordpress.com/2006/01/10/industrial-strength-bittorrent/feed would appear to be what you’re looking for. Just slap “/feed/” at the end of any URL.
Gary
http://www.garyslinger.com
January 11, 2006 at 9:28 am
Osprey is a download, Prodigem is a service. There is also the Digital Bicycle project which presumably has a BitTorrent tracker for the Drupal CMS. I *had* a blogging plugin for Azureus using metaWeblog API to upload, and Radio loved it, but the radio servers never accepted the .torrent extension at the time, so I sort of gave up. I hear it doesn’t work too well with the new Azureus. I’ve been working on other things, though…
January 11, 2006 at 10:12 am
Looking at this problem a bit more, the new Azureus has an RSS feed. There are also plugins to download from RSS feeds. Have users submit the RSS feed of what they’re doing, and have your Azureus “server” subscribe to all of them. Or, wrap up their items, and point Azureus to that new rollup.
So, your “server” that reads the RSS feed its supposed to pay attention to, permanently seeding everything. Azureus can also run headless.
January 11, 2006 at 11:00 am
Thom, that sounds very interesting. Where are you finding this information? Are there docs somewhre?
I found when I looked at the docs for Osprey, it used all kinds of terminology that I’m only vaguely aware of. RSS, of course, I totally understand.
January 11, 2006 at 12:53 pm
From the “core plugins” page the RSS publish feature was added:
0.10: RSS feed generation support; ability to run web on separate port from tracker; ability to specify categories to publish
– http://azureus.aelitis.com/wiki/index.php/Azureus_Core_Plugins
And there are *several* flavors of plugins that automatically download .torrent files that you can find on the main page.
The options for the RSS feed server are in the same place as the built-in webserver and tracker. Google around for how to run Azureus headless once you’ve got everything set up.
I’m really trying to look ahead in my mind to any problems… this is definately a permanent-seed option as you just subscribe Azureus to a feed of the .torrent files you want to seed, and just keep it going. You’ll want to look at the rules, and configure Azureus to either always seed files forever, or some other option (like only seed when there is no other seed.)
January 11, 2006 at 3:40 pm
Thom, we’ve drifted from where I must start this project:
1. I already have an RSS feed, but it doesn’t point to torrents, it points to MP3s.
http://www.morningcoffeenotes.com/rss.xml
2. I need a tool that automatically creates torrents and hosts them and maintains a permanent seed for each of the files.
3. Then I need to teach my aggregator how to catch these.
I need to do it in that order. #2 is where I am right now.
I’ve installed the RSSFeed plugin in Azureus, which I am familiar with, and it would be very useful if it could read a feed with enclosures if it has torrents as enclosures or the original objects as enclosures, and if it created a folder of torrents that I could monitor and generate the appropriate RSS when it changes. If I had that I’d be well on my way.
If this doesn’t make sense, ask questions. Thom is the only one who has been somewhat tracking the problem I’m trying to solve. People who recommend hosted solutions aren’t bothering to read the problem statement.
January 11, 2006 at 3:50 pm
What Azureus does could be very useful if we could communicate with it directly from newsRiver.root. It might be enough to send it an XML-RPC message saying “subscribe to this feed and put the resulting files in this folder.” Then we periodically check the folder, and when a new file appears, pass it through the synching code.
Also, if we’re going to build on it, we need to generate some end-user docs for the installation process and a simple explanation of what it does. Much of the information that’s needed is there, but I imgagine that a non-technical user wouldn’t be able to make sense of it.
January 11, 2006 at 7:29 pm
Well, reading this here… Azureus definately has an exposed Java API, but remember that BitTorrent, at core, is just “scripting” … The thing I’ve had a problem with is “watching” a directory. *nix has system-level calls for directory updates, but this is weird on Windows…. you could have a cron job of course that trys to look for changes.
But essentially, it would be a simple script to always make .torrent files from those files that exist in other directory. I made a drag & drop .torrent program for Windows.
Ironically, a Google employee wrote pyTivo, but the best I can gather is that you’re looking for the reverse! My Azureus plugin would attatch any .torrent you opened as an enclosure for a MetaWeblog API post…. perhaps I should dust it off and get it working again!
January 11, 2006 at 7:40 pm
Osprey is open source software for download that supports a service. You can add to it, change it, etc.
is a working implementation of that service offering a wide variety of stuff via bittorrent.
January 11, 2006 at 7:41 pm
make that http://torrent.ibiblio.org/ is a working implementation of that service…
January 11, 2006 at 8:14 pm
For creating the torrents, I would go all the way back to the beginning… to the original command-line BT software that comes with a script called maketorrent.py. A Frontier script could parse the RSS, call maketorrent.py for each MP3, and regenerate the new RSS file.
For seeding, permaseed from the Osprey project looks nice. Or maybe you could use the command-line launchmany-console.py that comes with BitTorrent.
January 12, 2006 at 3:13 am
Hey, you may want to check this out:
http://www.phpclasses.org/browse/package/2459.html
It’s a php class that will let you control a bittorrent client. It may be useful in what you’re trying to do.
Joel
January 12, 2006 at 10:52 am
Well, Azureus can create an RSS feed of what it’s currently doing, with links to the .torrent files. Currently, you have mp3 in RSS. Ideally, a tool that watched *that* feed and just made .torrent files of the MP3s would be great. I bet it would be pretty easy to munge the PyTivo code with the BitTorrent python script, and there you have .torrent files for each MP3.
Azureus can “watch” a folder for .torrent files. If you drop the .torrent file there, and put the MP3 in the default Azureus download folder, then it should pick that up and start seeding.
A work flow, and using folders to juggle around duplicates would be:
1. Obtain the MP3 file
… if its on another server, download from the feed using either XSLT or something that can find the new entries in the RSS, and download those files… if it’s on the same server, well, you already have it.
2. Create the .torrent file
Run the maketorrent.py script on each new file
3. Start seeding
Move the .mp3 file to the download folder of Azureus, and move the .torrent to folder that Azureus watches for new files. If these are one-in-the-same, that’s hip too.
I don’t know what your specific plans for automation are, or how much of this you could do directly. Can you regurgitate the RSS feed that Azureus makes into another place? Is this a requirement? You’d need some sort of re-blogging script, eh?
Are you wanting to come up with something that is easy for end-users? The simplest way to go in developing a full fledged BitTorrent implementation would be to tie into the Python scripts.
Well, I think these are the sorts of questions that will deal with the real nitty-gritty. I bet you could normalize and boil this all down into a lot of XSLT, sed and grep, and general file juggling, or whatever Frontier can do.
January 15, 2006 at 6:02 pm
Dave, per the first 2 of your needs, if you plug your already existant rss feed into your settings on Prodigem, it will automatically torrent each enclosure it sees. Here’s an image of what the settings switch looks like in your account: http://prodigem.com/torrents/images/autotorrent.jpg
Prodigem then also has its own user rss feeds which would represent the torrented translation of your original.
March 12, 2006 at 10:54 pm
Check out http://www.blogtorrent.com/
“Blog Torrent is free, open-source software from Downhill Battle that provides an easy way to share large files on your website without large storage or bandwidth costs.”